One of the exciting new features introduced in OpenCV 3 is called Seamless Cloning. With this new feature you can copy an object from one image, and paste it into another image making a composition that looks seamless and natural. The above image was created using a scene of a sky and that of an airplane. If I had simply overlaid the airplane image on top of the sky image, the result would look ridiculous (See Figure 2).
Now of course nobody in their right mind would do a composition like that. You would obviously mask the image out carefully, and perhaps after spending half a day in Photoshop get an image that looks like Figure 3.
If you are an artist you would spend another half day and carefully adjust the lighting on the airplane to the lighting of the sky image and create a beautiful composition.
There are two problems however. First, you don’t have half a day to spend. Second, you are probably not an artist!
Wouldn’t it be cool if you could make a very rough mask around the airplane, and yet create a beautiful composition that looks like Figure 1 ? What if you did it in just 10 lines of code ? Now that won’t just be cool, it would be seriously badass!
Before I show you the code, let me spend an hour explaining the exciting theory behind seamless cloning. Oh wait, I got a better idea. Let’s dive into the code first.
Seamless Cloning Example
A quick look at the usage first
Python
output = cv2.seamlessClone(src, dst, mask, center, flags)
C++
seamlessClone(Mat src, Mat dst, Mat mask, Point center, Mat output, int flags)
| src | Source image that will be cloned into the destination image. In our example it is the airplane. |
| dst | Destination image into which the source image will be cloned. In our example it is the sky image. |
| mask | A rough mask around the object you want to clone. This should be the size of the source image. Set it to an all white image if you are lazy! |
| center | Location of the center of the source image in the destination image. |
| flags | The two flags that currently work are NORMAL_CLONE and MIXED_CLONE. I have included an example to show the difference. |
| output | Output / result image. |
Now let’s look at the code that I used to generate the images above.
Python Example
# Standard imports
import cv2
import numpy as np
# Read images
src = cv2.imread("images/airplane.jpg")
dst = cv2.imread("images/sky.jpg")
# Create a rough mask around the airplane.
src_mask = np.zeros(src.shape, src.dtype)
poly = np.array([ [4,80], [30,54], [151,63], [254,37], [298,90], [272,134], [43,122] ], np.int32)
cv2.fillPoly(src_mask, [poly], (255, 255, 255))
# This is where the CENTER of the airplane will be placed
center = (800,100)
# Clone seamlessly.
output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
# Save result
cv2.imwrite("images/opencv-seamless-cloning-example.jpg", output);
C++ Example
using namespace cv;
// Read images : src image will be cloned into dst
Mat src = imread("images/airplane.jpg");
Mat dst = imread("images/sky.jpg");
// Create a rough mask around the airplane.
Mat src_mask = Mat::zeros(src.rows, src.cols, src.depth());
// Define the mask as a closed polygon
Point poly[1][7];
poly[0][0] = Point(4, 80);
poly[0][1] = Point(30, 54);
poly[0][2] = Point(151,63);
poly[0][3] = Point(254,37);
poly[0][4] = Point(298,90);
poly[0][5] = Point(272,134);
poly[0][6] = Point(43,122);
const Point* polygons[1] = { poly[0] };
int num_points[] = { 7 };
// Create mask by filling the polygon
fillPoly(src_mask, polygons, num_points, 1, Scalar(255,255,255));
// The location of the center of the src in the dst
Point center(800,100);
// Seamlessly clone src into dst and put the results in output
Mat output;
seamlessClone(src, dst, src_mask, center, output, NORMAL_CLONE);
// Save result
imwrite("images/opencv-seamless-cloning-example.jpg", output);
In the above example, the cloning type ( flags ) I used was NORMAL_CLONE. There is another type, MIXED_CLONE, that is subtly different from NORMAL_CLONE. Let’s see how the two types differ in detail.
Normal Cloning ( NORMAL_CLONE ) versus Mixed Cloning ( MIXED_CLONE )
I have a 5 year old son who gives me “I Love You” tickets if I treat him well. There was a time when kids yearned for their parent’s approval, but these days parents have to do their best to earn “I Love You” tickets. Anyway, back to cloning. I am going to use one of these “I Love You Tickets” in the service of Computer Vision (See Figure 4 ).
Let’s try to clone this image onto a wooden texture shown in Figure 5. We will be lazy and use a source mask that is all white, and clone the source image right in the center of the wooden texture image.
Python Example
import cv2
import numpy as np
# Read images : src image will be cloned into dst
im = cv2.imread("images/wood-texture.jpg")
obj= cv2.imread("images/iloveyouticket.jpg")
# Create an all white mask
mask = 255 * np.ones(obj.shape, obj.dtype)
# The location of the center of the src in the dst
width, height, channels = im.shape
center = (height/2, width/2)
# Seamlessly clone src into dst and put the results in output
normal_clone = cv2.seamlessClone(obj, im, mask, center, cv2.NORMAL_CLONE)
mixed_clone = cv2.seamlessClone(obj, im, mask, center, cv2.MIXED_CLONE)
# Write results
cv2.imwrite("images/opencv-normal-clone-example.jpg", normal_clone)
cv2.imwrite("images/opencv-mixed-clone-example.jpg", mixed_clone)
C++ Example
using namespace cv;
Mat src = imread("images/iloveyouticket.jpg");
Mat dst = imread("images/wood-texture.jpg");
// Create an all white mask
Mat src_mask = 255 * Mat::ones(src.rows, src.cols, src.depth());
// The location of the center of the src in the dst
Point center(dst.cols/2,dst.rows/2);
// Seamlessly clone src into dst and put the results in output
Mat normal_clone;
Mat mixed_clone;
seamlessClone(src, dst, src_mask, center, normal_clone, NORMAL_CLONE);
seamlessClone(src, dst, src_mask, center, mixed_clone, MIXED_CLONE);
// Save results
imwrite("images/opencv-normal-clone-example.jpg", normal_clone);
imwrite("images/opencv-mixed-clone-example.jpg", mixed_clone);
Normal Cloning Result
If we use Normal Cloning by using the NORMAL_CLONE flag, we will get the result shown in Figure 6. Now we did not use a good mask and you can see excessive smoothing between the words “I” and “Love”, and between “you” and “Paa”. Sure we were lazy. We could have created a rough mask and improved the result. But if you are lazy and smart, you would use Mixed Cloning.
Mixed Cloning Result
In Normal Cloning the texture ( gradient ) of the source image is preserved in the cloned region. In Mixed Cloning, the texture ( gradient ) of the cloned region is determined by a combination of the source and the destination images. Mixed Cloning does not produce smooth regions because it picks the dominant texture ( gradient ) between the source and destination images. The result of mixed cloning is shown in Figure 7. Notice the texture is no longer smooth between “I” and “Love”, and “you” and “Paa”. Lazy people cheer!
Seamless Cloning Video Result
I took the images of the airplane and the sky, and changed the position of the airplane to create this animation. MIXED_CLONE gave better results, and you hardly notice any artifacts. Cloning a 300×194 image (airplane) onto a 1000×560 image ( sky) takes approximately 0.4 seconds.
Poisson Image Editing
Seamless cloning in OpenCV is an implementation of an influential SIGGRAPH 2003 paper titled “Poisson Image Editing”, by Patrick Perez, Michel Gangnet, and Andrew Blake.
Now we know that if we blend the intensities ( RGB values ) of the source image ( the airplane ) with the destination image ( sky ) using a carefully created mask we will obtain a result like Figure 3. The central insight in the paper is that working with image gradients instead of image intensities can produce much more realistic results. After seamless cloning the intensity of the result image in the masked region is NOT the same as the intensity of the source region in the masked region. Instead, the gradient of the result image in the masked region is about the same as the gradient of the source region in the masked region. Additionally, the intensity of the result image at the boundary of the masked region is the same as the intensity of the destination image (sky).
The authors show that this is done by solving a Poisson equation, and hence the title of the paper — Poisson Image Editing. The theoretical and implementation details of the paper are actually very cool, but beyond the scope of this post. However, if you read the paper and have questions, feel free to ask them in the comments section.
Subscribe & Download Code
If you liked this article and would like to download code (C++ and Python) and example images used in this post, please click here. Alternately, sign up to receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.Image Credits
This is really cool! (but not enough of a reason to upgrade to OpenCV 3 for me)
True OpenCV 3 is still in beta with its share of bugs. I am just exploring new features. Here is another one you may find interesting — https://www.learnopencv.com/non-photorealistic-rendering-using-opencv-python-c/
Hello,
This is trully exciting new OpenCV feature to learn. Thank you for sharing it. I wonder if this can be successfully applied to copy an image, let us say, of a woman’s head for example with messed hair correctly to an other image (I mean if the segmentation behind it will work perfectly -precisely- well) ?
Begueradj.
Yes it will work just fine. I will actually add another section to this post where I show such examples.
I’m really looking forward to see that. Thank you very much in advance.
No problem. I usually send out updates to the mailing list. You can join the mailing list by scrolling up to the bottom of the post. Alternatively check back in a couple of weeks.
Hello again,
I just run your code and I had this error:
OpenCV Error: Assertion failed (minxd >= 0 && minyd >= 0 && maxxd <= dest.rows && maxyd <= dest.cols) in seamlessClone, file /root/opencv-3.0.0-beta/modules/photo/src/seamless_cloning.cpp, line 96
Traceback (most recent call last):
File "cloner.py", line 19, in
output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
cv2.error: /root/opencv-3.0.0-beta/modules/photo/src/seamless_cloning.cpp:96: error: (-215) minxd >= 0 && minyd >= 0 && maxxd <= dest.rows && maxyd <= dest.cols in function seamlessClone
How to fix it ?
Regards,
Begueradj
OpenCV 3 is till in beta and therefore buggy. You need to make sure your source image is entirely inside the destination image. In other words, make sure the points ( center.x + src.width / 2, center.y + src.height / 2 ) and ( center.x – src.width / 2, center.y – src.height / 2 ) are inside the destination image ( i.e. within the rectangle ( 0, 0, dst.width, dst.height). Hope this helps.
Thank you for the answer. I resolved the problem by: sudo apt-get install ocl-icd-opencl-dev . This is what worked for me. However I found other different solutions on internet. Only this one worked for me. Thank you. Begueradj.
Hello,I met a problem when I try to merge two image which come from the same human portrait, but one of it`s skin region is modified( darken or brighten), because my skin mask have many holes , i want to use seamlessClone to filter the holes. The result is disappointing. Can the algorithm do that? I brighten or darken the skin region in RGB(3 channel add the same offset), so there is no differece in gradient, how should I tune my method?
Can you post the images and the mask ?
thank you so much! it was very god for me! say hello to your sun! thank’s again!
Thanks!
When I run your code, I get these results. Do you know why?
Updated OpenCV. Problem solved.
Thanks for letting me know. I was scratching my head to understand what was going on.
what opencv version in this tutorial ?
It is for OpenCV 3
i using your python code got
AttributeError: ‘NoneType’ object has no attribute ‘shape’
Error
Are you sure you are using OpenCV 3 ? What is the output when you type the following on the command line.
python -c "import cv2; print cv2.__version__"
Check ur filename in folder
airplane.jpg?true=
could be airplane.jpg
Hei, run into the same error (possibly).
In my case, it’s because that I run the code in a different dictionary, so python failed load image via `imread`.
Hope it helps.
OpenCV 3.1.0 That is latest version.
The OpenCV 3.0.0 containing buggy
can i use opencv to make like this http://video.soundtooth.cn/play?url=http%3A%2F%2Fv.youku.com%2Fv_show%2Fid_XMzk4NDAzNTQ0.html%3Ffrom%3Ds1.8-1-1.2
how to tracking object in video ?? have any tutorial ?
Not yet, but I can write one.
OK, you get a ticket for this. I love you Satya.
🙂
I’m having this error :
output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
AttributeError: ‘module’ object has no attribute ‘seamlessClone’
You need OpenCV 3 for this.
that not problem
check ur filename in folder
airplane.jpg?true=
could be airplane.jpg
sir i need help?
facing error in blob detection… in real time blob is not detected but when I use picture as a reference it detect blob .. kindly help me that how to detect blob in real time video
why would it return me only backgroud image and won’t add other one on it.? I am using javaCV
Can u help me to convert from opencv to OpenCV3?
I can’t get size of 64×64
ball_original = cv.LoadImage(“Aqua-Red_Ball.png”)
ball = cv.CreateImage((64,64), ball_original.depth, ball_original.channels)
cv.Resize(ball_original, ball)
In OpenCV 3.1.0:
ball_original = cv.2.imread(“Aqua-Red_Ball.png”)
ball_img = ball_original.copy()
ball = ball_img.shape
ball = np.zeros((ball[0], ball[1]), np.uint8)
cv2.resize(ball_original, (ball[0], ball[1]))
Usually, OpenCV 3.1.0
How did u get airplane from right to left?
I only see one on right side.
I know how to do this.
Thanks!
Keep up good work.
Hey, thank you for your wonderful post! really helping me a lot.
I know it’s been a while you post this, but I really have to ask.
By the way, When I try to change im=cv2.imread() to my own image, it give me error :
normal_clone = cv2.seamlessClone(obj, im, mask, center, cv2.NORMAL_CLONE)
error: C:buildmaster_winpack-bindings-win32-vc14-staticopencvmodulescoresrcmatrix.cpp:433: error: (-215) u != 0 in function cv::Mat::create
my image has 300DPI, 4288×2848, in case you need it.
Can you teach me why? thanks!
always make sure, the file exist – using the os module.
https://uploads.disquscdn.com/images/26bfe0ad342c74984d5397a580427e45a6494f736a7225b788bedb70b509204d.png
I have done little different procedure, using contours points, converted them to array and then created mask image using cv2.fillpoly. My output image comes out the IMAGE 1 given in the above tutorial, it seems to blend with destination image. Am I doing something wrong, how can get IMAGE 3
Hi, i am new to python and i am learning from your articles, those are excellent. How to write the code for moving airplane and how to save video file from the images can you please tell me.
Thanks in advance
Hey! Satya! Thank you for the post! I really excited about the function. However,
I got “Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)” when using seamlessClone. Could you give me some help about it? Thanks a lot!
Can you use Artistic Style Transfer as an alternative to Seamless Cloning, especially in the application of face swapping?
why ?
opencv 3.4.1
MIXED https://uploads.disquscdn.com/images/8dfa7b8067b23dad84e81cefe6f7802ee149a50524288a5f453deaae369a6929.jpg
monochrome
https://uploads.disquscdn.com/images/d1258235febe1e74a672a1d977935af10a3621b2052a80659e9dff192f2ad006.jpg
normal
https://uploads.disquscdn.com/images/8af4804a7f261f844cff3a1ee3b83121691ab9f1780c74c633bdb9c64c8f551d.jpg
increase low sperm count how long to build up sperm sperm production process increase sperm count and motility more sperm when ejaculating how to increase seminal production vitamins to increase sperm count produce more sperm without pills sperm motility test ejaculate without an erection improve sperm taste increasing sperm count what vitamins increase sperm ejaculate harder natural ways to improve sperm count how to increase the amount you ejaculate vitamins to improve sperm quality how to improve ejaculation time how do i increase the amount i ejaculate ejaculate without erection how to improve ejaculation distance best sperm count improve sperm count morphology sperm improve low sperm count success stories sperm gain male sperm problems human sperm production rate increasing ejaculation fluid foods that increase sperm load