Learn OpenCV

Seamless Cloning using OpenCV ( Python , C++ )

Avatar Satya Mallick
March 2, 2015 45 Comments
how-to OpenCV 3

By 45 Comments

Seamless Cloning Example
Figure 1 : Seamless Cloning Example : An airplane cloned into the picture of an evening sky.

One of the exciting new features introduced in OpenCV 3 is called Seamless Cloning. With this new feature you can copy an object from one image, and paste it into another image making a composition that looks seamless and natural.  The above image was created using a scene of a sky and that of an airplane.  If I had simply overlaid the airplane image on top of the sky image, the result would look ridiculous (See Figure 2).

Sky with plane without seamless cloning.
Figure 2 : Sky with plane overlaid.

Now of course nobody in their right mind would do a composition like that. You would obviously mask the image out carefully, and perhaps after spending half a day in Photoshop get an image that looks like Figure 3.

Plane overlaid on sky with mask
Figure 3 : Sky image with airplane overlaid with careful masking.
I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI! Sign up now and take your skills to the next level!

If you are an artist you would spend another half day and carefully adjust the lighting on the airplane to the lighting of the sky image and create a beautiful composition.

There are two problems however. First, you don’t have half a day to spend. Second, you are probably not an artist!

Wouldn’t it be cool if you could make a very rough mask around the airplane, and yet create a beautiful composition that looks like Figure 1 ? What if you did it in just 10 lines of code ? Now that won’t just be cool, it would be seriously badass!

Before I show you the code, let me spend an hour explaining the exciting theory behind seamless cloning. Oh wait, I got a better idea. Let’s dive into the code first.

Seamless Cloning Example

A quick look at the usage first
Python 

output = cv2.seamlessClone(src, dst, mask, center, flags)

C++ 

seamlessClone(Mat src, Mat dst, Mat mask, Point center, Mat output, int flags)
src Source image that will be cloned into the destination image. In our example it is the airplane.
dst Destination image into which the source image will be cloned. In our example it is the sky image.
mask A rough mask around the object you want to clone. This should be the size of the source image. Set it to an all white image if you are lazy!
center Location of the center of the source image in the destination image.
flags The two flags that currently work are NORMAL_CLONE and MIXED_CLONE. I have included an example to show the difference.
output Output / result image.

Now let’s look at the code that I used to generate the images above.

Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE!

Python Example 

# Standard imports
import cv2
import numpy as np 

# Read images
src = cv2.imread("images/airplane.jpg")
dst = cv2.imread("images/sky.jpg")

# Create a rough mask around the airplane.
src_mask = np.zeros(src.shape, src.dtype)
poly = np.array([ [4,80], [30,54], [151,63], [254,37], [298,90], [272,134], [43,122] ], np.int32)
cv2.fillPoly(src_mask, [poly], (255, 255, 255))

# This is where the CENTER of the airplane will be placed
center = (800,100)

# Clone seamlessly.
output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)

# Save result
cv2.imwrite("images/opencv-seamless-cloning-example.jpg", output);

C++ Example 

using namespace cv;

// Read images : src image will be cloned into dst
Mat src = imread("images/airplane.jpg");
Mat dst = imread("images/sky.jpg");

// Create a rough mask around the airplane.
Mat src_mask = Mat::zeros(src.rows, src.cols, src.depth());

// Define the mask as a closed polygon
Point poly[1][7];
poly[0][0] = Point(4, 80);
poly[0][1] = Point(30, 54);
poly[0][2] = Point(151,63);
poly[0][3] = Point(254,37);
poly[0][4] = Point(298,90);
poly[0][5] = Point(272,134);
poly[0][6] = Point(43,122);

const Point* polygons[1] = { poly[0] };
int num_points[] = { 7 };

// Create mask by filling the polygon

fillPoly(src_mask, polygons, num_points, 1, Scalar(255,255,255));

// The location of the center of the src in the dst
Point center(800,100);

// Seamlessly clone src into dst and put the results in output
Mat output;
seamlessClone(src, dst, src_mask, center, output, NORMAL_CLONE);

// Save result
imwrite("images/opencv-seamless-cloning-example.jpg", output);

In the above example, the cloning type ( flags ) I used was NORMAL_CLONE. There is another type, MIXED_CLONE, that is subtly different from NORMAL_CLONE. Let’s see how the two types differ in detail.

Normal Cloning ( NORMAL_CLONE ) versus Mixed Cloning ( MIXED_CLONE )

I have a 5 year old son who gives me “I Love You” tickets if I treat him well. There was a time when kids yearned for their parent’s approval, but these days parents have to do their best to earn “I Love You” tickets. Anyway, back to cloning. I am going to use one of these “I Love You Tickets” in the service of Computer Vision (See Figure 4 ).

An "I Love You" Ticket.
Figure 4: An “I Love You” Ticket.

Let’s try to clone this image onto a wooden texture shown in Figure 5. We will be lazy and use a source mask that is all white, and clone the source image right in the center of the wooden texture image.

wood-texture
FIgure 5 : Wood Texture

Python Example 

import cv2
import numpy as np

# Read images : src image will be cloned into dst
im = cv2.imread("images/wood-texture.jpg")
obj= cv2.imread("images/iloveyouticket.jpg")

# Create an all white mask
mask = 255 * np.ones(obj.shape, obj.dtype)

# The location of the center of the src in the dst
width, height, channels = im.shape
center = (height/2, width/2)

# Seamlessly clone src into dst and put the results in output
normal_clone = cv2.seamlessClone(obj, im, mask, center, cv2.NORMAL_CLONE)
mixed_clone = cv2.seamlessClone(obj, im, mask, center, cv2.MIXED_CLONE)

# Write results
cv2.imwrite("images/opencv-normal-clone-example.jpg", normal_clone)
cv2.imwrite("images/opencv-mixed-clone-example.jpg", mixed_clone)

C++ Example 

using namespace cv;
Mat src = imread("images/iloveyouticket.jpg");
Mat dst = imread("images/wood-texture.jpg");

// Create an all white mask
Mat src_mask = 255 * Mat::ones(src.rows, src.cols, src.depth());

// The location of the center of the src in the dst
Point center(dst.cols/2,dst.rows/2);

// Seamlessly clone src into dst and put the results in output
Mat normal_clone;
Mat mixed_clone;

seamlessClone(src, dst, src_mask, center, normal_clone, NORMAL_CLONE);
seamlessClone(src, dst, src_mask, center, mixed_clone, MIXED_CLONE);

// Save results
imwrite("images/opencv-normal-clone-example.jpg", normal_clone);
imwrite("images/opencv-mixed-clone-example.jpg", mixed_clone);

Normal Cloning Result

If we use Normal Cloning by using the NORMAL_CLONE flag, we will get the result shown in Figure 6. Now we did not use a good mask and you can see excessive smoothing between the words “I” and “Love”, and between “you” and “Paa”. Sure we were lazy. We could have created a rough mask and improved the result.  But if you are lazy and smart, you would use Mixed Cloning.

OpenCV Normal Clone Example
Figure 6 : OpenCV Normal Clone Example

Mixed Cloning Result

In Normal Cloning the texture ( gradient ) of the source image is preserved in the cloned region. In Mixed Cloning, the texture ( gradient ) of the cloned region is determined by a combination of the source and the destination images. Mixed Cloning does not produce smooth regions because it picks the dominant texture ( gradient ) between the source and destination images. The result of mixed cloning is shown in Figure 7. Notice the texture is no longer smooth between “I” and “Love”, and “you” and “Paa”. Lazy people cheer!

OpenCV Mixed Cloning Example
Figure 7: OpenCV Mixed Cloning Example

Seamless Cloning Video Result

I took the images of the airplane and the sky, and changed the position of the airplane to create this animation. MIXED_CLONE gave better results, and you hardly notice any artifacts. Cloning a 300×194 image (airplane) onto a 1000×560 image ( sky) takes approximately 0.4 seconds.

Poisson Image Editing

Seamless cloning in OpenCV is an implementation of an influential SIGGRAPH 2003 paper titled “Poisson Image Editing”, by Patrick Perez, Michel Gangnet, and Andrew Blake.

Now we know that if we blend the intensities ( RGB values ) of the source image ( the airplane ) with the destination image ( sky ) using a carefully created mask we will obtain a result like Figure 3. The central insight in the paper is that working with image gradients instead of image intensities can produce much more realistic results. After seamless cloning the intensity of the result image in the masked region is NOT the same as the intensity of the source region in the masked region. Instead, the gradient of the result image in the masked region is about the same as the gradient of the source region in the masked region. Additionally, the intensity of the result image at the boundary of the masked region is the same as the intensity of the destination image (sky).

The authors show that this is done by solving a Poisson equation, and hence the title of the paper — Poisson Image Editing. The theoretical and implementation details of the paper are actually very cool, but beyond the scope of this post. However, if you read the paper and have questions, feel free to ask them in the comments section.



Image Credits

  • Sky Image : Licensed under the Creative Commons Attribution 2.0 Generic license. Attribution: epSos.de . Click here for details.
  • Airplane image : Public domain. Click here for details.

Comments

  2. Hello,

    This is trully exciting new OpenCV feature to learn. Thank you for sharing it. I wonder if this can be successfully applied to copy an image, let us say, of a woman’s head for example with messed hair correctly to an other image (I mean if the segmentation behind it will work perfectly -precisely- well) ?

    Begueradj.

    Reply

  3. Hello again,
    I just run your code and I had this error:

    OpenCV Error: Assertion failed (minxd >= 0 && minyd >= 0 && maxxd <= dest.rows && maxyd <= dest.cols) in seamlessClone, file /root/opencv-3.0.0-beta/modules/photo/src/seamless_cloning.cpp, line 96
    Traceback (most recent call last):
    File "cloner.py", line 19, in
    output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
    cv2.error: /root/opencv-3.0.0-beta/modules/photo/src/seamless_cloning.cpp:96: error: (-215) minxd >= 0 && minyd >= 0 && maxxd <= dest.rows && maxyd <= dest.cols in function seamlessClone

    How to fix it ?
    Regards,
    Begueradj

    Reply

    • OpenCV 3 is till in beta and therefore buggy. You need to make sure your source image is entirely inside the destination image. In other words, make sure the points ( center.x + src.width / 2, center.y + src.height / 2 ) and ( center.x – src.width / 2, center.y – src.height / 2 ) are inside the destination image ( i.e. within the rectangle ( 0, 0, dst.width, dst.height). Hope this helps.

      Reply

      • Thank you for the answer. I resolved the problem by: sudo apt-get install ocl-icd-opencl-dev . This is what worked for me. However I found other different solutions on internet. Only this one worked for me. Thank you. Begueradj.

        Reply

  4. Hello,I met a problem when I try to merge two image which come from the same human portrait, but one of it`s skin region is modified( darken or brighten), because my skin mask have many holes , i want to use seamlessClone to filter the holes. The result is disappointing. Can the algorithm do that? I brighten or darken the skin region in RGB(3 channel add the same offset), so there is no differece in gradient, how should I tune my method?

    Reply

  11. I’m having this error :

    output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
    AttributeError: ‘module’ object has no attribute ‘seamlessClone’

    Reply

  13. facing error in blob detection… in real time blob is not detected but when I use picture as a reference it detect blob .. kindly help me that how to detect blob in real time video

    Reply

  15. Can u help me to convert from opencv to OpenCV3?
    I can’t get size of 64×64
    ball_original = cv.LoadImage(“Aqua-Red_Ball.png”)
    ball = cv.CreateImage((64,64), ball_original.depth, ball_original.channels)
    cv.Resize(ball_original, ball)

    In OpenCV 3.1.0:
    ball_original = cv.2.imread(“Aqua-Red_Ball.png”)
    ball_img = ball_original.copy()
    ball = ball_img.shape
    ball = np.zeros((ball[0], ball[1]), np.uint8)
    cv2.resize(ball_original, (ball[0], ball[1]))

    Reply

  17. Hey, thank you for your wonderful post! really helping me a lot.
    I know it’s been a while you post this, but I really have to ask.
    By the way, When I try to change im=cv2.imread() to my own image, it give me error :

    normal_clone = cv2.seamlessClone(obj, im, mask, center, cv2.NORMAL_CLONE)
    error: C:buildmaster_winpack-bindings-win32-vc14-staticopencvmodulescoresrcmatrix.cpp:433: error: (-215) u != 0 in function cv::Mat::create

    my image has 300DPI, 4288×2848, in case you need it.
    Can you teach me why? thanks!

    Reply

  19. Hi, i am new to python and i am learning from your articles, those are excellent. How to write the code for moving airplane and how to save video file from the images can you please tell me.

    Thanks in advance

    Reply

  20. Hey! Satya! Thank you for the post! I really excited about the function. However,
    I got “Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)” when using seamlessClone. Could you give me some help about it? Thanks a lot!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Check out my latest course launching on Kickstarter
OpenCV For Beginners

Subscribe to receive the download link, receive updates, and be notified of bug fixes

Almost there! Please complete this form and click the button below to receive the download link.

  • We hate SPAM and promise to keep your email address safe.
Subscribe To Receive
  • C++ And Python Examples
  • Installation Packages
  • Resources Guide
  • Newsletter Insights
  • 20% Off OpenCV Courses
We hate SPAM and promise to keep your email address safe.​
  • We hate SPAM and promise to keep your email address safe.