Warp one triangle to another using OpenCV ( C++ / Python )

In this tutorial we will see how to warp a single triangle in an image to another triangle in a different image.

In computer graphics people deal with warping triangles all the time because any 3D surface can approximated by triangles. Images can be broken down into triangles and warped. However, in OpenCV there is no out of the box method that warps pixels inside a triangle to pixels inside another triangle.

This tutorial will explain step by step how to transform the triangle in the left image in Figure 1 to the right image.

Before we dive into code we need to understand what an affine transform is.

What is an Affine Transform ?

An Affine Transform is the simplest way to transform a set of 3 points ( i.e. a triangle ) to another set of arbitrary 3 points. It encodes translation ( move ), scale, rotation and shear. The image below illustrates how an affine transform can be used to change the shape of a square. Note that using an affine transform you can change the shape of a square to a parallelogram at any orientation and scale. However, the affine transform is not flexible enough to transform a square to an arbitrary quadrilateral. In other words, after an affine transform parallel lines continue to be parallel.

In OpenCV an affine transform is a 2×3 matrix. The first two columns of this matrix encode rotation, scale and shear, and the last column encodes translation ( i.e. shift ).

Triangle Warping using OpenCV

We now know that to warp a triangle to another triangle we will need to use the affine transform. In OpenCV, warpAffine allows you to apply an affine transform to an image, but not a triangular region inside the image. To overcome this limitation we find a bounding box around the source triangle and crop the rectangular region from the source image. We then apply an affine transform to the cropped image to obtain the output image. The previous step is crucial because it allows us to apply the affine transform to a small region of the image thus improving computational performance. Finally, we create a triangular mask by filling pixels inside the output triangle with white. This mask when multiplied with the output image turns all pixels outside the triangle black while preserving the color of all pixels inside the triangle.

Before we go into the details, let us read in input and output images, and define input and output triangles. For this tutorial our output image is just white, but you could read in another image if you wish.
C++

// Read input image and convert to float
Mat img1 = imread("robot.jpg");
img1.convertTo(img1, CV_32FC3, 1/255.0);

// Output image is set to white
Mat imgOut = Mat::ones(imgIn.size(), imgIn.type());
imgOut = Scalar(1.0,1.0,1.0);

// Input triangle
vector <Point2f> tri1;
tri1.push_back(Point2f(360,200));
tri1.push_back(Point2d(60,250));
tri1.push_back(Point2f(450,400));

// Output triangle
vector <Point2f> triOut;
tri2.push_back(Point2f(400,200));
tri2.push_back(Point2f(160,270));
tri2.push_back(Point2f(400,400));

Python

# Read input image and convert to float
img1 = cv2.imread("robot.jpg")

# Output image is set to white
img2 = 255 * np.ones(img_in.shape, dtype = img_in.dtype)

# Define input and output triangles
tri1 = np.float32([[[360,200], [60,250], [450,400]]])
tri2 = np.float32([[[400,200], [160,270], [400,400]]])

Our inputs and outputs are now defined and we are ready to go through the steps needed to transform all pixels inside the input triangle to output triangle.

Calculate the bounding boxes
In this step we calculate bounding boxes around triangles. The idea is to warp only a small part of the image and not the entire image for efficiency.

C++

// Find bounding rectangle for each triangle 
Rect r1 = boundingRect(tri1); 
Rect r2 = boundingRect(tri2);

Python

# Find bounding box. 
r1 = cv2.boundingRect(tri1) r2 = cv2.boundingRect(tri2)

Crop images & change coordinates
To efficiently apply affine transform to a piece of the image and not the entire image, we crop the input image based on the bounding box calculated in the previous step. The coordinates of the triangles also need to be modified so as to reflect their location in the new cropped images. This is done by subtracting the x and y coordinates of the top left corner of the bounding box from the x and y coordinates of the triangle.

C++

// Offset points by left top corner of the respective rectangles
vector<Point2f> tri1Cropped, tri2Cropped;
vector<Point> tri2CroppedInt;
for(int i = 0; i < 3; i++)
{
    tri1Cropped.push_back( Point2f( tri1[i].x - r1.x, tri1[i].y -  r1.y) );
    tri2Cropped.push_back( Point2f( tri2[i].x - r2.x, tri2[i].y - r2.y) );

    // fillConvexPoly needs a vector of Point and not Point2f
    tri2CroppedInt.push_back( Point((int)(tri2[i].x - r2.x), (int)(tri2[i].y - r2.y)) );
}

// Apply warpImage to small rectangular patches
Mat img1Cropped;
img1(r1).copyTo(img1Cropped);

Python

# Offset points by left top corner of the respective rectangles
tri1Cropped = []
tri2Cropped = []
    
for i in xrange(0, 3):
    tri1Cropped.append(((tri1[0][i][0] - r1[0]),(tri1[0][i][1] - r1[1])))
    tri2Cropped.append(((tri2[0][i][0] - r2[0]),(tri2[0][i][1] - r2[1])))

# Crop input image
img1Cropped = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]

Estimate the affine transform : We have just obtained the coordinates of input and output triangles in the cropped input and output images. Using these two triangles we can find the affine transform that will transform the input triangle to the output triangle in the cropped images using the following code.

C++

// Given a pair of triangles, find the affine transform. 
Mat warpMat = getAffineTransform( tri1Cropped, tri2Cropped );

Python

# Given a pair of triangles, find the affine transform.
warpMat = cv2.getAffineTransform( np.float32(tri1Cropped), np.float32(tri2Cropped) )

Warp pixels inside bounding box
The affine transform found in the previous step is applied to the cropped input image to obtain the cropped output image. In OpenCV you can apply an affine transform to an image using warpAffine.

C++

// Apply the Affine Transform just found to the src image
Mat img2Cropped = Mat::zeros(r2.height, r2.width, img1Cropped.type());
warpAffine( img1Cropped, img2Cropped, warpMat, img2Cropped.size(), INTER_LINEAR, BORDER_REFLECT_101);

Python

# Apply the Affine Transform just found to the src image
    img2Cropped = cv2.warpAffine( img1Cropped, warpMat, (r2[2], r2[3]), None, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101 )

Mask pixels outside the triangle
In the previous step we obtained the output rectangular image. However, we are interested in a triangle inside the rectangular region. So we create a mask using fillConvexPoly that is used to black out all pixels outside the triangle. This new cropped image can finally be put in the right location in the output image using top left corner of the output bounding rectangle.

C++

// Get mask by filling triangle
Mat mask = Mat::zeros(r2.height, r2.width, CV_32FC3);
fillConvexPoly(mask, tri2CroppedInt, Scalar(1.0, 1.0, 1.0), 16, 0);
    
// Copy triangular region of the rectangular patch to the output image
multiply(img2Cropped,mask, img2Cropped);
multiply(img2(r2), Scalar(1.0,1.0,1.0) - mask, img2(r2));
img2(r2) = img2(r2) + img2Cropped;

Python

# Get mask by filling triangle
mask = np.zeros((r2[3], r2[2], 3), dtype = np.float32)
cv2.fillConvexPoly(mask, np.int32(tri2Cropped), (1.0, 1.0, 1.0), 16, 0);

img2Cropped = img2Cropped * mask
    
# Copy triangular region of the rectangular patch to the output image
img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] * ( (1.0, 1.0, 1.0) - mask )
    
img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] + img2Cropped

This brings us to the end of this tutorial. I hope it was a good learning experience. I recommend you download the code and give it a try.

Image Credits

The robot image is in public domain. Many thanks to artist bamenny.