• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

Warp one triangle to another using OpenCV ( C++ / Python )

Satya Mallick
May 18, 2016 6 Comments
how-to Tutorial

May 18, 2016 By 6 Comments

Warp Triangle using OpenCV
Figure 1 : All pixels inside the blue triangle in the left image have been transformed to the blue triangle in the right image.

In this tutorial we will see how to warp a single triangle in an image to another triangle in a different image.

In computer graphics people deal with warping triangles all the time because any 3D surface can approximated by triangles. Images can be broken down into triangles and warped. However, in OpenCV there is no out of the box method that warps pixels inside a triangle to pixels inside another triangle.

This tutorial will explain step by step how to transform the triangle in the left image in Figure 1 to the right image.

Before we dive into code we need to understand what an affine transform is.

What is an Affine Transform ?

An Affine Transform is the simplest way to transform a set of 3 points ( i.e. a triangle ) to another set of arbitrary 3 points. It encodes translation ( move ), scale, rotation and shear. The image below illustrates how an affine transform can be used to change the shape of a square. Note that using an affine transform you can change the shape of a square to a parallelogram at any orientation and scale. However, the affine transform is not flexible enough to transform a square to an arbitrary quadrilateral. In other words, after an affine transform parallel lines continue to be parallel.

OpenCV Motion Models

In OpenCV an affine transform is a 2×3 matrix. The first two columns of this matrix encode rotation, scale and shear, and the last column encodes translation ( i.e. shift ).

    \[ S= \begin{bmatrix} a & b & t_x \\ c & d & t_y \end{bmatrix} \]

Given a point ( x, y), the above affine transform, moves it to point ( x_t, y_t ) using the equation given below

    \[ \begin{bmatrix} x_t \\ y_t \end{bmatrix} =\begin{bmatrix} a & b \\ c & d \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix}+\begin{bmatrix} t_x \\ t_y \end{bmatrix} \]

Triangle Warping using OpenCV

To understand the description below better, download the C++ and Python code and images by subscribing to our newsletter here.

We now know that to warp a triangle to another triangle we will need to use the affine transform. In OpenCV, warpAffine allows you to apply an affine transform to an image, but not a triangular region inside the image. To overcome this limitation we find a bounding box around the source triangle and crop the rectangular region from the source image. We then apply an affine transform to the cropped image to obtain the output image. The previous step is crucial because it allows us to apply the affine transform to a small region of the image thus improving computational performance. Finally, we create a triangular mask by filling pixels inside the output triangle with white. This mask when multiplied with the output image turns all pixels outside the triangle black while preserving the color of all pixels inside the triangle.

Before we go into the details, let us read in input and output images, and define input and output triangles. For this tutorial our output image is just white, but you could read in another image if you wish.
C++

// Read input image and convert to float
Mat img1 = imread("robot.jpg");
img1.convertTo(img1, CV_32FC3, 1/255.0);

// Output image is set to white
Mat imgOut = Mat::ones(imgIn.size(), imgIn.type());
imgOut = Scalar(1.0,1.0,1.0);

// Input triangle
vector <Point2f> tri1;
tri1.push_back(Point2f(360,200));
tri1.push_back(Point2d(60,250));
tri1.push_back(Point2f(450,400));
    
// Output triangle
vector <Point2f> triOut;
tri2.push_back(Point2f(400,200));
tri2.push_back(Point2f(160,270));
tri2.push_back(Point2f(400,400));

Python

# Read input image and convert to float
img1 = cv2.imread("robot.jpg")

# Output image is set to white
img2 = 255 * np.ones(img_in.shape, dtype = img_in.dtype)

# Define input and output triangles 
tri1 = np.float32([[[360,200], [60,250], [450,400]]])
tri2 = np.float32([[[400,200], [160,270], [400,400]]])

Our inputs and outputs are now defined and we are ready to go through the steps needed to transform all pixels inside the input triangle to output triangle.

  1. Calculate the bounding boxes
    In this step we calculate bounding boxes around triangles. The idea is to warp only a small part of the image and not the entire image for efficiency.

    C++

    // Find bounding rectangle for each triangle
    Rect r1 = boundingRect(tri1);
    Rect r2 = boundingRect(tri2);
    

    Python

     
    # Find bounding box. 
    r1 = cv2.boundingRect(tri1)
    r2 = cv2.boundingRect(tri2)
    
    
  2. Crop images & change coordinates
    To efficiently apply affine transform to a piece of the image and not the entire image, we crop the input image based on the bounding box calculated in the previous step. The coordinates of the triangles also need to be modified so as to reflect their location in the new cropped images. This is done by subtracting the x and y coordinates of the top left corner of the bounding box from the x and y coordinates of the triangle.

    C++

    // Offset points by left top corner of the respective rectangles
    vector<Point2f> tri1Cropped, tri2Cropped;
    vector<Point> tri2CroppedInt;
    
    for(int i = 0; i < 3; i++)
    {
        tri1Cropped.push_back( Point2f( tri1[i].x - r1.x, tri1[i].y -  r1.y) );
        tri2Cropped.push_back( Point2f( tri2[i].x - r2.x, tri2[i].y - r2.y) );
    
        // fillConvexPoly needs a vector of Point and not Point2f
        tri2CroppedInt.push_back( Point((int)(tri2[i].x - r2.x), (int)(tri2[i].y - r2.y)) );
    }
    
    // Apply warpImage to small rectangular patches
    Mat img1Cropped;
    img1(r1).copyTo(img1Cropped);
    

    Python

     
    # Offset points by left top corner of the 
    # respective rectangles
    
    tri1Cropped = []
    tri2Cropped = []
        
    for i in xrange(0, 3):
      tri1Cropped.append(((tri1[0][i][0] - r1[0]),(tri1[0][i][1] - r1[1])))
      tri2Cropped.append(((tri2[0][i][0] - r2[0]),(tri2[0][i][1] - r2[1])))
    
    # Apply warpImage to small rectangular patches
    img1Cropped = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]
    
  3. Estimate the affine transform : We have just obtained the coordinates of input and output triangles in the cropped input and output images. Using these two triangles we can find the affine transform that will transform the input triangle to the output triangle in the cropped images using the following code.

    C++

    // Given a pair of triangles, find the affine transform.
    Mat warpMat = getAffineTransform( tri1Cropped, tri2Cropped );
    

    Python

     
    # Given a pair of triangles, find the affine transform.
    warpMat = cv2.getAffineTransform( np.float32(tri1Cropped), np.float32(tri2Cropped) )
    
  4. Warp pixels inside bounding box
    The affine transform found in the previous step is applied to the cropped input image to obtain the cropped output image. In OpenCV you can apply an affine transform to an image using warpAffine.

    C++

     
    // Apply the Affine Transform just found to the src image
    Mat img2Cropped = Mat::zeros(r2.height, r2.width, img1Cropped.type());
    warpAffine( img1Cropped, img2Cropped, warpMat, img2Cropped.size(), INTER_LINEAR, BORDER_REFLECT_101);
    

    Python

    # Apply the Affine Transform just found to the src image
    img2Cropped = cv2.warpAffine( img1Cropped, warpMat, (r2[2], r2[3]), None, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101 )
    
  5. Mask pixels outside the triangle
    In the previous step we obtained the output rectangular image. However, we are interested in a triangle inside the rectangular region. So we create a mask using fillConvexPoly that is used to black out all pixels outside the triangle. This new cropped image can finally be put in the right location in the output image using top left corner of the output bounding rectangle.

    C++

    // Get mask by filling triangle
    Mat mask = Mat::zeros(r2.height, r2.width, CV_32FC3);
    fillConvexPoly(mask, tri2CroppedInt, Scalar(1.0, 1.0, 1.0), 16, 0);
        
    // Copy triangular region of the rectangular patch to the output image
    multiply(img2Cropped,mask, img2Cropped);
    multiply(img2(r2), Scalar(1.0,1.0,1.0) - mask, img2(r2));
    img2(r2) = img2(r2) + img2Cropped;
    

    Python

    # Get mask by filling triangle
    mask = np.zeros((r2[3], r2[2], 3), dtype = np.float32)
    cv2.fillConvexPoly(mask, np.int32(tri2Cropped), (1.0, 1.0, 1.0), 16, 0);
    
    # Apply mask to cropped region
    img2Cropped = img2Cropped * mask
    
    # Copy triangular region of the rectangular patch to the output image
    img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] * ( (1.0, 1.0, 1.0) - mask )
        
    img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] + img2Cropped
    
    

This brings us to the end of this tutorial. I hope it was a good learning experience. I recommend you download the code and give it a try.

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource guide. In our newsletter we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Image Credits

The robot image is in public domain. Many thanks to artist bamenny.

Tags: affine transform fillConvexPoly triangle warping warpAffine

Filed Under: how-to, Tutorial

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • RAFT: Optical Flow estimation using Deep Learning
  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy