fbpx

LearnOpenCV

Homography examples using OpenCV ( Python / C ++ )

Satya Mallick
January 3, 2016 48 Comments
Application Classical Computer Vision Image Alignment OpenCV OpenCV Tutorials Theory

By 48 Comments

Terms like “Homography” often remind me how we still struggle with communication. Homography is a simple concept with a weird name! In this post we will discuss Homography examples using OpenCV.

The Tower of Babel, according to a mythical tale in the Bible, was humans’ first engineering disaster. The project had all the great qualities of having a clear mission, lots of man power, no time constraint and adequate technology ( bricks and mortar ). Yet it failed spectacularly because God confused the language of the human workers and they could not communicate any longer.

What is Homography ?

Consider two images of a plane (top of the book) shown in Figure 1. The red dot represents the same physical point in the two images. In computer vision jargon we call these corresponding points. Figure 1. shows four corresponding points in four different colors — red, green, yellow and orange. A Homography is a transformation ( a 3×3 matrix ) that maps the points in one image to the corresponding points in the other image.

Homography examples using OpenCV. Image marked with four corresponding points in four different colors in both the images.
Figure 1 : Two images of a 3D plane ( top of the book ) are related by a Homography

Now since a homography is a 3×3 matrix we can write it as

H = \left[ \begin {array} {ccc} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & h_{22} \end{array} \right]

Let us consider the first set of corresponding points — (x_1,y_1) in the first image and (x_2,y_2)} in the second image. Then, the Homography H maps them in the following way

\left[ \begin{array}{c} x_1 \\ y_1 \\ 1 \end{array} \right]&= H \left[ \begin{array}{c} x_2 \\ y_2 \\ 1 \end{array} \right]&= \left[ \begin{array}{ccc} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & h_{22} \end{array} \right] \left[ \begin{array}{c}x_2 \\ y_2 \\1 \end{array} \right]

Homography examples using OpenCV – Image Alignment

The above equation is true for ALL sets of corresponding points as long as they lie on the same plane in the real world. In other words you can apply the homography to the first image and the book in the first image will get aligned with the book in the second image! See Figure 2.

Alignment of one image with another from same plane using homograpphy.
Figure 2 : One image of a 3D plane can be aligned with another image of the same plane using Homography

But what about points that are not on the plane ? Well, they will NOT be aligned by a homography as you can see in Figure 2. But wait, what if there are two planes in the image ? Well, then you have two homographies — one for each plane.

I’ve partnered exclusively with OpenCV.org to bring you official courses in AI, Computer Vision, and Deep Learning to take you on a structured path from first steps to mastery.

Homography examples using OpenCV – Panorama

In the previous section, we learned that if a homography between two images is known, we can warp one image onto the other. However, there was one big caveat. The images had to contain a plane ( the top of a book ), and only the planar part was aligned properly. It turns out that if you take a picture of any scene ( not just a plane ) and then take a second picture by rotating the camera, the two images are related by a homography!

In other words you can mount your camera on a tripod and take a picture. Next, pan it about the vertical axis and take another picture. The two images you just took of a completely arbitrary 3D scene are related by a homography. The two images will share some common regions that can be aligned and stitched and bingo you have a panorama of two images. Is it really that easy ? Nope! (sorry to disappoint) A lot more goes into creating a good panorama, but the basic principle is to align using a homography and stitch intelligently so that you do not see the seams. Creating panoramas will definitely be part of a future post.

How to calculate a Homography ?

To calculate a homography between two images, you need to know at least 4 point correspondences between the two images. If you have more than 4 corresponding points, it is even better. OpenCV will robustly estimate a homography that best fits all corresponding points. Usually, these point correspondences are found automatically by matching features like SIFT or SURF between the images, but in this post we are simply going to click the points by hand.

Let’s look at the usage first.

C++

// pts_src and pts_dst are vectors of points in source 
// and destination images. They are of type vector<Point2f>. 
// We need at least 4 corresponding points.

Mat h = findHomography(pts_src, pts_dst);

// The calculated homography can be used to warp
// the source image to destination. im_src and im_dst are
// of type Mat. Size is the size (width,height) of im_dst.
warpPerspective(im_src, im_dst, h, size);

Python

'''
pts_src and pts_dst are numpy arrays of points
in source and destination images. We need at least
4 corresponding points.
'''
h, status = cv2.findHomography(pts_src, pts_dst)

'''
The calculated homography can be used to warp
the source image to destination. Size is the
size (width,height) of im_dst
'''

im_dst = cv2.warpPerspective(im_src, h, size)

Let us look at a more complete example in both C++ and Python.

Homography examples using OpenCV C++

Images in Figure 2. can be generated using the following C++ code. The code below shows how to take four corresponding points in two images and warp image onto the other.

Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE! 

#include "opencv2/opencv.hpp"

using namespace cv;
using namespace std;

int main( int argc, char** argv)
{
    // Read source image.
    Mat im_src = imread("book2.jpg");
    // Four corners of the book in source image
    vector<Point2f> pts_src;
    pts_src.push_back(Point2f(141, 131));
    pts_src.push_back(Point2f(480, 159));
    pts_src.push_back(Point2f(493, 630));
    pts_src.push_back(Point2f(64, 601));

    // Read destination image.
    Mat im_dst = imread("book1.jpg");
    // Four corners of the book in destination image.
    vector<Point2f> pts_dst;
    pts_dst.push_back(Point2f(318, 256));
    pts_dst.push_back(Point2f(534, 372));
    pts_dst.push_back(Point2f(316, 670));
    pts_dst.push_back(Point2f(73, 473));

    // Calculate Homography
    Mat h = findHomography(pts_src, pts_dst);

    // Output image
    Mat im_out;
    // Warp source image to destination based on homography
    warpPerspective(im_src, im_out, h, im_dst.size());

    // Display images
    imshow("Source Image", im_src);
    imshow("Destination Image", im_dst);
    imshow("Warped Source Image", im_out);

    waitKey(0);
}

Homography examples using OpenCV Python

Images in Figure 2. can also be generated using the following Python code. The code below shows how to take four corresponding points in two images and warp image onto the other.

#!/usr/bin/env python

import cv2
import numpy as np

if __name__ == '__main__' :

    # Read source image.
    im_src = cv2.imread('book2.jpg')
    # Four corners of the book in source image
    pts_src = np.array([[141, 131], [480, 159], [493, 630],[64, 601]])

    # Read destination image.
    im_dst = cv2.imread('book1.jpg')
    # Four corners of the book in destination image.
    pts_dst = np.array([[318, 256],[534, 372],[316, 670],[73, 473]])

    # Calculate Homography
    h, status = cv2.findHomography(pts_src, pts_dst)

    # Warp source image to destination based on homography
    im_out = cv2.warpPerspective(im_src, h, (im_dst.shape[1],im_dst.shape[0]))

    # Display images
    cv2.imshow("Source Image", im_src)
    cv2.imshow("Destination Image", im_dst)
    cv2.imshow("Warped Source Image", im_out)

    cv2.waitKey(0)

Applications of Homography

The most interesting application of Homography is undoubtedly making panoramas ( a.k.a image mosaicing and image stitching ). Panoramas will be the subject of a later post. Let us see some other interesting applications.

Perspective Correction using Homography

Example base image for explaining perspective correction.
Figure 3. Perspective Correction

Let’s say you have a photo shown in Figure 1. Wouldn’t it be cool if you could click on the four corners of the book and quickly get an image that looks like the one shown in Figure 3. You can get the code for this example in the download section below. Here are the steps.

  1. Write a user interface to collect four corners of the book. Let’s call these points pts_src
  2. We need to know the aspect ratio of the book. For this book, the aspect ratio ( width / height ) is 3/4. So we can choose the output image size to be 300×400, and our destination points ( pts_dst ) to be (0,0), (299,0), (299,399) and (0,399)
  3. Obtain the homography using pts_src and pts_dst .
  4. Apply the homography to the source image to obtain the image in Figure 3.

Virtual Billboard

In many televised sports events, advertisement in virtually inserted in live video feed. E.g. in soccer and baseball the ads placed on small advertisement boards right outside the boundary of the field can be virtually changed. Instead of displaying the same ad to everybody, advertisers can choose which ads to show based on the person’s demographics, location etc.

In these applications the four corners of the advertisement board are detected in the video which serve as the destination points. The four corners of the ad serve as the source points. A homography is calculated based on these four corresponding points and it is used to warp the ad into the video frame.

After reading this post you probably have an idea on how to put an image on a virtual billboard. Figure 4. shows the first image uploaded to the internet.

Shot of Les Horribles Cernettes-first image to be uploaded on the internet.
Figure 4. First image uploaded to the internet.

And Figure 5. shows The Times Square.

Figure 5. The Times Square, Image Source

We can replace one of the billboards on The Times Square with the image of our choice. Here are the steps.

  1. Write a user interface to collect the four corners of the billboard in the image. Let’s call these points pts_dst
  2. Let the size of the image you want to put on the virtual billboard be w x h. The corners of the image ( pts_src ) are therefore to be (0,0), (w-1,0), (w-1,h-1) and (0,h-1)
  3. Obtain the homography using pts_src and pts_dst .
  4. Apply the homography to the source image and blend it with the destination image to obtain the image in Figure 6.

Notice in Figure 6. we have inserted image shown in Figure 4. into The Times Square Image.

Times square processed image showing a replaced image on one of the left side billboards.
Figure 6. Virtual Billboard. One of the billboards on the left hand side has been replaced with an image of our choice.


Image Credits

  1. The image in Figure 4. was the first photographic image uploaded to the internet. It qualifies as fair use. [ link ]
  2. The image used in Figure 5. ( The Time Square ) is licensed under the GFDL. [ link ]

Comments

  1. Cool! You remembered about the stitching! Haha. I really like your blog. Thank you

    BTW..I’ve developed my own algorithm for stitching and blending images 😀

    Reply

    • It is my favorite computer vision book. One of my minor regrets is that I turned down an offer to do an internship with Dr. Sing Bing Kang and Dr. Szeliski when I was a grad student. I used that summer to start a company instead :).

      Reply

  3. Is there a difference between using `findHomography` and `getPerspectiveTransform` when using four points to do a perspective transformation as in these examples?

    Reply

    • I don’t think there is any difference when you use exactly four points. But if you use more than 4 points you can choose different methods in case of findHomography ( e.g. CV_RANSAC ).

      Reply

  4. Hi. Thanks for the information. Is there any good order to see the all posts of this blog?
    I’m a kind of newbie, trying to learn computer vision (I’ve just understanded MLP) Am I allowed to know how the people do image net comp’, while I learn computer vision?

    Reply

    • Hi Yoon, Thanks for the kind words. This blog is not organized like a book but is a collection of articles. However, this year we will come up with some form of organized content for beginners.

      Reply

  5. Hi Satya, I would like to thank you for this sample, and i appreciate the way to explain homography.
    But i have one remark to add. The warped image is equal to the transformation of the source image to the destination by the homography matrix h. So to obtain that in C++, we should use the inverse of the matrix h in warpPerspective() function

    – warpPerspective(im_src, im_out, h.inv(), im_dst.size()); => transform source to destination
    – warpPerspective(im_dst, im_out, h, im_dst.size()); => transform destination to source

    Thank you again.

    Reply

  6. Hi sathya, when i am using the Homography and wrapPerspective for a video , the output video has a lot of flickering according to the camera angle. I think this technique is used for only static camera. Any ideas to eliminate the flickering. Really appreciate it.

    Thanks

    Reply

  7. Hi Satya,
    I used the above technique on the first image and aligned it with the second one. The aligned image is shown in the 3rd image with black areas. I need to compare the two images and show the difference (bitwise_xor or sub). One issue is part of the image is missing as it has negative coordinates after rotation. But if I put an offset to bring it into visible area, it will no longer be aligned. So what’s the best way to compare and show the difference? Any pointers?

    Reply

    • Actually you do not need a homography for the problem you have shown. You can simply use an affine transform ( check out getAffineTransform and warpAffine in OpenCV ). Hope that helps.

      Reply

  8. Hi Sathya , I would like to use Homography for a moving camera . Are the results going to be good ? How can I tell which movement is caused due to the camera movement? Thanks in advance !

    Reply

  9. Dear All,

    I am a beginner in computer vision. I am working on a dataset in which due to camera inclined position, there is perspective effect. I calculate optical flow, however, due to perspective distortion, optical flow is not correct. I attach figure here. Please see that optical flow for people near to camera is correct, whereas, people far from camera, optical flow is not calculated. I assume that this is due to the perspective distortion.
    Please suggest me that can I apply homography method explain in this post? Secondly do I need to apply this process before calculating optical flow?

    Many thanks in advance,
    Umer

    Reply

  10. I’ve got a problem to solve that I’m not sure if homography could help me.
    I have a mesh https://dl.dropboxusercontent.com/u/710615/tst_msh.jpg , which was created from this image : https://dl.dropboxusercontent.com/u/710615/FinalPCBwithFrame_.jpg .
    I need to fit and overlay the mesh on https://dl.dropboxusercontent.com/u/710615/Layer1.png .
    The goal is get something like this http://i.stack.imgur.com/fyixS.jpg.

    I’ve found the https://dl.dropboxusercontent.com/u/710615/3.jpg of the “source image” .
    I think that I need to remove a column of white pixels from the upper left corner of the mesh (I don’t know yet how to do it easily).
    Besides The mesh image and the original image from the mesh are equal. However, the image of layer has some similar holes but not equal, which is acceptable.

    Reply

  11. Hi Sir,
    Can Homography transform be used to correct the projector distortion when projected on planar-tilted surfaces

    Reply

      • Sir, can there be any homography between world plane and corresponding image Plane. If yes, then would be the coordinates in world dimension, will they be like (0, 0) (0, w) (w, h) (h, 0) where w and h are width and height of rectangular plane in real dimensions(mm/cm). Wouldn’t there be any inconsistency since image coordinates are in pixels.

        Reply

  14. This blog is incredible. I barely have a background in highschool lin
    algebra but can understand and implement the stuff you explain with
    ease.

    Reply

  15. Hi Satya, is it possible using homography (or another method) to form an image of say the front of a person from two images taken at +45deg and -45 deg of that person?

    Reply

    • the H Matrix is (matlab-like syntax):
      [a b c; d e f; g h 1]
      the sub-matrix [a b ; d e] can be interpreted as rotation matrix.
      We can interpret those values as:
      a,e = cos(t)
      b = -sin(t)
      c=sin(t)

      Reply

    • Usually those points are calculated using SFIT, but inorder to calculate those manually you may use ImageJ software.

      Reply

  18. Hi Satya,

    I would like to use homography to correct a distorted image. I would like the point A, B, C, D of the first image to correspond with the point A, B, C, D of the second image (square).

    I tried your code but, it resulted in a stronger distortion…

    Here are my code, could you tell me if I did something wrong.

    Thanks a lot for your help.

    import cv2
    import numpy as np
    import matplotlib.pyplot as plt

    if __name__ == ‘__main__’ :

    # Read source image.
    im_src = cv2.imread(‘points.jpg’, cv2.IMREAD_COLOR)
    # Four points of the miniR image
    pts_src = np.array([[744,255],[856,279],[1000,667],[926,741]], dtype=float)

    # Read destination image.
    im_dst = cv2.imread(‘rectangle.jpg’, cv2.IMREAD_COLOR)
    # Four points of the square
    pts_dst = np.array([[200,200],[1000,200],[1000,1000],[200,1000]], dtype=float)

    # Calculate Homography
    h, status = cv2.findHomography(pts_src, pts_dst)

    # Warp source image to destination based on homography
    im_out = cv2.warpPerspective(im_src, h, (im_dst.shape[1],im_dst.shape[0]))

    cv2.imwrite(‘corrected.jpg’, im_out)

    # Display images
    cv2.imshow(“Source Image”, im_src)
    cv2.imshow(“Destination Image”, im_dst)
    cv2.imshow(“Warped Source Image”, im_out)

    cv2.waitKey(0)

    #show plot with the coordinate
    plt.imshow(im_out, cmap=’gray’, interpolation=’bicubic’)

    https://uploads.disquscdn.com/images/f6ed7f2c2c4a2eb7da4420d178079cb452a3e142393d028e1cd1c66ec7a69ae6.jpg

    https://uploads.disquscdn.com/images/35ad70917ab6347e201722f3119e08d5a69acd379d5fd2b8940a39a9ecaac442.jpg

    https://uploads.disquscdn.com/images/4e169cdc266db535db20389b2eb8f5220cd9f692c801d4a950116c2a312c2cb5.jpg

    Reply

  19. Hello Satya,
    Is there any way to reduce the opacity of the im_dst image in the im_out image, so that I can see both im_src and im_dst images in im_out image? I tried adding an alpha channel to im_dst image just before using cv2.warpPerspective(), but all I got was a darkened im_dst image in im_out image! How to see both the images at the same time?

    Reply

    • Use cv2.addWeighted(im_dst, dst_opacity, im_out, out_opacity, 1), where dst_opacity and out_opacity are the weights for each image—you can use 0.5 for both for a start.

      Reply

  20. Just a small correction here.

    When an homography transforms pixel locations, it transforms them to homogenous coordinates, but they may be scaled by some scaling factor; you must divide by the scaling factor to get back to the correct coordinates in your image. So in your example, it should be [s*x’, s*y’, s] = H * [x, y, 1], and a division by s would give you the points [x’, y’, 1].

    Reply

  23. Hello Satya, thanks for your blog.
    I have trained a deep homography model based on the paper–[Deep image homography estimiation](https://arxiv.org/pdf/1606.03798.pdf)

    Results looks great,

    avg sqrt L2 loss of training set(500032 images from image net) : 0.8967
    avg sqrt L2 loss of test set(10000 images from image net) : 0.8448
    avg delta of test set(how many pixels different with the real delta) : 2.48

    Problem is I do not know how to convert the H4 matrix to homography matrix, do you know how to do it?

    You can find the details at fast ai forum–http://forums.fast.ai/t/how-to-transform-4-points-parameter-matrix-to-homography-matrix/5770

    Thanks

    Reply

  25. I have a small question. Generally Homography matrix is a camera projection matrix when the 3d scene lies on a 2d plane(for example on z==0). In that case, we can use the camera projection equations to find this H matrix. But how did we come up with this equation x’ = H*x. Can you give an explanation?

    Reply

  27. Great blog Satya ! just wondering if we can avoid knowing aspect ratio of the book. We know the book is rectangle may be something could be done with two vanishing points.

    Reply

  28. Is there a way to compute homography matrix without using matches points ?
    For example , can i compute homography matrix using Euler angles , position (x,y) , accelaration and speed ?

    Reply

  29. Thanks for this great tutorial !

    But can someone explain to me what the h11…h33 exactly are? And how they change while moving the image ?

    Reply

  30. Very Interesting!
    I had 2 quick questions :
    1] Can I use this to correct foreshortening any image (not a polygon) and how ?

    2] If the same image were at 15 deg to the left or right can i get the image at 0 deg ?

    Reply

  31. Hi satya,
    Thanks for wonderful article, My understanding about homography is that by using intrinsic and extrinsic camera calibration parameter , homography matrix calculated.

    so how findhomography function calculate camera matrix just by passing 8 pointes?

    Reply

Leave a Reply

Your email address will not be published.

This course is available for FREE only till 22nd Nov.
FREE Python Course
We have designed this Python course in collaboration with OpenCV.org for you to build a strong foundation in the essential elements of Python, Jupyter, NumPy and Matplotlib.
  • We hate SPAM and promise to keep your email address safe.
FREE OpenCV Crash Course
We have designed this FREE crash course in collaboration with OpenCV.org to help you take your first steps into the fascinating world of Artificial Intelligence and Computer Vision. The course will be delivered straight into your mailbox.
  • We hate SPAM and promise to keep your email address safe.
Subscribe To Receive
  • C++ And Python Examples
  • Installation Packages
  • Resources Guide
  • Newsletter Insights
  • 20% Off OpenCV Courses
We hate SPAM and promise to keep your email address safe.​
  • We hate SPAM and promise to keep your email address safe.

Subscribe to receive the download link, receive updates, and be notified of bug fixes

seperator

Almost there! Please complete this form and click the button below to receive the download link.

  • We hate SPAM and promise to keep your email address safe.
Subscribe To Receive
  • C++ And Python Examples
  • Installation Packages
  • Resources Guide
  • Newsletter Insights
  • 20% Off OpenCV Courses
We hate SPAM and promise to keep your email address safe.​
  • We hate SPAM and promise to keep your email address safe.
Subscribe Now
Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

Getting Started
Course
Information
About LearnOpenCV

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products.

Read More
Copyright © 2022 – BIG VISION LLC