Terms like “Homography” often remind me how we still struggle with communication. Homography is a simple concept with a weird name! In this post we will discuss Homography examples using OpenCV.
The Tower of Babel, according to a mythical tale in the Bible, was humans’ first engineering disaster. The project had all the great qualities of having a clear mission, lots of man power, no time constraint and adequate technology ( bricks and mortar ). Yet it failed spectacularly because God confused the language of the human workers and they could not communicate any longer.
What is Homography ?
Consider two images of a plane (top of the book) shown in Figure 1. The red dot represents the same physical point in the two images. In computer vision jargon we call these corresponding points. Figure 1. shows four corresponding points in four different colors — red, green, yellow and orange. A Homography is a transformation ( a 3×3 matrix ) that maps the points in one image to the corresponding points in the other image.
Now since a homography is a 3×3 matrix we can write it as
![H = \left[ \begin {array} {ccc} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & h_{22} \end{array} \right]](https://learnopencv.com/wp-content/ql-cache/quicklatex.com-39c024dba489da8900f1b7adf918fca5_l3.png)
Let us consider the first set of corresponding points — 
in the first image and 
in the second image. Then, the Homography 
maps them in the following way
![\left[ \begin{array}{c} x_1 \\ y_1 \\ 1 \end{array} \right]&= H \left[ \begin{array}{c} x_2 \\ y_2 \\ 1 \end{array} \right]&= \left[ \begin{array}{ccc} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{12} \\ h_{20} & h_{21} & h_{22} \end{array} \right] \left[ \begin{array}{c}x_2 \\ y_2 \\1 \end{array} \right]](https://learnopencv.com/wp-content/ql-cache/quicklatex.com-f3524d81bb8e73f2dc24afbad99fc374_l3.png)
Homography examples using OpenCV – Image Alignment
The above equation is true for ALL sets of corresponding points as long as they lie on the same plane in the real world. In other words you can apply the homography to the first image and the book in the first image will get aligned with the book in the second image! See Figure 2.
But what about points that are not on the plane ? Well, they will NOT be aligned by a homography as you can see in Figure 2. But wait, what if there are two planes in the image ? Well, then you have two homographies — one for each plane.
Master Generative AI for CV
Homography examples using OpenCV – Panorama
In the previous section, we learned that if a homography between two images is known, we can warp one image onto the other. However, there was one big caveat. The images had to contain a plane ( the top of a book ), and only the planar part was aligned properly. It turns out that if you take a picture of any scene ( not just a plane ) and then take a second picture by rotating the camera, the two images are related by a homography!
In other words you can mount your camera on a tripod and take a picture. Next, pan it about the vertical axis and take another picture. The two images you just took of a completely arbitrary 3D scene are related by a homography. The two images will share some common regions that can be aligned and stitched and bingo you have a panorama of two images. Is it really that easy ? Nope! (sorry to disappoint) A lot more goes into creating a good panorama, but the basic principle is to align using a homography and stitch intelligently so that you do not see the seams. Creating panoramas will definitely be part of a future post.
How to calculate a Homography ?
To calculate a homography between two images, you need to know at least 4 point correspondences between the two images. If you have more than 4 corresponding points, it is even better. OpenCV will robustly estimate a homography that best fits all corresponding points. Usually, these point correspondences are found automatically by matching features like SIFT or SURF between the images, but in this post we are simply going to click the points by hand.
Let’s look at the usage first.
C++
// pts_src and pts_dst are vectors of points in source
// and destination images. They are of type vector<Point2f>.
// We need at least 4 corresponding points.
Mat h = findHomography(pts_src, pts_dst);
// The calculated homography can be used to warp
// the source image to destination. im_src and im_dst are
// of type Mat. Size is the size (width,height) of im_dst.
warpPerspective(im_src, im_dst, h, size);
Python
'''
pts_src and pts_dst are numpy arrays of points
in source and destination images. We need at least
4 corresponding points.
'''
h, status = cv2.findHomography(pts_src, pts_dst)
'''
The calculated homography can be used to warp
the source image to destination. Size is the
size (width,height) of im_dst
'''
im_dst = cv2.warpPerspective(im_src, h, size)
Let us look at a more complete example in both C++ and Python.
Homography examples using OpenCV C++
Images in Figure 2. can be generated using the following C++ code. The code below shows how to take four corresponding points in two images and warp image onto the other.
#include "opencv2/opencv.hpp"
using namespace cv;
using namespace std;
int main( int argc, char** argv)
{
// Read source image.
Mat im_src = imread("book2.jpg");
// Four corners of the book in source image
vector<Point2f> pts_src;
pts_src.push_back(Point2f(141, 131));
pts_src.push_back(Point2f(480, 159));
pts_src.push_back(Point2f(493, 630));
pts_src.push_back(Point2f(64, 601));
// Read destination image.
Mat im_dst = imread("book1.jpg");
// Four corners of the book in destination image.
vector<Point2f> pts_dst;
pts_dst.push_back(Point2f(318, 256));
pts_dst.push_back(Point2f(534, 372));
pts_dst.push_back(Point2f(316, 670));
pts_dst.push_back(Point2f(73, 473));
// Calculate Homography
Mat h = findHomography(pts_src, pts_dst);
// Output image
Mat im_out;
// Warp source image to destination based on homography
warpPerspective(im_src, im_out, h, im_dst.size());
// Display images
imshow("Source Image", im_src);
imshow("Destination Image", im_dst);
imshow("Warped Source Image", im_out);
waitKey(0);
}
Homography examples using OpenCV Python
Images in Figure 2. can also be generated using the following Python code. The code below shows how to take four corresponding points in two images and warp image onto the other.
#!/usr/bin/env python
import cv2
import numpy as np
if __name__ == '__main__' :
# Read source image.
im_src = cv2.imread('book2.jpg')
# Four corners of the book in source image
pts_src = np.array([[141, 131], [480, 159], [493, 630],[64, 601]])
# Read destination image.
im_dst = cv2.imread('book1.jpg')
# Four corners of the book in destination image.
pts_dst = np.array([[318, 256],[534, 372],[316, 670],[73, 473]])
# Calculate Homography
h, status = cv2.findHomography(pts_src, pts_dst)
# Warp source image to destination based on homography
im_out = cv2.warpPerspective(im_src, h, (im_dst.shape[1],im_dst.shape[0]))
# Display images
cv2.imshow("Source Image", im_src)
cv2.imshow("Destination Image", im_dst)
cv2.imshow("Warped Source Image", im_out)
cv2.waitKey(0)
Applications of Homography
The most interesting application of Homography is undoubtedly making panoramas ( a.k.a image mosaicing and image stitching ). Panoramas will be the subject of a later post. Let us see some other interesting applications.
Perspective Correction using Homography
Let’s say you have a photo shown in Figure 1. Wouldn’t it be cool if you could click on the four corners of the book and quickly get an image that looks like the one shown in Figure 3. You can get the code for this example in the download section below. Here are the steps.
- Write a user interface to collect four corners of the book. Let’s call these points pts_src
- We need to know the aspect ratio of the book. For this book, the aspect ratio ( width / height ) is 3/4. So we can choose the output image size to be 300×400, and our destination points ( pts_dst ) to be (0,0), (299,0), (299,399) and (0,399)
- Obtain the homography using pts_src and pts_dst .
- Apply the homography to the source image to obtain the image in Figure 3.
Virtual Billboard
In many televised sports events, advertisement in virtually inserted in live video feed. E.g. in soccer and baseball the ads placed on small advertisement boards right outside the boundary of the field can be virtually changed. Instead of displaying the same ad to everybody, advertisers can choose which ads to show based on the person’s demographics, location etc.
In these applications the four corners of the advertisement board are detected in the video which serve as the destination points. The four corners of the ad serve as the source points. A homography is calculated based on these four corresponding points and it is used to warp the ad into the video frame.
After reading this post you probably have an idea on how to put an image on a virtual billboard. Figure 4. shows the first image uploaded to the internet.
And Figure 5. shows The Times Square.
We can replace one of the billboards on The Times Square with the image of our choice. Here are the steps.
- Write a user interface to collect the four corners of the billboard in the image. Let’s call these points pts_dst
- Let the size of the image you want to put on the virtual billboard be w x h. The corners of the image ( pts_src ) are therefore to be (0,0), (w-1,0), (w-1,h-1) and (0,h-1)
- Obtain the homography using pts_src and pts_dst .
- Apply the homography to the source image and blend it with the destination image to obtain the image in Figure 6.
Notice in Figure 6. we have inserted image shown in Figure 4. into The Times Square Image.
Subscribe & Download Code
If you liked this article and would like to download code (C++ and Python) and example images used in this post, please click here. Alternately, sign up to receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.Image Credits
Cool! You remembered about the stitching! Haha. I really like your blog. Thank you
BTW..I’ve developed my own algorithm for stitching and blending images 😀
Thanks Lucas. I would love to see some results of your algorithm here.
Computer Vision – Algorithms and Applications.
An excellent reading indeed! 🙂
It is my favorite computer vision book. One of my minor regrets is that I turned down an offer to do an internship with Dr. Sing Bing Kang and Dr. Szeliski when I was a grad student. I used that summer to start a company instead :).
Is there a difference between using `findHomography` and `getPerspectiveTransform` when using four points to do a perspective transformation as in these examples?
I don’t think there is any difference when you use exactly four points. But if you use more than 4 points you can choose different methods in case of findHomography ( e.g. CV_RANSAC ).
Hi. Thanks for the information. Is there any good order to see the all posts of this blog?
I’m a kind of newbie, trying to learn computer vision (I’ve just understanded MLP) Am I allowed to know how the people do image net comp’, while I learn computer vision?
Hi Yoon, Thanks for the kind words. This blog is not organized like a book but is a collection of articles. However, this year we will come up with some form of organized content for beginners.
Hi Satya, I would like to thank you for this sample, and i appreciate the way to explain homography.
But i have one remark to add. The warped image is equal to the transformation of the source image to the destination by the homography matrix h. So to obtain that in C++, we should use the inverse of the matrix h in warpPerspective() function
– warpPerspective(im_src, im_out, h.inv(), im_dst.size()); => transform source to destination
– warpPerspective(im_dst, im_out, h, im_dst.size()); => transform destination to source
Thank you again.
Hi sathya, when i am using the Homography and wrapPerspective for a video , the output video has a lot of flickering according to the camera angle. I think this technique is used for only static camera. Any ideas to eliminate the flickering. Really appreciate it.
Thanks
Hi Karthik,
Can you show a video so I better understand the flicker issue ?
Thanks
Satya
Hi Satya,
I used the above technique on the first image and aligned it with the second one. The aligned image is shown in the 3rd image with black areas. I need to compare the two images and show the difference (bitwise_xor or sub). One issue is part of the image is missing as it has negative coordinates after rotation. But if I put an offset to bring it into visible area, it will no longer be aligned. So what’s the best way to compare and show the difference? Any pointers?
Actually you do not need a homography for the problem you have shown. You can simply use an affine transform ( check out getAffineTransform and warpAffine in OpenCV ). Hope that helps.
Hi Sathya , I would like to use Homography for a moving camera . Are the results going to be good ? How can I tell which movement is caused due to the camera movement? Thanks in advance !
Hi George,
A homography can be used with a moving camera as long as the object you are looking at is a plane.
Dear All,
I am a beginner in computer vision. I am working on a dataset in which due to camera inclined position, there is perspective effect. I calculate optical flow, however, due to perspective distortion, optical flow is not correct. I attach figure here. Please see that optical flow for people near to camera is correct, whereas, people far from camera, optical flow is not calculated. I assume that this is due to the perspective distortion.
Please suggest me that can I apply homography method explain in this post? Secondly do I need to apply this process before calculating optical flow?
Many thanks in advance,
Umer
I’ve got a problem to solve that I’m not sure if homography could help me.
I have a mesh https://dl.dropboxusercontent.com/u/710615/tst_msh.jpg , which was created from this image : https://dl.dropboxusercontent.com/u/710615/FinalPCBwithFrame_.jpg .
I need to fit and overlay the mesh on https://dl.dropboxusercontent.com/u/710615/Layer1.png .
The goal is get something like this http://i.stack.imgur.com/fyixS.jpg.
I’ve found the https://dl.dropboxusercontent.com/u/710615/3.jpg of the “source image” .
I think that I need to remove a column of white pixels from the upper left corner of the mesh (I don’t know yet how to do it easily).
Besides The mesh image and the original image from the mesh are equal. However, the image of layer has some similar holes but not equal, which is acceptable.
Hi Sir,
Can Homography transform be used to correct the projector distortion when projected on planar-tilted surfaces
Yup. it is often used for that purpose
Sir, Can you describe the procedure briefly for doing the same.
Thanks in advance :-).
Sir, can there be any homography between world plane and corresponding image Plane. If yes, then would be the coordinates in world dimension, will they be like (0, 0) (0, w) (w, h) (h, 0) where w and h are width and height of rectangular plane in real dimensions(mm/cm). Wouldn’t there be any inconsistency since image coordinates are in pixels.
hello
how can i change camera matrix when i switch camer resolution after calibration?
thanks
Gabor
I have two images from a plane. How to get a 3D point?
This blog is incredible. I barely have a background in highschool lin
algebra but can understand and implement the stuff you explain with
ease.
Thanks for the kind words.
Hi Satya, is it possible using homography (or another method) to form an image of say the front of a person from two images taken at +45deg and -45 deg of that person?
Yes. The method is called View Morphing.
http://homes.cs.washington.edu/~seitz/vmorph/vmorph.htm
Very old version of OpenCV had an implementation. The new version has all the tools necessary to make the application.
awesome. thanks
Dear Sir,
how do we estimate rotation matrix from homograpy matrix H?
the H Matrix is (matlab-like syntax):
[a b c; d e f; g h 1]
the sub-matrix [a b ; d e] can be interpreted as rotation matrix.
We can interpret those values as:
a,e = cos(t)
b = -sin(t)
c=sin(t)
how does one pick the points coordinates? using paint?
Usually those points are calculated using SFIT, but inorder to calculate those manually you may use ImageJ software.
Hi Satya,
I would like to use homography to correct a distorted image. I would like the point A, B, C, D of the first image to correspond with the point A, B, C, D of the second image (square).
I tried your code but, it resulted in a stronger distortion…
Here are my code, could you tell me if I did something wrong.
Thanks a lot for your help.
import cv2
import numpy as np
import matplotlib.pyplot as plt
if __name__ == ‘__main__’ :
# Read source image.
im_src = cv2.imread(‘points.jpg’, cv2.IMREAD_COLOR)
# Four points of the miniR image
pts_src = np.array([[744,255],[856,279],[1000,667],[926,741]], dtype=float)
# Read destination image.
im_dst = cv2.imread(‘rectangle.jpg’, cv2.IMREAD_COLOR)
# Four points of the square
pts_dst = np.array([[200,200],[1000,200],[1000,1000],[200,1000]], dtype=float)
# Calculate Homography
h, status = cv2.findHomography(pts_src, pts_dst)
# Warp source image to destination based on homography
im_out = cv2.warpPerspective(im_src, h, (im_dst.shape[1],im_dst.shape[0]))
cv2.imwrite(‘corrected.jpg’, im_out)
# Display images
cv2.imshow(“Source Image”, im_src)
cv2.imshow(“Destination Image”, im_dst)
cv2.imshow(“Warped Source Image”, im_out)
cv2.waitKey(0)
#show plot with the coordinate
plt.imshow(im_out, cmap=’gray’, interpolation=’bicubic’)
https://uploads.disquscdn.com/images/f6ed7f2c2c4a2eb7da4420d178079cb452a3e142393d028e1cd1c66ec7a69ae6.jpg
https://uploads.disquscdn.com/images/35ad70917ab6347e201722f3119e08d5a69acd379d5fd2b8940a39a9ecaac442.jpg
https://uploads.disquscdn.com/images/4e169cdc266db535db20389b2eb8f5220cd9f692c801d4a950116c2a312c2cb5.jpg
Hello Satya,
Is there any way to reduce the opacity of the im_dst image in the im_out image, so that I can see both im_src and im_dst images in im_out image? I tried adding an alpha channel to im_dst image just before using cv2.warpPerspective(), but all I got was a darkened im_dst image in im_out image! How to see both the images at the same time?
Use cv2.addWeighted(im_dst, dst_opacity, im_out, out_opacity, 1), where dst_opacity and out_opacity are the weights for each image—you can use 0.5 for both for a start.
Just a small correction here.
When an homography transforms pixel locations, it transforms them to homogenous coordinates, but they may be scaled by some scaling factor; you must divide by the scaling factor to get back to the correct coordinates in your image. So in your example, it should be [s*x’, s*y’, s] = H * [x, y, 1], and a division by s would give you the points [x’, y’, 1].
That’s right! Also, there’s a function in OpenCV `convertPointsFromHomogeneous(..)`
Is it possible to reproject a panorama from equirectangular to rectilinear using opencv?
I cannot down your source code..
Hello Satya, thanks for your blog.
I have trained a deep homography model based on the paper–[Deep image homography estimiation](https://arxiv.org/pdf/1606.03798.pdf)
Results looks great,
avg sqrt L2 loss of training set(500032 images from image net) : 0.8967
avg sqrt L2 loss of test set(10000 images from image net) : 0.8448
avg delta of test set(how many pixels different with the real delta) : 2.48
Problem is I do not know how to convert the H4 matrix to homography matrix, do you know how to do it?
You can find the details at fast ai forum–http://forums.fast.ai/t/how-to-transform-4-points-parameter-matrix-to-homography-matrix/5770
Thanks
Great explanation, thank you for making it so readable!
I have a small question. Generally Homography matrix is a camera projection matrix when the 3d scene lies on a 2d plane(for example on z==0). In that case, we can use the camera projection equations to find this H matrix. But how did we come up with this equation x’ = H*x. Can you give an explanation?
Really good tutorial! Thanks.
Great blog Satya ! just wondering if we can avoid knowing aspect ratio of the book. We know the book is rectangle may be something could be done with two vanishing points.
Is there a way to compute homography matrix without using matches points ?
For example , can i compute homography matrix using Euler angles , position (x,y) , accelaration and speed ?
Thanks for this great tutorial !
But can someone explain to me what the h11…h33 exactly are? And how they change while moving the image ?
Very Interesting!
I had 2 quick questions :
1] Can I use this to correct foreshortening any image (not a polygon) and how ?
2] If the same image were at 15 deg to the left or right can i get the image at 0 deg ?
Hi satya,
Thanks for wonderful article, My understanding about homography is that by using intrinsic and extrinsic camera calibration parameter , homography matrix calculated.
so how findhomography function calculate camera matrix just by passing 8 pointes?