In this tutorial we will learn how to create an average face using OpenCV ( C++ / Python ).
Most people would agree that the woman in Figure 1 is pretty. Can you guess her ethnicity ? Why is her skin flawless ? Well, she is not real. She is not completely virtual either. Her face is the average face of all female employees who worked at my company, Sight Commerce Inc., around 2011. It is difficult to pin point her ethnicity because, thanks to our excellent diversity record, she is part Caucasian, part Hispanic, part Asian and part Indian!
This history of face averaging is fascinating.
It all started with Francis Galton ( cousin of Charles Darwin ) who, back in 1878, came up with a new photographic technique for compositing faces by aligning the eyes. He thought that by averaging faces of criminals he could create the prototypical criminal face which in turn would help predict if someone is a criminal based on their facial features. His hypothesis turned out to be wrong; you cannot predict if a person is a criminal by looking at their photos.
However, he noted that the average face was always more attractive than the faces it was the average of.
Several researchers in the 1990s showed that people find facial averages much more attractive than individual faces. In one amusing experiment researchers averaged the faces of 22 miss Germany finalists of 2002. People rated the average face to be much more attractive than every one of the 22 contestants, including miss Berlin who won the competition. Ouch! Turns out Jessica Alba’s face is attractive precisely because her face is close to the average!
Shouldn’t an average be mediocre by definition ? Why do we find an average face attractive ? According to an evolutionary hypothesis called Koinophilia, sexually reproducing animals seek mates with average features because deviations from the average could indicate disadvantageous mutations. An average face is also symmetric because the variations in the left side and the right side of the face are averaged out.
How to create an average face using OpenCV ?
This post is part of a series that includes Facial Landmark Detection, Delaunay Triangulation, Face Morphing and Face Swap. We build on some of the concepts introduced in these previous posts. In fact Face Averaging is no different in theory from Face Morphing.
Start your exciting journey from an absolute Beginner to Mastery in AI, Computer Vision & Deep Learning!
The steps for generating an average face given a set of facial images is described below. We make no assumptions about the size of the images or the size the faces in the images.
Step 1 : Facial Feature Detection
For each facial image we calculate 68 facial landmarks using dlib. Details about installing and using dlib can be found in my post on Facial Feature Detection. The figure on the left shows the 68 landmark points.
Step 2 : Coordinate Transformation
The input facial images can be of very different sizes. So we need a way to normalize the faces and bring them to the same reference frame. To achieve this we warp the faces to a 600×600 image such that the left corner of the left eye is at pixel location ( 180, 200 ) and the right corner of the right eye is at pixel location ( 420, 200 ). Let us call this coordinate system the output coordinate system and the coordinates of the original images the input coordinate systems.
How were the above points chosen ? I wanted to make sure the points were on a horizontal line, and the face was centered at about a third of the height from the top of the image. So I chose the corners of the eyes to be at ( 0.3 x width, height / 3 ) and ( 0.7 x width , height / 3 ).
We also know the location of the corners of the eyes in the original images; they are landmarks 36 and 45 respectively. We can therefore calculate a similarity transform ( rotation , translation and scale ) that transforms the points from the input coordinate systems to the output coordinate system.
or an entire image. The first two columns of this matrix encodes rotation and scale, and the last column encodes translation ( i.e. shift ). Let’s say you want to transform (move) the four corners of a square so that the square is scaled in the x and y direction by 
and 
respectively. At the same time it is rotated by an angle 
, and translated ( moved ) by 
and 
in the x and y directions. The similarity transform for this can be written as ![\[S=\begin{bmatrix}s_x \cos(\theta) & \sin(\theta) & t_x \\-\sin(\theta) & s_y \cos(\theta) & t_y\end{bmatrix}\]](https://learnopencv.com/wp-content/ql-cache/quicklatex.com-b6e614b5448854f2c83abcb6e5786774_l3.png)
, the above similarity transform, moves it to point 
using the equation given below ![\[\begin{bmatrix}x_t \\y_t\end{bmatrix}=\begin{bmatrix}s_x \cos(\theta) & \sin(\theta) \\-\sin(\theta) & s_y \cos(\theta)\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}+\begin{bmatrix}t_x \\t_y\end{bmatrix}\]](https://learnopencv.com/wp-content/ql-cache/quicklatex.com-46b93d3572bf6977c11de5e8d3da253d_l3.png)
To find the similarity transform that will transform the points from the input image coordinates to the output image coordinates we can use estimateRigidTransform
C++
// inPts and outPts are Vectors of points.
// The last parameter indicates we do not want a similarity
// transform and not a full affine transform.
cv::estimateRigidTransform(inPts, outPts, false);
Python
# inPts and outPts are numpy arrays of tuples
# The last parameter indicates we do not want a similarity
# transform and not a full affine transform.
cv2.estimateRigidTransform(inPts, outPts, False);
There is one little problem though. OpenCV requires you to supply at least 3 point pairs. This is silly because you can calculate a similarity transform using just two points. The good news is that we can simply hallucinate a third point such that it forms an equilateral triangle with the two known points and then use estimateRigidTransform as if we had three points pairs.
Once a similarity transform is calculated, it can be used to transform the input image and the landmarks to the output coordinates. The image is transformed using warpAffine and the points are transformed using transform.
Step 3 : Face Alignment
In the previous step we were able to transform all the images and the landmarks to the output image coordinates. All the images are now of the same size, and the two corners of the eyes are aligned. It may be tempting to obtain the average image by averaging pixel values of these aligned images. However if you did this, you will end up with an image shown on the left. Sure, the eyes are aligned, but other facial features are misaligned.
If you are a regular reader of this blog, you probably know the trick we are going to use next to bring the facial features in alignment.
If we knew which point in one input image corresponded to which point in another input image we could easily align the two images perfectly. However, we do not have that information. We only know the locations of 68 corresponding points in the input images. We will use these 68 points to divide the images into triangular regions. and align these regions before averaging pixel values.
You will benefit from reading my post on Face Morphing for more details on this alignment process. A less detailed description follows.
Calculate Mean Face Points
To calculate the average face where the features are aligned, we first need to calculate the average of all transformed landmarks in the output image coordinates. This is done by simply averaging the x and y values of the landmarks in the output image coordinates.
Calculate Delaunay Triangulation
In the previous step we obtained the landmark locations for the average face in the output image coordinates. We can use these 68 points (shown in blue in Figure 6 ), and 8 points on the boundary of the output image (shown in green) to calculate a Delaunay Triangulation ( shown in red ). More details about Delaunay Triangulation can be found here.
Delaunay triangulation allows us to break the image into triangles. The result of Delaunay triangulation is a list of triangles represented by the indices of points in the 76 points ( 68 face points + 8 boundary points ) array. In the triangulation example shown below in the yellow box, we see than landmarks 62, 68 and 60 form a triangle, and 32, 50, and 49 form another triangle and so on.
[
62 68 60
32 50 49
15 16 72
9 8 58
53 35 36
… ]
Warp Triangles
In the previous step we calculated the average location of facial landmarks and used these locations to calculate a Delaunay triangulation to divide the image into triangles. In Figure 7, the left image shows Delaunay triangles on the transformed input image and the middle image shows the triangulation on the average landmarks. Note that triangle 1 in the left image corresponds to triangle 1 in the middle image. The three vertices of triangle 1 in the left image and the corresponding vertices in the middle image can be used to calculate an affine transform. This affine transform can be used to transform all pixels inside triangle 1 in the left image to triangle 1 in the middle image. This procedure when repeated for every triangle in the left image, results in the right image. The right image is simply the left image warped to the average face.
Step 4 : Face Averaging
The previous step, when applied to all input images, gives us images that are correctly warped to the average image landmark coordinates. To calculate the average image, we can simply add the pixel intensities of all warped images and divide by the number of images. Figure 2 shows the result of this averaging. It looks much better than the average we obtained in Figure 5.
How do you think how the average President looks ? Would love to hear your response. To me, he looks fatherly and kind.
Face Averaging Results
How does an average successful male tech entrepreneur look like ? Figure 8 below shows the average face of Mark Zuckerberg, Larry Page, Elon Musk and Jeff Bezos. I can’t find anything remarkable to say about this average entrepreneur except that he still has some hair left despite Jeff Bezos trying to hold the average down.
How does an average Oscar’s best actress winner look like ? Figure 9 below shows the average face of Brie Larson, Julianne Moore, Cate Blanchett and Jennifer Lawrence. The average good actress looks very charming! She also has better teeth than the average entrepreneur. No surprises here!
Figure 8 – Facial Average of Mark Zuckerberg, Larry Page, Elon Musk and Jeff Bezos Figure 9 – Facial Average of last four best actress winners : Brie Larson, Julianne Moore, Cate Blanchett and Jennifer Lawrence
You can also make a symmetric face, by averaging a face and its mirror reflection. An example is shown below.
Subscribe & Download Code
If you liked this article and would like to download code (C++ and Python) and example images used in this post, please click here. Alternately, sign up to receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.Image Credits
All the images of current and former presidents are in public domain and were downloaded from wikipedia.
I got permission to use the image in Figure 1. from Sight Commerce Inc.
The average president struck me as mostly looking like Dubya. Then I realized that two of the six faces are father and son so maybe that isn’t so surprising. It would be interesting to take a facial average of a number of related people and see if it brings out familial features.
That’s true. The first time I looked at it, I thought there was a bug in my code. But after trying it on entrepreneurs and actresses set, I was convinced that the result was being of two similar faces of father and son.
in the github code, I see that there are a few .txt files, how these text files are created from the jpg files?
They were generated using dlib. You can see more details in this post
https://learnopencv.com/facial-landmark-detection/
Thank you for the time you put in, in creating and sharing this and what you know and have learned about this!
Thanks for the kind words!
Really Great Post! Thank you very much!!!
Thank you!
Thanks Dr. Satya. I’m trying to learn image morphing and your tutorials have been a great help! … I’ve a quick question for you 🙂 … For virtual makeup should I follow the same techniques or is there something else I should look at. Also I didn’t find a tutorial for “Virtual Makeup” on the site; I would be great if you can write one. Thanks again!
Hi Owais,
Thanks for the kind words. For Virtual makeup you need to start with landmark detection, but you need to go further and find contours around facial features ( e.g. contour around the lips ). I am the founder of a Virtual Makeup company ( taaz.com ) and unfortunately cannot reveal trade secrets 🙁
It’s alright, I understand … Still thanks for these amazing tutorials and keep em coming 🙂
Will the Delaunay triangulation always produce triangles with the same vertices? Imagine 4 points that are the corners of a rectangle – it seems the triangulation could choose the “right diagonal” or the “left diagonal” (sorry for the terrible ascii drawings…):
___
| |
—–
vs
___
| / |
—–
and then you wouldn’t have similar regions to warp and compare?
You are right. That is exactly why we do Delaunay triangulation only once — on the average landmark location. The triple of indices representing the triangulation are saved, and the same triangulation is applied to all faces.
Your ascii drawing was more than adequate to make the point :).
It would be interesting to know how will the average human face looks like…but it have to average across different groups.
You will find this interesting.
http://thechive.com/2015/09/13/heres-what-the-average-person-looks-like-in-each-country-11-photos/
Why does my result look like this.
This happens because “allPoints = readPoints(path);” and ” images =
readImages(path);” do not maintain proper correspondence. For some
images, the landmarks are wrongly loaded. Instead of using
“os.listdir(path)” to load the respective files for “allPoints” and
“images”, use some alternative method, for example, parsing a list of
file names which maintains proper correspondence. At least in Linux, the
code as it is generates wrong results like the above for both Python and C++.
Thanks Ram. This has been fixed. Please checkout the latest version of the code.
To fix this problem quickly, all you need to do is to change the call that navigates the file list from
for filePath in os.listdir(path):
to
for filePath in sorted(os.listdir(path)):
hi,man,I want to estimate how beautiful the person. Can I use the average face.
Check out this guest post
https://learnopencv.com/computer-vision-for-predicting-facial-attractiveness/
In the above post, only the facial landmark locations are used to predict attractiveness. But the right way to do it is to get a lot of images of faces and label them with a rating say 1-10, and then use Deep Learning to find an attractiveness score. The process will be a bit involved, but If you get a 1000 images of faces rated between 1-10, I would be happy to help with the machine learning part. Note : The attractiveness score should be based on the face alone.
sorry to bother you again. I’d like to know what’s the meaning of the numbers in obama.jpg.txt and something like that. Is it the coordinates of points in the picture?
Yes they are 68 points on the face.
Hi, I use opencv2.4.9 and I want to realize the face average. Is there any necessity for me to update to opencv3?
OpenCV 2.4.9 should work.
Hmmmh, hi again, how did you realize step 2 of face average, “Coordinate Transformation”. You used matlab, or python or what? And could you please tell me about those? Sorry for bother you.
Everything is done in OpenCV and dlib.
Amazing, job!
How to save 68 values pairs of landmark predicted by dlib to txt file?
print(“Part 0: {}, Part 1: {} …”.format(shape.part(0),
shape.part(1)))
You can also try my dlib fork that contains an example to save to file
https://github.com/spmallick/dlib/blob/master/examples/face_landmark_detection_to_file.cpp
Thank You!
Why did I get blue average face from normal fotos?
OpenCV uses BGR image format. Looks like your channels are swapped.
Thank You. I Added color conversion:
output = cv2.cvtColor(output, cv2.COLOR_BGR2RGB)
I’m getting the following error:
Traceback (most recent call last):
File “faceAverage.py”, line 215, in
pointsAvg = np.array([(0,0)]* ( len(allPoints[0]) + len(boundaryPts) ), np.float32());
IndexError: list index out of range
It’s worth noting I have 2611 pictures. Is there a way to fix this?
Add RAM to your PC
downsample pictures (pyrldown?) before averaging
I want to do face averaging for the same person. will the face average works with the different pose of same person.
Hi Mohanraj
It may give pretty weird results, but I would love to see the results if you try it.
I used ORL face data set of different pose for the same person. please find the below face image after the face average.
For frontal face images of same person am getting good results. please find the cal tech face data set result for frontal face images.
How to save the output image?
You can try the following code
cv2.imwrite("output.jpg", 255 * output);
Hello,
I am trying to figure out how one would save the output of this program to an image file. My first instinct was to use the following:
cv2.imwrite(“output.jpg”, output)
But this results in a completely black image being saved. After poking around online, it would seem that the format of image data for cv2.imshow is different than the format needed for cv2.imwrite. Specifically, that the image data is currently scaled between 0 – 1 and it needs to be scaled from 0 – 255 in order to write to the disk. Does anybody know how to reformat the image data such that cv2.imwrite will work on the output array and save the image to disk? Thanks!
Try this
cv2.imwrite("output.jpg", 255 * output);
Sir,
Can you please suggest some idea how to do age invariant face recognition
Hello Satya,
I was wondering why is the phantom point picked in such a way that the an equilateral triangle is formed between all three points? Is it related to the affine transform parameters that are estimated?
Any triangle would do. The points just should not be on a line. Skinny triangles are therefore not a good idea and the fattest triangle is an equilateral triangle.
i tried faceaverage cpp program and compile it by this command line in ubuntu :-
g++ -ggdb faceAverage.cpp -o faceAverage -lpthread `pkg-config –cflags –libs opencv`
and following error occured as attached in below screenshot
https://uploads.disquscdn.com/images/e028d2034f83551276e04e802086d6b93e267e02704ad49fb199aaabdfe5fd88.png
if there is error with this command please correct me
by providing suitable solution and also suggest whether we compile
this cpp program in opencv(2.4.x) or in dlib(section).??
what are the most effeiceint face warping algorithm?
I have used this triangulation method, but it fails in many cases especially if there is a facial expression in one face and the other face is neutral.
I have also used Thin-Plate Warping Algo but also fails in other cases.
Is there any more advanced face warping techniques that generalize better than these two.
You can try TPS ( as you suggested ) or Moving Least Squares (MLS). They both work ok, but will be computationally more expensive compared to the piecewise linear method we have used here.
Dear Mr.Mallick
Hello, thank you for your writing.
Could you let me know how to make .txt file(including 68points) of any other images?
I want to try this code with my selfies.
I only use python
They are created using Dlb’s facial landmark detector. https://learnopencv.com/facial-landmark-detection/
Hello! Thank you for this interesting post. How many photos will you recommend to use? For instance, I want to make an average face from 200+ photos. Will the amount make the result worse?
No, 200+ photos will make the skin region and background smoother. Some people have mentioned that code runs into memory issues with a large number of photos. If you encounter that, please let me know.
Satya
Getting this error when trying with different sets of images. Kindly help.
Traceback (most recent call last):
File “faceAverage.py”, line 215, in
pointsAvg = np.array([(0,0)]* ( len(allPoints[0]) + len(boundaryPts) ), np.float32());
IndexError: list index out of range
Tried with 3 images of female popstars, 5 Prime ministers.None worked.
https://uploads.disquscdn.com/images/743c03ff879ec1e221a2b65dbe02497131ce2b0543b5625f9d2d2baeac48180a.png
Why my result is like this? Thanks
@spmallick:disqus How to solve this? Thanks
I was able to get rid of this by changing my input images to be 600px by 600px.