• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

Face Reconstruction using EigenFaces (C++/Python)

Satya Mallick
January 26, 2018 5 Comments
Application Face Machine Learning Tutorial

January 26, 2018 By 5 Comments

Facial Reconstruction using Eigen Faces

Figure 1: On the left is the original image. The second image from left is constructed using 250 EigenFaces, the third using 1000 Eigenfaces and the image on the extreme right using 4000 Eigenfaces.

In this post, we will learn how to reconstruct a face using EigenFaces. This post is written for beginners. If you don’t know about Principal Component Analysis (PCA) or EigenFaces, I recommend you go through the following posts in the series.

  1. Principal Component Analysis (PCA)
  2. EigenFace using OpenCV (C++/Python)

What are EigenFaces?

In our previous post, we explained Eigenfaces are images that can be added to a mean (average) face to create new facial images. We can write this mathematically as,

    \[F = F_m + \sum^n_{i=1} \alpha_i F_i\]

where,

F is a new face.
F_m is the mean or the average face,
F_i is an EigenFace.
\alpha_i are scalar multipliers we can choose to create new faces. These can be positive or negative.

In our previous post, we explained how to calculate the EigenFaces F_i, how to interpret them and how to create new faces by changing weights \alpha_i.

Now suppose, we are given a new facial photo as shown in Figure 1. How can we reconstruct the photo using EigenFaces F_i? In other words, how do we find the weights \alpha_i that when used in the above equation will produce the facial image as an output? This is exactly the question covered in this post but before we attempt to do that, we need a little background in linear algebra.

Change of coordinates

Consider a 3D coordinate system X, Y, Z as shown using black in Figure 2. You can imagine another set of perpendicular axes that is rotated and translated (shifted) by (x_o, y_o, z_o) with respect to the original X, Y, Z frame. In Figure 2, we show the axes of this rotated and translated coordinate system X'Y'Z' in blue. Let us consider a point (shown using the red dot) whose coordinates in the XYZ coordinates is (x, y, z).

Change of coordinates

Figure 2.

How do we find the coordinates (x', y', z') of the point in the X'Y'Z' coordinate system? This can be done in two steps

  1. Translate : First, we can remove the translation component from (x, y, z) by subtracting the origin (x_o, y_o, z_o) of the new coordinate system. So we have a new vector (x-x_o, y-y_o, z-z_o).
  2. Project: Next, we need to project (x-x_o, y-y_o, z-z_o) onto X', Y', Z' which is nothing but the dot product of (x-x_o, y-y_o, z-z_o) with the direction X', Y' and Z' respectively. The green line in Figure 2 shows the projection of the point onto the Z' axis.

Let’s see how this technique applies to reconstructing faces.

Calculating PCA weights for a new facial image

As we had seen in the previous post, to calculate the principal components of facial data, we convert the facial images into long vectors. For example, if we have a collection of aligned facial images of size 100 x 100 x 3, each image can be thought as a vector of length 100 x 100 x 3 = 30,000. Just like a tuple of three numbers (x, y, z) represents a point in 3D, we can say that a vector of length 30,000 is a point in a 30,000 dimensional space. The axes of this high dimensional space are perpendicular to each other just like the axes X, Y and Z of a 3D dimensional space are perpendicular to each other. And just like X'Y'Z', the principal components (Eigenvectors) form a new coordinate system in this high dimensional space with the new origin being the mean vector.

Given a new image, here is how we can find the weights \alpha_i

  1. Vectorize image : We first create a long vector from image data. This is simple a rearrangement of data which requires just a line or two of code.
  2. Subtract mean vector
  3. Project onto Principal Components: This can be done by calculating the dot product of the mean subtracted vector with each of the principal components. This gives dot product is the weight \alpha_i
  4. Assemble face vector : Once the weights have been calculated, we can simply add the multiply each weight to the principal components ( or eigen faces ) and sum them all together. Finally, we need to add the average face vector to this sum.
  5. Reshape vector into facial image : As a result of the previous step, we obtain a vector that is 30k long and can be reshaped into a 100 x 100 x 3 image. This is the final image.

PCA for dimensionality reduction

In our example, a 100 x 100 x 3 image has 30k dimensions. After doing PCA on 2000 images, we can obtain a space that is 2000 dimensional and yet is able to reconstruct a new face to a reasonable level of accuracy. What used to take 30k numbers to represent is now represented using only 2k numbers (i.e. the weights \alpha_i). In other words, we just used PCA to reduce the dimension of the space of faces.

Code for Face Reconstruction using EigenFaces (C++/Python)

Download Code
To easily follow along this tutorial, please download code by clicking on the button below. It’s FREE!

Download Code

Assuming you have downloaded the code, we will go over important parts of the code. First, the code for calculating the mean face and the EigenFaces is shared in files createPCAModel.cpp and createPCAModel.py. The method was explained in our previous post and so we will skip that explanation. Instead, we will go over reconstructFace.cpp and reconstructFace.py.

C++

// Recontruct face using mean face and EigenFaces
void reconstructFace(int sliderVal, void*)
{
	// Start with the mean / average face
	Mat output = averageFace.clone();
	for (int i = 0;  i < sliderVal; i++)
	{
		// The weight is the dot product of the mean subtracted
		// image vector with the EigenVector
		double weight = imVector.dot(eigenVectors.row(i)); 

		// Add weighted EigenFace to the output
		output = output + eigenFaces[i] * weight; 
	}

	displayResult(im, output);
}
	

int main(int argc, char **argv)
{

	string modelFile("pcaParams.yml");
	cout << "Reading model file " << modelFile << " ... " ; 

	FileStorage file(modelFile, FileStorage::READ);
	
	// Extract mean vector
	meanVector = file["mean"].mat();

	// Extract Eigen Vectors
	eigenVectors = file["eigenVectors"].mat();

	// Extract size of the images used in training.
	Mat szMat = file["size"].mat();
	Size sz = Size(szMat.at<double>(1,0),szMat.at<double>(0,0));

	// Extract maximum number of EigenVectors. 
	// This is the max(numImagesUsedInTraining, w * h * 3)
	// where w = width, h = height of the training images. 
	int numEigenFaces = eigenVectors.size().height; 
	cout <<  "DONE" << endl; 

	cout << "Extracting mean face and eigen faces ... "; 
	// Extract mean vector and reshape it to obtain average face
	averageFace = meanVector.reshape(3,sz.height);
	
	// Reshape Eigenvectors to obtain EigenFaces
	for(int i = 0; i < numEigenFaces; i++)
	{
			Mat row = eigenVectors.row(i); 
			Mat eigenFace = row.reshape(3,sz.height);
			eigenFaces.push_back(eigenFace);
	}
	cout << "DONE" << endl; 

	// Read new test image. This image was not used in traning. 
	string imageFilename("test/satya1.jpg");
	cout << "Read image " << imageFilename << " and vectorize ... ";
	im = imread(imageFilename);
	im.convertTo(im, CV_32FC3, 1/255.0);
	
	// Reshape image to one long vector and subtract the mean vector
	imVector = im.clone(); 
	imVector = imVector.reshape(1, 1) - meanVector; 
	cout << "DONE" << endl; 


	// Show mean face first
	output = averageFace.clone(); 

	cout << "Usage:" << endl 
	<< "\tChange the slider to change the number of EigenFaces" << endl
	<< "\tHit ESC to terminate program." << endl;
	
	namedWindow("Result", CV_WINDOW_AUTOSIZE);
	int sliderValue; 

	// Changing the slider value changes the number of EigenVectors
	// used in reconstructFace.
	createTrackbar( "No. of EigenFaces", "Result", &sliderValue, numEigenFaces, reconstructFace);
	
	// Display original image and the reconstructed image size by side
	displayResult(im, output);
	

	waitKey(0);
	destroyAllWindows(); 
}

Python

# Recontruct face using mean face and EigenFaces
def reconstructFace(*args):
	# Start with the mean / average face
	output = averageFace
	
	for i in range(0,args[0]):
		'''
		The weight is the dot product of the mean subtracted
		image vector with the EigenVector
		'''
		weight = np.dot(imVector, eigenVectors[i])
		output = output + eigenFaces[i] * weight

	
	displayResult(im, output)
    


if __name__ == '__main__':

	# Read model file
	modelFile = "pcaParams.yml"
	print("Reading model file " + modelFile, end=" ... ", flush=True)
	file = cv2.FileStorage(modelFile, cv2.FILE_STORAGE_READ)
	
	# Extract mean vector
	mean = file.getNode("mean").mat()
	
	# Extract Eigen Vectors
	eigenVectors = file.getNode("eigenVectors").mat()
	
	# Extract size of the images used in training.
	sz = file.getNode("size").mat()
	sz = (int(sz[0,0]), int(sz[1,0]), int(sz[2,0]))
	
	''' 
	Extract maximum number of EigenVectors. 
	This is the max(numImagesUsedInTraining, w * h * 3)
	where w = width, h = height of the training images. 
	'''

	numEigenFaces = eigenVectors.shape[0]
	print("DONE")

	# Extract mean vector and reshape it to obtain average face
	averageFace = mean.reshape(sz)

	# Reshape Eigenvectors to obtain EigenFaces
	eigenFaces = [] 
	for eigenVector in eigenVectors:
		eigenFace = eigenVector.reshape(sz)
		eigenFaces.append(eigenFace)


	# Read new test image. This image was not used in traning. 
	imageFilename = "test/satya2.jpg"
	print("Read image " + imageFilename + " and vectorize ", end=" ... ");
	im = cv2.imread(imageFilename)
	im = np.float32(im)/255.0

	# Reshape image to one long vector and subtract the mean vector
	imVector = im.flatten() - mean; 
	print("Done");
	
	# Show mean face first
	output = averageFace
	
	# Create window for displaying result
	cv2.namedWindow("Result", cv2.WINDOW_AUTOSIZE)

	# Changing the slider value changes the number of EigenVectors
	# used in reconstructFace.
	cv2.createTrackbar( "No. of EigenFaces", "Result", 0, numEigenFaces, reconstructFace)

	# Display original image and the reconstructed image size by side
	displayResult(im, output)

	cv2.waitKey(0)
	cv2.destroyAllWindows()

You can create the model pcaParams.yml using createPCAModel.cpp and createPCAModel.py. The code uses the first 1000 images of the CelebA dataset and scales them to half the size first. So this PCA model was trained on images of size (89 x 109). In addition to the 1000 images, the code also used a vertically flipped version of the original images, and therefore we use 2000 images for training.

Back to the code shared above.

We first read the model file ( lines 24-42 in C++ and lines 23-43 in Python). It contains the mean vector of size 1 x 29,103, and a matrix of EigenVectors of size 2000 x 29,103. The model also includes the size of the images used in training.

Next, we reshape the mean vector to obtain the average face in line 46 of the C++ and Python code. We also, reshape, the Eigen Vectors to obtain the EigenFaces in lines 48-54 in both versions of the code.

Next, we read a new image that was not used in training. Note, the image is also of size 89×109 and the eyes were aligned with the images in the training set. This image is then vectorized (flattened) and the mean vector is subtracted from it. These operations are performed in lines 57-66 in C++ and lines 55-63 in Python.

The reconstruction is done in the function reconstructFace starting at line 2 in both versions of the code. A slider is provided which controls the number of EigenVectors to use. Since the model was trained on 2000 images, we can have a maximum of 2000 EigenVectors.

We start with the average face. The weights are calculated by the dot product of the mean subtracted image vector and the EigenVectors. Finally, the weighted EigenFaces are added to the average face.

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Tags: EigenFace PCA Principal Component Analysis

Filed Under: Application, Face, Machine Learning, Tutorial

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy