In this tutorial, we will learn about facial landmark detection using OpenCV with no external dependencies.
I have written several posts about Facial Landmark Detection and its applications. You can use landmark detection for face morphing, face averaging and face swapping. Until now, we had used the landmark detection that comes with Dlib. It works great, but wouldn’t it be nice if we did not have to depend on any external library.
Well, the wait is over. Almost!
OpenCV now supports several algorithms for landmark detection natively. However, the implementation needs some more work before it is ready for two reasons
- Python support: It appears that there is still no python support yet as of OpenCV 3.4
- Lack of trained models: Out of the three algorithms implemented for landmark detection, I could find only one trained model that works. This is not very hard to fix. In the worst case, we will try our own model and make it available for people to use.
Because of the two limitations above, this post will be updated a few times in the next few months.
Facemark API
OpenCV’s facial landmark API is called Facemark. It has three different implementations of landmark detection based on three different papers
- FacemarkKazemi: This implementation is based on a paper titled “One Millisecond Face Alignment with an Ensemble of Regression Trees” by V.Kazemi and J. Sullivan published in CVPR 2014. An alternative implementation of this algorithm can be found in DLIB
- FacemarkAAM: This implementation uses an Active Appearance Model (AAM) and is based on an the paper titled “Optimization problems for fast AAM fitting in-the-wild” by G. Tzimiropoulos and M. Pantic, published in ICCV 2013.
- FacemarkLBF: This implementation is based a paper titled “Face alignment at 3000 fps via regressing local binary features” by S. Ren published in CVPR 2014.
The three implementations follow similar patterns even though, at the time of writing this post, the FacemarkKazemi class does not seem to be derived from the base Facemark class, while the other two are.
Facemark trained models
Even though the Facemark API consists of three different implementations, a trained model is available for only FacemarkLBF. This post will be updated in future after we train our own model based on public datasets.
You can download the trained model from
- lbfmodel.yaml.
- lbfmodel.yaml (Forked Repo) ( in case the above link is taken down ).
Real-time facial landmark detection using OpenCV code
In this section, we will share the code for real-time facial landmark detection. This post will be updated when Python bindings are supported for the Facemark API.
The steps involved in calling the Facemark API for real-time landmark detection are listed with references to the code below.
- Load face detector: All facial landmark detection algorithms take as input a cropped facial image. Therefore, our first step is to detect all faces in the image, and pass those face rectangles to the landmark detector. We load OpenCV’s HAAR face detector (haarcascade_frontalface_alt2.xml) in line 14.
- Create Facemark Instance : In line 17 we create an instance of the Facemark class. It is wrapped inside OpenCV smart pointer (PTR) so you do not have to worry about memory management.
- Load landmark detector: Next, we load the landmark detector (lbfmodel.yaml) in line 20. This landmark detector was trained on a few thousand images of facial images and corresponding landmarks. Public datasets of facial images with annotated landmarks can be found here.
- Capture frames from webcam: The next step is to grab a video frame and process it. Line 23 sets up the video capture from the webcam connected to your machine. You can change it to read a video file instead by replacing line 23 with the code below.
VideoCapture cap("myvideo.mp4");
We then continuously grab frames from the video ( line 29 ) until ESC is pressed ( line 62 ).
5. Detect faces: We run the face detector on every frame of the video in lines 33-39. The output of a face detector is a vector of rectangles that contain one or more faces in the image.
6. Run facial landmark detector: We pass the original image and the detected face rectangles to the facial landmark detector in line 48. For every face, we get 68 landmarks which are stored in a vector of points. Because there can be multiple faces in a frame, we have to pass a vector of vector of points to store the landmarks ( see line 45 ).
7. Draw landmarks: Once we have obtained the landmarks, we can draw them on the frame for display. The code for drawing is inside drawLandmarks.hpp and is not shared here to keep things clean.
C++ code for OpenCV Facemark
#include <opencv2/opencv.hpp>
#include <opencv2/face.hpp>
#include "drawLandmarks.hpp"
using namespace std;
using namespace cv;
using namespace cv::face;
int main(int argc,char** argv)
{
// Load Face Detector
CascadeClassifier faceDetector("haarcascade_frontalface_alt2.xml");
// Create an instance of Facemark
Ptr<Facemark> facemark = FacemarkLBF::create();
// Load landmark detector
facemark->loadModel("lbfmodel.yaml");
// Set up webcam for video capture
VideoCapture cam(0);
// Variable to store a video frame and its grayscale
Mat frame, gray;
// Read a frame
while(cam.read(frame))
{
// Find face
vector<Rect> faces;
// Convert frame to grayscale because
// faceDetector requires grayscale image.
cvtColor(frame, gray, COLOR_BGR2GRAY);
// Detect faces
faceDetector.detectMultiScale(gray, faces);
// Variable for landmarks.
// Landmarks for one face is a vector of points
// There can be more than one face in the image. Hence, we
// use a vector of vector of points.
vector< vector<Point2f> > landmarks;
// Run landmark detector
bool success = facemark->fit(frame,faces,landmarks);
if(success)
{
// If successful, render the landmarks on the face
for(int i = 0; i < landmarks.size(); i++)
{
drawLandmarks(frame, landmarks[i]);
}
}
// Display results
imshow("Facial Landmark Detection", frame);
// Exit loop if ESC is pressed
if (waitKey(1) == 27) break;
}
return 0;
}
Facemark Demo
Here is a video demo of the above code.
Thank you, Satya, for another wonderful post.
I wonder if there is any comparison (accuracy/performance) between these algorithms and the one in Dlib? An OpenCV-based landmark detection will at least save some time on converting the cv::Mat to dlib::array2d.
Thanks. FacemarkKazemi is the exact same algorithm as Dlib’s, and if trained on the same data, the accuracy should be close. LBF and Kazemi papers both appeared in 2014 and have very close error measures on the 68-point dataset ( Kazemi : 0.064, LBF : 0.0632). This is assuming the implementation is close to the original author’s implementation.
What about the performance? OpenCV-based implementation is expected to be faster?
I don’t know for sure yet.
Is there a way to convert standard 68 landmark model that is available for free in dlib? Or it does not make any sense? Thanks
It does make sense because the algorithm is the same (FacemarkKazemi) and so one can write a converter that converts one model to the other. Although, I suspect, it would be just simpler to train a model from scratch which I plan to do soon.
Main long term issue would be, if it is technically feasible, that dlib is free to change its algorithms/formats…
Well, this is a good new, and I am very thankful of it.
I saw one disadvantage of dlib (though , if I had time, reading its examples would be more than useful and teach me c++ -at least-) :
it takes a bunch of time to compile on a nanopi (dlib compilation needs huge preprocessor? ressources ) ;
this has nothing to do with real time detection, but is annoying if one tries to adapt existing software to one’s needs (ex : using facial landmarks to deduce the extend of forehead and -maybe hair-, then separating (grabcut) faces from background, having “better ” averages and eigenfaces (less variance being explained by background : it takes a long time testing and typo chasing, just before decidiing whether means and eigen faces look “better”)
Compiling Dlib takes a bit of time, but nothing compared to compiling OpenCV. Also, it is only a one time effort, and after that you can simply link to the libraries.
Thanks a lot für this excellent post once again.
I have a question about landmark model. Do all algorithms use the same 68 landmark model with the same indexes. What is the best refernece on that?
Regards
Marcus
Thanks. You can train a landmark detector with any number of points. But most people use standard public datasets for training and therefore the order of the points is the same for all the models.
受益匪浅,感触良多!
Thank you, Doctor Satya, for your fabulous post.
When i downloaded your source with git and tried to build,
OpenCV(3.4.1-dev) Error: Assertion failed (mv[i].size == mv[0].size && mv[i].depth() == depth) in cv::merge, file e:opencvfinalopencv-mastermodulescoresrcmerge.cpp, line 458
(in all of source) I got the error message as above, could you help me?
Is your image grayscale by any chance?
https://uploads.disquscdn.com/images/749459a4c63085a4dfbe4f6284a473f5f12c87e99f276e2b68ca7641fecfca61.png
the preprocess for grayscale is already put on your code. right?
In order to debug this, please try and check the frame is read properly from the camera. Please let us know if it reads the frame from camera properly and still shows the error. Thanks!
Thanks Satya.
How to check a face to match with other face in dataset ? Could you help me about this ? This code, you just draw 68 point in a face. How to apply Facemark to recognize face
Face recognition is a whole different ball game. Facemark will not help you in that. However, it can be used as a preprocessing step for aligning faces to make recognition better.
You can align face following this article:
https://www.pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/
See this library, it’s use a landmark dataset to generate a face code, an array with 128 points, which is a unique one-sided code.
https://github.com/ageitgey/face_recognition
Thanks a lot for this great tutorial.
I got everything up and running on Window and MacOS.
But I fail building OpenCV for iOS with contribution modules that also include the face module.
I use the standard way with the python script:
build_framework.py –opencv ../.. –contrib ../../../opencv_contrib ./build.
The build goes through.
Everything links perfectly except for the functions from the face module.
Is there any restriction for the platform for this face module?
Regards
Marcus Hudritsch
Hi Marcus,
I have not tried it on iOS, so I am not sure why it does not work.
You may be aware that on iOS 11 and above, facial landmark detector is built into the operating system. Here is the link
https://medium.com/@kravik/ios-11-tutorial-vision-framework-3c836d5ecadd
Satya
Thx Satya, your’re great genious, you help us so much!
Thanks, Yuri.
Hi Satya,
Thanks for writing such informative articles.
I was trying to run this code but Could you please tell me in which module “opencv2/face.hpp” file is present in opencv?
Make sure you’ve OpenCV Installed along with opencv_contrib. opencv2/face.hpp is available in face module of opencv_contrib (Link : https://github.com/opencv/opencv_contrib/tree/master/modules/face/)
Hello Sir,
How did you compile the files? What command did you use?
Thank you!
For compilation :
g++ `pkg-config opencv –cflags –libs` -o facemark
./facemark
[Make sure you’ve the files in the current directory : haarcascade_frontalface_alt2.xml, lbfmodel.yaml]
Hello Doctor. I am honored to read your article. According to your code, I successfully implemented the facial landmark detection, but the accuracy of feature points on the contour of the face is not good. Is there any way to improve it? In addition, how do I train my own model?
The only way to improve this model will be to train your own model. We will write an article about it soon.
Thank you very much for your reply and look forward to learning your new article.
Thanks for the great article. I could able to test the landmarks. As a next step, I would like to extend this to recognise faces. Could you please write an article for that as well?
And based on my understanding, face recognition can be achieved by
1) Detect facial landmarks for all the images in a given folder
2) When a new image is given, compare new image’s face landmarks against the stored ones and say if that face can be recognised or not. For comparison, some kind of neighbour algorithm can be used.
I was able to do this way in Python. But I am clueless in how can I do this kind of comparison in Android. Given that I would like to develop an android app for face recognition, Is the above approach is correct or any other better approaches available?
Could you give any pointers?
Thanks in advance,
K
Hi K,
Unfortunately, face recognition using the method you described will not work well. You can use the Face Recognition implemented in OpenCV or a Deep Learning based approach like FaceNet
Dear Satya
we can use face landmarks for head pose estimation depending on the distance between points ?
For example, if (((landmarks[0][39].x + landmarks[0][42].x) / 2) – landmarks[0][27].x) > 0 so the face turned left, etc.
Best Regards
Hi Mohamad,
Please check our post on head pose estimation for the right way to do it
https://learnopencv.com/head-pose-estimation-using-opencv-and-dlib/
Satya
Hi, do you know is there any way that I could implement facial landmark detection using OpenCV in c# language? thank you
Support for C# is not there yet (for FacemarkLBF, Kazemi and AAM Algorithms). Stay in touch with OpenCV Tutorials and the upcoming releases for that.
How to train a custom landmark model?
Hi Raj,
We are still working on that post. Will post soon.
Satya
Hi, Satya,
I am very much looking forward to your post on training a custom-made landmark model. While it is still in the works, could you give some pointers to where we could find help in the documentation?
Thanks!
Marc
This is surely a great article for the beginners. I am facing the following problem while running the code. Can you tell the reason behind it please?
loading data from : C:Usersfs5veDocumentsvisual studio 2015Projectsfacelan
dmarkdetectionfacelandmarkdetectionlbfmodel.yaml
OpenCV(4.0.0-pre) Error: Unknown error code -49 (Input file is empty) in cvOpenF
ileStorage, file C:Usersfs5veDownloadsopencv-with-extraopencvmodulescore
srcpersistence_c.cpp, line 388
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.0.0-pre) Error: Unknown error code -49 (Input file is empty) in cvOpenF
ileStorage, file C:Usersfs5veDownloadsopencv-with-extraopencvmodulescore
srcpersistence_c.cpp, line 388
Please check your lbfmodel.yaml file, or else download the one from here: https://github.com/spmallick/GSOC2017/blob/master/data/lbfmodel.yaml and let us know if it works.
Thank you Dr. Satya Mallick
I have a question if you could help me with it:
i need a “facial component detector” which i can use it to store pixel’s information(pixels of components like: lip, mouth …) and separate facial components.
Can i use this land mark detectors above in your topic to do so? if yes, i’ll begin to learn how to work with, if no, could you guide me what can i do?
My best regards…Thanks in advance…!
Yes, you can use the landmark detected to find contours around facial components ( e.g. eye / nose / mouth etc. ). You can then use those contours to create binary masks for each component.
Thank you
i run
./facialLandmarkDetection haarcascade_frontalface_alt2.xml lbfmodel.yaml
loading data from : lbfmodel.yaml
and it error
OpenCV Error: Parsing error (lbfmodel.yaml(8365): Byte size not match elememt size) in icvYMLParseBase64, file /home/duadua/opencv-3.4.0/modules/core/src/persistence.cpp, line 1404
terminate called after throwing an instance of ‘cv::Exception’
what(): /home/duadua/opencv-3.4.0/modules/core/src/persistence.cpp:1404: error: (-212) lbfmodel.yaml(8365): Byte size not match elememt size in function icvYMLParseBase64
I am not quite sure. Have you tried OpenCV 3.4.1?
When will be Python bindings are available for the Facemark API. Are you working on it or anyone else?
We have opened an issue with OpenCV and helping them with testing it
https://github.com/opencv/opencv_contrib/issues/1661
However, the current fix, while it works is not satisfactory. So I am not sure when they will release it.
good afternoon, sorry to bother you, but I tried to execute this code and it marks me an error
undefined reference to `cv :: face :: FacemarkLBF :: Params :: Params () ‘
undefined reference to `cv :: face :: FacemarkLBF :: create (cv :: face :: FacemarkLBF :: Params,
in the part of
Ptr facemark = FacemarkLBF :: create ();
If you could help me I would be eternally grateful
Looks like you need to compile OpenCV with opencv_contrib.
i need x , y coordinates?if u have solution ,
cout<<x<<y; format.(landmark coordinates)
i need x , y coordinates?if u have solution ,
cout<<x<<y; format (landmark coordinates)
Nice
Thank you!
Can someone help me?
I has one error.
error LNK1112: module machine type ‘x64’ conflicts with target machine type ‘X86’ C:UsersLebedev’mlDocumentsVisual Studio 2013ProjectsProject1opencv_world342.lib(opencv_world342.dll) Project1
I download OpenCv for 64X
compilator can’t find bodies of those functions , where they describe?
1) cv::VideoCapture::read()
2)cv::imshow()
3)cv::cvtColor()
4)cv::circle()
5)cv::polylines()
6)cv::CascadClassifier::detectMultiScale()
You need to install OpenCV with opencv_contrib.
Hi Satya
I keep getting a “Exception thrown: read access violation. _Pnext was 0x7” error. This happens when I try load the “lbfmodel.yaml” model. It appears to be in the _Container_base12::_Orphan_all() function. Do you have any idea what could be causing this?
So I now know that the problem is the line”facemark->loadModel(“lbfmodel.yaml”);”. When I try load the model I get the error mentioned above. Is the model possibly in the wrong format? When I click download on github it opens the model data and then I right click save as. Is this the incorrect way to download the lbfmodel? If so how should I do it? Thank you for your time!
Hi Matthew,
I have not seen this problem before. Which version of OpenCV are you using?
Thanks
Satya
hi I am using version 4.2. but I have worked through the debug more and it is actually the the line before that is causing the program to crash. “// Create an instance of Facemark
Ptr facemark = FacemarkLBF::create();”. So its actually a problem with the pointer declaration. I’m currently using visual studio 2015. I’m going to update to 2017 and see if that is maybe the problem.
I am assuming it is 3.4.2. Please keep us informed how it goes so everybody can benefit. Thanks.
I had to change the following line: “Ptr facemark = FacemarkLBF::create();” to “Ptr facemark = createFacemarkLBF();”
Thanks. I have to look into this.
To check if the file is correct, you can see if it starts with
%YAML:1.0
thanks very much for such a post for beginners. I have installed the open cv 3.4.4 and open cv 3.4.3 contrib but won’t be able to execute the code
I’m receiving these
Severity Code Description Project File Line Suppression State
Error C3861 ‘circle’: identifier not found opencv-build template c:opencv-buildinstallincludeopencv2drawlandmarks.hpp 105
Severity Code Description Project File Line Suppression State
Error C3861 ‘faceDetector’: identifier not found opencv-build template c:opencv-buildinstallincludeopencv2coreutility.hpp 14
Severity Code Description Project File Line Suppression State
Error C2143 syntax error: missing ‘;’ before ‘<' opencv-build template c:opencv-buildinstallincludeopencv2dnndict.hpp 102
Hello Dr. Satya, is there a way I can store the landmark points found on each frame separately, I did try accessing the x and y members of the Point2f type vector but i got the error, “has no member function x”. Any help would be appreciated.
loading data from : lbfmodel.yaml
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(3.4.3) Error: Bad argument (No valid input file was given, please check the given filename.) in cv::face::FacemarkLBFImpl::loadModel
my program spews this error everytime I try to load the landmark model..I tried updating the .yaml file a few times but no change. I looked into the implementation of loadmodel and the file name string is correct . Any idea what may be going wrong?
Unhandled exception at 0x00007FF8AB2850D8 in FacialLandmarkDetection.exe: Microsoft C++ exception: cv::Exception at memory location 0x000000643F2FEAD0.