Who sees the human face correctly: the photographer, the mirror, or the painter? — Pablo Picasso
If Picasso was alive today, he would have definitely added one more profession to that list — a computer vision engineer!
As computer vision engineers and researchers we have been trying to understand the human face since the very early days. The most obvious application of facial analysis is Face Recognition. But to be able to identify a person in an image we first need to find where in the image a face is located. Therefore, face detection — locating a face in an image and returning a bounding rectangle / square that contains the face — was a hot research area. In 2001, Paul Viola and Michael Jones pretty much nailed the problem with their seminal paper titled “Rapid Object Detection using a Boosted Cascade of Simple Features.” In the early days of OpenCV and to some extent even now, the killer application of OpenCV was a good implementation of the Viola and Jones face detector.
Once you have a bounding box around the face, the obvious research problem is to see if you can find the location of different facial features ( e.g. corners of the eyes, eyebrows, and the mouth, the tip of the nose etc ) accurately. Facial feature detection is also referred to as “facial landmark detection”, “facial keypoint detection” and “face alignment” in the literature, and you can use those keywords in Google for finding additional material on the topic.
Applications of Facial Keypoint Detection
There are several interesting applications of keypoint detection in human faces. A few of them are listed below.
Master Generative AI for CV
Facial feature detection improves face recognition
Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same. Intuitively it makes sense that facial recognition algorithms trained with aligned images would perform much better, and this intuition has been confirmed by many research papers.
Head pose estimation
Once you know a few landmark points, you can also estimate the pose of the head. In other words you can figure out how the head is oriented in space, or where the person is looking. E.g. CLM-Framework described in this post also returns the head pose.
Face Morphing
Facial landmarks can be used to align faces that can then be morphed to produce in-between images. An example is shown in Figure 1.
Virtual Makeover
At my company ( TAAZ.com ) we had written our own facial landmark detector. The detected landmarks were used to the calculate contours of the mouth, eyes etc. to render makeup virtually. An example is shown in Figure 2.
Face Replacement
If you have facial feature points estimated on two faces, you can align one face to the other, and then seamlessly clone one face onto the other. You can also do something goofy like this
In a previous post, we showed how to use facial features to predict facial attractiveness.
Clearly, the ability to detect facial features in images and videos open up possibilities of a ton of interesting applications. Let us now get our hands dirty with some tools that will allow us to do this.
Facial Feature Detection & Tracking Libraries
There has been a flurry of activity in this area in the last 5 years. Part of the reason for this activity is the availability of large annotated datasets like LFPW and Helen. I have listed a bunch of papers in the next section. However, I do not recommend implementing these papers from scratch because now we have access to high quality open source implementations of some of these papers.
In the video below, you can see two of the libraries, Dlib and CLM-framework in action.
Dlib ( C++ / Python )
Dlib is a collection of miscellaneous algorithms in Machine Learning, Computer Vision, Image Processing, and Linear Algebra. Most of the library is just header files that you can include in your C++ application. Oh you prefer python ? No problem, it has a python API as well.
I personally like Dlib more than any other facial feature detection & tracking library because the code is very clean, well documented, the license permits use in commercial applications, the algorithm they have chosen to implement is very fast and accurate, and you can easily integrate the library in your C++ project by just including the header files.
How to compile Dlib ?
- Download a copy from github
git clone https://github.com/davisking/dlib.git
2. Build examples ( OSX / Linux )
cd dlib/examples mkdir build cd build cmake .. cmake --build . --config Release
These examples are a great way to start using dlib. Make a copy of an example cpp file, modify it, modify examples/CMakeLists.txt and compile again using the instructions above. Easy!
3. Compile dlib python module
cd dlib/python_examples ./compile_dlib_python_module.bat
4. Set PYTHONPATH environment variable
# Put the following line in .bashrc or .profile export PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH
5. Test python module
python -c "import dlib"
If the above line does not give an error, you are all set.
In case you run into any compilation issues, there are additional instructions at Dlib.net
How to run Dlib’s facial landmark detector ?
After you have built the examples, to run the facial landmark detector using a webcam, do the following.
cd examples/build/
#Download the face landmark model
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
tar xvjf shape_predictor_68_face_landmarks.dat.bz2
./webcam_face_pose_ex
If you want to run it on a single image, you can try
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
CLM-Framework (C++)
CLM-framework, also known as the Cambridge Face Tracker, is a C++ library for facial keypoint detection and head pose estimation. You can see how well it works in the included video. Compiling this library on OSX was bit of a challenge but it was not too bad. The library depends on OpenCV 3 and requires X11.
There are two important ways in which Dlib beats CLM-Framework. First, DLib is much faster than CLM-Framework. Second, Dlib’s license allows you to use it in commercial applications. If I had to pick, I would use Dlib. Interestingly, CLM-Framework depends on Dlib!
How to compile CLM-Framework ?
Compiling CLM-Framework was a bit involved for OSX. For windows and linux there are detailed instructions here. For compiling version 1.3.0 on OSX, I used the instructions for linux but made the following changes.
Most of the dependencies were installed using brew.
In file CMakeLists.txt ( red text was replaced with green text ).
find_package( OpenCV 3.0 REQUIRED )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)
In file exe/SimpleCLM/SimpleCLM.cpp
How to run CLM-Framework’s Facial Feature Detector ?
After compiling CLM-Framework, the executables are in the bin directory. For the webcam demo shown in this post, you can use
bin/SimpleCLM
Face++ ( FacePlusPlus ) : Web API
One of the best implementations of facial landmark detection is by FacePlusPlus. They won the 300 Faces In-the-Wild Landmark Detection Challenge, 2013. They provide an easy to use API. The problem is that you need to upload an image to their servers and that raises a lot of privacy concerns. But if privacy is not an issue, Face++ is very good option. You can see a demo at
http://www.faceplusplus.com/demo-landmark/
Facial Feature Detection Research
Many different approaches have been used to solve this problem and it is difficult to summarize them in a blog post. I am simply linking to some important papers ( with major bias toward recent work ) for people who want to investigate more.
- Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
- Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
- Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
- Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
- Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
- Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
- Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
- Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
- Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
- One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]
Nice write-up!
I actually explored a large number of open-source facial landmark detectors for a project, and found the CLM framework to outperform everything else (in terms of both speed and accuracy). We eventually used it in our project: http://www.technologyreview.com/news/541866/this-car-knows-your-next-misstep-before-you-make-it/
Some other interesting ones:
http://www.ics.uci.edu/~xzhu/face/
Very slow (~10 seconds an image after hyper threading on a 8-core CPU), but very accurate when it comes to high pose variations
https://github.com/patrikhuber/superviseddescent
Nicely written C++ code, though not very robust
http://www.vision.caltech.edu/xpburgos/ICCV13/
Specially designed for handling occlusions, but slow on account being written in MATLAB.
Thanks Avi. I had once implemented Supervised Descent in MATLAB and it gave very good results. Its theory is clean and easy to understand.
I had found and tested a few more, but decided to not include them in the post. See links below.
https://github.com/kylemcdonald/FaceTracker
https://github.com/uricamic/clandmark
Great Article
Everything installed on Ubuntu14.04.3 without a hitch. the example web application here: https://auduno.github.io/clmtrackr/examples/facesubstitution.html worked well with the web cam on my laptop howver the example: ./webcam_face_pose_ex did not, it produced:
~/dlib/examples/build$ ./webcam_face_pose_ex
Error detected at line 36.
Error detected in file /home/mark/dlib/dlib/../dlib/opencv/cv_image.h.
Error detected in function dlib::cv_image::cv_image(cv::Mat) [with pixel_type = dlib::bgr_pixel].
Failing expression was img.depth() == cv::DataType<typename pixel_traits::basic_pixel_type>::depth && img.channels() == pixel_traits::num.
The pixel type you gave doesn’t match pixel used by the open cv Mat object.
img.depth(): 0
img.cv::DataType<typename pixel_traits::basic_pixel_type>::depth: 0
img.channels(): 1
img.pixel_traits::num: 3
The following worked well with the image set from the lfw:
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
again, thanks for posting, much enjoyed
Mark
dlib uses OpenCV for the webcam demo. Are you sure your webcam works with OpenCV ?
plz plz plz …. write a post on aligning one face to another.i learnt alot from your post. i ‘m abhiginner in open cv want implement this thing in my face reco. program
Thanks Nilanjan. I am going step by step. Here is the next step
https://learnopencv.com/delaunay-triangulation-and-voronoi-diagram-using-opencv-c-python/
Hi,I want to build examples on mac using comment line “cmake –build .”but I got an error like this:
/Users/xxh/dlib/dlib/gui_widgets/nativefont.h:29:10: fatal error:
‘X11/Xlocale.h’ file not found
I don’t know how to fix it.
Hi Hank. You seem to be missing X11 for a mac. Download and install it from here http://www.xquartz.org/
Hi Satya, not only face morphing, how about virtual makeover? Could you share about that in following post?
Unfortunately, virtual makeover is something I cannot write about because my company would not like me to reveal our secret sauce :). It took years of fine tuning.
Oh, yes sorry fot that. I’m a CV engineer and want to make a makeover application at leisure time. So could you introduce some papers about this topic? or just some can-use but simple projects? Thank you!
Hi Satya, why not write a post on aligning one face to another 🙂 How to operate on the hair?
Thanks Pipi. Yes I will soon write an article about aligning faces. Here is a step forward in that direction https://learnopencv.com/delaunay-triangulation-and-voronoi-diagram-using-opencv-c-python/
Hi Satya, how can I detect lip movements ?I’m beginner and struggling
to understand it
Download dlib, you can track the landmarks. Compare the location of lip landmarks between two frames ( or maybe frames that are 10 frames apart ). This should tell you if the lip is moving.
you didn’t list up ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker
what a difference with this
ofxFaceTracker is based FaceTracker. I did try it out ( see my comment below ) and it did not give results anywhere close to the two approaches I have mentioned in this post.
excuse me, I want to do a face swapping. I detect the face with Viola Jones method, nut I don’t know how I should change it. could you please help me? I need a robust algorithm.
Face swapping requires many steps.
1. Use the landmark detector used in this post to detect landmarks in two faces.
2. Use delaunay triangulation to obtain a triangulation of the points.
3. Warp corresponding triangles from one face onto the other.
4. Seamlessly clone the warped face into the new image.
At some point I will write a post describing these steps in detail.
thanks for your reply. excuse me, could you please help me more? what their functions in opencv?
It can use the library “dlib” along with the opencv library?
There is a tutorial installation of dlib library with visual studio 2012?
Checkout the instructions here http://dlib.net/compile.html
Hello, i just looked into your post which is very interesting by the way, i am trying to compile the “dlib” library, i downloaded but when i try to run it, the example of face landmark detection it says that i do not have JPEG support, but i defined it on my source, i included the path to dlibexternallibjpeg on the include directory and also added all the files of that folder to my project together with the source.cpp file on the dlib folder, all of this on visual studio.
do you know maybe what i am doing wrong?
i wait for your answer. Regards from Mexico
Ricardo
Hi, thanks for the article, it was really helpful.
I got to run
the Dlib library and the project suggested in the article. Everything
went well. However, there is a huge difference between the frame rate
showed in your video and the example I ran on my computer. Is there any
config related to that or is it a matter of hardware?
Thanks in advance
First make sure you are NOT compiling with debug turned on. ( see here http://dlib.net/faq.html#Whyisdlibslow ). Second, most of the time is spent in displaying the lines on the video. If you don’t care about the display, turn it off and you will see a huge improvement. Third, dlib scales the frame by 2 and because this image is huge now, the processing speed decreases. Turn that off. Finally, I rewrote the display using OpenCV ( drawline ) and got further improvement. Hope this helps.
hi, Satya. Thanks for your answer.
I actually compiled it using cmake and ran this on the OSX terminal. By the way, I would like to compile it to Xcode project and use the landmark detection in another larger project, but I’m quite new to this image processing thing. Could you give me a hint of how compiling this to xcode?
Best,
Also a lot of time is spent in face detection ( and not landmark detection ). You can cheat and detect faces ever 5th frame but run the landmark detector on every frame using the location of the last detected face.
Hi!! Thanks for the information !
I am trying to create a lie detector with the movement of the iris so i hope this library let me detect and follow the iris
Very nice!
hi! can you use dlib on an existing project within XCode?
I used CMake in order to make it work with Xcode.
Has any one applied the techniques for other deformable objects other than faces?
I have applied a similar technique to other deformable objects. We were dealing with depth data. Unfortunately due to my confidentiality agreement I cannot talk about it in detail, but I am reasonably certain that this technique will work on other objects pretty well.
Thanks for your reply. I am trying extract keypoints to set of rigid objects (with lots of corners). I have to tune the parameters. Btw, how many labelled images does it generally require?
Thanks! Satya. I am trying extract keypoints to set of rigid objects (with lots of corners). I have to tune the parameters. Btw, how many labelled images does it generally require?
It depends a lot on the data. E.g. for facial landmark detection > 1000 images usually gives good results. But if you want a lot of non-frontal variations you can use about 2000 images. You can start with a few hundred images and have a large test set. If the algorithm fails on a test image, put that test image in the training set and train again after you have collected a hundred or so hard examples. This way you can systematically improve the quality of your results.
Thanks!
I’ve been trying to use train_shape_predictor_ex from Dlib, however it’s not working properly. In the xml file I have a list of images (each contain 68 fiducial points following the oder displayed in the attached picture). When I run face_landmark_detection_ex I get all points mixed up. Does anybody know the the fiducial points are arranged in wrong order?
How did you get the result of the last image? The one including the landmarks and their respectively point numbers?
Thanks in advance
I used TCDCN (http://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html), however I had to change the visualize.m file in order to represent facial landmarks with numbers instead of simple dots.
You might want try the imglab tool in the dlib tools to check if the points are in the same sequence. https://github.com/davisking/dlib/tree/master/tools/imglab
The images provided in the Dlib examples/faces folder are too small to see all facial landmarks in the imglab tool, however, they seem to be in the exactly same order. I just have no clue why they still look strange when I run face_landmark_detection_ex. The picture and the a XML file sample I’ve generated can be downloaded, respectively, at https://dl.dropboxusercontent.com/u/12547094/001_01.jpg and https://dl.dropboxusercontent.com/u/12547094/testing_with_face_landmarks_small.xml
The images provided in the Dlib examples/faces folder are too small to see all facial landmarks in the imglab tool, however, they seem to be in the exactly same order. I just have no clue why they still look strange when I run face_landmark_detection_ex. The picture and the a XML file sample I’ve generated can be downloaded at https://dl.dropboxusercontent.com/u/12547094/Archive.zip
Is the mixing up consistent for all images? Are the points locations correct?
How would you go about trying to find the top of the forehead? (Sorry I’m a beginner in CV)
I’d like to see a post on face morphing and substitution!
How about trying to use annotations to do that as well. There might be other approaches to do that but I think this might be more relevant to the discussion on the thread,
the step “std::vector faces = detector(cimg);” is too
cost time, need 25 seconds , but your demo is so fast, would you tell me
the reason, thank you very much.
hello how can i put the “PTS” into the “train.xml” by the imglab
Hi, I’m loving to follow your writings and it would be great if you could write an article about aligning faces 🙂
Thanks. Check out my post on face morphing ( https://learnopencv.com/face-morph-using-opencv-cpp-python/ ). With small modification, it will work for face alignment and face averaging.
when i build the train.xml ,the imglab only read the “*.jpg” but not “*.pts”, could you tell me how to put the annotations of the “*.pts” in the “train.xml ” by the imglab, or I should be write a new program for that， thanks。
Baokun : I have trained my own model but I can’t remember the details. I will look it up over the weekend and let you know.
thank you, I want train the model on the helen subset .the sunset includes “*.jpg” and “*.pts”
Is it possible to use http://dlib.net/train_shape_predictor_ex.cpp.html and http://dlib.net/face_landmark_detection_ex.cpp.html to training with different landmark numbers? Both detect 68 landmarks in a frontal face by default. I would like to detect 39 landmarks only (I can train all right using the train_shape_predictor_ex however face_landmark_detection_ex outputs the attached message.
Sorry for this very late reply. You have to change face_landmark_detection_ex.cpp so that it renders only 39 points and not 68 points.
I use qt on windows.I have included all header files and added source.cpp. What’s reason with the error?
#error “This file has been replaced. Instead you should add dlib/all/source.cpp to your project”
Hi I am currently working on a project where I have to extract the facial expression form video and images.
I tried Dlib but i didn’t know how to extract the coordinates of the corners of the eyes and mouth . Can you please help me.
Thanks
and can you please explain to me this output
are those the positions of the point compared to all the image or just the face??
when you detect facial landmark, you will receive 68 points. The points is fixed for every face. (i think so, because i check it with 3 faces). In your console, it only show 2 first points in received 68 points, and you can show more. To find the corners , you can choose left_most point and right_most point.
after calculating the distance between the points for the neutral face and for the face with an expression, how do we classifiy the results (how do i know witch emotion does the face express) ?
Hi sara. What do you use for learning algorithms? I think you need learning a distance area for each emotion face. If not, why don’t you use classification problem with superviser.
p/s: i am only a student.
i am just a newbie on this field so i don’t know exactly what to do next. should i use SVM for learning and what is the input for the learning ?
input is vector of faces. you try to use sift descriptor for each points which you just used facial landmark. Then, you classify all them by SVM.
EmoNets: Multimodal deep learning approaches for emotion recognition in video
http://arxiv.org/pdf/1503.01800v2
Hi. I just check with 2 faces. Result is:
sequence point: 17 oranges –>10 eyes brown –> 9 nose –>12 eyes–>20 mouth point == 68 point.
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Thank you Sir. Your instructions and explanations are really lucid and easy to follow.
Thanks Aakash 🙂
Sir,I am complete newbie to OpenCV. I have managed to configure code-blocks to compile and run OpenCV programs from
http://jonniedub.blogspot.in/2013/01/setting-up-codeblocks-ide-for-use-with.html
However i didn’t find any step by step instruction to configure it with the CLM Framework. Could you please help me.
Hello.
Are there analogs dlib on java?
I don’t know of a java library that does this, but I don’t usually code in Java. However, as you may be aware, you can use JNI to call C++ code from Java.
http://www.ibm.com/developerworks/java/tutorials/j-jni/j-jni.html
Hi, using dlib, can we able to save the image with face landmark overlay? I see it only displays in screen.
I am trying to get the face landmark from dlib then erase some of the facial features like eye brow, eye, mouth, … and fill that erased location with skin color.
Once you get the points, you can use OpenCV to do erasing / drawing. Check out OpenCV’s drawing methods here
http://docs.opencv.org/2.4/modules/core/doc/drawing_functions.html
Thanks Satya 🙂
Thanks for your great post, I have one question
“
Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same.
“
How to transform all faces to the mean face?
How to define the mean face shape?
Is there some code about this?
Thanks!
Thanks!
Yes, the post below and the associated code explains this alignment
https://learnopencv.com/average-face-opencv-c-python-tutorial/
The above post is about face averaging, but you have to do alignment before you can do averaging.
Thank you very much, I really learn a lot of things especially the code from your posts.
Thank you Sir. I’m newbie of OpenCV. Could you please write a tutorial using dlib for iOS?
Thats a great idea. I have been thinking about it for a while. Let me see if I can do that in the next month or two.
Thank Sir. Hope you could write it as soon as possible
So here is the code that does that does it
https://github.com/zweigraf/face-landmarking-ios
I have talked to the author. Someday if he gets a chance he may write an article for us.
How can I build as Objective-c
I have been waiting
thank you sir verry verrryy mutch your post was verry helpfull for me, and i will be glad if you can answer me:
i need to get the x or y coordinate separately for exemple when i write cout << shape.part(38)<< endl; it gives (160.115) but i want to have the 160 .
thank you sir verry much again and i wish good continuation for you.
Thanks.
You can just round the number round(shape.part(38))
http://www.cplusplus.com/reference/cmath/round/
thank you again sir thank you verry much !!!!!
it does not work for me :'( HELLLP PLEASE !!
when i tried with cout <<" the coordinate :" << round(shape.part(38))<< endl
i get x ans y coordinate together
but need to have x and y separately to continue my work HELLLP PLEASE !!
Hi Satya, thanks a lot for your wonderful post.
I am a beginner in opencv, so what I may be asking could be stupid, but is there a way to find the type of nose(pointed, hooked), color of the eyes, hair etc?, if so could you kindly post a way of how to do it
I wonder how DL approaches would perform for a custom classification task as opposed to the traditional CV approaches. How different would the algorithm be if the classification task /labels change.
Thanks for the reply sir, I didn’t quite understand what you meant, again I am a beginner in image recognization, I just wanted to know if it was feasible to categorize a nose(pointed, Roman), color of eyes, hair etc.
Sorry. Pls ignore my reply
Yes it is possible. To do that you have to get several images of pointed nose, Roman nose and other kinds of noses you want to classify. These days the state of the art algorithms for image classification are based on Deep Learning ( Google it ). There are some free frameworks to do Deep Learning. You can see my post on NVIDIA DIGITS here. The post is for beginners in mind.
https://learnopencv.com/deep-learning-example-using-nvidia-digits-3-on-ec2/
Thanks for the reply sir, I deeply appreciate it
Which of these approaches are the fastest on android if we could live with some loss in accuracy? Are there any open-source implementations for the same?
I would say Dlib is better, but I have not tried it on any other platform.
Thanks! Found this online. Need to check this out https://github.com/tzutalin/dlib-android
Very nice! Thanks for sharing.
Wc!
how can i use this landmark for facial emotion detection
You can use the library to get the facial landmarks, then match them to FACS (Facial Action Coding System). You can quantize the strength of the Action Unit between -1 and 1. Combinations of Action Units are marked as facial expressions. Use a database of emotion labeled portraits to extract the features of each emotion. Train a classifier to learn the emotion features and use it to classify on-line.
Thank
Here is another way to do it without using landmarks, https://github.com/mihaelacr/pydeeplearn
Thank you:)
Hello 🙂 I would like to print something on the screen using dlib functions (iOS project). Unfortunately, for some reason I can’t include in my project ‘dlib/gui_widgets.h’- I am getting the errors: ‘DLIB_NO_GUI_SUPPORT is defined so you can’t use the GUI code. Turn DLIB_NO_GUI_SUPPORT off if you want to use it.’ and ‘Also make sure you have libx11-dev installed on your system’. Do you have any other tip how I could print the coordinates of multiple points close to that points (drawn by ‘using draw_solid_circle’ function)? I would appreciate any help. 🙂
I am successfully done with “cmake –build . –config Release”, but still am not able to find “./compile_dlib_python_module.bat” in the folder “dlib/python_examples “. Could you please help me with it ?
You could use pip too. I am not sure if you could specify the optimization flag through pip though!
+1 I am having the same issue. There is no ‘compile_dlib_python_module.bat’ file in ‘dlib/python_examples’. What gives?
Thank you.!
did you solve this problem? I also have same issue..
Hi Satya, is there a way to extract the head pose (yaw, pitch and roll) from dlib’s result?
yup. That is my next post. Coming soon :).
But if you can’t wait go to my fork of dlib and see webcam_head_pose.cpp
https://github.com/spmallick/dlib/blob/master/examples/webcam_head_pose.cpp
I estimate a rotation matrix in that code using solvePnP. You can use the post below to convert rotation matrix to yaw, pitch, roll
https://learnopencv.com/rotation-matrix-to-euler-angles/
Hope that helps.
Satya
That’s great, I’m looking forward to the post. Meanwhile I’m gonna take a sneak peek into your code 🙂
Please edit this line :
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
this gives error of faces not being found. Since the executable is placed within build folder, the path to dataset becomes ../faces/*.jpg
Hence,
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat ../faces/*.jpg
Please correct me if I am wrong. I experienced this issue.
Thats right. Thanks for pointing out.
Hi Satya,
thanks for providing code to all your projects, I appreciate that!
Did you by any chance check out this project: https://github.com/ishay2b/VanillaCNN and know anything about performance and accuracy?
Since detecting the head pose of a profile face is quite hard for this algorithm (it works only when both eyes are visible). I was thinking to expand this tracking algorithm using multiple cameras (I think my application differs a bit). What would be your approach when using two or more cameras. Calibrating and independently evaluating and then comparing? That seems CPU-heavy. Any ideas or things you can point me to?
Many thanks!
We briefly looked at Deep Learning based approaches for Facial Landmark Detection but have not used dug deep into performance etc.
Multiple cameras add a lot of complexity. Before I answer that question, what application are you thinking about ?
Hello,
I am building an AAM (I noticed that AAMs are part of your research) , so I would like to ask you if I can use the Dlib library on Matlab in order to extract the shape parameters of a landmarked area from a picture.
Thank you.
Dlib is pure c++ and you can use mex to integrate it with MATLAB.
Hi Satya,
Really awesome library and thank you for you contribution.I’m implementing Dlib and it looks promising!
Is there any way to also detect hairline edge with Dlib? I tried to look into this but couldn’t find the right place. Appreciate if you can point me to the right direction.
Thank you
Hairline is tough. But if the hair is always pulled back, you can train a new landmark detector with a few points on the hairline.
Hi Satya, would love to learn how to train landmark detector for hairline. May you share some advice? Thank you!
Can I get more than 68 points of facial landmark. I see that in Face++, they can get 83 points. Is there any library help me find 83 points or even more (for example: some points on forehead)? Thanks!
You have to train your own model. Here is the code
http://dlib.net/train_shape_predictor_ex.cpp.html
Hey Satya, thanks for the great post and the instructions! This is so interesting that I wanted to try it myself. So I followed your instructions, and tried to build examples on OSX el capitan but failed. At the step “cmake –build . –config Release”, I get an error:
“[ 87%] Building CXX object CMakeFiles/dnn_imagenet_train_ex.dir/dnn_imagenet_train_ex.cpp.o
In file included from /Users/HomeFolder/dlib/examples/dnn_imagenet_train_ex.cpp:12:
In file included from /Users/HomeFolder/dlib/dlib/../dlib/dnn.h:14:
In file included from /Users/HomeFolder/dlib/dlib/../dlib/dnn/layers.h:8:
/Users/HomeFolder/dlib/dlib/../dlib/dnn/core.h:2444:45: fatal error: recursive template instantiation exceeded maximum depth of 256
using next_type = typename std::remove_reference::type;”
Do you happen to know why? I googled and looked at many posts from stackoverflow and none seems to be relevant to this issue. Thanks a lot!
How can I save eye area landmarks ?
Try this example. All landmarks are saved to a file.
https://github.com/spmallick/dlib/blob/master/examples/face_landmark_detection_to_file.cpp
Hello Satya Mallick,
Thank for your sharing.
I’ve built and run successfully some examples of Dlib such as face_detection_ex, face_landmark_detection_ex, webcam_face_pose_ex and your own example webcam_face_pose_fast.
But, on my PC, the examples don’t run smoothly as your above video. When I try to track facial landmark using webcam, it’s really slow. Maybe, just 3 or 4 frames per second.
What should I do to improve the performance? How can I allow the examples to use my GPU?
My System Information:
+ Windows 8.1 Pro
+ CPU i5-2500 3.3 GHZ (4 CPUs)
+ RAM 8G
+ GPU NVIDIA GeForce GTX 670
Thank you,
Please try the tricks described here and let me know if it helped.
https://learnopencv.com/speeding-up-dlib-facial-landmark-detector/
Thanks.
Hello there! Nice work!
The thing is i’m trying to get the landmarks from a video input (like a webcam), but i’m stucked on finding the code for Dlib-Python video landmark program (like the one you’re using in the video). All i found are designed for images. Can you please be kind and help me with some fast instructions or code advice?
Thanks!
Here you go.
http://dlib.net/webcam_face_pose_ex.cpp.html
Thanks for your fast response! But I’m not really sure how to implement it in python :(.
Nice library! I finally managed to run “./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat ../faces/*.jpg”. I can see the number of parts (which is 68) on screen. But where did all these landmarks go? Do I can get to store these information? Thanks!
You can checkout my fork of dlib that contains a cpp file that saves landmarks to disk
https://github.com/spmallick/dlib/blob/master/examples/face_landmark_detection_to_file.cpp
Thanks
Satya
Greetings. I’ve followed the instructions, but don’t know why there are no such ./webcam_face_pose_ex file. I run ls and get this list: 3d_point_cloud_ex matrix_expressions_ex
assignment_learning_ex max_cost_assignment_ex
bayes_net_ex member_function_pointer_ex
bayes_net_from_disk_ex mlp_ex
bayes_net_gui_ex model_selection_ex
bridge_ex mpc_ex
bsp_ex multiclass_classification_ex
CMakeCache.txt multithreaded_object_ex
CMakeFiles object_detector_advanced_ex
cmake_install.cmake object_detector_ex
compress_stream_ex one_class_classifiers_ex
config_reader_ex optimization_ex
custom_trainer_ex parallel_for_ex
dir_nav_ex pipe_ex
dlib_build pipe_ex_2
dnn_imagenet_ex quantum_computing_ex
dnn_imagenet_train_ex queue_ex
dnn_inception_ex random_cropper_ex
dnn_introduction2_ex rank_features_ex
dnn_introduction_ex running_stats_ex
dnn_mmod_dog_hipsterizer rvm_ex
dnn_mmod_ex rvm_regression_ex
dnn_mmod_face_detection_ex sequence_labeler_ex
empirical_kernel_map_ex sequence_segmenter_ex
face_detection_ex server_http_ex
face_landmark_detection_ex server_iostream_ex
fhog_ex shape_predictor_68_face_landmarks.dat
fhog_object_detector_ex sockets_ex
file_to_code_ex sockstreambuf_ex
graph_labeling_ex std_allocator_ex
gui_api_ex surf_ex
hough_transform_ex svm_c_ex
image_ex svm_ex
integrate_function_adapt_simp_ex svm_pegasos_ex
iosockstream_ex svm_rank_ex
kcentroid_ex svm_sparse_ex
kkmeans_ex svm_struct_ex
krls_ex svr_ex
krls_filter_ex threaded_object_ex
krr_classification_ex thread_function_ex
krr_regression_ex thread_pool_ex
learning_to_track_ex threads_ex
least_squares_ex timer_ex
linear_manifold_regularizer_ex train_object_detector
logger_custom_output_ex train_shape_predictor_ex
logger_ex using_custom_kernels_ex
logger_ex_2 video_tracking_ex
Makefile xml_parser_ex
matrix_ex
I’ve run all the instructions, which you have gave bellow((( What am i doing wrong?
this might be a silly question but how to run commands like these on cmd?
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
i m trying to run
./face_detection_ex faces/*.jpg
but it says
https://uploads.disquscdn.com/images/f08dde471997d751e8f59f934236d7abdedca58c82d5844e2d2d82bf4a369503.jpg
Hello @Marium it seems you’re trying to run those examples on Windows however these instructions are all for linux or mac osx. If you have built the examples on windows however you may try emitting the ./ from the begining and add .exe to the end of the program name once you are in the same folder as the examples. Just check in the folder to make sure that the examples have .exe at the end of their names otherwise they will not run on windows. If you need anymore help please run the command dir in cmd and post a screenshot of the output as a reply.
I am not able to successfully install dlib in my system.
I was able to make and get the dlib.so file generated. However, when I try to import dlib in my python code – I get the below error
ImportError: /usr/local/lib/python2.7/dist-packages/dlib-19.2.99-py2.7-linux-x86_64.egg/dlib/dlib.so: undefined symbol: _ZN4dlib12pixel_traitsINS_9rgb_pixelEE3numE
Thats a good one! Can you nm the dlib.so and see. You might also want to use C++filt to find the function prototype. It could be the order specified but thats strange.
This got resolved
https://github.com/davisking/dlib/issues/389
But what was the issue?
Hi SATYA MALLICK ,
Thanks for your great tutorial. But one thing I want to know. There is any example for export landmark in text file (for using with your Faceswap tutorial)?
Thanks!
Here is example code ( this was used to generate text files for dlib )
https://github.com/spmallick/dlib/blob/master/examples/face_landmark_detection_to_file.cpp
Satya
is there something similar for Python? If not, do you have suggestion to to access them in “shape” from “shape = predictor(img, d)”
and put them in list of tuples?
Did you find a way to do it in python? I need some help on that. Please can you let me know?
@spmallick:disqus Sir, render_face.hpp missing error is shown , please tell me how to solve this.
Hi Satya,
Thanks for your greate post! I have tested Dlib and found that it was really fast as I need. However, the face landmarks detected were jumping alot. Have you fix that problem? or could you recommend me how to fix it?
Thanks Hoang.
Here are a few suggestions
1. First check if the face detector bounding box is jumping around a lot. If so, you should not detect the face in each frame and use tracking instead ( Try MEDIANFLOW, https://learnopencv.com/object-tracking-using-opencv-cpp-python ). Alternatively you can use Kalman Filter to track the center of the box and that will smooth things out.
2. If the face box is constant, but the points are still jiggling you can use feature trackers. E.g. you can use calcOpticalFlowPyrLK to track a feature point. If the point tracked by optical flow and the landmark detector are within some threshold, you can use choose the location predicted by optical flow, else use the landmark detector result ( because the point might have moved significantly ).
Hope this helps.
Thanks Satya.
I think that using Optical Flow will be too slow with mobile device. However, the first suggesstion is a good idea. I will try this!
Keep in mind this is sparse optical flow. You are not calculating the flow on the entire image but only on 68 points using KLT .
Yep, I will try both suggesttions and reply you about the results. Thank you very much!
Hi, great article ! I’m wondering what kind of computer power is needed to run 24/7 facial landmarks calculations. I’m planning on runnning Debian, do I need a lot of horsepower ? Thanks a lot !
You don’t need a lot of horsepower. I can run this on Intel Core i7 ( 2.5 GHz ) in real time and if I don’t bother with the display of the image it is much much faster. The landmark detection is very fast 1000 fps and bottleneck the face detector needed to initialize it.
Hello, thanks for your quick answer, I really appreciate your help 🙂 I tried running it today on a dell Intel® Core™2 Quad CPU Q6600 @ 2.40GHz, and the result is catastrophic, I have about 8 seconds lag 😮 Any way to speed up the process other than buying a new computer ? Thanks a lot again for your time 🙂
Yes. Check this post.
https://learnopencv.com/speeding-up-dlib-facial-landmark-detector/
Just wondering how this compares with the 3000 fps paper. Ppl seem to achieve >200 fps. Haven’t tried it though! https://www.youtube.com/watch?v=TOVFOYrXdIQ
I want to querry landmark points using f200 realsense camera…Please help me on how to do it
Hola no me funciona para python ayuda por favor
hey guy, how can i make a lie detector with openCV? is there an algorithm? im using java. thanks
No, lie detection is not easily solvable using visual cues alone.
Hi, after the
facial landmark detection is there a way to extract the expression (happy , sad, anger … ) from dlib’s result?
Please help.. I have this problem..(ubuntu14.04)
[email protected]:~$ cd dlib/python_examples
[email protected]:~/dlib/python_examples$ ./compile_dlib_python_module.bat
bash: ./compile_dlib_python_module.bat: No such file or directory
Hello. I stuck.. please help.!
[email protected]:~/dlib/examples/build$ tar xvjf shape_predictor_68_face_landmarks.dat.bz2
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
—-> so I changed ‘tar xvjf’ to ‘bzip2 -d’
[email protected]:~/dlib/examples/build$ bzip2 -d shape_predictor_68_face_landmarks.dat.bz2
—-> it looks okay.
[email protected]:~/dlib/examples/build$ ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
bash: ./face_landmark_detection_ex: No such file or directory
—-> I don’t know how to solve this… please help.
Looks like you have not compiled. Here are the instructions.
http://dlib.net/compile.html
Hello, I’m using the dlib facial landmarks however the dots become unstable when I’m in a very clear environment or so if the background of the image is clear but works well where the background is dark like a black curtain for example. What can I do?
Hi Yan,
You can use stabilize the points using optical flow
http://docs.opencv.org/2.4/modules/video/doc/motion_analysis_and_object_tracking.html
In Facmorph example, I found points generated in a txt file for each image, how can I generate such text file for images?
68 points can be automatically generated using dlib’s facial landmark detector. But for that example, I hand marked a few more points.
How did you decide which additional 12 points were necessary to boost the effect?
I would also like to know
I am trying to execute this(https://github.com/spmallick/dlib) on my server (Ubuntu) and MAC terminal. Both cases I get this error-
(cv) MBP:build$ ./face_landmark_detection_to_file shape_predictor_68_face_landmarks.dat imgs/
No protocol specified
1463 FATAL [1] dlib.gui_core: Unable to connect to the X display.
exception thrown!
Failed to initialize X11 resources
—–
I have successfully done cmake and binaries have been generated. Please can you suggest on how to remove the dependencies of the X11? I have tried to do it, but, it throws up other exceptions(maybe I’m not doing it right)
Hi Satya,
I want to create my own face landmark detector that will detect about 90 points. I will prepare the file training_with_face_landmarks.xml according to my images and their associated landmark coordinates.
My question is: the code on page http://dlib.net/train_shape_predictor.py.html needs to be modified since I will detect more than 68 points?
Only the landmark rendering/display code needs to change.
hi SATYA MALLICK,
i configure the visual studio with the dlib library and i tested ./webcam_face_pose_ex and this works well, when i open the other example with a image, this does not show anything, and does not throw any errors
why?
What is the file name ?
the file is face_landmark_detection_ex, but when I run the program a black window appears and disappears without showing anything, also no mistakes
Hello. Thanks for this! 1: can I build a detection program to detect other than face and still go through the process you outlined? (I.e. morph two pants together)? 2: I don’t have the codes. I’ve subscribed. Thanks!
If you are looking for facial landmark detector code, it is in Dlib
http://dlib.net/
Please confirm your email address to receive the Welcome email.
Morphing something other than face is exactly the same as morphing face. You do need to build a shape predictor for that object. Here is an example code
http://dlib.net/train_shape_predictor_ex.cpp.html
Thanks! [email protected]
Hey Satya,
Thanks again for this! But I’m yet to receive the python file for this example with the politicians face morphing. Also, your dlib object detection technique is in C++ not python and I only use python :(. Please share these two codes sir if you can. Thanks and I appreciate that so much! [email protected].
Hello Satya Mallick,
Thanks for your tutorial. I have encountered some problems using dlib library on winform, when I include the header file in dlib, it causes heap corruption. How could I use this on winform?
Is there any open source library that gives more than 63 landmarks for the face.
I am looking for something similar to what Face++ API provides (83) landmarks that describe the face better and make its segmentation easier than the 63 landmarks.
For Example, sgementing the nose area from the 63 landmarks is not very effeicent.
Any suggestion Please?
I don’t know of any, but you can train Dlib to have any number of points.
Hi Satya,
Thanks a lot for the tutorial, I am able to run the “face_landmark_detection_ex” on different images but the problem is that it doesn’t save them in text format as you have in your tutorial, any idea how to save them in xyz.img.txt format
Can I save
cout << "pixel position of first part: " << shape.part(0) << endl;
cout << "pixel position of second part: " << shape.part(1) << endl;
to a file ?
I tried saving those values to file, but I getting the following error. One more points I wanted to ask my points are not matching with yours one, they are almost double.
OpenCV Error: One of arguments’ values is out of range () in locate, file /home/abhisek/Downloads/opencv-3.1.0/opencv-3.1.0/modules/imgproc/src/subdivision2d.cpp, line 286
terminate called after throwing an instance of ‘cv::Exception’
what(): /home/abhisek/Downloads/opencv-3.1.0/opencv-3.1.0/modules/imgproc/src/subdivision2d.cpp:286: error: (-211) in function locate
Aborted (core dumped)
Hi Satya,
If I half my points then the code works fine with new images also, was waiting for your reply, just wondering why I am getting the 2x values..
Are you resizing your image to speed up the process? Check if there is any image resizing going on.
hi, i have exactly the same issue! I resized the image… what does this mean? What is the problem?
Hey I want train my own images for face detection and recognition . Please mention the steps to do the same.
Thank you!
Hi , I’m trying to place a wig after facial landmark using dlib in ios. Any idea how can I find top position and try placing a wig.
Hello, I tried to implement Facial landmark detection in ios. I got the points. When I tried to use those points and place a UIView in ios the position is not matching. Any suggestion please ?
Question.. I’m trying to determine if the mount is “moving” across a series of frames in a video. How would you guys go about determining that? The tricky part is that the face could be moving.. so the face could be (but in my case, not very often) rotated, it could be turned slightly, or zooming in and out (i.e. person moving towards camera). So i guess you could look at the cluster of mouth points with respect to cluster of nose (or eye) points.
Any ideas?
hello thanku …this post is very helpful but before that i already find out the facial landmarks but now i want suggestion for how i can compare two facial images n find emotion …..n for comparison which algo/method is best.
can we crop only facial Landmark without the background?. Is it possible from CV:Mat object ?
Hi sir,
Could you do a short tutorial on how to retrain the shape_predictor_trainer model in Cpp? It’ll be a great help if we can learn how to reduce the number of landmarks to reduce the size of the .dat file, because 98MB is too large, especially on mobile phones, where Dlib is now becoming more and more popular.
Thanks for this amazing article. I want to know what paper implementations FACE++ uses in their work for “Facial Landmark Detection”?? Does anyone know? Thank you again