fbpx

LearnOpenCV

Deep Learning based Human Pose Estimation using OpenCV

Vikas Gupta
May 29, 2018 149 Comments
Application Deep Learning OpenCV OpenCV DNN OpenCV Tutorials Pose

By 149 Comments

Deep Learning based Human Pose Estimation using OpenCV OpenPose. You can see the detected keypoint locations marked on the image.

In this tutorial, Deep Learning based Human Pose Estimation using OpenCV. We will explain in detail how to use a pre-trained Caffe model that won the COCO keypoints challenge in 2016 in your own application. We will briefly go over the architecture to get an idea of what is going on under the hood.

This post has been tested on OpenCV 4.2.

1. Pose Estimation (a.k.a Keypoint Detection)

Pose Estimation is a general problem in Computer Vision where we detect the position and orientation of an object. This usually means detecting keypoint locations that describe the object.

For example, in the problem of face pose estimation (a.k.a facial landmark detection), we detect landmarks on a human face. We have written extensively on the topic. Please see our articles on ( Facial Landmark Detection using OpenCV and Facial Landmark Detection using Dlib )

A related problem is Head Pose Estimation where we use the facial landmarks to obtain the 3D orientation of a human head with respect to the camera.

In this article, we will focus on human pose estimation, where it is required to detect and localize the major parts/joints of the body ( e.g. shoulders, ankle, knee, wrist etc. ).

Remember the scene where Tony stark wears the Iron Man suit using gestures?

If such a suit is ever built, it would require human pose estimation!

For the purpose of this article, though, we will tone down our ambition a tiny bit and solve a simpler problem of detecting keypoints on the body. A typical output of a pose detector looks as shown below :

The sample skeleton output of pose estimation, and detected keypoints marked on the image.
Figure 1 : Sample Skeleton output of pose estimation. Image Credit: Oliver Sjöström, Instagram: @ollivves, Website: https://ollivves.com/

1.1. Keypoint Detection Datasets

Until recently, there was little progress in pose estimation because of the lack of high-quality datasets. Such is the enthusiasm in AI these days that people believe every problem is just a good dataset away from being demolished. Some challenging datasets have been released in the last few years which have made it easier for researchers to attack the problem with all their intellectual might.

Some of the datasets are :

  1. COCO Keypoints challenge
  2. MPII Human Pose Dataset
  3. VGG Pose Dataset

If we missed an important dataset, please mention in the comments and we will be happy to include in this list!

2. Multi-Person Pose Estimation model

The model used in this tutorial is based on a paper titled Multi-Person Pose Estimation by the Perceptual Computing Lab at Carnegie Mellon University. The authors of the paper train a very deep Neural Networks for this task. Let’s briefly go over the architecture before we explain how to use the pre-trained model.

2.1. Architecture Overview

Image showing the architecure of the Multi Person Pose Estimation model.
Figure 2: Multi-Person Pose Estimation model architecture

The model takes as input a color image of size w × h and produces, as output, the 2D locations of keypoints for each person in the image. The detection takes place in three stages :

  1. Stage 0: The first 10 layers of the VGGNet are used to create feature maps for the input image.
  2. Stage 1: A 2-branch multi-stage CNN is used where the first branch predicts a set of 2D confidence maps (S) of body part locations ( e.g. elbow, knee etc.). Given below are confidence maps and Affinity maps for the keypoint – Left Shoulder.
The confidence maps and Affinity maps for the keypoint Left Shoulder are marked on the image.
Figure 3 : Showing confidence maps for Left Shoulder for the given image

The second branch predicts a set of 2D vector fields (L) of part affinities, which encode the degree of association between parts. In the figure below part affinity between the Neck and Left shoulder is shown.

Image showing the Part Affinity Maps for Neck-Left Shoulder pair for the given image.
Figure 4 : Showing Part Affinity maps for Neck – Left Shoulder pair for the given image

Stage 2: The confidence and affinity maps are parsed by greedy inference to produce the 2D keypoints for all people in the image.

This architecture won the COCO keypoints challenge in 2016.

2.2 Pre-trained models for Human Pose Estimation

The authors of the paper have shared two models – one is trained on the Multi-Person Dataset ( MPII ) and the other is trained on the COCO dataset. The COCO model produces 18 points, while the MPII model outputs 15 points. The outputs plotted on a person is shown in the image below.

Image showing the keypoints plotted which are detected using two different models - the COCO and the MPI models.

COCO Output Format Nose – 0, Neck – 1, Right Shoulder – 2, Right Elbow – 3, Right Wrist – 4, Left Shoulder – 5, Left Elbow – 6, Left Wrist – 7, Right Hip – 8, Right Knee – 9, Right Ankle – 10, Left Hip – 11, Left Knee – 12, LAnkle – 13, Right Eye – 14, Left Eye – 15, Right Ear – 16, Left Ear – 17, Background – 18 MPII Output Format Head – 0, Neck – 1, Right Shoulder – 2, Right Elbow – 3, Right Wrist – 4, Left Shoulder – 5, Left Elbow – 6, Left Wrist – 7, Right Hip – 8, Right Knee – 9, Right Ankle – 10, Left Hip – 11, Left Knee – 12, Left Ankle – 13, Chest – 14, Background – 15

You can download the model weight files using the scripts provided at this location.

3. Code for Human Pose Estimation in OpenCV

In this section, we will see how to load the trained models in OpenCV and check the outputs. We will discuss code for only single person pose estimation to keep things simple. As we saw in the previous section that the output consists of confidence maps and affinity maps. These outputs can be used to find the pose for every person in a frame if multiple people are present. We will cover the multiple-person case in a future post.

First, download the code and model files from below. There are separate files for Image and Video inputs. Please go through the README file if you encounter any difficulty in running the code.

3.1. Step 1 : Download Model Weights

Use the getModels.sh file provided with the code to download all the model weights to the respective folders. Note that the configuration proto files are already present in the folders.

From the command line, execute the following from the downloaded folder.

sudo chmod a+x getModels.sh
./getModels.sh

Check the folders to ensure that the model binaries (.caffemodel files ) have been downloaded. If you are not able to run the above script, then you can download the model by clicking here for the MPII model and here for COCO model.

3.2 Step 2: Load Network

We are using models trained on Caffe Deep Learning Framework. Caffe models have 2 files –

  1. .prototxt file which specifies the architecture of the neural network – how the different layers are arranged etc.
  2. .caffemodel file which stores the weights of the trained model

We will use these two files to load the network into memory.

Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE!

C++

// Specify the paths for the 2 files
string protoFile = "pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt";
string weightsFile = "pose/mpi/pose_iter_160000.caffemodel";

// Read the network into Memory
Net net = readNetFromCaffe(protoFile, weightsFile);

Python

# Specify the paths for the 2 files
protoFile = "pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt"
weightsFile = "pose/mpi/pose_iter_160000.caffemodel"

# Read the network into Memory
net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)

3.3. Step 3: Read Image and Prepare Input to the Network

The input frame that we read using OpenCV should be converted to a input blob ( like Caffe ) so that it can be fed to the network. This is done using the blobFromImage function which converts the image from OpenCV format to Caffe blob format. The parameters are to be provided in the blobFromImage function. First we normalize the pixel values to be in (0,1). Then we specify the dimensions of the image. Next, the Mean value to be subtracted, which is (0,0,0). There is no need to swap the R and B channels since both OpenCV and Caffe use BGR format.
C++

//
Mat frame = imread("single.jpg");
// Specify the input image dimensions
int inWidth = 368;
int inHeight = 368;

// Prepare the frame to be fed to the network
Mat inpBlob = blobFromImage(frame, 1.0 / 255, Size(inWidth, inHeight), Scalar(0, 0, 0), false, false);

// Set the prepared object as the input blob of the network
net.setInput(inpBlob);

Python

# Read image
frame = cv2.imread("single.jpg")

# Specify the input image dimensions
inWidth = 368
inHeight = 368

# Prepare the frame to be fed to the network
inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), (0, 0, 0), swapRB=False, crop=False)

# Set the prepared object as the input blob of the network
net.setInput(inpBlob)

3.4. Step 4: Make Predictions and Parse Keypoints

Once the image is passed to the model, the predictions can be made using a single line of code. The forward method for the DNN class in OpenCV makes a forward pass through the network which is just another way of saying it is making a prediction.

C++

Mat output = net.forward()

Python

output = net.forward()

The output is a 4D matrix :

  1. The first dimension being the image ID ( in case you pass more than one image to the network ).
  2. The second dimension indicates the index of a keypoint. The model produces Confidence Maps and Part Affinity maps which are all concatenated. For COCO model it consists of 57 parts – 18 keypoint confidence Maps + 1 background + 19*2 Part Affinity Maps. Similarly, for MPI, it produces 44 points. We will be using only the first few points which correspond to Keypoints.
  3. The third dimension is the height of the output map.
  4. The fourth dimension is the width of the output map.

We check whether each keypoint is present in the image or not. We get the location of the keypoint by finding the maxima of the confidence map of that keypoint. We also use a threshold to reduce false detections.

Once the keypoints are detected, we just plot them on the image.
C++

    int H = output.size[2];
    int W = output.size[3];

    // find the position of the body parts
    vector<Point> points(nPoints);
    for (int n=0; n < nPoints; n++)
    {
        // Probability map of corresponding body's part.
        Mat probMap(H, W, CV_32F, output.ptr(0,n));

        Point2f p(-1,-1);
        Point maxLoc;
        double prob;
        minMaxLoc(probMap, 0, &prob, 0, &maxLoc);
        if (prob > thresh)
        {
            p = maxLoc;
            p.x *= (float)frameWidth / W ;
            p.y *= (float)frameHeight / H ;

            circle(frameCopy, cv::Point((int)p.x, (int)p.y), 8, Scalar(0,255,255), -1);
            cv::putText(frameCopy, cv::format("%d", n), cv::Point((int)p.x, (int)p.y), cv::FONT_HERSHEY_COMPLEX, 1, cv::Scalar(0, 0, 255), 2);

        }
        points[n] = p;
    }

Python

H = out.shape[2]
W = out.shape[3]
# Empty list to store the detected keypoints
points = []
for i in range(len()):
    # confidence map of corresponding body's part.
    probMap = output[0, i, :, :]

    # Find global maxima of the probMap.
    minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)

    # Scale the point to fit on the original image
    x = (frameWidth * point[0]) / W
    y = (frameHeight * point[1]) / H

    if prob > threshold :
        cv2.circle(frame, (int(x), int(y)), 15, (0, 255, 255), thickness=-1, lineType=cv.FILLED)
        cv2.putText(frame, "{}".format(i), (int(x), int(y)), cv2.FONT_HERSHEY_SIMPLEX, 1.4, (0, 0, 255), 3, lineType=cv2.LINE_AA)

        # Add the point to the list if the probability is greater than the threshold
        points.append((int(x), int(y)))
    else :
        points.append(None)

cv2.imshow("Output-Keypoints",frame)
cv2.waitKey(0)
cv2.destroyAllWindows()
Image plotted with keypoints detected when using the MPI model.
Image with kepoints detected using the MPI model plotted.

3.5. Step 5: Draw Skeleton

Final image with the output skeleton marked on it. The skeleton is drawn by joining the pairs using the indices of the points.

Since we know the indices of the points before-hand, we can draw the skeleton when we have the keypoints by just joining the pairs. This is done using the code given below.

C++

    for (int n = 0; n < nPairs; n++)
    {
        // lookup 2 connected body/hand parts
        Point2f partA = points[POSE_PAIRS[n][0]];
        Point2f partB = points[POSE_PAIRS[n][1]];

        if (partA.x<=0 || partA.y<=0 || partB.x<=0 || partB.y<=0)
            continue;

        line(frame, partA, partB, Scalar(0,255,255), 8);
        circle(frame, partA, 8, Scalar(0,0,255), -1);
        circle(frame, partB, 8, Scalar(0,0,255), -1);
    }

Python

for pair in POSE_PAIRS:
    partA = pair[0]
    partB = pair[1]

    if points[partA] and points[partB]:
        cv2.line(frameCopy, points[partA], points[partB], (0, 255, 0), 3)

Do checkout the Video demo using the video version of the code. We found that COCO model is 1.5 times slower than the MPI model. This is expected as we are using a stripped down version having 4 stages.

If you have ideas of some cool applications using these methods, do mention them in the comments!



References and Further Reading

Original Youtube Video Link used in the Sample Video
OpenPose
Pose Detection paper
Realtime multi-person Pose Estimation
OpenCV DNN Module
Loading Caffe models in OpenCV

Comments

  1. Satay,
    Another really interesting blog. Any tips for those of us who might run this under Windows? This looks to be aimed at the Linux world.

    Thanks,
    Doug

    Reply

      • Thank you for your interesting tutorial, after running “forward” function in C++ (Visual Studio 2017, OpenCV 3.4.0 and Windows X64)
        Mat output = net.forward();
        I get unhandled exception, and following error:

        OpenCV Error: Assertion failed (output_slice.isContinuous() && output_slice.size == curr_output.size) in fuseLayers, file C:buildmaster_winpack-build-win64-vc15opencvmodulesdnnsrcdnn.cpp, line 1430

        some users on the net said it only works with OpenCV 3.4.1, so I wondered is it because of OpenCV version?

        Could you please let me know what version of OpenCV is used to test the code?

        Thank you.

        Reply

      • sir i have been subscribed you but i did’t get the code please tell me where is the code please share me the link

        Reply

  2. Hi Satya,
    Thanks for sharing. Do you have any speed benchmarks for both models ? frame/sec for resolution

    Reply

    • for a resolution of 368×368,
      i7 and SSD – It takes 0.7 sec for MPI model and ~1 sec for COCO model.
      i7 and HDD – it takes 1.6 sec for MPI and 2.3 sec for COCO
      i5 and HDD – it takes ~6 sec for MPI

      Reply

  3. Hi Satya,
    Thanks for the quick response, I do have 3.4.1 OpenCV and Contrib built and installed. Obviously the shell script is not too useful as wget doesn’t exist in OpenCV. Found using CMake-Gui was need as I have multiple version of CUDA installed.

    Looks like all is well. As a point of reference on my system there is a bit ober 1sec difference between the two models per frame.

    Thanks,
    Doug

    Reply

    • Hi Doug, You can use the links given in Step 3.1 to download the models. Did you check how much time it takes per frame with/without GPU?

      Reply

  5. Hello Sir, I want to create a face aligner in 3D co-ordinates and frontal face alignment in 3D. How should I proceed? If a face is tilted towards left, right or any other direction the output should be frontal face and that to in 3D can you please suggest my method for this.
    thanks in Advance

    Reply

  6. Dear Satya or Vikas,
    Thank you for this ‘whetting the appetite’ introduction to gestures.
    My question is – suppose we want to analyse hand and finger gestures. That is recognize movement from the palm or dorsum (other side of the hand), fist and finger gestures – whether on the palm side or dorsum side, Can this current tutorial handle/estimate hand and finger gestures or where does one get a database of hand and finger gestures.
    Thank you,
    Anthony of Sydney.

    Reply

    • There is another model to do hand landmark detection. We are planning to have another post on that. In case it’s urgent, You can download the prototxt file from here and weights file from here.

      You can use the same code to get the hand landmarks, just by changing the model file names. Comment out the draw skeleton part. It will give you all the points on the fingers.

      Reply

      • Thank you for your reply, it is appreciated. My other question is related to speed of processing the images and the speed of displaying the skeleton/stick figures regardless of using the whole body (this tutorial) or hands (my question).

        In the above tutorial we see the moving picture of the dancer (top of the page) it appears that the skeleton/stick figure is moving in real time. Is it possible to get real-time displays of skeletons/stick figures?

        In other words, when you start the camera ‘rolling’ from time = t, when will I see in real time a person with skeleton/stick figure superimposed. If I wanted to do this on a Raspberry Pi, would I need the latest model 3?

        Thank you
        Anthony of Sydney

        Reply

        • You are right about the speed. It is not real-time. The model is very large, thus it takes a lot of time to process the image. Moreover, as far as I know, OpenCV does not allow using GPU for DNN module yet. o, raspberry pi is a BIG NO for now.

          You can instead use Tensorflow version of OpenPose with MobileNet version of Model, given here. This should run in near real-time even on raspberry-Pi.

          Reply

          • Dear Vikas,

            Thank you again for the reply. I also thank you for a future inclusion of a hand pose landmark tutorial described above.

            I go back (sorry) to my PC to attempt the exercise. I have Tensorflow, and I presume that I can load a PC version of OpenPose.

            Can the tutorial “Deep Learning based Human Pose Estimation using OpenCV” (this one) be conducted with Tensorflow and OpenPose and is there any necessary modifications to the code needed to be done. Given that near time processing and display can be done on an RPi, I presume that it may be faster with a PC.

            Thank you,

            Anthony of Sydney

          • Thanks Vikas! I took a look and it appears to be pretty much rooted into a Linux environment. I did not see a Windows port, which I would like to try, as I have TensorFlow-GPU running in Windows. That conclusion was reached by downloading a zip and running python setup.py install and it hangs. Has anyone successfully installed this under Windows that you know of? I do not think I will get any time to try as my plate is already overflowing! If someone gets this running under Windows please post here as I would love to know the secret.

            Thanks!

  7. Dear Satya,

    Awesome! Thanks for sharing a much easier version of OpenPose.
    May I ask you 2 things ?
    I did compile my OpenCV 3.4.1 with Cuda. But when I run the openposeimage or openposevideo it always starts with Initializing OpenCL Runtime.
    How can I change that to CUDA ?
    Second, do you have any blog such as : skin color, male/female, age, mood (happy, sad, anger, etc) ?

    Thank you once again!!!

    Reply

      • Hey Douglas, thanks for the reply.

        I thought they were totally different optimization tools.
        So, this frame rate that I am getting 2 frames per sec, is the best I can get in a I7, nVidia 1080 GTX Ti ? Sorry for the silly questions, I am new on GPU computing.

        Reply

        • We are still not clear on this. The discussion on this OpenCV issue says that DNN module supports GPU only through Halide Backend. Not through CUDA.

          Will update when we get a clear picture on the issue of “whether OpenCV DNN Modules use CUDA or not”.

          Reply

          • Thanks Vikas. One piece I can add is the last time I compiled 3.4.1 it went looking for the cuDNN library for Cuda 9.1. However, when I run OpenposeVideo and look at it with ProcExp, I see that it loads the Cuda Runtime, but I do not see cuDNN.

          • Thanks Doug, Might be helpful for someone. I don’t have a Windows system with a GPU to check 🙁

          • Thank you Douglas and Vikas! Although I didn’t understand half of what you guys said, I will study hard to catch up!
            So if I got this straight, so far the issue seems to be in OpenCV DNN handling or not CUDA.

        • That seems slow. On my laptop, i7 and a K5100M, which is not as powerful as your 1080, I am averaging about 2.4 seconds per frame. Vikas has some better numbers using a 1080Ti he might be better at answering this.

          Reply

  8. I want to show my sincere gratitude to your work on this page, I have definitely learnt a lot from you. I am sure you have helped many novices to start their great ideas and hopefully change the world. Thanks and continue the great content!

    Reply

  9. hi guys thanks for the awesome tutorial! i’m trying to understand what’s going on here is this model doing detection and tracking of a person? in the video it looks as if it knows it’s a single person that it has previously identified. can you clear this up pls? thanks again

    Reply

  10. @disqus_dqpsKEIOsm:disqus, thank you so much for this guide into the fascinating world of computer vision! I’d like to share my thoughts of how to apply such body estimation algorithm.

    You’re correct that there was some delay in solutions due no large datasets. I think that is why current approaches are too computationally consuming. Using OpenPose’s pose estimation network (MPI, 4 stages) I can achieve about 900ms per 368×368 frame (Intel® Core™ i5-4460 CPU @ 3.20GHz × 4). Even with acceleration backend ( https://github.com/opencv/opencv/wiki/Intel%27s-Deep-Learning-Inference-Engine-backend ) it takes no less than 550ms (the same CPU). One way is make it faster – do not process a whole image but crop a body using some person detector. This way resolution less than 368×368 can give more accurate results. There are a lot of lightweight SSD-based detectors for now. However most of them bound a body and could miss raised hands, in example.

    Despite an efficiency we can already use it offline (means not in real-time). In example, create a 2D skeleton animation. I’ve been trying to do an example ( https://github.com/dkurt/animator or https://github.com/dkurt/animator/tree/raw_anim ) but I just had no time to finish it =) Please find images attached (just a random image without any personal preferences to any of football teams). I hope it could be a good start for someone interested in after reading your tutorial.

    https://uploads.disquscdn.com/images/08d6d1b67824a586f0ca189a7d93d987802bf9bb6c1ce47785a1bfda2e5ab09e.png https://uploads.disquscdn.com/images/3cc0b6b66ae78bc28d2809f28be78f42d48b82c6080260e98e79686ca6bd5abd.png https://uploads.disquscdn.com/images/15728f0bbdecfb695b107bbfb5ad4f70ef1dc1870c6db3bb3ab637039c98af35.png

    Thanks again and good luck!

    Dmitry, OpenCV team member, Intel Corp.

    Reply

    • Glad to hear from you @dmitrykurtaev. Does it not support cuda enabled GPUs? I had a hard time finding whether it does or not. I saw this issue which kind of says cuda is not supported yet.

      Thanks for the Ideas.. This article would not have been possible, if not for you! Keep up the good work in OpenCV. Cheers!

      Reply

        • Hi, when I try net.setPreferableTarget(1), I get
          [ WARN:0] DNN: OpenCL target is not supported with current OpenCL device (tested with Intel GPUs only), switching to CPU.
          Anything I am doing wrong?

          Reply

          • Hi! Try to set an environment variable OPENCV_DNN_OPENCL_ALLOW_ALL_DEVICES to 1 (see dnn.cpp).

          • Thanks, really appreciate the reply!

            I still get the following messages:

            OpenCV(ocl4dnn): consider to specify kernel configuration cache directory via OPENCV_OCL4DNN_CONFIG_PATH parameter.
            OpenCL program build log: dnn/dummy
            Status -11: CL_BUILD_PROGRAM_FAILURE
            -cl-no-subgroup-ifp
            Error in processing command line: Don’t understand command line argument “-cl-no-subgroup-ifp”!

            I am a Python deep learning guy, don’t use C++ or OpenCV a lot, so I really don’t have much clue what all this means.

            But having said that, I have some interesting numbers:

            With OPENCV_DNN_OPENCL_ALLOW_ALL_DEVICES=1:

            Body keypoints forward pass: 0.76s (Coco)
            Face keypoints forward pass: 0.66s

            Without specifying the environment variable:

            Body keypoints forward pass: 2.53s (Coco)
            Face keypoints forward pass: 1.92s

            So about a ~3x speed up. Is this along expected lines?
            For a Pytorch version of OpenPose (from one Git repo – tensorboy), the speedup is like 2000x, but it’s super slow on CPU and just 0.025s (Coco) on CUDA GPU. The OpeCV dnn forward pass in comparison is pretty fast on CPU (just 1.5 to 2x the above numbers: 2.53 and 1.92).

            Thanks for reading the long comment. Would appreciate any response.
            My main question would be, is there any way OpenCV DNN could be as fast as 0.025s on CUDA GPUs?

            Thanks!

          • “is there any way OpenCV DNN could be as fast as 0.025s on CUDA GPUs”
            I think the short answer would be NO.

        • Hi @dmitrykurtaev:disqus I tried it ( with 3.4.1 as well as 4.0.0-pre )
          net.setPreferableTarget(DNN_TARGET_OPENCL) and
          export OPENCV_DNN_OPENCL_ALLOW_ALL_DEVICES=1

          Got similar performance as @chandrachudbasavaraj:disqus mentioned.

          Reply

  11. Thanks a lot
    I am having the following error
    TypeError: blobFromImage() takes at most 5 arguments (6 given)
    I appreciate if you advice How to solve it

    Thanks

    Reply

  12. Hi,

    For opencv version 3.4.0, I get an error in line “output = net.forward()”

    (-215) output_slice.isContinuous() && output_slice.size == curr_output.size in function cv::dnn::experimental_dnn_v3::Net::Impl::fuseLayers

    Reply

  13. How I can detect poses for multiple persons? If I have more than one then it ouputs “broken” pose where each person is used.

    Reply

    • It does output multi-person pose. But, the code needs to be modified to get them. I have mentioned this in the blog already

      Reply

      • “We will cover the multiple-person case in a future post”
        I cant find this article, maybe its not written?

        Do I need give seperate regions for multiple pose detection?

        Updated demo would be really appreciated . Thank You

        Reply

    • The output is a 4D matrix :

      The first dimension being the image ID ( in case you pass more than one image to the network ).
      The second dimension indicates the index of a keypoint. The model produces Confidence Maps and Part Affinity maps which are all concatenated. For COCO model it consists of 57 parts – 18 keypoint confidence Maps + 1 background + 19*2 Part Affinity Maps. Similarly, for MPI, it produces 44 points. We will be using only the first few points which correspond to Keypoints.
      The third dimension is the height of the output map.
      The fourth dimension is the width of the output map.

      Reply

      • sir one thing is confusing to me sir how does it konw about the point can we train our model on different pose

        Reply

  16. Hi Vikas / Satya,
    I am getting the following error when I run the following command:
    python OpenPoseVideo.py.

    [ INFO:0] Initialize OpenCL runtime…
    OpenCV Error: Assertion failed (output_slice.isContinuous() && output_slice.size == curr_output.size) in cv::dnn::experimental_dnn_v3::Net::Impl::fuseLayers, file C:projectsopencv-pythonopencvmodulesdnnsrcdnn.cpp, line 1430
    Traceback (most recent call last):
    File “OpenPoseVideo.py”, line 47, in
    output = net.forward()
    cv2.error: C:projectsopencv-pythonopencvmodulesdnnsrcdnn.cpp:1430: error: (-215) output_slice.isContinuous() && output_slice.size == curr_out
    put.size in function cv::dnn::experimental_dnn_v3::Net::Impl::fuseLayers

    My Python version:
    ‘3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]’
    My OpenCV version:
    ‘3.4.0’

    What am I doing wrong…?

    Thanks!
    Sachin

    Reply

  17. This is so helpful! thanks a lot for the post. Can you please make a post on multi-person pose detection also?

    Reply

  19. For taking measurement, you need a reference object in the image ( like a credit card ) at the same depth as the person. You can ask the person to hold the credit card on the palm or forehead so that it is easily detectable. Then you can find the distance between the points in pixels and convert to inches taking the credit card dimensions as reference. This is assuming that you know the credit card dimensions

    Reply

      • thanks for you reply sir.how to detect side pose in image because need to calculate chest and waist of human body?

        Reply

  22. when running the getmodels.sh code in ubuntu, i got the following error:

    ./getModels.sh: line 7: syntax error near unexpected token `newline’
    ./getModels.sh: line 7: `’

    Is there a more straightforward way to run this if I am using a PC?

    Reply

  23. Hi

    First off all, nice tutorial, pretty complete and interactive.

    on the other hand, I would like to ask some questions:

    1.-¿can I use this on windows 7?

    2.-¿How can I compile my projects directly on visual studio?

    Reply

  24. Hello Satya/Vikas and Team, Thank you for making very simple.
    I would like to know step by step from first step (images) to last step(pose on the image/video).
    I want to build the same thing with my custom images. how to create heat maps, how to join the key points.. and other steps…
    Please help me in this.

    Reply

  27. I like your posts!!!! I would like to learn about Hand-Keypoints
    Detection in Openpose like this post. Your
    explanation is very effective for me.

    Reply

  29. Thanks for your sharing. This web is useful.
    After run the program, I am wondering how to train a model to detect other objects, for example the hand. If you may provide another tutorial on that, it would be much better. Thanks.

    Reply

    • You can use the getModels.sh file to download the hand model. In the code you can change the model file names and use the same code for hand keypoints detection, with very small modifications!
      Regarding training, it should be done using other frameworks like Tensorflow / Caffe etc. OpenCV is to be used for inference or test time only.

      Reply

      • Sir, I have changed all the ‘pose’ to ‘hand’ in the .sh file, and I have tried to load the url into my web browser. However, they gave me 404 not found. I am not should whether the website is still available to not. Do you have any ideas?

        Reply

        • You don’t need to change anything in the .sh file. You should make changes in the python notebook file.
          The model is there at the link. You can use this link for downloading the model and this link for the prototxt file.

          Now, in the code, specify the paths to the hand model and prototxt file, along with other necessary changes and run the code.

          Reply

          • make these changes
            protoFile = “hand/pose_deploy.prototxt”
            weightsFile = “hand/pose_iter_102000.caffemodel”
            nPoints = 22
            POSE_PAIRS = [ [0,1],[1,2],[2,3],[3,4],[0,5],[5,6],[6,7],[7,8],[0,9],[9,10],[10,11],[11,12],[0,13],[13,14],[14,15],[15,16],[0,17],[17,18],[18,19],[19,20] ]

          • Sir, I am working on hand detection as part of my final year project for my undergraduate degree. It is really important to me. However, the model doesn’t work as well as I expect. I have more questions to ask to improve performance. May I send you emails to ask more questions?

  30. Plz help :
    Can’t open “pose/mpi/pose_iter_160000.caffemodel” in function ‘cv::dnn::ReadProtoFromBinaryFile’
    CV2 version 3.4.2 – Python
    Thanks in advance

    Reply

  31. i try to run this (Deep Learning based Human Pose Estimation using OpenCV ( C++ / Python )code in command prompt but i get this image. Please tell me why the key-points not appear in the image.

    Reply

  32. Dear Expertee in Pose Estimation, please im requring your assistance bcos i do not know where it get wrong after following the due process of what the above article and code said and in the Readme File i change what the readed said and im using python but the out come in both single picture and video are very very bad,

    please i need any body to help me out

    thanks for reading my comments https://uploads.disquscdn.com/images/f515fff1dd30e5b2c2e206f872ba6ef764fccddc41c520ba45fb60f0c8a95e0e.jpg https://uploads.disquscdn.com/images/f8913090dc827d850f87ff716892669888eeecb4aab133c6dbdaf920cc229d88.jpg

    And to produce this ugly pose is

    time taken by network : 17.194
    Total time taken : 17.506

    Reply

      • My project is about detecting human fall over(especially old people) so I want to adapt your code for my project but i’m not sure where should I change or adjust . Can u give a little suggestion?

        Reply

        • Nice Project!. You dont need the skeleton for doing this. Once you get the points from Step 3.4, You can check for certain points like neck, shoulders etc. If the y coordinates of most of them look same, then that means that probably the person has fallen.

          Reply

  37. Hi this is awesome and great. I want to know that when draw skeleton according to the human pose estimation can i cover that human with solid color? if it is yes how can i do it?

    Reply

  38. When I tried using the code, the output video was really slow, is there any way that I can make the output video faster?

    Reply

  39. @spmallick:disqus @disqus_dqpsKEIOsm:disqus Thanks for the great tutorial. I was wondering how these coordinates are measured.I already have pose coordinates. Suppose I want to know if a person is looking down. In this case I wanted to write a condition around neck,eyes,ears co-ordinates. Is there is a reference line against which these coordinates are measured? Sorry if my question does not make any sense.

    Reply

  40. @spmallick:disqus @disqus_dqpsKEIOsm:disqus Thanks for the great tutorial. is there a way to get the actions performed in the video using the pose estimated coordinates? I am using those coordinates to detect actions like sitting,standing. But it is not that accurate. Is there a better way to do it? Thank you

    Reply

Trackbacks

Leave a Reply to Walid Aly Cancel reply

Your email address will not be published. Required fields are marked *

Learn the state-of-the-art in AI: DALLE2, MidJourney, Stable Diffusion!
Claim Now
This course is available for FREE only till 22nd Nov.
FREE Python Course
We have designed this Python course in collaboration with OpenCV.org for you to build a strong foundation in the essential elements of Python, Jupyter, NumPy and Matplotlib.
  • We hate SPAM and promise to keep your email address safe.
FREE OpenCV Crash Course
We have designed this FREE crash course in collaboration with OpenCV.org to help you take your first steps into the fascinating world of Artificial Intelligence and Computer Vision. The course will be delivered straight into your mailbox.
  • We hate SPAM and promise to keep your email address safe.
 

Get Started with OpenCV

  • FREE OpenCV Crash Course
  • Getting Started Guides
  • Installation Packages
  • C++ And Python Examples
  • Newsletter Insights
  • We hate SPAM and promise to keep your email address safe.

Subscribe to receive the download link, receive updates, and be notified of bug fixes

seperator

Which email should I send you the download link?

  • We hate SPAM and promise to keep your email address safe.
Subscribe To Receive
  • FREE OpenCV Crash Course
  • Getting Started Guides
  • Installation Packages
  • C++ And Python Examples
  • Newsletter Insights
We hate SPAM and promise to keep your email address safe.​
  • We hate SPAM and promise to keep your email address safe.
Subscribe Now
Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

Getting Started
Course
Information
About LearnOpenCV

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products.

Read More
Copyright © 2023 – BIG VISION LLC