Pose Detection comparison : wrnchAI vs OpenPose

In our previous posts, we discussed how to perform Body and Hand pose estimation using the OpenPose library. Recently, as part of our consulting business, we got a chance to try the state-of-the-art pose-estimation system ( wrnchAI ) built by wrnch and compare it’s performance with OpenPose. We evaluated the Human Body Pose Estimation systems and report our findings by comparing wrnchAI vis-a-vis OpenPose on the following parameters :

1. Accuracy
2. Computation Speed
3. System Requirements
4. Model Size
5. Other Features
   • Mobile Support
   • Tracking Support.
   • Green Screen Segmentation.
   • Support for Application Development.
   • Licensing
   • Ease of Setup and Use

1. Accuracy Comparison

We use the COCO Dataset for the performance evaluation and comparison of wrnchAI with OpenPose. We use the following two datasets for evaluation of Accuracy.

COCO Validation Data ( COCO – Val )

It contains 5,000 images.

COCO Test-Dev Data ( COCO – Test-dev )

It contains 20,288 images.

1.1. Evaluation Metrics

COCO uses a metric called Object Keypoint Similarity(OKS). It gives a measure of how close the predicted Keypoint is with the ground truth. Higher OKS means higher overlap between predicted Keypoints and the ground truth.

COCO uses mean Precision and Recall as the primary metrics for evaluation for their yearly challenge. In addition, it also measures metrics like Precision and Recall at 50% and 75% OKS (AP_50, AP_75, AR_50, AR_75) and Precision and Recall for medium(AP_medium, AR_medium) and large objects (AP_large, AR_large).

We generate the results and use the COCO evaluation server for a fair evaluation.

1.2. Experimental Setup

We fix the shorter side of the image to a constant size ( net_size ) and resize the other side by preserving the aspect ratio. For example, suppose the net_size is kept at 320, and the input image is of height=640 and width=960, i.e. the image has an aspect ratio of 1.5; we resize the height to 320 and correspondingly resize the width to 480 ( maintaining the Aspect Ratio of 1.5 ).

For each dataset, we conduct two experiments : for two different input resolution. The net_size must be in multiples of 16 ( a requirement for OpenPose ). The two net_size used are 176 and 320.

1.3. Results on Validation Data

1.4. Results on Test-Dev Data

1.5. Observations

We can make the following observations from the above results and performance metrics. We have also given some visual results in support of the our observations.

1. The overall performance of both wrnchAI and OpenPose is similar for most of the metrics.

   Wrnch	OpenPose

2. wrnchAI performs better for smaller input images as suggested by the charts for input_size=176. This can also be inferred from the AP_medium and AR_medium numbers, which indicate how well a model performs for small to medium sized persons. wrnchAI outperforms OpenPose by ~4-10% for small-medium input images. 

   Wrnch	OpenPose

3. OpenPose performs better than wrnchAI for larger image size. OpenPose outperforms wrnchAI by ~2-4% for large input images.
4. On Visual inspection of the results we found the following :

• • wrnchAI is better at Keypoint Localization than the numbers suggest. 

    Wrnch	OpenPose

• • One of the factors why the numbers are low is that wrnchAI tries to predict key points for the occluded parts of the body. This is good for practical purposes but affects the performance numbers. 

    Wrnch	OpenPose

• • Another factor for the lower numbers for wrnchAI is False Positives. OpenPose handles False Positives slightly better than wrnchAI. 

    Wrnch	OpenPose

Overall, we can say that the accuracy of the two methods is similar, with wrnchAI being better at smaller images and OpenPose marginally better at large images.

2. Computation Speed Comparison

2.1. Experimental Setup

Given below is the system configuration used for evaluating the methods on the basis of computation speed.

Processor : Intel Core i7 6850K – 6 Core
RAM : 32 GB
GPU : NVIDIA GTX 1080 Ti with 11 GB RAM
OS : Linux 16.04 LTS
Programming Language : Python

We perform 2 experiments for evaluating the computation speed for both the models.

2.2. Speed comparison across different input size

We compute the time required to predict the key points for 1000 frames and divide it by 1000. This gives us the speed in terms of Frames per Second ( FPS ). The experiments are done at 3 different input resolution. The input resolutions used are – 320×224, 480×320, 720×480, 960×640.

Given below is the result of comparison of OpenPose and wrnchAI in terms of FPS.

Observations

It is clear that wrnchAI is much faster than OpenPose. wrnchAI is ~3.5x faster than OpenPose for small images and ~2x faster for medium to large images.

2.3. Speed Comparison for increasing number of persons

The goal of this experiment is to check if the inference time is dependent on the number of persons present, I.e. whether the time taken by the model increase if the number of people in the image increases. For this, we took images with different number of persons ranging from 1 to 20 and checked the inference time for each image. Given below is the graph showing the results of the experiment. The input resolution used was 480×320.