• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

CPU Performance Comparison of OpenCV and other Deep Learning frameworks

Satya Mallick
December 28, 2018 Leave a Comment
Deep Learning Image Classification Object Detection Performance Pose Tracking

December 28, 2018 By Leave a Comment

In this post, we will compare the performance of various Deep Learning inference frameworks on a few computer vision tasks on the CPU.

Surprisingly, with one exception, the OpenCV port of various deep learning models outperform the original implementation when it comes to performance on a CPU.

In this post, we will compare the CPU speed of the following Deep Learning algorithms :

  1. Image Classification
  2. Object Detection
  3. Object Tracking
  4. Pose Estimation
Become an expert in Computer Vision, Machine Learning, and AI in 12-weeks! Check out our course

Computer Vision Course

Testing Machine Configuration

Here is the configuration of the system we used.

  1. Machine : AWS t2.large instance. This instance has 2 vCPUs and 8 GB of RAM, but no GPU.
  2. Operating System : Ubuntu 16.04 LTS.
  3. OpenCV version : 3.4.3 ( installed using a slightly modified version of our OpenCV installation instructions ).
  4. Testing methodology : 100 cycles of the same test were performed and average time is reported.
Thanks to Vishwesh Ravi Srimali and Vikas Gupta who performed the experiments listed in this post.

Image Classification

The first application we compared is Image Classification on Caffe 1.0.0 , Keras 2.2.4 with Tensorflow 1.12.0, PyTorch 1.0.0 with torchvision 0.2.1 and OpenCV 3.4.3.

We used the pre-trained model for VGG-16 in all cases.

The results are shown in the Figure below.

PyTorch at 284 ms was slightly better than OpenCV (320ms). Keras came in third at 500 ms, but Caffe was surprisingly slow at 2200 ms.

Image Classification Speed Comparison on a CPU

Object Detection

The second application we chose was Object detection using YOLOv3 on Darknet.

The comparison was made by first importing the standard YOLOv3 object detector to OpenCV. If you are not familiar how to do this, please check out our post on Object detection using YOLOv3 and OpenCV

Darknet, when compiled without OpenMP, took 27.832 seconds per frame. Yes, that is not milliseconds, but seconds.

When compiled with OpenMP, Darknet was more than twice as fast with 12.730 seconds per frame.

But OpenCV accomplished the same feat at an astounding 0.714 seconds per frame.

Object Detection Speed Comparison on CPU

Object Tracking

The third application we tested was Object Tracking. For this, we chose a Deep Learning based object tracker called GOTURN. Now, this is not an apples-to-apples comparison because OpenCV’s GOTURN model is not exactly the same as the one published by the author. They were trained on different datasets. However, the underlying architecture is based on the same paper.

We compared the GOTURN Tracker in OpenCV with the Caffe based reference implementation provided by the authors of the GOTURN paper.

The OpenCV version ran at an impressive 50 ms per frame and was 6x faster than the reference implementation. Remember, these are both CPU implementations.

Object Tracking Speed Comparison on CPU

Pose Estimation

In the problem of Pose Estimation, given a picture of one or more people, you find the stick figure that shows the position of their various body parts.

If you have not read our post about Human Pose Estimation , you should check it out. We show how to import one of the best Pose Estimation libraries called OpenPose into an OpenCV application.

We compared the reference implementation of OpenPose in Caffe with the same model imported to OpenCV 3.4.3.

The reference implementation took 25.45 seconds while the OpenCV version took only 3.598 seconds.

Pose Estimation Speed Comparison on CPU

Why is OpenCV’s Deep Learning implementation on the CPU so fast? — The non-technical answer

In corporate America, whenever you see something unusual, you can find an answer to it by following the money.

Let’s start with the GPU story.

Which company is the top GPU seller in the world? Yup, it is NVIDIA. The company was very smart to realize the importance of GPUs in general purpose computing and more recently in Deep Learning.

In 2007, they released CUDA to support general purpose computing and in 2014 they released cuDNN to support Deep Learning on their GPUs. Consequently, the GPU implementation of all Deep Learning frameworks (Tensorflow, Torch, Caffe, Caffe2, Darknet etc.) is based on cuDNN. This has been a huge win for NVIDIA which has benefitted from the AI wave in addition to the cryptocurrency wave.

Now, which company is the top CPU seller in the world? Intel of course.

If you are new to OpenCV, you may not know OpenCV started at Intel Labs and the company has been funding its development for the most part. For a while, an independent company called Itseez was maintaining OpenCV, but recently it was acquired by — no points for guessing — Intel. The core OpenCV team is therefore at Intel.

As far as AI is concerned, Intel is in the inference business. Nobody uses Intel processors to train Deep Learning models, but a lot of people use their CPUs for inference. Therefore, Intel has a huge incentive to make OpenCV DNN run lightning fast on their CPUs.

Finally, the huge speed up also comes from the fact that the core team has deep optimization expertise on Intel CPUs.

OpenCV DNN : Insider’s knowledge

I had a call with Dmitry Kurtaev who is part of the core OpenCV team and has been working on the DNN module for about 2 years. He was kind enough to give me a quick overview of the DNN module. So here is some inside knowledge I acquired from Dmitry.

How it all started

I learned from Dmitry that the DNN module was started as part of Google Summer of Code (GSOC) by an intern Vitaliy Lyudvichenko who worked on it over two summers. Currently, the DNN module supports a few different backends

  1. Reference C++ implementation (default)
  2. Halide
  3. OpenCL
  4. OpenVINO

All results shown in this post used the reference C++ implementation.

Halide backend and why not to use it

There was an effort to make the DNN module faster using Halide but one fine day Vadim Pisarevsky optimized the reference CPU implementation to
the point that it was much faster than the Halide version! So, we are better off using the reference C++ implementation.

OpenCL backend and when to use it

The DNN module supports Intel GPUs with the OpenCL backend. Quite frankly, I am not impressed by the GPU support. The only silver lining is that OpenCV with OpenCL backend supports 16-bit floating point operations which can be 2x faster when using a GPU compared to the 32-bit version.

OpenVINO Toolkit

Remember, I mentioned how Intel has a huge incentive to make inference faster on CPUs. Well, the fastest DNN speed on Intel CPUs and other platforms like FPGAs and Neural Compute Stick is provided by Intel’s Open Visual Inference and Neural network Optimization ( OpenVINO ) toolkit. This will be covered in our next post.

Subscribe

If you liked this article, please subscribe to our newsletter. You will also receive a free Computer Vision Resource guide. In our newsletter, we share Computer Vision, Machine Learning and AI tutorials written in Python and C++ using OpenCV, Dlib, Keras, Tensorflow, CoreML, and Caffe.

Subscribe Now


Tags: deep learning Human Pose Estimation Image Classification Object Detection object tracking

Filed Under: Deep Learning, Image Classification, Object Detection, Performance, Pose, Tracking

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy