• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

MultiTracker : Multiple Object Tracking using OpenCV (C++/Python)

Satya Mallick
August 5, 2018 39 Comments
Application how-to Tracking Tutorial

August 5, 2018 By 39 Comments

In this post, we will cover how to use OpenCV’s multi-object tracking API implemented using the MultiTracker class. We will share code in both C++ and Python.

Before we dive into the details, please check previous posts listed below on Object Tracking to understand the basics of single object trackers implemented in OpenCV.

  1. Object Tracking using OpenCV
  2. GOTURN: Deep Learning based object tracking

Why do we need Multi Object Tracking

Most beginners in Computer Vision and Machine Learning learn about object detection. If you are a beginner, you may be tempted to think why do we need object tracking at all. Can’t we just detect objects in every frame?

Let’s explore a few reasons why tracking is useful.

First, when there are multiple objects (say people) detected in a video frame, tracking helps establish the identity of the objects across frames.

Second, in some cases, object detection may fail but it may still be possible to track the object because tracking takes into account the location and appearance of the object in the previous frame.

Third, some tracking algorithms are very fast because they do a local search instead of a global search. So we can obtain a very high frame rate for our system by performing object detection every n-th frame and tracking the object in intermediate frames.

So, why not track the object indefinitely after the first detection? A tracking algorithm may sometimes lose track of the object it is tracking. For example, when the motion of the object is too large, a tracking algorithm may not be able to keep up. So many real-world applications use detection and tracking together.

In this tutorial, we will focus on just the tracking part. The objects we want to track will be specified by dragging a bounding box around them.

MultiTracker : OpenCV’s Multiple Object Tracker

The MultiTracker class in OpenCV provides an implementation of multi-object tracking. It is a naive implementation because it processes the tracked objects independently without any optimization across the tracked objects.

Let’s go over the code step by step to find out how can we use OpenCV’s multi-object tracking API.

Download Code
To easily follow along this tutorial, please download code by clicking on the button below. It’s FREE!

Download Code

Step 1: Create a Single Object Tracker

A multi-object tracker is simply a collection of single object trackers. We start by defining a function that takes a tracker type as input and creates a tracker object. OpenCV has 8 different tracker types : BOOSTING, MIL, KCF,TLD, MEDIANFLOW, GOTURN, MOSSE, CSRT.

If you want to use the GOTURN tracker, please make sure to read this post and download the caffe model.

In the code below, given the name of the tracker class, we return the tracker object. This will be later used to populate the multi-tracker.

Python

from __future__ import print_function
import sys
import cv2
from random import randint

trackerTypes = ['BOOSTING', 'MIL', 'KCF','TLD', 'MEDIANFLOW', 'GOTURN', 'MOSSE', 'CSRT']

def createTrackerByName(trackerType):
  # Create a tracker based on tracker name
  if trackerType == trackerTypes[0]:
    tracker = cv2.TrackerBoosting_create()
  elif trackerType == trackerTypes[1]: 
    tracker = cv2.TrackerMIL_create()
  elif trackerType == trackerTypes[2]:
    tracker = cv2.TrackerKCF_create()
  elif trackerType == trackerTypes[3]:
    tracker = cv2.TrackerTLD_create()
  elif trackerType == trackerTypes[4]:
    tracker = cv2.TrackerMedianFlow_create()
  elif trackerType == trackerTypes[5]:
    tracker = cv2.TrackerGOTURN_create()
  elif trackerType == trackerTypes[6]:
    tracker = cv2.TrackerMOSSE_create()
  elif trackerType == trackerTypes[7]:
    tracker = cv2.TrackerCSRT_create()
  else:
    tracker = None
    print('Incorrect tracker name')
    print('Available trackers are:')
    for t in trackerTypes:
      print(t)
    
  return tracker

C++
Note: In addition to including opencv2/opencv.hpp, you also need to include opencv2/tracking.hpp.

#include <opencv2/opencv.hpp>
#include <opencv2/tracking.hpp>

using namespace cv;
using namespace std;

vector<string> trackerTypes = {"BOOSTING", "MIL", "KCF", "TLD", "MEDIANFLOW", "GOTURN", "MOSSE", "CSRT"}; 

// create tracker by name
Ptr<Tracker> createTrackerByName(string trackerType) 
{
  Ptr<Tracker> tracker;
  if (trackerType ==  trackerTypes[0])
    tracker = TrackerBoosting::create();
  else if (trackerType == trackerTypes[1])
    tracker = TrackerMIL::create();
  else if (trackerType == trackerTypes[2])
    tracker = TrackerKCF::create();
  else if (trackerType == trackerTypes[3])
    tracker = TrackerTLD::create();
  else if (trackerType == trackerTypes[4])
    tracker = TrackerMedianFlow::create();
  else if (trackerType == trackerTypes[5])
    tracker = TrackerGOTURN::create();
  else if (trackerType == trackerTypes[6])
    tracker = TrackerMOSSE::create();
  else if (trackerType == trackerTypes[7])
    tracker = TrackerCSRT::create();
  else {
    cout << "Incorrect tracker name" << endl;
    cout << "Available trackers are: " << endl;
    for (vector<string>::iterator it = trackerTypes.begin() ; it != trackerTypes.end(); ++it)
      std::cout << " " << *it << endl;
  }
  return tracker;
}

Step 2: Read First Frame of a Video

A multi-object tracker requires two inputs

  1. A video frame
  2. Location (bounding boxes) of all objects we want to track.

Given this information, the tracker tracks the location of these specified objects in all subsequent frames.

In the code below, we first load the video using the VideoCapture class and read the first frame. This will be used later to initialize the MultiTracker.

Python

  # Set video to load
  videoPath = "videos/run.mp4"
  
  # Create a video capture object to read videos
  cap = cv2.VideoCapture(videoPath)
 
  # Read first frame
  success, frame = cap.read()
  # quit if unable to read the video file
  if not success:
    print('Failed to read video')
    sys.exit(1)

C++

// set default values for tracking algorithm and video
  string videoPath = "videos/run.mp4";
  
  // Initialize MultiTracker with tracking algo
  vector<Rect> bboxes;

  // create a video capture object to read videos
  cv::VideoCapture cap(videoPath);
  Mat frame;

  // quit if unabke to read video file
  if(!cap.isOpened()) 
  {
    cout << "Error opening video file " << videoPath << endl;
    return -1;
  }

  // read first frame
  cap >> frame;
  

Step 3: Locate Objects in the First Frame

Next, we need to locate objects we want to track in the first frame. The location is simply a bounding box.

OpenCV provides a function called selectROI that pops up a GUI to select bounding boxes (also called a Region of Interest (ROI)).

In the C++ version, selectROI allows you to obtain multiple bounding boxes, but in the Python version, it returns just one bounding box. So, in the Python version, we need a loop to obtain multiple bounding boxes.

For every object, we also select a random color to display the bounding box.

The code is shown below.
Python

  ## Select boxes
  bboxes = []
  colors = [] 

  # OpenCV's selectROI function doesn't work for selecting multiple objects in Python
  # So we will call this function in a loop till we are done selecting all objects
  while True:
    # draw bounding boxes over objects
    # selectROI's default behaviour is to draw box starting from the center
    # when fromCenter is set to false, you can draw box starting from top left corner
    bbox = cv2.selectROI('MultiTracker', frame)
    bboxes.append(bbox)
    colors.append((randint(0, 255), randint(0, 255), randint(0, 255)))
    print("Press q to quit selecting boxes and start tracking")
    print("Press any other key to select next object")
    k = cv2.waitKey(0) & 0xFF
    if (k == 113):  # q is pressed
      break
  
  print('Selected bounding boxes {}'.format(bboxes))

C++

  // Get bounding boxes for first frame
  // selectROI's default behaviour is to draw box starting from the center
  // when fromCenter is set to false, you can draw box starting from top left corner
  bool showCrosshair = true;
  bool fromCenter = false;
  cout << "\n==========================================================\n";
  cout << "OpenCV says press c to cancel objects selection process" << endl;
  cout << "It doesn't work. Press Escape to exit selection process" << endl;
  cout << "\n==========================================================\n";
  cv::selectROIs("MultiTracker", frame, bboxes, showCrosshair, fromCenter);
  
  // quit if there are no objects to track
  if(bboxes.size() < 1)
    return 0;
  
  vector<Scalar> colors;  
  getRandomColors(colors, bboxes.size()); 

The getRandomColors function is rather simple

// Fill the vector with random colors
void getRandomColors(vector<Scalar>& colors, int numColors)
{
  RNG rng(0);
  for(int i=0; i < numColors; i++)
    colors.push_back(Scalar(rng.uniform(0,255), rng.uniform(0, 255), rng.uniform(0, 255))); 
}

Step 3: Initialize the MultiTracker

Until now, we have read the first frame and obtained bounding boxes around objects. That is all the information we need to initialize the multi-object tracker.

We first create a MultiTracker object and add as many single object trackers to it as we have bounding boxes. In this example, we use the CSRT single object tracker, but you try other tracker types by changing the trackerType variable below to one of the 8 tracker times mentioned at the beginning of this post. The CSRT tracker is not the fastest but it produces the best results in many cases we tried.

You can also use different trackers wrapped inside the same MultiTracker, but of course, it makes little sense.

The MultiTracker class is simply a wrapper for these single object trackers. As we know from our previous post, the single object tracker is initialized using the first frame and the bounding box indicating the location of the object we want to the track. The MultiTracker passes this information over to the single object trackers it is wrapping internally.

Python

  # Specify the tracker type
  trackerType = "CSRT"    

  # Create MultiTracker object
  multiTracker = cv2.MultiTracker_create()

  # Initialize MultiTracker 
  for bbox in bboxes:
    multiTracker.add(createTrackerByName(trackerType), frame, bbox)

C++

  // Specify the tracker type
  string trackerType = "CSRT";
  // Create multitracker
  Ptr<MultiTracker> multiTracker = cv::MultiTracker::create();

  // Initialize multitracker
  for(int i=0; i < bboxes.size(); i++)
    multiTracker->add(createTrackerByName(trackerType), frame, Rect2d(bboxes[i]));  

Step 4: Update MultiTracker & Display Results

Finally, our MultiTracker is ready and we can track multiple objects in a new frame. We use the update method of the MultiTracker class to locate the objects in a new frame. Each bounding box for each tracked object is drawn using a different color.

Python

  # Process video and track objects
  while cap.isOpened():
    success, frame = cap.read()
    if not success:
      break
    
    # get updated location of objects in subsequent frames
    success, boxes = multiTracker.update(frame)

    # draw tracked objects
    for i, newbox in enumerate(boxes):
      p1 = (int(newbox[0]), int(newbox[1]))
      p2 = (int(newbox[0] + newbox[2]), int(newbox[1] + newbox[3]))
      cv2.rectangle(frame, p1, p2, colors[i], 2, 1)

    # show frame
    cv2.imshow('MultiTracker', frame)
    

    # quit on ESC button
    if cv2.waitKey(1) & 0xFF == 27:  # Esc pressed
      break

C++

  while(cap.isOpened()) 
  {
    // get frame from the video
    cap >> frame;
  
    // Stop the program if reached end of video
    if (frame.empty()) break;

    //Update the tracking result with new frame
    multiTracker->update(frame);
  
    // Draw tracked objects
    for(unsigned i=0; i<multiTracker->getObjects().size(); i++)
    {
      rectangle(frame, multiTracker->getObjects()[i], colors[i], 2, 1);
    }
  
    // Show frame
    imshow("MultiTracker", frame);
    
    // quit on x button
    if  (waitKey(1) == 27) break;
    
   }

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Tags: MultiTracker object tracking OpenCV 3

Filed Under: Application, how-to, Tracking, Tutorial

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • RAFT: Optical Flow estimation using Deep Learning
  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy