In this post, we will cover how to use OpenCV’s multi-object tracking API implemented using the MultiTracker class. We will share code in both C++ and Python.
Before we dive into the details, please check previous posts listed below on Object Tracking to understand the basics of single object trackers implemented in OpenCV.
Why do we need Multi Object Tracking
Most beginners in Computer Vision and Machine Learning learn about object detection. If you are a beginner, you may be tempted to think why do we need object tracking at all. Can’t we just detect objects in every frame?
Let’s explore a few reasons why tracking is useful.
First, when there are multiple objects (say people) detected in a video frame, tracking helps establish the identity of the objects across frames.
Second, in some cases, object detection may fail but it may still be possible to track the object because tracking takes into account the location and appearance of the object in the previous frame.
Third, some tracking algorithms are very fast because they do a local search instead of a global search. So we can obtain a very high frame rate for our system by performing object detection every n-th frame and tracking the object in intermediate frames.
So, why not track the object indefinitely after the first detection? A tracking algorithm may sometimes lose track of the object it is tracking. For example, when the motion of the object is too large, a tracking algorithm may not be able to keep up. So many real-world applications use detection and tracking together.
In this tutorial, we will focus on just the tracking part. The objects we want to track will be specified by dragging a bounding box around them.
Recently, more advanced state of the art trackers have come into light. One of them is DeepSORT, which uses an advanced association matrix than it’s predecessor, SORT. It uses YOLO network as the object detection model, and can save the person identities for upto 30 frames (default, can be changed). FairMOT uses joint detection and re-ID approach to give real time results. It can use YOLOv5 or DLA-34 as the backbone, and is superior in the re-identification task.
Master Generative AI for CV
MultiTracker : OpenCV’s Multiple Object Tracker
The MultiTracker class in OpenCV provides an implementation of multi-object tracking. It is a naive implementation because it processes the tracked objects independently without any optimization across the tracked objects.
Let’s go over the code step by step to find out how can we use OpenCV’s multi-object tracking API.
Step 1: Create a Single Object Tracker
A multi-object tracker is simply a collection of single object trackers. We start by defining a function that takes a tracker type as input and creates a tracker object. OpenCV has 8 different tracker types : BOOSTING, MIL, KCF,TLD, MEDIANFLOW, GOTURN, MOSSE, CSRT.
If you want to use the GOTURN tracker, please make sure to download the caffe model.
In the code below, given the name of the tracker class, we return the tracker object. This will be later used to populate the multi-tracker.
Python
from __future__ import print_function
import sys
import cv2
from random import randint
trackerTypes = ['BOOSTING', 'MIL', 'KCF','TLD', 'MEDIANFLOW', 'GOTURN', 'MOSSE', 'CSRT']
def createTrackerByName(trackerType):
# Create a tracker based on tracker name
if trackerType == trackerTypes[0]:
tracker = cv2.TrackerBoosting_create()
elif trackerType == trackerTypes[1]:
tracker = cv2.TrackerMIL_create()
elif trackerType == trackerTypes[2]:
tracker = cv2.TrackerKCF_create()
elif trackerType == trackerTypes[3]:
tracker = cv2.TrackerTLD_create()
elif trackerType == trackerTypes[4]:
tracker = cv2.TrackerMedianFlow_create()
elif trackerType == trackerTypes[5]:
tracker = cv2.TrackerGOTURN_create()
elif trackerType == trackerTypes[6]:
tracker = cv2.TrackerMOSSE_create()
elif trackerType == trackerTypes[7]:
tracker = cv2.TrackerCSRT_create()
else:
tracker = None
print('Incorrect tracker name')
print('Available trackers are:')
for t in trackerTypes:
print(t)
return tracker
C++
Note: In addition to including opencv2/opencv.hpp, you also need to include opencv2/tracking.hpp.
#include <opencv2/opencv.hpp>
#include <opencv2/tracking.hpp>
using namespace cv;
using namespace std;
vector<string> trackerTypes = {"BOOSTING", "MIL", "KCF", "TLD", "MEDIANFLOW", "GOTURN", "MOSSE", "CSRT"};
// create tracker by name
Ptr<Tracker> createTrackerByName(string trackerType)
{
Ptr<Tracker> tracker;
if (trackerType == trackerTypes[0])
tracker = TrackerBoosting::create();
else if (trackerType == trackerTypes[1])
tracker = TrackerMIL::create();
else if (trackerType == trackerTypes[2])
tracker = TrackerKCF::create();
else if (trackerType == trackerTypes[3])
tracker = TrackerTLD::create();
else if (trackerType == trackerTypes[4])
tracker = TrackerMedianFlow::create();
else if (trackerType == trackerTypes[5])
tracker = TrackerGOTURN::create();
else if (trackerType == trackerTypes[6])
tracker = TrackerMOSSE::create();
else if (trackerType == trackerTypes[7])
tracker = TrackerCSRT::create();
else {
cout << "Incorrect tracker name" << endl;
cout << "Available trackers are: " << endl;
for (vector<string>::iterator it = trackerTypes.begin() ; it != trackerTypes.end(); ++it)
std::cout << " " << *it << endl;
}
return tracker;
}
Step 2: Read First Frame of a Video
A multi-object tracker requires two inputs
- A video frame
- Location (bounding boxes) of all objects we want to track.
Given this information, the tracker tracks the location of these specified objects in all subsequent frames.
In the code below, we first load the video using the VideoCapture class and read the first frame. This will be used later to initialize the MultiTracker.
Python
# Set video to load
videoPath = "videos/run.mp4"
# Create a video capture object to read videos
cap = cv2.VideoCapture(videoPath)
# Read first frame
success, frame = cap.read()
# quit if unable to read the video file
if not success:
print('Failed to read video')
sys.exit(1)
C++
// set default values for tracking algorithm and video
string videoPath = "videos/run.mp4";
// Initialize MultiTracker with tracking algo
vector<Rect> bboxes;
// create a video capture object to read videos
cv::VideoCapture cap(videoPath);
Mat frame;
// quit if unabke to read video file
if(!cap.isOpened())
{
cout << "Error opening video file " << videoPath << endl;
return -1;
}
// read first frame
cap >> frame;
Step 3: Locate Objects in the First Frame
Next, we need to locate objects we want to track in the first frame. The location is simply a bounding box.
OpenCV provides a function called selectROI that pops up a GUI to select bounding boxes (also called a Region of Interest (ROI)).
In the C++ version, selectROI allows you to obtain multiple bounding boxes, but in the Python version, it returns just one bounding box. So, in the Python version, we need a loop to obtain multiple bounding boxes.
For every object, we also select a random color to display the bounding box.
The code is shown below.
Python
## Select boxes
bboxes = []
colors = []
# OpenCV's selectROI function doesn't work for selecting multiple objects in Python
# So we will call this function in a loop till we are done selecting all objects
while True:
# draw bounding boxes over objects
# selectROI's default behaviour is to draw box starting from the center
# when fromCenter is set to false, you can draw box starting from top left corner
bbox = cv2.selectROI('MultiTracker', frame)
bboxes.append(bbox)
colors.append((randint(0, 255), randint(0, 255), randint(0, 255)))
print("Press q to quit selecting boxes and start tracking")
print("Press any other key to select next object")
k = cv2.waitKey(0) & 0xFF
if (k == 113): # q is pressed
break
print('Selected bounding boxes {}'.format(bboxes))
C++
// Get bounding boxes for first frame
// selectROI's default behaviour is to draw box starting from the center
// when fromCenter is set to false, you can draw box starting from top left corner
bool showCrosshair = true;
bool fromCenter = false;
cout << "\n==========================================================\n";
cout << "OpenCV says press c to cancel objects selection process" << endl;
cout << "It doesn't work. Press Escape to exit selection process" << endl;
cout << "\n==========================================================\n";
cv::selectROIs("MultiTracker", frame, bboxes, showCrosshair, fromCenter);
// quit if there are no objects to track
if(bboxes.size() < 1)
return 0;
vector<Scalar> colors;
getRandomColors(colors, bboxes.size());
The getRandomColors function is rather simple
// Fill the vector with random colors
void getRandomColors(vector<Scalar>& colors, int numColors)
{
RNG rng(0);
for(int i=0; i < numColors; i++)
colors.push_back(Scalar(rng.uniform(0,255), rng.uniform(0, 255), rng.uniform(0, 255)));
}
Step 3: Initialize the MultiTracker
Until now, we have read the first frame and obtained bounding boxes around objects. That is all the information we need to initialize the multi-object tracker.
We first create a MultiTracker object and add as many single object trackers to it as we have bounding boxes. In this example, we use the CSRT single object tracker, but you try other tracker types by changing the trackerType variable below to one of the 8 tracker times mentioned at the beginning of this post. The CSRT tracker is not the fastest but it produces the best results in many cases we tried.
You can also use different trackers wrapped inside the same MultiTracker, but of course, it makes little sense.
The MultiTracker class is simply a wrapper for these single object trackers. As we know from our previous post, the single object tracker is initialized using the first frame and the bounding box indicating the location of the object we want to the track. The MultiTracker passes this information over to the single object trackers it is wrapping internally.
Python
# Specify the tracker type
trackerType = "CSRT"
# Create MultiTracker object
multiTracker = cv2.MultiTracker_create()
# Initialize MultiTracker
for bbox in bboxes:
multiTracker.add(createTrackerByName(trackerType), frame, bbox)
C++
// Specify the tracker type
string trackerType = "CSRT";
// Create multitracker
Ptr<MultiTracker> multiTracker = cv::MultiTracker::create();
// Initialize multitracker
for(int i=0; i < bboxes.size(); i++)
multiTracker->add(createTrackerByName(trackerType), frame, Rect2d(bboxes[i]));
Step 4: Update MultiTracker & Display Results
Finally, our MultiTracker is ready and we can track multiple objects in a new frame. We use the update method of the MultiTracker class to locate the objects in a new frame. Each bounding box for each tracked object is drawn using a different color.
Python
# Process video and track objects
while cap.isOpened():
success, frame = cap.read()
if not success:
break
# get updated location of objects in subsequent frames
success, boxes = multiTracker.update(frame)
# draw tracked objects
for i, newbox in enumerate(boxes):
p1 = (int(newbox[0]), int(newbox[1]))
p2 = (int(newbox[0] + newbox[2]), int(newbox[1] + newbox[3]))
cv2.rectangle(frame, p1, p2, colors[i], 2, 1)
# show frame
cv2.imshow('MultiTracker', frame)
# quit on ESC button
if cv2.waitKey(1) & 0xFF == 27: # Esc pressed
break
C++
while(cap.isOpened())
{
// get frame from the video
cap >> frame;
// Stop the program if reached end of video
if (frame.empty()) break;
//Update the tracking result with new frame
multiTracker->update(frame);
// Draw tracked objects
for(unsigned i=0; i<multiTracker->getObjects().size(); i++)
{
rectangle(frame, multiTracker->getObjects()[i], colors[i], 2, 1);
}
// Show frame
imshow("MultiTracker", frame);
// quit on x button
if (waitKey(1) == 27) break;
}
I cannot find the Tracker definitions in OpenCV 3.4.1.
Did you compile with opencv_contrib?
i couldn’t find library (opencv_trackingxxxd.lib) for Ptr, VS when i trying to compile shows an errror
Hi, I have similar problem with Treshold module. I am using Python 3.6 / 64 bit, OpenCV 3.4.1. Instalation by anaconda (packages OpenCV, libopencv and py-opencv).
Any idea what is wrong?
Hi Roman! Sorry for the late reply. You should compile opencv from source with opencv_contrib.
Hello, thanks simplifying the process of learning Open CV with well documented code.
I wanted to know how can I detect tracking failure for a single object in the multi track so that I can remove it from the list of objects that are being tracked and perhaps re-initiate detection to add new objects ?
Thanks, Isaac. The multi-tracker does not do a good job in this respect. But it is a naive implementation, and so it is not doing anything very special. You can write a simple class that can hold an array of trackers and that will give you all the flexibility you are looking for.
Thank you, I was implementing that but have always been wondering if I wasn’t reinventing the wheel or whether there’s an inefficient way of doing so.
I was looking too for a way to cleanup a multitracker, but I did not succeed (as many others on the internet).
There is no method to remove the trackers from the multitracker, and every call to selectROIs adds new rectangles to the multitracker.
Maybe Isaac comes with a solution?
Thanks for the articles. This interface got me excited to try it out. Unfortunately, due to the issue discussed here, I think the openCV multi-tracker, as it exists, is of little value. I wonder what their imagined use model is. Thanks.
What happens if
a) a bounding box goes out of the domain ?
b) a bounding box (and the interesting things inside it) is hidden by another loving (edited : moving, of course) object?
Sometimes a tracker can recover when the object is occluded for a frame or two, but if the occlusion lasts a long time it cannot. That is why trackers are used in conjunction with detection.
Great article..my object trackers worked perfectly thanks Satya Mallick.
Thanks, Mark.
Thanks for making the tutorial however I can’t make a cv2.Tracker … in openCV 3.2.0 I also installed opencv-contrib-python but still does not exist. I am using python 3.6
I am not sure if the tracker class was available in OpenCV 3.2. I have tested it on 3.4.1
Hey Satya,
I tried compiling the code above but after selecting the first bounding box, it freezes. When I press q to stop selecting options, it freezes once again. When i simultaneously press q and esc, it outputs the the bounding box dimmensions however, theres always a (0,0,0,0) box and I get the following error: Process finished with exit code 136 (interrupted by signal 8: SIGFPE)
I want to mention that I’m trying to do this live, with VideoCapture instead of with a pre-recorded video
to accept rectangles in selectROIs(…) you have to press space, not q, to exit selectROIs(…) press esc.
I experimented with the MOSSE, KCF and MedianFlow trackers, all work fine. But only with MF update() returns rectangles, that vary with the size of the object.
This gives some clue for movements in the 3rd dimension (to and away from the camera). But after some time the rectangle(s) grow and do not return to their original size. I use the default params. Maybe someone knows how to change these to prevent the growing? (it looks as if increasing params.maxLevel f.i. from 5 to 50 has a positive effect…)
I think the tracker is “learning” about the environment of the detected objects, that causes growing. Maybe there is a way that the tracker returns now and then to the original rectangle from selectROIs(…) ?
CSRT and GOTURN also vary the size.
Most trackers fail when asked to track for very long durations. So, you have to update them with a detector once in a while.
For MedianFlow maxLevel will help track larger motions but it becomes computationally more expensive.
Thanks Satya,
CSRT does very litte variation on my rectangles, GOTURN does not work (at the moment). I read about the long duration problems, it’s not easy to solve by sophisticated algorithms.
There is also a MultiTrackerTLD ( MultiTracker_Alt ), maybe that offers more possibilities for timed updates with the original rectangles or pictures?
Can I have your C++ code? Thank you, my email is [email protected]
Follow the link below
Multi-Object Tracker
Satya Mallick~ CUDA with nvidia GPU graphics for “Multiple Object Tracking” can speed up camera tracking people! ? Thank you!
@spmallick:disqus
None of these trackers are production ready. Can you recommend a production grade algorithm ?
Strange developer problem:
while developing (OSX 10.11.6, Qt Creator, C++, OpenCV 3.4.1 and opencv_contrib) in my app the TrackerMedianFlow works fine. Making a standalone version (with all the frameworks, dylibs and plugins packed in an OSX app) only with this particular tracker I got:
dyld: lazy symbol binding failed: Symbol not found: __ZN2cv23buildOpticalFlowPyramidERKNS_11_InputArrayERKNS_12_OutputArrayENS_5Size_IiEEibiib Referenced from: […]/Contents/MacOS/./../Frameworks/libopencv_tracking.3.4.1.dylib.
libopencv_tracking.3.4.1.dylib is present in the app and other tracker types work fine.
Is something wrong with buildOpticalFlowPyramid?
Is this an OpenCV bug or do I miss some CMake flags while building the (shared) libraries?
This problem is called “name mangling”. I have a note on this problem from 10 years back :).
It says you encounter this issue if you have a function that is not defined in the right namespace or classname. For example, if a method was supposed to be MyClass::MyMethod(int, int) but while implementing it you forgot the class name and simply did MyMethod(int, int).
Not sure how to solve your exact problem, but I hope this pointer was useful.
Hmm, then it seems to be an OpenCV issue. I tested this with a simple app based on the code of your MultiObjecttracker and got exactly the same error, only with TrackerMedianFlow, not with the others.
I posted the issue on GitHub too:
https://github.com/opencv/opencv_contrib/issues/1723
I wonder how can one get object ID out of the Multitrack object – it helps to know if an object has left the scene or not.
For that I would suggest writing your own multi object tracker using the single object trackers in OpenCV.
https://learnopencv.com/object-tracking-using-opencv-cpp-python/
That will give you access to when individual trackers lose track. Unfortunately, the tracking failed flag output by these trackers is not always reliable.
Hello sir, Thanks for sharing this tutorial..Program works well with the given video with code. But when i’m using other video getting problem…i’m attaching snapshot..please reply
https://uploads.disquscdn.com/images/d55e9928156abe5484e5d1e311415df381a1cc9b9b84b4abd2a8bed7991dae4c.png
Can any body send me the zip of the code i tried to download by clicking the download code but it did not come
Sorry Benya,
I will check what is going on. Please use this link
https://github.com/spmallick/learnopencv/tree/master/MultiObjectTracker
Satya
Thanks the picture are displaying but not moving at all…what can i do again?
Everytime I tried to download the code, it always link me to the subscription page, but even I already subscribe, still can’t download the code. Anyone has same issue ?
I will check what is going on. Please use this link
https://github.com/spmallick/learnopencv/tree/master/MultiObjectTracker
Hi Satya,
is there anyway to retrieve the x and y cordinates of the tracking boxes(ROIs) in all frames?
Got it!
Hello
i have this exception “È stata generata un’eccezione non gestita: violazione di accesso in lettura.
**_Pnext** era 0x100000009.. Si è verificata l’eccezione ”
in english “there is an exception. reading access is violated **_Pnext** was 0x1000000009…there was an exceprion”
the program start , it allow me to select which element I want to track, but a little time after that i have push “esc” the exception is generated. I think that the problem is in vector standard header at the 1943 line.
how i can solve it?
Hello, Thanks for sharing well documented code….
I run the code but i am not able to select the multiple objects? how can i use the code for tracking multiple objects at a time….?
When I originally commented I clicked the -Notify me when new remarks are added- checkbox and now each time a comment is added I get four emails with the same comment. Is there any way you can remove me from that service? Thanks!