• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

Deep Learning based Text Detection Using OpenCV (C++/Python)

Vishwesh Shrimali
Vikas Gupta
January 28, 2019 Leave a Comment
Deep Learning Object Detection Text Recognition

January 28, 2019 By Leave a Comment

There is a common saying, “A picture is worth a thousand words“. In this post, we are going to take that literally and try to find the words in a picture! In an earlier post about Text Recognition, we discussed how Tesseract works and how it can be used along with OpenCV for text detection as well as recognition. This time, we are going to have a look at robust approach for detecting text, based on a recent paper : EAST: An Efficient and Accurate Scene Text Detector.

It should be noted that text detection is different from text recognition. In text detection we only detect the bounding boxes around the text. But, in text recognition, we actually find what is written in the box. For example, in the image given below, text detection will give you the bounding box around the word and text recognition will tell you that the box contains the word STOP.

Text Recognition engines such as Tesseract require the bounding box around the text for better performance. Thus, this detector can be used to detect the bounding boxes before doing Text Recognition.

A tensorflow re-implementation of the paper reported the following speed on 720p (resolution of 1280×720) images (source):

  • Graphic Card: GTX 1080 Ti
  • Network fprop: ~50 ms
  • NMS (C++): ~6 ms
  • Overall: ~16 fps

The tensorflow model has been ported to be used with OpenCV and they have also provided sample code. We will discuss how it works step by step. You will need OpenCV >= 3.4.3 to run the code. Let’s detect some text in images!

The steps involved are as follows:

  1. Download the EAST Model
  2. Load the Model into memory
  3. Prepare the input image
  4. Forward pass the blob through the network
  5. Process the output
Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE!

Download Code

Step 1: Download EAST Model

The EAST Model can be downloaded from this dropbox link : https://www.dropbox.com/s/r2ingd0l3zt8hxs/frozen_east_text_detection.tar.gz?dl=1.

Once the file has been downloaded (~85 MB), unzip it using

tar -xvzf frozen_east_text_detection.tar.gz

You can also extract the contents using the File viewer of your OS.

After unzipping, copy the .pb model file to the working directory.

Step 2: Load the network

We will use the cv::dnn::readnet or cv2.dnn.ReadNet() function for loading the network into memory. It automatically detects configuration and framework based on file name specified. In our case, it is a pb file and thus, it will assume that a Tensorflow Network is to be loaded.

C++

Net net = readNet(model);

Python

net = cv.dnn.readNet(model)

Step 3 : Prepare input image

We need to create a 4-D input blob for feeding the image to the network. This is done using the blobFromImage function.

C++

blobFromImage(frame, blob, 1.0, Size(inpWidth, inpHeight), Scalar(123.68, 116.78, 103.94), true, false);

Python

blob = cv.dnn.blobFromImage(frame, 1.0, (inpWidth, inpHeight), (123.68, 116.78, 103.94), True, False)

There are a few parameters we need to specify to this function. They are as follows :

  1. The first argument is the image itself
  2. The second argument specifies the scaling of each pixel value. In this case, it is not required. Thus we keep it as 1.
  3. The default input to the network is 320×320. So, we need to specify this while creating the blob. You can experiment with any other input dimension also.
  4. We also specify the mean that should be subtracted from each image since this was used while training the model. The mean used is (123.68, 116.78, 103.94).
  5. The next argument is whether we want to swap the R and B channels. This is required since OpenCV uses BGR format and Tensorflow uses RGB format.
  6. The last argument is whether we want to crop the image and take the center crop. We specify False in this case.

Step 4 : Forward Pass

Now that we have prepared the input, we will pass it through the network. There are two outputs of the network. One specifies the geometry of the Text-box and the other specifies the confidence score of the detected box. These are given by the layers :

  • feature_fusion/concat_3
  • feature_fusion/Conv_7/Sigmoid

This is specified in code as follows:

C++

std::vector<String> outputLayers(2);
outputLayers[0] = "feature_fusion/Conv_7/Sigmoid";
outputLayers[1] = "feature_fusion/concat_3";

Python

outputLayers = []
outputLayers.append("feature_fusion/Conv_7/Sigmoid")
outputLayers.append("feature_fusion/concat_3")

Next, we get the output by passing the input image through the network. As discussed earlier, the output consists of two parts : scores and geometry.

C++

std::vector<Mat> output;
net.setInput(blob);
net.forward(output, outputLayers);

Mat scores = output[0];
Mat geometry = output[1];

Python

net.setInput(blob)
output = net.forward(outputLayers)

scores = output[0]
geometry = output[1]

Step 5 : Process the output

As discussed earlier, we will use the outputs from both the layers ( i.e. geometry and scores ) and decode the positions of the text boxes along with their orientation. We might get many candidates for a text box. Thus, we need to filter out the best looking text-boxes from the lot. This is done using Non-Maximum Suppression.

Decode

C++

std::vector<RotatedRect> boxes;
std::vector<float> confidences;
decode(scores, geometry, confThreshold, boxes, confidences);

Python

[boxes, confidences] = decode(scores, geometry, confThreshold)

Non-Maximum Suppression

We use the OpenCV function NMSBoxes ( C++ ) or NMSBoxesRotated ( Python ) for filtering out the false positives and get the final predictions.

C++

std::vector<int> indices;
NMSBoxes(boxes, confidences, confThreshold, nmsThreshold, indices);

Python

indices = cv.dnn.NMSBoxesRotated(boxes, confidences, confThreshold, nmsThreshold)

Results

Given below are a few results.

As you can see, it is able to detect texts with varying Backgrounds, Fonts, Orientation, Size, Color. In the last one, it worked pretty well even for deformed Text. There are however, some mis-detections but we can say, overall it performs very well.

As the examples suggest, it can be used in a wide variety of applications such as Number plate Detection, Traffic Sign Detection, detection of text on ID Cards etc.

Become an expert in Computer Vision, Machine Learning, and AI in 12-weeks! Check out our course

Computer Vision Course

References

  1. EAST: An Efficient and Accurate Scene Text Detector
  2. Tensorflow Implementation
  3. OpenCV Samples [C++], [Python]

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now


Tags: deep learning text detection

Filed Under: Deep Learning, Object Detection, Text Recognition

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy