• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

Image Inpainting with OpenCV (C++/Python)

Satya Mallick
April 2, 2019 Leave a Comment
how-to Image Processing

April 2, 2019 By Leave a Comment

In today’s post we will describe a class of region filling algorithms called image inpainting.

Imagine finding an old family photograph. You scan it and it looks great except for a few scratches.

Of course you can load the photo in Photoshop and fix the scratches. But is that really cool? Hell no!

You are a super cool engineer! You have a reputation to live up to. You open your favorite editor and write 10 lines of code to solve the problem using an inpainting algorithm in OpenCV. If your friends do not look sufficiently impressed, you can tell them the method is based on the Navier Stokes equation they might have encountered in fluid dynamics!

But to be that cool, you need to read this post first.

What is Image Inpainting?

Image inpainting is a class of algorithms in computer vision where the objective is to fill regions inside an image or a video.

The region is identified using a binary mask, and the filling is usually done by propagating information from the boundary of the region that needs to be filled.

The most common application of image inpainting is restoration of old scanned photos. It is also used for removing small unwanted objects in an image.

Inpainting Algorithms

In this section, we will briefly discuss two inpainting algorithms implemented in OpenCV.

INPAINT_NS : Navier-Stokes based Inpainting

This method was published in 2001 in a paper titled “Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting”

Sometimes I feel like the field of Computer Vision is a field of immigrants from other fields like electrical engineering, computer science, physics, and mathematics.

They bring their ideas to the field and solve the same problem in very interesting and unique ways. An electrical engineer may see an image as a 2D signal and apply the theories of signal processing to solve computer vision problems. On the other hand, a mathematician may see an image as a connected graph and solve computer vision problems using graph theory.

So it isn’t surprising that theories developed for fluid dynamics also make their way into computer vision.

In the image below, our objective is to fill the dark region and obtain an image that looks like the one on the right.

Image Inpainting Example

How do we fill this black region? One constraint we would like is the edge entering point A should continue to the edge leaving point B. The other constraint we may want is that the region on the right of the curve joining A and B should be white, and the region on the left should be blue.

The above two constraints essentially state

  1. Preserve gradients (i.e. edge like features)
  2. Continue to propagate color information in smooth regions

The authors set up a partial differential equation (PDE) to update image intensities inside the region with the above constraints.

The image smoothness information is estimated by the image Laplacian and it is propagated along the isophotes (contours of equal intensities). The isophotes are estimated by the image gradient rotated by 90 degrees.

The authors show that these equations are closely related in form to the Navier-Stokes equations for 2D incompressible fluids.

The benefit of reducing the problem to one of fluid dynamics is that we benefit from well developed theoretical analysis and numerical tools.

INPAINT_TELEA : Fast Marching Method based

This implementation is based on a paper titled “An Image Inpainting Technique Based on the Fast Marching Method” by Alexandru Telea.

This implementation solves the same constraints using a different technique. Instead of using the image Laplacian as the estimator of smoothness, the author uses a weighted average over a known image neighborhood of the pixel to inpaint. The known neighborhood pixels and gradients are used to estimate the color of the pixel to be inpainted.

Once a pixel is inpainted, the boundary needs to updated. The author treats the missing region of the image as level sets and uses the fast marching method to update the boundary.

Pros and Cons

As per the theory and the papers, Navier-Stokes based inpainting is supposed to be slower and has a tendency to produce results that are blurrier than the Fast Marching based method.

In practice, we did not find that to be the case. INPAINT_NS produced better results in our tests and the speed was also marginally better than INPAINT_TELEA.

Inpainting Code in Python and C++

Download Code To easily follow along this tutorial, please download code by clicking on the button below. It's FREE!

Download Code

In OpenCV inpainting is implemented using the functioninpaint.

C++
void inpaint(
             Mat& src, 
             Mat& inpaintMask, 
             Mat& dst, 
             double inpaintRadius, 
             int flags)
Python
dst = cv2.inpaint(
             src, 
             inpaintMask, 
             inpaintRadius, 
             flags)

Where,

  • src = Source image
  • inpaintMask = A binary mask indicating pixels to be inpainted.
  • dst = Destination image
  • inpaintRadius = Neighborhood around a pixel to inpaint. Typically, if the regions to be inpainted are thin, smaller values produce better results (less blurry).
  • flags : INPAINT_NS (Navier-Stokes based method) or INPAINT_TELEA (Fast marching based method)

Inpainting Results

Let’s look at the result of applying inpainting to a historic image of President Lincoln. There is a fascinating history behind this photo which I have borrowed from Wikipedia

On Sunday, February 5, 1865, at Gardner’s Gallery in Washington DC, Alexander Gardner took several multiple-lens pictures of the President. Before this session ended, Gardner asked the president for one last pose. He moved his camera closer and took a photograph of Lincoln’s head, shoulders, and chest. Mysteriously the glass plate negative cracked. Gardner carefully took it to his dark room and was able to make one print, with an ominous crack across Lincoln’s face, before it broke completely and was discarded. This print, known as O-118, still exists to this day. Over the years many people have associated this crack with a symbolic foretelling of the assassin’s bullet that awaited Lincoln 10 weeks later.

Inpainting Result
Inpainting Results : The first image from the left is the input image, the second image is the mask, the third image is the result of INPAINT_TELEA and the final result is by INPAINT_NS.

Let’s look at a more complicated example. We have scribbled quite extensively over the image of a flower garden, but the results are still very compelling.

Image Inpainting using OpenCV
Left : Original Image with Scribbles. Middle : Inpainted using Fast Marching Method, Right : Inpainted using Navier-Stokes method.

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now


Tags: INPAINT_NS INPAINT_TELEA inpainting

Filed Under: how-to, Image Processing

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy