• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Learn OpenCV

OpenCV, PyTorch, Keras, Tensorflow examples and tutorials

  • Home
  • Getting Started
    • Installation
    • PyTorch
    • Keras & Tensorflow
    • Resource Guide
  • Courses
    • Opencv Courses
    • CV4Faces (Old)
  • Resources
  • AI Consulting
  • About

Automatic Red Eye Remover using OpenCV (C++ / Python)

Satya Mallick
March 7, 2017 18 Comments
Application Face Object Detection Tutorial

March 7, 2017 By 18 Comments

Automatic Red Eye Removal

In this tutorial, we will learn how to remove red eyes from a photo completely automatically.

Just thinking about this problem of red-eye removal brings back memories from my childhood. I grew up in small towns in India where not every family in my middle-class neighborhood had a camera. My family owned a point-and-shoot color camera called “Hotshot.” It was probably the only portable electronic device in our household, and it lasted a couple of decades. My mom kept the camera locked in a safe along with her gold jewelry. Photography was reserved for very special occasions because films were expensive. Sometimes months went by before the photos we took were developed. Naturally, it always broke my heart when the pictures we took at night had the red eye effect. Smiling people with bloody eyes reminded me of Dracula!

Sometime during my undergrad, I figured out a way to manually remove red-eyes using a photo editing tool. It brought me a lot of joy to learn the underlying principle, and I was amazed by its simplicity! It would take a few more years for me to figure out how to do it completely automatically.

Building a robust red eye removal application that will work on all kinds of images is beyond the scope of this post. However, we will learn the basic principles and build a proof of concept application.

What causes the Red-Eye Effect in flash photography?

FundusWhen you are in a dark room, your pupils are dilated (enlarged) to let in more light to help you see better. The flash on most cameras is very close to the lens. When you take a picture with the flash on, the light from the flash passes through the enlarged pupil to the back of the eyeball and back out through the pupil into the lens of the camera. The back of the eyeball is called the fundus. It is red in color because of an ample supply of blood in the fundus.

The image of the fundus is shown on the left. Examining the fundus can reveal a lot about the health of a person. You can even get smartphone apps that help you see the fundus with an attachment.

Most camera flashes these days flicker for a few seconds which make the pupils contract thus reducing the possibility of red eyes.

To easily follow the article below, please subsribe and download the code here.

How to remove Red-Eyes Automatically?

In this section, we will go step-by-step over the algorithm used for automatic red eye removal.

Step 1 : Eye detection

OpenCV Eye Detection

The first step is to detect eyes automatically. We use the standard OpenCV Haar detector (haarcascade_eye.xml) for eyes for finding the eyes. Sometimes it makes sense to run a face detector first and then detect the eyes inside the face region. To keep things simple, we are run the eye detector directly on the image. Skipping the face detector works when the input image is a portrait shot, or you have a closeup of the eyes.

You can also train your own HAAR object detector by following the instructions here.

I am sharing the code for eye detector below.
C++

// Read image
Mat img = imread("red_eyes.jpg",CV_LOAD_IMAGE_COLOR);

// Output image
Mat imgOut = img.clone();
    
// Load HAAR cascade 
CascadeClassifier eyes_cascade("haarcascade_eye.xml");

// A vector of Rect for storing bounding boxes for eyes.
std::vector<Rect> eyes;

// Detect eyes. 
eyesCascade.detectMultiScale( img, eyes, 1.3, 4, 0 | CASCADE_SCALE_IMAGE, Size(100, 100) );

Python

# Read image
img = cv2.imread("red_eyes.jpg", cv2.IMREAD_COLOR)

# Output image
imgOut = img.copy()
    
# Load HAAR cascade
eyesCascade = cv2.CascadeClassifier("haarcascade_eye.xml")

# Detect eyes
eyes = eyesCascade.detectMultiScale(img,scaleFactor=1.3, minNeighbors=4, minSize=(100, 100))

Step 2 : Masking red eyes

Next, we need to find the part of the pupil that is affected by the red eyes. There are many different ways of finding something red. One thing to note is that our color is not just red, it is bright red! You could convert the image to HSV color space and threshold based on hue and brightness. In this post, we have used a simpler heuristic. We say that he red channel should be greater than a threshold and also the sum of green and blue channels. For the purpose of a proof-of-concept system, the heuristic is adequate, but if you want to build automatic red-eye reduction for a commercial software package, you would need to gather thousands of images red eyes to come up with something better.

In the code below, we loop over all the eye rectangles we had detected in the previous step. We then split, the color image into its three channels using the command split. Finally, we create a mask that is 1 for every pixel where the red channel is above a threshold (150) and where the red channel is greater than the sum of green and blue channels.

C++

for( size_t i = 0; i &lt; eyes.size(); i++ )
{
    
  // Extract eye from the image.
  Mat eye = img(eyes[i]);
        
  // Split eye image into 3 channels.
  vector<Mat>bgr(3);
  split(eye,bgr);

  // Simple red eye detector
  Mat mask = (bgr[2] > 150) &amp; (bgr[2] &gt; ( bgr[1] + bgr[0] ));
}

Python

for (x, y, w, h) in eyes:
  
  # Extract eye from the image.
  eye = img[y:y+h, x:x+w]
  
  # Split eye image into 3 channels
  b = eye[:, :, 0]
  g = eye[:, :, 1]
  r = eye[:, :, 2]

  # Add the green and blue channels. 
  bg = cv2.add(b, g)

  # Simple red eye detector
  mask = (r > 150) &  (r > bg)
  
  # Convert the mask to uint8 format. 
  mask = mask.astype(np.uint8)*255

Step 3: Cleaning up pupil mask

Pupil Mask Processing
Left: Color-based pupil mask. Center: Pupil mask with holes filled. Right: Dilated pupil mask

The mask created in the previous step will most likely have holes. The left image in the figure above shows the raw mask obtained using color processing. We have removed the hole in the mask using the code shared below. To more about how it works, you can look at my post on filling holes.

C++

void fillHoles(Mat &amp;mask)
{
    
  Mat mask_floodfill = mask.clone();
  floodFill(mask_floodfill, cv::Point(0,0), Scalar(255));
  Mat mask2;
  bitwise_not(mask_floodfill, mask2);
  mask = (mask2 | mask);

}

Python

def fillHoles(mask):
    maskFloodfill = mask.copy()
    h, w = maskFloodfill.shape[:2]
    maskTemp = np.zeros((h+2, w+2), np.uint8)
    cv2.floodFill(maskFloodfill, maskTemp, (0, 0), 255)
    mask2 = cv2.bitwise_not(maskFloodfill)
    return mask2 | mask

In addition, it is a good idea to dilate the mask, so it covers a slightly larger region than necessary. This is because at the boundary the color fades away gradually and some redness might not have been captured in our original mask. In the figure above, the right image is the dilated the mask. We generate the dilated mask using the code shared below.

C++

// Clean up mask by filling holes and dilating
fillHoles(mask);
dilate(mask, mask, Mat(), Point(-1, -1), 3, 1, 1);

Python

# Clean up mask by filling holes and dilating
mask = fillHoles(mask)
mask = cv2.dilate(mask, None, anchor=(-1, -1), iterations=3, borderType=1, borderValue=1)

Step 4: Fix Red-Eyes

Now we have a mask that contains only the red region of each eye. We next show how to process the area inside this mask to fix red eyes.

We know that red eyes saturate the red channel in the image. In other words, all information in the red channel is destroyed. How can we recover some of this information back? While fixing red-eyes, we do not need to retrieve the true underlying texture in the red channels; we only need to find a texture that is plausible.

Fortunately, the red-eye effect destroys the texture only in the red channel; the blue and the green channels are still good. You can see this in the figure below where the red, green and blue channels of the image are shown.

Channels of an affected by Red-Eye
Left: Red channel. Center: Blue channel. Right: Green channel.

A combination of the green and blue channels can be used to come up with a plausible red channel. For example, we can create a red channel which is the average of the green and blue channels in the image. However, doing this may give the pupil a slight tinge which may look ok, but not great. Notice the purple tinge in the center image.

Red Eye removal
Left: Red eye. Center: Fixed by replacing red channel. Right: Fixed by replacing all channels.

That brings us to an important question. What should be the color of the pupil? The pupil is an opening in the eye, and the inside of the eye is completely dark. The pupils should, therefore, be colorless (grayscale) and dark. Instead of replacing only the red channel in the pupil region, we replace all the channels with the mean of the green and the blue channels. This eliminates the purple tinge.

The code below first creates the mean channel by averaging the green and the blue channels. It then replaces all the pixels inside the masked region of all channels with this mean channel.

C++

// Calculate the mean channel by averaging
// the green and blue channels
Mat mean = (bgr[0]+bgr[1])/2;

// Copy the mean image to blue channel with mask.
mean.copyTo(bgr[0], mask);

// Copy the mean image to green channel with mask.
mean.copyTo(bgr[1], mask);

// Copy the mean image to red channel with mask.
mean.copyTo(bgr[2], mask);

Python

# Calculate the mean channel by averaging
# the green and blue channels. Recall, bg = cv2.add(b, g)
mean = bg / 2
mask = mask.astype(np.bool)[:, :, np.newaxis]
mean = mean[:, :, np.newaxis]

# Copy the eye from the original image. 
eyeOut = eye.copy()

# Copy the mean image to the output image. 
np.copyto(eyeOut, mean, where=mask)

Step 5 : Replace the fixed eye region

In the previous step, we had fixed the three channels. The last step is to merge the three channels to create our RGB image and then put this fixed eye region back in the original image.

C++

// Merge the three channels
Mat eyeOut;
merge(bgr,eyeOut);
        
// Copy the fixed eye to the output image. 
eyeOut.copyTo(imgOut(eyes[i]));

Python

# Copy the fixed eye to the output image. 
imgOut[y:y+h, x:x+w, :] = eyeOut

Automatic Red-Eye Removal Results

We first show the results on the example image with which started.

Automatic Red Eye Removal

Notice that removing all color from the pupil region makes the image look pretty because the dot in the center of the eye is completely white. Also notice that the on the boundary of the pupil, the redness fades, but we still capture this region because of the dilation operation.

Next, we show the result on a closeup photo of the eyes. If we had used a face detector, it would not have detected a face, and the automatic eye detector would not have worked.

Automatic Red Eye Removal Example

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Image Credits

  1. The featured picture in this post is licensed under Creative Commons Attribution-Share Alike 2.5 Generic.
  2. The red-eye closeup and the fundus images are in the public domain.
Tags: C++ eyes OpenCV Python red-eye

Filed Under: Application, Face, Object Detection, Tutorial

About

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Getting Started

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

Resources

Download Code (C++ / Python)

ENROLL IN OFFICIAL OPENCV COURSES

I've partnered with OpenCV.org to bring you official courses in Computer Vision, Machine Learning, and AI.
Learn More

Recent Posts

  • Making A Low-Cost Stereo Camera Using OpenCV
  • Optical Flow in OpenCV (C++/Python)
  • Introduction to Epipolar Geometry and Stereo Vision
  • Depth Estimation using Stereo matching
  • Classification with Localization: Convert any Keras Classifier to a Detector

Disclaimer

All views expressed on this site are my own and do not represent the opinions of OpenCV.org or any entity whatsoever with which I have been, am now, or will be affiliated.

GETTING STARTED

  • Installation
  • PyTorch
  • Keras & Tensorflow
  • Resource Guide

COURSES

  • Opencv Courses
  • CV4Faces (Old)

COPYRIGHT © 2020 - BIG VISION LLC

Privacy Policy | Terms & Conditions

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.AcceptPrivacy policy