Automatic Red Eye Remover using OpenCV (C++ / Python)

In this tutorial, we will learn how to remove red eyes from a photo completely automatically.

Just thinking about this problem of red-eye removal brings back memories from my childhood. I grew up in small towns in India where not every family in my middle-class neighborhood had a camera. My family owned a point-and-shoot color camera called “Hotshot.” It was probably the only portable electronic device in our household, and it lasted a couple of decades. My mom kept the camera locked in a safe along with her gold jewelry. Photography was reserved for very special occasions because films were expensive. Sometimes months went by before the photos we took were developed. Naturally, it always broke my heart when the pictures we took at night had the red eye effect. Smiling people with bloody eyes reminded me of Dracula!

Sometime during my undergrad, I figured out a way to manually remove red-eyes using a photo editing tool. It brought me a lot of joy to learn the underlying principle, and I was amazed by its simplicity! It would take a few more years for me to figure out how to do it completely automatically.

Building a robust red eye removal application that will work on all kinds of images is beyond the scope of this post. However, we will learn the basic principles and build a proof of concept application.

What causes the Red-Eye Effect in flash photography?

When you are in a dark room, your pupils are dilated (enlarged) to let in more light to help you see better. The flash on most cameras is very close to the lens. When you take a picture with the flash on, the light from the flash passes through the enlarged pupil to the back of the eyeball and back out through the pupil into the lens of the camera. The back of the eyeball is called the fundus. It is red in color because of an ample supply of blood in the fundus.

The image of the fundus is shown on the left. Examining the fundus can reveal a lot about the health of a person. You can even get smartphone apps that help you see the fundus with an attachment.

Most camera flashes these days flicker for a few seconds which make the pupils contract thus reducing the possibility of red eyes.

How to remove Red-Eyes Automatically?

In this section, we will go step-by-step over the algorithm used for automatic red eye removal.

Step 1 : Eye detection

The first step is to detect eyes automatically. We use the standard OpenCV Haar detector (haarcascade_eye.xml) for eyes for finding the eyes. Sometimes it makes sense to run a face detector first and then detect the eyes inside the face region. To keep things simple, we are run the eye detector directly on the image. Skipping the face detector works when the input image is a portrait shot, or you have a closeup of the eyes.

You can also train your own HAAR object detector by following the instructions here. I am sharing the code for eye detector below.
C++

// Read image
Mat img = imread("red_eyes.jpg",CV_LOAD_IMAGE_COLOR);

// Output image
Mat imgOut = img.clone();

// Load HAAR cascade
CascadeClassifier eyes_cascade("haarcascade_eye.xml");

// A vector of Rect for storing bounding boxes for eyes.
std::vector<Rect> eyes;

// Detect eyes.
eyesCascade.detectMultiScale( img, eyes, 1.3, 4, 0 | CASCADE_SCALE_IMAGE, Size(100, 100) );

Python

# Read image
img = cv2.imread("red_eyes.jpg", cv2.IMREAD_COLOR)

# Output image
imgOut = img.copy()

# Load HAAR cascade
eyesCascade = cv2.CascadeClassifier("haarcascade_eye.xml")

# Detect eyes
eyes = eyesCascade.detectMultiScale(img,scaleFactor=1.3, minNeighbors=4, minSize=(100, 100))

Step 2 : Masking red eyes

Next, we need to find the part of the pupil that is affected by the red eyes. There are many different ways of finding something red. One thing to note is that our color is not just red, it is bright red! You could convert the image to HSV color space and threshold based on hue and brightness. In this post, we have used a simpler heuristic. We say that he red channel should be greater than a threshold and also the sum of green and blue channels. For the purpose of a proof-of-concept system, the heuristic is adequate, but if you want to build automatic red-eye reduction for a commercial software package, you would need to gather thousands of images red eyes to come up with something better.

In the code below, we loop over all the eye rectangles we had detected in the previous step. We then split, the color image into its three channels using the command split. Finally, we create a mask that is 1 for every pixel where the red channel is above a threshold (150) and where the red channel is greater than the sum of green and blue channels.

C++

for( size_t i = 0; i < eyes.size(); i++ )
{

  // Extract eye from the image.
  Mat eye = img(eyes[i]);

  // Split eye image into 3 channels.
  vector<Mat>bgr(3);
  split(eye,bgr);

  // Simple red eye detector
  Mat mask = (bgr[2] > 150) &amp; (bgr[2] &gt; ( bgr[1] + bgr[0] ));
}

Python

for (x, y, w, h) in eyes:

  # Extract eye from the image.
  eye = img[y:y+h, x:x+w]

  # Split eye image into 3 channels
  b = eye[:, :, 0]
  g = eye[:, :, 1]
  r = eye[:, :, 2]

  # Add the green and blue channels.
  bg = cv2.add(b, g)

  # Simple red eye detector
  mask = (r > 150) &  (r > bg)

  # Convert the mask to uint8 format.
  mask = mask.astype(np.uint8)*255

Step 3: Cleaning up pupil mask

The mask created in the previous step will most likely have holes. The left image in the figure above shows the raw mask obtained using color processing. We have removed the hole in the mask using the code shared below. To more about how it works, you can look at my post on filling holes.

C++

void fillHoles(Mat &mask)
{

  Mat mask_floodfill = mask.clone();
  floodFill(mask_floodfill, cv::Point(0,0), Scalar(255));
  Mat mask2;
  bitwise_not(mask_floodfill, mask2);
  mask = (mask2 | mask);
}

Python

def fillHoles(mask):
    maskFloodfill = mask.copy()
    h, w = maskFloodfill.shape[:2]
    maskTemp = np.zeros((h+2, w+2), np.uint8)
    cv2.floodFill(maskFloodfill, maskTemp, (0, 0), 255)
    mask2 = cv2.bitwise_not(maskFloodfill)
    return mask2 | mask

In addition, it is a good idea to dilate the mask, so it covers a slightly larger region than necessary. This is because at the boundary the color fades away gradually and some redness might not have been captured in our original mask. In the figure above, the right image is the dilated the mask. We generate the dilated mask using the code shared below.

C++

// Clean up mask by filling holes and dilating
fillHoles(mask);
dilate(mask, mask, Mat(), Point(-1, -1), 3, 1, 1);

Python

# Clean up mask by filling holes and dilating
mask = fillHoles(mask)
mask = cv2.dilate(mask, None, anchor=(-1, -1), iterations=3, borderType=1, borderValue=1)

Step 4: Fix Red-Eyes

Now we have a mask that contains only the red region of each eye. We next show how to process the area inside this mask to fix red eyes.

We know that red eyes saturate the red channel in the image. In other words, all information in the red channel is destroyed. How can we recover some of this information back? While fixing red-eyes, we do not need to retrieve the true underlying texture in the red channels; we only need to find a texture that is plausible.

Fortunately, the red-eye effect destroys the texture only in the red channel; the blue and the green channels are still good. You can see this in the figure below where the red, green and blue channels of the image are shown.

A combination of the green and blue channels can be used to come up with a plausible red channel. For example, we can create a red channel which is the average of the green and blue channels in the image. However, doing this may give the pupil a slight tinge which may look ok, but not great. Notice the purple tinge in the center image.

That brings us to an important question. What should be the color of the pupil? The pupil is an opening in the eye, and the inside of the eye is completely dark. The pupils should, therefore, be colorless (grayscale) and dark. Instead of replacing only the red channel in the pupil region, we replace all the channels with the mean of the green and the blue channels. This eliminates the purple tinge.

The code below first creates the mean channel by averaging the green and the blue channels. It then replaces all the pixels inside the masked region of all channels with this mean channel.

C++

// Calculate the mean channel by averaging
// the green and blue channels
Mat mean = (bgr[0]+bgr[1])/2;

// Copy the mean image to blue channel with mask.
mean.copyTo(bgr[0], mask);

// Copy the mean image to green channel with mask.
mean.copyTo(bgr[1], mask);

// Copy the mean image to red channel with mask.
mean.copyTo(bgr[2], mask);

Python

# Calculate the mean channel by averaging
# the green and blue channels. Recall, bg = cv2.add(b, g)
mean = bg / 2
mask = mask.astype(np.bool)[:, :, np.newaxis]
mean = mean[:, :, np.newaxis]

# Copy the eye from the original image.
eyeOut = eye.copy()

# Copy the mean image to the output image.
np.copyto(eyeOut, mean, where=mask)

Step 5 : Replace the fixed eye region

In the previous step, we had fixed the three channels. The last step is to merge the three channels to create our RGB image and then put this fixed eye region back in the original image.

C++

// Merge the three channels
Mat eyeOut;
merge(bgr,eyeOut);

// Copy the fixed eye to the output image.
eyeOut.copyTo(imgOut(eyes[i]));

Python

# Copy the fixed eye to the output image.
imgOut[y:y+h, x:x+w, :] = eyeOut

Automatic Red-Eye Removal Results

We first show the results on the example image with which started.

Notice that removing all color from the pupil region makes the image look pretty because the dot in the center of the eye is completely white. Also notice that the on the boundary of the pupil, the redness fades, but we still capture this region because of the dilation operation.

Next, we show the result on a closeup photo of the eyes. If we had used a face detector, it would not have detected a face, and the automatic eye detector would not have worked.

Image Credits

1. The featured picture in this post is licensed under Creative Commons Attribution-Share Alike 2.5 Generic.
2. The red-eye closeup and the fundus images are in the public domain.