Averaging face photos : mouth alignment


In a previous post, eyes of face photos were aligned to enable photo averaging as neat as possible around the eyes. Someone suggested that I could also align mouths by stretching a bit those faces for a neater averaging around the mouth. He was right, this is about to happen in this additional post. But it will happen without Masha Fishman because I intent to use colors in this post and she is in black and white.

Identify the mouth

First, all those mouths are about at the same place (under the nose!). It is thus possible to examine them with a simple -crop. Starting from the directory aligned_rotate_zoom, where eyes are already aligned:

gmic * -crop 180,517,455,700

28 mouths

Looking at those mouths, one can see that they have very different shapes, but that they all have a distinctive color : they are a bit more red. I didn’t find anything useful by watching at their red channel in the RGB space. But in the HSV space, their hue channel shows a different color for the lips. To study the hue channel, it is better to “rotate” it by 180° since the colors we want to observe are close to 0 and thus close to 360 also.

Mouth segmentation

Let’s try to segment those mouths, let’s make masks by thresholding the rotated hue channel. After some tests, it feels that 192 is a not so bad value for that:

mkdir mouth_mask
cd aligned_rotate_zoom
for i in *png
  gmic $i -crop 180,517,455,700 -rgb2hsv -channels 0 -+ 180 -mod 360 -blur 5 -threshold 192 -negative -o ../mouth_mask/$i

28 mouth masks

To see what it is worth, we use those masks to highlight the mouths:

cd aligned_rotate_zoom
gmic * -crop 180,517,455,700 -append_tiles 14 ../mouth_mask/* -append_tiles[1--1] 14 -+[-1] 1 -* -n 0,255

The highlighted zone is the mouth mask

The highlighted zone is the mouth mask

Well, it is far from a perfect segmentation, but our final goal is not segmentation but localization in order to be able to correctly align mouths. And with those masks they are localized, we actually only miss one lip : Harriet Sykes upper one.


The idea is to stretch the face according to mouth altitude. To estimate this altitude, we use -barycenter:

cd mouth_mask
gmic * -barycenter -rows 1 -a x

The command above displays a not so interesting row matrix, but it displays the information we want in the terminal : the mean value of this matrix, which is the average altitude of all the masks, for me : 74.

So now, we are ready to stretch. We’ll stretch only the area between the eyes and the mouth, which is for me about between altitude 376 and 500.

For every image in the directory aligned_rotate_zoom, the script below computes the difference between 74 and the altitude of the mouth mask. It then splits the image in 3 parts and stretches the middle part to get the mouth at the right altitude. Finally, it reassembles the 3 parts and resizes them so that every stretched images have the same definition.

mkdir stretched
cd aligned_rotate_zoom
for i in *png
  gmic $i ../mouth_mask/$i -barycenter[-1] -rows[-1] 1 --[-1] 74 \
  --rows[0] 0,375 --rows[0] 376,500 -resize[-1] '{w},{h-@{1,(0,0,0,0)}}' --rows[0] 501,100% \
  -a[-3--1] y -resize[-1] 644,766,1,3,0 -o[-1] ../stretched/$i


As you can see, no face is completely deformed. And the final averaging gives a much neater mouth than obtained in the previous post:


3 thoughts on “Averaging face photos : mouth alignment

  1. Pingback: Averaging face photos : eye alignment « Proof of concept of what?

  2. I guess you mean: who has the most averageness and who has least.

    Ok,let’s check, let’s compute the average difference between each of them and the averaged photo:

    cd stretched
    gmic * ../align_rotate_zoom_stretch.jpg --[0--2] [-1] -rm[-1] -abs -resize 1,1,1,1,2 -a x -display_graph ,,,,,,,,"image_number","mean_difference"

    So number 8 (Harriet Sykes) his the looser with a mean difference of 70.6 and the winner, with a mean difference of only 24.4 is number 1, congratulations Marc Jacobs.

    Well, this is a bit unfair, the background shouldn’t be taken into account and neither should black areas resulting from tilting and other deformation. But I don’t have time to take care of that now.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s