Averaging face photos : eye alignment

Intro

I stumbled upon a post from Patrick David where he takes front view close-ups photos of celebrities faces from Martin Schoeller and averages them with Image Magick.

I do the same here, but image alignment is made with a script and more image blendings are proposed. All that with G’mic.

Stealing the close-ups

Martin Schoeller shows his close-ups with a flash applet, it is thus not possible to just save them as usual. So, many screenshots and a big crop later and they are all 29 in a directory called faces:

  gmic screenshots*.png -crop 231,112,874,877 -o faces/faces.png

29 faces

Automatic alignment using -phase_correlation

Patrick David aligned images by hand using Gimp, he “tried to get eyes on the same level, and the same distance from the centers”. Let’s try to do it with a script.

The dumb way

“To estimate the relative translative offset between two similar images”, phase correlation is a good trick. G’mic has a command called -phase_correlation. Starting from two images, this command returns a single pixel image with 3 layers, each one being the estimated offset in the X, Y or Z direction.

Let’s try it. Let’s try to compare every images with the first one (Jack Nicholson) and to shift them according to what -phase_correlation find:

mkdir dumb_phase_correlation
cd faces
cp faces_000000.png ../dumb_phase_correlation
for i in $(ls|tail -n +2)
do
  gmic $i faces_000000.png --phase_correlation[-1,-2] -shift[0] '@{-1,(0,0,0,0)},@{-1,(0,0,0,1)}' -o[0] ../dumb_phase_correlation/$i
done

Obviously, this doesn’t work:

badly aligned

Aim at the eye

This failure just means that -phase_correlation should be used wisely.

There’s no way -phase_correlation can deal with different skin/hair colors, so we’ll use -gradient_norm to work on contours. And since we want to align eyes, we should ask -phase_correlation to focus on eyes only, let’s say the right one, we’ll do that with a wise -crop. It gives:

mkdir aligned_right_eye
cd faces
cp faces_000000.png ../aligned_right_eye
for i in $(ls|tail -n +2)
do
  gmic $i faces_000000.png --gradient_norm -n[-1,-2] 0,1 -crop[-1,-2] 96,258,322,444 --phase_correlation[-1,-2] -shift[0] '@{-1,(0,0,0,0)},@{-1,(0,0,0,1)}' -o[0] ../aligned_right_eye/$i
done

It looks much better:

All 29 images aligned on the right eye.

And if we try to average them:

  gmic * -+ -normalize 0,255 -o aligned_right_eye.jpg,90

it looks ok … for the right eye:

the 29 photos averaged after being aligned on the right eye.

the 29 photos averaged after being aligned on the right eye.

Align, rotate, zoom

The right eye is the first step, let’s try to align on both eyes. It won’t be easy because some of them don’t have both eyes at the same altitude and they all have different eye pitch.

Fortunately, they all have 2 eyes. If we suppose that the phase correlation focused on the right eye gives somehow the position of the right eye, we can do the same on the left eye and then align, rotate and zoom to get a perfect alignment on both eyes.

Since this is a much more complex issue, we’ll use a macro file called macro.gmic that will contain an custom command called align_rotate_zoom. The custom command first aligns according to both eye. Then it makes a rotation with an angle computed with eyes vertical shift and centered at mid distance between Jack Nicholson’s eyes. The zoom factor given to the -rotate is calculated with a comparison between the image eye pitch with Jack Nicholson’s one. At the end, the custom command is:

align_rotate_zoom:
  --gradient_norm -n[-1,-2] 0,1
  --crop[-1,-2] 322,258,548,444 -phase_correlation[-1,-2]  #left eye
  -crop[-2,-3] 96,258,322,444 -phase_correlation[-2,-3]  #right eye
#align
  vertical_shift={(@{-1,(0,0,0,1)}+@{-2,(0,0,0,1)})/2}  #average of the two dy
  horizontal_shift={(@{-1,(0,0,0,0)}+@{-2,(0,0,0,0)})/2}  #average of the two dx
  -shift[0] $horizontal_shift,$vertical_shift
#rotate & zoom
  eye_pitch=220 centerx=319 centery=337 
  angle={atan((@{-1,(0,0,0,1)}-@{-2,(0,0,0,1)})/$eye_pitch)/pi*180}
  zoom_factor={$eye_pitch/($eye_pitch-@{-1,(0,0,0,0)}+@{-2,(0,0,0,0)})}
  -rotate[0] $angle,0,2,$centerx,$centery,$zoom_factor
#clean a bit
  -cut[0] 0,255 -rm[-1,-2]

And the bash script to run becomes:

mkdir aligned_rotate_zoom
cd faces
cp faces_000000.png ../aligned_rotate_zoom
for i in $(ls|tail -n +2)
do
  gmic $i faces_000000.png ../macro.gmic -align_rotate_zoom -o[0] ../aligned_rotate_zoom/$i
done

It looks fine even if imperfect, for example, Hillary Clinton is too big. But when averaging is made on many photos, if only a few are wrong, you can’t really see it. And averaging could also be made without Hillary.

All 29 images aligned, rotated and zoom to fit both eyes.

And the final averaging gives:

the 29 photos averaged after being aligned on both eyes thanks to shifting, rotating and zooming.

the 29 photos averaged after being aligned on both eyes thanks to shifting, rotating and zooming.

Nice gaze isn’t it?

Other blending

Averaging is only one way of blending many images together, there are zillions of other ways. Here below is one using -compose_edges, which gives priority to edges:

gmic * -compose_edges 0.8 -o blend_edges.jpg,90

gmic * -compose_edges 0.8 -o blend_edges.jpg,90

I made some others based on image multiplication, averaging inverse of image values, minimum, maximum, minimum or maximum only on luminance:

Post Scriptum

In the next post, mouth alignment is added for event better averaging.

Advertisements

3 thoughts on “Averaging face photos : eye alignment

  1. Pingback: Averaging face photos : mouth alignment « Proof of concept of what?

  2. Pingback: High Multiples

  3. Pingback: High Multiples and Others | High Multiples

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s