Project 3: Face Morphing

Brandon Wong

For the morphing sequences, I will be using an image of Tyler the Creator and myself:

Part 1. Defining Correspondences

I ended up using this tool provided in the spec from last year to label my images, and I added the 4 corners after mannually annotating the points, resulting in the following labeling points. For the picture of myself, I had to estimate some of the points because the hair was blocking the details underneath.

For triangulating the mesh, I ended up taking the midpoint blend between the two point sets, and using the scipy.spatial.Delaunay function to get the triangles to be used later on in the project. Below is the resulting triangulation applied to the points of the original images.

Part 2. Computing the Mid-way Face

To compute the mid-way face, this involved warping both images to the mid-way face’s shape, then cross fading the pixel values of the warped images. Cross fading is pretty simple; not too much to explain as it’s a simple weighted average between the two images.

As for warping, this involved taking the mid-way face’s triangles and sampling from the original images in the corresponding triangles, doing this for each triangle. To do this for an individual triangle, I took the coordinates of each pixel in the triangle, using sk.draw.polygon to get such coordinates, and transformed them with some matrix $A$ (see below for details on how to calculate $A$).

With these transformed coordinates, I sampled from the original image, and populated the warped image with the sampled values. With this, we can warp an image by iterating through all triangles and populating the triangles with their corresponding values. Then, we can warp both images to the mid-way face and cross fade the two.

Warped images:

Final Result:

To calculate $A$, I implemented a function computeAffine(src_pts, dst_pts) which solved for the following linear equation:

$\left[ \begin{array}{ccc} t_1 & t_2 & t_3\\ t_4 & t_5 & t_6\\ 0 & 0 & 1\\ \end{array} \right] \left[ \begin{array}{ccc} a_1 & b_1 & c_1\\ a_2 & b_2 & c_2 \\ 1 & 1 & 1 \\ \end{array} \right] = \left[\begin{array}{ccc} x_1 & y_1 & z_1\\ x_2 & y_2 & z_2\\ 1 & 1 & 1 \end{array} \right]$

written as $TA = X$ which simplifies to $T = XA^{-1}$, where $A$ holds the cordinates for src_pts and $X$ holds the coordinates for dst_pts. (Math exchange post for reference)

Part 3. The Morph Sequence

Finishing up the rest of the morph sequence was pretty straight forward. This basically involved changing the mid-way points to a different set of points with different ratios of each image (instead of using pts1*0.5 + pts2*0.5, I would use pts1*(weight) + pts2*(1.0 - weight) , and similarly with cross fading the images, I used a weight as such: img1*(weight) + img2*(1.0 - weight). I then iterated through all the weights between 0.0 to 1.0 to create all the frames which would be used for the full video sequence as seen below:

Part 4. The Mean Face 😡

Using the FEI dataset with the annotations that came along with it, I was able to morph each individual face to the average geometry, then take the average of all the morphed images to produce the “mean face”:

One thing to note is that only the eyes seem to be aligned, and upon further investigation, while the warping seems to work, the bottom face lines seem to be blurry because the lines are blurry in some images.

Below are some images that have pretty sharp lines in the bottom portions:

Below are some images that where there is less of a clear distinct line:

Below is a warped version of myself to the average happy and neutral face using similar methods used in the morph sequence.

The happy face seems like an exageration of my features, and it works since I’m already sorta smiling, but the neutral face seems to not work as well (the wrinkles on my face seem unnatural). Overall, I think one issue is that my face might not be positioned in the same place as the rest of the faces, and since there are no anotations for the shape of the face, the forheads seem to be signficiantly warped and out of place compared to the other features.

Part 5. Caricatures: Extrapolating from the mean

To exprapolate myself, I essentially used ratios outside of the range [0.0,1.0] (ie. -0.5 or 1.0) when morphing my image into some other image, in this case the average neutral face.

Bells & Whistles: Gender/Age/Ethnicity change

I chose to change my ethnicity to Greek, and swap my gender to female. To do so, this simply involved taking the average greek female face, labeling it, then morphing my own face to the the mean greek female face. Below are some results using alpha=0.3:

One notable feature is how skinny my eyebrows became and also the shape of my eyes.

Side note on fixing the morphing of my face

I noticed that my face might simply just be out of place, or not positioned/normalized correctly, so I re-cropped my image to match the orientation of the average face, and re-assigned the labels respectively. The results didn’t seem to improve however…