For reasons we won't belabor, video conferencing has gained a tremendous user base in the last year. But despite its rise, it's not accessible to many because of the high network bandwidth required to carry both video and speech in real-time. Deep learning techniques (especially GANs) can deliver high-quality video via image compression at lower bit rates.

But before we dig into this research on a really interesting application of deep learning called talking head synthesis, we recommend checking out the brief video below. It'll help anchor the research as we dig a bit deeper into "One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing" (Wang et al., 2020).

Let's first level-set on the notations that we'll be using and get a little clarity on the goal of this research. From the paper:

Let s be an image of a person, referred to as the source image. Let \{d_1, d_2,...,d_N\} be a talking-head video, called the driving video, where d_i’s are the individual frames, and N is the total number of frames.

Our goal is to generate an output video \{y_1, y_2,...,y_N\}, where the identity in y_i’s is inherited from s and the motions are derived from d_i’s.

Depending on the s (i.e. the image of the person), the goal can be either one of the two broader deep learning tasks:

- If the person in the source image (s) is the same as in the driving video (d_i), then it's a video reconstruction task. The generated output video (y_i) still takes the identity information from s and motion information from d_i.
- If the person in s is not the same as in d_i, then it's a motion transfer task.

To inherit the features from the source image and control the novel synthesis of the talking head, the authors devised an unsupervised approach for learning a set of 3D keypoints and their decomposition.

The proposed method can be divided into three major steps:

- Source image feature extraction
- Driving video feature extraction
- Video generation

The beauty of the proposed solution is the joint training of all the architectures, in all three stages. We will look into the training details in a moment, but let's first quickly look at the architectural design for source image feature extraction.

Figure: Different features extracted from the source image. (Source)

Four separate neural network architectures (well, three actually) are used to extract identity-specific information. Digging in a bit:

Using a neural network F, the source image s is mapped to a 3D appearance feature volume f_s. The network F consists of multiple downsampling blocks followed by a number of 3D residual blocks to compute the 3D feature volume f_s.

Figure: Architectural design of F. (Source)

Using a canonical 3D keypoint detection network L, a set of K canonical 3D keypoints x_{c, k} \in \R^3 and their Jacobians J_{c, k} \in \R^{3 \times 3} are extracted from s.

The Jacobians represent how a local patch around the keypoint can be transformed into a patch in another image via an affine transformation.

The authors have used a U-Net style encoder-decoder to extract canonical keypoints.

Figure: Architectural design of L. (Source)

Our canonical keypoints are formulated to be independent of the pose and expression change. They should only contain a person’s geometry signature, such as the shapes of face, nose, and eyes.

A pose estimation network H is used to estimate the head pose of the person in s. It is parameterized by a rotation matrix R_s \in \R^{3 \times 3} and a translation vector t_s \in \R^3. The rotation matrix R_s in practice is composed of three matrices - yaw, pitch

Expression deformation estimation network \triangle is used to estimate deformation of keypoints from the neutral expression. Thus there are K 3D deformations \delta_{s, k}.

Note that the authors have used a common backbone with shared weights for both H and \triangle. This is evident from the proposed architecture for the same.

Note: The same architecture is used to extract motion-related information from the driving video.

Using the information from all 3 architectures, the authors have proposed a transformation T to obtain the final 3D keypoints x_{s, k} and their Jacobians J_{s, k} for the source image. T_x is applied to the keypoints and T_j to the Jacobians such that:

x_{s, k} = T_x(x_{c, k}, R_s, t_s, \delta_{s, k}) \equiv R_sx_{c, k} + t_s + \delta_{s, k}

J_{s, k} = T_j(J_{c, k}, R_s) \equiv R_sJ_{s, k}

Figure: Different features extracted from the driving video. (Source)

The driving video is used to extract motion-related information. To this end, head pose estimation network H and expression deformation estimator network \triangle is used. Note that 3D feature extractor (F) and canonical key point extraction network (L) are not used. This is inclined to the formulated goal. From the paper,

Instead of extracting canonical 3D keypoints from the driving image d using L, we reuse x_{c, k} and J_{c, k}, which were extracted from the source image s. This is because the face in the output image must have the same identity as the one in the source image s. There is no need to compute them again.

Using the identity-specific information (x_{c, k} and J_{c, k}) and motion-related information, final 3D keypoints x_{d, k} and their Jacobians J_{d, k} is computed for the driving video. The same transformations T_x and T_j are used such that,

x_{d, k} = T_x(x_{c, k}, R_d, t_d, \delta_{d, k}) \equiv R_dx_{c, k} + t_d + \delta_{d, k}
J_{d, k} = T_j(J_{c, k}, R_d) \equiv R_dJ_{d, k}

This 3D keypoint and its Jacobian are derived for every frame in the driving video. Since identity-specific information is used for computing final 3D keypoints and Jacobians for the driving video frame, we can provide user-specific rotation and translation matrix to change a person's head pose.

Our approach allows manual changes to the 3D head pose during synthesis. Let R_u and t_ube user-specified rotation and translation, respectively. The final head pose in the output image is given by R_d \leftarrow R_uR_d and t_d \leftarrow t_u + t_d. In video conferencing, we can change a person’s head pose in the video stream freely despite the original view angle.

Figure: Video synthesis pipeline. (Source)

The 3D keypoints and Jacobians extracted from the source image and the driving video frame are used to estimate warping flow maps. This flow map w_k is generated based on the k^{th} keypoint using the first-order approximation. This flow field w_k is used to warp the source feature w_k(f_s) where k \in \{1,2,..,K\}. First Order Motion Model for Image Animation by Siarohin et al. might be a useful read. You can check out this paper's summary by Lavanya Shukla here.

These warped features are fed to a motion field estimator network M which is a 3D U-Net style network. As shown in the figure, two outputs are estimated using this network - mask m and occlusion o.

Figure: Architectural design of M. (Source)

- Softmax activation is used to obtain the flow composition mask m, which consists of K 3D masks. These are again combined with K warping flow maps w_k to obtain the final composite flow field w. This is finally used to obtain the warped source feature w(f_s).
- Warping leads to occlusions. A 2D occlusion mask o is predicted to be inputted to the generator.

The authors have used a generator network G that takes the warped 3D source feature map w(f_s) and first projects them back to the 2D feature. This is then multiplied with the occlusion mask o followed by a series of 2D residual blocks and upsampling layers to obtain the final image.

Figure: Architectural design of G. (Source)

To summarize so far, we have source image s and driving video d. The task is to generate output video y such that it has the identity-specific information from s and motion-specific information from d. To obtain identity-specific information different neural networks are used and the same goes for motion-specific information. These pieces of information are used to obtain K 3D keypoints and Jacobians for both s and d.

These keypoints and Jacobians are then used to warp the source appearance feature f_sextracted from s from which they generate the final output image using a generator network G.

So how did the authors train this system? We'll cover the procedure, the dataset they used, and their losses.

The authors used a dataset of talking-head videos to train their models here. They mention the use of VoxCeleb2 and TalkingHead-1KH for evaluation, though it is a bit unclear which dataset they used for training.

For each video, two frames were sampled:

- one would act as a source image s,
- and the other as the frame from the driving video d.

The networks F, L, H, \triangle, M, and G are trained together by minimizing the following loss:

\mathcal{L} = \mathcal{L}_P(d, y) + \mathcal{L}_G(d, y) + \mathcal{L}_E(\{x_{d,k}\}, \{J_{d, k}\}) + \mathcal{L}_L(\{x_d, k\}) + \mathcal{L}_H(R_d, \bar{R}_d) + \mathcal{L}_{\triangle}(\{\delta_{d, k}\})

Let's go through each term one-by-one:

- Perpetual Loss (\mathcal{L}_P): Perpetual loss is commonly used in image reconstruction tasks. Here's a nice description of this loss function. In short, a pre-trained VGG network is used to extract features from both the ground truth image and the reconstructed image. The L_1 distance is computed between the features. The features are extracted from multiple hidden layers of varying resolutions. Besides regular VGG (trained on ImageNet) the authors have also used a pre-trained face VGG network for obvious reasons.
- GAN Loss (\mathcal{L}_G): The authors have used patch GAN implementation along with hinge loss. Check out this quick summary here.
- Equivalence Loss (\mathcal{L}_E): This loss ensures the consistency of the estimated keypoints. Let x_d be the detected keypoints for the input image d. When a known transformation T is applied to the image (T(d)), the detected keypoints should be transformed in the same way. L_1 distance is minimized such that ||x_d - T^{-1}(x_{T(d)}) ||_1 tends to zero. Here T^{-1} is the inverse of the known transform. The same logic is applicable to the jacobians of the keypoints.
- Key Prior Loss (L_L): This loss encourages the estimated image-specific keypoints x_{d, k}to spread out across the face region, instead of crowding around a small neighborhood. Distance is computed between the keypoint pairs and penalized if the distance is below some threshold.
- Head Pose Loss (L_H): L_1 distance is computed between the estimated head pose R_d and the one predicted by a pre-trained estimator \bar{R}_d. This approximation is as good as the pre-trained model head pose estimator.
- Deformation Prior Loss (L_{\triangle}): This loss is simply given as L_1 norm of the deviation such that, \mathcal{L}_{\triangle} = ||\delta_{d, k}||_1.

The models are trained using the coarse-to-fine technique. ADAM optimizer with a learning rate of 0.0002 is used to train the model on 256x256 resolution images for 100 epochs. This is then fine-tuned on 512x512 resolution images for 10 epochs.

The results of this research are really quite promising. The techniques we talked about here resulted in a 10X bandwidth reduction and, if you had a chance to look at the video in our introduction, you can see that the video quality is incredibly high considering that reduction. Models like this could make it possible to democratize access to video conferencing and reduce strain on networks, especially in residential and rural areas where bandwidth is already harder to come by.

The paper is packed with implementation details, failure modes, and other nitty-gritty. I highly recommend going through the paper especially the appendix section.

I would also like to thank Justin Tenuto for his edits.