One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

Project Page | Paper | Online Demo

Overview of the Proposed Method

Let$s$be an image of a person, referred to as the source image. Let$\{d_1, d_2,...,d_N\}$be a talking-head video, called the driving video, where$d_i$’s are the individual frames, and$N$is the total number of frames.

Our goal is to generate an output video$\{y_1, y_2,...,y_N\}$, where the identity in$y_i$’s is inherited from$s$and the motions are derived from$d_i$’s.

If the person in the source image ($s$) is the same as in the driving video ($d_i$), then it's avideo reconstruction task. The generated output video ($y_i$) still takes the identity information from$s$and motion information from$d_i$.
If the person in$s$is not the same as in$d_i$, then it's amotion transfer task.

Source image feature extraction
Driving video feature extraction
Video generation

Source Image Feature Extraction

Figure:Different features extracted from the source image. (Source)

3D Appearance Feature Extraction ($F$):

Figure:Architectural design of$F$. (Source)

$K$3D Canonical Keypoint Extraction ($L$):

The Jacobians represent how a local patch around the keypoint can be transformed into a patch in another image via an affine transformation.

Figure:Architectural design of$L$. (Source)

Our canonical keypoints are formulated to be independent of the pose and expression change. They should only contain a person’s geometry signature, such as the shapes of face, nose, and eyes.

Head Pose ($H$) and Expression Extraction ($\triangle$):

Figure:Architectural design of$H$and$\triangle$. (Source)

Driving Video Feature Extraction

Figure:Different features extracted from the driving video. (Source)

Instead of extracting canonical 3D keypoints from the driving image$d$using$L$, we reuse$x_{c, k}$and$J_{c, k}$, which were extracted from the source image$s$. This is because the face in the output image must have the same identity as the one in the source image$s$. There is no need to compute them again.

Our approach allows manual changes to the 3D head pose during synthesis. Let$R_u$and$t_u$be user-specified rotation and translation, respectively. The final head pose in the output image is given by$R_d \leftarrow R_uR_d$and$t_d \leftarrow t_u + t_d$. In video conferencing, we can change a person’s head pose in the video stream freely despite the original view angle.

Video Synthesis

Figure:Video synthesis pipeline. (Source)

Figure:Architectural design of$M$. (Source)

Softmax activation is used to obtain the flow composition mask$m$, which consists of$K$3D masks. These are again combined with$K$warping flow maps$w_k$to obtain the finalcompositeflow field$w$. This is finally used to obtain the warped source feature$w(f_s)$.
Warping leads to occlusions. A 2D occlusion mask$o$is predicted to be inputted to the generator.

Figure:Architectural design of$G$. (Source)

Training the Models

one would act as a source image$s$,
and the other as the frame from the driving video$d$.

Perpetual Loss ($\mathcal{L}_P$):Perpetual loss is commonly used in image reconstruction tasks. Here's a nice description of this loss function. In short, a pre-trained VGG network is used to extract features from both the ground truth image and the reconstructed image. The $L_1$distance is computed between the features. The features are extracted from multiple hidden layers of varying resolutions. Besides regular VGG (trained on ImageNet) the authors have also used a pre-trainedfaceVGG network for obvious reasons.
GAN Loss ($\mathcal{L}_G$):The authors have usedpatch GAN implementation along with hinge loss. Check out this quick summary here.
Equivalence Loss ($\mathcal{L}_E$):This loss ensures the consistency of the estimated keypoints. Let$x_d$be the detected keypoints for the input image$d$. When a known transformation$T$is applied to the image ($T(d)$), the detected keypoints should be transformed in the same way.$L_1$distance is minimized such that$||x_d - T^{-1}(x_{T(d)}) ||_1$tends to zero. Here$T^{-1}$is the inverse of the known transform. The same logic is applicable to the jacobians of the keypoints.
Key Prior Loss ($L_L$):This loss encourages the estimated image-specific keypoints$x_{d, k}$to spread out across the face region, instead of crowding around a small neighborhood. Distance is computed between the keypoint pairs and penalized if the distance is below some threshold.
Head Pose Loss ($L_H$):$L_1$distance is computed between the estimated head pose$R_d$and the one predicted by a pre-trained estimator$\bar{R}_d$. This approximation is as good as the pre-trained model head pose estimator.
Deformation Prior Loss ($L_{\triangle}$):This loss is simply given as$L_1$norm of the deviation such that,$\mathcal{L}_{\triangle} = ||\delta_{d, k}||_1$.

Conclusion