One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

Project Page | Paper | Online Demo

Overview of the Proposed Method

Let $s$ be an image of a person, referred to as the source image. Let $\{d_1, d_2,...,d_N\}$ be a talking-head video, called the driving video, where $d_i$’s are the individual frames, and $N$ is the total number of frames.

Our goal is to generate an output video $\{y_1, y_2,...,y_N\}$, where the identity in $y_i$’s is inherited from $s$ and the motions are derived from $d_i$’s.

If the person in the source image ($s$) is the same as in the driving video ($d_i$), then it's a video reconstruction task. The generated output video ($y_i$) still takes the identity information from $s$ and motion information from $d_i$.
If the person in $s$ is not the same as in $d_i$, then it's a motion transfer task.

Source image feature extraction
Driving video feature extraction
Video generation

Source Image Feature Extraction

Figure: Different features extracted from the source image. (Source)

3D Appearance Feature Extraction ($F$):

Figure: Architectural design of $F$. (Source)

$K$3D Canonical Keypoint Extraction ($L$):

The Jacobians represent how a local patch around the keypoint can be transformed into a patch in another image via an affine transformation.

Figure: Architectural design of $L$. (Source)

Our canonical keypoints are formulated to be independent of the pose and expression change. They should only contain a person’s geometry signature, such as the shapes of face, nose, and eyes.

Head Pose ($H$) and Expression Extraction ($\triangle$):

Figure: Architectural design of $H$ and $\triangle$. (Source)

Driving Video Feature Extraction

Figure: Different features extracted from the driving video. (Source)

Instead of extracting canonical 3D keypoints from the driving image $d$ using $L$, we reuse $x_{c, k}$ and $J_{c, k}$, which were extracted from the source image $s$. This is because the face in the output image must have the same identity as the one in the source image $s$. There is no need to compute them again.

Our approach allows manual changes to the 3D head pose during synthesis. Let $R_u$ and $t_u$be user-specified rotation and translation, respectively. The final head pose in the output image is given by $R_d \leftarrow R_uR_d$ and $t_d \leftarrow t_u + t_d$. In video conferencing, we can change a person’s head pose in the video stream freely despite the original view angle.

Video Synthesis

Figure: Video synthesis pipeline. (Source)

Figure: Architectural design of $M$. (Source)

Softmax activation is used to obtain the flow composition mask $m$, which consists of $K$ 3D masks. These are again combined with $K$ warping flow maps $w_k$ to obtain the final composite flow field $w$. This is finally used to obtain the warped source feature $w(f_s)$.
Warping leads to occlusions. A 2D occlusion mask $o$ is predicted to be inputted to the generator.

Figure: Architectural design of $G$. (Source)

Training the Models

one would act as a source image $s$,
and the other as the frame from the driving video $d$.

Perpetual Loss ($\mathcal{L}_P$): Perpetual loss is commonly used in image reconstruction tasks. Here's a nice description of this loss function. In short, a pre-trained VGG network is used to extract features from both the ground truth image and the reconstructed image. The $L_1$ distance is computed between the features. The features are extracted from multiple hidden layers of varying resolutions. Besides regular VGG (trained on ImageNet) the authors have also used a pre-trained face VGG network for obvious reasons.
GAN Loss ($\mathcal{L}_G$): The authors have used patch GAN implementation along with hinge loss. Check out this quick summary here.
Equivalence Loss ($\mathcal{L}_E$): This loss ensures the consistency of the estimated keypoints. Let $x_d$ be the detected keypoints for the input image $d$. When a known transformation $T$ is applied to the image ($T(d)$), the detected keypoints should be transformed in the same way. $L_1$ distance is minimized such that $||x_d - T^{-1}(x_{T(d)}) ||_1$ tends to zero. Here $T^{-1}$ is the inverse of the known transform. The same logic is applicable to the jacobians of the keypoints.
Key Prior Loss ($L_L$): This loss encourages the estimated image-specific keypoints $x_{d, k}$to spread out across the face region, instead of crowding around a small neighborhood. Distance is computed between the keypoint pairs and penalized if the distance is below some threshold.
Head Pose Loss ($L_H$): $L_1$ distance is computed between the estimated head pose $R_d$ and the one predicted by a pre-trained estimator $\bar{R}_d$. This approximation is as good as the pre-trained model head pose estimator.
Deformation Prior Loss ($L_{\triangle}$): This loss is simply given as $L_1$ norm of the deviation such that, $\mathcal{L}_{\triangle} = ||\delta_{d, k}||_1$.

Conclusion