Barbershop: Hair Transfer with GAN-Based Image Compositing Using Segmentation Masks

Introduction

Failure of Existing Approaches

Despite the recent success of face editing based on latent space manipulation (think Image2StyleGAN and Image2StyleGAN++), most editing tasks operate on an image by changing global attributes such as pose, expression, gender, or age instead of tackling hair editing.
Another approach to image editing is to select features from reference images and mix them together to form a single, composite image. Examples of composite image editing that have seen recent progress are problems of hair transfer and face-swapping. These tasks are extremely difficult mainly for the fact that the visual properties of different parts of an image are not independent of each other. For example:
The visual qualities of hair are heavily influenced by ambient and reflected light as well as transmitted colors from the underlying face, clothing, and background.
The pose of a head influences the appearance of the nose, eyes, and mouth, and the geometry of a person’s head and shoulders influences shadows and the geometry of their hair.
Other challenges include disocclusion (meaning when a previously occluded object becomes visible) of the background which happens when the hair region shrinks with respect to the background.
Disocclusion of the face region can expose new parts of the face, such as the ears, forehead, or jawline.
The shape of the hair is influenced by pose and also by the camera intrinsic parameters, and so the pose might have to change to adapt to the hair.
Failure to account for the global consistency of an image leads to noticeable artifacts such as the different regions of the image appearing disjointed, even if each part is synthesized with a high level of realism. In order for the composite image to seem plausible.
Previous methods of hair transfer based on GANs either use a complex pipeline of conditional generators in which each condition module is specialized to represent, process, and convert reference inputs with different visual attributes (such as MichiGAN), or make use of the latent space optimization with carefully designed loss and gradient orthogonalization to explicitly disentangle hair attributes (such as LOHO). While both of these methods show very promising initial results, the authors find that they could be greatly improved. For example,
Both methods need pre-trained inpainting networks to fill holes left over by misaligned hair masks, which may lead to blurry artifacts and unnatural boundaries.
The authors state that better results can be achieved without an auxiliary inpainting network to fill the holes, as transitions between regions have higher quality if they are synthesized by a single GAN.
The previous methods do not make use of a semantic alignment step to merge semantic regions from different reference images in latent space, e.g. to align a hair region and a face region from different images.
The concepts of identity, shape, structure, and appearance were introduced in MichiGAN and then used by other approaches including LOHO in order to describe different aspects of hair lack a precise definition.

The Novelty of Barbershop

In order for the composite image to seem plausible, the authors aim to make a single coherent composite image that balances the fidelity of each region to the corresponding reference image while also synthesizing an overall convincing and highly realistic image.
The authors provide precise and formal definitions for the concepts of identity, shape, structure, and appearance:
The shape of the hair is the binary segmentation region
The identity of an image of a head encompasses all the features one would need to identify an individual
The appearance broadly refers to the fine details (such as hair color)
The structure refers to coarser features (such as the form of locks of hair)
The authors propose a novel latent space, called the $FS$ space, for representing images. This new space is better at preserving details and is more capable of encoding spatial information.
The authors also propose a new GAN-embedding algorithm for aligned embedding. Similar to previous work, the algorithm can embed an image to be similar to an input image. In addition, the image is slightly modified to conform to a new segmentation mask.
The authors further propose a novel image compositing algorithm that can blend multiple images encoded in our new latent space to yield high-quality results.
The authors achieve a significant improvement in hair transfer, with the proposed approach being preferred over existing state-of-the-art approaches by over 95% of participants in a user study.

Barbershop: How Does the Shop Operate???

Overview



(a) reference images for the face (top) and hair (bottom) features
(b) reconstructed images using the $FS$ latent space
(c) a target mask with hair region (magenta) from the hair image and all other regions from the face image
(d) alignment in $W^{+}$ space
(e) a close-up view of the face (top) and hair (bottom) in $W^{+}$ space
(f) close-up views after details are transferred
(g) an entire image with details transferred
(h) the structure tensor is transferred into the blended image but the appearance code is from $S_{face}$
the appearance code is optimized

Generator Architecture

Source: Figure 2 from the paper

Main Steps in the Barbershop Pipeline

Reference images $I_{k}$ are segmented and a target segmentation is generated automatically, or optionally the target segmentation is manually edited.
Embed input reference images $I_{k}$ to find latent codes $C^{rec}_{k} = (F^{rec}_{k}, S_{k})$
Find latent codes $C^{align}_{k} = (F^{align}_{k}, S_{k})$ such that are embeddings of images which match the target segmentation $M$ while also being similar to the input images $I_{k}$ .
A combined structure tensor $F^{blend}$ is formed by copying region $k$ of $F^{align}_{k}$ for each $k = 1...K$.
Blending weights for the appearance codes $S_{k}$ are found so that the appearance code $S^{blend}$ is a mixture of the appearances of the aligned images. The mixture weights are found using a novel masked appearance loss function.

Results

Reconstruction Results in Different Spaces

In the top row we can see that, in the $W^{+}$ space, the structure of the subject’s curly hair on the left of the image is lost, and a wisp of hair on her forehead, as well as her necklace, is removed, but they are preserved in the $FS$ space.
In the middle row we can see that, the hair and brow furrows details are important to the expression of the subject, they are not preserved in $W^{+}$ space but they are in $FS$ space.
In the bottom row we can see that, the ground-truth image has freckles, without noise optimization this is not captured in $W^{+}$ space but it is preserved in $FS$ space.

Comparison of Barbershop with Prevision SoTA Methods

Qualitative Comparison

Quantitative Comparison

Root Mean Squared Error
Peak Signal-to-Noise Ratio
Structural Similarity
VGG-based Perceptual Similarity
Learned Perceptual Image Patch Similarity
Fréchet Inception Distance

Limitations of Barbershop

Even though the capacity of the latent space has been increased for Barbershop, it is difficult to reconstruct underrepresented features from the latent space such as jewelry.
Issues such as occlusion can produce confusing results. For example, thin wisps of hair which also partially reveal the underlying face are difficult to capture.
Many details such as the hair structure are difficult to preserve when aligning embeddings, and when the reference and target segmentation masks do not overlap perfectly the method may fall back to a smoother structure.
while Barbershop is tolerant of some errors in the segmentation mask input, large geometric distortions cannot be compensated.

Conclusion

In this report, we discuss Barbershop, a novel framework for GAN-based image editing.
Barbershop enables a user to interact with images by manipulating segmentation masks and copying content from different reference images.
The authors propose a new latent space that combines the commonly used $W^{+}$ style code with a structure tensor. The use of the structure tensor makes the latent code more spatially aware and enables us to preserve more facial details during editing.
The authors also propose a new GAN-embedding algorithm for aligned embedding. Similar to previous work, the algorithm can embed an image to be similar to an input image. In addition, the image can be slightly modified to conform to a new segmentation mask.
The authors also propose a novel image compositing algorithm that can blend multiple images encoded in our new latent space to yield a high-quality result.
The results produced by Barbershop show significant improvements over the current state of the art. In a user study, our results are preferred over 95% of the time.
Finally, we discussed the limitations posed by Barbershop.

Similar Reports