Digging Into StyleGAN-NADA for CLIP-Guided Domain Adaptation



Can a generative model be trained to produce images from a specific domain, guided only by a text prompt, without seeing any image? In other words: can an image generator be trained “blindly”?

Table of Contents

Let’s Talk About the Precursors

StyleGAN

A mapping network converts a latent code $z$, sampled from a Gaussian prior, to a vector $w$ in a learned latent space $\mathcal{W}$.
These latent vectors are then fed into the synthesis network, to control feature (or convolutional kernel) statistics.

StyleCLIP

Shortcomings of the Precursors

A Deep Dive Into StyleGAN-NADA

How can the semantic information encapsulated in CLIP be distilled in the best manner?
How should the optimization process be regularized in order to avoid adversarial solutions and mode collapse?

Training Setup

Source: Figure 2 from the paper

CLIP-based Guidance

Global Loss

$G(w)$ is the image generated by the latent code $w$ fed to the generator $G$
$t_{target}$ is the textual description of the target class
$D_{CLIP}$ is the CLIP-space cosine distance
the loss is named global, as it does not depend on the initial image or domain.

The loss leads to adversarial solutions. In the absence of a fixed generator that favors solutions on the real-image manifold, optimization overcomes the classifier (CLIP) by adding pixel-level perturbations to the image.
It sees no benefit in maintaining diversity. A mode-collapsed generator producing only one image may be the best minimizer for the distance to a given textual prompt.

Directional CLIP Loss

In order to do so, the authors first identify a cross-domain direction in CLIP-space by embedding a pair of textual prompts describing the source and target domains (e.g. “dog” and “cat”).
Then, the CLIP-space direction between images produced by the generator before and after fine-tuning must be determined. The authors do this using a two-generator setup. They begin with a generator pre-trained on a single source domain (e.g. faces, dogs, churches, or cars), and clone it.
One copy is kept frozen throughout the training process. Its role is to provide an image in the source domain for every latent code. This generator is referred to as $G_{frozen}$.
The second copy is trained. It is fine-tuned to produce images that, for any sampled code, differ from that of the source generator only along the textually described direction. This generator is referred to as $G_{train}$.
In order to maintain latent space alignment, the two generators share a single mapping network which is kept frozen throughout the process.

$\Delta T$ is the vector connecting the source and target space
$t_{source}$ and $t_{target}$ are the source and target class texts
$\Delta I$ is the vector connecting the images generated by both the generators $G_{frozen}$ and $G_{train}$ in the CLIP space
$E_{I}$ and $E_{T}$ are CLIP’s image and text encoders

It is adversely affected by mode-collapsed examples. If the target generator only creates a single image, the CLIPspace directions from all sources to this target image will be different. As such, they can’t all align with the textual direction.
It is harder for the network to converge to adversarial solutions because it has to engineer perturbations that fool CLIP across an infinite set of different instances.

Illustration of CLIP-space directions in the proposed directional loss. The images generated by both generators are embedded into CLIP-space and demand that the vector connecting them (denoted by $\Delta I$) is parallel to the vector connecting the source and the target space (denoted by $\Delta T$). Source: Figure 3 from the paper.

Layer Freezing

the $k$ most relevant layers are selected
a single training step is performed where only these relevant layers are optimized while freezing all others

Firstly,
$N_{w}$ number of latent codes belonging to $\mathcal{W}$ are randomly sampled
These latent codes are then converted to $\mathcal{W}+$ by replicating the same code for each layer
Secondly,
$N_{i}$ number of the StyleCLIP latent-code optimization method, using the aforementioned global loss
The $k$ layers for which the latent code changed most significantly are selected

An illustration of the two-step adaptive layer-freezing mechanism. Source: Figure 4 from the paper

Latent-Mapper Mining

Results

👉 Try Training StyleGAN-NADA on Google Colab

Results of Models adapted from StyleGAN2-FFHQ

Results of Models Adapted from StyleGAN2 and AFHQ-DOG

Out-of-Domain Editing

Advantages and Limitations of StyleGAN-NADA

Advantages

The proposed method is trained blindly without any image, depending on a pre-trained CLIP model as the only source of supervision. By using CLIP to guide the training of a generator rather than an exploration of its latent space, we are able to affect large changes in both styles and shape far beyond the generator’s original domain.
The ability to train generators without data leads to exciting new possibilities - from editing images in ways that are constrained almost only by the user’s creativity to synthesizing paired cross-domain data for downstream applications such as image-to-image translation.

Limitations

By relying on CLIP, the proposed approach is limited to concepts that CLIP has observed.
Textual guidance is also inherently limited by the ambiguity of natural language prompts. For example, when one describes a Raphael Painting, do they refer to the artistic style of the Renaissance painter, a portrait of Raphael's likeness, or an animated turtle bearing that name?
The proposed method works particularly well for style and fine details, but it may struggle with large-scale attributes or geometry changes. Such restrictions are common also in few-shot approaches. The authors find that a good translation often requires a fairly similar pre-trained generator as a starting point.

Conclusion

We discussed the precursors of the proposed approach: StyleGAN and StyleCLIP.
The proposed method is trained blindly without any image, depending on a pre-trained CLIP model as the only source of supervision. By using CLIP to guide the training of a generator, rather than an exploration of its latent space, we are able to affect large changes in both style and shape, far beyond the generator’s original domain.
We explored the training setup that consists of two generators initialized using the weights of a generator pre-trained on images from a source domain, such as FFHQ.
We discussed StyleCLIP's global loss as a naive solution to train the generators for StyleGAN-NADA and its shortcomings. We also discussed the Directional CLIP loss proposed by the authors to train the generators and how it overcomes the aforementioned shortcomings.
We further discussed the two-step adaptive layer-freezing mechanism in StyleGAN-NADA and how it resolves mode collapse or overfitting for domain shifts that are predominantly texture-based.
We also discussed the Latent-mapper mining trick used by the authors to ensure the completion of a full transformation.
We further showcased a wide range of out-of-domain adaptations that were enabled by the proposed method.
We discussed and noted several advantages and disadvantages of the proposed approach.
Finally, the authors hope this work can inspire others to continue exploring the world of textually-guided generation, particularly the astounding ability of CLIP to guide visual transformations.

"Perhaps, not too long in the future, our day-to-day efforts would no longer be constrained by data requirements - but only by our creativity." - The authors of the paper StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators.

Similar Articles