A Guide to Using Stable Diffusion XL with HuggingFace Diffusers and W&B

🎬 Introduction

The architecture of Stable Diffusion XL.
How we can use the 🧨 Diffusers library by 🤗 HuggingFace to generate stunning images.
How we can use Weights & Biases to manage our experiments and ensure reproducibility for our generations.
And how we can use Compel to perform prompt-weighing in order to achieve better control over our generated images with Stable Diffusion XL.



Table of Contents

🦄 A Brief Overview of Stable Diffusion XL

SDXL leverages a larger UNet backbone. Three times the size, in fact. The increase in parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder.
SDXL leverages multiple novel conditioning schemes and is trained on multiple aspect ratios.
The image generation pipeline also leverages a specialized high-resolution refiner model which is used to improve the visual fidelity of samples generated by SDXL using an image-to-image diffusion technique.
The base diffusion model generates initial latent tensors of size 128x128, which can be passed through a Variational Autoencoder (VAE) model with KL loss to generate the high-resolution image.
The latent tensors could also be passed on to the refiner model that applies SDEdit, using the same prompt. Although the base SDXL model is capable of generating stunning images with high fidelity, using the refiner model useful in many cases, especially to refine samples of low local quality such as deformed faces, eyes, lips, etc.
SDXL and the refinement model use the same autoencoder.

Visualization of the two-stage pipeline for SDXL. Source: Figure 1 from the paper.

💨 Creating the Diffusion Models

🎨 Performing Text-Conditional Image Generation

🪄 Exploring the Results on Weights & Biases 🐝

🦋 Using Compel to Assign Weights to the Tokens of Our Prompts

👨‍🎨 Some Tips to Generate the Best Images Using SDXL

SDXL is a machine-learning model and doesn't really know how to read our minds. Hence, we need to be extremely specific with our prompts.
Make use of negative prompts to remove the undesired qualities or features from the generated image.
Make use of prompt weighing mechanics from Compel to gain more fine-grained control over your images.
Do not expect the generated images to achieve perfect photorealism or render legible text.
Make use of the refiner whenever possible, in a lot of situations, it can help with deformed faces and other features from the output of the base model.
SDXL does not guarantee to be free of any kind of social or regional biases.
SDXL works especially well with images between 768 and 1024.
Since SDXL has two text encoders, we can pass a different prompt for each of the text encoders. We can even pass different parts of the same prompt to the text encoders.
Finally, generating the perfect image that you're imagining is going to be an iterative process, spanning several experiments. Check out how many iterations it took me to render the following image 👇

🏁 Conclusion

In this report, we discussed and explore the amazing text-conditional image generation capabilities of Stable Diffusion XL 1.0.
We briefly explored the architecture of SDXL as proposed by the paper SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.
We then explored how we can use the 🧨 Diffusers library by 🤗 HuggingFace to generate stunning images using SDXL.
We also explored how we can use Weights & Biases to manage our experiments and not only ensure reproducibility for our generations but maintain our own SDXL-generated art gallery!
We also discussed the different techniques to leverage the refiner model in the SDXL pipeline. For more detailed information on the usage of the refiner module, you can check out the original docs.
We also learned how we can use Compel to assign weight to the tokens in our prompts in order to achieve better control of the nature of our generations with SDXL.
I am immensely grateful to Sayak Paul for the guidance he provided while writing this report. I am also grateful to my friends Atanu, Arindam, and Dripto for sharing with me cool prompt ideas that helped me generate the stunning images used in this report.