A Guide to Prompt Engineering for Stable Diffusion



How we can establish simple image generation workflows using the 🧨 Diffusers library by 🤗 HuggingFace. We will primarily be sticking to the Stable Diffusion family of models for the scope of this report.
Managing and visualizing our image generation experiments with Weights & Biases.
The most basic "prompt engineering" technique to dramatically improve the quality of your images.
How to properly structure a great prompt for diffusion models.
How to further elevate our generated images using negative prompts.

Table of Contents (click to expand)

A Sneak Peek At The Code & Results





💻 The Image Generation Workflow

# Install all the dependencies
!pip install diffusers accelerate transformers wandb

# Imports
import torch
from diffusers import DiffusionPipeline
from wandb.integration.diffusers import autolog

# Define the Stable Diffusion XL pipeline
pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)

# Set the device for the pipeline
pipeline = pipeline.to("cuda")
# uncomment the following line if you encounter OOM issues
# pipeline.enable_sequential_cpu_offload()

# Set the prompt
prompt = "children dressed like a gangster"
# Set the negative prompt, or leave it `None` if you don't want to use it
negative_prompt = None

# Setting a seed would ensure reproducibility of the experiment.
generator = torch.Generator(device="cuda").manual_seed(42)

# Call WandB Autolog for Diffusers
autolog(init=dict(project="diffusers-prompt-engineering"))

# Add the callback to the pipeline, and execute the pipeline.
image = pipeline(
    prompt,
    negative_prompt=negative_prompt,
    generator=generator,
    height=1024,
    width=1024,
)





😈 The Devil's in the Details

🫀 The Anatomy of a Good Prompt

The Subject is the core idea behind the image. In the aforementioned demonstration, we used the subject "children dressed like a gangster".
The Medium that was used to deliver the artwork. Medium has a strong effect because one keyword alone can dramatically change the style. Some examples can be illustration, oil painting, 3d rendering, photography, cinematic shot, etc. In the aforementioned demonstration, we use the medium "photograph".
The artistic style that you want to capture, for example, impressionist, surrealist, gothic, art deco, etc. In the aforementioned demonstration, we used the style "hyper realistic". Note that we can also mention multiple styles in the prompts and experiment with mixing and matching different styles.
The Names of Artists whose unique aesthetic styles you might want to mimic. For example, you can add phrases like "in the style of Van Gogh", "in the style of Akira Kurosawa", "Studio Ghibli" or even "Abanindranath Tagore". You can also pass names of famous modern artists with unique styles from sites like Artstation, DeviantArt, etc.
Lighting and Color Palette are important aspects of any artwork that can be specified using phrases like surrealist lighting, dark lighting, muted color palette, vibrant, saturated, etc.
Additional Details such as names of cameras, names of rendering techniques, rendering engines, and resolution can also be included depending on the vision of the artist.

😉 A little Negativity goes a Long Way

🎨 Iterative Refinement of Diffusion Models with Prompt Engineering

🙏 Acknowledgements

This report wouldn't have been possible without the stream of constant advice and support from Sayak Paul.
Thanks to Arindam Das and Atanu Sarkar whose prompting experiments helped me a lot with this report.
I was hugely inspired to write this report after taking the Crash course in generative AI: Stable Diffusion, DALL·E & Midjourney by PromptHero. Adapting what I learned from this course in terms of MidJourney and A1111 to Diffusers is turning out to be a fun exercise.
The works of PromtHero community members FAngel, Atheist, and ArtEnthusiast inspired me a lot in crafting the prompts showcased in this report.
The article Stable Diffusion prompt: a definitive guide, served as an excellent guide in subject-driven prompt crafting, which is the prompting technique that is discussed in this report.

🏁 Conclusion

In this report, we discuss the art and science of prompt engineering for diffusion models primarily focussing on the Stable Diffusion family of models.
First, we establish a simple image generation workflow using the 🧨 Diffusers library by 🤗 HuggingFace. We also demonstrate the use of the Weights & Biases integration for Diffusers to automatically keep track of all aspects of our image generation experiments, including prompts, configs, and images.
We explore the most basic prompt engineering technique of stringing together descriptive phrases to a base prompt idea to dramatically improve the quality of our images.
We then explored what elements and phrases constitute the anatomy of a good prompt.
We then explored the use of negative prompts to both eliminate unwanted features from our images and also to improve their overall quality.
Finally, we explored the process of iterative refinement applied to the process of generating images using machine learning models.