Diffusion Models

My first attempts producing images at 256 x 256 weren't very successful, however lowering the resolution to 128 x 128 produced good results out of the box with the settings from the original paper.
While the paper doesn't report FID scores (at any resolution), they don't seem extremely competitive, nor do they continue going down with more training iterations.
The sample quality becomes okay after 50K steps (with 8 GPUs and batch size 16) and good to really good after 100K steps. The EMA model takes a bit longer to converge to high quality images, but suffers less from issues where all the sampled results tend to be biased towards a particular color.
Clipping $x_0$ to [-1, 1] during sampling was important to obtain high quality samples. This phenomena is discussed into more detailed in the Imagen paper.

Samples