How to Evaluate GANs using Frechet Inception Distance (FID)

Try to train your own simple GAN $\rightarrow$

Table of Contents

class GeneratedImageLogger(tf.keras.callbacks.Callback):
    def __init__(self, noise_dim, batch_size=32):
      super(GeneratedImageLogger, self).__init__()
      self.noise = tf.random.normal([batch_size, noise_dim])

    def on_epoch_end(self, logs, epoch):
      generated_image = generator(self.noise, training=False)

      wandb.log({"gen_images": [wandb.Image(image)
                            for image in generated_image]})

GAN Evaluation



In supervised image classification, evaluation is straightforward. We have to compare the predicted output to the actual output.
However, with a GAN, you pass in some random noise to get this fake(generated) image. We want this generated image to look as real as possible. So, how exactly can you quantify the realism of this generated image? Or how exactly can you evaluate GAN?

Fidelity: We want our GAN to generate high quality images.
Diversity: Our GAN should generate images that are inherent in the training dataset.

Pixel Distance: This is a naive distance measure where we subtract two images' pixel values. However, this is not a reliable metric.
Feature Distance: We use a pre-trained image classification model and use the activation of an intermediate layer. This vector is the high-level representation of the image. Computing a distance metric with such representation gives a stable and reliable metric.

Frechet Inception Distance(FID)

Evaluate your GAN using FID $\rightarrow$

Implement FID using TensorFlow

To evaluate your GAN using FID $\rightarrow$

To use the pre-trained Inception V3 model:

inception_model = tf.keras.applications.InceptionV3(include_top=False, 
                              weights="imagenet", 
                              pooling='avg')

Compute the embeddings for real images and generated images. Note that the authors of GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium have recommended using a minimum sample size of 10,000 to calculate the FID otherwise the true FID of the generator is underestimated.

def compute_embeddings(dataloader, count):
    image_embeddings = []

    for _ in tqdm(range(count)):
        images = next(iter(dataloader))
        embeddings = inception_model.predict(images)

        image_embeddings.extend(embeddings)

    return np.array(image_embeddings)

count = math.ceil(10000/BATCH_SIZE)

# compute embeddings for real images
real_image_embeddings = compute_embeddings(trainloader, count)

# compute embeddings for generated images
generated_image_embeddings = compute_embeddings(genloader, count)

real_image_embeddings.shape, generated_image_embeddings.shape

Here trainloader and genloader are tf.data dataset. Check out the colab notebook for implementation details.
With real and generated images embeddings, we will compute the FID score.

 def calculate_fid(real_embeddings, generated_embeddings):
     # calculate mean and covariance statistics
     mu1, sigma1 = real_embeddings.mean(axis=0), np.cov(real_embeddings, rowvar=False)
     mu2, sigma2 = generated_embeddings.mean(axis=0), np.cov(generated_embeddings,  rowvar=False)
     # calculate sum squared difference between means
    ssdiff = np.sum((mu1 - mu2)**2.0)
    # calculate sqrt of product between cov
    covmean = linalg.sqrtm(sigma1.dot(sigma2))
    # check and correct imaginary numbers from sqrt
    if np.iscomplexobj(covmean):
       covmean = covmean.real
     # calculate score
     fid = ssdiff + np.trace(sigma1 + sigma2 - 2.0 * covmean)
     return fid

 fid = calculate_fid(real_image_embeddings, generated_image_embeddings)

Observations

The FID score decreases with the better model checkpoint. One can pick a model checkpoint that generates a low FID score for inference.
The FID score is relatively high because the Inception model is trained on Imagenet which constitutes natural images while our GAN is trained on the FashionMNIST dataset.

Shortcomings of FID

It uses a pre-trained Inception model, which may not capture all features. This can result in a high FID score as in the above case.
It needs a large sample size. The minimum recommended sample size is 10,000. For a high-resolution image(say 512x512 pixels) this can be computationally expensive and slow to run.
Limited statistics(mean and covariance) are used to compute the FID score.