Understand how the DeOldify model works, reproduce and visualize the results obtained by the author. Made by Boris Dayma using Weights & Biases
Boris Dayma


DeOldify is a colorizer made by Jason Antic with fastai.
We feed black & white images and want to get them colorized in the most realistic manner.
The intent of this report is to understand how the model (which makes use of a GAN) works and reproduce the results obtained by the author.

Pre-training the Generator

The dataset is made of images taken from ImageNet.
Inputs are black & white versions of the images and outputs are the original colorized version.
The first step is to train a generator by feeding the black & white images and trying to predict the colorized version.
We can see below the training graph which contains 3 gaps due to the different steps of this training:
The predicted samples show images with a few colors but "brownish" overall which represents an average color predicted by the model when it is not sure of what to use.
The most vivid colors used are for the sky, water, people's skin, and vegetation which seems to be detected more easily and have lower variance in color (vs a T-shirt for example).

Pre-Training the Critic

We then pre-train a critic whose objective is to identify which images are real and which ones come from the pre-trained generator.
This will help in getting a quicker start on the traditional "GAN" training by having a pre-trained generator and a pre-trained critic.


Finally, traditional GAN training is performed.
The following sequence of steps is performed several times:
The author decides manually when to stop GAN training by looking at the current generated samples. There is a moment where the quality of generated pictures actually decreases.
For the purpose of this experiment, we pre-define a number of training updates that will be performed at each step. Below graphs represent several experiments with different batch sizes.
In practice, the author would perform the fastidious task of manually deciding the optimal number of updates to perform at each step over several experiments until reaching satisfactory results.

Predicted images

While not perfect, it is interesting to see how this method clearly generates more colorful samples.
The quality of the results will depend on the art of tweaking the training duration at each step and performing several experiments.