An Overview of Instance Aware Image Colorization



Paper | Code | Google Colab

Table of Contents

An Introduction to Image Colorization

Scribble-based colorization: The colorization process relies on high-level user scribbles like color paints or strokes to guide the process. This process thus reduces the plausible image space to generate convincing results. However, at the cost of intensive manual labor to provide careful guidance. Imagine applying this technique to an old classic black-and-white video (movie).
Example-based colorization: To reduce human efforts, this technique colorizes images with the color statistics transferred from a reference image. The result generated depends on the semantic similarity between the reference image and the grayscale input image. This process is not entirely automatic.
Learning-based colorization: Deep convolutional neural network-based frameworks have automated the colorization process in a true sense. Most of the existing architectures address two key components for convincing colorization - semantics, and multi-modality. Thus the framework should keep the semantics of the image intact, and it should be able to produce a wide variety of colors for the same object.
Instance-aware colorization: Learning-based colorization has achieved impressive performance but suffers visual artifacts when processing images with multiple foreground objects or multiple objects in a cluttered background. This brings us to this paper, where the authors' critical insight is to have a clear object-background separation that dramatically improves colorization performance.






Overview of the Proposed Method

Why?

Instead of learning to colorize the entire image, it learns to colorize individual instances(objects) in an image. This is a substantially easier task since the network is not confused with the background.
The instance-based colorization allows the instance network to learn object-level representations allowing better color distribution for that object.

How?

Figure 2: Overview of the proposed method. (Source)

Off-the-shelf pre-trained model to detect object instances and produce cropped object images.
Two backbone networks trained end to end, for instance, and full-image colorization, respectively. They are identical networks with different weights and can be any of the existing colorization architectures like DeOldify.
A fusion module to selectively blend features extracted from layers of the two colorization networks.

Object Detection

Figure 3: Object detection pipeline of the InstColorization framework.

Image Colorization Backbone

Figure 4: Network architecture contains two branches of colorization networks, one for colorizing the instance images and the other for colorizing the full image.

Fusion Module

Figure 5: The feature fusion module. (Source)

The module takes two inputs:
The feature map($f_j^X$) from full-image nework at the $j^{th}$ layer.
A bunch of instance features and corresponding bounding boxes $\{f_j^{X_i}, B_i\}_{i=1}^N$.
For both kinds of features, a small neural network with three convolutional layers are learned to predict full-image weight map $W_F$ and per-instance weight map $W_I^i$. These weight maps are used to fuse the features.
To fuse $f_j^{X_i}$ to $f_j^X$, the input bounding boxes $B_i$ are used, which determines the size and location of the instances. Note that the per-instance feature maps and the per-instance weight map are resized to match the size of the full image. The resized per-instance feature map is $f_j^{\bar X_i}$ and per-instance weight map is $\bar W_I^i$.
The weight maps are stacked, and softmax is applied on each pixel and obtains the fused feature using a weighted sum.
$f_j^{\tilde X} = f_j^X \bullet W_F + \sum_{i=1}^N f_j^{\bar X_i} \bullet \bar W_I^i$
Here, N is the number of instances per image.

Results



Training Procedure

Training Dataset

Evaluation Metrics

Training Details

First, the full-colorization network, a backbone network from an existing image colorization architecture like that of DeOldify, is trained. The authors have used the network proposed in Real-Time User-Guided Image Colorization with Learned Deep Priors. They have used the pre-trained weights to initialize the training process. They have trained the model for two epochs with a learning rate of 1e-5.
They then train the instance-aware network using the trained weights of the full-colorization network in the first step. This is more of a fine-tuning step than the extracted instances from the images instead of the entire image. They have trained the model for five epochs with a learning rate of 5e-5.
Finally, the fusion module is trained. The weights in both full and instance colorization networks are frozen.

Ending Note