Low-Light Image Enhancement: Lighting up Images in the Deep Learning Era



runs.summary["Dark-Images-Table"]

⏎

Filter

 - 6 of 6

Export as CSV

Columns...

Reset table

Type

Image

1

2

3

4

5

6

Dark-Images

Examples of images degraded by sub-optimal lighting conditions.

5



Table of Contents

Traditional Methods for Low-Light Enhancement

Histogram Equalization: this is a technique used to enhance the contrast of an image by redistributing the intensity values of the pixels. In low-light images, the majority of the pixel values are concentrated towards the lower end of the intensity range, resulting in a dark and dull appearance. Histogram equalization can be applied to these images to spread out the pixel values across the entire range of intensities, resulting in a brighter and more vibrant image.
Retinex Models: a typical Retinex model-based approach decomposes a lowlight image into a reflection component and an illumination component by priors or regularizations. The estimated reflection component is treated as the enhanced result.

Unrealistic Enhancement: both Histogram Equalization and Retinex models face the problem of over-enhancement of images. This can be especially problematic for images with sharp edges and textures, which may become exaggerated and appear artificial.
Noise: the noise already present in the images is usually ignored in such an approach, thus, it remains or becomes amplified in the enhanced results.
Computational complexity: the runtime is relatively long because of the complicated optimization processes in the case of Retinex models and computation of image histogram and cumulative distribution function in the case of Histogram Equalization.
Manual Tweaks: both approaches require manual tweaks and interventions to account for edge cases.

Deep Learning for Low-Light Enhancement

An ongoing timeline of deep learning models for low-light image enhancement

MirNetv2, as proposed by the paper Learning Enriched Features for Fast Image Restoration and Enhancement
Zero-DCE, as proposed by the paper Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement
NAFNet, as proposed by the paper Simple Baselines for Image Restoration

How to Train Your Dragons Models

Restorers currently provide clean and well-documented implementations of all of the aforementioned models using TensorFlow and Keras for easy training, inference, and benchmarking of such models.
In the near future, we also intend to publish high-quality pre-trained models for public usage as part of this project.
You can install restorers using:

pip install --upgrade pip setuptools
pip install git+https://github.com/soumik12345/restorers.git



The Dataset and the Input Pipeline

MirNetv2: Learning Enriched Features for Fast Image Restoration and Enhancement

A novel feature extraction model that obtains a complementary set of features across multiple spatial scales while maintaining the original high-resolution features to preserve precise spatial details.
A regularly repeated mechanism for information exchange, where the features from coarse-to-fine resolution branches are progressively fused together for improved representation learning.
A new approach to fuse multi-scale features using a selective kernel network that dynamically combines variable receptive fields and faithfully preserves the original feature information at each spatial resolution.
A recursive residual design that progressively breaks down the input signal in order to simplify the overall learning process and allows the construction of very deep networks.



Defining the MirNetv2 Model

# import MirNetv2 from restorers
from restorers.model import MirNetv2

# define the MirNetv2 model; this gives us a `tf.keras.Model`
model = MirNetv2(
    channels=80, # number of channels in the feature map
    channel_factor=1.5, # number of multi-scale residual blocks
    num_mrb_blocks=2, # factor by which number of the number of output channels vary
    add_residual_connection=True, # number of groups in which the input is split along the channel axis in the convolution layers.
)



The Charbonnier Loss

$x$ and $y$ are the input and target images, respectively
$\epsilon$ is a small constant to avoid division by zero

from restorers.losses import CharbonnierLoss

loss = CharbonnierLoss(
    # a small constant to avoid division by zero
    epsilon=1e-3,
    # type of reduction applied to the loss, it needs to be explicitly specified in case of distributed training
    reduction=tf.keras.losses.Reduction.SUM,
)

Training and Experiment Tracking with Weights & Biases

The WandbMetricsLogger callback automatically logs Keras' logs dictionary that callback methods such as on_epoch_end, on_batch_end, etc, take as an argument. Using this callback provides the following features:
train and validation metrics defined in model.compile
system (CPU/GPU/TPU) metrics
learning rate (both for a fixed value or a learning rate scheduler)
The WandbModelCheckpoint callback automatically saves the Keras model or model weights periodically and uploads them to W&B as a W&B artifact for model versioning. This callback provides the following features:
Save the model that has achieved "best performance" based on the "monitor".
Save the model at the end of every epoch regardless of the performance.
Save the model at the end of the epoch or after a fixed number of training batches.
Save only model weights, or save the whole model.
Save the model either in SavedModel format or in .h5 format.

# import Peak Signal-to-Noise Ratio and Structural Similarity metrics,
# implemented as part of restorers
from restorers.metrics import PSNRMetric, SSIMMetric
# import the wandb callbacks for keras
from wandb.keras import WandbMetricsLogger, WandbModelCheckpoint

# define the optimizer
optimizer = tf.keras.optimizers.experimental.AdamW(learning_rate=2e-4,)

# define the metrics
psnr_metric = PSNRMetric(max_val=1.0) # peak signal-to-noise ratio metric
ssim_metric = SSIMMetric(max_val=1.0) # structural similarity metric

# compile te model with loss, optimizer and metrics
model.compile(
    optimizer=optimizer, loss=loss, metrics=[psnr_metric, ssim_metric]
)

callbacks = [
    # define the metrics logger callback;
    # we set the `log_freq="batch"` explicitly
    # to the metrics are logged both batch-wise and epoch-wise
    WandbMetricsLogger(log_freq="batch"),
    # define the model checkpoint callback
    WandbModelCheckpoint(
        filepath="checkpoint",
        monitor="val_loss",
        save_best_only=False,
        save_weights_only=False,
        initial_value_threshold=None,
    )
]

# call model.fit()
model.fit(
    train_dataset,
    validation_data=val_dataset,
    epochs=50,
    callbacks=callbacks,
)



NAFNet: Nonlinear Activation Free Network for Image Restoration

NAFNet adopts a classic single-staged UNet architecture instead of the multi-scale residual architecture of MirNetv2. This architecture consists of NAFBlocks, which are the basic building blocks of NAFNet.



The authors experiment with multiple normalization techniques in the NAFBlock:
Batch Normalization is rejected since small batch sizes may introduce unstable statistics (check this paper for more details)
Although Instance Normalization is able to handle small batch sizes, it has been noted that
Instance Normalization does not always bring performance gains
Instance Normalization requires manual tuning (check this paper for more details).
The authors end up favoring Layer Normalization since,
It has been proven to handle small batch sizes
It can make training smooth, even with a 10 times increase in the learning rate, which can make the model converge faster.
Instead of any non-linear activation functions, the authors propose using SimpleGate, which directly divides the input feature map into two parts in the channel dimension and multiplies them element-wise.



Defining the NAFNet Model

# import NAFNet from restorers
from restorers.model import NAFNet

# define the NAFNet model; this gives us a `tf.keras.Model`
model = NAFNet(
    filters=16, # the starting filter size
    middle_block_num=1, # denotes the number of middle blocks, where each middle block is a single NAFBlock unit
    # the tuple size denotes the number of encoder blocks,
    # where each tuple entry denotes the number of NAFBlocks in the corresponding encoder block
    encoder_block_nums=(1, 1, 1, 1),
    # the tuple size denotes the number of decoder blocks,
    # where each tuple entry denotes the number of NAFBlocks in the corresponding decoder block
    decoder_block_nums=(1, 1, 1, 1),
)
# Note that len(decoder_block_nums) should be the same as the len(encoder_block_nums).

Training NAFNet





Zero-DCE: Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement

Understanding light-enhancement curves

Each pixel value of the enhanced image should be in the normalized range $[0,1]$, in order to avoid information loss induced by overflow truncation.
It should be monotonous to preserve the contrast between neighboring pixels.
The shape of this curve should be as simple as possible, and the curve should be differentiable to allow backpropagation.



The DCE-Net

The DCE-Net, or the Deep Curve-estimation Network, is a lightweight deep neural network that learns the mapping between an input image and its best-fitting curve parameter maps.
The input to the DCE-Net is a low-light image, while the outputs are a set of pixel-wise curve parameter maps for corresponding higher-order curves.
Some architectural details regarding DCE-Net:
It is a plain CNN of seven convolutional layers with symmetrical concatenation.
Each layer consists of 32 convolutional kernels of size 3×3 and stride 1, followed by the ReLU activation function.
The last convolutional layer is followed by the Tanh activation function, which produces 24 parameter maps for 8 iterations, where each iteration requires three curve parameter maps for the three channels.



# Import ZeroDCE from restorers
from restorers.model.zero_dce import ZeroDCE

# define the ZeroDCE model; this gives us a `tf.keras.Model`
model = ZeroDCE(
    num_intermediate_filters=32, # number of filters in the intermediate convolutional layers
    num_iterations=8, # number of iterations of enhancement
    decoder_channel_factor=1 # factor by which number filters in the decoder of deep curve estimation layer is multiplied
)



Non-Reference Loss Functions

$\Omega(i)$ is the four neighboring regions (top, down, left, right) centered at the region $i$
$Y$ and $I$ denote the average intensity value of the local region in the enhanced version and input image, respectively.

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    weight_exposure_loss=1.0, # weight of the exposure control loss
    weight_color_constancy_loss=0.5, # weight of the color constancy loss
    weight_illumination_smoothness_loss=20, # weight of the illumination smoothness loss
)



Training Zero-DCE

model.fit(
    train_dataset,
    validation_data=val_dataset,
    epochs=experiment_configs.training_configs.epochs,
    callbacks=callbacks,
)



How to Evaluate Your Dragons Models???

from restorers.evaluation import LoLEvaluator
from restorers.metrics import PSNRMetric, SSIMMetric

# define new wandb run for evaluation
wandb.init(project="low-light-enhancement", job_type="eval")

# Define the Evaluator for LoL dataset
evaluator = LoLEvaluator(
    # pass the list of Keras metrics to be evaluated for
    metrics=[PSNRMetric(max_val=1.0), SSIMMetric(max_val=1.0)],
    # pass the wandb artifact for the LoL dataset
    dataset_artifact_address=""ml-colabs/dataset/LoL:v0"",
    input_size=256,
)
# initialize model from wandb artifacts
evaluator.initialize_model_from_wandb_artifact("artifact-address-of-your-model-checkpoint")
# evaluate
evaluator.evaluate()





How to Use Your Dragons Models???

import wandb
# import Inferer class for low-light enhancement
from restorers.inference import LowLightInferer

# initialize a wandb run for inference
wandb.init(project="low-light-enhancement", entity="ml-colabs", job_type="inference")

# initialize the inferer
inferer = LowLightInferer(
    resize_factor=1,
    model_alias="Zero-DCE"
)
# intialize the model from wandb artifacts
inferer.initialize_model_from_wandb_artifact(
    "ml-colabs/low-light-enhancement/run_oaa25znm_model:v99"
)
# infer on a directory of images
inferer.infer("./dark_images")
# or infer on a single image
inferer.infer("./dark_images.png")





Standing on the Shoulders of Giants

This talk and, to some extent, the idea of building restorers came up after reading the paper Low-Light Image and Video Enhancement Using Deep Learning: A Survey by Chongyi Li, who also co-authored Zero-DCE.
This report primarily discusses the following 3 papers and their usage with restorers:
MirNetv2, as proposed by the paper Learning Enriched Features for Fast Image Restoration and Enhancement
Zero-DCE, as proposed by the paper Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement
NAFNet, as proposed by the paper Simple Baselines for Image Restoration
I'm grateful to have been able to collaborate on this project with:
Saurav Maheshkar
Samarendra Dash
Aritra Roy Gosthipaty
We are grateful to Sayak Paul for the TFLite Benchmarks of our implementation of MirNetv2, which can be found here.
Shoutout to Ayush Thakur, who has been working with me on improving the Weights & Biases callbacks for Keras.
We are grateful to Weights & Biases and the ML Developer Programs team at Google for providing us with cloud credits that made training these models a reality.
What's next with restorers??
We are training models on even larger datasets in order to increase the performance of the model; you can check them here.
We are adding implementations of more models not just for low-light enhancement but also for other image restoration tasks such as defocus-deblurring, deraining, denoising, super-resolution, and many more!!!
We are simultaneously also working on improving the API design and documentation of the library.
If you wish to contribute to restorers or, in general, are curious regarding what we are working on, check out:
https://github.com/soumik12345/restorers/issues
https://github.com/soumik12345/restorers/pulls