Dynamic Sky Replacement: The Sky Is Within Our Grasp!

Project Website | Paper | Colab Notebook

Table of Contents

An Introduction to Dynamic Sky Replacement

Purely vision-based,
Requires no user interactions,
Can be applied to either online or offline processing scenarios,
And does not require any special hardware.



Overview of the Proposed Method



Sky matting network
Motion estimator
Skybox

Sky Matting Network

Details On U-shaped Network



The authors used ResNet-50 as the encoder network.
The decoder is not symmetric to the encoder but uses convolutional layers with upsampling layers.
Since the sky region usually appears at the upper part of the image, the conventional convolutional layers are replaced with coordinate convolutional layers at the encoder's input layer and all the decoder layers.
Skip connections were applied between the encoder and the decoder layers with the same spatial size.
$L2$ loss was applied in the raw pixel space between the predicted sky matte and the ground truth matte.

Details of the Refinement Module

The coarse sky matte is first upsampled to the original input resolution in this stage.
The module takes in the upsampled coarse sky matte and the high-resolution input frame(guidance image). The authors have used the Guided Filtering technique instead of upsampling convolutional operation or adversarial training.
This technique is used because it has better behaviors near edges and has high efficiency and simplicity.
By using the blue channel(for better contrast) of the guidance image, the filtering transfers the structures of the guidance image to the low-resolution sky matte. It produces a more detailed result with minimal computational overhead.

Motion Estimation

The previous methods were used to estimate the motion of the real camera. However, the proposed method assumes that the sky and the in-sky objects are located at infinity, and their movement relative to the foreground is Affine. The method estimates the motion of these objects.
The motion is estimated using the frames of the input video. The iterative Lucas-Kanade method with pyramids is used to compute optical flow. Thus a set of sparse feature points can be tracked frame-by-frame. These feature points are located within the sky area. If there are not enough feature points, depth estimation is run on the current fame to compute Affine parameters.
For each pair of adjacent frames, given two sets of 2D feature points, RANSAC-based robust Affine estimation is used to compute the optimal 2D transformation.
A background template image is used to get the final sky background in the frame $t$. Using a simple method, the template is aligned using the computed Affine parameters.

Sky Image Blending

Let $I^t$, $A^t$, and $B^t$ be the video frame, predicted sky matte, and the aligned sky template image at time $t$. Let $Y^t$ be the output frame. It is a linear combination of $I^t$ and $B^t$, with $A^t$ as their pixel-wise combination weights. Thus $Y^t$ is given as,
$Y^t = (1 - A^t)I^t + A^tB^t$

From this equation we can understand that the soft matte is nothing but the probabilistic prediction of the encoder-decoder architecture and refined by the guided filter.
Simple linear combination might give unrealistic results due to different color tones and intensities of the input frame and the aligned background. Thus recoloring and relighting techniques are used to transfer colors and intensity from the background to the foreground.

Results

Reproduce the Results on Colab Notebook $\rightarrow$

Video Sky Augmentation

Weather/Lightning Translation