DeepMind Flamingo: A Visual Language Model for Few-Shot Learning



Single Image Samples

Step

Single Image samples: Gray boxes are user input and the pink boxes are Flamingo output.

1



Table of Contents

An Introduction to Flamingo

What is Flamingo?

Flamingo is a visually-conditioned autoregressive text generation model able to ingest a sequence of text tokens interleaved with images and/or videos, and produce text as output.

Challenges of Multimodal Generative Modeling

Unifying Strong Single-Modal Models

Training large language models is extremely computationally expensive.
The authors saved the computational resources by starting with a pretrained language model.
However, a text-only model has no built-in way to incorporate data from other modalities.
The authors wanted to enable this while retaining the knowledge of the original language model.

Supporting Both Images and Videos

Images and videos (of even modest resolution) are high dimensional.
Flattening them to 1D sequences (as used in unimodal text generation) is costly as the computation scales quadratically with the sequence length.
The authors also wanted to treat images and videos in a unified manner which is not straightforward.

Heterogeneous Training Data

Large models require huge datasets.
Paired image/caption datasets used in CLIP and ALIGN may not be general enough to reach GPT-3 style few-shot learning.
Large internet-based text-only dataset exists, but not for multimodal data.
One approach is to scrape webpages with interleaved images and text. Despite the generality of the data, the images and text are often weakly related.

Key Ideas of Flamingo

A novel architecture for accepting arbitrarily interleaved visual and text data as input and generating output text in an open-ended manner.
Architectural innovations and training strategies that effectively leverage large pre-trained vision-only and language-only models, saving tons of compute and preserving the benefits of these initial models while efficiently fusing the modalities. Specifically, the authors used Chinchilla, a 70B state-of-the-art LM (which is frozen in Flamingo) and trained Flamingo, an 80B parameter VLM.
Efficient ways to adapt to visual inputs of varying sizes, making Flamingo applicable to images and videos.

The Flamingo Model

Approach

The first goal of the authors is to leverage pre-trained language models without spending computing on training them from scratch. Specifically, they used a model called Chinchillah which was introduced recently by DeepMind. This allowed the Flamingo model to have strong generative language abilities and access to a large amount of knowledge stored in LM weights.
On the vision side, the authors pretrain a vision encoder with a contrastive text-image approach similar to CLIP. The role of this model is to extract rich semantic spatial features from the given images/videos.
The second goal was to bridge these two models harmoniously. For that, the authors freeze the weights of these models and link them via two learnable architectures.
The Perceiver Resampler receives spatiotemporal features from the Vision Encoder (obtained from a variable number of images or videos) and outputs a fixed-size set of visual tokens.
The visual tokens are then used to condition the frozen LM using freshly initialised cross-attention layers that are interleaved (or inserted) between the pre-trained LM layers. These layers offer LM a way to incorporate visual information for the next-token prediction task.

Vision Encoder: From Pixels to Features

The authors used an F6 Normalizer-Free ResNet (NFNet) as it gives an excellent trade-off between performance and efficiency given the hardware.
The vision encoder is pretrained as a dual encoder using contrastive loss employed by CLIP.
BERT is used for text encoder which is discarded after pretraining.
Contrastive similarities are computed as dot product of the mean pooling of the image encoder output, and the mean pooled output of BERT model.
In contrast to CLIP, global average pooling is used to produce the visual embedding (rather than the global attention pooling) for simplicity.
288 x 288 pixels image resolution is used and the joint embedding space size is 1376.
The final output is a 2D spatial grid of features $X_f$, which is further flattened to 1D as shown in Figure 4.
For video inputs, the frames are sampled at 1 FPS and are encoded independently to obtain a sequence of $T$ feature maps $X_f$ which are then concatenated.

Vision Encoder Details

Perceiver Resampler: From Varying-Size Large Feature Maps to Few Visual Tokens

The Perceiver Resampler is based on DeepMind's paper Perceiver: General Perception with Iterative Attention.
The Flamingo model takes as input a variable number of image or video features from the vision encoder and outputs a fixed number of visual tokens.
The visual inputs are re-sampled to a fixed and small number (64 in practice) of outputs to significantly reduce the computational complexity of vision-text cross-attention.
The visual features which are fed to Perceiver Resampler are obtained by first adding a learnt temporal position ($t=0, t=1, t=2$ ) to each spatial grid of features corresponding to a given frame of the video as seen in Figure 4.
The authors only used temporal encodings and no spatial grid position encodings as the latter didn't brought improvements.
These visual features are then flattened as $X_f$.
Similar to the original Perceiver, the model learns a predefined number of latent input queries.
These latent queries are fed to a transformer stack and cross-attend to the flattened visual features $X_f$.
The keys and values computed from the learnt latents are concatenated to the keys and values obtained from $X_f$.

Interleaving New GATED XATTN-DENSE Layers Within a Frozen Pretrained LM.

Text generation is performed by a Transformer decoder, conditioned on the visual representations $X$ produced by the Perceiver Resampler.
The authors used a 70B parameter Chinchilla model for the largest Flamingo model as the language model.
The pretrained blocks are frozen during the training of Flamingo to preserve the information and text generation abilities in the text-only language model.
In order to condition the LM on the visual inputs, the authors inserted gated cross-attention dense (GATED XATTN-DENSE illustrated in Figure 5) blocks in between the original self-attention layers. Note that the original self-attention layers are frozen during the training of Flamingo while the newly inserted cross-attention layers are trained from scratch.
LayerNorm is applied to all attention inputs and the feed-forward layers (GPT-2 style).
The authors also added tanh getting mechanisms to preserve the original language model behaviour at initialisation and not catastrophically change the learned features by the LM.
The outputs of the newly added cross-attention layers are multiplied by tanh($\alpha$), where $\alpha$ is a layer-specific learnable scalar initialized at 0.
During training, the model smoothly transitions from a fully trained text-only model to a visual language model, thanks to the gating mechanism.

Per-Image/Video Attention Masking

The training data also consists of interleaved sequences scraped and processed from webpages.
Each interleaved example consists of: a sequence of text $y$, a sequence of images $x$, and the sequence of positions of the images in the text.
Based on the visual data positions, the authors define a function $\phi:[1, L] \mapsto[0, N]$ that assigns to each text position the index of the last image or video appearing before this position (or 0 if no visual data appears before the position).
The function 𝜙 defines which visual inputs are considered usable to predict token in Equation - (1): the set of preceding tokens $y_{<\ell} \triangleq\left(y_1, \ldots, y_{\ell-1}\right)$, and the set of preceding images or videos $x_{\leq \ell} \triangleq\left\{x_i \mid i \leq \phi(\ell)\right\}$.
Multi-image attention is implemented with the gated xattn-dense layers with causal masking over tokens from the Perceiver Resampler.
By default, each token is only allowed to attend to the visual tokens of the image that appeared immediately before it (this restriction improved performance).
Although direct attention is over a single image, there is still a causal dependency on previous images (due to causal self-attention in the text decoder).
Furthermore, experiments show that the model can train on 5 images, but generalize up to 32.

Training Data

Flamingo Training Details

Task Adaptation With Few-Shot In-Context Learning

The authors evaluate the ability of the model to rapidly adapt to new tasks using in-context learning, popularised by GPT-3.
The model is given a set of support-examples in the form of (image, text) or (video, text) (where the image or video is the input visual and the text is the expected response) and a single visual query for which the model needs to make a prediction.
Given a few support-examples, the authors build a multimodal prompt by concatenating the support examples followed by the visual query as illustrated in the Figure 8.

Flamingo Models

Qualitative Results

Conclusion

References

DeepMind blog on Flamingo
Flamingo official paper
A digest of the Flamingo model by Samuel Albanie
Unofficial implementation of the key ideas of Flamingo