Fun With Neural Cellular Automata
In this article, we take a look at how to make pretty pictures using differentiable self-organizing systems, using Weights & Biases to keep track of our results.
Created on May 22|Last edited on February 10
Comment
Below are two examples of neural cellular automata. Click and drag on each to 'erase' part of the image, and watch as the patterns try to re-form. Each cell (pixel) sees only its immediate neighbors, yet together they can organize to produce these dynamic patterns and textures.
Interactive HTML Previews
Run set
2
Come along for the ride as we discuss what neural cellular automata are, how to train them, and some creative ways in which we can put these to use. Here's what we'll cover:
Table of Contents
Background to Neural Cellular Automata
I was first introduced to this idea in this fantastic Distil.pub article, in which Alexander Mordvintsev and colleagues were able to train NCAs that could work together to build an image and repair it if damaged. Their work was inspired by biological systems, where individual units ('cells') can only sense their immediate surroundings and yet are able to organize and adjust their behavior to accomplish some overall goal.
My attempt at a diagram of how these work
For this project, the overall architecture looks something like the diagram above. The grid consists of four channels (corresponding to the RGB and alpha channels in an image).
Each cell 'looks' at itself and its 8 immediate neighbors by applying four 3x3 kernels, giving a total of 16 inputs (4 channels x 4 kernels) fed into a tiny little neural network brain. The output of this network is the cell's proposed update.
Training NCAs
So how do we make these cellular automata LOOK like something? In this paper, they use a 'style loss' based on a pre-trained image classification network to target the 'style' (lower level patterns and structure) of an input image, similar to how style transfer works.
During training, grids at various stages are sampled from a pool of training examples, and updates are applied. Then, the style loss is calculated based on how well the resulting images match the target style image.
I modified this approach to add a second loss, using CLIP from OpenAI to compare the generated images with a text description. This works together with the style loss. Thus, for example, we could provide an image of leaves or grass but add the prompts 'Mossy stones' or 'red flames' to achieve two very different types of outputs.
Run set
4
By expanding the Run set above, you can view the prompts used to train these particular examples. I leave it to you to judge how well it has worked! It's easy to log images and other media with W&B. I went a little wild tracking everything - you can view intermediate results and many other colorful outputs on the project workspace.
Making Fractals
Run set
3
By repeatedly scaling the grid while applying updates, we can create high-resolution images which show patterns at multiple scales. The results are these fantastical fractal landscapes, which often look quite different from the images obtained by applying the NCA at only one scale.
Adding Video
I wanted a way to have the behavior of these NCA controllable by an external signal such as a video. One way to do this is to add additional input channels which tweak the objective (inspired by this paper). During training, we pass a single rule to the entire grid and use a different prompt as the objective for each rule.
Then later, we can use a video to control how much each rule is expressed at a given location. I think the results are pretty neat:
Run set 2
2
Learn More
The code for this project is available as a Colab notebook here, and you can see all the different results in this project (basic NCA) and this one (videos). For those who want to dive deeper, I covered Neural Cellular Automata in lesson eight of my AIAIArt course in a lot more detail.
Key references:
I hope this post inspires you to go and make something pretty! Feel free to reach out to me @johnowhitaker with questions, feedback, and any outputs you're particularly proud of.
Add a comment
Tags: Computer Vision
Iterate on AI agents and models faster. Try Weights & Biases today.