Modern Evolution Strategies for Creativity

👉 Try Concrete Image Fitting on Google Colab

👉 Try Abstract Concept Fitting on Google Colab

Table of Contents

The Proposed Evolution Strategies for Creativity

A configuration of triangles can be represented in a parameter space which composes of the positions and colors of triangles. Concretely, a configuration of $N$ triangles is parameterized by a collection of $\left(x_{1}, y_{1}, x_{2}, y_{2}, x_{3}, y_{3}, r, g, b, a\right)$ for each of the triangles, which are vertex coordinates and the RGBA color, totally making $10N$ parameters.
We render such a configuration onto a canvas.
Then we calculate the fitness of the rendered canvas based on how well it fits a target image or a concept in the form of a text prompt.
The Evolution Strategy algorithm keeps a pool of candidate configurations and uses mutations to evolve better ones measured by the said fitness.
In the Evolution Strategy, all the parameters are updated, and a fixed hyper-parameter is used, which is the number of triangles $N$.
In order to have better creative results, the authors use a modern Evolution Strategy algorithm, Policy Gradients with Parameter-based Exploration or PGPE, optimized by the ClipUp.

Evolutionary Strategy in Practice

They augment the rendered canvas by random cropping in calculating the fitness and average the fitness on each augmented canvas, as used in the case of VQGAN + CLIP. This would prevent the rendered canvas from overfitting and increase the stability in the optimization.
The triangles are rendered on top of a background with a uniform distribution noise. Mathematically, this equals modeling the uncertainty of parts in the canvas not covered by triangles with a max-entropy assumption and using the Monte Carlo method for approximation.
The maximal alpha value for each triangle is limited to $0.1$, which prevents front triangles from (overly) shadowing the back ones.

What Exactly Does "Fitting" Mean Here?

Fitting a concrete image: The pixel-wise L2 loss between the rendered canvas and the target image is the fitness. This is demonstrated by the lower branch in the aforementioned figure describing the proposed architecture.
Fitting a concept: The authors represent the concept as a text prompt and embed the text prompt using the text encoder in CLIP. The rendered canvas is then embedded using the image encoder available in CLIP. Since the CLIP models are trained so that both embedded images and texts are comparable under cosine distance for similarity, such distance can be used as the fitness.

The proposed Evolution Strategy approach for Computational Creativity

Result

Fitting Concrete Target Image

Fitting Abstract Concept With CLIP

Conclusion

The proposed method offers considerable improvements in quality and efficiency compared to traditional genetic algorithms and is comparable to gradient-based methods.
The authors also demonstrate that the Evolution Strategy algorithm could produce diverse, distinct geometric abstractions aligned with human interpretation of language and images.
The authors observe that the Modern Evolution Strategy approach produces very different and sometimes better results compared to gradient-based methods, arguably due to the intrinsical behavior of the optimization mechanism. However, it remains an open problem to understand how in a general setting, the Modern Evolution Strategy approach compares with gradient-based methods.
The way the authors deal with evolutionary algorithms provides insight into a different paradigm that can be applied to computational creativity.
The authors note that widely adopted gradient-based methods are fine-tuned for specific domains, i.e., diff rendering for edges, parameterized shapes, or data-driven techniques for rendering better textures. Each of the applications requires tunes and tweaks that are domain-specific and hard to transfer. In contrast, the Modern Evolution Strategy approach is agnostic to the domain, i.e., how the renderer works.
The authors envision that approaches inspired by Evolution Strategies could potentially unify various domains with significantly less effort for adaptation in the future.

Similar Reports