Mood Board Search: An Open-Source AI Model For Finding Images By Mood When Words Don't Cut It
New experimental ML model project Mood Board Search can learn the mood of a collection of images and help you find more like it.
Created on July 9|Last edited on July 9
Comment
We've all been in a situation where you want to convey a feeling, but words can't do it justice. Human emotion is complex, and sometimes the human-construct of language can't portray it well enough. Luckily, we can understand emotion more abstractly through the senses, which is where mood boards can come into play.
Mood boards, as a visual medium, can portray emotion when words fail us; They are a useful tool used by creatives during the art-making process, or a common way for internet subcultures to present themselves on social media. Either way, when you want to find images matching some feeling, what better way to search for it than with your own mood board of example images?
As part of Experiments with Google, ML engineers at Google Brain have put together a machine learning model for this exact thing. With Mood Board Search, you can use handmade mood boards to use as a search query to find more images of the same mood, and much more. All the code is open-source and available on the project GitHub if you want to take it for a spin yourself. You can read the blog post about this release by clicking here.
How does Mood Board Search work?
Mood Board Search is powered by CAV (Concept Activation Vectors), which are essentially groupings of neurons in the embedding space of a model which pertain to certain concepts - think all the neurons in a handwritten-digit classifier model that light up when it sees a "3". The same idea extends to the moods represented by images as mentioned before.
The trainable portion of this setup is the CAVs, which are created by feeding your mood board images into a pre-trained model and keeping track of which neurons light up. When searching for new images, the images are fed into the same pre-trained model and the pattern of lit-up neurons are compared against the trained CAV patterns. Comparing these light-up patterns lets the program decide a CAV score which shows how similar the found image is to the established mood. If the score is high, it's deemed to fit the mood.
How can I use Mood Board Search?
First, you'll need to train yourself a CAV matching your mood. The code required for this is all available on the GitHub repository, though you'll need some understanding of Python and Tensorflow. They've included a convenient locally-hosted web interface so you won't have to do absolutely everything with code.
Once you've created a CAV, there is an experimental app called CAVcamera which was developed in conjunction with the project by Nord Projects. CAVcamera is a photo app that will compare what the lens sees against your trained CAV to see if it matches the mood well. Beyond CAVcamera, a framework called CAVlib was also created so that developers can create their own CAV-based applications.
Find out more
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.