Visualize Audio Data in W&B

Interactively explore ML data and predictions in the audio domain. Made by Stacey Svetlichnaya using Weights & Biases

W&B Tables—our latest feature for dataset and prediction visualization—enable interactive exploration and analysis of audio data. In this short example, I render whale song as human music: I synthesize melodies from the vocalization of whales and other marine mammals as they would sound on a violin, trumpet, etc. I use Differentiable Digital Signal Processing from Tensorflow's Magenta (resources, colab demo) to generate the music from original recordings in the Watkins Marine Mammal Sound Database.

For this project, I store my toy dataset in a remote bucket and version it in a reference artifact. The change from logging a regular W&B Artifact is minimal: instead of adding a local path with artifact.add_artifact([your local file path]), add a remote path (generally a URI) with artifact.add_reference([your remote path]). You can read more about reference artifacts here.

List of file paths and sizes in this reference bucket. Note that these are merely references to the contents, not actual files stored in W&B, so they are not available for download from this view

In this example, I've manually uploaded some marine mammal vocalizations to a public storage bucket on GCP. The full dataset is available from the Watkins Marine Mammal Sound Database, and you can play the sample songs directly from W&B. With a dataset visualization table, you can see and interact with your data directly: listen to audio samples, play videos, see images, and more. This means you don't need to fill up local storage, wait for files to download, open media in a different app, or navigate multiple windows/browser tabs of file directories.

Press play/pause on any song and view additional metadata

You can group by any column: say, group by "species" to listen to different samples from the same marine mammal in one row.

Group by "species" (I also removed the id column which isn't relevant to this view)

Of course you can still fetch the files from the reference artifact and use the data locally:

Instead of uploading your content to W&B and versioning with regular Artifacts, you can visualize data and predictions via reference paths (URIs) to remote storage using Artifacts by Reference as I do in this example. This mode is currently in beta.

Synthesized samples grouped by instrument. Scroll down inside each panel for more.

Beyond raw training data, you may want to visualize training results: model predictions over the course of training, examples generated with different hyperparameters, etc. You can join these to existing data tables to set up powerful interactive visualizations and analysis. In this case, I have synthesized a few renditions of the marine mammal melodies in different human instruments like violin, flute, and tenor sax, via the amazing DDSP library and Colab Notebook from Magenta for timbre transfer (with a WIP W&B Colab here). These synthetic songs are local .wav files created in a Colab or my local dev environment. Each file is associated with the original song_id and the target instrument.

Play and pause the songs and optionally download the files

To see and interact with audio files, log them directly into a wandb.Table associated with an artifact. To visualize a piece of media such as an image, video, or song (audio file) in the browser, we need to wrap it in a wandb object of the matching type—in this case, wandb.Audio(). The wandb object takes in a file path to render the contents of the file. Sample code, assuming my songs live in a local folder called whalesong/synth:

Group by song_id to see all the transformations of a given song in one row (the same melody played on a flute, violin, trumpet, or tenor sax). You can also group by instrument to compare timbre across melodies.

Find the header of the column you'd like to group by, click on the three dot menu on the right of the column name, and select "Group by" from the dropdown. Try it here.

Compare melodies across different instruments/timbres

Compare timbre across different melodies

To listen to both song versions side-by-side, I can join the table of original songs to the table of generated songs:

Query across existing tables to create a new wandb.JoinedTable without duplicating data

Join across tables you've logged in earlier artifacts to efficiently create new views for analysis—without duplicating your data. I've logged all the information about the original marine songs in a song_samples table of my playable_songs artifact and about the synthesized songs in a synth_song_samples table of my synth_samples artifact. To compare the original and synthesized versions, I can join these tables on a single key (or a list of two keys) and even change the join type for the sub-tables (inner, outer, etc) from the browser:

This is an early proof of concept to illustrate the power of W&B Dataset and Prediction Visualization for the audio space. I hope to explore timber transfer, DDSP, and the Tensorflow Magenta toolset in more depth—and more serious applications like identifying and tracking different marine mammal species based on underwater recordings—in future reports.