Read on to learn what you can do with our new Visualization IDE, and see interactive examples. Learn by doing? Try our example colab for an end-to-end walk through.

Building and understanding ML models is inherently visual. As opposed to classical software engineering, an ML model has no source code to inspect and debug. Instead we use statistics and visualizations to gain insight and inform our decisions.

Weights & Biases is the system-of-record for storing every piece of data and tracking every computation that goes into producing models. Our UI provides a rich toolset for querying, visualizing, presenting, and sharing anything saved to W&B.

W&B Reports, like the one you're reading right now, are "reproducible documents". Every figure you see in this report is backed by the actual code and data that produced it. Our customers use them to track their day-to-day work, collaborate within teams, and publish research findings.

Today, we're incredibly excited to launch our new Visualization Development Environment (VDE). You can use it to construct completely custom visualizations for understanding and presenting your results. You can log them from Python, add them to W&B workspaces and reports, and make them available for anyone to use.

Here's how it works in a nutshell:

  1. Log some data to W&B from a Python script.
  2. Use our Visualization IDE to fetch your data and create a Vega spec to visualize the data.
  3. Optionally use wandb.plot_table() to log your own custom visualization from any Python script.

Want to try it yourself?

Run our example colab for an end-to-end walk through.

Making new tools for model understanding

Let's look at some examples of useful visualizations we've made using our VDE.

Our first example was created by Kyle Goyette, an engineer on our team, for understanding attention mechanisms. It helps us understand how gradient flows between time-steps in an attentive RNN.

Hover over a point with your mouse to see the strength of connection between that timestep and all other timesteps.

Section 8

Each visualization we've made comes with a report to show you what it looks like and provide usage examples. Check out Kyle's Visualizing the Effect of Attention on Gradient Flow report to see how to use the example above.

If you make something useful, send a link to your report to c@wandb.com. We'll feature it in our Gallery!

You can use Kyle's next example to understand why seq2seq models make specific predictions. The weight of the connecting lines shows how much attention the decoder paid to a given input word (on the bottom) when producing an output word (on the top).

Hover over a word to highlight the connections for that word.

Section 8

Check out Kyle's Visualizing NLP Attention Based Models report to read about how to use this tool in your own work.

New charts in the wandb library

We've built six ready-to-use charts. You can add them anywhere in the UI by creating a Custom Chart panel and then picking them from the dropdown, or log them from Python with wandb 0.10.5 or later.

The first four look like this:

Section 4

All of our built-in plots support smooth pan and zoom interactions, and have useful hover states. Try clicking and dragging, or scrolling, on the plots above.

Our built-in plots are always comparable across runs, and respond to your run color settings. Click on the eyeballs and colors swatches in the table above to update the plots.

You can log the "Height v. Distance" plot above yourself, by running the following code:

import wandb, random, math

wandb.init()
data = [i, random.random() + math.sin(i / 10) for i in range(100)]
table = wandb.Table(data=data, columns=["distance", "height"])
wandb.log({'line-plot':
    wandb.plots.line(table, x="distance", y="height", title="Height v. Distance")
})

To learn about the built-in plots and how to use them, check out the detailed wandb.plot reports by Stacey Svetlichnaya, another engineer on our team. You can find them here in our Gallery. Each report shows the plot in context, describes its usage, and gives examples showing how to customize them to your needs.

Customizable chart presets

In addition to the plots above, we've updated our PR and ROC plots to use this new system.

Section 10

Check out Stacey's documentation reports for examples and usage information:

Keep an an eye on the Gallery for announcements about new built-in plots!

Data Layout Independence

The visualizations created with this tool are decoupled from the underlying data they display. To use a custom visualization, you construct a query in our new visual query editor. We'll fetch the data from our graphql API, and feed it into your visualization.

You can modify the query a plot uses by editing its panel.

demo - using the visual query editor.gif

To try this yourself, clone this report into your own project, then click the pencil icon on one of the panels contained in this report.

Check out Stacey's guide to visualizing varying levels of structure in Weights & Biases, from individual tables logged in runs, to high-level summary comparisons.

The W&B ML Visualization Development Environment

Custom visualizations are built using the Vega and Vega-lite languages. With Vega, you can specify every detail about how your data is transformed into pixels on the screen.

Our Visualization Development Environment lets you define Vega specs and save them for reuse, all in the context of your actual data. You can reach the IDE via any custom visualization panel you have edit access to.

2020-10-08 00.14.18.gif

To try it yourself, clone this report, edit one of the panels, and then click the “Edit” button to open our Visualization IDE.

From there you can change everything about your visualization, from details like line thicknesses and strokes, to how it responds to mouse interactions, and how it processes and transforms input data on its way to the screen.

Resources:

Beyond simple plots

The powerful Vega library can do much more than what we've shown above. But just how far can you go?

John Qian, another engineer on our team, set out to answer that question. He decided to make a basic side-scrolling game. Use the keys WASD to move, and press Z to get a boost.

Section 10

Like the other visualizations in this document, this one is driven by data logged to W&B. The panel on the right shows the underlying pixel data used to render the game! Turn the eyeballs on and off or change a color swatch to see how it affects your terrain.

This example isn't all that useful for explaining ML results. Who wants to have to avoid falling in pits to determine how their model training run progressed? But it does show off the power of Vega, to smoothly process and render large numbers of points.

What's next?

We're really excited to see what you can do with our VDE. Here's what we'll be doing in the mean-time:

Export pixel-perfect charts for publication

Academic researchers often include W&B data in figures for their published papers. For example, one of our users Amir Abdi wrote this NeurIPS paper based on work he did in Weights & Biases. Authors like Amir often ask us to add fine-grained plot style controls, so that W&B visualizations can be used directly in their papers.

Our new Visualization IDE is a step toward solving that problem. With the Vega language you get the pixel-accurate level of detail you need, for any aspect of your work. We'd love to hear from users who use this tool to customize charts for published papers (email c@wandb.com), and we're excited to build a set of reusable charts that make research writing easier and more reproducible.

The Vega library has excellent support for pixel-perfect export of both SVGs and PNGs. We're working on hooking up export support now.

Beyond Vega: Custom panels

We think our users will take Vega a very long way, and do things we've never dreamed of. But like all things, it will reach its limits.

We're hard at work on opening up our panel system so that users can write completely custom panel plugins in JavaScript and other languages. We want to make sure these provide an excellent experience, and work seamlessly in reports and the rest of W&B. This will come with a whole host of new data storage and query features to allow you to process large scale data.

Stay tuned!

Reports and the future of publishing

The Weights & Biases toolset makes reproducibility the default for ML researchers and practitioners.

Visualizations are more powerful when they're interactive, and directly connected with the data that drives them. Readers can inspect and confirm work, and reproduce results themselves.

We believe the future of model research and development is live, reproducible and interactive, and we're building it here at W&B.

We build the best tools for machine learning, and we're hiring! If you're a designer or engineer who thinks there's always a better way, we want to hear from you. Drop us a line at jobs@wandb.com.