Powerful Custom Charts To Debug Model Performance

Easily log custom, interactive visualizations to understand & present your results. Made by Carey Phelps using Weights & Biases
Carey Phelps

Upgrade your ML workflow from tinkering with matplotlib to automatically generating interactive charts straight from your ML training script. I’ll show you how to:

  1. Log custom charts with just 3 lines of code
  2. Play with the code that generates the visualization
  3. Save custom presets to use from your script

report 2.gif

Log charts in 2 minutes

In your Python script, specify a table of data to visualize. The columns can be anything you like, and you’ll use them later in the chart to specify the different features. For example, for a scatter plot you would want to specify an x and y axis, as well as an optional color axis.

To follow this tutorial, open this Colab and run the cells. Re-run the script multiple times to get data you can compare across runs in the dashboard.

Play with the notebook →

table = wandb.Table(data=data_1, columns=["step", "height"])
histogram = wandb.plot.histogram(table, value='height', title='Histogram')
wandb.log({'my_histo': histogram})

image.png

Customize the visualizations

To tailor the visualizations, we’re going to use Vega. It’s a declarative visualization grammar that lets you specify exactly how you want your data to be visualized.

  1. Edit chart: Click the pencil icon to edit the chart. You’ll see a preview of the chart on the left, and an interactive query on the right. The query lets you specify exactly what fields to visualize in the chart, and the dropdown boxes below give you a way to map the query results to the chart inputs.
  2. Open spec: Inside the modal, you’ll see the name of a chart preset in the upper left corner. Open the chart definition by clicking that Edit button next to the dropdown. Now you’ll see the Vega spec on the left — it’s just fancy JSON.
  3. Edit spec: The Vega spec defines the details of how the histogram is displayed. Try tinkering with the code to see what changes in the preview chart on the right. For example, you can remove the line "stack": null to make the chart default back to stacking on top of each other.

report3.gif

Create a chart using that preset

Now that I have a new custom chart ID, I can call that chart from my script. You can see below how I follow the same simple steps:

  1. Create a new wandb run with wandb.init()
  2. Log some data to a table with wandb.Table()
  3. Map the table columns to the fields in the chart you created
  4. Copy the chart ID to use in wandb.plot_table()
  5. Log the chart you created with wandb.log()

import wandb
import random
import math

# Start a new run
run = wandb.init(project='custom-charts',
                 notes='Custom stacked bar chart')
offset = random.random()

# Set up data to log in custom charts
data = []
for i in range(100):
  data.append([i, random.random() + math.log(1 + i) + offset + random.random()])

# Create a table with the columns to plot
table = wandb.Table(data=data, columns=["step", "height"])

# Map from the table's columns to the chart's fields
fields = {"x": "step",
          "value": "height"}

# Use the table to populate the new custom chart preset
# To use your own saved chart preset, change the vega_spec_name
my_custom_chart = wandb.plot_table(vega_spec_name="carey/stacked_bar_chart",
              data_table=table,
              fields=fields,
              )

# Log the plot to have it show up in the UI
wandb.log({"custom_chart": my_custom_chart})

# Finally, end the run. We only need this ine in Jupyter notebooks.
run.finish()

Results of a custom chart

In this screenshot you can see the new histogram appear on the scene — looking dapper! If you want to further customize the chart, you can overwrite this chart definition by logging to the same key in wandb.log(). In my case in the script above this would be “custom_chart”.

image.png

Section 7

You did it! 🎉

That’s it, that’s the whole tutorial. Reach out to me at c@wandb.com if you have any questions or suggestions. If you create a cool chart preset, send me an email with a link to the chart. If I like what I see, I’ll send you a hand written note and stickers, straight from W&B virtual HQ!

More resources

  1. Live report with great examples of custom charts in Weights & Biases
  2. Custom Charts documentation for details on what’s possible
  3. Vega documentation for custom chart ideas
  4. Weights & Biases documentation for a broader look at the ML developer tools available
  5. Gallery of machine learning projects for inspiration from ML researchers