Logging Arbitrary Curves

How to plot various types of metrics in W&B with custom charts. Made by Stacey Svetlichnaya using Weights & Biases
Stacey Svetlichnaya

Base case: Log metric Y vs time

In the simple scenario, we want to log a metric like loss or accuracy over the course of training a model. The metric of interest, say “train_loss”, is logged to wandb every timestep (e.g. at the end of each epoch of training). Use the native W&B “Line plot” to visualize the full list of “train_loss” values logged to wandb over time. Click on "Add visualization" > "Line plot" > set Y = “train_loss”. This plot extends easily to multiple runs in a Workspace.

Modify X-axis: Log metric Y vs arbitrary X

What if we need units other than time on the x-axis? There are several ways to log a metric y versus an arbitrary metric x.

Single (x, y) data point per run

Use the native W&B “Scatter plot” and select the desired metrics for x and y. By default these will be pulled from the wandb run summary field, which stores only the last value logged to a given key (i.e. every new value logged overwrites the previous one). Thus each run will become a single data point of the form (summary.x, summary.y).

Multiple (x, y) data points per run (across time)

What if we need to log a data point multiple times over the course of each run, say at the end of every epoch? In this case, the run summary data is insufficient. We need the run history data, which we can access via the "Custom Chart" > "history" field and display easily with the "line plot" or line/v0 builtin. Note that this assume the two metrics are joined by time—each point (x, y) is logged at a particular time step.

Add a dimension: Log metrics Y, Z vs arbitrary X

What if the typical dimensionality of wandb (2 dimenisons: y vs x) is not enough? Add extra lines or colors to accommodate an extra dimension of data.

Multiple (x, y, z...) data points, once per run

To log multi-dimensional data at each timestep, use a wandb.Table(), which accepts a 2D array of data. Each row is a data point, and each column is a dimension of that data point. This code snippet will make the dimensions/fields "x", "y", and "z" available under the table key "custom_data_table":
# Logging a custom table of datamy_custom_data = [[x0, y0, z0], [x1, y1, z1]]wandb.log({“custom_data_table”: wandb.Table(data=my_custom_data, columns = ["x", "y", "z"])})
To generate these PR and ROC curves, I initially used sklearn.metrics.precision_recall_curve and roc_curve then log the arrays of (recall, precision, class name) and (false positive rate, true positive rate, class name) to a wandb.Table(). This table is then accessible by key from the run's summaryTable and rendered with a Vega spec very similar to the previous one.
While this is still useful for illustrating the power of custom charts, we've made this process much easier and faster with area under curve Custom Charts presets. You can now plot a PR curve with wandb.plot.pr_curve() or ROC curve with wanb.plot.roc_curve(). You can also make minimal modifications to the Area Under Curve preset to get the charts below, using colors instead of line stroke types to denote a class: change the color field to the following, and reconnect the class field from the dropdown on the right. This will render the full class names if you pass them in to the optional labels argument of the wandb.plot call.
"color": { "type": "nominal", "field": "${field:class}", "title" : "Class name"}

Multiple (x, y, z...) data points, multiple times per run

These plots are similar to the previous section, except each set of curves is plotted once at the end of each of 10 training epochs. If you click on the gear settings icon in the top right of each chart, you will see a menu to "Select epoch". You can use this to focus each chart on a particular epoch and compare across them. Here we use the run historyTable instead of just summaryTable to get the 2D array of values logged for each epoch. An extra key for the training epoch can condition the opacity to filter the points to one epoch at a time (t_epoch in this case).
You can see the query mapping and the full Vega below the charts. Note the selection key used to create the dropdown menu and the opacity key used to condition the opacity to render one epoch at a time.

Add two dimensions: x, y, time, and multiple runs

Using opacity to indicate time and color to indicate runs can give you four dimensions in one chart! Here I log microaveraged precision for each model at the end of every epoch. The more opaque/saturated the color, the later the epoch. Darker lines tend towards the top right of the chart, with higher average precision as the number of training examples and/or epochs increases—and with substantial noise since this is a toy experiment :)

What's your dream visualization?

Between wandb and Vega, you can log virtually anything. Try making your own custom chart and let us know how it goes!