Show Training Plots within Documentation
Make your library more reproducible and transparent by easily including training details, model weights, and sample model predictions within your documentation
Created on August 29|Last edited on October 10
Comment
Like real zoos, model zoos should have some standards if they want their visitors to leave happy. 🐒
Before W&B:
Ok, this is a bit extreme... but you get the point 😁
Experimentation is fundamental to machine learning so your experimental results, logs and hyperparameters are vital for people to build on top of your work. Including them within documentation improves the library visitors ability to analyze and reproduce your work.
Hiding them within Pull Requests or READMEs isn't the answer either 🫣
In this tutorial, we'll show how to embed W&B within the technical documentation of your library to:
- Track and display the experimental results of the models and techniques included
- Provide all the info used to train the model - hyperparameters, data, system info, etc.
- Allow users to automatically download the latest model weights
- Show 1000s of example predictions in an interactive table
TL;DR - track with W&B and include your models and plots within embedded Reports
Run set 2
89
The plots above are dynamically updated whenever there new experiments ran.
You can see the configurations, summary metrics and live training logs. When embedded within your docs, your readers can verify the hyperparameters used to train any models included, along with seeing any metrics that were logged during training or testing.
💡
A System of Record for your Library's Experimental Results
With a few lines of code, you can track and visualize your model training and evaluation.
W&B is easy to integrate and to make it even easier, we already have integrations will lots of the frameworks you know including PyTorch, Tensorflow, Lightning etc.
It's also free for academic or personal use.
import wandbwandb.init('project-name', config={'lr': 0.01, 'bs'=64})for epoch in epochs:wandb.log({'loss': loss, 'validation_accuracy': val_acc})
Get Started
Share everywhere with W&B Reports
Everything in this post is based on using W&B Reports within your docs, see below how to create them.
Here's how CleanRL, a popular RL library, are using W&B Reports within their docs to show benchmark results, predictions and configurations. https://docs.cleanrl.dev
Share Training Logs and Analysis
In the plot below, you can click on the Run set to navigate to the run that produced each line in this charts.
Sweep: r00a5sn2 1
11
Share your Latest Model Weights
If you log models as W&B Artifacts, you can embed model cards so that visitors can download the model weights via the UI or the W&B API. They can also see the config that was used to train the model.
model.pth
Artifact overview
Type
model
Created At
May 20th, 2022
Description
Versions
Version
Aliases
Logged By
Tags
Created
TTL Remaining
# of Consuming Runs
Size
Loading...
To log models:
art = wandb.Artifact("my-object-detector", type="model")art.add_file("saved_model_weights.pt")wandb.log_artifact(art)
To embed the model card in your Report,
- Type /
- Choose Weave panel
- In the expression editor, type:
project(<your username/team name>).artifact(<artifact name>)
Showcase Model Predictions
W&B supports logging rich media (images, videos, audio etc.) with support for many popular label types, like image segmentation or bounding boxes.
For example, the team behind the Craiyon (formerly Dall-E mini) put the predictions of their latest model in a W&B Table so they could also evaluate how it was performing and share their results with the world.
my_data = [[0, wandb.Image("img_0.jpg"), 0, 0],[1, wandb.Image("img_1.jpg"), 8, 0],[2, wandb.Image("img_2.jpg"), 7, 1],[3, wandb.Image("img_3.jpg"), 1, 1]]columns=["id", "image", "prediction", "truth"]test_table = wandb.Table(data=my_data, columns=columns)
How to Embed W&B within your Documentation
To embed dynamic W&B plots within your documentation, you can create a W&B Report with the summary you want to include.
How to Create a Report
- Click Create Report from your workspace
- Choose which plots to include
- Or you can create a blank Report and add plots yourself by typing / within the Report and choosing Panel grid > Import panel.
Make the Report Public
For people to view the report, you'll need to make it public in one of two ways:
- Option 1: Make your project public (view-only) by clicking the lock 🔒 icon beside the project name. This will make your project workspace visible too.
How to make a W&B project public
- Option 2: Make a magic link for your report.
Click Share on the Report and toggle anyone with the magic link can view, you'll need to paste this into the iframe next.
Embed the Report iframe
To make embedding easier, you can copy the embed code when you click Share on a Report.
It'll look like this:
<iframe src="https://wandb.ai/_scott/reports-in-docs/reports/Use-W-B-within-Documentation--VmlldzoyNTQzOTUy" style="border:none;height:1024px;width:100%">
If you're using a magic link, you'll need to replace the scr link to include the access code
💡
Conclusion
Using W&B, you can make your library more reproducible and transparent by including the exact configurations and logs from your training runs.
In this tutorial, we've seen how to embed Reports and include plots, predictions and model weights so that your users can understand and use your models without having to run opaque model binaries.
You can:
- Embed Reports in your docs
- Include W&B panels to show logs and configurations
How many discoveries were lost because they weren't written down?
With Reports you can save your ingenious ideas and insights alongside a snapshot of your work and then share them with your team. Or just keep them all for when you need them.
Collaboration in ML made easy with W&B Teams
How to use W&B Teams on your teams next machine learning project
Add a comment