Visualize & Debug Machine Learning Models

A guide to getting started with Weights & Biases in 5 minutes. Made by Lavanya Shukla using Weights & Biases
Lavanya Shukla

Never lose your progress again.

Save everything you need to debug, compare and reproduce your models — architecture, hyperparameters, weights, model predictions, GPU usage, git commits, and even datasets — with a few lines of code.

Share model insights with your team with Reports like this one. Reports combine interactive plots for performance metrics, predictions, hyperparameters with explanatory text to help you tell the story of your model!

Used by the likes of OpenAI, Lyft, Github and researchers at top machine learning labs across the world, W&B is part of the new standard of best practices for machine learning.

Screen Shot 2020-10-25 at 4.16.28 AM.png

Explore live dashboard →

Track your model training in 5 minutes

Install wandb library and login:

pip install wandb
wandb login

Flexible integration for any Python script:

import wandb

# 1. Start a W&B run
wandb.init(project='gpt3')

# 2. Save model inputs and hyperparameters
config = wandb.config
config.learning_rate = 0.01

# Model training code here
‍
# 3. Log metrics over time to visualize performance
for i in range (10):
    wandb.log({"loss": loss})

Pick your framework → | Try in a colab →

If you have any questions, please don't hesitate to ask in our Slack community.

Free for individuals, academics

... and open source projects. We are committed to staying free to support the advancement of machine learning.

If you'd like a free academic account for your research group, reach out to us →.

Debug model performance in real time

Log predictions to see how your model is performing in realtime, and identify problem areas during training. We support rich media including images, video, audio, bounding boxes, segmentation maps and 3D objects.

Try in a colab →

System metrics – CPU and GPU usage

Visualize live metrics like GPU utilization to identify training bottlenecks and avoid wasting expensive resources with automatically generated system metrics.

Keep GPU costs down, and increase iteration speeds by efficiently allocating resources in distributed training pipelines.

wandb is framework agnostic, to start capturing your system metrics, simply add the following lines to your training script:

import wandb
wandb.init(project="my-project")

# model training code here 

Initializing wandb automatically logs system metrics every 2 seconds, averaged over a 30 second period. The metrics include:

Powerful custom charts

Easily log custom, interactive visualizations to understand & present your results. Share customized plots with your team, instead of matplotlib screenshots.

For example, Kyle Goyette built this plot to understand why seq2seq models make specific predictions. The weight of the connecting lines shows how much attention the decoder paid to a given input word (on the bottom) when producing an output word (on the top).

Hover over a word to highlight the connections for that word.

Try custom charts →

Share model insights interactively

It's never been easier to share model insights with your coworkers, stakeholders, or even the whole world.

Reports (like this one) combine interactive plots for performance metrics, predictions, hyperparameters with explanatory text to help you tell the story of your model. You can use them to explain how your model works, show plots of how model versions improved, discuss bugs, and demonstrate progress towards milestones with W&B.

Here are two reports our users have created. Find more reports in our Gallery.

Creating a report is simple.

Collaboration with W&B.gif

Efficient hyperparameter optimization

Find the best hyperparameters for your model and data pipelines, in a few lines of code. Use Weights & Biases Sweeps to automate hyperparameter optimization and explore the space of possible models.

Sweeps are lightweight, fast to set up and plug into your existing infrastructure.

Try sweeps →

Hyperparameter Optimization with W&B 2.gif

Pipeline tracking and production model management

Use W&B Artifacts to store and keep track of datasets, models, and evaluation results across machine learning pipelines with just a few lines of code. This allows you to trace the flow of data through your pipeline, so you know exactly which datasets feed into your models.

Think of an artifact as a versioned folder of data. You can store entire datasets directly in artifacts, or use artifact references to point to data in other systems.

Learn more →

Simple integration with any framework

🏗 Keras

In Keras, you can use our callback to automatically save all the metrics tracked in model.fit. To get you started here's a minimal example:

# Import W&B
import wandb
from wandb.keras import WandbCallback

# Step1: Initialize W&B run
wandb.init(project='project_name')

# 2. Save model inputs and hyperparameters
config = wandb.config
config.learning_rate = 0.01

# Model training code here ...

# Step 3: Add WandbCallback 
model.fit(X_train, y_train,  validation_data=(X_test, y_test),
          callbacks=[WandbCallback()])

🔥 PyTorch

W&B provides first class support for PyTorch. To automatically log gradients and store the network topology, you can call .watch and pass in your PyTorch model. Then use .log for anything else you want to track, like so:

import wandb

# 1. Start a new run
wandb.init(project="gpt-3")

# 2. Save model inputs and hyperparameters
config = wandb.config
config.dropout = 0.01

# 3. Log gradients and model parameters
wandb.watch(model)
for batch_idx, (data, target) in enumerate(train_loader):
  ...  
  if batch_idx % args.log_interval == 0:      
    # 4. Log metrics to visualize performance
    wandb.log({"loss": loss})

🌊 TensorFlow

The simplest way to log metrics in TensorFlow is by logging tf.summary with our TensorFlow logger:

import wandb

# 1. Start a W&B run
wandb.init(project='gpt3')

# 2. Save model inputs and hyperparameters
config = wandb.config
config.learning_rate = 0.01

# Model training here

# 3. Log metrics over time to visualize performance
with tf.Session() as sess:
  # ...
  wandb.tensorflow.log(tf.summary.merge_all())

💨 fastai

Visualize, compare, and iterate on fastai models using Weights & Biases with the WandbCallback.

import wandb
from fastai2.callback.wandb import WandbCallback

# 1. Start a new run
wandb.init(project="gpt-3")

# 2. Automatically log model metrics
learn.fit(..., cbs=WandbCallback())

🤗 HuggingFace

Just run a script using HuggingFace's Trainer in an environment where wandb is installed and we'll automatically log losses, evaluation metrics, model topology and gradients:

# 1. Install the wandb library
pip install wandb

# 2. Run a script that has the Trainer to automatically logs metrics, model topology and gradients
python run_glue.py \
 --model_name_or_path bert-base-uncased \
 --task_name MRPC \
 --data_dir $GLUE_DIR/$TASK_NAME \
 --do_train \
 --evaluate_during_training \
 --max_seq_length 128 \
 --per_gpu_train_batch_size 32 \
 --learning_rate 2e-5 \
 --num_train_epochs 3 \
 --output_dir /tmp/$TASK_NAME/ \
 --overwrite_output_dir \
 --logging_steps 50

Free for academics and individuals

W&B is free for individuals and academic teams, and will always be. We are committed to staying free to support the advancement of machine learning, and we make it easy to export data with our Export API.

Built for scale

We handle millions of models every month for teams doing some of the most cutting-edge deep learning research.

Data & Privacy

We take security very seriously, and our cloud-hosted dashboard uses industry standard best practices for encryption. If you're working with datasets that cannot leave your enterprise cluster, we have on-prem installations available.

It's also easy to download all your data and export it to other tools— like custom analysis in a Jupyter notebook. Here's more on our API.

Advice from experts

See how teams working on cutting edge deep learning projects use W&B to train, collaborate on, and debug their workflows.

Managing Large Deep Learning Projects – OpenAI

How the OpenAI Robotics team uses Weights and Biases to run a large deep learning project – like training a robot hand to manipulate physical objects like the Rubik's cube.

Science of Debugging with W&B – Latent Space

How the LatentSpace team uses Weights and Biases to quickly identify, communicate, and iteratively debug their models and build on top of each other's work.

Experiment Tracking Best Practices – Lukas Biewald

Why is experiment tracking so important for doing real world machine learning?

image.png

Join our communities

FAQ

Please see our technical FAQ for commonly asked questions about the API and product.

Who has rights to the data?

You can always export and delete your data at any time. We will never share data associated with private projects. We hope that when you can, you will make your work public so that other practitioners can learn from it.

We hope to discover and share high level patterns to move the field of machine learning forward. For example, we wrote this article on how people are not fully utilizing their GPUs. We want to do this in a way that respects your privacy and feels honest. If you have any concerns about data privacy, we'd love to hear from you. Reach out at contact@wandb.com.

Why are you building these tools?

At Weights & Biases our mission is to build the best tools for machine learning. Our experienced technical cofounders built Figure Eight, and our tools are being used by cutting-edge machine learning teams including OpenAI and Toyota. We enjoy making useful tools, and the best part of our day is interacting with people who are building real models using our product.

https://gblobscdn.gitbook.com/assets%2F-Lqya5RvLedGEWPhtkjU%2F-M0e0PqYEmqY7IY6on_G%2F-Lr50PBpxE-Hby1AyJFJ%2Fimage.png?alt=media

Questions?

If you have any questions, please don't hesitate to ask in our Slack community.