Weights & Biases 💜 Ray Tune

Weights & Biases helps your ML team unlock their productivity by optimizing, visualizing, collaborating on, and standardizing their model and data pipelines – regardless of framework, environment, or workflow. Used by the likes of OpenAI, Toyota and Github, W&B is part of the new standard of best practices for machine learning. By saving everything you need to track and compare models — architecture, hyperparameters, weights, model predictions, GPU usage, git commits, and even datasets – W&B makes your ML workflows reproducible.

Today we're announcing an integration with a tool our community adores – Ray/Tune is one of first and most respected libraries for scalable hyperparameter optimization. With just a few lines of code Ray/Tune helps researchers optimize their models with state-of-the-art algorithms and scale their hyperparameter optimization process to hundreds of nodes and GPUs.

Why we chose Ray Tune – Delivering model development and hyperparameter optimization at scale

We're especially excited about the possibilities this collaboration with our friends at Ray/Tune opens up. Both Weights and Biases and Ray/Tune are built for scale and handle millions of models every month for teams doing some of the most cutting-edge deep learning research.

Whereas W&B is a centralized repository for everything you need to track, reproduce and gain insights from your models easily; Ray/Tune provides a simple interface for scaling and running distributed experiments. A few reasons why our community likes Ray/Tune –

Simple Distributed execution: Ray Tune makes it easy to scale from a single node, to multiple GPUs, and further multiple nodes
Large number of algorithms: Ray Tune has a huge number of algorithms including Population Based Training, ASHA, and HyperBand
Framework agnostic: Ray Tune works across frameworks including PyTorch, Keras, Tensorflow, XGBoost, and PyTorchLightning.
Fault-tolerance: Ray Tune is built on top of Ray, providing fault tolerance out of the box.

Getting Started

There are 2 ways you can use the wandb integration with Ray Tune.

1. The WandbLogger

tune.run(
train,
loggers=[WandbLogger],
config={
"wandb": {"project": "rayTune", "monitor_gym": True}},
})

WandbLogger automatically logs the metrics reported to the W&B dashboard of the project.

2. The wandb_mixin

You can also use wandb_mixin function decorator when you need to log any custom metrics, charts and other visualizations

@wandb_mixin
def train(...):
...
wandb.log({...})
tune.report(metric = score)
...

Read the full post →

‍