Skip to main content
tim-w
Projects
MNIST-v4
Reports
Demo Report

Demo Report

Tim Whittaker
Created on May 9|Last edited on May 9

Weights and Biases (W&B) 🪄🐝

Weights and Biases is a ML Ops platform built to facilitate collaboration and reproducibility across the machine learning development lifecycle. Machine learning projects can quickly become a mess without some best practices in place to aid developers and scientists as they iterate on models and move them to production.
W&B is lightweight enough to work with whatever framework or platform teams are currently using, but enables teams to quickly start logging their important results to a central system of record. On top of this system of record, W&B has built visualization, automation, and documentation capabilities for better debugging, model tuning, and project management.

Logging to the W&B System of Record

Experiment Tracking

W&B has a few core primitives which comprise the experiment tracking logging system of the SDK. You can log pretty much anything with W&B: scalar metrics, images, video, custom plots, etc.
To get an idea of the variety of data types you can log, check out the below report, which has code snippets for different media types that may pertain to your use case.

The canonical sections of code which require logging are the training loop and an evaluation job on validation or golden dataset, but you can log any piece of code in your workflow such as data pre-processing, augmentation, generation, etc. All you have to do is call wandb.init() and log diagnostic charts, metrics, and mixed media with wandb.log(). An executed piece of code contextualized by wandb.init() is called a run. 
wandb.init(project='gpt4')
config = wandb.config
config.learning_rate = 0.01
# 2. Save model inputs and hyperparameters
# Model training here
# 3. Log metrics over time to visualize performance
wandb.log({"loss": loss})
wandb.finish()
For example, we could get off to the races with a toy example

This set of panels contains runs from a private project, which cannot be shown in this report

For example, we can get up and running with something simple
For example, we can get up and running with something simple For example, we can get up and running with something simple

Artifact Tracking and Versioning

Artifacts are inputs and outputs of each part of your machine learning pipeline, namely datasets and models. Training datasets change over time as new data is collected, removed, or re-labeled, models change with new architectures being implemented along with continuous re-retraining. With these changes, all downstream tasks utilizing the changed datasets and models will be affected and understanding this dependency chain is critical for debugging effectively. W&B can log this dependency graph easily with a few lines of code.
Let's say we have a directory "sample_images" which stores a set of images and labels in our local development environment
import wandb
with wandb.init(project="my_project", job_type="model_training") as run:
# Create Artifact
training_images = wandb.Artifact(name='training_images', type="training_data")
# Add serialized data
training_images.add_dir('sample_images')
# Log to W&B, automatic versioning
wandb.log_artifact(training_images)
The "sample_images" directory more often exists in some cloud object store like s3 or remote file system, in which case W&B can track references to the respective artifacts. In this case, W&B will still automatically version and provide durable URI's and user-defined aliases to the underlying artifacts.

### Generic Training Loop
with wandb.init(project="my_project", entity="my_team", job_type="data-prep") as run:
	## do something with your data 
	train_data_artifact = wandb.Artifact("train_data", type="dataset")
	train_data_artifact.add_dir("/data/train")
	val_data_artifact = wandb.Artifact("val_data", type="dataset")
	val_data_artifact.add_dir("/data/val")
	run.log({"train_dataset": train_dataset_artifact, "val_data": test_data_artifact})

with wandb.init(project="my_project", entity="my_team", job_type="training") as run:
	train_data_artifact.use_artifact()
	for i in range(epochs):
		optimizer.step()
		wandb.log({"train_loss": train_loss,
			   "val_loss": val_loss})
	model_artifact = wandb.Artifact("my_model", type = "model")
	model_artifact.add_dir("/path/to/model")
	run.log({"model": model_artifact")

### Evaluation Script to Assess Model Errors
with wandb.init(project="my_project", entity="my_team", job_type="evaluation") as run:
        model_artifact.use_artifact()
	val_data_artifact.use_artifact()
	test_data = load(test_data_artifact)
	val_preds = self.model.predict(test_data)
    	# log validation predictions alongside the run
    	columns=["id", "image", "guess", "truth"]
    	predictions_table = wandb.Table(columns = columns)
    	# log image, predicted and actual labels, and all scores to an interactive table
    	for filepath, img, top_guess, scores, truth in zip(self.generator.filenames,
                                                       val_data, 
                                                       max_preds, 
                                                       val_preds,
                                                       true_ids):
      		row = [img_id, wandb.Image(img), top_guess, truth, scores]
      		predictions_table.add_data(row)
    	wandb.log({"evaluation_table" : predictions_table})
You can also embed rich media and plots into W&B Tables, which provide a persistent, interactive evaluation store for your models. More on them below 👇



dSection 1


Run set
63



Run set
63



Run set
63



Run set
63


Created with ❤️ on Weights & Biases.

https://wandb.ai/tim-w/MNIST-v4/reports/Demo-Report--VmlldzoxOTc2OTE4
Made with Weights & Biases. Sign up or log in to create reports like this one.