W&B Guided Evaluation (Getting started on SaaS)

A guide to getting started with the cloud instance of Weights & Biases (e.g. https://www.wandb.ai).
Created on August 13|Last edited on August 13
Comment
﻿
Table of ContentsTable of ContentsWeights and Biases (W&B) 💫Supporting the key principles of an ML workflow15 minute overview of W&B end-to-endGetting startedYour account teamHow do I create accounts for my team?SDK Installation and LoginExperiment TrackingWhat can I log and how do I log itW&B TablesData and Model (Artifact) Tracking and VersioningSlack channel and support detailsHow to ... Examples of using W&B Reports as a log book / training journalProgrammatic Reports using the W&B APIExamples of using W&B Tables Model CI/CDBest Practice GuidesTips and TricksML coursesOther useful links
﻿
﻿
Weights and Biases (W&B) 💫ML practitioners, MLOps teams, and ML leaders at the most cutting-edge of AI and LLM organisations rely on W&B as a definitive system of record, while boosting individual productivity, scaling production ML, and enabling a streamlined CI/CD ML workflow. Offering over 19,000 integrations with ML frameworks and libraries, and a robust partner ecosystem with all the major cloud and infra providers, the W&B platform seamlessly fits into any industry or use case, and across the entire ML lifecycle. 
﻿
﻿
Supporting the key principles of an ML workflow
15 minute overview of W&B end-to-end
Getting started
Your account team
How do I create accounts for my team?
SDK Installation and Login
Experiment Tracking
What can I log and how do I log itScalar metrics can be logged by passing them in to  wandb.log as a dictionary with a name. 
wandb.log({"my_metric": some_scalar_value})
Each time wandb.log is called, that increments a variable W&B keeps track of called step. This is the (x-axis) you see with all the time-series charts. If you call wandb.log every epoch, then the step represents the epoch count, but you may be calling it other times in validation or testing loops, in which case the step is not as clear. To pass a step manually (simply add step = my_int_variable) to wandb.log. This can be important to getting your charts at the resolution you want. 
In Pytorch Lightning modules, you may want to set step to trainer.global_step  for example. It is recommended to pack as many metrics as you can into a single dictionary and logging them in one go vs. separate wandb.log calls, each of which increment the step. ﻿﻿
﻿
This set of panels contains runs from a private project, which cannot be shown in this report
﻿
﻿
You will notice that if you log a scalar metric multiple times in a run, it will appear as a line chart with the step as the x-axis, and it will also appear in the Runs Table. The entry in the Runs Table is the summary metric, which defaults to the last value logged during the course of the run. You can change this behavior by setting the summary metric in the run using run.summary["my_metric_name"]=some_value . This is useful if you want to compare runs according to different aggregations of a given metric (e.g. mean, max, min) as opposed to simply the last one. 
wandb.init()
﻿
for i in range(5):
    wandb.log({"my_metric": i})
﻿
wandb.summary["my_metric"] = 2 # 2 instead of the default 4
﻿
wandb.finish()
﻿
W&B TablesTables are a special wandb Data Type, which allow you to log data, including other wandb Data Types, into an interactive dataframe in the workspace. This is especially useful for logging model predictions in order to filter them and inspect errors. To log a table you can add data row-by-row or as a pandas dataframe or Python lists. The elements of the dataframe can be any wandb Data Type (e.g. wandb.Image, wandb.Html, wandb.Plotly) or simple scalar or text values:
# Add data as a list of lists or pandas dataframe
my_data = [
  [0, wandb.Image("img_0.jpg"), 0, 0],
  [1, wandb.Image("img_1.jpg"), 8, 0],
  [2, wandb.Image("img_2.jpg"), 7, 1],
  [3, wandb.Image("img_3.jpg"), 1, 1]
]
          
# create a wandb.Table() with corresponding columns
columns=["id", "image", "prediction", "truth"]
test_table = wandb.Table(data=my_data, columns=columns)
﻿
# Add data incrementally
for img_id, img in enumerate(mnist_test_data):
    true_label = mnist_test_data_labels[img_id]
    guess_label = my_model.predict(img)
    test_table.add_data(img_id, wandb.Image(img), \
                         guess_label, true_label)
﻿
wandb.log({"test_table": test_table})
﻿
Use tables to log validation, sample predictions, or model errors, not entire training datasets. They can handle up to 200k rows but UI performance will vary depending on how many rich media types you have embedded. Here is a comprehensive guide to logging tables. 
Note on Tables: when logging tables you will see in the workspace wandb.summary["my_table_name"] like below. This is using a weave expression to query logged data in W&B and render it appropriately. Read more about weave here.  W&B by default only renders the last version of a table (the summary one) logged in a run. So if you are logging the same table (name) multiple times throughout a run, you will only see the last one by default. There are different ways to approach this if you need to see all of them, ping us to discuss.
﻿
﻿
project("dummy-team", "that_was_easy").runs.summary["test_predictions"]
﻿
Data and Model (Artifact) Tracking and Versioning
Slack channel and support details
How to ... 
Examples of using W&B Reports as a log book / training journal
Programmatic Reports using the W&B API
Examples of using W&B Tables 
Model CI/CD
Best Practice Guides
Tips and Tricks
ML courses
Other useful links﻿
Add a comment