Skip to main content
wandb
Projects
report
Reports
Foundation Model Builder Tips & Best Practices

Foundation Model Builder Tips & Best Practices

This report contains some helpful tips and best practices for those who are building foundation models, kicking off long and extensive training runs, or just utilizing W&B at scale
Uma Krishnaswamy, Abraham Leal, Mohammad Bakir, Travis Depuy, Wyler Zahm, Venky Yerneni
Created on August 21|Last edited on February 12
Programmatic Access to MetricsQuerying sampled points in a run (more performant)Querying all points in a run:Querying a summary metric (final point, best point, min/max) Downloading System Metrics DataSaved ViewsCreate a new saved workspace viewProgrammatic WorkspacesPublic Preview: W&B Charts Full Fidelity plots 📈 Panel ManagementRun PreemptionPrivate Preview: W&B Run Rewind ⏪Private Preview: W&B Run Forking 🌿 Best Practices for Best PerformanceHardware RecommendationsLogging RecommendationsHiding panels programmaticallySection CountSearching Runs & Looking for Metrics


Programmatic Access to Metrics

If you'd like to query points via the W&B API, you can do so via the following:

Querying sampled points in a run (more performant)

If you'd like to return sampled history metrics for a run, you can do the following:
import wandb

api = wandb.Api()

# access the run object via the api 
run = api.run('<entity>/<project>/<run_id>')

# history is a dataframe where each column corresponds to a metric
history = run.history()

Querying all points in a run:

import wandb

api = wandb.Api()

# access the run object via the api 
run = api.run('<entity>/<project>/<run_id>')

# full_history is an iterator that yields dictionaries from the run's history, one at a time.
full_history = run.scan_history()

# list of dictionaries where each dictionary represents a single step's data from the run's history.
losses = [row for row in full_history]

Querying a summary metric (final point, best point, min/max)

The last value logged with wandb.log is automatically set as the summary dictionary in a W&B Run.
If you would like to utilize custom summary metrics to capture model performance at the best, min, or max step, check out the docs here.
import wandb


api = wandb.Api()


# access the run object via the api 
run = api.run('<entity>/<project>/<run_id>')


# run.summary is a dictionary of the final points in training
run.summary
Note: We impose a 10.4MB limit on summary metrics
💡

Downloading System Metrics Data

This snippet produces a dataframe with all the system resource consumption metrics for a run and then saves it to a CSV.
import wandb

api = wandb.Api()

run = api.run("<entity>/<project>/<run_id>")
system_metrics = run.history(stream="events")
system_metrics.to_csv("sys_metrics.csv")

Saved Views

In large workspaces (many runs, many metrics, many steps), it can be tedious to filter & group & toggle the visibility of runs to display repeatedly. Create a Saved View to organize your preferred setup of charts and data, so you won't have to redo any of the filtering, grouping, or visibility operations when loading your workspace. For more details, read the documentation here

Create a new saved workspace view

  1. Navigate to a personal workspace or a saved view.
2. Make edits to the workspace.
3. Click on the meatball menu (three horizontal dots) at the top right corner of your workspace. Click on Save as a new view.
New saved views appear in the workspace navigation menu:


Programmatic Workspaces

You can organize and visualize your machine learning experiments more effectively by programmatically creating, managing, and customizing workspaces by defining configurations, setting panel layouts, and organizing sections with the wandb-workspaces W&B library.
wandb-workspaces is a Python library for programmatically creating and customizing W&B Workspaces and Reports.
For more details, read the documentation here

Public Preview: W&B Charts Full Fidelity plots 📈

Currently Active For: SaaS, Dedicated Cloud
What are Full Fidelity Plots? A plot point aggregation method, which plots average values while preserving minimum and maximum values across x-axis point buckets. Providing a complete and precise representation of training runs, allowing users to see detailed outliers and zoom in without losing data integrity.
Highlights:
  • Show all outliers/spikes: Significantly enhance the visibility of training metrics and results, especially for long training runs or large training jobs. Users can now visualize outliers and spikes alongside all other metrics in a single, comprehensive line plot without random sampling.
  • Improved line detail: Aggregates data along the x-axis, showing minimum, maximum, and average values for a complete representation.
  • Full fidelity: Spot outliers and zoom into high-fidelity details without data loss.
For more details on enabling, check out the documentation. See the examples below

Run set
1


Panel Management

Currently Active For: SaaS
Panel Management allows you to quickly curate your workspace by clearing all the panels and rapidly adding back only the metrics and keys you care about. It not only helps improve workspace usability by curating a workspace with only those metrics that matter most to you, but it also improves overall performance of W&B by reducing the amount of data necessary to load in a Workspace

Current Behavior
By default, when new data is logged to your workspace, panels are automatically generated to display it, ensuring you're always up-to-date with the latest metrics. This is especially useful for getting quick insights without extra setup.
However, in larger workspaces , this can become overwhelming as panels accumulate, making it harder to focus on the metrics that matter most. In such cases, you might want to turn off automatic panel generation for a more tailored view and greater control over your workspace.

Clearing All Panels
Workspaces can get cluttered with too many panels over time. You now have the option to clear all panels in a single step, giving you a fresh start to curate a more focused workspace.

How to clear panels: Go to the workspace action bar and select "Clear all panels" from the overflow menu. This action is undoable by clicking on `undo` in the Workspace.

  • What happens: All panels are removed, and you can begin adding only the panels you want.
  • When a workspace is cleared, the action can be reversed by clicked `undo`.
Note: Clearing all panels also disables automatic panel generation.

Bulk adding panels
After clearing panels or turning off automatic generation, you can manually add the panels that matter most to you.
How to add panels: Use the `Add panels` button in the workspace action bar or section headers to add specific panels. Refine your search to less than 500 metrics for bulk add action.
  • Quick add: When you need to add multiple metrics, use the `Quick add` feature to easily select and display multiple panels at once.


Want to turn it off?
You can disable automatic panel generation from your workspace settings ⚙️ if you prefer to manage your panels manually or from a user-prompted banner when logging sufficient amount of data to the project.

Check out this video for a full demo
💡

Run Preemption

Weights & Biases allows users to efficiently manage and restart interrupted or preempted training runs without losing progress or prior information. This feature ensures continuity in long-running experiments by saving and restoring the state of a run, enabling seamless resumption and reducing wasted computational resources.

Private Preview: W&B Run Rewind ⏪

Currently Active For: SaaS, Dedicated Cloud
What is Rewind? Rewind allows you to "rewind" a run’s history to a specific point, enabling truncation or modification without losing original data.
This compliments the run forking capability below, allowing you to correct or modify the run history itself!
Highlights:
  • History Truncation: Modify run history without losing data. Truncate run history to the rewind point, allowing new data logging and recomputed summary metrics.
  • Config Preservation: Preserve and merge original configurations with new ones.
  • Run & Artifact Management: Artifacts are associated with the source run that produced them.
  • Consistent Management: Maintain consistent run IDs for traceable experiments.
For more details and code examples, check out the documentation
How to enable? Contact W&B Support at support@wandb.com to request access to this feature and upgrade the SDK to >=0.17.1

Private Preview: W&B Run Forking 🌿

Currently Active For: SaaS, Dedicated Cloud
What is Run Forking? Run forking allows you to create a new run, or fork from an existing W&B run, enabling users to explore different parameters or models from a specific step without impacting the original run. This is particularly useful for continuous experimentation and iterative improvements with longer training runs.
This compliments the run rewind capability above, providing more flexibility in managing and experimenting with your runs as opposed to modifying their history!
Highlights:
  • History Preservation: Forking preserves the integrity of an experiment appending its history and configuration to the new run, enabling easy comparisons and adjustments.
  • Simplified Experience: Forking simplifies testing alternate hypotheses or parameters from any _step, preserving the original source run's integrity.
  • Incremental Analysis: Forking offers an incremental view of your training experiment, capturing the essence of each variation and its impact to the full training run.
For more details and code examples, check out the documentation
How to enable? Contact W&B Support at support@wandb.com to request access to this feature and upgrade the SDK to >=0.16.5
In the example below, multiple runs are forked from the run_fork_parent run at step 60, preserving the original run's state. This approach allows for iterative experimentation while maintaining a consistent baseline. By forking at a particular step, you ensure that the new runs inherit all the parameters, and data up to that point, enabling focused modifications and optimizations from a known state.

Run set
21



Best Practices for Best Performance

For users who are logging projects with many experiments, metrics, and steps per metric - these are recommendations for keeping workspaces performant

Hardware Recommendations

Viewing large workspaces can be memory intensive, we typically recommend the following to ensure you're getting the best experience:
  • Minimal tabs open at once while viewing the workspace
  • Using Chrome on a laptop or desktop (or any bigger machine with more memory)
  • We advise against viewing large workspaces on mobile devices

Logging Recommendations

The size of metrics (number of metrics, number of steps per metric) has a large impact on read and write performance.
You can achieve more scale and better UI performance by reducing the size of the metrics per run - whether thats reducing the frequency of logging, the number of metrics visible in the UI, or the number of sections.

Hiding panels programmatically

In general, if there are metrics in your workflow you know you won't be viewing later, you can set the hidden=True argument within run.define_metric() to automatically move the panel(s) to the hidden section of the UI. 
import wandb
import numpy as np

run = wandb.init()

run.define_metric("custom_step", hidden=True)
run.define_metric("loss", step_metric="custom_iter_step", hidden=True)

for i in range(10):
    run.log({"custom_step": i**3})
    run.log({"loss": i})

run.finish()

Section Count

If you find you have too many sections and performance is slow, consider the workspace setting to create sections by prefix rather than suffix, which can result in fewer sections and better performance.


Having hundreds of sections in a workspace can hurt performance. Consider creating sections based on high-level groupings of metrics and avoiding an anti-pattern of one section for each metric.

Searching Runs & Looking for Metrics

In general, using regex for run or panel searches can be an intensive operation. If you experience slow loading times when searching, the following can help make navigating the W&B UI a more pleasant experience:
  • Instead of repeatedly searching for a run, we recommend identifying the run through a filter, then saving that filter in a saved view - this way, you can load that saved view directly without having to reconfigure your workspace
  • Utilizing Reports to pull in the panels, runs, and metrics across projects that are relevant
  • Utilizing Panel Management to select the metrics you care about

Created with ❤️ on Weights & Biases.

https://wandb.ai/wandb/report/reports/Foundation-Model-Builder-Tips-Best-Practices--Vmlldzo5MTA3MDI2
Made with Weights & Biases. Sign up or log in to create reports like this one.