Collaborative Research and Publication-Ready Graphics with W&B

Real-time collaboration and publication-ready graphics with a few mouse clicks or lines of code
Created on November 22|Last edited on January 10
Comment
In this report, we'll look at how Weights & Biases can make your research more collaborative and reproducibly all while creating publication-ready graphics that showcase your insights and performance. Let's jump right in: 
Creating and Collaborating on W&B Reports To begin, we'll show you how to create and work collaboratively in a Report, export that Report, and, in the final section we'll learn how to export components of a Report – images, charts, etc. – for use in publication. 
We're going to assume, for starters, that your model training run(s) have completed. At that point, just clikc on the Create Report button from your home dashboard (see below). For users who are comfortable with programmatically defining and using reports W&B now offers the Reports API allowing you to use the wandb library to create, reuse, and programmatically edit Reports.
Using the checkboxes, select which categories of charts – eval (evaluation performance)  train (training performance), or System (system metrics such as GPU utilization, power consumption, etc.) – you'd like to display in the Report and click on that Create Report button one last time. 
﻿
At this point, we have a Report containing charts––you can see an example here. Your collaborators can comment in the comment section like you would on a blog, but it's usually a lot more useful collaborating on specific elements. You can comment on a chart (by clicking the speech bubble icon in the upper right-hand corner of each chart cell) or on certain words or paragraphs in the Report itself simply by highlighting the text that you wish to comment on. Please note that users must be logged into their Weights & Biases account in order to be able to comment on a Report.
﻿
Sharing a ReportIf you don't yet have a Team set up in Weights & Biases, you can still share individual Reports with collaborators. 
On any Report page click on the Share button toward the upper left of the Report. This creates a "magic link" (shown below) that you can send to people with or without a Weights & Biases account. This works in an identical fashion to when you share a Google Docs link with 'Anyone who has the link'. Users needn't be logged in to view this Report but they will be required to be logged in and added to your team before leaving comments and/or making edits. Users who are not added to your team (with or without a W&B account) will have view-only access to the Report.
When sharing a report you may also want to embed it inside of another webpage – perhaps on your team's content management system, e.g., Confluence, Notion, or as an iFrame on a HTML page. The step for performing that operation is illustrated in the animation below and a documentation reference page on how to Embed reports can be found here.
﻿
On the off chance that your collaborators don't have access to the Weights & Biases website you can export your Report as well!
Exporting Results for PublicationSometimes you or your collaborators will want to export your findings for use outside of the Reports ecosystem. Weights & Biases offers the ability to export single charts, datasets, experiment metrics (accuracy, etc.), hyperparameter lists, and more with a few mouse clicks or programmatically via the wandb API.
Exporting DataTo start, let's say you want to export data out of an experiment. To download data across several runs, for example, you'd utilize the code below. 
Calling api.runs returns a Runs object that is iterable and acts as a list. By default the object loads 50 runs at a time in sequence, but you can change the number of runs loaded per page by altering the per_page keyword argument. api.runs also accepts an order keyword argument. The default order is descending order, -created_at ; specify +created_at to get results in ascending order. You can also sort by config or summary values e.g. summary.val_acc or config.experiment_name. 
The code below is also auto-generated by clicking on the More Actions > Export Panel menu, which is shown in the first two steps of the animation below in the Exporting a single chart's data section:
import pandas as pd 
import wandb
﻿
api = wandb.Api()
﻿
# Project is specified by <entity/project-name>
runs = api.runs("<entity>/<yourproject>")
summary_list = [] 
config_list = [] 
name_list = [] 
for run in runs: 
    # run.summary are the output key/values like accuracy.
    # We call ._json_dict to omit large files 
    summary_list.append(run.summary._json_dict) 
﻿
    # run.config is the input metrics.
    # We remove special values that start with _.
    config = {k:v for k,v in run.config.items() if not k.startswith('_')}
    config_list.append(config) 
﻿
    # run.name is the name of the run.
    name_list.append(run.name)       
﻿
﻿
summary_df = pd.DataFrame.from_records(summary_list) 
config_df = pd.DataFrame.from_records(config_list) 
name_df = pd.DataFrame({'name': name_list}) 
all_df = pd.concat([name_df, config_df,summary_df], axis=1)
﻿
all_df.to_csv("project.csv")
To learn more about about exporting data for use in other visualizations tools such as matplotlib, seaborn, etc. please reference the documentation section here. For additional examples using the API to export data, en masse, reference this documentation page.
Exporting a single chart's dataInstead of exporting entire projects data, you can easily export a single chart's data for use in other charting libraries, e.g., matplotlib, seaborn , etc.﻿﻿
﻿
﻿
﻿
Exporting FiguresSimilar to how you exported data from your experiments in the previous step you can export figures – in a variety of formats – via the same menu. When exporting figures you can choose between PNG, SVG, or PDF formats.
﻿
﻿﻿﻿
Exporting Figures - TroubleshootingYou may encounter instances in which the default logging is slower than what you would like. As a rule, keep the total number of distinct metrics under 10,000. Additionally, new users are limited to approximately 200 requests per minute via the API, allowing you to run approximately 15 processes in parallel. If you would like to run more than 15 processes in parallel as a new user please reach out to us at contact@wandb.com﻿. Additional information on limits and best practices can be found here.
Exporting Experiment ResultsNow that we've covered exporting portions of an experiment – data, followed by figures – you may be wondering how to export an entire Report. After creating a Report, simply click on the three dots menu item. 
﻿
From there you can choose to duplicate a Report, which is great for when you want to create multiple versions of a Report, perhaps distributing slightly different copies for review to different audiences, or for when you've created an assignment with a list of homework exercises and your students want to create their own copy to fill out. 
Instead of duplicating a Report we want to Export the 'Reporting on first findings' Report, choosing between PDF or LaTeX\LaTeXLATE​X﻿. We choose the LaTeX\LaTeXLATE​X﻿ option, selecting the Reproducibility Template, which generates a conference-ready series of performance curves, any code snippets that you've included in your Report (complete with line numbers), as well as any text that you've written in your Report. A zip file, ready for use in Overleaf or your TeXTeXTeX﻿ editor of choice will be downloaded to your local machine. Make any additions there and then your paper will be ready for publication. 
﻿
﻿
Citing Us﻿
Last, if you'd like to cite W&B, here are some pre-generated ways for you to do so:
"We used Weights & Biases for experiment tracking and visualizations to develop insights for this paper."
﻿
﻿BibTeXBibTeXBibTeX﻿citation@article{biewald2020experiment,
  title={Experiment tracking with weights and biases},
  author={Biewald, Lukas},
  journal={Software available from wandb.com},
  volume={2},
  pages={233},
  year={2020}
}
IEEE CitationL. Biewald, “Experiment Tracking with Weights and Biases,” Weights & Biases. [Online]. Available: http://wandb.com/. [Accessed: Add date here].
Software available from wandb.com﻿
Related Reading:
Weights & Biases: Your Automated Logging Companion for Reproducible Research
How W&B helps students, researchers, teachers, and academics of all stripes log research for easy collaboration, sharing, and reproducibility
How Many Discoveries Were Lost Because They Weren't Written Down?
With Reports you can save your ingenious ideas and insights alongside a snapshot of your work and then share them with your team. Or just keep them all for when you need them.
﻿
﻿