charles edits on Announcing W&B Tables: Iterate on your data
Created on July 18|Last edited on July 18
Comment
Today we're excited to launch W&B Tables, a new tool for data iteration and model evaluation. Tables will help you achieve faster model development cycles and a deeper understanding of how your models work.
With Tables, you can log, version, organize, and visualize structured data. Our earliest adopters have used it to track how their training datasets evolve, to record every prediction made during model evaluation, and more. The powerful new Table Visualizer helps them find slices of data that models find difficult and unearth patterns in model behavior that guide dataset construction.
That's a Toadstool, Not a Toad: Visualizing Model Mistakes with Tables
Here's an example Tables visualization comparing the errors made by five different models trained to classify living things using a subset of the iNaturalist 2017 dataset.
With this table, we can find out which validation examples cause our models the most trouble and search for patterns in our models' errors.
Check out this report by Stacey to see more in-depth, real-world examples from this project demonstrating how Tables can help you understand your models in the context of your data.
Mendeleev 500val runs
4
Since this document is a W&B Report, the panel above is both interactive and computed live, directly from tables of model predictions. Let's walk through its construction.
Each table contains rich information about a single model -- you can check out one of those tables here. But we're searching for trends in errors across models. So from these tables, we construct a new table by concatenating the predictions across models, filtering to predictions that didn't match the ground truth, and grouping by the predicted class. From the raw predictions, we've created a table where each row contains input data and output metrics for all our models' errors for a single class.
You can create a new table right here in this Report: click a column header to play with grouping, create a new derived column as an expression over the data in the other columns, or change the filters to focus on a different slice of the data.
For inspiration, below is another visualization built from the same data, showing the images where a model thought it saw an amphibian (filter on the model's guess) and the confidence was high (sort by the score), but the image actually contained a fungus (filter on the ground truth). Lo, and behold: some rather froglike fungi, perched on a log!
Mendeleev 500val runs
4
What's Inside Tables
For now, here's a brief overview.
A Table is well, a table, with rows and columns. You can add rich media values like images with bounding box controls, and cross-table references for seamlessly connecting one table to another. To store a Table with W&B, construct the wandb.Table class, and then add your Table to a wandb.Artifact, or log it in a run with wandb.log.
Tables comes with a powerful type system that you don't need to learn. It will infer the types of data you give it automatically.
Since Tables is built on top of W&B Artifacts, you get efficient versioning and data lineage tracking automatically.
Part of the Artifacts DAG in Stacey's Mendeleev project
The Table Visualizer is built using a new system we've developed called Weave, that allows us (and eventually you!) to create flexible, composable and scalable data UIs.
You can build derived columns in the Table Visualizer using Weave's powerful data-aware auto-suggest capabilities. The Table Visualizer keeps track of types and automatically uses appropriate visualizations for each column. And you can filter, group, sort, and compare tables by interacting with the controls.
We're just getting started. Over the coming months, we'll:
- Scale Tables up with the Weave backend. We'll remove the current artificial limit of 200k rows per Table, and make everything really fast.
- Release a new way to plot your data using Weave. You'll be able to plot data from Tables, runs, artifacts and more. And you'll be able to perform projections like T-SNE to get a new perspective on your data.
Head over to the Tables docs to get started! And as always, we love any and all feedback. If you've got questions, ideas, bug reports, or anything else you'd like to discuss, please don't hesitate to ping us at support@wandb.com.
Add a comment