Shorter charles edits on Announcing W&B Tables: Iterate on your data
Created on July 18|Last edited on July 18
Comment
alt-subtitles: {Dig/Dive} into data
Today, we're excited to launch W&B Tables, a new tool for inspecting and managing rich data about how models and datasets interact. Tables will help you achieve faster model development cycles and a deeper understanding of how your models work.
cool screenshot here; e.g. a mosaic of Tables from Stacey's reports
With Tables, you can log, version, organize, and visualize structured data. Our earliest adopters have used it to track how their training datasets evolve, to record every prediction made during model evaluation, and more. The powerful new Table Visualizer helps them find slices of data that models struggle with and unearth patterns in model behavior that guide dataset construction.
That's a Toadstool, Not a Toad: Visualizing Model Mistakes with Tables
Check out this report from Stacey to see a cornucopia of real-world examples of how Tables will help you understand your models in the context of your data.
Here, we have a simple example to whet your appetite. It uses the Tables Visualizer to dive deep on the errors made by four different models trained to classify pictures of living things using the iNaturalist 2017 dataset.
With this table, we can find out which validation examples cause our models the most trouble and search for patterns in our models' errors. We've filtered the table down to images where a model thought it saw an amphibian (filter on the model's guess) and the confidence was high (sort by the score), but the image actually contained a fungus (filter on the truth).
Lo, and behold: some rather froglike fungi, perched on a log!
Mendeleev 500val runs
4
Tables support the full gamut of operations on structured data. And since this document is a W&B Report, the panel above is both interactive and computed live, directly from the raw tables of model predictions, like this one.
That means anyone reading this Report can visualize and drill down on other aspects of the models' behavior on this data. Change the filter's guess and truth values to see other cases of mistaken identity -- like a plant that looks like a bird. Or create an entirely different table! Click "Reset & Automate Columns", then change the filters to focus on a different slice of the data, click a column header to play with grouping, or even create a new derived column as an expression over the data in the other columns.
Getting Started with Tables
alt-titles: Under the Tables, How Tables Works, Inside Tables
For now, here's a brief overview.
A Table is well, a table, with rows and columns. You can add rich media values, like images with bounding box controls, and cross-table references for seamlessly connecting one table to another. To store a Table with W&B, construct the wandb.Table class, and then add your Table to a wandb.Artifact, or log it in a run with wandb.log.
Tables comes with a powerful type system that you don't need to learn. It will infer the types of data you give it automatically.
Since Tables is built on top of W&B Artifacts, you get efficient versioning and data lineage tracking automatically.
Part of the Artifacts DAG in Stacey's Mendeleev project
The Table Visualizer is built using a new system we've developed called Weave, that allows us (and eventually you!) to create flexible, composable and scalable data UIs.
You can build derived columns in the Table Visualizer using Weave's powerful data-aware auto-suggest capabilities. The Table Visualizer keeps track of types and automatically uses appropriate visualizations for each column. And you can filter, group, sort, and compare tables by interacting with the controls.
We're just getting started. Over the coming months, we'll:
- Scale Tables up with the Weave backend. We'll remove the current artificial limit of 200k rows per Table, and make everything really fast.
- Release a new way to plot your data using Weave. You'll be able to plot data from Tables, runs, artifacts and more. And you'll be able to perform projections like t-SNE to get a new perspective on your data.
Head over to the Tables docs to get started! And as always, we love any and all feedback. If you've got questions, ideas, bug reports, or anything else you'd like to discuss, please don't hesitate to ping us at support@wandb.com.
Add a comment