Machine Learning Experiment Tracking

Why is experiment tracking so important for doing real world machine learning?

At first glance, building and deploying machine learning models looks a lot like writing code. But there are some key differences that make machine learning harder:

1. Machine Learning projects have far more branching and experimentation than a typical software project.
2. Machine Learning code generally doesn’t throw errors, it just underperforms, making debugging extra difficult and time consuming.
3. A single small change in training data, training code or hyperparameters can wildly change a model’s performance, so reproducing earlier work often requires exactly matching the prior setup.
4. Running machine learning experiments can be time consuming and just the compute costs can get expensive.

Tracking experiments in an organized way helps with all of these core issues. Weights and Biases (wandb) is a simple tool that helps individuals to track their experiments — I talked to several machine learning leaders of different size teams about how they use wandb to track their experiments.

The essential unit of progress in an ML project is an experiment, so most people track what they’re doing somehow — generally I see practitioners start with a spreadsheet or a text file to keep track of what they’re doing.

Spreadsheets and docs are incredibly flexible — what’s wrong with this approach? See more at Machine Learning Experiment Tracking.

‍