Configuring W&B Projects with Hydra
A brief guide on how to use W&B and Hydra together, complete with code and a few known pitfalls to work around
Created on January 31|Last edited on February 28
Comment
Introduction
Hydra (by Meta Research) is a highly versatile configuration and experiment manager designed to offload the stress of managing, comparing and even scaling experiments. Likewise, the motivation to optimize modern day ML workflows to allow practitioners focus on the real problem lies at the heart of Weights and Biases. The question is how we can best marry these tools to create optimal workflows.
That's what we'll be discussing today.
In this report, we'll describe how individuals and teams can seamlessly combine both of these amazing tools in their own projects and workflows to boost their productivity. Specifically, we'll use Hydra and W&B to address the problem of finding the right kind of Normalization in Convolutional classification pipelines for different datasets. (namely MNIST and CIFAR10).
First though, let's cover both solutions in a little more detail before we dive deeper.
Weights & Biases
Tracking and interpreting ML experiments can become extremely complicated as your experiments scale up. Weights & Biases provides an ecosystem of tools to solve all your tracking, visualization, exploration and versioning needs seamlessly with minimal amount of code.
Used by ML teams from large companies like NVIDIA, OpenAI, Toyota, Facebook, and more, W&B has become the new standard in modern day ML pipelines. In the long run, W&B strives to make ML experiments reproducible by allowing practitioners to save and track everything including parameters, hyper-parameters, git commits, invoking command, system stats and so on.
Hydra
Another important problem in ML pipelines is actually building multitudes of experiments and running them. An easy alternative to that is building an experimental test-bed and configuring each experiment separately.
This process can become highly redundant and difficult to manage as your number of experiments increases. Hydra not only provides an intuitive way to manage configurations, but also makes it extremely easy to launch experiments on local and remote clusters.
configs/├── dataset│ ├── cifar10.yaml│ └── mnist.yaml├── defaults.yaml├── hydra│ ├── defaults.yaml│ └── with_ray.yaml├── model│ ├── cifar10.yaml│ └── mnist.yaml├── normalization│ ├── batch.yaml│ ├── default.yaml│ ├── group.yaml│ ├── instance.yaml│ └── nonorm.yaml├── train│ └── defaults.yaml└── wandb└── defaults.yaml
Hydra configurations are stored in a very intuitive directory structure as shown above. Most ML practitioners tend to like this because they can swap out configurations in just one command line argument. For example, if a user wants to change the dataset on which the attached codebase runs from the default MNIST to CIFAR 10, it can done just by adding a command line argument:
$ python main.py dataset=cifar10
Or if the user wants to run the codebase on multiple datasets, it can be done easily by:
$ python main.py -m dataset=cifar10,mnist
Implementation
As always, integrating W&B with a Hydra project is pretty straight forward. All we need to do is:
import wandb@hydra.main(config_path="configs/", config_name="defaults")def run_experiment(cfg):run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)wandb.log({"loss": loss})
Here, wandb.entity and wandb.project are defined as a hydra configuration file for wandb logging. (Ref: Source). Despite the simplicity, there are certain pitfalls that you need to avoid to fully utilize the W&B toolset with Hydra. Let's walk through those:
Pitfall #1: Using Hydra's Config with wandb.config
Hydra uses omegaconf as the default way to interface with the configuration dictionaries. However, it is very important to keep in mind that OmegaConf's dictionary is not a subclass of primitive dictionaries (unlike tools like Addict). Hence directly passing Hydra's Config to wandb.config leads to unexpected results on the dashboard. It's necessary to convert omegaconf.DictConfig to primitive dict() type, before passing to wandb.config.
@hydra.main(config_path="configs/", config_name="defaults")def run_experiment(cfg):wandb.config = omegaconf.OmegaConf.to_container(cfg, resolve=True, throw_on_missing=True)run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)wandb.log({"loss": loss})model = Model(**wandb.config.model.configs)
Pitfall #2: Non-Responsiveness During the Start of Training
This is a known pitfall which sometimes occurs in distributed training scenarios (docs) where wandb's multiprocessing interferes with the multiprocessing provided by Hydra's Multirun, specifically in the distributed launchers like Ray's local cluster.
To solve this, try to changing wandb's multiprocessing protocol either by adding an extra settings parameter to wandb.init as:
wandb.init(settings=wandb.Settings(start_method="thread"))
or by setting a global environment variable from your shell:
$ export WANDB_START_METHOD=thread
A Demonstration
The attached codebase demonstrates how different kinds of normalizations (Batch Norm, Instance Norm and Group Norm) performs on both MNIST and CIFAR10 classification problems. Invoking the experiments can be easily done by:
$ python main.py -m dataset=cifar10,mnist +experiments="glob(*)" +model.num_groups=8
MNIST
39
CIFAR10
142
Weights and Biases Sweeps with Hydra
Sweeps by W&B is a highly scalable hyperparameter search platform, which provides interesting insights and visualization about W&B experiments with minimal requirements code real-estate. Just like any other W&B tool, Sweeps integrates seamlessly with Hydra projects, with no-coding requirements. The only thing one needs to do, is create a configuration file describing the various parameters to sweep upon.
A simple example sweep.yaml file in this project, would be:
program: main.pymethod: bayesmetric:goal: maximizename: test/accuracyparameters:dataset:values: [mnist, cifar10]command:- ${env}- python- ${program}- ${args_no_hyphens}
Invoking the sweep is also pretty straight forward:
$ wandb sweep sweep.yaml
Once you call this, W&B automatically creates a sweep inside your project and returns a command for you to run (like the one in the last line in the screenshot below).
In this case, invoking the command wandb agent adrishd/hydra-example/12xsrs0b automatically launches the sweeps for you and all the telemetry can be easily monitored on the web app https://wandb.ai/adrishd/hydra-example/sweeps/12xsrs0b.
A wandb sweep works by executing the command env python main.py dataset=<value> ... on the shell to generate each runs and uses the search strategy (for example bayesian optimization) to reach the goal for a given metric (here, it tries to maximize the test/accuracy metric).
💡
MNIST Sweep
98
CIFAR Sweep
153
Pitfalls with Sweeps Integration
Although configuring W&B sweeps with Hydra is extremely simple, there are a couple of caveats that needs to be addressed.
Pitfall #3 (Sweep): Passing parameters not present in defaults
Hydra supports passing extra parameters through the command line which aren't present in the default configuration file, by using a + before command. For example, we can pass an extra parameter (let's say experiment) with some value by simply calling:
$ python program.py +experiment=some_experiment
However, the use of +configurations cannot be used for W&B sweeps. Although you can of course use +configuration for hard coding configuration, that doesn't need to be swept over. Simply put, if you needs to add extra parameter(s), like let's say the number of GPUs, you can easily do that by adding +num_gpus=1 to the argument list as follows.
command:- ${env}- python- ${args_no_hyphens}- +num_gpus=1
However you cannot sweep over such + configurations similar to what one does while configuring Hydra Experiments. Speaking of Hydra experiments, many practioners prefer to create Hydra experiments & use Hydra sweeper to sweep over various experiments by passing something like a +experiment=name1,name2,... argument to the script.
This kind of argument passing is not possible with W&B Sweeps, as one cannot pass the + prefix with an argument.
Workaround
There's a rather simple workaround to this where you can initialize the experiment parameter with a default empty file and use W&B sweep to override those empty configs on each call.
Here's how:
First, create an empty default.yaml file inside your experiments/ (or equivalent) folder. Next, in the main defaults file of the project config (here defaults.yaml), point add an extra item in the defaults: list as:
defaults:- dataset: mnist- ... # other defaults- experiment: default
This will allow the W&B sweep to override the experiment parameter with whatever experiment is present in the values: list.
In this project, our experiments directory is called normalization/. As one can see, in both the sweep_mnist.yaml and defaults.yaml file we have a normalization parameter. In the first file, it comes as a W&B Sweep parameter with a list of values to experiment on. And in the next one, we put a use a blank default configuration to initalize normalization as a hydra parameter.
💡
Pitfall #4: W&B's Sweeps with Hydra's Sweep
Currently W&B Sweeps doesn't integrate natively with Hydra's sweeping mechanism. We are currently working on a way to figure out the best way to do that. Stay tuned for more updates on this end!
Conclusion
Hopefully, this quick guide on a few smart ways to use W&B and Hydra together (and a few pitfalls to avoid) helps improve your machine learning workflows. If you'd like any additional information about how to use Hydra with W&B, feel free to drop a question in the comments and we'll do our best to help you out. Thanks for reading!
Add a comment
Aml Hassan Esmail • *
Hi Adrish Dey,
Thank you very much for this amazing article!
There's only one thing that's not clear, in the GitHub repo -the minimal implementation checkout-, the default.yaml file for wandb is missing, where could I find it?
Reply
Iterate on AI agents and models faster. Try Weights & Biases today.