Table of Contents

Our Hyperparameter Tuning Experiment

Grid Search
Bayesian Search
Population-Based Training

Ray Tune and W&B

Ray Tune provides implementations of State-of-the-Art hyperparameter tuning algorithms that scale
Experiments can be scaled easily from a notebook to GPU-powered servers without any change in code
Experiments can be parallelized across GPUs in 2 lines of code
With W&B Experiment Tracking, you have all your stats in one place for making useful inferences
Using W&B with Ray Tune, you never lose any progress

Enable W&B tracking

You can pass WandbLogger as a logger when calling tune.run. This tracks all the metrics reported to Tune.
You can use @wandb_mixin function decorator and invoke wandb.log to track your desired metrics. Tune initializes W&B run using the information passed in the config dictionary.

     config = {
       ....
          "wandb":{
            "project": "Project_name",
            "api_key": #Your W&B API KEY,
            #Additional wandb.init() parameters
        }
}


Grid Search (Baseline):

{
  "per_gpu_batch_size": [16, 32],
  "learning_rate": [2e-5, 3e-5, 5e-5],
  "num_epochs": [2, 3, 4]
}

analysis = tune.run(
    ...
    resources_per_trial={'gpu': #num_gpu,
      'cpu': #num_cpu }, # Tune will use this information to parallelize the tuning operation
    config=config
    ...
)

Bayesian Search with Asynchronous HyperOpt



{
  "per_gpu_batch_size": (16, 64),
  "weight_decay": (0, 0.3),
  "learning_rate": (1e-5, 5e-5),
  "warmup_steps": (0, 500),
  "num_epochs": (2, 5)
}

Population-Based Training



Run the hyperparameter optimization process for some samples for a given time step (or iterations) T.
After every T iterations, compare the runs and copy the weights of good-performing runs to the bad-performing runs and change their hyperparameter values to be close to the runs' values that performed well.
Terminate the worst-performing runs. Although the algorithm's idea seems simple, there is a lot of complex optimization math that goes into building this from scratch. Tune provides a scalable and easy-to-use implementation of the SOTA PBT algorithm

{
  "per_gpu_batch_size": [16, 32, 64],
  "weight_decay": (0, 0.3),
  "learning_rate": (1e-5, 5e-5),
  "num_epochs": [2, 3, 4, 5]
}

The Winner: Population Based Training of Our Hugging Face Model

Avoiding local minima with Bayesian Optimization: When using a Bayesian Optimization method, it is essential to provide an initial set of “random guesses”. Intuitively, this provides a more informative prior for the Bayesian Optimization to start with. Otherwise, the Optimizer can be myopic and overfit to a small number of samples.
Cutting down iteration time is super important: Always ensure that you utilize all of our machine's computing resources. Anything that can be run in parallel should be run in parallel.
Tweaking the perturbation/mutation interval for PBT: With PBT, an important consideration is the perturbation interval, or how frequently we want to exploit and explore our hyperparameters. For our experiments, we performed this mutation after every epoch. However, doing this too frequently is counterproductive since model performance is noisy if only trained for a few batch steps.
Random seeds also factor into our accuracy results. In addition to tuning the hyperparameters above, it might also be worth sweeping over different random seeds to find the best model. A two-step approach could work best here: First, use an early stopping algorithm to train over many different seeds, and then selecting just the best performing seeds, use Population Based Training to tune the other hyperparameters.

A Few Final Thoughts

All the experiments were parallelized across 8 GPUs by using Tune.
These experiments can be scaled up or down without changing the code.
We have all the important metrics, inferences and even this report in one place and can be easily shared
These inferences can be used to accurately quantify the resources that will be saved by using the suitable search method.
This overall structure leads to more productivity among the teams