A complete YOLOX experiment and model tracking pipeline
YOLOX now comes with a full-fledged Weights & Biases integration which allows you to track experiment metrics, hardware statistics, config parameters, and more. Here's an example.
Created on October 5|Last edited on October 11
Comment
YOLOX now supports an easy-to-use full-fledged Weights & Biases integration allowing one to track experiment metrics and hardware statistics real time on to their centralised W&B project dashboard. It also comes with a whole host of features ranging from logging config variables of each run for reproducibility to an interactive bounding box debugger for visualising model predictions along with W&B Tables which allows you to inspect each sample and the corresponding model predictions on it giving you more control over the failure modes of the model. Lastly but not the least, the integration also allows you to visualise model's gradients per layer and also saves the checkpoints using W&B Artifacts.
Getting started
There are only 2 required steps to have all your model metrics tracked in your W&B Dashboard:
- Install Weights & Biases: Run pip install wandb to fetch and install the latest stable release of WandB.
- Once WandB is successfully installed on the server/ environment, you can simply turn on the WandB flag in the training file for YOLOX. There are two aliases (-wb/ --wandb) for the W&B flags to activate W&B for your experiment:
python tools/train.py -n yolox-s -d 8 -b 64 --fp16 -o -wb [--cache]yolox-myolox-lyolox-x
or
python tools/train.py -n yolox-s -d 8 -b 64 --fp16 -o --wandb [--cache]yolox-myolox-lyolox-x
This assumes that you have already installed the required packages and set-up the dataset correctly in the appropriate path and format. More details on this can be found in the instructions on the readme of the YOLOX repository.
Using the WandB flag triggers the wandb.login() which logins to your wandb account and starts logging the experiment run with the default parameters as specified in the YOLOX/yolox/utils/wandb/wandb_utils.py file which can be changed based on one's requirements. For more info regarding these parameters, visit the wandb.init() docs page here.
W&B Dashboard
YOLOX main project dashboard with all logged experiment runs.
Using W&B logger in YOLOX will result in the creation of a project dashboard on your profile or the entity specified in the wandb.init() parameters in the logger. By default, the logger will create the project under the entity W&B had previously logged to. The default name of the project is YOLOX which can also be changed along with the run-name in the __init__ arguments of the W&B logger class.
The main dashboard allows you to view all the panels of all runs logged under that project. You can further sort, re-arrange or group runs based on your preference and requirement.
Metrics Logging
Once W&B Logger is initiated in YOLOX, by default it will log all available training and evaluation metrics on to your dashboard where you can view them collectively for all runs. You can edit these panels based on range values or group them or use smoothing to make noisy plots more interpretable.
Few of the training metrics saved to the dashboard are shown above. You can further export them to use for documentation purposes.
Tables
W&B Tables allows you to comb through the samples in your dataset and analyse model predictions on them, giving you more control over your model's failure modes and allows you to detect patterns in outliers.
With the W&B Tables integration in YOLOX, you can analyse samples individually in the form of tables which stores various metadata in each column along with the bounding box predictions on the sample as shown above.
Bounding Box Debugger
With the BBox debugger in the W&B YOLOX integration, you can now easily analyse the model's bounding box predictions on each samples and interact with the observations which offers the functionalities including selecting specific class bounding box to be shown or showing bounding boxes with a minimum confidence threshold.
Hardware Statistics
WandB logger by default stores and tracks all the hardware statistics real-time for all given runs. This allows you to monitor your system in the form of memory usage, GPU usage, power consumption and thermals, all at one place.
Model Gradients
With the wandb.watch() in the YOLOX integration, WandB tracks and stores the gradients of all the layers for any given model trained.
Config Parameters
All YOLOX runs with W&B logger enabled will save all the training hyper-parameters in the config section of the run on the W&B project page. It also showcases the summary of the metrics that were logged for the run and shows the OS/ python versions used along with the executable code to replicate that run for reproducibility.
Model Checkpointing
With W&B Artifacts, in every training run of YOLOX with W&B initialised, the model weights will be saved and stored as artifacts on your W&B artifacts page in the run dashboard. This allows you to download the artifact and resume training or even construct a compute graph using all artifacts logged to reproduce the training pipeline. The artifacts include metadata providing information on which epoch it was logged at and whether it is the best performance weights or not.
Reports
Finally, you can write a report just like the one you read now and export all your panels on to it to share your amazing research projects and findings in one single place.
Add a comment