Automate Your Experiment Tracking with ChatGPT Custom Instructions and Weights & Biases
ChatGPT is great at writing code. Here's how to get even more automation using custom prompts and Weights & Biases!
Created on August 7|Last edited on August 29
Comment
Introduction
Logging and tracking the results of machine learning experiments can be tedious.By utilizing Weights & Biases (wandb) and custom instructions, this process can be streamlined and automated, saving you time while also ensuring consistency and accuracy.
This post is meant to be a short tutorial (with an example of training on good old MNIST) that explains how to teach ChatGPT to log the results of your experiments using Weights & Biases.
Before Adding the Custom Instruction
Before proceeding with the rest of the tutorial, let's see how ChatGPT writes a simple training example on the MNIST dataset for the prompt “write a short script that trains a conv net on the MNIST dataset using torch."
Note - this is without the custom prompt instruction.
*Response with omitted model and data loading code for brevity
Results without custom instructions
As you can see, there's no logging besides a print statement. That's not ideal!
In my opinion, every ML training loop should have a logging scheme that saves data in a viewable fashion. In addition, I usually prefer a validation loop within my training loop (which I could have specified in my prompt, but I usually follow the format of doing validation loops within the training loop. So I'll add this to my custom instructions as well).
Now: let’s see how we can add a custom prompt that automatically will be used by ChatGPT to use Wandb to log the results of our experiments.
The Prompt
I made a prompt that seems to work well for GPT-4. To add this, go to “Custom instructions” in the settings section.
Here is our prompt:
When writing a machine learning training loop, always log results using wandb. Or perhaps if I ask you to add logging to it, you can use the following instructions. I usually like to log loss and accuracy for the training and validation loops (you will need to add the calculation of accuracy if it is not currently used), and if you see other metrics that may be useful to log, ask me about them. Also if you are unsure about calculating the accuracy, please ask me to confirm your work.
for example "" run = wandb.init(project='{exp_name}', entity='byyoung3', name=experiment_name) "" where exp_name will be an experiment name that you come up with based on the code I give you.
Note that I gave it some instructions and my Weights & Biases username so you will need to replace your username with your (byyoung3) if you want to try this out yourself.
The Result
Using the same prompt as before
Let’s see how ChatGPT responds when requesting the same task as before!
As per the previous instructions we added, the prompt handles naming our experiment as well as initializing it for Weights & Biases!
ChatGPT written code with our custom instructions
Below, you will see that ChatGPT has added functionality for calculating accuracy as well as logging the error.
ChatGPT response with logging!
You may find that the custom prompt provided here is just a starting point. Different projects and workflows might require modifications to the prompt to suit specific needs.
Whether tracking different metrics, integrating with various data preprocessing steps, or adapting to different frameworks and datasets, the custom instructions can be tailored to fit. This works great for writing training loops from scratch and modifying existing loops to add Weights & Biases logging!
Add a comment
Iterate on AI agents and models faster. Try Weights & Biases today.