Model CI/CD with webhook automations on Weights & Biases

How to leverage automations for continuous integration and deployment in your organization
Created on July 25|Last edited on June 21
Comment
﻿
What is CI/CD for ML? When changes are made to code in traditional software, a user will typically commit their changes to a git repo and submit a pull request. This may trigger tests on the new code changes to make sure it works and the pull request is often reviewed by members of the team before final approval. The code is then merged into the main branch. 
﻿
In ML development, code is not the only major asset being deployed. Artifacts, namely model weights, play a much bigger role. Testing and deploying a model often involves handing off those weights to a separate team who must run tests, containerize and deploy the model to production. 
This hand-off procedure is rife with issues: context is often lost, files are being juggled around, and it can turn into a scavenger hunt to find the files you need to deploy the model. 
As a ML system of record, W&B has all the context you need about a model's creation: its training and evaluation metrics, dataset dependencies, and other vital metadata. With W&B Model Registry and Webhooks you can now implement a clean hand-off point between ML Engineering and DevOps. 
Today, we'll show you how to do just that. Here's what we'll be covering: 
What is CI/CD for ML? Using webhook automations in Weights & Biases1. Create a GitHub fine-grained Personal Access Token (PAT)2. Go to your Team Settings3. Add your GitHub PAT to the Team Secrets4. Create a webhook5. Go to the W&B Model Registry and create an Automation6. Write GitHub Action Workflow YAMLs to perform CI/CDExample: Automate Model Testing and ReportingExample: Automate Model DeploymentRetrieving Artifacts from W&B in a CI/CD ToolAPICLIConclusion
﻿
Using webhook automations in Weights & Biases﻿Webhook automations allow DevOps teams to easily automate the hand-off of model checkpoints from development to testing to deployment. When new models get linked to the W&B Model Registry, a webhook POST request can automatically trigger a GitHub Action workflow to consume the model, run tests on it, containerize it in an inference server, or even deploy it. 
Here's how to set up a Webhook Automation in W&B and connect to GitHub Actions:
1. Create a GitHub fine-grained Personal Access Token (PAT)First, check the docs here and make sure the PAT has the proper permissions to trigger webhook events. The required permissions are to trigger a repo dispatch are Read and write to Contents of the repo:
﻿
You may want to add other permissions as well. 
2. Go to your Team Settings
﻿
You'll want to look for "Team secrets" and "Webhooks" about two thirds of the way down the page: 
﻿
﻿
3. Add your GitHub PAT to the Team Secrets﻿
﻿
4. Create a webhookFor GitHub Actions you will want to add your PAT as the Access token and the repo url for repo dispatches which is typically https://api.github.com/repos/<org>/<repo_name>/dispatches
﻿
﻿
Once you've filled out the relevant info, click the "Create webhook" button. 
5. Go to the W&B Model Registry and create an AutomationClick on "View details" for a Registered Model you want to create a Webhook Automation for:
﻿
Click "New Automation":
﻿
Choose the triggering event. This can be when a new model version is added to the registry or when an alias is added to an existing version:
﻿
Choose "Webhooks" as the Action Type:
﻿
Choose "Webhooks" as the Action type:
﻿
Select the GitHub Actions Webhook we created earlier and add your payload. You can add template strings to pass W&B context to GitHub Actions for later use. 
﻿
﻿
Note: ensure that the "event_type" field matches the GitHub Action yaml field "type" in the repo dispatch:
💡
on:
  repository_dispatch:
    types: "test-and-report"  # Should match the Automation payload's "event_type"
Lastly, name and describe your automation and you are good to go:
﻿
Here are a couple example GitHub Action Workflow YAMLs to get you started in implementing best practices for Model Registry in CI/CD. 
6. Write GitHub Action Workflow YAMLs to perform CI/CDHere are some examples below to get you started:
Example: Automate Model Testing and Reporting﻿Video Tutorial ﻿
name: Test ML Model
﻿
on:
  repository_dispatch:
    types: "test-and-report"
﻿
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  test_and_report:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    env:
      WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
﻿
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    - uses: actions/checkout@v3 
﻿
    - name: Set up Python     
      uses: actions/setup-python@v4
      with: 
        python-version: '3.7.13'
        cache: 'pip'
    
    - name: Install Dependencies
      run: pip install wandb
      # run: pip install -r ./.github/workflows/requirements.txt
﻿
    - name: Login to W&B
      run: wandb login
﻿
﻿
    # Use the payload to pull down relevant artifacts, test and deploy them! 
    - name: Echo Template Strings from W&B Webhook
      run: |
        echo -e "${{ github.event.client_payload.my_custom_string1 }}\n\
        ${{ github.event.client_payload.my_custom_string2 }}\n\
        ${{ github.event.client_payload.event_type }}\n\
        ${{ github.event.client_payload.event_author }}\n\
        ${{ github.event.client_payload.artifact_version }}\n\
        ${{ github.event.client_payload.artifact_version_string }}\n\
        ${{ github.event.client_payload.artifact_collection_name }}\n\
        ${{ github.event.client_payload.project_name }}\n\
        ${{ github.event.client_payload.entity_name }}"
    
    - name: Retrieve Artifacts
      run: python .github/workflows/retrieve_artifacts.py --registered_model_name "${{ github.event.client_payload.artifact_version_string }}"
﻿
    - name: Test Model Loading and Inference
      run: pytest -s --artifact_path ./wandb_artifact_cache
﻿
    - name: Generate W&B Report with Reports API
      run: python generate_report.py # Check out https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Report_API_Quickstart.ipynb for more details﻿
Example: Automate Model Deployment﻿Video Tutorial﻿
name: Containerize Model in Inference Server
﻿
on:
  repository_dispatch:
    types: "build-inference-server"
﻿
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  build_inference_server:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    env:
      WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
﻿
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    - uses: actions/checkout@v3 
﻿
    - name: Set up Python     
      uses: actions/setup-python@v4
      with: 
        python-version: '3.7.13'
        cache: 'pip'
    
    - name: Install Dependencies
      run: pip install wandb
      # run: pip install -r ./.github/workflows/requirements.txt
﻿
    - name: Login to W&B
      run: wandb login
﻿
﻿
    # Use the payload to pull down relevant artifacts, test and deploy them! 
    - name: Echo Template Strings from W&B Webhook
      run: |
        echo -e "${{ github.event.client_payload.my_custom_string1 }}\n\
        ${{ github.event.client_payload.my_custom_string2 }}\n\
        ${{ github.event.client_payload.event_type }}\n\
        ${{ github.event.client_payload.event_author }}\n\
        ${{ github.event.client_payload.artifact_version }}\n\
        ${{ github.event.client_payload.artifact_version_string }}\n\
        ${{ github.event.client_payload.artifact_collection_name }}\n\
        ${{ github.event.client_payload.project_name }}\n\
        ${{ github.event.client_payload.entity_name }}"
    
   - name: Retrieve Artifacts
     run: python .github/workflows/retrieve_artifacts.py --registered_model_name "${{ github.event.client_payload.artifact_version_string }}"
﻿
    - name: Containerize the artifact in our inference server
      uses: docker/build-push-action@v4
      with:
        context: ./docker_context
        file: src/torchserve/Dockerfile
        push: true
        tags: ${{ env.GAR_LOCATION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.SERVICE }}:${{ github.sha }}
Retrieving Artifacts from W&B in a CI/CD Tool
API
CLIThe wandb CLI can be used to retrieve an artifact with wandb artifact get <entity>/<project>/<name>:<alias> --root <target_dir> . This is convenient but the downside is you won't see the lineage of the usage of that artifact downstream. For instance you can use in a github action workflow YAML
  run: |
     ARG="${{ github.event.client_payload.artifact_version_string }}" # This avoids space escaping in registered model name
     wandb artifact get "$ARG" --root ./wandb_artifact_cache
ConclusionThis concludes our brief introduction to using webhook automations for CI/CD on Weights & Biases. If you'd like to learn more about how they work on our platform, we recommend checking out our docs. We also have a new free, interactive course on model CI/CD where we get into greater detail on webhooks and a plenty more. You can register for free here. 
﻿
﻿
Add a comment
Tags: Articles, Intermediate, W&B Features, MLOps
Iterate on AI agents and models faster. Try Weights & Biases today.