GSK Weave PoC Guide

One stop shop for everything you need to test out during the W&B Pilot.
Created on April 25|Last edited on April 25
Comment
﻿
Access Weights & Biases here: https://gsk.wandb.io﻿
💡
For Any Questions, please reach out via the Teams chat.﻿﻿
💡
Weights and Biases (W&B) 💫Quick Documentations LinksAccess to W&BEnvironment VariablesW&B Installation & AuthenticationTrack and evaluate GenAI applications via W&B Weave Get started with W&B Weave - Quickstart guideFAQsW&B Weave
﻿
Weights and Biases (W&B) 💫Weights and Biases is an MLOps and LLMOps platform built to facilitate collaboration and reproducibility across the machine learning development lifecycle. Machine learning projects can quickly become a mess without some best practices in place to aid developers and scientists as they iterate on models and move them to production. 
W&B is lightweight enough to work with whatever framework or platform teams are currently using, but enables teams to quickly start logging their important results to a central system of record. On top of this system of record, W&B has built visualization, automation, and documentation capabilities for better debugging, model tuning, and project management. 
Quick Documentations LinksAccess the main Weave documentation page: https://weave-docs.wandb.ai/﻿﻿﻿
Learn how to integrate W&B Weave with your stack: https://weave-docs.wandb.ai/guides/integrations/﻿
W&B Weave Integration with OpenAI: https://weave-docs.wandb.ai/guides/integrations/openai﻿﻿﻿
Trace LLM Applications: https://weave-docs.wandb.ai/tutorial-tracing_2﻿﻿﻿
W&B Weave Cookbooks: https://weave-docs.wandb.ai/reference/gen_notebooks/intro_notebook﻿
RAG App Evaluation Tutorial: https://weave-docs.wandb.ai/tutorial-rag﻿
Handle and Redact PII data with W&B Weave: https://weave-docs.wandb.ai/reference/gen_notebooks/pii﻿
﻿
Access to W&BIf you do not already have a W&B account please ask an admin to send you an invite. 
Our suggestion is to create a team that you can then log projects to. This will need to be done by an admin.
Environment VariablesIt will make things easier to set the following environment variables. This will save you having to specify the host and key as detailed further down in this document.
os.environ["WANDB_HOST"].    = "https://gsk.wandb.io"
os.environ["WANDB_BASE_URL"] = "https://gsk.wandb.io"
os.environ["WANDB_API_KEY"]  = "your-wandb-api-key"
﻿﻿Of course you may need quotes depending how you set these. Your API key is available at https://gsk.wandb.io/authorize﻿
Optionally you can also set the following or define these as part of the weave init call e.g. weave.init(team-name/project-name)
os.environ["WANDB_ENTITY"]  = "your-wandb-team-name"
os.environ["WANDB_PROJECT"] = "your-weave-project-name"
W&B Installation & AuthenticationTo start using W&B, you first need to install the Python package (if it's not already there)
pip install wandb weave
Once it's installed, authenticate your user account by logging in through the CLI or SDK. You should have receive an email to sign up to the platform, after which you can obtain your API token (The API token is in your "Settings" section under your profile)
wandb login --host <YOUR W&B HOST URL> <YOUR API TOKEN>
OR through Python:
﻿﻿wandb.login(host=os.getenv("WANDB_BASE_URL"), key=os.getenv("WANDB_API_KEY"))
In headless environments, you can instead define the WANDB_API_KEY environment variable.﻿
Once you are logged in, you are ready to track your workflows!
﻿
Track and evaluate GenAI applications via W&B Weave ﻿
﻿
Weave is a lightweight toolkit for tracking and evaluating GenAI applications
The goal is to bring rigour, best-practices, and composability to the inherently experimental process of developing GenAI applications, without introducing cognitive overhead.
﻿
﻿
Weave can be used to:
Log and debug model inputs, outputs, and traces
Build rigorous, apples-to-apples evaluations for language model use cases
Capture valuable feedback that can be used to build new training and evaluation sets
Organise all the information generated across the LLM workflow, from experimentation to evaluations to production
A quick-start guide to weave can be found here. 
Get started with W&B Weave - Quickstart guideOnce you have authenticated with W&B, you can start by creating a Weave project with the following command
import weave
weave.init('<entity-name>/<project-name>')
# this ensures the project is created in the relevant GSK team (entity). 
# If you have WANDB_ENTITY and WANDB_PROJECT set as env vars then you won't need to specify these.
# You may need to create the team. 
Now you can decorate the functions you want to track by adding this one line decorator weave.op() to your functions.
Here's what an example script would look like (feel free to copy paste this in your IDE and run this script)
import weave
from openai import OpenAI
﻿
client = OpenAI()
﻿
# Weave will track the inputs, outputs and code of this function
@weave.op()
def extract_dinos(sentence: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": """In JSON format extract a list of `dinosaurs`, with their `name`,
their `common_name`, and whether its `diet` is a herbivore or carnivore"""
            },
            {
                "role": "user",
                "content": sentence
            }
            ],
            response_format={ "type": "json_object" }
        )
    return response.choices[0].message.content
﻿
﻿
# Initialise the weave project
weave.init('<entity-name>/jurassic-park')
﻿
sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""
﻿
result = extract_dinos(sentence)
print(result)
FAQs
W&B Weave1. How does Tracing with W&B Weave work
﻿This loom video (~4mins) walks through how tracing works with W&B Weave 
2. How can I add a custom cost for my GenAI model?
You can add a custom cost by using the add_cost method. This guide walks you through the steps of adding a custom cost. Additionally we also have this cookbook on Setting up a custom cost model with associated notebook.
3. How can I create my own custom Scorers with W&B Weave?
W&B Weave has it's own predefined scorers that you use as well as create your own Scorers. This documentation walks through creating your own scorers with W&B Weave
4. Can I control/customize the data that is logged?
Yes, If you want to change the data that is logged to weave without modifying the original function (e.g. to hide sensitive data), you can pass postprocess_inputs and postprocess_output to the op decorator.
﻿Here's more details on how to do so
5. How to publish prompts to W&B Weave?
W&B Weave support Prompts as first class object. You can use weave.publish() to log prompts or any object as well (eg: Datasets, Models etc.) to Weave. This guide walks into details on publishing prompts to W&B Weave
6. How do I handle PII/sensitive data so it's not logged to W&B Weave?
W&B Weave has different ways to redact sensitive data, this guide walks through the details on different ways to Handle and Redact PII data.
﻿
Add a comment