WandBot: GPT-4 Powered Chat Support

Table of Contents

Working with Data

Data Collection and Preprocessing

from langchain.text_splitter import MarkdownTextSplitter, PythonCodeTextSplitter, NotebookTextSplitter

markdown_splitter = MarkdownTextSplitter()
python_code_splitter = PythonCodeTextSplitter()
notebook_splitter = NotebookTextSplitter()

MarkdownTextSplitter: Parses and processes markdown files, ensuring that the bot can understand and generate responses related to content within these files. This is particularly useful when dealing with W&B documentation, as a significant portion is written in Markdown.
PythonCodeTextSplitter: Processes Python code files, enabling the bot to understand and generate responses based on code snippets and programming concepts. This is crucial for addressing technical questions related to the W&B API and its usage in Python projects.
NotebookTextSplitter: Handles Jupyter Notebook files, which are a common format for sharing code, data, and documentation in the machine learning community. This splitter ensures that the bot can understand and generate responses based on the content within these notebooks, including code, markdown, and outputs.

Data Ingestion

Higher dimensionality: HydeEmbeddings use a higher-dimensional vector space, allowing them to capture more nuanced relationships between words and phrases.
Context-awareness: HydeEmbeddings are designed to incorporate contextual information, resulting in a better understanding of the meaning and intent behind the text. By using an LLM-generated document based on a specific question or topic, HydeEmbeddings can capture relevant patterns that help find similar documents in a trusted knowledge base
Robustness: HydeEmbeddings are more resistant to noise and ambiguity, making them better suited for handling complex language structures and diverse document formats. The use of hypothetical answers in the HyDE method helps mitigate the risk of "hallucinations" from the LLM, which can be especially useful in sensitive applications where precise information is critical, such as medicine.

from langchain.embeddings import HydeEmbeddings

hyde_embeddings = HydeEmbeddings()

# Ingest data into Langchain framework using HydeEmbeddings
processed_data = []
for document in preprocessed_data:
    embedding = hyde_embeddings.get_embedding(document)
    processed_data.append({"text": document, "embedding": embedding})

Storing our Vectors in a Vector Store

Storing document embeddings into our FAISS Index

faiss_index = FAISSWithScore()
retriever = VectorStoreRetrieverWithScore(faiss_index)

Storing the FAISS index and embeddings in Weights & Biases Artifacts

import wandb

# Log the FAISS index and embeddings to W&B Artifacts
artifact = wandb.Artifact("faiss_index_and_embeddings", type="data")
artifact.add_file("faiss_index_file.faiss")
artifact.add_file("embeddings.npy")

run = wandb.init()
run.log_artifact(artifact)
run.finish()

Creating WandBot

Designing a Robust Prompt for the LLM

from langchain import ChatPromptTemplate

# Create a custom prompt for the Q&A bot
prompt_template = ChatPromptTemplate(
    user_prompt="User: {question}",
    assistant_prompt="Assistant: {answer}",
    token_constraints={"max_length": 2048}
)

Creating the Q&A Pipeline

Improved search efficiency: By using the FAISS index and OpenAI embeddings, the pipeline can search through a large number of documents quickly, yielding relevant results.
Contextual understanding: Since the pipeline incorporates the HydeEmbeddings, it has a better understanding of the context and can provide more accurate responses.
Scoring mechanism: The RetrievalQAWithSourcesChainWithScore class also provides a scoring mechanism, allowing the system to rank the relevance of the retrieved documents and filter the documents by the similarity scores.
Usage with Weights & Biases: Storing the pipeline components–such as the FAISS index and embeddings–in W&B Artifacts allows for better version control, collaboration, and data portability. The use of W&B Artifacts ensures that the pipeline can be easily updated and shared among team members, facilitating the continuous improvement of the Q&A bot. In the code, the pipeline loads the artifacts using the run.use_artifact method, which simplifies the process of accessing the required data and components.

# Load FAISS index and embeddings from W&B Artifacts
run = wandb.init()
artifact = run.use_artifact("faiss_index_and_embeddings:latest")

faiss_index = FAISSWithScore.load(artifact.get_path("faiss_index_file.faiss").download())
embeddings = np.load(artifact.get_path("embeddings.npy").download())

# Create the Q&A pipeline
pipeline = RetrievalQAWithSourcesChainWithScore(
    retriever=faiss_index.as_retriever(),
    embeddings=embeddings,
    prompt_template=prompt_template
)

Creating the Chat Interface

Interactive experience: The stateful storage can enable a more interactive and dynamic chat experience, as the model can eventually be improved to generate context-aware responses based on previous interactions with the user. This is great for diving deeper or refining queries to get the answer you really want.
Flexibility: The Chat class can be easily adapted to work with various user interfaces, such as Discord and Slack applications, allowing developers to integrate the Q&A bot into different platforms seamlessly.

from langchain import Chat

# Instantiate the Chat class with the Q&A pipeline
chat = Chat(pipeline)

Model Selection and Fallback Mechanism

Service Continuity: By using GPT-4 as the primary model and GPT-3.5 Turbo as a fallback, the Q&A bot can ensure continuous operation even if the primary model is unavailable or encounters issues. This is particularly important for maintaining a consistent user experience and preventing downtime, which can negatively impact user satisfaction and trust in the system.
Performance Optimization: GPT-4 provides state-of-the-art performance in natural language understanding and generation. By default, our Q&A bot leverages this model to deliver the highest-quality responses to user queries. However, GPT-3.5 Turbo, while slightly less powerful, still offers a high level of performance. Utilizing this fallback mechanism allows the Q&A bot to maintain its effectiveness even when the primary model is not available. Speaking of which:
Resource Management: In some cases, the availability of the primary model, GPT-4, might be limited due to resource constraints or other factors. By incorporating a fallback mechanism, the Q&A bot can seamlessly switch to GPT-3.5 Turbo, ensuring that users continue to receive responses to their queries without being negatively impacted by resource limitations.
Flexibility and Scalability: The inclusion of a fallback mechanism provides the Q&A bot with the flexibility to adapt to changes in the underlying language models or infrastructure. This makes it easier to scale the system, accommodate new models or updates, and ensure that the bot remains up-to-date with the latest advancements in natural language processing.

Deploying the WandBot on Discord and Slack

Integrating with Discord

Import the required libraries: Import the discord library, along with any other necessary libraries, such as the Chat class from the Q&A bot implementation.

import discord
from qa_bot import Chat

chat = Chat(**kwargs)

async def on_message(message):
    if message.author == client.user:
        return

    response = chat.handle_message(message.content)
    await message.channel.send(response)

client = discord.Client()

@client.event
async def on_ready():
    print(f'{client.user} has connected to Discord!')

client.run('your_bot_token')

Integrating with Slack

Import the required libraries: Import the slack_bolt library and its components, along with any other necessary libraries, such as the Chat class from the Q&A bot implementation.

from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler
from qa_bot import Chat

chat = Chat(**kwargs)

@app.event("app_mention")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...
def command_handler(body, say):
    message = body['event']['text']
    response = chat.handle_message(message)
    say(response)

app = App()

handler = SocketModeHandler(app, 'your_app_token')
handler.start()

Logging and Analysis with Weights & Biases Stream Table

Model Analysis: By logging user queries and the bot's responses, the team can analyze the performance of the model, identify areas for improvement, and gain insights into how well the model is meeting users' needs. This information can be used to iteratively refine the model and its underlying components for better performance and user satisfaction.
Debugging: The W&B StreamTable serves as a central repository of user interactions, allowing developers to identify and diagnose any issues that arise during the bot's operation. This is particularly useful for understanding and resolving edge cases or unexpected behavior that may occur in real-world use.
Downstream Evaluation Tasks: The logged data can be utilized for downstream evaluation tasks such as refining the prompt, adjusting the pipeline's components, or even training new models based on specific user requirements. This continuous feedback loop enables the development team to adapt and improve the Q&A bot in response to real-world user interactions.
Monitoring and Reporting: The W&B StreamTable provides a visual and easily accessible platform for monitoring the bot's performance over time, enabling the team to track improvements, spot trends, and generate reports as needed.

Evaluation of the Q&A Bot

Load the evaluation dataset: We utilize the evaluation dataset stored as a Weights & Biases Artifact, which includes a set of questions, original responses, and document sources.
Test the chat model: For each question in the evaluation dataset, we use the chat model to generate a response along with the retrieved documents and their scores.
Calculate retrieval accuracy: We assess the retrieval accuracy by checking if the original document is present in the retrieved documents.
Calculate string similarity: We use the FuzzyWuzzy library to compute the similarity between the chatbot-generated response and the original response. This similarity score ranges from 0 to 100, with higher scores indicating greater similarity.
Grade the chatbot's responses: We use the language model to evaluate the chatbot's responses by comparing them with the original answers. The Language Model is instructed to grade the chatbot's answers as either CORRECT or INCORRECT.

Generate a large dataset of simulated user questions for each code snippet in the W&B documentation. The snippet serves as the "ideal response," providing both the question and answer.
Utilize the existing Wandbot code and algorithm to generate a response for each question.
Implement an automated evaluation using a GPT-3.5 model-based approach.
Perform human labeling on a subset of questions and model answers by our MLEs (Machine Learning Engineers). We filtered out examples where our experts assessed the question or "ideal response" to be incorrect.

Conclusion