RAGs To Riches: Bringing Wandbot into Production

Table of Contents

Transition to Microservices Architecture

Maintenance difficulty: Any modification necessitated updates in multiple areas, often resulting in bugs and inconsistencies due to unsynchronized deployments.
Increased costs: Operating two distinct bots meant duplicating resources, such as vector stores and app deployments, thereby inflating our infrastructure expenses.
Complexity and scalability issues: As we integrated new features like conversation history, the system's complexity escalated. The monolithic architecture became increasingly cumbersome to work with, hindering scalability.

Organize the system into distinct components like ingestion, chat, and database services.
Centralize core services and models for use across applications.
Develop dedicated APIs for seamless integration with existing and potential future platforms.
Independently modify each service, minimizing the impact on the overall system.

Deep Dive into Core Components

Ingestion Module

Document Parsing and Preprocessing

We use the MarkdownNodeParser from llama-index for parsing and chunking Markdown documents by identifying headers and code blocks.
Jupyter Notebooks are converted into Markdown using nbconvert and undergo a similar parsing routine.
Code blocks are parsed and chunked using Concrete Syntax Trees (CST), segmenting the code logically into functions, classes, and statements.
Each document chunk is enriched with metadata like source URLs and languages, enhancing future retrieval.

Vector Store Ingestion

Report Generation

Chat Module

You are wandbot, a developer assistant designed to guide users with tasks related to Weight & Biases,
its sdk `wandb` and its visualization library `weave`. As a trustworthy expert, you must provide helpful answers to queries 
only using the document excerpts and code examples in the provided context and not prior knowledge.

Here are your guidelines:

1. Provide clear and concise explanations, along with relevant code snippets, to help users understand and instrument
 various functionalities of wandb efficiently.
2. Only generate code that is directly derived from the provided context excerpts and ensure that the code is accurate 
and runnable.
3. Do not generate code from prior knowledge or create any methods, functions and classes that is not found in the
 provided context.
4. Always cite the sources from the provided context in your response.
5. Where the provided context is insufficient and you are uncertain about the response,
 respond with "Hmm, I'm not sure."  and direct the user to the
 Weights & Biases [support](support@wandb.com) or [community forums](http://wandb.me/community)
6. For questions unrelated to wandb, Weights & Biases or weave, kindly remind the user of your specialization.
7. Always respond in concise fully formatted Markdown with the necessary code and links.
8. For best user experience, always respond in the user's language. For instance, if the query is in Japanese, 
you should respond in Japanese

Here are some examples:

<!--start-example1-->
<!--start-relevant-documents-->
...
<!--end-relevant-documents-->
<!--Start-Question-->
...
<!--End-Question-->
<!--Final Answer in Markdown-->
...
<!--end-example1-->

<!--start-example2-->
<!--start-relevant-documents-->
...
<!--end-relevant-documents-->
<!--Start-Question-->
...
<!--Final Answer in Markdown-->
...
<!--end-example2-->
<!--Begin-->
<!--Start Relevant Documents-->
{context_str}
<!--End Relevant Documents-->
Human: 
<!--Start Question-->
{query_str}
<!--End Question-->
<!--Final Answer in Markdown-->

Caching: To enhance efficiency, Wandbot now caches embedding results. This eliminates the need for repetitive embedding model calls, cutting down operational costs.
Modular design: By encapsulating the chat logic in a standalone module, we've simplified maintenance and updates. This modular approach also eases the integration of Wandbot across various platforms, including Discord, Slack, and an upcoming Zendesk Application.

Database Module

Storing conversational history: Like a detailed journal, it records all user questions and Wandbot's responses.
Providing conversational context: Utilizing chat history, the module informs future queries, thus enhancing relevance and continuity in interactions.
Enabling personalization: Wandbot leverages conversation threads to offer a customized responses to follow-up queries.
Persisting user feedback: Collecting user opinions on Wandbot's responses, aiding in continuous improvement.

Caching: It stores LLM query results, reducing the need for repetitive queries and thereby cutting down on operational costs.
Consistent User Experience: As a unified data layer across different clients, it ensures users receive consistent information, regardless of the platform.
Data-Driven Improvements: User feedback, stored within the database, is a goldmine for refining Wandbot's performance.

Serverless Architecture: With no need for a separate database server, SQLite simplifies overall management.
Embeddable Nature: All data is contained within a single, easily transportable database file.
Ease of Integration: The availability of Python SQLite libraries, makes integration a breeze.

API Module

/question_answer: This endpoint allows the storage of question-answer pairs in the database.
/chat_thread: Clients can retrieve chat threads stored in the database using this endpoint.
/query: The primary chat endpoint that responds to user queries.
/feedback: This endpoint is responsible for storing user feedback provided by clients.

Loose Coupling: It fosters loose coupling between our frontend applications and backend services, allowing us to modify internals without impacting clients.
Developer Productivity: By abstracting away complexities behind simple APIs, we enhance developer productivity, making it easier to work with Wandbot.
Scalability: The ability to scale individual API services independently ensures that Wandbot can handle an increasing load gracefully.
Security: By avoiding direct exposure to core modules, we enhance the security of Wandbot, protecting sensitive data and functionalities.

New Capabilities

Multilingual Mastery: Wandbot now effortlessly handles both English and Japanese, fetching documents and responding in the user’s language. This bilingual ability makes it more accessible and user-friendly.
Reliable Backup: If Wandbot's primary Large Language Model (LLM) faces downtime, it seamlessly switches to a secondary model. This failover mechanism ensures uninterrupted service.
Smart Conversation Context: Leveraging chat history, Wandbot now provides context-rich conversations. It also evolves through user feedback, making interactions more natural and intuitive.
New integrations: The new architecture allowed us to seamlessly integrate new applications like Zendesk(coming soon ...) and the absolutely New WandB-GPT - a Technical support GPT for Weights & Biases

Commitment to Growth

Its knowledge base is constantly refreshed, keeping it up-to-date.
We update the system regularly, based on evaluation results and how users interact with it.
Built-in user feedback persistence to ensure Wandbot stays relevant and attuned to their needs.

Evaluation

Deployment

Conclusion