RAG techniques: From naive to advanced

Imagine you’re demoing your company’s new AI chatbot to a potential client. You ask it about their latest product, the one they’ve been working on for months, and what does it return? Information from two years ago about a product they don’t even sell anymore. Frustrating, right?

This is a good example of what retrieval augmented generation (RAG) prevents. It gives LLMs access to content not included in their training data, either because it could not access it, did not access it, or if it was created after the LLMs training date.

In this article we’re going to explore retrieval augmented generation, and some common RAG techniques.

Let’s jump in and understand the core concepts of modern RAG.

What is retrieval augmented generation (RAG)?

Retrieval augmented generation (RAG) is a method that combines information retrieval and language generation to produce more accurate and context-aware responses. It uses a retriever to fetch relevant data from a knowledge base, which a generator then uses to create informed outputs.

The retriever is responsible for finding relevant information from the knowledge base, which is typically stored in a vector database for efficient similarity search. Think of vector database as a special kind of database that helps you find similar semantically matching items very quickly. The generator, usually a large language model (LLM), then uses this retrieved information to produce informed and contextually relevant responses.

The basic workflow of a RAG system is straightforward yet powerful. When a query is received, the system first converts it into a vector representation. This query vector is then used to search the vector database for similar content. The most relevant information is retrieved and fed into the generator along with the original query. For example, if asked about recent climate change policies, a RAG system might retrieve the latest environmental reports and use them to generate an up-to-date, factual response.

Source: link

While the basic workflow of retrieval augmented generation provides a robust foundation for combining retrieval and generation, the real-world applications of RAG are far more nuanced and diverse. As the technology has matured, experts have discovered that a one-size-fits-all approach doesn’t suffice for the myriad of challenges presented across different industries and data types. This evolution brings us to an important question:

Isn’t RAG a single technique? Why do we need multiple RAG techniques?

RAG as a subject has evolved and people have developed numerous techniques to boost the performance of the traditional retrieval strategies. RAG has evolved into a diverse family of techniques, each tailored to address specific challenges or optimize for particular use cases. The one-size-fits-all approach of early systems quickly revealed its limitations when faced with the vast diversity of real-world applications. Different industries, varying data types, and unique retrieval challenges all demanded more specialized solutions.

The result is a RAG ecosystem that’s as diverse as it is powerful. Some RAG techniques focus on improving retrieval accuracy, others on reducing computational costs, and still others on handling multi-modal data like images or audio. This toolkit approach allows developers to mix and match RAG techniques based on their specific needs, enhancing accuracy, relevance, and efficiency across a wide range of applications.

Let’s first look at the most basic form of retrieval augmented generation, naive RAG.

Naive RAG

Naive RAG is the simplest form of retrieval augmented generation, involving indexing, retrieval, and generation. When a query arrives, similar data chunks are retrieved and combined to generate a response. However, naive RAG can face challenges with retrieval accuracy and response quality.

This is the simplest form of RAG. This was the methodology which gained popularity shortly after ChatGPT was revealed to the world. This vanilla RAG follows a typical process that includes indexing, retrieval, and generation.

Indexing: The initial phase involves processing and preparing the data. Generally the embedding models and language models have a certain context limitation. In simple words, they can only understand a certain amount of text at a time. If the text is too long, these models might not be as accurate or they might not be able to process the text at all. To align with the context limitations of language models, the text is divided into smaller, manageable segments called as chunks. These chunks are then transformed into vector representations using an embedding model and stored in a vector database for efficient similarity searches in the subsequent retrieval stage.

Retrieval: When a user query is received, it is converted it into a vector representation using the same embedding model used to encode the chunks. Next, the model calculates similarity scores between this query vector and the vectors of the indexed chunks. It then identifies and retrieve the top K chunks that exhibit the highest similarity to the query, which are incorporated as expanded context in the prompt.

Generation: This process involves synthesizing the original query and the retrieved documents into a cohesive prompt for the large language model to process. The model’s response strategy may vary based on task-specific requirements, allowing it to either leverage its inherent knowledge or confine its answers to the information provided in the retrieved documents. Additionally, one can adjust the tone and format of how the LLM should respond, depending on the query and context and the specific use-case being served.

However, naive RAG can have several drawbacks:

• Retrieval challenges: Getting relevant documents is challenging for a simple similarity search and it will often struggle with precision and recall. This might cause the passing of irrelevant chunks or context to LLM.
• Generation problems: Sometimes the LLM, even if provided with relevant context, can suffer from hallucinations, toxicity, biases in outputs which can lead to a bad user experience.
• Complexity limitations: For more complex use-case, a single retrieval based on the original query may not be sufficient to gather the required contextual information for the LLM. Complex problems often demand multiple retrieval steps or more sophisticated query processing to provide adequate context for accurate responses.

Advanced RAG

Advanced RAG enhances basic RAG by improving how the most relevant context is retrieved for the LLM to generate better answers. The focus is on ensuring the most pertinent data is fetched for queries across various scenarios. Since RAG starts with data processing, it’s important to discuss this in detail.

Any RAG pipeline starts with the data and processing it. So, let’s start there.

Chunking

Chunking in retrieval augmented generation (RAG) is the process of splitting large texts into smaller, manageable pieces called chunks. This is essential because embedding models and language models have context length limitations. Effective chunking preserves semantic meaning, enabling accurate retrieval and response generation.

Most embedding models used to process the documents have a fixed context length. Even large language models, despite their impressive capabilities, have limitations on how much context they can handle at once. The challenge, then, is to chunk your data without losing its semantic meaning.

Typically, most of the popular encoder models used for creating embeddings can handle about 512 tokens at a time. So, we need to choose a chunking strategy that gives enough context for our language models to work with, but still allows the embedding model to create meaningful representations. Ideally, you want to keep semantically related pieces together. But what’s “semantically related” really depends on your data and use-case.

Unfortunately, there’s no one-size-fits-all chunking strategy. It really depends on the data you’re dealing with. For example, if you’re working with well-formatted data that has a clear structure (like markdown documents), you might chunk based on markdown headers. There are various splitting techniques available in frameworks like LangChain and LlamaIndex. While we can’t dive into all of them here, some popular ones include:

• RecursiveCharacterTextSplitter: This is often recommended as a starting point. It splits text based on a list of user-defined characters, trying to keep related pieces of text together.
• HTMLHeaderTextSplitter and MarkdownHeaderTextSplitter: These split text based on HTML or markdown-specific characters, and include information about where each chunk came from.
• SemanticChunker: This first splits on sentences, then combines ones next to each other if they’re semantically similar enough.

One thing to consider is to keep some overlap text between chunks. This can help ensure that semantic context doesn’t get lost between chunks. There are many more text splitters out there, and I’d recommend exploring them to find what works best for your use-case. If you’re feeling overwhelmed, the RecursiveCharacterTextSplitter is often a good place to start.

Indexing

After chunking, we move on to indexing. This is where we embed (convert to a vector representation) our chunks using an embedding encoder model. Modern embedding models like bge-base are specifically optimized for search tasks, which is exactly what we need in RAG.

If you’re wondering which model to use, the MTEB leaderboard is a great place to start. It can help you find a model that strikes the right balance between latency and accuracy for your specific needs. But don’t just blindly trust the leaderboard, it is always a good practice to test some of these models on your data and see which one works best for your use case.

It’s worth noting that for a small number of documents, something as simple as a numpy array can work just fine as a vector database. The main point of using a vector database is to create an index to allow for Approximate Search, so you don’t have to compute too many cosine similarities. For many use-cases, datasets with around some thousand vectors don’t even require index creation. If you can live with up to 100ms latency, skipping index creation can simplify your workflow while still guaranteeing 100% recall.

But what if you’re dealing with millions or billions of vectors? That’s when building an ANN (Approximate Nearest Neighbor) index starts to make sense. An ANN or vector index is a special data structure designed to efficiently organize and search vector data based on their similarity. It narrows down the search space, so you don’t have to scan the entire vector space. While it’s faster than exhaustive search (like kNN or flat search), it’s worth noting that it’s slightly less accurate.

If you want to dive deeper into how ANN indexes work, there are some great resources out there (link). But for now, the key takeaway is this: indexing is crucial for efficient searching in RAG, but the complexity of your indexing solution should match the scale of your data.

Once you have our vector database ready, you can start querying it to start getting relevant documents.

We’re all set, right ? Not even close! We are just getting started. There’s a whole world of techniques we can use to supercharge our RAG system and take its performance to the next level. In the upcoming sections, we’ll explore a variety of strategies that can dramatically improve our retrieval and generation process. Let’s go….

Metadata filtering

In a retrieval augmented generation (RAG) system, metadata is supplementary information attached to data chunks or documents, like titles, authors, timestamps, or tags. This metadata enhances retrieval accuracy by providing additional context, helping the system match queries with the most relevant information.

By attaching relevant metadata to each chunk of information in your knowledge base, you’re essentially creating additional layers of context. These layers can be leveraged during retrieval, acting as a set of filters that narrow down the search space before you even start the semantic search process. In other words, you only perform the semantic search within the documents that match your query criteria.

Let’s look at some practical applications:

• Temporal filtering: If your chunks are associated with dates, you can easily filter results based on recency or specific time periods. This is super useful for time-sensitive information like news articles or scientific papers. For example, if a user asks about “recent advancements in AI,” you could trigger a filter for chunks dated within the last year, ensuring the most up-to-date information is prioritized.
• Jurisdictional segmentation: This is particularly powerful in legal applications. If a user inquires about “property laws in California,” your system can immediately filter for chunks tagged with the California jurisdiction, dramatically reducing the search space and improving relevance.
• Domain-specific categorization: For multifaceted knowledge bases, you might tag chunks with domain categories. In a medical database, for instance, chunks could be tagged with specialties like “cardiology,” “neurology,” or “pediatrics.” A query about “heart disease treatments” would then prioritize cardiology-tagged chunks.

Source: link

Let’s walk through a concrete example to illustrate this:

Imagine you’re doing RAG on your company’s financial documents, and you have a legal department that handles all legal matters. A user asks, “Can you get the legal division financial report for Q2 2024?”

There are many ways semantic search could fail here. The model needs to accurately understand and represent “financial report,” “legal division,” “Q2,” and “2024” all in a single vector representation. It might focus on “legal division” and retrieve irrelevant financial reports from other years, leaving your generator (an LLM in our case) to figure out which documents are actually relevant to Q2 2024.

To mitigate this, we can use metadata filtering, assuming you’ve already stored dates and relevant metadata with your documents and vectors.

You have a couple of options here:

• Use an LLM call to extract entities like Q2 2024 and legal division from the user query.
• Use zero-shot entity detection models like GliNER. For example, given below are the Gliner extracted entities for our example query.

The choice depends on your specific use case and what works best for your data.

Once you’ve extracted these entities, you can use them to pre-filter your documents or vector database. This ensures that you only perform your vector search on documents whose metadata matches your filters. Most of the vector databases today support these filtering features. For example, you can take a look at how LanceDB (a popular multi-modal vector database) does it here.

BM25: The old king still reigns

In the world of RAG, we often get excited about fancy embedding techniques. But let’s not forget about the good old full-text search – it’s still the king in many scenarios. And when we talk about full-text search, BM25 is the algorithm that stands out.

Why should we care about BM25 when we have cool embedding techniques? Well, here’s the thing: while embeddings are indeed cool, they have their limitations. It’s very hard to encode all the textual information in a mere 768-dimensional vector (768 is just a approximate figure here, many embedding models have varying embedding vector size). It’s a lossy compression.

Another challenge with embeddings is that the training data used for pre-training these embedding models will never fully represent the specific data in your company. Sure, you can fine-tune your embedding model on your data to improve performance, but that’s a lot of extra steps. You need to prepare the data, fine-tune your model, validate it, and then finally use it. It’s not exactly a walk in the park.

This is where keyword search or full-text search comes in. It might seem basic at first glance compared to vector search, but it’s still incredibly powerful. And at the heart of many full-text search systems is BM25, an algorithm powered by tf-idf (term frequency-inverse document frequency).

In fact, there’s a running joke in the information retrieval community that progress in the field has been slow because BM25 is such a strong baseline that it’s hard to beat. It’s particularly powerful for long documents and those that contain a lot of domain-specific jargon, like medical and legal acronyms. Embedding models might fail to encode the acronym “IRA” (it’s an acronym in legal domain for Individual Retirement Account) in your embedding model, but keyword search can search it without any problems.

To capture all the strengths and mitigate the pitfalls of embedding search, it’s a good idea to include keyword search in your pipeline. The current best practice is to use both keyword and vector-based search, retrieve documents using both methods, and then combine the results to get the final context. This is also referred as hybrid retrieval or hybrid search in many places.