Agentic AI Engineering

Lesson 9: RAG Focus

Most LLMs today are trained once on a fixed corpus, which makes their internal knowledge static. Pre-training is essentially taking a "closed-book exam" on the world's information. This leads to familiar limits; models won’t natively know about post-cutoff events or private/proprietary data, and when forced to guess, they may hallucinate [1], [2].

Why not update the weights again? Methods like Supervised fine-tuning (SFT) can help, but in practice, it's resource-heavy and slow. Teams must curate datasets, run multi-day training jobs, and manage risks like catastrophic forgetting (improving performance on one task while losing it on previously learned ones) [21] [19]. Ideally, we’d like models to adapt more like humans, updating from feedback and updating weights continuously. Reinforcement-learning approaches move in that direction, but they’re outside our scope here and remain an open research question [22], [23].

Today’s practical workaround is the context window , the model’s working memory (RAM). Instead of retraining, we supply the facts each time we call the model. But context is finite and costly: performance degrades with oversized prompts, and models are known to under-use information buried in the middle of very long inputs (“lost in the middle”) [20]. So “just put everything in the prompt” doesn’t scale. Even as context windows grow, selection beats stuffing; the practical path is to fetch only what’s needed, just in time, rather than cram everything in.

This is where Retrieval-Augmented Generation (RAG) comes in. With RAG, we store millions of tokens of external knowledge in a database and retrieve just the right snippets at runtime. We then place those snippets in context so that the model can answer accurately [3]. In other words, we turn a closed-book exam into an open-book one. In Lesson 3, we called this Context Engineering, the work of curating the information flow into the model. With RAG, our applications become more accurate, verifiable, and trustworthy because answers can be grounded in and cite their sources [3].

We’ve made a video about RAG if you’d like a quick refresh:

Since the external knowledge base lives outside the model, the natural move in an agentic system is to expose it as a tool. In Lesson 6 we learned how to build tools, and in Lesson 8 we wired them into a ReAct loop so the model can decide when to call them. Give your assistant a retrieval tool that queries an external DB, and you’ve got what the industry calls “Agentic RAG”. An agent that thinks, issues retrieval calls when it needs external information, reads the results, and uses them to ground its answers with no manual document stuffing required.

From here, a new need will naturally emerge. As agents look things up, make decisions, and interact with users, people expect them to remember what mattered: preferences, past conversations, learned procedures, and facts discovered along the way. That’s the role of a memory system. In practice, we pair the read-path of RAG with a write-path that proposes additions, updates, or deletions to an external store, also called Long-Term memory. In this lesson, we’ll focus on retrieval. In the next lesson, Memory for Agents, we’ll see what a memory system is and how they are built.

We’ll start by decomposing RAG into its fundamental components and architecture. From there, we’ll explore the techniques required to make RAG production-ready and show how it becomes a great tool inside the agentic frameworks we’ve seen.

The RAG System: Core Components

To properly design and implement a RAG system, you first need to understand its three conceptual pillars. These components work together as part of the Context Engineering process we discussed in Lesson 3 to ground an LLM's response in external facts. By breaking the system into these distinct parts, you can isolate responsibilities, troubleshoot issues more effectively, and ensure that the LLM always receives the most relevant and accurate information.

The first pillar is Retrieval. This component acts as the search engine of your RAG system. Its primary job is to take a user's query and efficiently find the most relevant information from your knowledge base [4].

The dominant technique is semantic search, which relies on vector embeddings. These are numerical representations of text where words and phrases with similar meanings are mapped to nearby points in a high-dimensional space. An embedding model processes an input, usually just text, capturing its semantic meaning and converting it into these dense vectors.

Figure 1: Different words “embedded” and projected into a new multidimensional space. Notice how words with similar meanings are closer together in the “embedding space”.Figure 1: Different words “embedded” and projected into a new multidimensional space. Notice how words with similar meanings are closer together in the “embedding space”.

We create these embeddings for all your documents during an offline ingestion process and store them in a specialized vector database. When a user query arrives, we convert it into a vector using the same embedding model. Then, we compare this query vector against the document vectors in the database to find those that are semantically closest, often using a metric such as cosine similarity [1].

The next pillar is Augmentation. Once the retrieval component finds the most relevant document chunks, we prepare this information for the LLM. Augmentation is the process of carefully formatting the retrieved text into the prompt sent to the language model. This step plays a key role in effective Context Engineering, as it involves creating a structured prompt that includes the original user query, the retrieved context, and clear instructions for the LLM on how to use that context to generate an accurate answer [4].

Finally, we have the Generation phase. We send the augmented prompt, now rich with relevant external information, to the LLM. The model's task is to synthesize this information and generate a coherent, factually grounded answer to the user's original question [1]. The quality of the final output depends heavily on the relevance of the retrieved context and the clarity of the instructions we provide in the augmented prompt.

Figure 2: The three core components of a Retrieval-Augmented Generation system.Figure 2: The three core components of a Retrieval-Augmented Generation system.

Understanding these three components is the first step. Now, let us see how they fit into the end-to-end operational pipeline of a real-world RAG system.

The RAG Pipeline: Ingestion and Retrieval

A production-ready RAG system operates in two distinct phases: an offline process for preparing data and an online process for answering queries in real-time. Separating these workflows is important for efficiency and scalability, as it allows you to handle computationally intensive data preparation independently from the low-latency demands required when the user interacts with your application [5].

The first phase is Offline Ingestion & Indexing. This is where you build your knowledge base. The process begins by loading documents from various sources, such as PDFs, websites, or databases.

Next, you split documents into smaller, manageable chunks. We do this to work around the LLM’s context window limit; very long PDFs or books won’t fit in their entirety, and just as importantly to improve retrieval precision.

Most embedding models have an effective window and a training-length “sweet spot.” If a model mostly saw ~500 tokens per example during training, it tends to capture meaning most faithfully at roughly that scale. Too small and you lose context; too large and unrelated ideas get averaged together, weakening the signal. Aim for chunks that map to a single logical unit (a paragraph, a self-contained step, a short subsection), and use a small overlap (e.g., 10–20%) to avoid cutting ideas in half. In practice, we tune size empirically: we test a few ranges (e.g., 400–1,500, sometimes up to ~2,000) on a labeled set and pick what maximizes retrieval metrics (hit rate/recall, plus a ranked metric like MRR@k or nDCG@k).

Figure 3: The end-to-end RAG pipeline, split into its offline ingestion and online inference phases.Figure 3: The end-to-end RAG pipeline, split into its offline ingestion and online inference phases.

With this end-to-end path in place, the next challenge is ensuring quality. A naive RAG pipeline often struggles with the messiness of real-world data, which is why we need advanced techniques to improve retrieval accuracy. If you’re curious about implementation details, we’ll build a minimal, from-scratch multimodal RAG and agentic RAG in Lesson 11.

Advanced RAG Techniques

While a basic RAG pipeline is a good starting point, production systems need more advanced techniques to handle diverse data and complex queries. These methods improve the quality of the retrieved context, which directly impacts the accuracy of the final answer.

Figure 4 Example of a more robust RAG pipeline.Figure 4 Example of a more robust RAG pipeline.

A helpful way to reason about improvements is to look at where they act on the pipeline. Some optimizations happen before retrieval (shaping the query, filtering the corpus), others during retrieval itself (how we search and score), and others after retrieval (re-ranking or restructuring context before generation). Thinking in these three categories makes diagnosing failures easier and deciding what to try next.

Now, let’s list some of these advanced techniques and see how they work.

Hybrid search combines traditional keyword-based search, such as BM25, with modern vector search. BM25 excels at exact terms, acronyms, codes, and jargon, while vector search captures semantic meaning and synonyms [6]. In practice, the most common way to improve retrieval breadth is to attach multiple indexes to your knowledge base so different signals can vote. A classic pairing is a search index (BM25 + field filters) alongside a vector index. But you don’t have to stop there; you can also fan out to SQL (for policy or catalog lookups), time-series queries, or any mechanism your data already supports, then fuse the results at scoring time.

For example, if a user asks about “RMD rules for 2025”, a keyword index surfaces pages that literally mention RMD (Required Minimum Distribution) and “2025”. A vector index brings in passages that say “mandatory retirement withdrawals next year” or “minimum distribution requirements” without the acronym. Similarly, a query for “GDPR Article 30 records” benefits from BM25 on “Article 30” while vectors catch “records of processing activities.” Combining both gives coverage and precision [7]

Figure 5: A hybrid retrieval flow combining keyword and vector search, followed by a re-ranking step.Figure 5: A hybrid retrieval flow combining keyword and vector search, followed by a re-ranking step.

Re-ranking

Re-ranking adds a second stage to the retrieval process. After an initial retrieval fetches a broad set of candidate documents, a more advanced model, like a cross-encoder, re-evaluates and re-orders these candidates. The cross-encoder examines the query and each document together to produce a more accurate relevance score, ensuring the most relevant documents are placed at the top [8]. This is important for the LLM's attention and improves the signal-to-noise ratio of the context [9]. Because a cross-encoder scores each (query, chunk) pair, it introduces meaningful latency and should be used when accuracy is critical or k is large. A tiny example:

Initial retrieval returns:

chunk1
chunk2
chunk3

The reranker returns:

chunk2
chunk3
chunk1

and we might choose to keep (Top 2):

chunk2
chunk3

In other words, we retrieve with a generous top-k, then keep the top-r only after re-ranking to ensure the context is small and relevant.

Query Transformations

Query transformations modify the user's input to improve retrieval. One method is decomposition , which breaks a complex question into simpler sub-questions [10]. Another method is Hypothetical Document Embeddings (HyDE). Here, an LLM first generates a hypothetical ideal answer to the query. The embedding of this hypothetical answer is then used for the search, which often retrieves more relevant documents than the original, sometimes ambiguous, query [11].

Suppose the question is, “What’s our travel policy for European conferences this year?” You can decompose to: where the policy lives, what qualifies as a “conference,” which rules apply to Europe, and what changed “this year.” Retrieve for each sub-question, to get the location of the authoritative doc, definition section, region-specific clauses, and the changelog, and then synthesize the final answer, citing the relevant excerpts.

Advanced Chunking Strategies

Advanced chunking strategies move beyond fixed-size splitting, which can awkwardly cut sentences or ideas in half. Semantic chunking splits documents based on topical shifts, keeping related sentences together. For structured documents like PDFs or tables, layout-aware chunking preserves the document's inherent structure. Another approach is context-enriched chunking , which adds explanatory context to each chunk before embedding, improving retrieval accuracy [12].

GraphRAG

GraphRAG uses knowledge graphs to answer questions about relationships that are hard to capture in isolated chunks. Conceptually, it is another index type alongside vector and keyword search: the index here is a graph of entities and typed edges, and retrieval is graph traversal plus path scoring [13]. It's the type of index that can be useful when the relationships are discovered from messy, unstructured text.

Consider the question, “Which incidents were caused by weekend deployments that also touched the login service?” In a graph, you might model:

  • Entities: Deployment, Service, Incident, ChangeTicket, CalendarDay.
  • Relationships: Deployment --affects--> Service, Incident --linked_to--> Deployment, Deployment --has_change--> ChangeTicket, Deployment --happened_on--> CalendarDay, CalendarDay --is_weekend--> true.

A traversal answers the question by filtering Deployment nodes where CalendarDay.is_weekend = true, intersecting with Deployment --affects--> Service(name="login"), and following Incident --linked_to--> Deploymentto return the matching Incidentnodes, optionally scoring by proximity or edge confidence [14].

Metadata Filtering

One of the most effective levers in production is metadata filtering. If each chunk carries fields like source, department, country, language, policy_version, or effective_date, you can filter the search space before scoring. This is especially important when documents are similar but scoped differently. The filters can be extracted using an LLM from the user query or programmatically injected from the UI or other custom business logic.

Temporal filters are particularly handy. If a user asks, “What changed between March and June 2025?”, restrict retrieval to chunks with effective_date within 2025-03-01 to 2025-06-30, then retrieve within that slice. You can also apply a ‘two-time’ logic: we can first restrict by effective_date (what was valid then), then use indexed_at (when we ingested it) to pick the freshest version within that window.

These techniques improve retrieval quality. Next, we will see how retrieval becomes one of many tools an intelligent agent can choose to use as it reasons through a problem.

Agentic RAG

As we explored in Lessons 7 and 8, the ReAct framework enables an agent to reason, act, and observe in a loop. Agentic RAG is the practical application of this concept, where RAG is not a rigid pipeline but a dynamic tool that a ReAct-style agent can choose to use. The core distinction is the shift from a linear workflow to an adaptive, iterative process controlled by an agent [15].

In practice, you expose your retrieval logic as a tool, for example, retrieve_documents() with arguments for query text, index choice, and metadata filters. This way, the agent can decide when to call it, with what parameters, and how many iterations to run. The agent can choose whether a single pass is sufficient or whether to refine the query and call the tool again [16]. It can also decide which knowledge source to consult based on the user’s goal, and whether external verification is required [17].

This agentic approach enables several advanced capabilities. The agent can iteratively use the RAG tool, refining its query based on initial results. It can also choose which knowledge base to search. For example, for an IT outage, it might query search_incident_runbooks instead of search_marketing_materials. Furthermore, the agent can fuse information from its RAG tool with outputs from other tools, like a web search or a code interpreter [18]. The agent can even decide to update its knowledge base with new information it learns, a concept related to memory that we will explore in the next lesson.

Consider this conceptual thought process for an agent:

  • Thought: The user is asking about "2024 EU data retention rules." Our internal policy cites 2023, which is likely outdated.
  • Action: retrieve(internal_policy, query="EU data retention 2024")
  • Observation: The retrieved document mentions amendments but is missing citations.
  • Thought: I need external verification.
  • Action: web_search("EU data retention 2024 official")
  • Observation: Found a new directive published by the EU.
  • Thought: I will synthesize and cite both, highlighting changes from 2023.

This transforms RAG from a simple database lookup into a conversation with a knowledgeable research assistant.

Figure 6: An agent's reasoning loop, showing its ability to choose between multiple tools, including a RAG tool for internal knowledge.Figure 6: An agent's reasoning loop, showing its ability to choose between multiple tools, including a RAG tool for internal knowledge.

You now understand both a linear RAG pipeline and how an agent can control retrieval when needed. Let us wrap up by situating RAG in the wider AI Engineering toolkit and previewing what comes next.

Conclusion

In this lesson, we covered RAG, from fundamentals to its role in advanced agentic systems. RAG mitigates core limitations like knowledge cutoffs and hallucinations by grounding models in external, verifiable data. This not only improves accuracy but also builds user trust by enabling systems to cite their sources.

For production-grade applications, moving beyond a basic implementation is necessary. Advanced techniques like hybrid search, re-ranking, and sophisticated chunking are important for achieving the quality and relevance needed in the real world. As we have seen, the future of knowledge retrieval is agentic. By treating RAG as a tool within an intelligent agent's toolkit, we move from rigid, linear pipelines to dynamic, adaptive systems that can reason about their information needs.

Mastering RAG is a foundational competency for any AI Engineer. It is an essential component of the broader discipline of Context Engineering, which is crucial to developing effective and trustworthy AI applications.

This lesson sets the stage for our next topic. In Lesson 10, we will explore Memory for Agents, examining how short-term and long-term memory systems complement the on-demand retrieval capabilities of RAG. We will also cover later in the course concepts such as retrieval quality evaluations and production monitoring, which are crucial for maintaining high-performing systems.

References

Previous

Lesson 8: ReAct Practice

Next

Lesson 10: Memory for Agents

On this page