Retrieval Augmented Generation

Overview

Everything the agent has fetched so far, it fetched with a tool call. Retrieval Augmented Generation does it without one: before the model runs, code searches a collection of documents and inserts the passages it judges possibly relevant into the prompt. The agent asks for nothing; the context is simply already there when it starts. That makes RAG someone else writing the opening of the conversation — and every failure in this lesson follows from that: the chunk boundary that cuts a rule from its exception, the setting that floods the prompt with near-misses, the search that always returns something even when the collection holds nothing useful.

Where this fits

RAG is context that arrives provided — the agent never asks for it, because code searched and inserted it before the model ran. That code is placing a bet on "what do I need" before the agent has a chance to ask the question itself. When the bet is right, the context is simply there; when it is wrong, the agent starts the task with someone else's guess.

Retrieval without a tool call

Up to now the agent has done its own fetching: read the manual, searched the document, asked the library. Every one of those was a tool call the agent chose to make.

RAG removes the choosing. A piece of code runs first, searches, and pastes what it found into the prompt:

Step by step:

The model did not learn the documents. It was not trained,
not extended, not connected to a database.

We changed the TEXT IT RESPONDS TO. Retrieval is prompt
construction — the one big prompt, built partly by machine.

So whoever writes the chunker, the search, and the insertion decides what the agent knows before it speaks. That is why the small choices below matter so much. Let's break it, one choice at a time.

"Use RAG" is not a method

Every arrow in the diagram hides a decision, and each decision can quietly change what the model believes. Watch four of them fail.

This is the tearing principle from earlier, now with a sharper consequence: at ingestion, a bad tear costs a strip its aboutness; at retrieval, it delivers half a rule as if it were whole. Chunking is a decision about which sentences are allowed to be separated — and rules travel with their exceptions or they lie.

That last line deserves its own alarm: retrieval always returns something. Nearest is not the same as near. The most dangerous RAG answer is the one assembled from confidently-inserted chunks that were merely the least-far-away.

How the text is introduced

One more decision, and it may matter most of all: how the inserted text is labeled when it enters the prompt. Anything that enters the conversation can instruct, and these chunks enter in the user's message. Watch the same chunk under two labels:

Inserted text arrives in the user's message, with the power
to instruct. Label it as EVIDENCE, not ORDERS:

  "possibly relevant · retrieved automatically · ignore freely"

And remember what else rides in on inserted text: anything
anyone ever wrote into the collection. A stranger's words,
delivered to the seat that steers.

Inserted for the agent, or fetched by it

So which is right — text inserted automatically before the model runs, or the agent fetching for itself with a tool call? Put them side by side:

Neither wins outright, and production systems almost always mix them: insert the plausible; let the agent fetch the rest. The prompt starts with the obvious passages, and search tools stay available for when what was inserted turns out to be wrong or thin.

Retrieval before the question is a BET.
Retrieval after the question is a PURCHASE.
Bets are cheap and sometimes wrong; purchases are exact and
cost iterations. Budget for both — and give the agent a way
to notice when the bet missed.