An LLM Has No Memory

A pure function

There is no special memory inside an LLM. The simplest accurate picture: the LLM is a function applied to the prompt to produce an output. It takes in a prompt; it produces a response; and in doing so it changes nothing inside itself. It is what mathematicians call a pure function — no internal state, no side effects, nothing left behind that could serve as a memory of the exchange.

(You may have seen products that advertise memory. Whatever they do, it is separate machinery outside this process — we will open that box later in the course. The function itself stays pure.)

"Remembering" is reading

Then how does it remember what you said at the beginning of a conversation? It doesn't — it reads it. Its memory of your earlier words, and of its own, is built by reading the conversation and seeing who said what:

When the answer isn't in the conversation

Look again at what the model did when the information wasn't there. It said "you haven't told me anything yet." That was a good answer — but it was not the only thing that could have happened. When a conversation doesn't contain what's needed to answer, an LLM has three ways out:

Here is the uncomfortable part. All three are the same kind of act: the model continuing the document with its most probable next words. "You haven't told me anything yet" and a confidently invented answer differ in one respect only — we judge one as honest and the other as worthless. The model is not choosing between truth and fiction; it is completing a script.

So the lever is not "make the model honest." The lever is the conversation itself. If we want it to say "you haven't told me that yet" instead of inventing, we have to build a conversation in which saying so is the most probable continuation — where the context makes honesty fit the script better than invention does.

We will return to this again and again: hallucination is not a bug to be scolded out of the model — it is what happens when the conversation leaves the right answer unconstrained. Engineering the context so the truthful answer is the fitting answer is much of what prompt and agent design is about.

The journal

Here is the picture to keep. Imagine you had no memory of your life — no recollection of your history at all — but I handed you a journal of everything you had done and said. Asked "what were you thinking that day?", you would open the journal and look it up. Asked "what did you promise those people?", you would find the page and read your own words. You would trust the journal to be your memory.

That is the LLM's situation on every single turn — with one difference: the entire journal is handed to it at once, every time. Its memory of its own past actions is it reading what it said earlier in the conversation. Its memory of you is reading what you said. That reflection into the prompt is the only concept of memory it has.