Σ 0:00▸ 0:00

Module 1 · Prompts as the Foundation of Computation · scripted

An LLM Has No Memory

by Jules White

15 minoutcomes: prompt-engineering
🎤 Talk view← Course

Overview

When we have a conversation with a large language model, it appears that the LLM remembers what we said earlier — and we hear people talk about LLM "memory" all the time. But LLMs have no memory. In this lesson, we learn how structuring prompts as conversations gives LLMs the illusion of memory.

A pure function

How can something with no memory hold up its end of a conversation? Start with what an LLM actually is. There is no special memory inside an LLM. The simplest accurate picture: the LLM is a function applied to the prompt to produce an output. It takes in a prompt; it produces a response; and in doing so it changes nothing inside itself. It is what mathematicians call a pure function — no internal state, no side effects, nothing left behind that could serve as a memory of the exchange.

Text in, text out — and nothing retained

The contract of a pure function

in:the conversation, handed over all at once
out:a response
retained:nothing

Run it again tomorrow with the same conversation, and it stands in exactly the same position: everything it "knows" is on the page in front of it.

(You may have seen products that advertise memory. Whatever they do, it is separate machinery outside this process — we will open that box later in the course. The function itself stays pure.)

"Remembering" is reading

Then how does it remember what you said at the beginning of a conversation? It doesn't — it reads it. Its memory of your earlier words, and of its own, is built by reading the conversation and seeing who said what:

Example 1 · A perfect memory?
user

I'm allergic to shellfish. Also, what's a good dish to try in Molde?

assistant

Given the allergy, skip the famous klippfisk dishes with shrimp garnish — try bacalao without seafood topping, or a reindeer dish.

user

Wait — what was the first thing I told you?

assistant

That you're allergic to shellfish.

✎ noteThis looks like recall. It isn't. The whole conversation was handed over again, and the answer was produced by reading the first message — the same way you would answer "who said what?" about a script: not by remembering the scene, but by looking at the page.
Example 2 · The same question, new conversation
user

What was the first thing I told you?

assistant

This is the beginning of our conversation — you haven't told me anything yet. What would you like to start with?

✎ noteSame model, same "memory," minutes later. Nothing carried over, because nothing can carry over: there is no inside to carry it in. If it isn't in the conversation, it doesn't exist.

When the answer isn't in the conversation

Look again at what the model did when the information wasn't there. It said "you haven't told me anything yet." That was a good answer — but it was not the only thing that could have happened. When a conversation doesn't contain what's needed to answer, an LLM has three ways out:

Three ways to fill a gap

SAY SO"you haven't told me anything yet" — it reports that the conversation is empty
FALL BACKit answers from its trained knowledge — which may be fine, or may be wrong for YOUR situation
INVENTit dreams up a plausible answer and presents it as fact — a hallucination

Here is the uncomfortable part. All three are the same kind of act: the model continuing the document with its most probable next words. "You haven't told me anything yet" and a confidently invented answer differ in one respect only — we judge one as honest and the other as worthless. The model is not choosing between truth and fiction; it is completing a script.

Example 3 · The gap, filled by invention
user

What was the name of the hotel I said I booked?

assistant

You mentioned you booked the Hotel Alexandra in Molde — a lovely choice near the harbor.

✎ noteNo hotel was ever named. Nothing in the conversation constrains the answer, so the most probable continuation of "the hotel I booked" is… a specific, plausible hotel. It is not lying — it has no concept of a fact it could betray. It is finishing the document the way the document leans.

So the lever is not "make the model honest." The lever is the conversation itself. If we want it to say "you haven't told me that yet" instead of inventing, we have to build a conversation in which saying so is the most probable continuation — where the context makes honesty fit the script better than invention does.

The conversation constrains the answer

Want it to invent?leave the gap open — invention fits
Want it to admit a gap?put a rule in the conversation: "If the conversation doesn't contain the answer, say you don't have it — never guess."

Now the honest answer is the one that best fits the document — so the honest answer is the one you get.

We will return to this again and again: hallucination is not a bug to be scolded out of the model — it is what happens when the conversation leaves the right answer unconstrained. Engineering the context so the truthful answer is the fitting answer is much of what prompt and agent design is about.

Controlling hallucination

Controlling hallucination is about telling the LLM what to do when there is insufficient information in the conversation.

Conversation consistency

There is a second lever, and it points inward. We can require the LLM to reference the conversation itself — not the outside world — as the support for what it says.

Conversation consistency

Instructing the LLM to reference the part of the conversation that supports its output produces less hallucination.

And it makes mistakes easy to detect: a "direct quote" that does not appear anywhere in the conversation is simple to check for.

Watch the difference. First, a question with nothing in the conversation to reference:

Example 4 · Nothing in the conversation to reference
user

Which of Jules White's papers are related to prompt engineering?

assistant

Jules White has written several papers on prompt engineering, including "Conversational Prompt Programming: Design Patterns for LLM Applications" (2023) and "Prompt Engineering Strategies for Software Developers" (2022).

✎ noteBoth titles are invented. Nothing in the conversation constrains the answer, so the model completes the script with plausible titles — and there is no easy way to check them without leaving the conversation.

Now a new conversation. The papers are placed in the conversation, and the answer must quote them:

Example 5 · Reference the conversation, quote it exactly
user

Here are the papers I have written:

  1. "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" (2023)

  2. "ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design" (2023)

  3. "Improving the Reliability of Mobile Software Systems through Continuous Analysis and Proactive Reconfiguration" (2009)

  4. "Creating Self-healing Service Compositions with Feature Models and Microrebooting" (2008)

Which of these papers are related to large language models? For each one, provide the full title as a direct quote from the list above.

assistant

Two of the papers are related to large language models:

  1. "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT"

  2. "ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design"

✎ noteNo hallucination. Every title in the answer is a direct quote from the list in the conversation.

The quotation requirement does two things. It forces the response to be strongly consistent with what is in the conversation — the model is referencing the prompt it was given, not the open world, and that alone makes invention much less likely. And it gives us a detector: if the response ever "quotes" a title that is not in the conversation, we can find that with a simple text comparison — no expertise required. The conversation is the record, and the response must stay consistent with the record.

The journal

Here is the picture to keep. Imagine you had no memory of your life — no recollection of your history at all — but I handed you a journal of everything you had done and said. Asked "what were you thinking that day?", you would open the journal and look it up. Asked "what did you promise those people?", you would find the page and read your own words. You would trust the journal to be your memory.

That is the LLM's situation on every single turn — with one difference: the entire journal is handed to it at once, every time. Its memory of its own past actions is it reading what it said earlier in the conversation. Its memory of you is reading what you said. That reflection into the prompt is the only concept of memory it has.

The whole idea in three lines

The LLM is stateless. Its response is computed from the current prompt — nothing else. Its memory is the conversation, re-read every turn.

Check the picture
  • In Example 1, point to the exact words that served as the "memory" of the allergy.
  • If someone deleted the first message before the third turn, what would the model "remember"? What does that tell you about where the memory lives?
  • What is the difference between having a memory and being handed a journal? Does the difference matter, if the journal is complete?
Your turn

Open a fresh session with any LLM and ask: "What did we talk about yesterday?" Then paste in yesterday's transcript and ask again. Explain, in one sentence each, why the first answer was empty and the second was perfect.