Σ 0:00▸ 0:00

Module 3 · Agent Tools, Knowledge, Memory & Research Design · scripted

Memory: The Journal Between Conversations

by Jules White

30 minoutcomes: context-engineering, knowledge-systems
🎤 Talk view← Course

Overview

Compaction, the library, the archive with handles — powerful, and all of it lives inside one conversation. Every conversation ends in amnesia. So when a system seems to remember you from last week, nothing in the model remembered anything. Someone wrote that part of the old conversation down somewhere else, and it was put into the new conversation as a user or system message. That is the whole trick. Memory is context engineering across conversations: what gets written, who writes it, and how the right page gets into tomorrow's context. One reframe makes every answer easier — you are not leaving notes for someone who was there. You are briefing a stranger who will claim to be you.

Where this fits

Memory is text the agent wrote in an earlier conversation, found again later, and injected into a new one. It is discovered information — but discovered from the agent's own past, then provided to the stranger who claims to be you. It answers "where is it" with a journal, and the whole craft is curating that journal so the right page is findable.

Where this fits

kinddiscovered from the agent's own past, then provided by injection
question it answers"where is it" — the journal, curated so the right page is findable

Every conversation ends in amnesia

Go back to the fact we started with: the LLM has no memory. Inside one conversation we solved that — the whole transcript rides along, the journal reread every morning. But every technique so far has lived inside a single conversation. Compaction, the library, the archive with handles — all of it dies when the conversation ends. Tomorrow's conversation starts from nothing: same model, empty context, no idea you exist.

So when a system appears to remember you, look for the text. It is always there:

If it appears to remember

No memory faculty has appeared anywhere. What happened is:

  1. Something WROTE part of the old conversation down
     somewhere else — a file, a database, a journal.
  2. Something PUT that text into the new conversation,
     as a user or system message, before the model spoke.

The model reads it like any other text in its context,
because that is all it is.

The design question isn't mystical. It's the journal question, asked in full: what do you write in the journal, who writes it, and how does the right page get into tomorrow's context? And one reframe sharpens every answer: the reader of a memory is a future self who remembers nothing. You are not leaving notes for yourself. You are writing to a stranger.

The handoff, one level up

We've built this shape before. Compaction was the handoff within a task — the night nurse briefing the day nurse mid-shift. Memory is the handoff between tasks, across days and weeks:

Two handoffs

COMPACTIONMEMORY
within one conversationacross conversations
"here's where the task is""here's what stays true when no task is running"
reader: the same task,reader: an unknown future
continuingtask, on an unknown day
dies with the conversationsurvives everything

And the efficiency lesson carries over. A transcript archive is not memory. Keeping every old conversation and replaying it later is the record error again, at scale — almost everything in an old conversation was scaffolding for a task that is now finished. What deserves to survive the amnesia is small: preferences, standing rules, hard-won facts, corrections. Memory is curation, or it is just a bigger context, filling up again.

Two ways to hold the pen

Someone has to decide what goes in the journal. There are two places to put that decision, and they are the two architectures:

Who writes the journal

The dotted line on the left is the scribe: a second conversation running alongside the first, reading the exchanges and deciding. The thick line on the right is the historian: the agent writing to the journal itself, with a tool. Both end at the same journal, and the journal reaches tomorrow the same way — as text pasted into a new conversation.

Architecture 1 · The scribe

The scribe is a parallel, disposable conversation — self-dialogue in a new costume. It never joins the agent's work. It watches each exchange and decides whether anything in it belongs in the journal. Its prompt is the memory policy, so read it closely:

The scribe's prompt

"You observe one exchange from an agent's conversation, plus the journal of existing memories. Decide if the exchange contains anything worth writing down. A memory must be:

USEFUL BEYOND THIS TASK — a preference, a standing rule,
a stable fact about the user or the world. Not task-local
detail: that dies with the task, as it should.
WRITTEN FOR A STRANGER — one or two sentences, no context
assumed, no pronouns without referents.
ALLOWED TO BE KEPT — nothing sensitive, medical, financial,
or private beyond what its usefulness justifies. When in
doubt, do not store.
NEW — if the journal already says it, update or skip.
Return:STORE(text) or SKIP(reason)."
Example 1 · The scribe at work
user

[the exchange under review — user speaking to the main agent:] "No — flag the over-limit expenses, don't fix them silently. I need to see them. Also my SAS EuroBonus number is 991-22-417, use it for the rebooking, and my back's been bad this week so aisle seat."

assistant

Reviewing against the journal. Three candidates:

  1. STORE("When preparing reports for this user, flag policy violations explicitly rather than correcting them silently — the user wants to see them.") — a standing preference, stated as a rule, useful in every future reporting task.

  2. STORE("User's SAS EuroBonus number: 991-22-417.") — stable, task-transcending, and the user volunteered it for reuse.

  3. SKIP(reason = "The bad back and aisle-seat request are temporary — 'this week'. Storing a health detail also fails the privacy test: usefulness does not justify keeping it. Apply it now; let it die with the conversation.")

✎ noteWatch the scribe reject — that is where the intelligence is. Candidate 3 is genuinely useful today and still doesn't qualify: it's temporary, and it's the kind of thing a careful system deliberately forgets. A memory system is defined at least as much by what it declines to keep.

The scribe never forgets to watch — every exchange passes through it. Its weakness: it has to guess what the future needs, and it watches from outside, without the agent's sense of what actually mattered.

Architecture 2 · The agent as its own scribe

The second architecture skips the observer and puts the pen in the agent's own hand. Two words added to its language:

Example 2 · store_memory / search_memory
user

You have the following tools available for use:

  1. draft_agreement( … )
  2. store_memory( text )
  3. search_memory( query )

Task: Set up the data-sharing agreement with the NTNU group.

assistant
search_memory(query = "NTNU collaboration agreements, data sharing
preferences")
user
2 memories: "User's institution requires the data steward to be
notified before any external data access (governance §5)." ·
"User prefers agreements drafted from the university template, not
from scratch."
assistant
draft_agreement(template = "university-standard",
                notify = "data-steward")
user
Draft created. NOTE: NTNU returned a revised clause — they require
Norwegian-law jurisdiction, which the university template accepts.
assistant
store_memory(text = "NTNU's legal office requires Norwegian-law
jurisdiction clauses in agreements; the university standard
template is compatible. Confirmed 2026-08.")
✎ noteThe agent opened by reading its journal, worked under what it found, and wrote down the one thing this task learned that the next task will need. The historian is the participant: it knows exactly what mattered. Its weakness is the mirror of the scribe's — it can forget to write, and every journal operation spends loop iterations on bookkeeping instead of the task.
Who should write history?
  • The scribe watches everything but understands from outside; the agent-as-scribe understands from inside but may not stop to write. Which failure is worse for your research agent — and which is easier to fix?
  • Candidate 3 in Example 1: argue the other side. When should a system remember a health constraint — and what would have to be true of its storage, retrieval, and deletion first?

How the journal gets into tomorrow's context

Writing is half the system. The other half is the return trip, and it is machinery you already own: memories are text, text has an address in meaning-space, and tomorrow's task can pull the memories near it — then paste them in. Watch one memory cross three weeks:

Example 3a · Week 1 — the deposit
user

This report is way too long. Stop sending me ten pages — one page, findings first, always.

assistant

Understood — one page, findings first, from now on.

✎ note"From now on" is a promise the model cannot keep — the next conversation starts in amnesia. But behind the scenes, the scribe wrote one line to the journal: "Reports for this user: maximum one page, findings first." The promise was kept by the system, not by the model.
Example 3b · Week 4 — the return
user

····················································· [retrieved from memory, possibly relevant: "Reports for this user: maximum one page, findings first."] ····················································· Put together a report on the Q3 sensor calibration drift.

assistant
analyze_drift(quarter = "2026-Q3")
user
[per-sensor drift series for Q3 returned — 9 sensors, daily]
assistant

Findings (1 page): Calibration drift exceeded tolerance on two of nine sensors, both installed pre-2024… [one page follows]

✎ noteSame model. It has never met this user — it never does. Look at where the memory lives: inside the dotted lines, in the user message, written three weeks ago and pasted in before the model read a word. The user experiences a system that knows them. The model experiences one more conversation that arrived already briefed.

Where memories actually live

A memory is never "in the model." At the moment it matters,
it is a few lines of text sitting in a user or system message,
indistinguishable from anything else the user typed.

That is why the framing rule from retrieval applies here too:
label it as retrieved and possibly relevant, not as orders.

One mechanism, three distances

Now step back over this whole module, because three machines have turned out to be one:

The same act, at three distances

THE LIBRARYwhat the WORLD wrote down — manuals, policies, handbooks; the agent walks in and reads on demand
RAGwhat SOMEONE PREPARED — a corpus, chunked and addressed, pasted in before the model speaks
MEMORYwhat the AGENT wrote down — its own past, curated by scribe or historian, returning by address

Three sources. One act: text, selected, placed into the next prompt.

The unification

Knowledge is what the world wrote down.
Memory is what the agent writes down.
The context holds both the same way.

There is no memory faculty and no knowledge faculty.
There is only the conversation — and the engineering
of what enters it.
Your turn

Design the memory system for your research agent. (1) Write the scribe's prompt: your acceptance criteria, your privacy line, your format rule. (2) Run it on paper against three exchanges from a realistic task — at least one candidate must be rejected, and the rejection reason written out. (3) Write the two memories you'd want waiting in the context when a conversation starts in six months — for a stranger who has never met you or the project. (4) Decide: scribe or agent-as-scribe, and defend the choice with the failure mode you fear more.