Exercise: The Archive

The goal: make self-training cheap

"Read the Manual" made a claim: an agent can train itself at the moment of need — if the archive is shaped for a reader who pays per look. The table of contents sharpened it: one small routing file beats a pile the agent has to dig through. This exercise attaches numbers to both claims, and puts two design levers in your hands:

Open the simulator: /exercises/archive. You build a fake documentation tree — invent paths and contents freely, or load a template — pick a task, and press run. No approval gates: the agent works alone, and the lab meters every look it takes. Each run lands in a table so designs can be compared on the same question.

Experiment 1 — organization

The three templates hold the same facts in three shapes: one giant file (everything.md), the filing cabinet (small files behind a README.md index), and the attic (the same files scattered under unhelpful names, no index). Run The powdered sample against each and compare the input-token column. The facts never changed; only the architecture did — the difference is what the shape itself costs. Then look at how the agent moved: did it read the index first, or gamble on search_files, or swallow the whole handbook?

Experiment 2 — instructions

Open the agent's instructions and edit them. The default is deliberately generic. Try standing orders like:

• "Always read README.md or an index first if one exists."
• "Search before you list; never read a file over 2,000 chars without first checking whether a smaller one answers."
• "Prefer the most recently dated document when sources disagree."

Re-run the attic with your instructions. The question this experiment asks: can instructions rescue a badly organized archive — and what do those instructions cost on a well-organized one? Runs record which instructions they used, so the table shows default vs custom side by side.

Experiment 3 — tasks that want different shapes

Not every question rewards the same organization:

• The visiting professor (wifi) is a point lookup — search plus good file names win; an index is barely used.
• The first week is a synthesis task — the answer lives in four files. An index earns its keep; the attic makes the agent crawl.
• Old vs new springs the trap: every template contains a superseded 2019 handbook with wrong values. Whether the agent notices depends on labeling — the cabinet says "SUPERSEDED" at the top; the attic says nothing.

Run all three against your favorite shape. If you had to pick one organization for a mixed stream of questions, which survives best?

Design your own

Pick a domain you actually know — your group's onboarding notes, an instrument's documentation, a project wiki — and rebuild it as a tree with synthetic contents only (invent every fact; an LLM you already use can generate plausible fake documents). Write tasks a newcomer would really ask, add answer checks if you like, and iterate on both levers:

• Add a table of contents. What did it cost, and what did it save?
• Rename files from the searcher's point of view.
• Split the big file. Where does splitting stop paying?
• Tune the instructions until a fresh agent bootstraps in two calls.
• Ask something the archive cannot answer — does the agent say so, or improvise?

One run proves nothing; models are probabilistic. Run each comparison a few times before you trust it — and bring your best before-and-after numbers to the discussion.