Rough Proof of Concept: Trying Your Agent for Real

75 min

Overview

Yesterday you designed the study you would run; today you find out — roughly, honestly, ad hoc — whether your agent is even in the neighborhood. Take two or three cases from your own mini-benchmark and run them in the Forge with your best build. Does it solve them? How many turns? How many tokens? Run one again: same answer, or noise? Then try one variation from this morning — give it retrieval, give it memory, make it plan first — and see whether anything visibly changes. This is not the study. This is the proof-of-concept evidence that goes into Friday's skeleton with exactly the right modesty: "on a handful of real cases, the agent did this — here's the tally, here's the best excerpt, here's where it broke."

Honest and rough beats rigorous and fake

Small tallies, honestly kept, are real evidence. A handful of runs dressed up as a study is not.

Two rules: declare which cases you're running BEFORE you run them — and report every run, including the embarrassing ones. A rescue destroys the data point; let it fail and write down where.

The session at a glance

1
Pick 2–3 cases from your mini-benchmark — the ones concrete enough to run today.
2
Run each with your best build. Tally: worked? turns? tokens? excerpt.
3
Run at least one case AGAIN — is the result stable, or was it luck?
4
Try one variation from this morning — knowledge, memory, or plan-first — on the same cases.
5
Write the honest summary: what this shows, and what it doesn't.

Part 1 · Pick the cases and run them

From your Proposed Study, take the two or three cases you made concrete last night. Give each to your best build from yesterday, exactly as written, and let it run.

The tally — one row per run

workeddid it solve the case? fully / partly / no — by the "better looks like" sentence you wrote for that case

turnshow many back-and-forths it took

tokenswhat it cost — the Forge shows you; write it down

the momentone line — the smartest or dumbest thing it did

Use the output like the person it's for: would you accept this? File it? Act on it? "Technically completed the task" and "I would actually use this" are different findings — note which one you got.

Part 2 · Run it again

Pick the most interesting case and run it again, unchanged. Agents are probabilistic: one different early response and the whole trajectory diverges.

What the second run tells you

same result, similar pathsome evidence the first run wasn't luck — say "twice," not "reliably"

same result, wild paththe agent finds its way by different routes — interesting, and worth an excerpt

different resultyour single-run tallies are weather, not climate — which is a finding: the real study needs repetition, and now you know

This is the difference between "it worked" and "we have some evidence this isn't just statistical variation" — and it costs one run.

Part 3 · Try one variation

This morning covered knowledge, memory, and retrieval; you know your agent's weakest moment from yesterday. Pick one thing to try:

One dial — pick what your question needs

knowledgegive it a document or retrieval over real material — does the librarian who reads ahead visibly change the verdict?

memorylet it keep notes across cases — does remembering the last case help the next, or contaminate it?

plan firstask it to plan before acting — different trajectory, or just a longer one?

the toolsetyesterday's other build, if your hunch still points there — now on your real benchmark cases

Write one sentence predicting what will happen before you run — then run the same cases and tally the same way. A prediction on record is what turns tinkering into exploration with a spine.

Part 4 · The honest summary

Read your tally and write three short paragraphs:

What this does and doesn't show

what it showed"on these N cases, the agent ⟨did what⟩ — best moment, worst moment." Claim the tally, nothing more

what changedwhat the variation did — visibly better, visibly worse, or honestly indistinguishable on this few runs

what it doesn't showsmall N, rough measures, maybe fake data — and the specific thing the real Proposed Study would establish that this can't

The modesty is not a weakness in the paper. "Rough proof of concept plus a well-designed study" is the strongest true thing you can say after four days — and stronger than an inflated claim a reviewer punctures in one question.

Before we regroup

💾 Capture

Bring the tally.

Write down:

1The tally table — case × run × (worked / turns / tokens), with the variation runs included.
2Two or three verbatim excerpts — the best moment, the most instructive failure, and the run-two divergence if you got one.
3Your prediction from Part 3, and one line on how it fared.
4The honest summary — the three paragraphs.

Be ready to discuss:

This afternoon you assemble the paper skeleton, and this tally goes in as your preliminary explorations. Before you write the introduction, reread your summary's third paragraph — the introduction must promise only what the first paragraph holds.