Σ 0:00▸ 0:00

Module 2 · Building Agents · scripted

Agent Design, Prototyping, and Informal Benchmarking Session

90 min

Overview

This afternoon you build agents — but the agents are not the point. This is a laboratory session about your research domain: you will define four problems from it, choose the most interesting, and use quick, throwaway agents to probe them. The instrument is the Agent Forge — the workbench from this morning, where your tools carry real code and the agent actually runs. You are not trying to build something that works. You are trying to find out what building it teaches you about the problem — where it is harder than it looks, easier than it looks, or different than it looks. Your proof of concept is not code: it is a set of conversations, and the evidence you bring back will be an excerpt from one. This exploration session repeats every afternoon this week with new instruments; today's discoveries become candidates for your research project.

Work in groups of 2–3 where research domains align; solo is fine where they don't.

The contract

The deliverable is NOT the agent. The deliverable is what building the agent taught you about the problem.

A broken agent that taught you something beats a working agent that taught you nothing. Do not polish. Do not optimize. Explore.

The exercise at a glance

1
Define four problems from your research domain — one per value category, each clear and specific.
2
Choose the 2–3 you most want to probe.
3
In the Agent Forge, build the quickest agent that could possibly touch each problem.
4
Push each agent: change one thing at a time, and watch what breaks, opens, or surprises.
5
Capture your best discovery as one slide — and bring all four problem statements with you.

Part 1 · Define four problems

Before you touch the Forge, write four problem statements from your own research domain. Not four tasks you'd like automated — four problems, each primarily made of a different kind of limit:

The four categories — what is the problem primarily made of?

efficiencythe thing is well-defined and gets done — but slowly or expensively. Remove the speed limit and the problem disappears

capacitythe method exists and works — but cannot reach the volume that matters. Remove the scale limit and the problem disappears

possibilitythe thing does not exist. Nobody does it, at any speed or price. There is no slow version of it

qualitythe thing gets done, at scale — but with errors, inconsistency, or shallowness. Throughput is fine; trust is not

A problem statement passes when…

It names a CONCRETE activity or outcome in your domain — a nameable thing, not an area.

It states the limiting fact — and that limit IS its category. The "primarily" test: if the category's constraint vanished, would the problem essentially disappear? If it reads equally well under two categories, keep cutting until one dominates.

It says what BETTER would look like, in one checkable sentence.

The specificity test: if two different agents could both "solve" it and you couldn't say which did better — it isn't specific yet.

The same territory cuts four ways — and good problem statements are detailed. Here are two worked domains, four problems each. The first is deliberately ordinary — a family kitchen — because the categories have nothing to do with technology:

A family kitchen, four problems

efficiencyour family plans meals for the week and buys food — then life shows up. Practice runs late, a meeting eats an evening, ingredients get used or spoil — and the plan no longer fits what is in the fridge, what everyone wants, or the time we actually have. The plan was fine on Sunday; the week changed underneath it. Re-planning every day against everything that changed is well-defined work nobody has time for — so we do it badly, and the waste is the bill

capacityreal planning works for dinners — seven meals a week. Extending it to all twenty-one meals plus school lunches, snacks, and prep-ahead is beyond anyone's evening: the method (plan, shop, prep) is fine, the volume is the wall

qualitywe cook every night, but the meals drift toward the same eight dishes, and how well a dinner lands — too spicy for one kid, not filling enough after practice — never systematically shapes the next week's plan. Output is fine; consistency of "everyone actually enjoyed it" is the problem

possibilitywhat we want is a personal chef who knows every family member's dietary restrictions and preferences, remembers who liked what at the dinner table last week, watches this week's sales at our local grocery store and what is in season in our region right now, keeps surprising us with new and interesting dishes while mixing in the favorites, reads our calendar to know how much cooking time each evening actually allows, plans the leftovers so the whole week just works — and REPLANS on the fly, every time the week shifts, fusing information no person has time to gather each day. That person cannot be hired at any reasonable price — and no amount of money puts one in every family's kitchen. Not slow — nonexistent

A research lab, four problems

efficiencyassembling the weekly lab-meeting summary — collating every member's updates, figures, and blockers into one document — takes two hours of the PI's week. The work is fully defined; it is just slow

capacitythe PI can give deep, line-by-line feedback on about three student manuscripts a month; the lab produces ten drafts a month plus revisions. The feedback method is fine — the volume is the wall

qualitystatistical review of student analyses is inconsistent: the same class of error — pseudo-replication, wrong error bars — gets caught in some drafts and slips through in others, depending on who reviewed, when, and under what deadline. Throughput is fine; consistency is not

possibilitywhat the lab wants is a dedicated scientific scout for every member: something that continuously watches new papers, preprints, talks, and datasets across the six subfields the lab touches, relates each one to the lab's active hypotheses, running experiments, and past results, flags which ongoing experiment might be scooped or contradicted this week, and briefs each student differently based on their project and what they already read. No lab could ever staff this. Not slow — nonexistent

Notice the shape of the possibility statements: they are the longest, and that is not an accident. "A personal chef" is a wish; the fully specified version — feedback from last week's table, this week's sales, this evening's calendar — is a design. Impossible things need the most specification, because specificity is what makes them explorable.

Capture — end of Part 1

💾 Capture

Write down all four problem statements, labeled by category — including the ones you will not explore today.

Part 2 · Into the Forge

Choose the 2–3 problems you most want to probe, and open the Agent Forge — the same workbench as this morning: you describe tools, an LLM writes their code, and when your agent speaks a tool, the code actually runs and its result joins the conversation.

Build quickly — for each chosen problem

1
Open the Forge: /exercises/agent-forge — or scan the QR code on the next card.
2
Give the agent the smallest possible version of the problem as its task.
Not the whole problem — one representative case of it. If your problem is checking hundreds of citations, the task is checking three.
3
Declare 2–4 tools — no more.
Describe each in one or two sentences and let the Forge write the code. Where the real tool would need data you don't have, make the tool return realistic fake data — a fake result teaches almost as much as a real one, and costs nothing. And for a tool too hard to build even in the Forge, simulate it and its response: make it a tool where you, the human, type in the result.
4
Run it.
Do not fix what breaks — read it. Every failure is data about the problem: write one line about what actually went wrong before touching anything.
5
Timebox: 15 minutes per problem, then move on.
You are drilling test holes, not building the well.

Scan to open the Agent Forge — ai-agents-seminar.vercel.app/exercises/agent-forge

Part 3 · Push it

Now run the informal benchmark this session is named for. For each agent, change one thing at a time, re-run the same task, and compare the two trajectories — same problem, one variable, two conversations side by side:

Measure the impact of one dimension

Measure how manipulating ONE dimension of the agent's design impacts its operation.

Choose a dimension. Define a measure for it. Then, keeping everything else the same, manipulate that dimension and observe its impact.

Then step back across everything you tried, and write one line per answer while it's fresh:

The questions to answer

Which workedwhich proof-of-concept agents actually worked — and why?

Which didn'twhat were the limiting factors and the gaps in capability?

Code, or researchwas the gap just code to be written and access to a system — or reasoning, or a hint at a research gap worth investigating in your domain?

What surprised youthe thing you did not predict

Part 4 · Capture your best opportunity

Choose the most interesting thing you discovered — not the most polished agent, the most interesting finding — and compress it into one slide.

The slide — five fields

user prompt

Problem: what problem were you exploring? Agent idea: what did you build or try? Discovery: what you learned that you did not know before Opportunity: the new capability, research question, or design challenge this revealed Evidence: one concrete excerpt from a conversation in your experiment

A worked example

Problemresearchers cannot keep up with hundreds of new papers relevant to a narrow research question

Agent ideaan agent that searches a set of papers, extracts claims, compares findings, and identifies contradictions

Discoverythe difficult part was not summarization — it was deciding which claims were actually comparable

Opportunityhow should an agent represent and compare scientific claims across heterogeneous studies?

Evidencetwo papers appeared contradictory until the agent was given explicit criteria for population, outcome, and experimental setting

Read that example again and notice what makes it good: the discovery is a finding about the problem, and the opportunity is a research question. That is the shape to aim for.

Before we regroup

💾 Capture

Bring your slide and your four problem statements to the group discussion.

Write down:

1All four problem statements from Part 1, labeled by category.
2Your one slide — the five fields, filled in.
3The evidence: the actual conversation excerpt, copied verbatim from the Forge.
4One line per agent you built: the single most useful thing its failure or success told you about the problem.

Be ready to discuss:

For one problem you did NOT explore today: which dimension of the agent's design do you predict would matter most, and why? Be ready to defend the prediction; later this week you can test it.