Σ 0:00▸ 0:00

Module 2 · Building Agents · scripted

Positioning: How It's Done Today — and the Research Opportunity for Tomorrow

60 min

Overview

Yesterday you considered how agents might change your domain and developed several possible research directions. Today you built prototype agents to explore specific problems and see what you could learn from them.

In this session, you will choose an initial research direction to work on for the rest of the week.

This is a starting point, not a final research question. The goal is not necessarily to study or build the agent itself. Instead, start with a real problem in your domain where an agent might be useful. Building and experimenting with an agent gives you something concrete to study. As you work, the more important research contribution may turn out to be a gap you discover: a missing method, framework, measurement approach, theory, design principle, safety mechanism, or other idea needed to make agents effective in this setting.

The agent and problem provide a way to ground the research. They give you something concrete to build, observe, measure, and question. Your research direction should evolve as you learn from that work.

The deliverable is a one-page Positioning Memo. This work will be the background and motivation for a possible research paper.

Choose a starting point

Write two sentences.

One: identify a problem in your domain and an agent that might help address it.

Two: identify an initial research question you want to investigate through that work.

These are starting points. You should expect to refine them as you build, test, measure, and discover what the important research problems actually are.

:::steps The session at a glance 1. Choose a problem, an agent that might apply to it, and an initial research question. 2. Map the state of practice: how is this work done in your domain today? 3. Find evidence: identify two or three real numbers that show the size or importance of the problem. 4. Identify what you can borrow, what needs to be adapted, and what appears to be missing. 5. Identify the main risks and failure modes. 6. Assemble the Positioning Memo — one page.

Part 1 · Choose

Review what you already have: the research direction and three research questions from The Day After, and what you learned from today's prototyping session.

Choose a problem in your domain where an agent might be useful. Then identify an agent you can build and experiment with to explore that problem.

You are not deciding what your final paper will be about. You are choosing something concrete enough to begin investigating.

Three tests for the starting point you choose

interestis the underlying problem important or interesting enough to investigate further?

leverageis there a reason agents might change what is possible — through scalable cognitive work, tool use, reasoning, coordination, adaptation, or another capability?

buildabilitycan you build and test a rough version in the Forge this week — using simulated data or tools where necessary?

Then identify an initial research question. This question gives you something to investigate as you build, but it can change as you discover more important gaps or questions.

Five kinds of initial question

capabilitycan an agent do this work in my domain — and where does it fail?

designwhich design choices — tools, knowledge, instructions, or other components — make the agent work better?

methodologywhat process or method produces effective agents, tools, or skills for this class of task?

safetywhat can go wrong when an agent performs this work, and what would be required to use it safely?

assessmenthow should we measure agent performance on this kind of work?

These questions are starting points. For example, you might begin by asking whether an agent can perform a task and discover that the real problem is that your field has no good way to measure success. Or you might discover that existing agent designs fail because a new methodology, framework, or representation is needed.

Those discoveries may become the research.

Part 2 · The state of practice

Next, understand how this problem is handled today. This gives you a baseline for understanding what agents might change and where important gaps may exist.

Map how it's done today

1
Who does this work today? Identify the specific roles involved — for example, an analyst, reviewer, coordinator, clinician, or researcher.
2
What methods and tools do they use? Describe the actual workflow, including spreadsheets, email, manual handoffs, specialized software, or undocumented knowledge.
3
What does the current process cost? Consider time, money, expertise, error rates, delays, backlog, or other relevant costs.
4
What is not done today because it is too expensive, slow, or difficult? Consider analyses, reviews, checks, reports, or other work that would be useful but is rarely or never performed.

Understanding the current process also helps you avoid assuming that the agent itself is the contribution. The important question may instead be what has to change — technically, methodologically, organizationally, or theoretically — for agents to work well in this setting.

Part 3 · The evidence hunt

Spend twenty minutes looking for two or three real, citable numbers that help establish the size or importance of the problem. Use an agent to help you search.

What counts as evidence

a volumehow many cases, papers, claims, incidents, requests, or other units of work pass through the current process

a costhours per task, cost per review, budget, staffing requirements, or another measure of resources required

a failure ratehow often the current process is wrong, late, inconsistent, incomplete, or skipped

a citationa report, survey, dataset, or paper from your field that provides evidence about the problem

Verify your evidence

Record the source for each number. If you find a number that you cannot verify, mark it "to verify" rather than treating it as evidence.

If you cannot find useful numbers, record what you searched for and what important measurement appears to be missing. The absence of a useful measurement may itself point to an interesting research problem.

Part 4 · Borrow, reinvent, invent

Break the problem and your proposed agent approach into their important pieces. For each piece, determine whether you can borrow an existing approach, need to reinvent or adapt one, or need to invent something new.

The triage

borrowan existing method, tool, metric, dataset, framework, or result already addresses this part of the problem and can be reused

reinventsomething similar exists, but it must be adapted, re-validated, or redesigned for your domain or problem

inventan important piece does not appear to exist and may require a new method, measurement, framework, theory, design principle, technique, or other contribution

Look carefully at the invent column. This is one place where your research direction may begin to change.

You may have started with "Can an agent do X?" but discover that answering that question requires a new way to measure X, a new method for designing agents for X, a framework for describing X, or a theory explaining when and why agents succeed at X.

That missing piece may ultimately be more important than the original agent.

If there is nothing in the invent column, consider whether there is a deeper question to investigate. If there are many things in the column, narrow the scope and identify the one that seems most important.

Part 5 · The risk register

Identify the main ways applying agents to this problem could fail or create new problems.

Write the risk register

1
List the three most plausible failure modes. Be specific about what could happen.
2
For each failure mode, decide whether it is primarily an implementation or deployment problem, or whether it exposes something we do not yet know how to solve.
3
If a failure exposes a missing method, measurement, framework, theory, or safety mechanism, consider whether that gap should become part of your research direction.

A risk can therefore become a research question. You might begin by trying to build an agent and discover that the more important question is how to determine when that agent can be trusted, how to detect a particular failure, or how to design the surrounding system so that the failure is contained.

The deliverable: the Positioning Memo

Write a one-page memo. Use prose rather than fragments where possible.

Treat the memo as a record of where your research direction stands now, not as a final statement of the project. You will continue to revise the direction as you build and learn.

The Positioning Memo

user prompt

The problem: the problem in your domain that you want to investigate The agent: an agent that might help address or explore this problem The initial research question: the question you currently want to investigate through the agent Why now: what agents make possible or change that makes this problem worth investigating How it's done today: the current practice — who does the work, how they do it, what it costs, and what is not currently done The evidence: two or three relevant numbers with sources, or an important measurement you could not find Borrow / reinvent / invent: what already exists, what needs to be adapted, and what appears to be missing Potential research contribution: a method, framework, measurement, theory, design principle, safety mechanism, or other contribution that your investigation may require Top risks: the main failure modes and whether any of them suggest additional research questions

Before we regroup

💾 Capture

Bring the Positioning Memo.

Write down:

1Complete all sections of the memo.
2Add one line to your build notebook describing how your research direction changed based on today's prototyping and investigation.
3Identify the strongest piece of evidence you found — or the most important measurement you could not find.
4Identify one thing you now think may need to be developed — a method, framework, measurement, theory, design principle, or other missing piece.

Be ready to discuss:

As you listen to other teams, look for gaps that appear across multiple domains. If several groups discover that they need the same kind of missing method, framework, or measurement, that may point to a broader research problem.

:::