Σ 0:00▸ 0:00

Module 2 · Building Agents · scripted

The Economics of the Conversation

by Jules White

20 minoutcomes: context-engineering
🎤 Talk view← Course

Overview

Every turn, the whole conversation is sent to the model again — and we pay per word, both in and out. Conversation design is therefore economic design: the conversation you launch is a bet on everything its trajectory will go on to cost. In this lesson, we see why agent steps are expensive output, find the recurrence hiding in the bill, and learn how caching makes re-reading the conversation cheap while rewriting it stays costly.

We pay per word

The conversation has a limit — and it also has a meter running. Roughly speaking, we pay per word: per word for the prompts we send to the LLM, and per word for what it sends back.

The meter

Input:the prompt going in → paid per token (≈ word)
Output:the response coming out → paid per token (≈ word)

Output tokens are typically several times more expensive than input tokens.

This makes conversation design an economic act. An agent conversation is self-modifying — the trajectory writes itself — so what we are really choosing is an initial conversation that launches a trajectory whose total cost, tokens in plus tokens out, is as small as we can make it.

But not smallness at any price. Cost is one dimension among several: we also care about accuracy, and usually about qualities of the output itself — what a reader feels, what a non-expert understands. The conversation's design is the lever for all of them at once: it decides how much goes in, how much comes out, and how good what comes out is.

What the design is balancing

costtokens in + tokens out for the prompt
accuracydoes the agent reach the right answer, reliably
qualitythe properties of the output we actually care about

We are trying to design conversations that provide high accuracy and quality at low cost.

Agent steps are output — and output costs more

For an AI agent, the words coming out are typically tool calls.

Agent steps are output

Each step, the agent acts by generating a tool call.

A tool call is OUTPUT — billed at the premium rate.

Every step the agent takes is an output purchase.

Watch a small run, and count what the model produces:

Example 1 · A four-step run
user

You have the following tools available for use:

  1. list_files( folder )
  2. read_file( path )

Task: One of last month's expense reports is missing its receipts. Find which one.

assistant
list_files(folder = "/expenses/july")
user
report-alpha.csv, report-beta.csv, receipts.txt
assistant
read_file(path = "/expenses/july/receipts.txt")
user
[a long file: fourteen receipt entries for report-alpha,
 line after line — and not a single entry for report-beta]
assistant
read_file(path = "/expenses/july/report-beta.csv")
user
[41 expense rows; the receipt column is empty in every one]
assistant

report-beta.csv is the one missing its receipts: receipts.txt has no entries for it, and its receipt column is empty on all 41 rows.

✎ noteFour steps, four pieces of model output: three tool calls and one answer. Every one of them is billed at the output rate.

Each tool call is output; each one costs money — at the premium rate. So there is a rule of thumb (not a guarantee): the more steps the agent takes to solve the problem, the more expensive the solution.

And tool design decides how big each of those outputs has to be. Watch what happens when the tools can't say a small change:

Example 2 · The edit that cost the whole file
user

You have the following tools available for use:

  1. read_file( path )
  2. write_file( path, content )

Task: In transactions.csv — about 10,000 rows — transaction TX-88213 has the wrong date. Move it forward one day.

assistant
read_file(path = "transactions.csv")
user
[all 10,000 rows of the file — TX-88213 is row 4,507, dated 2026-07-14]
assistant
write_file(path = "transactions.csv", content = "id,date,amount,merchant
TX-00001,2026-07-01,439.00,Vestland Foods
TX-00002,2026-07-01,89.50,Molde Kaffebar
[… all 10,000 rows, regenerated — identical except one date …]
TX-88213,2026-07-15,1210.00,Romsdal Logistikk
[…]")
user
OK.
✎ noteThe change was one field of one row. But write_file only speaks whole files — so the agent had to generate every row again as OUTPUT, at the premium rate, to move one date by one day.

The task was tiny; the bill was not. Output cost is a property of tool design: if the only way to say a small change is to say the whole file, you pay for the whole file — at the output rate. A tool like edit_row( id, date ) would have made the same fix a one-line output.

But the real cost story is subtler than "more steps, more calls" — and you already know the equation that tells it.

The recurrence in the bill

Remember how the conversation evolves: the next prompt is the prior conversation, plus the model's output, plus what came back —

The recurrence, now with a price on it

Cₜ₊₁ = Cₜ + toolcallₜ + resultₜ

Each step the agent takes —

INPUTpaid for all of Cₜ — the ENTIRE conversation so far
OUTPUTpaid for the tool call the model produces
and thenCₜ grows, so the next step's input bill is bigger

It looks like the conversation is simply evolving — but each new tool call is produced by feeding the entire prior prompt back in. So the input bill is not per-step constant; it grows with every step. Here is Example 1 again, unrolled as a ledger:

Example 1, unrolled as a ledger

step 1in 1,200 (task + tools) · out 40
step 2in 1,300 (step 1 + call + result) · out 35
step 3in 2,900 (step 2 + call + result) · out 45
step 4in 3,600 (step 3 + call + result) · out 30
whole run9,000 input tokens · 150 output tokens

Each step costs more than the one before, because the prompt is longer than it was. And notice step 4: it is still paying input for receipts.txt — a file the agent already read. Every result stays in the conversation and is re-billed on every step after it arrives. This is exactly why deep investigations are expensive: every observation the agent acquires joins the conversation, and the recurrence re-bills it on every step that follows.

One more thing the ledger shows: the bill cannot be computed in advance. The recurrence has no clean solution. Nothing guarantees the agent is converging on the goal at any given step; the same task may take three steps today and ten tomorrow. We can shape the economics — smaller results, tighter tools, fewer steps — but we cannot schedule them.

Caching: cheap re-reading, expensive rewriting

Modern models add one more rule to the economics: caching. You pay slightly more for input tokens the first time you introduce them — and then far less, often an order of magnitude less, every time the model re-reads the identical tokens afterward.

The caching rule

Introducing contextexpensive — first-time input pays a premium
Re-reading the conversationcheap — identical tokens re-read at a deep discount, often 10× less

For an agent, which re-feeds the whole growing conversation at every step, this is what makes long trajectories affordable: step 9 re-reads steps 1–8 at the discount.

Here is what that looks like across two successive steps of an agent. The prompt at step 5 is the prompt at step 4 plus one new thing — and we pay less for everything that matches:

Two successive prompts

The green blocks match the previous prompt exactly, so they re-read at the deep discount. Only the orange block — the one new thing — pays full price. Only the new parts of the conversation are charged at the full rate; the older parts are cheap.

But the discount has one condition, and it changes how you treat history: caching works on the matched prefix of the conversation. Edit anything early — swap the system instruction, reword a message, reorder anything — and nothing after the edit matches anymore:

The same prompt, with the history edited

The prefix no longer matches; every token repays at full price. So from an economics perspective, the rule is simple: preserve the history. Keep long prefixes stable, append rather than rewrite, and let the cache carry the weight of the growing conversation.