The Economics of the Conversation

We pay per word

The wall has edges — and it also has a meter running. Roughly speaking, we pay per word: for every word we send into the model, and for every word that comes out of it.

This turns conversation design into an economic act. Since an agent conversation is self-modifying — the trajectory writes itself — what we are really choosing is an initial conversation that launches a trajectory whose total cost, tokens in plus tokens out, is as small as we can make it.

But not smallness at any price. We are simultaneously balancing accuracy and correctness, and often other qualitative properties we care about — if the agent generates marketing copy, we care what readers feel; if it summarizes for a safety review, we care what a non-expert understands. It is a multidimensional optimization problem, and the design of the conversation is the lever: it influences how much we pay going in, how much comes out, and how good what comes out is.

Agent steps are output — and output costs more

For an AI agent, the words coming out are typically tool calls. Each tool call is output; each one costs money. And because output is priced above input, there is a rule of thumb (not a guarantee): multiple tool calls cost more than one, and the more steps the agent takes to solve the problem, the more expensive the solution.

But the real cost story is subtler than "more steps, more calls" — and you already know the equation that tells it.

The recurrence in the bill

Remember how the conversation evolves: the next prompt is the prior conversation, plus the model's output, plus what came back —

It looks like the conversation is simply evolving — but each new tool call is produced by feeding the entire prior prompt back in. So the input bill is not per-step constant; it grows with every step:

Each step costs more than the one before, because the prompt is longer than it was. (The exception is when the window was already full from the start — then we face a different problem, deciding what to remove each step, which is its own topic later.)

And here is why you cannot just compute the bill in advance: the recurrence has no clean solution. At any given step there is no guarantee the agent is converging on the goal. The same task may take three steps today and ten tomorrow; every run's trajectory is different. We can shape the economics — smaller results, tighter tools, fewer steps — but we cannot schedule them.

Caching: cheap re-reading, expensive rewriting

Modern models add one more rule to the economics, and it matters enormously for agents: caching. The idea: you pay slightly more for input tokens the first time you introduce them — and then far less, often an order of magnitude less, every time the model re-reads the identical tokens afterward.

Think of it as: introducing context is expensive; re-reading the existing conversation is cheap. For an agent — which re-feeds the whole growing conversation at every step — this is the difference that makes long trajectories affordable: step 9 re-reads steps 1 through 8 at the discount.

But the discount has one condition, and it changes how you treat history: caching works on the unchanged prefix of the conversation. The moment you edit history, the cache is invalid from the edit onward:

Edit the beginning — swap the system instruction, reword an early message, reorder anything — and there is no common prefix left; the entire conversation re-prices at full rate. You can do all kinds of surgery on a conversation's history and make it look however you want. But from an economics perspective, the rule is simple: preserve the history. Keep long prefixes stable, append rather than rewrite, and let the cache carry the weight of the growing conversation.