Module 3 · Agent Tools, Knowledge, Memory & Research Design · scripted
Correct Information About the Wrong Thing
Overview
An agent can search well, find a real document, read a true number — and still answer the wrong question. In this lesson we see how a successful search can mislead, why every task must be interpreted before any search begins, how an agent can drift into an excellent investigation of the wrong problem, and when the best acquisition action is simply to ask.
The search succeeds and the answer is wrong
An agent can acquire completely correct information and still be solving the wrong problem. Watch a search that succeeds in every visible way:
You have the following tools available for use:
- search_documents( query )
- read_document( name )
Task: Find out what it costs to rent the main conference room at the Grand Hotel.
search_documents(query = "Grand Hotel conference room price")2 matches: "Grand Hotel Conference Facilities — Price List" ·
"Grand Hotel Meeting & Event Packages (current)"read_document(name = "Grand Hotel Conference Facilities — Price List")[dated March 2022] Main Conference Room: €900 per day.Found it. The main conference room at the Grand Hotel costs €900 per day.
The search worked. The document was real. €900 really was the price — three years ago. And so the answer is wrong, with nothing anywhere that looks like a failure. Notice which failure is easier to catch:
The misleading success
An agent can acquire completely correct information and still be solving the wrong problem.
A failed search is obvious — "I couldn't find the conference-room price."
A misleading successful search feels better — "I found it! The room costs €900."
Apparent success can increase the agent's confidence in a wrong answer.
The request needs interpreting first
The hotel example failed on time — the world moved on and the document did not. But a task can go wrong even earlier, before any document is touched. Consider: Find our current customer churn rate. The agent finds a board report that says "2025 Board Metrics — Customer churn: 4.2%." Perfectly accurate. But perhaps we meant churn this month rather than in 2025. Enterprise customers rather than all customers. Revenue churn rather than customer-count churn. The phrase current customer churn rate requires interpretation before acquisition even begins.
Interpretation comes first
Before any search, the agent forms an interpretation of what the task means.
The search is aimed by that interpretation — a search can only find what the interpretation asks for.
"Find our current customer churn rate" already needs interpreting — which month, which customers, which kind of churn.
From task to observation
Every observation the agent makes arrives through this pipeline. If the interpretation step goes wrong, everything downstream is aimed at the wrong target.
Correct is not the same as relevant
The examples so far separate two properties that are easy to blur together. Correct and relevant are different properties, and a piece of information can have one without the other.
Correct vs. relevant
The 2022 hotel price is perfectly correct — and almost useless.
Verification tools check correctness. Nothing in the search itself checks relevance, because relevance is a relationship between the information and the intended problem — and the intended problem lives in the user's head, not in any document.
One request, many readings
Vague requests are the normal case, not the exception. Suppose someone asks: Find out why our membership declined. A single plain-sounding request can carry several different questions inside it.
Readings of "membership declined"
Fewer new members joined.
More existing members canceled.
Renewals declined.
One membership category shrank.
The total number of members fell.
Membership revenue fell even though member count did not.
The agent implicitly chooses one of these readings. The danger is that its choice is not the intended one. Once that happens, the agent may conduct an excellent investigation of the wrong question.
Acquisition drift
Now watch that danger play out. The agent picks a reading and starts searching — and every search comes back with true, confirming results.
You have the following tools available for use:
- query_metrics( metric, period )
- think( thought )
Task: Find out why our membership declined.
think(thought = "Membership declined — so fewer people are joining.
I'll check new-member counts.")OK.query_metrics(metric = "new_members", period = "last 6 months")New-member signups down 18% versus the prior 6 months.think(thought = "Confirmed. Why are fewer people joining? Check
advertising.")OK.query_metrics(metric = "advertising_spend", period = "last 6 months")Advertising spending down 30%.query_metrics(metric = "website_traffic", period = "last 6 months")Website visits down 22%.Membership declined because reduced advertising caused fewer prospective members to visit the website and join.
Everything the agent found about advertising was correct. It simply followed the wrong branch of the problem — and each true observation made the wrong branch look more right.
Acquisition drift
The agent's story was coherent, well-evidenced, and wrong.
Everything it found was correct.
Sometimes the best move is to ask
If the interpretation is the weak point, the user can strengthen it. Suppose someone asks: Find me the best flight to Oslo. Best could mean cheapest, shortest, fewest connections, most reliable, best departure time. The agent could immediately search hundreds of flights. Or it could do this:
You have the following tools available for use:
- search_flights( destination, sort_by )
- ask_user( question )
Task: Find me the best flight to Oslo.
ask_user(question = "When you say best, should I prioritize total
travel time or price?")Price. I'm flexible on timing.search_flights(destination = "Oslo", sort_by = "price")Cheapest option: €142, one stop in Copenhagen, arrives 18:40.The cheapest flight to Oslo is €142 with one stop in Copenhagen, arriving at 18:40.
Clarification is itself an acquisition action, and the user is a source of information.
The clarification rule
Clarification is an acquisition action.
The user is another source of information.
Ask when uncertainty about the interpretation is large enough that clarifying beats proceeding on an assumption.
Do not ask about every possible ambiguity.
The fourth question
Our acquisition framework so far asked three questions: what information do I need, where can I find it, and how can I acquire it. This lesson adds a fourth. More searching does not answer the fourth question — if the interpretation is wrong, more acquisition can reinforce the error.
The four questions of acquisition
The first three are answered by searching. The fourth is not — if the interpretation is wrong, additional acquisition can reinforce the error.