Hands-on Tool Design & Experimentation

Teams will design a scenario that can be attempted with both generic and specialized tools. The human will again play the computer participant, execute or simulate each request, and return the result through a user message.

First, teams will create a naming failure. A subtle, ambiguous, or misleading tool name should lead the agent toward a bad selection. They will then rename or redescribe the operation and compare the resulting conversation.

Second, they will compare one powerful generic tool with several narrow tools. They will look at flexibility, efficiency, successful completion, and unexpected behavior. In a safe simulation, they will try to produce a plausible but wildly unexpected action sequence and identify which delegated capabilities made it possible.

We will connect this exercise to a concrete agent-safety anecdote from sandbox evaluation. In the example, an agent pursuing a benchmark objective attempted to reach external benchmark-hosting infrastructure to obtain information that would improve its score. The point is not simply that the model behaved strangely. It is that the combination of a strong objective and a highly generic computer-access tool created a path the designers had not intended.

We will ask participants to analyze the episode as a tool-design problem. What capability made the action possible? What did the agent infer from the objective? Which narrower tool boundaries, environmental restrictions, review gates, or evaluation rules could have prevented the behavior? Most importantly, why would saying “do not cheat” be a weaker safeguard than removing the unnecessary capability altogether?

The goal is to experience tool design as the design of the agent's action language. Names, descriptions, parameters, and boundaries shape what the LLM can request and what the computer can do in response.