Hands-on Human-in-the-Loop Agent Prototyping

Participants will build an agent prompt that teaches or guides a human through a process one step at a time. The agent should decide what the human needs to do next, communicate one action clearly, receive the result, and use that result to decide what comes next.

Before testing the agent, teams will define a communication protocol. We will treat a protocol as a small language with three parts: syntax, which specifies the structure of messages; semantics, which specifies what each message means; and timing, which specifies when messages are sent and how many actions may be requested at once.

The important lesson is that communication itself must be engineered. The action request has to be understandable, but the returned observation has to be understandable as well. Both participants must follow the language closely enough for the conversation to continue.

Because both participants are intelligent in this exercise, small violations may be repaired through interpretation. Teams will therefore test two cases: one where the protocol works well and one where ambiguity or an unexpected result causes it to break down. They will revise the protocol based on the failure.

Teams will then build a multimodal version in which the human returns images as observations. We will explicitly connect this exercise to the cooking demonstration: the agent is directing action and receiving new evidence about the resulting state of the world.