Exercise: The Agent Forge

Overview

In the tools exercise, you played the computer's side of the conversation. Now you build it: your tools carry real code, written by an LLM from your description, and when the agent speaks a tool, the code runs and its answer joins the conversation. Your job moves up a level — from speaking for the world to designing the language of action, and governing when it may be spoken.

From playing the computer to building it

Back in the tools exercise, every call paused while you typed what the world said back. Now we take your hands off the keyboard: your tools carry real code — JavaScript written by an LLM from your description — and when the agent calls a tool, that code actually runs and its return value goes back into the conversation. You have just seen the agent loop; now you are going to stand inside one.

The exercise playground

This exercise needs its playground — the Agent Forge — because something has to actually execute the code your tools carry. An agent here is a system prompt plus a set of forged tools; name it and save it as you go. (If you prefer, an LLM of your choice can help with the forging half: describe the tool and ask for a JavaScript async function run(params, store) plus a JSON Schema for its parameters, then paste both into the Forge — the code box is editable. The loop half runs here.)

Two audiences, two texts

Each tool has a description and implementation notes — and they go to different readers. The description is what the agent sees: the promise, in the vocabulary of action. The notes are what the code generator sees: the engineering ticket. Keeping them separate is the whole discipline of interface versus implementation, in miniature — the agent should never need to know that reading time is computed at 200 words per minute; the generator absolutely does.

The careful eval

When the agent calls a forged tool, the Forge runs the code inside a sandboxed worker with a hard timeout — no page access, no stored keys, and a kill switch for infinite loops. The code gets two things: the params the agent chose, and a store — a plain object that survives between calls of that tool, which is how a tool can keep a counter, a fake database, or notes to itself. Whatever the function returns is translated to text and appended to the conversation, exactly as the loop lesson described.

What every Forge agent starts with

Before you forge anything, the agent already speaks four families of built-in tools. They are armed by default — but the tool set is a design surface: the BUILT-IN TOOLS panel lets you remove any of them from this agent (what changes when the agent loses fs_write?), and a removed built-in comes back through the "add from library…" picker.

The tool-smithing family closes a loop worth pausing on: the agent can call the same code generator you use, read the draft, tweak the schema or the code itself, and then propose the result — one tool or a set that works together. Nothing changes until you approve, tool by tool, with the code on screen; a freshly approved tool still asks permission every time it runs until you say otherwise. The tool set can grow mid-conversation — but only through you.

Tools that work together can be grouped into a tool set — a named grouping with a description, saved outside any one agent so every agent can enable it, and the same tool can belong to many sets. Sets can hold built-ins too, and the Forge ships starter sets as worked examples: filesystem (read-only), filesystem (read-write), visualization & tables, tool creation, attachments, and a mixed web research kit. Sets toggle per agent: enable adds a set's tools to the current agent (and arms its built-ins), disable removes them again — so "filesystem (read-only)" becomes a real constraint by disabling the read-write set and enabling the read-only one. You curate sets in the TOOL SETS panel; the agent can group its own tools with save_toolset and ask for a set with enable_toolset — which walks through the same approval gate, tool by tool.

The teaming family makes the Forge multi-agent — and who may talk to whom is part of the design. Each agent carries an AGENT TEAM: the saved agents it may start conversations with, optionally pinned to a version (2), a range (1-3, >=2), or latest — every Save of an agent bumps its version (v1, v2, …) and archives the old one. The agent fans work out to teammates — several in parallel — then await_conversations joins on all of them, or on whichever finishes first. Teammates run autonomously with their own tools plus the shared filesystem (the coordination medium); a teammate's approval-required or human-answered tool pauses and surfaces to you — edit the arguments, allow (once or for that conversation), answer, or deny. A teammate whose own saved definition has a team can delegate further down (max depth 3) — organizations of agents, each layer scoped by its own roster. The TEAM CONVERSATIONS panel shows every conversation live, with depth, version, and the full transcript.

Vector databases — search by meaning, filter by fact

The vector-database family gives the agent a miniature database system whose tables carry one vector column. Start with the primitive: embed_text turns a text into the vector that positions its meaning in the model's space — texts that mean similar things get nearby vectors, and that single fact is the entire technology. (The tool returns the dimension count and a preview, never the full 1536 numbers: vectors are for comparing, not for reading.)

On top of that primitive sits the database: create_vector_database takes a name, a description, and typed columns — text and number attributes plus exactly one vector column. db_insert hands the vector column a chunk of text and the Forge embeds it on the way in, storing the full vector next to the attributes. db_search is the payoff: give it a query and results come back ordered by similarity of meaning; give it a where filter (equality, comparisons, contains) and only matching rows are ranked — search by meaning, filter by fact, in one call. Omit the query and it's a plain table query. At this scale the search is exact brute force — every vector compared, no approximation; production systems trade that exactness for approximate indexes (HNSW, IVF) only when the vectors number in the millions.

Databases are scoped like the filesystem: shared (every agent, persists across everything — the coordination medium), agent (only its owner sees it, across that agent's runs), or instance (this run only, purged on New run). The VECTOR DATABASES panel shows every database, its schema, and every record — including which text each vector points back to.

The web, and how secrets stay secret

The web family puts the agent on the open internet through the course proxy. http_get fetches a page or API and returns it as markdown (structure kept: headings, links, lists) or plain text, chunked like attachments. http_request is the raw instrument — method, headers, body, your call — and because a raw request can spend, send, or mutate, it is the one built-in that always asks the human first, exactly like a "needs approval" forged tool, wherever it runs (a teammate's request pauses and surfaces to you too).

Authentication is its own lesson. The SECRETS panel takes named values — an API key, a token — that live only in this tab's memory: never saved, never exported, gone on reload. The agent sees the names (list_secrets) and writes a placeholder — "Authorization": "Bearer {{secret:MY_KEY}}" — and the Forge injects the real value at send time, then redacts it out of everything that comes back. The agent authenticates without ever holding the credential: the same separation production systems draw between the model and the vault.

Skills — instructions as a unit

Tools are what an agent can do; a skill is packaged knowledge of how — following the Agent Skills standard: a folder with a SKILL.md at its root (YAML frontmatter carrying name and description, body carrying the instructions) plus supporting files under paths — references/checklist.md, scripts/convert.js. Upload one as a zip, build one in the AGENT SKILLS panel, or let the agent author its own with create_skill and write_skill_file.

The standard's engine is progressive disclosure, and the Forge lets you watch it: an enabled skill contributes only its name and description to the agent's system prompt (level 1); when a task calls for it, the agent loads the full instructions with read_skill (level 2), and individual files on demand (level 3). Costs stay flat until knowledge is actually needed. A skill's JavaScript is executable: run_skill_script runs a scripts/*.js file in the same sandbox as forged tools — with the human's approval each time, since that code arrived with the skill, not through the tool gate.

Skills attach per agent: the panel's checkboxes (or enable_skill/disable_skill) decide which agents carry which skills, the attachment is saved and versioned with the agent, and a teammate spawned from a skilled agent brings its skills along. enable_skill is human-gated — attaching a skill is prompt self-modification, so the gate shows you the full SKILL.md before you approve. In one line each: a tool is an action, a tool set is a bundle of actions, a skill is the know-how for wielding them.

There is also the bare instrument under all of this: run_javascript executes arbitrary code in the sandbox and returns the result — the agent's own eval, with fetchText, fetchMarkdown, htmlToMarkdown, the filesystem, and llm() in scope, and your approval in front of every run.

Lifecycle hooks — agents on the loop itself

The loop has joints, and you can put agents in them. The LIFECYCLE HOOKS panel assigns any saved agent (version-pinnable, like a teammate) to a point in the bench agent's loop. At that point the hook agent receives the event as an ordinary message — its own system prompt stays fully in charge — and runs like a teammate, with its own tools and the shared filesystem; its conversation appears in TEAM CONVERSATIONS with a ⚡ badge. To modify the loop it must call a directive tool — an explicit, named action offered only at that event, and only if the hook's assignment allows it. This is the interceptor idea: the main agent never knows the hook ran; it just experiences a world that has been edited around it.

What a hook may change is part of its assignment. Each hook row has a may: line — untick a directive tool and the hook simply isn't offered it (calling it anyway earns a refusal). Untick them all and the hook is observe-only: it can still read, compute, and write files — it just cannot touch the loop. The purest hook is exactly that: a recorder that watches and files things away, modifying nothing.

The rules that keep this honest: hooks fire in declaration order, each seeing the previous one's edits; a hook run gets the hook agent's own tools and the shared filesystem but no teaming and no tool-smithing (hooks never cascade, never fire hooks of their own); its approval-required tools still pause and surface to you; a hook that errors or runs out of rounds changes nothing — and every firing lands in the HOOK ACTIVITY log with its work tallied, directives or not. And tool.before runs before your approval gate, so what you approve is what will actually run.

You have already built this pattern by hand: the Memory Lab's recall agent is a turn.start hook (read the memory files, append what's relevant), and its extraction agent is a turn.end hook (write what just happened to the filesystem). Rebuild it here from parts you own: save a memory-recall agent (hook it to turn.start, allowed append_context) and a memory-record agent (hook it to turn.end, observe-only — its whole job is fs_write), and your bench agent "remembers" — while the panel shows you exactly which agent did the remembering, and what it cost. And the vector twin: let the recorder db_insert what happened into a shared vector database, and the recaller db_search it by the meaning of the incoming message — retrieval by similarity instead of by file name, at the same two joints of the loop.

There is also a shortcut on your side of the bench: the add from library… picker drops in a ready-made tool (a web-page reader, a calculator, note memory, CSV statistics, a human-answered expert), or enables a whole tool set, and everything it adds is editable like anything you forged.

Files on the bench

You can also hand the agent real material: the 📎 button attaches a file — CSV, PDF, Word, or any text file — and its text is extracted in your browser (nothing is uploaded anywhere). The agent is told only the name and size; the attachment built-ins let it look inside, and your forged tools can compute over the contents directly. Attach a photo and it goes to the agent's eyes instead: the image rides into your next message as a multimodal prompt, for the model to look at directly.

Step 1 · Forge one tool well

Create a single tool and take it around the full cycle. Pick something honestly computable in a browser — reading statistics, a unit or currency-table converter, a date calculator, a dice roller with odd dice, a grade-curve calculator from your own field. Generate it, then test it before any agent touches it: run it with easy values, edge values, and at least one input you expect to break it. Every run lands in the test history. Then improve it without editing the code yourself: write one line of feedback, tick the failing tests as evidence, and regenerate.

Step 2 · Run the loop in step mode

Give the agent two or three forged tools and a task that needs them, and run in step-through mode: every call waits for your Allow — with the arguments editable before it runs — and every result pauses for your review before the agent sees it, editable in transit. Both edits are strange powers worth feeling once. Watch the loop you studied on paper happen in front of you: call, execute, translate, append, decide. Between step-through and autonomous sits run to breakpoints: mark any tool — forged or built-in — with a 🔴 and the loop runs free until it hits one, then gives you the full stop-edit-continue treatment. In autonomous mode only "needs approval" tools pause, and once allowed, results flow straight back. And whatever the mode, ⏹ Stop is the kill switch: it terminates the loop wherever it is — the model call is aborted on the wire, a tool mid-execution has its result and side effects discarded, pending calls are visibly cancelled, teammates and hooks stop — and the bench stays open for you to continue or start over.

Step 3 · Govern, then let go

Now design the policy. For each tool, decide: does this need a human, every time? Tick needs approval on the ones that do — anything that writes, spends, sends, or deletes deserves the checkbox; pure reads and pure computation usually don't. Then switch to autonomous mode and run a fresh task: unchecked tools fire without you, checked ones still stop and ask — and the Allow for this conversation option lets you lift the gate for the rest of the run once trust is earned. This checkbox is your first governance decision as an agent designer, and it is exactly the decision production teams make tool by tool.