Module 1 · Foundations of Agentic AI · scripted
Multimodal Prompts: Images as Context
Prompts don't have to be text
Every prompt so far has been text. But LLMs can take non-text inputs too — other modalities — and when a prompt mixes text with another modality, we call it a multimodal prompt. We will focus on the one that matters most for agents: images as input. A prompt can be text, text plus an image, or just an image.
The reason this matters follows from everything we know: the LLM is driven by context — information about the situation, what to do, how to reason. And an image, as people say, is often worth a thousand words.
Situational context in one snapshot
Think back to the cooking agent. Describing a real kitchen in text — every ingredient, every container, every tool — is a laborious process. Or:
What can I actually make tonight? Here's my kitchen:
📷 the refrigerator shelf
📷 the pantry
📷 the kitchen drawers
Plenty to work with. From the fridge: hummus, olives, peas, cucumber, pesto, a big bowl of pineapple, and assorted leftovers in those containers — and that half lemon is already drying out, so use it tonight. The pantry has oils, vinegars, and dry goods in the canisters; your drawers show pots, steel bowls, and full cutlery. I'd do a warm grain bowl: rice from the pantry, peas and cucumber, hummus and pesto as the dressing base, lemon over the top — pineapple after. Want it step by step?
Seeing what the human cannot
Here is the deeper power, and it comes from a true story. A house's air conditioning stopped working one evening. The owner went to the basement and took one photo of the unit. The model noticed something the owner had not: a small indicator light, on, on a switch wired into the unit — and knew that this light meant the condensate pump had likely failed, because a float valve turns it on when water builds up in the drip pan. The owner didn't know what that light was. They didn't know it existed.
Example 2 recreates that moment with a different mechanical room — and a different tell. Look at the photo before reading the reply: what would you have mentioned?
Our cooling stopped working this evening. Here's the mechanical room in the basement:
📷 basement mechanical room — insulated pipes, copper lines, a
grey electrical panel, a white wall-mounted unit
One detail in your photo stands out: the drain pipe in the foreground carries a yellow CONDENSATE label. Cooling systems produce condensate water, and they usually have a safety float switch that shuts the system down if that water backs up — typically because the condensate pump has failed or the drain is clogged. Before calling anyone: find where that condensate line drains, check for standing water in the collection pan, and see if the pump hums when the system tries to start. That's the most likely reason for the shutdown.
This is the crucial asymmetry. When a human translates a situation into text, the human is the filter: only what they notice, and know to be important, makes it into the description. A photo skips the filter. It carries information the human didn't know was information — and lets the agent bring knowledge to bear that the human doesn't have. That is exactly what you want from agents whose expertise goes beyond the person using them.
- In Example 2, list what the human knew, what the photo knew, and what the model knew. Where did the solution actually come from?
- What is the text-only version of this conversation? How many turns does it take, and does it ever succeed?
The world, without the translation step
The same property matters for automation. Agents acting in the physical world — with no human present at all — need to know the state of that world, and one of the easiest ways to capture it is a camera. Normally we translate the world into text, and the text becomes what the model sees. An image lets us skip the translation: the world's state enters the conversation directly, captured by a device instead of narrated by a person, with no human in the loop to filter it.
Images as interaction
Images aren't only context — they are also a natural way to express things to an agent that are painful to say in words:
Here's our workshop whiteboard from this afternoon. Turn it into an organized project backlog.
📷 whiteboard with two marker-written columns of project ideas,
several colors of ink, one item circled in red
Reading your board: two streams — "Wikipedia ideas" on the left (translating key Ohio articles to Spanish, a CultureCrawl worklist, template improvements, library partnerships) and "Wikidata ideas" on the right (a completeness check for Henri Matisse's paintings, accuracy sweeps of the Columbus museums, maps of Ohio, OSU site items). The red ink looks like a second pass — additions and owners. And one item is circled for emphasis: the OSM + Wikidata name suggestion. I'd structure it as two workstreams with that circled item as the flagged priority. Want owners and rough sizes next?
Yes — and the circled one first. Scope it.
The circle told me where to look: a tool suggesting names for OpenStreetMap features from Wikidata. Scope: match OSM objects to Wikidata entities, pull multilingual labels, propose names for unnamed features, human review before any edit. Small pilot: one city, one week.
One image, three critics
The board didn't change. What you ask the agent to be changes everything it gives you back:
Same whiteboard. Act as a skeptic: try to poke holes in this plan. How does it fail in ways we haven't thought of?
📷 the same workshop whiteboard
Holes I can poke, from the board itself:
- "Translate key Ohio articles to Spanish" — translations rot. Who re-syncs them when the English articles change? A one-time push leaves stale pages that look authoritative.
- "Check all paintings by Henri Matisse for completeness" assumes a canonical list of Matisse paintings exists to check against. It doesn't — catalogs disagree. Completeness against what?
- Your circled priority — OSM + Wikidata name suggestion — writes names into a live map other communities maintain. Automated naming without local consent is exactly the kind of edit OSM has revolted against before.
- Most red-ink owners are one name per item. Every workstream here has a bus factor of one.
Same whiteboard again. What are the gaps and ambiguities in what's written here?
Ambiguities, reading closely:
- "MXN" next to the circled item — initials? A currency? Nothing on the board resolves it.
- "Merrike (OCLC)" — a person at OCLC, or a system? The red pass added it without saying.
- "Interesting maps of Ohio" — interesting is doing all the work in that sentence, and it's undefined.
- The pencil box at bottom right (crowdsourcing research between Wikidata, Wikiversity, Wikipedia) sits outside both columns. In scope or parked?
- No dates anywhere on the board. Two hashtags, zero deadlines.
One more pass. What are the critical questions we should be asking about this plan?
The questions I'd put to the room:
- Who owns the items with no name next to them — and is one owner per item enough for any of this to survive the month after the conference?
- What does "done" look like for the Matisse completeness check and the Columbus museum sweep — a number, a list, a report?
- Which items need community consent before anyone edits — the OSM naming tool, the template changes?
- Which single item, if it shipped, would make this board a success? Is it the one you circled?
We sketched this on paper. All resistors are the same value R. What's the equivalent resistance between A and B?
📷 hand-drawn resistor network on paper, terminals labeled A and
B
Reading your sketch: from A, one resistor leads into a diamond of five — that's a bridge configuration, so it won't reduce by simple series-parallel rules; the bridging resistor couples the two paths. Then the network continues through a second bridge-like cluster and a final parallel pair before B. To solve it we apply a delta–wye transformation to the first diamond, collapse the result, and repeat for the second cluster. Shall I work the algebra symbolically in R?
Whiteboard sketches, pointing at the broken part, circling the region of a chart, a hand-drawn diagram of the system you want — visual language is fast, precise about where, and effortless for the human. Multimodal prompts make it part of the conversation.
Do one of each in your own domain: (1) give an agent a situational photo — your equipment, workspace, or setup — and ask what it notices that you didn't mention; (2) sketch a system or process on paper, photograph it, and ask the agent to formalize it. Where did the image beat any text you could have written?
Photos via Wikimedia Commons: USDA/FSIS refrigerator (public domain); EFTA pantry and basement utility room (public domain); "Integrated kitchen with lower cabinets open" (CC BY-SA 3.0); WikiConference North America 2018 hackathon whiteboard (CC BY-SA 4.0); circuit sketch (CC0).
