Walkthrough: Multimodal Rube Goldberg Machine

Walkthrough: multimodal Rube Goldberg machine

This is an interactive exercise for a shared screen. You will need a chalkboard or whiteboard, and a way to photograph it and send the photo to AI. Any chat tool that accepts photos works. On a phone, the attach button opens the camera.

Together, the class will draw one Rube Goldberg machine on the board and photograph it. Then the class will ask AI to do each of the five things from the lecture with that one picture: extract its parts, interpret it to find its weak spot and review it, transform it into build instructions, simulate it, and extend it with three more steps.

The steps build on each other. Step 1 gets the drawing to a point where AI understands it. Each step after that starts a fresh conversation with the same photo and asks AI to do one more thing with it.

Step 1 · Draw the machine and check that AI understands it

Ask the class for a goal: ring a bell, turn on a lamp, drop a marble in a cup. Then build the machine on the board one step at a time, with a different student adding each step. Aim for six to eight steps, and let at least one be questionable.

The point of this step is to put as much as possible on the board rather than into the prompt. A few tips for the drawing:

• Label the parts. Write "marble," "ramp," "dominoes," "lever" next to each one.
• Draw a line or a dashed line from each part to a short note explaining what it does, such as "car rolls into cup" or "cup drops and pulls string."
• Add arrows showing the direction things move.
• Add a legend in the corner for any symbol you use more than once.
• If one part is hard to draw clearly, draw it again, larger, off to the side, and label it "detail of step 4."

Photograph the board and send the photo with only this prompt:

[attach the photo]

Here is our Rube Goldberg machine. Look at the diagram and the notes and explain what you see and how you think it works.

Read AI's explanation aloud. If it understood every step the way the class meant it, move on. If it misunderstood a step, do not explain the step in text. Improve the drawing instead: add a note, add an arrow, add a legend entry, take a clearer photo, or draw a second diagram of the tricky part. Then send the new photo with the same prompt. Repeat until AI's explanation matches the machine. This is the same loop as the persona and context lessons: when AI gets it wrong, the fix is better context.

Step 2 · Extract

Once AI seems to understand the machine, start a new conversation and send the photo with this prompt. Starting fresh matters: it shows that the drawing alone, with its notes and labels, now carries the whole machine.

[attach the photo]

List every part in this drawing and what each one connects to, in order from the start of the chain to the end.

Compare the list to the board. This is the same thing AI did with the fridge and the receipts in the lecture: it pulled out every item of one kind and put them in a useful order.

Step 3 · Interpret: what will fail

In the same conversation as Step 2, ask AI to interpret the machine against a question. Now that it has the list of parts, it can reason about each one.

Which step of this machine is most likely to fail, and why? Rank the three weakest steps.

Ask whether the class agrees with the ranking. Usually the step a student drew as a joke is on the list, and usually there is one the class did not expect. This is interpretation: AI is not describing the drawing, it is judging it.

Step 4 · Interpret: two reviewers

Still in the same conversation, give AI two roles. This is the persona lesson applied to the photo, the way the travel planner reviewed the Nashville list.

Review this machine twice. First as a physics teacher grading it for whether each step would actually transfer enough energy to the next. Then as a nine-year-old who wants it to be more fun.

Ask what each review noticed that the other did not. The physics teacher and the nine-year-old are looking at the same lines. The role decides what they see.

Step 5 · Transform

Start a new conversation and send the photo again. This time ask for something of a different kind: not a description or a review, but build instructions, the way a photo of a stool became instructions in the lecture.

[attach the photo]

Write step-by-step instructions to build this machine using only these materials: [the class lists what is actually in the room: rulers, books, tape, string, paper cups, pencils, a water bottle]. Substitute freely, but keep every step of the chain.

Read the substitutions aloud. Ask whether any of them is smarter than what was drawn.

Step 6 · Simulate

Start a new conversation and send the photo. Now ask AI to run the machine, the way it ran the maze game in the lecture.

[attach the photo]

Run this machine. Narrate what happens step by step, then show the key moments as ASCII frames. If a step would fail, stop there and say why.

If AI stops at a failing step, it has found a design problem from the drawing alone, before anything was built. Compare where it stopped with the ranking from Step 3.

Step 7 · Extend

In the same conversation as Step 6, ask AI to add to the machine. The lecture extended a half-filled calendar. Here the machine ends somewhere; ask AI to keep going.

Look at where our machine currently ends and design the next three steps to do something bold and amazing. Describe each new step, then draw the whole machine, old steps and new, in ASCII.

Read the three new steps aloud. Ask whether they connect properly to the last step on the board, and whether the class would add them to the drawing.

End by asking the class to name the five things AI did with one photograph. Then ask which of them they could do with a photo of something on their desk right now.