Σ 0:00▸ 0:00

Multimodal AI Context · scripted

Practice

What students will do

Working in pairs, students take or draw pictures and ask AI to do each of the five things from the lecture with them. Each station matches one of the lecture's examples, so students already know what a good result looks like. By the end, every pair has had AI extract information from a picture, interpret a picture against a question, transform a picture into something new, simulate a drawing, and extend a picture that is unfinished.

Notes for teachers

Students need a device that can take or upload a photo and a chat tool that accepts one. Remind them not to photograph people or anything with personal information, such as names, addresses, or card numbers on receipts.

Set this up as stations or as a sequence. Keep the order. Each station is more surprising than the one before it. Students may use a photo they already have on their phone, such as their own fridge or a receipt, as long as it has no personal information in it.

Station 1 · Extract

In the lecture, a photo of a fridge became an inventory and a photo of receipts became an expense table.

Turn a picture into a table

  1. 1
    Photograph something with many items in it.
    The inside of a fridge or pantry, a shelf, a toolbox, the contents of a bag. Ask: "Create an inventory of what you can see so I can avoid buying things I already have." Ask for it as a table, organized by where each item is.
  2. 2
    Photograph two or three receipts together. Cover any card numbers first.
    Ask for a table of date, store, category, and total, with totals by category. Tell AI to say which numbers it cannot read rather than guessing.
  3. 3
    Count what it got right, what it missed, and what it said it could not read.
Discuss
  • Did AI list anything that was not there? Did it miss anything that was?
  • Did it say what it could not see or read? Why does that matter for an inventory or an expense table?

Station 2 · Interpret: find time and use a machine

In the lecture, AI found three hours of study time in a calendar screenshot and explained a washing machine from a photo of its panel.

Read a picture against a question

  1. 1
    Screenshot or photograph a week of your calendar, or draw one on paper with your real commitments.
    Ask: "I need [an amount of time] this week for [something]. When do you recommend?" Check whether the times AI picks are actually free.
  2. 2
    Photograph something with controls and no instructions.
    A microwave panel, a thermostat, a printer, a bike's gear shifters, a coffee maker. Ask: "How do I use this to [do the thing]?" Check the steps against the actual buttons.
Discuss
  • Which calendar entries did AI have to read and reason about to find the free time?
  • Did AI mention any button or control that is not in your photo? Did it say when something was out of frame?

Station 3 · Interpret: review a plan

In the lecture, a travel planner persona found the problems in a handwritten Nashville trip list.

Get a plan reviewed

  1. 1
    Write a plan by hand on a sheet of paper and photograph it.
    A day trip, a weekend, a party, a project timeline, a training week. Give it times. Put at least one real problem in it on purpose, such as two things far apart scheduled back to back.
  2. 2
    Send the photo with: "Act as a [travel planner / event planner / coach] and identify any gaps or problems with what I am proposing."
  3. 3
    Write down which problems AI found, whether it found the one you planted, and whether it found any you did not plant.
Discuss
  • Which of AI's points came from the picture, and which came from what it already knew about the places or activities?
  • Did AI tell you to check anything it was not sure about? Should it have?

Station 4 · Transform

In the lecture, a photo of ingredients became a recipe and a photo of a stool became build instructions.

The transformation

  1. 1
    Photograph a few ingredients together and ask for a recipe with an ingredients list and numbered steps. Check that every ingredient in the recipe is either in the photo or stated as an assumption.
  2. 2
    Photograph an object and ask for instructions to build it from an absurd set of materials you choose. Read the answer for whether the engineering is real, even if the materials are not.
Discuss
  • Which transformation surprised you most? What did AI have to understand about what it saw to do it?

Station 5 · Simulate

In the lecture, a maze game drawn on a whiteboard became a game AI could run.

Make the drawing go

  1. 1
    Draw a small game on paper or a whiteboard: a maze, a board game, a simple level.
    Write the rules beside it, with a legend for the symbols. Put everything on the drawing, not in the prompt.
  2. 2
    Send the photo with: "Here is a picture of my game. I am ready to play." Play three or four moves.
  3. 3
    Check each move against your rules. If AI got a rule wrong, fix the drawing, not the prompt, and try again.
Discuss
  • Did AI state any assumptions about rules you did not write down? Were they reasonable?
  • What did you have to add to the drawing to make AI play it correctly?

Station 6 · Extend

In the lecture, a half-filled calendar became a full week, with AI inferring the patterns behind the entries that were already there.

Finish what is started

  1. 1
    Screenshot or draw a week's calendar with only some things filled in, but with patterns in it, such as the same activity on two days or nothing early on weekends.
  2. 2
    List four to six things you still need to schedule, with how long each takes. Ask AI to schedule them "based on the preferences you can see in what I already have."
  3. 3
    Write down which patterns AI inferred and whether each one was right.
Wrap up
  • List the things you asked AI to do with pictures today. Which one would you actually use this week?
  • In every case, what did AI use from the picture that you would not have typed?