Instructing AI by Example

Two ways to teach

So far, almost every program we have written has been a set of instructions. Double the number. Turn these notes into a flash card. Organize the observations into a table. We say what to do, and AI does it. This is the most natural way to tell anyone what we want, and it takes us a long way.

But we can also teach AI what to do without giving it instructions. In an earlier lesson we saw that when we work with ChatGPT, Claude, or Gemini, we are working with a large language model, or LLM. Large language models are experts at recognizing patterns. Showing an LLM what to do, instead of writing instructions for what to do, turns out to be extraordinarily powerful. That second way, showing it what to do, is teaching by example. Teaching by example is what this lesson is about.

Teaching by example is also known as in-context learning. The LLM learns how to perform the task from the context in the prompt, the examples we show it, rather than from any change to the model itself. Nothing is retrained. The examples are in the conversation, and the LLM uses them the same way it uses any other context. You may also see the term few-shot prompting, which means a prompt that includes a few examples, or a "few shots," of the task before asking for a new one. A prompt with no examples at all is sometimes called zero-shot. These are all names for the same idea: the examples in the prompt teach the LLM what to do.

How would you teach someone to write like your favorite author?

Suppose that is the assignment. The first way we might reach for is to write down the rules.

What would we actually do in the real world? We would show someone a short story by Hemingway. We would have them read it, think about how Hemingway writes, and then try to mimic the style. That might be an assignment you are given: write in the style of Hemingway. Nobody hands you a thousand rules to read first. They have you read some of his writing, because humans are good at recognizing patterns.

We can teach an LLM the same way. We show it examples of what we want it to do and let it figure out, on its own, how to do that. We never write the rules down. For example, we can show it a sample of how we want it to write and then ask it to write in the same style. We can show it a sample angry email from a customer and how we think the reply should be worded, and it learns how to respond to the next angry email. We can show it pictures of things and the names we use for them, and it learns to identify and name those things.

We are going to explore how to show an LLM examples of what we want and how to correctly perform the work. Sometimes we show it correctly completed work. Sometimes we show it incorrectly completed work and tell it that this is wrong. An LLM can learn from correct examples, and it can learn from incorrect ones.

Provide examples rather than rules

Start with a task we can teach both ways, so we can compare them. The task is to reverse a sentence character by character. The first conversation teaches it by instruction.

The second conversation teaches the same task by example. There is no instruction anywhere in it. There are only pairs of input and output, and then a new input with the output left blank.

Both conversations accomplish the same thing. The first is teaching by instruction: we stated the rule, and the LLM applied it. The second is teaching by example: we never stated the rule. The LLM looked at two pairs, inferred the pattern, and completed the third pair. In this case the two ways take about the same effort, because the rule is easy to write down. Next we look at a case where the rule is not easy to write down.

A classifier, taught in one prompt

Classification means putting something into a category. You can think of it as categorization: we take an item and give it a label, and the label is the category it belongs to. Sorting email into “urgent” and “later” is classification. So is deciding whether a review is positive or negative.

Now imagine we want an LLM to learn what music we like. Writing down the rules for how we decide what music we like is hard. We may not know the rules ourselves. But we can go through a list of songs we have heard, give each one a label, and let the LLM learn from the examples. The labels are love, like, okay, don’t like, and can’t stand. (Every song and artist below is invented.)

Nobody explained the labels. Nobody wrote a rule such as “fast songs with a big chorus are loved, and slow instrumental songs are not.” Seven labeled examples were enough for the LLM to see the pattern and apply it to a song it had never seen. It answered with the label and nothing else, because every example ended with the label and nothing else. If we wanted the classifier to be more accurate, we would add more examples, not more rules.

Examples can teach a boundary

Sometimes the examples we have are not enough. They do not make it clear where the boundary is: the line between what belongs in a category and what does not. When every example sits comfortably inside the category, the LLM has never seen the edge, so it has to guess where the edge is. The fix is one more example that sits on the edge and shows which side it belongs on.

Suppose we want an LLM to pull a shopping list out of text messages from home. The first conversation gives it two examples. In both of them, everything mentioned in the message goes into the list.

The LLM did exactly what the examples showed. In every example, every item after “grab” or “out of” went into the list, so it put every item after “pick up” into the list, including a person. We knew the list was for groceries. The examples never showed that, because none of them contained anything that was not a grocery. The boundary was never taught.

The second conversation adds one example. The new example contains something that should be left out, and the list leaves it out. Then it sends the same message again.

One example did the work. It showed the LLM a message with something in it that is not a grocery, and a list that leaves that thing out. The LLM inferred the boundary from that one example and applied it to the message about the sister. We could have written the boundary as a rule, such as “only include things you can buy at a store.” But we did not have to, and often we do not notice a boundary until the LLM crosses it. When that happens, the repair is the same every time: add one example that shows the correct side.

A gallery of little machines

Once you see the pattern, it is hard to stop building these. Each of the following would have been a serious project a few years ago. All three are taught the same way: examples first, then a new case.

Three prompts. The first decodes meaning that lives under the words. Five examples taught it, and the LLM decoded a phrase that was not in the examples. The second sorts a new excuse into a group. Seven examples taught the groups, and the LLM put the new excuse in the right one without anyone defining what “calendar confusion” means. The third spots something slightly off. Two normal nights taught what normal looks like, and the LLM found the one reading that did not fit and explained why. Notice that the odd reading is not wildly wrong. It is off by less than a degree, and the LLM found it because the examples showed how tight “normal” is.

Before large language models, doing what we just did in these examples required complex AI skills. You had to know how to train deep learning models, which people studied for years, and training one took huge amounts of computing power and time. Now you can build and do many of the same things, often at better accuracy than the old techniques could reach, in a single prompt. When someone talks about training an AI model, ask yourself whether it is something you could teach an LLM to do through examples. You might be able to build it for them in one message.

Think step by step

So far the examples taught answers. The next idea teaches a process for reasoning through a problem. Consider a person: if you have them write down their plan first, and the plan is right, the answer that follows is much more likely to be right. The same holds for an LLM. An LLM predicts the next words from the words that came before. If the LLM writes down a step-by-step plan to solve the problem first, and the plan is correct, it is much more likely to output the correct solution as the next words it predicts.

Forced to jump straight to the answer, the LLM had to land on the number in one move, and it got the answer wrong. Now the only change is one sentence at the end of the prompt.

The steps are not decoration. Each correct step makes the next correct words more likely. By the time the LLM writes the answer, the answer is simply the next thing that follows a correct plan.

This technique is called chain of thought. A chain of thought is the sequence of reasoning steps the LLM writes down on the way to an answer. Because an LLM produces each word based on the words before it, the chain of thought becomes part of the context for the answer. When the chain is right, the answer that follows it is usually right. When the LLM is forced to skip the chain and jump straight to the answer, it has to get there in one guess, and on anything with more than one step, the guess is often wrong.

"Think step by step" is the simplest way to ask for a chain of thought. It leaves the LLM to decide what the steps are. That works for many problems. For problems where we care about how the reasoning is done, we can go further and show the chain of thought we want, which is the next idea.

Teaching the reasoning for solving a type of problem by example

Now put the two ideas together. “Think step by step” asks for some reasoning, and the LLM decides what the steps are. We can go further and show a worked example of the reasoning we want, and the LLM will reason our way.

Compare this with the bus. There, the LLM invented its own steps. Here it walked our four steps, in order, applying our tests, including the same skepticism about what was measured and for whom. “Think step by step” asks for a chain of reasoning. A worked example designs it.

Teaching a voice

Now the case where rules fail almost completely. Try to write rules for how you write. Every style guide sounds the same, and none of them captures a voice. Samples do.

Short sentences. A concrete scene first, the lesson second. A turn in the last line. None of that was stated. All of it was shown, and the LLM absorbed it from three passages. Some knowledge is like this: you can recognize it and demonstrate it, but you cannot fully write it down. Examples are the only language it has.

Refining by pointing

So far the examples came first and the result came after. Now reverse it, because this becomes the simplest way to improve a prompt you already have. Instead of explaining what was wrong, point at it.

The important part is step 3. Every preferred and not-preferred sentence is copied word for word from the response the LLM just produced. We do not invent them.

All four quoted sentences came from the first response. Two of them were plain and specific, so we marked them preferred. Two of them were the kind of sentence that could be on any flyer for any business, so we marked them not preferred. The LLM kept the two preferred sentences almost word for word, dropped the two not-preferred ones, and wrote the rest to match the preferred ones. “Avoid buzzwords” is itself vague. “Never sentences like these” is exact, because the LLM can see exactly what we mean. When an LLM produces something you do not want, the strong move is to copy the mistake itself into the prompt as a not-preferred example, and to copy the best sentence it has produced as a preferred one.

Examples are the densest instruction we have. A rule describes a pattern and leaks at every edge. An example shows the pattern, edges included. The best prompts often mix them: a short rule for the intent, and examples for everything the rule cannot say.