Next-Word Prediction: Learning from Patterns

How does the LLM formulate its response?

LLMs are taught, as part of their training process, to predict the next word. The model is shown a sentence where we already know what the next word should be, and it is trained until its prediction matches. Repeat this at enormous scale and something remarkable happens: the model becomes incredibly sensitive to the patterns in our words.

Part of the underlying architecture is a mechanism for learning what to pay attention to — words it saw earlier, even much further back in the text — and how to use them to figure out what's coming next. That attention to distant context is a large part of what makes the prediction so powerful.

The simple case: patterns it already knows

Beyond memorized phrases: patterns established on the fly

Memorized rhymes are the trivial case. The interesting case is a pattern that exists only inside the current conversation. Watch a transcript establish two characters — their voices, their relationship, their situation — and then hand the model just two words of the next line:

We don't actually know how it does this

Here is the honest and fascinating part: we don't know how the model does this internally. We know how to set these systems up, train them, and build them — but how the structure inside gives rise to this capability is an area of active research. Much of that research looks strikingly like neuroscience: researchers run ablation studies, disabling components to see what breaks, the same way we probe the brain. We built it, we can train it, and we are still reverse-engineering what it actually learned.

We know how to build one — not what it learned

Hold both of these at once, because both are true. We know how to train one: the architecture, the training loop, the data pipeline — the mechanics of building these systems are established engineering, written down and reproducible. And at the same time: we don't know what the learned weights represent or how they operate. The recipe is understood; the thing the recipe produces is not. Everything in this course sits on that strange foundation.

A conversation is a trajectory

Now connect this back to an earlier idea. If the prompt is the whole conversation, and the model continues patterns, then a conversation is not a container of facts — it is a trajectory. Every message pushes the model in a direction; the model tries to complete the conversation along the path the document has established, until it reaches a natural stopping point (or a word limit). Same question, different trajectory, different continuation:

The trajectory is assembled from the conversation's parts

Where did that trajectory come from? Not from any single sentence. Read the conversation again and you can pick out the components — a place, a task, a voice, a track record — each one embedded somewhere in the text. Together they aim the document. When you read a conversation and sense where it could head next — that sense is the trajectory.

The trajectory is not a rail

Two more things before the contrast example, and they matter. First: the trajectory does not have to be deterministic or tightly constrained — it is a direction, not a track. Second, the consequence: the same conversation can yield a different trajectory each time we generate. Send the identical document twice and watch:

And now the contrast: the same subject, aimed down a completely different trajectory.

The takeaway

Prompt engineering is trajectory engineering. Every technique to come — examples, roles, tools, plans, retrieved knowledge — is a way of shaping the document so that the most probable continuation is the one we want.