Language, Vision, Bodies

This is Unit Nine of Artificial Intelligence: language, vision and bodies. The unit covers three subjects, so it comes in two parts, and this is the first — language. The task is narrower than people expect. A language model assigns probabilities to what comes next in a running text. That is the whole objective, and everything these systems do arrives out of doing it at scale. Text is unusual as training material because it labels itself. Every sentence already contains its own answers — the words so far are the input and the next word is the correct output — so the supervision is free. Three error entries here, and they correct beliefs that are extremely widespread. These systems consult no database when answering. Correcting one in conversation teaches it nothing. And when the training residue runs thin, the model does not stop or hedge — it continues, in the same confident grammar it uses when right. That last fact is the unit's most important, and Unit One prepared for it: our species used fluency as its proxy for understanding, and industrial language models have broken the connection. By the end you will be able to do five things. One: say why text is free training data, and what counting models die of. Two: say what an embedding is, and what it inherits. Three: say why there is no lookup at answer time. Four: say why correcting one in conversation teaches it nothing. Five: state confabulation, and why the confidence is not a fault.

Five entries. The task, the property that makes text ideal training material, a working model built by pure counting — and then the two reasons that model cannot scale.

The task.

AI 0501 The Next-Word Task

Assigning probabilities to what follows in a running text. Modern language machines are built on this one objective, pursued at very large scale.

Why text is ideal.

AI 0502 Text Labels Itself

Text trains this task for nothing — every sentence is already a supervised example, with the words so far as input and the next word as the answer.

Counting, as a model.

AI 0503 The Bigram Table

Counting how often each word follows each other word gives a working language model. It genuinely functions — which is what makes its failure informative.

The first failure.

AI 0504 The Counting Wall

Counting models die on sparsity. With a realistic vocabulary, almost every legal sequence of words has never appeared, so the counts are zero nearly everywhere.

The second.

AI 0505 The Missing Ruler

A counting model has no way to notice that an unseen sentence is plausible because its words resemble seen ones. There is no ruler for similarity anywhere in it.

Five entries on the repair. The idea is old and simple — a word is characterised by the company it keeps — and turning it into positions supplies exactly the missing ruler. Then two things those positions inherit.

The idea.

AI 0506 Known by Its Company

A word is known by what it appears alongside — similar contexts, similar meaning. That claim is what makes numeric positions for words possible at all.

The representation.

AI 0507 Words as Rows

Every word gets a row of numbers arranged so that words used similarly sit near each other. Similarity becomes distance — which is the ruler the counting model lacked.

Where the rows come from.

AI 0508 Learning the Rows

The rows are learned, not designed — descent adjusts a few hundred numbers per word so that prediction error falls, and the arrangement emerges from that.

What they inherit.

AI 0509 The Rows Carry Fingerprints

Rows built from text carry the text's associations, complimentary and otherwise. These are company-kept statistics, so whatever the source material associates, the positions record.

And their limit.

AI 0510 One Row per Word

A fixed embedding gives each word one position permanently, so a word with two meanings occupies a single spot regardless of which sense a sentence intends.

Three entries. The mechanism that fixes the one-position limit, the architecture built around it, and the training regime — and the mechanism is simpler than its reputation.

The mechanism.

AI 0511 Attention

Attention rewrites each word's row as a weighted average of the other rows in the passage, with the weights computed from the passage itself. Context now moves the position.

The architecture.

AI 0512 The Transformer

Attention layers stacked with ordinary weighted-sum-and-bend layers, plus information about word order. That stack is the architecture the field now runs on.

The training.

AI 0513 Pretraining

The stack trained on next-word prediction across enormous text collections. Nothing about the objective changed from entry one — only the quantity did.

Two entries answering a question people ask constantly. What is actually inside a trained model, and an error entry about where its answers come from — with a point that cuts both ways.

What remains.

AI 0514 Statistical Residue

The training text is not inside the model. What remains is statistical residue distilled into the settings — which is why it can produce text nobody wrote, and why it cannot reliably reproduce text somebody did.

And the error.

AI 0515 "It Looks the Answers Up in a Database"

Nothing is consulted at answer time — there is no database and no lookup step. That single fact explains both the unprecedented reach and the characteristic failures, which are two sides of one design.

Three entries on how these become the things people actually use, and then the sharpest correction in the unit — what does and does not happen when you talk to one.

Becoming an assistant.

AI 0516 Becoming an Assistant

A pretrained predictor becomes an assistant through further training on demonstrations of assistant behaviour. The predictor and the assistant are the same machine at different stages.

During a conversation.

AI 0517 Context Is Not Training

While talking, the model's settings do not change at all. What changes is the context — the words attention can see — and that is a different thing from learning.

And the error.

AI 0518 "It Learns from Talking to Me"

Correcting a model in conversation teaches it nothing in the training sense. Training and use are separate regimes — the correction affects the current context and nothing beyond it.

Three entries, and the first is the failure mode everything else in this part has been building toward. Where a model's residue is thin, prediction does not stop — and the tone does not change.

The failure.

AI 0519 Confabulation

Where the residue is thin, prediction carries on regardless — producing confident text with the same grammar it uses when correct. There is no internal signal marking the difference.

Why that is dangerous.

AI 0520 Fluency, Decoupled Again

Fluency was our species' working proxy for understanding, and industrial language models have decoupled the two. The habit is old and reliable and no longer tracks what it used to.

And the open question.

AI 0521 The Open Question

Whether anything deserving the name understanding forms inside these systems is a genuine scientific disagreement — and the course presents it as open rather than settling it.

One capstone, tracing a single invented citation from the request that produced it through to the remedy — which makes the mechanism visible rather than merely described.

The capstone. One invented citation.

AI 0522 The Invented Citation

One fabricated reference followed from the request that produced it, through the mechanism that produced it, to what actually prevents it. The misreadings along the way are as instructive as the mechanism.

Five things you can now do — and Part Two covers vision and bodies. You can say why text is free training data — every sentence contains its own answers — and what kills counting models: sparsity, and no notion of resemblance. You can say what an embedding is and what it inherits — the associations of whatever text built it, flattering and otherwise. You can say why there is no lookup at answer time, and that this explains both the reach and the characteristic failures. You can say why correcting one in conversation teaches it nothing: the settings do not move, only the visible context does. And you can state confabulation — confident continuation where the residue is thin — and say why the confidence is a property of the design rather than a malfunction.