Artificial Intelligence / Unit 9 AI 0511 worked example, 60 to 90 seconds
Attention
The idea
Attention rewrites every word's row as a weighted average of the other rows in the passage — with the weights computed from how well the rows match, so each word assembles its own context.
In practice
Read "the melon that the farmer thumped was ripe" and ask what "was ripe" must attend to: melon, not farmer. Attention's pair-matching, tuned by descent until prediction succeeds, learns to weight exactly so — the melon row pours heavily into the rewrite, the farmer row barely at all. And the one-row problem dissolves: after an attention layer, "bank" in a river sentence and "bank" in a loan sentence have been rewritten by different company, which repairs the flaw video 0510 exposed. The weighted average is Unit 6's oldest tool; the novelty is that the weights are not knobs set once by training but numbers produced on the fly, from the words themselves, at every reading.
Where people slip
No grammar rule was written anywhere — the weighting was learned because it made prediction better.
Your place on the trail
Every lesson rests on earlier ones and opens later ones. Tap any to jump.
Something wrong in this lesson? Flag it · Sources for Artificial Intelligence