Artificial Intelligence / Unit 9 AI 0504 short lesson, about 20 seconds
The Counting Wall
Captions available
The idea
Counting-based language models die on sparsity — with a realistic vocabulary, almost every legal word sequence has never been written, so its count is zero.
In practice
A 50,000-word vocabulary makes the bigram table two and a half billion cells; three-word contexts push past a hundred trillion.
Where people slip
No pile of text on Earth fills those tables — "unseen" and "impossible" collapse into the same zero.
Your place on the trail
Every lesson rests on earlier ones and opens later ones. Tap any to jump.
Unlocks
Something wrong in this lesson? Flag it · Sources for Artificial Intelligence