Limits and Society

This is Unit Ten of Artificial Intelligence: limits and society. The unit covers two subjects, so it comes in two parts, and this is the first — how to evaluate a claim about what a system can do. The unit opens by taking apart a sentence type you encounter constantly. Any statement that a system can do something is three claims at once: a task, measured somehow; a distribution of cases it was measured on; and a level of reliability. Separating those three turns a headline into a list of questions that have answers. And most of the disagreement about this field comes from people arguing about a sentence without having separated it. The part's central technical result is that no learner is generally best. Every method makes assumptions about what the world is like, and works where those assumptions hold. Capability is therefore jagged — excellent where the bet fits, and dropping off sharply where it does not. And there is a rule that keeps the whole apparatus honest. The same ten questions deflate an overclaim and a dismissal, and a dismissal that cannot answer them is no better than the claim it rejects. By the end you will be able to do five things. One: unpack a capability claim into its three claims. Two: say what a benchmark score establishes, and what erodes it. Three: say why every learner is a bet. Four: price a multi-step task by multiplying along it. Five: say why overclaiming and dismissal are twin errors.

Two entries, and this is the unit's central instrument. Learn to hear three separate claims inside one sentence and most arguments about this field become tractable — because each of the three has an answer.

The three claims.

AI 0561 A Capability Claim Is Three Claims

Any statement that a system can do something contains a task and how it was measured, a distribution of cases it was measured over, and a level of reliability. All three are usually left implicit.

Expanded.

AI 0562 The Bar-Exam Claim, Expanded

Taking a familiar headline and writing out its three clauses turns it into a list of questions that can actually be answered — which the original sentence prevented.

Six entries on the shared exam, and four separate ways its meaning degrades over time. The instructive part is that all four are structural — none requires anybody to behave badly.

What a benchmark is.

AI 0563 The Benchmark

A shared held-out exam — Unit Eight's locked drawer, standardised so rival systems can be compared on the same cases.

The first erosion.

AI 0564 Contamination

The exam's text reaching the training pile. Public benchmarks circulate and training piles are enormous, so the overlap happens without anyone arranging it.

The second.

AI 0565 The Frozen Exam

A benchmark freezes while the world moves. Its cases stay typical of the year it was assembled, and what counts as typical does not hold still.

The third.

AI 0566 Communal Overfitting

A held-out grade is spent the moment it guides tuning — and thousands of teams tuning against one public leaderboard spend it collectively, without any single team overfitting.

The pattern.

AI 0567 The Benchmark Life Cycle

A benchmark starts out discriminating, becomes a target, and ends as a formality. So the same score means different things depending on where in that cycle it was earned.

And the error.

AI 0568 "Benchmark Score Equals Capability"

The score is a real measurement and equating it with capability is the mistake — the exam may be contaminated, aged, or collectively tuned against, and the number reports none of that.

Four entries containing the part's deepest result. Averaged across all possible worlds, no method beats any other — which means every method that works does so because it assumed something true about this world.

The result.

AI 0569 Every Learner Is a Bet

With no assumptions, no learner outperforms any other across all possible worlds. Learning works only where its assumptions hold — so every method is a bet about what reality is like.

And no escape through simplicity.

AI 0570 No Refuge in Simplicity

Preferring simple explanations does not rescue it, because simplicity depends on the vocabulary. A rule that is short in one set of features is enormous in another, so the preference is itself an assumption.

The shape that follows.

AI 0571 The Jagged Frontier

Capability runs jagged along the edges of each method's assumptions — excellent where they fit and dropping sharply where they do not, often without warning.

And the limits themselves.

AI 0572 The Walls, Dated

The reliable limits of current systems are mechanisms rather than accidents — confabulation, fragility outside the training distribution, opacity. Each has a cause this course has already explained.

Two entries with a result that prices long-running autonomy in three multiplications. Every step in a multi-step task has to succeed, and reliabilities multiply — which turns a good per-step number into a poor overall one.

The arithmetic.

AI 0573 Multiply Along the Chain

A task with many steps succeeds only if all of them do, and per-step reliabilities multiply along the chain rather than averaging.

Priced.

AI 0574 The Hundred-Step Task

Chain arithmetic prices long-horizon autonomy in three multiplications, and the price is steep — a per-step rate that sounds excellent produces an unusable whole.

The capstone gathers everything into ten questions, and then one entry stops the whole instrument from becoming a rhetorical weapon — which is the difference between an auditor and a sceptic.

The capstone. Ten questions.

AI 0575 The Auditor's Checklist

Ten questions compressing the course's evaluation machinery, each with a mechanism behind it and a claim it deflates. They are the assembled form of everything this part covered.

The rule.

AI 0576 The Symmetry Rule

Overclaiming and dismissal are twin errors, and the same ten questions deflate both. A dismissal that cannot answer them is worth no more than the claim it rejects.

Three entries closing the part by turning the apparatus on predictions about the field itself — which are, structurally, exactly the kind of thing this unit has been examining.

What a forecast is.

AI 0577 Forecasts Are Extrapolations

Forecasts about this field are themselves predictions from past distributions — so every lesson about distribution shift applies to them directly.

The stance that survives.

AI 0578 Conditioned Watching

Conditioning rather than forecasting — name the mechanism each current wall rests on, and watch for evidence that a specific mechanism has moved. That is checkable in a way a date is not.

And the error.

AI 0579 "The Curve Tells You Where It's Going"

Extrapolating a capability curve is a bet that the future resembles the past — worth exactly what that bet is worth, and no more, however smooth the curve looks.

Five things you can now do — and Part Two covers deployment. You can unpack a capability claim into three claims — a task and its measure, a distribution of cases, and a reliability — each of which has an answer. You can say what erodes a benchmark score: contamination, age, collective tuning, and the life cycle that makes the same number mean different things. You can say why every learner is a bet, and that simplicity offers no escape because it depends on the vocabulary you chose. You can price a multi-step task by multiplying along the chain, and say why an excellent per-step rate produces an unusable whole. And you can say why overclaiming and dismissal are twin errors, deflated by the same questions — so a dismissal that cannot answer them is worth nothing extra.