Next 15 is learned.
Next 16 isn't, in one place.
next 16.2.11 · claude-opus-5
AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.
SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.
This page is Next.js. Thirteen ordinary App Router jobs — read a cookie, write a route handler, redirect, clear a cache — one shot each, no docs, no retries.
12 of the 13 compiled. The one miss is the newest thing in the set: Next 16 gave revalidateTag() a second argument, and the model still calls it with one.
This measures the model, not Next.js.
One task: clearing a cache tag
Next 16 reworked caching. revalidateTag(tag) became revalidateTag(tag, profile) — the second argument ties the clear to a cache profile. The model writes the one-argument call that was right through Next 15.
next 16.2.11
import { revalidateTag } from "next/cache";
export async function refreshUsers() {
revalidateTag("users");
}error TS2554: Expected 2 arguments, but got 1.
next 16.2.11
import { revalidateTag } from "next/cache";
export async function refreshUsers() {
revalidateTag("users", "max");
}
// 2nd argument: a cache profile,
// either a name or a config objecttsc --noEmit — 0 errors
Reading the error. TS2554 is the compiler saying a function got the wrong number of arguments. It is the whole failure — the import is right, the function is right, the count is not.
The Next 15 async change is fully absorbed
Next 15's biggest change made the request helpers async — you have to await cookies(), await headers() and await draftMode(). That trips a lot of hand-written code. The model got all three right without being told.
Route handlers, middleware, redirects and path revalidation all compiled on the first try too. The prompts name the job — "read the session cookie", "send a 307" — never the function signature.
Which is the pattern in one library: the change from two years ago is learned, the change from this year is not. Give it another model release and this miss will most likely close on its own.
One sentence from Next's own docs fixes it
Next.js serves a Markdown version of every docs page. The revalidateTag page states the two-argument signature in a line. I put that line in the prompt and re-ran the failing task ten times.
Then I padded it — same sentence, more text around it — to test whether long documents bury a correction.
| What was in the prompt | Size | Where the sentence sat | Compiled |
|---|---|---|---|
| nothing — the score above | — | — | 0 / 10 |
| the sentence alone | 137 B | — | 10 / 10 |
| sentence, then unrelated Next.js docs | 25 KB | first | 10 / 10 |
| the same docs, then the sentence | 25 KB | last, at character 24,960 | 10 / 10 |
| sentence, then twice the padding | 50 KB | first | 9 / 10 |
Length is not what buries a correction. Twenty-five kilobytes of unrelated Next.js documentation left the fix working, whether the sentence went first or last. That result is the control that killed the obvious explanation — and it matters for two other libraries here, where the same sentence inside their own docs does nothing at all.
Worth copying regardless: a Markdown version next to every docs page is 6.6 KB where the HTML is 490 KB. A model that can fetch one page gets something it can use. Most libraries in the nine-library survey serve either a list of links or a five-megabyte bundle.
The compiler has the last word
Thirteen realistic App Router tasks, written by claude-opus-5, each dropped into a small project with next 16.2.11, react and @types/react actually installed, then run through tsc --noEmit. A task passes only if it compiles.
- One model
- Everything here is
claude-opus-5. Another model will score differently. - One shot, no docs
- The model gets the task and nothing else. The table above shows what one sentence of documentation does to the one failing task.
- Small numbers
- Thirteen tasks. A 92 and a 93 are the same thing. Read the gap between 0 and 100 as the signal.
- Compiles is not correct
tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code, so nothing here says a route behaves properly.- Refusals
- A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile, and refused tasks drop out of the denominator. Next.js had none. Stripe had five, which is why its 100 is out of 10 rather than 15.
- It measures the model
- 92 is a statement about what
claude-opus-5remembers of Next.js, not about Next.js itself.
Related findings
Other libraries, scored the same way
v9 renamed the main hook. The model wrote v8 every time.
Queries are clean. It still builds the client the v6 way.
meta() still reads the removed data argument.
Every v4 to v5 rename written unprompted.
Score my library
Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.
There's no npm package. Clone the repo, point it at a library, run it. The compiler is the judge.