A library's own docs
bury its own correction.
AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.
SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — which turns "did the docs help?" into something you can count.
So I counted. Three libraries, each with one job the model reliably gets wrong, and each with a sentence in its own documentation that names the fix.
The sentence on its own fixes the failure, ten times out of ten. Two of the three ship that same sentence inside their own larger docs, and there it fixes nothing.
It's not the length. That was tested and it doesn't hold.
One failing task per library, ten tries per cell
The only thing that changes between columns is how the sentence reaches the model. Same task, same model, same installed package, same compiler.
| Library and the task it fails | No context | The sentence alone | Inside the library's own docs |
|---|---|---|---|
React Router — the meta() argument | 0 / 10 | 10 / 10 (76 B) | 0 / 10 (25 KB pack) |
| Prisma — building the client | 0 / 10 | 10 / 10 (174 B) | 5 / 10 (25 KB pack) |
Next.js — revalidateTag argument count | 0 / 10 | 10 / 10 (137 B) | 10 / 10 (6.6 KB page) |
The one thing that holds across all three: the sentence works. None of these were written for the experiment — every one is quoted word for word from the library's own documentation. The model reads the instruction and follows it.
It is not length
The obvious reading of that table is that long documents swallow a correction. React Router's sentence sits at line 182 of a 213-line file, and there it never works.
That was tested directly, and it's wrong. Same Next.js task, same sentence, nothing varying but how much text surrounded it and where the sentence sat. The padding was real Next.js documentation for other APIs, checked to never state the signature under test.
| What was in the prompt | Size | Where the sentence sat | Compiled |
|---|---|---|---|
| nothing | — | — | 0 / 10 |
| the sentence alone | 137 B | — | 10 / 10 |
| sentence, then padding | 25 KB | first | 10 / 10 |
| padding, then sentence | 25 KB | last, at character 24,960 | 10 / 10 |
| sentence, then twice the padding | 50 KB | first | 9 / 10 |
Neither size nor position brought the failure back. So the question became: what is different about the documents that do smother it?
They agree, and they point at the same thing
Both run on React Router, on the one task it fails, ten tries per row.
| What was in the prompt | Size | What it was about | Compiled |
|---|---|---|---|
| the sentence alone | 76 B | — | 10 / 10 |
| the section it sits in | 215 B | routing | 10 / 10 |
| 25 KB of Zod docs, then the sentence | 25 KB | unrelated | 8 / 10 |
| the file the sentence lives in | 7.3 KB | routing | 0 / 10 |
| all five files of the pack | 25 KB | routing | 0 / 10 |
Read the two middle rows together. Twenty-five kilobytes of schema-validation documentation leaves the correction working. Seven kilobytes of the library's own routing documentation destroys it. Same sentence, same file, both times.
So it's neither length nor topic on its own. 25 KB of unrelated text is harmless. 215 bytes of the library's own routing text is harmless. About 7 KB of it is not. What crowds out a correction is how much material about the same subject sits around it — and the other four files in React Router's pack turned out to be irrelevant. Its own containing file is enough on its own.
That also explains Next.js, which looked like a counterexample. Its 6.6 KB page is the same subject as the correction but it is entirely about the failing function — nothing in it competes for the same decision, so there's nothing to crowd anything out.
A confound worth naming
The three failures are not equally easy to correct. Next.js's is an argument-count error, and "the second argument is required" is about as direct as a fix gets. React Router's is a renamed field inside a destructured argument.
It is entirely possible that some corrections survive their surroundings and others are fragile, and that this explains the tables above better than anything about the documents themselves. I can't rule it out with three libraries.
What can be said, and what cannot
Supported
A short, specific sentence naming a change fixes the failure it names, on all three libraries measured, 9 or 10 times out of ten. It is worth shipping one.
Supported
Two of these three libraries already contain that sentence somewhere in their own documentation, and it is not reaching the model there.
Supported
The cause is not length. It's how much same-subject material surrounds the correction — 25 KB of unrelated docs leaves it working, 7.3 KB of the library's own routing docs does not.
Not supported, and withdrawn
That document length alone is the cause. That was an earlier conclusion on this page. The size-and-position sweep did not reproduce it, so it's gone.
What libraries publish today
Nine libraries, two filenames that promise the same thing. The filename is a convention. The contents are not. Sizes are in characters; "links" is the share of the file that is a link index rather than prose.
| Library | llms.txt | llms-full.txt |
|---|---|---|
| Prisma | 2,450 · 71% links | 6,915 · 0% links |
| tRPC | 15,699 · 0% links | 599,762 |
| Zod | 21,522 · 94% links | 259,341 · 1% links |
| Vercel AI SDK | 2,217 · 33% links | 5,673,960 |
| Drizzle ORM | 37,377 · 84% links | 3,553,515 |
| Next.js | 8,792 · 58% links | — (404) |
| Stripe | 93,150 · 87% links | — (404) |
| TanStack Query | 11,396 · 78% links | identical file |
| React Router | — (404) | — (404) |
Sizes run from 2,217 characters to 5,673,960, under two filenames that claim to be the same thing. A model that can't fetch anything gets nothing usable out of a link index, and nothing at all out of a five-megabyte bundle. Next.js is worth copying on one point regardless: it serves a Markdown version next to every docs page — 6.6 KB where the HTML is 490 KB — and that file fixed its failure ten times out of ten.
Two commands
The size difference in the last row of that table is one curl away.
measure the two files
curl -sL https://nextjs.org/docs/app/api-reference/functions/revalidateTag.md | wc -c
curl -sL https://ai-sdk.dev/llms-full.txt | wc -cSurvey script: scripts/survey-agent-docs.mjs → data/agent-docs.json. The context runs are the --with-context path in the same repo. Per-library numbers sit on the React Router, Prisma and Next.js scorecards.
What this does not tell you
- One model
- Everything here is
claude-opus-5. Another model may read a long document differently. - Three libraries
- One failing task each, ten tries per cell. That's enough to see a 0-versus-10 gap and not enough to explain it.
- Close to handing over the answer
- In the single-sentence run the model's entire context is the correction. So it measures whether an obvious instruction gets followed, not whether documentation works in normal use.
- One shot, not an agent
- These are single prompts. A real agent can search, fetch and re-read, which is a different situation and probably a kinder one.
- Compiles is not correct
tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code.- Refusals
- A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile, and refused tasks drop out of the denominator. None of the tasks here were refused. Stripe is the one library where that happens, which is why its 100 is out of 10 rather than 15.
- It measures the model
- Everything on this page is a statement about how one model reads documentation, not about how well any of these libraries are documented.
Revised twice on 7 August 2026. An earlier version concluded that document size was the cause; a controlled size-and-position sweep did not reproduce it, and that claim is withdrawn. The current claim rests on two controlled comparisons — one holding size steady, one holding subject steady — and on narrowing the effect down to a single file.
The follow-on question
Score my library
Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.
There's no npm package. Clone the repo, point it at a library, run it. The compiler is the judge.