SDKProof All eight scores Source on GitHub

Finding · 8 August 2026

Scoring a library twice
can't tell you if its agent docs work.

AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.

SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.

Some libraries now ship files written for AI agents to read. The obvious way to check whether those files help is to score the library bare, score it again with its own files in the prompt, and report the difference.

I did that across three libraries and six arms, and not one difference is distinguishable from zero. That is not a sample-size problem. For React Router, a perfect fix would not register either.

Measure the one task the library actually fails, and the same files move it 50 to 100 points.

Measure the failing task, not the whole scorecard.

The question

A maintainer asked it, and it is the right question

On 31 July 2026, about an earlier scorecard, a Prisma maintainer asked: “does it still do that even with the skills that prisma init installs?”

It is a better question than the one this site had been answering. How well a model knows your API is not something you control — that is a fact about a training cut-off. Whether the files you ship close the gap is entirely under your control, and nobody was measuring it.

So: score every library twice, bare and with its own published agent files, and report the difference.

Three libraries publish something. They publish three different things.

Prisma 7: From v7.9, prisma init writes a skill tree into the user's project — .agents/skills/, .claude/skills/, .windsurf/skills/, 9 packs. The only one of the three that puts files on a user's disk.
React Router 8: A skill pack exists at .agents/skills/react-router/ in the repo, but it is not shipped: npm i react-router does not install it, and reactrouter.com/llms.txt returns 404. Measured anyway, because it is what they wrote.
Next.js 16: A Markdown version of every docs page, served alongside the HTML — 6.6 KB where the page itself is 490 KB. Published on the web, not installed.

The obvious method

Every whole-scorecard arm comes back empty

Fifteen tasks per library, three trials each, so 45 generations per arm. The baseline is the same task set with no context at all. The interval is Newcombe's on the difference of two proportions.

Library	What was in the prompt	Compiled	vs bare	95% CI on the difference	Verdict
Prisma 7	`prisma-client-api` pack	39 / 45	+0.0 pp	−14.7 … +14.7	no effect
Prisma 7	plus the setup & upgrade docs	42 / 45	+6.7 pp	−6.6 … +20.2	no effect
React Router 8	`SKILL.md` only	42 / 45	+2.2 pp	−10.2 … +14.8	no effect
React Router 8	the full five-file pack	41 / 45	+0.0 pp	−13.0 … +13.0	no effect

Prisma's baseline is 39 of 45, React Router's is 41 of 45. Every interval contains zero. Two older arms from a Prisma 6 run at n=8 come back empty too, so across every scorecard-scale arm ever run here the count is 0 of 6.

The tempting misreading is “+6.7 points, so the skills help a bit.” That difference is three generations out of forty-five. React Router's “+2.2” is one generation.

Published without an interval — which is how this site published them until today — they read as small improvements rather than as noise.

The obvious objection, tested

It is not the sample size. The instrument can't see a success.

The natural objection is that 45 generations is too few. So ask what the metric would report if the agent files worked perfectly — every task passing, in every trial.

Library	Best possible arm	vs bare	95% CI on the difference	Would it register?
Prisma 7	45 / 45 vs 39 / 45	+13.3 pp	+2.8 … +26.2	barely
React Router 8	45 / 45 vs 41 / 45	+8.9 pp	−0.6 … +20.7	no

For React Router, a total success is indistinguishable from doing nothing. Prisma's ceiling clears zero, but only just, and only if both of its failures are fixed in all three trials.

Why

The passes you already had sit in the denominator

Prisma 7 passes 13 of 15 tasks with no context at all. React Router 8 passes 14 of 15. Documentation can only fix what is broken, so those thirteen and fourteen untouched passes sit in the denominator of both arms and dilute the one or two tasks that can actually move.

The better a library scores, the less able this metric is to detect its docs working. That is backwards, and it is structural. No number of trials removes it, because the diluting tasks are not noise — they are correct answers.

The method that works

Isolate the one task that fails

Same libraries, same files, same model. The only change is the denominator: one task, the one the library actually gets wrong, at ten trials per cell.

Library & the task it fails	What was in the prompt	Compiled	vs bare	Verdict
Prisma — `construct-client`	nothing	0 / 10	—	baseline
	`prisma-client-api` pack	0 / 10	+0.0 pp	no effect
	plus the setup & upgrade docs	5 / 10	+50.0 pp	real
	the adapter sentence alone	10 / 10	+100.0 pp	real
React Router — `meta-from-loader`	nothing	0 / 10	—	baseline
	the sentence alone	10 / 10	+100.0 pp	real
	the section it sits in	10 / 10	+100.0 pp	real
	the 7.3 KB file it sits in	0 / 10	+0.0 pp	no effect
	the full 25 KB pack	0 / 10	+0.0 pp	no effect
Next.js — `revalidate-tag`	nothing	0 / 10	—	baseline
	the sentence alone	10 / 10	+100.0 pp	real
	plus 25 KB of padding, first	10 / 10	+100.0 pp	real
	plus 25 KB of padding, last	10 / 10	+100.0 pp	real
	plus 50 KB of padding, first	9 / 10	+90.0 pp	real

Eight of these eleven arms clear zero. None of the six scorecard arms did. Same files, same model, same measurements underneath.

The only difference is whether the twelve or thirteen tasks that were never broken are allowed into the denominator.

The three arms that show no effect are the three that genuinely fixed nothing: prisma-client-api at 0 of 10, and React Router's containing file and full pack, both at 0 of 10. An empty result and a real zero look the same here, and they should — the point of the interval is that it refuses to invent a direction in either case.

What comes out of it

Three findings a maintainer can act on

Not “your skills moved your score 87 → 93.” That sentence is unsupported, and it is also not useful. What the failing-task measurement produces is specific enough to fix.

1. Prisma ships a pack named for the task that does not fix the task

prisma-client-api contains constructor.md, and it is what an agent routing on a skill's name loads for “construct a PrismaClient”. It moves the number by zero, on the scorecard and on the isolated task.

What does fix it lives in prisma-postgres-setup/references/prisma7-client.md and prisma-upgrade-v7/references/driver-adapters.md. That is a routing problem, not a writing problem, and it is fixable in an afternoon.

2. React Router's correct sentence is already written, and its own file buries it

Alone, the sentence fixes the failure 10 times out of 10. Inside the 7.3 KB file it sits in, 0 times out of 10. Publishing a short file would work. Publishing the pack would not.

3. Next.js is the one that already works

Its per-page Markdown fixes its failure 10 times out of 10, and keeps working with 50 KB of unrelated documentation in front of it.

Limits

What this does not establish

One model: Everything here is Claude Opus 5. Another model will behave differently.
Three libraries: One failing task each. Ten trials per cell on the isolated tasks, three on the scorecards.
One-shot prompts: Not an agent that can search. The files are pasted straight in, which is the most favourable delivery there is — and the pack arms still sit at zero.
The single-sentence arms are close to handing over the answer: Their whole context is the correction. They establish a ceiling — the words work — not a shipping recommendation.
Most libraries publish nothing to measure: Only three of nine libraries surveyed ship any agent context at all, so “do my agent docs work” is a question most maintainers cannot yet ask.

Check it yourself

Every number here is recomputed from files in the repo

No number on this page comes from a fresh run. Each one is recomputed from result files already committed.

Two commands from a clone of the repo

npx tsx scripts/skills-delta.ts   # every arm on disk, with intervals
npm test                          # includes the interval arithmetic

Arms and provenance: fixtures/*/agent-context/PROVENANCE.md. Interval maths: armDelta() in src/stats.ts, Newcombe's method over the Wilson intervals. Per-library numbers on the Prisma, React Router and Next.js scorecards; the surrounding-text result is in the agent-docs survey.

Published 8 August 2026. The scorecard deltas on this page were previously shown without intervals, including as “87 → 93” on the Prisma page. That reading is withdrawn here and corrected there.

Elsewhere

Related findings

Agent docs One sentence from a library's own docs fixes the failure 10 times out of 10. The same sentence buried in their full docs pack fixes nothing. Read the numbers → The board Eight libraries, each scored by compiling the model's answers against the real installed package. See every score →

Score my library

Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.

Request a scorecard → Read the code

There's no npm package. Clone the repo, point it at a library, run it. The compiler is the judge.