All eight scores Source on GitHub
Scorecard · React Router 8

Fourteen of fifteen.
The fifteenth fails every time.

react-router 8.3.0 · claude-opus-5

AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.

SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.

This page is React Router. Fifteen ordinary jobs — load data for a route, submit a form, read a search param, set the page title — one shot each, no docs, no retries.

14 of the 15 compiled. The miss is a rename: the meta function used to receive an argument called data, v8 calls it loaderData, and the model still writes data. Re-run three more times and it failed 3 of 3.

This measures the model, not React Router.

93/100 14 of 15 compiled
14 compiled 1 did not compile 0 refused
Model: claude-opus-5 Tasks: 15 Package: react-router 8.3.0 Run: 6 August 2026 Pass: it compiles
What went wrong

meta() still reads the argument v8 deleted

v8 renamed the data field on the meta arguments to loaderData, so it lines up with Route.ComponentProps. The rename touched MetaArgs, MetaMatch, UIMatch and Route.ComponentProps.matches together.

What the model wrote react-router 8.3.0 
export const meta: MetaFunction<typeof loader> =
  ({ data }) => {
    return [{ title: data?.user?.name ?? "User" }];
  };

error TS2339: Property 'data' does not exist on type 'MetaArgs<() => Promise<{ user: … }>, Record<string, unknown>>'.

What v8 compiles react-router 8.3.0 
export const meta: MetaFunction<typeof loader> =
  ({ loaderData }) => {
    return [{ title: loaderData?.user?.name ?? "User" }];
  };

tsc --noEmit — 0 errors

Why this one slips past a human reviewer. data is the more natural word, and it sits inside a destructured argument where nothing looks unusual. You read it and see nothing wrong. The compiler doesn't read, it checks — TS2339 means the property isn't on the type.

The other 14

The removals it could have tripped on, it didn't

v8 deleted a lot: the json() and defer() helpers, the whole react-router-dom package, and the plain-object route context. The model wrote none of them.

It returns bare objects from loaders instead of json(). For streaming it hands back an un-awaited promise instead of defer(). It wraps a value in data() when it needs to set a status code, calls createContext() without the old unstable_ prefix, and reads middleware state with context.get(userContext) rather than as a property.

loader / action redirect() useLoaderData useFetcher useSubmit useSearchParams isRouteErrorResponse createBrowserRouter no json() — plain objects no defer() — raw promise data() for status codes createContext, no unstable_ context.get()

93 on a release this new is the surprise here. The likely reason: v8 mostly finished removals that v7 had already been warning about for a long time, so the model had years of "stop using this" to learn from. The one thing it gets wrong is the one thing that was a straight rename with no warning period.

Tested, not guessed

The fix is one sentence you've already written

.agents/skills/react-router/references/framework-mode.md says: "Important: meta receives loaderData; do not use deprecated data args." That file isn't installed by npm i react-router, and reactrouter.com/llms.txt returns a 404. Of nine libraries surveyed, this is the only one publishing neither.

So I put the sentence in the prompt myself and re-ran the one failing task, ten times per row.

What was in the promptSizeWhat it was aboutCompiled
nothing — the score above——0 / 10
that one sentence, alone76 B—10 / 10
the Metadata section it sits in215 Brouting10 / 10
25 KB of Zod docs, then the sentence25 KBunrelated8 / 10
the file the sentence lives in7.3 KBrouting0 / 10
all five files of the pack25 KBrouting0 / 10

Seventy-six characters fix it ten times out of ten. The identical sentence inside its own 7.3 KB file fixes it zero times out of ten. The model isn't refusing the instruction. It isn't reaching it.

And it isn't length. Twenty-five kilobytes of Zod documentation with the same sentence at the end still passed 8 of 10. It's the routing material around it — the other four files in the pack turned out to be irrelevant, the containing file alone is enough to smother the line.

What that means in practice

The useful half is settled. A short standalone file naming the v8 renames would work, and it is far less effort than publishing everything. That is measured, not assumed.

Prisma shows the same shape: its adapter requirement passes 10 of 10 alone, and 5 of 10 inside its own 25 KB pack. Why the full pack fails is still an open question — the controlled comparisons are here.

Put plainly: line 182 does not survive the other 212 lines of the file it lives in.

This section was rewritten twice on 6 August 2026. It first said "publish the file", then — after the buried versions failed — said documentation does not help at all. Testing the sentence on its own showed both were wrong. The numbers above are the third and current answer.

How this was measured

The compiler has the last word

Fifteen realistic React Router tasks, written by claude-opus-5, each dropped into a small project with react-router 8.3.0 actually installed, then run through tsc --noEmit. A task passes only if it compiles.

One model
Everything here is claude-opus-5. Another model will score differently.
One shot, no docs
The model gets the task and nothing else. The table above shows what happens when it gets one sentence more.
Small numbers
Fifteen tasks. A 92 and a 93 are the same thing. Read the gap between 0 and 100 as the signal.
Compiles is not correct
tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code, so nothing here says a route behaves properly.
One miss it can't see
v8 deleted react-router-dom and asks you to import RouterProvider from react-router/dom. The model imported it from react-router, which still compiles because the root entry re-exports it. That task passed, and the stale habit it was written to catch is invisible to a type-checker.
Refusals
A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile, and refused tasks drop out of the denominator. React Router had none. Stripe had five, which is why its 100 is out of 10 rather than 15.
It measures the model
93 is a statement about what claude-opus-5 remembers of React Router, not about React Router itself.
Elsewhere

Related findings

Agent docs One sentence from a library's own docs fixes the failure 10 times out of 10. The same sentence buried in their full docs pack fixes nothing. Read the numbers → Agent skills Three libraries ship files meant for AI agents. Scored with and without them, across six runs, not one difference clears zero. Read the numbers →
The rest of the board

Other libraries, scored the same way

TanStack Table 90/100
0 of 12 compiled

v9 renamed the main hook. The model wrote v8 every time.
Prisma 787/100
13 of 15 compiled

Queries are clean. It still builds the client the v6 way.
Next.js 1692/100
12 of 13 compiled

One miss: revalidateTag() takes two arguments now.
Stripe 22100/100
10 of 10 compiled, 5 refused

Clean on everything it wrote. It would not write a third of the set.

All eight scores are on the home page →

Score my library

Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.

Request a scorecard → Read the code

There's no npm package. Clone the repo, point it at a library, run it. The compiler is the judge.