The code looks right.It doesn't compile.
AI coding assistants write library code from memory. Ship a big release that renames or removes things & the assistant keeps writing the old version — it reads fine, and it doesn't compile. Point this at your package and it tells you which parts.
──────────────────────────────────────────────────────────── @apollo/client v3.14.1 -> v4.2.12 ──────────────────────────────────────────────────────────── v4 landed 12 months ago 228 exported symbols -> 134 (entry-only diff) WHAT LEFT (23) — functions, hooks and classes gone from the entrypoint with no deprecation first This is what a model trained on v3 will still write. ApolloConsumer ApolloProvider DocumentType createQueryPreloader getApolloContext makeReference ... 17 more Deprecated first, then removed (49) — these rarely produce drift: ApolloError, BackgroundQueryHookFetchPolicy, BackgroundQueryHookOptions, BaseMutationOptions, BaseQueryOptions, BaseSubscriptionOptions, ExecutionPatchIncrementalResult, ExecutionPatchInitialResult, ExecutionPatchResult, IncrementalPayload, InteropApolloQueryResult, InteropExecutionPatchResult, InteropLazyQueryExecResult, InteropMutateResult, InteropMutationExecutionPatchIncrementalResult, InteropMutationExecutionPatchInitialResult, InteropQueryResult, InteropSubscribeResult, LazyQueryExecFunction, LazyQueryHookExecOptions ... (48 type-only export(s) also left the entrypoint. They are listed in --json; a model writes a hook far more often than it writes a type name.) WORTH SCORING — 23 functions, hooks and classes gone from the entrypoint with no deprecation first, in a major that is 12 months old Next: npx sdkproof @apollo/client
Reads two published versions off npm & diffs their type declarations — nothing is installed and no model is called. The other half, npx sdkproof <package>, gives a model real coding jobs and puts every answer through tsc, the TypeScript compiler. Pass means it compiled.
What was asked basic-table
Write buildUserTable(data) that creates a TanStack React Table instance over a list of users with id and name columns, and returns the instance.
… 39 more lines.
What the compiler said
$ tsc -p fixtures/react-table/tsconfig.json --pretty false
fixtures/react-table/candidate.ts(2,3): error TS2724: '"@tanstack/react-table"' has no exported member named 'createTable'. Did you mean 'ReactTable'?
did not compile
Table v9 has no createTable on its main entry point. The model wrote the v8 API, and 0 of 12 answers compiled.
Five of the 20 answers that did not compile, one from each library that has them. Pass means the answer compiled against the real installed package — nothing else counts.
TanStack Table v9 renamed the hook. The model kept writing v8.
v9 dropped useReactTable and createTable from the main entry point, and replaced the getCoreRowModel() family with a features map you pass in yourself. Every type now takes that feature set as its first type argument.
I gave Claude Opus 5 twelve ordinary table jobs: build a table, sort it, paginate it. All twelve failed to compile. It wrote the v8 API every time, so that's 0 out of 100.
@tanstack/react-table 9.1.2
import {
createTable,
getCoreRowModel,
type ColumnDef,
type Table,
} from "@tanstack/react-table";
export function buildUserTable(data: User[]): Table<User> {
const columns: ColumnDef<User, any>[] = [
{ id: "id", accessorKey: "id", header: "ID" },
{ id: "name", accessorKey: "name", header: "Name" },
];
return createTable<User>({
data,
columns,
getCoreRowModel: getCoreRowModel<User>(),
state: {},
onStateChange: () => {},
renderFallbackValue: null,
});
}error TS2724: '"@tanstack/react-table"' has no exported member named 'createTable'. Did you mean 'ReactTable'? error TS2558: Expected 2 type arguments, but got 1.
@tanstack/react-table 9.1.2
import type { ColumnDef } from "@tanstack/react-table";
import {
useTable,
createColumnHelper,
rowSortingFeature,
} from "@tanstack/react-table";
const features = { rowSortingFeature };
type Features = typeof features;
const helper = createColumnHelper<Features, User>();
const columns: ColumnDef<Features, User, any>[] = [
helper.accessor("id", { header: "ID" }),
helper.accessor("name", { header: "Name" }),
];
export function buildUserTable(data: User[]) {
return useTable<Features, User>({ features, columns, data });
}tsc --noEmit — 0 errors
Left is the model's real answer, shortened. Right is the reference that does compile, kept in the repo. Both went through the compiler against the same installed copy of @tanstack/react-table 9.1.2.
Eight libraries, scored the same way
Every score is claude-opus-5 on 10–15 small coding tasks. The number is how many answers compiled out of how many it wrote, so 100 means every one did. Each tick below is one task.
v9 renamed the main hook & swapped the getCoreRowModel() family for a features map. The model wrote v8 every single time.
View scorecard →Inline callbacks still infer fine. It only breaks when you give a callback an explicit type — v7 removed the type names it reaches for.
View scorecard →Queries & $extends are clean. It still writes the v6 client setup and leaves out the driver adapter that v7 now requires.
View scorecard →Async cookies & headers are absorbed. The one miss is Next 16's new 2-argument revalidateTag().
View scorecard →Drops the deleted json() & defer() on its own. But meta() still takes the removed data argument. It's loaderData now.
View scorecard →Clean on everything it wrote, including the pinned apiVersion. It refused 5 of the 15 tasks outright, so this score covers 10.
View scorecard →Fully absorbed. Every v4→v5 rename written unprompted: gcTime, placeholderData, 'pending'.
View scorecard →Fully absorbed. The unified error option & the 2-argument z.record(), both unprompted.
View scorecard →Want your library on this board? Name any TypeScript package & I'll have a model write code for it, then compile every answer against the real thing. React Router 8 is here because someone asked.
Request a scorecard →Read the tasks. Read the errors.
Every score above is built from small coding jobs, like “fetch a user and show their name”. Each job names the function to write & nothing else. No option names, no docs, one attempt.
The answer goes into a project with the real package installed, then through tsc, the TypeScript compiler. A task passes only if it compiles.
Pick a library to see the jobs the model was given, which answers compiled, and the compiler's own error for the ones that did not.
Three steps. The compiler has the last word.
Ask
A model gets 10–15 small realistic jobs for one library. The prompt names the function to write, never the option names. One shot, no docs, no retries.
Compile
Each answer drops into a small project with the real package installed, then goes through tsc --noEmit. Same package your users install.
Count
Pass = it compiles. Nothing else counts. The failures are the compiler's own errors: exports that are gone, wrong arguments, options that got removed.
It measures the model, not your library
A low score doesn't mean the library is bad. It means the model's memory of it is out of date. TanStack Query v5 landed in 2023 and scores 100 — the model has seen years of it. Table v9 is fresh and scores 0.
So the number moves. The gap is widest right after a big release, narrows as models retrain, then opens again on your next one. It also shifts every time a new model ships.
Which is why this is worth watching rather than auditing once.
Schematic. It shows the shape, not measured data.
What this does not tell you
The numbers are small & the setup is narrow on purpose. Here is where it stops.
- One model
- Everything on this page is
claude-opus-5. Another model will score differently. - One shot, no docs
- The model gets the task & nothing else. Hand it the release notes and most of these misses go away.
- Small numbers
- 10–15 tasks per library. A 92 and a 93 are the same thing. Treat the gap between 0 and 100 as the signal, not the gap between 92 and 93.
- Compiles is not correct
tsc, the TypeScript compiler, checks that the API exists & the types line up. It never runs the code, so nothing here says the code does the right thing.- Refusals
- Stripe 22 is the one case where the model would not answer. It refused 5 of 15 tasks, so its 100 covers the 10 it did write.
Related findings
Scoring next
auth() is async now
Drizzle ORM — fast-moving schema & query API
Score my library
Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.
npx sdkproof <package> runs the whole thing on any typed package on npm. The source is here too — the compiler is the judge either way.