Ten jobs. Ten compiled.

zod 4.4.3 · claude-opus-5

AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.

SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.

This page is Zod, the schema library. Ten ordinary jobs — describe an object, parse it, pull the type back out — one shot each, no docs, no retries.

All ten compiled. The model wrote Zod 4's shapes, not Zod 3's, including the two renames it had the best excuse to get wrong.

This measures the model, not Zod.

100~~/100~~ 10 of 10 compiled

10 compiled 0 did not compile 0 refused

Model: claude-opus-5 Tasks: 10 Package: zod 4.4.3 Run: 26 July 2026 Pass: it compiles

What it could have got wrong

Zod 4 renamed two things a model would reach for from memory

Zod 4 folded the three separate error options into one called error, and made z.record() take a key type as well as a value type. Both are valid Zod 3 on the left and errors against the installed Zod 4.

The model wrote the right-hand column, unprompted. The prompts name the job, never the option names.

What a Zod 3 memory writes fails against zod 4.4.3

// custom message for a missing field
z.string({
  required_error: "Name is required",
});

// a map of string to number
z.record(z.number());

// an IP address
z.string().ip();

Every line here was correct in Zod 3. Against 4.4.3 the option name, the argument count and the method are all wrong.

What the model actually wrote zod 4.4.3

// one unified error option
z.string({
  error: "Name is required",
});

// key type, then value type
z.record(z.string(), z.number());

// the version-specific checks
z.string().ipv4();

tsc --noEmit — 0 errors

Reading the errors. TS2769 means no version of that call accepts those options. TS2554 means the wrong number of arguments. TS2339 means the method isn't there any more. All three are the compiler's own words, from a real run against zod 4.4.3.

It moved

This was 90 one model earlier

The same ten tasks on the previous model, Opus 4.8, scored 90 — 9 of 10 compiled. The one miss was the error option: it wrote required_error, the Zod 3 name, and the compiler rejected it.

Opus 5 was trained later and writes error. Nothing about Zod changed between the two runs. The library sat still and the score moved, which is the clearest way to say what this number is: it's a reading of the model's memory, taken against a fixed package.

Expect it to move the other way too. When Zod ships its next release that removes or renames something, the model's memory goes stale again and the score drops until models retrain.

All ten

What the model wrote

Marked ones are the version-specific probes — tasks written to catch a Zod 3 answer.

z.object .parse / .safeParse z.infer z.array + z.enum .optional().default() .email() nested objects z.record(key, value) — two arguments ipv4 check the unified error option

How this was measured

The compiler has the last word

Ten realistic Zod tasks, written by claude-opus-5, each dropped into a small project with zod 4.4.3 actually installed, then run through tsc --noEmit. A task passes only if it compiles.

One model: Everything here is claude-opus-5. Another model will score differently — Opus 4.8 scored 90 on this same set.
One shot, no docs: The model gets the task and nothing else. No release notes, no editor rules, no retries.
Small numbers: Ten tasks. A clean sweep of ten is a weaker claim than it looks — it rules out a common failure, not a rare one.
Compiles is not correct: tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code, so nothing here says a schema validates the right things.
Refusals: A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile, and refused tasks drop out of the denominator. Zod had none. Stripe had five, which is why its 100 is out of 10 rather than 15.
It measures the model: 100 says claude-opus-5 knows current Zod. It says nothing about whether Zod is well designed.

This run was rebuilt on 6 August 2026 from the committed render of the original 26 July run, after a smoke test overwrote the result file. Every task passed in that run, so the counts are exact rather than reconstructed.

Elsewhere