Every answer compiled.
A third of the questions got no answer.
stripe 22.4.0 · claude-opus-5
AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.
SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.
This page is Stripe. Fifteen ordinary payments jobs — create a customer, refund a charge, check a webhook signature — one shot each, no docs.
The model wrote code for 10 of the 15, and all 10 compiled. On the other five it declined to answer at all. That's not a failure and it's not a pass, so this page reads 100 out of 10, not out of 15.
This measures the model, not Stripe.
What a refusal is, and why it changes the denominator
On five of the fifteen tasks the model returned no code. Not wrong code — none. The response came back with a stop reason of refusal and an empty body.
There is nothing to compile, so there is nothing to score. Counting a refusal as a failure would blame Stripe's API for something the API was never shown. Counting it as a pass would be worse. So refused tasks drop out of the denominator, and the 100 above covers the 10 that produced code.
| The two rates from the same run | Rate | Out of |
|---|---|---|
| The code it wrote uses the current API | 100% | 10 answers that produced code |
| Tasks that came back with working code | 67% | 15 tasks asked, refusals included |
Both are true of the same run, and either one alone misleads. The 100 says every line of Stripe code the model wrote uses the current API. The 67 says a third of the job came back empty. Squashing them into one number hides which of the two you are looking at, so both are published here and in scores.json.
Both come from ten to fifteen samples. A clean 100% at that size is a weaker claim than it looks — it rules out a common failure, not a rare one.
Five tasks the model would not write
Re-sampling the identical prompt did not clear them. To see how consistent that was, each of the five was run ten more times, one attempt per run.
| Task | Refused | What it was written to check |
|---|---|---|
| payment-intent | 10 / 10 | automatic payment methods, client secret |
| auto-paginate | 10 / 10 | async iteration over list() |
| connect-account | 10 / 10 | v22: stripeAccount as a second-argument option |
| per-request-key | 10 / 10 | v22: apiKey moved into RequestOptions |
| expand-customer | 9 / 10 | the expand parameter |
Ten trials is ten trials
A 10-out-of-10 row is not proof of a 100% refusal rate. It's consistent with anything above about 72%. The tasks not listed refused 0 of 10, which is consistent with anything up to about 28% — so read those as "did not refuse in ten tries", not "never refuses".
It's this library, not the setup
Same harness, same model, same style of prompt, run against the seven other libraries on this board: zero refusals, on any task, on any of them. Stripe is the only one that produces any.
It isn't about missing context either
The obvious explanation was that these prompts arrive with no project and no stated reason, so "list another account's customers" reads like the code half of a fraud job.
Each of the five was rewritten with one clause of ownership — "our platform onboards sellers as connected accounts, for the seller's own dashboard…" — and run as a paired test, both versions mixed into the same batch. Both versions refused at least 9 times out of 10, and four of the five refused 10 out of 10 either way. The reason isn't missing context. It's the shape of the operation.
Every version-specific probe that ran, passed
Stripe ships releases that remove or rename things quickly — v21 and v22 landed eight days apart. Three of these tasks exist purely to catch a model writing the older shapes. All three compiled.
The model wrote the exact pinned apiVersion string the installed SDK expects, where any remembered older string is a type error. It treated decimal_string fields as Stripe.Decimal, which v21 changed from plain string. And it put idempotencyKey in the second argument rather than mixed into the parameters, which is the v22 split.
Taking money refuses. Giving it back does not.
Create a PaymentIntent: refused 10 times out of 10. Issue a refund: refused once in 10. Same SDK, same money, opposite direction.
It holds across the set. Creating a customer never refuses; reading every customer always does. Pulling one customer's full record through expand refuses 9 times in 10. Acting on another account, or with another key, refuses every time. Configuring the client, checking a webhook signature, finalising an invoice, reading an exchange rate — all clean.
So the trigger isn't "Stripe". It's narrower: moving money toward you, reading customer data in bulk or in full, or acting with credentials that might not be yours. Reasonable things for a safety system to be careful about, and awkward for a measurement that deliberately supplies no context. The same stripped-down prompt that makes the number clean is the one that arrives looking unexplained.
The compiler has the last word
Fifteen realistic Stripe tasks, written by claude-opus-5, each dropped into a small project with stripe 22.4.0 actually installed, then run through tsc --noEmit. A task passes only if it compiles. The prompts name the resource — customers, checkout sessions, webhooks — never the signature.
- One model
- Everything here is
claude-opus-5. Another model will score differently, and may not refuse at all. - Two retry policies
- The scored run re-asks a refused prompt up to 4 times at identical wording before recording a refusal. The 10-trial table above is 1 attempt per trial. Those measure different things, so neither rate is quoted without saying which it was.
- Narrower than the rest
- A third of the set produced no code, so this page covers less ground than the other scorecards. Two of the refused tasks probe
apiKeyandstripeAccountas request options — though that same v22 change is covered by the idempotency task, which ran and passed. - Refusals
- A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile. Refused tasks drop out of the denominator, which is why the 100 here is out of 10 rather than 15.
- Compiles is not correct
tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code, so nothing here says a charge would go through.- It measures the model
- Both numbers are statements about
claude-opus-5— what it remembers of Stripe's API, and what it is willing to write. Neither is a statement about Stripe.
Related findings
Other libraries, scored the same way
v9 renamed the main hook. The model wrote v8 every time.
Breaks the moment a callback gets an explicit type.
Queries are clean. It still builds the client the v6 way.
meta() still reads the removed data argument.
Score my library
Name any TypeScript package & I'll run it. Or do it yourself — it's all open source.
There's no npm package. Clone the repo, point it at a library, run it. The compiler is the judge.