Stripe 22 scores 100. The model refused a third of the tasks.
Every task Claude Opus 5 was willing to write compiles clean against stripe@22.4.0 — including all three that probe what v21 and v22 actually changed. But five of the fifteen it declined outright, returning no code at all. Those are not failures. Nothing was measured, and the score below covers only what ran.
These returned stop_reason: "refusal" — a thinking block and no text. Not a failure, not drift, and not evidence that Stripe's API is hard to use. Simply nothing to measure. Re-sampling the identical prompt four times did not clear them.
Measured over 10 trials each, on the exact prompts this scorecard uses.
Task
Refused
What it was written to probe
payment-intent
10 / 10
automatic payment methods, client secret
auto-paginate
10 / 10
async iteration over list()
connect-account
10 / 10
v22: stripeAccount as a second-argument option
per-request-key
10 / 10
v22: apiKey moved into RequestOptions
expand-customer
9 / 10
the expand parameter
It is specific to this library, not to the harness. The same rig, same prompts, same model, run against Zod's task set: 0 refusals out of 100. Against Stripe's full set: 62 out of 150, 41.3%.
And it is not about missing context. The obvious explanation was that these prompts arrive with no project, no ownership, no stated reason — so "list another account's customers" reads like the code half of a fraud task. Each of the five was rewritten with one clause of ownership context ("our platform onboards sellers as connected accounts, for the seller's own dashboard…") and run as a paired A/B, both variants interleaved in the same batch. Every pair came back 10/10 refused on both arms. The hypothesis is wrong: the trigger is the shape of the operation, not the absence of a reason.
Clean
What it gets right — 10 of 10
Every version-specific probe that ran, passed.
Stripe ships breaking majors fast — v21 and v22 landed eight days apart — and the three tasks written specifically to catch that drift all compile clean. The model writes the exact pinned apiVersion literal the installed SDK expects, where any remembered older string is a type error. It treats decimal_string fields as Stripe.Decimal, which v21 changed from string. And it puts idempotencyKey in the second argument, not mixed into params — the v22 separation that removed the old blended call shape.
customers.createcheckout.sessionsrefunds.createinvoices.finalizeInvoicewebhooks.constructEventStripeCardError narrowingexact pinned apiVersion literalStripe.Decimal, not stringRequestOptions as the 2nd arg
The shape of it
Taking money refuses. Returning it does not.
Create a PaymentIntent: refused 10 times out of 10. Issue a refund: refused once out of 10. Same SDK, same money, opposite direction.
It holds across the set. Creating a customer never refuses; reading every customer always does. Pulling one customer's full record through expand refuses 9 times in 10. Acting on another account, or with another key, refuses every time. Configuring the client, verifying a webhook signature, finalizing an invoice, reading an exchange rate — all clean.
So the trigger is not "Stripe". It is a narrower shape: moving money toward you, reading customer data in bulk or in full, or acting with credentials that might not be yours. Which is a reasonable thing for a safety system to be careful about, and an awkward one for a benchmark that deliberately supplies no context — the same stripped-down prompt that makes the measurement clean is the one that arrives looking unexplained.
No opinions — just the compiler
How this was measured
15 realistic Stripe tasks → generated by claude-opus-5 → each written into a project with the real installed stripe v22.4.0 and type-checked with tsc --noEmit.
A task passes only if it compiles clean. Prompts name the resources (customers, checkout.sessions, webhooks, …) but never the signatures, so this measures what the model reaches for. Refusals are recorded and excluded — never counted as a pass or a failure.
Read this score as covering 10 of 15 written tasks. A third of the set produced no code, so this page reports a narrower measurement than every other scorecard here. The two tasks probing apiKey and stripeAccount as request options are among them — though that same v22 change is covered by the idempotency task, which ran and passed.
Same limit applies board-wide: tsc catches "this does not exist" and "wrong shape". It cannot catch code that compiles and means something else.
Is your SDK ready for AI agents?
Right after a major release, agents keep shipping your old API. SDKProof scores it, tracks it across model releases, and tells you exactly what to fix.
