SDKProof All eight scores Source on GitHub

Scorecard · Prisma 7

Every query compiles.
The client setup doesn't.

@prisma/client 7.8.0 · claude-opus-5

AI coding assistants write library code from memory. When a library ships a big release that renames or removes things, the assistant keeps writing the old version. It reads fine & it doesn't build.

SDKProof measures how often that happens. It gives a model real coding jobs for one library, then compiles every answer against the real installed package with tsc, the TypeScript compiler. A task passes only if it compiles — no AI judges another AI.

This page is Prisma. Fifteen ordinary jobs — write a row, read across a relation, run a transaction — one shot each, no docs, no retries.

13 of the 15 compiled. Both misses are the same thing. The model still builds the client the Prisma 6 way, and Prisma 7 changed that.

This measures the model, not Prisma.

87~~/100~~ 13 of 15 compiled

13 compiled 2 did not compile 0 refused

Model: claude-opus-5 Tasks: 15 Package: @prisma/client 7.8.0 Run: 6 August 2026 Pass: it compiles

What went wrong

Two tasks failed, and it's the same mistake twice

Prisma 7 dropped the bundled query engine. The client now needs a driver adapter — a small object that holds the database connection and gets passed in when you build the client. The model has not learned that, so it writes the Prisma 6 setup.

Both panes below are real output. Left is what the model wrote, right is the version that compiles.

Task: build the client @prisma/client 7.8.0

import { PrismaClient } from "@prisma/client";

// nothing here sets up a database driver
const prisma = new PrismaClient({
  log: ["warn", "error"],
});

error TS2345: Argument of type '{ log: ("error" | "warn")[]; }' is not assignable to parameter of type 'Subset<PrismaClientOptions, PrismaClientOptions>'.

What Prisma 7 compiles @prisma/client 7.8.0

import { PrismaClient } from "@prisma/client";
import { PrismaPg } from "@prisma/adapter-pg";

const adapter = new PrismaPg({
  connectionString: process.env.DATABASE_URL,
});

const prisma = new PrismaClient({
  adapter,
  log: ["warn", "error"],
});

tsc --noEmit — 0 errors

Task: point the client at a database URL @prisma/client 7.8.0

const prisma = new PrismaClient({
  datasourceUrl: process.env.DATABASE_URL,
});

error TS2353: Object literal may only specify known properties, and 'datasourceUrl' does not exist in type 'Subset<PrismaClientOptions, PrismaClientOptions>'.

What Prisma 7 compiles @prisma/client 7.8.0

// the URL lives on the adapter now
const adapter = new PrismaPg({
  connectionString: process.env.DATABASE_URL,
});

const prisma = new PrismaClient({ adapter });

tsc --noEmit — 0 errors

Reading the errors. TS2345 means the options object is the wrong shape — the required adapter is missing. TS2353 means the option name doesn't exist at all: datasourceUrl worked in Prisma 5 and 6 and was removed in 7. Both are the compiler's own words, not mine.

The other 13

The query surface is clean

Every reading and writing task compiled on the first try, including the newer $extends API that replaced $use middleware.

Nothing here needed a hint. The model wrote nested relation writes, cursor pagination and $transaction without being told the option names — the prompts name the job, never the API.

create / update upsert findUnique / findMany where + orderBy nested relation writes include cursor pagination $transaction count $extends — computed field $extends — query middleware $queryRaw

Tested, not guessed

Do Prisma's own agent files fix it?

From v7.9, prisma init installs a set of files written for AI agents. I ran the same 15 tasks again with those files in the prompt, three times each. Nothing else changed. A Prisma maintainer asked for this, which is why it exists.

What was in the prompt	Score	Change	What moved
nothing — the score above	87	—	both setup tasks fail
`prisma-client-api`	87	+0	fixes neither, 0 of 3 tries on both
plus the v7 setup & upgrade docs	93	+6	fixes `datasourceUrl`, 3 of 3

The files work. The one an agent picks up first doesn't.

prisma-client-api holds a file called constructor.md. If an agent chooses a file by its name, that is the one it loads to build a client. Loading it moved the score by zero.

What actually fixed a task was prisma-postgres-setup/references/prisma7-client.md and prisma-upgrade-v7/references/driver-adapters.md. Both state the adapter requirement flatly. So the right answer is in the box, filed under the wrong label. That's a routing problem, not a writing problem.

One failure survives the whole pack, but not a single sentence.

With all of those files in context, a bare new PrismaClient() still fails. Measured on that one task at ten tries: 5 of 10 with the full 25 KB pack, and 10 of 10 when the adapter sentence is the only thing in the prompt. Same words. 174 characters instead of 25,000.

In one of the failing runs the model wrote the correct form into a comment and shipped the wrong one right under it:

It documented the rule, then broke it full 25 KB pack in context

/** Prisma 7 requires a driver adapter:
 *   const prisma = new PrismaClient({ adapter });
 */
const prisma = new PrismaClient({
  log: ["warn", "error"],
  errorFormat: "pretty",
});

error TS2345 — the adapter it just described is not there

The same model, given only the sentence 174 characters in context

const adapter = new PrismaPg({
  connectionString: process.env.DATABASE_URL,
});

const prisma = new PrismaClient({ adapter });

10 of 10 tries compiled

It had read the requirement well enough to write it down. It did not apply it.

What would move the number

Four changes, each one from a measurement above

Not general advice about writing good docs. Every item is something this run measured, and every one is checkable against Prisma's own shipped files.

1. `constructor.md` never mentions `datasourceUrl`

The failing task writes new PrismaClient({ datasourceUrl }), a field v7 removed. It is fixed only in the run that includes prisma-postgres-setup/references/prisma7-client.md — the one file in the whole set that names datasourceUrl at all. The pack an agent would reach for to build a client mentions it zero times.

So the fix arrived from a file about Postgres setup, which an agent working on client construction has no reason to open. Move the removed-field list into constructor.md and the answer sits where the question gets asked.

2. The file agents reach for hedges. The file they skip is direct.

constructor.md heads the field adapter (Required for the SQL provider workflow) — a condition the reader has to decide applies to them. prisma7-client.md says: "Adapter is mandatory: new PrismaClient() with no arguments throws."

Same requirement, two registers, and the plain one is in the file an agent is least likely to load. Stating the failure instead of the condition costs one sentence.

3. The strongest one: make it shorter

The adapter sentence on its own fixes bare new PrismaClient() 10 times out of 10. The same sentence inside the 25 KB pack fixes it 5 times out of 10. The wording never changed — only how much sat around it.

So this is not comprehension and not refusal. The model follows the instruction when it reaches it. React Router shows the same shape, harder: 10 of 10 from a single sentence, 0 of 10 from that sentence inside its own file.

Length is the obvious culprit and it does not hold up. A controlled run on Next.js put the same sentence at the end of 25 KB of unrelated docs and it still passed 10 of 10 — the numbers are here. Ship the short file, because that provably works. Why the full pack does not is still open.

4. Worth saying plainly: the docs themselves are good

Across nine libraries surveyed, Prisma's llms-full.txt is one of only three files a model can use without fetching anything else — 6,915 characters of prose, no link index. The same file from another library on this board is 5.7 MB. This is a routing problem, not a writing problem.

How this was measured

The compiler has the last word

Fifteen realistic Prisma tasks, written by claude-opus-5, each dropped into a small project with @prisma/client 7.8.0 actually installed, then run through tsc --noEmit. A task passes only if it compiles. Every failure above is the compiler's own error text.

One model: Everything here is claude-opus-5. Another model will score differently.
One shot, no docs: The model gets the task and nothing else — no AGENTS.md, no editor rules, no agent files. That's the point: it measures what a model reaches for unaided. Hand it the release notes and both misses go away.
Small numbers: Fifteen tasks. Read the gap between 0 and 100 as the signal, not the gap between 87 and 93.
Compiles is not correct: tsc, the TypeScript compiler, checks that the API exists and the types line up. It never runs the code, so nothing here says the code does the right thing.
Refusals: A refusal is neither a pass nor a fail — the model writes no code, so there is nothing to compile, and refused tasks drop out of the denominator. Prisma had none. Stripe had five, which is why its 100 is out of 10 rather than 15.
It measures the model: 87 is a statement about what claude-opus-5 remembers of Prisma, not about Prisma's design. A number this size mostly means the release is recent.