Open source · MIT
One rubric, every decision model.
Write the rubric once. Verdict asks Jev, Clef and any other decision model the same questions, and shows which one agrees with your labels. It runs on your machine, with your own keys.
$ verdict run -r intent.yaml -d cases.jsonl --models jev,clef,clef-flash
Wants a human × cases.jsonl: 4 case(s), 3 model(s), the same questions to each
model answered by agreed with labels failed input tokens time
jev typesafe/jev-1.14.0 7/8 (88%) 0 2,041 0.5 s
clef cloudflare/clef 7/8 (88%) 0 1,547 0.9 s
clef-flash cloudflare/clef-flash 8/8 (100%) 0 1,547 0.4 s
the models gave one answer on 7 of 8 question(s) they all answered; they differ on 1:
explicit · urgency expected medium jev low (0.67) | clef low (0.53) | clef-flash medium (0.11)The models you can use
Name any of them after --models. The rubric does not change.
| Name it | Model | Service | Checked against the live service | Images |
|---|---|---|---|---|
jev | Jev | TypeSafe | Yes | No, text only |
clef | Clef | Cloudflare Workers AI | Yes | Up to 4 |
clef-flash | Clef-flash | Cloudflare Workers AI | Yes | Up to 4 |
openrouter:<model> | For example typesafe/jev-1.13 | OpenRouter | Not yet | No |
openai:<model> | OpenAI's Decisions API | OpenAI | Not yet | No |
systemone:<model> | Any endpoint that takes the same request | Yours | You check it | No |
Each model is called with the key in your own environment. Rules, people and prompted models can judge a criterion too.
What you can hand it
One subject can be several of these. Each model receives what it can read.
| Input | On the command line | What the judge receives |
|---|---|---|
| Text | text:…, a text file, or - | The text |
| File | A .pdf, .html, .md, .txt or .json path | Its text |
| JSON | json:{…} | The document, checkable against a schema |
| Web page | A URL | The page's visible text; with --screenshot, a picture too |
| YouTube | A YouTube URL | The transcript |
| Image | A .png, .jpg or .webp path | The picture, for models that take images |
You write the rubric. Every model gets the same questions.
A rubric is criteria, scales and what each level means. Verdict turns it into the typed questions decision models take, so you keep one file instead of one integration per vendor.
name: Wants a human
criteria:
- id: wants_human
name: Asks for a person
scale: { type: binary, labels: ["no", "yes"] }
anchors:
"no": A question the bot can answer.
"yes": Asks for an agent or a human.
- id: urgency
name: Urgency
scale: { type: ordinal, labels: [low, medium, high] }
anchors:
low: No time pressure.
medium: Wants it soon.
high: Blocked now.intent.yaml, shortened from the example.{ "state": "please connect me to an agent",
"questions": {
"wants_human": {
"type": "noul",
"instructions": "… \"yes\" for this criterion?",
"criteria": { "true": "Asks for an agent or a human.",
"false": "A question the bot can answer." } },
"urgency": {
"type": "score",
"instructions": "… on the ordered levels …",
"criteria": [ "low: No time pressure.",
"medium: Wants it soon.",
"high: Blocked now." ] } } }No rubric edits to switch models
Name the models on the command line. The rubric file stays as it is, and each model's run is kept so you can compare any two case by case.
Answers you can compare
Every model's answer comes back on the rubric's own scale, with its probability. Agreement with your labels is printed with its denominator.
Checked against the live services
Jev, Clef and Clef-flash. Adapters for OpenRouter and OpenAI's decisions endpoints are in the repository and marked as not yet checked.
Three steps, on your machine
Nothing passes through us. Each model is called from where you run the command, with the key in your environment.
-
1 · Install
Get the command line
Node 20 or newer. The engine, the SDK and the server are in the same repository.
npm install -g @verdict-ai/cli -
2 · Write
A rubric and a file of cases
The command writes a starter rubric. Cases are lines of JSON; add the label you expect, where you have one.
verdict rubric init intent.yaml -
3 · Run
One run per model, one table
Agreement with your labels, tokens and time per model, then every case the models answered differently.
verdict run -r intent.yaml -d cases.jsonl --models jev,clef
Once you have chosen a model
The same rubric keeps working as the thing your product is judged by.
Versions that never change
Freeze a rubric and it gets an id made from its content. The same id is the same document on any machine.
Diff two versions, case by case
Fixed, regressed, moved close to flipping. A GitHub Action posts it on the pull request and fails on regressions.
Judges that are not models
Rules for what a rule can decide, people for what needs a person, and a hand-off to a person when the model is unsure.
Check it against people
Run a rubric over cases people already judged and see where it disagrees, before you trust it as a judge.
From code, not only the terminal
A TypeScript SDK, a REST API you can host, and an MCP server for agents. One engine behind all of them.
A server for teams
Keep every version where any machine can reach it by name, and say which one production runs. Self-host it from the repository.
Rather not run the server?
Verdict Cloud is the same server, hosted: sign-in for your team, roles, and a console that shows which rubric version each environment runs and who changed it. Your cases still never leave your machines.
It is in private preview, by invitation. Join the waiting list, or write to support@verdict-ai.dev.
