Calibrating a judge for legal work

It's becoming increasingly common for us to do a first pass with AI before asking lawyers for advice, but can you really trust an AI's legal advice?

Over the last month, I partnered with Legal Benchmarks to explore this question. Legal Benchmarks' community of lawyers contributes legal requests, examples of AI failures, and tasks for model evaluation. I helped improve their task definitions, environment, and graders for an updated leaderboard.

The strongest frontier model, Astra, achieved a Criterion Pass² rate of 88%. With the new evaluation complete, I want to share how we designed it and some of the lessons I learned along the way.

The challenge of tacit knowledge

When I first started designing legal AI tasks, I had an "engineer-brain" instinct: if law is code for humans, couldn't we just reduce law to code and create unit tests to make it verifiable? Why not just write contracts in something like Prolog? It turns out that's a naive understanding of a lawyer's job.

Anna Guo, founder of Legal Benchmarks, taught me that even simple legal queries, like asking for a contract’s effective date, can require nuance: “when a contract is signed, the date it starts to be enforceable and when it is effective can all be different dates.”

Much of this knowledge is tacit: practitioners learn it from their seniors and through real-life cases, and it can be hard to spell out. To make the problem even more wicked, the right legal response often goes beyond a binary yes-or-no answer. It depends on context, risks, and other qualifications.

That nuance makes a deterministic legal verifier hard to build. Coding environments such as SWE-Gym and SWE-smith use executable tests to check proposed solutions. Our verifier needs to apply a lawyer's tacit judgment when evaluating an agent's answer.

Given these challenges, how would one define a task to test an AI's legal capabilities?

Task formulation

In Legal Benchmarks, we formulate a task as follows:

1. An agent receives a lawyer’s request as a prompt, which may come with source files.
2. The agent works through the provided context in a workspace and submits an answer, which may include output files.
3. We grade the answer on two metrics, substance and form, defined below.

Tasks include contract drafting and document extraction. We run them in the LB Harness, our shared environment for evaluating models on legal work.

The Legal Benchmarks team and I explored different tool and harness setups. Choosing the right affordances matters because harnesses can have a large impact on model performance.

One choice for benchmark design is to make the harness as close as possible to an end-user application, such as Claude Cowork or Codex. While this is useful for measuring real user experience, differences in tools and document handling make it harder to isolate a model's legal capabilities.

For this benchmark, we decided to focus on legal interpretation and reasoning. I built the LB Harness based on the setup from Harvey LAB, using the Tardigrade framework. I gave every model the same document reader, with OCR for scans and preserved edit markers. Each model gets a simple filesystem with basic shell commands via Just Bash and no network access.

We run each task in 2 fresh attempts, each with a time limit of 15 minutes. The LB Harness includes a clock that shows the remaining time in subsequent model requests.

To measure reliability, we follow τ-bench’s Passᵏ definition: the probability of completing a task successfully in every one of k independent attempts. Task Pass² applies this to whole tasks: the share of tasks that satisfy every substance criterion in both attempts. Criterion Pass² applies the same requirement to each criterion: we count a criterion only if it passes in both attempts, then divide that count by the total number of criteria. Form is scored separately.

With the LB Harness defined, the next step was to design a grading mechanism that captured what lawyers considered an acceptable answer.

Designing the judge

We first had to define what a good answer meant:

• Substance covers legal correctness, completeness, depth of reasoning, and whether the answer addresses the request. An answer can reach the right conclusion and still fail if it leaves out the reasoning or qualifications a lawyer needs.
• Form covers how the work is presented: its clarity, structure, and usability as a professional deliverable.

The simplest approach would be to give a judge these definitions and ask it to assign a score to each answer. Instead, inspired by CheckEval, we break our evaluation into a list of predicates. This structures the judge's output as a binary vector. When all elements of this vector are true, the answer passes the task.

We applied that approach to substance and form, expressing each requirement as a pass/fail criterion. Together, these criteria form the task's checklist. The leaderboard lets you compare Task Pass² and Criterion Pass² for substance. Form is the share of form checks passed, averaged over attempts.

Turning lawyer feedback into checklists

Legal Benchmarks already had binary substance criteria, alongside lawyer-written rubrics, open-ended reviews, and human decisions on cases escalated after models disagreed on prior runs. Some criteria bundled several requirements, repeated a check, or referred to evidence without attaching it for the judge.

My goal was to turn these semi-structured inputs into checks that an LLM judge could use reliably. Unlike a human who can fill in gaps when a criteria is underspecified, an LLM judge needs explicit expectations and the reference files necessary to reach a reliable verdict.

We worked toward a few principles for these criteria:

• Atomic. Each criterion should ask for one decision, with a clear pass/fail boundary. Compound criteria should be split into smaller ones where doing so preserves their meaning.
• Self-contained. Each criterion should spell out what counts as a pass, so the judge can evaluate the answer using a fixed set of reference files.
• Non-duplicative. Criteria should cover distinct requirements without checking the same result twice. Different requirements can still fail together.

To make this transformation scalable, I built meta-verifiers to check criteria against these principles. I then asked a frontier model to propose rewrites that followed them.

This sounded straightforward until I showed the initial transformation results to a lawyer. They rejected several rewrites where making the criteria cleaner changed their meaning. In one case, a rewrite replaced a required address in the criteria with placeholder text. It became clear that keeping an expert in the loop was necessary for crafting good verifiers.

What worked much better was separating mechanical transformations, such as straightforward atomic splits, from changes that required legal judgment, such as deciding whether a reference supported a required value. Even apparently mechanical rewrites had changed content, so we used a dedicated split checker and had engineers review the criteria diff.

The remaining changes needed lawyer review. To make the most of their limited time, I used model disagreement to flag uncertain cases for human escalation, borrowing from active learning.

The loop ran over several rounds, with each lawyer review informing the next proposals.

Screening the models

The next step was to choose an LLM to act as our judge. We screened 5 different models as potential judge candidates, using lawyer-labeled answers from Legal Benchmarks. Each review paired an AI answer with one substance criterion and a lawyer's PASS or FAIL decision about whether the answer satisfied that criterion. The reviewing lawyers were different from the lawyers who authored the tasks.

The review set contained 107 answer–criterion pairs across 31 tasks. We focused on difficult cases: answers where model judges disagreed and answers that came close to satisfying the requirements. These were useful places to look for weaknesses in a judge because a small omission or an ambiguous qualification could change the verdict. The scores below measure agreement with the lawyers on this selected set.¹

I tested the candidate judges with the same baseline instruction and compared their verdicts with the lawyer labels. A judge could disagree in either direction: it could reject an answer the lawyer considered sufficient, or accept one the lawyer considered incomplete. PASS recall measures how often it agrees with the lawyer on a PASS case, and FAIL recall measures how often it agrees on a FAIL case. Our main measure, balanced accuracy, is the average of those two recalls. This gives both kinds of decision equal weight even though the review set contains more FAIL labels.

Sol had the highest measured balanced accuracy, but the two recalls show why the choice of judge was less straightforward than that ranking suggests. GLM agreed with more of the lawyer-PASS decisions, while Sol agreed with more of the lawyer-FAIL decisions. In the disagreement analysis, GLM sometimes credited an answer for providing the main fact even when it omitted a required citation or qualification. Sol's stricter reading helped it catch those omissions, but it also rejected more answers that the lawyers had accepted.

GLM had the next-highest balanced accuracy after Sol, alongside its stronger PASS recall. That made it a promising starting point for calibration: it recognized many of the answers lawyers considered sufficient, and the disagreement analysis gave us a specific behavior to work on.

Its open weights also made it attractive for future reproducibility. We chose GLM for those reasons, then tested whether we could optimize its system prompt to catch omissions it was overlooking.

Refining the prompt

The disagreement cases suggested that GLM needed a more explicit rule for handling partially satisfied criteria. I added an instruction to require evidence for every part of the criterion, without filling in gaps for the answer. The aim was to catch incomplete answers while preserving its ability to recognize the answers lawyers had approved.

For this experiment, I split the same review pool into development and held-out tasks.¹ The prompt variants were written before scoring them on the development set. I selected the variant with the highest balanced accuracy that met a minimum PASS recall requirement, then compared it with baseline GLM and Sol on the held-out tasks. All three configurations below used high reasoning effort.

The revised instruction helped GLM catch more lawyer-FAIL cases, with a smaller loss in PASS recall. Baseline GLM already led Sol on this held-out subset, so tuning accounts for only the additional gain over baseline. Sol still caught more lawyer-FAIL cases, but rejected more than half of the lawyer-PASS cases. Its stricter judgments had helped on the development tasks and hurt on the held-out tasks; neither judge dominated across both sets.

Agreement between models alone would have hidden this tradeoff: in an earlier comparison, two judges agreed most of the time but still produced meaningfully different pass rates.²

I also experimented with two ways of calling the judge: one criterion per call, or a list of verdicts for all criteria in a task. We found no measurable accuracy advantage for either approach against the lawyer labels. Batching reduced the number of requests, but produced more lenient verdicts and was less reliable on long checklists. We chose one criterion per call.

Results

I ended up with 1,032 criteria after splitting and reviewing the requirements. I also turned open-ended lawyer-written descriptions of form into explicit checklists for how the work should be presented.

I then ran the model leaderboard with these criteria and the calibrated judge. A few results stood out:

• Similar Task Pass² rates hid different failure patterns. Astra had a Criterion Pass² rate of 87.9%, compared with 77.2% for Qwen Flash, yet both had the same Task Pass² rate. Qwen Flash also passed more whole attempts. This suggests more near misses from Astra and a spikier mix of complete answers and larger misses from Qwen Flash.
• The top score carried a steep cost premium. Opus's estimated collection cost per attempt was about 24 times Qwen Flash's, for a gain of 11.5 percentage points in Task Pass². Qwen's lower cost came with slower completion and lower form scores. Reliability, speed, and presentation each matter when weighing these tradeoffs.
• Faster did not mean cheaper. Astra averaged 92 seconds per attempt against Fable's 180, yet its estimated collection cost per attempt was about 51% higher. Latency and cost need to be measured separately for the workflow being evaluated.³

I also looked through selected answers to understand what the models got wrong:

• The best models still missed requirements in detailed drafts. A model could correctly draft a negotiated royalty term but leave its financial risk unexplained in the cover memo. Another left old cross-references in the final document. Getting the main terms right did not ensure that the accompanying advice and document edits were complete.
• Lower-scoring models struggled with basic extraction and finishing the work. Examples included reporting the wrong commitment amount, omitting a required party's role, and supplying figures unsupported by the source document. In one run, repeated document reads exhausted the tool-call allowance before the model returned an incomplete answer. These mistakes weren't exclusive to weaker models: a top performer also misread a count as an area measurement and still passed the relevant comparison check.

You can follow the results on the Legal Benchmarks leaderboard.

Working on legal AI?

If you're working on evaluating models for legal tasks or are interested in our dataset, you can reach me at contact@clavia.ai.