Can a 9-billion-parameter model beat a frontier model at reading contracts?

Finding the needle in a contract

CUAD is 510 real commercial contracts, hand-annotated by lawyers, marked up across 41 clause types — anti-assignment, cap on liability, non-compete, and so on. The job: hand the model a full contract and one clause type, and ask it to find that clause, verbatim, if it's there.

Every answer is a small JSON object. Two ways to fail: say a clause exists when it doesn't (or miss one that does), or find the right clause but misquote it.

SYSTEM: You are a contract analyst.
Given a contract and a clause type,
return JSON: {"found": bool, "quote": string}.
The quote must be an exact, verbatim
excerpt from the contract, or "" if
not found.

CONTRACT: "This Distributor Agreement
is made effective as of the 31st day
of March, 2000, by and between
Airspan Networks Incorporated..."
[...11,400 more tokens...]

CLAUSE TYPE: Agreement Date

{
  "found": true,
  "quote": "31st day of March, 2000"
}

Four contenders, one blind test

Same contracts, same prompts, same scoring — the only variable is the model.

The fine-tune wins — on a frontier model's home turf

On a 1,639-question paired sample scored across all four active models:

One frozen recipe, one reported run

All results in this article use seed 43: the same data pipeline, prompt, scorer, and paired question set for every active model.

The active frontier comparison uses the identical 1,639-question sample for base, fine-tuned, Luna, and Terra. The fine-tune used LoRA over 4,058 examples for 3 epochs, at about $8.67 and 2.2 H100-hours as recorded by the platform ledger.

The checks behind the numbers

A benchmark win is only as good as the design underneath it. Here's what was — and wasn't — done to keep this one honest.

• All 41 clause types scored, all 101 held-out contracts, at natural clause prevalence — no category selection.
• Contract-level split: the same document never appears in both train and test, audited for near-duplicates — max 8-word-shingle Jaccard similarity 0.256 against the training set. In law, template reuse is the deployment distribution; the binding check is document-level split integrity, and it's enforced here.
• One identical zero-shot prompt for every model. No per-model prompt tuning.
• Paired design: every model answers the same questions, so any difference is the model, not the test.
• gpt-5.6-luna rejects temperature 0 and was run at its forced default — a vendor-imposed decoding difference, disclosed rather than hidden.
• One parser, equal leniency for every model. An invalid or unparseable output always scores against whichever model produced it.
• Contract-clustered bootstrap confidence intervals, not naive per-question resampling: fine-tuned beat base in 1000 / 1000 paired resamples (detection F1 +0.099 [+0.080, +0.120], span F1 +0.204 [+0.178, +0.234]).
• All raw model outputs are saved and re-scorable. Costs come from the Overmind platform's billing ledger; frontier prices come from published rate cards.

What the difference looks like on the page

Same contract, same clause type, same instructions — different sense of where the clause actually ends.

Same accuracy tier, your own weights

The cost figures below apply measured tokens from the active paired sample to the relevant published rates. Terra is much more expensive on this workload, while the case for owning the 9B weights isn't only about the sticker price: you hold the weights, avoid someone else's rate limits, and keep contract text on your own infrastructure.