Scorecard: fine-tuned 9B vs base 9B vs two frontier models
We fine-tuned a small open model — Qwen3.5-9B — to pull clauses out of commercial contracts, then scored it against the same 9B model untouched and two frontier models from closed APIs: GPT-5.6-luna and GPT-5.6-terra.
This dashboard uses the active, like-for-like 1,639-question paired sample. The fine-tuned model and base model also have complete full-run results; Sonnet 5 is intentionally excluded from this active comparison.
What exactly was tested?
- The data
- CUAD — 510 real commercial contracts read and marked up by lawyers across 41 clause types (anti-assignment, cap on liability, non-compete, and so on). Scoring is against those lawyers' highlights.
- The task
- Give the model a full contract plus one clause type. It returns
{"found": true/false, "quote": "…"}— the verbatim passage, or nothing if the clause is absent. - The fine-tune
- LoRA over 4,058 training examples, 3 epochs, ~2.2 H100-hours, about $9 as billed by the platform for seed 43. No prompt tricks: all four active models get the identical zero-shot prompt.
- The test set
- 101 contracts held out at the contract level, so no question from a training contract can leak in.
Methodology & fairness checklist
- Coverage
- All 41 clause types scored, all 101 held-out contracts, at natural clause prevalence — no category selection, no cherry-picking.
- Split integrity
- Contract-level split: the same document never appears in both train and test. Audited for near-duplicates — max 8-word-shingle Jaccard similarity 0.256 against the training set. In law, template reuse is the deployment distribution; the binding check is document-level split integrity, and it is enforced here.
- Prompting
- One identical zero-shot prompt for every model. No per-model prompt tuning.
- Pairing
- Every model answers the same questions, so any difference is the model, not the test.
- Decoding
- gpt-5.6-luna rejects temperature 0 and was run at its forced default (1). That vendor constraint is disclosed, not hidden.
- Scoring
- One parser, equal leniency for every model. An invalid or unparseable output always counts against whichever model produced it.
- Confidence
- Contract-clustered bootstrap, not naive per-question resampling. Fine-tuned beat base in 1000/1000 paired resamples — detection F1 +0.099 [+0.080, +0.120], span F1 +0.204 [+0.178, +0.234].
- Reproducibility
- All raw model outputs are saved and re-scorable. Costs come from the Overmind platform's billing ledger; frontier prices come from published rate cards.
| Metric | Fine-tuned 9B | Base 9B | GPT-5.6-luna | GPT-5.6-terra |
|---|
Amber and ★ mark the best model in each row. For false alarms, cost and latency, lower is better. Terra’s cost is an estimate using its published API input/output rates and this run’s measured tokens.
Seed 43 — complete four-model comparison
All figures on this dashboard come from the seed-43 run: 1,639 paired questions for FT, base, Luna, and Terra, with full-run coverage for FT and base.
One metric at a time, with the line to use
The Overview shows every metric at once. Here you take them one at a time — what each measures, and a sentence you can lift straight into a deck.
Five views of the trade-off
Every chart below is drawn from the same 1,639 paired questions, except the last, which uses the full held-out run.
Where does fine-tuning help most?
Base vs fine-tuned 9B on every clause type, scored over the full held-out run (~4,100 questions). Click any column heading to sort, or filter by name.
| Clause type | Base 9B | Fine-tuned 9B | Change | GPT-5.6-luna | GPT-5.6-terra |
|---|
Base and fine-tuned are scored on the full held-out run; Luna and Terra are scored on the same 1,639-question paired sample. Category rows are indicative because the frontier sample is smaller. The Change column is fine-tuned minus base.
What would this cost you per month?
One question = one contract read for one clause type. A team reviewing 250 contracts a month against all 41 clause types asks about 10,000. The scale is logarithmic, so each tick is 10×.
Luna and Terra figures use exact measured token usage priced at published OpenAI API rates ($0.20/$1.20 and $2/$12 per 1M input/output tokens, respectively); no cache discount is assumed. The 9B figure uses published open-model reference rates, not an Overmind serving bill. Training was a one-time $8.67 reference from the earlier ledger.
What the difference looks like on the page
Four questions from the held-out contracts. Left is the lawyer's highlight; the rest is what each model returned, unedited. The base model usually finds the right region and then quotes far too much of it.
The frontier model finds more — and flags more that isn't there
On the active 1,639-question paired sample, Luna and Terra both find most real clauses, but each raises more false alarms than the fine-tuned model. The fine-tuned model is the most precise of the four active models; see the scorecard for the exact counts.
- — This page reports seed 43 only, with the same prompt and scorer across all four active models.
- — gpt-5.6-luna was run at a forced temperature of 1 with the same zero-shot prompt as everyone else; no frontier-specific prompt tuning was attempted.
- — The four-model comparison uses a 1,639-question paired sample (cost-limited), not the full ~4,100-question run.
- — Luna and Terra costs are token-based estimates using the published OpenAI API rates; the dashboard shows the token counts and assumptions.
- — Terra is included in the active comparison. Its quality and latency are measured, and its cost is estimated from the published API rate card.