Reference-fix scorer

The judge is an LLM — expensive, non-deterministic, sometimes overconfident. Correctness gets a second opinion from the reference-fix scorer: a deterministic diff comparison, no model calls involved.

What it measures

Given the scenario’s REFERENCE_FIX.md and the run’s final git diff HEAD, the scorer emits:

{
  "value": 0.812,
  "structural": 1.0,
  "textual": 0.65,
  "matched_files": ["store/pricing.py"],
  "missing_files": [],
  "extra_files": []
}

• structural — file-set Jaccard: matched / (matched + missing + extra). 1.0 means the agent touched exactly the files the reference touches; 0.0 means no overlap.
• textual — mean character-level similarity across matched files’ added-line sets (via similar::TextDiff::from_chars). Character-level rather than line-level so a semantic-equivalent reword still scores > 0.
• value — weighted combination. Default is 0.6 * structural + 0.4 * textual.
• matched / missing / extra files — for narrating what the agent got right, what it missed, and what it spuriously changed.

Why it matters

The judge sees this in AUTHORITATIVE_FACTS and is instructed to weight correctness + minimality against it. Two consequences:

• Correctness ceiling — the judge can’t score correctness higher than the scorer supports. Bounds a common failure mode where the judge sees test_passed=true and gives an easy 9/10 to an over-scoped diff.
• Minimality signal — extra_files catches “the agent also refactored something adjacent” without needing the judge to spot it in the diff text.

The cross-layer-tax story

The v0 Python-era archive had 7 runs of cross-layer-tax. All 7 took the proximate fix at apply_tax (which passes some tests but leaves others red), and none touched the reference-fix files rates.py + display.py. Manual code review noticed. The scorer, when we backfilled it, made it a first-class metric:

Scenario	Archive avg score
asd-vs-baseline	0.812
inheritance-bug	0.764
audit-migration	0.752
cross-layer-tax	0.063

That 0.063 is the money finding. Rust-era gpt-4o-mini runs now score ~0.75 on that same scenario — either the model got better at cross-layer reasoning or the harness’s tool packs route it more effectively, but either way we can now measure it automatically.

Limitations

• Semantic equivalence — two fixes that behave identically but look nothing alike (e.g., raw * (1 - pct) vs raw - raw * pct) will score low on textual similarity even though correctness is fine. The judge’s justification field is where you catch that.
• Diff order sensitivity — the scorer is unordered on files but the character-level ratio is order-sensitive. A refactor that reorders lines will score lower than intuition suggests.
• Multi-hunk files — currently one hunk-set per file. A file with independent bugs and independent fixes gets aggregated.

Fixing any of these is a scorer PR away — see crates/crucible-scorer/src/score.rs.