Evaluation That Deserves Trust
Deterministic tests first, calibrated judges second, cost third, human review for high risk edits.
A fixed order removes debate
Deterministic checks come first because they are cheap and strict. Unit tests, format checks, schema checks, and hidden transfer cases form the base. A patch that fails here stops at once.
Model judges come second for qualities that tests cannot see, such as clarity, redundancy, and task fit. Cost and latency come third so gains are weighed against price. A change that helps quality but doubles cost is marked as a trade. Human review comes last for high risk edits and release notes.
Judges need judges
Meta evaluation keeps judges honest. Agreement is tracked across cycles. Drift triggers recalibration. A judge that approves everything is treated as broken. A judge that blocks everything is treated as stuck.
Calibration sets with known good and known bad cases run on a schedule. The loop may improve workers and prompts, but it cannot silently lower the bar. Threshold moves are logged with reasons and reviewer names.
Transfer proves generality
Held out cases stay hidden during search. The generator never sees them while it invents patches. Promotion runs them once with full logs. This simple rule blocks most forms of quiet gaming.
Cost trends sit beside quality trends. Token use, latency, and cache behavior are recorded per generation. A quality gain with falling cost is a strong win. A quality gain with rising cost needs an explicit trade note.
Verification hierarchy from strong to weak
| Signal | Strength | Role in promotion |
|---|---|---|
| Formal checks and full tests | Strongest | Required for every accept |
| Held out transfer cases | Strong | Required for model and risky edits |
| Calibrated model judges | Medium | Supports clarity and fit review |
| Uncalibrated self praise | Weakest | Never promotes alone |