Benchmarks And Transfer Checks
Which public benchmarks prove what, plus how to hold out transfer cases that block quiet gaming.
What each benchmark proves
Coding benchmarks such as SWE bench and Polyglot test whether an agent can read, edit, and repair real code under tests. Reasoning sets such as GSM8K and MATH test step by step problem solving. Decision suites such as HotPotQA and AlfWorld test search plus multi step choices. HumanEval era code tasks test single shot generation with hidden tests.
No single suite proves recursion. A benchmark proves a skill at one moment. Recursion is proven by the diary across cycles, where the same suite plus fresh transfer cases move together.
Holding out transfer the right way
Split every fixture into public checks the generator may see and hidden checks it never sees during search. Promotion runs hidden checks once with full logs. A patch that passes public checks but fails hidden ones returns to the archive with its residue.
Rotate hidden sets on a schedule and track the gap between public and hidden scores. A widening gap means the loop is learning the school instead of the subject, so difficulty steps down and diversity checks tighten.
Benchmark map used across the atlas
| Benchmark | Skill under test | Atlas use |
|---|---|---|
| SWE bench | Real code repair under tests | Archive search validation |
| Polyglot | Multilingual code tasks | Second validation axis |
| GSM8K | Math word problems | Reasoning fixtures |
| ARC AGI | Visual pattern generalization | Transfer probe |
| HotPotQA | Multi hop search answers | Decision memory test |
| AlfWorld | Household multi step tasks | Reflection memory test |
| HumanEval | Single shot code generation | Classic baseline |