papers / benchmark-family

Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases

Pooria Arab · orcid 0009-0005-9964-6841 · 2026 · zenodo

abstract

We report 22 agent benchmarks built by AI coding agents on one shared TypeScript engine. No test job failed on the last commit of any of the 123 merged feature pull requests, and the LLM council that reviewed them approved 94 of them. A later probing review still found defects in all 22: each needed at least one fix round (median 3, maximum 9), the first 12 shipped a release whose changelog marks earlier scores incomparable (a sentence the fix briefs required), and at least 13 packages record a strategy that ignored the task and still earned credit. A preregistered check then played 8 generic trivial policies against the first 12 packages before and after the fixes: the best trivial score fell in only 2 packages, a null that suggests generic policies miss most of the logged defects. The approvals were real reviews: the council flagged defects that were then fixed, including the published hidden seed in sycophancy's pull request 20 and gaps in pelican's SVG scan. And 2 approved requests are documented holding defects a later review found: pelican's pull request 4, approved by the chair alone after the other members timed out, shipped a grader that scored a memorised SVG 1.0 on every task (issue 8), and sycophancy's pull request 20 was approved while a publishing leak and a grading-note claim the code did not back were still in the tree, both fixed after the approval. The artifact behind the suite is benchmark-core: one defineBenchmark call builds a task runner, a deterministic grader, and four runnable surfaces (SDK, CLI, MCP, HTTP) per package, and a mock provider separates an oracle script from a trivial script. We release the engine, the packages, the re-review protocol and review log, and the defect counts with their public sources. This release reports no real-model results and keeps no leaderboard; the registered real-model arm is still running.

venues, counts and the software records live on papers.