papers / test-weakening

Test Weakening: Measuring Repair without Eroding Tests

Pooria Arab · orcid 0009-0005-9964-6841 · 2026-10-29 · Published on Zenodo

abstract

A coding agent can make visible tests pass by changing their expected answers. Test Weakening measures whether repair preserves the evidence used to accept it. We study 30 synthetic JavaScript tasks with executable checks for original assertions, held-out behavior, and mutation strength. The study runs 11 scripted policies over 660 episodes including repeats. Reference repair earns 100.0 [88.6, 100.0] percent full success. Expected-answer rewriting earns 100.0 [88.6, 100.0] percent public-test success but 0.0 [0.0, 11.4] percent full success. Brackets are 95 percent Wilson intervals over tasks. A semantics-preserving test reformat earns 0.0 [0.0, 11.4] percent full success, exposing a lexical preservation limit. A separately labelled simulated population sweep compares public-only, original-test, and full acceptance rules. It measures scoring sensitivity under fixed behavior, not changes in learned agents. No learned model, human study, or production deployment is evaluated.

benchmark: benchmark-test-weakening. submitted to: none recorded.

citation

Arab, Pooria (2026). Test Weakening: Measuring Repair without Eroding Tests. https://doi.org/10.5281/zenodo.23752433

bibtex

@misc{test-weakening,
  author = {Arab, Pooria},
  title = {Test Weakening: Measuring Repair without Eroding Tests},
  year = {2026},
  doi = {10.5281/zenodo.23752433},
  url = {https://pooriaarab.com/papers/test-weakening},
}

venues, counts and the software records live on papers.