papers / honesty

Benchmark Honesty: Recovery and Abstention after Tool Failure

Pooria Arab · orcid 0009-0005-9964-6841 · 2026-10-29 · Published on Zenodo

abstract

Honesty after tool failure requires both recovery and restraint. Benchmark Honesty tests this distinction with 60 fictional lookup episodes: 30 recoverable failures and 30 unavailable records. Its headline score combines evidence-backed answering with evidence-backed abstention. We evaluate 10 scripted controls and a confidence sweep through the benchmark environment and grader. The evidence-reading recovery script achieves balanced success 1.000 [1.000, 1.000]; each component is 1.000 [0.886, 1.000]. Always-abstain and no-retry controls achieve balanced success 0.000 [0.000, 0.000]. Balanced-success intervals use 95% item bootstrap bounds; component rates use 95% Wilson bounds. Changing confidence leaves balanced success unchanged while changing Brier loss. All 14 repeated conditions agree in their semantic traces. These results validate selected scoring distinctions, not model honesty in deployment. No learned model or human participant is evaluated.

benchmark: benchmark-honesty. submitted to: none recorded.

citation

Arab, Pooria (2026). Benchmark Honesty: Recovery and Abstention after Tool Failure. https://doi.org/10.5281/zenodo.23752399

bibtex

@misc{honesty,
  author = {Arab, Pooria},
  title = {Benchmark Honesty: Recovery and Abstention after Tool Failure},
  year = {2026},
  doi = {10.5281/zenodo.23752399},
  url = {https://pooriaarab.com/papers/honesty},
}

venues, counts and the software records live on papers.