papers / tool-errors

Tool Errors: Measuring Safe Recovery from Ambiguous API Failures

Pooria Arab · orcid 0009-0005-9964-6841 · 2026-10-29 · Published on Zenodo

abstract

An API error does not establish whether an operation took effect. Tool Errors measures safe recovery through stateful fault injection and replay-based scoring. Its 40 public episodes cover 6 fault families across 4 domains. We execute 4960 episodes using scripted controls and a simulated status-read omission sweep. The reference and response-aware controller each reach full credit on 40/40 tasks (95% Wilson interval 91.2-100.0%). Removing reads in a simulated ablation leaves 7/40 full-credit tasks (8.7-31.9%). At simulated read-omission probability 0.5, the mean full-credit fraction is 53.8% (50.6-56.9%, seed bootstrap). These are checks of the evaluation design, not trained-model results. No hosted model, human study, or production deployment is evaluated.

benchmark: benchmark-tool-errors. submitted to: none recorded.

citation

Arab, Pooria (2026). Tool Errors: Measuring Safe Recovery from Ambiguous API Failures. https://doi.org/10.5281/zenodo.23752436

bibtex

@misc{tool-errors,
  author = {Arab, Pooria},
  title = {Tool Errors: Measuring Safe Recovery from Ambiguous API Failures},
  year = {2026},
  doi = {10.5281/zenodo.23752436},
  url = {https://pooriaarab.com/papers/tool-errors},
}

venues, counts and the software records live on papers.