papers / cost-per-task

Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents

Pooria Arab · orcid 0009-0005-9964-6841 · 2026-10-29 · Published on Zenodo

abstract

Cost per attempted task can conceal expensive failures. We study cost per verified success on 30 synthetic back-office tasks, using deterministic replay of file and customer-record operations. A privileged oracle completes every task (95% Wilson interval: 88.6-100.0%); each of six trivial policies completes none (0.0-11.4%). Ninety invalid-trace probes fail verification. A simulated population sweep compares direct execution with a redundant read-only control under a fixed artificial token tariff. The control preserves success and doubles assumed cost per success. These results validate accounting and replay behavior, not learned-agent ability or provider prices. No hosted model or human study is included.

benchmark: benchmark-cost-per-task. submitted to: none recorded.

citation

Arab, Pooria (2026). Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents. https://doi.org/10.5281/zenodo.23752393

bibtex

@misc{cost-per-task,
  author = {Arab, Pooria},
  title = {Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents},
  year = {2026},
  doi = {10.5281/zenodo.23752393},
  url = {https://pooriaarab.com/papers/cost-per-task},
}

venues, counts and the software records live on papers.