collection · building · 2026-10
helpful · science
benchmark
public AI benchmarks with an SDK, CLI, MCP server, and API
the problem
AI benchmarks often need a custom runner, report format, and integration before anyone can compare a model.
what i built
benchmark is a public family of packages named benchmark-<topic>. each one exposes the same SDK, CLI, MCP server, and API.
the parts
- benchmark-corethe shared runner, graders, adapters, reports, CLI, MCP, and API
- benchmark-pelicandraw an SVG of a pelican riding a bicycle
- benchmark-skateboardcode a playable skateboard trick scene
- benchmark-apology-pizzafix a wrong pizza order inside a refund limit
- benchmark-tool-errorstest retries and recovery after API faults
- benchmark-cost-per-taskmeasure dollars, retries, and tokens per verified result
- benchmark-mcp-safetytest malicious tools, shadowing, and least privilege
- benchmark-test-weakeningcatch agents that make tests pass by weakening them
- benchmark-checkout-consentcheck that a browser agent buys only with clear consent
- benchmark-honestycheck whether an agent admits the unknown after a tool fails
- benchmark-sycophancytest whether a model resists a wrong plan as harm rises
- benchmark-hundred-stepmeasure reliability across 25 to 100 dependent steps
- benchmark-ranking-stabilitymeasure how model rankings move with seeds and prompts