benchmark

the problem

AI benchmarks often need a custom runner, report format, and integration before anyone can compare a model.

what i built

benchmark is a public family of packages named benchmark-<topic>. each one exposes the same SDK, CLI, MCP server, and API.

how it is built

every package is deterministic and reproducible, and each one was independently reviewed before release.

the datasets on hugging face

the parts

• benchmark-corethe shared runner, graders, adapters, reports, CLI, MCP, and API
• benchmark-pelicandraw an SVG of a pelican riding a bicycle
• benchmark-skateboardtest the math behind two dozen skateboard tricks
• benchmark-apology-pizzafix a wrong pizza order inside a refund limit
• benchmark-tool-errorstest retries and recovery after API faults
• benchmark-cost-per-taskmeasure dollars, retries, and tokens per verified result
• benchmark-mcp-safetytest malicious tools, shadowing, and least privilege
• benchmark-test-weakeningcatch agents that make tests pass by weakening them
• benchmark-checkout-consentcheck that a browser agent buys only with clear consent
• benchmark-honestycheck whether an agent admits the unknown after a tool fails
• benchmark-sycophancytest whether a model resists a wrong plan as harm rises
• benchmark-hundred-stepmeasure reliability across 25 to 100 dependent steps
• benchmark-ranking-stabilitymeasure how model rankings move with seeds and prompts
• benchmark-latency-budgetscore an answer only when it lands before the deadline
• benchmark-crm-truthfulnesscheck that a sales agent claims only what the evidence supports
• benchmark-instruction-hierarchykeep user and system rules when context fights back
• benchmark-memory-conflictresolve changing preferences and disagreements in memory
• benchmark-accessibility-fixfix a broken page so keyboard, screen-reader, and contrast checks pass
• benchmark-silent-constraintsmeet the goal without breaking rules nobody mentioned
• benchmark-rollback-recoverycheck that an agent notices, repairs, and discloses its own mistakes
• benchmark-schema-mutationcall the current tool schema after it migrates
• benchmark-payment-delegationtest spend limits, allowlists, and disputes for an agent with a card