What a row is

Each row in results/ is one cell: one graded attempt of one task by one harness configuration and one model. It records the suite, task and task_version; the harness name, version and configuration hash; the model's provider id; the verdict and partial-credit score; tokens (fresh input, cache reads, output); wall time; the run id and the Paddock commit it ran at; and its cost.

What is scored

• A verifier outside the agent's sandbox grades every attempt from its submission and the harness's own tool-call record. Only passed is a pass; failed, format_failed and no_submission count in n.
• Infrastructure failures and quarantines (agent_error, error, integrity_hold, skipped_no_taint_gate) are never scored. They carry excludedBecause: infra:<reason> and are counted in each board's excluded column.
• Rows graded before a fix that changed what a pass means carry the fix's tag, for example pre-GYMDOCS-01, and are not scored. Each tag is a grading changelog entry.
• Results compare only within one task version. A task whose content changes gets a new task_version; rows at different versions are never pooled, so a suite with rows at two versions has two boards.
• Rows from different harness configurations are different rows, even with the same harness name and model: the configuration hash is part of the row.

Statistics

Pass rate is passed / n with a 95% Wilson score interval (z = 1.96). Mean is partial credit, the fraction of a task's sub-checks passed, averaged over scored attempts that report one. Boards rank by pass rate, then by the interval's lower bound. Most cells are small (n of 4 to 18): overlapping intervals are not a ranking.

Tokens and cost

• Tokens are what the harness reported: fresh input, cache-read input and output (reasoning included). A count nobody measured is null, never zero.
• Tokens per pass is the tokens of every scored attempt, failed attempts included, divided by passes. It is shown only when every scored attempt in the row was measured.
• A row's cost is its tokens priced at the dated price table its results document names. The table gives each price a source: provider-invoice-reconciled (checked against a provider invoice), published-rate (the provider's list price) or unknown. Cost per pass is shown only when every scored attempt is priced; any missing token count, missing price or unknown source makes it "unknown", never a number. Gateway or proxy cost estimates are never a source.
• Costs are API-equivalent: a run on a subscription plan is priced at the API rate.

Price table 2026-10-10

Per-token rates in USD per million tokens: input is fresh (uncached) input, cache is cache-read input, output includes reasoning tokens. Cache writes count as input at the input rate, so a provider's cache-write premium (Anthropic: 1.25x for 5-minute writes) is not included. A model absent from this table, or with source unknown, is priced unknown, never zero. Rates are what an API customer would pay; a run on a subscription plan is still priced at these API-equivalent rates. Proxy or gateway cost estimates are never a source.

Price table 2026-10-18

Per-token rates in USD per million tokens: input is fresh (uncached) input, cache is cache-read input, output includes reasoning tokens. Cache writes count as input at the input rate, so a provider's cache-write premium (Anthropic: 1.25x for 5-minute writes) is not included. A model absent from this table, or with source unknown, is priced unknown, never zero. Rates are what an API customer would pay; a run on a subscription plan is still priced at these API-equivalent rates. Proxy or gateway cost estimates are never a source.

Price table 2026-10-25

Per-token rates in USD per million tokens: input is fresh (uncached) input, cache is cache-read input, output includes reasoning tokens. Cache writes count as input at the input rate, so a provider's cache-write premium (Anthropic: 1.25x for 5-minute writes) is not included. A model absent from this table, or with source unknown, is priced unknown, never zero. Rates are what an API customer would pay; a run on a subscription plan is still priced at these API-equivalent rates. Proxy or gateway cost estimates are never a source.

Harness fairness

Before results are published, every harness's tool and permission configuration is audited against its vendor defaults with this checklist, and the audit is recorded here.

Each finding is pass, fail or not-verifiable. An audited harness has evidence for every check passing; it remains provisional until a maintainer signs off. A pending harness has unresolved checks. The audit date records the evidence review, not sign-off. Vendor documentation describes the defaults on that date; without a recorded CLI version it cannot establish historical defaults. Current runner code does not establish how an imported run executed.

1. tools: The harness exposes its vendor-default tool set: no tool removed, none added, and native web access left at the vendor default and recorded.
2. permissions: The harness runs in its vendor's non-interactive mode with equivalent authority; the sandbox boundary is the container, the same for every harness.
3. model: The model the harness actually ran, read from its own telemetry, is the one recorded (CCMODEL-01).
4. reasoning: The reasoning or effort level is recorded, and is the vendor default or the same named level across compared rows.
5. prompt: Every harness gets the same task prompt and no harness-specific instructions beyond its vendor default system prompt.
6. limits: Wall-clock timeout, container CPU and memory, and turn or token limits are the same for every harness, or the vendor default and recorded.
7. telemetry: The harness's tool-call record is captured whole, never truncated, so a read-evidence check cannot fail on capture (OCEXPORT-01).
8. transport: When a model is reached through a proxy or gateway, any request rewriting it does is part of the recorded configuration (KIMIEMPTY-01).
9. provenance: Harness CLI version and configuration hash are recorded with every result.

antigravity: checklist evidence

claude-code: checklist evidence

codex: checklist evidence

opencode: checklist evidence

pi: checklist evidence

Harness configurations

A row's configuration hash is the SHA-256 of this canonical JSON descriptor.

Sources

The first results were imported from the eval bench that preceded Paddock, keeping each source's exclusions. Those rows carry no token or wall-time measurements and no harness version, so their tokens and cost read "not measured" and "unknown". Published aggregates with no per-attempt rows are not imported: only rows can be recomputed.

Reproduce this site

git clone https://github.com/laceyenterprises/agentpaddock
cd agentpaddock
python3 scripts/leaderboard.py validate
python3 scripts/build_site.py

To submit results or a vendor correction, see CONTRIBUTING.