Paddock

Grading changelog

Every change to grading, tasks, harness capture or pricing that affects what a result means, with the comparability boundary it draws. Rows on the wrong side of a boundary keep their data and carry the tag shown; they are not scored.

date change what changed comparability boundary excluded

2026-10-18 READEV-01
grading All 15 review tasks using review_checks advance one task_version. Read evidence recognizes command prefixes and literal for-loop bindings, including value-preserving default/error parameter expansions. Shell comments and escaped dollars cannot supply loop expansion evidence; path-changing parameter operators remain unsupported. hard-review-multi-defect-tool-calls, review-corrupt-refresh-checkpoint, review-rescue-local-tip and review-stale-lock-recovery move from version 2 to 3; the other 11 review tasks move from version 1 to 2. Earlier results keep their original versions and are not pooled with the new versions; saved submissions can be regraded. —

2026-10-18 PDKRESULTS-01
infrastructure Early gym versions did not record a task's version. Such runs are version 1 when they ran before the first version change (2026-10-10) or when the task never changed; the few that cannot be placed are not imported. Results from the per-attempt export carry measured tokens and wall time; earlier imports keep null tokens. —

2026-10-15 HARDFIX-01
task Five new hard-fix -v3 task ids add contract interactions and declared plausible wrong fixes, checked through the task verifier. The regression tasks require submitted tests to distinguish three additional wrong implementations. The new -v3 ids start at task_version 1 on separate boards. Original and -v2 tasks, versions and published results stay unchanged; no previous row is regraded or excluded. —

2026-10-14 DOCSCHECK-02
grading docs-retry-behavior-diff, docs-snapshot-new-command and docs-gym-baseline-keys move to task_version 3. Fact checks accept negated alternatives, contrasting outcome-specific exit codes, subject-scoped boolean --json documentation, header-selected table defaults and equivalent model-qualified key forms. Table descriptions reject conflicting --json defaults and argument-error codes; table column delimiters do not count as baseline key separators. Version 2 results for these three tasks are not comparable with version 3 results. Saved submissions can be regraded under version 3; old results retain their original version. The other three docs tasks remain at version 2. —

2026-10-13 DOCSCHECK-01
grading All six docs tasks move to task_version 2. Required facts accept ordinary equivalent technical wording, and independent facts have separate entries. Docs task_version 1 results measured narrower phrasing and are not comparable with task_version 2 results. Saved submissions can be re-checked under version 2; old results retain their original version. —

2026-10-10 ZAIPRICE-02
pricing glm-5.2 cache reads are priced at Z.ai's published $0.26 per million again. The provider's charges for the period were higher than the recorded tokens cost even at the full input rate, so no cache rate explains them; the proxy's records omit failed and abandoned calls, which a provider may still bill. zai/glm-5.2 is priced published-rate from the 2026-10-10 price table on. It becomes provider-invoice-reconciled only after a day-by-day reconciliation against the provider's invoice. —

2026-10-10 PDK-03
task Seven tasks derived from a private review were scrubbed for publication and moved to task_version 2: fix-corrupt-refresh-checkpoint, fix-rescue-local-tip, fix-stale-lock-recovery, review-corrupt-refresh-checkpoint, review-rescue-local-tip, review-stale-lock-recovery and hard-review-multi-defect-tool-calls. Imported rows of those tasks stay at task_version 1. Results compare only within one task version, so version 2 rows form their own boards. —

2026-10-10 HRX-01
task The hard-fix rework added regression-test and wrong-fix variants as new tasks with a -v2 suffix in their id. The -v2 tasks are different tasks, not new versions of the originals, so their rows are never pooled with the original tasks' rows. —

2026-10-07 ZAIPRICE-01
pricing glm-5.2 cache reads were priced at the full input rate, on the reading that Z.ai billed them that way. That reading came from fitting a single rate to a single billing total, which cannot establish a rate, and the provider's published cache rate contradicts it. Retracted on 2026-10-10 (ZAIPRICE-02). No row in the results store was ever priced at that rate: every zai/glm-5.2 row imported up to 2026-10-10 has cost unknown, because its tokens were not measured. —

2026-10-06 OCEXPORT-01
harness opencode's tool-call export was read through a pipe truncated at 64 KiB, so long sessions lost read evidence. It is now captured through files. opencode rows before this date are kept as published, but a failed opencode row before it may be a capture failure rather than a model failure: a pass needs captured read evidence. —

2026-10-06 LITELLMPRICE-01
pricing The model proxy billed Fireworks cached input at the full input rate (about 3x) and under-billed Together. Proxy cost estimates are never a leaderboard source. Costs are recomputed from measured tokens at the dated price table. —

2026-10-06 GYMSKEW-02
infrastructure A deploy landing in the middle of a run mixed two gym-library versions in one run. A run now pins one library version. Runs it voided are infrastructure failures, not model results: they carry an infra: excludedBecause tag or were never exported by the source. —

2026-10-05 RFDFIX-01
task fix-corrupt-refresh-checkpoint's agent-visible finding contradicted its hidden contract: it recommended quarantining on any read error, while a hidden case required read errors to propagate. A fixture-contract line now states the contract. Rows of fix-corrupt-refresh-checkpoint before the fix carry excludedBecause pre-RFDFIX-01; before it, no combination could pass the task. 6 rows tagged pre-RFDFIX-01

2026-10-05 GYMAGY-02
grading Antigravity tool calls were collected but not recognised as reads; they are now normalised to the gym's tool-call shapes. Gemini rows on read-evidence suites (review, coding, injection) count only after this fix; earlier rows carry excludedBecause pre-GYMAGY-02. 12 rows tagged pre-GYMAGY-02

2026-10-04 GYMDOCS-01
grading Docs fact checks required a fact's terms to co-occur on a single line; they now match within bounded sections. Docs rows graded before the fix carry excludedBecause pre-GYMDOCS-01 and are not scored. 35 rows tagged pre-GYMDOCS-01

2026-10-04 GYMAGY-01
harness Antigravity (Gemini) tool calls were not collected, so read-evidence checks could not see them. Superseded by GYMAGY-02, which is the gate for Gemini rows. —

2026-10-04 CCMODEL-01
harness The claude-code harness ignored the requested model, so early Sonnet and Haiku rows actually ran Opus. Claude non-Opus rows count only after this fix; earlier rows carry excludedBecause pre-CCMODEL-01. 0 rows tagged pre-CCMODEL-01

2026-09-30 OMBPARSE-01
infrastructure The local model backend did not parse Qwen3-Coder-Next tool calls (with OMBTOK-01 and OMBPASS-01, the same day). The first local run submitted nothing on every task. That run is superseded and not imported. Every local row in results/ ran after all three fixes. —

2026-09-29 KIMIEMPTY-01
harness Kimi K3 on Moonshot answered some tool calls with empty text; codex echoed the empty turn back and Moonshot refused the next request, so most codex workers on that route crashed. The model proxy now drops empty assistant text before the request reaches Moonshot. codex x moonshot/kimi-k3 rows before the fix and the post-fix rerun carry different harness config hashes and are never pooled. Other harness and route combinations were unaffected. —

date	change	what changed	comparability boundary	excluded
2026-10-18	READEV-01 grading	All 15 review tasks using review_checks advance one task_version. Read evidence recognizes command prefixes and literal for-loop bindings, including value-preserving default/error parameter expansions. Shell comments and escaped dollars cannot supply loop expansion evidence; path-changing parameter operators remain unsupported.	hard-review-multi-defect-tool-calls, review-corrupt-refresh-checkpoint, review-rescue-local-tip and review-stale-lock-recovery move from version 2 to 3; the other 11 review tasks move from version 1 to 2. Earlier results keep their original versions and are not pooled with the new versions; saved submissions can be regraded.	—
2026-10-18	PDKRESULTS-01 infrastructure	Early gym versions did not record a task's version. Such runs are version 1 when they ran before the first version change (2026-10-10) or when the task never changed; the few that cannot be placed are not imported.	Results from the per-attempt export carry measured tokens and wall time; earlier imports keep null tokens.	—
2026-10-15	HARDFIX-01 task	Five new hard-fix -v3 task ids add contract interactions and declared plausible wrong fixes, checked through the task verifier. The regression tasks require submitted tests to distinguish three additional wrong implementations.	The new -v3 ids start at task_version 1 on separate boards. Original and -v2 tasks, versions and published results stay unchanged; no previous row is regraded or excluded.	—
2026-10-14	DOCSCHECK-02 grading	docs-retry-behavior-diff, docs-snapshot-new-command and docs-gym-baseline-keys move to task_version 3. Fact checks accept negated alternatives, contrasting outcome-specific exit codes, subject-scoped boolean --json documentation, header-selected table defaults and equivalent model-qualified key forms. Table descriptions reject conflicting --json defaults and argument-error codes; table column delimiters do not count as baseline key separators.	Version 2 results for these three tasks are not comparable with version 3 results. Saved submissions can be regraded under version 3; old results retain their original version. The other three docs tasks remain at version 2.	—
2026-10-13	DOCSCHECK-01 grading	All six docs tasks move to task_version 2. Required facts accept ordinary equivalent technical wording, and independent facts have separate entries.	Docs task_version 1 results measured narrower phrasing and are not comparable with task_version 2 results. Saved submissions can be re-checked under version 2; old results retain their original version.	—
2026-10-10	ZAIPRICE-02 pricing	glm-5.2 cache reads are priced at Z.ai's published $0.26 per million again. The provider's charges for the period were higher than the recorded tokens cost even at the full input rate, so no cache rate explains them; the proxy's records omit failed and abandoned calls, which a provider may still bill.	zai/glm-5.2 is priced published-rate from the 2026-10-10 price table on. It becomes provider-invoice-reconciled only after a day-by-day reconciliation against the provider's invoice.	—
2026-10-10	PDK-03 task	Seven tasks derived from a private review were scrubbed for publication and moved to task_version 2: fix-corrupt-refresh-checkpoint, fix-rescue-local-tip, fix-stale-lock-recovery, review-corrupt-refresh-checkpoint, review-rescue-local-tip, review-stale-lock-recovery and hard-review-multi-defect-tool-calls.	Imported rows of those tasks stay at task_version 1. Results compare only within one task version, so version 2 rows form their own boards.	—
2026-10-10	HRX-01 task	The hard-fix rework added regression-test and wrong-fix variants as new tasks with a -v2 suffix in their id.	The -v2 tasks are different tasks, not new versions of the originals, so their rows are never pooled with the original tasks' rows.	—
2026-10-07	ZAIPRICE-01 pricing	glm-5.2 cache reads were priced at the full input rate, on the reading that Z.ai billed them that way. That reading came from fitting a single rate to a single billing total, which cannot establish a rate, and the provider's published cache rate contradicts it. Retracted on 2026-10-10 (ZAIPRICE-02).	No row in the results store was ever priced at that rate: every zai/glm-5.2 row imported up to 2026-10-10 has cost unknown, because its tokens were not measured.	—
2026-10-06	OCEXPORT-01 harness	opencode's tool-call export was read through a pipe truncated at 64 KiB, so long sessions lost read evidence. It is now captured through files.	opencode rows before this date are kept as published, but a failed opencode row before it may be a capture failure rather than a model failure: a pass needs captured read evidence.	—
2026-10-06	LITELLMPRICE-01 pricing	The model proxy billed Fireworks cached input at the full input rate (about 3x) and under-billed Together.	Proxy cost estimates are never a leaderboard source. Costs are recomputed from measured tokens at the dated price table.	—
2026-10-06	GYMSKEW-02 infrastructure	A deploy landing in the middle of a run mixed two gym-library versions in one run. A run now pins one library version.	Runs it voided are infrastructure failures, not model results: they carry an infra: excludedBecause tag or were never exported by the source.	—
2026-10-05	RFDFIX-01 task	fix-corrupt-refresh-checkpoint's agent-visible finding contradicted its hidden contract: it recommended quarantining on any read error, while a hidden case required read errors to propagate. A fixture-contract line now states the contract.	Rows of fix-corrupt-refresh-checkpoint before the fix carry excludedBecause pre-RFDFIX-01; before it, no combination could pass the task.	6 rows tagged `pre-RFDFIX-01`
2026-10-05	GYMAGY-02 grading	Antigravity tool calls were collected but not recognised as reads; they are now normalised to the gym's tool-call shapes.	Gemini rows on read-evidence suites (review, coding, injection) count only after this fix; earlier rows carry excludedBecause pre-GYMAGY-02.	12 rows tagged `pre-GYMAGY-02`
2026-10-04	GYMDOCS-01 grading	Docs fact checks required a fact's terms to co-occur on a single line; they now match within bounded sections.	Docs rows graded before the fix carry excludedBecause pre-GYMDOCS-01 and are not scored.	35 rows tagged `pre-GYMDOCS-01`
2026-10-04	GYMAGY-01 harness	Antigravity (Gemini) tool calls were not collected, so read-evidence checks could not see them.	Superseded by GYMAGY-02, which is the gate for Gemini rows.	—
2026-10-04	CCMODEL-01 harness	The claude-code harness ignored the requested model, so early Sonnet and Haiku rows actually ran Opus.	Claude non-Opus rows count only after this fix; earlier rows carry excludedBecause pre-CCMODEL-01.	0 rows tagged `pre-CCMODEL-01`
2026-09-30	OMBPARSE-01 infrastructure	The local model backend did not parse Qwen3-Coder-Next tool calls (with OMBTOK-01 and OMBPASS-01, the same day). The first local run submitted nothing on every task.	That run is superseded and not imported. Every local row in results/ ran after all three fixes.	—
2026-09-29	KIMIEMPTY-01 harness	Kimi K3 on Moonshot answered some tool calls with empty text; codex echoed the empty turn back and Moonshot refused the next request, so most codex workers on that route crashed. The model proxy now drops empty assistant text before the request reaches Moonshot.	codex x moonshot/kimi-k3 rows before the fix and the post-fix rerun carry different harness config hashes and are never pooled. Other harness and route combinations were unaffected.	—