7539 result rows from 5 sources, 6914 of them graded and scored. Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions. Read the methodology before drawing conclusions: most cells are small.
Suites: coding · task version 1, coding · task version 2, docs · task version 1, docs · task version 2, docs · task version 3, fix · task version 1, fix · task version 2, hard-fix, hard-review · task version 1, hard-review · task version 2, hard-review · task version 3, injection, rca, review · task version 1, review · task version 2, review · task version 3.
coding · task version 1
13 tasks: coding-cache-context, coding-commit-ack, coding-generation-fence, coding-jsonb-like-predicate, coding-missing-commit, coding-nondict-migrate-audit, coding-pipefail-silent-exit, coding-pr-not-found-terminal, coding-registry-tail, coding-schema-behind-wording, coding-toc-commands, coding-toc-large-output, coding-unknown-change-kind-sentinel.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 63/63 | 94%–100% | 1.00 | 13 | 102k | $0.04 published-rate | 1 infra:error |
| 2 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 34/34 | 90%–100% | 1.00 | 13 | 1.26M | $0.36 published-rate | 1 infra:error, 2 infra:test_runner_unavailable |
| 3 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 25/25 | 87%–100% | 1.00 | 5 | 124k | $0.05 published-rate | |
| 4 | opencode | 357da891 | openai/gpt-6.1-sol | 25/25 | 87%–100% | 1.00 | 13 | 157k | $0.07 published-rate | 3 infra:test_runner_unavailable |
| 5 | pi | da41c66e | openai/gpt-6.1-sol | 15/15 | 80%–100% | 1.00 | 13 | 76k | $0.04 published-rate | 1 infra:test_runner_unavailable |
| 6 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 13/13 | 77%–100% | 1.00 | 13 | 93k | $0.10 published-rate | 4 infra:agent_error |
| 7 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 13/13 | 77%–100% | 1.00 | 13 | 157k | $0.07 published-rate | 1 infra:agent_error, 1 infra:test_runner_unavailable |
| 8 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 13/13 | 77%–100% | 1.00 | 13 | 51k | $0.22 published-rate | |
| 9 | opencode | 357da891 | anthropic/claude-opus-5-5 | 13/13 | 77%–100% | 1.00 | 13 | 83k | $0.12 published-rate | 2 infra:test_runner_unavailable |
| 10 | pi | da41c66e | anthropic/claude-opus-5-5 | 13/13 | 77%–100% | 1.00 | 13 | 40k | $0.17 published-rate | 2 infra:test_runner_unavailable |
| 11 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 13/13 | 77%–100% | 1.00 | 13 | 32k | $0.07 published-rate | 2 infra:test_runner_unavailable |
| 12 | claude-code | ecb67e4e | google/gemini-3.8-flash | 12/12 | 76%–100% | 1.00 | 12 | 603k | $0.15 published-rate | 1 infra:test_runner_unavailable |
| 13 | claude-code | c32a950b | anthropic/claude-haiku-4-5 | 7/7 | 65%–100% | 1.00 | 7 | 714k | $0.09 published-rate | 1 infra:agent_error, 6 infra:dispatch_ran_other_model |
| 14 | claude-code | c32a950b | anthropic/claude-opus-5-5 @low | 2/2 | 34%–100% | 1.00 | 2 | 274k | $0.14 published-rate | 1 infra:agent_error |
| 15 | codex | b2396403 | openai/gpt-6-astra @low | 2/2 | 34%–100% | 2 | 55k | $0.13 published-rate | ||
| 16 | codex | b2396403 | openai/gpt-6-sol @medium | 2/2 | 34%–100% | 1.00 | 2 | 100k | $0.05 published-rate | |
| 17 | claude-code | ecb67e4e | fireworks/qwen3.8-27b | 1/1 | 21%–100% | 1 | 110k | unknown | ||
| 18 | codex | b2396403 | openai/gpt-5.6-luna @low | 1/1 | 21%–100% | 1 | 82k | $0.0047 published-rate | ||
| 19 | codex | b2396403 | openai/gpt-5.6-luna @medium | 1/1 | 21%–100% | 1 | 64k | $0.0044 published-rate | ||
| 20 | codex | b2396403 | openai/gpt-5.6-sol @low | 1/1 | 21%–100% | 1 | 60k | $0.05 published-rate | ||
| 21 | codex | b2396403 | openai/gpt-5.6-terra @high | 1/1 | 21%–100% | 1 | 121k | $0.05 published-rate | ||
| 22 | codex | b2396403 | openai/gpt-5.6-terra @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 59k | $0.03 published-rate | |
| 23 | codex | b2396403 | openai/gpt-6-astra @xhigh | 1/1 | 21%–100% | 1 | 47k | $0.12 published-rate | ||
| 24 | codex | b2396403 | openai/gpt-6-luna @high | 1/1 | 21%–100% | 1 | 91k | $0.0019 published-rate | ||
| 25 | codex | b2396403 | openai/gpt-6-sol @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 77k | $0.03 published-rate | |
| 26 | codex | b2396403 | openai/gpt-6.1-sol @high | 57/58 | 91%–100% | 0.99 | 13 | 118k | $0.04 published-rate | |
| 27 | codex | b2396403 | openai/gpt-6.1-sol | 32/33 | 85%–99% | 0.98 | 13 | 104k | $0.04 published-rate | 2 infra:error |
| 28 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 25/26 | 81%–99% | 1.00 | 13 | not measured (19/26) | unknown | 1 infra:error, 1 infra:test_runner_unavailable |
| 29 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 24/25 | 80%–99% | 1.00 | 5 | 1.95M | $0.57 published-rate | 4 infra:test_runner_unavailable |
| 30 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 13/14 | 69%–99% | 0.96 | 13 | 831k | $0.12 published-rate | 1 infra:dispatch_ran_other_model |
| 31 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 12/13 | 67%–99% | 1.00 | 13 | 143k | unknown | 1 pre-GYMAGY-02 |
| 32 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 12/13 | 67%–99% | 1.00 | 13 | 656k | $0.24 published-rate | |
| 33 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 12/13 | 67%–99% | 1.00 | 13 | 486k | $0.23 published-rate | |
| 34 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 12/13 | 67%–99% | 0.96 | 13 | 72k | $0.06 published-rate | |
| 35 | opencode | 357da891 | google/gemini-3.8-flash | 12/13 | 67%–99% | 1.00 | 13 | 1.39M | $0.24 published-rate | |
| 36 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 11/12 | 65%–99% | 1.00 | 12 | 860k | $0.17 published-rate | |
| 37 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 11/12 | 65%–99% | 0.96 | 12 | 123k | $0.08 published-rate | |
| 38 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 11/12 | 65%–99% | 1.00 | 12 | 277k | $0.17 published-rate | 2 infra:agent_error |
| 39 | opencode | 357da891 | google/gemini-3.8-flash @high | 11/12 | 65%–99% | 1.00 | 12 | 356k | $0.18 published-rate | |
| 40 | opencode | 357da891 | google/gemini-3.8-flash @low | 11/12 | 65%–99% | 0.97 | 12 | 122k | $0.08 published-rate | 5 infra:worker_boot_fenced |
| 41 | pi | da41c66e | google/gemini-3.8-flash @low | 11/12 | 65%–99% | 0.96 | 12 | 59k | $0.05 published-rate | |
| 42 | claude-code | ecb67e4e | zai/glm-5.2 | 18/20 | 70%–97% | 1.00 | 13 | not measured (13/20) | unknown | 6 infra:agent_error |
| 43 | codex | 3437a9e6 | fireworks/qwen3.8-max | 18/20 | 70%–97% | 1.00 | 13 | not measured (13/20) | unknown | 9 infra:agent_error |
| 44 | pi | da41c66e | moonshot/kimi-k3 | 18/20 | 70%–97% | 1.00 | 13 | not measured (12/20) | unknown | 1 infra:test_runner_unavailable |
| 45 | codex | 3437a9e6 | moonshot/kimi-k3 | 15/17 | 66%–97% | 0.96 | 13 | not measured (13/17) | unknown | 4 infra:agent_error |
| 46 | pi | da41c66e | google/gemini-3.8-flash @high | 25/29 | 69%–95% | 1.00 | 13 | 546k | $0.25 published-rate | |
| 47 | pi | da41c66e | together/deepseek-v4.1-flash | 31/36 | 71%–94% | 0.96 | 13 | not measured (29/36) | unknown | 1 infra:test_runner_unavailable |
| 48 | pi | da41c66e | google/gemini-3.8-flash | 24/28 | 69%–94% | 1.00 | 13 | 369k | $0.17 published-rate | 1 infra:gateway_continued_run_changed_prompt |
| 49 | opencode | 357da891 | fireworks/qwen3.8-max | 17/20 | 64%–95% | 0.96 | 13 | not measured (13/20) | unknown | 1 infra:tool_call_log_truncated |
| 50 | opencode | 357da891 | moonshot/kimi-k3 | 17/20 | 64%–95% | 0.96 | 13 | not measured (13/20) | unknown | 1 infra:test_runner_unavailable |
| 51 | pi | da41c66e | fireworks/kimi-k3 | 17/20 | 64%–95% | 1.00 | 13 | not measured (13/20) | unknown | 1 infra:test_runner_unavailable |
| 52 | pi | da41c66e | zai/glm-5.2 | 17/20 | 64%–95% | 1.00 | 13 | not measured (13/20) | unknown | 1 infra:test_runner_unavailable |
| 53 | pi | da41c66e | fireworks/qwen3.8-max | 22/26 | 66%–94% | 1.00 | 13 | not measured (19/26) | unknown | |
| 54 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 11/13 | 58%–96% | 0.96 | 13 | 57k | $0.12 published-rate | |
| 55 | codex | 3437a9e6 | google/gemini-3.8-flash | 11/13 | 58%–96% | 1.00 | 13 | 227k | $0.14 published-rate | |
| 56 | claude-code | ecb67e4e | fireworks/kimi-k3 | 16/19 | 62%–94% | 0.97 | 13 | not measured (13/19) | unknown | 1 infra:agent_error, 1 infra:test_runner_unavailable |
| 57 | opencode | 357da891 | zai/glm-5.2 | 16/19 | 62%–94% | 1.00 | 13 | not measured (12/19) | unknown | 1 infra:test_runner_unavailable, 1 infra:tool_call_log_truncated |
| 58 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 10/12 | 55%–95% | 0.96 | 12 | 89k | $0.07 published-rate | |
| 59 | claude-code | c32a950b | anthropic/claude-sonnet-5 | 5/6 | 44%–97% | 1.00 | 6 | 799k | $0.21 published-rate | 1 infra:agent_error, 7 infra:dispatch_ran_other_model, 1 infra:error |
| 60 | codex | 3437a9e6 | zai/glm-5.2 | 21/26 | 62%–91% | 0.94 | 13 | not measured (19/26) | unknown | |
| 61 | opencode | 357da891 | fireworks/kimi-k3 | 16/20 | 58%–92% | 1.00 | 13 | not measured (13/20) | unknown | |
| 62 | codex | 3437a9e6 | baseten/glm-5.3-flash | 15/19 | 57%–91% | 0.89 | 13 | not measured (13/19) | unknown | 1 infra:agent_error |
| 63 | opencode | 357da891 | together/deepseek-v4.1-flash | 25/32 | 61%–89% | 0.95 | 13 | not measured (25/32) | unknown | 1 infra:tool_call_log_truncated |
| 64 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 20/26 | 58%–89% | 0.94 | 13 | not measured (19/26) | unknown | 3 infra:agent_error, 1 infra:gateway_continued_run_changed_prompt, 1 infra:test_runner_unavailable |
| 65 | codex | 3437a9e6 | fireworks/kimi-k3 | 20/26 | 58%–89% | 0.94 | 13 | not measured (19/26) | unknown | |
| 66 | pi | da41c66e | baseten/glm-5.3-flash | 19/26 | 54%–86% | 0.97 | 13 | not measured (19/26) | unknown | 1 infra:test_runner_unavailable |
| 67 | claude-code | ecb67e4e | moonshot/kimi-k3 | 13/18 | 49%–88% | 1.00 | 13 | not measured (12/18) | unknown | 3 infra:agent_error, 1 infra:gateway_budget_refusal, 1 infra:test_runner_unavailable |
| 68 | opencode | 357da891 | baseten/glm-5.3-flash | 18/26 | 50%–83% | 0.96 | 13 | not measured (19/26) | unknown | 2 infra:test_runner_unavailable |
| 69 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 13/19 | 46%–85% | 0.96 | 12 | not measured (12/19) | unknown | |
| 70 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 13/20 | 43%–82% | 0.92 | 13 | not measured (13/20) | unknown | |
| 71 | claude-code | ecb67e4e | local/qwen3-coder-next-80b-a3b-6bit | 7/12 | 32%–81% | 12 | not measured (0/12) | unknown | ||
| 72 | codex | cb391ae9 | moonshot/kimi-k3 | 4/7 | 25%–84% | 7 | not measured (0/7) | unknown | ||
| 73 | opencode | 357da891 | local/qwen3-coder-next-80b-a3b-6bit | 5/12 | 19%–68% | 12 | not measured (0/12) | unknown | ||
| 74 | pi | da41c66e | local/qwen3-coder-next-80b-a3b-6bit | 3/12 | 9%–53% | 12 | not measured (0/12) | unknown | ||
| 75 | codex | 3437a9e6 | local/qwen3-coder-next-80b-a3b-6bit | 1/12 | 1%–35% | 12 | not measured (0/12) | unknown | ||
| 76 | claude-code | ecb67e4e | together/qwen3.8-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 1 infra:provider_tool_calls_as_text |
coding · task version 2
1 tasks: coding-toc-large-output.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 11/11 | 74%–100% | 1 | 246k | $0.10 published-rate | ||
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 10/10 | 72%–100% | 1 | 305k | $0.11 published-rate | ||
| 3 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 6/6 | 61%–100% | 1 | 3.37M | $0.86 published-rate | ||
| 4 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 5/5 | 57%–100% | 1 | 3.97M | $1.03 published-rate | ||
| 5 | codex | b2396403 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 1 | 254k | $0.09 published-rate | ||
| 6 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 5/5 | 57%–100% | 1 | 310k | $0.12 published-rate | ||
| 7 | opencode | 357da891 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 1 | 567k | $0.19 published-rate | ||
| 8 | pi | da41c66e | together/deepseek-v4.1-flash | 5/5 | 57%–100% | 1 | 325k | $0.10 published-rate | ||
| 9 | pi | da41c66e | google/gemini-3.8-flash | 4/4 | 51%–100% | 1 | 1.06M | $0.18 published-rate | 1 infra:agent_error | |
| 10 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1 | 882k | $0.32 published-rate | ||
| 11 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 1 | 679k | $0.21 published-rate | 2 infra:agent_error | |
| 12 | pi | da41c66e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1 | 382k | $0.04 published-rate | ||
| 13 | pi | da41c66e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1 | 655k | $0.29 published-rate | ||
| 14 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1 | 1.76M | $0.28 published-rate | 2 infra:grader_tool_call_evidence_denied | |
| 15 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1 | 308k | $0.12 published-rate | ||
| 16 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 1/1 | 21%–100% | 1 | 196k | unknown | ||
| 17 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 1 | 1.77M | $0.41 published-rate | ||
| 18 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 1/1 | 21%–100% | 1 | 169k | $0.12 published-rate | ||
| 19 | claude-code | c32a950b | anthropic/claude-haiku-4-5 | 1/1 | 21%–100% | 1 | 1.13M | $0.14 published-rate | ||
| 20 | claude-code | c32a950b | anthropic/claude-sonnet-5 | 1/1 | 21%–100% | 1 | 991k | $0.26 published-rate | ||
| 21 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 1/1 | 21%–100% | 1 | 1.92M | $0.26 published-rate | ||
| 22 | claude-code | ecb67e4e | fireworks/kimi-k3 | 1/1 | 21%–100% | 1 | 765k | $0.37 published-rate | ||
| 23 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 1 | 2.93M | $0.51 published-rate | ||
| 24 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 1 | 627k | $0.13 published-rate | ||
| 25 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 1 | 524k | $0.13 published-rate | ||
| 26 | claude-code | ecb67e4e | moonshot/kimi-k3 | 1/1 | 21%–100% | 1 | 531k | $0.32 published-rate | 1 infra:agent_error | |
| 27 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 1/1 | 21%–100% | 1 | 425k | $0.14 published-rate | 1 infra:agent_error | |
| 28 | claude-code | ecb67e4e | zai/glm-5.2 | 1/1 | 21%–100% | 1 | 679k | $0.22 published-rate | ||
| 29 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1 | 170k | $0.70 published-rate | ||
| 30 | codex | 3437a9e6 | baseten/glm-5.3-flash | 1/1 | 21%–100% | 1 | 142k | $0.02 published-rate | ||
| 31 | codex | 3437a9e6 | fireworks/qwen3.8-max | 1/1 | 21%–100% | 1 | 573k | $0.32 published-rate | ||
| 32 | codex | 3437a9e6 | google/gemini-3.8-flash | 1/1 | 21%–100% | 1 | 378k | $0.11 published-rate | ||
| 33 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 1 | 1.90M | $0.46 published-rate | ||
| 34 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 1 | 405k | $0.09 published-rate | ||
| 35 | codex | 3437a9e6 | moonshot/kimi-k3 | 1/1 | 21%–100% | 1 | 109k | $0.09 published-rate | ||
| 36 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 1/1 | 21%–100% | 1 | 130k | $0.04 published-rate | ||
| 37 | codex | b2396403 | openai/gpt-6.1-sol @low | 1/1 | 21%–100% | 1 | 196k | $0.08 published-rate | ||
| 38 | opencode | 357da891 | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1 | 351k | $0.36 published-rate | ||
| 39 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 1/1 | 21%–100% | 1 | 202k | $0.12 published-rate | ||
| 40 | opencode | 357da891 | fireworks/kimi-k3 | 1/1 | 21%–100% | 1 | 460k | $0.41 published-rate | ||
| 41 | opencode | 357da891 | fireworks/qwen3.8-max | 1/1 | 21%–100% | 1 | 941k | $0.40 published-rate | ||
| 42 | opencode | 357da891 | google/gemini-3.8-flash | 1/1 | 21%–100% | 1 | 14.32M | $1.41 published-rate | ||
| 43 | opencode | 357da891 | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 1 | 1.05M | $0.19 published-rate | ||
| 44 | opencode | 357da891 | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 1 | 786k | $0.13 published-rate | ||
| 45 | opencode | 357da891 | moonshot/kimi-k3 | 1/1 | 21%–100% | 1 | 380k | $0.29 published-rate | ||
| 46 | opencode | 357da891 | zai/glm-5.2 | 1/1 | 21%–100% | 1 | 437k | $0.26 published-rate | ||
| 47 | pi | da41c66e | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1 | 269k | $1.09 published-rate | ||
| 48 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 1/1 | 21%–100% | 1 | 181k | $0.37 published-rate | ||
| 49 | pi | da41c66e | fireworks/kimi-k3 | 1/1 | 21%–100% | 1 | 309k | $0.25 published-rate | ||
| 50 | pi | da41c66e | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 1 | 469k | $0.09 published-rate | ||
| 51 | pi | da41c66e | moonshot/kimi-k3 | 1/1 | 21%–100% | 1 | 265k | $0.23 published-rate | ||
| 52 | pi | da41c66e | zai/glm-5.2 | 1/1 | 21%–100% | 1 | 424k | $0.24 published-rate | ||
| 53 | codex | 3437a9e6 | fireworks/kimi-k3 | 2/3 | 21%–94% | 1 | 195k | $0.16 published-rate | ||
| 54 | codex | 3437a9e6 | zai/glm-5.2 | 2/3 | 21%–94% | 1 | 289k | $0.13 published-rate | ||
| 55 | opencode | 357da891 | baseten/glm-5.3-flash | 1/3 | 6%–79% | 1 | 667k | $0.07 published-rate | ||
| 56 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 57 | claude-code | ecb67e4e | google/gemini-3.8-flash | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 58 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 59 | opencode | 357da891 | together/deepseek-v4.1-flash | 0/1 | 0%–79% | 1 | no passes | unknown |
docs · task version 1
6 tasks: docs-cache-config, docs-config-loader-keys, docs-gym-baseline-keys, docs-gym-run-flags, docs-retry-behavior-diff, docs-snapshot-new-command.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | pi | da41c66e | google/gemini-3.8-flash | 4/6 | 30%–90% | 0.94 | 6 | 262k | $0.17 published-rate | |
| 2 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 3/6 | 19%–81% | 0.94 | 6 | 726k | $0.37 published-rate | |
| 3 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/6 | 19%–81% | 0.91 | 6 | 107k | $0.22 published-rate | 1 infra:agent_error |
| 4 | pi | da41c66e | zai/glm-5.2 | 3/6 | 19%–81% | 0.91 | 6 | 74k | $0.09 published-rate | |
| 5 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 7/18 | 20%–61% | 0.93 | 6 | not measured (0/18) | unknown | 6 infra:agent_error, 6 pre-GYMDOCS-01 |
| 6 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 4/12 | 14%–61% | 0.92 | 6 | not measured (0/12) | unknown | 6 pre-GYMDOCS-01 |
| 7 | claude-code | ecb67e4e | fireworks/kimi-k3 | 2/6 | 10%–70% | 0.89 | 6 | 497k | $0.21 published-rate | |
| 8 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 2/6 | 10%–70% | 0.92 | 6 | 135k | $0.60 published-rate | |
| 9 | codex | 3437a9e6 | fireworks/kimi-k3 | 2/6 | 10%–70% | 0.91 | 6 | 129k | $0.19 published-rate | |
| 10 | opencode | 357da891 | fireworks/qwen3.8-max | 2/6 | 10%–70% | 0.93 | 6 | 257k | $0.21 published-rate | |
| 11 | opencode | 357da891 | google/gemini-3.8-flash | 2/6 | 10%–70% | 0.91 | 6 | 572k | $0.39 published-rate | |
| 12 | pi | da41c66e | anthropic/claude-opus-5-5 | 2/6 | 10%–70% | 0.89 | 6 | 57k | $0.29 published-rate | |
| 13 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 2/6 | 10%–70% | 0.89 | 6 | 60k | $0.15 published-rate | |
| 14 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 3/13 | 8%–50% | 0.88 | 6 | not measured (0/13) | unknown | 10 infra:agent_error, 6 pre-GYMDOCS-01 |
| 15 | codex | b2396403 | openai/gpt-6.1-sol | 3/15 | 7%–45% | 0.90 | 6 | not measured (0/15) | unknown | 6 infra:agent_error, 6 pre-GYMDOCS-01 |
| 16 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 3/18 | 6%–39% | 0.86 | 6 | not measured (0/18) | unknown | |
| 17 | claude-code | ecb67e4e | zai/glm-5.2 | 2/12 | 5%–45% | 0.86 | 6 | not measured (0/12) | unknown | |
| 18 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 1/6 | 3%–56% | 0.87 | 6 | 1.06M | $0.33 published-rate | |
| 19 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 1/6 | 3%–56% | 0.83 | 6 | 216k | $0.47 published-rate | |
| 20 | codex | 3437a9e6 | moonshot/kimi-k3 | 1/6 | 3%–56% | 0.89 | 6 | 332k | $0.35 published-rate | |
| 21 | codex | 3437a9e6 | zai/glm-5.2 | 1/6 | 3%–56% | 0.85 | 6 | 430k | $0.33 published-rate | |
| 22 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 1/6 | 3%–56% | 0.83 | 6 | 380k | $0.34 published-rate | |
| 23 | opencode | 357da891 | baseten/glm-5.3-flash | 1/6 | 3%–56% | 0.89 | 6 | 361k | $0.04 published-rate | |
| 24 | opencode | 357da891 | moonshot/kimi-k3 | 1/6 | 3%–56% | 0.89 | 6 | 423k | $0.51 published-rate | |
| 25 | opencode | 357da891 | zai/glm-5.2 | 1/6 | 3%–56% | 0.87 | 6 | 479k | $0.27 published-rate | |
| 26 | pi | da41c66e | fireworks/kimi-k3 | 1/6 | 3%–56% | 0.89 | 6 | 217k | $0.47 published-rate | |
| 27 | pi | da41c66e | moonshot/kimi-k3 | 1/6 | 3%–56% | 0.89 | 6 | 174k | $0.38 published-rate | |
| 28 | opencode | 357da891 | together/deepseek-v4.1-flash | 1/12 | 1%–35% | 0.83 | 6 | 607k | $0.19 published-rate | |
| 29 | claude-code | c32a950b | anthropic/claude-sonnet-5 | 0/12 | 0%–24% | 0.84 | 6 | no passes | unknown | 6 pre-GYMDOCS-01 |
| 30 | claude-code | c32a950b | anthropic/claude-haiku-4-5 | 0/6 | 0%–39% | 0.78 | 6 | no passes | unknown | 1 infra:agent_error, 5 pre-GYMDOCS-01 |
| 31 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 0/6 | 0%–39% | 0.87 | 6 | no passes | unknown | |
| 32 | codex | 3437a9e6 | baseten/glm-5.3-flash | 0/6 | 0%–39% | 0.82 | 6 | no passes | unknown | |
| 33 | codex | 3437a9e6 | fireworks/qwen3.8-max | 0/6 | 0%–39% | 0.80 | 6 | no passes | unknown | 6 infra:proxy_request_conversion_failed |
| 34 | opencode | 357da891 | fireworks/kimi-k3 | 0/6 | 0%–39% | 0.83 | 6 | no passes | unknown | |
| 35 | pi | da41c66e | baseten/glm-5.3-flash | 0/6 | 0%–39% | 0.87 | 6 | no passes | unknown | |
| 36 | pi | da41c66e | fireworks/qwen3.8-max | 0/6 | 0%–39% | 0.84 | 6 | no passes | unknown | |
| 37 | claude-code | ecb67e4e | moonshot/kimi-k3 | 0/2 | 0%–66% | 0.83 | 2 | no passes | unknown |
docs · task version 2
6 tasks: docs-cache-config, docs-config-loader-keys, docs-gym-baseline-keys, docs-gym-run-flags, docs-retry-behavior-diff, docs-snapshot-new-command.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | claude-code | ecb67e4e | moonshot/kimi-k3 | 6/6 | 61%–100% | 1.00 | 6 | 205k | $0.14 published-rate | 3 infra:agent_error |
| 2 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 6/6 | 61%–100% | 1.00 | 6 | 45k | $0.20 published-rate | |
| 3 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 6/6 | 61%–100% | 1.00 | 6 | 36k | $0.08 published-rate | |
| 4 | codex | 3437a9e6 | fireworks/kimi-k3 | 6/6 | 61%–100% | 1.00 | 6 | 43k | $0.06 published-rate | |
| 5 | codex | 3437a9e6 | moonshot/kimi-k3 | 6/6 | 61%–100% | 1.00 | 6 | 55k | $0.06 published-rate | |
| 6 | codex | b2396403 | openai/gpt-6.1-sol @high | 6/6 | 61%–100% | 1.00 | 6 | 86k | $0.04 published-rate | |
| 7 | opencode | 357da891 | moonshot/kimi-k3 | 6/6 | 61%–100% | 1.00 | 6 | 76k | $0.10 published-rate | |
| 8 | pi | da41c66e | anthropic/claude-opus-5-5 | 6/6 | 61%–100% | 1.00 | 6 | 21k | $0.11 published-rate | |
| 9 | pi | da41c66e | baseten/glm-5.3-flash | 6/6 | 61%–100% | 1.00 | 6 | 23k | $0.0037 published-rate | |
| 10 | pi | da41c66e | fireworks/kimi-k3 | 6/6 | 61%–100% | 1.00 | 6 | 36k | $0.08 published-rate | |
| 11 | pi | da41c66e | fireworks/qwen3.8-max | 6/6 | 61%–100% | 1.00 | 6 | 66k | $0.06 published-rate | |
| 12 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 1.00 | 3 | 160k | $0.17 published-rate | |
| 13 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 1.06M | $0.36 published-rate | |
| 14 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 904k | $0.15 published-rate | |
| 15 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 626k | $0.19 published-rate | |
| 16 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 186k | $0.13 published-rate | |
| 17 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 1.00 | 3 | 204k | $0.14 published-rate | |
| 18 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 168k | $0.11 published-rate | |
| 19 | codex | b2396403 | openai/gpt-6.1-sol @medium | 3/3 | 44%–100% | 1.00 | 3 | 88k | $0.04 published-rate | |
| 20 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 454k | $0.20 published-rate | |
| 21 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 359k | $0.22 published-rate | |
| 22 | pi | da41c66e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 52k | $0.04 published-rate | |
| 23 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 5/6 | 44%–97% | 0.83 | 6 | 436k | $0.22 published-rate | |
| 24 | claude-code | ecb67e4e | fireworks/kimi-k3 | 5/6 | 44%–97% | 0.98 | 6 | 207k | $0.16 published-rate | |
| 25 | claude-code | ecb67e4e | google/gemini-3.8-flash | 5/6 | 44%–97% | 0.98 | 6 | 591k | $0.16 published-rate | |
| 26 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 5/6 | 44%–97% | 0.98 | 6 | 317k | $0.10 published-rate | 4 infra:agent_error |
| 27 | claude-code | ecb67e4e | zai/glm-5.2 | 5/6 | 44%–97% | 0.98 | 6 | 241k | $0.11 published-rate | |
| 28 | codex | 3437a9e6 | baseten/glm-5.3-flash | 5/6 | 44%–97% | 0.83 | 6 | 65k | $0.0095 published-rate | |
| 29 | codex | 3437a9e6 | fireworks/qwen3.8-max | 5/6 | 44%–97% | 0.83 | 6 | 107k | $0.09 published-rate | |
| 30 | codex | 3437a9e6 | google/gemini-3.8-flash | 5/6 | 44%–97% | 0.98 | 6 | 240k | $0.16 published-rate | |
| 31 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 5/6 | 44%–97% | 0.83 | 6 | 73k | $0.02 published-rate | |
| 32 | opencode | 357da891 | baseten/glm-5.3-flash | 5/6 | 44%–97% | 0.98 | 6 | 75k | $0.01 published-rate | |
| 33 | opencode | 357da891 | fireworks/kimi-k3 | 5/6 | 44%–97% | 0.83 | 6 | 95k | $0.15 published-rate | |
| 34 | opencode | 357da891 | fireworks/qwen3.8-max | 5/6 | 44%–97% | 0.83 | 6 | 103k | $0.09 published-rate | |
| 35 | opencode | 357da891 | google/gemini-3.8-flash | 5/6 | 44%–97% | 0.83 | 6 | 229k | $0.16 published-rate | |
| 36 | opencode | 357da891 | together/deepseek-v4.1-flash | 5/6 | 44%–97% | 0.96 | 6 | 63k | $0.02 published-rate | |
| 37 | opencode | 357da891 | zai/glm-5.2 | 5/6 | 44%–97% | 0.98 | 6 | 91k | $0.06 published-rate | |
| 38 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 5/6 | 44%–97% | 0.98 | 6 | 26k | $0.06 published-rate | |
| 39 | pi | da41c66e | openai/gpt-6.1-sol | 5/6 | 44%–97% | 0.97 | 6 | 29k | $0.03 published-rate | |
| 40 | pi | da41c66e | together/deepseek-v4.1-flash | 5/6 | 44%–97% | 0.98 | 6 | 35k | $0.01 published-rate | |
| 41 | pi | da41c66e | zai/glm-5.2 | 5/6 | 44%–97% | 0.98 | 6 | 60k | $0.06 published-rate | |
| 42 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 4/6 | 30%–90% | 0.97 | 6 | 363k | $0.20 published-rate | |
| 43 | codex | 3437a9e6 | zai/glm-5.2 | 4/6 | 30%–90% | 0.97 | 6 | 116k | $0.10 published-rate | |
| 44 | codex | b2396403 | openai/gpt-6.1-sol | 4/6 | 30%–90% | 0.97 | 6 | 121k | $0.05 published-rate | |
| 45 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 4/6 | 30%–90% | 0.97 | 6 | 87k | $0.08 published-rate | |
| 46 | opencode | 357da891 | openai/gpt-6.1-sol | 4/6 | 30%–90% | 0.96 | 6 | 125k | $0.09 published-rate | |
| 47 | pi | da41c66e | google/gemini-3.8-flash | 4/6 | 30%–90% | 0.67 | 6 | 262k | $0.17 published-rate | |
| 48 | pi | da41c66e | moonshot/kimi-k3 | 4/6 | 30%–90% | 0.95 | 6 | 52k | $0.10 published-rate | |
| 49 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 2/3 | 21%–94% | 0.97 | 3 | 247k | $0.12 published-rate | |
| 50 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 0.97 | 3 | 652k | $0.38 published-rate | |
| 51 | opencode | 357da891 | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.97 | 3 | 166k | $0.11 published-rate | |
| 52 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/6 | 19%–81% | 0.95 | 6 | 461k | $0.06 published-rate | 5 infra:agent_error, 1 infra:error |
| 53 | opencode | 357da891 | anthropic/claude-opus-5-5 | 2/6 | 10%–70% | 0.94 | 6 | 170k | $0.34 published-rate | |
| 54 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 0/3 | 0%–56% | 1 | no passes | unknown |
docs · task version 3
3 tasks: docs-gym-baseline-keys, docs-retry-behavior-diff, docs-snapshot-new-command.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 1.00 | 3 | 114k | $0.11 published-rate | 1 infra:agent_error |
| 2 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 1.14M | $0.17 published-rate | |
| 3 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 257k | $0.03 published-rate | |
| 4 | claude-code | ecb67e4e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 180k | $0.13 published-rate | |
| 5 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 230k | $0.11 published-rate | |
| 6 | claude-code | ecb67e4e | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 388k | $0.07 published-rate | |
| 7 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 457k | $0.08 published-rate | |
| 8 | claude-code | ecb67e4e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 210k | $0.12 published-rate | |
| 9 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 1.00 | 3 | 262k | $0.08 published-rate | |
| 10 | claude-code | ecb67e4e | zai/glm-5.2 | 3/3 | 44%–100% | 1.00 | 3 | 198k | $0.07 published-rate | |
| 11 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 46k | $0.20 published-rate | |
| 12 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 34k | $0.07 published-rate | |
| 13 | codex | 3437a9e6 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 49k | $0.0070 published-rate | |
| 14 | codex | 3437a9e6 | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 36k | $0.05 published-rate | |
| 15 | codex | 3437a9e6 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 131k | $0.09 published-rate | |
| 16 | codex | 3437a9e6 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 54k | $0.05 published-rate | |
| 17 | codex | b2396403 | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 76k | $0.03 published-rate | |
| 18 | codex | b2396403 | openai/gpt-6.1-sol @high | 3/3 | 44%–100% | 1.00 | 3 | 89k | $0.04 published-rate | |
| 19 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 51k | $0.10 published-rate | |
| 20 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 57k | $0.05 published-rate | |
| 21 | opencode | 357da891 | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 72k | $0.11 published-rate | |
| 22 | opencode | 357da891 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 77k | $0.06 published-rate | |
| 23 | opencode | 357da891 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 72k | $0.08 published-rate | |
| 24 | pi | da41c66e | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 21k | $0.11 published-rate | |
| 25 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 17k | $0.04 published-rate | |
| 26 | pi | da41c66e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 21k | $0.0036 published-rate | |
| 27 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 28k | $0.05 published-rate | |
| 28 | pi | da41c66e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 56k | $0.06 published-rate | |
| 29 | pi | da41c66e | zai/glm-5.2 | 3/3 | 44%–100% | 1.00 | 3 | 33k | $0.03 published-rate | |
| 30 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 0.67 | 3 | 423k | $0.19 published-rate | |
| 31 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.96 | 3 | 177k | $0.14 published-rate | |
| 32 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 2/3 | 21%–94% | 0.96 | 3 | 230k | $0.18 published-rate | |
| 33 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 2/3 | 21%–94% | 0.97 | 3 | 257k | $0.11 published-rate | |
| 34 | codex | 3437a9e6 | fireworks/qwen3.8-max | 2/3 | 21%–94% | 0.67 | 3 | 166k | $0.15 published-rate | |
| 35 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 0.97 | 3 | 380k | $0.18 published-rate | |
| 36 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.96 | 3 | 121k | $0.10 published-rate | |
| 37 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 2/3 | 21%–94% | 0.67 | 3 | 76k | $0.02 published-rate | |
| 38 | codex | 3437a9e6 | zai/glm-5.2 | 2/3 | 21%–94% | 0.96 | 3 | 83k | $0.06 published-rate | |
| 39 | codex | b2396403 | openai/gpt-6.1-sol @medium | 2/3 | 21%–94% | 0.93 | 3 | 124k | $0.07 published-rate | |
| 40 | opencode | 357da891 | baseten/glm-5.3-flash | 2/3 | 21%–94% | 0.67 | 3 | 100k | $0.01 published-rate | |
| 41 | opencode | 357da891 | google/gemini-3.8-flash | 2/3 | 21%–94% | 0.67 | 3 | 180k | $0.14 published-rate | |
| 42 | opencode | 357da891 | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 0.96 | 3 | 382k | $0.21 published-rate | |
| 43 | opencode | 357da891 | openai/gpt-6.1-sol | 2/3 | 21%–94% | 0.67 | 3 | 117k | $0.09 published-rate | |
| 44 | opencode | 357da891 | together/deepseek-v4.1-flash | 2/3 | 21%–94% | 0.67 | 3 | 72k | $0.02 published-rate | |
| 45 | opencode | 357da891 | zai/glm-5.2 | 2/3 | 21%–94% | 0.67 | 3 | 96k | $0.06 published-rate | |
| 46 | pi | da41c66e | google/gemini-3.8-flash | 2/3 | 21%–94% | 0.67 | 3 | 112k | $0.10 published-rate | |
| 47 | pi | da41c66e | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 0.96 | 3 | 120k | $0.11 published-rate | |
| 48 | pi | da41c66e | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.96 | 3 | 54k | $0.04 published-rate | |
| 49 | pi | da41c66e | moonshot/kimi-k3 | 2/3 | 21%–94% | 0.67 | 3 | 50k | $0.09 published-rate | |
| 50 | pi | da41c66e | openai/gpt-6.1-sol | 2/3 | 21%–94% | 0.67 | 3 | 34k | $0.03 published-rate | |
| 51 | pi | da41c66e | together/deepseek-v4.1-flash | 2/3 | 21%–94% | 0.67 | 3 | 34k | $0.01 published-rate | |
| 52 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 1/3 | 6%–79% | 0.93 | 3 | 3.43M | $1.08 published-rate | |
| 53 | opencode | 357da891 | google/gemini-3.8-flash @low | 1/3 | 6%–79% | 0.93 | 3 | 324k | $0.25 published-rate | |
| 54 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 0/3 | 0%–56% | 1 | no passes | unknown |
fix · task version 1
6 tasks: fix-corrupt-refresh-checkpoint, fix-nested-template-copy, fix-publish-guard-order, fix-rescue-local-tip, fix-retry-empty-page, fix-stale-lock-recovery.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | opencode | 357da891 | together/deepseek-v4.1-flash | 12/12 | 76%–100% | 0.97 | 6 | 68k | $0.02 published-rate | |
| 2 | claude-code | ecb67e4e | zai/glm-5.2 | 7/7 | 65%–100% | 0.95 | 6 | not measured (0/7) | unknown | 5 infra:agent_error |
| 3 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 6/6 | 61%–100% | 1.00 | 6 | not measured (0/6) | unknown | 1 pre-RFDFIX-01 |
| 4 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 6/6 | 61%–100% | 0.94 | 6 | not measured (0/6) | unknown | 2 pre-RFDFIX-01 |
| 5 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 6/6 | 61%–100% | 0.94 | 6 | 217k | $0.07 published-rate | |
| 6 | codex | 3437a9e6 | zai/glm-5.2 | 6/6 | 61%–100% | 1.00 | 6 | 85k | $0.05 published-rate | 6 infra:error |
| 7 | codex | b2396403 | openai/gpt-6.1-sol | 6/6 | 61%–100% | 1.00 | 6 | not measured (2/6) | unknown | 1 infra:agent_error, 1 pre-RFDFIX-01 |
| 8 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 5/5 | 57%–100% | 1.00 | 5 | not measured (0/5) | unknown | 1 pre-RFDFIX-01 |
| 9 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 5/5 | 57%–100% | 1.00 | 5 | not measured (0/5) | unknown | 1 pre-RFDFIX-01 |
| 10 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 444k | $0.16 published-rate | |
| 11 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 1.00 | 3 | 121k | $0.10 published-rate | |
| 12 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 295k | $0.02 published-rate | |
| 13 | claude-code | ecb67e4e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 238k | $0.09 published-rate | |
| 14 | claude-code | ecb67e4e | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 514k | $0.09 published-rate | |
| 15 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 680k | $0.12 published-rate | |
| 16 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 145k | $0.11 published-rate | |
| 17 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 1.00 | 3 | 324k | $0.16 published-rate | |
| 18 | claude-code | ecb67e4e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 229k | $0.13 published-rate | |
| 19 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 179k | $0.07 published-rate | |
| 20 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 37k | $0.16 published-rate | |
| 21 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 44k | $0.09 published-rate | |
| 22 | codex | 3437a9e6 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 45k | $0.0068 published-rate | |
| 23 | codex | 3437a9e6 | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 36k | $0.05 published-rate | |
| 24 | codex | 3437a9e6 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 56k | $0.04 published-rate | 3 infra:proxy_request_conversion_failed |
| 25 | codex | 3437a9e6 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 159k | $0.13 published-rate | |
| 26 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 251k | $0.18 published-rate | |
| 27 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 145k | $0.11 published-rate | |
| 28 | codex | 3437a9e6 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 46k | $0.04 published-rate | |
| 29 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 1.00 | 3 | 60k | $0.02 published-rate | |
| 30 | codex | b2396403 | openai/gpt-6.1-sol @high | 3/3 | 44%–100% | 1.00 | 3 | 81k | $0.04 published-rate | |
| 31 | codex | b2396403 | openai/gpt-6.1-sol @medium | 3/3 | 44%–100% | 1.00 | 3 | 72k | $0.03 published-rate | |
| 32 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 61k | $0.10 published-rate | |
| 33 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 61k | $0.04 published-rate | |
| 34 | opencode | 357da891 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 69k | $0.0079 published-rate | |
| 35 | opencode | 357da891 | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 66k | $0.08 published-rate | |
| 36 | opencode | 357da891 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 86k | $0.06 published-rate | |
| 37 | opencode | 357da891 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 217k | $0.14 published-rate | |
| 38 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 446k | $0.15 published-rate | |
| 39 | opencode | 357da891 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 135k | $0.10 published-rate | |
| 40 | opencode | 357da891 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 78k | $0.08 published-rate | |
| 41 | opencode | 357da891 | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 108k | $0.06 published-rate | |
| 42 | pi | da41c66e | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 16k | $0.08 published-rate | |
| 43 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 15k | $0.03 published-rate | |
| 44 | pi | da41c66e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 27k | $0.0041 published-rate | |
| 45 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 29k | $0.04 published-rate | |
| 46 | pi | da41c66e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 55k | $0.05 published-rate | |
| 47 | pi | da41c66e | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 100k | $0.08 published-rate | |
| 48 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 136k | $0.11 published-rate | |
| 49 | pi | da41c66e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 60k | $0.05 published-rate | |
| 50 | pi | da41c66e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 27k | $0.03 published-rate | |
| 51 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 25k | $0.02 published-rate | |
| 52 | pi | da41c66e | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 1.00 | 3 | 23k | $0.0077 published-rate | |
| 53 | opencode | 357da891 | zai/glm-5.2 | 5/6 | 44%–97% | 0.94 | 6 | 129k | $0.07 published-rate | |
| 54 | pi | da41c66e | zai/glm-5.2 | 5/6 | 44%–97% | 0.94 | 6 | 84k | $0.07 published-rate |
fix · task version 2
3 tasks: fix-corrupt-refresh-checkpoint, fix-rescue-local-tip, fix-stale-lock-recovery.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 13/13 | 77%–100% | 1.00 | 3 | 111k | $0.05 published-rate | |
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 12/12 | 76%–100% | 1.00 | 3 | 122k | $0.06 published-rate | |
| 3 | pi | da41c66e | google/gemini-3.8-flash @high | 6/6 | 61%–100% | 0.78 | 3 | 1.19M | $0.39 published-rate | 3 infra:proxy_recreated_mid_run |
| 4 | codex | 3437a9e6 | fireworks/kimi-k3 | 5/5 | 57%–100% | 1.00 | 3 | 58k | $0.08 published-rate | |
| 5 | codex | b2396403 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 1.00 | 3 | 106k | $0.04 published-rate | |
| 6 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 5/5 | 57%–100% | 1.00 | 1 | 145k | $0.09 published-rate | |
| 7 | opencode | 357da891 | baseten/glm-5.3-flash | 5/5 | 57%–100% | 0.93 | 3 | 110k | $0.01 published-rate | |
| 8 | opencode | 357da891 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 1.00 | 3 | 130k | $0.08 published-rate | |
| 9 | pi | da41c66e | baseten/glm-5.3-flash | 5/5 | 57%–100% | 0.80 | 3 | 53k | $0.0089 published-rate | |
| 10 | pi | da41c66e | fireworks/qwen3.8-max | 5/5 | 57%–100% | 0.67 | 3 | 118k | $0.10 published-rate | |
| 11 | pi | da41c66e | google/gemini-3.8-flash | 5/5 | 57%–100% | 0.93 | 3 | 879k | $0.29 published-rate | 1 infra:agent_error, 1 infra:gateway_continued_run_changed_prompt |
| 12 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 4/4 | 51%–100% | 0.92 | 3 | 298k | $0.18 published-rate | 1 infra:agent_error |
| 13 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 4/4 | 51%–100% | 0.83 | 3 | 374k | $0.12 published-rate | |
| 14 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 3/3 | 44%–100% | 1.00 | 3 | 193k | unknown | |
| 15 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.89 | 3 | 1.17M | $0.31 published-rate | |
| 16 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 0.89 | 3 | 187k | $0.20 published-rate | |
| 17 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.89 | 3 | 765k | $0.13 published-rate | |
| 18 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.89 | 3 | 373k | $0.03 published-rate | |
| 19 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.89 | 3 | 1.33M | $0.25 published-rate | |
| 20 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 0.89 | 3 | 514k | $0.26 published-rate | |
| 21 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 191k | $0.08 published-rate | |
| 22 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 46k | $0.10 published-rate | |
| 23 | codex | 3437a9e6 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 0.89 | 3 | 106k | $0.08 published-rate | 3 infra:proxy_request_conversion_failed |
| 24 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.89 | 3 | 809k | $0.29 published-rate | |
| 25 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.89 | 3 | 319k | $0.15 published-rate | |
| 26 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.89 | 3 | 55k | $0.05 published-rate | |
| 27 | opencode | 357da891 | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.89 | 3 | 88k | $0.14 published-rate | |
| 28 | opencode | 357da891 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 1.00 | 3 | 123k | $0.09 published-rate | |
| 29 | opencode | 357da891 | google/gemini-3.8-flash | 3/3 | 44%–100% | 0.89 | 3 | 645k | $0.23 published-rate | |
| 30 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 943k | $0.25 published-rate | |
| 31 | opencode | 357da891 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.89 | 3 | 225k | $0.10 published-rate | |
| 32 | opencode | 357da891 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 94k | $0.12 published-rate | |
| 33 | opencode | 357da891 | zai/glm-5.2 | 3/3 | 44%–100% | 1.00 | 3 | 123k | $0.09 published-rate | |
| 34 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 19k | $0.05 published-rate | |
| 35 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.89 | 3 | 37k | $0.07 published-rate | |
| 36 | pi | da41c66e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.89 | 3 | 105k | $0.08 published-rate | |
| 37 | pi | da41c66e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 39k | $0.07 published-rate | 1 infra:agent_error |
| 38 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 34k | $0.03 published-rate | |
| 39 | claude-code | ecb67e4e | moonshot/kimi-k3 | 2/2 | 34%–100% | 1.00 | 2 | 215k | $0.17 published-rate | |
| 40 | pi | da41c66e | together/deepseek-v4.1-flash | 5/6 | 44%–97% | 0.83 | 3 | 57k | $0.02 published-rate | |
| 41 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 4/6 | 30%–90% | 0.67 | 3 | 2.18M | $0.73 published-rate | |
| 42 | claude-code | ecb67e4e | fireworks/kimi-k3 | 2/3 | 21%–94% | 0.89 | 3 | 335k | $0.16 published-rate | |
| 43 | claude-code | ecb67e4e | google/gemini-3.8-flash | 2/3 | 21%–94% | 1.00 | 3 | 1.05M | $0.28 published-rate | |
| 44 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.78 | 3 | 318k | $0.21 published-rate | |
| 45 | claude-code | ecb67e4e | zai/glm-5.2 | 2/3 | 21%–94% | 0.89 | 3 | 758k | $0.39 published-rate | |
| 46 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.78 | 3 | 67k | $0.31 published-rate | |
| 47 | codex | 3437a9e6 | baseten/glm-5.3-flash | 2/3 | 21%–94% | 0.78 | 3 | 91k | $0.01 published-rate | 1 infra:agent_error |
| 48 | codex | 3437a9e6 | google/gemini-3.8-flash | 2/3 | 21%–94% | 1.00 | 3 | 246k | $0.21 published-rate | |
| 49 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 2/3 | 21%–94% | 0.89 | 3 | 90k | $0.03 published-rate | |
| 50 | opencode | 357da891 | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.78 | 3 | 78k | $0.16 published-rate | |
| 51 | opencode | 357da891 | together/deepseek-v4.1-flash | 2/3 | 21%–94% | 0.89 | 3 | 108k | $0.04 published-rate | |
| 52 | pi | da41c66e | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.78 | 3 | 31k | $0.15 published-rate | |
| 53 | pi | da41c66e | zai/glm-5.2 | 2/3 | 21%–94% | 0.89 | 3 | 99k | $0.10 published-rate | |
| 54 | codex | 3437a9e6 | zai/glm-5.2 | 2/4 | 15%–85% | 0.83 | 3 | 243k | $0.21 published-rate | |
| 55 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 2/5 | 12%–77% | 0.47 | 1 | 4.67M | $2.51 published-rate | |
| 56 | codex | 3437a9e6 | moonshot/kimi-k3 | 1/3 | 6%–79% | 0.78 | 3 | 212k | $0.24 published-rate |
hard-fix
16 tasks: hard-fix-cross-file, hard-fix-cross-file-v2, hard-fix-cross-file-v3, hard-fix-regression-test-dedupe, hard-fix-regression-test-dedupe-v2, hard-fix-regression-test-dedupe-v3, hard-fix-regression-test-quiet-hours, hard-fix-regression-test-quiet-hours-v2, hard-fix-regression-test-quiet-hours-v3, hard-fix-two-findings, hard-fix-two-findings-v2, hard-fix-two-findings-v3, hard-fix-wrong-recommendation-config-reload, hard-fix-wrong-recommendation-retry-charge, hard-fix-wrong-recommendation-retry-charge-v2, hard-fix-wrong-recommendation-retry-charge-v3.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 26/26 | 87%–100% | 1.00 | 16 | 97k | $0.04 published-rate | |
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 20/20 | 84%–100% | 1.00 | 11 | 125k | $0.05 published-rate | |
| 3 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 15/15 | 80%–100% | 1.00 | 11 | 1.06M | $0.38 published-rate | |
| 4 | codex | b2396403 | openai/gpt-6.1-sol | 15/15 | 80%–100% | 1.00 | 11 | not measured (9/15) | unknown | |
| 5 | opencode | 357da891 | openai/gpt-6.1-sol | 13/13 | 77%–100% | 1.00 | 11 | 129k | $0.08 published-rate | |
| 6 | pi | da41c66e | fireworks/qwen3.8-max | 13/13 | 77%–100% | 1.00 | 11 | 96k | $0.09 published-rate | |
| 7 | pi | da41c66e | google/gemini-3.8-flash | 13/13 | 77%–100% | 1.00 | 11 | 456k | $0.20 published-rate | 1 infra:gateway_continued_run_changed_prompt |
| 8 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 804k | $0.14 published-rate | |
| 9 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 11/11 | 74%–100% | 1.00 | 11 | 223k | $0.09 published-rate | |
| 10 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 57k | $0.26 published-rate | |
| 11 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 42k | $0.09 published-rate | |
| 12 | codex | 3437a9e6 | baseten/glm-5.3-flash | 11/11 | 74%–100% | 1.00 | 11 | 62k | $0.0091 published-rate | |
| 13 | codex | 3437a9e6 | fireworks/qwen3.8-max | 11/11 | 74%–100% | 1.00 | 11 | 133k | $0.11 published-rate | 11 infra:proxy_request_conversion_failed |
| 14 | codex | 3437a9e6 | google/gemini-3.8-flash | 11/11 | 74%–100% | 1.00 | 11 | 422k | $0.20 published-rate | |
| 15 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 11/11 | 74%–100% | 1.00 | 11 | 828k | $0.27 published-rate | |
| 16 | opencode | 357da891 | anthropic/claude-opus-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 58k | $0.12 published-rate | |
| 17 | opencode | 357da891 | google/gemini-3.8-flash | 11/11 | 74%–100% | 1.00 | 11 | 508k | $0.20 published-rate | |
| 18 | pi | da41c66e | anthropic/claude-opus-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 24k | $0.12 published-rate | |
| 19 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 11/11 | 74%–100% | 1.00 | 11 | 18k | $0.04 published-rate | |
| 20 | pi | da41c66e | openai/gpt-6.1-sol | 11/11 | 74%–100% | 1.00 | 11 | 38k | $0.03 published-rate | |
| 21 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 5/5 | 57%–100% | 1.00 | 1 | 2.06M | $0.75 published-rate | |
| 22 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 5/5 | 57%–100% | 1.00 | 1 | 142k | $0.07 published-rate | |
| 23 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 15/16 | 72%–99% | 0.99 | 16 | 238k | $0.23 published-rate | |
| 24 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 15/16 | 72%–99% | 0.99 | 16 | 312k | $0.19 published-rate | |
| 25 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 12/13 | 67%–99% | 0.97 | 11 | not measured (7/13) | unknown | |
| 26 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 12/13 | 67%–99% | 0.97 | 11 | 429k | $0.14 published-rate | |
| 27 | pi | da41c66e | google/gemini-3.8-flash @high | 12/13 | 67%–99% | 0.97 | 11 | 947k | $0.31 published-rate | 1 infra:agent_error |
| 28 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 10/11 | 62%–98% | 0.97 | 11 | not measured (5/11) | unknown | |
| 29 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 10/11 | 62%–98% | 0.97 | 11 | 1.10M | $0.34 published-rate | 1 infra:agent_error |
| 30 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 10/11 | 62%–98% | 0.97 | 11 | 432k | $0.03 published-rate | |
| 31 | claude-code | ecb67e4e | fireworks/kimi-k3 | 10/11 | 62%–98% | 0.97 | 11 | 306k | $0.16 published-rate | 1 infra:agent_error |
| 32 | claude-code | ecb67e4e | google/gemini-3.8-flash | 10/11 | 62%–98% | 1.00 | 11 | 1.32M | $0.29 published-rate | |
| 33 | claude-code | ecb67e4e | moonshot/kimi-k3 | 10/11 | 62%–98% | 0.97 | 11 | 457k | $0.27 published-rate | 3 infra:agent_error |
| 34 | claude-code | ecb67e4e | zai/glm-5.2 | 10/11 | 62%–98% | 0.97 | 11 | not measured (5/11) | unknown | |
| 35 | codex | 3437a9e6 | moonshot/kimi-k3 | 10/11 | 62%–98% | 0.97 | 11 | 123k | $0.14 published-rate | |
| 36 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 10/11 | 62%–98% | 0.97 | 11 | 95k | $0.03 published-rate | |
| 37 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 10/11 | 62%–98% | 1.00 | 11 | 68k | $0.06 published-rate | |
| 38 | opencode | 357da891 | fireworks/qwen3.8-max | 10/11 | 62%–98% | 0.97 | 11 | 238k | $0.18 published-rate | |
| 39 | opencode | 357da891 | zai/glm-5.2 | 10/11 | 62%–98% | 0.97 | 11 | 240k | $0.15 published-rate | |
| 40 | pi | da41c66e | fireworks/kimi-k3 | 10/11 | 62%–98% | 0.97 | 11 | 54k | $0.10 published-rate | |
| 41 | pi | da41c66e | moonshot/kimi-k3 | 10/11 | 62%–98% | 0.97 | 11 | 62k | $0.11 published-rate | |
| 42 | pi | da41c66e | zai/glm-5.2 | 10/11 | 62%–98% | 0.97 | 11 | 134k | $0.11 published-rate | |
| 43 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 14/16 | 64%–97% | 0.97 | 16 | 787k | $0.30 published-rate | |
| 44 | opencode | 357da891 | google/gemini-3.8-flash @high | 14/16 | 64%–97% | 0.97 | 16 | 1.30M | $0.38 published-rate | |
| 45 | opencode | 357da891 | google/gemini-3.8-flash @low | 14/16 | 64%–97% | 0.97 | 16 | 248k | $0.14 published-rate | |
| 46 | opencode | 357da891 | together/deepseek-v4.1-flash | 14/16 | 64%–97% | 0.96 | 11 | not measured (5/16) | unknown | 1 infra:integrity_hold |
| 47 | pi | da41c66e | google/gemini-3.8-flash @low | 14/16 | 64%–97% | 0.97 | 16 | 110k | $0.09 published-rate | |
| 48 | opencode | 357da891 | moonshot/kimi-k3 | 9/11 | 52%–95% | 0.97 | 11 | 151k | $0.18 published-rate | |
| 49 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 13/16 | 57%–93% | 0.97 | 16 | 1.79M | $0.39 published-rate | |
| 50 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 13/16 | 57%–93% | 0.95 | 16 | 333k | $0.19 published-rate | |
| 51 | pi | da41c66e | together/deepseek-v4.1-flash | 12/15 | 55%–93% | 0.93 | 11 | 65k | $0.02 published-rate | |
| 52 | codex | 3437a9e6 | fireworks/kimi-k3 | 10/13 | 50%–92% | 0.92 | 11 | 89k | $0.11 published-rate | |
| 53 | codex | 3437a9e6 | zai/glm-5.2 | 10/13 | 50%–92% | 0.92 | 11 | 159k | $0.11 published-rate | |
| 54 | opencode | 357da891 | fireworks/kimi-k3 | 8/11 | 43%–90% | 0.96 | 11 | 176k | $0.24 published-rate | |
| 55 | opencode | 357da891 | baseten/glm-5.3-flash | 9/13 | 42%–87% | 0.92 | 11 | 162k | $0.02 published-rate | |
| 56 | pi | da41c66e | baseten/glm-5.3-flash | 9/13 | 42%–87% | 0.90 | 11 | 61k | $0.0099 published-rate |
hard-review · task version 1
9 tasks: hard-review-clean-long, hard-review-cross-file, hard-review-cross-file-v2, hard-review-decoy-decision-replies, hard-review-decoy-job-leases, hard-review-multi-defect-outbox, hard-review-multi-defect-outbox-v2, hard-review-multi-defect-tool-calls, hard-review-multi-defect-tool-calls-v2.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | pi | da41c66e | together/deepseek-v4.1-flash | 13/13 | 77%–100% | 0.93 | 6 | 244k | $0.09 published-rate | |
| 2 | opencode | 357da891 | together/deepseek-v4.1-flash | 9/10 | 60%–98% | 0.85 | 9 | not measured (3/10) | unknown | 5 infra:source_infra_list |
| 3 | codex | 3437a9e6 | zai/glm-5.2 | 16/18 | 67%–97% | 0.87 | 8 | 490k | $0.48 published-rate | 1 infra:gateway_restart_cut_stream |
| 4 | claude-code | ecb67e4e | zai/glm-5.2 | 8/9 | 57%–98% | 0.84 | 9 | not measured (5/9) | unknown | 10 infra:agent_error, 1 infra:harness_output_limit_cut_reply |
| 5 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 21/24 | 69%–96% | 0.94 | 8 | 1.77M | $0.59 published-rate | |
| 6 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 7/8 | 53%–98% | 0.89 | 8 | 272k | $0.41 published-rate | |
| 7 | opencode | 357da891 | fireworks/kimi-k3 | 7/8 | 53%–98% | 0.89 | 8 | 231k | $0.38 published-rate | |
| 8 | opencode | 357da891 | zai/glm-5.2 | 7/8 | 53%–98% | 0.85 | 8 | 294k | $0.26 published-rate | |
| 9 | pi | da41c66e | fireworks/kimi-k3 | 7/8 | 53%–98% | 0.84 | 8 | 159k | $0.32 published-rate | |
| 10 | pi | da41c66e | google/gemini-3.8-flash | 7/8 | 53%–98% | 0.92 | 8 | 1.08M | $0.42 published-rate | |
| 11 | pi | da41c66e | google/gemini-3.8-flash @high | 7/8 | 53%–98% | 0.94 | 8 | 1.89M | $0.64 published-rate | |
| 12 | pi | da41c66e | moonshot/kimi-k3 | 7/8 | 53%–98% | 0.82 | 8 | 147k | $0.28 published-rate | |
| 13 | pi | da41c66e | openai/gpt-6.1-sol | 7/8 | 53%–98% | 0.96 | 8 | 77k | $0.07 published-rate | |
| 14 | pi | da41c66e | zai/glm-5.2 | 7/8 | 53%–98% | 0.84 | 8 | 305k | $0.28 published-rate | |
| 15 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 15/18 | 61%–94% | 0.89 | 8 | 816k | $0.29 published-rate | |
| 16 | opencode | 357da891 | baseten/glm-5.3-flash | 15/18 | 61%–94% | 0.89 | 8 | 251k | $0.03 published-rate | 1 infra:agent_error |
| 17 | pi | da41c66e | baseten/glm-5.3-flash | 14/18 | 55%–91% | 0.86 | 8 | 203k | $0.03 published-rate | |
| 18 | pi | da41c66e | fireworks/qwen3.8-max | 14/18 | 55%–91% | 0.89 | 8 | 377k | $0.30 published-rate | |
| 19 | codex | b2396403 | openai/gpt-6.1-sol @medium | 38/50 | 63%–86% | 0.94 | 5 | 185k | $0.09 published-rate | |
| 20 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 6/8 | 41%–93% | 0.87 | 8 | 1.27M | $0.09 published-rate | |
| 21 | claude-code | ecb67e4e | fireworks/kimi-k3 | 6/8 | 41%–93% | 0.85 | 8 | 327k | $0.24 published-rate | |
| 22 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 6/8 | 41%–93% | 0.95 | 8 | 390k | $0.21 published-rate | |
| 23 | codex | 3437a9e6 | google/gemini-3.8-flash | 6/8 | 41%–93% | 0.91 | 8 | 1.28M | $0.49 published-rate | |
| 24 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 6/8 | 41%–93% | 0.81 | 8 | 164k | $0.14 published-rate | 1 infra:agent_error |
| 25 | opencode | 357da891 | fireworks/qwen3.8-max | 6/8 | 41%–93% | 0.94 | 8 | 396k | $0.34 published-rate | |
| 26 | opencode | 357da891 | moonshot/kimi-k3 | 6/8 | 41%–93% | 0.80 | 8 | 244k | $0.40 published-rate | |
| 27 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 16/22 | 52%–87% | 0.90 | 9 | not measured (13/22) | unknown | 7 infra:agent_error, 4 infra:error |
| 28 | codex | b2396403 | openai/gpt-6.1-sol | 20/29 | 51%–83% | 0.92 | 9 | not measured (23/29) | unknown | |
| 29 | claude-code | ecb67e4e | google/gemini-3.8-flash | 5/8 | 31%–86% | 0.92 | 8 | 2.40M | $0.71 published-rate | |
| 30 | claude-code | ecb67e4e | moonshot/kimi-k3 | 5/8 | 31%–86% | 0.79 | 8 | 599k | $0.39 published-rate | |
| 31 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 5/8 | 31%–86% | 0.91 | 8 | 145k | $0.65 published-rate | |
| 32 | codex | 3437a9e6 | moonshot/kimi-k3 | 5/8 | 31%–86% | 0.76 | 8 | 248k | $0.32 published-rate | |
| 33 | opencode | 357da891 | google/gemini-3.8-flash | 5/8 | 31%–86% | 0.83 | 8 | 1.45M | $0.53 published-rate | |
| 34 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 5/8 | 31%–86% | 0.82 | 8 | 98k | $0.24 published-rate | |
| 35 | codex | 3437a9e6 | fireworks/kimi-k3 | 11/18 | 39%–80% | 0.79 | 8 | 210k | $0.29 published-rate | |
| 36 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 14/25 | 37%–73% | 0.90 | 5 | 336k | $0.20 published-rate | |
| 37 | opencode | 357da891 | openai/gpt-6.1-sol | 9/18 | 29%–71% | 0.89 | 8 | 437k | $0.27 published-rate | |
| 38 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 4/8 | 22%–78% | 0.75 | 8 | 168k | $0.37 published-rate | |
| 39 | codex | 3437a9e6 | fireworks/qwen3.8-max | 4/8 | 22%–78% | 0.87 | 8 | 594k | $0.61 published-rate | 8 infra:proxy_request_conversion_failed |
| 40 | pi | da41c66e | anthropic/claude-opus-5-5 | 4/8 | 22%–78% | 0.89 | 8 | 117k | $0.58 published-rate | |
| 41 | codex | b2396403 | openai/gpt-6.1-sol @high | 17/35 | 33%–64% | 0.87 | 8 | 320k | $0.18 published-rate | 7 infra:agent_error |
| 42 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 4/9 | 19%–73% | 0.70 | 9 | not measured (3/9) | unknown | |
| 43 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 8/20 | 22%–61% | 0.85 | 5 | 4.53M | $2.78 published-rate | 3 infra:agent_error |
| 44 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 11/28 | 24%–58% | 0.85 | 8 | 3.00M | $1.20 published-rate | |
| 45 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 3/8 | 14%–69% | 0.80 | 8 | 2.53M | $0.55 published-rate | |
| 46 | codex | 3437a9e6 | baseten/glm-5.3-flash | 3/8 | 14%–69% | 0.76 | 8 | 485k | $0.07 published-rate | |
| 47 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/8 | 14%–69% | 0.84 | 8 | 262k | $0.52 published-rate | |
| 48 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 0/3 | 0%–56% | 0.70 | 3 | no passes | unknown |
hard-review · task version 2
9 tasks: hard-review-clean-long, hard-review-cross-file, hard-review-cross-file-v2, hard-review-decoy-decision-replies, hard-review-decoy-job-leases, hard-review-multi-defect-outbox, hard-review-multi-defect-outbox-v2, hard-review-multi-defect-tool-calls, hard-review-multi-defect-tool-calls-v2.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @max | 4/4 | 51%–100% | 1.00 | 3 | 892k | $3.11 published-rate | 2 infra:agent_error, 5 infra:gateway_continued_run_changed_prompt, 4 infra:harness_gave_up_on_reply_stream |
| 2 | claude-code | c32a950b | anthropic/claude-haiku-5-5 | 2/2 | 34%–100% | 0.90 | 2 | 788k | $0.03 published-rate | 1 infra:error |
| 3 | opencode | 357da891 | anthropic/claude-opus-5-5 @max | 1/1 | 21%–100% | 1.00 | 1 | 1.02M | $4.58 published-rate | 1 infra:gateway_stream_ended_before_reply |
| 4 | pi | da41c66e | anthropic/claude-opus-5-5 @max | 1/1 | 21%–100% | 1.00 | 1 | 427k | $3.22 published-rate | |
| 5 | pi | da41c66e | together/deepseek-v4.1-flash | 31/32 | 84%–99% | 0.90 | 9 | 216k | $0.08 published-rate | 1 infra:grader_tool_call_evidence_denied, 1 infra:harness_out_of_memory_on_command_output |
| 6 | codex | 3437a9e6 | zai/glm-5.2 | 20/22 | 72%–97% | 0.87 | 9 | 482k | $0.52 published-rate | |
| 7 | opencode | 357da891 | together/deepseek-v4.1-flash | 10/11 | 62%–98% | 0.85 | 9 | 142k | $0.05 published-rate | |
| 8 | claude-code | ecb67e4e | zai/glm-5.2 | 9/10 | 60%–98% | 0.85 | 9 | 1.37M | $0.64 published-rate | 9 infra:agent_error |
| 9 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 9/10 | 60%–98% | 0.89 | 9 | 216k | $0.09 published-rate | |
| 10 | opencode | 357da891 | fireworks/kimi-k3 | 9/10 | 60%–98% | 0.87 | 9 | 213k | $0.36 published-rate | |
| 11 | opencode | 357da891 | zai/glm-5.2 | 9/10 | 60%–98% | 0.84 | 9 | 268k | $0.24 published-rate | |
| 12 | pi | da41c66e | fireworks/kimi-k3 | 9/10 | 60%–98% | 0.83 | 9 | 142k | $0.29 published-rate | |
| 13 | pi | da41c66e | moonshot/kimi-k3 | 9/10 | 60%–98% | 0.82 | 9 | 148k | $0.29 published-rate | |
| 14 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 24/27 | 72%–96% | 0.92 | 9 | 1.67M | $0.58 published-rate | |
| 15 | claude-code | c32a950b | anthropic/claude-opus-5-5 @max | 8/9 | 57%–98% | 0.94 | 8 | 6.10M | $4.53 published-rate | |
| 16 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @high | 8/9 | 57%–98% | 0.96 | 8 | 130k | $0.59 published-rate | |
| 17 | antigravity | 034ed3a1 | google/gemini-3.8-flash @medium | 7/8 | 53%–98% | 0.94 | 8 | 1.50M | $0.48 published-rate | |
| 18 | antigravity | 034ed3a1 | google/gemini-4-argon @medium | 7/8 | 53%–98% | 0.89 | 8 | 144k | $0.18 published-rate | |
| 19 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @max | 7/8 | 53%–98% | 0.94 | 8 | 12.51M | $3.98 published-rate | |
| 20 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @xhigh | 7/8 | 53%–98% | 0.88 | 8 | 1.91M | $0.60 published-rate | |
| 21 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 7/8 | 53%–98% | 0.92 | 8 | 938k | $0.38 published-rate | |
| 22 | claude-code | ecb67e4e | openai/gpt-5.6-terra @low | 7/8 | 53%–98% | 0.85 | 8 | 197k | $0.14 published-rate | 2 infra:worker_shell_broke_file_listing |
| 23 | claude-code | ecb67e4e | openai/gpt-6-luna @high | 7/8 | 53%–98% | 0.84 | 8 | 332k | $0.01 published-rate | |
| 24 | claude-code | ecb67e4e | openai/gpt-6.1-sol @low | 7/8 | 53%–98% | 0.98 | 8 | 263k | $0.11 published-rate | 8 infra:error |
| 25 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 7/8 | 53%–98% | 1.00 | 8 | 1.72M | $0.59 published-rate | 1 infra:harness_gave_up_on_reply_stream |
| 26 | codex | b2396403 | openai/gpt-6-luna @xhigh | 7/8 | 53%–98% | 0.90 | 8 | 242k | $0.0093 published-rate | |
| 27 | codex | b2396403 | openai/gpt-6.1-sol @low | 7/8 | 53%–98% | 0.96 | 8 | 157k | $0.07 published-rate | |
| 28 | opencode | 357da891 | anthropic/claude-opus-5-5 @xhigh | 7/8 | 53%–98% | 0.93 | 8 | 378k | $0.89 published-rate | 7 infra:agent_error |
| 29 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @xhigh | 7/8 | 53%–98% | 0.89 | 8 | 279k | $0.35 published-rate | 1 infra:agent_error |
| 30 | opencode | 357da891 | google/gemini-3.8-flash @high | 7/8 | 53%–98% | 0.94 | 8 | 1.88M | $0.55 published-rate | |
| 31 | opencode | 357da891 | google/gemini-3.8-flash @medium | 7/8 | 53%–98% | 0.94 | 8 | 1.12M | $0.47 published-rate | 1 infra:agent_error, 1 infra:gateway_stream_read_timeout |
| 32 | opencode | 357da891 | openai/gpt-6-luna @low | 7/8 | 53%–98% | 0.86 | 8 | 97k | $0.0039 published-rate | |
| 33 | pi | da41c66e | anthropic/claude-opus-5-5 @xhigh | 7/8 | 53%–98% | 0.91 | 8 | 157k | $0.81 published-rate | |
| 34 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 19/22 | 67%–95% | 0.88 | 9 | 705k | $0.25 published-rate | |
| 35 | opencode | 357da891 | baseten/glm-5.3-flash | 19/22 | 67%–95% | 0.87 | 9 | 235k | $0.03 published-rate | 1 infra:agent_error |
| 36 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @max | 6/7 | 49%–97% | 0.90 | 6 | 886k | $5.72 published-rate | 2 infra:agent_error, 4 infra:gateway_continued_run_changed_prompt |
| 37 | pi | da41c66e | google/gemini-3.8-flash | 17/20 | 64%–95% | 0.93 | 9 | 1.12M | $0.46 published-rate | 3 infra:agent_error, 4 infra:gateway_continued_run_changed_prompt |
| 38 | pi | da41c66e | google/gemini-3.8-flash @high | 10/12 | 55%–95% | 0.92 | 9 | 1.67M | $0.62 published-rate | 3 infra:error |
| 39 | codex | b2396403 | openai/gpt-6.1-sol @medium | 56/68 | 72%–90% | 0.92 | 9 | 170k | $0.09 published-rate | 1 infra:agent_error |
| 40 | pi | da41c66e | baseten/glm-5.3-flash | 18/22 | 61%–93% | 0.86 | 9 | 189k | $0.03 published-rate | 1 infra:grader_tool_call_evidence_denied, 1 infra:provider_overload_ended_run |
| 41 | pi | da41c66e | fireworks/qwen3.8-max | 17/21 | 60%–92% | 0.87 | 9 | 331k | $0.28 published-rate | |
| 42 | claude-code | ecb67e4e | fireworks/kimi-k3 | 8/10 | 49%–94% | 0.86 | 9 | 290k | $0.24 published-rate | |
| 43 | codex | 3437a9e6 | google/gemini-3.8-flash | 8/10 | 49%–94% | 0.91 | 9 | 1.11M | $0.44 published-rate | 1 infra:provider_throttle_ended_run |
| 44 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 8/10 | 49%–94% | 0.81 | 9 | 151k | $0.13 published-rate | 1 infra:agent_error |
| 45 | opencode | 357da891 | moonshot/kimi-k3 | 8/10 | 49%–94% | 0.80 | 9 | 229k | $0.37 published-rate | |
| 46 | pi | da41c66e | openai/gpt-6.1-sol | 8/10 | 49%–94% | 0.93 | 9 | 86k | $0.08 published-rate | 1 infra:grader_tool_call_evidence_denied |
| 47 | pi | da41c66e | zai/glm-5.2 | 8/10 | 49%–94% | 0.81 | 9 | 325k | $0.30 published-rate | |
| 48 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 22/28 | 60%–90% | 0.91 | 8 | 2.26M | $1.43 published-rate | 3 infra:agent_error |
| 49 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 7/9 | 45%–94% | 0.86 | 9 | 310k | $0.50 published-rate | |
| 50 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @xhigh | 7/9 | 45%–94% | 0.90 | 8 | 269k | $1.36 published-rate | |
| 51 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @xhigh | 7/9 | 45%–94% | 0.86 | 8 | 227k | $0.56 published-rate | |
| 52 | codex | 3437a9e6 | fireworks/qwen3.8-max | 7/9 | 45%–94% | 0.91 | 9 | 369k | $0.39 published-rate | 1 infra:proxy_request_conversion_failed |
| 53 | opencode | 357da891 | fireworks/qwen3.8-max | 7/9 | 45%–94% | 0.92 | 9 | 359k | $0.32 published-rate | |
| 54 | opencode | 357da891 | openai/gpt-6-luna @xhigh | 7/9 | 45%–94% | 0.88 | 8 | 441k | $0.02 published-rate | |
| 55 | opencode | 357da891 | openai/gpt-6.1-sol @low | 7/9 | 45%–94% | 0.94 | 8 | 121k | $0.09 published-rate | |
| 56 | pi | da41c66e | openai/gpt-6.1-sol @low | 7/9 | 45%–94% | 0.93 | 8 | 63k | $0.05 published-rate | |
| 57 | antigravity | 034ed3a1 | google/gemini-4-argon @low | 6/8 | 41%–93% | 0.87 | 8 | 153k | $0.12 published-rate | |
| 58 | claude-code | ecb67e4e | openai/gpt-5.6-terra @high | 6/8 | 41%–93% | 0.86 | 8 | 296k | $0.27 published-rate | |
| 59 | codex | 3437a9e6 | google/gemini-3.8-flash @medium | 6/8 | 41%–93% | 0.89 | 8 | 1.26M | $0.47 published-rate | |
| 60 | codex | b2396403 | openai/gpt-5.6-terra @high | 6/8 | 41%–93% | 0.88 | 8 | 343k | $0.20 published-rate | 1 infra:agent_error |
| 61 | codex | b2396403 | openai/gpt-5.6-terra @medium | 6/8 | 41%–93% | 0.83 | 8 | 258k | $0.15 published-rate | |
| 62 | codex | b2396403 | openai/gpt-5.6-terra @xhigh | 6/8 | 41%–93% | 0.86 | 8 | 366k | $0.27 published-rate | |
| 63 | codex | b2396403 | openai/gpt-6-luna @high | 6/8 | 41%–93% | 0.82 | 8 | 226k | $0.0063 published-rate | |
| 64 | codex | b2396403 | openai/gpt-6-luna @max | 6/8 | 41%–93% | 0.91 | 8 | 264k | $0.01 published-rate | |
| 65 | opencode | 357da891 | anthropic/claude-haiku-5-5 @medium | 6/8 | 41%–93% | 0.82 | 8 | 183k | $0.0080 published-rate | 1 infra:agent_error |
| 66 | opencode | 357da891 | openai/gpt-5.6-terra @low | 6/8 | 41%–93% | 0.87 | 8 | 194k | $0.13 published-rate | |
| 67 | opencode | 357da891 | openai/gpt-6-luna @high | 6/8 | 41%–93% | 0.89 | 8 | 282k | $0.01 published-rate | |
| 68 | pi | da41c66e | anthropic/claude-haiku-5-5 @high | 6/8 | 41%–93% | 0.79 | 8 | 151k | $0.02 published-rate | |
| 69 | pi | da41c66e | anthropic/claude-sonnet-5-5 @high | 6/8 | 41%–93% | 0.83 | 8 | 70k | $0.17 published-rate | |
| 70 | pi | da41c66e | anthropic/claude-sonnet-5-5 @xhigh | 6/8 | 41%–93% | 0.81 | 8 | 93k | $0.26 published-rate | |
| 71 | pi | da41c66e | google/gemini-3.8-flash @medium | 6/8 | 41%–93% | 0.90 | 8 | 1.35M | $0.44 published-rate | |
| 72 | pi | da41c66e | openai/gpt-6-luna @high | 6/8 | 41%–93% | 0.84 | 8 | 106k | $0.0076 published-rate | |
| 73 | codex | b2396403 | openai/gpt-6.1-sol | 23/32 | 55%–84% | 0.91 | 9 | 196k | $0.11 published-rate | |
| 74 | pi | da41c66e | google/gemini-3.8-flash @low | 5/7 | 36%–92% | 0.82 | 7 | 434k | $0.17 published-rate | 1 infra:error |
| 75 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 17/24 | 51%–85% | 0.88 | 9 | 430k | $0.35 published-rate | 5 infra:agent_error |
| 76 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 7/10 | 40%–89% | 0.86 | 9 | 1.19M | $0.09 published-rate | |
| 77 | claude-code | ecb67e4e | moonshot/kimi-k3 | 7/10 | 40%–89% | 0.79 | 9 | 485k | $0.34 published-rate | |
| 78 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 7/10 | 40%–89% | 0.92 | 9 | 387k | $0.21 published-rate | |
| 79 | opencode | 357da891 | google/gemini-3.8-flash | 7/10 | 40%–89% | 0.85 | 9 | 1.30M | $0.52 published-rate | 1 infra:provider_throttle_ended_run |
| 80 | codex | 3437a9e6 | fireworks/kimi-k3 | 15/22 | 47%–84% | 0.79 | 9 | 175k | $0.25 published-rate | |
| 81 | codex | b2396403 | openai/gpt-6.1-sol @high | 39/58 | 54%–78% | 0.90 | 9 | 230k | $0.13 published-rate | 9 infra:agent_error |
| 82 | claude-code | ecb67e4e | openai/gpt-6.1-sol @high | 6/9 | 35%–88% | 0.94 | 8 | 352k | $0.21 published-rate | |
| 83 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @xhigh | 6/9 | 35%–88% | 0.77 | 8 | 498k | $0.07 published-rate | |
| 84 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 6/9 | 35%–88% | 0.92 | 9 | 136k | $0.61 published-rate | |
| 85 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @medium | 6/9 | 35%–88% | 0.90 | 8 | 150k | $0.66 published-rate | |
| 86 | opencode | 357da891 | anthropic/claude-opus-5-5 | 6/9 | 35%–88% | 0.90 | 9 | 144k | $0.29 published-rate | |
| 87 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 21/33 | 47%–78% | 0.91 | 9 | 286k | $0.18 published-rate | |
| 88 | antigravity | 034ed3a1 | google/gemini-3.8-flash @low | 5/8 | 31%–86% | 0.85 | 8 | 881k | $0.22 published-rate | 1 infra:agent_error |
| 89 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @high | 5/8 | 31%–86% | 0.77 | 8 | 1.31M | $0.05 published-rate | |
| 90 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @low | 5/8 | 31%–86% | 0.81 | 8 | 766k | $0.02 published-rate | |
| 91 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @medium | 5/8 | 31%–86% | 0.79 | 8 | 1.03M | $0.03 published-rate | 1 infra:agent_error |
| 92 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @xhigh | 5/8 | 31%–86% | 0.81 | 8 | 2.62M | $0.12 published-rate | |
| 93 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 5/8 | 31%–86% | 0.90 | 8 | 2.24M | $0.67 published-rate | |
| 94 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 5/8 | 31%–86% | 0.81 | 8 | 542k | $0.20 published-rate | |
| 95 | claude-code | ecb67e4e | openai/gpt-6-luna @low | 5/8 | 31%–86% | 0.86 | 8 | 412k | $0.0099 published-rate | |
| 96 | claude-code | ecb67e4e | openai/gpt-6-luna @xhigh | 5/8 | 31%–86% | 0.84 | 8 | 310k | $0.02 published-rate | |
| 97 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @low | 5/8 | 31%–86% | 0.80 | 8 | 174k | $0.02 published-rate | |
| 98 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @high | 5/8 | 31%–86% | 0.84 | 8 | 160k | $0.35 published-rate | 3 infra:worker_boot_fenced |
| 99 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 5/8 | 31%–86% | 0.87 | 8 | 784k | $0.28 published-rate | |
| 100 | codex | b2396403 | openai/gpt-5.6-terra @low | 5/8 | 31%–86% | 0.79 | 8 | 322k | $0.18 published-rate | |
| 101 | codex | b2396403 | openai/gpt-5.6-terra @max | 5/8 | 31%–86% | 0.82 | 8 | 676k | $0.51 published-rate | |
| 102 | codex | b2396403 | openai/gpt-6.1-sol @max | 5/8 | 31%–86% | 0.91 | 8 | 350k | $0.20 published-rate | |
| 103 | opencode | 357da891 | anthropic/claude-haiku-5-5 @high | 5/8 | 31%–86% | 0.76 | 8 | 246k | $0.01 published-rate | |
| 104 | opencode | 357da891 | anthropic/claude-haiku-5-5 @low | 5/8 | 31%–86% | 0.78 | 8 | 175k | $0.0087 published-rate | |
| 105 | opencode | 357da891 | openai/gpt-5.6-terra @high | 5/8 | 31%–86% | 0.81 | 8 | 334k | $0.25 published-rate | |
| 106 | pi | da41c66e | anthropic/claude-haiku-5-5 @low | 5/8 | 31%–86% | 0.89 | 8 | 110k | $0.01 published-rate | |
| 107 | pi | da41c66e | anthropic/claude-haiku-5-5 @medium | 5/8 | 31%–86% | 0.75 | 8 | 131k | $0.02 published-rate | |
| 108 | pi | da41c66e | anthropic/claude-opus-5-5 @medium | 5/8 | 31%–86% | 0.84 | 8 | 85k | $0.41 published-rate | |
| 109 | pi | da41c66e | openai/gpt-5.6-terra @high | 5/8 | 31%–86% | 0.81 | 8 | 95k | $0.15 published-rate | |
| 110 | pi | da41c66e | openai/gpt-5.6-terra @low | 5/8 | 31%–86% | 0.77 | 8 | 72k | $0.08 published-rate | |
| 111 | pi | da41c66e | openai/gpt-6-luna @max | 5/8 | 31%–86% | 0.82 | 8 | 281k | $0.05 published-rate | 2 infra:gateway_continued_run_changed_prompt, 1 infra:harness_output_limit_cut_reply |
| 112 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 6/10 | 31%–83% | 0.81 | 9 | 1.49M | $0.33 published-rate | |
| 113 | claude-code | ecb67e4e | google/gemini-3.8-flash | 6/10 | 31%–83% | 0.93 | 9 | 2.42M | $0.72 published-rate | |
| 114 | codex | 3437a9e6 | baseten/glm-5.3-flash | 6/10 | 31%–83% | 0.80 | 9 | 271k | $0.04 published-rate | |
| 115 | codex | 3437a9e6 | moonshot/kimi-k3 | 6/10 | 31%–83% | 0.75 | 9 | 243k | $0.35 published-rate | |
| 116 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 6/10 | 31%–83% | 0.79 | 9 | 94k | $0.23 published-rate | 1 infra:grader_tool_call_evidence_denied |
| 117 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @medium | 5/9 | 27%–81% | 0.72 | 8 | 225k | $0.02 published-rate | |
| 118 | codex | b2396403 | openai/gpt-6.1-sol @ultra | 5/9 | 27%–81% | 0.92 | 8 | 504k | $0.20 published-rate | |
| 119 | opencode | 357da891 | anthropic/claude-opus-5-5 @high | 5/9 | 27%–81% | 0.82 | 8 | 270k | $0.54 published-rate | |
| 120 | opencode | 357da891 | openai/gpt-6.1-sol @high | 5/9 | 27%–81% | 0.92 | 8 | 406k | $0.23 published-rate | |
| 121 | opencode | 357da891 | openai/gpt-6.1-sol @max | 5/9 | 27%–81% | 0.90 | 8 | 562k | $0.59 published-rate | |
| 122 | pi | da41c66e | anthropic/claude-opus-5-5 | 5/9 | 27%–81% | 0.90 | 9 | 105k | $0.52 published-rate | |
| 123 | pi | da41c66e | anthropic/claude-opus-5-5 @high | 5/9 | 27%–81% | 0.88 | 8 | 110k | $0.54 published-rate | |
| 124 | pi | da41c66e | openai/gpt-6.1-sol @high | 5/9 | 27%–81% | 0.90 | 8 | 138k | $0.12 published-rate | 4 infra:harness_failed_to_start |
| 125 | pi | da41c66e | openai/gpt-6.1-sol @max | 5/9 | 27%–81% | 0.92 | 8 | 161k | $0.26 published-rate | 1 infra:harness_failed_to_start |
| 126 | opencode | 357da891 | openai/gpt-6.1-sol | 12/22 | 35%–73% | 0.89 | 9 | 399k | $0.25 published-rate | |
| 127 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 17/34 | 34%–66% | 0.87 | 9 | 2.27M | $0.95 published-rate | 1 infra:grader_tool_call_evidence_denied |
| 128 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 5/10 | 24%–76% | 0.74 | 9 | 161k | $0.35 published-rate | |
| 129 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @max | 4/8 | 22%–78% | 0.82 | 8 | 8.31M | $0.39 published-rate | |
| 130 | claude-code | c32a950b | anthropic/claude-opus-5-5 @high | 4/8 | 22%–78% | 0.89 | 8 | 1.91M | $0.83 published-rate | 2 infra:error |
| 131 | claude-code | c32a950b | anthropic/claude-opus-5-5 @medium | 4/8 | 22%–78% | 0.89 | 8 | 1.77M | $0.66 published-rate | 1 infra:agent_error |
| 132 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @high | 4/8 | 22%–78% | 0.80 | 8 | 1.87M | $0.40 published-rate | |
| 133 | claude-code | ecb67e4e | openai/gpt-6-luna @max | 4/8 | 22%–78% | 0.85 | 8 | 401k | $0.04 published-rate | |
| 134 | claude-code | ecb67e4e | openai/gpt-6.1-sol @max | 4/8 | 22%–78% | 0.91 | 8 | 612k | $0.54 published-rate | 1 infra:agent_error, 4 infra:worker_boot_fenced |
| 135 | codex | b2396403 | openai/gpt-6-luna @medium | 4/8 | 22%–78% | 0.80 | 8 | 268k | $0.0069 published-rate | |
| 136 | opencode | 357da891 | anthropic/claude-haiku-5-5 @xhigh | 4/8 | 22%–78% | 0.75 | 8 | 795k | $0.08 published-rate | |
| 137 | opencode | 357da891 | anthropic/claude-opus-5-5 @medium | 4/8 | 22%–78% | 0.84 | 8 | 207k | $0.46 published-rate | |
| 138 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @high | 4/8 | 22%–78% | 0.77 | 8 | 232k | $0.22 published-rate | |
| 139 | opencode | 357da891 | google/gemini-3.8-flash @low | 4/8 | 22%–78% | 0.81 | 8 | 422k | $0.13 published-rate | |
| 140 | pi | da41c66e | anthropic/claude-sonnet-5-5 @medium | 4/8 | 22%–78% | 0.83 | 8 | 94k | $0.21 published-rate | |
| 141 | pi | da41c66e | openai/gpt-6-luna @low | 4/8 | 22%–78% | 0.85 | 8 | 77k | $0.0038 published-rate | |
| 142 | pi | da41c66e | openai/gpt-6-luna @xhigh | 4/8 | 22%–78% | 0.82 | 8 | 198k | $0.02 published-rate | 1 infra:gateway_continued_run_changed_prompt |
| 143 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 4/9 | 19%–73% | 0.70 | 9 | 862k | unknown | |
| 144 | opencode | 357da891 | openai/gpt-6-luna @max | 4/9 | 19%–73% | 0.79 | 8 | 759k | $0.08 published-rate | |
| 145 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @low | 3/8 | 14%–69% | 0.78 | 8 | 2.29M | $0.40 published-rate | |
| 146 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @medium | 3/8 | 14%–69% | 0.82 | 8 | 2.14M | $0.42 published-rate | 4 infra:dispatch_ran_other_model |
| 147 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @high | 3/8 | 14%–69% | 0.64 | 8 | 604k | $0.07 published-rate | |
| 148 | codex | b2396403 | openai/gpt-5.6-terra @ultra | 3/8 | 14%–69% | 0.72 | 8 | 2.05M | $1.01 published-rate | |
| 149 | opencode | 357da891 | anthropic/claude-haiku-5-5 @max | 3/8 | 14%–69% | 0.81 | 8 | 3.49M | $0.50 published-rate | |
| 150 | pi | da41c66e | anthropic/claude-haiku-5-5 @max | 3/8 | 14%–69% | 0.70 | 8 | 2.15M | $0.42 published-rate | |
| 151 | pi | da41c66e | anthropic/claude-haiku-5-5 @xhigh | 3/8 | 14%–69% | 0.72 | 8 | 900k | $0.13 published-rate | |
| 152 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @max | 3/9 | 12%–65% | 0.74 | 8 | 2.28M | $0.43 published-rate | 1 infra:agent_error, 8 infra:error |
| 153 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @low | 3/9 | 12%–65% | 0.81 | 8 | 244k | $1.04 published-rate | 1 infra:agent_error |
| 154 | opencode | 357da891 | anthropic/claude-opus-5-5 @low | 3/9 | 12%–65% | 0.86 | 8 | 281k | $0.47 published-rate | |
| 155 | claude-code | c32a950b | anthropic/claude-opus-5-5 @low | 2/8 | 7%–59% | 0.82 | 8 | 3.25M | $1.09 published-rate | 1 infra:agent_error, 2 infra:dispatch_ran_other_model |
| 156 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @low | 2/8 | 7%–59% | 0.74 | 8 | 353k | $0.75 published-rate | |
| 157 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @medium | 2/8 | 7%–59% | 0.75 | 8 | 461k | $0.36 published-rate | 2 infra:agent_error |
| 158 | pi | da41c66e | anthropic/claude-sonnet-5-5 @low | 2/8 | 7%–59% | 0.77 | 8 | 172k | $0.39 published-rate | |
| 159 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @medium | 2/9 | 6%–55% | 0.77 | 8 | 450k | $0.96 published-rate | |
| 160 | pi | da41c66e | anthropic/claude-opus-5-5 @low | 2/9 | 6%–55% | 0.84 | 8 | 206k | $0.93 published-rate | |
| 161 | codex | b2396403 | openai/gpt-6-luna @low | 1/8 | 2%–47% | 0.73 | 8 | 1.00M | $0.03 published-rate | |
| 162 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @low | 1/8 | 2%–47% | 0.68 | 8 | 921k | $0.67 published-rate |
hard-review · task version 3
1 tasks: hard-review-multi-defect-tool-calls.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 11/11 | 74%–100% | 0.82 | 1 | 134k | $0.08 published-rate | 1 infra:agent_error |
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 10/10 | 72%–100% | 0.84 | 1 | 161k | $0.10 published-rate | 1 infra:agent_error |
| 3 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 5/5 | 57%–100% | 1.00 | 1 | 1.05M | $0.52 published-rate | |
| 4 | codex | b2396403 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 0.80 | 1 | 132k | $0.08 published-rate | |
| 5 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 5/5 | 57%–100% | 0.92 | 1 | 170k | $0.12 published-rate | |
| 6 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.80 | 1 | 972k | $0.50 published-rate | |
| 7 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 0.80 | 1 | 337k | $0.12 published-rate | |
| 8 | codex | 3437a9e6 | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.80 | 1 | 85k | $0.15 published-rate | |
| 9 | codex | 3437a9e6 | zai/glm-5.2 | 3/3 | 44%–100% | 0.80 | 1 | 552k | $0.91 published-rate | |
| 10 | opencode | 357da891 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.80 | 1 | 165k | $0.02 published-rate | |
| 11 | opencode | 357da891 | openai/gpt-6.1-sol | 3/3 | 44%–100% | 0.93 | 1 | 190k | $0.15 published-rate | |
| 12 | pi | da41c66e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.80 | 1 | 143k | $0.02 published-rate | |
| 13 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 2/2 | 34%–100% | 0.80 | 1 | 360k | unknown | |
| 14 | claude-code | ecb67e4e | zai/glm-5.2 | 2/2 | 34%–100% | 0.80 | 1 | 538k | $0.59 published-rate | 1 infra:agent_error |
| 15 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 2/2 | 34%–100% | 0.80 | 1 | 117k | $0.05 published-rate | |
| 16 | opencode | 357da891 | together/deepseek-v4.1-flash | 2/2 | 34%–100% | 0.80 | 1 | 156k | $0.06 published-rate | |
| 17 | antigravity | 034ed3a1 | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 0.80 | 1 | 2.41M | $0.66 published-rate | |
| 18 | antigravity | 034ed3a1 | google/gemini-4-argon @low | 1/1 | 21%–100% | 0.80 | 1 | 85k | $0.08 published-rate | |
| 19 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 430k | $0.02 published-rate | |
| 20 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @medium | 1/1 | 21%–100% | 0.80 | 1 | 366k | $0.01 published-rate | |
| 21 | claude-code | c32a950b | anthropic/claude-opus-5-5 @high | 1/1 | 21%–100% | 1.00 | 1 | 771k | $0.49 published-rate | 1 infra:error |
| 22 | claude-code | c32a950b | anthropic/claude-opus-5-5 @max | 1/1 | 21%–100% | 1.00 | 1 | 4.18M | $4.50 published-rate | |
| 23 | claude-code | c32a950b | anthropic/claude-opus-5-5 @medium | 1/1 | 21%–100% | 1.00 | 1 | 1.26M | $0.47 published-rate | |
| 24 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 1/1 | 21%–100% | 0.80 | 1 | 1.04M | $0.23 published-rate | |
| 25 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 1.20M | $0.25 published-rate | |
| 26 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 897k | $0.18 published-rate | |
| 27 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @max | 1/1 | 21%–100% | 1.00 | 1 | 6.26M | $3.98 published-rate | |
| 28 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @medium | 1/1 | 21%–100% | 0.80 | 1 | 737k | $0.15 published-rate | |
| 29 | claude-code | ecb67e4e | fireworks/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 266k | $0.27 published-rate | |
| 30 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 0.80 | 1 | 1.43M | $0.50 published-rate | |
| 31 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 0.80 | 1 | 4.81M | $0.62 published-rate | |
| 32 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 0.80 | 1 | 835k | $0.48 published-rate | |
| 33 | claude-code | ecb67e4e | moonshot/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 274k | $0.22 published-rate | |
| 34 | claude-code | ecb67e4e | openai/gpt-5.6-terra @high | 1/1 | 21%–100% | 1.00 | 1 | 128k | $0.11 published-rate | |
| 35 | claude-code | ecb67e4e | openai/gpt-5.6-terra @low | 1/1 | 21%–100% | 1.00 | 1 | 146k | $0.11 published-rate | |
| 36 | claude-code | ecb67e4e | openai/gpt-6-luna @high | 1/1 | 21%–100% | 1.00 | 1 | 153k | $0.0063 published-rate | |
| 37 | claude-code | ecb67e4e | openai/gpt-6-luna @max | 1/1 | 21%–100% | 1.00 | 1 | 133k | $0.02 published-rate | |
| 38 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 1/1 | 21%–100% | 0.80 | 1 | 259k | $0.14 published-rate | |
| 39 | claude-code | ecb67e4e | openai/gpt-6.1-sol @high | 1/1 | 21%–100% | 0.80 | 1 | 176k | $0.14 published-rate | |
| 40 | claude-code | ecb67e4e | openai/gpt-6.1-sol @low | 1/1 | 21%–100% | 1.00 | 1 | 244k | $0.10 published-rate | 1 infra:error |
| 41 | claude-code | ecb67e4e | openai/gpt-6.1-sol @max | 1/1 | 21%–100% | 0.80 | 1 | 227k | $0.32 published-rate | 1 infra:worker_boot_fenced |
| 42 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 139k | $0.02 published-rate | |
| 43 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 133k | $0.01 published-rate | |
| 44 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @medium | 1/1 | 21%–100% | 0.80 | 1 | 94k | $0.01 published-rate | |
| 45 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 285k | $0.04 published-rate | |
| 46 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1.00 | 1 | 91k | $0.44 published-rate | |
| 47 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @high | 1/1 | 21%–100% | 1.00 | 1 | 116k | $0.54 published-rate | |
| 48 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @low | 1/1 | 21%–100% | 1.00 | 1 | 108k | $0.46 published-rate | |
| 49 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @medium | 1/1 | 21%–100% | 1.00 | 1 | 117k | $0.52 published-rate | |
| 50 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 246k | $1.32 published-rate | |
| 51 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 86k | $0.19 published-rate | 1 infra:worker_boot_fenced |
| 52 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @medium | 1/1 | 21%–100% | 0.80 | 1 | 62k | $0.14 published-rate | |
| 53 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 114k | $0.31 published-rate | |
| 54 | codex | 3437a9e6 | fireworks/qwen3.8-max | 1/1 | 21%–100% | 0.80 | 1 | 204k | $0.31 published-rate | |
| 55 | codex | 3437a9e6 | google/gemini-3.8-flash | 1/1 | 21%–100% | 0.80 | 1 | 1.01M | $0.47 published-rate | |
| 56 | codex | 3437a9e6 | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 0.80 | 1 | 591k | $0.42 published-rate | |
| 57 | codex | b2396403 | openai/gpt-5.6-terra @low | 1/1 | 21%–100% | 0.80 | 1 | 144k | $0.08 published-rate | |
| 58 | codex | b2396403 | openai/gpt-5.6-terra @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 255k | $0.17 published-rate | |
| 59 | codex | b2396403 | openai/gpt-6-luna @high | 1/1 | 21%–100% | 0.80 | 1 | 130k | $0.0040 published-rate | |
| 60 | codex | b2396403 | openai/gpt-6-luna @max | 1/1 | 21%–100% | 0.80 | 1 | 159k | $0.0079 published-rate | |
| 61 | codex | b2396403 | openai/gpt-6-luna @medium | 1/1 | 21%–100% | 0.80 | 1 | 124k | $0.0032 published-rate | |
| 62 | codex | b2396403 | openai/gpt-6-luna @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 202k | $0.0096 published-rate | |
| 63 | codex | b2396403 | openai/gpt-6.1-sol @low | 1/1 | 21%–100% | 0.80 | 1 | 108k | $0.07 published-rate | |
| 64 | codex | b2396403 | openai/gpt-6.1-sol @max | 1/1 | 21%–100% | 1.00 | 1 | 255k | $0.16 published-rate | |
| 65 | codex | b2396403 | openai/gpt-6.1-sol @ultra | 1/1 | 21%–100% | 1.00 | 1 | 294k | $0.16 published-rate | |
| 66 | opencode | 357da891 | anthropic/claude-haiku-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 111k | $0.0053 published-rate | |
| 67 | opencode | 357da891 | anthropic/claude-haiku-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 94k | $0.0048 published-rate | |
| 68 | opencode | 357da891 | anthropic/claude-haiku-5-5 @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 243k | $0.03 published-rate | |
| 69 | opencode | 357da891 | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1.00 | 1 | 79k | $0.20 published-rate | |
| 70 | opencode | 357da891 | anthropic/claude-opus-5-5 @high | 1/1 | 21%–100% | 1.00 | 1 | 142k | $0.27 published-rate | |
| 71 | opencode | 357da891 | anthropic/claude-opus-5-5 @medium | 1/1 | 21%–100% | 1.00 | 1 | 98k | $0.22 published-rate | |
| 72 | opencode | 357da891 | anthropic/claude-opus-5-5 @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 203k | $0.54 published-rate | 3 infra:agent_error |
| 73 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 1/1 | 21%–100% | 0.80 | 1 | 90k | $0.09 published-rate | |
| 74 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 99k | $0.11 published-rate | |
| 75 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 140k | $0.20 published-rate | |
| 76 | opencode | 357da891 | fireworks/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 107k | $0.25 published-rate | |
| 77 | opencode | 357da891 | fireworks/qwen3.8-max | 1/1 | 21%–100% | 0.80 | 1 | 139k | $0.19 published-rate | |
| 78 | opencode | 357da891 | google/gemini-3.8-flash | 1/1 | 21%–100% | 0.80 | 1 | 809k | $0.41 published-rate | |
| 79 | opencode | 357da891 | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 0.80 | 1 | 691k | $0.48 published-rate | |
| 80 | opencode | 357da891 | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 0.80 | 1 | 587k | $0.37 published-rate | 1 infra:agent_error |
| 81 | opencode | 357da891 | moonshot/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 140k | $0.28 published-rate | |
| 82 | opencode | 357da891 | openai/gpt-5.6-terra @high | 1/1 | 21%–100% | 0.80 | 1 | 147k | $0.12 published-rate | |
| 83 | opencode | 357da891 | openai/gpt-5.6-terra @low | 1/1 | 21%–100% | 1.00 | 1 | 54k | $0.06 published-rate | |
| 84 | opencode | 357da891 | openai/gpt-6-luna @high | 1/1 | 21%–100% | 0.80 | 1 | 305k | $0.01 published-rate | |
| 85 | opencode | 357da891 | openai/gpt-6-luna @low | 1/1 | 21%–100% | 0.80 | 1 | 146k | $0.0041 published-rate | |
| 86 | opencode | 357da891 | openai/gpt-6-luna @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 177k | $0.01 published-rate | |
| 87 | opencode | 357da891 | openai/gpt-6.1-sol @high | 1/1 | 21%–100% | 0.80 | 1 | 215k | $0.16 published-rate | |
| 88 | opencode | 357da891 | openai/gpt-6.1-sol @low | 1/1 | 21%–100% | 0.80 | 1 | 83k | $0.06 published-rate | |
| 89 | opencode | 357da891 | openai/gpt-6.1-sol @max | 1/1 | 21%–100% | 0.80 | 1 | 316k | $0.35 published-rate | |
| 90 | opencode | 357da891 | zai/glm-5.2 | 1/1 | 21%–100% | 0.80 | 1 | 199k | $0.19 published-rate | |
| 91 | pi | da41c66e | anthropic/claude-haiku-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 62k | $0.0074 published-rate | |
| 92 | pi | da41c66e | anthropic/claude-haiku-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 56k | $0.0065 published-rate | |
| 93 | pi | da41c66e | anthropic/claude-opus-5-5 | 1/1 | 21%–100% | 1.00 | 1 | 58k | $0.30 published-rate | |
| 94 | pi | da41c66e | anthropic/claude-opus-5-5 @high | 1/1 | 21%–100% | 1.00 | 1 | 53k | $0.30 published-rate | |
| 95 | pi | da41c66e | anthropic/claude-opus-5-5 @low | 1/1 | 21%–100% | 0.80 | 1 | 35k | $0.16 published-rate | |
| 96 | pi | da41c66e | anthropic/claude-opus-5-5 @medium | 1/1 | 21%–100% | 1.00 | 1 | 60k | $0.29 published-rate | |
| 97 | pi | da41c66e | anthropic/claude-opus-5-5 @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 115k | $0.81 published-rate | |
| 98 | pi | da41c66e | anthropic/claude-sonnet-5-5 @high | 1/1 | 21%–100% | 0.80 | 1 | 43k | $0.11 published-rate | |
| 99 | pi | da41c66e | anthropic/claude-sonnet-5-5 @medium | 1/1 | 21%–100% | 0.80 | 1 | 46k | $0.11 published-rate | |
| 100 | pi | da41c66e | anthropic/claude-sonnet-5-5 @xhigh | 1/1 | 21%–100% | 0.80 | 1 | 55k | $0.16 published-rate | |
| 101 | pi | da41c66e | fireworks/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 121k | $0.32 published-rate | |
| 102 | pi | da41c66e | google/gemini-3.8-flash @high | 1/1 | 21%–100% | 0.80 | 1 | 782k | $0.61 published-rate | |
| 103 | pi | da41c66e | google/gemini-3.8-flash @medium | 1/1 | 21%–100% | 0.80 | 1 | 879k | $0.48 published-rate | |
| 104 | pi | da41c66e | moonshot/kimi-k3 | 1/1 | 21%–100% | 0.80 | 1 | 109k | $0.29 published-rate | |
| 105 | pi | da41c66e | openai/gpt-5.6-terra @low | 1/1 | 21%–100% | 1.00 | 1 | 33k | $0.05 published-rate | |
| 106 | pi | da41c66e | openai/gpt-6-luna @high | 1/1 | 21%–100% | 1.00 | 1 | 43k | $0.0034 published-rate | |
| 107 | pi | da41c66e | openai/gpt-6-luna @max | 1/1 | 21%–100% | 0.80 | 1 | 101k | $0.01 published-rate | |
| 108 | pi | da41c66e | openai/gpt-6-luna @xhigh | 1/1 | 21%–100% | 1.00 | 1 | 60k | $0.0071 published-rate | |
| 109 | pi | da41c66e | openai/gpt-6.1-sol | 1/1 | 21%–100% | 0.80 | 1 | 75k | $0.09 published-rate | |
| 110 | pi | da41c66e | openai/gpt-6.1-sol @high | 1/1 | 21%–100% | 1.00 | 1 | 66k | $0.07 published-rate | |
| 111 | pi | da41c66e | openai/gpt-6.1-sol @low | 1/1 | 21%–100% | 1.00 | 1 | 43k | $0.04 published-rate | |
| 112 | pi | da41c66e | openai/gpt-6.1-sol @max | 1/1 | 21%–100% | 1.00 | 1 | 85k | $0.16 published-rate | |
| 113 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 4/5 | 38%–96% | 0.92 | 1 | 1.78M | $1.27 published-rate | 1 infra:error |
| 114 | pi | da41c66e | together/deepseek-v4.1-flash | 4/5 | 38%–96% | 0.76 | 1 | 198k | $0.08 published-rate | |
| 115 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 3/4 | 30%–95% | 0.75 | 1 | 348k | $0.33 published-rate | 2 infra:agent_error |
| 116 | pi | da41c66e | google/gemini-3.8-flash | 3/4 | 30%–95% | 0.75 | 1 | 844k | $0.53 published-rate | 1 infra:gateway_continued_run_changed_prompt |
| 117 | pi | da41c66e | fireworks/qwen3.8-max | 2/3 | 21%–94% | 0.73 | 1 | 177k | $0.27 published-rate | |
| 118 | antigravity | 034ed3a1 | google/gemini-3.8-flash @low | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 119 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 120 | antigravity | 034ed3a1 | google/gemini-4-argon @medium | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 121 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @high | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 122 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @max | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | 1 infra:agent_error |
| 123 | claude-code | c32a950b | anthropic/claude-haiku-5-5 @xhigh | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 124 | claude-code | c32a950b | anthropic/claude-opus-5-5 @low | 0/1 | 0%–79% | 0.80 | 1 | no passes | unknown | |
| 125 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 @xhigh | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 126 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 127 | claude-code | ecb67e4e | google/gemini-3.8-flash | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 128 | claude-code | ecb67e4e | openai/gpt-6-luna @low | 0/1 | 0%–79% | 0.40 | 1 | no passes | unknown | |
| 129 | claude-code | ecb67e4e | openai/gpt-6-luna @xhigh | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 130 | codex | 3437a9e6 | anthropic/claude-haiku-5-5 @max | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | 1 infra:error |
| 131 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 132 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @low | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 133 | codex | 3437a9e6 | baseten/glm-5.3-flash | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 134 | codex | 3437a9e6 | google/gemini-3.8-flash @high | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 135 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 136 | codex | 3437a9e6 | moonshot/kimi-k3 | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 137 | codex | b2396403 | openai/gpt-5.6-terra @high | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 138 | codex | b2396403 | openai/gpt-5.6-terra @max | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 139 | codex | b2396403 | openai/gpt-5.6-terra @medium | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 140 | codex | b2396403 | openai/gpt-5.6-terra @ultra | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 141 | codex | b2396403 | openai/gpt-6-luna @low | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 142 | opencode | 357da891 | anthropic/claude-haiku-5-5 @max | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 143 | opencode | 357da891 | anthropic/claude-haiku-5-5 @medium | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 144 | opencode | 357da891 | anthropic/claude-opus-5-5 @low | 0/1 | 0%–79% | 0.80 | 1 | no passes | unknown | |
| 145 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @low | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 146 | opencode | 357da891 | anthropic/claude-sonnet-5-5 @medium | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 147 | opencode | 357da891 | google/gemini-3.8-flash @low | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 148 | opencode | 357da891 | openai/gpt-6-luna @max | 0/1 | 0%–79% | 0.80 | 1 | no passes | unknown | |
| 149 | pi | da41c66e | anthropic/claude-haiku-5-5 @max | 0/1 | 0%–79% | 0.80 | 1 | no passes | unknown | |
| 150 | pi | da41c66e | anthropic/claude-haiku-5-5 @medium | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 151 | pi | da41c66e | anthropic/claude-haiku-5-5 @xhigh | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 152 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 153 | pi | da41c66e | anthropic/claude-sonnet-5-5 @low | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 154 | pi | da41c66e | openai/gpt-5.6-terra @high | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 155 | pi | da41c66e | openai/gpt-6-luna @low | 0/1 | 0%–79% | 1 | no passes | unknown | ||
| 156 | pi | da41c66e | zai/glm-5.2 | 0/1 | 0%–79% | 0.60 | 1 | no passes | unknown | |
| 157 | codex | 3437a9e6 | anthropic/claude-opus-5-5 @max | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:gateway_continued_run_changed_prompt | |
| 158 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 @max | — | 0%–0% | 0 | not measured (0/0) | unknown | 1 infra:harness_gave_up_on_reply_stream | |
| 159 | pi | da41c66e | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 1 infra:agent_error |
injection
2 tasks: injection-exfil-via-telegram, injection-memory-write-injection.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 2 | antigravity | 034ed3a1 | google/gemini-4-argon @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 3 | claude-code | c32a950b | anthropic/claude-haiku-4-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 4 | claude-code | c32a950b | anthropic/claude-opus-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 5 | claude-code | c32a950b | anthropic/claude-sonnet-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 6 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 7 | claude-code | ecb67e4e | baseten/glm-5.3-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 8 | claude-code | ecb67e4e | fireworks/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 9 | claude-code | ecb67e4e | fireworks/qwen3.8-max | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 10 | claude-code | ecb67e4e | google/gemini-3.8-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 11 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 12 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 13 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 14 | claude-code | ecb67e4e | openai/gpt-6.1-sol | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 15 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 16 | claude-code | ecb67e4e | zai/glm-5.2 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 17 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 18 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 19 | codex | 3437a9e6 | baseten/glm-5.3-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 20 | codex | 3437a9e6 | fireworks/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 21 | codex | 3437a9e6 | fireworks/qwen3.8-max | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 22 | codex | 3437a9e6 | google/gemini-3.8-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 23 | codex | 3437a9e6 | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 24 | codex | 3437a9e6 | moonshot/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 25 | codex | 3437a9e6 | together/deepseek-v4.1-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 26 | codex | 3437a9e6 | zai/glm-5.2 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 27 | codex | b2396403 | openai/gpt-6.1-sol | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 28 | codex | b2396403 | openai/gpt-6.1-sol @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 29 | codex | b2396403 | openai/gpt-6.1-sol @medium | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 30 | opencode | 357da891 | anthropic/claude-opus-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 31 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 32 | opencode | 357da891 | baseten/glm-5.3-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 33 | opencode | 357da891 | fireworks/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 34 | opencode | 357da891 | fireworks/qwen3.8-max | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 35 | opencode | 357da891 | google/gemini-3.8-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 36 | opencode | 357da891 | google/gemini-3.8-flash @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 37 | opencode | 357da891 | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 38 | opencode | 357da891 | moonshot/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 39 | opencode | 357da891 | openai/gpt-6.1-sol | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 40 | opencode | 357da891 | together/deepseek-v4.1-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 4 infra:skipped_no_taint_gate | |
| 41 | opencode | 357da891 | zai/glm-5.2 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 42 | pi | da41c66e | anthropic/claude-opus-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 43 | pi | da41c66e | anthropic/claude-sonnet-5-5 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 44 | pi | da41c66e | baseten/glm-5.3-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 45 | pi | da41c66e | fireworks/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 46 | pi | da41c66e | fireworks/qwen3.8-max | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 47 | pi | da41c66e | google/gemini-3.8-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 48 | pi | da41c66e | google/gemini-3.8-flash @high | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 49 | pi | da41c66e | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 50 | pi | da41c66e | moonshot/kimi-k3 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 51 | pi | da41c66e | openai/gpt-6.1-sol | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 52 | pi | da41c66e | together/deepseek-v4.1-flash | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate | |
| 53 | pi | da41c66e | zai/glm-5.2 | — | 0%–0% | 0 | not measured (0/0) | unknown | 2 infra:skipped_no_taint_gate |
rca
3 tasks: rca-admit-pressure-crash-loop, rca-fk-drain, rca-litellm-watchdog-restart-storm.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 14/14 | 78%–100% | 1.00 | 3 | 66k | $0.03 published-rate | |
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 12/12 | 76%–100% | 1.00 | 3 | 67k | $0.03 published-rate | 3 infra:agent_error |
| 3 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 7/7 | 65%–100% | 1.00 | 3 | 146k | $0.09 published-rate | 3 infra:agent_error |
| 4 | pi | da41c66e | google/gemini-3.8-flash | 7/7 | 65%–100% | 1.00 | 3 | 85k | $0.07 published-rate | |
| 5 | pi | da41c66e | together/deepseek-v4.1-flash | 7/7 | 65%–100% | 1.00 | 3 | 17k | $0.0061 published-rate | |
| 6 | opencode | 357da891 | together/deepseek-v4.1-flash | 6/6 | 61%–100% | 1.00 | 3 | 38k | $0.01 published-rate | |
| 7 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 5/5 | 57%–100% | 1.00 | 1 | 637k | $0.24 published-rate | |
| 8 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 5/5 | 57%–100% | 1.00 | 3 | 175k | $0.05 published-rate | |
| 9 | codex | 3437a9e6 | fireworks/kimi-k3 | 5/5 | 57%–100% | 1.00 | 3 | 33k | $0.04 published-rate | |
| 10 | codex | 3437a9e6 | zai/glm-5.2 | 5/5 | 57%–100% | 1.00 | 3 | 54k | $0.04 published-rate | |
| 11 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 5/5 | 57%–100% | 1.00 | 1 | 71k | $0.04 published-rate | |
| 12 | opencode | 357da891 | baseten/glm-5.3-flash | 5/5 | 57%–100% | 1.00 | 3 | 32k | $0.0042 published-rate | |
| 13 | opencode | 357da891 | openai/gpt-6.1-sol | 5/5 | 57%–100% | 1.00 | 3 | 54k | $0.04 published-rate | |
| 14 | pi | da41c66e | baseten/glm-5.3-flash | 5/5 | 57%–100% | 1.00 | 3 | 13k | $0.0022 published-rate | |
| 15 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 3/3 | 44%–100% | 1.00 | 3 | 82k | unknown | 1 infra:agent_error |
| 16 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 410k | $0.26 published-rate | |
| 17 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 1.00 | 3 | 102k | $0.14 published-rate | |
| 18 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 506k | $0.08 published-rate | |
| 19 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 182k | $0.02 published-rate | |
| 20 | claude-code | ecb67e4e | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 402k | $0.17 published-rate | |
| 21 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 1.09M | $0.38 published-rate | |
| 22 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 67k | $0.06 published-rate | |
| 23 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 1.00 | 3 | 398k | $0.26 published-rate | |
| 24 | claude-code | ecb67e4e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 123k | $0.08 published-rate | |
| 25 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 123k | $0.06 published-rate | |
| 26 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 40k | $0.17 published-rate | |
| 27 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 38k | $0.08 published-rate | |
| 28 | codex | 3437a9e6 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 1.00 | 3 | 25k | $0.0040 published-rate | |
| 29 | codex | 3437a9e6 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 188k | $0.15 published-rate | |
| 30 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 57k | $0.04 published-rate | |
| 31 | codex | 3437a9e6 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 33k | $0.04 published-rate | 3 infra:harness_failed_to_start |
| 32 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 3/3 | 44%–100% | 1.00 | 3 | 30k | $0.0098 published-rate | |
| 33 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 47k | $0.10 published-rate | |
| 34 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 44k | $0.04 published-rate | |
| 35 | opencode | 357da891 | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 56k | $0.09 published-rate | |
| 36 | opencode | 357da891 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 167k | $0.12 published-rate | |
| 37 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 451k | $0.30 published-rate | |
| 38 | opencode | 357da891 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 1.00 | 3 | 76k | $0.06 published-rate | |
| 39 | opencode | 357da891 | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 61k | $0.07 published-rate | |
| 40 | opencode | 357da891 | zai/glm-5.2 | 3/3 | 44%–100% | 1.00 | 3 | 49k | $0.03 published-rate | |
| 41 | pi | da41c66e | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 18k | $0.09 published-rate | |
| 42 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 17k | $0.04 published-rate | |
| 43 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 20k | $0.04 published-rate | |
| 44 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 216k | $0.18 published-rate | |
| 45 | pi | da41c66e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 23k | $0.04 published-rate | 1 infra:agent_error |
| 46 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 16k | $0.01 published-rate | |
| 47 | pi | da41c66e | zai/glm-5.2 | 3/3 | 44%–100% | 1.00 | 3 | 27k | $0.04 published-rate | |
| 48 | codex | b2396403 | openai/gpt-5.6-luna @medium | 1/1 | 21%–100% | 1.00 | 1 | 55k | $0.0040 published-rate | |
| 49 | codex | b2396403 | openai/gpt-5.6-luna @xhigh | 1/1 | 21%–100% | 1 | 55k | $0.0033 published-rate | ||
| 50 | codex | b2396403 | openai/gpt-5.6-sol @high | 1/1 | 21%–100% | 1.00 | 1 | 61k | $0.05 published-rate | |
| 51 | pi | da41c66e | google/gemini-3.8-flash @low | 1/1 | 21%–100% | 1.00 | 1 | 12k | $0.0094 published-rate | 2 infra:error |
| 52 | opencode | 357da891 | fireworks/qwen3.8-max | 5/6 | 44%–97% | 3 | 55k | $0.05 published-rate | 1 infra:agent_error | |
| 53 | codex | b2396403 | openai/gpt-6.1-sol | 9/11 | 52%–95% | 1.00 | 3 | 81k | $0.03 published-rate | 1 infra:error |
| 54 | claude-code | ecb67e4e | zai/glm-5.2 | 7/9 | 45%–94% | 1.00 | 3 | 223k | $0.10 published-rate | |
| 55 | pi | da41c66e | fireworks/qwen3.8-max | 5/7 | 36%–92% | 1.00 | 3 | 45k | $0.05 published-rate | |
| 56 | claude-code | c32a950b | anthropic/claude-haiku-4-5 | 2/3 | 21%–94% | 2 | 724k | $0.10 published-rate | 3 infra:dispatch_ran_other_model | |
| 57 | claude-code | c32a950b | anthropic/claude-sonnet-5 | 2/3 | 21%–94% | 3 | 975k | $0.27 published-rate | 3 infra:dispatch_ran_other_model | |
| 58 | claude-code | ecb67e4e | fireworks/kimi-k3 | 2/3 | 21%–94% | 1.00 | 3 | 197k | $0.09 published-rate | |
| 59 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 7/11 | 35%–85% | 1.00 | 3 | 1.06M | $0.36 published-rate | |
| 60 | codex | 3437a9e6 | fireworks/qwen3.8-max | 3/6 | 19%–81% | 3 | 130k | $0.13 published-rate | ||
| 61 | codex | b2396403 | openai/gpt-6-luna @high | 0/1 | 0%–79% | 1 | no passes | unknown |
review · task version 1
6 tasks: review-clean-retention-plan, review-corrupt-refresh-checkpoint, review-publish-guard-order, review-rescue-local-tip, review-retry-empty-page, review-stale-lock-recovery.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 9/9 | 70%–100% | 0.91 | 3 | 350k | $0.18 published-rate | |
| 2 | opencode | 357da891 | together/deepseek-v4.1-flash | 8/8 | 68%–100% | 0.93 | 5 | 47k | $0.02 published-rate | 1 infra:agent_error, 3 infra:tool_call_log_truncated |
| 3 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 6/6 | 61%–100% | 0.93 | 6 | not measured (0/6) | unknown | 1 infra:agent_error, 11 pre-GYMAGY-02 |
| 4 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 6/6 | 61%–100% | 0.93 | 6 | not measured (0/6) | unknown | |
| 5 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 6/6 | 61%–100% | 0.97 | 6 | not measured (0/6) | unknown | |
| 6 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 6/6 | 61%–100% | 0.97 | 6 | 217k | $0.08 published-rate | |
| 7 | codex | b2396403 | openai/gpt-6.1-sol | 6/6 | 61%–100% | 0.97 | 6 | not measured (0/6) | unknown | |
| 8 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 0.93 | 3 | 104k | $0.10 published-rate | |
| 9 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.93 | 3 | 234k | $0.03 published-rate | |
| 10 | claude-code | ecb67e4e | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 136k | $0.06 published-rate | |
| 11 | claude-code | ecb67e4e | google/gemini-3.8-flash | 3/3 | 44%–100% | 0.93 | 3 | 417k | $0.13 published-rate | |
| 12 | claude-code | ecb67e4e | moonshot/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 176k | $0.10 published-rate | |
| 13 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 0.87 | 3 | 145k | $0.07 published-rate | |
| 14 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 0.87 | 3 | 51k | $0.21 published-rate | |
| 15 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.93 | 3 | 44k | $0.09 published-rate | |
| 16 | codex | 3437a9e6 | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 31k | $0.05 published-rate | |
| 17 | codex | 3437a9e6 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 0.87 | 3 | 56k | $0.05 published-rate | 3 infra:proxy_request_conversion_failed |
| 18 | codex | 3437a9e6 | google/gemini-3.8-flash | 3/3 | 44%–100% | 0.93 | 3 | 346k | $0.19 published-rate | |
| 19 | codex | 3437a9e6 | moonshot/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 36k | $0.04 published-rate | |
| 20 | codex | b2396403 | openai/gpt-6.1-sol @high | 3/3 | 44%–100% | 0.87 | 3 | 76k | $0.04 published-rate | 1 infra:agent_error, 1 infra:error |
| 21 | opencode | 357da891 | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 0.87 | 3 | 56k | $0.09 published-rate | |
| 22 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.93 | 3 | 57k | $0.04 published-rate | |
| 23 | opencode | 357da891 | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.93 | 3 | 48k | $0.0061 published-rate | |
| 24 | opencode | 357da891 | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 59k | $0.10 published-rate | |
| 25 | opencode | 357da891 | fireworks/qwen3.8-max | 3/3 | 44%–100% | 0.87 | 3 | 69k | $0.06 published-rate | |
| 26 | opencode | 357da891 | google/gemini-3.8-flash | 3/3 | 44%–100% | 0.87 | 3 | 117k | $0.09 published-rate | |
| 27 | opencode | 357da891 | moonshot/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 59k | $0.08 published-rate | |
| 28 | opencode | 357da891 | openai/gpt-6.1-sol | 3/3 | 44%–100% | 0.87 | 3 | 62k | $0.05 published-rate | |
| 29 | pi | da41c66e | anthropic/claude-opus-5-5 | 3/3 | 44%–100% | 0.87 | 3 | 22k | $0.10 published-rate | |
| 30 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.93 | 3 | 18k | $0.04 published-rate | |
| 31 | pi | da41c66e | baseten/glm-5.3-flash | 3/3 | 44%–100% | 0.93 | 3 | 17k | $0.0027 published-rate | |
| 32 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 18k | $0.05 published-rate | |
| 33 | pi | da41c66e | fireworks/qwen3.8-max | 3/3 | 44%–100% | 0.87 | 3 | 41k | $0.04 published-rate | |
| 34 | pi | da41c66e | google/gemini-3.8-flash | 3/3 | 44%–100% | 0.93 | 3 | 111k | $0.11 published-rate | |
| 35 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.93 | 3 | 188k | $0.15 published-rate | |
| 36 | pi | da41c66e | moonshot/kimi-k3 | 3/3 | 44%–100% | 0.93 | 3 | 23k | $0.05 published-rate | |
| 37 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 0.93 | 3 | 19k | $0.02 published-rate | |
| 38 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 5/6 | 44%–97% | 0.83 | 6 | not measured (0/6) | unknown | 6 infra:agent_error |
| 39 | codex | 3437a9e6 | zai/glm-5.2 | 5/6 | 44%–97% | 0.90 | 6 | 315k | $0.39 published-rate | |
| 40 | claude-code | ecb67e4e | fireworks/qwen3.8-27b | 4/5 | 38%–96% | 0.84 | 5 | not measured (0/5) | unknown | 1 infra:agent_error |
| 41 | claude-code | ecb67e4e | zai/glm-5.2 | 4/6 | 30%–90% | 0.80 | 6 | not measured (0/6) | unknown | 1 infra:agent_error |
| 42 | pi | da41c66e | zai/glm-5.2 | 4/6 | 30%–90% | 0.80 | 6 | 236k | $0.28 published-rate | |
| 43 | codex | 3437a9e6 | baseten/glm-5.3-flash | 2/3 | 21%–94% | 0.85 | 3 | 60k | $0.0094 published-rate | |
| 44 | opencode | 357da891 | zai/glm-5.2 | 3/6 | 19%–81% | 0.63 | 6 | 243k | $0.23 published-rate |
review · task version 2
6 tasks: review-clean-retention-plan, review-corrupt-refresh-checkpoint, review-publish-guard-order, review-rescue-local-tip, review-retry-empty-page, review-stale-lock-recovery.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 23/23 | 86%–100% | 0.99 | 5 | 103k | $0.04 published-rate | |
| 2 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 18/18 | 82%–100% | 0.96 | 6 | 589k | $0.29 published-rate | |
| 3 | codex | b2396403 | openai/gpt-6.1-sol @high | 14/14 | 78%–100% | 0.97 | 6 | 101k | $0.05 published-rate | 2 infra:agent_error |
| 4 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 12/12 | 76%–100% | 0.97 | 6 | 757k | $0.30 published-rate | |
| 5 | codex | b2396403 | openai/gpt-6.1-sol | 12/12 | 76%–100% | 0.98 | 6 | 94k | $0.04 published-rate | |
| 6 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 10/10 | 72%–100% | 1.00 | 2 | 114k | $0.06 published-rate | |
| 7 | opencode | 357da891 | openai/gpt-6.1-sol | 10/10 | 72%–100% | 0.96 | 6 | 109k | $0.08 published-rate | |
| 8 | pi | da41c66e | together/deepseek-v4.1-flash | 10/10 | 72%–100% | 0.98 | 6 | 89k | $0.03 published-rate | |
| 9 | opencode | 357da891 | together/deepseek-v4.1-flash | 9/9 | 70%–100% | 0.93 | 6 | 60k | $0.02 published-rate | |
| 10 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 6/6 | 61%–100% | 0.93 | 6 | 278k | unknown | |
| 11 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 6/6 | 61%–100% | 0.97 | 6 | 176k | $0.32 published-rate | |
| 12 | claude-code | ecb67e4e | google/gemini-3.8-flash | 6/6 | 61%–100% | 0.97 | 6 | 610k | $0.23 published-rate | |
| 13 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 6/6 | 61%–100% | 0.93 | 6 | 169k | $0.08 published-rate | |
| 14 | codex | 3437a9e6 | google/gemini-3.8-flash | 6/6 | 61%–100% | 0.97 | 6 | 423k | $0.23 published-rate | |
| 15 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 6/6 | 61%–100% | 0.93 | 6 | 67k | $0.06 published-rate | |
| 16 | opencode | 357da891 | google/gemini-3.8-flash | 6/6 | 61%–100% | 0.93 | 6 | 283k | $0.15 published-rate | |
| 17 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 6/6 | 61%–100% | 0.97 | 6 | 22k | $0.05 published-rate | |
| 18 | pi | da41c66e | fireworks/kimi-k3 | 6/6 | 61%–100% | 0.97 | 6 | 42k | $0.13 published-rate | |
| 19 | pi | da41c66e | google/gemini-3.8-flash | 6/6 | 61%–100% | 0.97 | 6 | 273k | $0.17 published-rate | |
| 20 | pi | da41c66e | google/gemini-3.8-flash @high | 6/6 | 61%–100% | 0.97 | 6 | 440k | $0.26 published-rate | |
| 21 | pi | da41c66e | moonshot/kimi-k3 | 6/6 | 61%–100% | 0.97 | 6 | 41k | $0.12 published-rate | |
| 22 | pi | da41c66e | openai/gpt-6.1-sol | 6/6 | 61%–100% | 0.97 | 6 | 29k | $0.03 published-rate | |
| 23 | claude-code | ecb67e4e | fireworks/qwen3.8-27b | 3/3 | 44%–100% | 0.93 | 3 | 192k | unknown | |
| 24 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.93 | 3 | 523k | $0.15 published-rate | |
| 25 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.87 | 3 | 118k | $0.08 published-rate | |
| 26 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 0.93 | 3 | 209k | $0.14 published-rate | |
| 27 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.93 | 3 | 82k | $0.06 published-rate | |
| 28 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 0.93 | 3 | 336k | $0.17 published-rate | |
| 29 | opencode | 357da891 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.93 | 3 | 87k | $0.07 published-rate | |
| 30 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 7/8 | 53%–98% | 0.93 | 6 | 268k | $0.09 published-rate | |
| 31 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 5/6 | 44%–97% | 0.93 | 6 | 930k | $0.16 published-rate | |
| 32 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 5/6 | 44%–97% | 0.90 | 6 | 435k | $0.05 published-rate | |
| 33 | claude-code | ecb67e4e | fireworks/kimi-k3 | 5/6 | 44%–97% | 0.90 | 6 | 274k | $0.17 published-rate | |
| 34 | claude-code | ecb67e4e | moonshot/kimi-k3 | 5/6 | 44%–97% | 0.90 | 6 | 260k | $0.23 published-rate | |
| 35 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 5/6 | 44%–97% | 0.87 | 6 | 65k | $0.29 published-rate | |
| 36 | codex | 3437a9e6 | moonshot/kimi-k3 | 5/6 | 44%–97% | 0.90 | 6 | 58k | $0.12 published-rate | |
| 37 | opencode | 357da891 | anthropic/claude-opus-5-5 | 5/6 | 44%–97% | 0.87 | 6 | 74k | $0.13 published-rate | |
| 38 | opencode | 357da891 | fireworks/kimi-k3 | 5/6 | 44%–97% | 0.90 | 6 | 98k | $0.23 published-rate | 1 infra:agent_error |
| 39 | opencode | 357da891 | fireworks/qwen3.8-max | 5/6 | 44%–97% | 0.87 | 6 | 118k | $0.12 published-rate | |
| 40 | opencode | 357da891 | moonshot/kimi-k3 | 5/6 | 44%–97% | 0.90 | 6 | 95k | $0.18 published-rate | |
| 41 | pi | da41c66e | anthropic/claude-opus-5-5 | 5/6 | 44%–97% | 0.87 | 6 | 28k | $0.14 published-rate | |
| 42 | codex | 3437a9e6 | fireworks/kimi-k3 | 8/10 | 49%–94% | 0.86 | 6 | 80k | $0.15 published-rate | |
| 43 | pi | da41c66e | fireworks/qwen3.8-max | 8/10 | 49%–94% | 0.86 | 6 | 255k | $0.21 published-rate | |
| 44 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 7/10 | 40%–89% | 0.80 | 6 | 357k | $0.38 published-rate | 3 infra:agent_error |
| 45 | opencode | 357da891 | baseten/glm-5.3-flash | 7/10 | 40%–89% | 0.84 | 6 | 142k | $0.02 published-rate | 1 infra:agent_error |
| 46 | claude-code | ecb67e4e | zai/glm-5.2 | 4/6 | 30%–90% | 0.80 | 6 | 445k | $0.39 published-rate | |
| 47 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 4/6 | 30%–90% | 0.90 | 6 | 70k | $0.15 published-rate | |
| 48 | codex | 3437a9e6 | baseten/glm-5.3-flash | 4/6 | 30%–90% | 0.83 | 6 | 65k | $0.01 published-rate | |
| 49 | codex | 3437a9e6 | fireworks/qwen3.8-max | 4/6 | 30%–90% | 0.87 | 6 | 170k | $0.21 published-rate | 3 infra:proxy_request_conversion_failed |
| 50 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 4/6 | 30%–90% | 0.83 | 6 | 101k | $0.04 published-rate | |
| 51 | opencode | 357da891 | zai/glm-5.2 | 4/6 | 30%–90% | 0.80 | 6 | 190k | $0.16 published-rate | |
| 52 | pi | da41c66e | zai/glm-5.2 | 4/6 | 30%–90% | 0.80 | 6 | 132k | $0.17 published-rate | |
| 53 | codex | 3437a9e6 | zai/glm-5.2 | 5/8 | 31%–86% | 0.82 | 6 | 397k | $0.54 published-rate | |
| 54 | pi | da41c66e | baseten/glm-5.3-flash | 6/10 | 31%–83% | 0.84 | 6 | 60k | $0.01 published-rate | |
| 55 | pi | da41c66e | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 3 infra:error |
review · task version 3
3 tasks: review-corrupt-refresh-checkpoint, review-rescue-local-tip, review-stale-lock-recovery.
| # | harness | config | model | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex | b2396403 | openai/gpt-6.1-sol @medium | 23/23 | 86%–100% | 1.00 | 3 | 107k | $0.05 published-rate | |
| 2 | codex | b2396403 | openai/gpt-6.1-sol @high | 21/21 | 85%–100% | 1.00 | 3 | 108k | $0.05 published-rate | 1 infra:agent_error |
| 3 | claude-code | c32a950b | anthropic/claude-opus-5-5 | 12/12 | 76%–100% | 1.00 | 3 | 789k | $0.32 published-rate | |
| 4 | codex | b2396403 | openai/gpt-6.1-sol | 12/12 | 76%–100% | 1.00 | 3 | 96k | $0.05 published-rate | |
| 5 | claude-code | c32a950b | anthropic/claude-opus-5-5 @xhigh | 10/10 | 72%–100% | 0.98 | 2 | 1.26M | $0.75 published-rate | |
| 6 | codex | b2396403 | openai/gpt-6.1-sol @xhigh | 10/10 | 72%–100% | 1.00 | 2 | 114k | $0.06 published-rate | |
| 7 | pi | da41c66e | google/gemini-3.8-flash | 10/10 | 72%–100% | 1.00 | 3 | 397k | $0.23 published-rate | 1 infra:gateway_continued_run_changed_prompt |
| 8 | antigravity | 034ed3a1 | google/gemini-3.8-flash @high | 9/9 | 70%–100% | 1.00 | 3 | 827k | $0.40 published-rate | |
| 9 | opencode | 357da891 | openai/gpt-6.1-sol | 7/7 | 65%–100% | 1.00 | 3 | 129k | $0.09 published-rate | |
| 10 | antigravity | 034ed3a1 | google/gemini-3.1-pro @high | 6/6 | 61%–100% | 1.00 | 3 | 288k | unknown | |
| 11 | claude-code | c32a950b | anthropic/claude-sonnet-5-5 | 6/6 | 61%–100% | 1.00 | 3 | 767k | $0.15 published-rate | |
| 12 | opencode | 357da891 | together/deepseek-v4.1-flash | 5/5 | 57%–100% | 1.00 | 3 | 82k | $0.03 published-rate | 1 infra:agent_error |
| 13 | antigravity | 034ed3a1 | google/gemini-4-argon @high | 3/3 | 44%–100% | 1.00 | 3 | 249k | $0.53 published-rate | |
| 14 | claude-code | ecb67e4e | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 802k | $0.32 published-rate | |
| 15 | claude-code | ecb67e4e | google/gemini-3.8-flash @medium | 3/3 | 44%–100% | 1.00 | 3 | 481k | $0.32 published-rate | |
| 16 | claude-code | ecb67e4e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 194k | $0.09 published-rate | |
| 17 | codex | 3437a9e6 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 501k | $0.27 published-rate | |
| 18 | codex | 3437a9e6 | google/gemini-3.8-flash @low | 3/3 | 44%–100% | 0.93 | 3 | 244k | $0.15 published-rate | |
| 19 | opencode | 357da891 | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 0.93 | 3 | 77k | $0.07 published-rate | |
| 20 | opencode | 357da891 | google/gemini-3.8-flash | 3/3 | 44%–100% | 1.00 | 3 | 450k | $0.22 published-rate | |
| 21 | opencode | 357da891 | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 565k | $0.32 published-rate | |
| 22 | pi | da41c66e | anthropic/claude-sonnet-5-5 | 3/3 | 44%–100% | 1.00 | 3 | 26k | $0.07 published-rate | |
| 23 | pi | da41c66e | fireworks/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 67k | $0.21 published-rate | |
| 24 | pi | da41c66e | google/gemini-3.8-flash @high | 3/3 | 44%–100% | 1.00 | 3 | 693k | $0.37 published-rate | |
| 25 | pi | da41c66e | moonshot/kimi-k3 | 3/3 | 44%–100% | 1.00 | 3 | 59k | $0.18 published-rate | |
| 26 | pi | da41c66e | openai/gpt-6.1-sol | 3/3 | 44%–100% | 1.00 | 3 | 39k | $0.04 published-rate | |
| 27 | pi | da41c66e | together/deepseek-v4.1-flash | 11/12 | 65%–99% | 0.97 | 3 | 129k | $0.05 published-rate | 1 infra:error |
| 28 | claude-code | ecb67e4e | together/deepseek-v4.1-flash | 7/8 | 53%–98% | 0.95 | 3 | 351k | $0.13 published-rate | |
| 29 | codex | 3437a9e6 | fireworks/kimi-k3 | 5/7 | 36%–92% | 0.83 | 3 | 109k | $0.21 published-rate | |
| 30 | pi | da41c66e | fireworks/qwen3.8-max | 5/7 | 36%–92% | 0.86 | 3 | 382k | $0.31 published-rate | |
| 31 | claude-code | ecb67e4e | baseten/glm-5.3-flash | 2/3 | 21%–94% | 0.87 | 3 | 737k | $0.08 published-rate | |
| 32 | claude-code | ecb67e4e | fireworks/kimi-k3 | 2/3 | 21%–94% | 0.87 | 3 | 481k | $0.34 published-rate | |
| 33 | claude-code | ecb67e4e | google/gemini-3.8-flash @high | 2/3 | 21%–94% | 1.00 | 3 | 779k | $0.35 published-rate | |
| 34 | claude-code | ecb67e4e | moonshot/kimi-k3 | 2/3 | 21%–94% | 0.87 | 3 | 385k | $0.42 published-rate | |
| 35 | codex | 3437a9e6 | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.87 | 3 | 86k | $0.41 published-rate | |
| 36 | codex | 3437a9e6 | baseten/glm-5.3-flash | 2/3 | 21%–94% | 0.80 | 3 | 71k | $0.01 published-rate | |
| 37 | codex | 3437a9e6 | moonshot/kimi-k3 | 2/3 | 21%–94% | 0.87 | 3 | 92k | $0.24 published-rate | |
| 38 | opencode | 357da891 | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.87 | 3 | 100k | $0.21 published-rate | |
| 39 | opencode | 357da891 | fireworks/kimi-k3 | 2/3 | 21%–94% | 0.87 | 3 | 156k | $0.44 published-rate | 1 infra:agent_error |
| 40 | opencode | 357da891 | fireworks/qwen3.8-max | 2/3 | 21%–94% | 0.87 | 3 | 192k | $0.22 published-rate | |
| 41 | opencode | 357da891 | google/gemini-3.8-flash @low | 2/3 | 21%–94% | 0.87 | 3 | 144k | $0.10 published-rate | |
| 42 | opencode | 357da891 | moonshot/kimi-k3 | 2/3 | 21%–94% | 0.87 | 3 | 150k | $0.32 published-rate | |
| 43 | pi | da41c66e | anthropic/claude-opus-5-5 | 2/3 | 21%–94% | 0.87 | 3 | 38k | $0.21 published-rate | |
| 44 | claude-code | ecb67e4e | fireworks/qwen3.8-max | 6/10 | 31%–83% | 0.78 | 3 | 467k | $0.49 published-rate | 3 infra:agent_error |
| 45 | opencode | 357da891 | baseten/glm-5.3-flash | 4/7 | 25%–84% | 0.80 | 3 | 212k | $0.03 published-rate | 1 infra:agent_error |
| 46 | codex | 3437a9e6 | zai/glm-5.2 | 4/8 | 22%–78% | 0.80 | 3 | 772k | $1.08 published-rate | |
| 47 | claude-code | ecb67e4e | fireworks/qwen3.8-27b | 1/2 | 9%–91% | 0.70 | 2 | 323k | unknown | 1 infra:agent_error |
| 48 | pi | da41c66e | baseten/glm-5.3-flash | 3/7 | 16%–75% | 0.80 | 3 | 103k | $0.02 published-rate | |
| 49 | claude-code | ecb67e4e | zai/glm-5.2 | 2/6 | 10%–70% | 0.73 | 3 | 1.52M | $1.03 published-rate | 1 infra:agent_error |
| 50 | codex | 3437a9e6 | together/deepseek-v4.1-flash | 2/6 | 10%–70% | 0.77 | 3 | 283k | $0.14 published-rate | |
| 51 | pi | da41c66e | zai/glm-5.2 | 2/6 | 10%–70% | 0.73 | 3 | 636k | $0.76 published-rate | |
| 52 | claude-code | ecb67e4e | google/gemini-3.8-flash @low | 1/3 | 6%–79% | 0.73 | 3 | 358k | $0.28 published-rate | |
| 53 | codex | 3437a9e6 | anthropic/claude-sonnet-5-5 | 1/3 | 6%–79% | 0.87 | 3 | 149k | $0.33 published-rate | |
| 54 | codex | 3437a9e6 | fireworks/qwen3.8-max | 1/3 | 6%–79% | 0.87 | 3 | 510k | $0.70 published-rate | |
| 55 | opencode | 357da891 | zai/glm-5.2 | 1/6 | 3%–56% | 0.57 | 3 | 1.09M | $1.08 published-rate | |
| 56 | pi | da41c66e | google/gemini-3.8-flash @low | — | 0%–0% | 0 | not measured (0/0) | unknown | 3 infra:error |