Board · task version 1
coding
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
13 tasks
- coding-cache-context
- coding-commit-ack
- coding-generation-fence
- coding-jsonb-like-predicate
- coding-missing-commit
- coding-nondict-migrate-audit
- coding-pipefail-silent-exit
- coding-pr-not-found-terminal
- coding-registry-tail
- coding-schema-behind-wording
- coding-toc-commands
- coding-toc-large-output
- coding-unknown-change-kind-sentinel
coding · task version 1
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 63 / 63 | 94%–100% | 1.00 | 13 | 102k | $0.04 list | 1 | |
| 2 | anthropic/claude-opus-5-5claude-code · config c32a950b | 34 / 34 | 90%–100% | 1.00 | 13 | 1.26M | $0.36 list | 3 | |
| 3 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 25 / 25 | 87%–100% | 1.00 | 5 | 124k | $0.05 list | 0 | |
| 4 | openai/gpt-6.1-solopencode · config 357da891 | 25 / 25 | 87%–100% | 1.00 | 13 | 157k | $0.07 list | 3 | |
| 5 | openai/gpt-6.1-solpi · config da41c66e | 15 / 15 | 80%–100% | 1.00 | 13 | 76k | $0.04 list | 1 | |
| 6 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 13 / 13 | 77%–100% | 1.00 | 13 | 93k | $0.10 list | 4 | |
| 7 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 13 / 13 | 77%–100% | 1.00 | 13 | 157k | $0.07 list | 2 | |
| 8 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 13 / 13 | 77%–100% | 1.00 | 13 | 51k | $0.22 list | 0 | |
| 9 | anthropic/claude-opus-5-5opencode · config 357da891 | 13 / 13 | 77%–100% | 1.00 | 13 | 83k | $0.12 list | 2 | |
| 10 | anthropic/claude-opus-5-5pi · config da41c66e | 13 / 13 | 77%–100% | 1.00 | 13 | 40k | $0.17 list | 2 | |
| 11 | anthropic/claude-sonnet-5-5pi · config da41c66e | 13 / 13 | 77%–100% | 1.00 | 13 | 32k | $0.07 list | 2 | |
| 12 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 12 / 12 | 76%–100% | 1.00 | 12 | 603k | $0.15 list | 1 | |
| 13 | anthropic/claude-haiku-4-5claude-code · config c32a950b | 7 / 7 | 65%–100% | 1.00 | 7 | 714k | $0.09 list | 7 | |
| 14 | anthropic/claude-opus-5-5 @lowclaude-code · config c32a950b | 2 / 2 | 34%–100% | 1.00 | 2 | 274k | $0.14 list | 1 | |
| 15 | openai/gpt-6-astra @lowcodex · config b2396403 | 2 / 2 | 34%–100% | 2 | 55k | $0.13 list | 0 | ||
| 16 | openai/gpt-6-sol @mediumcodex · config b2396403 | 2 / 2 | 34%–100% | 1.00 | 2 | 100k | $0.05 list | 0 | |
| 17 | fireworks/qwen3.8-27bclaude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 110k | unknown | 0 | ||
| 18 | openai/gpt-5.6-luna @lowcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 82k | $0.0047 list | 0 | ||
| 19 | openai/gpt-5.6-luna @mediumcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 64k | $0.0044 list | 0 | ||
| 20 | openai/gpt-5.6-sol @lowcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 60k | $0.05 list | 0 | ||
| 21 | openai/gpt-5.6-terra @highcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 121k | $0.05 list | 0 | ||
| 22 | openai/gpt-5.6-terra @xhighcodex · config b2396403 | 1 / 1 | 21%–100% | 1.00 | 1 | 59k | $0.03 list | 0 | |
| 23 | openai/gpt-6-astra @xhighcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 47k | $0.12 list | 0 | ||
| 24 | openai/gpt-6-luna @highcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 91k | $0.0019 list | 0 | ||
| 25 | openai/gpt-6-sol @xhighcodex · config b2396403 | 1 / 1 | 21%–100% | 1.00 | 1 | 77k | $0.03 list | 0 | |
| 26 | openai/gpt-6.1-sol @highcodex · config b2396403 | 57 / 58 | 91%–100% | 0.99 | 13 | 118k | $0.04 list | 0 | |
| 27 | openai/gpt-6.1-solcodex · config b2396403 | 32 / 33 | 85%–99% | 0.98 | 13 | 104k | $0.04 list | 2 | |
| 28 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 25 / 26 | 81%–99% | 1.00 | 13 | not measured (19/26) | unknown | 2 | |
| 29 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 24 / 25 | 80%–99% | 1.00 | 5 | 1.95M | $0.57 list | 4 | |
| 30 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 13 / 14 | 69%–99% | 0.96 | 13 | 831k | $0.12 list | 1 | |
| 31 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 12 / 13 | 67%–99% | 1.00 | 13 | 143k | unknown | 1 | |
| 32 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 12 / 13 | 67%–99% | 1.00 | 13 | 656k | $0.24 list | 0 | |
| 33 | google/gemini-3.8-flash @highcodex · config 3437a9e6 | 12 / 13 | 67%–99% | 1.00 | 13 | 486k | $0.23 list | 0 | |
| 34 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 12 / 13 | 67%–99% | 0.96 | 13 | 72k | $0.06 list | 0 | |
| 35 | google/gemini-3.8-flashopencode · config 357da891 | 12 / 13 | 67%–99% | 1.00 | 13 | 1.39M | $0.24 list | 0 | |
| 36 | google/gemini-3.8-flash @highclaude-code · config ecb67e4e | 11 / 12 | 65%–99% | 1.00 | 12 | 860k | $0.17 list | 0 | |
| 37 | google/gemini-3.8-flash @lowclaude-code · config ecb67e4e | 11 / 12 | 65%–99% | 0.96 | 12 | 123k | $0.08 list | 0 | |
| 38 | google/gemini-3.8-flash @mediumclaude-code · config ecb67e4e | 11 / 12 | 65%–99% | 1.00 | 12 | 277k | $0.17 list | 2 | |
| 39 | google/gemini-3.8-flash @highopencode · config 357da891 | 11 / 12 | 65%–99% | 1.00 | 12 | 356k | $0.18 list | 0 | |
| 40 | google/gemini-3.8-flash @lowopencode · config 357da891 | 11 / 12 | 65%–99% | 0.97 | 12 | 122k | $0.08 list | 5 | |
| 41 | google/gemini-3.8-flash @lowpi · config da41c66e | 11 / 12 | 65%–99% | 0.96 | 12 | 59k | $0.05 list | 0 | |
| 42 | zai/glm-5.2claude-code · config ecb67e4e | 18 / 20 | 70%–97% | 1.00 | 13 | not measured (13/20) | unknown | 6 | |
| 43 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 18 / 20 | 70%–97% | 1.00 | 13 | not measured (13/20) | unknown | 9 | |
| 44 | moonshot/kimi-k3pi · config da41c66e | 18 / 20 | 70%–97% | 1.00 | 13 | not measured (12/20) | unknown | 1 | |
| 45 | moonshot/kimi-k3codex · config 3437a9e6 | 15 / 17 | 66%–97% | 0.96 | 13 | not measured (13/17) | unknown | 4 | |
| 46 | google/gemini-3.8-flash @highpi · config da41c66e | 25 / 29 | 69%–95% | 1.00 | 13 | 546k | $0.25 list | 0 | |
| 47 | together/deepseek-v4.1-flashpi · config da41c66e | 31 / 36 | 71%–94% | 0.96 | 13 | not measured (29/36) | unknown | 1 | |
| 48 | google/gemini-3.8-flashpi · config da41c66e | 24 / 28 | 69%–94% | 1.00 | 13 | 369k | $0.17 list | 1 | |
| 49 | fireworks/qwen3.8-maxopencode · config 357da891 | 17 / 20 | 64%–95% | 0.96 | 13 | not measured (13/20) | unknown | 1 | |
| 50 | moonshot/kimi-k3opencode · config 357da891 | 17 / 20 | 64%–95% | 0.96 | 13 | not measured (13/20) | unknown | 1 | |
| 51 | fireworks/kimi-k3pi · config da41c66e | 17 / 20 | 64%–95% | 1.00 | 13 | not measured (13/20) | unknown | 1 | |
| 52 | zai/glm-5.2pi · config da41c66e | 17 / 20 | 64%–95% | 1.00 | 13 | not measured (13/20) | unknown | 1 | |
| 53 | fireworks/qwen3.8-maxpi · config da41c66e | 22 / 26 | 66%–94% | 1.00 | 13 | not measured (19/26) | unknown | 0 | |
| 54 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 11 / 13 | 58%–96% | 0.96 | 13 | 57k | $0.12 list | 0 | |
| 55 | google/gemini-3.8-flashcodex · config 3437a9e6 | 11 / 13 | 58%–96% | 1.00 | 13 | 227k | $0.14 list | 0 | |
| 56 | fireworks/kimi-k3claude-code · config ecb67e4e | 16 / 19 | 62%–94% | 0.97 | 13 | not measured (13/19) | unknown | 2 | |
| 57 | zai/glm-5.2opencode · config 357da891 | 16 / 19 | 62%–94% | 1.00 | 13 | not measured (12/19) | unknown | 2 | |
| 58 | google/gemini-3.8-flash @lowcodex · config 3437a9e6 | 10 / 12 | 55%–95% | 0.96 | 12 | 89k | $0.07 list | 0 | |
| 59 | anthropic/claude-sonnet-5claude-code · config c32a950b | 5 / 6 | 44%–97% | 1.00 | 6 | 799k | $0.21 list | 9 | |
| 60 | zai/glm-5.2codex · config 3437a9e6 | 21 / 26 | 62%–91% | 0.94 | 13 | not measured (19/26) | unknown | 0 | |
| 61 | fireworks/kimi-k3opencode · config 357da891 | 16 / 20 | 58%–92% | 1.00 | 13 | not measured (13/20) | unknown | 0 | |
| 62 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 15 / 19 | 57%–91% | 0.89 | 13 | not measured (13/19) | unknown | 1 | |
| 63 | together/deepseek-v4.1-flashopencode · config 357da891 | 25 / 32 | 61%–89% | 0.95 | 13 | not measured (25/32) | unknown | 1 | |
| 64 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 20 / 26 | 58%–89% | 0.94 | 13 | not measured (19/26) | unknown | 5 | |
| 65 | fireworks/kimi-k3codex · config 3437a9e6 | 20 / 26 | 58%–89% | 0.94 | 13 | not measured (19/26) | unknown | 0 | |
| 66 | baseten/glm-5.3-flashpi · config da41c66e | 19 / 26 | 54%–86% | 0.97 | 13 | not measured (19/26) | unknown | 1 | |
| 67 | moonshot/kimi-k3claude-code · config ecb67e4e | 13 / 18 | 49%–88% | 1.00 | 13 | not measured (12/18) | unknown | 5 | |
| 68 | baseten/glm-5.3-flashopencode · config 357da891 | 18 / 26 | 50%–83% | 0.96 | 13 | not measured (19/26) | unknown | 2 | |
| 69 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 13 / 19 | 46%–85% | 0.96 | 12 | not measured (12/19) | unknown | 0 | |
| 70 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 13 / 20 | 43%–82% | 0.92 | 13 | not measured (13/20) | unknown | 0 | |
| 71 | local/qwen3-coder-next-80b-a3b-6bitclaude-code · config ecb67e4e | 7 / 12 | 32%–81% | 12 | not measured (0/12) | unknown | 0 | ||
| 72 | moonshot/kimi-k3codex · config cb391ae9 | 4 / 7 | 25%–84% | 7 | not measured (0/7) | unknown | 0 | ||
| 73 | local/qwen3-coder-next-80b-a3b-6bitopencode · config 357da891 | 5 / 12 | 19%–68% | 12 | not measured (0/12) | unknown | 0 | ||
| 74 | local/qwen3-coder-next-80b-a3b-6bitpi · config da41c66e | 3 / 12 | 9%–53% | 12 | not measured (0/12) | unknown | 0 | ||
| 75 | local/qwen3-coder-next-80b-a3b-6bitcodex · config 3437a9e6 | 1 / 12 | 1%–35% | 12 | not measured (0/12) | unknown | 0 | ||
| 76 | together/qwen3.8-flashclaude-code · config ecb67e4e | — | 0%–0% | 0 | not measured (0/0) | unknown | 1 |
list = published-rate · invoice = provider-invoice-reconciled