Board · task version 1
hard-fix
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
16 tasks
- hard-fix-cross-file
- hard-fix-cross-file-v2
- hard-fix-cross-file-v3
- hard-fix-regression-test-dedupe
- hard-fix-regression-test-dedupe-v2
- hard-fix-regression-test-dedupe-v3
- hard-fix-regression-test-quiet-hours
- hard-fix-regression-test-quiet-hours-v2
- hard-fix-regression-test-quiet-hours-v3
- hard-fix-two-findings
- hard-fix-two-findings-v2
- hard-fix-two-findings-v3
- hard-fix-wrong-recommendation-config-reload
- hard-fix-wrong-recommendation-retry-charge
- hard-fix-wrong-recommendation-retry-charge-v2
- hard-fix-wrong-recommendation-retry-charge-v3
hard-fix
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 26 / 26 | 87%–100% | 1.00 | 16 | 97k | $0.04 list | 0 | |
| 2 | openai/gpt-6.1-sol @highcodex · config b2396403 | 20 / 20 | 84%–100% | 1.00 | 11 | 125k | $0.05 list | 0 | |
| 3 | anthropic/claude-opus-5-5claude-code · config c32a950b | 15 / 15 | 80%–100% | 1.00 | 11 | 1.06M | $0.38 list | 0 | |
| 4 | openai/gpt-6.1-solcodex · config b2396403 | 15 / 15 | 80%–100% | 1.00 | 11 | not measured (9/15) | unknown | 0 | |
| 5 | openai/gpt-6.1-solopencode · config 357da891 | 13 / 13 | 77%–100% | 1.00 | 11 | 129k | $0.08 list | 0 | |
| 6 | fireworks/qwen3.8-maxpi · config da41c66e | 13 / 13 | 77%–100% | 1.00 | 11 | 96k | $0.09 list | 0 | |
| 7 | google/gemini-3.8-flashpi · config da41c66e | 13 / 13 | 77%–100% | 1.00 | 11 | 456k | $0.20 list | 1 | |
| 8 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 11 / 11 | 74%–100% | 1.00 | 11 | 804k | $0.14 list | 0 | |
| 9 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 11 / 11 | 74%–100% | 1.00 | 11 | 223k | $0.09 list | 0 | |
| 10 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 57k | $0.26 list | 0 | |
| 11 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 42k | $0.09 list | 0 | |
| 12 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 62k | $0.0091 list | 0 | |
| 13 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 133k | $0.11 list | 11 | |
| 14 | google/gemini-3.8-flashcodex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 422k | $0.20 list | 0 | |
| 15 | google/gemini-3.8-flash @highcodex · config 3437a9e6 | 11 / 11 | 74%–100% | 1.00 | 11 | 828k | $0.27 list | 0 | |
| 16 | anthropic/claude-opus-5-5opencode · config 357da891 | 11 / 11 | 74%–100% | 1.00 | 11 | 58k | $0.12 list | 0 | |
| 17 | google/gemini-3.8-flashopencode · config 357da891 | 11 / 11 | 74%–100% | 1.00 | 11 | 508k | $0.20 list | 0 | |
| 18 | anthropic/claude-opus-5-5pi · config da41c66e | 11 / 11 | 74%–100% | 1.00 | 11 | 24k | $0.12 list | 0 | |
| 19 | anthropic/claude-sonnet-5-5pi · config da41c66e | 11 / 11 | 74%–100% | 1.00 | 11 | 18k | $0.04 list | 0 | |
| 20 | openai/gpt-6.1-solpi · config da41c66e | 11 / 11 | 74%–100% | 1.00 | 11 | 38k | $0.03 list | 0 | |
| 21 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 5 / 5 | 57%–100% | 1.00 | 1 | 2.06M | $0.75 list | 0 | |
| 22 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 5 / 5 | 57%–100% | 1.00 | 1 | 142k | $0.07 list | 0 | |
| 23 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 15 / 16 | 72%–99% | 0.99 | 16 | 238k | $0.23 list | 0 | |
| 24 | google/gemini-3.8-flash @lowclaude-code · config ecb67e4e | 15 / 16 | 72%–99% | 0.99 | 16 | 312k | $0.19 list | 0 | |
| 25 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 12 / 13 | 67%–99% | 0.97 | 11 | not measured (7/13) | unknown | 0 | |
| 26 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 12 / 13 | 67%–99% | 0.97 | 11 | 429k | $0.14 list | 0 | |
| 27 | google/gemini-3.8-flash @highpi · config da41c66e | 12 / 13 | 67%–99% | 0.97 | 11 | 947k | $0.31 list | 1 | |
| 28 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 10 / 11 | 62%–98% | 0.97 | 11 | not measured (5/11) | unknown | 0 | |
| 29 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 10 / 11 | 62%–98% | 0.97 | 11 | 1.10M | $0.34 list | 1 | |
| 30 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 10 / 11 | 62%–98% | 0.97 | 11 | 432k | $0.03 list | 0 | |
| 31 | fireworks/kimi-k3claude-code · config ecb67e4e | 10 / 11 | 62%–98% | 0.97 | 11 | 306k | $0.16 list | 1 | |
| 32 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 10 / 11 | 62%–98% | 1.00 | 11 | 1.32M | $0.29 list | 0 | |
| 33 | moonshot/kimi-k3claude-code · config ecb67e4e | 10 / 11 | 62%–98% | 0.97 | 11 | 457k | $0.27 list | 3 | |
| 34 | zai/glm-5.2claude-code · config ecb67e4e | 10 / 11 | 62%–98% | 0.97 | 11 | not measured (5/11) | unknown | 0 | |
| 35 | moonshot/kimi-k3codex · config 3437a9e6 | 10 / 11 | 62%–98% | 0.97 | 11 | 123k | $0.14 list | 0 | |
| 36 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 10 / 11 | 62%–98% | 0.97 | 11 | 95k | $0.03 list | 0 | |
| 37 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 10 / 11 | 62%–98% | 1.00 | 11 | 68k | $0.06 list | 0 | |
| 38 | fireworks/qwen3.8-maxopencode · config 357da891 | 10 / 11 | 62%–98% | 0.97 | 11 | 238k | $0.18 list | 0 | |
| 39 | zai/glm-5.2opencode · config 357da891 | 10 / 11 | 62%–98% | 0.97 | 11 | 240k | $0.15 list | 0 | |
| 40 | fireworks/kimi-k3pi · config da41c66e | 10 / 11 | 62%–98% | 0.97 | 11 | 54k | $0.10 list | 0 | |
| 41 | moonshot/kimi-k3pi · config da41c66e | 10 / 11 | 62%–98% | 0.97 | 11 | 62k | $0.11 list | 0 | |
| 42 | zai/glm-5.2pi · config da41c66e | 10 / 11 | 62%–98% | 0.97 | 11 | 134k | $0.11 list | 0 | |
| 43 | google/gemini-3.8-flash @mediumclaude-code · config ecb67e4e | 14 / 16 | 64%–97% | 0.97 | 16 | 787k | $0.30 list | 0 | |
| 44 | google/gemini-3.8-flash @highopencode · config 357da891 | 14 / 16 | 64%–97% | 0.97 | 16 | 1.30M | $0.38 list | 0 | |
| 45 | google/gemini-3.8-flash @lowopencode · config 357da891 | 14 / 16 | 64%–97% | 0.97 | 16 | 248k | $0.14 list | 0 | |
| 46 | together/deepseek-v4.1-flashopencode · config 357da891 | 14 / 16 | 64%–97% | 0.96 | 11 | not measured (5/16) | unknown | 1 | |
| 47 | google/gemini-3.8-flash @lowpi · config da41c66e | 14 / 16 | 64%–97% | 0.97 | 16 | 110k | $0.09 list | 0 | |
| 48 | moonshot/kimi-k3opencode · config 357da891 | 9 / 11 | 52%–95% | 0.97 | 11 | 151k | $0.18 list | 0 | |
| 49 | google/gemini-3.8-flash @highclaude-code · config ecb67e4e | 13 / 16 | 57%–93% | 0.97 | 16 | 1.79M | $0.39 list | 0 | |
| 50 | google/gemini-3.8-flash @lowcodex · config 3437a9e6 | 13 / 16 | 57%–93% | 0.95 | 16 | 333k | $0.19 list | 0 | |
| 51 | together/deepseek-v4.1-flashpi · config da41c66e | 12 / 15 | 55%–93% | 0.93 | 11 | 65k | $0.02 list | 0 | |
| 52 | fireworks/kimi-k3codex · config 3437a9e6 | 10 / 13 | 50%–92% | 0.92 | 11 | 89k | $0.11 list | 0 | |
| 53 | zai/glm-5.2codex · config 3437a9e6 | 10 / 13 | 50%–92% | 0.92 | 11 | 159k | $0.11 list | 0 | |
| 54 | fireworks/kimi-k3opencode · config 357da891 | 8 / 11 | 43%–90% | 0.96 | 11 | 176k | $0.24 list | 0 | |
| 55 | baseten/glm-5.3-flashopencode · config 357da891 | 9 / 13 | 42%–87% | 0.92 | 11 | 162k | $0.02 list | 0 | |
| 56 | baseten/glm-5.3-flashpi · config da41c66e | 9 / 13 | 42%–87% | 0.90 | 11 | 61k | $0.0099 list | 0 |
list = published-rate · invoice = provider-invoice-reconciled