Board · task version 1
hard-review
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
9 tasks
- hard-review-clean-long
- hard-review-cross-file
- hard-review-cross-file-v2
- hard-review-decoy-decision-replies
- hard-review-decoy-job-leases
- hard-review-multi-defect-outbox
- hard-review-multi-defect-outbox-v2
- hard-review-multi-defect-tool-calls
- hard-review-multi-defect-tool-calls-v2
hard-review · task version 1
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | together/deepseek-v4.1-flashpi · config da41c66e | 13 / 13 | 77%–100% | 0.93 | 6 | 244k | $0.09 list | 0 | |
| 2 | together/deepseek-v4.1-flashopencode · config 357da891 | 9 / 10 | 60%–98% | 0.85 | 9 | not measured (3/10) | unknown | 5 | |
| 3 | zai/glm-5.2codex · config 3437a9e6 | 16 / 18 | 67%–97% | 0.87 | 8 | 490k | $0.48 list | 1 | |
| 4 | zai/glm-5.2claude-code · config ecb67e4e | 8 / 9 | 57%–98% | 0.84 | 9 | not measured (5/9) | unknown | 11 | |
| 5 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 21 / 24 | 69%–96% | 0.94 | 8 | 1.77M | $0.59 list | 0 | |
| 6 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 7 / 8 | 53%–98% | 0.89 | 8 | 272k | $0.41 list | 0 | |
| 7 | fireworks/kimi-k3opencode · config 357da891 | 7 / 8 | 53%–98% | 0.89 | 8 | 231k | $0.38 list | 0 | |
| 8 | zai/glm-5.2opencode · config 357da891 | 7 / 8 | 53%–98% | 0.85 | 8 | 294k | $0.26 list | 0 | |
| 9 | fireworks/kimi-k3pi · config da41c66e | 7 / 8 | 53%–98% | 0.84 | 8 | 159k | $0.32 list | 0 | |
| 10 | google/gemini-3.8-flashpi · config da41c66e | 7 / 8 | 53%–98% | 0.92 | 8 | 1.08M | $0.42 list | 0 | |
| 11 | google/gemini-3.8-flash @highpi · config da41c66e | 7 / 8 | 53%–98% | 0.94 | 8 | 1.89M | $0.64 list | 0 | |
| 12 | moonshot/kimi-k3pi · config da41c66e | 7 / 8 | 53%–98% | 0.82 | 8 | 147k | $0.28 list | 0 | |
| 13 | openai/gpt-6.1-solpi · config da41c66e | 7 / 8 | 53%–98% | 0.96 | 8 | 77k | $0.07 list | 0 | |
| 14 | zai/glm-5.2pi · config da41c66e | 7 / 8 | 53%–98% | 0.84 | 8 | 305k | $0.28 list | 0 | |
| 15 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 15 / 18 | 61%–94% | 0.89 | 8 | 816k | $0.29 list | 0 | |
| 16 | baseten/glm-5.3-flashopencode · config 357da891 | 15 / 18 | 61%–94% | 0.89 | 8 | 251k | $0.03 list | 1 | |
| 17 | baseten/glm-5.3-flashpi · config da41c66e | 14 / 18 | 55%–91% | 0.86 | 8 | 203k | $0.03 list | 0 | |
| 18 | fireworks/qwen3.8-maxpi · config da41c66e | 14 / 18 | 55%–91% | 0.89 | 8 | 377k | $0.30 list | 0 | |
| 19 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 38 / 50 | 63%–86% | 0.94 | 5 | 185k | $0.09 list | 0 | |
| 20 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 6 / 8 | 41%–93% | 0.87 | 8 | 1.27M | $0.09 list | 0 | |
| 21 | fireworks/kimi-k3claude-code · config ecb67e4e | 6 / 8 | 41%–93% | 0.85 | 8 | 327k | $0.24 list | 0 | |
| 22 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 6 / 8 | 41%–93% | 0.95 | 8 | 390k | $0.21 list | 0 | |
| 23 | google/gemini-3.8-flashcodex · config 3437a9e6 | 6 / 8 | 41%–93% | 0.91 | 8 | 1.28M | $0.49 list | 0 | |
| 24 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 6 / 8 | 41%–93% | 0.81 | 8 | 164k | $0.14 list | 1 | |
| 25 | fireworks/qwen3.8-maxopencode · config 357da891 | 6 / 8 | 41%–93% | 0.94 | 8 | 396k | $0.34 list | 0 | |
| 26 | moonshot/kimi-k3opencode · config 357da891 | 6 / 8 | 41%–93% | 0.80 | 8 | 244k | $0.40 list | 0 | |
| 27 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 16 / 22 | 52%–87% | 0.90 | 9 | not measured (13/22) | unknown | 11 | |
| 28 | openai/gpt-6.1-solcodex · config b2396403 | 20 / 29 | 51%–83% | 0.92 | 9 | not measured (23/29) | unknown | 0 | |
| 29 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 5 / 8 | 31%–86% | 0.92 | 8 | 2.40M | $0.71 list | 0 | |
| 30 | moonshot/kimi-k3claude-code · config ecb67e4e | 5 / 8 | 31%–86% | 0.79 | 8 | 599k | $0.39 list | 0 | |
| 31 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 5 / 8 | 31%–86% | 0.91 | 8 | 145k | $0.65 list | 0 | |
| 32 | moonshot/kimi-k3codex · config 3437a9e6 | 5 / 8 | 31%–86% | 0.76 | 8 | 248k | $0.32 list | 0 | |
| 33 | google/gemini-3.8-flashopencode · config 357da891 | 5 / 8 | 31%–86% | 0.83 | 8 | 1.45M | $0.53 list | 0 | |
| 34 | anthropic/claude-sonnet-5-5pi · config da41c66e | 5 / 8 | 31%–86% | 0.82 | 8 | 98k | $0.24 list | 0 | |
| 35 | fireworks/kimi-k3codex · config 3437a9e6 | 11 / 18 | 39%–80% | 0.79 | 8 | 210k | $0.29 list | 0 | |
| 36 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 14 / 25 | 37%–73% | 0.90 | 5 | 336k | $0.20 list | 0 | |
| 37 | openai/gpt-6.1-solopencode · config 357da891 | 9 / 18 | 29%–71% | 0.89 | 8 | 437k | $0.27 list | 0 | |
| 38 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 4 / 8 | 22%–78% | 0.75 | 8 | 168k | $0.37 list | 0 | |
| 39 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 4 / 8 | 22%–78% | 0.87 | 8 | 594k | $0.61 list | 8 | |
| 40 | anthropic/claude-opus-5-5pi · config da41c66e | 4 / 8 | 22%–78% | 0.89 | 8 | 117k | $0.58 list | 0 | |
| 41 | openai/gpt-6.1-sol @highcodex · config b2396403 | 17 / 35 | 33%–64% | 0.87 | 8 | 320k | $0.18 list | 7 | |
| 42 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 4 / 9 | 19%–73% | 0.70 | 9 | not measured (3/9) | unknown | 0 | |
| 43 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 8 / 20 | 22%–61% | 0.85 | 5 | 4.53M | $2.78 list | 3 | |
| 44 | anthropic/claude-opus-5-5claude-code · config c32a950b | 11 / 28 | 24%–58% | 0.85 | 8 | 3.00M | $1.20 list | 0 | |
| 45 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 3 / 8 | 14%–69% | 0.80 | 8 | 2.53M | $0.55 list | 0 | |
| 46 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 3 / 8 | 14%–69% | 0.76 | 8 | 485k | $0.07 list | 0 | |
| 47 | anthropic/claude-opus-5-5opencode · config 357da891 | 3 / 8 | 14%–69% | 0.84 | 8 | 262k | $0.52 list | 0 | |
| 48 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 0 / 3 | 0%–56% | 0.70 | 3 | no passes | unknown | 0 |
list = published-rate · invoice = provider-invoice-reconciled