Board · task version 3
review
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
3 tasks
- review-corrupt-refresh-checkpoint
- review-rescue-local-tip
- review-stale-lock-recovery
review · task version 3
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 23 / 23 | 86%–100% | 1.00 | 3 | 107k | $0.05 list | 0 | |
| 2 | openai/gpt-6.1-sol @highcodex · config b2396403 | 21 / 21 | 85%–100% | 1.00 | 3 | 108k | $0.05 list | 1 | |
| 3 | anthropic/claude-opus-5-5claude-code · config c32a950b | 12 / 12 | 76%–100% | 1.00 | 3 | 789k | $0.32 list | 0 | |
| 4 | openai/gpt-6.1-solcodex · config b2396403 | 12 / 12 | 76%–100% | 1.00 | 3 | 96k | $0.05 list | 0 | |
| 5 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 10 / 10 | 72%–100% | 0.98 | 2 | 1.26M | $0.75 list | 0 | |
| 6 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 10 / 10 | 72%–100% | 1.00 | 2 | 114k | $0.06 list | 0 | |
| 7 | google/gemini-3.8-flashpi · config da41c66e | 10 / 10 | 72%–100% | 1.00 | 3 | 397k | $0.23 list | 1 | |
| 8 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 9 / 9 | 70%–100% | 1.00 | 3 | 827k | $0.40 list | 0 | |
| 9 | openai/gpt-6.1-solopencode · config 357da891 | 7 / 7 | 65%–100% | 1.00 | 3 | 129k | $0.09 list | 0 | |
| 10 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 6 / 6 | 61%–100% | 1.00 | 3 | 288k | unknown | 0 | |
| 11 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 6 / 6 | 61%–100% | 1.00 | 3 | 767k | $0.15 list | 0 | |
| 12 | together/deepseek-v4.1-flashopencode · config 357da891 | 5 / 5 | 57%–100% | 1.00 | 3 | 82k | $0.03 list | 1 | |
| 13 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 3 / 3 | 44%–100% | 1.00 | 3 | 249k | $0.53 list | 0 | |
| 14 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 802k | $0.32 list | 0 | |
| 15 | google/gemini-3.8-flash @mediumclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 481k | $0.32 list | 0 | |
| 16 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 194k | $0.09 list | 0 | |
| 17 | google/gemini-3.8-flashcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 501k | $0.27 list | 0 | |
| 18 | google/gemini-3.8-flash @lowcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 0.93 | 3 | 244k | $0.15 list | 0 | |
| 19 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 3 / 3 | 44%–100% | 0.93 | 3 | 77k | $0.07 list | 0 | |
| 20 | google/gemini-3.8-flashopencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 450k | $0.22 list | 0 | |
| 21 | google/gemini-3.8-flash @highopencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 565k | $0.32 list | 0 | |
| 22 | anthropic/claude-sonnet-5-5pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 26k | $0.07 list | 0 | |
| 23 | fireworks/kimi-k3pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 67k | $0.21 list | 0 | |
| 24 | google/gemini-3.8-flash @highpi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 693k | $0.37 list | 0 | |
| 25 | moonshot/kimi-k3pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 59k | $0.18 list | 0 | |
| 26 | openai/gpt-6.1-solpi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 39k | $0.04 list | 0 | |
| 27 | together/deepseek-v4.1-flashpi · config da41c66e | 11 / 12 | 65%–99% | 0.97 | 3 | 129k | $0.05 list | 1 | |
| 28 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 7 / 8 | 53%–98% | 0.95 | 3 | 351k | $0.13 list | 0 | |
| 29 | fireworks/kimi-k3codex · config 3437a9e6 | 5 / 7 | 36%–92% | 0.83 | 3 | 109k | $0.21 list | 0 | |
| 30 | fireworks/qwen3.8-maxpi · config da41c66e | 5 / 7 | 36%–92% | 0.86 | 3 | 382k | $0.31 list | 0 | |
| 31 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 2 / 3 | 21%–94% | 0.87 | 3 | 737k | $0.08 list | 0 | |
| 32 | fireworks/kimi-k3claude-code · config ecb67e4e | 2 / 3 | 21%–94% | 0.87 | 3 | 481k | $0.34 list | 0 | |
| 33 | google/gemini-3.8-flash @highclaude-code · config ecb67e4e | 2 / 3 | 21%–94% | 1.00 | 3 | 779k | $0.35 list | 0 | |
| 34 | moonshot/kimi-k3claude-code · config ecb67e4e | 2 / 3 | 21%–94% | 0.87 | 3 | 385k | $0.42 list | 0 | |
| 35 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 2 / 3 | 21%–94% | 0.87 | 3 | 86k | $0.41 list | 0 | |
| 36 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 2 / 3 | 21%–94% | 0.80 | 3 | 71k | $0.01 list | 0 | |
| 37 | moonshot/kimi-k3codex · config 3437a9e6 | 2 / 3 | 21%–94% | 0.87 | 3 | 92k | $0.24 list | 0 | |
| 38 | anthropic/claude-opus-5-5opencode · config 357da891 | 2 / 3 | 21%–94% | 0.87 | 3 | 100k | $0.21 list | 0 | |
| 39 | fireworks/kimi-k3opencode · config 357da891 | 2 / 3 | 21%–94% | 0.87 | 3 | 156k | $0.44 list | 1 | |
| 40 | fireworks/qwen3.8-maxopencode · config 357da891 | 2 / 3 | 21%–94% | 0.87 | 3 | 192k | $0.22 list | 0 | |
| 41 | google/gemini-3.8-flash @lowopencode · config 357da891 | 2 / 3 | 21%–94% | 0.87 | 3 | 144k | $0.10 list | 0 | |
| 42 | moonshot/kimi-k3opencode · config 357da891 | 2 / 3 | 21%–94% | 0.87 | 3 | 150k | $0.32 list | 0 | |
| 43 | anthropic/claude-opus-5-5pi · config da41c66e | 2 / 3 | 21%–94% | 0.87 | 3 | 38k | $0.21 list | 0 | |
| 44 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 6 / 10 | 31%–83% | 0.78 | 3 | 467k | $0.49 list | 3 | |
| 45 | baseten/glm-5.3-flashopencode · config 357da891 | 4 / 7 | 25%–84% | 0.80 | 3 | 212k | $0.03 list | 1 | |
| 46 | zai/glm-5.2codex · config 3437a9e6 | 4 / 8 | 22%–78% | 0.80 | 3 | 772k | $1.08 list | 0 | |
| 47 | fireworks/qwen3.8-27bclaude-code · config ecb67e4e | 1 / 2 | 9%–91% | 0.70 | 2 | 323k | unknown | 1 | |
| 48 | baseten/glm-5.3-flashpi · config da41c66e | 3 / 7 | 16%–75% | 0.80 | 3 | 103k | $0.02 list | 0 | |
| 49 | zai/glm-5.2claude-code · config ecb67e4e | 2 / 6 | 10%–70% | 0.73 | 3 | 1.52M | $1.03 list | 1 | |
| 50 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 2 / 6 | 10%–70% | 0.77 | 3 | 283k | $0.14 list | 0 | |
| 51 | zai/glm-5.2pi · config da41c66e | 2 / 6 | 10%–70% | 0.73 | 3 | 636k | $0.76 list | 0 | |
| 52 | google/gemini-3.8-flash @lowclaude-code · config ecb67e4e | 1 / 3 | 6%–79% | 0.73 | 3 | 358k | $0.28 list | 0 | |
| 53 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 1 / 3 | 6%–79% | 0.87 | 3 | 149k | $0.33 list | 0 | |
| 54 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 1 / 3 | 6%–79% | 0.87 | 3 | 510k | $0.70 list | 0 | |
| 55 | zai/glm-5.2opencode · config 357da891 | 1 / 6 | 3%–56% | 0.57 | 3 | 1.09M | $1.08 list | 0 | |
| 56 | google/gemini-3.8-flash @lowpi · config da41c66e | — | 0%–0% | 0 | not measured (0/0) | unknown | 3 |
list = published-rate · invoice = provider-invoice-reconciled