Board · task version 1
rca
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
3 tasks
- rca-admit-pressure-crash-loop
- rca-fk-drain
- rca-litellm-watchdog-restart-storm
rca
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 14 / 14 | 78%–100% | 1.00 | 3 | 66k | $0.03 list | 0 | |
| 2 | openai/gpt-6.1-sol @highcodex · config b2396403 | 12 / 12 | 76%–100% | 1.00 | 3 | 67k | $0.03 list | 3 | |
| 3 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 7 / 7 | 65%–100% | 1.00 | 3 | 146k | $0.09 list | 3 | |
| 4 | google/gemini-3.8-flashpi · config da41c66e | 7 / 7 | 65%–100% | 1.00 | 3 | 85k | $0.07 list | 0 | |
| 5 | together/deepseek-v4.1-flashpi · config da41c66e | 7 / 7 | 65%–100% | 1.00 | 3 | 17k | $0.0061 list | 0 | |
| 6 | together/deepseek-v4.1-flashopencode · config 357da891 | 6 / 6 | 61%–100% | 1.00 | 3 | 38k | $0.01 list | 0 | |
| 7 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 5 / 5 | 57%–100% | 1.00 | 1 | 637k | $0.24 list | 0 | |
| 8 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 5 / 5 | 57%–100% | 1.00 | 3 | 175k | $0.05 list | 0 | |
| 9 | fireworks/kimi-k3codex · config 3437a9e6 | 5 / 5 | 57%–100% | 1.00 | 3 | 33k | $0.04 list | 0 | |
| 10 | zai/glm-5.2codex · config 3437a9e6 | 5 / 5 | 57%–100% | 1.00 | 3 | 54k | $0.04 list | 0 | |
| 11 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 5 / 5 | 57%–100% | 1.00 | 1 | 71k | $0.04 list | 0 | |
| 12 | baseten/glm-5.3-flashopencode · config 357da891 | 5 / 5 | 57%–100% | 1.00 | 3 | 32k | $0.0042 list | 0 | |
| 13 | openai/gpt-6.1-solopencode · config 357da891 | 5 / 5 | 57%–100% | 1.00 | 3 | 54k | $0.04 list | 0 | |
| 14 | baseten/glm-5.3-flashpi · config da41c66e | 5 / 5 | 57%–100% | 1.00 | 3 | 13k | $0.0022 list | 0 | |
| 15 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 3 / 3 | 44%–100% | 1.00 | 3 | 82k | unknown | 1 | |
| 16 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 3 / 3 | 44%–100% | 1.00 | 3 | 410k | $0.26 list | 0 | |
| 17 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 3 / 3 | 44%–100% | 1.00 | 3 | 102k | $0.14 list | 0 | |
| 18 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 3 / 3 | 44%–100% | 1.00 | 3 | 506k | $0.08 list | 0 | |
| 19 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 182k | $0.02 list | 0 | |
| 20 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 402k | $0.17 list | 0 | |
| 21 | google/gemini-3.8-flash @highclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 1.09M | $0.38 list | 0 | |
| 22 | google/gemini-3.8-flash @lowclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 67k | $0.06 list | 0 | |
| 23 | google/gemini-3.8-flash @mediumclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 398k | $0.26 list | 0 | |
| 24 | moonshot/kimi-k3claude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 123k | $0.08 list | 0 | |
| 25 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1.00 | 3 | 123k | $0.06 list | 0 | |
| 26 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 40k | $0.17 list | 0 | |
| 27 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 38k | $0.08 list | 0 | |
| 28 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 25k | $0.0040 list | 0 | |
| 29 | google/gemini-3.8-flashcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 188k | $0.15 list | 0 | |
| 30 | google/gemini-3.8-flash @lowcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 57k | $0.04 list | 0 | |
| 31 | moonshot/kimi-k3codex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 33k | $0.04 list | 3 | |
| 32 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 3 / 3 | 44%–100% | 1.00 | 3 | 30k | $0.0098 list | 0 | |
| 33 | anthropic/claude-opus-5-5opencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 47k | $0.10 list | 0 | |
| 34 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 44k | $0.04 list | 0 | |
| 35 | fireworks/kimi-k3opencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 56k | $0.09 list | 0 | |
| 36 | google/gemini-3.8-flashopencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 167k | $0.12 list | 0 | |
| 37 | google/gemini-3.8-flash @highopencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 451k | $0.30 list | 0 | |
| 38 | google/gemini-3.8-flash @lowopencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 76k | $0.06 list | 0 | |
| 39 | moonshot/kimi-k3opencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 61k | $0.07 list | 0 | |
| 40 | zai/glm-5.2opencode · config 357da891 | 3 / 3 | 44%–100% | 1.00 | 3 | 49k | $0.03 list | 0 | |
| 41 | anthropic/claude-opus-5-5pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 18k | $0.09 list | 0 | |
| 42 | anthropic/claude-sonnet-5-5pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 17k | $0.04 list | 0 | |
| 43 | fireworks/kimi-k3pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 20k | $0.04 list | 0 | |
| 44 | google/gemini-3.8-flash @highpi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 216k | $0.18 list | 0 | |
| 45 | moonshot/kimi-k3pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 23k | $0.04 list | 1 | |
| 46 | openai/gpt-6.1-solpi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 16k | $0.01 list | 0 | |
| 47 | zai/glm-5.2pi · config da41c66e | 3 / 3 | 44%–100% | 1.00 | 3 | 27k | $0.04 list | 0 | |
| 48 | openai/gpt-5.6-luna @mediumcodex · config b2396403 | 1 / 1 | 21%–100% | 1.00 | 1 | 55k | $0.0040 list | 0 | |
| 49 | openai/gpt-5.6-luna @xhighcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 55k | $0.0033 list | 0 | ||
| 50 | openai/gpt-5.6-sol @highcodex · config b2396403 | 1 / 1 | 21%–100% | 1.00 | 1 | 61k | $0.05 list | 0 | |
| 51 | google/gemini-3.8-flash @lowpi · config da41c66e | 1 / 1 | 21%–100% | 1.00 | 1 | 12k | $0.0094 list | 2 | |
| 52 | fireworks/qwen3.8-maxopencode · config 357da891 | 5 / 6 | 44%–97% | 3 | 55k | $0.05 list | 1 | ||
| 53 | openai/gpt-6.1-solcodex · config b2396403 | 9 / 11 | 52%–95% | 1.00 | 3 | 81k | $0.03 list | 1 | |
| 54 | zai/glm-5.2claude-code · config ecb67e4e | 7 / 9 | 45%–94% | 1.00 | 3 | 223k | $0.10 list | 0 | |
| 55 | fireworks/qwen3.8-maxpi · config da41c66e | 5 / 7 | 36%–92% | 1.00 | 3 | 45k | $0.05 list | 0 | |
| 56 | anthropic/claude-haiku-4-5claude-code · config c32a950b | 2 / 3 | 21%–94% | 2 | 724k | $0.10 list | 3 | ||
| 57 | anthropic/claude-sonnet-5claude-code · config c32a950b | 2 / 3 | 21%–94% | 3 | 975k | $0.27 list | 3 | ||
| 58 | fireworks/kimi-k3claude-code · config ecb67e4e | 2 / 3 | 21%–94% | 1.00 | 3 | 197k | $0.09 list | 0 | |
| 59 | anthropic/claude-opus-5-5claude-code · config c32a950b | 7 / 11 | 35%–85% | 1.00 | 3 | 1.06M | $0.36 list | 0 | |
| 60 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 3 / 6 | 19%–81% | 3 | 130k | $0.13 list | 0 | ||
| 61 | openai/gpt-6-luna @highcodex · config b2396403 | 0 / 1 | 0%–79% | 1 | no passes | unknown | 0 |
list = published-rate · invoice = provider-invoice-reconciled