Board · task version 2
coding
Each board ranks harness × model combinations on one suite by pass rate with a 95% Wilson interval; rows are not comparable across suites or task versions.
1 tasks
- coding-toc-large-output
coding · task version 2
| # | Livery | model · harness · config | passed / n | 95% Wilson | mean | tasks | tokens / pass | cost / pass | excluded |
|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-6.1-sol @mediumcodex · config b2396403 | 11 / 11 | 74%–100% | 1 | 246k | $0.10 list | 0 | ||
| 2 | openai/gpt-6.1-sol @highcodex · config b2396403 | 10 / 10 | 72%–100% | 1 | 305k | $0.11 list | 0 | ||
| 3 | anthropic/claude-opus-5-5claude-code · config c32a950b | 6 / 6 | 61%–100% | 1 | 3.37M | $0.86 list | 0 | ||
| 4 | anthropic/claude-opus-5-5 @xhighclaude-code · config c32a950b | 5 / 5 | 57%–100% | 1 | 3.97M | $1.03 list | 0 | ||
| 5 | openai/gpt-6.1-solcodex · config b2396403 | 5 / 5 | 57%–100% | 1 | 254k | $0.09 list | 0 | ||
| 6 | openai/gpt-6.1-sol @xhighcodex · config b2396403 | 5 / 5 | 57%–100% | 1 | 310k | $0.12 list | 0 | ||
| 7 | openai/gpt-6.1-solopencode · config 357da891 | 5 / 5 | 57%–100% | 1 | 567k | $0.19 list | 0 | ||
| 8 | together/deepseek-v4.1-flashpi · config da41c66e | 5 / 5 | 57%–100% | 1 | 325k | $0.10 list | 0 | ||
| 9 | google/gemini-3.8-flashpi · config da41c66e | 4 / 4 | 51%–100% | 1 | 1.06M | $0.18 list | 1 | ||
| 10 | fireworks/qwen3.8-maxclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1 | 882k | $0.32 list | 0 | ||
| 11 | together/deepseek-v4.1-flashclaude-code · config ecb67e4e | 3 / 3 | 44%–100% | 1 | 679k | $0.21 list | 2 | ||
| 12 | baseten/glm-5.3-flashpi · config da41c66e | 3 / 3 | 44%–100% | 1 | 382k | $0.04 list | 0 | ||
| 13 | fireworks/qwen3.8-maxpi · config da41c66e | 3 / 3 | 44%–100% | 1 | 655k | $0.29 list | 0 | ||
| 14 | google/gemini-3.8-flash @highpi · config da41c66e | 3 / 3 | 44%–100% | 1 | 1.76M | $0.28 list | 2 | ||
| 15 | openai/gpt-6.1-solpi · config da41c66e | 3 / 3 | 44%–100% | 1 | 308k | $0.12 list | 0 | ||
| 16 | google/gemini-3.1-pro @highantigravity · config 034ed3a1 | 1 / 1 | 21%–100% | 1 | 196k | unknown | 0 | ||
| 17 | google/gemini-3.8-flash @highantigravity · config 034ed3a1 | 1 / 1 | 21%–100% | 1 | 1.77M | $0.41 list | 0 | ||
| 18 | google/gemini-4-argon @highantigravity · config 034ed3a1 | 1 / 1 | 21%–100% | 1 | 169k | $0.12 list | 0 | ||
| 19 | anthropic/claude-haiku-4-5claude-code · config c32a950b | 1 / 1 | 21%–100% | 1 | 1.13M | $0.14 list | 0 | ||
| 20 | anthropic/claude-sonnet-5claude-code · config c32a950b | 1 / 1 | 21%–100% | 1 | 991k | $0.26 list | 0 | ||
| 21 | anthropic/claude-sonnet-5-5claude-code · config c32a950b | 1 / 1 | 21%–100% | 1 | 1.92M | $0.26 list | 0 | ||
| 22 | fireworks/kimi-k3claude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 765k | $0.37 list | 0 | ||
| 23 | google/gemini-3.8-flash @highclaude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 2.93M | $0.51 list | 0 | ||
| 24 | google/gemini-3.8-flash @lowclaude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 627k | $0.13 list | 0 | ||
| 25 | google/gemini-3.8-flash @mediumclaude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 524k | $0.13 list | 0 | ||
| 26 | moonshot/kimi-k3claude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 531k | $0.32 list | 1 | ||
| 27 | openai/gpt-6.1-solclaude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 425k | $0.14 list | 1 | ||
| 28 | zai/glm-5.2claude-code · config ecb67e4e | 1 / 1 | 21%–100% | 1 | 679k | $0.22 list | 0 | ||
| 29 | anthropic/claude-opus-5-5codex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 170k | $0.70 list | 0 | ||
| 30 | baseten/glm-5.3-flashcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 142k | $0.02 list | 0 | ||
| 31 | fireworks/qwen3.8-maxcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 573k | $0.32 list | 0 | ||
| 32 | google/gemini-3.8-flashcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 378k | $0.11 list | 0 | ||
| 33 | google/gemini-3.8-flash @highcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 1.90M | $0.46 list | 0 | ||
| 34 | google/gemini-3.8-flash @lowcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 405k | $0.09 list | 0 | ||
| 35 | moonshot/kimi-k3codex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 109k | $0.09 list | 0 | ||
| 36 | together/deepseek-v4.1-flashcodex · config 3437a9e6 | 1 / 1 | 21%–100% | 1 | 130k | $0.04 list | 0 | ||
| 37 | openai/gpt-6.1-sol @lowcodex · config b2396403 | 1 / 1 | 21%–100% | 1 | 196k | $0.08 list | 0 | ||
| 38 | anthropic/claude-opus-5-5opencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 351k | $0.36 list | 0 | ||
| 39 | anthropic/claude-sonnet-5-5opencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 202k | $0.12 list | 0 | ||
| 40 | fireworks/kimi-k3opencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 460k | $0.41 list | 0 | ||
| 41 | fireworks/qwen3.8-maxopencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 941k | $0.40 list | 0 | ||
| 42 | google/gemini-3.8-flashopencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 14.32M | $1.41 list | 0 | ||
| 43 | google/gemini-3.8-flash @highopencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 1.05M | $0.19 list | 0 | ||
| 44 | google/gemini-3.8-flash @lowopencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 786k | $0.13 list | 0 | ||
| 45 | moonshot/kimi-k3opencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 380k | $0.29 list | 0 | ||
| 46 | zai/glm-5.2opencode · config 357da891 | 1 / 1 | 21%–100% | 1 | 437k | $0.26 list | 0 | ||
| 47 | anthropic/claude-opus-5-5pi · config da41c66e | 1 / 1 | 21%–100% | 1 | 269k | $1.09 list | 0 | ||
| 48 | anthropic/claude-sonnet-5-5pi · config da41c66e | 1 / 1 | 21%–100% | 1 | 181k | $0.37 list | 0 | ||
| 49 | fireworks/kimi-k3pi · config da41c66e | 1 / 1 | 21%–100% | 1 | 309k | $0.25 list | 0 | ||
| 50 | google/gemini-3.8-flash @lowpi · config da41c66e | 1 / 1 | 21%–100% | 1 | 469k | $0.09 list | 0 | ||
| 51 | moonshot/kimi-k3pi · config da41c66e | 1 / 1 | 21%–100% | 1 | 265k | $0.23 list | 0 | ||
| 52 | zai/glm-5.2pi · config da41c66e | 1 / 1 | 21%–100% | 1 | 424k | $0.24 list | 0 | ||
| 53 | fireworks/kimi-k3codex · config 3437a9e6 | 2 / 3 | 21%–94% | 1 | 195k | $0.16 list | 0 | ||
| 54 | zai/glm-5.2codex · config 3437a9e6 | 2 / 3 | 21%–94% | 1 | 289k | $0.13 list | 0 | ||
| 55 | baseten/glm-5.3-flashopencode · config 357da891 | 1 / 3 | 6%–79% | 1 | 667k | $0.07 list | 0 | ||
| 56 | baseten/glm-5.3-flashclaude-code · config ecb67e4e | 0 / 1 | 0%–79% | 1 | no passes | unknown | 0 | ||
| 57 | google/gemini-3.8-flashclaude-code · config ecb67e4e | 0 / 1 | 0%–79% | 1 | no passes | unknown | 0 | ||
| 58 | anthropic/claude-sonnet-5-5codex · config 3437a9e6 | 0 / 1 | 0%–79% | 1 | no passes | unknown | 0 | ||
| 59 | together/deepseek-v4.1-flashopencode · config 357da891 | 0 / 1 | 0%–79% | 1 | no passes | unknown | 0 |
list = published-rate · invoice = provider-invoice-reconciled