Benchmarks
What the tests in AI launch posts actually measure, which ones still tell models apart, and five questions to ask before trusting any score.
Five questions to ask about any score
Composite indexes
One number built from many tests. Useful for a first comparison; they disagree with each other because they weight different work.
Intelligence Index v5
Artificial Analysis (independent)
- Measures
- General capability across knowledge work, coding, science and general knowledge.
- Built from
- Nine evaluations, 25% per category. 60% of the weight uses private or held-out test sets.
- Reports
- Score 0–100 and cost per task, at every reasoning-effort setting.
- Leader
- Gemini 4 Carbon 60 (restricted access); Claude Opus 5.5 59. Read Nov 5, 2026.
Vals Index v2.2
Vals AI (independent)
- Measures
- Agent work weighted by the size of the industry it comes from.
- Built from
- Finance, coding, legal and tax tasks, weighted 8.0 : 5.6 : 1.2 : 0.5. Mostly private test sets.
- Reports
- Accuracy, cost per test, and time per test.
- Leader
- Claude Sonnet 5.5 at 70.85% ($18.70 per test), Opus 5.5 70.18%, Gemini 4 Argon 69.24%. Oct 27, 2026.
Inside the Intelligence Index
Terminal-Bench 4.0
The model gets a real command line and a job to finish — install packages, debug configurations, chain commands. Each task runs five times.
Watch for: Version matters: scores on 2.1 and 4.0 are not comparable. Gemini 3.8 Flash scored 87.6% on 2.1 and 19.1% on 4.0.
FrontierCode 2.0
Coding tasks graded the way a human code reviewer would grade them, not just by whether tests pass. Built by Cognition.
Watch for: Strict by design: even the best models score far from 100%, so small gaps matter.
Terminal-Bench-Science
Scientific computing tasks done through a terminal.
Watch for: Uses about a third of all output tokens in the index, so it drives a lot of the cost-per-task figure.
HLE-Diamond
A curated subset of Humanity's Last Exam, a set of 2,500 expert-written questions across maths, science and the humanities.
Watch for: With tools (search, code) scores run far higher than without. Always check which was used.
AA-Omniscience
Factual questions where a confident wrong answer costs points, so saying 'I don't know' is better than guessing.
Watch for: One of the few tests that rewards a model for not making things up.
AA-Briefcase, AutomationBench, GDP.pdf
Office work: multi-step analysis deliverables, automating workflows across apps, and document tasks graded all-or-nothing.
Watch for: The closest of these tests to the work most MBA graduates will hand to a model.
Other tests you'll see in launch posts
SWE-bench Pro
Fix real issues in actively maintained open-source code repositories; graded by the project's own tests.
Watch for: Labs run it in their own setups. Compare numbers only when the harness and effort level match.
DeepSWE
Longer agentic software-engineering tasks. OpenAI, Meta and DeepSeek quoted it for their autumn launches.
Watch for: Mostly vendor-reported so far; few independent runs.
OSWorld
Operate a desktop computer from screenshots — open apps, fill forms, move files.
Watch for: Several versions and 'strict' vs 'partial' grading. Claude Opus 5.5's card lists 48.7% strict and 81.8% partial on the same test.
ARC-AGI
Abstract visual puzzles that are easy for people and were hard for models.
Watch for: Setup changes everything: GPT-6 Astra's 99.9% came from a specialized setup; a neutral harness measured 62.7%.
Retired: too easy to tell models apart
SWE-bench Verified
A human-checked set of GitHub issues — the standard coding test of 2025.
Watch for: Saturated: three models above 95%. Labs moved to SWE-bench Pro.
MMLU, GPQA, AIME
Multiple-choice knowledge, graduate science questions, and competition maths.
Watch for: Top models score near 100%, so differences are noise. Still common in marketing.