Trends
The simple-evals-mm Index sums up each model in one number: its capability, estimated from all questions of the leaderboard benchmarks at once. The idea comes from Epoch AI's Capabilities Index (ECI) and Artificial Analysis' Progress Index: rather than averaging benchmark scores, item response theory (the method behind standardized tests) estimates every model's capability and every question's difficulty together.
Why not a plain mean? A question that every model solves, or none, says nothing about which model is stronger, yet it still counts in a mean, and as benchmarks saturate such questions squeeze the differences together. The Index weighs each question by how well it separates the models, so the questions that some models solve and others miss decide the order. A gap of 10 points means the same anywhere on the scale, also near the top where percentages bunch up, and every benchmark counts equally whatever its size. How it is computed.
simple-evals-mm Index over time
Every setting
All effort levels, with 90% intervals (bootstrap over questions).
simple-evals-mm Index vs benchmark score
Each benchmark's fitted curve: its expected score at each capability. A curve already at the top for every model is saturated; one that rises late still separates the models.
The fitted formula for every benchmark
Each curve is summarized by one formula:
where is a model's simple-evals-mm Index and is the S-shaped logistic function, rising from 0 to 1.
- , guessing rate: the score a model gets by answering at random: 1 / (number of options) for each multiple-choice question, 0 for a free-form answer, averaged over the benchmark. GPQA Diamond (four options) is 25%; MMMU-Pro (2 to 12 options, mostly 10) is 13%; MathVision (some questions have five options, the rest free-form) is 12.5%.
- , difficulty: the index at which a model is expected to get half of the way from guessing to 100%. For a free-form benchmark that is a 50% score; for GPQA Diamond, 62.5% (halfway from 25% to 100%). A higher is a harder benchmark: ZeroBench ( ≈ 162) lies beyond today's best models (≈ 150), while JGraphQA ( below 50) is already solved by the weakest models here.
- , slope: how fast the score rises around . A steep benchmark (large ) jumps from near guessing to near 100% within a few index points, so it separates models only near its difficulty and saturates quickly.
- Fit error: how far the formula is from the fitted curve (root mean square, in score points); small values mean the summary is faithful.
How the simple-evals-mm Index is computed
The idea. Think of a standardized test. Each model has one hidden number, its capability (each effort level of a model counts as a model of its own). Each question has a difficulty and a discrimination : how sharply it separates weaker from stronger models. The chance that a model answers a question correctly rises smoothly with the gap between its capability and the question's difficulty:
A model far above a question's difficulty almost always solves it, one far below almost never; sets how steep that change is. Fitting this to every attempt of every model on all questions gives the capabilities and the difficulties together. A model earns credit for solving questions that others miss, and little for questions everyone solves. The design follows Artificial Analysis' Progress Index and is close to Epoch AI's ECI.
Keeping it fair. Three adjustments make the comparison fair.
- A lucky guess is not capability, so multiple-choice scores are rescaled so that random guessing counts as 0. Each question is rescaled by its own number of options.
- With only a few dozen models, one question's difficulty is uncertain. Each question is therefore pulled toward its benchmark's average until the answers show otherwise (partial pooling).
- Each benchmark's share of the fit is divided by its number of questions, so a 2,000-question benchmark weighs no more than a 30-question one.
Reading the numbers. The scale has no natural zero; only differences matter. Two anchors fix it: GPT-6 Astra (max) at 150 and Qwen3.5-9B (thinking) at 100. Adding models leaves the anchors in place, so values stay comparable across updates. The bars are 90% intervals from refits on resampled questions, resampled within each benchmark. As a check, fitting one score per benchmark instead of every question (as ECI does with published scores) gives nearly the same Index; the largest difference is points.
The fit is sem site trends in simple-evals-mm (src/simple_evals_mm/capability.py).
