Methodology

Scope

Open-weight language models with under 100B total parameters. Mixture-of-Experts models count by total parameters, not active ones, because total size decides the memory you need to serve them. Models are grouped intoTiny (< 5B), Small (5–15B), Medium (15–40B), Large (40–100B).

Regions and openness

• Region is where the lab is based: United States, China, Europe, Rest of Asia, Other. "Rest of Asia" covers Korea, Japan, India and others; "Other" covers labs in Canada and the Middle East, and labs that don't publish a location.
• Fully open means the training data and code are public too, not just the weights (OLMo, Apertus, EuroLLM, SmolLM, K2, Baguettotron).
• Fine-tunes are models built on another lab's weights (for example Edge0 35B on Qwen3.6, Hermes 4.3 on Seed-OSS). The "Hide fine-tunes" filter removes them.

The Small LM Index

The index is the average score a model is expected to get on six core benchmarks, on a 0–100 scale:

• MMLU-Pro (Knowledge): Harder, 10-option version of MMLU across 14 subjects.
• GPQA Diamond (Knowledge): Graduate-level, Google-proof science questions.
• Humanity's Last Exam (Reasoning): Frontier-difficulty expert questions (no tools).
• AIME (Math): American Invitational Mathematics Examination (year noted per model).
• LiveCodeBench (Coding): Contamination-free competitive programming problems.
• SWE-bench Verified (Coding): Resolving real GitHub issues in an agent harness.

Missing benchmarks

Labs report different benchmarks, and they tend to report the ones they do well on. A plain average of whatever was reported would reward that: a model that publishes only three hard benchmarks it tops would beat a model that publishes all six. So the index doesn't average reported scores. It estimates the scores that are missing.

1. Every reported score, on all 18 benchmarks we track, is treated as a noisy reading of one underlying ability per model. A statistical model (a one-factor model on the logit scale, similar to item response theory) learns how hard each benchmark is, how strongly it tracks that ability, and how noisy it is.
2. Each model's ability is estimated from the scores it reports. Models with few scores are pulled toward the average until the evidence says otherwise, so publishing less never helps.
3. The index is the ability converted back into the expected average on the six core benchmarks. An estimate never exceeds the best score anyone has reported on that benchmark, because predicting past the strongest models isn't reliable.
4. A model needs at least 2 core benchmarks to get a score. With fewer, the table shows "—".

How accurate are the estimates? We hide each reported core score in turn and predict it from the model's other scores. The average error is about 5–6 points, against about 13 points for just using the benchmark's average across models. Coding-agent scores (SWE-bench) are the hardest to predict, at about 10 points. Runnpm run validate:index to reproduce this.

Every index comes with a likely range (about 95%). It is narrow for models that report all six core benchmarks and wide for those that report two. The leaderboard shows how many core benchmarks were reported (e.g. "3/6"), and model pages list the estimated scores, marked as estimates.

Because the scale is fitted on all models together, adding a model can move other scores slightly, usually by well under a point. Rankings are what to compare, not single decimals.

Supplementary benchmarks, which help estimate ability but are not part of the six-benchmark average:Terminal-Bench, τ²-Bench, IFBench, MMMU-Pro, MMLU, Multilingual MMLU, IFEval, BFCL, MATH-500, HMMT, AA-LCR, SciCode.

Where the numbers come from

• Benchmarks come from official model cards and launch posts, which means mostly vendor-reported numbers.
• Who measured it. Every score is tagged with its source. Most are self-reported by the lab that made the model. Some come from another lab, usually a rival's comparison table. A few are independently measured by a neutral evaluator. In the leaderboard, ✓ marks independent scores and * marks another lab's; the "Independently measured" filter keeps only models with at least one independent score.
• Independent scores win. When a benchmark has been measured independently, that number replaces the lab's in the table, the charts and the index, and the model page still shows what the lab reported. Two independent sources, in order of preference: Tiny Analysis evals, our own runs of every model on one harness with the same settings (first run pending), andArtificial Analysis, imported through their free API (first sync pending). Treat models whose scores are all self-reported with extra care.
• Labels on every score. Model pages tag each score:Lab-reported (the model's own lab),Other lab (a rival's comparison table, often run without the model's best settings, so usually on the low side), andIndependent (a neutral evaluator, named on hover). Scores in a dashed outline under "Not reported: estimated" are the index model's predictions, not measurements.
• Earlier versions. When a new release has no published results but its previous version does (for example Apertus 1.5 70B and the first Apertus 70B), the model page shows those scores underEarlier version. They are context only: they never count toward the index and never appear in the leaderboard or charts as the new model's scores.
• Checked dates. Each model page shows when its entry was last checked against the sources listed at the bottom of the page.
• Variants differ. AIME may be the 2024, 2025 or 2026 exam, and LiveCodeBench may be v5 or v6, depending on when the model shipped. Hover a score in the table or see the model page for the exact variant.
• Prices are synced from the public OpenRouter models API (last sync 2026-11-03). Blended price = (3 × input + output) / 4.
• Hosted API providers counts the distinct providers serving the model on OpenRouter.
• Downloads and likes come from the Hugging Face API (last sync 2026-11-03). Downloads are a rolling 30-day count for the main repository only; quantized copies (GGUF, MLX) uploaded by others are not included, so popular local-use models are undercounted.
• Speed (output tokens per second and time to first token) will be measured by streaming a fixed prompt through OpenRouter's fastest provider. The speed axes appear once the first measurement run has finished. Where we have no measurement, Artificial Analysis's median across providers fills the gap; each model page names the source.
• Memory and speed for running locally add up the weights (parameters × bits per weight, e.g. 4.85 bits for Q4_K_M), the context cache for the chosen context length, and about 0.5 GB plus 2% of the weights for runtime buffers. The context cache uses each model's real attention layout from its config.json (hybrid and sliding-window models need far less), synced from Hugging Face (2026-11-03). Speed assumes generation is limited by memory bandwidth, with typical efficiency per device and a penalty for Mixture-of-Experts layers; treat it as a rough guide. Try it in Can I run it?
• 49 entries are flagged Needs verification: some figures came from secondary sources or memory and still need checking against the official card.

Limitations

• Vendor-reported scores use each lab's own prompts, harnesses and reasoning budgets. They are not a controlled head-to-head. Independent re-measurement covers only some models so far.
• Coding-agent results (SWE-bench, Terminal-Bench) depend heavily on the scaffold.
• Benchmarks can be contaminated, so treat gaps of 1–2 points as noise.

Corrections

Every figure is checked against a source before it goes in. To report a wrong score or ask for a model to be added, use the correction form and include a link to the source.