Comparing AI Systems: What Benchmarks and Leaderboards Show

No AI system is best in general. How HELM tests and Chatbot Arena votes compare models, and where memorization and uneven data skew rankings.

Guide: AI BasicsLast verified: October 20266 min read

No AI system is superior in general, because every comparison depends on the task, the test, and the measure used. Two public methods dominate: fixed test sets such as those in Stanford's Holistic Evaluation of Language Models (HELM), which scored 30 models on the same 42 scenarios, and preference voting such as Chatbot Arena, which ranks models from pairwise human votes. Each method has documented weak spots, so a ranking is best read as evidence about one test, not a verdict on a product.

Key takeaways

  • HELM raised the share of core scenarios on which models were measured from an average of 17.9 percent to 96.0 percent by running all 30 models on identical tests.
  • Chatbot Arena converts human votes between two anonymous answers into a ranking, and its founding paper reported over 240,000 votes.
  • A 2025 study estimated that two developers received 19.2 and 20.4 percent of all Arena data, which can tilt results toward models tuned for that platform.
  • A 2024 test on freshly written grade-school math problems found accuracy drops of up to 8 percent for some model families, a sign of partial memorization.
  • A ranking is only useful for a decision if its tasks, scoring, and date resemble the intended use.

Why one winner does not exist

Choosing between AI systems resembles choosing between vehicles. A delivery van and a sports car are both vehicles, yet "better" has no meaning until the job is named. The same holds for language models, image generators, and forecasting systems. A system that writes fluent summaries may do poorly at arithmetic, one that scores well in English may stumble in French, and one that is accurate may be slow or expensive to run.

Researchers therefore compare systems on defined tasks and publish the method. The two approaches below are the best documented, and each answers a different question.

Fixed test sets: what HELM changed

Before HELM, models were often reported on whichever benchmarks their builders chose, which made side-by-side reading difficult. The Stanford Center for Research on Foundation Models paper, first submitted on November 16, 2022, counted this gap: on average, models had been evaluated on only 17.9 percent of the core scenarios, and some prominent models shared no scenarios at all. HELM defined 16 core scenarios, measured each on seven things (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency), and benchmarked 30 models on the same set, lifting coverage to 96.0 percent. A further 26 targeted scenarios probed areas such as reasoning and disinformation.

The lesson is structural. Accuracy was only one of seven columns, so a model that tops one column can sit mid-table on another. The HELM paper on arXiv reports 25 top-level findings, and none of them is "model X wins."

Preference voting: how Arena rankings are built

Chatbot Arena takes the opposite route. A visitor types any prompt, receives answers from two anonymous models, and votes for the better one. The founding paper, posted on March 7, 2024 by Wei-Lin Chiang and colleagues, reported over 240,000 votes and applied statistical methods to turn pairwise outcomes into a ranking. The authors also checked that crowd questions were diverse and that crowd votes agreed with expert raters.

Voting captures what people prefer, including tone, length, and formatting, which a multiple-choice test cannot. That is also its limit. A preferred answer is not necessarily a correct one, and the prompts come from whoever visits the site.

Where leaderboards go wrong

A 2025 paper titled The Leaderboard Illusion, by Shivalika Singh and 12 co-authors, examined Arena's data. It identified 27 private model variants that one developer tested before a release, estimated that Google and OpenAI received 19.2 and 20.4 percent of all Arena data, and estimated that 83 open-weight models together received 29.7 percent. The authors calculated that limited extra data can bring relative gains of up to 112 percent on the Arena distribution, a gain in the platform's own test that need not carry over to other uses.

Fixed test sets have a different problem: leakage. If benchmark questions appear in training data, a model may recall answers instead of solving problems. In 2024, Hugh Zhang and colleagues wrote GSM1k, a new set of grade-school math problems matched to the popular GSM8k set. Several model families dropped by up to 8 percent, and the researchers linked a model's likelihood of reproducing GSM8k examples to its score gap, which suggests partial memorization. The same abstract notes that many frontier models showed minimal overfitting, so the finding applies to some models, not all.

Comparison methods side by side

MethodWhat it measuresMain limit
HELM-style fixed scenariosAccuracy plus calibration, robustness, fairness, bias, toxicity, and efficiency on shared tasksScenarios freeze at a point in time, and models can be tuned to them
Arena-style preference votesWhich answer visitors prefer in a blind comparisonPrompts are self-selected, and access to testing data was uneven in 2025
Fresh held-out sets such as GSM1kWhether a score holds on unseen problemsCovers one skill, and the set must stay unpublished to stay useful
Own trial on real workFit with a specific task, data, and budgetTakes time and needs a clear scoring rule

A short checklist before trusting any AI ranking

  • Which tasks were tested, and do they resemble the planned use?
  • What was the date, and which model version was measured?
  • Who ran the test, and did the developer of a listed model have a role in it?
  • Is accuracy the only column, or are cost, speed, and error behavior reported?
  • Could the test questions have appeared in training data?
  • Does a small trial on real examples agree with the published ranking?

The last question matters most. A school district comparing writing assistants, or a clinic comparing transcription tools, gets a better answer from a blind trial on its own material, scored by the staff who will use the output, than from any public table.

FAQ

Which AI model is the best right now?

No source can answer that without a task. Public rankings change as new versions are released, so any list older than a few months is stale. A ranking is informative only for the tasks it tested.

Are leaderboard scores reliable?

They are reliable for what they measure and unreliable as a general verdict. Research from 2024 and 2025 documents memorization on fixed benchmarks and uneven data access on a voting platform.

What does HELM measure?

HELM measures language models on shared scenarios across seven categories: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The 2022 paper evaluated 30 models.

How can a non-expert compare two AI tools?

Write 20 to 50 real examples from the intended work, run both tools on them, hide the tool names from the reviewers, and score the outputs against a rubric agreed in advance.

Last verified: October 2026

Sources

  • Holistic Evaluation of Language Models, Liang, Bommasani et al., Stanford CRFM, Transactions on Machine Learning Research, 2023 (arXiv:2211.09110)
  • Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, Chiang et al., 2024 (arXiv:2403.04132)
  • The Leaderboard Illusion, Singh et al., 2025 (arXiv:2504.20879)
  • A Careful Examination of Large Language Model Performance on Grade School Arithmetic, Zhang et al., NeurIPS Datasets and Benchmarks Track, 2024 (arXiv:2405.00332)

New to artificial intelligence?

Start with the basics, then follow the guides to tools, rules, and effects on society.

Start with AI Basics