Research does not support naming one best AI image generator: in the Holistic Evaluation of Text-to-Image Models (HEIM) benchmark, published at NeurIPS 2023, 26 models were tested across 12 aspects and no single model excelled in all of them. Image quality is also hard to score with a formula, since the long-standing automatic metric, Fréchet Inception Distance (FID), has been shown in later research to contradict human raters. Any claim that one tool is "the ultimate" generator therefore depends on which aspect was measured and who judged it.
Key takeaways
- HEIM evaluated 26 text-to-image models on 12 aspects across 62 scenarios and found different models strong in different aspects.
- FID, introduced in 2017, compares statistics of generated and real images and became the default quality number.
- A 2023 preprint reports that FID contradicts human raters and proposes a replacement called CMMD.
- Aspects beyond appearance, such as bias, toxicity, originality, and efficiency, are part of a fair comparison.
- Results describe the models and prompts tested at a given date, and versions change often.
What "high quality" has to mean before it can be measured
An image can be sharp and still wrong. It may ignore half of the prompt, draw the wrong number of objects, or reproduce a stereotype. Early evaluation of image generators concentrated on two things: how realistic the pictures look, and how well they match the text prompt. HEIM's authors argue that this leaves out much of what matters and widen the scope to include aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilingual prompts, and efficiency.
The practical lesson is that "quality" is a bundle. A person choosing a tool to draw a logo, a classroom diagram, or a product photo is weighting these aspects differently, and a single score hides the weighting.
FID: the number most papers report
The Fréchet Inception Distance was introduced in a 2017 paper by Martin Heusel and colleagues, presented at the NIPS conference. It compares the statistical distribution of features extracted from generated images with the same features from real images; a lower distance is read as "more like real photos." The authors reported that it captured similarity better than the earlier Inception Score. The metric was built when the main task was generating images without text prompts, and it spread because it was cheap to compute and easy to compare across papers.
Why researchers now question FID
In the paper Rethinking FID (arXiv, submitted November 30, 2023, revised January 2024), Sadeep Jayasumana and co-authors list several weaknesses. The features come from an older network, Inception-v3, which they say cannot represent the variety of content in modern text-to-image output. The metric assumes the feature distribution is normal, which they argue does not hold. It also needs many samples to be stable. In their experiments FID contradicted human raters, failed to track gradual improvements in iterative models, and gave inconsistent results when the sample size changed.
The authors propose CMMD, which uses CLIP embeddings and a statistical test called maximum mean discrepancy. The paper is a preprint on arXiv, so it has not been through the same review as a journal article, and CMMD is one proposal among several.
What a holistic benchmark adds
The HEIM paper, with Percy Liang and Li Fei-Fei among its authors, evaluates 26 models on 12 aspects using 62 scenarios. The authors released the generated images and the human ratings, and the code sits inside the HELM evaluation codebase, so other groups can rerun or extend it. The main finding, stated in the abstract, is that no model excels at everything and different models show different strengths.
This type of result is useful because it replaces a single winner with a profile. It also has limits: the 26 models were those available in 2023, and several have been updated or retired since.
Illustrative example: choosing criteria for a task
The table below is illustrative. It does not rate any product, and the weights are examples of how a reader might think about a task, not findings from either paper.
| Task | Aspects likely to matter most | What to check |
|---|---|---|
| Classroom illustration | Text-image alignment, knowledge, bias | Does the picture show the concept correctly and avoid stereotypes? |
| Marketing mock-up | Aesthetics, originality, robustness | Does it hold up across several rephrased prompts? |
| Multilingual project | Multilinguality | Are non-English prompts understood as well as English ones? |
| High-volume use | Efficiency, cost | How long does each image take, and at what price? |
Questions to ask before trusting a "best generator" claim
- Which aspects were measured, and which were left out?
- Who judged the images: people, a formula, or another AI model?
- How many prompts and samples were used, and are they published?
- Which model versions were tested, and on what date?
- Does the publisher of the ranking also sell one of the tools?
FAQ
Is there an objective score for AI image quality?
Not a single one. Automatic metrics such as FID are widely used, but later research reports that they can disagree with human raters. Benchmarks like HEIM combine several metrics with human ratings.
What does a lower FID mean?
A lower FID means the statistics of generated images are closer to those of a set of real images. It does not say whether an individual image fits a prompt or is free of errors.
Did HEIM rank the 26 models?
The abstract reports that no single model excelled in all aspects, so the result is a set of strengths per model, not one leaderboard.
Do these results still describe today's tools?
Only partly. The papers date from 2023 and 2024, and image generators are updated often. They remain useful for the method: which aspects to test and why one number is not enough.
Related reading
More in AI Basics.
Last verified: October 2026
Sources
- Lee T, Yasunaga M, Meng C, et al., Holistic Evaluation of Text-To-Image Models, NeurIPS 2023 (arXiv:2311.04287)
- Jayasumana S, Ramalingam S, Veit A, et al., Rethinking FID: Towards a Better Evaluation Metric for Image Generation, arXiv:2401.09603, 2023, revised 2024
- Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S, GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NIPS 2017 (arXiv:1706.08500)