Image hallucination is when an AI system describes an object that is not in the picture, and researchers measure it by checking a model's words against verified labels for the image. A 2018 study by Anna Rohrbach and colleagues introduced the CHAIR metric for captioning models, and a 2023 study by Yifan Li and colleagues introduced POPE, a yes-or-no test for large vision-language models. Both found that models score well on ordinary quality measures while still naming things that are absent.
Key takeaways
- CHAIR counts the share of mentioned objects that are not in the image (CHAIRi) and the share of sentences containing such an object (CHAIRs).
- In the 2018 study, models that scored best on standard caption metrics did not always hallucinate less.
- POPE asks simple questions such as "Is there a chair in the image?" and uses random, popular, and adversarial choices of absent objects.
- In POPE's 2023 tests, three of five models answered "Yes" to more than 95 percent of questions in every setting, and accuracy fell as the absent objects became more plausible.
- These results come from specific older models, so they show how the problem is measured, not how a current product performs.
What an image hallucination is
An image captioning system takes a photo and writes a sentence. A hallucination in this setting is a described object that is not there, such as a fork in a caption for a plate of broccoli. Rohrbach and her co-authors called the failure object hallucination and published the work at EMNLP 2018. They framed it as a relevance problem: a caption can read smoothly, match human reference sentences closely, and still mention something the picture does not contain.
Large vision-language models, which pair a language model with an image encoder, show the same behavior. Li and colleagues described hallucination in these systems as describing objects that do not match the target image, and their paper appeared at EMNLP 2023.
How CHAIR scores a caption
CHAIR stands for Caption Hallucination Assessment with Image Relevance. It restricts the check to the 80 object categories in the MSCOCO segmentation challenge. A caption's words are reduced to base forms and matched through a synonym list, so "player" maps to "person." Ground truth combines two sources, the segmentation labels and the human reference captions, because each alone gives a distorted answer: annotators call many things a "table" that the segmentation labels do not count, and people often leave real objects unmentioned.
The two scores are simple fractions. CHAIRi is hallucinated object mentions divided by all object mentions. CHAIRs is the number of sentences with a hallucinated object divided by all sentences. Lower is better. On the paper's harder test split, where object pairs in test captions do not overlap with those in training, a standard TopDown model's CHAIRs rose from 8.4 percent to 11.3 percent of sentences, a drop the authors connect to over-reliance on word patterns learned from text.
What the 2018 study found about causes
The authors compared architectures and training goals on MSCOCO. Models that used bounding-box attention, which ties words to detected regions, hallucinated less than models working from coarse feature maps. Training with a self-critical loss that rewards the CIDEr score often increased hallucination, which the authors suggested is because CIDEr does not penalize it enough. They also found that models that hallucinate more tend to make mistakes a language model alone would make: the word "fork" follows "plate of food" in text, whether or not a fork is pictured.
The same paper reports that the best scorers on standard metrics were not always the least prone to hallucination. That is the reason a separate measure was needed.
POPE: asking yes-or-no questions instead
Li and colleagues found that checking free-form captions depends on parsing rules, caption length, and prompt wording. POPE turns the problem into a binary question per object. Present objects are asked about directly. Absent objects are chosen in three ways: random, popular (the most frequent objects in the dataset that are not in the image), and adversarial (objects that most often appear together with the objects that are in the image). The authors drew 500 MSCOCO validation images with more than three labeled objects and wrote six questions per image.
The results show two patterns. First, accuracy dropped as the absent objects became more plausible. InstructBLIP's F1 score went from 89.29 under random sampling to 83.45 under popular and 78.45 under adversarial. Second, some models said "Yes" almost regardless of the picture. LLaVA answered "Yes" to 95.37, 97.37, and 99.10 percent of questions across the three settings, and MultiModal-GPT answered "Yes" to between 99.97 and 100 percent, with accuracy at about 50 percent, the level of a coin flip on a balanced test.
Two measures side by side
| Feature | CHAIR (2018) | POPE (2023) |
|---|---|---|
| What is checked | Objects named in a generated caption | Yes or no answers about listed objects |
| Ground truth | MSCOCO segmentation labels plus reference captions | MSCOCO object annotations |
| Limitation | Depends on object lists and hand-built synonym rules | Covers only the objects chosen for the questions |
| Metric | CHAIRi and CHAIRs, lower is better | Accuracy, precision, recall, F1, and share of "Yes" |
| Models tested | Captioning models such as FC, Att2In, TopDown, NBT | Five vision-language models, including LLaVA and InstructBLIP |
Why it matters beyond the lab
Captions and image descriptions are often used as text alternatives for people who cannot see an image. The Web Content Accessibility Guidelines 2.2, success criterion 1.1.1, require non-text content to have a text alternative that serves the equivalent purpose. By that wording, an automatic description that invents an object does not serve the purpose, although the criterion itself does not mention AI. This is a plain reading, not a legal finding.
The limits of the evidence are clear. The 2018 and 2023 studies tested everyday objects in one photo collection with models available then. They do not rate a current product, a medical image tool, or a text-only chatbot. This article gives general information and is not legal advice about accessibility duties, which differ by country, province, state, and sector.
Checks before relying on an AI description of an image
- Does the description name objects that are hard to see or only commonly found in similar scenes?
- Can the same question be asked a second time in yes-or-no form, and does the answer stay the same?
- Is a person reviewing descriptions used in public content, such as a web page or a record?
- Does the tool publish a hallucination test result on a named benchmark, and which version of the model was tested?
FAQ
Is image hallucination the same as a text hallucination?
They are related. In both, a model produces content unsupported by its input or by fact. In the image case, the input is a picture and the unsupported content is usually an object or detail that is absent from it.
Does a higher caption quality score mean fewer hallucinations?
Not necessarily. The 2018 study reports that models scoring best on standard caption metrics did not always hallucinate less, which is why CHAIR was proposed as a separate check.
What does a very high "Yes" rate in POPE mean?
It means the model agreed that nearly every asked object was present. With half the questions about absent objects, answering "Yes" every time leaves accuracy near 50 percent.
Do newer models still hallucinate objects?
The sources here do not test current models, so they cannot say. A 2024 revision of a large survey on hallucination lists hallucination in vision-language models among open research directions.
Related reading
- Hallucinations and their implications
- How computer vision works and where it fails
- Deep learning applications and limits
- Ethical questions in AI-generated art
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- Rohrbach A, Hendricks LA, Burns K, Darrell T, Saenko K. Object Hallucination in Image Captioning, EMNLP 2018
- Li Y, Du Y, Zhou K, Wang J, Zhao WX, Wen J-R. Evaluating Object Hallucination in Large Vision-Language Models, EMNLP 2023
- Huang L et al. A Survey on Hallucination in Large Language Models, ACM Transactions on Information Systems, arXiv version revised November 19, 2024
- Web Content Accessibility Guidelines 2.2, success criterion 1.1.1 Non-text Content, World Wide Web Consortium