Current AI models imitate the steps of human reasoning, but published tests do not show that they reason the way people do. Prompting a 540-billion-parameter model to write out intermediate steps raised its solve rate on grade-school math problems from 18 percent to 57 percent in 2022, yet later studies found that the written steps can misstate what drove the answer and that accuracy can collapse when puzzles get harder.
Key takeaways
- Chain-of-thought prompting, described in a January 2022 paper, improved results on arithmetic, commonsense, and symbolic tasks, and its authors left open whether the network is actually reasoning.
- A 2023 study found that written explanations can leave out the real cause of an answer, with accuracy falling by as much as 36 percent when inputs were biased.
- A 2025 study of reasoning models reported complete accuracy collapse beyond certain puzzle complexities.
- The ARC-AGI benchmark is built from tasks that people solve easily and that AI systems found hard.
- A fluent explanation from a model is evidence of fluent text, not proof of the process behind it.
What chain-of-thought prompting changed
The phrase "AI reasoning" in current research mostly refers to chain-of-thought prompting. In a paper first posted on arXiv on January 28, 2022 and presented at NeurIPS 2022, Jason Wei and colleagues showed that adding a few worked examples with intermediate steps to a prompt improved large language models on arithmetic, commonsense, and symbolic tasks. On GSM8K, a set of math word problems, the PaLM 540B model scored 18 percent with standard prompting and 57 percent with eight step-by-step examples. The previous best published result was 55 percent, and a fine-tuned GPT-3 with a verifier scored 33 percent.
The authors were direct about the limits. They wrote that the method emulates the thought process of human reasoners but does not answer whether the network is actually reasoning, and they noted there is no guarantee that a written path is correct. The gain also appeared with very large models, so it described one family of systems at one point in time.
When the written steps do not match the cause
A human who explains a decision may be wrong about their own motives, and models show a similar gap. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman tested this in a paper posted in May 2023 and published at NeurIPS 2023. They added a biasing feature to multiple-choice questions, such as reordering the options so the correct answer was always "(A)". Model answers shifted strongly toward the bias, yet the step-by-step explanations rarely mentioned it, and when the bias pointed to a wrong answer the models often produced a plausible justification for it. Across 13 tasks from the BIG-Bench Hard collection, tested on GPT-3.5 and Claude 1.0, accuracy fell by as much as 36 percent.
The finding concerns models from 2023. It does not say every later system behaves the same way, but it shows why a readable explanation cannot be treated as a record of the computation.
A ceiling on harder problems
Parshin Shojaee, Iman Mirzadeh, and co-authors studied reasoning-tuned models in a paper submitted on June 7, 2025 and titled "The Illusion of Thinking." They used puzzle environments where difficulty can be raised in controlled steps while the logic stays the same. The abstract reports three regimes: standard models did better on low-complexity tasks, reasoning models did better on medium-complexity tasks, and both collapsed on high-complexity tasks. Reasoning effort rose with difficulty up to a point and then fell even though token budget remained, and the models did not reliably use explicit algorithms.
These results come from specific puzzles, and the abstract gives no numeric thresholds. The paper is best read as one controlled probe rather than a final verdict.
Tasks that people solve and models struggled with
François Chollet introduced the Abstraction and Reasoning Corpus in 2019 together with a definition of intelligence as skill-acquisition efficiency on unknown tasks. ARC-AGI puzzles rely on core knowledge priors rather than learned cultural knowledge, and the stated design principle is that they are easy for humans and hard for AI. The ARC Prize site reports that OpenAI's o3-preview scored 75 percent at low compute and 87 percent at higher compute on ARC-AGI-1 in December 2024. For the harder ARC-AGI-2, the organizers tested more than 400 members of the public in San Diego in early 2025 and report that every task was solved by at least two people in two attempts or fewer.
Four studies side by side
| Study | What was tested | Reported result | Main limit |
|---|---|---|---|
| Wei et al., 2022 | PaLM 540B on GSM8K | 18 percent with standard prompting, 57 percent with chain of thought | Authors leave open whether the model reasons |
| Turpin et al., 2023 | 13 BIG-Bench Hard tasks, GPT-3.5 and Claude 1.0 | Accuracy down by up to 36 percent under biased inputs; bias rarely named | Older models |
| Shojaee et al., 2025 | Controlled puzzles of rising difficulty | Accuracy collapse beyond certain complexities | Specific puzzle types |
| ARC Prize, 2024 to 2025 | Grid puzzles built for fluid intelligence | Human panel solved all ARC-AGI-2 tasks; AI scores vary by compute | One benchmark family |
Questions to ask about a claim that AI reasons like a person
- Which task was tested, and could the answer appear in training data?
- Was accuracy measured only on final answers, or were the written steps also checked?
- Does the result hold when the problem is made harder or the wording is changed?
- Which humans served as the comparison, how many, and under what conditions?
- How much computing was spent per task, and does the claim state it?
The old phrase "deep human reasoning" has no agreed measurement. Reports that blend human judgment with model output, for example a clinician reviewing a draft, are studies of teamwork and need their own evidence.
FAQ
Does chain-of-thought prompting prove that an AI model reasons?
No. The authors of the original paper said it does not answer whether the model is reasoning. It shows that producing intermediate steps improves scores on certain benchmarks.
Can an AI explanation be trusted as the real reason for an answer?
Not automatically. In the 2023 bias experiments, the explanations seldom mentioned the feature that changed the answer, so a clear explanation can still be incomplete.
Why do researchers use puzzles like ARC-AGI?
Such tasks are designed to avoid reliance on memorized knowledge and to be easy for people, which makes failures informative. A high score on one benchmark does not establish general ability.
Are these findings still current?
They are dated measurements. Models change quickly, and new tests can overturn old limits, so each result should be read with its year attached.
Related reading
- How deep learning works and where it is applied
- Natural and artificial intelligence compared
- What vectors do inside AI models
- Limits of AI emotion recognition
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- Wei J. et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022 (arXiv:2201.11903)
- Turpin M. et al., Language Models Don't Always Say What They Think, NeurIPS 2023 (arXiv:2305.04388)
- Shojaee P. et al., The Illusion of Thinking, 2025 (arXiv:2506.06941)
- ARC-AGI overview and ARC-AGI-1 and ARC-AGI-2 pages, ARC Prize Foundation, 2026
- Chollet F., On the Measure of Intelligence, 2019