Language models can solve multi-step math word problems and, in at least one case, five of six problems at the International Mathematical Olympiad, yet their accuracy can drop sharply when a question is reworded or padded with a harmless extra sentence. Researchers disagree about whether that pattern shows reasoning that is fragile or pattern matching that only resembles reasoning. The most informative tests are the ones that change a problem without changing its logic.
Key takeaways
- A 2024 study built 5,000 variants of grade-school math problems and found that swapping the numbers moved model scores more than swapping the names.
- Adding one irrelevant sentence lowered accuracy across the models in the study's main table, by up to 65 percent according to the abstract.
- On July 21, 2025, Google DeepMind announced that an advanced Gemini model scored 35 of 42 points at the 2025 International Mathematical Olympiad, with the answers graded by the competition's coordinators.
- Both results can be true at once, because a model can handle hard, well-posed problems and still be unstable under small changes.
- A single score is weak evidence; the useful question is how a score changes when only surface details change.
What counts as reasoning in a test
In everyday use, reasoning means reaching a conclusion by steps that follow from stated facts, and doing it again correctly when the details change. A student who understands that a total is a price multiplied by a count can solve the problem whether the item is apples or pencils and whether the price is 4 or 7. That second part, the stability under change, is what separates understanding a method from remembering a worked example.
This is why researchers test models on problems with the same logical structure but different surface content. If a model has learned the method, its score should barely move. If it has mostly matched familiar patterns, the score should wobble. The two studies below are built around that idea, one from the failure side and one from the success side.
The GSM-Symbolic experiment
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar posted GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models on arXiv on October 7, 2024, and revised it on August 27, 2025. They took 100 questions from GSM8K, a widely used set of 1,319 grade-school math test problems, and turned each into a template with blanks for names and numbers. From each template they generated 50 versions, which gives 5,000 questions. They then tested 25 models, from small open models to GPT-4o, o1-mini, and o1-preview.
Two findings stand out. First, scores varied across versions of the same question, and the variation was smaller when only names changed than when numbers changed. Second, the authors added one clause that sounds relevant but does not change the answer. The abstract reports drops of up to 65 percent across state-of-the-art models. In the paper's table, one small open model fell from 80.7 percent to 18.0 percent, GPT-4o from 94.9 to 63.1, and o1-preview from 92.7 to 77.4.
The authors concluded that model reasoning is fragile and may be closer to sophisticated pattern matching than to logical reasoning. That is an interpretation of the data, and the models tested were the ones available in 2024.
The Olympiad result
The opposite kind of evidence came from the 2025 International Mathematical Olympiad. Google DeepMind announced on July 21, 2025 that an advanced version of Gemini with Deep Think had solved five of the six problems for 35 of 42 points, working in natural language within the 4.5-hour limit. The competition's coordinators graded the answers using the same criteria applied to student work. The year before, a Google DeepMind system had needed problems translated into the formal proof language Lean and had taken days of computation.
The competition's president, Gregor Dolinar, said the solutions were astonishing in many respects. The announcement also states that the review confirmed the submitted answers, not the system or process behind them. The result shows that models can solve problems that stump most contest-trained students. It does not show how they get there, and it does not settle whether they would hold up if the problems were rephrased.
How the two kinds of evidence fit together
| Test | What was held constant | What changed | What it can support |
|---|---|---|---|
| GSM-Symbolic, 2024 | The logic of 100 problems | Names, numbers, and one added clause | Scores are sensitive to surface details |
| Olympiad, 2025 | Official grading rules | Problem difficulty far above grade school | Models can solve very hard, well-posed problems |
The two rows do not contradict each other. They measure different things on different models a year apart. A system tuned for long, careful solution attempts may handle a hard proof problem and still be thrown off by an irrelevant sentence in a word problem, because training that helps one does not guarantee the other. That is a hypothesis, not a finding of either study.
Questions to ask about any reasoning claim
- Was the same problem tested in more than one wording, or only once?
- Could the question or its answer have been in the training data?
- Did a person or an independent body check the answers, or did the developer grade its own work?
- Does the score come from one attempt or from the best of many?
- Which model version was tested, and on what date?
FAQ
Does a wrong answer after a rewording prove the model cannot reason?
No. It shows the model was not stable under that change. People also make errors when a problem includes distracting details, so the useful comparison is how often and how badly each group is thrown off.
Does winning an Olympiad medal mean a model reasons like a mathematician?
Not by itself. The graded result confirms correct answers on six specific problems. It does not describe the internal process, and the competition's own statement limited its review to the submitted answers.
Are the 2024 results out of date?
The tested models are older, and later systems may behave differently. The paper was revised in August 2025, but its main table still reports the models studied in 2024, so the figures describe a dated snapshot.
Why do researchers disagree?
The word reasoning has no single test. Some researchers treat stable, correct multi-step answers as sufficient, while others require evidence about how the answer was reached.
Related reading
- Whether AI can think for itself
- Could AI outsmart people
- AI evolving beyond human capability
- Machine consciousness and sentience
More in AI Policy and Ethics.
Last verified: October 2026
Sources
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, Mirzadeh et al., arXiv 2410.05229, 2024, revised 2025
- Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad, Google DeepMind, July 21, 2025