Whether AI calculates accurately depends on how the answer is produced. A calculator or a Python program applies fixed rules and returns exact results within its number format, while a chatbot built on a language model writes digits the way it writes words, so long multiplications and chains of steps can come out wrong. Accuracy rises sharply when the model passes the arithmetic to a calculator or a code interpreter instead of doing it in text.
Key takeaways
- In a May 2023 preprint, ChatGPT and GPT-4 multiplied two three-digit numbers correctly only 55 percent and 59 percent of the time, and accuracy fell toward zero as the problems got larger.
- Anthropic's March 27, 2025 interpretability research found that its model adds numbers through parallel internal paths, then describes the textbook method of carrying the one when asked how it did it.
- Toolformer (February 2023) and PAL (November 2022) are research systems that hand arithmetic to a calculator or a Python interpreter.
- Exact tools have limits too: in Python, 0.1 + 0.1 + 0.1 does not equal 0.3 because of binary floating-point storage.
- The numbers above describe models from 2022 and 2023, so they show a pattern in how errors arise, not the current accuracy of any product.
Two kinds of machines that both get called AI
A pocket calculator runs a small circuit that adds, multiplies, and divides digits by rule. Every time it receives the same input, it returns the same output. A large language model is a different kind of system. It reads text, breaks it into pieces called tokens, and predicts the next token from patterns learned in training. When such a model is asked for 4,317 times 628, it is generating digits one after another, and each digit is a prediction, not a step in a long multiplication.
The model can still be right, because short sums appear constantly in training text and the patterns are strong. Trouble appears as the number of digits grows and as one problem needs many connected steps, since an error early in the chain carries into everything after it.
What a 2023 test of multiplication showed
Nouha Dziri and 15 co-authors posted "Faith and Fate: Limits of Transformers on Compositionality" on arXiv on May 29, 2023. The paper tested GPT-3, ChatGPT, and GPT-4 on multi-digit multiplication, logic grid puzzles, and a dynamic programming task. For three-digit by three-digit multiplication, it reports that ChatGPT and GPT-4 reached only 55 percent and 59 percent accuracy. When prompted without worked steps or with a few examples, accuracy dropped to near zero as problems grew. Letting the model write out intermediate steps helped, but that gain shrank toward zero at higher complexity.
The authors also fine-tuned GPT-3 on a range of smaller multiplications. It became nearly perfect on problems like those in training and fell sharply on larger ones it had not seen. Their conclusion is that these models handle multi-step problems by matching pieces of computations they have seen, not by carrying out a general procedure. That is the authors' interpretation of their results, and other researchers read the evidence differently.
The model that explains a method it did not use
Anthropic published research on March 27, 2025, tracing how its Claude model adds two numbers such as 36 and 59. The company reports that the model uses several paths at once: one computes a rough approximation of the sum, and another handles the last digit. The paths then combine into the answer, 95. Asked how it got that result, the model described the standard method of carrying the one. The researchers suggest that the model learned to explain arithmetic by imitating human explanations while learning to compute by its own internal route.
This matters for checking work. A chatbot's written explanation of a calculation is not a record of how the number was reached, so a tidy explanation next to a wrong answer is not evidence that the answer is right.
Handing the arithmetic to a tool
A different line of research avoids the problem by not asking the model to calculate at all. Toolformer, posted by Timo Schick and colleagues on February 9, 2023, trained a language model to decide when to call outside tools, including a calculator, a search engine, and a translation system. The abstract reports substantially improved zero-shot performance on a range of tasks and says arithmetic is among the basic tasks where plain language models fail.
PAL, posted by Luyu Gao and colleagues on November 18, 2022, takes another route. The language model reads a word problem and writes a short program as its reasoning, and a Python interpreter runs it. On the GSM8K set of grade-school math word problems, PAL with the Codex model beat chain-of-thought prompting on the 540-billion-parameter PaLM model by an absolute 15 percent, according to the paper. The model still has to set up the problem correctly. Only the arithmetic is made exact.
Exact does not mean error-free
Python's documentation gives a plain example of how even exact machinery has rules. Most decimal fractions cannot be stored exactly in binary, so the value held for 0.1 is slightly larger than one tenth, and 0.1 + 0.1 + 0.1 == 0.3 evaluates to False. The same documentation points to the decimal module for work that needs exact decimal arithmetic, such as accounting. A program that calculates correctly under its own rules can still surprise a reader who expects decimal behavior.
Where each approach stands
The table summarizes the approaches described above. It is a simplified orientation, not a ranking.
| Approach | How the answer is produced | Typical weakness |
|---|---|---|
| Calculator or spreadsheet | Fixed rules applied to digits | Rounding, number-format limits, wrong input |
| Chatbot answering in text | Digits predicted token by token | Errors grow with digits and steps |
| Chatbot with step-by-step text | Intermediate lines written out, then combined | Gain shrinks on larger problems; one wrong line spoils the rest |
| Model plus calculator or code | Model sets up the problem, a tool computes | Problem may be set up incorrectly |
A short routine for checking an AI's arithmetic
- Recompute any number that will be used in a decision with a calculator or spreadsheet.
- Estimate the size of the answer first; a result with the wrong number of digits is wrong.
- Check the last digit by hand, since it is quick and catches many slips.
- If the product offers a code or calculation tool, ask for the code and read it, not just the number.
- Do not treat a fluent explanation as proof of the result.
FAQ
Can ChatGPT do math correctly?
It can be right on short problems and often is. The 2023 test above found it correct on only 55 percent of three-digit multiplications, and current products differ by model and by whether they use a calculation tool. Anything important should be rechecked.
Why do chatbots make arithmetic mistakes?
They generate digits as text predictions, not by running the long-multiplication steps a calculator performs. Mistakes become more likely as the numbers lengthen and as more connected steps are needed.
Is a calculator a form of AI?
A basic calculator follows fixed rules and does not learn from data, so it is ordinarily not called AI. It is, however, the kind of exact tool that AI systems can be built to call.
Do tools fix the problem completely?
They remove arithmetic errors but not setup errors. The model still has to translate a word problem into the correct operations, and the tool still follows its own number-format rules.
Related reading
- How much math AI depends on
- Vectors and embeddings in AI
- How AI programs process large data sets
- The parts an AI system is built from
More in AI Basics.
Last verified: October 2026
Sources
- Faith and Fate: Limits of Transformers on Compositionality, Dziri et al., arXiv, 2023
- Floating-Point Arithmetic: Issues and Limitations, Python Software Foundation documentation
- Toolformer: Language Models Can Teach Themselves to Use Tools, Schick et al., arXiv, 2023
- PAL: Program-aided Language Models, Gao et al., arXiv, 2022
- Tracing the thoughts of a large language model, Anthropic research, March 27, 2025