A May 2025 study by researchers at Microsoft Research and Salesforce Research found that 15 large language models scored about 39 percent lower, on average, when a task was revealed across several conversational turns than when the same task arrived in one complete message. Most of the drop came from inconsistency: the same model could succeed in one run and fail in another on identical input. The finding explains why a chatbot that handles a full question well can still wander when a request is built up piece by piece.
Key takeaways
- The study tested 15 models on six tasks across more than 200,000 simulated conversations and reported an average 39 percent drop in multi-turn settings.
- Best-case ability fell by about 16 percent, while the gap between best and worst runs grew by about 112 percent.
- Models often guessed missing details early, then built on those guesses.
- Lowering the randomness setting did not fix the problem, and two reasoning models degraded in a similar way.
- The conversations were simulated, so the results describe a controlled test rather than every real chat. As of October 2026 the arXiv record still shows only the May 2025 version.
How the researchers tested a conversation
Most chatbot benchmarks give a model a complete, carefully worded instruction and score the reply. Real users rarely begin that way. They start with a loose request, add a constraint, correct a misunderstanding, and supply a number later. The paper, LLMs Get Lost In Multi-Turn Conversation (Laban, Hayashi, Zhou, and Neville, posted on arXiv on May 9, 2025), set out to measure what happens in that second situation.
The team took tasks that already had fully specified instructions and split each one into small pieces called shards. A second language model played the user and revealed at most one shard per turn. The six tasks were code generation, database queries, API-style actions, grade-school math, data-to-text writing, and summarization. A single-turn control that handed over every shard at once, in a bulleted list, scored 95.1 percent of the original single-turn result, so the drop cannot be blamed on the instructions being cut up.
What the numbers show
Averaged across the models, performance fell from roughly 90 percent in the single-turn setting to roughly 65 percent in the sharded setting, which is the 39 percent relative drop the abstract reports. The paper separates this into two measures. Aptitude is the 90th-percentile score across repeated simulations, a stand-in for what a model can do on a good run. Unreliability is the distance between the 90th and 10th percentile scores, a stand-in for how much results swing from run to run.
Aptitude fell by about 16 percent. Unreliability rose by about 112 percent. The authors conclude that the models did not lose most of their skill; they became much less predictable. A user who tries the same conversation twice may therefore see one good outcome and one poor one.
Why the models got lost
The paper names several behaviors seen in the transcripts, summarized below.
| Behavior | What it looks like in a chat |
|---|---|
| Premature answers | The model proposes a final solution after the first message, filling gaps with assumptions |
| Reliance on earlier attempts | Later replies patch the first answer instead of starting over, so errors accumulate and answers grow longer |
| Uneven attention to turns | The first and last messages weigh heavily, and details from the middle turns are dropped |
| Verbosity | Long replies add new assumptions that can pull the thread away from what the user said |
In the authors' words, once a model takes a wrong turn it tends not to recover. The pattern is a description of observed behavior in this test, not a claim about the internal mechanism.
What did not help
Setting the sampling temperature to zero, which makes output more deterministic, left unreliability near 30 percent for the models examined, and the gain for GPT-4o was modest. The two reasoning models in the set, OpenAI's o3 and DeepSeek-R1, degraded in much the same way as the others; the authors link this partly to replies that ran about 33 percent longer.
A prompting tactic that repeats information gave partial relief. In a "snowball" setup, each turn restated everything the user had said so far, and it recovered an estimated 15 to 20 percent of the gap between sharded and single-turn performance. Performance still stayed below the single-turn level.
Limits of the study
The users in the experiment were simulated by another model, and the tasks had checkable answers, such as code that either passes tests or does not. Open-ended conversations about advice, brainstorming, or opinions were not scored. The models tested were the ones available in early 2025, and releases from the following 18 months may behave differently; the paper does not speak to them, and no follow-up measurement from the authors was found when this page was checked in October 2026. The results are best read as evidence that a long, piecemeal exchange adds risk, not as a measurement of how often any particular chatbot fails in daily use.
A checklist drawn from the findings
- State the full task, with every constraint, in one message when the answer matters.
- When a thread has drifted, copy the settled requirements into a new chat and start again rather than patching the old one.
- Ask for questions before an answer if details are missing, since early guessing was a main failure pattern.
- Repeat the key facts periodically in a long session, which is the idea behind the snowball approach.
- Check outputs that depend on details from the middle of a long exchange.
FAQ
Does this mean chatbots cannot hold a conversation?
No. The study measured accuracy on tasks with checkable answers, and the models still performed well on single, complete requests. The result concerns reliability when information arrives gradually.
Which chatbots were tested?
The paper reports 15 models from eight families, both open-weight and closed-weight, including two reasoning models. The authors report the drop as an average and describe it as affecting every top model tested.
Is a longer context window a fix?
The study does not test that directly. The failures appeared in conversations short enough to fit comfortably in current context limits, which suggests that memory capacity alone is not the explanation.
Was this peer reviewed?
As of October 27, 2026, the arXiv page lists a single version from May 9, 2025 and no journal or conference reference. This article therefore treats it as a preprint and relies on its stated figures.
Related reading
- Replika and human-like chatbot conversations
- AI chatbots in school settings
- What OpenAI's chat product can do
- Character AI's cloud agreement with Google
More in AI Tools and Industry.
Last verified: October 2026
Sources
- LLMs Get Lost In Multi-Turn Conversation, Laban, Hayashi, Zhou, and Neville, arXiv:2505.06120, May 9, 2025
- LLMs Get Lost In Multi-Turn Conversation, full-text HTML version (ar5iv), arXiv, 2025