A bigger context window does not mean a model reads all of it equally well. In "Lost in the Middle," published in Transactions of the Association for Computational Linguistics in 2024, accuracy was highest when the relevant passage sat at the start or end of the input and fell when it sat in the middle, even in models built for long inputs. A July 2025 study from Chroma, which tested 18 newer models, found a different weakness: accuracy slipped as inputs grew longer, especially with look-alike distractor passages. How much text is supplied, and what else is in it, matters as much as how much fits.
Key takeaways
- A language model draws on knowledge stored in its weights, on the text supplied in the current prompt, and, in some designs, on an external index it searches.
- In the Lost in the Middle tests, accuracy followed a U shape: strongest for information at the beginning or end of the input and weakest in the middle.
- For GPT-3.5-Turbo, the worst-case result with 20 or 30 documents was lower than its 56.1 percent score with no documents at all.
- Models with extended context windows often matched their shorter-window counterparts when the input fit in both.
- The Lost in the Middle models date from 2023. A July 2025 Chroma study of 18 newer models found that accuracy still declined with input length, and that distractors made it worse.
Three places a model keeps information
Discussions of "memory" in AI mix several things. The table separates three, using the terms from the research papers cited here.
| Kind of memory | Where it lives | Limit |
|---|---|---|
| Parametric memory | The model's weights, set during training | Fixed until the model is retrained; precise recall of facts is imperfect |
| Input context | The text supplied in the current prompt | Capped by the context window and, as the study below shows, used unevenly |
| Non-parametric memory | An external index the system searches, such as a document store | Quality depends on what the search retrieves |
The phrase "limited memory" usually refers to the second row, the finite amount of text a model can read at once. The first and third rows explain why systems rarely depend on that window alone.
What "Lost in the Middle" tested
Nelson Liu and six co-authors tested how well models find and use information in long inputs. The paper is available in the ACL Anthology, and a preprint first appeared on arXiv on July 6, 2023. Two tasks were used. In multi-document question answering, a model receives a question plus 10, 20, or 30 documents (roughly 2,000, 4,000, or 6,000 tokens) and only one contains the answer; the researchers moved that document from first to last position. In key-value retrieval, a model receives a block of random identifier pairs and must return the value for one key.
The models included GPT-3.5-Turbo and a 16K-token version, Claude-1.3 and a 100K-token version, MPT-30B-Instruct, and LongChat-13B (16K). In the question answering task, accuracy was highest when the answer sat at the beginning or end, forming a U-shaped curve. GPT-3.5-Turbo's performance could drop by more than 20 percent, and in the worst case with 20 or 30 documents it fell below its closed-book result of 56.1 percent, where the model received no documents. Key-value retrieval showed a mixed picture: both Claude versions did nearly perfectly across the tested lengths, while other models struggled at 140 and 300 pairs.
Longer windows and the retrieval alternative
Two further findings matter for anyone choosing a tool. First, when the input fit in both a model and its extended-context counterpart, performance was often identical, so the larger window was not necessarily better at using the text. Second, placing the question both before and after the documents did not improve the question answering results substantially, although it lifted GPT-3.5-Turbo (16K) to perfect accuracy on 300 key-value pairs in the retrieval task.
A different strategy is to avoid loading everything. The 2020 NeurIPS paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, paired a pretrained generator (parametric memory) with a dense vector index of an online encyclopedia (non-parametric memory) and reported state-of-the-art results on three open-domain question answering tasks at that time. Retrieval narrows the text given to the model, which is one response to a limited window, though its answers are only as good as the passages retrieved.
What changed since the 2023 tests
The Lost in the Middle paper is the latest peer-reviewed edition of that study, and the models it tested are all from 2023. Follow-up work has used newer systems. On July 14, 2025, Kelly Hong, Anton Troynikov, and Jeff Huber of the vector database company Chroma published "Context Rot," which tested 18 models, among them GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, and Qwen3-235B. Their tasks were deliberately simple, so input length was the main variable.
Three results stand out. Performance fell as input length grew, even on easy tasks. A single topically related distractor lowered accuracy compared with the baseline, and four distractors lowered it more. Questions with a weak semantic match to the answer passage degraded faster at long lengths. Position behaved differently from 2023: across 11 positions in the needle-in-a-haystack task, the authors saw no notable variation. The Chroma report is a company study and not a peer-reviewed paper, and no equivalent test of models released in 2026 was found for this article, so how today's systems behave is still something to check directly.
A simple position test for any chat tool
The following check is an illustrative classroom or office exercise derived from the study's design, not a validated benchmark.
- Prepare a long document of about 20 short paragraphs with one planted fact, such as an invented date.
- Ask the same question three times, placing the planted fact near the start, the middle, and the end.
- Record whether each answer is right, and repeat with several different planted facts, since a single trial proves little.
- If middle placement fails more often, put key instructions and facts at the beginning or end, or supply fewer documents.
FAQ
What does limited memory mean for a language model?
It usually means the model can read only a finite amount of text per request, called the context window. Knowledge learned in training sits separately in the model's weights.
Does a bigger context window fix the problem?
Not by itself. In the 2024 study, extended-context models often performed identically to shorter-window versions on inputs that fit both, and accuracy still varied with position.
Do these results apply to current models?
Only in part. The Lost in the Middle models were from 2023. In 2025, Chroma tested 18 newer models and saw no clear position effect on a simple needle task, but it did see accuracy fall with length and with distractors. No study of 2026 models was found, so a quick test on the tool in use is the safest check.
Is retrieval a replacement for a long context?
It is an alternative that supplies fewer, more relevant passages. Its quality depends on the search step, so it needs its own testing.
Related reading
- Examples of limited memory AI
- Limited memory AI and restricted data storage
- Artificial and natural intelligence compared
- Core components of AI systems
More in AI Basics.
Last verified: October 2026
Sources
- Lost in the Middle: How Language Models Use Long Contexts, Liu et al., Transactions of the Association for Computational Linguistics, 2024
- Lost in the Middle (preprint v3), arXiv 2307.03172, November 20, 2023
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020, arXiv 2005.11401
- Context Rot: How Increasing Input Tokens Impacts LLM Performance, Hong, Troynikov, and Huber, Chroma, July 14, 2025