Vector search finds stored text whose list of numbers (its vector) points in nearly the same direction as the vector of a question, and a retrieval-augmented system passes the best matches to a language model as reference material. The approach was described for language tasks in a 2020 NeurIPS paper by Patrick Lewis and colleagues, which paired a model's learned weights with a dense vector index of an online encyclopedia. The method retrieves the nearest text, not necessarily the correct text, so its results need checking.
Key takeaways
- Retrieval-augmented generation (RAG) splits memory in two: the model's weights and an outside store that is searched by vector similarity.
- Ranking by cosine similarity and ranking by dot product can disagree, because the dot product also rewards long vectors.
- Large collections are usually searched approximately, for example with Hierarchical Navigable Small World (HNSW) graphs, which trade a small chance of a miss for speed.
- Stored embeddings are not anonymous: a 2023 study recovered 92 percent of 32-token texts exactly from their embeddings.
Why a language model needs a searchable store
A language model keeps what it learned in training inside its weights, and those weights cannot be edited line by line. The 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" calls this parametric memory and adds a second, non-parametric memory: a dense vector index of an online encyclopedia that a neural retriever searches. The authors reported state-of-the-art results on three open-domain question answering tasks and found that the combined system produced more specific, diverse, and factual text than a model using weights alone.
The practical appeal is easy to see. A company policy, a course syllabus, or a clinic's intake guide can be placed in the outside store and replaced when it changes, with no retraining. The model never needs to memorize the document; it needs to receive the right passage at the right moment. Vectors are what make "the right passage" computable.
The pipeline in five steps
A typical retrieval system works as follows. First, documents are cut into passages of a few hundred words. Second, an embedding model turns each passage into a vector, and the vectors are stored with a pointer to the original text. Third, when a question arrives, the same embedding model turns it into a vector. Fourth, the system looks for the stored vectors closest to the question vector. Fifth, the text of the top few passages is placed in the prompt, and the language model writes an answer from that material.
Two details matter more than they seem. The question and the passages must be embedded by the same model, since each model builds its own coordinate system. And the cut points matter: a passage that splits a rule from its exception may be retrieved without the exception.
A worked example with three numbers per passage
Real embeddings run to hundreds of dimensions. The table below uses three, with invented values, so the arithmetic can be checked by hand. It is an illustrative example, not output from any real model. The query is "What is the return deadline for an item?" and its vector is (0.9, 0.1, 0.3). Cosine similarity is the dot product divided by the product of the two vector lengths; the query's length is 0.954.
| Passage (invented) | Vector | Dot product with query | Cosine similarity |
|---|---|---|---|
| A: Returns accepted within 30 days | (0.8, 0.2, 0.1) | 0.770 | 0.972 |
| B: Store opening hours | (0.1, 0.9, 0.4) | 0.300 | 0.318 |
| C: Shipping and delivery times | (0.7, 0.0, 0.6) | 0.810 | 0.921 |
| D: Gift card terms | (-0.5, 0.4, 0.8) | -0.170 | -0.174 |
By cosine similarity the order is A, C, B, D, and passage A is correctly first. By raw dot product passage C (0.810) edges ahead of A (0.770), because C's vector is longer. Which measure is right depends on how the embedding model was trained, so the model's documentation should be followed. Taking the top two results also shows the limit of the method: C is close to the question in direction, yet it answers a different question. The language model must still be instructed to say when the retrieved text does not contain an answer.
Exact search versus approximate search
Comparing a question with every stored vector is exact and, for a few thousand passages, perfectly fast. For millions, systems switch to approximate nearest neighbor search. One widely used method is described in a paper by Yu. A. Malkov and D. A. Yashunin, first posted in March 2016. HNSW builds several layers of proximity graphs over nested subsets of the stored items. Search starts in a sparse top layer, takes long jumps to reach the right neighborhood, and then moves down to denser layers for fine steps. The authors report that search cost grows roughly logarithmically with the size of the collection.
"Approximate" has a concrete meaning: the true closest vector is sometimes not returned. The share of true neighbors that are found is called recall, and it can be raised by searching more of the graph at the price of time. A team should measure recall on its own data, since results on a benchmark do not transfer automatically.
What stored vectors can reveal
It is tempting to treat an embedding as a scrambled, harmless summary. A 2023 paper by John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush, "Text Embeddings Reveal (Almost) As Much As Text," tested that assumption. Their method iteratively proposes text, re-embeds it, and corrects it. It recovered 92 percent of 32-token inputs exactly, and it recovered full names from a dataset of clinical notes. The result applies to the models and settings they studied, but it is enough to show that an embedding of a sensitive document should be handled with the same access controls as the document.
Checks before trusting a retrieval system
- Which similarity measure does the embedding model expect, cosine or dot product?
- Were test questions run against the store, including questions whose answer is not in it?
- Does the answer show which passage it came from, so a reader can verify it?
- How are vectors refreshed when a document changes or the embedding model is replaced?
- Who can read the vector store, and does it hold personal or confidential text?
FAQ
Does retrieval stop a language model from making things up?
It reduces the problem but does not end it. The model can still misread a passage, merge two passages, or answer from memory when nothing relevant was retrieved. The 2020 paper reported more factual generation than a weights-only model, not error-free output.
What is a vector database?
It is software that stores vectors with their source text and answers nearest-neighbor queries quickly, often with an approximate method such as HNSW. Ordinary databases and search libraries can also do this for smaller collections.
Why not just search for keywords?
Keyword search matches words, so a question about "returning an item" can miss a passage that says "refund window." Vector search can match by meaning, but it can also miss exact terms such as product codes. Many systems combine both.
Can the numbers be turned back into text?
Sometimes. The 2023 study above recovered most short texts exactly from their embeddings, so vectors from private documents deserve the same protection as the documents.
Related reading
- What vectors do in AI, with embeddings and similarity measures
- Why the problem domain shapes an AI project
- Human-centered AI and keeping people in the loop
- Education codes and their role in schools
More in AI Basics.
Last verified: October 2026
Sources
- Lewis P et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401, NeurIPS 2020
- Malkov Yu A, Yashunin D A., Approximate nearest neighbor search using Hierarchical Navigable Small World graphs (title shortened), arXiv:1603.09320, 2016 (revised 2018)
- Morris J X, Kuleshov V, Shmatikov V, Rush A M., Text Embeddings Reveal (Almost) As Much As Text, arXiv:2310.06816, EMNLP 2023