In artificial intelligence, a vector is an ordered list of numbers that stands for a word, an image, a user, or any other item, placed so that similar items end up close together. A system can then judge how alike two items are by comparing their vectors, for example with cosine similarity, which measures the angle between them. Google's machine learning course notes that embeddings for words often have 256, 512, or 1024 dimensions, far more than any chart can show.
Key takeaways
- An embedding is a vector produced by a model, and distance between embeddings is read as relative similarity.
- Euclidean distance, cosine similarity, and the dot product answer slightly different questions, and the dot product also grows with vector length.
- Word vectors from 2013 research showed that relations such as male to female appear as consistent offsets between vectors.
- The original Transformer converts tokens into vectors of 512 numbers, and similarity search over such vectors is a field of its own.
- A single static vector per word cannot separate a word's different meanings.
What a vector holds
A vector is a row of numbers, and each position is one dimension. Google's machine learning course describes an embedding as a vector representation of data in an embedding space, where a model typically projects high-dimensional data into a lower-dimensional space. Each item is described by a fixed count of floating-point numbers, usually between negative one and one or between zero and one.
The course illustrates the idea with food. In a two-dimensional picture, apple strudel is placed at (0.5, 0.3) and hot dog at (0.2, -0.5). Items that sit near each other, such as a hot dog and shawarma, are treated as more closely related than items far apart, such as apple strudel and borscht. Real spaces have many more dimensions, and the course adds that single dimensions are rarely interpretable and that an embedding is specific to the task that produced it.
Three ways to compare two vectors
Google's course on measuring similarity from embeddings lists three measures. Euclidean distance is the length of the straight line between the ends of the two vectors, and it shrinks as similarity grows. Cosine similarity is the cosine of the angle between them, calculated as the dot product divided by the product of the two lengths, and it grows as similarity grows. The dot product multiplies matching components and adds them up; it equals the two lengths times the cosine, so it rises with vector length as well as direction.
The table applies the three measures to a small made-up case. Apple strudel and hot dog use the coordinates quoted above; the third item, a bratwurst at (0.3, -0.4), is an invented point added only for this illustrative example, and the results are rounded.
| Pair | Euclidean distance | Cosine similarity | Dot product |
|---|---|---|---|
| Apple strudel and hot dog | 0.854 | -0.159 | -0.050 |
| Apple strudel and bratwurst (invented) | 0.728 | 0.103 | 0.030 |
| Hot dog and bratwurst (invented) | 0.141 | 0.966 | 0.260 |
All three measures agree on the ranking: the two savory items are the closest pair. They would disagree if one vector were stretched. Doubling the hot dog vector to (0.4, -1.0) would leave its cosine with apple strudel at -0.159 but double the dot product to -0.100, which is why cosine is often preferred when only direction should matter.
How words became vectors
The idea of learning word vectors from text reached wide attention in 2013. A paper by Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, first submitted to arXiv on January 16, 2013, proposed two model designs for computing continuous word vectors from very large datasets and reported that high-quality vectors could be learned from a 1.6 billion word set in less than a day.
A companion paper, "Linguistic Regularities in Continuous Space Word Representations" by Mikolov, Wen-tau Yih, and Geoffrey Zweig, was presented at NAACL in June 2013. It found that a relationship such as male to female is characterized by a relation-specific vector offset, so that "King - Man + Woman" gives a vector very close to "Queen." The authors reported that the vectors answered almost 40 percent of their syntactic analogy questions correctly. That is a measured result on one test set, not a claim that vectors capture meaning in general.
Vectors inside a language model
Modern language models still start from vectors. The 2017 Transformer paper, "Attention Is All You Need," states that the model uses learned embeddings to convert input and output tokens to vectors of dimension d_model, and in the base model that dimension is 512. The paper also shares one weight matrix between the two embedding layers and the output layer that predicts the next token.
Here the vectors are not fixed labels for words. They are parameters that training adjusts, and later layers refine them. Google's course contrasts this with a static embedding, where each word has exactly one vector; it says word2vec is an example and that a single point per word is a limitation, since all of a word's senses are squeezed into one location.
Searching through millions of vectors
Once items are vectors, a common task is nearest-neighbor search: given one vector, find the stored vectors most like it. Checking every stored vector is simple but slow at scale. A 2017 paper by Jeff Johnson, Matthijs Douze, and Hervé Jégou, "Billion-scale similarity search with GPUs," describes exact and approximate methods and reports building a graph that connects 1 billion vectors in less than 12 hours on four GPUs. The authors also released their approach as open source.
The trade-off in approximate search is speed against the chance of missing a true nearest neighbor. A team choosing a vector database or library should measure that trade-off on its own data rather than rely on published speed figures.
Checks before relying on embeddings
- Which model produced the vectors, and were they trained for the same task they are now used for?
- Do all stored vectors come from the same model and dimension count? Vectors from different models cannot be compared.
- Does the similarity measure match the goal, direction only (cosine) or direction and magnitude (dot product)?
- Has retrieval quality been tested on real queries, including rare words and ambiguous ones?
- Is any sensitive text being turned into vectors that are stored and shared, and who can read them?
FAQ
Is a vector the same as an embedding?
Not quite. A vector is any ordered list of numbers. An embedding is a vector that a model learned to produce for an item so that distances between items carry meaning.
Why do embeddings have so many dimensions?
More dimensions give a model more room to separate items and relationships. Google's course gives 256, 512, or 1024 as common sizes for word embeddings, and the original Transformer used 512. The right size depends on the task and the model.
Can the numbers in an embedding be read directly?
Usually not. Google's course says individual dimensions are rarely interpretable, so the useful information lies in the distances and angles between vectors.
Does "King - Man + Woman = Queen" always work?
No. The 2013 paper reported that the result was very close to "Queen" in its vectors and measured an accuracy of almost 40 percent on its syntactic analogy set. Results vary by model, training data, and test.
Related reading
- The concept and importance of vectors in AI
- Essential components of AI systems
- What artificial intelligence is, with a Harvard guide
- Khan Academy and learning about AI
More in AI Basics.
Last verified: October 2026
Sources
- Embeddings: embedding space and static embeddings, Machine Learning Crash Course, Google for Developers
- Measuring similarity from embeddings, Machine Learning Crash Course, Google for Developers
- Mikolov T, Chen K, Corrado G, Dean J., Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781, 2013
- Mikolov T, Yih W, Zweig G., Linguistic Regularities in Continuous Space Word Representations, NAACL-HLT, Association for Computational Linguistics, 2013
- Vaswani A et al., Attention Is All You Need, arXiv:1706.03762, 2017
- Johnson J, Douze M, Jégou H., Billion-scale similarity search with GPUs, arXiv:1702.08734, 2017