A GPT-style chatbot writes a reply one token at a time, where each token is a word or word fragment chosen from probabilities that a neural network computes from the text so far. Because the process imitates statistical patterns in training text and does not check facts, the U.S. National Institute of Standards and Technology (NIST) calls confident but false output, known as confabulation, a natural result of how these models are designed.
Key takeaways
- Chatbot replies are built token by token, with each choice conditioned on everything written before it.
- The Transformer architecture, published on June 12, 2017, is the design behind GPT-style models.
- Human feedback training made GPT-3 follow instructions better, but its authors noted that simple mistakes remained.
- How the next token is picked changes output quality even when the underlying model is identical.
- NIST lists confabulation as a distinct risk of generative AI, strongest in long, open-ended answers.
One token at a time
The architecture behind GPT-style chatbots was introduced in a paper titled "Attention Is All You Need," first submitted to arXiv on June 12, 2017, by eight authors. It described the Transformer, a network that relies on attention instead of recurrence or convolutions, and reported a score of 28.4 BLEU on the WMT 2014 English-to-German translation task, more than two points above the best earlier results.
OpenAI's GPT-3 paper, "Language Models are Few-Shot Learners," followed on May 28, 2020. It describes GPT-3 as an autoregressive language model with 175 billion parameters. Autoregressive means that the model predicts the next piece of text from the pieces before it, appends the result, and repeats. The paper also reports that tasks and examples were supplied purely as text in the prompt, with no gradient updates or fine-tuning for each task. In general terms, a token is a chunk of text, often a short word or part of a longer one, and a chatbot reply of a few hundred words is the product of several hundred such steps.
From text predictor to chatbot
A model trained only to continue text does not automatically behave like an assistant. On March 4, 2022, a team led by Long Ouyang posted "Training language models to follow instructions with human feedback." The authors fine-tuned GPT-3 first on demonstrations written by human labelers and then with reinforcement learning from human feedback, which uses human rankings of model outputs. In their human evaluations, outputs from a 1.3 billion parameter InstructGPT model were preferred to outputs from the 175 billion parameter GPT-3, a model with about 100 times more parameters.
The same abstract reports gains in truthfulness and less toxic output, and it also says the models still make simple mistakes. That sentence matters for anyone using a chatbot for factual work: instruction tuning changes style and helpfulness more reliably than it guarantees accuracy.
Why the same prompt can give different answers
At each step the model assigns a probability to every possible next token. Software then has to pick one, and the rule for picking is called decoding. Always taking the single most likely token sounds safe, but a 2019 paper by Ari Holtzman and colleagues, published at ICLR 2020 as "The Curious Case of Neural Text Degeneration," found that maximizing likelihood produces "bland and strangely repetitive" text. The authors proposed nucleus sampling, which draws from a changing set of the most probable tokens and cuts off the unreliable tail, and showed that the decoding strategy alone can substantially change quality with the same model.
The table is an illustrative simplification, not a description of any specific product.
| Decoding approach | How the next token is chosen | Typical effect on text |
|---|---|---|
| Greedy | Always the single most probable token | Repeatable, but can turn dull or loop |
| Pure random sampling | Any token, weighted by probability | More varied, with a risk of odd word choices from the unlikely tail |
| Nucleus sampling | Random choice among the smallest set of tokens covering most of the probability | Varied but fluent, which is why repeated runs differ |
This is the main reason two identical questions can produce two different answers. Different wording is not evidence that one reply was looked up and the other invented; both come from the same sampling process.
What NIST says about confabulation
NIST's Generative Artificial Intelligence Profile (NIST AI 600-1), published in July 2024, defines confabulation as the production of confidently stated but erroneous or false content, known colloquially as hallucinations or fabrications, by which users may be misled or deceived. It explains that generative models produce outputs approximating the statistical distribution of their training data, and that large language models predict the next token or word. Such prediction can yield accurate and consistent text, but it can also yield text that is factually inaccurate or internally inconsistent.
The profile singles out open-ended prompts that ask for long-form responses and domains that need contextual or expert knowledge as the settings where this matters most. Fluent wording is therefore a poor signal of correctness, since fluency is exactly what the training objective rewards.
A short checklist for reading chatbot output
- Treat names, dates, citations, and numbers as claims to confirm in a primary source.
- Ask for sources, then open them; a listed reference can itself be invented.
- Prefer short, specific questions over open-ended requests for long answers.
- Regenerate or rephrase once; large differences between runs show the answer is not settled.
- Keep personal, confidential, or regulated information out of prompts unless the provider's terms and the organization's policy allow it.
FAQ
Does a chatbot look facts up when it answers?
Not by default. A plain language model generates text from patterns learned in training. Some products add search or document retrieval as a separate step, and the product's own documentation says whether that happens.
Why do chatbots sound so confident when wrong?
The model optimizes for plausible continuations, and plausible text reads confidently. NIST describes confabulations as confidently stated errors that can mislead users.
Did human feedback training remove false statements?
It reduced some problems. The InstructGPT authors reported better truthfulness and less toxic output than GPT-3, and they also reported that the models still make simple mistakes.
Is a token the same as a word?
No. A token is a unit of text that the model works with, which can be a whole short word, part of a longer word, or punctuation. The exact splitting rules differ between models.
Related reading
- GPT-3 and the language model behind it
- GPT-4.5 and later model releases
- The chatbot experience powered by GPT
- What OpenAI chat tools can do
More in AI Tools and Industry.
Last verified: October 2026
Sources
- Vaswani et al., Attention Is All You Need, arXiv:1706.03762, 2017
- Brown et al., Language Models are Few-Shot Learners, arXiv:2005.14165, 2020
- Ouyang et al., Training language models to follow instructions with human feedback, arXiv:2203.02155, 2022
- Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020 (arXiv:1904.09751)
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), National Institute of Standards and Technology, July 2024