A model with more than one trillion parameters first appeared in the research literature in 2021, when Google researchers described Switch Transformers, a family that included a 1.6 trillion parameter model with 2048 experts. Such models are "sparse": for each token, only a small share of the parameters does any work, so the headline parameter count says little about how much computing a model uses. A second Google model, GLaM, reached 1.2 trillion parameters but used about 96.6 billion of them per token.
Key takeaways
- The Switch Transformer paper (January 2021) reports models of up to 1.6 trillion parameters, built with a mixture-of-experts design.
- In a mixture-of-experts model, a routing layer sends each token to a few experts, so total parameters and active parameters differ by a large factor.
- GLaM has 1.2 trillion parameters in total and about 96.6 billion active per token, roughly 8 percent.
- Kimi K2, an open model described in July 2025, lists 1 trillion total and 32 billion activated parameters.
- No authoritative public ranking of "the largest model" exists, because many developers do not publish parameter counts.
What a parameter is, and why counting them is tricky
A parameter is a number a model learns during training, such as a weight that scales the signal between two units. A "dense" model uses every parameter for every input. GPT-3, which has 175 billion parameters, is dense, and the GLaM authors describe their 1.2 trillion parameter model as about seven times its size.
That comparison hides a difference in design. In a sparse model, the parameters are divided into groups called experts, and a small routing network decides which experts handle each token. Parameters in the other experts sit idle for that token. Total parameters measure how much the model can store. Active parameters measure how much arithmetic happens per token. Both numbers matter, and a trillion-parameter label usually refers to the first one.
Switch Transformers: the 1.6 trillion parameter model
William Fedus, Barret Zoph, and Noam Shazeer submitted the Switch Transformers paper to arXiv on January 11, 2021, and it was published in the Journal of Machine Learning Research in 2022. The design sends each token to a single expert (top-1 routing), which the authors argue cuts communication and computing cost compared with earlier mixture-of-experts designs. They pre-trained models of up to a trillion parameters on the C4 text corpus and report a four-times speedup over the T5-XXL model.
Inside the paper, the largest model is called Switch-C: 1.6 trillion parameters and 2048 experts. A smaller companion, Switch-XXL, has 395 billion parameters but applies about 10 times more floating-point operations per token. The authors found that Switch-XXL scored 89.6 on the SQuAD reading benchmark against 87.7 for Switch-C, and they describe the dependence between quality, compute per token, and parameter count as poorly understood. More parameters did not automatically mean better results.
GLaM and Kimi K2: other published trillion-scale models
GLaM, from a team at Google, was posted to arXiv on December 13, 2021, and accepted to ICML 2022. Its largest version has 1.2 trillion parameters, 64 experts in each mixture-of-experts layer, and two experts chosen for each token. The authors state that it uses about one third of the energy needed to train GPT-3 and about half the inference compute. These claims come from the paper itself and were not independently checked for this article.
In July 2025, the Kimi K2 report described an open mixture-of-experts language model with 1 trillion total parameters and 32 billion activated. The pattern across all three is the same: very large total counts, modest compute per token.
Total and active parameters side by side
| Model | Reported total parameters | Active per token | Source type |
|---|---|---|---|
| Switch-C (2021) | 1.6 trillion, 2048 experts | Single expert per token; about one tenth the operations per token of Switch-XXL | Peer-reviewed paper (JMLR 2022) |
| Switch-XXL (2021) | 395 billion | About 10 times the operations per token of Switch-C | Same paper |
| GLaM (2021) | 1.2 trillion | About 96.6 billion (8 percent) | Paper accepted to ICML 2022 |
| Kimi K2 (2025) | 1 trillion | 32 billion | Technical report on arXiv |
| GPT-3 (dense, for comparison) | 175 billion | All 175 billion | As cited in the GLaM paper |
Why storing a trillion parameters is its own problem
The following arithmetic is an illustrative example, not a figure from the papers. At 16 bits, or two bytes, per parameter, 1.6 trillion parameters occupy 3.2 terabytes before any training state is counted. If one accelerator chip had 80 gigabytes of memory, the weights alone would need at least 40 such chips. Sparse routing reduces the computing per token, but the memory for all experts still has to exist somewhere, and tokens must be shipped between chips to reach their experts. The Switch authors frame their simplified routing as a way to limit that communication.
Why "largest model" claims need care
- Check whether the number is total or active parameters.
- Check whether the model is dense or sparse, since the two are not comparable on size alone.
- Look for a paper or technical report. Many commercial developers do not publish parameter counts, so figures in news coverage may be estimates.
- Check whether the claim covers training only or a released, usable model.
- Note the date. Counts and architectures from 2021 describe that year's research, not today's products.
FAQ
Which was the first trillion-parameter model?
The Switch Transformers paper of January 2021 is an early peer-reviewed report of models reaching 1.6 trillion parameters. Other organizations announced large models around the same period, and priority claims depend on what counts as a trained, evaluated model, so a single winner cannot be confirmed from the sources reviewed.
Does a bigger parameter count mean a smarter model?
Not by itself. In the Switch paper, a model with four times fewer unique parameters scored higher on one benchmark while using about 10 times more computation per token. Training data and compute budget also matter.
Are today's chatbots trillion-parameter models?
For most commercial systems the developer has not published a parameter count, so the question cannot be answered from primary sources. Open reports such as Kimi K2 show that total counts at or above one trillion exist in practice.
What is a mixture of experts?
It is a network design in which many specialized sub-networks (experts) exist but a router activates only a few for each token. The experts are not assigned human-defined topics; they specialize in whatever patterns help the training objective.
Related reading
- How deep learning works and where it is used
- What the GPT-3 paper and API showed
- Google's LaMDA language model
- Google's Gemini model announcement
More in AI Basics.
Last verified: October 2026
Sources
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus W, Zoph B, Shazeer N, arXiv 2101.03961, 2021; Journal of Machine Learning Research 23(120), 2022. source page
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts, Du N et al., arXiv 2112.06905, 2021; ICML 2022. source page
- Kimi K2: Open Agentic Intelligence, Kimi Team, arXiv 2507.20534, 2025