Trillion-Parameter AI Models: What the Count Means

Google's Switch Transformers reported 1.6 trillion parameters in 2021. Sparse designs activate only a small share per token, so size alone misleads.

Guide: AI BasicsLast verified: October 20265 min read

A model with more than one trillion parameters first appeared in the research literature in 2021, when Google researchers described Switch Transformers, a family that included a 1.6 trillion parameter model with 2048 experts. Such models are "sparse": for each token, only a small share of the parameters does any work, so the headline parameter count says little about how much computing a model uses. A second Google model, GLaM, reached 1.2 trillion parameters but used about 96.6 billion of them per token.

Key takeaways

  • The Switch Transformer paper (January 2021) reports models of up to 1.6 trillion parameters, built with a mixture-of-experts design.
  • In a mixture-of-experts model, a routing layer sends each token to a few experts, so total parameters and active parameters differ by a large factor.
  • GLaM has 1.2 trillion parameters in total and about 96.6 billion active per token, roughly 8 percent.
  • Kimi K2, an open model described in July 2025, lists 1 trillion total and 32 billion activated parameters.
  • No authoritative public ranking of "the largest model" exists, because many developers do not publish parameter counts.

What a parameter is, and why counting them is tricky

A parameter is a number a model learns during training, such as a weight that scales the signal between two units. A "dense" model uses every parameter for every input. GPT-3, which has 175 billion parameters, is dense, and the GLaM authors describe their 1.2 trillion parameter model as about seven times its size.

That comparison hides a difference in design. In a sparse model, the parameters are divided into groups called experts, and a small routing network decides which experts handle each token. Parameters in the other experts sit idle for that token. Total parameters measure how much the model can store. Active parameters measure how much arithmetic happens per token. Both numbers matter, and a trillion-parameter label usually refers to the first one.

Switch Transformers: the 1.6 trillion parameter model

William Fedus, Barret Zoph, and Noam Shazeer submitted the Switch Transformers paper to arXiv on January 11, 2021, and it was published in the Journal of Machine Learning Research in 2022. The design sends each token to a single expert (top-1 routing), which the authors argue cuts communication and computing cost compared with earlier mixture-of-experts designs. They pre-trained models of up to a trillion parameters on the C4 text corpus and report a four-times speedup over the T5-XXL model.

Inside the paper, the largest model is called Switch-C: 1.6 trillion parameters and 2048 experts. A smaller companion, Switch-XXL, has 395 billion parameters but applies about 10 times more floating-point operations per token. The authors found that Switch-XXL scored 89.6 on the SQuAD reading benchmark against 87.7 for Switch-C, and they describe the dependence between quality, compute per token, and parameter count as poorly understood. More parameters did not automatically mean better results.

GLaM and Kimi K2: other published trillion-scale models

GLaM, from a team at Google, was posted to arXiv on December 13, 2021, and accepted to ICML 2022. Its largest version has 1.2 trillion parameters, 64 experts in each mixture-of-experts layer, and two experts chosen for each token. The authors state that it uses about one third of the energy needed to train GPT-3 and about half the inference compute. These claims come from the paper itself and were not independently checked for this article.

In July 2025, the Kimi K2 report described an open mixture-of-experts language model with 1 trillion total parameters and 32 billion activated. The pattern across all three is the same: very large total counts, modest compute per token.

Total and active parameters side by side

ModelReported total parametersActive per tokenSource type
Switch-C (2021)1.6 trillion, 2048 expertsSingle expert per token; about one tenth the operations per token of Switch-XXLPeer-reviewed paper (JMLR 2022)
Switch-XXL (2021)395 billionAbout 10 times the operations per token of Switch-CSame paper
GLaM (2021)1.2 trillionAbout 96.6 billion (8 percent)Paper accepted to ICML 2022
Kimi K2 (2025)1 trillion32 billionTechnical report on arXiv
GPT-3 (dense, for comparison)175 billionAll 175 billionAs cited in the GLaM paper

Why storing a trillion parameters is its own problem

The following arithmetic is an illustrative example, not a figure from the papers. At 16 bits, or two bytes, per parameter, 1.6 trillion parameters occupy 3.2 terabytes before any training state is counted. If one accelerator chip had 80 gigabytes of memory, the weights alone would need at least 40 such chips. Sparse routing reduces the computing per token, but the memory for all experts still has to exist somewhere, and tokens must be shipped between chips to reach their experts. The Switch authors frame their simplified routing as a way to limit that communication.

Why "largest model" claims need care

  • Check whether the number is total or active parameters.
  • Check whether the model is dense or sparse, since the two are not comparable on size alone.
  • Look for a paper or technical report. Many commercial developers do not publish parameter counts, so figures in news coverage may be estimates.
  • Check whether the claim covers training only or a released, usable model.
  • Note the date. Counts and architectures from 2021 describe that year's research, not today's products.

FAQ

Which was the first trillion-parameter model?

The Switch Transformers paper of January 2021 is an early peer-reviewed report of models reaching 1.6 trillion parameters. Other organizations announced large models around the same period, and priority claims depend on what counts as a trained, evaluated model, so a single winner cannot be confirmed from the sources reviewed.

Does a bigger parameter count mean a smarter model?

Not by itself. In the Switch paper, a model with four times fewer unique parameters scored higher on one benchmark while using about 10 times more computation per token. Training data and compute budget also matter.

Are today's chatbots trillion-parameter models?

For most commercial systems the developer has not published a parameter count, so the question cannot be answered from primary sources. Open reports such as Kimi K2 show that total counts at or above one trillion exist in practice.

What is a mixture of experts?

It is a network design in which many specialized sub-networks (experts) exist but a router activates only a few for each token. The experts are not assigned human-defined topics; they specialize in whatever patterns help the training objective.

Last verified: October 2026

Sources

  • Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus W, Zoph B, Shazeer N, arXiv 2101.03961, 2021; Journal of Machine Learning Research 23(120), 2022. source page
  • GLaM: Efficient Scaling of Language Models with Mixture-of-Experts, Du N et al., arXiv 2112.06905, 2021; ICML 2022. source page
  • Kimi K2: Open Agentic Intelligence, Kimi Team, arXiv 2507.20534, 2025

New to artificial intelligence?

Start with the basics, then follow the guides to tools, rules, and effects on society.

Start with AI Basics