No published source can name one AI software package as the leader in performance and efficiency, because those words describe a measurement taken on a specific model, chip, and workload. What the research literature does document is a set of engineering techniques that cut the time or memory a model needs, with reported gains such as a three-times speedup for GPT-2 training and two to four times higher serving throughput for large language models.
Key takeaways
- "Performance" can mean accuracy, speed, or throughput, and "efficiency" can mean memory, cost, or energy. A claim is only comparable when it names the metric, hardware, and baseline.
- FlashAttention reported a three-times speedup on GPT-2 training with exact results, by reducing data movement on the GPU.
- vLLM reported two to four times higher serving throughput at the same latency, by managing the key-value cache more carefully.
- LLM.int8() reported roughly halving inference memory for a 175-billion-parameter model with no performance degradation.
- Each gain was measured against named systems in a paper, so it describes that comparison and not every deployment.
What a performance or efficiency claim has to specify
Speed and quality pull in different directions. A smaller or compressed model may answer faster and still answer worse, so a useful claim reports both. Three details separate a measurement from a slogan: the task and data, the hardware (a particular GPU model and count), and the baseline that the new result is compared against. A "twice as fast" figure against an old implementation says little about a well-tuned current one.
Throughput and latency are also different. Throughput counts how many requests or tokens a system handles per second across many users. Latency is how long one user waits. Serving systems can raise throughput by batching requests, which may lengthen the wait for an individual request, so papers that report throughput gains usually state the latency they held constant.
Reducing data movement: FlashAttention
Attention, the core operation in transformer models, moves large arrays between a GPU's main memory and its small, faster on-chip memory. Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Re submitted FlashAttention to arXiv on May 27, 2022. It splits the computation into tiles so fewer reads and writes cross that boundary, and it still computes exact attention, not an approximation. The authors reported a 15 percent end-to-end speedup on BERT-large at sequence length 512 compared with the MLPerf 1.1 training speed record, three times on GPT-2 at sequence length 1K, and 2.4 times on the long-range arena benchmark at lengths 1K to 4K.
Serving many users: PagedAttention and vLLM
When a language model generates text, it keeps a key-value cache for each request. That cache is large and changes size as the answer grows, so memory gets wasted through fragmentation and duplication, which limits how many requests fit in one batch. A team led by Woosuk Kwon posted Efficient Memory Management for Large Language Model Serving with PagedAttention on September 12, 2023, and the work appeared at SOSP 2023. Their vLLM system reported two to four times higher throughput at the same latency compared with systems such as FasterTransformer and Orca, with larger gains for longer sequences, bigger models, and more complex decoding.
Using fewer bits: quantization and distillation
Quantization stores numbers at lower precision. In LLM.int8(), submitted August 15, 2022, Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer reported that their method roughly halves inference memory for feed-forward and attention projection layers. A 175-billion-parameter checkpoint can be converted to 8-bit and used without performance degradation, according to the abstract, because a small set of outlier dimensions is kept in 16-bit while more than 99.9 percent of values are multiplied in 8-bit. They released their software and said it lets models of that size run on one server with consumer GPUs.
Distillation takes a different route: a small model is trained to imitate a large one. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean described the idea for neural networks in a paper submitted March 9, 2015. Its abstract says distilling an ensemble into a single model can significantly improve the acoustic model of a heavily used commercial speech system, without giving a number.
Summary of the four techniques
| Technique | What it changes | Reported result | Condition to check |
|---|---|---|---|
| FlashAttention (2022) | Memory traffic inside attention | Three times faster GPT-2 training, exact output | Sequence length and GPU generation |
| vLLM and PagedAttention (2023) | Key-value cache memory | Two to four times higher serving throughput | Latency held equal, compared systems named |
| LLM.int8() (2022) | Numeric precision at inference | About half the memory, no degradation reported | Tested model families and tasks |
| Distillation (2015) | Model size | Better single model from an ensemble, no figure in abstract | Task and teacher model |
A checklist for reading an efficiency claim
- Which metric moved: speed, memory, cost, or quality?
- What was the baseline, and was it a current, tuned system?
- Which hardware and which model sizes were tested?
- Did quality get measured on the same tasks before and after?
- Is the source a peer-reviewed paper, a preprint, or a seller's own benchmark?
- Has anyone else reproduced the result on different hardware?
FAQ
Is there an official ranking of the best AI software for efficiency?
No neutral ranking covers all workloads. Standardized benchmarks such as MLPerf measure defined tasks on defined systems, and a winner on one task can trail on another.
Do these techniques change what the model answers?
FlashAttention computes exact attention, so its output matches the standard method. Quantization and distillation can change outputs, which is why the papers measure accuracy alongside memory or speed.
Are the 2022 and 2023 numbers still current?
They are dated measurements. Hardware and software have changed since, and later work may report different gains, so each figure should be read with its paper date.
Does a faster system always cost less?
Not automatically. Cost depends on hardware prices, utilization, and engineering time to adopt a technique, and none of the papers above report a total cost of ownership.
Related reading
- AI hardware in 2022 and what MLPerf measured
- How AI is applied to complex problems
- Companies that design and make AI chips
- Setting the learning rate when training a model
More in AI Tools and Industry.
Last verified: October 2026
Sources
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao T, Fu D, Ermon S, Rudra A, Re C, arXiv 2205.14135, 2022
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon W et al., arXiv 2309.06180, SOSP 2023
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, Dettmers T, Lewis M, Belkada Y, Zettlemoyer L, arXiv 2208.07339, NeurIPS 2022
- Distilling the Knowledge in a Neural Network, Hinton G, Vinyals O, Dean J, arXiv 1503.02531, 2015