A large language model is built in two broad phases: pre-training on trillions of tokens of text, then post-training with human preference data to make it follow instructions. Meta's 2024 technical report on Llama 3 is one of the most detailed public accounts. It describes a 405-billion-parameter model pre-trained on 15.6 trillion tokens with a compute budget of 3.8 x 10^25 floating-point operations, followed by several rounds of post-training.
Key takeaways
- Llama 3 was pre-trained on about 15 trillion multilingual tokens, compared with 1.8 trillion tokens for Llama 2.
- Much of the work is data cleaning: the report describes deduplication at the URL, document, and line level, plus quality classifiers.
- Model size was chosen by fitting scaling laws on smaller training runs before the flagship run began.
- During a 54-day stretch of pre-training, the job was interrupted 466 times, and about 78 percent of the unexpected stops were attributed to confirmed or suspected hardware faults.
- Post-training combines supervised fine-tuning, rejection sampling, and direct preference optimization, repeated over several rounds.
Step one: collecting and cleaning the text
Pre-training data starts as a large web crawl and has to be filtered before it is useful. The Llama 3 report lists three levels of deduplication: pages with the same URL, near-duplicate documents found with a technique called MinHash, and repeated lines, where text that appeared more than six times in a bucket of 30 million documents was removed. The authors note that line-level removal also deletes some good text, but their tests showed clear gains overall.
Heuristic filters then drop documents with long repeated passages or many unusual tokens, and classifiers trained on labeled examples score the remaining text for quality. Separate pipelines pick out code and mathematics pages, and a language identifier sorts documents into 176 languages. The final mix is stated as roughly 50 percent general knowledge, 25 percent mathematical and reasoning tokens, 17 percent code, and eight percent multilingual text.
Step two: deciding how big to make the model
A model that is too small wastes data, and one that is too large wastes compute. The report fits scaling laws, which are curves relating training compute to the best model size and token count, using many smaller runs. Extrapolating to the planned budget of 3.8 x 10^25 operations suggested a model of about 402 billion parameters trained on 16.55 trillion tokens. Because the curves flatten near their minimum, the team judged the result insensitive to small shifts and chose 405 billion parameters, a dense Transformer with 126 layers.
Step three: running the training job
Llama 3 405B was trained on up to 16,000 H100 GPUs. The report says the complexity of failures at this size exceeds that of larger CPU clusters the authors had run, because training is synchronous and a single GPU failure can force a restart. Across a 54-day snapshot there were 466 interruptions: 47 planned, such as firmware updates, and 419 unexpected. GPU problems made up 58.7 percent of the unexpected ones, and manual intervention was needed only three times. Automation kept effective training time above 90 percent.
What each stage produces
The table restates the report's stages in plain terms. It is a simplified summary of one developer's account, and other developers organize the work differently.
| Stage | What goes in | What comes out | What can go wrong |
|---|---|---|---|
| Data preparation | Web crawl, code, multilingual text | Filtered, deduplicated token mix | Filters remove good text along with bad |
| Scaling study | Small training runs | Chosen model size and token count | Predictions from small runs can be noisy |
| Pre-training | 15.6 trillion tokens, 16,000 GPUs | A model that predicts next tokens | Hardware failures, lost time |
| Post-training | Human preference data, synthetic data | An assistant that follows instructions | Training data can contain errors |
| Evaluation | Benchmarks and adversarial tests | Scores and a contamination check | Benchmark text may appear in training data |
Step four: teaching the model to follow instructions
A pre-trained model completes text, but does not answer questions on request. Post-training changes that. A reward model is first trained on human-annotated preference data, which marks which of two answers is better. For each prompt, the team samples between 10 and 30 candidate responses and uses the reward model to keep the best one, a step called rejection sampling. Supervised fine-tuning on the kept responses is followed by direct preference optimization, and the cycle runs for several rounds, with the stronger model from each round helping to produce data for the next. For code, the report adds checks: generated programs go through a parser, a linter, and automatically written unit tests, and about 20 percent of solutions were initially wrong but corrected after error feedback.
Limits of this account
The report was written by the developer and covers one model family, released in July 2024, so it cannot be treated as a description of every system. The cleaning and filtering choices described are judgment calls, and the report itself says its manual review found that line-level deduplication removes some high-quality text. Readers evaluating any model can ask which data was excluded, how evaluation sets were kept apart from training data, and whether the evaluation results were produced by an independent party.
FAQ
Does building an AI model mean writing rules for each answer?
No. The model learns statistical patterns from data, and people decide the data, the objective, and the checks. Hand-written code covers the training pipeline, filters, and evaluation harness.
Why does training need so many GPUs?
The compute budget in the Llama 3 report is 3.8 x 10^25 operations, almost 50 times that of the largest Llama 2. Spreading the work across thousands of processors is the only way to finish in weeks, and it brings the failure problems described above.
Is more data always better?
Not by itself. The report spends heavy effort on removing duplicates and low-quality text, and it uses scaling experiments to balance model size against token count.
Who checks that the finished model is safe?
The report includes evaluation sections, including adversarial benchmarks and a contamination analysis, but these are produced by the developer. Independent testing is a separate step that the paper does not replace.
Related reading
- Whether AI can create and improve itself
- Why language models state false things
- The origins of artificial intelligence
- Who owns an artificial intelligence system
More in AI and Society.
Last verified: October 2026