Skip to content
Papers.

Training Compute-Optimal Large Language Models research paper by Google DeepMind, 2022

Google DeepMind · Mar 29, 2022 · Training and scaling · 3,755 citations · 13 upvotes

Read on arXiv

What it shows

Chinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.

Summarised by hand from the abstract.

More from Google DeepMind

All 3
PaperCitations
Gemma 4 Technical ReportOpen multimodal models from 2.3B to 31B parameters, with a thinking mode and image and audio input.Foundation models · Jul 2026106
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextRecalls details across millions of tokens of text, hours of video and days of audio.Foundation models · Mar 20244,002

Other training and scaling papers

Topic
PaperCitations
LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 202123.2k
Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 20209,061
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 20193,177
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024405
About this paper
Authors
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch and 19 more
arXiv
2203.15556 · PDF
Venue
Advances in Neural Information Processing Systems 35
Citations
3,755, 352 influential · Semantic Scholar
Upvotes
13 · Hugging Face
Code
github.com/karpathy/llama2.c
Lab
Google DeepMind · on Companies · on Acquisitions

Changes

What changed
New paperAdded to the listSep 24, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.