Skip to content
Papers.

Scaling Laws for Neural Language Models research paper by OpenAI, 2020

OpenAI · Jan 23, 2020 · Training and scaling · 9,061 citations · 12 upvotes

Read on arXiv

What it shows

Language model loss falls as a smooth power law as model size, data and compute grow.

Summarised by hand from the abstract.

More from OpenAI

All 5
PaperCitations
Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.Alignment and safety · Sep 2025329
GPT-4 Technical ReportA multimodal model that passes a simulated bar exam with a score around the top 10% of test takers.Foundation models · Mar 202327.3k
Training language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.Alignment and safety · Mar 202224.2k
Language Models are Few-Shot LearnersGPT-3, a 175B-parameter model, does new tasks from a few examples in the prompt, with no fine-tuning.Foundation models · May 202063.6k

Other training and scaling papers

Topic
PaperCitations
LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 202123.2k
Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 20223,755
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 20193,177
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024405
About this paper
Authors
Jared Kaplan, Sam McCandlish, Tom Henighan and 7 more
arXiv
2001.08361 · PDF
Venue
arXiv.org
Citations
9,061, 631 influential · Semantic Scholar
Upvotes
12 · Hugging Face
Lab
OpenAI · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf

Changes

What changed
New paperAdded to the listSep 24, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.