Skip to content
Papers.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism research paper by NVIDIA, 2019

NVIDIA · Sep 17, 2019 · Training and scaling · 3,177 citations · 5 upvotes

Read on arXiv

What it shows

Megatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.

Summarised by hand from the abstract.

More from NVIDIA

All 3
PaperCitations
Cosmos 3: Omnimodal World Models for Physical AIOne model that reads and generates text, images, video, audio and robot actions for physical AI.Multimodal and robotics · Jun 202690
Nemotron-4 340B Technical ReportOpen models sized for one 8-GPU server, aligned with over 98% synthetic data.Foundation models · Jun 2024138

Other training and scaling papers

Topic
PaperCitations
LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 202123.2k
Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 20209,061
Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 20223,755
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024405
About this paper
Authors
Mohammad Shoeybi, Mostofa Patwary, Raul Puri and 3 more
arXiv
1909.08053 · PDF
Venue
arXiv.org
Citations
3,177, 393 influential · Semantic Scholar
Upvotes
5 · Hugging Face
Lab
NVIDIA · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf

Changes

What changed
New paperAdded to the listSep 24, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.