LoRA Learns Less and Forgets Less research paper by Databricks, 2024

Databricks · May 15, 2024 · Training and scaling · 405 citations · 91 upvotes

LoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.

Summarised by hand from the abstract.

Date	Paper	Lab	Citations
Jun 17, 2021	LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 2021Microsoft	Microsoft	23.2k
Jan 23, 2020	Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 2020OpenAI	OpenAI	9,061
Mar 29, 2022	Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 2022Google DeepMind	Google DeepMind	3,755
Sep 17, 2019	Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 2019NVIDIA	NVIDIA	3,177

About this paper

Authors: Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes and 9 more
arXiv: 2405.09673 · PDF
Venue: Trans. Mach. Learn. Res.
Citations: 405, 38 influential · Semantic Scholar
Upvotes: 91 · Hugging Face
Code: github.com/danbider/lora-tradeoffs
Lab: Databricks · on Companies · on Acquisitions · on Rounds · on Paydays · on Releases · on TechConf

Changes

Date	What changed
Sep 24, 2026	New paperAdded to the listSep 24, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

What it shows

Other training and scaling papers

Changes

New papers by email