Skip to content
Papers.

LoRA: Low-Rank Adaptation of Large Language Models research paper by Microsoft, 2021

Microsoft · Jun 17, 2021 · Training and scaling · 23,235 citations · 66 upvotes

Read on arXiv

What it shows

LoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.

Summarised by hand from the abstract.

More from Microsoft

All 4
PaperCitations
StudentSim: Training LLM-based Student SimulatorsStudentSim trains per-student simulators that answer like a given learner and change their answers under a tutor's guidance.Applied AI · Sep 20260
From Local to Global: A Graph RAG Approach to Query-Focused SummarizationGraphRAG builds a knowledge graph of a document set so a model can answer questions about the whole collection.Retrieval and data · Apr 20242,191
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your PhoneSmall models trained on filtered and synthetic data that run on a phone and rival much larger ones.Foundation models · Apr 20242,497

Other training and scaling papers

Topic
PaperCitations
Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 20209,061
Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 20223,755
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 20193,177
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024405
About this paper
Authors
Edward J. Hu, Yelong Shen, Phillip Wallis and 5 more
arXiv
2106.09685 · PDF
Venue
International Conference on Learning Representations
Citations
23,235, 3,222 influential · Semantic Scholar
Upvotes
66 · Hugging Face
Lab
Microsoft · on Companies · on Acquisitions · on Quarterly · on Paydays · on Releases · on TechConf

Changes

What changed
New paperAdded to the listSep 24, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.