Notable data and AI papers

Papers627Labs45

627 papers · 45 labs

LoRA Learns Less and Forgets Less research paper by Databricks, 2024

Databricks · Training and scaling · May 15, 2024 · 486 citations · 91 upvotes

Read on arXiv

What it shows

LoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.

By Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes and 9 more · arXiv 2405.09673 · PDF · Code

Summarised by hand from the abstract.

More from Databricks

All 3
Agents and evaluationUnverifiedMar 9, 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded ReasoningOfficeQA Pro evaluates AI agents on multi-document reasoning across historical financial documents, revealing persistent challenges in grounded reasoning despite advanced model capabilities.Databricks48 citations3 upvotesRead →Agents and evaluationUnverifiedMar 5, 2026KARL: Knowledge Agents via Reinforcement LearningA reinforcement learning system for enterprise search agents achieves superior performance through diverse training data generation and multi-task learning approaches.Databricks2 citations7 upvotesRead →

Other training and scaling papers

Topic
Training and scalingJun 17, 2021LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft25.6k citations66 upvotesRead →Training and scalingJan 23, 2020Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI9,592 citations12 upvotesRead →Training and scalingMar 29, 2022Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind4,035 citations13 upvotesRead →Training and scalingSep 17, 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA3,387 citations5 upvotesRead →Training and scalingUnverifiedJan 5, 2024DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.DeepSeek872 citations56 upvotesRead →Training and scalingUnverifiedSep 16, 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAjina-embeddings-v3, a large-scale text embedding model, achieves state-of-the-art performance in multilingual and long-context retrieval tasks using Low-Rank Adaptation and Matryoshka Representation Learning.Jina AI221 citations35 upvotesRead →
About this paper
Venue
Trans. Mach. Learn. Res.
Citations
486, 46 influential · Semantic Scholar
Upvotes
91 · Hugging Face
Lab
Databricks · on Companies · on Acquisitions · on Rounds · on Paydays · on Releases · on TechConf

Changes

CitationsNov 9, 2026Citations rose from 479 to 486.
CitationsNov 2, 2026Influential citations rose from 39 to 46.
CitationsNov 2, 2026Citations rose from 441 to 479.
CitationsOct 12, 2026Influential citations rose from 38 to 39.
CitationsOct 12, 2026Citations rose from 422 to 441.
CitationsOct 5, 2026Citations rose from 409 to 422.
CitationsSep 28, 2026Citations rose from 407 to 409.
CitationsSep 25, 2026Citations rose from 405 to 407.
New paperSep 24, 2026Added to the list.