Papersby GlyphWhen Less is More: Investigating Data Pruning for Pretraining LLMs at Scale research paper by Cohere, 2023
Cohere · Training and scaling · Sep 8, 2023 · 144 citations · 17 upvotes
What it shows
Perplexity outperforms more complex methods for data quality estimation in pruning large language model training datasets, resulting in performance improvement with reduced data.
By Max Marion, Ahmet Üstün, Luiza Pozzobon and 3 more · arXiv 2309.04564 · PDF
Hugging Face's summary; not yet checked by hand.
More from Cohere
All 30Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language ReasoningOptimized supervised fine-tuning data composition enables reasoning models to consistently process and respond in diverse non-English languages without requiring reasoning supervision in each target language.
Cohere33 upvotesRead →The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic ReasoningAn open competition on unseen Linguistics Olympiad puzzles, graded by the official jury; small 14B systems beat models twice their size.Cohere1 citationsRead →CALIBER: Calibrating Confidence Before and After Reasoning in Language ModelsTrains reasoning models to state their confidence twice, before thinking and after answering, halving calibration error against the best single-estimate method.Cohere1 citationsRead →AI Exposure Scores: what they measure, what they miss, and what comes nextReviews how widely cited scores of which jobs AI can assist are used in policy, what they miss, and newer measures that address it.CohereRead →The Culture Funnel: You Can't Align What isn't in the DataModern LLM pipelines experience a cultural data funnel where explicit cultural signals diminish during post-training, necessitating shifts in training data approaches for better cultural alignment.Cohere1 citationsRead →Soft-SVeRL: Self-Verified Reinforcement Learning with Soft RewardsTurns each prompt into a checklist scored item by item, giving reinforcement learning partial-credit rewards for tasks that cannot be checked automatically.Cohere1 citationsRead →Agents Explore but Agents Ignore: LLMs Lack Environmental CuriosityLLM-based agents fail to exploit discovered unexpected information despite recognizing it, indicating a lack of environmental curiosity that depends on tools, compute, and training data distribution.Cohere2 citations8 upvotesRead →Tiny Aya: Bridging Scale and Multilingual DepthTiny Aya demonstrates high-quality multilingual capabilities with 3.35 billion parameters through region-aware posttraining and balanced language performance.Cohere18 citations10 upvotesRead →
Other training and scaling papers
TopicLoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.
Microsoft25.6k citations66 upvotesRead →Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.
OpenAI9,592 citations12 upvotesRead →Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.
Google DeepMind4,035 citations13 upvotesRead →Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.
NVIDIA3,387 citations5 upvotesRead →DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.
DeepSeek872 citations56 upvotesRead →LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.
Databricks486 citations91 upvotesRead →
About this paper
- Citations
- 144, 13 influential · Semantic Scholar
- Upvotes
- 17 · Hugging Face
- Lab
- Cohere · on Companies · on Acquisitions · on Releases
Changes
Citations rose from 124 to 144.
A first count of 13 influential citations.
A first count of 124 citations.
Added to the list.