Papersby GlyphBack to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs research paper by Cohere, 2024
Cohere · Alignment and safety · Feb 22, 2024 · 804 citations · 18 upvotes
What it shows
Reinforcement Learning from Human Feedback (RLHF) benefits from simpler REINFORCE-style optimization methods over computationally expensive PPO or "RL-free" methods, enhancing performance and reducing cost in large language models.
By Arash Ahmadian, Chris Cremer, Matthias Gallé and 4 more · arXiv 2402.14740 · PDF
Hugging Face's summary; not yet checked by hand.
More from Cohere
All 30Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language ReasoningOptimized supervised fine-tuning data composition enables reasoning models to consistently process and respond in diverse non-English languages without requiring reasoning supervision in each target language.
Cohere33 upvotesRead →The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic ReasoningAn open competition on unseen Linguistics Olympiad puzzles, graded by the official jury; small 14B systems beat models twice their size.Cohere1 citationsRead →CALIBER: Calibrating Confidence Before and After Reasoning in Language ModelsTrains reasoning models to state their confidence twice, before thinking and after answering, halving calibration error against the best single-estimate method.Cohere1 citationsRead →AI Exposure Scores: what they measure, what they miss, and what comes nextReviews how widely cited scores of which jobs AI can assist are used in policy, what they miss, and newer measures that address it.CohereRead →The Culture Funnel: You Can't Align What isn't in the DataModern LLM pipelines experience a cultural data funnel where explicit cultural signals diminish during post-training, necessitating shifts in training data approaches for better cultural alignment.Cohere1 citationsRead →Soft-SVeRL: Self-Verified Reinforcement Learning with Soft RewardsTurns each prompt into a checklist scored item by item, giving reinforcement learning partial-credit rewards for tasks that cannot be checked automatically.Cohere1 citationsRead →Agents Explore but Agents Ignore: LLMs Lack Environmental CuriosityLLM-based agents fail to exploit discovered unexpected information despite recognizing it, indicating a lack of environmental curiosity that depends on tools, compute, and training data distribution.Cohere2 citations8 upvotesRead →Tiny Aya: Bridging Scale and Multilingual DepthTiny Aya demonstrates high-quality multilingual capabilities with 3.35 billion parameters through region-aware posttraining and balanced language performance.Cohere18 citations10 upvotesRead →
Other alignment and safety papers
TopicTraining language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.
OpenAI25.8k citations26 upvotesRead →Direct Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns a model to human preferences with a simple classification loss, no reward model or RL loop.
Stanford University11.6k citations73 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.
Anthropic3,987 citations4 upvotesRead →Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Anthropic657 citations30 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.OpenAI391 citations200 upvotesRead →
About this paper
- Citations
- 804, 87 influential · Semantic Scholar
- Upvotes
- 18 · Hugging Face
- Lab
- Cohere · on Companies · on Acquisitions · on Releases
Changes
Influential citations rose from 84 to 87.
Citations rose from 780 to 804.
A first count of 84 influential citations.
A first count of 780 citations.
Added to the list.