Papersby GlyphSleeper Agents: Training Deceptive LLMs that Persist Through Safety Training research paper by Anthropic, 2024
Anthropic · Alignment and safety · Jan 10, 2024 · 657 citations · 30 upvotes
What it shows
Models trained with a hidden backdoor kept their deceptive behaviour through standard safety training.
By Evan Hubinger, Carson Denison, Jesse Mu and 36 more · arXiv 2401.05566 · PDF · Code
Summarised by hand from the abstract.
More from Anthropic
All 6Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?This testbed evaluates how well AI agents can improve scientific machine learning models and identifies their reliance on engineering tricks over research insight.
Anthropic18 upvotesRead →Shaping capabilities with token-level data filteringToken filtering during pretraining effectively reduces unwanted language model capabilities while maintaining alignment, becoming more effective at larger scales and tolerating noisy labels with sufficient compute.Anthropic13 citations31 upvotesRead →Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red TeamingClassifiers trained from a written constitution held off universal jailbreaks through more than 3,000 hours of red teaming.Anthropic221 citations10 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.Anthropic3,987 citations4 upvotesRead →
Other alignment and safety papers
TopicTraining language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.
OpenAI25.8k citations26 upvotesRead →Direct Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns a model to human preferences with a simple classification loss, no reward model or RL loop.
Stanford University11.6k citations73 upvotesRead →Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsReinforcement Learning from Human Feedback (RLHF) benefits from simpler REINFORCE-style optimization methods over computationally expensive PPO or "RL-free" methods, enhancing performance and reducing cost in large language models.
Cohere804 citations18 upvotesRead →Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.OpenAI391 citations200 upvotesRead →GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL OptimizationMulti-reward reinforcement learning suffers from reward normalization collapse in GRPO, which GDPO addresses by decoupling reward normalization for improved training stability and performance across reasoning tasks.
NVIDIA234 citations235 upvotesRead →Aya Expanse: Combining Research Breakthroughs for a New Multilingual FrontierA new family of 8B and 32B parameter multilingual language models, Aya Expanse, achieves state-of-the-art performance across multiple languages and outperforms larger models in evaluations.Cohere153 citations9 upvotesRead →
About this paper
- Venue
- arXiv.org
- Citations
- 657, 61 influential · Semantic Scholar
- Upvotes
- 30 · Hugging Face
- Lab
- Anthropic · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf
Changes
Citations rose from 647 to 657.
Influential citations rose from 59 to 61.
Citations rose from 616 to 647.
Influential citations rose from 55 to 59.
Citations rose from 595 to 616.
Influential citations rose from 54 to 55.
Citations rose from 580 to 595.
Influential citations rose from 53 to 54.
Citations rose from 578 to 580.
Influential citations rose from 52 to 53.
Citations rose from 576 to 578.
Added to the list.