Papersby GlyphWhy Language Models Hallucinate research paper by OpenAI, 2025
OpenAI · Alignment and safety · Sep 4, 2025 · 391 citations · 200 upvotes
What it shows
Models hallucinate because training and benchmarks reward confident guessing over saying they do not know.
By Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and 1 more · arXiv 2509.04664 · PDF
Summarised by hand from the abstract.
More from OpenAI
All 8Reasoning Models Struggle to Control their Chains of ThoughtChain-of-thought controllability measures how effectively models can be constrained to follow reasoning steps, with findings showing significantly lower controllability in reasoning versus output generation, and varying impacts from model size, training methods, and task complexity.
OpenAI37 citations41 upvotesRead →OpenAI o1 System CardThe o1 model series, trained with reinforcement learning and chain of thought, enhances safety and robustness by reasoning about policies, leading to superior performance on risk benchmarks while highlighting the need for robust alignment and risk management.OpenAI38 upvotesRead →GPT-4o System CardGPT-4o is an omnimodal autoregressive model trained to handle text, audio, image, and video inputs, offering high-performance outputs across these modalities, with particular strengths in vision and audio.OpenAI5,388 citations88 upvotesRead →GPT-4 Technical ReportA multimodal model that passes a simulated bar exam with a score around the top 10% of test takers.OpenAI28.4k citations7 upvotesRead →Training language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.OpenAI25.8k citations26 upvotesRead →Language Models are Few-Shot LearnersGPT-3, a 175B-parameter model, does new tasks from a few examples in the prompt, with no fine-tuning.OpenAI66.4k citations21 upvotesRead →Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI9,592 citations12 upvotesRead →
Other alignment and safety papers
TopicDirect Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns a model to human preferences with a simple classification loss, no reward model or RL loop.
Stanford University11.6k citations73 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.
Anthropic3,987 citations4 upvotesRead →Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsReinforcement Learning from Human Feedback (RLHF) benefits from simpler REINFORCE-style optimization methods over computationally expensive PPO or "RL-free" methods, enhancing performance and reducing cost in large language models.
Cohere804 citations18 upvotesRead →Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Anthropic657 citations30 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL OptimizationMulti-reward reinforcement learning suffers from reward normalization collapse in GRPO, which GDPO addresses by decoupling reward normalization for improved training stability and performance across reasoning tasks.
NVIDIA234 citations235 upvotesRead →
About this paper
- Venue
- arXiv.org
- Citations
- 391, 23 influential · Semantic Scholar
- Upvotes
- 200 · Hugging Face
- Lab
- OpenAI · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf
Changes
Citations rose from 386 to 391.
Influential citations rose from 21 to 23.
Citations rose from 363 to 386.
Influential citations rose from 20 to 21.
Citations rose from 353 to 363.
Influential citations rose from 19 to 20.
Citations rose from 335 to 353.
Citations rose from 331 to 335.
Citations rose from 329 to 331.
Added to the list.