Papersby GlyphOpenAI o1 System Card research paper by OpenAI, 2024
OpenAI · Alignment and safety · Dec 21, 2024 · 0 citations · 38 upvotes · unverified
What it shows
The o1 model series, trained with reinforcement learning and chain of thought, enhances safety and robustness by reasoning about policies, leading to superior performance on risk benchmarks while highlighting the need for robust alignment and risk management.
By OpenAI, Aaron Jaech, Adam Kalai and 262 more · arXiv 2412.16720 · PDF
UnverifiedHugging Face's summary; not yet checked by hand.
More from OpenAI
All 8Reasoning Models Struggle to Control their Chains of ThoughtChain-of-thought controllability measures how effectively models can be constrained to follow reasoning steps, with findings showing significantly lower controllability in reasoning versus output generation, and varying impacts from model size, training methods, and task complexity.
OpenAI37 citations41 upvotesRead →Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.OpenAI391 citations200 upvotesRead →GPT-4o System CardGPT-4o is an omnimodal autoregressive model trained to handle text, audio, image, and video inputs, offering high-performance outputs across these modalities, with particular strengths in vision and audio.OpenAI5,388 citations88 upvotesRead →GPT-4 Technical ReportA multimodal model that passes a simulated bar exam with a score around the top 10% of test takers.OpenAI28.4k citations7 upvotesRead →Training language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.OpenAI25.8k citations26 upvotesRead →Language Models are Few-Shot LearnersGPT-3, a 175B-parameter model, does new tasks from a few examples in the prompt, with no fine-tuning.OpenAI66.4k citations21 upvotesRead →Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI9,592 citations12 upvotesRead →
Other alignment and safety papers
TopicDirect Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns a model to human preferences with a simple classification loss, no reward model or RL loop.
Stanford University11.6k citations73 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.
Anthropic3,987 citations4 upvotesRead →Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsReinforcement Learning from Human Feedback (RLHF) benefits from simpler REINFORCE-style optimization methods over computationally expensive PPO or "RL-free" methods, enhancing performance and reducing cost in large language models.
Cohere804 citations18 upvotesRead →Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Anthropic657 citations30 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL OptimizationMulti-reward reinforcement learning suffers from reward normalization collapse in GRPO, which GDPO addresses by decoupling reward normalization for improved training stability and performance across reasoning tasks.
NVIDIA234 citations235 upvotesRead →
About this paper
- Citations
- 0, 0 influential · Semantic Scholar
- Upvotes
- 38 · Hugging Face
- Lab
- OpenAI · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf
Changes
A first count of 0 influential citations.
A first count of 0 citations.
Found by the weekly Hugging Face scan; not yet checked by hand.