Papersby GlyphAlignment and safety
Making models follow intent and resist misuse: RLHF, preferences, jailbreaks, deception, hallucination. 32 papers from 15 labs.
OTel: Open Telco AI Datasets, Benchmarks, and ModelsOpen Telco provides open telecom AI datasets and post-trained baselines for retrieval, reranking, and instruction-tuned language models, improving benchmark performance across all three families.
AT&T1 citationsRead →Last Secure Code Benchmark: Is the Code You Vibe Secure?This paper introduces a benchmark showing that current coding agents frequently produce functional but insecure code, highlighting significant security risks in automated coding.
Alibaba (Qwen)77 upvotesRead →ActiveSaddler: Automated Curriculum Learning for Agent Harness OptimizationActiveSaddler optimizes agent training by dynamically adapting the curriculum of scenarios based on discovered failure patterns.
Microsoft86 upvotesRead →Sharpening Tax in Post-TrainingThis paper introduces a metric to quantify the loss in solution coverage after reinforcement learning and proposes a sampler to mitigate it.
Meta1 citations111 upvotesRead →VeriHarness: Scaling Agentic Verification for Long-Horizon TasksVeriHarness improves long-horizon agent performance by using the base model as a verifier to resolve conflicting claims and challenge consensus.
Google1 citations62 upvotesRead →Mid-Harness: Scaling Actions Between Model and Harness for Terminal AgentsMid-Harness improves agent reliability by sampling and verifying candidate actions before execution, rather than relying on a single generation.
NVIDIA3 citations120 upvotesRead →OTel: Building Domain-Specialized Telecom LLM Foundations for Intelligent NetworksOpen Telco provides open datasets and trained models for telecom AI tasks, improving retrieval, reranking, and language model performance.AT&TRead →AISPA: User-Centric System Prompt Auditing for Large Language Model ApplicationsAISPA is a framework for auditing system prompts in commercial AI products to ensure they are protective of users.
Stanford University36 upvotesRead →Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer DecodingAutoregressive generation in large language models traditionally uses the final layer for token prediction, but a new decoding strategy dynamically selects more reliable intermediate layers based on entropy-guided search, improving reasoning performance with minimal computational overhead.Alibaba (Qwen)3 citations27 upvotesRead →The Culture Funnel: You Can't Align What isn't in the DataModern LLM pipelines experience a cultural data funnel where explicit cultural signals diminish during post-training, necessitating shifts in training data approaches for better cultural alignment.
Cohere1 citationsRead →Native Audio-Visual Alignment for GenerationNAVA enables joint audio-video generation with improved synchronization and controllability through native audio-visual alignment and context-conditioned denoising.
Baidu8 citations31 upvotesRead →RefAlign: Representation Alignment for Reference-to-Video GenerationReference-to-video generation is enhanced through explicit feature alignment between diffusion transformers and visual foundation models, improving identity consistency and semantic discriminability while maintaining text controllability.Baidu9 citations2 upvotesRead →SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video EditingSAMA presents a factorized approach to video editing that separates semantic anchoring from motion modeling, enabling instruction-guided edits with preserved motion through pre-trained motion restoration tasks and sparse anchor frame prediction.Baidu9 citations38 upvotesRead →Voxtral RealtimeVoxtral Realtime is a streaming speech recognition model trained end-to-end for sub-second latency with performance matching offline systems.
Mistral AI10 citations28 upvotesRead →GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL OptimizationMulti-reward reinforcement learning suffers from reward normalization collapse in GRPO, which GDPO addresses by decoupling reward normalization for improved training stability and performance across reasoning tasks.NVIDIA234 citations235 upvotesRead →Black-Box On-Policy Distillation of Large Language ModelsGenerative Adversarial Distillation (GAD) enhances black-box distillation by framing the student model as a generator and using a discriminator to provide adaptive feedback, surpassing traditional sequence-level knowledge distillation.Microsoft63 citations54 upvotesRead →Instella: Fully Open Language Models with Stellar PerformanceInstella, a family of fully open large language models, achieves state-of-the-art performance using open data and is competitive with leading open-weight models, with specialized variants for long context and mathematical reasoning.
AMD7 citations7 upvotesRead →FineVision: Open Data Is All You NeedFineVision, a large-scale and curated dataset, enhances vision-language models through rigorous data collection, de-duplication, and human oversight, leading to improved performance.
Hugging Face53 citations81 upvotesRead →DocReward: A Document Reward Model for Structuring and StylizingDocReward, a document reward model, evaluates and enhances the structural and stylistic quality of generated documents, outperforming GPT-4o and GPT-5 in both accuracy and human-preferred document generation.Microsoft1 citations27 upvotesRead →The Alignment Waltz: Jointly Training Agents to Collaborate for SafetyWaltzRL, a multi-agent reinforcement learning framework, improves LLM safety and helpfulness by collaboratively training a conversation agent and a feedback agent, reducing unsafe responses and overrefusals.Meta13 citations41 upvotesRead →Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.
OpenAI391 citations200 upvotesRead →Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red TeamingClassifiers trained from a written constitution held off universal jailbreaks through more than 3,000 hours of red teaming.
Anthropic221 citations10 upvotesRead →OpenAI o1 System CardThe o1 model series, trained with reinforcement learning and chain of thought, enhances safety and robustness by reasoning about policies, leading to superior performance on risk benchmarks while highlighting the need for robust alignment and risk management.OpenAI38 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →Aya Expanse: Combining Research Breakthroughs for a New Multilingual FrontierA new family of 8B and 32B parameter multilingual language models, Aya Expanse, achieves state-of-the-art performance across multiple languages and outperforms larger models in evaluations.Cohere153 citations9 upvotesRead →Self-Improving Robust Preference OptimizationSRPO, a self-improving offline RLHF framework, achieves robustness to out-of-distribution tasks by optimizing a min-max objective that jointly enhances self-improvement and generative policies, leading to superior performance compared to DPO.Cohere25 citations19 upvotesRead →Jina CLIP: Your CLIP Model Is Also Your Text RetrieverA novel multi-task contrastive training method improves CLIP model performance on both text-image and text-text retrieval tasks.
Jina AI61 citations37 upvotesRead →Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsReinforcement Learning from Human Feedback (RLHF) benefits from simpler REINFORCE-style optimization methods over computationally expensive PPO or "RL-free" methods, enhancing performance and reducing cost in large language models.Cohere804 citations18 upvotesRead →Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Anthropic657 citations30 upvotesRead →Direct Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns a model to human preferences with a simple classification loss, no reward model or RL loop.Stanford University11.6k citations73 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.Anthropic3,987 citations4 upvotesRead →Training language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.OpenAI25.8k citations26 upvotesRead →