Papersby GlyphShaping capabilities with token-level data filtering research paper by Anthropic, 2026
Anthropic · Training and scaling · Jan 29, 2026 · 13 citations · 31 upvotes · unverified
What it shows
Token filtering during pretraining effectively reduces unwanted language model capabilities while maintaining alignment, becoming more effective at larger scales and tolerating noisy labels with sufficient compute.
By Neil Rathi, Alec Radford · arXiv 2601.21571 · PDF · Code
UnverifiedHugging Face's summary; not yet checked by hand.
More from Anthropic
All 6Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?This testbed evaluates how well AI agents can improve scientific machine learning models and identifies their reliance on engineering tricks over research insight.
Anthropic18 upvotesRead →Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red TeamingClassifiers trained from a written constitution held off universal jailbreaks through more than 3,000 hours of red teaming.Anthropic221 citations10 upvotesRead →Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Anthropic393 citations10 upvotesRead →Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Anthropic657 citations30 upvotesRead →Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.Anthropic3,987 citations4 upvotesRead →
Other training and scaling papers
TopicLoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.
Microsoft25.6k citations66 upvotesRead →Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.
OpenAI9,592 citations12 upvotesRead →Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.
Google DeepMind4,035 citations13 upvotesRead →Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.
NVIDIA3,387 citations5 upvotesRead →DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.
DeepSeek872 citations56 upvotesRead →LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.
Databricks486 citations91 upvotesRead →
About this paper
- Venue
- arXiv.org
- Citations
- 13, 1 influential · Semantic Scholar
- Upvotes
- 31 · Hugging Face
- Lab
- Anthropic · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf
Changes
Influential citations rose from 0 to 1.
Citations rose from 12 to 13.
Citations rose from 10 to 12.
A first count of 0 influential citations.
A first count of 10 citations.
Found by the weekly Hugging Face scan; not yet checked by hand.