Skip to content
Papers.

Shaping capabilities with token-level data filtering research paper by Anthropic, 2026

Anthropic · Jan 29, 2026 · Training and scaling · 10 citations · 31 upvotes · unverified

Read on arXiv

What it shows

Token filtering during pretraining effectively reduces unwanted language model capabilities while maintaining alignment, becoming more effective at larger scales and tolerating noisy labels with sufficient compute.

UnverifiedHugging Face's summary; not yet checked by hand.

More from Anthropic

All 5
PaperCitations
Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red TeamingClassifiers trained from a written constitution held off universal jailbreaks through more than 3,000 hours of red teaming.Alignment and safety · Jan 2025199
Alignment faking in large language modelsClaude 3 Opus sometimes went along with a training goal it disagreed with, to avoid being changed, without being told to.Alignment and safety · Dec 2024343
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingModels trained with a hidden backdoor kept their deceptive behaviour through standard safety training.Alignment and safety · Jan 2024578
Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains a harmless assistant from AI feedback guided by a short list of written principles, not human harm labels.Alignment and safety · Dec 20223,721

Other training and scaling papers

Topic
PaperCitations
LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 202123.3k
Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 20209,065
Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 20223,756
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 20193,179
DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.DeepSeek · Jan 2024 · Unverified855
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024407
About this paper
Authors
Neil Rathi, Alec Radford
arXiv
2601.21571 · PDF
Venue
arXiv.org
Citations
10, 0 influential · Semantic Scholar
Upvotes
31 · Hugging Face
Code
github.com/neilrathi/token-filtering
Lab
Anthropic · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf

Changes

What changed
Influential citationsfirst count: 0Sep 25, 2026
Citationsfirst count: 10Sep 25, 2026
New paperFound by the weekly scan, unverifiedSep 25, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.