Skip to content
Papers.

Residual Context Diffusion Language Models research paper by UC Berkeley, 2026

UC Berkeley · Jan 30, 2026 · Training and scaling · 8 citations · 36 upvotes · unverified

Read on arXiv

What it shows

Residual Context Diffusion (RCD) enhances diffusion large language models by recycling discarded token information through contextual residuals, improving accuracy with minimal computational overhead.

UnverifiedHugging Face's summary; not yet checked by hand.

More from UC Berkeley

All 12
PaperCitations
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive ExecutionFreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines.Inference and efficiency · Aug 2026 · Unverified2
Playful Agentic Robot LearningEmbodied robots learn reusable skills through self-directed play and exploration, then apply these skills to improve performance on downstream tasks without additional training.Agents and evaluation · Jun 2026 · Unverified7
Agents' Last ExamOver 1,000 real, checkable professional tasks written with 250+ industry experts; the hardest tier is far from solved.Agents and evaluation · Jun 202615
Flash-KMeans: Fast and Memory-Efficient Exact K-MeansFlash-kmeans enables efficient online k-means clustering on GPUs through novel kernel-level optimizations that eliminate I/O bottlenecks and reduce atomic write contention.Inference and efficiency · Mar 2026 · Unverified6
dLLM: Simple Diffusion Language ModelingA unified open-source framework is presented that standardizes core components of diffusion language modeling for reproduction, customization, and accessible development of both large and small models.Foundation models · Feb 2026 · Unverified12
SLA2: Sparse-Linear Attention with Learnable Routing and QATSLA2 improves sparse-linear attention in diffusion models by introducing a learnable router, direct attention formulation, and quantization-aware fine-tuning for enhanced efficiency and quality.Inference and efficiency · Feb 2026 · Unverified20
Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache QuantizationQuant VideoGen addresses KV cache memory limitations in autoregressive video diffusion models through semantic-aware smoothing and progressive residual quantization, achieving significant memory reduction with minimal latency impact.Inference and efficiency · Feb 2026 · Unverified16
Latent Implicit Visual ReasoningWhile Large Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on language as their core reasoning modality.Reasoning · Dec 2025 · Unverified10

Other training and scaling papers

Topic
PaperCitations
LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft · Jun 202123.3k
Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI · Jan 20209,065
Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind · Mar 20223,756
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA · Sep 20193,179
DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.DeepSeek · Jan 2024 · Unverified855
LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks · May 2024407
About this paper
Authors
Yuezhou Hu, Harman Singh, Monishwaran Maheswaran and 10 more
arXiv
2601.22954 · PDF
Venue
arXiv.org
Citations
8, 0 influential · Semantic Scholar
Upvotes
36 · Hugging Face
Code
github.com/yuezhouhu/residual-context-diffusion
Lab
UC Berkeley

Changes

What changed
Influential citationsfirst count: 0Sep 25, 2026
Citationsfirst count: 8Sep 25, 2026
New paperFound by the weekly scan, unverifiedSep 25, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.