Notable data and AI papers

Papers627Labs45

627 papers · 45 labs

Residual Context Diffusion Language Models research paper by UC Berkeley, 2026

UC Berkeley · Training and scaling · Jan 30, 2026 · 13 citations · 36 upvotes · unverified

Read on arXiv

What it shows

Residual Context Diffusion (RCD) enhances diffusion large language models by recycling discarded token information through contextual residuals, improving accuracy with minimal computational overhead.

By Yuezhou Hu, Harman Singh, Monishwaran Maheswaran and 10 more · arXiv 2601.22954 · PDF · Code

UnverifiedHugging Face's summary; not yet checked by hand.

More from UC Berkeley

All 14
ArchitecturesUnverifiedOct 14, 2026Correction Training for Diffusion ModelsThis method adds a correction step to diffusion models to fix errors that accumulate during the image generation process.UC Berkeley42 upvotesRead →Multimodal and roboticsUnverifiedSep 29, 2026OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?This paper maps how visual generation tasks improve visual understanding capabilities, revealing specific task-dependent transfer benefits.UC Berkeley57 upvotesRead →Inference and efficiencyUnverifiedAug 17, 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive ExecutionFreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines.UC Berkeley8 citations86 upvotesRead →Agents and evaluationUnverifiedJun 17, 2026Playful Agentic Robot LearningEmbodied robots learn reusable skills through self-directed play and exploration, then apply these skills to improve performance on downstream tasks without additional training.UC Berkeley37 citations49 upvotesRead →Agents and evaluationJun 3, 2026Agents' Last ExamOver 1,000 real, checkable professional tasks written with 250+ industry experts; the hardest tier is far from solved.UC Berkeley42 citations393 upvotesRead →Inference and efficiencyUnverifiedMar 10, 2026Flash-KMeans: Fast and Memory-Efficient Exact K-MeansFlash-kmeans enables efficient online k-means clustering on GPUs through novel kernel-level optimizations that eliminate I/O bottlenecks and reduce atomic write contention.UC Berkeley16 citations64 upvotesRead →Foundation modelsUnverifiedFeb 26, 2026dLLM: Simple Diffusion Language ModelingA unified open-source framework is presented that standardizes core components of diffusion language modeling for reproduction, customization, and accessible development of both large and small models.UC Berkeley18 citations136 upvotesRead →Inference and efficiencyUnverifiedFeb 13, 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATSLA2 improves sparse-linear attention in diffusion models by introducing a learnable router, direct attention formulation, and quantization-aware fine-tuning for enhanced efficiency and quality.UC Berkeley1 citations59 upvotesRead →

Other training and scaling papers

Topic
Training and scalingJun 17, 2021LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.Microsoft25.6k citations66 upvotesRead →Training and scalingJan 23, 2020Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.OpenAI9,592 citations12 upvotesRead →Training and scalingMar 29, 2022Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.Google DeepMind4,035 citations13 upvotesRead →Training and scalingSep 17, 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.NVIDIA3,387 citations5 upvotesRead →Training and scalingUnverifiedJan 5, 2024DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.DeepSeek872 citations56 upvotesRead →Training and scalingMay 15, 2024LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.Databricks486 citations91 upvotesRead →
About this paper
Venue
arXiv.org
Citations
13, 0 influential · Semantic Scholar
Upvotes
36 · Hugging Face
Lab
UC Berkeley

Changes

CitationsNov 2, 2026Citations rose from 9 to 13.
CitationsOct 5, 2026Citations rose from 8 to 9.
CitationsSep 25, 2026A first count of 0 influential citations.
CitationsSep 25, 2026A first count of 8 citations.
New paperSep 25, 2026Found by the weekly Hugging Face scan; not yet checked by hand.