Papersby GlyphInference and efficiency
Serving, attention kernels, KV caches, quantisation: the cost of running models. 72 papers from 23 labs.
EmbeddingGemma 2: Unified and Efficient Multimodal RepresentationsEmbeddingGemma 2 is an open multimodal embedding model that unifies text, images, audio, and video representations with efficient resource-flexible design and state-of-the-art benchmark performance.
Google34 upvotesRead →SWIFT: Self-guided Weak-to-strong dIFfusion models via Target extrapolationThis training method uses a smaller, frozen model to guide the training of a larger diffusion model, improving quality without increasing inference costs.
Alibaba (Qwen)30 upvotesRead →EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud DelegationThis paper introduces a framework that reduces inference costs by using a small local model for most tasks and delegating only critical steps to the cloud.
NVIDIA30 upvotesRead →FlashMask-3: Efficient and Expressive Mask-Aware Distributed AttentionFlashMask-3 accelerates long-context attention via a Hopper-optimized sparse-mask kernel and distributed context-parallel extensions that eliminate mask-communication overhead and improve training throughput.
Baidu11 upvotesRead →Simplifying and Improving On-Policy Distillation for Few-Step DiffusionThis paper introduces a simplified regression-based method for distilling diffusion models that improves generation speed without requiring complex training techniques.
Microsoft27 upvotesRead →Virtual Head AttentionVirtual Head Attention improves long-context model efficiency by reducing KV cache and compute while preserving quality through lightweight linear query and output transformations.Baidu17 upvotesRead →NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter ScaleNeMo-DCR enables efficient, bit-exact synchronization of trillion-parameter model checkpoints by transmitting only the sparse weight changes between clusters.NVIDIA1 citations29 upvotesRead →TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language ModelsTRACE improves reinforcement learning for large language models by aligning quantization during training and rollout to reduce performance loss in low-precision execution.Alibaba (Qwen)1 citations92 upvotesRead →VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMsThis paper introduces a method for multimodal models to adaptively allocate visual detail, reducing token usage while maintaining performance.Alibaba (Qwen)1 citations71 upvotesRead →LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient SketchesLoGRA reduces memory usage during reinforcement learning by using low-rank gradient sketches and step control to stabilize training.NVIDIA153 upvotesRead →Decoding Looped Transformers Better for (Almost) FreeLoopCD improves decoding quality in looped transformers by contrasting final predictions with earlier recurrent states without extra training.
Apple1 citations46 upvotesRead →In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video DiffusionFlashForward accelerates video diffusion by reusing in-flight KV caches and using sparse clean anchors to maintain structural consistency.
Meta1 citations34 upvotesRead →DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionA 552B mixture-of-experts model with a 1M-token context, built to shrink the KV cache for agent workloads.
DeepSeek158 citations188 upvotesRead →SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent HarnessSoL-Pi scales auto-research loops for coding agents by optimizing harness-layer mechanisms to reduce token traffic and costs.NVIDIA13 citations137 upvotesRead →Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and InferenceLayer dropout improves large language model training efficiency and enables faster inference via early exit and speculative decoding without sacrificing accuracy.
Cerebras25 upvotesRead →FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive ExecutionFreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines.
UC Berkeley8 citations86 upvotesRead →OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse PrefetchingOasisKV improves LLM inference throughput by storing full KV caches in lower memory tiers and prefetching only relevant entries into HBM using speculative-decoding lookahead predictions.Microsoft1 citations25 upvotesRead →UEmbed: Unified Sparse and Dense Multimodal EmbeddingsUEmbed is a decoder-only multimodal model that jointly produces dense and sparse embeddings in a single forward pass, extending sparse retrieval to unified text and multimodal inputs.Alibaba (Qwen)4 citations51 upvotesRead →Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation ModelMage-VL is a codec-native streaming vision-language model that uses motion-aware tokenization and a dual-system architecture to enable efficient real-time multimodal perception and reasoning.Microsoft17 citations36 upvotesRead →Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention SparsificationSol-Attn improves training-free sparse attention for diffusion transformers by combining dynamic block routing, sparse computation, and approximation correction in a single online pass to accelerate video generation without sacrificing quality.NVIDIA11 citations37 upvotesRead →SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video GenerationSANA-Video 2.0 is a hybrid video diffusion transformer that combines linear and softmax attention to generate high-resolution video efficiently on a single GPU.NVIDIA6 citations38 upvotesRead →Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and EditingMage-Flow is a compact 4B generative stack combining a lightweight latent tokenizer and a native-resolution diffusion transformer to enable efficient high-resolution text-to-image generation and instruction-based editing.Microsoft4 citations77 upvotesRead →DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive GenerationDSpark enhances LLM inference speed by combining parallel draft generation with adaptive verification that reduces waste and improves throughput in high-concurrency settings.DeepSeek77 citations49 upvotesRead →Unlimited OCR WorksUnlimited OCR introduces Reference Sliding Window Attention to eliminate growing memory consumption during long-sequence OCR tasks, enabling efficient transcription of multiple pages in a single forward pass.Baidu9 citations90 upvotesRead →FastContext: Training Efficient Repository Explorer for Coding AgentsFastContext separates repository exploration from code solving in LLM agents using specialized exploration models that reduce token consumption and improve resolution rates.Microsoft5 citations94 upvotesRead →PiD: Fast and High-Resolution Latent Decoding with Pixel DiffusionPiD introduces a pixel diffusion decoder that reformulates latent decoding as conditional pixel diffusion, enabling fast and high-quality image synthesis at high resolutions with reduced computational requirements.NVIDIA10 citations45 upvotesRead →LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMsLiteFrame, a lightweight video encoder with Compressed Token Distillation training method, reduces latency and increases frame processing capacity for long-form video understanding in Video LLMs while maintaining accuracy.
Google DeepMind2 citations26 upvotesRead →Measuring Maximum Activations in Open Large Language ModelsModern open LLMs exhibit wide variation in activation magnitudes across families and training stages, with maximum values spanning four orders of magnitude and showing complex scaling patterns that differ from simple size-based expectations.Baidu17 upvotesRead →EndPrompt: Efficient Long-Context Extension via Terminal AnchoringEndPrompt extends large language model context windows by using a two-segment training approach that preserves semantic continuity while enabling efficient long-range positional learning through sparse supervision.Baidu15 upvotesRead →SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion TransformerSANA-WM is an efficient 2.6B-parameter world model that generates high-fidelity 720p videos with precise camera control, achieving industrial-level quality while significantly reducing computational requirements through hybrid attention, dual-camera branches, two-stage generation, and robust annotation pipelines.NVIDIA70 citations92 upvotesRead →FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout ScalingA novel two-stage reinforcement learning framework called Sol-RL integrates FP4 quantization with diffusion model alignment to accelerate training while maintaining high-fidelity performance.NVIDIA3 citations29 upvotesRead →Flash-KMeans: Fast and Memory-Efficient Exact K-MeansFlash-kmeans enables efficient online k-means clustering on GPUs through novel kernel-level optimizations that eliminate I/O bottlenecks and reduce atomic write contention.UC Berkeley16 citations64 upvotesRead →ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuningResearchers address imbalance in routing weights of Mixture-of-LoRAs models by proposing Reinforcement Routing (ReMix), which uses non-learnable weights and reinforcement learning techniques to improve model expressiveness and performance.Meta3 citations26 upvotesRead →TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for RoboticsTOPReward is a probabilistically grounded temporal value function that uses pretrained video Vision-Language Models to estimate robotic task progress through internal token logits, achieving superior performance in zero-shot evaluations across diverse real-world tasks.
Ai243 citations26 upvotesRead →DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and InferenceDUET-VLM presents a dual compression framework that reduces visual tokens while maintaining high accuracy in vision-language models through coordinated vision and language backbone processing.
AMD2 citations5 upvotesRead →Experiential Reinforcement LearningExperiential Reinforcement Learning introduces an explicit experience-reflection-consolidation loop that improves learning efficiency and performance in sparse-reward environments by enabling structured behavioral revision without additional inference costs.Microsoft18 citations76 upvotesRead →CoPE-VideoLM: Codec Primitives For Efficient Video Language ModelsCoPE-VideoLM improves video language model efficiency by using video codec primitives instead of full-image encoding for most frames.Microsoft14 citations34 upvotesRead →SLA2: Sparse-Linear Attention with Learnable Routing and QATSLA2 improves sparse-linear attention in diffusion models by introducing a learnable router, direct attention formulation, and quantization-aware fine-tuning for enhanced efficiency and quality.UC Berkeley1 citations59 upvotesRead →OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every IterationOPUS is a dynamic data selection framework that improves pre-training efficiency by scoring data candidates based on optimizer-induced update projections in a stable proxy-derived target space, achieving superior performance with reduced computational overhead.Alibaba (Qwen)21 citations274 upvotesRead →Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache QuantizationQuant VideoGen addresses KV cache memory limitations in autoregressive video diffusion models through semantic-aware smoothing and progressive residual quantization, achieving significant memory reduction with minimal latency impact.UC Berkeley29 citations35 upvotesRead →Self-Hinting Language Models Enhance Reinforcement LearningSAGE is an on-policy reinforcement learning framework that enhances GRPO by injecting self-hints during training to increase outcome diversity under sparse rewards, improving alignment of large language models.Microsoft21 citations31 upvotesRead →Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKITOptiKIT is a distributed framework that automates LLM optimization processes, enabling efficient GPU resource utilization and consistent performance improvements across heterogeneous infrastructure without requiring specialized optimization expertise.
eBay2 upvotesRead →Stabilizing Reinforcement Learning with LLMs: Formulation and PracticesThe paper provides a theoretical foundation for optimizing sequence-level rewards in reinforcement learning using token-level objectives, highlighting the importance of techniques like importance sampling correction, clipping, and Routing Replay for stabilizing training, especially with large language models.Alibaba (Qwen)49 citations109 upvotesRead →ToolOrchestra: Elevating Intelligence via Efficient Model and Tool OrchestrationA small orchestrator using ToolOrchestra method coordinates various intelligent tools with reinforcement learning, achieving higher accuracy and efficiency in solving complex tasks like Humanity's Last Exam compared to larger models.NVIDIA52 citations129 upvotesRead →Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language ModelsThe study identifies key architectural factors and efficient operators to optimize small language models for real-device latency, introducing the Nemotron-Flash family for improved accuracy and efficiency.NVIDIA19 citations37 upvotesRead →RDMA Point-to-Point Communication for LLM SystemsTransferEngine provides a uniform interface for flexible point-to-point communication in large language models, supporting disaggregated inference, reinforcement learning, and Mixture-of-Experts routing across different hardware.
Perplexity16 citations8 upvotesRead →E^2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise RerankerA unified framework extends a single text embedding model to perform both retrieval and listwise reranking, achieving state-of-the-art results with low latency.Alibaba (Qwen)3 citations33 upvotesRead →DeepSeek-OCR: Contexts Optical CompressionDeepSeek-OCR uses optical 2D mapping to compress long contexts, achieving high OCR precision with reduced vision tokens and demonstrating practical value in document processing.DeepSeek200 citations98 upvotesRead →Chronos-2: From Univariate to Universal ForecastingChronos-2, a pretrained model with a group attention mechanism, achieves state-of-the-art performance in zero-shot univariate, multivariate, and covariate-informed forecasting tasks.
Amazon276 citations28 upvotesRead →EAGER: Entropy-Aware GEneRation for Adaptive Inference-Time ScalingEAGer, a training-free method, uses token-wise entropy to optimize computational resources and improve performance on complex reasoning tasks.
Cohere17 citations3 upvotesRead →
Amazon276 citations28 upvotesRead →EAGER: Entropy-Aware GEneRation for Adaptive Inference-Time ScalingEAGer, a training-free method, uses token-wise entropy to optimize computational resources and improve performance on complex reasoning tasks.