Papersby GlyphEfficient Memory Management for Large Language Model Serving with PagedAttention research paper by UC Berkeley, 2023
UC Berkeley · Inference and efficiency · Sep 12, 2023 · 9,965 citations · 72 upvotes
What it shows
vLLM manages the KV cache like pages of virtual memory, serving models with 2 to 4 times the throughput.
By Woosuk Kwon, Zhuohan Li, Siyuan Zhuang and 6 more · arXiv 2309.06180 · PDF · Code
Summarised by hand from the abstract.
More from UC Berkeley
All 14Correction Training for Diffusion ModelsThis method adds a correction step to diffusion models to fix errors that accumulate during the image generation process.
UC Berkeley42 upvotesRead →OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?This paper maps how visual generation tasks improve visual understanding capabilities, revealing specific task-dependent transfer benefits.UC Berkeley57 upvotesRead →FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive ExecutionFreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines.UC Berkeley8 citations86 upvotesRead →Playful Agentic Robot LearningEmbodied robots learn reusable skills through self-directed play and exploration, then apply these skills to improve performance on downstream tasks without additional training.UC Berkeley37 citations49 upvotesRead →Agents' Last ExamOver 1,000 real, checkable professional tasks written with 250+ industry experts; the hardest tier is far from solved.UC Berkeley42 citations393 upvotesRead →Flash-KMeans: Fast and Memory-Efficient Exact K-MeansFlash-kmeans enables efficient online k-means clustering on GPUs through novel kernel-level optimizations that eliminate I/O bottlenecks and reduce atomic write contention.UC Berkeley16 citations64 upvotesRead →dLLM: Simple Diffusion Language ModelingA unified open-source framework is presented that standardizes core components of diffusion language modeling for reproduction, customization, and accessible development of both large and small models.UC Berkeley18 citations136 upvotesRead →SLA2: Sparse-Linear Attention with Learnable Routing and QATSLA2 improves sparse-linear attention in diffusion models by introducing a learnable router, direct attention formulation, and quantization-aware fine-tuning for enhanced efficiency and quality.UC Berkeley1 citations59 upvotesRead →
Releases that name it
From releases.fru.devOther inference and efficiency papers
TopicFlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessFlashAttention computes exact attention with far fewer GPU memory reads and writes, making long sequences faster.
Stanford University5,830 citations16 upvotesRead →Group Sequence Policy OptimizationGroup Sequence Policy Optimization (GSPO) is a reinforcement learning algorithm that improves training efficiency and performance of large language models by using sequence-level importance ratios and operations.
Alibaba (Qwen)814 citations323 upvotesRead →Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionNSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.
DeepSeek596 citations157 upvotesRead →SmolVLM: Redefining small and efficient multimodal modelsSmolVLM, a series of compact multimodal models, achieves high performance with minimal GPU memory usage, making efficient deployment on mobile and edge devices possible.
Hugging Face330 citations212 upvotesRead →Chronos-2: From Univariate to Universal ForecastingChronos-2, a pretrained model with a group attention mechanism, achieves state-of-the-art performance in zero-shot univariate, multivariate, and covariate-informed forecasting tasks.
Amazon276 citations28 upvotesRead →Inference-Time Scaling for Generalist Reward ModelingSelf-Principled Critique Tuning enhances pointwise generative reward modeling for large language models, improving scalability and quality compared to existing methods.DeepSeek259 citations59 upvotesRead →
Amazon276 citations28 upvotesRead →Inference-Time Scaling for Generalist Reward ModelingSelf-Principled Critique Tuning enhances pointwise generative reward modeling for large language models, improving scalability and quality compared to existing methods.About this paper
- Venue
- Symposium on Operating Systems Principles
- Citations
- 9,965, 1,375 influential · Semantic Scholar
- Upvotes
- 72 · Hugging Face
- Lab
- UC Berkeley
Changes
Influential citations rose from 1,331 to 1,375.
Citations rose from 9,683 to 9,965.
Influential citations rose from 1,240 to 1,331.
Citations rose from 9,009 to 9,683.
Influential citations rose from 1,210 to 1,240.
Citations rose from 8,815 to 9,009.
Influential citations rose from 1,172 to 1,210.
Citations rose from 8,538 to 8,815.
Influential citations rose from 1,166 to 1,172.
Citations rose from 8,481 to 8,538.
Influential citations rose from 1,165 to 1,166.
Citations rose from 8,469 to 8,481.
Added to the list.