Papersby GlyphSwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation research paper by Snowflake, 2024
Snowflake · Inference and efficiency · Oct 4, 2024 · 21 citations · 2 upvotes
What it shows
SwiftKV lets prompt tokens skip later layers and merges KV caches, cutting prefill cost for long-prompt workloads.
By Aurick Qiao, Zhewei Yao, Samyam Rajbhandari and 1 more · arXiv 2410.03960 · PDF
Summarised by hand from the abstract.
More from Snowflake
All 5Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document CollectionsOn 2,250 questions over 800 PDFs, the best agents match human accuracy but lean on brute-force search rather than planning.
Snowflake3 citations66 upvotesRead →Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement LearningLarge language model agents trained in synthetic environments with code-driven simulations and database-backed state transitions demonstrate superior out-of-distribution generalization compared to traditional benchmark-specific approaches.Snowflake81 citations53 upvotesRead →Language Models Model LanguageThe paper advocates for an empiricist approach to evaluating language models, emphasizing frequency of use over traditional theoretical frameworks.Snowflake26 upvotesRead →Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding ModelsOpen text embedding models from 22M to 334M parameters that led MTEB retrieval for their size at release.Snowflake84 citations2 upvotesRead →
Other inference and efficiency papers
TopicEfficient Memory Management for Large Language Model Serving with PagedAttentionvLLM manages the KV cache like pages of virtual memory, serving models with 2 to 4 times the throughput.
UC Berkeley9,965 citations72 upvotesRead →FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessFlashAttention computes exact attention with far fewer GPU memory reads and writes, making long sequences faster.
Stanford University5,830 citations16 upvotesRead →Group Sequence Policy OptimizationGroup Sequence Policy Optimization (GSPO) is a reinforcement learning algorithm that improves training efficiency and performance of large language models by using sequence-level importance ratios and operations.
Alibaba (Qwen)814 citations323 upvotesRead →Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionNSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.
DeepSeek596 citations157 upvotesRead →SmolVLM: Redefining small and efficient multimodal modelsSmolVLM, a series of compact multimodal models, achieves high performance with minimal GPU memory usage, making efficient deployment on mobile and edge devices possible.
Hugging Face330 citations212 upvotesRead →Chronos-2: From Univariate to Universal ForecastingChronos-2, a pretrained model with a group attention mechanism, achieves state-of-the-art performance in zero-shot univariate, multivariate, and covariate-informed forecasting tasks.
Amazon276 citations28 upvotesRead →
Amazon276 citations28 upvotesRead →About this paper
- Venue
- Conference on Empirical Methods in Natural Language Processing
- Citations
- 21, 1 influential · Semantic Scholar
- Upvotes
- 2 · Hugging Face
- Lab
- Snowflake · on Companies · on Acquisitions · on Quarterly · on Paydays · on Releases · on TechConf
Changes
Citations rose from 19 to 21.
Citations rose from 18 to 19.
Added to the list.