Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention research paper by DeepSeek, 2025

DeepSeek · Inference and efficiency · Feb 16, 2025 · 596 citations · 157 upvotes · unverified

NSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.

By Jingyang Yuan, Huazuo Gao, Damai Dai and 12 more · arXiv 2502.11089 · PDF

UnverifiedHugging Face's summary; not yet checked by hand.

More from DeepSeek

All 22

Inference and efficiencySep 17, 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionA 552B mixture-of-experts model with a 1M-token context, built to shrink the KV cache for agent workloads.

DeepSeek158 citations188 upvotesRead →Inference and efficiencyUnverifiedJul 6, 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive GenerationDSpark enhances LLM inference speed by combining parallel draft generation with adaptive verification that reduces waste and improves throughput in high-concurrency settings.

DeepSeek77 citations49 upvotesRead →Agents and evaluationUnverifiedFeb 25, 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM InferenceDualPath addresses KV-cache storage I/O bottlenecks in multi-turn LLM inference by introducing dual-path loading and dynamic load balancing across prefill and decode engines.

DeepSeek22 citations55 upvotesRead →Multimodal and roboticsUnverifiedJan 28, 2026DeepSeek-OCR 2: Visual Causal FlowDeepSeek-OCR 2 introduces DeepEncoder V2 that dynamically reorders visual tokens based on semantic content, enabling more human-like causal reasoning in 2D image understanding through cascaded 1D causal structures.

DeepSeek79 citations73 upvotesRead →Training and scalingUnverifiedDec 31, 2025mHC: Manifold-Constrained Hyper-ConnectionsManifold-Constrained Hyper-Connections (mHC) stabilize and scale residual connection architectures by restoring identity mapping properties through manifold projection and infrastructure optimization.

DeepSeek108 citations334 upvotesRead →Agents and evaluationUnverifiedDec 2, 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.

DeepSeek887 citations275 upvotesRead →ReasoningUnverifiedNov 27, 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical ReasoningA self-verifying large language model for theorem proving improves mathematical reasoning by incentivizing rigorous step-by-step derivations and achieves high scores in international competitions.

DeepSeek83 citations98 upvotesRead →Inference and efficiencyUnverifiedOct 21, 2025DeepSeek-OCR: Contexts Optical CompressionDeepSeek-OCR uses optical 2D mapping to compress long contexts, achieving high OCR precision with reduced vision tokens and demonstrating practical value in document processing.

DeepSeek200 citations98 upvotesRead →

Topic

Inference and efficiencySep 12, 2023Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM manages the KV cache like pages of virtual memory, serving models with 2 to 4 times the throughput.

UC Berkeley9,965 citations72 upvotesRead →Inference and efficiencyMay 27, 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessFlashAttention computes exact attention with far fewer GPU memory reads and writes, making long sequences faster.

Stanford University5,830 citations16 upvotesRead →Inference and efficiencyUnverifiedJul 24, 2025Group Sequence Policy OptimizationGroup Sequence Policy Optimization (GSPO) is a reinforcement learning algorithm that improves training efficiency and performance of large language models by using sequence-level importance ratios and operations.

Alibaba (Qwen)814 citations323 upvotesRead →Inference and efficiencyUnverifiedApr 7, 2025SmolVLM: Redefining small and efficient multimodal modelsSmolVLM, a series of compact multimodal models, achieves high performance with minimal GPU memory usage, making efficient deployment on mobile and edge devices possible.

Hugging Face330 citations212 upvotesRead →Inference and efficiencyUnverifiedOct 17, 2025Chronos-2: From Univariate to Universal ForecastingChronos-2, a pretrained model with a group attention mechanism, achieves state-of-the-art performance in zero-shot univariate, multivariate, and covariate-informed forecasting tasks.

Amazon276 citations28 upvotesRead →Inference and efficiencyUnverifiedOct 1, 2025ACON: Optimizing Context Compression for Long-horizon LLM AgentsAgent Context Optimization (ACON) compresses context in large language models for efficient long-horizon tasks by analyzing failure cases and distilling the compressor into smaller models.

Microsoft162 citations36 upvotesRead →

About this paper

Venue: Annual Meeting of the Association for Computational Linguistics
Citations: 596, 50 influential · Semantic Scholar
Upvotes: 157 · Hugging Face
Lab: DeepSeek · on Companies · on breaches · on Rounds · on labs · on Leaders · on Arena · on legislations · on Paydays · on Metered · on problems · on repos · on socials · on standards · on terms

Changes

CitationsNov 2, 2026Influential citations rose from 46 to 50.

CitationsNov 2, 2026Citations rose from 550 to 596.

CitationsOct 12, 2026Citations rose from 531 to 550.

CitationsOct 5, 2026Influential citations rose from 44 to 46.

CitationsOct 5, 2026Citations rose from 508 to 531.

CitationsSep 28, 2026Influential citations rose from 43 to 44.

CitationsSep 28, 2026Citations rose from 507 to 508.

CitationsSep 25, 2026A first count of 43 influential citations.

CitationsSep 25, 2026A first count of 507 citations.

New paperSep 25, 2026Found by the weekly Hugging Face scan; not yet checked by hand.

What it shows

More from DeepSeek

Other inference and efficiency papers

Changes