Virtual Head Attention research paper by Baidu, 2026

Baidu · Inference and efficiency · Oct 13, 2026 · 0 citations · 17 upvotes · unverified

Virtual Head Attention improves long-context model efficiency by reducing KV cache and compute while preserving quality through lightweight linear query and output transformations.

By Xibo Ding, Guoxia Wang, Jinle Zeng and 4 more · arXiv 2610.17959 · PDF · Code

UnverifiedHugging Face's summary; not yet checked by hand.

More from Baidu

All 16

Inference and efficiencyUnverifiedOct 13, 2026FlashMask-3: Efficient and Expressive Mask-Aware Distributed AttentionFlashMask-3 accelerates long-context attention via a Hopper-optimized sparse-mask kernel and distributed context-parallel extensions that eliminate mask-communication overhead and improve training throughput.

Baidu11 upvotesRead →Inference and efficiencyUnverifiedJun 22, 2026Unlimited OCR WorksUnlimited OCR introduces Reference Sliding Window Attention to eliminate growing memory consumption during long-sequence OCR tasks, enabling efficient transcription of multiple pages in a single forward pass.

Baidu9 citations90 upvotesRead →Multimodal and roboticsUnverifiedJun 12, 2026Memento: Reconstruct to Remember for Consistent Long Video GenerationMemento is a subject-reconstruction-guided framework that improves long-form video generation by preserving recurring subjects through memory-based reconstruction and dual-query mechanisms.

Baidu6 citations18 upvotesRead →Agents and evaluationUnverifiedJun 5, 2026DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded ReasoningA multi-agent framework for deep research tasks that addresses planning, evidence acquisition, and report synthesis through decoupled components and dynamic optimization mechanisms.

Baidu2 citations9 upvotesRead →Agents and evaluationUnverifiedJun 4, 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM AgentsToolMaze benchmark reveals that real-world tool failures significantly degrade TIR performance, with implicit semantic failures causing the most severe drops and dynamic replanning emerging as a key bottleneck.

Baidu28 citations27 upvotesRead →Agents and evaluationUnverifiedJun 3, 2026MapAgent: An Industrial-Grade Agentic Framework for City-scale Lane-level Map GenerationMapAgent is an industrial-grade agentic architecture that combines vision-language processing with constraint-aware reasoning to produce specification-compliant lane maps, achieving high automation rates in large-scale urban mapping.

Baidu2 citations18 upvotesRead →Alignment and safetyUnverifiedMay 28, 2026Native Audio-Visual Alignment for GenerationNAVA enables joint audio-video generation with improved synchronization and controllability through native audio-visual alignment and context-conditioned denoising.

Baidu8 citations31 upvotesRead →Inference and efficiencyUnverifiedMay 15, 2026Measuring Maximum Activations in Open Large Language ModelsModern open LLMs exhibit wide variation in activation magnitudes across families and training stages, with maximum values spanning four orders of magnitude and showing complex scaling patterns that differ from simple size-based expectations.

Baidu17 upvotesRead →

Topic

Inference and efficiencySep 12, 2023Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM manages the KV cache like pages of virtual memory, serving models with 2 to 4 times the throughput.

UC Berkeley9,965 citations72 upvotesRead →Inference and efficiencyMay 27, 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessFlashAttention computes exact attention with far fewer GPU memory reads and writes, making long sequences faster.

Stanford University5,830 citations16 upvotesRead →Inference and efficiencyUnverifiedJul 24, 2025Group Sequence Policy OptimizationGroup Sequence Policy Optimization (GSPO) is a reinforcement learning algorithm that improves training efficiency and performance of large language models by using sequence-level importance ratios and operations.

Alibaba (Qwen)814 citations323 upvotesRead →Inference and efficiencyUnverifiedFeb 16, 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionNSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.

DeepSeek596 citations157 upvotesRead →Inference and efficiencyUnverifiedApr 7, 2025SmolVLM: Redefining small and efficient multimodal modelsSmolVLM, a series of compact multimodal models, achieves high performance with minimal GPU memory usage, making efficient deployment on mobile and edge devices possible.

Hugging Face330 citations212 upvotesRead →Inference and efficiencyUnverifiedOct 17, 2025Chronos-2: From Univariate to Universal ForecastingChronos-2, a pretrained model with a group attention mechanism, achieves state-of-the-art performance in zero-shot univariate, multivariate, and covariate-informed forecasting tasks.

Amazon276 citations28 upvotesRead →

About this paper

Citations: 0, 0 influential · Semantic Scholar
Upvotes: 17 · Hugging Face
Lab: Baidu · on Companies · on Quarterly

Changes

CitationsNov 9, 2026A first count of 0 influential citations.

CitationsNov 9, 2026A first count of 0 citations.

New paperNov 9, 2026Found by the weekly Hugging Face scan; not yet checked by hand.

What it shows

More from Baidu

Other inference and efficiency papers

Changes