Notable data and AI papers

Papers627Labs45

627 papers · 45 labs

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models research paper by DeepSeek, 2024

DeepSeek · Architectures · Jan 11, 2024 · 1,268 citations · 65 upvotes · unverified

Read on arXiv

What it shows

The DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.

By Damai Dai, Chengqi Deng, Chenggang Zhao and 14 more · arXiv 2401.06066 · PDF · Code

UnverifiedHugging Face's summary; not yet checked by hand.

More from DeepSeek

All 22
Inference and efficiencySep 17, 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionA 552B mixture-of-experts model with a 1M-token context, built to shrink the KV cache for agent workloads.DeepSeek158 citations188 upvotesRead →Inference and efficiencyUnverifiedJul 6, 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive GenerationDSpark enhances LLM inference speed by combining parallel draft generation with adaptive verification that reduces waste and improves throughput in high-concurrency settings.DeepSeek77 citations49 upvotesRead →Agents and evaluationUnverifiedFeb 25, 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM InferenceDualPath addresses KV-cache storage I/O bottlenecks in multi-turn LLM inference by introducing dual-path loading and dynamic load balancing across prefill and decode engines.DeepSeek22 citations55 upvotesRead →Multimodal and roboticsUnverifiedJan 28, 2026DeepSeek-OCR 2: Visual Causal FlowDeepSeek-OCR 2 introduces DeepEncoder V2 that dynamically reorders visual tokens based on semantic content, enabling more human-like causal reasoning in 2D image understanding through cascaded 1D causal structures.DeepSeek79 citations73 upvotesRead →Training and scalingUnverifiedDec 31, 2025mHC: Manifold-Constrained Hyper-ConnectionsManifold-Constrained Hyper-Connections (mHC) stabilize and scale residual connection architectures by restoring identity mapping properties through manifold projection and infrastructure optimization.DeepSeek108 citations334 upvotesRead →Agents and evaluationUnverifiedDec 2, 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.DeepSeek887 citations275 upvotesRead →ReasoningUnverifiedNov 27, 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical ReasoningA self-verifying large language model for theorem proving improves mathematical reasoning by incentivizing rigorous step-by-step derivations and achieves high scores in international competitions.DeepSeek83 citations98 upvotesRead →Inference and efficiencyUnverifiedOct 21, 2025DeepSeek-OCR: Contexts Optical CompressionDeepSeek-OCR uses optical 2D mapping to compress long contexts, achieving high OCR precision with reduced vision tokens and demonstrating practical value in document processing.DeepSeek200 citations98 upvotesRead →

Other architectures papers

Topic
ArchitecturesJun 12, 2017Attention Is All You NeedIntroduced the Transformer, the attention-only architecture behind nearly every large language model since.Google201k citations142 upvotesRead →ArchitecturesDec 1, 2023Mamba: Linear-Time Sequence Modeling with Selective State SpacesA selective state space model that scales linearly with sequence length and matches Transformers on language.Carnegie Mellon University10.2k citations152 upvotesRead →ArchitecturesSep 2, 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationCodeT5, a unified encoder-decoder Transformer model, improves code understanding and generation by leveraging semantic information from identifiers and user comments.Salesforce2,633 citations3 upvotesRead →ArchitecturesUnverifiedMar 14, 2025VGGT: Visual Geometry Grounded TransformerVGGT, a feed-forward neural network, efficiently infers multiple 3D attributes from single or multiple views, outperforming alternatives and enhancing downstream tasks without post-processing.Meta2,132 citations40 upvotesRead →ArchitecturesMay 13, 2023CodeT5+: Open Code Large Language Models for Code Understanding and GenerationCodeT5+, a family of flexible encoder-decoder LLMs initialized with frozen LLMs and trained with a mixture of pretraining objectives, achieves state-of-the-art performance across various code-related tasks.Salesforce836 citations7 upvotesRead →ArchitecturesUnverifiedOct 9, 2019HuggingFace's Transformers: State-of-the-art Natural Language ProcessingTransformers library provides state-of-the-art Transformer architectures and pretrained models for natural language processing tasks with a unified API and emphasis on extensibility and robust deployment.Hugging Face251 citations34 upvotesRead →
About this paper
Venue
Annual Meeting of the Association for Computational Linguistics
Citations
1,268, 132 influential · Semantic Scholar
Upvotes
65 · Hugging Face
Lab
DeepSeek · on Companies · on breaches · on Rounds · on labs · on Leaders · on Arena · on legislations · on Paydays · on Metered · on problems · on repos · on socials · on standards · on terms

Changes

CitationsNov 2, 2026Influential citations rose from 125 to 132.
CitationsNov 2, 2026Citations rose from 1,199 to 1,268.
CitationsOct 12, 2026Citations rose from 1,168 to 1,199.
CitationsOct 5, 2026Influential citations rose from 123 to 125.
CitationsOct 5, 2026Citations rose from 1,127 to 1,168.
CitationsSep 28, 2026Citations rose from 1,123 to 1,127.
CitationsSep 25, 2026A first count of 123 influential citations.
CitationsSep 25, 2026A first count of 1,123 citations.
New paperSep 25, 2026Found by the weekly Hugging Face scan; not yet checked by hand.