Papersby GlyphArchitectures
The shapes of models: Transformers, state space models, mixture of experts. 23 papers from 14 labs.
Correction Training for Diffusion ModelsThis method adds a correction step to diffusion models to fix errors that accumulate during the image generation process.
UC Berkeley42 upvotesRead →Learning Representations to Guide Pixel Diffusion TransformersThis paper introduces a guidance method for diffusion transformers that uses compact internal representations to improve image generation quality and efficiency.
Alibaba (Qwen)34 upvotesRead →ALoDLM: Adaptively Looped Diffusion Language ModelsALoDLM improves diffusion language models by using token-adaptive latent recurrence to allocate more computation to difficult tokens.
Amazon1 citations80 upvotesRead →Context Language ModelsContext Language Models manage their own context by treating it as an editable file, improving performance and efficiency.
Meta5 citations47 upvotesRead →Flattening Every Memory Peak in Long-Context Mixture-of-Experts TrainingBounds the four memory peaks that crash long-context mixture-of-experts training, from expert dispatch to optimizer state, with fixed-size GPU schedules.
Salesforce19 upvotesRead →On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityQwen3.8-Flash-Next: a 125B mixture-of-experts model with 6B active that nearly matches its 397B predecessor at 1/9 the training compute.Alibaba (Qwen)76 citations62 upvotesRead →Chiaroscuro Attention: Spending Compute in the DarkCHIAR-Former uses spectral entropy-based routing to dynamically select between DCT, RBF, and self-attention operators, achieving improved efficiency on large text datasets while maintaining performance through hybrid attention mechanisms.
Accenture1 upvotesRead →Dynamic Chunking Diffusion TransformerDynamic Chunking Diffusion Transformer adapts token sequence length based on image content and diffusion timestep, improving efficiency and performance over fixed-token approaches.
AMD3 citations16 upvotesRead →Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-ExpertsImbalanced expert routing in Mixture-of-Experts models leads to computational inefficiencies in expert parallelism, which are addressed by a dynamic rerouting algorithm that balances workload and reduces memory usage.Salesforce11 citations5 upvotesRead →NVIDIA Nemotron 3: Efficient and Open IntelligenceNemotron 3 is a family of efficient models using a hybrid Mamba-Transformer architecture for strong agentic and reasoning capabilities.
NVIDIA60 citations44 upvotesRead →Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningNemotron 3 Nano is an efficient Mixture-of-Experts model that achieves high throughput and strong reasoning performance for agentic tasks.NVIDIA112 citations45 upvotesRead →Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon LayersControlled synthetic pretraining reveals CANON LAYERS, lightweight architectural components that enhance information flow and model capabilities across various sequence architectures.Meta48 citations29 upvotesRead →OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMOmniVinci, an open-source omni-modal LLM, enhances cross-modal understanding and performance across audio, vision, and robotics applications with innovative architecture and efficient data curation.NVIDIA73 citations93 upvotesRead →VGGT: Visual Geometry Grounded TransformerVGGT, a feed-forward neural network, efficiently infers multiple 3D attributes from single or multiple views, outperforming alternatives and enhancing downstream tasks without post-processing.Meta2,132 citations40 upvotesRead →ORIGAMI: A generative transformer architecture for predictions from semi-structured dataA transformer that learns directly from nested JSON documents, treating prediction as next-token generation, and stays competitive on tabular benchmarks.
MongoDB2 citationsRead →Hymba: A Hybrid-head Architecture for Small Language ModelsHymba, a family of small language models with a hybrid-head architecture combining transformer attention and state space models, achieves state-of-the-art performance with improved efficiency and reduced cache size.NVIDIA126 citations50 upvotesRead →Toto: Time Series Optimized Transformer for ObservabilityToto, a Time Series Optimized Transformer for Observability, achieves state-of-the-art performance in observability and general-purpose forecasting using a vast dataset of time series data.
Datadog47 citations33 upvotesRead →DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsThe DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.
DeepSeek1,268 citations65 upvotesRead →Mamba: Linear-Time Sequence Modeling with Selective State SpacesA selective state space model that scales linearly with sequence length and matches Transformers on language.
Carnegie Mellon University10.2k citations152 upvotesRead →CodeT5+: Open Code Large Language Models for Code Understanding and GenerationCodeT5+, a family of flexible encoder-decoder LLMs initialized with frozen LLMs and trained with a mixture of pretraining objectives, achieves state-of-the-art performance across various code-related tasks.Salesforce836 citations7 upvotesRead →CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationCodeT5, a unified encoder-decoder Transformer model, improves code understanding and generation by leveraging semantic information from identifiers and user comments.Salesforce2,633 citations3 upvotesRead →HuggingFace's Transformers: State-of-the-art Natural Language ProcessingTransformers library provides state-of-the-art Transformer architectures and pretrained models for natural language processing tasks with a unified API and emphasis on extensibility and robust deployment.
Hugging Face251 citations34 upvotesRead →Attention Is All You NeedIntroduced the Transformer, the attention-only architecture behind nearly every large language model since.
Google201k citations142 upvotesRead →
Amazon1 citations80 upvotesRead →Context Language ModelsContext Language Models manage their own context by treating it as an editable file, improving performance and efficiency.