Skip to content
Papers.

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers research paper by Meta, 2025

Meta · Dec 19, 2025 · Architectures · 40 citations · 29 upvotes · unverified

Read on arXiv

What it shows

Controlled synthetic pretraining reveals CANON LAYERS, lightweight architectural components that enhance information flow and model capabilities across various sequence architectures.

UnverifiedHugging Face's summary; not yet checked by hand.

More from Meta

All 34
PaperCitations
WearableQA: A Benchmark for Health Reasoning over Real-World Wearable DataWearableQA is a benchmark of multiple-choice questions derived from real longitudinal wearable data that evaluates large language model reasoning across data and health dimensions.Agents and evaluation · Sep 2026 · Unverified0
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and RecipesControlled experiments on when text, image understanding and image generation help or compete when trained together.Multimodal and robotics · Aug 20262
HumanCLAW: Can Vision-Language Models Act Through a Body?HumanCLAW decouples high-level vision-language decisions from low-level motor execution to evaluate embodied action intelligence, revealing that current vision-language models lack embodied self-awareness.Multimodal and robotics · Jul 2026 · Unverified2
TUA-Bench: A Benchmark for General-Purpose Terminal-Use AgentsTUA-Bench presents a comprehensive benchmark for evaluating general-purpose terminal-use agents across diverse digital activities and specialized workflows, revealing significant performance gaps among current frontier agents.Agents and evaluation · Jun 2026 · Unverified2
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved ReasoningProcess-driven image generation decomposes synthesis into iterative steps involving textual planning, visual drafting, textual reflection, and visual refinement, with step-wise supervision ensuring consistency and interpretability.Reasoning · Apr 2026 · Unverified5
V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised LearningV-JEPA 2.1 is a self-supervised model that learns dense visual representations for images and videos through a combination of dense predictive loss, deep self-supervision, multi-modal tokenizers, and effective scaling.Multimodal and robotics · Mar 2026 · Unverified83
ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuningResearchers address imbalance in routing weights of Mixture-of-LoRAs models by proposing Reinforcement Routing (ReMix), which uses non-learnable weights and reinforcement learning techniques to improve model expressiveness and performance.Inference and efficiency · Mar 2026 · Unverified3
Beyond Language Modeling: An Exploration of Multimodal PretrainingControlled multimodal pretraining experiments reveal key insights about unified visual representations, data complementarity, world modeling emergence, and efficient scaling through mixture-of-experts architectures.Multimodal and robotics · Mar 2026 · Unverified32

Other architectures papers

Topic
PaperCitations
Attention Is All You NeedIntroduced the Transformer, the attention-only architecture behind nearly every large language model since.Google · Jun 2017194k
Mamba: Linear-Time Sequence Modeling with Selective State SpacesA selective state space model that scales linearly with sequence length and matches Transformers on language.Carnegie Mellon University · Dec 20239,203
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsThe DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.DeepSeek · Jan 2024 · Unverified1,123
Hymba: A Hybrid-head Architecture for Small Language ModelsHymba, a family of small language models with a hybrid-head architecture combining transformer attention and state space models, achieves state-of-the-art performance with improved efficiency and reduced cache size.NVIDIA · Nov 2024 · Unverified115
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMOmniVinci, an open-source omni-modal LLM, enhances cross-modal understanding and performance across audio, vision, and robotics applications with innovative architecture and efficient data curation.NVIDIA · Oct 2025 · Unverified58
Toto: Time Series Optimized Transformer for ObservabilityToto, a Time Series Optimized Transformer for Observability, achieves state-of-the-art performance in observability and general-purpose forecasting using a vast dataset of time series data.Datadog · Jul 2024 · Unverified43
About this paper
Authors
Zeyuan Allen-Zhu
arXiv
2512.17351 · PDF
Venue
Neural Information Processing Systems
Citations
40, 3 influential · Semantic Scholar
Upvotes
29 · Hugging Face
Code
github.com/facebookresearch/PhysicsLM4
Lab
Meta · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf

Changes

What changed
Influential citationsfirst count: 3Sep 25, 2026
Citationsfirst count: 40Sep 25, 2026
New paperFound by the weekly scan, unverifiedSep 25, 2026

Sources: each lab's own papers and arXiv, with citation and upvote counts from Semantic Scholar and Hugging Face. One-line summaries are for orientation, not a substitute for the paper. Logos via logo.dev; trademarks belong to their owners.

New papers by email

Monday afternoons, only in weeks with new papers from the labs.

Double opt-in. Unsubscribe any time.