Papersby Glyph

Notable data and AI papers

Papers627 Labs45

627 papers · 45 labs

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism research paper by NVIDIA, 2019

NVIDIA · Training and scaling · Sep 17, 2019 · 3,387 citations · 5 upvotes

Megatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.

By Mohammad Shoeybi, Mostofa Patwary, Raul Puri and 3 more · arXiv 1909.08053 · PDF

Summarised by hand from the abstract.

More from NVIDIA

Multimodal and roboticsUnverifiedNov 5, 2026VFlash: Fast, Accurate Visual Planning via Flash Primitive DecodingA method called VFlash speeds up and improves visual motion planning for robots by predicting entire spatial shapes at once.

NVIDIA109 upvotesRead →Foundation modelsUnverifiedOct 27, 2026Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy DistillationNLD-Uni is a unified discrete diffusion language model that combines visual understanding and generation via cross-paradigm on-policy distillation and progressive self-refining diffusion.

NVIDIA12 upvotesRead →ReasoningUnverifiedOct 27, 2026ThinkWorld: World Imagination Reinforces Chain-of-Dynamics ReasoningThinkWorld improves robotic generalization by representing actions as dynamics chains and using world imagination to learn from unlabeled human videos.

NVIDIA57 upvotesRead →Multimodal and roboticsUnverifiedOct 20, 2026Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary RigsThis model generates both video and 3D motion for any character rig, allowing the motion to be replayed accurately within a 3D scene.

NVIDIA26 upvotesRead →Inference and efficiencyUnverifiedOct 13, 2026EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud DelegationThis paper introduces a framework that reduces inference costs by using a small local model for most tasks and delegating only critical steps to the cloud.

NVIDIA30 upvotesRead →Training and scalingUnverifiedOct 8, 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often BetterThis paper shows that distilling from offline student rollouts is often more efficient and accurate than standard on-policy distillation for many model pairs.

NVIDIA28 upvotesRead →Multimodal and roboticsUnverifiedOct 7, 2026Long-WAM: Scaling the Context of World-Action ModelsLong-WAM improves robot control by using autoregressive pretraining to effectively leverage long visual histories for real-time action prediction.

NVIDIA2 citations114 upvotesRead →Inference and efficiencyUnverifiedOct 6, 2026NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter ScaleNeMo-DCR enables efficient, bit-exact synchronization of trillion-parameter model checkpoints by transmitting only the sparse weight changes between clusters.

NVIDIA1 citations29 upvotesRead →

Training and scalingJun 17, 2021LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.

Microsoft25.6k citations66 upvotesRead →Training and scalingJan 23, 2020Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.

OpenAI9,592 citations12 upvotesRead →Training and scalingMar 29, 2022Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.

Google DeepMind4,035 citations13 upvotesRead →Training and scalingUnverifiedJan 5, 2024DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.

DeepSeek872 citations56 upvotesRead →Training and scalingMay 15, 2024LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.

Databricks486 citations91 upvotesRead →Training and scalingUnverifiedSep 16, 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAjina-embeddings-v3, a large-scale text embedding model, achieves state-of-the-art performance in multilingual and long-context retrieval tasks using Low-Rank Adaptation and Matryoshka Representation Learning.

Jina AI221 citations35 upvotesRead →

About this paper

Venue: arXiv.org
Citations: 3,387, 417 influential · Semantic Scholar
Upvotes: 5 · Hugging Face
Lab: NVIDIA · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf

Changes

CitationsNov 9, 2026Influential citations rose from 411 to 417.

CitationsNov 9, 2026Citations rose from 3,352 to 3,387.

CitationsNov 2, 2026Influential citations rose from 403 to 411.

CitationsNov 2, 2026Citations rose from 3,265 to 3,352.

CitationsOct 12, 2026Influential citations rose from 400 to 403.

CitationsOct 12, 2026Citations rose from 3,236 to 3,265.

CitationsOct 5, 2026Influential citations rose from 395 to 400.

CitationsOct 5, 2026Citations rose from 3,187 to 3,236.

CitationsSep 28, 2026Influential citations rose from 394 to 395.

CitationsSep 28, 2026Citations rose from 3,179 to 3,187.

CitationsSep 25, 2026Influential citations rose from 393 to 394.

CitationsSep 25, 2026Citations rose from 3,177 to 3,179.

New paperSep 24, 2026Added to the list.