Papersby GlyphTraining Compute-Optimal Large Language Models research paper by Google DeepMind, 2022
Google DeepMind · Training and scaling · Mar 29, 2022 · 4,035 citations · 13 upvotes
What it shows
Chinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.
By Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch and 19 more · arXiv 2203.15556 · PDF · Code
Summarised by hand from the abstract.
More from Google DeepMind
All 12DiffusionGemma Technical ReportDiffusionGemma is a fine-tuned mixture-of-experts language model that uses discrete diffusion to generate text blocks in parallel, achieving high speed while preserving capabilities like multimodal inputs and reasoning.
Google DeepMind14 citations40 upvotesRead →Gemma 4 Technical ReportOpen multimodal models from 2.3B to 31B parameters, with a thinking mode and image and audio input.Google DeepMind356 citations81 upvotesRead →LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMsLiteFrame, a lightweight video encoder with Compressed Token Distillation training method, reduces latency and increases frame processing capacity for long-form video understanding in Video LLMs while maintaining accuracy.Google DeepMind2 citations26 upvotesRead →Understanding the Challenges in Iterative Generative Optimization with LLMsGenerative optimization using large language models faces challenges due to implicit design decisions about artifact modification and learning evidence that significantly impact success across different applications.Google DeepMind11 citations27 upvotesRead →LoGeR: Long-Context Geometric Reconstruction with Hybrid MemoryLoGeR enables long-term 3D video reconstruction by combining bidirectional priors with a hybrid memory system that includes parametric Test-Time Training and non-parametric sliding window attention mechanisms.Google DeepMind46 citations63 upvotesRead →SIMA 2: A Generalist Embodied Agent for Virtual WorldsSIMA 2, built on a Gemini foundation model, interacts in 3D virtual worlds, reasons about goals, handles complex instructions, and autonomously learns new skills through open-ended self-improvement.Google DeepMind27 citations26 upvotesRead →Robot Learning from a Physical World ModelPhysWorld integrates video generation and physical world modeling to enable accurate robotic manipulation from visual demonstrations without real robot data.Google DeepMind23 citations32 upvotesRead →Vibe Checker: Aligning Code Evaluation with Human PreferenceVibe Checker evaluates LLMs by combining functional correctness and instruction following to better align with human coding preferences.Google DeepMind2 citations34 upvotesRead →
Other training and scaling papers
TopicLoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.
Microsoft25.6k citations66 upvotesRead →Scaling Laws for Neural Language ModelsLanguage model loss falls as a smooth power law as model size, data and compute grow.
OpenAI9,592 citations12 upvotesRead →Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.
NVIDIA3,387 citations5 upvotesRead →DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.
DeepSeek872 citations56 upvotesRead →LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.
Databricks486 citations91 upvotesRead →jina-embeddings-v3: Multilingual Embeddings With Task LoRAjina-embeddings-v3, a large-scale text embedding model, achieves state-of-the-art performance in multilingual and long-context retrieval tasks using Low-Rank Adaptation and Matryoshka Representation Learning.
Jina AI221 citations35 upvotesRead →
About this paper
- Venue
- Advances in Neural Information Processing Systems 35
- Citations
- 4,035, 375 influential · Semantic Scholar
- Upvotes
- 13 · Hugging Face
- Lab
- Google DeepMind · on Companies · on Acquisitions
Changes
Influential citations rose from 365 to 375.
Citations rose from 3,876 to 4,035.
Influential citations rose from 361 to 365.
Citations rose from 3,835 to 3,876.
Influential citations rose from 352 to 361.
Citations rose from 3,756 to 3,835.
Citations rose from 3,755 to 3,756.
Added to the list.