Papersby Glyph

Notable data and AI papers

Papers627 Labs45

627 papers · 45 labs

Scaling Laws for Neural Language Models research paper by OpenAI, 2020

OpenAI · Training and scaling · Jan 23, 2020 · 9,592 citations · 12 upvotes

Language model loss falls as a smooth power law as model size, data and compute grow.

By Jared Kaplan, Sam McCandlish, Tom Henighan and 7 more · arXiv 2001.08361 · PDF

Summarised by hand from the abstract.

More from OpenAI

ReasoningUnverifiedMar 5, 2026Reasoning Models Struggle to Control their Chains of ThoughtChain-of-thought controllability measures how effectively models can be constrained to follow reasoning steps, with findings showing significantly lower controllability in reasoning versus output generation, and varying impacts from model size, training methods, and task complexity.

OpenAI37 citations41 upvotesRead →Alignment and safetySep 4, 2025Why Language Models HallucinateModels hallucinate because training and benchmarks reward confident guessing over saying they do not know.

OpenAI391 citations200 upvotesRead →Alignment and safetyUnverifiedDec 21, 2024OpenAI o1 System CardThe o1 model series, trained with reinforcement learning and chain of thought, enhances safety and robustness by reasoning about policies, leading to superior performance on risk benchmarks while highlighting the need for robust alignment and risk management.

OpenAI38 upvotesRead →Multimodal and roboticsUnverifiedOct 25, 2024GPT-4o System CardGPT-4o is an omnimodal autoregressive model trained to handle text, audio, image, and video inputs, offering high-performance outputs across these modalities, with particular strengths in vision and audio.

OpenAI5,388 citations88 upvotesRead →Foundation modelsMar 15, 2023GPT-4 Technical ReportA multimodal model that passes a simulated bar exam with a score around the top 10% of test takers.

OpenAI28.4k citations7 upvotesRead →Alignment and safetyMar 4, 2022Training language models to follow instructions with human feedbackInstructGPT: fine-tuning on human feedback made a 1.3B model preferred over the 175B GPT-3.

OpenAI25.8k citations26 upvotesRead →Foundation modelsMay 28, 2020Language Models are Few-Shot LearnersGPT-3, a 175B-parameter model, does new tasks from a few examples in the prompt, with no fine-tuning.

OpenAI66.4k citations21 upvotesRead →

Training and scalingJun 17, 2021LoRA: Low-Rank Adaptation of Large Language ModelsLoRA fine-tunes a large model by training small low-rank matrices, cutting trainable parameters by 10,000 times.

Microsoft25.6k citations66 upvotesRead →Training and scalingMar 29, 2022Training Compute-Optimal Large Language ModelsChinchilla: for a fixed compute budget, train a smaller model on more data; parameters and tokens should grow together.

Google DeepMind4,035 citations13 upvotesRead →Training and scalingSep 17, 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMegatron-LM splits each Transformer layer across GPUs to train models with billions of parameters.

NVIDIA3,387 citations5 upvotesRead →Training and scalingUnverifiedJan 5, 2024DeepSeek LLM: Scaling Open-Source Language Models with LongtermismDeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.

DeepSeek872 citations56 upvotesRead →Training and scalingMay 15, 2024LoRA Learns Less and Forgets LessLoRA learns less than full fine-tuning on code and math, but forgets less of what the model already knew.

Databricks486 citations91 upvotesRead →Training and scalingUnverifiedSep 16, 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAjina-embeddings-v3, a large-scale text embedding model, achieves state-of-the-art performance in multilingual and long-context retrieval tasks using Low-Rank Adaptation and Matryoshka Representation Learning.

Jina AI221 citations35 upvotesRead →

About this paper

Venue: arXiv.org
Citations: 9,592, 649 influential · Semantic Scholar
Upvotes: 12 · Hugging Face
Lab: OpenAI · on Companies · on Acquisitions · on Paydays · on Releases · on TechConf

Changes

CitationsNov 9, 2026Influential citations fell from 652 to 649.

CitationsNov 9, 2026Citations rose from 9,533 to 9,592.

CitationsNov 2, 2026Influential citations rose from 636 to 652.

CitationsNov 2, 2026Citations rose from 9,270 to 9,533.

CitationsOct 12, 2026Influential citations rose from 635 to 636.

CitationsOct 12, 2026Citations rose from 9,190 to 9,270.

CitationsOct 5, 2026Influential citations rose from 631 to 635.

CitationsOct 5, 2026Citations rose from 9,085 to 9,190.

CitationsSep 28, 2026Citations rose from 9,065 to 9,085.

CitationsSep 25, 2026Citations rose from 9,061 to 9,065.

New paperSep 24, 2026Added to the list.