A2, Lesson 3: Scale: Bigger Gets Better, Up to a Point
Tags: Sourced / General / Our view / Illustrative. About 12 minutes. No math or code.
1. The story: three papers in two years (2020 to 2022)
In January 2020, a group of researchers (several at OpenAI) posted "Scaling Laws for Neural Language Models." Its abstract says the loss (the model's error in predicting text) "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." It adds that "optimally compute-efficient training involves training very large models on a relatively modest amount of data." [Sourced: Kaplan et al., Jan 23, 2020, https://arxiv.org/abs/2001.08361] In May 2020 (arXiv posting), OpenAI's GPT-3 paper, "Language Models are Few-Shot Learners," trained a model with 175 billion parameters, which the authors describe as "10x more than any previous non-sparse language model." Instead of retraining for each task, the model was given a few examples in the text prompt and was tested "without any gradient updates or fine-tuning." The authors say it performed strongly on many tasks, and also "some datasets where GPT-3's few-shot learning still struggles." All GPT-3 models were trained for 300 billion tokens. [Sourced: Brown et al., 2020, https://arxiv.org/abs/2005.14165 ; results are the authors' own] On March 29, 2022 (arXiv 2203.15556), a DeepMind team reported "Training Compute-Optimal Large Language Models." After training over 400 models, it concluded that "current large language models are significantly undertrained," and that for the best use of a given compute budget, "for every doubling of model size the number of training tokens should also be doubled." It trained a 70-billion-parameter model, Chinchilla, on 1.4 trillion tokens, with the same compute as its 280-billion-parameter predecessor Gopher (trained on 300 billion), and reported that Chinchilla "uniformly and significantly outperforms" Gopher, GPT-3 and others on a broad range of tasks. [Sourced: Hoffmann et al., 2022, https://arxiv.org/abs/2203.15556 ; the authors' own evaluation] The second paper disagreed with part of the first: for a 10x bigger compute budget, the 2020 paper suggested model size grow 5.5x and training data only 1.8x; the 2022 paper found they should grow in equal proportion. [Sourced: Hoffmann et al., introduction] Others have since checked both. In 2024, two separate research teams reported that much of the gap between the two papers comes from how parameters were counted and from setup choices at small scale, and that with those fixed the results agree with the 2022 paper. A third team that tried to rebuild one of the 2022 paper's estimates found its reported numbers did not fit the data they extracted, though the rebuilt result was compatible with the paper's other two methods. These checks cover the setups the researchers tested, not today's largest models. [Sourced: Pearce and Song, arXiv 2406.12907 (published in TMLR 2024); Porian et al., arXiv 2406.19146; Besiroglu et al., arXiv 2404.10102] The story: "bigger is better" was an empirical finding, then it was refined, and one of the refinements was "you were not feeding them enough." [Our view]
2. The one idea
For these models, performance improves smoothly and predictably as you add size, data and compute, but with diminishing returns, and the best mix of size and data is a research finding that changed within two years. Scale explains much of the progress between 2018 and 2022. It does not explain everything, and it does not promise that more will always keep working. [Our view]
3. The kitchen-table version
A student gets better with more study hours, but each extra hour helps a bit less. Also, a bigger brain with too few books reads worse than a smaller brain with plenty of books. [General]
4. What "few-shot" meant
Before GPT-3, getting a model to do a new task usually meant collecting thousands of examples and retraining (Lesson 2). GPT-3 showed that, at its size, you could show a few examples in the prompt and it often picked up the pattern. The authors call this "in-context learning." It is the root of everyday prompting (the topic of course A4). [Sourced: GPT-3 paper abstract and Figure 1.1; our gloss]
5. Worked example: a toy scaling curve
All numbers [Illustrative], computed by script. The exponent is invented; real curves come from the papers. Suppose each 10-times increase in model size cuts the remaining error gap to 85% of what it was. Starting with a gap of 1.0:
- 1x size: 1.00; 10x: 0.85; 100x: 0.72; 1,000x: 0.62; 10,000x: 0.53; 100,000x: 0.45. Each step costs ten times more than the last and buys a smaller improvement: the first 10x removes 0.15, the fifth removes 0.07. Tokens per parameter. GPT-3: 300 billion tokens for 175 billion parameters: about 1.7 tokens per parameter. Chinchilla: 1.4 trillion for 70 billion: 20 tokens per parameter. [Our arithmetic on the papers' stated numbers] The later paper's lesson in one number: the same compute went further when spent on more reading and a smaller model. What a curve does not tell you: whether a new ability will appear, or whether an ability that matters to your business is on the curve at all. [Our view]
6. What it costs and what it breaks
- Money and energy: the 2022 paper itself says compute and energy cost for training these models "is substantial."
- Concentration: few organizations can afford top-end training, so a few set the pace.
- Data supply: more tokens means more text; where it comes from, who owns it and whether it will run out are open questions, and some are in court (see C3 for rights and rules).
- Benchmarks: the same papers rely on benchmark tests; benchmark gains do not guarantee gains at your task.
- Hype risk: "scaling will solve everything" is a claim, not a result from these papers. [Our view]
7. Monday checklist
- When a vendor says "our model is bigger," ask: bigger in parameters, training data, or both?
- Ask what test shows it is better for your task.
- Try a smaller, cheaper model on your task before paying for the biggest.
- Check cost per use, not just quality.
- Watch for claims that extend a curve past where it was measured.
8. Try it yourself (no code, 20 minutes)
Take a task you do with AI. Run it on a small and a large version of any tool and score ten outputs. Is the gap big enough to pay for?
9. Self-check
- What did the 2020 scaling-law paper say about loss? It follows a power law in model size, data and compute.
- What was GPT-3's size, and how was it used on tasks? 175 billion parameters; few examples in the prompt, no fine-tuning.
- What did the 2022 paper conclude about earlier large models? They were undertrained; model size and training tokens should grow in equal proportion.
- In the toy curve, what happens to the gain at each 10x? It shrinks.
- What does a scaling curve not guarantee? That a new ability will appear or that it fits your job.
10. Where this is going
Whether further scale keeps helping, and how much, is argued by researchers; so is how much comes from better data and training methods instead. The next lessons cover the other ingredients: human feedback, images, reasoning and tools. [Our view]
Sources
- Kaplan et al., Scaling Laws for Neural Language Models, Jan 23, 2020: https://arxiv.org/abs/2001.08361
- Brown et al., Language Models are Few-Shot Learners, 2020: https://arxiv.org/abs/2005.14165
- Hoffmann et al., Training Compute-Optimal Large Language Models, 2022: https://arxiv.org/abs/2203.15556
- Pearce and Song, Reconciling Kaplan and Chinchilla Scaling Laws, 2024: https://arxiv.org/abs/2406.12907
- Porian et al., Resolving Discrepancies in Compute-Optimal Scaling of Language Models, 2024: https://arxiv.org/abs/2406.19146
- Besiroglu et al., Chinchilla Scaling: A replication attempt, 2024: https://arxiv.org/abs/2404.10102
Review flags
- RESOLVED (Oct 7, 2026): neutral review. Three 2024 follow-up papers (abstracts read) by other research teams are now cited. Two reconcile Kaplan and Chinchilla; one reports problems in a single Chinchilla estimate while finding its rebuilt result compatible with the paper's other two methods.
- RESOLVED: dates. Kaplan posted Jan 23, 2020; GPT-3 first posted May 28, 2020; Chinchilla posted Mar 29, 2022 (arXiv submission pages). Quotes "relatively modest amount of data," "significantly undertrained," 5.5x/1.8x and "substantial" energy cost checked in the paper texts/abstracts.
- The three main papers are the authors' own results; the follow-ups test setups the researchers chose, not today's frontier models. The lesson says so.
- The statement that GPT-3 "started" prompting is our gloss and is tagged that way.
- The toy exponent is invented; do not read it as the real exponent.
Claim ledger (A2 Lesson 3)
| # | Claim | Source | Verified | Status | Re-check |
|---|---|---|---|---|---|
| 1 | Loss follows power law in size, data, compute over 7+ orders of magnitude | arXiv 2001.08361 abstract | 2026-10-06 | Read | none |
| 2 | GPT-3: 175B parameters, 300B tokens, few-shot without fine-tuning, struggles on some tasks | arXiv 2005.14165 | 2026-10-06 | Read | none |
| 3 | Chinchilla 70B on 1.4T tokens vs Gopher 280B on 300B; double size means double tokens; Kaplan suggested 5.5x/1.8x | arXiv 2203.15556 | 2026-10-07 | Re-read (text lines checked) | none |
| 5 | 2024 follow-ups reconcile Kaplan and Chinchilla; one replication reports a fit problem in one estimate | arXiv 2406.12907, 2406.19146, 2404.10102 abstracts | 2026-10-07 | Read; neutral review added | none |
| 4 | Toy curve and tokens-per-parameter arithmetic | Our computation | 2026-10-06 | Illustrative, script-checked | none |