A2, Lesson 2: Pretraining: Learn First, Specialize Later
Tags: Sourced / General / Our view / Illustrative. About 12 minutes. No math or code.
1. The story: two ways to read, in two months (2018) and a held-back model (2019)
In late 2018, Google researchers posted the paper that introduced BERT. Its abstract says BERT is "designed to pre-train deep bidirectional representations from unlabeled text," and that the pre-trained model "can be fine-tuned with just one additional output layer" for a wide range of tasks. The paper reports new state-of-the-art results on eleven language tasks, including a GLUE score of 80.5% (a 7.7-point absolute improvement) and SQuAD v1.1 question answering F1 of 93.2. It was first posted on October 11, 2018. [Sourced: Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," https://arxiv.org/abs/1810.04805 ; these results are the authors' own, on benchmarks of the time] The paper compares itself with OpenAI's earlier GPT, which read text left to right only. BERT instead hid some words and asked the model to guess them from the words on both sides, "inspired by the Cloze task." [Sourced: same paper] On February 14, 2019, OpenAI announced GPT-2: a model with 1.5 billion parameters "trained simply to predict the next word in 40GB of Internet text," from a dataset of 8 million web pages. The company said it was "a direct scale-up of GPT, with more than 10X the parameters and trained on more than 10X the amount of data," and that, "due to our concerns about malicious applications of the technology, we are not releasing the trained model." It released a much smaller version and a technical paper instead, calling this "an experiment in responsible disclosure" (OpenAI's own phrase). OpenAI released the full 1.5-billion-parameter model on November 5, 2019, as the last step of a staged release. [Sourced: OpenAI, "GPT-2: 1.5B release," Nov 5, 2019, https://openai.com/index/gpt-2-1-5b-release/] The announcement also reported failure modes (repetition, "fires happening under water," unnatural topic switching) and that it took "a few tries to get a good sample." [Sourced: OpenAI, "Better language models and their implications," Feb 14, 2019, https://openai.com/index/better-language-models/ ; company statements about the model's abilities and risks are the company's own]
Not everyone agreed with the decision. The Verge reported on February 21, 2019 that "many AI researchers have criticized the decision, accusing the lab of exaggerating the danger," while OpenAI said it wanted time to assess misuse risks such as misleading news articles and spam. [Sourced: The Verge, Feb 21, 2019, https://www.theverge.com/2019/2/21/18234500/ai-ethics-debate-researchers-harmful-programs-openai ; both sides are opinions]
2. The one idea
Pretraining means a model first learns general language from a huge amount of unlabeled text, and only afterward gets a small amount of extra training for your specific job. The expensive part is done once; you reuse it. This changed AI from "build a new model for each task" to "start from a general one." [General]
3. The kitchen-table version
A new hire who has read a thousand books already knows the language, so you only teach the shop's own rules. A hire who has never read anything needs everything taught from scratch. Pretraining is the thousand books. [General]
4. Two ways to learn from raw text
- Fill in the blank (BERT style): hide a word, guess it from both sides. Good for understanding a sentence: classifying, finding answers in a passage.
- Predict the next word (GPT style): guess the next word from the words before. Good for writing text, because writing is next-word guessing repeated. Both need no human labels. The text is its own answer key, which is why they could use so much of it. [General; the two objectives are as described in the BERT paper and OpenAI's post]
5. Worked example: why reuse beats starting over
All numbers [Illustrative], computed by script. Not from either paper; real needs vary a lot by task. A small store wants to sort 5,000 customer reviews into "complaint," "praise" and "question." Suppose labeling costs $0.08 per review (paid by the piece).
- From scratch: a model with no prior language knowledge might need on the order of 100,000 labeled examples: $8,000 of labeling.
- Fine-tune a pretrained model: suppose 1,000 labeled examples are enough: $80.
- Just label all 5,000 yourself: $400, and no model needed. So for a job this small, labeling by hand is cheaper than building any model from scratch, and a pretrained model makes the model route cheaper still. Many shops would skip fine-tuning altogether and use a general assistant with a few examples in the prompt (Lesson 4). Scale of the reading: 40 GB of text, at roughly one byte per character and a 500,000-character book, is about 80,000 books. [Illustrative arithmetic on OpenAI's stated 40GB]
6. What it costs and what it breaks
- Pretraining is expensive; only a few organizations do it. Most businesses use someone else's.
- Inherited habits: a pretrained model carries whatever is in its training text, including errors and bias. The GPT-2 post itself notes weak spots and bias risks in generated text.
- Staged release: OpenAI's decision to hold back GPT-2 sparked a debate about whether such caution was warranted. The post calls it an experiment. Per the verifier, The Verge (Feb 21, 2019) reported critics saying the risk was exaggerated, and MIT Technology Review (Aug 29, 2019) covered both camps. [Our view]
- Benchmarks flatter: high scores on tests like GLUE do not mean the model works for your task.
7. Monday checklist
- When you use an AI tool, ask: is it a general model or one built for my task?
- Before paying to fine-tune anything, count the examples you would need and price a hand-labeled sample.
- Try a general assistant on 20 of your real examples first.
- Keep a list of the tool's wrong answers.
- Treat any benchmark number as a claim about the benchmark.
8. Try it yourself (no code, 20 minutes)
Write ten sentences with one word missing. Fill each by guessing from both sides. Then write ten sentence beginnings and guess the next word. Notice which was easier, and why both can be trained without human labels.
9. Self-check
- What is pretraining? Learning general language from lots of unlabeled text before a small specialization.
- How did BERT learn? By guessing hidden words from both sides.
- How did GPT-2 learn? By predicting the next word.
- Why did OpenAI hold back GPT-2's full model in February 2019? It cited concerns about malicious uses.
- Why might hand-labeling beat a model for a small job? The cost of building and checking the model can exceed the labeling cost.
10. Where this is going
Pretraining plus later adjustment became the standard recipe. The next lesson asks what happens when you just make the pretrained model much bigger. [Our view]
Sources
- BERT paper: https://arxiv.org/abs/1810.04805
- OpenAI GPT-2 announcement, Feb 14, 2019: https://openai.com/index/better-language-models/
- OpenAI, GPT-2: 1.5B release, Nov 5, 2019: https://openai.com/index/gpt-2-1-5b-release/
- The Verge, Feb 21, 2019 (secondary): https://www.theverge.com/2019/2/21/18234500/ai-ethics-debate-researchers-harmful-programs-openai
Review flags
- Both main sources are by the model builders; results and risk claims are theirs. The lesson labels them that way. The GPT-2 technical paper and the original GPT paper were not read; the "GPT read left to right" statement rests on the BERT paper's own description and is tagged to it.
- RESOLVED (Oct 7, 2026): BERT first posted Oct 11, 2018 (arXiv submission page). GPT-2 full release Nov 5, 2019 (OpenAI page read). Neutral coverage: The Verge, Feb 21, 2019 (page read) is now in the lesson. "Responsible disclosure" is still OpenAI's phrase and is labeled.
- The MIT Technology Review Aug 2019 piece was not read and is not cited.
- The example labeling counts and prices are invented.
Claim ledger (A2 Lesson 2)
| # | Claim | Source | Verified | Status | Re-check |
|---|---|---|---|---|---|
| 1 | BERT: bidirectional pretraining, masked words, fine-tune with one extra layer; eleven tasks; GLUE 80.5% (+7.7), SQuAD v1.1 F1 93.2 | arXiv 1810.04805 | 2026-10-06 | Read abstract and introduction | none |
| 2 | GPT-2 announced Feb 14, 2019: 1.5B parameters, 40GB, 8M pages, >10x GPT; full model withheld; failure modes | OpenAI post | 2026-10-06 | Read | none |
| 3 | Labeling-cost and book-count arithmetic | Our computation | 2026-10-06 | Illustrative, script-checked | none |
| 4 | BERT first posted Oct 11, 2018 | arXiv submission history | 2026-10-07 | Read | none |
| 5 | GPT-2 1.5B released Nov 5, 2019; critics said danger exaggerated | OpenAI release page; The Verge Feb 21, 2019 | 2026-10-07 | Read | none |