A2, Lesson 1: "Attention Is All You Need," and Then What?
Tags: Sourced / General / Our view / Illustrative. About 11 minutes. No math or code. Builds on A1 (the Transformer idea in plain words) and does not repeat it.
1. The story: a translation paper (2017)
In 2017, eight researchers, most at Google Brain and Google Research, published a paper titled "Attention Is All You Need." Its goal was narrow: build a better machine-translation system. The abstract says the dominant models were "complex recurrent or convolutional neural networks," and proposes "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." [Sourced: Vaswani et al., "Attention Is All You Need," 2017, https://arxiv.org/abs/1706.03762] What did they claim? On the standard WMT 2014 English-to-German test, the big model scored 28.4 BLEU (a score that compares machine translation to human translations), "improving over the existing best results, including ensembles, by over 2 BLEU." On English-to-French, the big model scored 41.0 BLEU, a new best for a single model, "after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature." [Sourced: the paper as published at NeurIPS 2017 and the first arXiv version (June 12, 2017), both 41.0. Later arXiv revisions print 41.8 in the abstract and table but 41.0 in the text, so we use the published figure.] The paper's authors note that they equally contributed and the listing order is random. One contribution note says one of them "proposed replacing RNNs with self-attention and started the effort to evaluate this idea." [Sourced: same paper, author note] Nothing in the paper says "chatbot." It reports translation, plus a test on grammar parsing. The jump from there to today's assistants took several more steps, and the rest of this course is those steps. [Our view]
2. The one idea
The Transformer's practical win was speed of training: it reads all the words of a sentence at once and lets each word look directly at every other word, instead of reading one word at a time. That made it possible to train much larger models on much more text in a reasonable time. Size, not just cleverness, is what followed. [General; our reading of the paper's stated aims]
3. The kitchen-table version
Old way: a reader goes through a book one word at a time, carrying notes in her head, and by the end the early words are fuzzy. New way: a team reads every page at the same time, and each reader can glance at any other page. It is faster, and it keeps long-range connections better. [General]
4. What the paper actually measured
- Speed: the paper's table compares layer types: a recurrent layer needs a number of sequential operations that grows with sentence length; a self-attention layer needs a constant number. The paper lists maximum path length between any two words as constant for self-attention and growing with length for recurrent layers. [Sourced: same paper, Table 1]
- Cost: it reports training on one machine with 8 NVIDIA P100 GPUs; the base model took 12 hours (100,000 steps at about 0.4 seconds), the big model 3.5 days (300,000 steps at 1.0 second). [Sourced: same paper, training section]
- Quality: translation scores (BLEU) above earlier systems on two language pairs.
- What it did not measure: whether such models could chat, reason, write code, follow instructions or stay factual. Those came later, from different work. [Our view]
5. Worked example: why "all at once" mattered
All numbers [Illustrative], computed by script, from the paper's description of sequential steps.
- A 50-word sentence through a recurrent model needs about 50 steps in sequence per layer; the steps cannot run at the same time. A self-attention layer does its work for all 50 words in 1 step (in parallel on enough hardware).
- Distance between the first and last word: 49 hops in a recurrent model, 1 hop in self-attention.
- The cost: self-attention compares every word with every other. For 50 words that is 2,500 comparisons; for 500 words, 250,000; for 5,000 words, 25,000,000. That cost grows with the square of length, which is why very long documents were hard and why later work on long inputs mattered. [Our reading of the paper's Table 1, which lists per-layer cost growing with n squared] Take-away: the trade was parallel speed for more comparison work. Hardware that does many things at once made that a good trade. [Our view]
6. What it costs and what it breaks
- Compute appetite: the design rewards bigger models and more data, which means more money and energy. The paper's own training costs were small by today's standards; later models are far larger (see Lesson 3).
- Long inputs: the all-pairs cost limits how much text a model can look at at once.
- Not understanding: the paper does not claim the model understands anything; it claims better scores on tests.
- Credit: the paper itself says attention was already "an integral part" of translation models, used together with recurrent networks, and the authors cite earlier attention work. "Invented overnight" is a myth. [Sourced: the paper's introduction (references 2 and 19) and its authors' own wording]
7. Monday checklist
- Pick one AI tool you use. Ask: what task was it built for, and what was it later stretched to do?
- When someone says "the paper that started it all," ask what the paper measured.
- Remember the test: a score on a benchmark is not the same as doing your job.
- Note one thing the tool does well and one it does badly; save them.
- Read the first page of the paper (the abstract); it is short.
8. Try it yourself (no code, 20 minutes)
Read the abstract of the Transformer paper. Underline each claim. Then write two columns: "measured in the paper" and "claimed later by others." Fill both with at least two items.
9. Self-check
- What task was the 2017 paper about? Machine translation (plus a parsing test).
- What did it drop? Recurrence and convolutions; it used attention only.
- What was the paper's main practical advantage? More parallelizable and faster to train, with better scores.
- What does all-pairs attention cost as the text gets longer? Comparisons grow with the square of length.
- Name something the paper did not test. Chat, instruction following, reasoning, factual accuracy.
10. Where this is going
The Transformer remains the base of most large models in this course; whether it stays that way is open, as researchers are testing other designs. The next lessons show what was built on top of it: pretraining, scale, assistants, images, reasoning, tools and agents. Some expect far more capable systems later; Lesson 7 covers that briefly and neutrally. [Our view]
Sources
- Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, "Attention Is All You Need," 2017: https://arxiv.org/abs/1706.03762 (full text read at https://arxiv.org/pdf/1706.03762); published version: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Review flags
- RESOLVED (Oct 7, 2026): English-to-French BLEU. The NeurIPS 2017 PDF (proceedings.neurips.cc) and arXiv v1 (June 12, 2017) say 41.0 in the abstract, table and text. arXiv revisions from v5 on (latest v7, Aug 2, 2023) print 41.8 in the abstract and table but still 41.0 in the text. The lesson uses 41.0.
- RESOLVED: arXiv first posting is June 12, 2017; the conference is NIPS 2017 (Advances in Neural Information Processing Systems 30). The lesson says "2017" only.
- "Most at Google" reflects the author lines: six list Google Brain or Google Research, one lists the University of Toronto, one an independent address; two footnotes say the work was done while at Google.
- RESOLVED: the earlier-attention statement now rests on the paper's own introduction, not our view.
- The step and comparison counts are our arithmetic from the paper's Table 1; real systems vary.
Claim ledger (A2 Lesson 1)
| # | Claim | Source | Verified | Status | Re-check |
|---|---|---|---|---|---|
| 1 | Paper proposes attention-only Transformer; drops recurrence and convolutions | arXiv 1706.03762 abstract | 2026-10-06 | Read | none |
| 2 | 28.4 BLEU English-German (over 2 BLEU better); 41.0 English-French after 3.5 days on 8 GPUs | NeurIPS 2017 PDF; arXiv v1 and v7 compared | 2026-10-07 | Resolved: published version says 41.0 | none |
| 3 | Table 1: sequential operations and path lengths | Same | 2026-10-06 | Read | none |
| 4 | 8 P100 GPUs; 12 hours base, 3.5 days big | Same | 2026-10-06 | Read | none |
| 5 | Step and comparison counts | Our arithmetic | 2026-10-06 | Illustrative, script-checked | none |