← Ten breakthroughs that brought AI hereA2 · LESSON 4 OF 7

A2, Lesson 4: From Text Engine to Assistant

Tags: Sourced / General / Our view / Illustrative. About 12 minutes. No math or code.

1. The story: the smaller model that people preferred (January to November 2022)

Raw language models complete text; they do not follow instructions. OpenAI's InstructGPT paper (posted in March 2022) opens with this: "Making language models bigger does not inherently make them better at following a user's intent," and large models "can generate outputs that are untruthful, toxic, or simply not helpful." The team's fix was to collect examples of good answers written by human labelers, fine-tune GPT-3 on them, then collect human rankings of model outputs and use "reinforcement learning from human feedback" to push the model toward preferred answers. The headline result: "outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters." The authors also report improvements in truthfulness and reductions in toxic output, and say the model "still makes simple mistakes." [Sourced: Ouyang et al., "Training language models to follow instructions with human feedback," https://arxiv.org/abs/2203.02155 ; the authors' own evaluation, with preferences measured on their own prompt distribution] On November 30, 2022, OpenAI introduced ChatGPT as "a sibling model to InstructGPT," in a free "research preview." The announcement says the dialogue format lets it "answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests." Its own limitations list says: "ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers"; it is "sensitive to tweaks to the input phrasing"; it is "often excessively verbose" because trainers "prefer longer answers that look more comprehensive"; and it "usually" guesses what the user meant instead of asking. These are the 2022 limitations; OpenAI's page now carries a note that "ChatGPT has evolved substantially since then." [Sourced: OpenAI, "Introducing ChatGPT," Nov 30, 2022, https://openai.com/index/chatgpt/ ; company statements; note read Oct 7, 2026] This lesson's point: the step from "text engine" to "assistant" was not mainly more size (Lesson 3). It was human judgment fed back into training. [Our view]

2. The one idea

After pretraining, people showed the model what good answers look like and ranked its attempts. The model learned to produce answers people tend to prefer. That is why assistants feel helpful, and also why they can sound confident and polished even when wrong: "preferred by raters" is not the same as "true." [General; supported by the paper's and the post's own statements]

3. The kitchen-table version

A new employee who has read everything still needs coaching: "answer the customer's question first, be polite, say when you don't know." You show examples, then compare her answers and say which is better. After many rounds she answers the way your team likes. Those answers are good for your team, but if the team likes long, confident replies, she will give them. [General]

4. The three steps (plain)

  1. Demonstrations: labelers write good answers to sample prompts; the model is trained to imitate them.
  2. Rankings: for each prompt the model writes several answers; labelers rank them.
  3. Reward and practice: a second model learns to predict the rankings, and the assistant model practices producing answers that score well. [Sourced: InstructGPT abstract; our plain-words summary]

5. Worked example: turning a ranking into pairs

All numbers [Illustrative], computed by script. A labeler ranks four answers to one customer question, best to worst: C, A, D, B. That ranking contains 6 pairwise preferences: C beats A, D and B; A beats D and B; D beats B. Counting wins: C 3, A 2, D 1, B 0. A reward model is trained to give C the highest score and B the lowest. Four answers produce 6 pairs; nine answers would produce 36. For 1,000 prompts with four answers each, one ranking per prompt yields 6,000 pairs of training signal from only 1,000 ranking tasks. So rankings are an efficient way to collect judgments. [Our arithmetic] The catch, which the ChatGPT post names: if raters prefer longer answers that look comprehensive, the reward model learns to like length. The system learns what raters reward, not necessarily what is true. [Sourced: OpenAI post, limitations; our inference]

6. What it costs and what it breaks

  • Human labor: labelers write and rank; quality depends on who they are and what instructions they get.
  • Whose preferences: the model reflects its raters' choices. Different raters could produce a different assistant.
  • Confident errors: polish can hide mistakes (the post's own warning).
  • Refusals: the post notes that training the model to be more cautious makes it decline questions it could answer.
  • Over-optimization: pushing hard on a score can warp behavior; the post mentions "well-known over-optimization issues."
  • Your use: never treat a fluent answer as a checked answer.

7. Monday checklist

  1. For any AI answer that matters, ask for sources and check one.
  2. Notice length: ask for short, direct answers and see if quality changes.
  3. Rephrase a question two ways and compare answers (the post says phrasing matters).
  4. Write down three questions where the tool is usually wrong for your business.
  5. Give feedback buttons honest use; they feed the same process.

8. Try it yourself (no code, 20 minutes)

Ask an assistant the same business question three times. Rank the answers best to worst, and write one sentence on why. You just did step 2.

9. Self-check

  1. What did InstructGPT show about size? A 1.3B model was preferred over the 175B GPT-3 on the authors' prompts.
  2. What are the three steps? Demonstrations, rankings, reward and practice.
  3. What did OpenAI say ChatGPT sometimes writes? Plausible-sounding but incorrect or nonsensical answers.
  4. Why can assistants be verbose? Trainers preferred longer answers that look comprehensive.
  5. Is "preferred by raters" the same as "true"? No.

10. Where this is going

Feedback methods keep changing, and so does what raters are asked to prefer. The next lessons look at images, open models, and then reasoning and tools. [Our view]

Sources

Review flags

  • RESOLVED (Oct 7, 2026): InstructGPT paper first posted Mar 4, 2022 (arXiv submission page). Quotes in the lesson checked against the arXiv abstract and the ChatGPT post (all phrases found).
  • RESOLVED: later ChatGPT changes. OpenAI's own page now says ChatGPT "has evolved substantially since" 2022; the lesson presents the limitations list as 2022 only.
  • Both sources are by the builder; results and limits are theirs, and the lesson says so. The full InstructGPT paper (labeler details, its own caveats) was not read; the lesson claims only what the abstract says.
  • "InstructGPT started the assistant era" is not claimed; the lesson says ChatGPT is a sibling model.
  • The ranking example is invented.

Claim ledger (A2 Lesson 4)

#ClaimSourceVerifiedStatusRe-check
1Bigger does not mean following intent; three-step method; 1.3B preferred to 175BarXiv 2203.02155 abstract2026-10-06Readnone
2ChatGPT introduced Nov 30, 2022 as sibling to InstructGPT; free research previewOpenAI post2026-10-06Readnone
3Limitations: plausible-sounding wrong answers, phrasing sensitivity, verbosity from rater preference, guesses on ambiguityOpenAI post2026-10-06Readnone
4Ranking-to-pairs arithmeticOur computation2026-10-06Illustrative, script-checkednone
3InstructGPT first posted Mar 4, 2022; ChatGPT page notes it has evolved since 2022arXiv page; OpenAI post2026-10-07Readnone