← AI, commerce and societyA6 · LESSON 2 OF 7

A6, Lesson 2: When the algorithm learns the wrong lesson

Tags: Sourced stories / Our view / Illustrative examples. Self-paced reading and practice. No math or code. Every case or news claim carries its source and the date we read it; invented figures are labeled invented.

1. The story

In 2014, Amazon started building a tool to read job applications. It learned from ten years of resumes sent to Amazon, and most of those came from men, because most of the tech industry was men. By 2015 the company noticed the tool had taught itself a lesson nobody meant to teach: it marked down resumes containing the word "women's," as in "women's chess club captain." Amazon edited the tool, never regained confidence in it, and disbanded the team by early 2017, reported Reuters. Amazon says the tool was never used to judge real candidates.

2. The one idea

An algorithm learns the lesson its examples and rewards teach, not the lesson you meant. Amazon wanted "who is a good engineer." Its historical data reflected who had been hired before, so the tool learned to imitate yesterday's decisions, including their bias. You get what you optimize for, whether that is engagement, margin or trust.

3. The kitchen-table version

A teacher is told to raise her class's test average, no questions asked. The fastest way is not to teach better - it is to quietly move the strongest students into her room. The average rises, and nothing was taught. Reward a number alone and you invite the shortcut that produces the number without the thing you actually wanted.

4. Worked example

Our invented candle shop wants more repeat buyers, so it gives a tool one goal - pick customers for the VIP early-access email to maximize repeat purchases. The shop has 2,000 customers; the tool picks 300, and 68% of its picks buy again within a month, against 22% across the whole shop. Those figures are invented, but the shape is the classic trap: the tool did not create a single new loyal customer. It read the order history and selected the people who already buy every month. Picture the counterfactual as a stipulated thought experiment - the week no email went out: in that stipulation, 204 of the 300 picks would have come back anyway. That 204 is invented ground truth for the illustration, not an observation, and the raw 68% repeat rate alone cannot prove the tool caused anything. The metric says the tool tripled repeat buying. The tool changed nothing but the guest list. Amazon's resume reader made the same mistake in a darker key: it mistook imitating the past for predicting the future.

Instructor: "So measure repeat purchases differently?" Learner: "Measure what changed, not what was already true. Compare the tool's picks against similar customers who got no email - otherwise you are grading the shortcut, not the work."

5. Why this matters to a shop

The same trap sits inside every commerce tool that "optimizes" something. A subject-line optimizer rewarded for opens can drift toward clickbait your best customers resent. A recommendation engine rewarded for clicks can learn to push whatever is loudest, not what your shop stands behind. A discount tool rewarded for repeat orders can quietly train your best customers to never pay full price: by next quarter the metric it was rewarded for looks fine and your margin is gone. The tool did its job. The job was described badly.

6. What it costs and what it breaks

A narrow metric is fast to optimize and easy to report; a balanced set of goals is slower and harder to explain, and a fair before-and-after comparison takes weeks of discipline. Checking edge cases takes real hours. But the alternative is discovering, like Amazon, that the tool learned a lesson you would never have taught on purpose - and in hiring, credit, pricing and customer treatment, the wrong lesson can be a legal and reputational break, not just a commercial one.

7. Monday checklist

List every tool in your shop that optimizes a number, and the number. For each, write the real goal in one plain sentence, then add the never-do clause ("and never train customers to wait for discounts"). Ask of each tool: does this metric change behavior, or just select it? Pick the edge case that would reveal what the tool really learned. Decide who reviews the tool's choices monthly, and put it on a calendar.

8. Try it yourself (five minutes)

Pick one number your business tries to grow, like repeat orders or email sign-ups. Write down two ways a single-minded machine could grow that number while quietly hurting your customers or your brand. If you can think of them, an optimizer can find them.

9. Self-check

  1. What lesson did Amazon's data actually teach? Answer: To imitate past hiring decisions, which reflected a mostly male industry - not to recognize good engineers.
  2. Why is "women's chess club captain" the detail that matters? Answer: It shows the tool learned surface markers from old decisions, not merit; editing out markers could not restore confidence in what else it had learned.
  3. In the VIP example, why is 68% versus 22% misleading? Answer: The tool selected yesterday's loyal customers rather than creating new ones; nothing changed but the guest list.
  4. What is each tool in your shop currently rewarded for, in one sentence?
  5. What is your shop's "women's chess club captain" - the edge case that would show you what a tool really learned?

Sources

  • Reuters, "Amazon scraps secret AI recruiting tool that showed bias against women," October 2018: built from 2014, trained on ten years of resumes, penalized "women's," team disbanded by early 2017; Amazon said it was never used to evaluate candidates.
  • Read 3 Nov 2026 (via Reuters listing and The Verge's account; Reuters full text would not render).
  • Wick and Ember, the VIP figures (2,000 customers, 300 picks, 68% vs 22%, 204 returning) and the discount scenario are invented and illustrative, not a reported story.
  • Review flags: the legal-exposure sentence is our framing, not a legal conclusion.
  • Go deeper, free: Harvard's session 3, The Alignment Problem, covers how labs try to align AI with human preferences, and its slide deck walks the paper-clip thought experiment. Watch it here.

Review flags

  • the legal-exposure sentence is our framing, not a legal conclusion.

Claim ledger

Claim or exampleBasisChecked
Recruiting project began 2014; bias found by 2015; team disbanded by early 2017.Reuters, Amazon recruiting3 Nov 2026
Invented VIP thought experiment: 204/300 = 68%; causality not established by a repeat rate.Invented, illustrative; arithmetic script-checked3 Nov 2026

Independence

Independently developed by Commerce Stories, not affiliated with Harvard. We link to the archived 2024 course; we do not reproduce its videos, slides or exercises. The linked videos were not watched for this written-material review.