← AI in online sellingC1 · LESSON 2 OF 7

C1, Lesson 2: Search and Recommendations: Ranking What the Shopper Sees

Tags: Sourced / General / Our view / Illustrative. Reading time about 13 minutes.

1. The story: the $1 million algorithm Netflix never used

On October 2, 2006, Netflix announced a prize of $1 million for the first team to beat its own movie-recommendation system, Cinematch, by 10% in prediction accuracy. It released 100 million anonymous ratings to contestants. [Sourced: Netflix announcement posted Oct 2, 2006; NYT Oct 2, 2006] Accuracy was measured as RMSE, the average size of the error between predicted and real ratings. Netflix's FAQ said Cinematch scored 0.9525 on the test set, 9.6% better than just guessing each movie's average rating (1.0540). A 10% win meant getting to about 0.857. [Sourced: Netflix Prize FAQ, archived Aug 2007; our arithmetic] On September 21, 2009 the team BellKor's Pragmatic Chaos won. [Sourced: Netflix Prize forum announcement (Wayback, Sept 24, 2009); New York Times, Sept 22, 2009] Then the surprise. In 2012 Netflix's own engineers wrote that they evaluated the winning ensemble offline, but the added accuracy "did not seem to justify the engineering effort needed to bring them into a production environment," and that by then the business had moved on to streaming, which launched in 2007 and changed what data existed. [Sourced: Netflix Tech Blog, "Netflix Recommendations: Beyond the 5 Stars (Part 1)," 2012] The same post says 75% of what members watch comes from some sort of recommendation. [Sourced: same post; a company claim] The lesson: a better score on a test is not the same as a better store. What the shopper does next, and what it costs to run, decide it. [Our view]

2. The one idea

Search answers "I know what I want." Recommendations answer "show me what I did not know to ask for." Both are the same job underneath: put a ranked list in front of a shopper, then learn from what they click and buy. [General]

3. The kitchen-table version

You run a small shop and a customer says "something to carry water." Your shelf has "hydration bottle," "flask" and "tumbler," but the label says "bottle." A good clerk knows the three words mean the same thing and shows all three. A great clerk also remembers the customer bought a cap last time and says, "same size cap for this one?" Search is the first clerk. Recommendations are the second. [General]

4. How a store search actually works

  1. Match: find products whose words match the query (keywords), or whose meaning is close (embeddings: lists of numbers that put similar meanings near each other).
  2. Rank: order the matches using signals: text match, popularity, price, stock, margin, past clicks.
  3. Adjust: filters, business rules ("do not show out of stock first"), merchandiser pins.
  4. Learn: record clicks, add-to-carts and purchases; use them to rank better next time. [General] Where the common failures sit: Baymard Institute's benchmark of 50 top US e-commerce sites found that 70% of desktop search implementations could not return relevant results for product-type synonyms, and 34% did not handle model numbers or misspellings well. This is an older benchmark (2014-2015), so read it as a warning about a basic failure, not a current rate. [Sourced: Baymard Institute, "The State of Mobile E-Commerce Search and Category Navigation," Nov 17, 2015, which gives both figures for desktop sites; the 70% also appears in its June 18, 2014 article]

5. Recommendations: where they appear and what they do

PlaceQuestion it answersTypical signal
Home pageWhat should this shopper see first?Past views, purchases
Product pageWhat goes with this? What else is like this?Co-purchase, similar content
CartWhat would complete this order?Complements, accessories
Post-purchase emailWhat next?Replenishment timing
[General] Amazon's 2003 paper described recommendations on the home page, search results, the cart and the order-confirmation page. [Sourced: Smith and Linden 2017, see Lesson 1]
Three methods you will meet: collaborative (people who bought X also bought Y, as in Lesson 1), content-based (this item is like the ones you liked, by its attributes), and popular (what sells most). New shoppers and new products have no history, the cold start problem, so stores start with popular and content-based, and switch as data arrives. [General]

6. Worked example A: is the ranking any good? (the $12 water bottle)

All numbers [Illustrative], computed by script. Shopper searches "water bottle." Two rankings of the first five results, 1 = relevant:

  • Ranking A: relevant, not, relevant, not, not. Precision@5 = 2/5 = 0.4. The first relevant result is in position 1.
  • Ranking B: relevant, relevant, relevant, not, not. Precision@5 = 3/5 = 0.6. Precision@5 is the share of the top five that are relevant. Another common score, reciprocal rank, is 1 divided by the position of the first relevant result (A: 1.0; if the first relevant were at position 2: 0.5). Caution: offline scores like these are what Netflix measured with RMSE. They are a guide, not proof. The proof is an A/B test on real shoppers (Lesson 4).

7. Worked example B: what do "no results" cost?

[Illustrative] A store gets 20,000 searches a month. 12% return zero results: 2,400. Suppose adding synonyms ("flask", "tumbler" to "bottle") fixes two thirds of them: 1,600 searches. A fixed search converts at half the normal 4% rate, so 2%: 32 extra orders. At $36 an order and 40% margin, that is 32 x $36 x 0.40 = $461 a month gross margin. Compare the cost: someone spends two hours a week reviewing the zero-result list and adding synonyms. At $30 an hour that is about $260 a month. The synonym work pays; a fancy new search vendor at $2,000 a month would need to do far better. [Our view] This is the cheapest AI-adjacent win in the store: read your zero-result queries.

8. What it costs and what it breaks

  • Cost: data cleaning (titles, attributes: see Commerce Module 1), tuning time, and the tool.
  • Popularity trap: ranking by clicks pushes bestsellers up, which makes them sell more, which makes them rank higher. New products never get a chance unless you reserve exposure for them.
  • Echo trap: recommend only what a shopper already bought, and you teach them to buy the same thing. For a durable like a bottle, someone who just bought one probably does not want another.
  • Margin and stock: a ranking that ignores stock sends shoppers to out-of-stock pages. [General; our view]
  • Failure story: the Netflix winner was never deployed. Complex ranking models can cost more to run than the lift pays. [Sourced: Netflix Tech Blog 2012]

9. Monday checklist for a practitioner

  1. Export last month's searches. List the top 20 that returned nothing or few results.
  2. For each, ask: synonym, misspelling, model number, or product we do not sell?
  3. Check the top 10 recommendations on your three best-selling product pages. Would a human clerk say that?
  4. Look at your recommendations for a product bought once. Does it suggest a second one?
  5. Write down what each fix would earn (use Section 7) before buying any tool.

10. Try it yourself (no code, 30 minutes)

Search your own store (or one you use) for ten everyday words a shopper might use, including one misspelling and one synonym. Score each result list with precision@5 by hand. Note which failures were vocabulary problems and which were ranking problems.

11. Self-check

  1. What accuracy target did the Netflix Prize set? 10% better than Cinematch.
  2. Why did Netflix not deploy the winning algorithm? The extra accuracy did not seem to justify the engineering effort, and the business had moved to streaming (per Netflix).
  3. A ranking returns relevant results in positions 1, 3, 4 and not 2, 5. Precision@5? 0.6.
  4. What is the cold start problem? No history on a new shopper or product, so history-based methods have nothing to work with.
  5. Why can ranking by clicks hurt new products? Popular items keep getting more exposure, so new ones never get a chance.

12. Where this is going

Shopping agents (Lesson 7) will run searches for the buyer. They will not click through your ranking; they will read your product data. That makes clean attributes and honest descriptions the part of search that matters most. [Our view]

Sources

Review flags

  • RESOLVED (Oct 7, 2026): winner date. BellKor's Pragmatic Chaos, Sept 21, 2009, per the Netflix Prize forum (Wayback Sept 24, 2009) and NYT Sept 22, 2009 (first verifier). The Netflix FAQ numbers (Cinematch 0.9525; movie-average 1.0540; "9.6% improvement") and the Tech Blog quotes ("did not seem to justify the engineering effort"; "75% of what people watch is from some sort of recommendation"; streaming launched 2007) were re-read on the archived FAQ and the blog.
  • RESOLVED: Baymard dates. Both the 70% (synonyms) and the 34% (model number or one-character misspelling) appear in Baymard's Nov 17, 2015 article; the 70% is also in its June 18, 2014 article (the 34% was not found in the 2014 PDF). The lesson cites the 2015 article and calls the figures old.
  • RESOLVED: the RMSE figures are labeled as the 2007 FAQ test-set figures; the lesson does not compare them with final contest results.
  • "75% of what members watch" is a Netflix claim and is labeled as one.
  • All worked-example numbers (precision, zero-result rate, $461, $260, $2,000 vendor) are illustrative.

Claim ledger (Lesson 2)

#ClaimSourceVerifiedStatusRe-check
1Prize announced Oct 2, 2006: $1M, 10% over Cinematch, 100M ratingsNetflix announcement, NYT2026-10-06Read excerpts of bothnone
2Cinematch RMSE 0.9525, 9.6% better than movie-average 1.0540Netflix FAQ 20072026-10-06Read; arithmetic checked (9.63%)none
3Winner BellKor's Pragmatic Chaos, Sept 21, 2009Netflix Prize forum (Wayback Sept 24, 2009); NYT Sept 22, 20092026-10-06Corrected by verifiernone
4Netflix: accuracy gain did not justify engineering effort; streaming launched 2007Netflix Tech Blog2026-10-06Read, primarynone
575% of viewing from recommendationsNetflix Tech Blog2026-10-06Company claimnone
670% of desktop sites fail on synonyms; 34% model numbers/misspellingsBaymard Nov 17, 2015 article (70% also June 18, 2014)2026-10-07Re-read; datednone
7Amazon recommendations on home, search, cart, order pages (2003)Smith and Linden 20172026-10-06Readnone
8Worked-example numbersOur computation2026-10-06Illustrative, script-checkednone