← AI in online sellingC1 · LESSON 4 OF 7

C1, Lesson 4: Pricing, Promotions and Personalization Online: Testing Without Burning Trust

Tags: Sourced / General / Our view / Illustrative. Reading time about 13 minutes.

1. The story: 68 DVDs, five days, 6,896 refunds

In September 2000, Amazon ran a price test on 68 DVD titles over five days, showing different shoppers discounts between 20% and 40% to see how much price moved sales. Shoppers noticed that the same movie cost different amounts. On September 27, 2000, Amazon issued a statement calling the random testing "a mistake" and said it had not used customer demographics. CEO Jeff Bezos said: "We've never tested and we never will test prices based on customer demographics." Amazon said it had refunded 6,896 customers who paid more than the lowest price offered. [Sourced: Amazon press statement, Sept 26-27, 2000; CNN/Reuters, Sept 28, 2000] Two things to take from it. The test itself was ordinary: any store learning about price sensitivity wants to see what happens at different prices. What hurt was that shoppers could see each other's prices and had no reason to believe the difference was fair. A price is not just a number; it is a message. [Our view]

2. The one idea

Pricing online is a series of experiments. The skill is designing an experiment that answers the question without a false win, and without making shoppers feel cheated. AI helps at three points: suggesting prices, choosing who sees which offer, and reading test results. None of them removes the need for the test. [General]

3. The kitchen-table version

You sell lemonade. Saturday you charge $2 and sell 40 cups. Sunday you charge $3 and sell 30. Did price cause the difference? Maybe. Maybe Sunday was rainy. A real test charges both prices at the same time to similar people, for long enough that luck evens out. That is all an A/B test is. [General]

4. Three tools, three questions

ToolQuestionCaution
Price setting (list price, markdowns)What should this item cost?Elasticity is not the same for every item (Commerce Module 17B)
PromotionsWhich offer moves volume without giving away margin?Discounts pull sales forward and train shoppers to wait
PersonalizationWho should see which offer, and when?Different prices for the same item to different people carry legal and trust risk
[General]

5. Worked example A: price test for the $12 water bottle

All numbers [Illustrative]. Control: $12, 100 sales a week, cost $7. Profit per unit $5. Weekly profit $500. Test: $10, sales rise to 140. Profit per unit $3. Weekly profit $420. The discount lost money even though sales rose 40%. Break-even: a $2 cut shrinks unit profit from $5 to $3, so you need 500 / 3 = about 167 sales (a 67% rise) to match the old profit. A rule worth remembering: a price cut must raise volume by more than the margin you give up.

6. Worked example B: how many shoppers does a test need?

Computed by script with the standard two-proportion formula, 5% significance, 80% power. [Illustrative inputs, standard method] Baseline conversion 2%. To detect a 10% relative lift (2.0% to 2.2%) you need about 80,700 visitors per variant, 161,000 in total. For a 20% lift, about 21,100 per variant. For 5%, about 315,000 per variant. So what: our small store with 10,000 visits a month (Lesson 1) cannot reliably detect a 10% lift in a month. A small store should test big changes, run tests longer, or test on a step with higher volume (such as add-to-cart) rather than final purchase. [Our view]

7. Worked example C: the false win

Test five offers against control at once, each at the usual 5% false-alarm rate. The chance that at least one looks like a winner by luck alone is 1 - 0.95^5 = 22.6%. With three it is 14.3%. If you pick the best-looking one and call it a win, about one in four or five "wins" is noise. Fixes: decide the single success measure before the test, run a pre-set length, and re-test any surprising winner. [General; our arithmetic] AI test-reading tools can make this worse: they will happily hunt for segments where one variant won. Slicing the data many ways is the same false-win problem in disguise.

8. Personalization and the law

  • New York passed an Algorithmic Pricing Disclosure Act (signed May 9, 2025, in effect November 10, 2025). It requires businesses that set prices using algorithms and personal data to show: "THIS PRICE WAS SET BY AN ALGORITHM USING YOUR PERSONAL DATA." The National Retail Federation sued to block it; on October 8, 2025 Judge Rakoff of the Southern District of New York dismissed the challenge. NRF filed a notice of appeal on October 31, 2025 (Second Circuit No. 25-2818), and the state's brief in that appeal is dated February 10, 2026. We did not obtain the Second Circuit's current docket, so check it for the latest status. A separate NY bill, the One Fair Price Act (S.8623B/A.9349B), passed both chambers on June 4, 2026. As of October 7, 2026, the official Assembly action log and Senate page show no signed or chaptered entry; do not treat this separate bill as enacted law. [Current official status: https://assembly.state.ny.us/leg/?Actions=Y&Summary=Y&Text=Y&bn=A9349&default_fld= ; https://www.nysenate.gov/legislation/bills/2025/A9349/amendment/B] [Sourced: NY General Business Law 349-a; NY Attorney General alert, Nov 2025; Governor's release, Nov 24, 2025; Reuters, Oct 8, 2025; S.D.N.Y. docket 1:25-cv-05500 on CourtListener; NY Attorney General's Second Circuit brief, Feb 10, 2026; NY Senate bill page A9349B, read Oct 7, 2026]
  • FTC: in January 2025 FTC staff published early findings from a study of "surveillance pricing" and found that intermediaries can use many kinds of personal data, such as precise location or browser history, to influence the price a consumer sees. These are staff findings from an ongoing study, not rules. [Sourced: FTC press release, Jan 17, 2025] Practical rule: test prices at random across all shoppers, or personalize offers (a coupon) rather than prices for the same item. Check local law. This is not legal advice. [Our view]

9. What it costs and what it breaks

  • Trust: shoppers screenshot prices. The Amazon episode ended in refunds and an apology.
  • Margin: promotions teach shoppers to wait. Track what share of sales happen on discount.
  • Cannibalization: a promotion on one item can pull sales from another.
  • Data: a pricing tool is only as good as clean sales, cost and stock data (Commerce Modules 3 and 9).
  • Tool fees: price-optimization tools often charge a share of the gains; ask how they measure "gains."

10. Monday checklist

  1. What decision will this test change? If none, do not run it.
  2. What is the one success measure (profit per visitor, not just sales)?
  3. How many visitors do you need (use Section 6)? Can you get them?
  4. Would a shopper who saw both prices feel cheated?
  5. Is any offer based on personal data? Check local law and write it down.

11. Try it yourself (no code, 30 minutes)

Pick one product. Write its price, unit cost and weekly sales. Compute profit. Compute the volume you would need to break even on a 10% price cut. Then ask: from what you know about your shoppers, is that rise realistic?

12. Self-check

  1. What made the Amazon 2000 test a problem? Shoppers saw different prices for the same DVD and felt cheated; Amazon called it a mistake and refunded 6,896.
  2. Price $12 to $10, unit cost $7, sales 100 to 140. Profit goes up or down? Down, $500 to $420.
  3. Why can a small store not detect a 10% lift in a month? It needs about 80,700 visitors per variant at 2% conversion.
  4. Five variants tested together, 5% alarm each. Chance of a false winner? About 22.6%.
  5. Safer way to personalize? Offers (coupons) rather than different prices for the same item, with legal check.

13. Where this is going

Shopping agents (Lesson 7) will compare prices across stores in seconds. A hidden price difference is easier to find when software asks every store at once. Stable, honest pricing gets more valuable. [Our view]

Sources

Review flags

  • RESOLVED by rewrite (Oct 7, 2026): appeal. The Second Circuit's own docket could not be retrieved (CourtListener's public RECAP copy covers only the district court). What is verified: Judge Rakoff's Oct 8, 2025 judgment for the Attorney General; NRF's notice of appeal entered Oct 31, 2025 (S.D.N.Y. docket); the appeal number 25-2818; amicus briefs Jan 13, 2026 (Washington Legal Foundation, U.S. Chamber) and the State's brief Feb 10, 2026 (DktEntry 56.1). The lesson says only that and tells readers to check the docket. It makes no claim that the appeal is undecided. Current appellate status remains unverified; the text makes only dated historical claims, not a claim about the appeal outcome or current enforcement.
  • RESOLVED: One Fair Price Act. The NY Senate page for A9349B reads "Passed Senate & Assembly" with no governor action shown (read Oct 7, 2026). The Assembly action log confirms both-chamber passage on June 4, 2026 (rechecked Oct 7); no signed/chaptered entry is shown.
  • Amazon 2000 details are Amazon's account plus Reuters; the dispute over whether prices tied to demographics rests on Amazon's denial. We state it as Amazon's statement.
  • Sample-size figures use the standard two-proportion formula (normal approximation); calculators differ slightly. Illustrative inputs.
  • Worked-example prices, costs and volumes are illustrative.
  • Not legal advice.

Claim ledger (Lesson 4)

#ClaimSourceVerifiedStatusRe-check
168 DVDs, five days, 20-40% discountsAmazon statement; CNN/Reuters2026-10-06Readnone
2Amazon called test a mistake; Bezos quoteAmazon statement2026-10-06Read, company statementnone
36,896 customers refundedCNN/Reuters2026-10-06Readnone
4NY law signed May 9, 2025, effective Nov 10, 2025, disclosure textNY AG alert Nov 2025; governor release Nov 24, 20252026-10-06Corrected by verifier2027-01-06
5Rakoff dismissed NRF v. James Oct 8, 2025; notice of appeal Oct 31, 2025 (2d Cir. 25-2818); State brief Feb 10, 2026Reuters; S.D.N.Y. docket (CourtListener); EPIC-hosted brief2026-10-07Rewritten to verified facts; appellate docket not retrieved2027-01-06
5bOne Fair Price Act passed both chambers June 4, 2026; no signed/chaptered entry as of Oct 7NY Senate page A9349B and Assembly action log2026-10-07Read; not stated as law2026-12-06
6FTC staff early findings Jan 17, 2025FTC release2026-10-06Read headline and summarynone
7Sample sizes (80,700 etc.), 22.6% false winOur computation2026-10-06Script-checkednone