C4, Lesson 2: Forecasting with AI: What Actually Beats a Simple Method
Tags: Sourced / General / Our view / Illustrative. About 13 minutes.
1. The story: the forecasting contest where plain machine learning lost, 2018
The M4 Competition, organized by Spyros Makridakis and colleagues, asked teams to forecast 100,000 time series with 61 methods; it continued three earlier contests going back more than 45 years, and for the first time it included machine learning methods. In the authors' summary of the results (International Journal of Forecasting, 2018), five findings stood out: of the 17 most accurate methods, 12 were combinations of mostly statistical approaches; the biggest surprise was a "hybrid" method using both statistical and machine learning features, whose average sMAPE (an error measure) was close to 10% better than the combination benchmark; the second-best method combined seven statistical methods and one machine learning method; the two best methods also did well at 95% prediction intervals; and the six pure machine learning methods "performed poorly," none beating the combination benchmark and only one beating a naive method (Naive2). [Sourced: Makridakis, Spiliotis and Assimakopoulos, M4 Competition results and findings, International Journal of Forecasting 2018; The M4 Competition: 100,000 time series and 61 forecasting methods, 2020] The M4 result is not the last word. In the M5 competition (2020), which used daily retail sales data, the winner was a forecasting novice using the machine learning method LightGBM, and the organizers wrote that machine learning methods, trained across many series at once, had reversed the earlier finding. But less than half of the 2,666 teams beat a naive baseline, and only 7.5% beat the best statistical benchmark. The gains were large at the top of the product hierarchy (about 53%) and small at the lowest level (about 3%). [Sourced: Makridakis, Spiliotis, Assimakopoulos, "M5 accuracy competition: Results, findings, and conclusions," Int. J. Forecasting, 2022, read at the author post-print https://purehost.bath.ac.uk/ws/portalfiles/portal/233855891/IJF_2022_M5_Conclusions_post_print_.pdf; the organizers' own account] The same test applies to any newer model: beat the simple baseline on data it has not seen. The lesson is not "AI forecasting is bad." The best method was part machine learning. It is that how you combine methods and check them against a simple baseline matters more than which fashionable tool you use. [Our view]
2. The one idea
A forecast is only useful if it beats a simple rule at a cost you can afford, and you must measure it the same way every time. Use a naive baseline (last period, or same week last year), test any AI forecaster against it on data it has not seen, and prefer combinations over one fancy model. [General]
3. The kitchen-table version
You need to know how many bottles to order for next month. "Same as last month" is a forecast. So is "average of the last four months." A clever program may do better, but only if it beats those two on months it has not seen. If it does not, save the money. [General]
4. How forecasts are judged
- Error (WAPE): total absolute error divided by total actual sales. Easy to explain.
- Bias: whether you tend to over- or under-forecast. A biased forecaster costs more than a noisy one.
- Forecast value added: how much better than the naive baseline.
- Interval: a range, not just one number; the M4 winners also did well at 95% intervals. [Sourced: M4 findings] This builds on the forecasting module in the Commerce course; see Commerce Module 4. [General]
5. Worked example: simple methods against each other
All numbers [Illustrative], from a script with random demand. Weekly sales with a gentle upward trend and a bump every 13 weeks. Train on 39 weeks, test on the next 13.
| Method | WAPE (error) | Bias |
|---|---|---|
| Naive: repeat last week | 14.0% | +14.0% |
| Moving average, last 4 weeks | 7.3% | +6.3% |
| Linear trend on last 26 weeks | 7.3% | +6.5% |
| Average of moving average and trend | 7.2% | +6.4% |
| The simple methods cut the naive error in half. The combination barely beat either of its parts, and all of them were biased high, because the test weeks missed the bump. A machine learning model would need to beat 7.2% on weeks it has never seen and also fix that bias to earn its cost. If it cannot, a spreadsheet is the right tool. [Our view] | ||
| What a point of error costs. On $5,000,000 of annual sales, one percentage point of error is $50,000 of sales misforecast. If a quarter of that turns into markdowns or lost margin, a point costs about $12,500 a year. Compare that to the license and staff cost of a forecasting tool. [Illustrative] |
6. When AI forecasting earns its keep
- Many items (thousands of SKUs) where hand-tuning is impossible.
- Rich signals: price, promotion, weather, events, stockouts.
- New items with no history (using similar items).
- Combination and hybrid setups with a simple baseline as a sanity check. When it does not: few items, stable demand, bad data, or no one to act on the forecast. [General]
7. What it costs and what it breaks
- Data cleaning: stockouts show up as zero demand; promotions distort history.
- Bias: nobody notices a steady over-forecast until stock piles up.
- Opacity: a black box is hard to question; keep a baseline beside it.
- Overfitting: a model that fits history perfectly can fail on new weeks. Test on held-out weeks.
- The bullwhip: forecasting orders instead of sales feeds distortion (Lesson 1).
8. Monday checklist
- Pick your 20 top items. Compute naive and 4-week-average forecast error for the last 13 weeks.
- Compute bias, not just error.
- Ask any vendor to show their method against your naive baseline on your data, on weeks it did not see.
- Mark stockout weeks so they do not look like low demand.
- Decide who acts on the forecast and how.
9. Try it yourself (no code, 30 minutes)
In a spreadsheet, forecast the last 13 weeks of one item with "last week" and "average of last 4 weeks." Compute total absolute error divided by total sales for each. Which wins?
10. Self-check
- What did M4 find about pure machine learning methods? They performed poorly; none beat the combination benchmark.
- What type of method was the surprise winner? A hybrid of statistical and machine learning.
- In the example, did the combination clearly beat the simple methods? No; 7.2% against 7.3%.
- Why check bias? A steady over- or under-forecast costs money even if error is low.
- What baseline should any new forecaster beat? Naive or a short moving average.
11. Where this is going
Forecasting tools will get easier to use. The discipline of testing against a baseline on unseen data will stay the dividing line between value and hype. [Our view]
Sources
- Makridakis, Spiliotis and Assimakopoulos, The M4 Competition: Results, findings, conclusion and way forward (Int. J. Forecasting, 2018): https://ideas.repec.org/a/eee/intfor/v34y2018i4p802-808.html ; abstract read at: https://pure.unic.ac.cy/en/publications/the-m4-competition-results-findings-conclusion-and-way-forward/
- The M4 Competition: 100,000 time series and 61 forecasting methods (2020): https://www.sciencedirect.com/science/article/pii/S0169207019301128
Review flags
- I read the abstract of the 2018 paper (five findings) and the summary of the 2020 paper; I did not read either paper in full.
- The competition ran in 2018; the lesson avoids day-level dates.
- RESOLVED (Oct 7, 2026): M5 results added to the lesson from the organizers' IJF 2022 paper (read in full at the post-print). Per the verifier: the M6 competition (2022-23) is about stock prices, not retail demand, so it is not used. Newer deep-learning or foundation forecasters were not reviewed; the lesson makes no claim about them and tells readers to test any model against a naive baseline.
- The example is from a script with random demand; all numbers are illustrative.
Claim ledger (C4 Lesson 2)
| # | Claim | Source | Verified | Status | Re-check |
|---|---|---|---|---|---|
| 1 | M4: 100,000 series, 61 methods, built on 3 earlier contests | ScienceDirect 2020; IJF 2018 | 2026-10-06 | Read abstracts | none |
| 2 | 12 of top 17 were combinations; hybrid 9.4% better (sMAPE, per verifier); second best 7 statistical plus 1 ML | IJF 2018 abstract | 2026-10-06 | Read abstract | read paper |
| 3 | Six pure ML methods performed poorly; one beat Naive2 | Same | 2026-10-06 | Read abstract | read paper |
| 4 | Forecast-error table and cost-of-a-point | Our computation | 2026-10-06 | Illustrative, script-checked | none |
| 5 | M5 (2020): LightGBM winner; 7.5% beat best statistical benchmark; ~53% top-level vs ~3% bottom-level gain | Makridakis et al., IJF 2022 post-print | 2026-10-07 | Read in full | none |