Every forecasting project I review reports an accuracy number. Almost none report what a naive repeat of last year would have scored on the same data. Without that second number, the first one means very little. Forecast accuracy without a baseline is theater.
An accuracy number says more about the demand than the model
Take two product lines. One is a staple with steady weekly sales, and almost any method will forecast it within 10%. The other is promotional and lumpy, and a strong method might still miss by 40%. A report showing 90% accuracy on the first and 60% on the second says nothing about where the forecasting effort paid off. The stable line may have been just as accurate with no model at all. The difficult line may be where the model earned its cost.
Accuracy on its own measures how forecastable the demand is. It does not measure what the forecasting added.
The baseline is the simplest forecast you can defend
The fix is a baseline: the simplest forecast anyone could produce from the same history. For most businesses that is “copy last year, shifted for calendar,” so that the same weekdays and holidays line up. It needs no software and takes an afternoon to compute.
If your model can’t beat it, you paid for complexity, not for foresight. And the gap between model and baseline, not the accuracy number itself, is the only honest measure of what the investment returned.
Example
What 5,507 teams learned on Walmart data
The largest public test of this idea is the M5 competition, run in 2020 by the forecasting researcher Spyros Makridakis and colleagues. Teams received several years of daily unit sales for 42,840 product and store series from Walmart and were asked to forecast the following 28 days. The organizers scored a set of simple benchmarks on the same data, among them a naive forecast, a seasonal naive forecast and plain exponential smoothing.
The competition drew 5,507 teams, many of them using machine learning. Fewer than half, 48.4%, beat the naive benchmark. About a third, 35.8%, beat the seasonal naive one. Only 7.5% beat exponential smoothing, a method that dates from the 1950s.
The winners did beat it clearly, by more than 20%, so the lesson is not that sophisticated methods fail. It is that sophistication carries no guarantee. Most of the effort in that competition produced forecasts that a textbook method would have matched or beaten, and the only reason anyone knows is that the benchmarks were scored alongside.
What to ask before approving the next initiative
Before approving your next forecasting initiative, ask one question: what does the dumbest possible forecast score? If nobody in the room knows, the project isn’t ready. Then make the comparison a standing practice.
- Score the baseline every cycle, on the same items and periods as the model, and report both side by side.
- Report the gap by product family or demand pattern. The average hides where the model helps and where it hurts.
- Count planner overrides the same way. An adjustment that makes the forecast worse than the baseline is a cost, however well intended.
- Pay for the gap. A vendor or an internal team should commit to an improvement over the baseline, not to an accuracy figure.
We hold our own work to the same test. In a recent validation for a consumer goods business, the forecast was scored on held-out months against both same-month-last-year and the company’s existing planning rule, and the comparison is published with its limits. A forecast that cannot show its gap over the simple alternative has not yet shown its value.
Sources: Makridakis, Spiliotis and Assimakopoulos, “M5 accuracy competition: Results, findings, and conclusions,” International Journal of Forecasting, 2022.