A demand forecast is not an inventory decision. A forecasting model estimates future demand; a replenishment policy decides how much to order, when to order, and which constraints to respect. The distinction matters because a model that predicts sales more accurately on average can still produce higher operating costs or worse service levels.
The M5 retail forecasting competition provides a substantial reference point for evaluating forecasting models, including machine-learning approaches. But forecasting competitions are not inventory-control experiments. Research using M5 data has subsequently examined how forecast quality interacts with inventory policies and their costs. The relationship is not uniform across products or operating conditions.
A forecast becomes valuable through a decision
Consider a distributor replenishing a frequently ordered spare part. Its model predicts demand over the supplier lead time. The manager must then choose an order-up-to level while accounting for stock already available, outstanding orders, warehouse space, and the relative costs of shortages and excess stock.
Two models may have similar average errors but different behavior in the upper tail of demand. If shortages are expensive, correctly representing high-demand outcomes may matter more than achieving the lowest average absolute error. Conversely, systematically excessive safety stock can be costly for slow-moving products.
A small numerical illustration makes the point. In a one-period newsvendor setting, suppose an unmet unit costs eight monetary units and an excess unit costs two. The cost-optimal target is the 80th percentile of demand under the model’s distribution: 8/(8+2) = 0.80. The mean forecast is not automatically the appropriate order quantity. This is an illustrative calculation, not a measured business outcome.
Compare learning models with credible baselines
A sensible evaluation starts with transparent alternatives: seasonal naive forecasts, exponential smoothing, and an established regression or tree-based method. More complex supervised models may use promotions, calendar effects, store attributes, product hierarchy, stock availability, and supplier conditions. The objective is not to maximize model complexity but to test whether additional inputs improve decisions on unseen periods.
The split must respect time. Training on later observations and testing on earlier ones creates leakage. Promotional variables and supplier lead times must be represented as they would have been known at the time of a real decision. If recorded sales were limited by stockouts, observed sales should not be treated automatically as uncensored demand.
For probabilistic forecasts, evaluate quantiles and calibration as well as point error. A forecast claiming that 80% of future demand lies below a threshold should be checked against outcomes from comparable held-out periods. Average performance alone can conceal poor behavior in intermittent or exceptional-demand regimes.
Connect forecasting to replenishment
The next stage is to place each forecast into the same replenishment procedure. Keep the order constraints, holding costs, supplier assumptions, and exception-handling rules identical. Change only the forecast inputs under comparison. If each model is also permitted a separately tuned policy, report that as a second experiment rather than confusing model quality with policy tuning.
Useful operational measures include total holding and ordering cost, lost-sales or backorder cost, fill rate, stockout frequency, service-level violations, inventory turnover, and working capital tied up in inventory. Show these measures by product group and lead-time regime; aggregate averages may hide commercially important failures.
Use paired simulation, not optimistic extrapolation
Historical backtests are valuable but limited. A stochastic inventory simulator can stress the forecast-policy combination under lead-time disruptions, demand shifts, stockouts, minimum-order quantities, and capacity limits. Evaluate competing policies on the same independent demand paths or random seeds. This paired comparison reduces noise when estimating differences between decision rules.
A robust protocol reports more than one attractive scenario: include the baseline, adverse operating conditions, confidence intervals or uncertainty bands, and sensitivity to shortage-cost assumptions. Distinguish model fitting, policy selection, and final held-out evaluation. A result tuned repeatedly against the test set is not a genuine out-of-sample result.
The study by Theodorou and colleagues illustrates why accuracy and inventory performance should be assessed separately: relative cost structures and inventory policies change which forecast is preferable.
What a decision-maker should request
A practical acceptance report should identify the business decision; the forecasting horizon; input-data availability; baseline models; holdout periods; the optimization or replenishment rule; operational costs; uncertainty; and conditions under which the recommendation fails. A model should earn deployment by improving the decision it supports, not by producing a visually impressive error chart.
This framework applies beyond inventory—to production scheduling, staffing, capacity planning, and distribution—where supervised learning estimates uncertain inputs and optimization determines action.
Evaluate Distribution Policies Before Operational Deployment
Are you managing inventory across multiple warehouses, distribution centers, or retail locations? Are you considering machine learning to improve demand estimates, or simulation to compare replenishment and allocation policies?
Bamboos Consulting can help formulate distribution problems, integrate forecasting models with agent-based simulation, and develop controlled experiments to compare operational alternatives. The assessment can examine costs, service levels, stockouts, delivery constraints, and the robustness of decisions under uncertain demand and supply conditions.
The objective is to identify which policies perform better, under which assumptions, and whether the available evidence is sufficient to support an operational pilot.
Explore our Machine Learning & Optimization resources, discover related simulation and computational methods in the Technical Resources library, and follow further research and industrial applications on Substack.
References
- Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4), 1346–1364.
- Theodorou, E., Spiliotis, E., & Assimakopoulos, V. (2025). Forecast accuracy and inventory performance: Insights on their relationship from the M5 competition data. European Journal of Operational Research, 322(2), 414–426.