Out-of-sample validation vs overfitting: how to tell a forecast will actually predict

A location model is only worth trusting if it predicts stores it has never seen. Out-of-sample validation tests exactly that: you hide some real openings from the model, have it forecast them using only what was known before they opened, then compare the forecast to what actually happened. A model that scores well on stores it was trained on but poorly on hidden ones is overfit, and overfitting is the single most common reason a confident-looking accuracy claim falls apart in practice.

What is overfitting?

Overfitting is when a model learns the noise in its training data instead of the pattern. Give a flexible model enough freedom and it can "explain" every historical store almost perfectly, including the quirks that will never repeat. It looks brilliant in the demo and then misses on the next real site, because it memorised the past rather than learning what generalises. Any vendor can show you a chart where predicted matches actual on the data the model already saw. That chart tells you almost nothing.

What does out-of-sample validation actually involve?

The honest test is to evaluate the model on data it did not learn from:

• Hold out real openings. Set aside a set of stores the model never trained on.
• Forecast them blind. Use only the information available before each opened, so the model can't "cheat" with hindsight.
• Compare to reality. Measure the gap between forecast and actual revenue across those held-out sites, and report the distribution, not just the best cases.
• Do it on your estate. Accuracy on someone else's stores in another market is not accuracy on yours.

This is how IRIS reports performance: calibration, whether the 80% range we quote contains the real result about four times in five, rather than a single accuracy figure. That is the figure we are computing across every deployed model, checked out-of-sample on openings the model has not seen, and we will publish it with the misses included. See the methodology page for the full picture, including where the model is weaker.

How do I check a location-intelligence vendor's accuracy claim?

Use this as a checklist. The answers separate a validated model from a good-looking demo:

A vendor who can answer the right-hand column is describing a model that predicts. A vendor who can only answer the left-hand one is describing a model that fits. The gap between the two is where expansion budgets get lost.