Re-run and check
How IRIS validates a forecast out of sample
Any vendor can show you a chart where the forecast lines up with reality. The trick is which reality. Show a model the data it was trained on and ask it to reproduce it, and of course it can; a flexible model can explain almost every store it has already seen, including the quirks that will never repeat. That chart is a demo of memory, not of prediction, and it tells you close to nothing about the next site you sign. Validation is the discipline of refusing to be impressed by it.
Validation means forecasting real openings using only the information that would have been available before they opened, then comparing that forecast to what actually happened. The distinction is the whole game. A model that fits your historical stores is not the same as a model that predicts a store it has never seen, so we test the second, not the first.
Every forecast IRIS returns is shaped like this: a value, and the interval around it.
The Amsterdam Centrum forecast, illustrative. A single confident number has no width; the world does.
That is the object we validate. Not the point on its own, which precision alone can fake, but the point together with the range that says how much to trust it.
The test reads your holdout three ways
Run on your own estate, an out-of-sample test is the number that matters, and it asks three questions at once.
- The point. How close each forecast lands to the real outcome.
- The range, not just the point. A forecast's 80% interval should contain the real result about four times in five. The test asks whether the interval itself is honest, not only whether the midpoint lands close.
- The order. Whether IRIS ranks your held-out stores in the order their real results bear out.
Read against the line of perfect agreement, a well-calibrated forecast clusters along it: the closer a point sits to the line, the closer that forecast landed.
Illustrative. The shape a well-calibrated out-of-sample result takes, not a specific measured result.
That third read, the order, is the one an expansion decision actually turns on. A point forecast a year out has to fight the economy: conditions move between the day you sign a lease and the day the doors open, and no model controls that. What holds through the noise is the ordering.
Getting your strongest candidate correctly ranked as your strongest is far steadier than any single revenue figure.
A shift that pushes every forecast up or down together barely moves the order at all. So the ranking survives exactly the macro noise that a lone point estimate cannot.
Coverage is the meta-test
There is one more question, and it is the hardest to fake. Coverage asks not whether one point lands close, but whether the confidence we quote is the confidence you actually get. Say 80%, and the truth should land inside the band about four times in five, across many stores. Plot the confidence stated against the confidence delivered, and honest calibration hugs the diagonal.
Illustrative, not a measured IRIS result. A point on the diagonal means an interval we label X% contains the real outcome about X% of the time: say 80%, and roughly four in five land inside.
That is the difference between a confident number and an honest one. A confident number cannot be wrong on the page; an interval can, so it is the thing we can actually be held to. Treating an honest interval as worth more than a confident point is not a house style. It is the peer-reviewed research IRIS is built on: work on when a quantified claim deserves trust and when it does not.
Why there is no headline accuracy figure
We do not lead with a single site-wide accuracy percentage, and the reason is a matter of principle, not modesty. A number like that usually describes fit rather than prediction, and it is quoted before anyone has seen your data, your formats, or your market. Accuracy is not a property a model has in the abstract; it is what happens when a model meets your estate. So the honest version of this test is one we run with you, on your own held-out stores, during the Validation Sprint, and the accuracy we agree there is committed in writing when you implement.
Where the answer should be uncertain, it says so. IRIS has to beat the simple benchmarks you would otherwise lean on, footfall alone, a bought demographics report, last year's rule of thumb, or it has not earned the decision. And the interval is deliberately wider in a new market or with a small store count, because there the honest answer is less certain. New to the distinction between fitting and predicting? Start with out-of-sample validation vs overfitting.