Normalized demand history becomes forecasts that can defend themselves: SBC-routed candidate batteries, rolling-origin cross-validation, a finite-only championship, and a CI gate that fails the build if the median stops beating seasonal-naive on 787 real-shaped series.
M2 ends at a confirmed mapping. M3 is everything between that click and a per-series forecast that can defend itself: a complete panel, a routed battery of candidate models, a rolling-origin backtest, and a champion chosen by measurement rather than by preference.
Every month between the first and the last is reconstructed and zero-filled, so a gap means “no demand” rather than “no row”. ADI and CV² are computed on that panel and each (sku, customer) lands in an SBC quadrant.
Rolling-origin cross-validation scores every candidate by MASE with WAPE alongside; the best finite candidate wins, is refit on the full history, and carries its whole candidate list into the payload so the choice stays auditable.
Confirming a mapping spawns the Modal forecast job; the signed callback persists
series, forecasts and dq_report, flips the job to complete
and deletes extracted.json — the second half of the retention promise.
Same two-stage shape as ingestion, same signing primitive, a different job. Hover any node to isolate its edges.
mapping_json, opens a stage='forecast' job and invokes Modal. A failed invocation marks both rows failed rather than leaving a job hanging.extracted.json M2 wrote — the single input contract of this stage.R2.delete() runs once the forecasts are durable, so the intermediate parse artifact is destroyed too. After this only derived data exists.requires_proxy_auth=True on the forecast endpoint, same credential class as parse.apply_mapping with the confirmed columns, then the signed callback. Any exception becomes a failed job, never a silent stall.series.json, forecasts.json, dq_report.json under datasets/{uid}/{ds}/. M4 serves exactly these.revision — the cache key M4 is built on.run_forecast directly and fails the build outside the band.
Its input is extracted.json plus the columns the user confirmed — not the workbook,
which no longer exists by then. Re-forecasting a dataset therefore never needs the original file,
which is exactly what makes the retention promise survivable as a product decision.
Average demand interval on one axis, squared coefficient of variation on the other. The thresholds are the classical Syntetos-Boylan-Croston cut-offs, and the two batteries deliberately overlap.
Orders arrive with gaps but in similar sizes. Sparse battery: CrostonClassic,
CrostonSBA, TSB(0.2, 0.2), ADIDA, IMAPA,
plus Naive and HistoricAverage.
Gaps and wildly varying sizes — the hardest quadrant and the one where a normal interval is most confidently wrong. Same sparse battery.
Regular orders, stable sizes. Dense battery: AutoETS(12),
AutoTheta(12), a constrained AutoARIMA
(max_p/q=2, max_P/Q=1, season 12, nmodels=20,
approximation on), plus CrostonSBA and HistoricAverage.
Orders every month, sizes all over the place. Same dense battery — the volatility is in the magnitude, not in the arrival process.
SeasonalNaive(12) joins both batteries only above 12 months
CrostonSBA sits in the dense battery and Naive/HistoricAverage
sit in the sparse one, so an intermittent series can still be won by a smooth-family model and the
reverse. The quadrant narrows the search — it never decides the answer. Below 12 months of
history the dense battery collapses to CrostonSBA + HistoricAverage and no
season-12 model runs at all.
Every candidate is backtested on the same folds, scored, then filtered by a rule that has nothing to do with accuracy: it must produce finite numbers everywhere.
| months | h | windows | step | MASE season |
|---|---|---|---|---|
| ≥ 24 | 6 | 4 | 3 | 12 |
| ≥ 18 | 6 | 3 | 3 | 1 |
| ≥ 12 | 3 | 3 | 2 | 1 |
| < 12 | no cross-validation — every series takes the named fallback | |||
Below 24 months the MASE denominator is the one-step difference rather than the 12-month one, so scores stay comparable inside a panel but not across panels of different lengths. If a seasonal difference is identically zero — a perfectly flat series — the scale falls back to the mean absolute first difference, so the series is still scored instead of dividing by zero.
| 1 | Sort candidates by mean MASE across folds; ties break toward the simpler model via MODEL_PREFERENCE. |
| 2 | Walk that order and take the first candidate whose CV predictions are all finite and whose full-history forecast is all finite. |
| 3 | No candidate survives → named fallback with an explicit fallback_reason. |
| 4 | Champion is refit on the whole history; means are clamped at zero. |
| 5 | Intervals: the model's own 80% band for AutoETS/AutoTheta/AutoARIMA, otherwise the 0.8 quantile of absolute CV residuals, clamped so lo ≤ mean ≤ hi. |
| 6 | The full candidate list, both scores, the interval method and the seasonal-naive comparison ship inside the payload. |
A season-12 model on a barely-longer-than-a-season panel returns NaN for the folds it cannot fill. Asserting finiteness at the end would fail the whole job for one bad series; making it part of the championship simply moves that series to its next-best candidate. The regression test locks the behaviour, and the payload is guaranteed strictly-finite JSON — which is what lets M4 render charts and M5 write a spreadsheet without defensive checks. See finding R1.
“The model adapts to their data” is a marketing sentence until something measures it. The benchmark job runs the engine directly against a committed 24-month fixture on every CI run and fails the build outside the band.
The champion median is roughly 29% better than the
seasonal-naive baseline measured on the same folds. The two champion rows are the same engine before
and after the honesty fix in #5: the baseline used to be credited with a fabricated 0.00 wherever
SeasonalNaive was excluded from the battery, and removing that lie moved the reported
median from 0.720 to 0.745. See finding R2.
modal-app/tests/test_benchmark.py runs in its own CI job
(uv run pytest -q tests/test_benchmark.py -s) and is excluded from the fast suite via
-k "not benchmark", so a 100-second engine run never blocks the unit tests.
The second assertion is the one that matters: an absolute ceiling can be met by a lucky fixture, but beating the seasonal baseline on the same folds cannot.
scripts/qa/m3.sh drives upload → confirm → forecast against a real Worker with
the real Python jobs, then asserts every series carries a champion, a backtest score or a named
fallback, and a 6-month horizon — and that the intermediate artifact is gone.
17 files, +20,049 −6 — of which 18,889 lines are the committed benchmark fixture. The engine itself is 203 lines.
Committed rather than generated so the benchmark measures the engine and nothing else. See finding N5.
Two defects the milestone found in itself — both about numbers that looked fine — and five decisions that explain the engine. Click any row to expand.
Every series now carries a champion chosen by a backtest, the candidates it beat, both error measures, the interval method, and a seasonal-naive comparison — and CI refuses to go green if the median stops beating that baseline. Two of the milestone's own findings are cases where a number looked right and was not; both were caught by running the thing rather than by reading it, and both are now locked by tests.
revision, which becomes the cache key.