Profiling a facility-demand dataset, testing what actually correlates, fitting a seasonal model and deciding what counts as an outlier — in that order.
What is in the table before anything is modelled — size, shape and whether the target is skewed enough to matter.
Recomputed from the rows the filters leave behind
Counts per bin, with the mean and median marked
Pearson correlation across the candidate features, computed on the current selection. Two features that carry the same signal should not both go into the model unchanged.
Ranked by absolute correlation
Trend plus an additive monthly component, fitted on the selection and scored against a seasonal-naive baseline on a held-out tail.
Shaded band is the 80% prediction interval
Whether the residuals behave. A model is only worth reporting once its errors look like noise.
Dashed lines mark the current threshold; shaded tails are what it flags
Individual observations beyond the threshold, worst first. Each row is one record with its own expected value — not a monthly total.
| Record | Building | Space type | Month | Observed | Expected | z | Direction |
|---|
The point of the project was the sequence, not the model: understand the distribution first, decide what counts as an outlier and why, and only then fit anything.
# 1. Profile before modelling — shape, gaps, and what is skewed df.describe(include="all") df.isna().mean().sort_values(ascending=False) # 2. Expected value per GROUP, not per month. # A residual against the month alone makes every record in that month # identical, which hides the variation the outlier test is looking for. grp = ["building", "space_type", "month"] df["expected"] = df.groupby(grp)["demand"].transform("mean") # 3. Robust standardisation. A handful of extreme records inflate the # standard deviation enough that they stop looking extreme. df["resid"] = df["demand"] - df["expected"] # Spread differs by space, so standardise inside the group too — # one global MAD just flags whichever spaces are biggest. med = df.groupby(grp)["resid"].transform("median") mad = (df["resid"] - med).abs().groupby(df[grp].apply(tuple)).transform("median") df["z"] = 0.6745 * (df["resid"] - med) / mad outliers = df[df["z"].abs() > 3.0] # 4. Only now fit. Trend + additive monthly component. from statsmodels.tsa.holtwinters import ExponentialSmoothing model = ExponentialSmoothing(train["demand"], trend="add", seasonal="add", seasonal_periods=12).fit() forecast = model.forecast(3) # 5. Score on a held-out tail against the baseline worth beating. mape = (abs(test["demand"] - pred) / test["demand"]).mean()
Why the median absolute deviation and not the standard deviation. A few genuinely extreme records inflate the standard deviation enough that they stop looking extreme — the outliers hide inside the threshold meant to catch them. MAD is not pulled around by the tail, so the flag list stays stable when a bad month arrives.
The baseline worth defending is seasonal-naive. A model that cannot beat "the same month last year" is not adding anything, and on facility data that baseline is surprisingly strong — checking against it first stops the whole exercise being a wasted week.
Built with pandas, NumPy, statsmodels and Matplotlib. The charts here are the same figures redrawn as SVG so the analysis reads in a browser without running a notebook.