Ten Forecasting Tips & Caveats

1. On aggregating item (sku) forecasts.

Many companies that require both item-level and aggregate (e.g brand or product group) forecasts proceed via a bottom-up strategy: they forecast each item and then add the item forecasts to obtain a group forecast. Normally, however, the item-level data are difficult to forecast because they are volatile or short or interrupted. In such cases, a better strategy is the top-down strategy: the group is forecast directly (because group data are longer and more stable) and the item-level forecasts are adjusted so that they sum to the group forecast. The improvement in forecasting accuracy can be dramatic.

2. Excel and forecasting.

Excel will readily fit a Trend Line to the series. The forecaster needs to be aware however that the Trend Line is calculated by giving equal weight to all the data—the distant past is given as much weight as the recent past. So despite that trends change over time, the Trend Line does not adapt very fast and can produce outdated forecasts. Perhaps that is why the Trend Line was the least accurate of two-dozen extrapolative procedures compared in the famous M-Competition. Excel’s Trend Line is to be avoided for time series forecasting. Better to replace it with Exponential Smoothing procedures.

3. Consider Exponential Smoothing.

Exponential Smoothing is frequently an efficient, practical and effective way to forecast time series. It can be automated for batch processing, it can be applied to relatively short series and it can be adapted to different patterns of trend and seasonality. Forecasting competitions have shown it to be more reliable than more complicated methods of extrapolation. Perhaps its strongest attribute is its logical and tangible basis. In exponential smoothing, you estimate the current Level of your time series and then adjust it for Trends, Seasonality, and Special Events.

4. On measurement of forecasting accuracy.

Do not base your judgments of forecast accuracy on statistical measures of goodness of fit, which are also called within-sample statistics. Within-sample statistics only tell you how well a forecasting method can reproduce the historical data. To properly gauge forecasting accuracy, you need to perform out-of-sample tests: you must withhold some data from the historical series and use these data as test cases.

5. On acquisition of new forecasting software.

Do not expect instant gratification in terms of improved forecasting accuracy. You will need to effectively teach your forecasting engine about your data. This, in turn, can require a significant investment of your time in trial-and-error research. And the more you know about forecasting methodology, the better you’ll be able to train your forecasting engine.

6. On automatic forecasting.

Automatic forecasting relies on built-in features of forecasting software to choose an appropriate method for each of your time series. Like an automatic camera, automatic forecasting will work well in normal and usual circumstances. For difficult conditions, however, there is no substitute for your careful examination, judgment and expertise.

7. Special Events and Automatic Forecasting.

Statistical forecasting can never be relegated completely to the automatic forecasting capability of forecasting software. These automatic algorithms show great promise and their potential forecasting accuracy has been given a great boost by the M3-Competition. However, they are not generally capable of recognizing and adjusting for “special events.” A one-time special event occuring in a recent October, for example, may well lead to a model that erroneously projects upward spikes in all Octobers to come.

8. Don’t overtax the data.

Forecasting methods that can perform very effectively on good data—see the next tip—may break down when applied to time series that are short or interrupted. For example, the ARIMA models of Box and Jenkins should not be applied to annual or quarterly data that contain less than 20 observations or monthly data that contain less that 36 observations. Software may well go ahead and fit these models on request; however the results may ignore key patterns in the data and lead to implausible forecasts.

9. How much data is enough for monthly forecasts?

Monthly data are likely to be seasonal. To reliably fit and test statistical models, a time series of 48-60 months is desirable. At least 3 seasonal cycles are in order for estimating a seasonal model. With 48 months, for example, the first 36 can be used to fit the model and the last 12 to test the model’s forecasting accuracy. For short-term forecasting, going back more than 60 months is unlikely to be helpful, since most statistical methods assign more weight to the recent than to the distant past.

10. Beware of the prediction intervals (PIs) produced by forecasting software, especially for regression models.

Without exception, the PIs ignore sources of error that can be as serious as the components of error they represent. The result is that they mislead the forecaster into believing that the forecasts will be within a narrower range of the truth than is actually the case. For a discussion and proposed solution, see the 2000 article by Len Tashman, “Effect of Regressor Forecast Error on the Variance of Regression Forecasts,” with Thorodd Bakken and Jeff Buzas, Journal of Forecasting, 19, 587-600.