OR/MS Today - February 2000

February 2000

Forecasting 2000
Predicting which product is best for your needs may be a daunting task

By Jack Yurkiewicz

Forecasting as a formal discipline has arrived. Almost all graduate and undergraduate programs cover some portion of forecasting in their quantitative modules. The three major quantitative areas (statistics, operations research, operations management) all call forecasting their own. As evidence, virtually every statistics, management science and operations management text on the market today has at least one chapter devoted to forecasting. So do most books on economics, finance and marketing. Receiving an undergraduate or graduate degree in business without being exposed to at least some basic forecasting concepts and techniques is thus almost impossible. Business practitioners have made both time-series analysis and causal forecasting essential factors of their analysis of the firm, and financial analysts routinely use forecasting techniques.

Many business practitioners use the forecasting component that comes with a general statistics program. Programs such as SPSS (with its Trends module), MINITAB, SAS, NCSS, etc. offer capable and robust forecasting features. Statistical practitioners of these programs have the advantage of no additional learning. That is, if the user knows and uses the program for statistics, then he or she can use its forecasting capabilities with no new training. However, these programs may not offer all the sophisticated forecasting methodologies available beyond exponential smoothing and Box-Jenkins models. They also generally do not offer the "automatic" features that a dedicated forecasting package routinely has.

As in the previous surveys, I have classified the software into three broad categories. I call the categories "automatic," "semi-automatic" and "manual." In automatic software, the user enters or imports data and asks the program to "analyze" it. Considering various diagnostic tests, the software responds with a "recommended" methodology (exponential smoothing, Box-Jenkins, etc.) that should give the "best" forecasts. If the user concurs with the recommendation, the program will then find the optimal parameters for the proposed procedure, get the forecasts and corresponding statistics (mean-square-error, Ljung-Box statistics, mean-absolute-deviation, etc.) and make forecast plots. The user can, of course, manually "override" the recommended procedure and choose his or her own preferred methodology. The software will then get the optimal parameters for the model and the associated output.

No general statistics programs with forecasting capabilities falls into this "automatic" group, while many dedicated forecasting programs do. Thus, these dedicated products offer even "statistically-challenged" users the promise of making accurate and useful forecasts. In the past, using "messy" data, I found that it was not difficult to get better results (lower MSE, or MAD, or Schwarz's BIC, or whatever the software's optimizing criterion was) by manually choosing a model and letting the program find the best parameters. However, that is not so with the most recent versions of forecasting software. Usually, the model that I proposed coincided with the one the program recommended. The key word in the above is "almost," and so the two generally accepted cautions are in force: 1. Statistical knowledge of forecasting is still necessary, and 2. Be wary if you are using these products as "black-boxes."

We call the second category "semi-automatic." Here, the user enters the data, but the program does not recommend a procedure. The user must choose some appropriate model from a list. The software will then find the optimal parameters for the model chosen by minimizing some criterion, make the forecasts, and get the appropriate statistics and plots. The user of such software must obviously have a solid knowledge of forecasting and the various associated techniques. All of the general statistics programs with forecasting modules fall into this category, as do some dedicated forecasting programs.

Using semi-automatic software, or automatic software in the "semi-automatic mode," it is important to know how the program finds the optimal parameters. For example, some software will find the optimal smoothing parameters for Winters' seasonal forecasting method using a nonlinear optimization approach, thus getting the results in a single iteration. Other products will use a grid search. These ask the user for lower and upper bounds for the parameters. The software then finds the optimal parameters within these ranges. Such products recommend that the user use a grid search with larger step sizes first and then "fine-tune" the search with smaller step sizes. Thus, it may take several "passes" before the user gets these parameters to three-decimal accuracy.

There are also different levels on just how "semi-automatic" a program is. Some products may automatically find the optimal parameters for Brown's simple exponential smoothing and for Holt's method, but not for Winters' model. Some will find the optimal parameters for Box-Jenkins models, but not for exponential smoothing ones.

We call the final category "manual." Here the user must specify both the method and the parameters. Thus, the user must execute many "runs" for a time series, each time noting the corresponding output statistics. There are few statistics programs that fall into this group.

I wish...

I have used many forecasting programs and general statistics programs with their forecasting modules. Using the software in the semi-automatic mode (I choose the model, the program optimizes the parameters), I noticed some disconcerting results. Taking a time series from a forecasting textbook that had obvious trend and seasonality, I wanted the software to find the appropriate multiplicative Winters' model. Rarely did two programs give the same forecasts for identical data when using an exponential smoothing technique. In particular, the optimal parameters were different and the resulting forecasts were different. Even when I entered identical smoothing parameters manually for the different products, invariably I got different forecasts and different mean squared errors. However, the programs did give identical results when using Box-Jenkins methodology. The problem, I believe, is that there are various ways to "initialize" the procedure. For example, with Winters' method, we have to make initial estimates for the level, trend and seasonal factors. Forecasts are functions of data set size and smoothing constants [1]. That is, if the data set is not large and the smoothing parameters are close to zero, these initial conditions play a larger role in the final forecasts. Unfortunately, many programs do not tell the user how they initialize the procedure. No program allows the user to choose a particular initialization scheme. Thus, I was infrequently unable to duplicate the results found in the forecasting text. I wish the software vendors would allow the user to choose a particular initialization method, or at least explain in the documentation what that procedure is.

Another bothersome detail is that some programs do not specifically tell the user what statistic they are trying to optimize. It could be the mean squared error, or mean absolute deviation, or the residual standard error or something else [2]. For some products, even when they did specify the optimizing criterion, I got a different result using the same specified parameters. I wish the vendors would document their optimizing criterion, and give the mathematical formula for that criterion, so users can duplicate the software's results.

While many programs claim to read Excel files, not all can read current Excel 8 format files (from Office 97 or Office 2000). Some programs require Excel 4.0 files or earlier. In addition, not all programs import Excel files in its "native" format, but require additional elaborate header information indicating a name for the data, the number of periods per year, number of periods per seasonal cycle and other information. This makes data importation more cumbersome. I wish the vendors would allow their products to import the latest Excel files seamlessly and get additional information via dialog boxes.

Survey Results

As with the previous surveys, the reader should understand that the results given are just summaries of the information supplied by the vendors of the software. Moreover, we only considered products from commercial vendors, and we identified those from advertising, word-of-mouth and displays at professional conferences. We did not attempt to verify the information supplied, so this is obviously not a critical review of the software.

How do you choose forecasting software? Besides choosing one of the three categories described, you should also determine the overall capabilities of the program. That is, what methodologies are available? However, more is not necessarily better. If you know that your techniques are confined to regression, exponential smoothing models and Box-Jenkins procedures, then perhaps a general statistics program with forecasting capabilities would suffice. On the other hand, if you generally use state-space models or neural networking, and you want some "advice" if such procedures might be appropriate, then your choices are more limited to a dedicated forecasting product. Even if the program does have the methods you require, the capabilities of these methods vary. Some programs limit the number of observations or data points, and if your data set is routinely large, then such programs become useless. Perhaps the program can do Winters' method, but will it permit damped or nonlinear trend in addition to the standard linear trend? Can it find smoothing parameters outside the standard zero-one interval?

The graphics output of these programs can vary. All will give you a time plot of the data and most can give a plot of the fitted results and the data. However, some can show forecasts, while others can show forecasts and confidence intervals for the forecasts. Some can give a simple plot of the autocorrelation and partial autocorrelation function, while others show the confidence limits for the correlations. These and other issues can be easy to resolve, as most vendors will readily supply you with the needed information. However, resolving some others may be harder. Just how easy is it to learn the program, and how easy is it to use, especially to the casual user who does not do forecasting every day. The documentation quality varies. Also, does the vendor supply help, and what is the level of support? Does technical support mean just a Web presence with a series of generic FAQ's, or can you actually get to speak with someone for answers to your specific questions? Is there a charge for live technical support, and is there an expiration date?

Finally, many of these products are expensive. See if you can download a trial "try-before-you-buy" copy from the web. SPSS, MINITAB, NCSS and others have such features. These copies are generally either the complete product that is limited to 30 days or a certain number of runs, or have minor forms of "crippling," such as not printing or saving. Those that force you to use the vendor's supplied data and do not allow you to use your own are little more than glorified slide shows and may not help you in your quest.

Be sure to read the accompnaying software survey.

References:

Makridakis, S., S. Wheelwright, R. Hyndman, ³Forecasting, Methods and Applications,² Third Edition, John Wiley and Sons, N.Y., 1998, page 150.

DeLurgio, S., ³Forecasting Principles and Applications,² Irwin/McGraw-Hill, N.Y., pages 51-56.

Jack Yurkiewicz (yurk@pace.edu) is a professor of Management Science at the Lubin School of Business at Pace University.

Table of Contents

OR/MS Today Home Page

Lionheart Publishing, Inc.
506 Roswell Street, Suite 220, Marietta, GA 30060, USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com