Software Survey
Forecasting Software Survey
Predicting which product is right for you
By Jack Yurkiewicz
The analysis of time-series data permeates most arenas in business, science, social science, medicine, even sports. Movie studios chart the weekly box office returns of their films, the government monitors unemployment figures and inflation rates, the Los Angeles Lakers track Shaquille O'Neill's free-throws, doctors monitor cholesterol values as patients try different medicines. The examples go on and on. Thus, forecasting theory and methodology is a basic staple in most business school curricula, even as other quantitative areas have seen a decrease in coverage.
All business statistics textbooks now include at least one chapter on forecasting, most management science texts do the same, and ditto for all operations management books. Getting an undergraduate or graduate degree in business without at least some rudimentary coverage of forecasting is unlikely, and many business schools are offering an elective course on time and frequency domain forecasting theory and procedures.
Whether practiced by students or stock market analysts, forecasting is completed with the aid of computer software. Recognizing this, over the past two decades, many companies introduced dedicated forecasting software into the market. These products provided many forecasting methodologies, and were touted as easy-to-learn and easy-to-use.
Companies that had general statistical analysis programs, such as SPSS, Minitab, SAS, NCSS, etc., incorporated forecasting features or modules into their general products. Since users generally go to one source for their statistical and forecasting needs, these products have flourished, and some dedicated forecasting companies have floundered. Compared to our previous survey, made two years ago, the list of dedicated forecast vendors has decreased. However, those that have survived, along with new products that came into the market, generally offer far more forecasting "firepower" than the general statistical competition.
If you want to do a state-space analysis, for example, you will not find that capability in Minitab, SPSS, NCSS or most other general statistics program. Generally, all-purpose statistical software do not offer anything beyond exponential smoothing models and univariate Box-Jenkins procedures. Also, the dedicated products are frequently (but not always) easier to use, chiefly because of their enhanced "automation" power.
Automation is the key in assessing a program's "ease-of-use" rating. As in the previous surveys, we have classified the software into three broad classifications. Readers have shown that they find this grouping useful in helping them choose a product. We call these categories "automatic," "semi-automatic" and "manual." In automatic software, the user enters or imports data and asks the program to "analyze" it. Considering various diagnostic tests, the software responds with a "recommended" methodology (exponential smoothing, Box-Jenkins, etc.) that should give the "best" forecasts. If the user concurs with the recommendation, the program will then find the optimal parameters for the proposed procedure, get the forecasts and corresponding statistics (mean-square-error, Ljung-Box statistics, mean-absolute-deviation, etc.), and make forecast plots. The user can, of course, manually override the recommended procedure and choose the preferred model; the software does the rest.
None of the general statistics programs with forecasting capabilities can be classified as "automatic." For example, when I used SPSS with Trends and wanted to use a Box-Jenkins model, the program asked me to specify a value for autoregressive, difference, moving average (the traditional "p," "d," and "q" values). Thus, before a practitioner can use Trends, he or she must have some knowledge of what these values mean and do, and how to estimate them.
When I tried the same data set in Forecast Pro, an automatic product, it recommended Winters' method for my data. I specified Box-Jenkins instead, and it found the parameters for the model. It then compared various statistical measures for the two procedures to justify its choice of Winters' model. Thus, these dedicated products allow even statistically challenged users the ability to make accurate and useful forecasts. In fact, a few advertisements for some of these products essentially promise just that. This, at best, is misleading because a novice, without some statistical knowledge, may use the product as a "black-box," and not understand some of the ramifications and pitfalls of statistical forecasting.
In "semi-automatic" software, the user enters the data, but the program does not recommend a procedure. The user must choose some appropriate model from a list. The software will then find the optimal parameters for the model chosen by minimizing some statistical criterion, make the forecasts, and get the appropriate statistics and plots. The user of such software must obviously have a solid knowledge of forecasting and the various associated techniques. Almost all of the general statistics programs (SPSS, NCSS, Minitab, etc.) with forecasting modules fall into this category, as do some dedicated forecasting programs.
When using forecasting software, it is important to know how the program finds the optimal parameters for the chosen model. Most programs find the parameters by minimizing the MSE (mean-square-error), or related measures (SSE, or sum-or-squared errors, or RSE, or residual-standard-error). Some will try to minimize MAD (mean absolute deviation) or MAPE (mean absolute percentage error). Other programs will try to minimize the BIC (Bayesian Information Criterion) or AIC (Akaike Information Criterion) [1]. A few programs will give you a choice of which statistic to minimize, while others don't even tell you what they are minimizing. As a result, depending on the software product, you may get different "optimal" parameters and different forecasts for the same model.
Another issue is how the programs find their optimal parameters. For example, some software will find the optimal smoothing parameters for Winters' seasonal forecasting method using a nonlinear optimization approach, thus getting the results in a single iteration. Other products will use a grid search, asking the user for lower and upper bounds for the parameters. These products recommend a grid search with larger step sizes first and then "fine-tune" the search with smaller step sizes. Thus, it may take several "passes" before the user gets these parameters to three-decimal accuracy.
For "manual" software, the user must specify both the method and the parameters. Thus, the user must execute many runs for a time series, each time noting the corresponding output statistics, and eventually choosing the "optimal" parameters after he or she is satisfied ...or sufficiently tired.
The degree of automation can vary within a program. Minitab, for example, will automatically find the optimal parameters for Holt's method, but not for Winters' model. Some programs will automatically get the optimal parameters for exponential smoothing models, but not for Box-Jenkins models.
Another issue that may befuddle users is that different products give different parameters (and thus different forecasts) for the same exponential smoothing model and the identical minimizing statistic. The explanation comes from how the program sets its initial conditions to find the "correct" parameters. To test this, I wrote an Excel template that employs the Winters method, using the initial conditions as originally proposed by Winters [2]. The template then uses Excel's Solver to minimize the MSE, gets the model's parameters, makes forecasts and plots. The results I got were similar to those from some commercial programs, and vastly different from those of others. Which program is "correct?" Technically, they all are, based on the different assumptions of how they start the search. What is disconcerting is that some programs say nothing in their documentation about how the model's parameters are obtained, including what statistic is being minimized.
The Survey
As in the past, we attempted to identify as many forecasting vendors as possible. We then e-mailed a simple questionnaire, which asked for the capabilities and features of the software, to those vendors we could find. If a vendor failed to respond, we usually followed up with another e-mail and a phone call. The resulting list is almost surely not comprehensive, and we will always get readers and vendors later who complain that we did not include a certain product. To those we can only apologize and promise to add them to the online version and include them in the next survey. We identified the vendors from advertising, word-of-mouth, displays at professional conferences and previous surveys.
The Results
As with the previous surveys, the reader should understand that the results given are just summaries of the information supplied by the vendors of the software. We did not attempt to verify the information supplied, and so, this is obviously not a critical review of the software.
How do you choose forecasting software? I believe the advice given in our past surveys is still applicable. Choose an automatic or semi-automatic program, but you should also determine the overall capabilities of the product. That is, what methodologies are available? More is not necessarily better. If you know that your techniques are confined to regression, exponential smoothing models and Box-Jenkins procedures, then perhaps a general statistics program with forecasting capabilities will suffice. On the other hand, if you generally use state-space models or neural networking, and you want some advice if such procedures are appropriate, then your choices are limited to a dedicated forecasting product.
Even if the program does have the methods you require, the features of these methods vary. Some programs limit the number of observations or data points, and if your data set is routinely large, then such programs become useless. If you need to periodically forecast numerous times and you want to automate the process, pick a program that can do batch forecasting. Perhaps the program can do Holt-Winters' method, but will it permit damped or nonlinear trend in addition to the standard linear trend? It may permit a multiplicative model, but perhaps not an additive one.
These and other issues can be easy to resolve, as most vendors will readily supply you with the needed information. However, resolving some others issues may be harder. Just how easy is it to learn the program, and how easy is it to use, especially to the casual user who does not do forecasting every day? I have found the documentation quality to vary widely. Some are little pamphlets and reiterate what is in the on-disk help system, while others will give good tutorials on how to use the package. Others go as far as to give advice on the art of forecasting itself and teach the methodology. Also, does the vendor supply help, and what is the level of support? One extreme for technical support is just a Web presence with a series of generic FAQs, while at the other extreme you actually get to speak with someone for answers to your specific questions. What is the charge, if any, for live technical support?
Finally, many of these products are expensive. See if you can download a trial "try-before-you-buy" copy from the Web, usually called a "demo." Autobox, SPSS, MINITAB, NCSS and others allow you to do this. These copies are generally either the complete product that is limited to some time duration or a certain number of runs, or have minor forms of "crippling," such as not printing or saving. Beware of those "demos" that are little more than sales brochures, or force you to use only the vendor's sample data.
References
- DeLurgio, Stephen A., 1998, "Forecasting Principles and Applications," Irwin McGraw-Hill.
- Winters, P.R., 1960, "Forecasting Sales by Exponentially Weighted Moving Average," Management Science, Vol. 6, pp. 324-342.
Jack Yurkiewicz (yurk@pace.edu) is a professor of management science at the Lubin School, Pace University, in New York City. His current main interest is "technology in the classroom." In particular, he designed and teaches the Web-assisted courses in management science and statistics in the MBA program, and is working on learning assessment.
OR/MS Today copyright © 2003 by the Institute for Operations Research and the Management Sciences. All rights reserved.
Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com
Web Site © Copyright 2003 by Lionheart Publishing, Inc. All rights reserved.