December 1996 € Volume 23 € Number 6

Forecasting Software Survey

Guide to a fast-growing discipline: How to identify the appropriate product for your needs

By Jack Yurkiewicz

Forecasting is a discipline that has grown enormously in the past few years. Since 1986, the American Statistical Association, in its sponsored series of meetings, "Making Statistics More Effective in Schools of Business," has stressed, among other things, that instructors should expand series forecasting coverage in traditional business statistics courses. Today, virtually all statistics texts devote at least one chapter to the topic, and most operations management books also cover the subject. Business practitioners have made either time-series or causal (explanatory) forecasting an integral part of their analysis of the firm.

Because of these developments, many forecasting programs have entered the market. Some of these are stand-alone, dedicated forecasting programs, while others are modules of general statistical software. Most products are Windows' programs, and many have roots in DOS versions and even mainframe counterparts of a generation ago. Almost all allow the user to enter the data easily, either by typing in the values in a pseudo-spreadsheet, or by directly importing spreadsheets (Excel, Lotus, etc.), ASCII files and other file formats. All offer a variety of time-series and regression techniques. There is, however, variation in the scope of their capabilities.

We can categorize forecasting software into three groups. The first category can best be called "automatic" software. The user enters the data and tells the programs to analyze it. Considering various diagnostic tests, the software suggests that a particular methodology (Box-Jenkins, exponential smoothing, etc.) should be used on that data to get the best forecasts. If the user concurs, the software will then go on to find the optimal parameters for that method, get the forecasts and corresponding statistics (mean square error, mean absolute deviation, Ljung-Box statistics, etc.) and make forecast plots.

The user can, of course, ignore the recommendation made by the software and manually specify the technique. The software will then continue to get the necessary output. Thus, the user can use the product as a sort of "black box" to get forecasts, or use his or her understanding of forecasting techniques to choose the appropriate method. Forecast Pro is perhaps the archetype of automatic forecasting software.

I call the second software category "semiautomatic." The user enters the data but the software does not make recommendations on the appropriate procedure to use. The practitioner must specify the procedure. The software will then find the appropriate optimal parameters for that model, make forecasts, and get the corresponding statistics and plots.

Clearly, to use such software effectively, the user must have an understanding of forecasting and its various techniques. Many forecasting products fall into this category. However, there is a degree of how "semiautomatic" the program is. For instance, suppose the user wants to use Winters' method to get a forecast on the seasonal data in the Trends module of SPSS.

Because SPSS uses a grid search to find the optimal smoothing parameters, the user must first specify that these constants should be found, and then indicate the range for the search.

If additional accuracy is needed, the user can, after Trends finds the parameters to the nearest tenth, specify a finer step-size in the search. On the other hand, a program such as Sibyl Runner, which uses a nonlinear programming approach, will get the optimal smoothing parameters in one "iteration." The user need not specify the range for the search.

I label the third category "manual." Here the user specifies the model and the parameters. The software will not find the optimal parameters. Thus, the user must execute several "runs," each time noting the corresponding output statistics, and end the procedure either when the software deems these statistics best or when the user becomes tired, whichever comes first.

There are few commercial statistics or forecast products on the market that fall into the "manual" category. The forecasting component of MINITAB, for example, will find the optimal parameters for Box-Jenkins methods, simple exponential smoothing, Holt's method and others, but not for Winters' procedure.

What does a prospective purchaser and user look for in forecasting software? Besides knowing which of the above categories the program falls in, the user needs to know the overall capabilities of the product. One criterion is the breadth of methods that are available. A comprehensive "laundry" list of techniques is a first start, but such a cursory list can be misleading or incomplete. For example, all programs have regression analysis, but can the software find prediction intervals for new observations? Surely the program can perform a variety of exponential smoothing techniques, but does it allow a nonlinear or damped trend, or find smoothing parameters outside the zero-one interval (based on Gardner's 1985 research [1])?

Graphics output is another important consideration when choosing forecasting software. All products give time series plots of the data as well as the forecasts. But are other relevant graphs given? For example, does the program give a plot of the sample autocorrelation and partial autocorrelation functions, or just give a table of the results? If the software makes the plots, will it show the confidence limits for the correlations also? Finally, users may want to export these graphs into another application, such as a word processor, to include in a report. Can the program easily and effectively do this?

Such issues are easy to resolve, as most vendors can readily supply the pertinent information. Other issues are harder to ascertain. The program should be easy to learn and to use. The documentation should be complete. Ideally it should not just describe the software, but give users advice on the art of making forecasts, and perhaps even teach the methodology. Is there "on-line" help? Is the module in a general statistics product such as Statgraphics Plus, MINITAB or SPSS sufficient, or is a dedicated forecasting package needed? Finally, consider the price. Some products cost as much as a computer; is such a figure justifiable? Computer prices continue to fall, but not software.

The survey
In the survey we attempted to find as many forecasting vendors as possible. Using the results of an earlier OR/MS Today [2] survey as a springboard, we sought additional products. Many were no longer in business, and new companies have appeared in the interim. We mailed a simple one-page questionnaire to as many vendors as we could identify, and if a vendor failed to respond, a follow-up call was made directly. The resulting list is almost surely not comprehensive, and the reader may wonder why we do not list his or her favorite program. Perhaps we did not know the program, or the vendor failed to respond to the questionnaire. We only considered products from commercial vendors, identifying them through advertising or displays at professional conferences. Software bundled with texts, or written by a professor and used at his or her university but not elsewhere, while perhaps competitive, were not considered.

The results
The reader should understand that the results of the survey are just a summarization of the information supplied by the vendors of the software. We did not attempt to verify the information supplied. As such, this is not a review of the products. However, because I own and use several of the products, I would like to amplify some of the "yes" answers given by the vendors.

For example, all programs claim to read ASCII files. However, some require "special handling" before the software can read them. Sibyl/Runner, for instance, requires that the ASCII file consist of a column of numbers, with the first element an optional variable name. This is far more straightforward to prepare than that required by Forecast Pro, where the first six elements form a "header" for the data. This header consists of a variable name (which must be in quotes), variable description (again in quotes), starting year (an integer), starting period (also an integer), periods per year (an integer) and periods per seasonal cycle (an integer). It is difficult to remember all this without the documentation, and so preparing the data "outside" the software becomes more tedious.

Another point to make is the software's inflexibility in choosing initial conditions. Most programs will not let you set the initial conditions for the class of exponential smoothing models (Brown's, Holt's, Winters' procedures). They do not all use the same initial conditions. Thus, if you want to compare results from the software against some known example, perhaps from a textbook, you invariably will get different results. This can be frustrating.

You cannot even compare one product with another. Rarely do two programs get the same "optimal" parameters. I found the answers to vary widely on my test data, especially when doing Winters' method on erratic seasonal data. Who is "right?" The problem is that some products may try to minimize the mean square error of the residuals, while others try to minimize the mean absolute deviation, and still others try to minimize the Akaike Information Criterion. Generally, the user cannot specify what statistic should be optimized, and thus comparing results becomes difficult, if not impossible.

Be careful when you use automatic software. I found, especially when the time series was erratic, that the so-called "optimal" procedure and parameters chosen by the software could be bettered manually, using the software's own criterion statistic. That is, I could find a better model and parameters manually, while still optimizing the same statistic the software does when it makes its recommendation.

Obviously, the results of a one-page questionnaire cannot tell you everything there is to know about a product. A vendor may claim that his or her product "does trend analysis." To you that may mean the program will give the equation of the S-curve that fits the data, while a vendor may infer that only the linear trend line is found. Call the companies before you order. Try the inexpensive "student" version (if it exists) first. This usually limited version of the product (data sets may be smaller, or certain features omitted) can give you a good idea if the "professional" version is right for you. The purpose of this survey is to give interested readers an overview of what is currently on the market and to help them identify appropriate products for their research and teaching. You can contact me via e-mail, and perhaps I can help you with your quest. You can reach me at: j.yurkiewicz@worldnet.att.net.

References

1. Gardner, E.S., (1985), "Exponential Smoothing: The State of the Art," Journal Forecasting, Vol. 4, No. 1, pp. 1-28.

2. "Forecasting Software: Clearing Up a Cloudy Picture," OR/MS Today, Feb. 1993, pp. 64-75.

Click here to view the online version of the 1996 Forecasting Software Survey

Jack Yurkiewicz is a professor of Management Science at the Lubin School of Business, Pace University, New York. His current interests include technology in education. He is active in revising the curriculum to make the teaching and learning of quantitative methods easier and more meaningful with new technologies.

E-mail to the Editorial Department of OR/MS Today: orms@lionhrtpub.com

Lionheart Publishing, Inc.
2555 Cumberland Parkway, Suite 299, Atlanta, GA 30339 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com