The forecaster who needs forecasting software has different choices from those of just a few years ago. The dealer market is segmented into two groups: those who sell stand-alone, dedicated forecasting software (e.g., Forecast Pro) and those who offer a general statistical analysis program (e.g., SAS, Minitab, NCSS, Systat, etc.) that includes forecasting techniques. The former group is smaller and says it caters to different and additional needs of users who do forecasting and might not be satisfied with the features and capabilities of the latter group. One company, SPSS, with its Trends module, says it falls into both groups. In the previous survey, I mentioned a "newer branch," programs that are add-ins to Microsoft Excel. This branch has grown; typically these are statistical offerings (e.g., StatTools) with forecasting features, but are less comprehensive than the stand-alone general statistical programs.

What to Consider


Broadly speaking, the main differences between the stand-alone, dedicated forecasting software and a statistical analysis program with forecasting features (whether it is stand-alone or an Excel add-in) can be categorized into "breadth of techniques" and "ease of use." Both groups of products will usually offer the time-series models for exponential smoothing or Box-Jenkins techniques (the Excel Add-Ins that I have worked do not have Box-Jenkins capabilities), but if you require a more exotic methodology, such as state-space analysis, a dedicated forecasting product is more likely to satisfy that need. In the past, it was in the "ease of use" category where the dedicated forecasting products excelled, but new versions of statistics products have narrowed that gap and may even have eliminated it.

As in previous surveys, I have broken down the forecasting software into three groups: designated automatic, semiautomatic and manual. Automatic software performs some diagnostic test on the data and then recommends the forecasting technique. It then optimizes the parameters of that technique by minimizing some statistic and gives the forecasts, plot and summary statistical measures. For example, using a data set that tracked motion picture box-office returns, Forecast Pro, an automatic program, working in its "Expert" mode, gave the results shown in Figures 1 and 2.

Figure 1: Forecast Pro's analysis of the data.

Figure 2: Forecast Pro's recommendation of the model and optimal parameters.

The software also gave a graph including 95 percent confidence intervals for the forecasts, various "goodness of fit" statistics [mean absolute percentage error (MAPE), mean absolute deviation (MAD), Box-Ljung, Bayesian Information Criterion (BIC) and others] and forecasts for the periods I specified. Other automatic products offer similar output. Of course, with automatic software, the user can veto the recommended technique and manually specify the forecasting methodology.

Semiautomatic software asks the user to specify the forecasting methodology for the input data, and the program then finds the optimal parameters by minimizing some statistic. For example, Figure 3 shows the dialog box from NCSS, a general statistical analysis program that falls into the semiautomatic group. Using the same box-office data, which had both trend and seasonality, I chose that (Trend-Season or Winters') method. The program permits work in the manual mode if I enter the three smoothing parameters, or offers to find the optimal ones if I choose a specific search method (e.g., minimize mean square error). The software then gives optimal parameters, the resulting forecasts, graphs and various statistical measures.

Figure 3: Dialog box from NCSS.

Finally, manual software asks the user to choose the model and the parameters for that model. For example, Figure 4 shows the dialog box from Systat, a general statistical analysis program that falls into the manual group. The user chooses exponential smoothing as the methodology, and the dialog box asks for the data and values for the three smoothing parameters. The software then gives the graph, forecasts and the accompanying statistics.

Figure 4: Dialog box from Systat.

All three groups have certain advantages but come with pitfalls. The automatic products may allow the novice or inexperienced user to make forecasts, but could lull some users into using the program as a crutch. For example, when I recently asked a particular practitioner why he chose the Box-Jenkins model, he responded, "The computer told me." Manual software gives the user complete control of the forecasting process, but finding the correct parameters of the model can be a tedious trial-and-error procedure.

The software should allow the user to withhold some portion of the data and find a fit for the rest. Many programs do not permit this. The software output can vary widely. Some products just give the forecasts, a graph of the data with the forecasts and a few statistical results. For example, almost all will give the MSE or the MDD, but far fewer give the Akaike Information Criterion (AIC), Ljung-Box statistic or the BIC. Experienced forecasters may want statistical tests on the within-sample errors, and, after the model is found, may also want the statistics for the out-of-sample errors. Many products will not do this.

The user should be aware of the nature of the time-series that he or she is analyzing. It may be too small for a particular procedure or have missing values that may preclude using the software at all. It may exhibit extreme volatility or have statistical outliers that could fool some programs, yielding poor forecasts. If the data does exhibit seasonality, the user should know the periodicity. Some products make this "pre-analysis" easier than others. All programs will give a time plot of the data so the user can visually detect anomalies, while a few offer warnings and suggestions.

One item that is difficult to ascertain is the "ease-of-use" of the software. If you use the product all the time, you may become acclimated to the software's idiosyncrasies and your "complaint level" may decrease. However, the occasional user may eventually stop using the product if the drawbacks outweigh the benefits. Since our last survey, some products that I tried and that previously exceeded my "complaint-level bar" have improved greatly. For example, all the products I tried imported my Excel 11 files seamlessly, and the operation was intuitive (sorry, I read the manual only as a last resort). My biggest complaint with some products is that I could not get the time-forecast plot into Excel for enhancement. With other programs, I found that saving the graphics output and trying to import it into Word became cumbersome at least and impossible at worst; the copy-paste sequence was the only solution.

One issue that is still open from our last survey is "duplication." By this I mean, if you and I both choose the same model with the same data, will two automatic forecasting programs give the same results (i.e., yield the same model, identical parameters for the model and the same forecasts)? Invariably, the answer is "no." The reasons vary, from which statistic the software (e.g., MSE or BIC) is trying to minimize to the number of data points the software is using to start the procedure (the initial conditions). I have found that even when I tried this test on pairs of semiautomatic or manual software, and I worked in the manual mode (i.e., choosing the model and specifying the parameters), the resulting forecasts differed, probably because the products differed on their choice of initial conditions. The documentation frequently gave no clue as to how the software made its calculations, and the program often did not allow the user a choice on which statistic it should monitor.

The Survey


We tried to identify as many forecasting vendors as possible, using advertising, displays at professional conferences, information from previous surveys and product reviews. We then e-mailed the vendors and asked them to respond to our online questionnaire. The survey asked for the capabilities and features of the software and allowed the vendor to include additional details not addressed by the questions. If a vendor failed to respond, we usually followed up with an e-mail and, at times, a telephone call.

Based on reader and vendor feedback, we have tried to make this edition as comprehensive as possible. Still, finding all available products on the market is problematical. So for those users and dealers who feel slighted by our omission, please accept our apology, let us know of your existence, and we will try our best to include you in our next roundup.

The Results


While the purpose of this survey is to let the reader become aware of some of the offerings, it does not purport to rate or review these products. However, the advice to help you choose forecasting software given in our past survey is still applicable. Experienced users will want to look at the forecasting techniques that are available in the software. Next, consider the level of automation of the product. I recommend a semi-automatic or automatic program for most users. However, other issues, such as the ease-of-learning and the ease-of-use of the software may be harder to evaluate. Getting a trial version of the software, if one is available, is the best way to resolve this. Almost all general statistics programs, but very few dedicated forecasting programs, offer these trial versions for downloading. Typically, they last for a short but adequate time for testing (10 to 30 days), but they may not be full-featured.

Even if the program does have the techniques and the level of automation you require, some features may vary. For example, Minitab finds the optimal exponential smoothing parameters for Holt's method, but not for Winters' method. Some programs limit the number of observations. A program might do Winters' method, but may permit only linear but not curvilinear trend, and the model may only be multiplicative, but not additive.

Vendor support is important. Is it free for some period? Is it "live?" That is, do you actually get to speak with another person, or is support limited to e-mail exchanges? Check to see if there are free patches for the current and previous versions of the software. Before you buy, call the vendor and ask questions. My experiences and feedback from other users have shown vendors to be uniformly helpful and eager to assist you in making the correct purchase for your needs.



Jack Yurkiewicz (yurk@pace.edu) is a professor of management science at the Lubin School, Pace University, in New York City. His current interest is "teaching with technology." His courses are built around Excel and add-ins. One class, euphemistically called "statistics without formulas," focuses primarily on data analysis, and his management science class emphasizes models and simulation. He is also working on learning assessment.

Table of Contents

OR/MS Today Home Page


OR/MS Today copyright © 2006 by the Institute for Operations Research and the Management Sciences. All rights reserved.


Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com


Web Site © Copyright 2006 by Lionheart Publishing, Inc. All rights reserved.