Operations Research Management Science - Decision Analysis Software Survey

<BODY bgcolor="#ffffff" link="cc0000" vlink="cc0000" text="#000000"> <img src="../cleardot.gif" width=1 height=1 vspace=4 border=0><br>   <table border=0 cellspacing=0 cellpadding=0 width=480><tr valign=top> <td width=10><img src="../cleardot.gif" width=1 height=1 hspace=4 border=0></td> <td width=470> <FONT face="verdana, arial, helvetica" SIZE=2><B><I><A HREF="../../ORMS.shtml" TARGET="_top">OR/MS Today</A> - June 2002</I></B></font><BR> <hr size="1"> <img src="cleardot.gif" width=1 height=1 vspace=4 border=0><br> <font face="times new roman, times"> <img src="../cleardot.gif" width=1 height=1 vspace=4 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=2><B>Software Survey</B></FONT><br> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=8 border=0><br> <FONT SIZE="+2"><B>Decision Analysis:<BR> Aiding Insight VI</B></FONT><BR> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> <FONT SIZE="+1"><I>'It's Not Your Grandfather's Decision Analysis Software'</I></FONT><BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=2><B>By Daniel T. Maxwell</B></FONT><br> <hr size="1"> <img src="../cleardot.gif" width=1 height=1 vspace=12 border=0><br>    This is the sixth biennial survey of decision analysis software published in <I>OR/MS Today.</I> Over the years, these surveys have been published as the market for decision-supporting software has continued to expand, as has the variety of modeling techniques that are being identified as decision analytic by software vendors. In fact, many of the software packages identified in this survey would be barely recognizable as decision analysis software to many "old school" decision analysts. The emerging tools employ optimization, simulation and artificial intelligence techniques that blur the historical distinction between decision analysis and other analysis disciplines. The removal of these historical boundaries can have mixed results in individual analyses. On the positive side, the creative combination of techniques complemented by the extremely powerful computing and visualization capability that is available potentially provides very robust tools to a broader community of analysts. That said, all of the modeling techniques encoded in software have their own underlying axioms, assumptions and limiting conditions. And, the risk of unwittingly creating an apparently elegant and informative model that violates these conditions  potentially misinforming decision-makers  has never been higher.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> In response to these changes in both the state-of-the-practice and the marketplace, we will first briefly discuss traditional decision analysis and the relationship to other techniques that are being blended into "decision analysis" software. We will then discuss the survey and its results, and conclude with some considerations and recommendations for analysts, software consumers, and decision-makers that strive for high-quality analysis results.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=12 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=3><B>'Decision Analysis'</B></FONT><BR> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> Historically, decision analysis has been thought of as a process supported by quantitative modeling and structured elicitation techniques which are intended to improve the quality of complex decisions. Complex decisions in this context refer to ill-structured situations possessing multiple (usually competing) objectives and uncertainty concerning outcomes. The intent of the process is to provide a decision-maker or group of stakeholders with insight into their preferences as they relate to the relative priority they place among objectives, as well as concerning their attitudes toward risk. The decision analytic literature identifies four basic stages of the process (often using slightly different terminology): 1. problem structuring, 2. value elicitation and modeling, 3. probability elicitation and modeling, and 4. exploration of the computed results (sensitivity analysis). The process is necessarily iterative. For instance, insights achieved through sensitivity analysis might cause a decision-maker to reassess their value judgments; likewise, disciplined thinking about values might stimulate the identification of a new alternative. While these techniques have become increasingly powerful, the focus in the decision analysis community has been on developing prescriptive modeling and elicitation techniques, and supporting software that enhances human judgment and expertise.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> Operations research analysts, computer scientists, decision analysts and other professionals have been reaching into the decision analytic sciences from across the boundaries of traditionally independent intellectual areas with increasing frequency. The result has been the development of models and software applications that address even more complex decision situations. The focus of many of these efforts has been to apply techniques rooted in other disciplines to enhance the robustness of models and the model development process. For instance, brainstorming techniques that have their intellectual roots in the psychological and business administration literature now often support the structuring stage of the decision analysis process. Similarly, "traditional" decision analysis emphasized selecting a single alternative from a set of possible courses of actions. Now, optimization techniques such as mathematical programming are increasingly being integrated with decision analytic techniques to support resource allocation modeling involving very complex sets of constraints and potential combinations of alternatives. Group tools now support the value elicitation stage of the process using techniques that have their roots in voting theory  the psychology of consensus building  and are enabled by engineering achievements that allow distributed group activities and model development.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> The ability to address uncertainty is an essential tenet of decision analysis. Historically, decision analysts addressed uncertainty through the structured elicitation of subjective probability distributions. These judgments were then represented in decision trees and later in influence diagrams. Advances within the decision analytic and computer sciences have yielded Bayesian Networks capable of representing and solving very complex probability models, and capable of learning probability distributions from data. Additionally, descriptive modeling techniques, such as Monte Carlo simulation, are being coupled with multi-attributed functions to provide insight into very complex processes and systems with unprecedented ease.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> Graphical user interfaces are significantly simplifying the model elicitation process. Graphical model structuring tools are often now supported by underlying data models that support full numerical specification of a model using the graphical structure as a foundation. This functionality has led to the use of decision analysis techniques like influence diagrams to represent cognitive models of complex systems, and to then use the underlying computational engine to explore the behavior of the system. Similarly, GUI-based probability and value elicitation has reached a level of maturity that can in some circumstances support "machine assisted" elicitation, potentially easing the labor burden on analysts and increasing the "reach" of subjective data collection. Finally, the combination of powerful visualization and GUI-based models now allows analysts to conduct sensitivity analysis in real time, often with decision-makers and stakeholders as active participants. All of these advances significantly increase the decision analysis techniques and software in support of the difficult, ill-structured problems.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> There are risks and issues that accompany these technological advances. It can almost be the case that constructing a decision analytic model can be too easy. Many of the insights decision-makers achieve are because of the focus provided by a trained decision analyst. A good model structuring tool and process does not just easily place and populate nodes with data. It ensures that the nodes represent the stakeholders' essential objectives and the key variables that affect outcomes with respect to those objectives, and captures the key relationships among model variables. A skilled decision analyst probes for these relationships and then selects a tool that is consistent with the conditions that are present. A novice user of a hierarchical decision-modeling tool may not recognize dependence relationships that exist among variables, or how to compensate for those conditions in the model. Similarly, one can not expect software to ensure that an attribute developed by a novice passes a clarity test, or is operationally sound. These are common shortcomings in models developed by beginners. They could potentially cause a model to mislead, rather than inform, a decision-maker. Also, care must be taken to ensure that decision analysis software is applied in a manner that is consistent with the underlying logic of the modeling approach implemented.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=12 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=3><B><A HREF="../surveys/das/das.html" TARGET="_top">The Survey</A></B></FONT><BR> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> <I>OR/MS Today</I> Managing Editor Tracy Jean Benn administered the survey. The questions were similar to those in the previous surveys, although there were a few additional questions. In particular, vendors were asked to identify, in a few words, the industries or market segments they have targeted and the specific applications for which the software was most widely used. The responses to these questions are intended to provide potential users of the software with some insight into the types of decision situations and decision-makers for which the software was designed.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> The survey responses presented in this article are "as is," in that no supplemental verification was attempted. The information should be used to support initial product screening only. Further research and testing is advised, particularly before undertaking a major analysis effort. For those interested in researching the principles of decision analysis underlying many of these packages, a couple of recommended references are: "Multiple Criteria Decision Analysis" by Valerie Belton and Theodor J. Stewart, "Making Hard Decisions: An Introduction to Decision Analysis" by Robert T. Clemen and "Value Focused Thinking" by Ralph Keeney.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=12 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=3><B>The Results</B></FONT><BR> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> There were 28 packages identified by 19 different companies in response to the survey. Advertised software prices ranged from $29 to more than $1,000; many respondents indicated that potential users should call for price quotes. Most respondents have been in the marketplace for a number of years and participated in multiple surveys. Many of the vendors of multiple packages have developed very robust interfaces between their products. These features would allow a user to implement a particular package for its intended purpose, and then efficiently share the required information with another specialized product.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> The vast majority of the software is written for use with the Windows operating system. Three packages (Analytica, Netica and Treeplan) have Mac OS versions. Netica has a version available that runs under both the LINUX and Solaris operating systems. Three packages (Joint Gains, Opinions-Online and Web-HIPRE) are offered as Web-based applications.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> The continually increasing computational power of processors, advances in computer visualization techniques and the Web are enabling structuring support that was previously unimaginable. The survey respondents indicate that 17 packages explicitly support structuring and brainstorming. Ten of the packages have variants designed to support group elicitation, and eight packages support decentralized group activities.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> Some of the packages indicate that there are limits to the model size and complexity that the software will support. Potential users should research these constraints carefully in the context of the types of models that they are intending to construct. In some instances, these constraints are limiting; in others, the vendor is imposing limits that are consistent with "best practices" for the type of modeling on which they are focused.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> In general, decision analysis software emphasizes either: 1. the elicitation and analysis of complex multi-criteria value functions, focusing on the second (and fourth) stage of the decision analysis process, or 2. they focus on the elicitation and analysis of uncertainties, emphasizing the third (and fourth) stage. The split across packages is roughly even, with 15 vendors offering approaches for eliciting value functions graphically and 10 vendors indicating they can do so for probabilities. One package, Expert Choice 2000 2nd Edition, indicates that they support the entire spectrum of elicitation graphically, as well as displaying analysis results graphically. In fact, the vast majority of the packages indicate that analysis results can be displayed graphically.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> There are two survey questions that shed some light on a software package's ability to support sensitivity analysis. The first question asked if probabilities and/or weights could be defined as variables, to which 10 vendors indicated that this functionality existed in their software. The second question asked if graphical sensitivity analysis is supported, and 21 of the vendors indicated that this functionality is provided.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> The new features and comments sections provide some significant insight into the advertised strengths of the software and emerging trends of new functionality. Resource allocation features continue to gain popularity as both a specialized package and as new functionality from existing vendors. The ability to exchange data with commercial applications, such as EXCEL and ACCESS, continues to emerge. One package, Decision Explore, provides XML-based import and export functionality. Approximately five packages indicate they support Monte Carlo simulation as a method for solving their models, and a significant number indicate that linear programming is now integrated into their tools. Netica continues to offer algorithms that present probability distribution for a network using data, subjective techniques or a mixture of both. Also, a number of packages advertise improved sensitivity analysis capabilities.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=12 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=3><B>Conclusions</B></FONT><BR> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> The cost of software as part of conducting a comprehensive decision analysis modeling effort is most likely the smallest component of the cost, even if one purchases the more expensive packages listed in the survey. For large efforts, the costs associated with analyst labor, subject matter expertise and stakeholder time, and generating data will far exceed software costs. That said, there is not always a correlation between the cost and the performance of the software, or suitability to the analysis problem. And, a well-constructed decision analytic model can provide senior decision-makers and other stakeholders with valuable, cost-effective insight into complex, high-value decision situations.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> When shopping for decision analysis software, evaluate the software in relation to the parts of the decision analysis process that are most relevant to you. If you seek an addition or two to your general tool kit, then a package or combination of packages that provide balanced support across all stages of the process at a modest cost may be most appropriate. If the types of models you wish to employ involve multiple stakeholders and multiple competing attributes, then tools that emphasize group support and value elicitation are probably most appropriate. Problems involving large uncertainties, diagnosis, complex interdependencies or risk analysis would benefit most from tools like Influence Diagrams, Bayesian Networks or one of the Monte Carlo modeling tools. Whichever tools are selected, they should be intuitive to the user and support easy iteration among the various stages of the decision analysis process.<BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=2><B><A HREF="../surveys/das/das.html" TARGET="_top">View the Decision Analysis Software Survey</A></B></font><BR> <img src="../cleardot.gif" width=1 height=1 vspace=6 border=0><br> <IMG SRC="../barbluehook.gif" WIDTH="468" HEIGHT="4"><br> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> <FONT face="verdana, arial, helvetica" SIZE=2><I>Dan Maxwell (<A HREF="mailto:maxwell@ebrinc.com">maxwell@ebrinc.com</A>) is a Division Director for Military Studies at Evidence Based Research in Vienna, Va. The author acknowledges Steve Shaker, Joseph Lewis and the rest of the EBR staff for their assistance in the preparation of this article.</i><BR>  <img src="../cleardot.gif" width=1 height=1 vspace=5 border=0><br>  <img src="../barblue.gif" width=468 height=1 border=0 alt=" "><br> <img src="../cleardot.gif" width=1 height=1 vspace=8 border=0><br> <font face="verdana, arial, helvetica" size="1"> <CENTER><script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script></CENTER><br> <img src="../cleardot.gif" width=1 height=1 vspace=8 border=0><br> <LI> <A HREF="tocfr.html" Target="_top"><I>Table of Contents</I></A><br> <img src="../cleardot.gif" width=1 height=1 vspace=4 border=0><br> <LI> <A HREF="../../ORMS.shtml" TARGET="_top"><I>OR/MS Today</I> Home Page</A><br>  <img src="../cleardot.gif" width=1 height=1 vspace=8 border=0><br> <img src="../barblue.gif" width=468 height=1 border=0 alt=" "><br> <b><i>OR/MS Today</i></b> copyright � 2002 by <A HREF="http://www.informs.org/" target="_new">the Institute for Operations Research and the Management Sciences</a>. All rights reserved.<br> <img src="../cleardot.gif" width=1 height=1 vspace=3 border=0><br> <img src="../bargrey.gif" width=468 height=1 border=0 alt=" "><br> <b>Lionheart Publishing, Inc.</b><br> 506 Roswell Rd., Suite 220, Marietta, GA 30060 USA<br> Phone: 770-431-0867 | Fax: 770-432-6969<br> E-mail: <a href="mailto:lpi@lionhrtpub.com">lpi@lionhrtpub.com</a><br> URL: <A HREF="http://www.lionhrtpub.com" target="_top">http://www.lionhrtpub.com</A><br> <img src="../cleardot.gif" width=1 height=1 vspace=3 border=0><br> <img src="../bargrey.gif" width=468 height=1 border=0 alt=" "><br> Web Site � Copyright 2002 by Lionheart Publishing, Inc. All rights reserved. </td></tr></table> <img src="../cleardot.gif" width=1 height=1 vspace=2 border=0><br> </BODY>