Teaching Trends
Defining Success
When it comes to measuring the success of management science courses in MBA programs, look at what our students achieve on the job
By Steve Powell
Anyone who teaches must ask periodically what it means for a student to succeed in their course. Success, certainly, can be measured in many ways, and the success of an individual student depends both on their background and effort, and on the design of the course. As teachers we control the design and delivery of our courses. We may also be able to influence the effort level of our students, but rarely can we control their backgrounds. This lack of complete control of the process is part of what makes teaching a challenge.
My purpose in this article is to raise some questions about how we define success in the MBA management science course and to share my own perspectives. Although I hope my thoughts will be stimulating for those who teach in other types of programs, I cannot claim any special expertise outside this context. In the MBA program, we typically teach the more applied side of our discipline, stressing management science over operations research, if you will. Few of our students will specialize in our subject or use our subject more than intermittently on the job. The special challenge of teaching management science to MBAs is to make our subject interesting and useful to an audience that has no natural affinity for it.
How do you measure the success of your students? Consider the following alternatives.
1. The quality of student evaluations. At my school, student evaluations are essentially the only information the administration uses to evaluate teaching performance. This reinforces the already high salience of this direct, and often personal, feedback in our minds. But how meaningful are these evaluations, which are completed on the last day of class in each course and therefore cannot measure the value of a course within the curriculum or on the job?
2. Student performance on exams. If exam performance is vital feedback to students, it is also vital feedback for teachers. But do we focus on the average grade, the 90th percentile or the 10th? Some students do well in our courses because of their own backgrounds or aptitudes; we should not fool ourselves by taking too much credit in their cases. Some students will struggle no matter what we do. Perhaps our biggest successes are the students we help not to fail. Perhaps the fundamental problem with using exam scores is that no one knows how well they predict future success.
3. The level of passive understanding of management science achieved. For years, the goal of the management science course was not to teach students to actually build models but rather to appreciate models; to be "intelligent consumers" of management science. (For a critique of this approach, see my article, "From Intelligent Consumer to Active Modeler: Two MBA Success Stories," Interfaces, Vol. 27. No. 3, 1997.) This kind of passive understanding might be a sufficient justification for having a management science course in the MBA curriculum, but only if many students actually encountered management science in their jobs. There seems to be little evidence for this happening.
4. Enrollments in second-year quantitative electives. One way we measure our success at Tuck is by the level of interest in second-year electives in management science and modeling. We currently offer four or five courses in this area, and about half of the class takes at least one of these courses.
5. Success of students using management science on the job. How many of our students owe their jobs to their knowledge of management science? How many apply our methods directly on the job? How many attribute some of their career success to management science? Very little effort has been expended over the years in measuring the impact of our teaching on the careers of our students. My school periodically sends a survey to alumni five years out asking which courses had the biggest impact on them, but management science is not always listed. Certainly, most of our students would not cite management science as having had a major impact on their careers. But are we having any impact?
I believe that the only legitimate measure of success for my courses, ultimately, is student success on the job. All the other measures are helpful; I would certainly rather have high student evaluations than low ones, for example. But all other measures are means to an end. An MBA program is by design a practical, job-centered training program. Time spent on management science in the curriculum is time that could be spent on accounting, marketing, finance or strategy, all fields whose direct relevance to the student's job performance is accepted. Therefore, to justify the place of management science in the curriculum, we must be prepared to demonstrate that students actually use our subject on the job. If we accept this premise, then we must find out how students currently use our subject on the job, as well as how they can do so more effectively, and design our teaching to enhance these outcomes.
The potential of end-user modeling
Everyone is aware that spreadsheets have taken over the task of business analysis in most firms in the last two decades. The MIS department is still there, for sure, but the explosion in end-user computing has lead to the development of millions of small-scale (and some large-scale) spreadsheet programs, mostly built by business people, without any involvement of the MIS department. While little attention has been paid to these small-scale computer programs, Michael Schrage in "Serious Play" (Harvard Business School Press, 2000) cites evidence that spreadsheets are used across a wide range of application areas and have a major impact within the decision-making processes of most firms. However, the evidence is mounting of the threats posed by poorly designed and bug-riddled spreadsheets in use every day throughout industry. (The authority on spreadsheet errors is Ray Panko, http://panko.cba.hawaii.edu/ssr/Mypapers/whatknow.htm. At least two organizations are studying spreadsheet risks: the European Spreadsheet Risks Interest Group, www.eusprig.org/, and the Spreadsheet Research Interest Group, www.calgary.ca/mg/grossman/grossman_srig.html.)
What is the current state of spreadsheet use by end-user modelers? The evidence suggests that, despite the widespread use of spreadsheets, the quality with which they are engineered is only poor to adequate. The evidence suggests there are four major problem areas:
- End-user spreadsheets frequently have major bugs.
- End-users are overconfident about the quality of their own spreadsheets.
- The process end-users employ to create their spreadsheets is inefficient at best and chaotic at worst.
- End-users fail to employ the most productive methods for generating insight from their spreadsheets.
The management science community can offer the end-user community powerful instruction in at least three areas:
- problem formulation: how to distill a complex business problem into a model;
- spreadsheet engineering: how to use spreadsheets efficiently and effectively to build models; and
- generating insights: how to analyze a business situation using a spreadsheet model to generate insights, not just numbers.
Examples from Tuck
The difficulties of tracking our students years after they graduate and determining how they may be using our subject are obvious. However, in residential MBA programs we have a built-in, if somewhat limited, opportunity to do the next best thing. Our students take a summer job between the first and second years of the program. Many of my students work in jobs in which they are business analysts, so they are highly likely to use spreadsheets. When they return to campus after the summer their experiences using spreadsheets are strong in their minds. Each year I poll our returning students and ask them if they used any of what we taught them in our Decision Science course. This course covers Excel modeling, optimization and simulation, all in 18 class sessions. I usually get 15 to 20 responses out of 220 students. (If this seems like a low response rate, you might ask how many would say they made direct use of their class work in marketing or strategy.) I'll share some of the most recent responses to this informal poll to indicate how some of my students are using management science on the job.
Student 1: This student built and used valuation models in the Mergers and Acquisitions Group of an investment bank. Among the specific Excel tools used: Data Table, Goal Seek and Solver.
Student 2: This student ranked Decision Science as the most useful course in the first year, along with Managerial Accounting. He built a model to allocate sales and marketing resources among 15 business segments for a major mutual fund company. As to the level of success achieved: "They acted as if I were Moses bringing down the Ten Commandments from Mt. Sinai."
Student 3: Working for a consulting company, this student built models that determined fixed and variable costs for a variety of alternative power generation technologies. The results were used in making decisions on power generation capital for commercial properties. The student commented: "I was just so proud of myself."
Student 4: This student's consulting client had acquired another large firm and wanted to know what it would take to integrate the new firm into the existing one. It also wanted a quantitative framework to help it plan for future acquisitions. This was a very different task from the ones the student had studied in Decision Science, since it was not motivated by a single, clearly stated decision.
Student 5: This student used modeling to conduct a competitive analysis of a potential entrant into an important consumer goods product category. One key was forecasting market share for the new entrant. The major consumer goods firm the student worked for had proprietary forecasting software, but the time available was too short to use it. Instead, this student conducted a simulation study using the Excel add-in Crystal Ball. In the limited time available, he was able to build a model, assess the necessary probability inputs, run the model, analyze the results and present his recommendations in a form understandable to management.
Student 6: This student used Crystal Ball for an analysis for a major manufacturer of farm equipment. The motivating question was whether to put individual component parts up for electronic auction. The student commented: "The people at (my company) were pretty impressed with the analysis — Decision Science was a great course!"
Student 7: This student's consulting client was an automotive supplier. Each division in this firm has to bid for the right to sell a certain product to the automotive manufacturers. Each division has a certain success rate with each of the manufacturers; also, the price discounts granted to the manufacturers differ by division. All of these factors were taken into account in a model that determined the net present value for each division as a function of the products it sold. The student commented: "I felt that my skills in modeling helped me a lot in this first assignment. I did indeed surprise my manager, since he didn't expect I would know this."
Student 8: The high-end vacation industry caters to wealthy individuals who want access to vacation services in the most elite locations. This student worked for a firm that sells vacation rentals to a small client base. They own some 60 vacation properties in the Western United States, the Caribbean and other prime locations. They currently have around 400 clients who have paid a membership fee of as much $1 million for the right to take weeklong vacations in any of the firms' properties. The firm offers the guarantee that if any of its properties are booked for the period an individual client requests, they will provide equivalent space. To fulfill this promise when their own properties are fully booked, they rent expensive space on short notice, often during high-demand periods. The problem the student analyzed using simulation was to measure the costs of overbooking, and develop a recommendation for how fast the firm should expand its holdings as a function of the rate at which its client base expanded.
These examples are a sample from the reports I receive each year from students returning from their summer jobs. They suggest to me several conclusions. One general observation is that these students are having success, often-impressive success, using tools that we teach them in a very short course. End-users can make powerful use of management science, without years of training. But to accept this conclusion, we have to be willing to recognize management science not just in an application of linear programming but also in a well-built and effectively used spreadsheet analysis that does not employ any of the specialized management science tools.
Another conclusion I draw is that general problem-solving skills are as important as specific management science tools for this group. By general skills I mean the ability to recognize a problem that can be modeled and the ability to abstract enough of the essence of that problem into a model that will ultimately provide insight.
Finally, my experience suggests that the specialized tools of management science can be deployed effectively by these end-users. Many of my students who build spreadsheet models on the job do not use optimization or simulation. However, those that do often achieve extremely impressive results, sometimes even of biblical proportions!
Incidentally, success on the job seems to translate into success on the other measures of success we discussed before. We make a point of citing some of these stories to incoming students, and we tell them to seek out returning second-year students to ask them about their experiences. This helps motivate the first-years to take the required management science course seriously. Success on the summer job also translates directly into interest in second-year electives, and sometimes into students focusing their job search on a job that will call on these skills.
I do not claim that our success at Tuck is unique. If it were, it could be due to the particulars of our students or the jobs they take. But others have reported similar successes, at quite dissimilar institutions. Sonntag and Grossman, for example, cite the success of one of their students in using spreadsheet modeling and management science on the job ("End User Modeling Improves R&D Management at AgrEvo Canada, Inc.," Interfaces, Vol. 29, No. 5, pp. 132-142, 1999). Liberatore and Nydick, whose teaching approach is much less focused on spreadsheet modeling than my own, also report great success with executive MBA students using modeling for practical projects in their own companies ("Breaking the Mold: A New Approach for Teaching the First MBA Management Science Course," Interfaces, Vol. 29, No. 4, pp. 99-116, 1999). The point I wish to make is that success on the job is a practical and achievable goal, at least for some proportion of our students. Certainly if we define success in these terms we are more likely to achieve it. And on the job success must count as the weightiest form of impact on the lives of MBA students.
Implications for MBA teaching
What does all this imply for our teaching? Perhaps a reevaluation of the role of the management science course in the MBA curriculum is in order. We used to view management science as comprised of its powerful tools: optimization, simulation, queuing, forecasting and so on. When we taught management science, our goal was to impart an understanding (and, if possible, some facility in using) these tools. But too few of our MBA students had the ability and opportunity to use these tools on the job to really justify the time spent on the topic in the curriculum. This eventually led to a concerted movement to remove the course from the curriculum. (See Tom Grossman's analysis of this process in "Causes of the decline of the business school management science course," INFORMS Transactions on Education, Vol. 1, No. 2: http://ite.informs.org/Vol1No2/Grossman/Grossman.html, December 2000.)
The phenomenon of end-user (spreadsheet) computing has changed all that. All of our students want, and need to know, how to use Excel. They may not recognize building a spreadsheet as modeling, but we should recognize it for what it is and exploit the opportunity it represents.
Given the sorry state of spreadsheet engineering in most companies today, our first task should be to teach our students how to build simple spreadsheet models efficiently and effectively. This includes topics not found in the current textbooks, such as how to effectively design a spreadsheet, how to build it without introducing bugs, and how to debug it to ensure that any bugs are caught.
Next, we should learn how to teach modeling in the broader context of problem solving. This includes not only how to simplify complex problem descriptions effectively, but also how to extract useful managerial insights from model results and to use those insights to enlighten managerial discussions.
Again, consult the textbooks and you will find that most model presentations stop once an answer is found. Practicing management scientists know that the "answer" is really just the beginning of the analysis process. How are students to learn how to extract and communicate managerial insights from models if we don't teach them?
Finally, we should continue to teach the specialized analytical tools of management science, but we may have to teach fewer tools, and teach them in fewer hours than we once did, in order to accommodate the spreadsheet engineering and problem-solving skills students also need. This is a trade-off I am willing to make, not because I value management science less but because I value student success on the job more.
Stephen G. Powell is a professor at the Amos Tuck School of Business at Dartmouth College. He has an M.S. and Ph.D. from Stanford in Engineering-Economic Systems, and a B.A. from Oberlin College in Economics. At Tuck he has developed a variety of courses in management science, including the core Decision Science course (with Ken Baker and Tom Grossman) and electives in the Art of Modeling, Business Process Redesign and Applications of Simulation. In 2001, Powell was awarded the INFORMS Prize for the Teaching of OR/MS Practice. His research has ranged from energy economics through advertising and promotion planning, studies of manufacturing production lines, and service sector business processes. His current interests are primarily in improving end-user modeling: training individuals and groups to use modeling more effectively. The author acknowledges Tom Grossman's contributions to this essay.
OR/MS Today copyright © 2002 by the Institute for Operations Research and the Management Sciences. All rights reserved.
Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com
Web Site © Copyright 2002 by Lionheart Publishing, Inc. All rights reserved.