Innovative Education
Online Exams Put to the Test
Despite some technical obstacles, concept computes for introductory management science course
By Erhan Erkut and Armann Ingolfsson
On Dec. 17, 1996, we administered a synchronous online final exam for the first time to a class of 350 undergraduate business students. We did this because we believed it was the appropriate way to test students in our course, but we must admit that we had some trepidation going in because of the multitude of things that could go wrong. It would be too much to say that the exam went off without a hitch, but by and large it was successful. Since then, we (and four colleagues) have used online exams and assignments in six different subjects, and more than 4,000 undergraduate and MBA students have gone through the experience. We believe online exams accomplished what we wanted them to, and have gradually been getting better at preparing, administering and marking such exams.
What is an Online Exam?
Students in our introductory management science course complete several assignments, three 50-minute quizzes and a three-hour final exam. Most assignments, all quizzes and the final exam are online, meaning that students use a computer to complete and submit their work through the Internet. All our assignments and exams use the same procedure for access and submission, as illustrated in Figure 1.
Figure 1: Submission process for an online exam
At the beginning of an exam, students receive two handouts — the exam questions and an answer sheet. They then access an Exams page on the course Web site and type in their ID number (and possibly other identifying information) to access their exam on the computer. The exam is a Web page containing questions (identical to the student's handout), a link to download a Microsoft Excel workbook containing problem data and an electronic version of the answer sheet. The workbook may also contain partially or fully completed models and templates. Students do most of their work in the workbook. We put no restrictions on how students solve problems; they are free to apply any technique. (Students never cease to amaze us with the different ways they approach a problem. They are sometimes able to generate surprisingly good solutions using tools that, to us, seem inappropriate.)
Students enter their answers on the electronic answer sheet as they generate them. They then save their workbook and submit it to our server through the exam Web page. In response, the server extracts the answers and displays them to allow students to check that the submission was successful (students are also encouraged to fill in the paper answer sheet, as backup). Multiple submissions are permitted and encouraged to prevent lost work due to computer crashes. Only the last submission is marked.
Why Online Exams?
Our introductory management science course (taught to 300 students each term) emphasizes spreadsheet-based modeling and problem solving. In a typical lecture, a spreadsheet model is developed from scratch. Students are encouraged to participate in building the model, exploring the model through what-if analysis, and generating recommendations using (for example) optimization or simulation. During computer labs, students recreate some of the models from lecture or work on related problems. Hence, paper-and-pencil exams are incompatible with the goals of our course.
It is unreasonable to ask students to analyze large data sets (for example, calculating summary statistics, plotting histograms or fitting curves) in a paper-and-pencil exam, whereas these tasks can be accomplished in a few minutes with a spreadsheet. Similarly, asking students to perform what-if analysis on models of moderate size using paper and pencil is unrealistic, while spreadsheets make such analysis instantaneous. With optimization or simulation, the limitations of paper and pencil become even more apparent — for example, solving a 20-variable, 10-constraint linear program manually in a three-hour exam is impossible, while this takes a few seconds on a spreadsheet. The old adage "what gets measured gets done" is one reason we feel the need for online exams: paper-and-pencil exams provide incentives to students to focus on skills that are different from the ones our course emphasizes — namely, modeling and solving problems on the computer.
We do not mean to suggest that paper-and-pencil exams should never be used in management science courses. A paper exam may be ideal for a course that focuses on algorithms and theory, or for a course on strategy. However, when the primary focus is to teach computer-based modeling and problem solving, paper exams seem unsuitable. While every course is different, we suspect that some of our arguments for online exams may apply equally well for many other courses, for example in mathematics, statistics, engineering and computer science.
Online Exam Questions
We believe that a well-designed online exam problem consists of several parts. If students were asked only for the "final answer" to a problem (possibly a single number), most students would receive a zero because of a (possibly trivial) error made at some point. While this allows us to identify students whose work is perfect, it does not allow us to differentiate between "almost perfect" and "no knowledge of the subject." Segmenting a problem and asking a sequence of questions of varying difficulty that test different areas of competence, helps differentiate between abilities and allows for diagnosis of weaknesses.
For example, we give students a partially completed model for monthly production planning and ask questions that can be answered simply by studying the data. This tests understanding of the components of the model (transition from text to spreadsheet). Then we ask questions that require adding or changing numbers or formulas in the model (for example, production amounts) and reporting how profit (or cost) changes. This tests understanding of how the model components interact. Then we ask for the solution that maximizes profit, to test the ability to apply optimization. As follow-up, we ask questions about the solution (for example, about utilization of resources or patterns in the solution), to test whether students can generate managerial insights from the mathematical solution of an abstract model. Finally, to test the level of command of the model, we ask students to make a nontrivial change to the model (for example, include the possibility of increasing demand through advertising) and find the new optimal solution. Very few students get full credit on such a problem; however, most get partial credit based on their command of the subject.
We have discovered that with online exams, one can easily do things that are difficult to do with traditional exams, and we are gradually exploring these possibilities. We routinely prepare a few slightly different versions of each exam, by perturbing problem data and questions without affecting the difficulty level. A student's ID number determines which version of the exam they see and the incentive to peek at a neighbor's screen is removed. A natural extension is to give different problem data to every student, and we have done this in an elective on Monte Carlo simulation. For example, Excel macros can generate different data sets for input analysis for every student, in a way that we can replicate for marking purposes. We have also experimented with problems that test students' ability to debug spreadsheet models — a crucial skill for spreadsheet modelers.
Online Exam Logistics
Giving an online final exam in several computer labs simultaneously is a significant logistical and technical undertaking, and an excellent opportunity for us operations researchers to practice what we preach! We use project management methods to ensure preparations are on schedule, capacity planning to decide how many labs we need, integer programming to assign students to labs, and even rudimentary vehicle routing techniques to plan routes for "roving professors" who travel between labs to answer questions.
A final exam in our introductory course usually requires a dozen computer labs (if the class fits in one lab, many of the logistical problems disappear). The number of students assigned to each lab is usually 80 percent to 90 percent of lab capacity, allowing us to move students who experience hardware or software problems to another computer.
We assign proctors (usually former students) to every lab. Proctors check student IDs, start and end the exam on time, move students having equipment breakdowns, and check screens to make sure no prohibited software is used. A professor is assigned to a cluster of two to four labs. Professors answer content questions and deal with emergencies, such as a partial or complete server failure. Proctors communicate by e-mail with a student assistant that is in frequent contact with the lead instructor. This assistant e-mails announcements that need to be made during the exam simultaneously to all proctors.
Marking Online Exams
After the exam, we extract the responses from the electronic answer sheets and collect all of them in one Excel workbook for marking. The marking can best be described as semi-automatic. For most answers, we prepare a marking key that includes several possible answers and the marks (full or partial) for each. We have never been able to anticipate all of the distinct answers that students will give, so we complete the marking key after the exam. It usually takes at least 10 hours to prepare the marking key, but once it is ready, the entire exam can be marked in a few seconds. In some cases, we are able to post the marks less than 24 hours after the exam; a turnaround time that students appreciate greatly. Once the marking is completed, we populate students' workbooks with their marks (and possibly some comments) and e-mail them to the student. This enhances learning since many students check to see what they did wrong after the exam.
When we see an answer that we did not anticipate (Excel's data filter tool comes in handy to scan the range of answers), we try to reconstruct the path a student followed to end up with that answer, especially if many students answered the same. We may then give partial credit for that answer. This reverse engineering is often straightforward, but sometimes it becomes a real challenge. This is where we spend most of our time and energy in marking.
Some problem segments are designed with the intent of awarding partial credit. For example, one question may ask for a parameter and the next, for a figure that depends on the parameter. A student may compute the parameter incorrectly, but use the correct procedure in the following question. To give the student credit for the second part, we recreate their use of the formula — a very simple task in a spreadsheet, but often impractical when grading manually. All features of Excel, including macros, can be used to make this "conditional marking" process as sophisticated as the instructor desires. Conditional marking can be used within a question as well. If a student is asked to report the values of the decision variables and the objective function for an optimization problem, we can award marks separately for correct calculation of the objective function, feasibility and degree of suboptimality. Thus, a student may receive close to full credit for a feasible but slightly suboptimal solution.
For most questions, we determine three or four different answers that we award full or partial marks. We use nested IF formulas and simple macros to convert answers to marks. During this process, one has to be careful about rounding (35.45 vs. 35.5), units (0.35 vs. 35 percent) and formats (35,000 vs. 35 000, or 35% vs. 35 percent). We may spend an hour to generate the different numerical answers that deserve credit for one question, but once that is done the marking of that question for 300 students is accomplished in a second — a very satisfying experience.
Just like with more traditional exams, our marking procedure is not perfect — occasionally we make both type I and type II errors. However, we believe our semi-automated marking procedure has the major advantage that it is free of biases and it uses an identical marking logic for every student.
Online Exam Challenges
The first challenge is to prepare good problems. A good online problem is different from a good paper-and-pencil problem, and it takes time to become adept at preparing one. The second challenge is logistics. This requires considerable planning, starting several weeks in advance of the exam, and considerable labor (more than a dozen assistants and a team of three or four professors). The third challenge is to manage hardware and software problems. We prepare the students before the exam by telling them that technical problems can and do occur during any three-hour computer session. We tell them the ways we deal with problems and ask them to stay calm (easy for us to say, of course). Students are encouraged to visit their labs ahead of time to familiarize themselves with the location, surroundings and computers. A fourth challenge is perceptions of inequity by students who are asked to write the exam in a lab they have not used before. This was a major problem on one of our exams when some of the students had to write the exam on slow machines with small monitors. Although we have never been able to detect any systematic difference in performance between labs, some students thought they were treated unfairly. This challenge seems to diminish as the age of computers on our campus has become more uniform.
Security is an ever-present challenge. We use ASP (active server page) files to limit exam access to students on our class list using computers in the appropriate labs. We assign students to labs to break up coalitions, and we have at least two versions of the exam. As further measures, we forbid the use of communication programs and watch the screens carefully. Finally, we check the IP address for each submission to ensure different students do not submit from the same computer.
We have thought of many ways to cheat on an online exam and have put many prevention measures in place. However, a competent hacker can probably find a way around our defenses. Although high-tech ways of cheating on an online exam exist, low-tech methods, such as passing pieces of paper or peeking at nearby monitors, are more common. We believe that our low-tech method of vigilant proctoring will remain the most effective way to prevent cheating, as with traditional exams.
Future Improvements
We have improved the process of preparing, administering and marking online exams considerably during the last five years. Still, it is always possible to do better. An area of particular interest to us is attempting to mark students' problem-solving approaches, as opposed to just the numerical answers they generate. For example, when asked for an optimal solution, one might wish to give partial credit for a problem formulation that is nearly correct except for reversing one constraint inequality, even if this resulted in a nonsensical solution. As most instructors who use spreadsheets know, untangling another's spreadsheet model can be frustrating. Yet, it is possible to write macros that elicit some of the structure of a spreadsheet model, including solver constraints and settings, to facilitate this type of marking. We have done this on a limited basis with online homework assignments and would like to explore this approach further.
To summarize, we believe the online format is considerably superior to paper-and-pencil exams for our courses, and we suspect that this is the case for many other instructors and subjects. We encourage other instructors to experiment with this format, and we are willing to share our experiences. The Web page http://www.bus.ualberta.ca/aingolfsson/online_exams/ provides an example of how our online exam submission process works.
Erhan Erkut (erhan.erkut@ualberta.ca) and Armann Ingolfsson (armann.Ingolfsson@ualberta.ca) are professors at the University of Alberta School of Business.
OR/MS Today copyright © 2001 by the Institute for Operations Research and the Management Sciences. All rights reserved.
Lionheart Publishing, Inc.
506 Roswell Street, Suite 220, Marietta, GA 30060, USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com
Web Site © Copyright 2001 by Lionheart Publishing, Inc. All rights reserved.