Missing the mark with multiple guess
What do students know when they do well on exams, particularly those with multiple-choice questions?
It is now exam time for colleges and universities. The increased public focus on our educational system has increased the focus on exams, and a particularly pernicious aspect of exams: Multiple-choice questions.
Worshipping a false prophet
I spent more than a decade as a college instructor, and I got to see first-hand the damage that examinations do to education. This damage takes many forms: the administrative workload associated with preparing, safely storing and marking exams, the psychological stress imposed on students, teachers and administrators and of course the blunted prospects for students who did not do well on the exams.
But don't we need exams to make sure that students know something when they graduate? Yes, students need to "know something."
But what do students know when they do well on exams, particularly those with multiple-choice questions? The simple and most accurate answer is that the students who do well know how to successfully write an exam, and of course they may have some knowledge of the subject — just not as much knowledge as you might think.
Putting the guesswork into education
In my undergraduate days, we used to love what we called multiple-guess exams, because that's what they were. If you were a skilled guesser or knew how to skim a textbook or lecture notes, it was relatively easy to cram for an exam.
Remembering anything useful a week later was highly unlikely, but a passing grade on a transcript was one more step on the road to success. We'd save for later the embarrassment of actually holding down a job and having a manager or co-worker look at the fruits of our labour and ask, "Where did you go to school?"
The thing is, there are strategies for maximizing the odds of finding the right answer. One thing you can be certain of in a multiple-choice exam is that the correct answer is in there somewhere; you just have to find it. The so-called 'advantage' of multiple-choice exams is that they are efficient: it does not take long to write one and with modern technology it takes even less time to mark one — or 100.
The disadvantage is that it is very difficult to write "good" exam questions, and even if you do, it is even more difficult to write questions that evaluate high-level learning. And then there's the drawback (some would say the benefit) that you really don't need to know much to pass a multiple-choice exam.
Exams as lottery tickets
If a six-year-old wrote a university-level exam that had 100 multiple-choice questions, each with four possible answers, the child could expect to score 25 per cent on average by filling in one box for each question. (Of course, it is not likely that a six-year-old would score 50 per cent or more, but as we shall see, that does not add much to the integrity of the process).
Given that random selection will yield the correct answer 25 per cent of the time, how much do you actually have to know to score 50 per cent?
I will save you the algebra by telling you that if you know the answer to 34 questions and guess on the other 66, odds are that you will score a 50 or better. Knowing barely a third of the material, you get a marginal credit — assuming that the questions accurately assess subject knowledge.
OK, you might say, this regime would still result in a failing grade for 50 percent of the students who have mastered only a third of the course material, but why pass any of them?
But it's worse than that. Most multiple-choice exams follow a pattern of having one ridiculous answer, one highly unlikely answer, and two that require some expertise, either in language or subject area knowledge to sort out which is correct.
A person who has been moderately attentive during the school year and done a reasonable amount of cramming will be able to find the final two pretty quickly, making the guessing part a 50-50 proposition.
Just what are we measuring?
Yes, there is a technique to writing exams, and students who learn it can outperform more skilled or knowledgeable students. Educators regard exam-writing skills as so important that in my time at Red River College, first-year students received a booklet called Making your Mark that gave tips on how to study and how to write exams with a lot of detail on multiple-choice and true or false questions.
Essentially this well-meaning practice helped many students to falsely inflate their grades. Grade inflation is only one side of this dark coin, however; some students are not skilled at exam writing or are psyched out by the process and do not choose options wisely. These students' grades devalue the amount of learning they have done.
I'd be upset about this inflation if I believed that exams were a reliable or even useful tool for evaluating student learning. But if you believe it is a useful exercise to spend as much as a quarter of the time allotted in a semester to writing and marking exams, ask yourself how comfortable you would be if before going under for heart surgery, your surgeon were to tell you, "Don't worry. I scored 96 per cent on a multiple choice exam on bypass surgery."
Kevin Longfield is a retired college instructor who relied on guesswork for at least part of his university degrees.