13
$\begingroup$

This is perhaps the question that some thought Grading scale: how to handle multiple choice questions with different number of choices really was. I'm of the opinion that multiple choice questions should not be used for summative assessment. However, I have no actual evidence to back up that view.

Has there been actual research on this, specifically with regards to mathematics?

I don't know if the use of multiple choice questions is different in mathematics to other subjects by dint of sheer ignorance in how other subjects are examined. I do know that often multiple choice questions in mathematics are not simply "Do you happen to know the answer?" questions where the route to the answer is irrelevant but the goal is more that by seeing the choice of answer one can deduce the route that they took and so there is no need to see the details (this assumes that they took a route and did not simply guess).

A sub-issue in this is the numerous schemes to discourage guessing. I'd happily also learn about any research as to the effectiveness of these. Again, my opinion is that they do not correct for the failings of the use of multiple choice questions but, again, I have no research-based evidence for this.

Let me conclude by re-emphasising that I am asking about summative assessment and not formative assessment and about actual research. I'm not interested in answers that are purely anecdotal or opinion-based (I'll happily hear those in another venue, though).

In thinking about MattF's comment about correlation, let me try to focus it more precisely. I have no doubt that multiple choice test scores are correlated with every other type of test. That doesn't speak to their efficacy and fairness though.

Consider the following two types of question:

  1. Multiple choice, where the options include "Some of the above" and "All of the above".

  2. No partial credit, but credit only given for an answer with reasoning.

In both, it is an "all or nothing". I've posited the extra options for the "multiple choice" variant to make it so that it isn't just a closed list to choose between and so to know the answer then a student ought to have worked it out. In the first, a student can guess. In the second, they can't.

My question, then, could be phrased as: how much effect does the fact that students can guess an answer have on a multiple choice test's ability to report students' abilities when compared with a "no partial credit" test?

Improve this question
$\endgroup$
17
  • 3
    $\begingroup$ @MattF. Effectiveness in assessing what the students have actually learnt. I would not be surprised to learn that all of the situations you cite use multiple choice exams because they are cheap to grade, not because they actually measure what they want to measure. $\endgroup$
    – Andrew Stacey
    Commented May 28, 2014 at 12:33
  • 2
    $\begingroup$ @MattF. Please cite those. I'll only be able to assess if that's what I'm after if I see them. $\endgroup$
    – Andrew Stacey
    Commented May 28, 2014 at 13:04
  • 2
    $\begingroup$ As a grade school student, I tended to treat multiple-choice exams like a game, particularly because mathematics competitions in those grades tended to be multiple-choice. $\endgroup$
    – Joe Z.
    Commented May 28, 2014 at 14:51
  • 4
    $\begingroup$ A possible problem with multiple choice tests is not necessarily about whether students can guess an answer, is't about whether the approach to learning has been to get students to answer this question correctly (teaching to the test) and sacrificing teaching for understanding along the way. It's conceivable (and shown in research) that students can get correct answers (legitimately) and lack understanding. The question "effective for what" still stands. Even if they do "measure what they measure" effectively, they may be measuring the wrong thing. $\endgroup$
    – JPBurke
    Commented May 28, 2014 at 20:00
  • 3
    $\begingroup$ Probably students who are "exam-clever" do well at both multiple choice (any kind) or not multiple choice, and students who are not exam-clever do badly at both. So most assessments of exam-cleverness will give highly correlated results. But the type of exam given distorts what students take the trouble to learn, and often what is taught. $\endgroup$
    – André Nicolas
    Commented May 29, 2014 at 0:48

3 Answers 3

Reset to default
5
$\begingroup$

I've read your question a few times, plus the comments. As best I can determine, your gut feeling is that constructed response exams should be a much better indicator of student understanding than multiple choice exams in math, because a) students cannot guess and b) partial learning can be recognized by analyzing the worked problem.

Therefore, you want to see a comparison of student performance on multiple choice versus short-answer questions testing the same concepts within the same exam. If the average student's performance is similar on the two parts of the exam, this would disprove your hypothesis that short-answer is better. Is this correct?

This question is a month old, but I'll assume you as OP think this does state your question and I will continue. I've listed some research articles below. The short answer is that when the question stems are matched, there is little difference in student strategy or student performance. It is possible that very carefully written CR exams might produce more information about student learning, but it is also likely that this careful question-writing hardly ever happens. My 20-min review indicates there is surprisingly little evidence that CR exams are better than multiple choice.

Lukhele, R., Thissen, D., & Wainer, H. (1994). On the Relative Value of Multiple‐Choice, Constructed Response, and Examinee‐Selected Items on Two Achievement Tests. Journal of Educational Measurement, 31(3), 234-250. They found student answers for AP Chem and AP History did not differ significantly between MC and constructed response.

Katz, I. R., Friedman, D. E., Bennett, R. E., & Berger, A. E. (1996). Differences in Strategies Used to Solve Stem-Equivalent Constructed-Response and Multiple-Choice SAT [R]-Mathematics Items. RESEARCH REPORT-EDUCATIONAL TESTING SERVICE PRINCETON RR. Found that matched-stem problems triggered similar student problem-solving and similar results.

Shepard, L. A. (2008). Commentary on the National Mathematics Advisory Panel recommendations on assessment. Educational Researcher, 37(9), 602-609. This is a short critique of research that matches MC vs CR question stems. They argue that CR allows for different sorts of testing and this should be taken into account.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ long-term, the problem with MC, especially if they become standardized and known to instructors, is that the instructors cease teaching the full spectrum of concepts and yield to the temptation to teach to the test. Do the studies above try to quantify this effect? $\endgroup$
    – James S. Cook
    Commented Jun 25, 2014 at 14:12
  • $\begingroup$ Sorry - that's a completely different experimental test, and not part of these studies (or the OP's original question). I would argue (as someone who regularly administers exams to 800 undergraduates) that there are many ways to teach badly and we shouldn't throw out MC exams because some instructors teach to them. They can be written carefully and used effectively. $\endgroup$
    – Adrienne
    Commented Jun 25, 2014 at 14:22
  • $\begingroup$ Thanks for the answer, and much to read through. However, your second paragraph gets it almost completely wrong: "If the average student's performance is similar on the two parts of the exam" The problem that I have with MC is that there is an unnecessary additional random factor introduced. On average, this will have no effect. But on individual students then it could potentially have a large effect. $\endgroup$
    – Andrew Stacey
    Commented Jun 25, 2014 at 19:11
  • $\begingroup$ As best I can tell, these studies compare within-student results for both MC and CR, and find a single student produces similar scores in both types of exams. I'm unclear why you expect randomness to have an effect -- if an exam has 50 questions, wouldn't randomness have essentially equal positive and negative effects, and a student score be pretty close to their ability to solve math problems correctly? $\endgroup$
    – Adrienne
    Commented Jun 25, 2014 at 20:11
  • $\begingroup$ I think Loop Space is worried about the fact that, for example, in the exam I just recently graded, there were ~10 students (out of 400) who completely, randomly guessed on a 30 question MC exam. From those students alone I have a swing of 15 points out of 100: one received 8 points more than complete random guessing would give on average, another received 7 points fewer. To some, a 15% swing is acceptable. To some, it is not. $\endgroup$
    – Willie Wong
    Commented Jun 30, 2014 at 13:49
4
$\begingroup$

I don't know of research on the use of multiple choice questions for summative assessment specifically in mathematics. However, what's important is the approach to construction of multiple choice questions, if they are to overcome challenges like the ones you list. Following course design basics from sources like Wiggins & McTighe, what we know about assessments are that they are most effective when they are aligned with your learning goals.

With this in mind, I did come across one article that looks promising, but at the moment I only have access to the abstract, so I can't guarantee its value. (Edit: The article is available in full here.) From the abstract it appears the authors are taking a somewhat nuanced look at assessment in mathematics, and considering connections between the types of things you might want to assess, and how different assessment tools (in particular MCQs) can be of value.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Welcome to MESE! $\endgroup$
    – Mark Fantini
    Commented Jun 24, 2014 at 19:30
0
$\begingroup$

I prefer giving multiple choice exams. I can give frequent feedback to students, since the grading is easier. Below is a study the indicates no difference between multiple choice and short answer.

http://www.ncbi.nlm.nih.gov/pubmed/24274234

Chris

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the link. The abstract of the article makes it pretty clear that this is about formative assessment which I specifically rule out. Your answer also seems more formative in spirit since you talk about giving feedback to the students afterwards. $\endgroup$
    – Andrew Stacey
    Commented Aug 28, 2014 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.