How to design multiple choice questions

Question

I've seen the previously-asked questions about the effectiveness of multiple choice problems, but I want to know about design. What are best practices in multiple choice problem design? The "common" understanding I've come to is based on perennial complaints -- "hard to write well"; "punishes students for small mistakes (like sign errors)"; "great when well-written"

Do you know of studies related to the design of questions so they address particular outcomes/skills, particularly in mathematics? Is there a manual on the topic?

Back Story: Our department has decided to face an impending assessment edict by trying a new thing: common multiple-choice section on all final exams. Therefore, limit suggestions to either general theory on writing MC questions, or resources somehow focusing on writing MC in summative (not formative) assessments.

Answer 1

Personally in my own math courses, I have found that the gold-standard is to have an advance cycle of short-answer responses, document the most common student responses, and then turn those into the multiple-choice options in the future.

Disclaimer: In general I frown upon multiple-choice testing, since the math discipline is inherently about explaining/justifying things in writing. However, my department has dictated in certain courses that the final exams be all multiple choice, for reasons similar to those expressed by the OP.

I do believe that in practice, the most common way to write math multiple-choice questions is for the writer to basically flip some of the signs in the work at some point (this is certainly a very common source of errors). However, I think that multiple-choice defenders overlook how multitudinous the misunderstandings can really be, and these are masked unless one gives short-answer questions.

For example, in the spring of 2015 on a remedial algebra test I posed the short-answer problem, "Factor completely: $50a^4 - 18b^2$" and tallied 21 distinct incorrect answers (from among 54 students). Other problems may have more distinct responses than I had time or patience to distinguish (e.g., solving an equation by factoring, or finding a linear equation given two points). This brings me back to my original thesis: If multiple-choice tests must be given, and the required time is made available, then an advance cycle of short-answer testing is an objective way of witnessing the most common misunderstandings at least once.

Answer 2

Use them for what they're best at testing, like conceptual understanding questions.

One way to do this is by having students evaluate statements they would never be expected to come up with on their own, but should be able to understand the truth or falsehood of. This can also be useful for questions that have many different way of getting to an answer that have nonequivalent difficulty.

For example, consider the following question given to a first year calculus class that has discussed concavity.

Assume that $f(x)$ has a critical point at $x=c$ and that $f''(x) > 0$ for all $x<c$ and $f''(x)<0$ for all $x>c$. What could be the behavior around the critical point?

(a) $f(x)$ has a local minimum at $x=c$.

(b) $f(x)$ has a local maximum at $x=c$.

(c) $f(x)$ is non-increasing with a stationary point at $x=c$.

(d) $f(x)$ is non-decreasing with a stationary point at $x=c$.

If you just asked the question students would be completely lost, and wouldn't know how to express an answer (unless you'd basically explained the question to them in class). However, they should be able to think critically about such a question and determine what of the answers is plausible.

I think such questions get at whether or not the students have a conceptual understanding of the material. Most students (at the college level anyway) have been trained to do computations but have never really been forced to think about the conceptual side of things. I think one reason for this is that instructors don't want a student losing all points on a question because of a conceptual error at the beginning. Multiple choice can provide a less punishing place to determine students conceptual understanding.

Answer 3

Consider how a student can solve the problems you present.

Identify the potential errors they may commit - forgetting to carry the tens or acting on addition before multiplication, for example.

For each error, work out the result from making this error. It is helpful to make a list of answers and the "cause" of that answer - remember to put the correct one in this list as well.

Now include the answers that come from the errors you want to find and fix. Depending on the question, this can be anything from using algorithms wrongly to not grasping an appropriate intuition for the concepts needed.

Given that it is multiple choice, you won't always fit every wrong option into the list presented to students. However, you can prioritise for the errors that are most commonly made, for those that best make a point you want to emphasise, or that force a student to compound errors or find and fix their own errors in order to answer.

Answer 4

This is very hard to do, but I would suggest a simple rule: Multiple choice questions should be trivial to answer if you understand the concept and very difficult otherwise. I can't think of a great example off the top of my head, but when I do, I'll post here!