11
$\begingroup$

Suppose you have been told to give a multiple choice exam, and to nullify the effects of random guessing by penalizing incorrect answers. Suppose there are $N$ available choices, of which exactly one is correct.

One common approach is to give $P$ points for the correct answer, $p = -P\, /\, (N-1)$ points for an incorrect answer, and 0 points for not answering or choosing more than one answer.*

Now suppose we have some questions with 4 available choices, and some true/false questions with only 2 available choices. How should we assign the corresponding correct answer scores $P_2$ and $P_4$ (and the respective penalties for incorrect answers $p_2$ and $p_4$) to make the questions "worth the same"?**

For convenience we can normalise $P_4 = 3$ the corresponding penalty $p_4 = -1$:

  1. Set $P_2=1, p_2 = -1$, so the penalties are the same for incorrect answers.
  2. Set $P_2=2, p_2 = -2$, so the differences are the same between correct and incorrect answers.
  3. Set $P_2=3, p_2 = -3$, so the differences are the same between answering correctly and not answering.
  4. Set $P_2=\sqrt{3}, p_2 = -\sqrt{3}$ so the variances are the same for random guesses.

Is there a compelling reason to prefer any of the above options, or something else altogether?


* To prevent hurting the students' feelings by giving them a negative score on a test, we can equivalently give 0 points for incorrect answer and P/N points for null/multiple answers. Whether we normalize the score around 0 or P/N is inconsequential for the discussion here.

** Without penalties (for example, in the case of free-response questions) the answer is simple: just make the maximum number of points awarded for each question be the same. The question here can be re-interpreted as: does it make more sense to count the weight of each question relative to the theoretical minimum score or the theoretical average score, or something else altogether?

$\endgroup$
  • 6
    $\begingroup$ What part of "Never use multiple choice for summative assessment" don't you understand?! $\endgroup$ – Loop Space May 27 '14 at 13:05
  • 3
    $\begingroup$ What?? You expect me to read the whole question? What sort of crazy exams do you write where you expect your students to do that? More seriously: yes, that's why I left it as a comment not an answer. Also, I would worry that you're reaching the point where you care more about this than either the students or the administration. $\endgroup$ – Loop Space May 27 '14 at 13:29
  • 1
    $\begingroup$ For the actual exam, the grading scale is already published (+3/-1 for multiple choice and +1/-1 for true-false); my gut feeling is that is almost certainly weighing the true-false question too little and will compensate for this decision by making the true-false section "easier". $\endgroup$ – Willie Wong May 27 '14 at 13:32
  • 1
    $\begingroup$ The German Kangaroo competition explains its scoring system here: mathe-kaenguru.de/wettbewerb/auswertung/index.html It values the questions by perceived difficulty with $P=3$, $P=4$ and $P=5$, giving students an offset of 24 or 30 Points, so that negative scores cannot be reached. $\endgroup$ – Toscho May 27 '14 at 13:45
  • 1
    $\begingroup$ @WillieWong The justification for the value of a question lies in the didactics, not in the maths. $\endgroup$ – Toscho May 27 '14 at 14:58
4
$\begingroup$

Let's consider an extreme case: An exam with two questions. Questions 1 has 2 options (only one is correct). Question 2 has 1000 options (only one is correct). Let $c_i>0$ be the points awarded for a correct answer to problem $i$ and $w_i>0$ the points subtracted for a wrong answer. Without loss of generality, we can scale $c_1=1$.

Then there are some desirable aspects we wish to achieve:

1) Random guessing for each problem has an expected value of 0 points. For this to occur we have to satisfy: \begin{eqnarray*} 0.5c_1-0.5w_1 &=&0\\ 0.001c_2-0.999w_2 &=&0 \end{eqnarray*} Then $w_1=c_1=1$.

2) Each question carries the same relative weight in damaging/helping the overall score. For this to occur we have to satisfy: \begin{eqnarray*} c_1+w_2 = w_1+c_2 \end{eqnarray*} Since then missing exactly one question results in the same score. But, in light of $c_1=w_1$ above, this reduces to \begin{eqnarray*} w_2 =c_2 \end{eqnarray*} This, of course, contradicts the null expected value condition. So we can't do (1) and (2) at the same time.

Two possible solutions/alterations:

1') Random guessing throughout the exam has an expected value of 0 points for the whole exam. For (1') and (2) to occur we have to satisfy: \begin{eqnarray*} 0.5c_1-0.5w_1+0.001c_2-0.999w_2&=&0\\ c_1+w_2 &=& w_1+c_2 \end{eqnarray*} With $c_1=1$, this reduces to \begin{eqnarray*} -0.5w_1+0.001c_2-0.999w_2&=&-0.5\\ w_1+c_2- w_2 &=&1 \end{eqnarray*} This has solution set $w_1=1-\frac{998}{1499}c_2$ and $w_2=\frac{501}{1499}c_2$. To keep $w_1>0$, then $c_2$ should be chosen less than $\frac{1499}{998}$. Maybe $c_2=1$ makes sense, in which case $w_1=w_2=\frac{501}{1499}$.

2') The other possible solution is not to weigh each question equally (e.g. weigh easy ones less or something). Suppose we want Question 2 to weigh twice as much as Question 1 (I mean, c'mon, 1000 choices makes it hard to find just the right one.) Then (1) and (2') give: \begin{eqnarray*} 0.5c_1-0.5w_1 &=&0\\ 0.001c_2-0.999w_2 &=&0\\ 2(c_1+w_2) &=& w_1+c_2 \end{eqnarray*} Setting $c_1=1$ and solving gives: $w_1=1$, $c_2=\frac{999}{997}$, $w_2 = \frac{1}{997}$.

That's about the best analysis I can think of. Of course, the above numbers can be modified for your case of 4 choices; I just think better in extremes. I'm guessing if you start to have more than two questions, the systems of equations get bigger but stay linear and solvable if you choose one of the suggested alterations.

$\endgroup$
  • 2
    $\begingroup$ Question 1 is a true/false, and question 2 is an AIME problem. $\endgroup$ – Joe Z. May 29 '14 at 18:56
2
$\begingroup$

I believe option number 2 is the correct one, given the circumstances that you describe.


At its base, you want each question on this multiple-choice test, regardless of how many possible answers it has, to be worth $1$ point if the student has demonstrated he knows the answer to the question, and $0$ points otherwise. This is the only fundamental way to make each question "worth" the same. Anything else is just clever rescaling.

The most common way to formulate this is to give $1$ point for a correct answer, and $0$ points for an incorrect answer. However, complicating this is the fact that a student might get the correct answer purely through guessing. So, you decide to give them an "incentive" (really just removing the edge from the former choice) to not answer if they have no idea, by giving them $1/n$ points, the expected value from guessing randomly, for leaving the question blank. In your scenario, this is $1/4$ point for a question with four choices, and $1/2$ point for a true-false question.

If you scale this up by a factor of $4$ to make the point values all integers, you get:

$$\begin{matrix} P_4 = 4 & O_4 = 1 & p_4 = 0\\ P_2 = 4 & O_2 = 2 & p_2 = 0 \end{matrix}$$

And if you subtract enough from each row to make $O_n$ equal to $0$ in all cases, you get:

$$\begin{matrix} P_4 = 3 & O_4 = 0 & p_4 = -1\\ P_2 = 2 & O_2 = 0 & p_2 = -2 \end{matrix}$$

which is exactly your second option.

Another, less common way to formulate this is to give $1$ point for a correct answer, and $0$ points to the expected value of random guessing. This also fits the description, assuming that you equate random guessing to "not knowing anything about the question". If this is the case, then you'll want a matrix where the following is true:

$$\begin{matrix} P_4 = 1 & O_4 = 0 & 3p_4 + P_4 = 0\\ P_2 = 1 & O_2 = 0 & p_2 + P_2 = 0 \end{matrix}$$

Solve those two equations on the right and we get:

$$\begin{matrix} P_4 = 1 & O_4 = 0 & p_4 = -1/3\\ P_2 = 1 & O_2 = 0 & p_2 = -1 \end{matrix}$$

And scale it up by three to make the point values integers:

$$\begin{matrix} P_4 = 3 & O_4 = 0 & p_4 = -1\\ P_2 = 3 & O_2 = 0 & p_2 = -3 \end{matrix}$$

and we get your third option.

So we've narrowed our choices down to your options 2 and 3 because the first two don't account for the model of "$1$ point for knowing, $0$ points for not knowing".

At this point, I'd go with option 2 for two reasons:

  1. It follows the more conventional formulation of $1$ point for a correct answer and $0$ points for an incorrect answer.

  2. It feels more aesthetically pleasing. Imposing a penalty of $-3$ points for an incorrect answer to a true-and-false feels like way too much gravity to put onto a true-false question.

Note that the AP exams, before they dropped penalties altogether, went with option 3, taking away $1/4$ point for incorrect answers to questions with $5$ choices and $1/3$ point for incorrect answers to question with $4$ choices. However, there's not much difference between $1/4$ and $1/3$, while there is a huge difference between $-1/3$ and $-1$.


That being said, I really do need to criticize the usage of penalties for incorrect answers on multiple-choice exams here. I completely fail to understand how getting an answer incorrect somehow demonstrates that the student knows less than nothing about the question, or conversely how leaving the answer blank means that the student knows any more about the question than how to take the test.

You may be (and most people are) doing this to compensate for the fact that getting an answer correct does not necessarily demonstrate that the student knows the answer to the question. But that is a flaw inherent to multiple-choice tests. You can't get rid of that just by making leaving an answer blank an equal alternative (which is what the expected-value penalty really does).

$\endgroup$
1
$\begingroup$

I think a well-written test should not have the need for that kind of penalty.

If the test were a full problem solve, the student would be encouraged to "show all work." and odds are, they'd simply do their best. At least they'd feel that if they got an incorrect answer, the teacher might give partial credit. I've seen 10 pt questions get 9 pts as it was clear the error was careless, and not a lack of understanding.

In the case of multiple choice, the negative points for a wrong answer creates a different layer of test anxiety as the student will have to ponder each question to be 'darn sure' they are right, and perhaps leave questions blank because they aren't 100% certain. Can a student really be expected to calculate their odds for each question, to decide how right they feel about an answer?

Let me add a bit to my own objection. Say there are 4 choices for each question. I am going to get 80% correct. I then have 20 more questions which I have no clue on. Guessing should give me 5% more in undeserved points. So wrong guesses should count as -1/3 to negate that extra 5% from my guessing. In reality, depending on the nature of the questions, a bright student should be able to narrow down the answer to the best 2 of 4, in which case the penalty factor should be higher, -1, so the wrong guesses cancel the 50% that were guessed right.

Depending on the level of the course, I can spend a bit of effort coaching a student to beat a multiple choice test with the same strategy I teach them for checking their own answers on the 'long' tests.

$\endgroup$
  • 3
    $\begingroup$ @Geoff - I am of the opinion that multiple choice tests are the sign of a lazy teacher. If the nature of mult choice pushes scores up, then a simple curve can pull it down to something reasonable. $\endgroup$ – JTP - Apologise to Monica May 27 '14 at 16:54
  • 1
    $\begingroup$ I don't see what you mean by "burden on the student". Say a question has five options: if they know the answer, fine, they tick it (4 pts); if they know it is between two options, fine, then they tick them both (3pts); if they know it is definitely not one option, fine, then they tick the rest (1pt). A student gets the chance to always show what they know and what they don't know. Expectation of guessing is zero. $\endgroup$ – Geoff May 27 '14 at 17:48
  • 1
    $\begingroup$ -1, although I think it's a fair point that multiple choice exams in mathematics are a bad idea, this answer explicitly does not answer the question. $\endgroup$ – Chris Cunningham May 27 '14 at 23:41
  • 1
    $\begingroup$ @ChrisCunningham - I appreciate a DV coming with a reason, and I respect you for it. On Money.SE when a member asks "How do I XXX?" there are often times an answer is a long explanation of why it's a bad idea. In my opinion, creating an exam whose grading structure is so contrived, it's just a bad idea. Admittedly, my opinion. $\endgroup$ – JTP - Apologise to Monica May 28 '14 at 1:27
  • 1
    $\begingroup$ @WillieWong One of the problems with giving partial credit for eliminating certain wrong answers is that those answers may not have been plausible in the first place. If I ask a student who the Prime Minister of Canada was from 1926 to 1931, and he remarks "well, it certainly wasn't Thomas Mulcair", does that actually show that he knows anything about who it is? What's so different about eliminating this choice on a multiple-choice exam? $\endgroup$ – Joe Z. May 28 '14 at 18:21
1
$\begingroup$

To me, it seems desirable to make each question have the same weight in determining a student's final grade. That implies that which rubric is best depends on the way you assign grades for the course. Based on your first footnote, I assume that a student's grade for the course depends on the difference between his total score and the total scores of other students. Based on that, it seems that option #2 is the most reasonable -- it will make each question have equal weight in determining the student's grade.

$\endgroup$
  • 1
    $\begingroup$ That is a good point. Yes, I expect the final "grade" to be "fitted to a curve" based on the numeric values of the test. So your argument is: since the grades are "relative", one should use the "maximum amount by which a question can advantage a student compared to his peers" and that is the difference between the correct and the incorrect score. $\endgroup$ – Willie Wong May 28 '14 at 8:23
-1
$\begingroup$

The worth of a question is $P$. Score and penalty for an answer are proportional to $P$.

This worth should not be dependant on the number of possible answers, but on the didactic worth you see in that question.

Suppose you have a question:

Compute 2+3.

with the possible answers: 1,3,5,7 or 5,7. Why should it's worth depend on the number of possible answers given? What, if you give 100 possible answers? Should the worth be 50 times that of a question with only two possible answers?

An extreme example would be a test with one question with 100 possible answers and 100 (or 50?) questions with 2 possible answers.

Valueing these questions to either of your proposed versions will render some possible outcome unfair.

$\endgroup$
  • $\begingroup$ To be more precise: I've already postulated that the questions should be worth the same. I am asking for guidance on how to translate that to setting a numeric score, preferably with justification. As far as I can tell only the first sentence of your answer has bearing on my question. And seeing that your answer is choosing one of either my option 2 or 3, I am rather confused by your final sentence. $\endgroup$ – Willie Wong May 27 '14 at 13:04
  • $\begingroup$ @WillieWong You stated in your question, that you award $P$ points for a correct answer and give a penalty of $-\frac{1}{N-1}\cdot P$ for a wrong answer. As $-\frac{1}{N-1}$ is just the factor to get random answering to an Expected value of 0, the worth or value of the answer is $P$. $\endgroup$ – Toscho May 27 '14 at 13:35
  • $\begingroup$ If you want to interpret my answer as choosing one of either of your proposed versions, than it's answer 3: Setting $P$ equal to all questions. $\endgroup$ – Toscho May 27 '14 at 13:37
  • $\begingroup$ I've assigned -1 to this answer as well because again we seem to be trying to dismiss the question rather than answer it. $\endgroup$ – Chris Cunningham May 27 '14 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.