Grading scale: how to handle multiple choice questions with different number of choices

Question

Suppose you have been told to give a multiple choice exam, and to nullify the effects of random guessing by penalizing incorrect answers. Suppose there are $N$ available choices, of which exactly one is correct.

One common approach is to give $P$ points for the correct answer, $p = -P\, /\, (N-1)$ points for an incorrect answer, and 0 points for not answering or choosing more than one answer.^*

Now suppose we have some questions with 4 available choices, and some true/false questions with only 2 available choices. How should we assign the corresponding correct answer scores $P_2$ and $P_4$ (and the respective penalties for incorrect answers $p_2$ and $p_4$) to make the questions "worth the same"?^**

For convenience we can normalise $P_4 = 3$ the corresponding penalty $p_4 = -1$:

1. Set $P_2=1, p_2 = -1$, so the penalties are the same for incorrect answers.
2. Set $P_2=2, p_2 = -2$, so the differences are the same between correct and incorrect answers.
3. Set $P_2=3, p_2 = -3$, so the differences are the same between answering correctly and not answering.
4. Set $P_2=\sqrt{3}, p_2 = -\sqrt{3}$ so the variances are the same for random guesses.

Is there a compelling reason to prefer any of the above options, or something else altogether?

---

^* To prevent hurting the students' feelings by giving them a negative score on a test, we can equivalently give 0 points for incorrect answer and P/N points for null/multiple answers. Whether we normalize the score around 0 or P/N is inconsequential for the discussion here.

^** Without penalties (for example, in the case of free-response questions) the answer is simple: just make the maximum number of points awarded for each question be the same. The question here can be re-interpreted as: does it make more sense to count the weight of each question relative to the theoretical minimum score or the theoretical average score, or something else altogether?

Answer 1

Let's consider an extreme case: An exam with two questions. Questions 1 has 2 options (only one is correct). Question 2 has 1000 options (only one is correct). Let $c_i>0$ be the points awarded for a correct answer to problem $i$ and $w_i>0$ the points subtracted for a wrong answer. Without loss of generality, we can scale $c_1=1$.

Then there are some desirable aspects we wish to achieve:

1) Random guessing for each problem has an expected value of 0 points. For this to occur we have to satisfy: \begin{eqnarray*} 0.5c_1-0.5w_1 &=&0\\ 0.001c_2-0.999w_2 &=&0 \end{eqnarray*} Then $w_1=c_1=1$.

2) Each question carries the same relative weight in damaging/helping the overall score. For this to occur we have to satisfy: \begin{eqnarray*} c_1+w_2 = w_1+c_2 \end{eqnarray*} Since then missing exactly one question results in the same score. But, in light of $c_1=w_1$ above, this reduces to \begin{eqnarray*} w_2 =c_2 \end{eqnarray*} This, of course, contradicts the null expected value condition. So we can't do (1) and (2) at the same time.

Two possible solutions/alterations:

1') Random guessing throughout the exam has an expected value of 0 points for the whole exam. For (1') and (2) to occur we have to satisfy: \begin{eqnarray*} 0.5c_1-0.5w_1+0.001c_2-0.999w_2&=&0\\ c_1+w_2 &=& w_1+c_2 \end{eqnarray*} With $c_1=1$, this reduces to \begin{eqnarray*} -0.5w_1+0.001c_2-0.999w_2&=&-0.5\\ w_1+c_2- w_2 &=&1 \end{eqnarray*} This has solution set $w_1=1-\frac{998}{1499}c_2$ and $w_2=\frac{501}{1499}c_2$. To keep $w_1>0$, then $c_2$ should be chosen less than $\frac{1499}{998}$. Maybe $c_2=1$ makes sense, in which case $w_1=w_2=\frac{501}{1499}$.

2') The other possible solution is not to weigh each question equally (e.g. weigh easy ones less or something). Suppose we want Question 2 to weigh twice as much as Question 1 (I mean, c'mon, 1000 choices makes it hard to find just the right one.) Then (1) and (2') give: \begin{eqnarray*} 0.5c_1-0.5w_1 &=&0\\ 0.001c_2-0.999w_2 &=&0\\ 2(c_1+w_2) &=& w_1+c_2 \end{eqnarray*} Setting $c_1=1$ and solving gives: $w_1=1$, $c_2=\frac{999}{997}$, $w_2 = \frac{1}{997}$.

That's about the best analysis I can think of. Of course, the above numbers can be modified for your case of 4 choices; I just think better in extremes. I'm guessing if you start to have more than two questions, the systems of equations get bigger but stay linear and solvable if you choose one of the suggested alterations.

Answer 2

I believe option number 2 is the correct one, given the circumstances that you describe.

---

At its base, you want each question on this multiple-choice test, regardless of how many possible answers it has, to be worth $1$ point if the student has demonstrated he knows the answer to the question, and $0$ points otherwise. This is the only fundamental way to make each question "worth" the same. Anything else is just clever rescaling.

The most common way to formulate this is to give $1$ point for a correct answer, and $0$ points for an incorrect answer. However, complicating this is the fact that a student might get the correct answer purely through guessing. So, you decide to give them an "incentive" (really just removing the edge from the former choice) to not answer if they have no idea, by giving them $1/n$ points, the expected value from guessing randomly, for leaving the question blank. In your scenario, this is $1/4$ point for a question with four choices, and $1/2$ point for a true-false question.

If you scale this up by a factor of $4$ to make the point values all integers, you get:

$$\begin{matrix} P_4 = 4 & O_4 = 1 & p_4 = 0\\ P_2 = 4 & O_2 = 2 & p_2 = 0 \end{matrix}$$

And if you subtract enough from each row to make $O_n$ equal to $0$ in all cases, you get:

$$\begin{matrix} P_4 = 3 & O_4 = 0 & p_4 = -1\\ P_2 = 2 & O_2 = 0 & p_2 = -2 \end{matrix}$$

which is exactly your second option.

Another, less common way to formulate this is to give $1$ point for a correct answer, and $0$ points to the expected value of random guessing. This also fits the description, assuming that you equate random guessing to "not knowing anything about the question". If this is the case, then you'll want a matrix where the following is true:

$$\begin{matrix} P_4 = 1 & O_4 = 0 & 3p_4 + P_4 = 0\\ P_2 = 1 & O_2 = 0 & p_2 + P_2 = 0 \end{matrix}$$

Solve those two equations on the right and we get:

$$\begin{matrix} P_4 = 1 & O_4 = 0 & p_4 = -1/3\\ P_2 = 1 & O_2 = 0 & p_2 = -1 \end{matrix}$$

And scale it up by three to make the point values integers:

$$\begin{matrix} P_4 = 3 & O_4 = 0 & p_4 = -1\\ P_2 = 3 & O_2 = 0 & p_2 = -3 \end{matrix}$$

and we get your third option.

So we've narrowed our choices down to your options 2 and 3 because the first two don't account for the model of "$1$ point for knowing, $0$ points for not knowing".

At this point, I'd go with option 2 for two reasons:

1. It follows the more conventional formulation of $1$ point for a correct answer and $0$ points for an incorrect answer.

2. It feels more aesthetically pleasing. Imposing a penalty of $-3$ points for an incorrect answer to a true-and-false feels like way too much gravity to put onto a true-false question.

Note that the AP exams, before they dropped penalties altogether, went with option 3, taking away $1/4$ point for incorrect answers to questions with $5$ choices and $1/3$ point for incorrect answers to question with $4$ choices. However, there's not much difference between $1/4$ and $1/3$, while there is a huge difference between $-1/3$ and $-1$.

---

That being said, I really do need to criticize the usage of penalties for incorrect answers on multiple-choice exams here. I completely fail to understand how getting an answer incorrect somehow demonstrates that the student knows less than nothing about the question, or conversely how leaving the answer blank means that the student knows any more about the question than how to take the test.

You may be (and most people are) doing this to compensate for the fact that getting an answer correct does not necessarily demonstrate that the student knows the answer to the question. But that is a flaw inherent to multiple-choice tests. You can't get rid of that just by making leaving an answer blank an equal alternative (which is what the expected-value penalty really does).

Answer 3

I think a well-written test should not have the need for that kind of penalty.

If the test were a full problem solve, the student would be encouraged to "show all work." and odds are, they'd simply do their best. At least they'd feel that if they got an incorrect answer, the teacher might give partial credit. I've seen 10 pt questions get 9 pts as it was clear the error was careless, and not a lack of understanding.

In the case of multiple choice, the negative points for a wrong answer creates a different layer of test anxiety as the student will have to ponder each question to be 'darn sure' they are right, and perhaps leave questions blank because they aren't 100% certain. Can a student really be expected to calculate their odds for each question, to decide how right they feel about an answer?

Let me add a bit to my own objection. Say there are 4 choices for each question. I am going to get 80% correct. I then have 20 more questions which I have no clue on. Guessing should give me 5% more in undeserved points. So wrong guesses should count as -1/3 to negate that extra 5% from my guessing. In reality, depending on the nature of the questions, a bright student should be able to narrow down the answer to the best 2 of 4, in which case the penalty factor should be higher, -1, so the wrong guesses cancel the 50% that were guessed right.

Depending on the level of the course, I can spend a bit of effort coaching a student to beat a multiple choice test with the same strategy I teach them for checking their own answers on the 'long' tests.

Answer 4

To me, it seems desirable to make each question have the same weight in determining a student's final grade. That implies that which rubric is best depends on the way you assign grades for the course. Based on your first footnote, I assume that a student's grade for the course depends on the difference between his total score and the total scores of other students. Based on that, it seems that option #2 is the most reasonable -- it will make each question have equal weight in determining the student's grade.

Answer 5

The worth of a question is $P$. Score and penalty for an answer are proportional to $P$.

This worth should not be dependant on the number of possible answers, but on the didactic worth you see in that question.

Suppose you have a question:

Compute 2+3.

with the possible answers: 1,3,5,7 or 5,7. Why should it's worth depend on the number of possible answers given? What, if you give 100 possible answers? Should the worth be 50 times that of a question with only two possible answers?

An extreme example would be a test with one question with 100 possible answers and 100 (or 50?) questions with 2 possible answers.

Valueing these questions to either of your proposed versions will render some possible outcome unfair.