A Method to Grade a Multiple Statement or True and False Question

Question

I am trying to come up a method to grade a multiple choice question. I prepared a test that consists of some questions and each question has 10 options for which 3 to 5 options are correct. When I grade them, I want to give high points if the student picked correct options, and give penalty for options selected incorrectly.

Here's an example question:

Which of the below numbers are even? 
 1.) 17 
 2.) 22
 3.) 33
 4.) 42
 5.) 57
 6.) 61
 7.) 49
 8.) 99
 9.) 13
10.) 30

Actual correct answers are options 2, 4, 10. So, there are 3 options are correct.

Let's say a student picked 4 options (2, 4, 5, 10). 3 of them are correct, 1 incorrect.

Is there a method in the literature for grading this kind of questions?

Edit: The range of grade should be 0 to 1 (regardless of number of correct answers). Also, can we use an exponential function where we award each correct answers and penalize each incorrect answers?

Answer 1

For multiple choice questions it is much better to ask in a slightly different way, namely

Are the following numbers even?
     yes  no
 1.)  O   O   17 
 2.)  O   O   22
 3.)  O   O   33
 4.)  O   O   42
 5.)  O   O   57
 6.)  O   O   61
 7.)  O   O   49
 8.)  O   O   99
 9.)  O   O   13
10.)  O   O   30

The advantage is, that you explicitly open the option "I don't know." (and also the possibility to say "Both"). Consequently, every cross set by the student has a meaning and a missing cross also says something.

I read on the German Wikipedia site on multiple choice questions that a fair grading that takes guessing into account would be:

Every correct cross gives one point and every wrong cross results in negative points. In this example with only two possibilities for every answer each wrong cross should give one negative point (hence, one correct and one incorrect cross cancel each other).

For questions with more possible choices, a wrong crosses should give less negative points.

It seems that there are (at least in Germany) quite some court decisions on how multiple choice test have to be graded (and not all make perfect sense…).

I don't know any English resources on the topic…

Answer 2

As Michael E2's comment already says: How you grade depends on what you want to achieve. One goal might be, that pure guessing should have the same mean value as not doing anything at all.

In a yes-no-scheme like in Dirk's answer, that is easy. Every wrong answer gives the same points negativ, which the correct answer gives positive.

If only correct answers are to be marked (and wrong answers are indistinguishable from no answer), this needs to be different.

Suppose a question has $t$ correct and $f$ wrong answer options and the question has a weight of $w$, which means giving exactly the correct answers yields $w$ points. Then each correct mark gives $\frac{w}{t}$ points and each wrong mark gives $-\frac{w}{f}$ points.

Example: I take your example. The weight is $1$, there are $t=3$ correct answers and $f=7$ wrong answers. So each correct mark gives $\frac{1}{3}$ point, each wrong mark gives $-\frac{1}{7}$ point. The students marks all three correct options and one wrong option, so he should get $3\cdot\frac{1}{3}+1\cdot\left(-\frac{1}{7}\right)=\frac{6}{7}$ points.

Reasoning: If you don't discriminate the answer options, then you get the following crosstab for the points to be awarded:

|       |mark|no mark|
----------------------
|correct| x  | 0     |
|wrong  | y  | 0     |

The 0 on the right side are necessary, because you don't award someone points (neither positive nor negative) for not answering. The $y$ is gonna be negative, of course.

Let's take a random guesser. Each correct options gives him an expected value of $\frac{x}{2}$. Each wrong option gives him an expected value of $\frac{y}{2}$. Suppose, that the random guesser answers each option independantly (by no way realistic if $t=1$ is not known). Than the expected value of the whole question is $E=t\cdot\frac{x}{2}+f\cdot\frac{y}{2}$. This should be $0$, so you get $$tx+fy=0$$ Furthermore, giving exactly the correct answers earns $tx$ points, which should be the weight $w$: $$tx=w$$ Solving these equations for $x$ and $y$ yields: $$x=\frac{w}{t},\quad y=-\frac{w}{f}=-\frac{t}{f}x$$

Answer 3

Some people have given this question (and a lot of similar ones) a great deal of thought and come up with industry standards: QTI.

1. Yes, there is something in the literature about it.
2. You can use an exponential function (or just about anything you want). But I would caution it against it. Do you want to spend more time explaining the subject matter or more time explaining the grade?

Answer 4

Really great question with a great set of answers. I would like to offer another form of multiple choice assessment that I use frequently that I am really pleased with. I know that this will not directly answer your question as per your description, but given the title I think this answer can be useful for future MESE users who come across this question. Ultimately, I believe that you are trying to create a more rigorous multiple-choice assessment that actually digs into a students thinking and is not just a "black-box of guessing", hence the penalty for incorrect answers and having multiple correct answers per question (a really great strategy that I am definitely going to use ;), but also remaining as fair as possible. What I do is three-fold...

1. Each question has five options for answers. Four are potential solutions to the problem and the last is described in 2. For each question there is one correct answer and one "almost correct" answer. If the student chooses the correct answer they get full credit and if they choose the AC answer they get half credit. The AC answer will typically involve making one standard error such as missing a sign, confusing operations, or a logic error you have noticed frequently happening with the class.

2. Provide an option E for each questions that allows a student to write in an answer that is different from those provided. Generally, I only do this on one question per assessment and I tell the students ahead of time that there is exactly one question that requires a write-in. If a student gets the question correct, it proves a high level of understanding with the topic of the question. If they get it wrong, they still have a chance to prove some level of understanding, i.e. they pick the AC answer or their explanation is logical (see 3.). Sometimes, usually with accelerated classes I will have multiple write-ins required and not inform the students of how many there are. It is really up to you and what you on which way to go, depending on your class.

3. Each question has a space underneath to write one to two sentences justifying the students selected answer. This not only shows a students real understanding but it also gives really good insight into which questions were guessed on. It also gives students a chance to express math in words, which I believe to be a very powerful thing to learn and something shirked by many educators. As far as grading this part, I usually will do it with a pass-fail mentality, i.e. if it is "more than half logical" the student will receive full credit, but anything that is illogical is circled and a note is written either explaining the fallacy if it is brief or "ok but see me" if it is a more serious misunderstanding. I have had these explanations factor into the grade for the question in two ways in the past: worth some amount of bonus points (usually a fractional amount) or worth some percentage of the questions total value.

A few notes on this strategy..

• It takes a lot of time and energy to both create and grade, I will usually only do a fully multiple choice assessment like this for cumulative assessments such as midterms and finals
• It is very different from most multiple choice assessments your students have likely ever taken. Because of that, ease them in putting a few questions like this on other assessments to get them used to the style of MC question
• Because of the nature of these questions, they should be deep and meaningful. As a result of this, these questions will usually take students two to three times as long to complete so keep that in mind when creating these questions

Answer 5

In https://educationrealist.wordpress.com/2014/01/17/multiple-answer-math-tests/ a California high school math teacher found that the best way to grade multiple answer tests is:

Each answer is an individual True/False question. That works really well. So if you have a six-option question, you can get 6/6, 5/6, 4/6 etc. Then you assign point totals for each option.