11
$\begingroup$

I am trying to come up a method to grade a multiple choice question. I prepared a test that consists of some questions and each question has 10 options for which 3 to 5 options are correct. When I grade them, I want to give high points if the student picked correct options, and give penalty for options selected incorrectly.

Here's an example question:

Which of the below numbers are even? 
 1.) 17 
 2.) 22
 3.) 33
 4.) 42
 5.) 57
 6.) 61
 7.) 49
 8.) 99
 9.) 13
10.) 30

Actual correct answers are options 2, 4, 10. So, there are 3 options are correct.

Let's say a student picked 4 options (2, 4, 5, 10). 3 of them are correct, 1 incorrect.

Is there a method in the literature for grading this kind of questions?

Edit: The range of grade should be 0 to 1 (regardless of number of correct answers). Also, can we use an exponential function where we award each correct answers and penalize each incorrect answers?

$\endgroup$
  • 3
    $\begingroup$ If I were setting up a scheme like this, then I might put the scoring system on the test. E.g., 2 points for each correct answer, -1 for each incorrect answer. The numbers I've provided are intended as placeholders; you'd have to determine what you think is fair. (Note that with +2 for correct and -1 for incorrect, someone who picks all 10 answers, in this particular case, would end up with -1 points; so it's marginally better to skip the question if one doesn't know what is meant by even/odd. Probably this is a desirable feature!) $\endgroup$ – Benjamin Dickman Mar 19 '15 at 9:27
  • $\begingroup$ @MichaelE2, the thing is that we do not know a priori number of correct answers for different questions. So, it would not be fair to apply the same rubrics for all questions. I think the method should be proportion to number of correct answers. $\endgroup$ – Co Koder Mar 19 '15 at 17:32
  • $\begingroup$ @BenjaminDickman, I think I forgot to mention that each question should have a range of 0 to 1. Also, your method could have a potential problem such that if there are 5 options correct, and a student picks all 10 options, he will get 5*2 - 5*1 = 5 points, which is not fair, right? $\endgroup$ – Co Koder Mar 19 '15 at 17:37
  • $\begingroup$ I adjusted the title to more reflect the question and answers. It differs slightly with a multiple choice question with picking one. $\endgroup$ – Chris C Mar 22 '15 at 16:07
  • 1
    $\begingroup$ There is a recent blog post by T Tao entitled How to assign partial credit on an exam of true-false questions? that may be of some interest... $\endgroup$ – Benjamin Dickman Jun 2 '16 at 5:49
11
$\begingroup$

For multiple choice questions it is much better to ask in a slightly different way, namely

Are the following numbers even?
     yes  no
 1.)  O   O   17 
 2.)  O   O   22
 3.)  O   O   33
 4.)  O   O   42
 5.)  O   O   57
 6.)  O   O   61
 7.)  O   O   49
 8.)  O   O   99
 9.)  O   O   13
10.)  O   O   30

The advantage is, that you explicitly open the option "I don't know." (and also the possibility to say "Both"). Consequently, every cross set by the student has a meaning and a missing cross also says something.

I read on the German Wikipedia site on multiple choice questions that a fair grading that takes guessing into account would be:

Every correct cross gives one point and every wrong cross results in negative points. In this example with only two possibilities for every answer each wrong cross should give one negative point (hence, one correct and one incorrect cross cancel each other).

For questions with more possible choices, a wrong crosses should give less negative points.

It seems that there are (at least in Germany) quite some court decisions on how multiple choice test have to be graded (and not all make perfect sense…).

I don't know any English resources on the topic…

$\endgroup$
4
$\begingroup$

As Michael E2's comment already says: How you grade depends on what you want to achieve. One goal might be, that pure guessing should have the same mean value as not doing anything at all.

In a yes-no-scheme like in Dirk's answer, that is easy. Every wrong answer gives the same points negativ, which the correct answer gives positive.

If only correct answers are to be marked (and wrong answers are indistinguishable from no answer), this needs to be different.

Suppose a question has $t$ correct and $f$ wrong answer options and the question has a weight of $w$, which means giving exactly the correct answers yields $w$ points. Then each correct mark gives $\frac{w}{t}$ points and each wrong mark gives $-\frac{w}{f}$ points.

Example: I take your example. The weight is $1$, there are $t=3$ correct answers and $f=7$ wrong answers. So each correct mark gives $\frac{1}{3}$ point, each wrong mark gives $-\frac{1}{7}$ point. The students marks all three correct options and one wrong option, so he should get $3\cdot\frac{1}{3}+1\cdot\left(-\frac{1}{7}\right)=\frac{6}{7}$ points.

Reasoning: If you don't discriminate the answer options, then you get the following crosstab for the points to be awarded:

|       |mark|no mark|
----------------------
|correct| x  | 0     |
|wrong  | y  | 0     |

The 0 on the right side are necessary, because you don't award someone points (neither positive nor negative) for not answering. The $y$ is gonna be negative, of course.

Let's take a random guesser. Each correct options gives him an expected value of $\frac{x}{2}$. Each wrong option gives him an expected value of $\frac{y}{2}$. Suppose, that the random guesser answers each option independantly (by no way realistic if $t=1$ is not known). Than the expected value of the whole question is $E=t\cdot\frac{x}{2}+f\cdot\frac{y}{2}$. This should be $0$, so you get $$tx+fy=0$$ Furthermore, giving exactly the correct answers earns $tx$ points, which should be the weight $w$: $$tx=w$$ Solving these equations for $x$ and $y$ yields: $$x=\frac{w}{t},\quad y=-\frac{w}{f}=-\frac{t}{f}x$$

$\endgroup$
  • $\begingroup$ Can you a give us a numerical example? $\endgroup$ – Co Koder Mar 19 '15 at 17:28
  • 1
    $\begingroup$ This answer could be improved by giving the result at the top, with an example, and then doing the proof. Something like: "If the question is worth w in total, and there are t correct answers and f wrong answers in the list, then the score for each correct answer selected should be w/t and the score for each incorrect answer selected should be -w/f. In your example, there are 3 correct and 7 incorrect and the total for the question is 1, so each correct is 1/3 and each incorrect is -1/7. If a student selects 3 correct and 1 incorrect so the score would be 3*1/3 - 1/7 = 6/7. The proof is..." $\endgroup$ – DavidButlerUofA Mar 19 '15 at 22:12
  • $\begingroup$ @DavidButlerUofA Thanks, I adopted your proposal. $\endgroup$ – Toscho Mar 20 '15 at 9:50
  • $\begingroup$ @Toscho, I think your approach assumes that the number of correct options are always greater than number of wrong answers. For example, let's assume that there are 9 options are correct and 1 incorrect. If a student picks 8 correct options and 1 incorrect options, he will get 8*1/9 - 1*1/1 = -1/8 which is negative. Do you think it is fair? He got 8 out of 9 options correct and 1 wrong. $\endgroup$ – Co Koder Mar 22 '15 at 2:29
  • $\begingroup$ @CoKoder You're right, that it's possible to get negative results per question. As I point out in my first sentence, the solution depends on your aim. I didn't say, that fairness is an aim for my solution. That'd be a bad aim, as it cannot be modeled truely with mathematics. And beware: If negative points are not possible, then pure guessing is always better than not doing anything and so you encourage pure guessing in case of lacking knowledge. $\endgroup$ – Toscho Mar 24 '15 at 15:56
2
$\begingroup$

Some people have given this question (and a lot of similar ones) a great deal of thought and come up with industry standards: QTI.

  1. Yes, there is something in the literature about it.
  2. You can use an exponential function (or just about anything you want). But I would caution it against it. Do you want to spend more time explaining the subject matter or more time explaining the grade?
$\endgroup$
  • 2
    $\begingroup$ Can you give us an example that uses exponential function? $\endgroup$ – Co Koder Mar 19 '15 at 22:02
2
$\begingroup$

Really great question with a great set of answers. I would like to offer another form of multiple choice assessment that I use frequently that I am really pleased with. I know that this will not directly answer your question as per your description, but given the title I think this answer can be useful for future MESE users who come across this question. Ultimately, I believe that you are trying to create a more rigorous multiple-choice assessment that actually digs into a students thinking and is not just a "black-box of guessing", hence the penalty for incorrect answers and having multiple correct answers per question (a really great strategy that I am definitely going to use ;), but also remaining as fair as possible. What I do is three-fold...

  1. Each question has five options for answers. Four are potential solutions to the problem and the last is described in 2. For each question there is one correct answer and one "almost correct" answer. If the student chooses the correct answer they get full credit and if they choose the AC answer they get half credit. The AC answer will typically involve making one standard error such as missing a sign, confusing operations, or a logic error you have noticed frequently happening with the class.

  2. Provide an option E for each questions that allows a student to write in an answer that is different from those provided. Generally, I only do this on one question per assessment and I tell the students ahead of time that there is exactly one question that requires a write-in. If a student gets the question correct, it proves a high level of understanding with the topic of the question. If they get it wrong, they still have a chance to prove some level of understanding, i.e. they pick the AC answer or their explanation is logical (see 3.). Sometimes, usually with accelerated classes I will have multiple write-ins required and not inform the students of how many there are. It is really up to you and what you on which way to go, depending on your class.

  3. Each question has a space underneath to write one to two sentences justifying the students selected answer. This not only shows a students real understanding but it also gives really good insight into which questions were guessed on. It also gives students a chance to express math in words, which I believe to be a very powerful thing to learn and something shirked by many educators. As far as grading this part, I usually will do it with a pass-fail mentality, i.e. if it is "more than half logical" the student will receive full credit, but anything that is illogical is circled and a note is written either explaining the fallacy if it is brief or "ok but see me" if it is a more serious misunderstanding. I have had these explanations factor into the grade for the question in two ways in the past: worth some amount of bonus points (usually a fractional amount) or worth some percentage of the questions total value.

A few notes on this strategy..

  • It takes a lot of time and energy to both create and grade, I will usually only do a fully multiple choice assessment like this for cumulative assessments such as midterms and finals
  • It is very different from most multiple choice assessments your students have likely ever taken. Because of that, ease them in putting a few questions like this on other assessments to get them used to the style of MC question
  • Because of the nature of these questions, they should be deep and meaningful. As a result of this, these questions will usually take students two to three times as long to complete so keep that in mind when creating these questions
$\endgroup$
1
$\begingroup$

In https://educationrealist.wordpress.com/2014/01/17/multiple-answer-math-tests/ a California high school math teacher found that the best way to grade multiple answer tests is:

Each answer is an individual True/False question. That works really well. So if you have a six-option question, you can get 6/6, 5/6, 4/6 etc. Then you assign point totals for each option.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.