3
$\begingroup$

Is there a common and simple way of assigning points for a test item based on the percent of students who answered the item correctly ($p$-value)? For simplicity, let us disregard for a moment the practicality (e.g., time and effort) needed and the pedagogical consequences of grading. Let us just imagine that we are constructing the scoring system in a math competition where difficult questions are rewarded more points, where the difficulty is based on the $p$-value.

(Remark: In a math competition, as opposed to a classroom setting, making the students compete for points is encouraged. So finding such a scoring system can be valuable.)

For example, consider this hypothetical five-item test and their $p$-values.
Item 1: 100%
Item 2: 80%
Item 3: 50%
Item 4: 20%
Item 5: 5%

What immediately comes to mind is to assign $\frac{1}{x}$ points for an item with a $p$-value of $x$. So the scores will be
Item 1: 1
Item 2: 1.25
Item 3: 2
Item 4: 5
Item 5: 20

It is apparent that the number of points dramatically increases as the $p$-value approaches 0. We can of course introduce immediate modifications such as getting the square root for instance, but that seems too artificial.

PS: The Harvard-MIT Mathematics Tournament offers such a scoring system. Based on the overview, I trust that their scoring system is indeed very effective, but I just find it too difficult to understand.

$\endgroup$
  • 5
    $\begingroup$ Why should a more difficult question give more points? Isn't it enough that only few students get points for the hard questions? In your example correctly answering question 5 is more valuable than questions 1-4 together, which feels wrong to me. Especially in a multiple choice situation being lucky on the almost impossible question shouldn't count that much. Can you try to explain what you are trying to achieve with the system? $\endgroup$ – Joonas Ilmavirta Apr 26 '15 at 12:51
  • $\begingroup$ Teachers assign points to questions in an exam. Questions deemed easy by the teacher are worth less points than harder ones. Although done with all sincerity and care, this is can be bit arbitrary. So assigning points can alternatively be done (albeit more tedious) after the exam is administered. Such a scoring system becomes especially useful in math competitions, for example. $\endgroup$ – Mark Lao Apr 26 '15 at 16:51
  • 4
    $\begingroup$ I'm more used to the way of thinking that easier and harder problems are worth the same amount. To get a good score, one has to answer well questions of all levels. An outstanding solution to a difficult problem can be canceled by a terrible answer to a simple one. I see no reason to exaggerate the point gap between the top students and the mediocre ones. Another point is that it is fair if the students know in advance the maximum points for each problem and can divide their time accordingly. If a question is important, it should give many points even if it is easy. (In my opinion, anyway.) $\endgroup$ – Joonas Ilmavirta Apr 26 '15 at 17:05
  • $\begingroup$ I completely agree with everything you said Joonas. In fact, I apply the principles you mentioned in all my classes. This is the reason why I put this disclaimer: For simplicity, let us disregard for a moment the practicality (e.g., time and effort) needed and the pedagogical consequences of grading. I asked this question just as a mental exercise after seeing the Harvard-MIT Mathematics Tournament scoring system, which I find very interesting. $\endgroup$ – Mark Lao Apr 26 '15 at 17:10
1
$\begingroup$

You could start by assigning the same point value to each question. Then run a multiple regression with total score earned as the dependent variable and the score earned on each question as independent variables. Perhaps then you would assign new point values to each question as a function of its R-squared value in the multiple regression.

This function could be as simple as a direct proportion. In that case, any item that explained 0% of the variation in total score (such as an item that all students answered correctly) would be worth 0 points in the new set of point values.

Whatever function you choose, you would then have the option to iterate the procedure, running a new multiple regression based on the new point values of the items. My only concern with that iterative approach is that the limit of the iterations might yield a scoring system in which the total score is essentially determined by a single item.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.