Way to assign points for an item based on the percent of students who answered the item correctly

Question

Is there a common and simple way of assigning points for a test item based on the percent of students who answered the item correctly ($p$-value)? For simplicity, let us disregard for a moment the practicality (e.g., time and effort) needed and the pedagogical consequences of grading. Let us just imagine that we are constructing the scoring system in a math competition where difficult questions are rewarded more points, where the difficulty is based on the $p$-value.

(Remark: In a math competition, as opposed to a classroom setting, making the students compete for points is encouraged. So finding such a scoring system can be valuable.)

For example, consider this hypothetical five-item test and their $p$-values.
Item 1: 100%
Item 2: 80%
Item 3: 50%
Item 4: 20%
Item 5: 5%

What immediately comes to mind is to assign $\frac{1}{x}$ points for an item with a $p$-value of $x$. So the scores will be
Item 1: 1
Item 2: 1.25
Item 3: 2
Item 4: 5
Item 5: 20

It is apparent that the number of points dramatically increases as the $p$-value approaches 0. We can of course introduce immediate modifications such as getting the square root for instance, but that seems too artificial.

PS: The Harvard-MIT Mathematics Tournament offers such a scoring system. Based on the overview, I trust that their scoring system is indeed very effective, but I just find it too difficult to understand.

Answer 1

You could start by assigning the same point value to each question. Then run a multiple regression with total score earned as the dependent variable and the score earned on each question as independent variables. Perhaps then you would assign new point values to each question as a function of its R-squared value in the multiple regression.

This function could be as simple as a direct proportion. In that case, any item that explained 0% of the variation in total score (such as an item that all students answered correctly) would be worth 0 points in the new set of point values.

Whatever function you choose, you would then have the option to iterate the procedure, running a new multiple regression based on the new point values of the items. My only concern with that iterative approach is that the limit of the iterations might yield a scoring system in which the total score is essentially determined by a single item.

Answer 2

From an information theory perspective, if the proportion of people getting a question right is $p$, then someone getting that question right is worth $-\log p$ units of information, while someone getting it wrong is worth $-\log (1-p)$ units of information. This does go to infinity as $p$ goes to $0$, but not as quickly as $1/x$. So using this formula, someone who gets a question correct would gain $-\log p$ points, while someone who gets the problem wrong would lose $-\log (1-p)$. As $p$ goes to $0$, the penalty for getting the problem wrong would go to zero. However, as $p$ goes to $1$, the penalty would go to infinity. If that doesn't appeal to you, you can get rid of the penalty, and adjust the scores of only the people who get the problem right, or make sure that all of your problems have $p$ less than $0.5$ (after all, the purpose of questions is to separate the best students from the rest, and if most of the students are getting the question right, then that's not separating out the best students).

Answer 3

Preface

This question is a bit old, so maybe I should just leave it be, but it intrigued me as I'd never seen points assigned like this. As I've never seen points assigned like this, I can't comment on common scoring systems, but I can come up with simple systems as you've emphasized in the question.

Of possible note, I'm unable to access the Harvard scoring system linked in the question. It gives me an error that I'm not authenticated, which likely means I need a student or faculty account at Harvard to access it, which I don't have.

The Simple Function

Let:
$n$ be the normal score for the question,
$max$ be the maximum score a hard question can be worth,
$M$ be the multiplier for the maximum score, so $M=\frac{max}{n}$,
$m$ be the additive portion of the multiplier, so $m=M-1$,
$p$ be the p-value as a percent of students who got the question right,
and $v$ be the actual value of the question after p-values are considered.

Then use:
$v=n(1+m-mp)$

which gives a range of $n$ to $M\cdot n$.

Then you can set $M$ to something reasonable, like 1.5, and a student who just answers the hardest questions can only get 50% more points than a student who just answers the easiest questions.

Existing Scoring Methods

I've seen points assigned by subjective difficulty, or by length of time taken (which is still somewhat subjective, but can be measured for an "average" case). The idea being that you get a fairly constant number of points based on the effort or time spent, regardless of which questions you worked on.

For example, if questions 1-10 take one minute each, and questions 11-15 take five minutes each, we could assign 3 points each for the first 10, and 15 points each for the last 5. A student who answers all questions should spend about 35 minutes and could get up to 105 points (which could be capped, allowing a mistake or two while still getting a perfect score, or not). That gives 20 extra minutes for the slow students, etc.

A person who only spends 20 minutes on the test could answer all the short questions and two long questions for 60 points, or they could answer four long questions for the same 60 points. Etc.

Points by Percent Answered Correctly

If the goal is to objectively assign difficulty using the percentage of students who answered the question correctly, the trick is to find a function that assigns the range 0-1 in a way you find fair.

Your point system says the value, v, of an answer is given by $v=\frac{n}{p}$, where n is the number of points normally graded, and p is the percent of students who got the question right.

This puts a lot of extra weight on the questions only a few students get right. The person who gets all five questions right gets 100%, someone who only gets question 5 right gets 68%, and someone who only misses question 5 (which is 95% of the class) gets a maximum of 32%. That's clearly a bad scoring system.

Scaling the P-Factor

We can use scaling and offsets to tweak that system. Currently, since the denominator, p, can scale down to 0, we can get arbitrarily large score values. If we put p through a function such that the denominator has a limited range, we can limit the score of any particular answer.

For example, $v=\frac{n}{0.9 + \frac{p}{10}}$ gives us a minimum score of $n$ when $p=1$, and a maximum score of $\frac{n}{0.9}\approx 1.1n$ when $p=0$.

I prefer to think in terms of multipliers, instead of divisors, so I'd modify it a bit to $v=n\cdot(1.1-\frac{p}{10})$, giving a score range of exactly $\{n, 1.1n\}$.

We can further extend that to any arbitrary maximum relative score. Let $M$ be the maximum relative score we can give, so $M=2$ would give a final score of $2n$. Then let $m=M-1$, so it's the additional relative score. So for $M=2$, $m=1$, giving 100% extra score for the question nobody answered.

Now, for any given $m$, we can write $v=n(1+m-mp)$.

Examples:

$M=2$, so $m=1$:
$v=n(1+1-1p)$
$=n(2-\{1,0\})$
$=\{1,2\}n$.

$M=1.5$, so $m=0.5$:
$v=n(1+0.5-0.5p)$
$=n(1.5-\{0.5, 0\})$
$=\{1, 1.5\}n$.

Negative Offsets

We can offset the result so easy questions are worth less than $n$ and hard questions are worth more than $n$, but we have to be careful that we're not assigning negative points for large values of $m$.

For $m<2$, we can do $v=n(1+m-mp-\frac{m}{2})$ $=1+\frac{m}{2}-mp$.

For $m=1$, that gives
$v=n(\{1,2\} - 0.5)$
$=\{0.5, 1.5\}n$.

For $m=0.5$, it's
$v=n(\{1,1.5\} - 0.25)$
$=\{0.75, 1.25\}n$.

Setting $m=0.4$, gives
$v=\{0.8, 1.2\}n$
which gives the hardest questions 50% more points than the easiest.

Piecewise Offsets

For large values of m, it's probably better to have offsets greater than 1 be multiplicatively proportional to offsets less than 1. So if the highest score is $n\cdot M$, the lowest score is $\frac{n}{M}$.

Of course, now we're getting into the same issue as your original function: large discrepancies make hard questions too profitable. But the math still works, so we'll do it for the sake of discussion. Plus it might be useful for something like Jeopardy, where hard questions are supposed to be worth a lot. Setting M to 2.45 gives hard answers a 50% bonus over easy answers like we did above, but at that point you might as well just save yourself some headache and use the easy formula.

Modify the above function (aka "create a new function from scratch and don't bother writing down the derivation") to get
$\require{amsmath}v=\begin{cases} n\cdot[\frac{m+1}{2} - p\cdot(m-1)] & p\leq 0.5 \\ n\cdot[1-(2\cdot p - 1)\cdot(1-\frac{2}{m+1})] & p\gt 0.5 \end{cases}$

Example, $M=5$, so $m=4$, we want our maximum score to be $5n$ and our minimum score to be $\frac{n}{5}$. Because we're offsetting, actual max is $\frac{5n}{2}=2.5n$ and actual min is $2\cdot\frac{n}{5}=\frac{n}{2.5}$. ¹ The system works best for $M>2$; otherwise, you get lower scores for hard questions.

When $p=0$, we use
$v=n\cdot[\frac{m+1}{2} - p\cdot(m-1)]$
$=n\cdot[\frac{4+1}{2} - 0\cdot 3]$
$=n\cdot[2.5-0]$
$=2.5\cdot n$.

When $p=0.25$, we use
$v=n\cdot[\frac{m+1}{2} - p\cdot(m-1)]$
$=n\cdot[\frac{4+1}{2} - 0.25\cdot 3]$
$=n\cdot[2.5-0.75]$
$=1.75\cdot n$.

When $p=0.5$, we use
$v=n\cdot[\frac{m+1}{2} - p\cdot(m-1)]$
$=n\cdot[\frac{4+1}{2} - 0.5\cdot 3]$
$=n\cdot[2.5-1.5]$
$=1\cdot n$.

When $p=0.5$, we could use
$v=n\cdot[1-(2\cdot p - 1)\cdot(1-\frac{2}{m+1})]$
$=n\cdot[1-(2\cdot 0.5 - 1)\cdot(1-\frac{2}{4+1})]$
$=n\cdot[1-(0)\cdot(1-0.4)]$
$=n\cdot[1 - 0]$
$=1\cdot n$
showing there's continuity at the break.

When $p=0.75$, we use
$v=n\cdot[1-(2\cdot p - 1)\cdot(1-\frac{2}{m+1})]$
$=n\cdot[1-(2\cdot 0.75 - 1)\cdot(1-\frac{2}{4+1})]$
$=n\cdot[1-(0.5)\cdot(1-0.4)]$
$=n\cdot[1 - 0.5\cdot 0.6]$
$=n\cdot[1-0.3]$
$=0.7\cdot n$.

When $p=1$, we use
$v=n\cdot[1-(2\cdot p - 1)\cdot(1-\frac{2}{m+1})]$
$=n\cdot[1-(2\cdot 1 - 1)\cdot(1-\frac{2}{4+1})]$
$=n\cdot[1-(1)\cdot(1-0.4)]$
$=n\cdot[1 - 0.6]$
$=0.4\cdot n$.

¹ If you prefer, you can define $M$ as actual max, then $m=2\cdot M-1$, $M>1$ gives good results, and the rest follows properly.

Selecting Sample Data

There are a few different ways to assign the actual p-values. First, values can be measured against the current class, previous class(es), or some standard.

The easiest way is to give the test questions to some volunteers and see how many they get right. Record the p-values and assign scores to each question as above. Now, each time you grade future tests, the score multiplier for that question is already known.

To get a larger sample set, you can aggregate the scores over each class. After a few semesters, you've got hundreds of samples for each question instead of the four samples from TAs you coerced into helping you.

At some point, it might be good to drop your earliest classes, so you have a rolling average over 5 years or something. This keeps scores current, while also smoothing out anomalies.

You can also assign the p-value based on how many students got it right on this exact test. This is probably fine for a competition, but leads to a big problem for classwork: if my score is based on a p-value from other people in the class, it's in my best interest to not help them, or to even mislead them into scoring poorly on certain topics I put more effort into.

By scoring this test based on previous classes, I have no reason to deliberately sabotage my current classmates, which is generally a good thing.

Scoring Partial Credit

The second thing to consider when assigning p-values is partial credit. The above scoring assumes questions are pass/fail. In the real world, we often assign some points to answers that are incomplete or partially wrong, but got some parts right.

In this case, instead of $p$ being the percent of right answers, we simply make it the average percent of maximum score attained. If a question is normally worth 10 points, and the average score is 6.2, then $p=62\%$ for that question.

The Pendulum Effect

When we set pressures on a population based on previous performance, we can introduce a pendulum effect where the population bounces between different extremes.

Classes one year might do well in one metric, then we change the scoring to motivate the next year's class to do better in areas the first class scored poorly. Then the second class performs poorly in the area the first class did well on. So we change the scoring again and we're back where we started.

Instead of radically changing scoring each semester, it's better to slowly change scoring. So we only slightly incentivize year 2 in the areas year 1 was bad at. If year 2 still does badly in that area, we raise the incentive a bit more for year 3, etc.

The idea is to get to a point where no topic has particularly higher p-values than another topic, implying that students are getting a well-rounded experience from the class.

By aggregating p-values over several years, it helps smooth these jumps so we tend towards an equilibrium in the middle, instead of ping-ponging between extremes.

Of course, this requires that students are aware of the scoring, and which topics are worth more points. Otherwise they won't know to spend a bit more time on the harder topics.

It also requires consistent questions over many years. If you constantly change your test questions, it will likely throw the entire system off.

Answer 4

If you only care about scoring a single assessment that has already happened, you could grade each question on a curve: the scores (just 0s and 1s) get shifted and rescaled to have a fixed mean and standard deviation.

This has the effect of increasing the number of points very significantly (if I've done my math right, something like: https://www.desmos.com/calculator/pf33jysqex) where $x$ is the percent correct) for problems that are very commonly gotten right or wrong.

Of course, if a problem is gotten entirely correct or entirely incorrect, you should drop it when scoring since it doesn't have any data.

Aside: I'm curious how extrinsic motivation in math competitions affects motivation in math competitions and in the classroom. The typical relationship is that extrinsic motivation undermines intrinsic motivation and creates a dependence on extrinsic motivation. But are there studies about this specific type of extrinsic motivation for this specific audience?