Item Analysis in Mathematics Exams

Question

I'd like to process data like the following, from my most recent seven-question exam.

       Item     1     2     3     4     5     6     7
PtsPossible     9    13    11    12    12    11    12
     Median     8     9     8     7    10    10    12
       Mean  7.10  9.10  7.25  6.65  9.27  9.39 10.57   
      Stdev  2.22  2.05  3.10  2.86  2.46  1.94  1.89          n = 51

Essentially I'd like to know what this data table can tell me about my exam design.

I've read a very small amount about item analysis in general, but I lose patience with the authors immediately when I see things like: "The question difficulty is defined as the percentage of students who selected the correct response." This definition makes sense if the question is asking for factual recall, but mathematics questions with partial credit are not like this.

What useful item analysis can be done post-exam, besides seeing which questions were the "hardest?"

Answer 1

To answer your direct question, not much. But, assuming you have available the analytical grades and not just the statistics:

In the specific example, questions are distinguished by their maximum value "PtsPossible". So they are different objects, because they have a possibly different range -the "maximum points awarded" are not necessarily the same (even though the variability is not large). Now, you may have a detailed analysis and discussion to offer about the "difficulty" of each answer (importance, length of requirements to answer, nature of the question etc etc), but "frankly my dear I don't give a damn" :) -no matter what your considerations were, for my distant statistical eyes, you synthesized them and compacted them all in one single value - the "PtsPossible" per question.
Although this ranking is important, for the moment, "standardize" your data to make the grades comparable: Express the grading of each answer by each student as a percentage of the maximum points that the specific question gives. You have transformed your sample into "relative achievement scores" (or whatever you want to call it). Then recalculate the descriptive statistics of your sample. Now all your data range in 0-1 (or 0-100%), and now they are comparable.

Now use the fact that each question has a different maximum value: order the questions ranking them by their "PtsPossible", and see whether the descriptive statistics follow the same pattern (e.g. the more "valuable" the question the higher the average relative achievement), or the inverse pattern, or no pattern at all. Although at this level of aggregation you have only 7 data points (the 7 "PtsPossible" and the 7 values of each descriptive statistic), you can calculate their correlation coefficient. Even better, graph the 7-point series of PtsPossible pair-wise with each series of a descriptive statistic in a X-Y scatter-plot, to see visually whether there is a clear linear or non-linear association. If there is an association, how can you interpret it? E.g., if more valuable questions exhibit lower variability in the grades (e.g. lower st.dev.), it is an indication that you have designed the test so that the students had a relatively easy day... Is that so? etc.

Create histograms with bars of length 0.05 or 0.1 for each question and look at them -are they unimodal, bimodal? Unimodal shapes will tell you that students are relatively more homogeneous regarding their knowledge/performance, compared to a question whose answer grades have a bimodal histogram. Does this accords with, or contradicts, any related opinion you had prior to the test?

Get centered visual insights: subtract from all data points the relevant sample mean (per question), obtaining a zero-mean data sample. Create here too histograms, and graph together, not the bars, but all the lines that each connects the middle point of each bar (i.e a simple empirical "probability density function" for each question), for this data sample. You will immediately see whether they more or less "look the same" or whether some have a visibly different shape, without being confused by possible different mean values. This will tell you immediately whether the distribution of the questions differ in terms of skewness, i.e. you will obtain visually the information that the comparison between mean and median usually gives.

As you will be doing all these, your brain will start to combine the new information with the presumably good knowledge you have about the intricacies of each question, as well as about your students -and conclusions will emerge, usually richer than "which question was the hardest". And richer conclusions require richer analysis.

Answer 2

I've got no suggestions for item analysis on your presented data; stop reading if the rest is irritating because it doesn't answer your question.

I've read a fair bit on test design, and the vast majority of it is based upon analysis of multiple-choice questions... which you clearly do not have. And, sadly, I don't think the total points you've shown will help give you the formative assessment you want and deserve.

I've done some analysis of short-answer biology questions, and the errors come down to things like a) not understanding the biology, b) understanding the biology but being unable to work through the logic needed to determine the answer, or c) understanding the biology and likely able to do the work but not understanding the question. Obviously, "c" requires immediate rectification (even when grading that exam), and the other two provide feedback for me to improve the next iteration of the course.

A classic element of test analysis compares the errors of top students to the errors of bottom students. How hard would it be for you (or a TA) to parse out where the points were lost for each question, particularly for the top 10 and bottom 10 students? That might be a shortcut to find the difference between simple unpreparedness and poor question-writing.

Answer 3

The comment of Mon Kee Poo raises an important point: The actual distributions of the points per item might offer some more insight than just the median and the standard deviation.

For example, if there is a 'hit or miss' question, the points will not be normally distributed, but rather bimodal, i.e. one portion of students will have almost 0 points and the other portion of students will have almost full points. Such situation can occur, for example, if a specific isolated technique is asked for, such as finding an integrating factor to solve a differential equation.

Having this in mind, the mean and the sd. might not be the best way to discribe your distributions. In particular, you don't see if the results are skewed to the right or skewed to the left (although you can calculate the nonparametric skew, which gives an idea of the skewness, but is imprecise).