I am interested in looking at any design resources or "guiding principles" on the distribution of different types of question difficulties on evaluative examinations.

We can use Item Response Theory and other data analysis techniques to get a fairly good idea of how discriminatory and difficult questions on our test are, so I was curious in moving a step forward into test design: given a set of questions with different discrimination indexes and difficulty indexes what should the distribution of questions on an actual exam be? Should you just go for highest discrimination and call it a day? This would seem to ignore "A"-maker and "F"-maker questions that have high discrimination on one end of the spectrum, possibly giving you more granular data about the students.

Does anyone know of any research on test design that addresses distribution of difficult and or discrimination on exams? Or even if there is no concrete research any documents outlining guiding principles?

Edit: I've done a literature search and so I'm going to edit this question with what I've found. I hope that this is a useful jumping off point for anyone trying to understand modern testing theory.

TLDR: Read “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development” by Hambleton and Jones and look at Evidence Centered Design.

Most modern testing theory comes from psychology, where tests are the primary tool of research. There are two main schools of thought on modern testing metrics: Classical Test Theory (CTT) and Item Response Theory (IRT). The goal of CTT is to create reliable tests (ie tests with reproducible results) while the goal of IRT is to create reliable items (questions) by understanding their relationship to an underlying measure of aptitude called ability. These theories have substantively different definitions of discrimination and difficulty. CTT:

  • Discrimination: Pearson correlation between item and total test score
  • Difficulty: Percentage of students who answered question correctly.

IRT: In IRT, a function called an Item Response Function (IRF) $P_i(\theta)$ is fitted to the $i$'th question. This function takes a number $\theta$ called “ability” which is a measure of a students knowledge collected from assignments, interviews, etc. The function returns the percentage chance that a student will get the question correct. Since a student with low ability should have a low percentage change of answering a question correctly and a student with high ability should have high chance, this function (for dichotomous data) should interpolate between 0 at the low end and 1 at the high end.

The IRF may depend on many other variables, for a great rundown of different IRF's and their uses see “A Visual Guide to Item Response Theory” by Ivailo Partchev. For IRT,

  • Discrimination: The maximum slope of the IRF.
  • Difficulty: The ability level at which a student has a .5 chance of answering correctly.

Note: These metrics can be very different between CTT and IRT. For a description and comparison of the models and their uses see “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development” (http://psycnet.apa.org/record/1994-08136-001) by Hambleton and Jones.

So now the question: given a set of questions we want to make a test. How should we proceed and what effected does Difficulty and Discrimination have? From Hambleton and Jones and others we have the following criteria:

(CTT) High discrimination, difficulty right in the middle. This is to maximize variance giving the final score more meaning. This is also thought to increase reliability, as was partially vindicated in “The relationship between the distribution of item difficulties and test reliability” by Feldt, although he also showed that while restricting item difficult to the middle of the spectrum did increase reliability, it did so almost imperceptibly.

(IRT) In general, high discrimination. However, IRT has a device called an Item Information Function which allows you to roughly view the amount of information your test will gather about students at different ability levels. You can add in items to tune this function to test an ability range you are interested in.

(ECD) The third model was developed by ETS and is called Evidence Centered Design. For their introduction see their research report. The idea is that a test should measure competency in certain areas, and that you as the administrator want to prove on some level that is it. ECD used IRT to gives guidelines to construct and evaluate test items that measure student ability on specific topics in a verifiable way. Although it requires multiple rounds of testing using the same items.

Conclusion: Are you trying to

  • Maximize meaningfulness of the score? Use Classical Test Theory, questions in middle difficulty (.3-.8) and high discrimination. This maximizes variance in the score while preserving reliability.

  • Tests discriminating between certain ability levels? Use Item Response Theory, matching your test items to the desired Test Information Function.

  • Are you trying to test specific knowledge, accurately across time, while allowing yourself to reuse questions on different populations? Use the Evidence Centered Design process to provide evidence that test actually evaluates desired area of knowledge.

While all of the above is useful to understand how one thinks about test construction, I have yet to find any guidelines specific to mathematics. Although that might be fodder for another question here.

If anyone is interested in digging deeper and answering more specific questions, the best place to find articles are

Articles mentioned here:

  • 1
    $\begingroup$ The guiding principles you're seeking would depend on the objective of your assessment. For example, is your objective to assess students' learning over a period of time, or identifying the smartest student in the class? $\endgroup$
    – yathish
    Commented Dec 5, 2017 at 5:30
  • $\begingroup$ That's a really good point, at this point I'd take any and all resources on guiding principles for any purpose, but specifically I'm looking at single use, non-progressive testing, like a final exam. The goal being less the differentiation of students and more the assigning of a meaning number that describes their knowledge of the material. $\endgroup$
    – Nate Bade
    Commented Dec 7, 2017 at 16:32
  • $\begingroup$ This is an awesome question in the amount of work already put into it. $\endgroup$
    – guest
    Commented Dec 8, 2017 at 20:39
  • $\begingroup$ No direct answer on the specific question, but I do find value in tests that do more than classify the population into fractions but that also have some thought behind them in terms of what level is needed. IOW, a bit of a "bar" for performance rather than just curving. $\endgroup$
    – guest
    Commented Dec 8, 2017 at 20:43
  • $\begingroup$ Thank you, I was putting together a talk and thought that what I found shound't go to waste. I'm really interested in how to design tests that both give specific information to students (and myself) about what both they and I know, and give them a grade the actually reflects their knowledge and ability. The latter is a necessary evil but there it is. $\endgroup$
    – Nate Bade
    Commented Dec 8, 2017 at 21:26


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.