I am interested in looking at any design resources or "guiding principles" on the distribution of different types of question difficulties on evaluative examinations.
We can use Item Response Theory and other data analysis techniques to get a fairly good idea of how discriminatory and difficult questions on our test are, so I was curious in moving a step forward into test design: given a set of questions with different discrimination indexes and difficulty indexes what should the distribution of questions on an actual exam be? Should you just go for highest discrimination and call it a day? This would seem to ignore "A"-maker and "F"-maker questions that have high discrimination on one end of the spectrum, possibly giving you more granular data about the students.
Does anyone know of any research on test design that addresses distribution of difficult and or discrimination on exams? Or even if there is no concrete research any documents outlining guiding principles?
Edit: I've done a literature search and so I'm going to edit this question with what I've found. I hope that this is a useful jumping off point for anyone trying to understand modern testing theory.
TLDR: Read “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development” by Hambleton and Jones and look at Evidence Centered Design.
Most modern testing theory comes from psychology, where tests are the primary tool of research. There are two main schools of thought on modern testing metrics: Classical Test Theory (CTT) and Item Response Theory (IRT). The goal of CTT is to create reliable tests (ie tests with reproducible results) while the goal of IRT is to create reliable items (questions) by understanding their relationship to an underlying measure of aptitude called ability. These theories have substantively different definitions of discrimination and difficulty. CTT:
- Discrimination: Pearson correlation between item and total test score
- Difficulty: Percentage of students who answered question correctly.
IRT: In IRT, a function called an Item Response Function (IRF) $P_i(\theta)$ is fitted to the $i$'th question. This function takes a number $\theta$ called “ability” which is a measure of a students knowledge collected from assignments, interviews, etc. The function returns the percentage chance that a student will get the question correct. Since a student with low ability should have a low percentage change of answering a question correctly and a student with high ability should have high chance, this function (for dichotomous data) should interpolate between 0 at the low end and 1 at the high end.
The IRF may depend on many other variables, for a great rundown of different IRF's and their uses see “A Visual Guide to Item Response Theory” by Ivailo Partchev. For IRT,
- Discrimination: The maximum slope of the IRF.
- Difficulty: The ability level at which a student has a .5 chance of answering correctly.
Note: These metrics can be very different between CTT and IRT. For a description and comparison of the models and their uses see “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development” (http://psycnet.apa.org/record/1994-08136-001) by Hambleton and Jones.
So now the question: given a set of questions we want to make a test. How should we proceed and what effected does Difficulty and Discrimination have? From Hambleton and Jones and others we have the following criteria:
(CTT) High discrimination, difficulty right in the middle. This is to maximize variance giving the final score more meaning. This is also thought to increase reliability, as was partially vindicated in “The relationship between the distribution of item difficulties and test reliability” by Feldt, although he also showed that while restricting item difficult to the middle of the spectrum did increase reliability, it did so almost imperceptibly.
(IRT) In general, high discrimination. However, IRT has a device called an Item Information Function which allows you to roughly view the amount of information your test will gather about students at different ability levels. You can add in items to tune this function to test an ability range you are interested in.
(ECD) The third model was developed by ETS and is called Evidence Centered Design. For their introduction see their research report. The idea is that a test should measure competency in certain areas, and that you as the administrator want to prove on some level that is it. ECD used IRT to gives guidelines to construct and evaluate test items that measure student ability on specific topics in a verifiable way. Although it requires multiple rounds of testing using the same items.
Conclusion: Are you trying to
Maximize meaningfulness of the score? Use Classical Test Theory, questions in middle difficulty (.3-.8) and high discrimination. This maximizes variance in the score while preserving reliability.
Tests discriminating between certain ability levels? Use Item Response Theory, matching your test items to the desired Test Information Function.
Are you trying to test specific knowledge, accurately across time, while allowing yourself to reuse questions on different populations? Use the Evidence Centered Design process to provide evidence that test actually evaluates desired area of knowledge.
While all of the above is useful to understand how one thinks about test construction, I have yet to find any guidelines specific to mathematics. Although that might be fodder for another question here.
If anyone is interested in digging deeper and answering more specific questions, the best place to find articles are
Articles mentioned here:
- First place to go to understand CTT and IRT methodology: Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development, Hambleton and Jones.
- A great introduction to the nuts and bolts of IRT: A Visual Guide to Item Response Theory, Ivailo Partchev.
- A review of CTT Test Design with empirical results: The relationship between the distribution of item difficulties and test reliability, Feldt.
- Introduction to Evidence Centered Design A Brief Introduction to
Evidence-centered Design , Mislevy, Almond, Lukas
- University of Washington's Guidelines on item level analysis