Skip to main content
added 219 characters in body
Source Link
mweiss
  • 17.4k
  • 1
  • 42
  • 89

I don't know that there are any standardized measures that work by analyzing the questions themselves (I doubt it) but there are a number of statistical approaches that work by analyzing the test-takers' performance. For example, depending on the size of the shifts one could look to see if there are statistically significant differences in mean score or mean time to complete. (You did not indicate whether the shifts are all taught by the same instructors; if not, then it will be hard to argue that the differences in outcomes should be ascribed to the test rather than to the teacher.)

You could also pre-evaluate the different versions by gathering a sample of people who successfully took the exam the previous year, randomly assign each them 1 of the 4 different versions of the exam, and look to see if there are statistically significant differences in time to complete or score. If it is possible to design the sample so that everybody in it took the same version of the exam the previous year, you could also correct based on past performance.

You may want to look at Wikipedia's articles on on Psychometrics and Item Response Theory.

I don't know that there are any standardized measures that work by analyzing the questions themselves (I doubt it) but there are a number of statistical approaches that work by analyzing the test-takers' performance. For example, depending on the size of the shifts one could look to see if there are statistically significant differences in mean score or mean time to complete. (You did not indicate whether the shifts are all taught by the same instructors; if not, then it will be hard to argue that the differences in outcomes should be ascribed to the test rather than to the teacher.)

You could also pre-evaluate the different versions by gathering a sample of people who successfully took the exam the previous year, randomly assign each them 1 of the 4 different versions of the exam, and look to see if there are statistically significant differences in time to complete or score. If it is possible to design the sample so that everybody in it took the same version of the exam the previous year, you could also correct based on past performance.

I don't know that there are any standardized measures that work by analyzing the questions themselves (I doubt it) but there are a number of statistical approaches that work by analyzing the test-takers' performance. For example, depending on the size of the shifts one could look to see if there are statistically significant differences in mean score or mean time to complete. (You did not indicate whether the shifts are all taught by the same instructors; if not, then it will be hard to argue that the differences in outcomes should be ascribed to the test rather than to the teacher.)

You could also pre-evaluate the different versions by gathering a sample of people who successfully took the exam the previous year, randomly assign each them 1 of the 4 different versions of the exam, and look to see if there are statistically significant differences in time to complete or score. If it is possible to design the sample so that everybody in it took the same version of the exam the previous year, you could also correct based on past performance.

You may want to look at Wikipedia's articles on on Psychometrics and Item Response Theory.

Source Link
mweiss
  • 17.4k
  • 1
  • 42
  • 89

I don't know that there are any standardized measures that work by analyzing the questions themselves (I doubt it) but there are a number of statistical approaches that work by analyzing the test-takers' performance. For example, depending on the size of the shifts one could look to see if there are statistically significant differences in mean score or mean time to complete. (You did not indicate whether the shifts are all taught by the same instructors; if not, then it will be hard to argue that the differences in outcomes should be ascribed to the test rather than to the teacher.)

You could also pre-evaluate the different versions by gathering a sample of people who successfully took the exam the previous year, randomly assign each them 1 of the 4 different versions of the exam, and look to see if there are statistically significant differences in time to complete or score. If it is possible to design the sample so that everybody in it took the same version of the exam the previous year, you could also correct based on past performance.