4
$\begingroup$

For an online assignment application where you can configure different versions of a single assignment/test to have different questions, should it be possible to also have the different versions have different number of questions?

Details:

We have an online assignment/test application that lets you make assignments with different versions. These versions can have different numeric values in the questions and can even have different questions between them (chosen from a common pool of questions for that assignment).

Is it also reasonable to have different numbers of questions in the different versions? Or should the number of questions be enforced to be the same across all versions of a particular assignment/test?

$\endgroup$
  • $\begingroup$ Note: The application we have is available for use but the purpose of this SE question is to help determine the best way for the application to work. $\endgroup$ – Joel Boulet Apr 28 at 16:12
  • $\begingroup$ Questions could have different weights, so as long as the total weight is the same, the number of questions would not matter to me. For example, I would be completely fine if version 1 had five questions worth 4 each (totaling 20) and version 2 had eight questions, each worth 2 or 3 (also totaling 20). $\endgroup$ – Andrew Chin Apr 28 at 16:58
6
$\begingroup$

When I write an assessment, I am trying to understand how well the students have mastered some particular body of material. In principle, a "perfect designer" could write multiple assessments with wildly different questions or numbers of questions, and consistently obtain valid results.

However, I am not a perfect designer—nor, frankly, is anyone else. Therefore when I design assessments, one of the things that I do in order to ensure that the assessment will be consistent (at least) is to ask every student the same questions. If there is some systemic flaw in my design, at least it will have the same effect on all of the students, and can be (hopefully) corrected for.

Of course, there are other practical concerns which sometimes compel instructors to write multiple assessments for the same material. The most common reason, I suspect, is to prevent cheating, and it is my assumption that this is the rationale which motivates the question. In this situation, the usual practice is to write multiple versions of the same assessment so that students cannot simply copy off of each other. In this setting, you have already given up on 100% consistency. However, there are still things which can be made to reduce the chances of errors biasing the assessment in favor of (or against) students who take any particular version of the assessment. The chief technique is to write one exam, then modify questions by, for example, changing some of the constants. Asking completely different questions may also prevent cheating, but it runs the risk of interfering with the consistency of the assessment.

Thus, after all of that, my short answer is no, I do not think that one should create different versions of assessments with variable numbers of questions.

That being said, I think that there are questions which one can ask to arrive at an answer of their own:

  1. What is the goal of the assessment? Is it formative (designed to check in on student progress toward understanding) or summative (designed to determine whether or not the students have mastered the material and assign a grade)? If the assessment is summative, then I would suggest that consistency is much more important—the stakes are higher for the students, and the risk of constructing an inconsistent assessment is commensurably higher. On the other hand, if the assessment is summative, it might make sense to give wildly different assessments, on the principle that you are polling or sampling the class in some way, and that the students taking one version of the exam are representative of the whole class. This might be a good way to catch errors in teaching without making any one student feel overwhelmed.

  2. What is the goal of writing multiple versions of the assessment? If the goal is to prevent cheating, then it seems to me that consistency is desirable. As such, it seem like poor practice to give wildly different versions of an assessment to different students. On the other hand, maybe the assessment is something that students will be building on in the future in some kind of group activity, e.g. you give two very different assessments, then ask students to work in groups to share solutions to these different versions.

  3. What is lost by writing multiple versions? This relates some to the first two questions. How much time are you going to lose (both writing the assessment and grading it later)? Will you lose consistency? validity? Are you potentially introducing errors in one version of the assessment and not the other? do these errors matter? etc.

  4. What is gained by writing multiple versions? Again, consider this in the context of the answers to the first two questions. Will you really prevent any cheating? Are you really concerned that cheating is going to take place (in a precalculus class where most of the students are non-majors and are only taking the class because they are required to, cheating might be a big problem; in an upper division class for mathematics majors, cheating is likely much less prevalent). Are you exposing students to a wider range of material?

| improve this answer | |
$\endgroup$
1
$\begingroup$

I took the computerized GRE 25 years ago and it had different numbers of questions because it used an algorithm to hone in on how strong you were. Basically it would keep pecking at you until it honed in on a score, then stop. So, sure "it's possible". Even "has been done". And a generation ago.

[But I'm sure a lit search (gentle hint/nudge) would show examples of variable length testing from much older dates, even pre-computers. It's not like it's something that is hard to imagine.]

But like Xander, I think you are not asking the question properly if you don't describe more what is the purpose of your application (for assessment testing, grading, learning or some mix of those). As this may influence how you decide to construct the thing.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.