What is a fair way of constructing exams with tiered levels of difficulty?

Question

I would like to construct an exam that has C-level, B-level, and A-level questions.

I would like a student who answers all C-level questions correctly a passing grade, a student who answers all C and B-level questions a B grade, and a student who answers all questions correctly an A-grade.

However, I would also like students to be able to combine problems; for instance, those who answer 70-80% of the C-questions and 1-2 B questions should still pass.

What is a fair way of constructing such an exam? I'm primarily interested in answers drawn from experience. Edit: By fair, I mean that a student that really doesn't understand the material won't automatically get a high grade, and that a student who knows the material well and slips up one one or two problems won't end up with a very low grade.

My ideas so far: Have, say, 6 problems of each type. To get a C, you must get 6 questions right from any of the three categories. Similarly, to get a B you must get 6 questions right out of the A and B categories. To get an A, you would need to get all 6 A questions right.

Answer 1

There is at least one way to make this question more reconciliable with reality if the implicit question is partly about how to make the students less stressed... by more-clearly delineating what their grade(s) will be at a given level of effort. (On the whole, it's a fine impulse to want to implement this, I think.)

However, I very-strongly suspect that it will mostly not be received as such, but as merely a more-game-able system by being more laid-open. And then you yourself have the increased moral burden of figuring out what to do with the students who're less able to, or less inclined to, try to game the system.

An entirely analogous problem, that puts too much burden on students' larger sensibilities, is the common situation that homework does not contribute to the grade, but "is recommended". Most students misjudge the situation, to their own detriment, but not intentionally!

Instead, for your issue, more to the point would be to give an estimation of the "difficulty" of the various questions, if you are confident of your appraisals of it/them, to literally give information to your students... rather than giving a little information, but "in return" requiring additional exercise of judgement from them. Still, if you have a plan about scaled grading of more-difficult questions... you should be upfront about it... but, again, will students be able to gauge things properly? Especially the weaker students? Requiring additional exercise-of-judgement is a further test, after all.

In summary, in a course where any significant fraction of the population is near "failure", such a strategy would likely amplify the failings of the weaker people, confuse the middle-ground, etc.

Answer 2

Any way of breaking up questions and then assigning point values is going to result in the ability to break it.

If you go the route you mention (answer 6, minimum numbers for a grade), how will you deal with a student who answers the 6, but doesn't answer all of them correctly? Do they now get a C for attempting the harder ones? Is it an F because you can't tell? If you start assigning point values and then just adding them up, there is no difference than any other method.

I'd suggest that you make 3 questions. One that you'll call "A" level, one you'll call "B" level, and one you'll call "C" level. They can choose one to work through. If they get it correct (that means completely correct work or with minor algebraic/labeling errors at most) then they get the grade that they attempted. If they get it wrong (that means correct method but problem-breaking levels of misconceptions) then they get a "D". If they can't even attempt the problem correctly (that means blank or no work shown) give them an "F".

What reason do you have for giving them 6 questions at each level if you just want to see what they know?

Answer 3

I once took a Topology course at the University of Michigan in which the exam questions were constructed in roughly the form of a menu: some questions were worth 10 points, some were worth 20, etc. There were roughly 160 points worth of questions available, and students were instructed to answer any combination of questions that added up to a maximum of 100 points. So, for example, a student who wanted to mainly stick to routine exercises could answer eight 10-point problems and one 20-point problem; a student who wanted to really challenge him or herself could answer two 40-point problems and one 20-point problem. (These are just examples; I don't recall the exact configuration of options.)

Many years later, when I taught a Math for Elementary Teachers course, I adapted the scheme, but I found that virtually without exception every single student chose to answer only low-value questions, so after a couple of years I abandoned the model, as I was putting a lot of effort into designing the exams and not getting much out of it.

Answer 4

The exam construction strategy which I have been using recently, and which seems to work pretty well, is as follows:

To make the numbers easy, suppose that I want to write a 10 question exam, where a student who gets 90% or more of the possible points gets an A, 80-90% a B, 60-80% a C, and anything less is D/F territory (obviously, this can be scaled up to make longer exams). This exam will consist of four categories:

• Passing level questions: on the exam described above, 6 of the 10 questions will be in this section. These questions represent the bare minimum of material that needs to be mastered in order to pass the class at the lowest possible level. These questions typically test recall and very simple computations---students should not be required to show any work here, and should be able answer these questions in less than 30 seconds---it really should be just a matter of writing down the answer. On a 10 question exam, the whole section should take less than 5 minutes. In a recent business calculus exam, such questions included

  • Determine $\frac{\mathrm{d}}{\mathrm{d}x} \pi^2$.
  • Complete the statement of the product rule: if $f$ and $g$ are differentiable functions, then $(fg)'(x) = $___.

• C level questions: the next 2 questions are in this section. These questions are typically very computational in nature, and relatively straight-forward and mechanical, but might require a bit more work and should depend on more than just recall. These questions can also be taken word-for-word from the homework assignments (maybe fudging some numbers here and there). I expect a typical student to be able to answer one of these questions in about 5 minutes (or, hopefully, less). These are applications of ideas in relatively simple settings. For example

  • Find $\frac{\mathrm{d}}{\mathrm{d}x} 3\sqrt{9-x^2}.$
  • Alice paints and sells her own paintings. Each painting that she produces requires \$150 in supplies, and can be sold for \$300. If she has fixed costs of \$1,350 per month, how many paintings must she sell each month in order to break even?

• B level questions: on this 10 question exam, there is only one B level question. The goal of a B level question is, in general, to go a little bit beyond just the mechanics and see whether or not a student can do more than one step at a time. These questions ask students to link several ideas to solve a single problem, without any extra scaffolding given. For example:

  • Use the limit definition of the derivative to compute $\frac{\mathrm{d}}{\mathrm{d}x} \frac{1}{\sqrt{x}}$.

  I expect a B level question to take about 10 minutes to complete.

  They might know this from the power rule, but would not have seen this particular derivative computed "by hand" in class. The question requires them to know a definition, to be able to apply that definition, and to actually compute a limit. Another example (from a precalc class):

  • Find an equation for a line tangent to the graph of $2x^3 + 2$ which passes through the point $(1,4)$.

  This is actually an appropriate B level question in the precalc class (it would be a C level question in a calc class), as they have been given an algebraic notion of tangency, but no derivative function. Hence they have to remember that definition, apply it to the function, and work through the computations.

• A level questions: on a 10 question exam, there would be one of these. An A level question is meant to push the students to use what they know in a novel situation. I don't actually expect many students to give a complete answer to an A level question, and I expect that the ones that can answer these questions will spend 15-20 minutes on each of them. The goal is to get the students to show how they think about hard problems. A level questions might require knowing multiple definitions and applying them to a novel type of problem. There may be some computation in such a question, but the real impediment to answering should be conceptual rather than computational. For example

  • Suppose that $$f(x) = \begin{cases} 5x + 4 & \text{if $x \le 0$, and} \\ ax^2 + bx + 7 & \text{if $x > 0$.} \end{cases}$$ Are there values of $a$ and $b$ such that $f$ is differentiable at zero? If so, find them. If not, explain why not.
  • (For a precalc class:) Suppose that $x$ is a real number, and let $\lceil x \rceil$ denote the ceiling of $x$. That is, $\lceil x \rceil$ is the smallest integer that is greater than $x$, or what we get when we "round $x$ up" to the next integer. For example, $\lceil -1.5 \rceil = -1$ and $\lceil \pi \rceil = 4$. Note that integers "round up" to themselves, e.g. $\lceil -2 \rceil = -2$. Let $f:\mathbb R\to\mathbb R$ be defined by the formula \begin{equation*} f(x) = \lceil x \rceil - x. \end{equation*} For what values of $p$ is $f$ $p$-periodic? Does $f$ have a fundamental period? If so, what is it?

Every question is worth exactly the same quantity of points. So the A level question which requires 20 minutes to answer is just as "valuable" as the passing level question which can be dashed off in 10 seconds.

Students seem responsive to this scheme. They really like to know exactly how much effort they have to put into something in order to get the result that they want, and they really like to know where they stand. I've tried other schema where, for example, students choose some number of problems to complete; or where there are hard cutoffs for the number of questions at a given level which must be answered in order to earn a particular grade; or where questions at different levels are weighted differently, but these seem to confuse students. Students also really like percentage-based grading scales---every time I have ever tried to move away from such scales, I've been creamed in evaluations (not that one should base their teaching on getting good numerical evaluations, but this doesn't seem like a good hill to die on).

On the instructor side of things, I don't really think that these exams are "game-able" to any significant degree. You can't get the A without making significant progress on the A level question(s). They are also not hard to grade, since there is no choice involved on the part of the students, and one doesn't have to go back and forth between pages of the exam to count the number of questions answered in each section. You also don't lose time explaining the structure of the exam to the students---it is an exam like any other that they have ever seen (just with sections labeled "passing", "C", "B", and "A").

Answer 5

I don't think this is really needed/optimal for the following rationale. In general, you are covering a few select topics and student competency varies on those topics. The bell curve will sort itself out and the general theme of someone getting 70% of the easy right and 30% of the hard right will be more common than the opposite: students getting all difficulty at easy, zero at harder.

Instead I would just think of the exam construction this way. Normal books for classes from grade school through diffyQs tend to have problems at the I/II/III levels. I is complete plug and chug. II is slightly harder, some manipulation. III can be challenging (at different levels of challenging but still not as easy).

I would just look at all the topics you covered (say 5 lessons from a MWF class, 2 weeks with test on last F). Design a 50 minute exam to cover the 5 topics. It is probable 5 II problems (one for each section) as well as a smattering of I problems, perhaps to hit subtopics in the lessons, to the extent time allows.

If you want to assign one III problem, do it at the end of the exam (on any of the topics). The purpose of that is mostly student differentiation (B versus A or +/- shades). It can be required or extra credit as you think best (if timing tight or students struggling to reach reasonable percentage of 80%/90% cuttofs, make it extra credit).

I do think it is reasonable to give different points for the II and I problems. Just do it based on rough time required. As a ballpark, 2:1, but use your judgment and keep it simple.

The III problem can probably be 3 in 3:2:1. (15, 10, 5 is easy point math for students under the gun.) Given how this drives the % cuttoffs, it is another argument for making that problem EC.

[When I say I/II/III, I mean probably very close analogs of the problems, but you can even consider to include unassigned HW problems that are right out of the book, especially if the drill materials are lengthy.]

Avoid the temptation to get cute and assign problems different in form from the homework or requiring a lot of synthesis of various skills. Test what was in the lessons! That is hard enough, usually and the students have accomplished a lot if they master that. If you can't resist the temptation to get cute, creative, and show how smart you are, then do it with the III problem at the end (and probably DO make it extra credit).

In terms of A/B/C differentiation, it will handle itself fine accross the questions. Getting 90% right of a bunch of II problems with a few Is mixed in is non trivial in a tested 50 minute environment, with no reference materials. Kid probably deserves an A for that.