# How can I learn to write better questions to test for conceptual understanding?

I'm worried that I'm bad at realizing when a question I've written requires little or no conceptual understanding to answer. Like, when I'm writing a question for a homework assignment or exam, I'll be thinking of it conceptually because that's how I've learned mathematics. But I don't know how to ensure that the question demands that a student think conceptually. Here's a silly minimal example calculus question to illustrate what I mean:

Suppose that a truck's distance from you in meters at a time $$t$$ seconds after the big bang is given by the function $$p(t) = \sin(t)+42 + \mathrm{e}^t$$. What is the velocity of the truck $$t=19$$?

I wrote this question thinking that you need to understand the whole "velocity is the derivative of position" idea to answer. But really a student could answer this question by noticing that it has a function $$p$$ in it, take that function's derivative because that's the only thing you do to functions in an differential calculus class, and then plug in $$t=19$$ because why else would that $$t=19$$ be there.

How can I learn to write questions that require a student to think conceptually/deeply to answer?

Or if you want to ask this question from a pessimistic/adversarial perspective, how can I learn to write questions that'll thwart some learners' habits of cramming immediately before tests or exams so they can robotically reproduce what they've crammed, without ever really having understood it?

I'm interested in resources that address this, but I'm not sure such resources exist, so sage advice is very welcome. :)

• The answers in this MathEdSE post have a bunch of good examples of conceptual calculus question. – Mike Pierce Oct 7 at 3:35
• You could just outright ask the students to explain something instead of doing it. For example, for the problem you mentioned, instead ask: "Suppose I have a function p(t) that outputs the distance from me to an object, with t measured in seconds and p measured in meters. What does p'(19) represent? Explain in a few sentences as if you were teaching a classmate this concept." – Brendan W. Sullivan Oct 7 at 16:54
• Time might act differently near the time of the big bang. I'm guessing you thought that was amusing. But I would worry that the big bang had something to do with the problem. I advise leaving distractions like that out. – Sue VanHattum Oct 7 at 18:38
• A related question is how to write such questions without using most of your available free time in the process. – kcrisman Oct 9 at 16:42
• @kcrisman Right!? I takes a long time to write good questions. That's one reason I was hoping for resources or documentation on how to write better examination questions; if there are resources then it's easier to learn to write good questions as a skill, and to teach others to do it, so it takes less time. – Mike Pierce Oct 9 at 16:53

Agreeing with comments and other posts: If you want more conceptual answers, give them less details in the set-up.

Using your velocity problem, here are a couple of examples of making it more conceptual:

• Suppose that a truck's distance from you in meters at a time $$t$$ seconds after the big bang is given by the function $$p(t)$$. What does $$p'(19)$$ tell you (and what are its units)?
• Suppose that a truck's distance from you in meters at a time $$t$$ seconds after the big bang is given by the function $$p(t)$$. What does it mean if $$p'(a)<0$$?

In the absence of details, the students cannot just 'plug and chug' naively and are forced to actually think about the pertinent concepts.

• +1 for asking for the units! Minor nitpick: It should be "given by the function value $p(t)$" in both points. And where does the $a$ in $p'(a) < 0$ come from? – ComFreek Oct 8 at 8:42
• @ComFreek Wording like "Given by the function f(x)" is completely ordinary; a quick Google reveals that "given by the function" is a vastly more common than "given by the function value". What's your object to the former? As for the point about a, isn't it sufficient to leave it implicit that it's just some arbitrary value? If not, why not? What sort of explanation of that do you think is warranted, and how would it add clarity? – Mark Amery Oct 8 at 10:23
• @MarkAmery Most results for that search term are in a different context than the one used in the answer. Here, it's said that the "truck's distance [...] $t$ seconds after the big bang is given by the function $p(t)$". That's wrong, since the concrete distance after $t$ seconds is given by the value the function attains at $t$. Generally, I try to distinguish between the function as such and a specific value very much. I don't think mixing those concepts is beneficial for beginners even if some senior authors still do this, unfortunately. – ComFreek Oct 8 at 13:31
• @MarkAmery If I see unbound variables in a text, I immediately think "did I overlook the introduction of this variable somehow?". If it's arbitrary, I would specify that - especially in an exam, e.g. "Let $a \in [0, \infty)$ be some arbitrary time position. If $p'(a) < 0$, what could you say about the truck?". (As done in the other answer, I'd introduce the function type at the very beginning.) – ComFreek Oct 8 at 13:37
• @ComFreek Distinguishing the function and its value is worth doing, yes. But you seem not to be distinguishing the phrase "given by" from the phrase "equal to". Your objection only makes sense if we insist that technically Aeryk's words must mean that the distance is the function itself, rather than its value... but the common usage of the phrasing "given by the function" in a way that is inconsistent with this suggests that this is frequently not, in fact, what "given by" means. What basis do you have to suggest that this usage - more common, I think, than yours - is incorrect? – Mark Amery Oct 8 at 13:51

Asking students to explain why something happens can be useful for assessing understanding, although it is often harder to grade and works best with many demonstrations before the exam. (Students need to know what your expectations for a thorough explanation are.) I have found that asking students to critique a process will sometimes help me assess their conceptual understanding. I give them a sample problem and solution, and ask them to analyze the work. As an example, after introducing the indefinite integral but before discussing techniques like $$u$$-substitution I will give my students the following problem.

• A student is trying to find the following antiderivative $$\int xe^{x^2}\;dx$$ and comes up with $$\int xe^{x^2}\;dx = 2e^{x^2}+C.$$ Is the student's work correct? Convince me either way.

Answering this correctly requires students to (1) understand that differentiation of the antiderivative should result in the original integrand, (2) differentiate correctly, and (3) state a conclusion. Simply finding the derivative of $$2e^{x^2}+C$$ isn't enough.

I also second Aeryk's suggestions, and would add that asking about units can be a good way to go, such as

• If $$r(t)$$ represents the rate in grams per minute at which a bacterial colony is growing at time $$t$$, what are the units of $$\int_2^7 r(t)\;dt$$?
• "If $r(t)$ represents the rate in grams per minute" Does that mean that $r(t)$ is unitless? If not, what is the purpose of specifying the units of $r$, shouldn't you say "the rate in mass per time"? – JiK Oct 8 at 16:24

I agree with @BrendanW.Sullivan's comment. That is, when teaching an undergraduate course, like calculus, students need more than procedural knowledge. For a deeper understanding, and efforts to evaluate such, students should be asked on exams to answer a few "free form" questions, like the one Brendan suggested.

A good question to ask following any question you ask of students in class, in questions you assign for homework, and/or in questions you construct for an exam: "How/Why does your solution make sense to you?"

Conceptual understanding refers to an integrated and functional grasp of mathematical ideas. Students with conceptual understanding know more than isolated facts and methods. They understand why a mathematical idea is important and the kinds of contexts in which is it useful, and why it is useful.

If you do the above regularly and consistently, your students will be more familiar questions to assess conceptual understanding in an exam.

Also see Assessing conceptual understanding in Mathematics, where you can download the full pdf.

• Sorry I can't ping you on the other site, and I'd like to ask your opinion on this. In chat I complained that it is pedagogically nonsensical even if we disregard the basic error that I stated in my comment, but inexplicably some professional mathematician cannot seem to grasp the fact that most students would be misled by such ill-defined 'definition' of 0 in terms of subtraction of naturals. And almost surely 3 people in that room (not just him) upvoted that post. Is there a convoluted way they are reading the post to make it correct? – user21820 Nov 25 at 4:16
• I didn't intend to offend you in my comments on the recent MathEd post nor in the above one. If you're referring to the above one, I'm sorry to disagree with you again. I understand my teaching experience may differ from yours, but my remarks are based on my experience. I admit I'm a bit surprised at your tone in your recent comments, but maybe it's just my misinterpretation. – user21820 Nov 30 at 21:41
• The most sound practices in math ed are not grounded in one person's "teaching experience", @user21820. They are based on research, rather than one person's subjective impression. – Namaste Nov 30 at 22:44
• @user21820 I have deleted my last comment, because I'd have rather expressed it to you individually, and not publicly. I agree that we only very occasionally disagree. – Namaste Dec 2 at 16:09
• Okay I've deleted my last comment as well. I definitely agree that the best pedagogical practices are based on rigorous research and not one opinion, but I simply have strong convictions, at least about what works for those I have taught. In any case, I respect your viewpoint, and I think it would also be nice if one day we could discuss mathematics and pedagogy in person. I don't profess to be good at online communication. =) – user21820 Dec 2 at 16:17

As stated in other comments, try not to "invite" your students to merely apply rules they do know. In the specific example you mention, I would prefer a multiple choice question of the form:

A truck's distance in metres from you as a function of time $$t$$ (in seconds) is given by a smooth function $$p:[0,+\infty)\to\mathbb{R}$$. Knowing that $$p(19)=12$$, $$p'(19)=24$$ and $$p''(19)=3$$, the velocity of the truck at $$t=19s$$ is:

1. 12 m/s,
2. 24 m/s,
3. 3 m/s?

Under such a framework, one cannot avoid some conceptual processing - or exhibit, at least, the absence of it.

In general, the more abstract the context of a question is, the more possible it is that a student invokes conceptual processing.

Also, when you invoke some context - e.g. motion - do it for a reason and not just so as to make the question more "fancy". I've made this mistake several times myself.

Consider, for instance, the following questions, which both refer to the same issue:

• Find an antiderivative $$F$$ of $$f(x)=4x+3$$ such that $$F(0)=3$$.

• Let $$f(t)=4t+3$$ describe the way a particle's acceleration changes over time. Is it possible that the particle has a zero velocity at some time? From all the antiderivatives of $$f$$, which one describes a particle that starts from inertia and constantly accelerates with positive acceleration?

The second involves much more in-depth knowledge of the relation between a function and its antiderivative than the first one.

• Your implementation of the question has a 1/3 chance of the student not exhibiting his lack of understanding. Why would you suggest multiple choice? – svavil Oct 8 at 8:03
• I'd suggest changing "given by a smooth function $p(t)$" to "given by a smooth function $p: \mathbb{R} \to \mathbb{R}$" since $p(t)$ usually denotes a specific value. (I know some senior authors still write $p(t)$ to actually mean the function $t \mapsto p(t)$, but I don't think students should be instructed to use this overloaded syntax.) – ComFreek Oct 8 at 8:47
• Yeap, that's right, I'll fix it, thanks for the recommendation! – Βασίλης Μάρκος Oct 8 at 10:43
• If we're asking about units, then one should be certain that the units in the question statement make sense. There's no mention of a metre in the definition of $p$, so how would the student know the answer is not $24$ furlongs/second. – JiK Oct 8 at 14:33
• @ΒασίληςΜάρκος thanks, I understand that students will still have to explain and expand their answer. Maybe open questions vs. multiple choice is just a stylistic choice then. – svavil Oct 8 at 18:28

What one has to do to test for conceptual understanding is hard to state in terms of general principles (although Polya's books on Plausible Reasoning do a pretty good job of addressing the issue) and maybe is best addressed via examples.

Here is one example. Consider a cubic polynomial in one variable that is increasing as a function of its argument. Choose one that is not easily factored. One could ask to find its critical points and decide their nature. Or one could ask how many real roots the polynomial has. The first question is mechanical. One has to know the definition of critical point and some test to decide the type of a critical point. To answer the second question one has to realize that for the indicated polynomial to have more than one root, it must have a local minimum at which the value is negative. Then one has to do the same mechanical operations as in the first question to decide whether this is the case. The way the problem is posed forces much more serious thinking. Usually one has to ask both sorts of questions because the student who cannot solve the first has no hope of solving the second.

Another example. Students can be asked to evaluate an integral whose evaluation can be effected by making a change of variables. An alternative, which both isolates the change of variables issue and avoids recipes is the following. Give them the value of an integral such as $$\int_{-\infty}^{\infty}e^{-x^{2}/2}\,dx$$ and ask them to find the value of $$\int_{-\infty}^{\infty}e^{-3x^{2}}\,dx$$ or $$\int_{0}^{\infty}e^{-x^{2}/2}\,dx$$ or $$\int_{0}^{\infty}e^{-3x^{2}}\,dx$$ (the last is slightly harder than the first two, combining the issues present in both).

I have often used variations of both of these examples (and there are many more). They are effective for discriminating among those who understand more and those who merely have mastered formal manipulations. On the other hand, a test full of questions like these will not yield good results in most classrooms in most universities (it would be too hard).

These sorts of questions force deeper thinking without requiring that the student make it explicit, something which is out of reach for all but the most interested and enthusiastic students. Understanding, at whatever level, has to be evaluated operationally. At an introductory level it rarely works to ask "explain why ..." When many students can answer this it is usually the case that they are parroting phrases repeated often by the teacher, and then not much is being learned.

If your assumption on how the students may get the right answer, maybe rephrase the question in a way where doing that is the wrong answer. For instance, you could ask for acceleration rather than velocity.

Still, I don't think many students will get the right answer by following the logic you mentiond without having an understanding on what's behind!

• Experience grading exams tells me that they do follow that logic. Like, grading a problem similar to the one you mention for example, I've been shocked at how many students respond by just taking the first derivative and plugging in the initial condition. – Mike Pierce Oct 7 at 15:44
• @MikePierce So... There you have a possible answer I guess! – David Oct 8 at 6:20

One way is to use simple, atypical mathematical objects, fringe cases, non-examples and counterexamples, etc.

• Can you add some examples to this answer? – Jasper Oct 7 at 20:31
• @Jasper Specifically, I meant the class of examples which some people refer to as pathological. They're replete. – Allawonder Oct 8 at 6:09
• If they are replete, it shouldn't be hard to provide some. Your answer isn't very helpful as it stands. – mephistolotl Oct 8 at 16:56
• I think examples such as $x^2 \sin(1/x)$ whose derivatives might have the intermediate value property but not be continuous is what is meant here. However, for a more basic course this wouldn't be particularly useful. – kcrisman Oct 9 at 16:41

The "deeper" or at least off-beaten path solution should be much quicker than a standard but boring approach to create a system of equations or calculate a derivative or something like that. Give two-three problems and limit time so that they can solve them in time if they apply some sort of a trick. You'll get a lot of angry students :)

Here is an middle-school example, no derivatives, it can be solved with a system of linear equations:

Every day an engineer arrives to a station at 8 a.m. by train. Exactly at the same time a car, sent from a factory, drives up to the station, picks up the engineer and takes him to the factory. One day the engineer arrived at 7 a.m., decided not to wait for the car, and started walking towards the car. When the car met the engineer, it picked him up, turned back and arrived to the factory 20 minutes earlier than usual. For how long did the engineer walk? Consider the speeds of the engineer and the car constant.

Or you can solve it literally in 30 seconds if you think about what is happening, and use 3rd grade arithmetic.

• There's an analogous type of exercise commonly asked in integral calculus. Something like "Verify that $\int f(x) \,\mathrm{d}x = F(x)$," for some specific $f$ and $F$. Taking an anti-derivative of $f$ might be really tough, whereas taking the derivative of $F$ is probably much easier. – Mike Pierce Oct 7 at 20:40
• I'm not sure how I feel about this sort of question though. Like, if you put such a question on an exam, then the students who answer the question "the long way" won't be penalized for points on that question, but instead will be penalized in terms of time, and so will be penalized on points on other questions that they don't have time for. Is it fair to assess the students this way? – Mike Pierce Oct 7 at 20:43
• @MikePierce I think it was George Polya who said that the goal of school math is to teach students to build equations out of word problems and then solve them (I don't know what he thought about college math). This is how I was trained, equations were my hammer that I would use for every nail. When I see elegant simple solutions to the problems like the above, I think that I missed something when I was at school. If anything, problems like this help thinking outside the box. Maybe do not give them on test, but discuss on a practice session. – Rusty Core Oct 7 at 21:13
• @Namaste I suggest re-reading what I wrote, in particular the part about me missing something when I was in school. This is one of the few cases when I consider my school education lacking ;-) – Rusty Core Oct 8 at 21:52
• As for the question, I must be missing something. Does the car get back to the factory at exactly eight too? – Allawonder Oct 9 at 18:32