To which degree should exams test intelligence vs mastery of the material

Question

I think it happens to a lot of students that (at least it feels like) really understanding the material does not really contribute to having a decent grade. It seems to me that quite often exams are more a test of general intelligence than a test whether a student really mastered the material. Of course its usually the smarter students that also master the covered material more deeply than the rest, but still. It seems to miss the point of a final exam, which is to test whether the students mastered the material. How do you guys handle the problem of designing exams that avoid this pitfall? Moreover do you think that testing mastery of the material is the right goal to aim for?

EDIT: One example I could give are proofs that only require knowledge of a few basic operations that one is allowed to perform, but demand that one sees some sort of trick. Almost no (deep) understanding of the material is required and almost no advantage is gained from understanding the ideas behind the concepts. Someone smart without deep knowledge will easily outperform someone less talented who spent a lot of time understanding the concepts. But not only does it not reward understanding, but tricks that are really hard to see are heavily dependent on luck. I might see the trick one day but I won`t on another. Of course both factors are impossible to rule out completely but the might well be mitigated if the teacher would try. If exams are designed like the one I complain about, they put too much focus on the performence of students under exam situations while a lot of other qualities are neglected.

Take the following question:

Determine the radius of convergence for the following infinite sum:

$$\sum_{k=1}^\infty \frac{\pi^k}{k^{1/4}}(e-x)^k$$

One day I might do the right transformations and on another day I might not. But I am not sure whether questions like this test deep understanding of mathematical concepts.

On the other hand I think that there are also exam questions that are too easy. For example if one wants to test whether the students understood the jordan canonical form, I think it is a very poor test to simply require the students to calculate the matrix and cernels etc. . This can be done by any student who memorizes the required algorithm without understanding what it actually means.

Answer 1

I think majority of the questions on an exam should be pretty straight, not tricky. Remember all the material is new to the students so in that sense it is already a "trick". You can still have a few questions that are harder by doing ones that are complex in terms of manipulation or that combine two methods or that are word problems. But even that I would go easy on.

That is unless you are teaching a Putnam class or the like.

The smarter kids will still do better because they learn the material better and have better memories for the overall class (as usually more than one idea is tested).

But I think there is a bad tendency to want to be cute with tricky problems and then have to curve the exams. And not even really test well if the basic concepts are learned. This appeals to people who already know the material and want to do tricky stuff with it (most instructors) but is not sound pedagogy. It would be like an NFL coach teaching super complicated plays to a group of players who need to master the center-quarterback exchange.

Answer 2

Have a large portion of your test consist of written explanations of the concepts.

For example, here is a typical plug and chug exam problem:

"Consider the region bounded by $y=x^2+x+10$, $x=1$, $x=3$ and $y=2$. Find the volume of the solid obtained by revolving this region about the $x$-axis "

Many students will get this correct by memorizing a particular form of the solution. You could make it "tricky" by having them revolve about some line other than the $x$-axis (if they have not memorized a formula for that as well).

This same problem can be transformed into a test of conceptual understanding by adding the following:

"Give a sum (using sigma notation) which approximates the volume of this solid by using $N$ slices. Draw a picture of an arbitrary slice and explain why each part of your formula makes sense. Use your calculator to evaluate this sum for $N=10$, $N=100$, and $N=1000$. Finally, write the limit of this sum as a definite integral and evaluate using the Fundamental Theorem of Calculus. Does this exact answer agree with your approximations?"

Certainly, you cannot include as many questions on a test if this is the kind of depth you are looking for. However, it doesn't require exceptional intelligence: just hard work and dedication to understanding the concepts involved.