How to assess an undergraduate mathematics program?

Question

Our university and department, like most, are required to "assess" our program annually. In the past our department has done this by e.g. picking one of the learning outcomes for our program and a final exam question in one or two classes that are related to that outcome, and coming up with a numerical rubric to evaluate performance on those exam questions. But this doesn't really tell us much that we didn't already know, and no matter how carefully chosen, one exam question can be misleading and not representative of an entire program.

What successful ways have other mathematics departments found to assess their undergraduate programs and determine whether learning outcomes are being achieved?

Answer 1

Two questions from Lynn Arthur Steen are perhaps not addressed frequently, but seem important to me:

(1) Are students learning how to use mathematics in other subjects? Do students recognize similar mathematical concepts and methods in different contexts?

(2) Do faculty take responsibility for the quality of students’ learning?

These are from "Asking the Right Questions," in Supporting Assessment in Undergraduate Mathematics. MAA Supporting link.

The MAA (Math Assoc Amer) also published Assessment Practices in Undergraduate Mathematics. MAA Practices link. The latter publication likely addresses your "What successful ways" question, but I am in no position to summarize this 292-page book.

Answer 2

I was put in charge of assessment for our community-college math department as of last year. One resource that was immensely useful was the UMass "Program-Based Review and Assessment" handbook (link). There's an example of a math department assessment plan on p. 28.

That said, the process highlighted there is exactly as you describe: score selected questions embedded on final exams. On the one hand, we're fortunate in the math discipline that assessment is so amenable to our existing testing practices; on the other, it does look a bit silly and redundant. The silver lining for me is to know about specific skill mastery and not overall test or course averages.

The one thing that mystifies me is the practice of checking only one objective per year; that doesn't seem like the intent to me (although the UMass document, for example, does say "if you and other members of your department can agree on only one goal, don’t let this stall your progress. Focus on that one goal – more will come later." [p. 11]). Assessing just one goal per year seems like you'd be going a half-decade between pings on any objective to know whether they'd degraded or improved or whatnot, which seems crazy.

What I'm doing is to establish fixed questions in each of our terminal courses, and so assess every one of our objectives every year in a consistent manner. Hopefully this actually leads to less effort, because we won't have to spend mental energy creating new plans (re-inventing the wheel) every year; will also provide a "dashboard" of information on critical skills to compare year-to-year; and serves as a resolution to your very valid "one exam question can be misleading and not representative of an entire program" issue. Note also this is just the terminal courses (last ones to complete the degree), which I think meets the spirit of program assessment (what do students know when they complete our time with us), and also conveniently has a relatively smaller number of sections, and students, and instructors to wrangle for the task.

As extra steps I'm also receiving prerequisite grades and academic majors for all our students (to perform cross-correlations from the prerequisite course to identify particular strength or weakness), and sending out a survey to alumni this month for observations, how they did afterward, things they wish they were improved, and perceived strength on each of our program objectives. But that's definitely extra-credit for this task (I was expressly told by our Dean of assessment not to do that because it would be too much work), and can certainly be skipped if you just want to minimally meet accreditation demands. However, I did want to add a bit more depth/texture to our understanding of how our students are doing in later academic programs.

Answer 3

Honestly, if you really want to see how your program is doing you need to rely on some external testing. Perhaps the major field assessment test (MFT), or, the subject math GRE. These are a start, but, I don't think they quite uncover the higher end of undergraduate learning. For example, I took courses on manifold theory, general relativity, fiber bundles, axiomatic classical mechanics and a few other things. None of that would have even registered a blip in those tests, yet, I can tell you to a certainty they were huge in my overall education. Often these assessments are about the bulk of the students, but, what really speaks to the quality of a program is not average student's progress. In my estimation, the bigger question, what are you offering the best students? What are you offering the students whose heart is more with the study of mathematics than the acquisition of a credential? I'm not sure I know how to "assess" that, but, I know it matters.

Answer 4

You declare a number of learning outcomes already (for the program and presumably for the courses too). The learning outcomes for each course should be assessed by the grading (could e.g. track them by looking at the grades in individual questions of midterms/finals, check some other outcomes by results in term papers or such).