Practical ways to check if a student has mastered a particular topic

Question

I've been thinking that in an ideal world math classes would be self-paced, with a student being required to demonstrate mastery in a collection of topics and techniques in order to complete the course. Some students might complete the course much faster than others, and the slower students would not be forced to try to learn more advanced material when they are still struggling with the basics.

The difficulty with implementing this idea is: how do you check whether the student has mastered a given topic? They could take an exam, but if they don't do well on the exam what then? I suppose then they would take a different exam which covers the same topics. The student might end up taking 5 exams or more on the same topics before they demonstrate mastery. This poses a practical problem, as it would be difficult to create so many exams and also difficult to grade so many exams.

In the Khan Academy learning system (at least when I tried it out several years ago) students demonstrated mastery by getting a sufficiently long streak of questions correct. For example, if they get ten chain rule questions correct, then they have demonstrated mastery. This wasn't a perfect system, but it was at least an interesting step towards practical self-paced math courses.

Question: Has there been much discussion or research on how students can demonstrate mastery so that self-paced math courses can be implemented in a practical way? What are some of the ideas people have come up with? Have there been any efforts to create large banks of exams that can be used for self-paced math courses?

Answer 1

I presume you are fishing for practical ways in an online (digital) environment. First off, there's no objective definition of mastery. At the most rudimentary level you have standardised assessments that rely on overall accuracy to indicate mastery.

As you said Khan Academy uses an enhanced model that factors in for the success streak and not just accuracy. Duolingo (although unrelated to math) uses a similar model with another sophisticated layer added to it - which is to account for forgetting what you've learned with time (which happens naturally without sustained practice).

ALEKS uses something called Knowledge Space Theory which uses stochastic methods to efficiently assess student proficiency in a given set of topics. It's able to do this by asking the least number of questions. More importantly, the result of ALEKS is not a numerical score but a detail of what the student has mastered, and what he is ready to learn next. Lastly, Cognitive Tutor goes even further to consider step by step attempt of student, along with his hint seeking behaviour and other parameters to navigate a student through a course.

There must be many other burgeoning ideas in this field that I haven't mentioned above. Research on this is thriving mostly from the perspective of developing Intelligent Tutoring Systems (ITS) with the aid of big data and machine learning. The idea is to capture as much metadata about the student as possible and use it to strengthen the metric defining mastery.

Finally, regarding your question on availability of question banks, do appreciate that with the help of programming, making a question bank is not such a big issue. This is particularly true for maths and even more so for primary and secondary level education. For instance, ALEKS question bank has dynamic questions based on the problem type. Every time a problem type is invoked, a randomly generated problem (with the numbers being randomised) is thrown to the student.

This answer would be incomplete if I don't mention that defining mastery will be ultimately specific to the subject, the level of education and the assessors' objectives.