25
$\begingroup$

What research has been done on how much and in what circumstances various non-lecture types of teching are effective with regards to student knowledge and performance? Meta/review studies preferred over single case studies, of course.

I am thinking of methods like learning stations, flipped classroom, discovery learning, project work, group learning, even the more common seminars and lab work.

When and in what doses are these methods provide higher results than traditional lectures and exercise (problem solving) sessions?

I understand that nearly any method has it's place for some topics and to some amount. But is there any single method that can achieve same results when replacing most of the lessons in a course? I am thinking in parallels with biology - if there are 30 lessons in the course, a single field trip will be more useful than 2 more lessons. But replace all of the lessons with field work and you'll have spent a lot of time and leave with many gaps in knowledge. It seems that the concept of diminishing returns is somehow at play here, isn't it?

$\endgroup$
  • 2
    $\begingroup$ You should find this MAA report on teaching calculus helpful, esp. chapters 7 ("Good teaching") and 8 ("Ambitious teaching"). (The term "ambitious teaching," while not a neutral description, comprises "active-learning" strategies and other things.) $\endgroup$ – user1027 Dec 27 '19 at 2:37
  • 10
    $\begingroup$ Note that education research is similar to psychological research, which is now known to have a replication crisis, positive-effect publishing bias, low effect values, etc. After a rather large amount of reading I've come away fairly skeptical (excepting one hard fact: online courses are disasters for weak/remedial students, approx. doubling fail rates). All the local attempts at modern techniques that I've witnessed went nowhere/were abandoned. But definitely read some stuff, be aware, and make up your own mind. $\endgroup$ – Daniel R. Collins Dec 27 '19 at 16:12
  • 5
    $\begingroup$ How People Learn is a good reference. $\endgroup$ – skullpatrol Dec 27 '19 at 20:58
15
$\begingroup$

Consider a paper from this year: Setren, et. al., "Effects of the Flipped Classroom: Evidence from a Randomized Trial", Annenberg Institute at Brown University (2019). In their introduction, the authors write:

Despite the proliferation of the flipped classroom, little well-identified evidence exists on its impact on student learning.

In this study a large (N = 1328 students), randomized trial was conducted at the West Point military academy in the United States, in economics and math classes. Their findings were that there is a small short-term boost in the math classes, none in economics classes, and those effects became negligible by the end of the course. About half of the students in flipped classes did the advance preparation. Flipped classrooms also increased inequity, in the sense that high-performing, white, male students did better, while weaker and minority students had no improvement.

The authors speculate that the short-term benefit in math classes may be due largely to the excitement level of the instructors trying a new method.


In comments, the question has been raised whether the model of using videos for flipped classrooms, as tested in the study, is a fair one or not. Indeed, this seems to be the standard and widespread practice. From the bibliography, Schaffhauser, Dian and Rhea Kelly, 2016, “55 Percent of Faculty Are Flipping the Classroom”, Campus Technology:

Fifty-five percent of the survey respondents said they are somewhere along the spectrum of flipping all or some of their courses, in which they ask their students to view videos or some other digital matter online before coming to school and then use class time for other activities, such as hands-on and team projects or discussions. Twenty-five percent intend to introduce the flipped model into their courses over the next year or are exploring that possibility.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ the authors write: "Despite the proliferation of the flipped classroom, little well-identified evidence exists on its impact on student learning." Their definition of a flipped classroom is very narrow: it involves students watching a video of a lecture at home. That seems to me like an absurd method, since this is what books are for. So I'm not surprised that an absurd method doesn't produce great results. $\endgroup$ – Ben Crowell Dec 29 '19 at 1:15
  • 1
    $\begingroup$ @BenCrowell: Any time I hear people advocating for flipped classrooms, video is always the go-to delivery method. E.g., Wikipedia, "In a flipped classroom, 'content delivery' may take a variety of forms. Often, video lessons prepared by the teacher or third parties are used to deliver content..." I'd be surprised if there's evidence that anything else is more standard practice? $\endgroup$ – Daniel R. Collins Dec 29 '19 at 2:23
  • 1
    $\begingroup$ The OP gives a long list of many different phrases that could be applied to a whole menagerie of very different teaching techniques. The paper describes one very specific teaching technique which may be what people often mean when they say "flipped classroom," I don't know. The practices they tried seem self-evidently cringeworthy to me, and I'm not surprised they were no more effective than lecturing. I teach mostly physics, not math, but the active learning techniques that have shown documented success in physics look nothing like this. $\endgroup$ – Ben Crowell Dec 29 '19 at 16:13
  • 1
    $\begingroup$ Some common denominators in the techniques for which I've seen evidence of effectiveness are: (1) words come out of students' mouths, and those words are about concepts; (2) students read a textbook before they come to class, and this reading requirement is enforced through grading (e.g., an easy multiple-choice test given at the beginning of the class). The Setren study tested a technique that didn't include either of these elements. $\endgroup$ – Ben Crowell Dec 29 '19 at 16:19
  • 1
    $\begingroup$ @BenCrowell: The OP asked explicitly about flipped classrooms, so my answer addresses that. I agree with you that I'm not surprised the effect is poor; but the study models the standard widespread practice so far as I'm aware. I've added information from the bibliography to that effect, above. $\endgroup$ – Daniel R. Collins Dec 30 '19 at 2:38
13
$\begingroup$

Re active learning, the best meta-analysis of the evidence that I know of is Freeman et al., "Active learning increases student performance in science, engineering, and mathematics," http://www.pnas.org/content/early/2014/05/08/1319030111 .

Active learning is not the opposite of fully guided instruction. The successful active learning techniques for which I've seen good empirical evidence are mostly done using fully guided instruction. Students have a textbook, and the textbook presents the principles, gives examples, and so on. Students just don't get the textbook spoon-fed to them again during class time.

Active learning is not the same as "edutainment" practices such as the use of electronic clickers. Clickers are neither necessary nor sufficient for implementing active learning well.

| improve this answer | |
$\endgroup$
  • 7
    $\begingroup$ (1) "Thus, this study’s intent was to evaluate the average effect of any active learning type and intensity contrasted with traditional lecturing." It's probably worth mentioning that so-called "active learning" courses included as little as 10% of the time spent on learning activities, since the OP seems new to the general idea. (2) It may also be of interest to the OP that "traditional lecturing," whatever that is, has been criticized for hundreds of years (and for a couple of millennia if you want to include Plato's Phaedrus, Meno and probably other dialogues). $\endgroup$ – user1027 Dec 27 '19 at 2:15
  • 6
    $\begingroup$ Echoing @user1527, this study states that the time of active learning was variable and "we were not able to evaluate the relationship between the intensity (or type) of active learning and student performance". It appears that a fair summary would be "some active learning is better than none". Am I reading that correctly or is there more nuance to conclude? It might still be true that 52,11% time on passive learning is actually the sweet spot, right? $\endgroup$ – Džuris Dec 27 '19 at 3:56
  • 10
    $\begingroup$ I just realized that their definition of "traditional lecturing" excludes any exercises, even homework: "student activity was assumed to be limited to taking notes and/or asking occasional and unprompted questions of the instructor". Thus the article is not really an argument against lecturing, it's a statement that lecturing is not enough. $\endgroup$ – Džuris Dec 27 '19 at 15:56
9
$\begingroup$

A supplement to @Ben's PNAS article citation:

Hodges, Linda C. "Contemporary Issues in Group Learning in Undergraduate Science Classrooms: A Perspective from Student Engagement." CBE—Life Sciences Education 17, no. 2 (2018): es3. Journal link.

The focus is on group work. At my college we have a study group of faculty exploring issues arising with group work, for example, how best to form groups. The Hodges paper has a section entitled "How does the way groups are formed affect student outcomes?":

"Should students self-select into groups or should instructors form groups? How do group size and composition affect the functioning and learning in the group? Should group composition be permanent or changing throughout a term?"

Another section asks, "How do learning spaces impact group work?"---a particular focus of our study group.

| improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ My current practice is to randomize groups at least once per meeting, sometimes more. I find it works much better than self-selection. $\endgroup$ – Ben Crowell Dec 27 '19 at 2:58
  • $\begingroup$ @BenCrowell: Same: How To Help a Quiet Class. $\endgroup$ – Joseph O'Rourke Dec 27 '19 at 13:20
9
$\begingroup$

The Education Endowment Foundation has a 'Teaching and Learning Toolkit' which aims to assess the effectiveness of various teaching methods or types of intervention, including collaborative learning, altering teaching for individual learning styles, and small group learning. It is focused on teaching 5–16 year olds, and not specific to mathematics education, but it is all evidence-based and (in my opinion) persuasively presented.

It is available here: https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit and the citation is:

Higgins, S., Katsipataki, M., Kokotsaki, D., Coleman, R., Major, L.E., & Coe, R. (2014). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation.

| improve this answer | |
$\endgroup$
7
$\begingroup$

I would like to echo Peter Saveliev's point in the comments: You are asking about "higher results" but you do not specify what qualities you are interested in measuring.

Do you care about grades? Performance on standardized tests of content knowledge? Student success in courses which depend on the given course as a prerequisite? Student perception of how interesting the material is? Likelihood that the student will sign up for optional successor courses? Ability to speak coherently about the material? Student perception of themselves as an independent authority on the subject matter?

There are a million distinct questions you can ask. Different instructors have radically different values, and "success" can mean something very different to different instructors.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I didn't want to be too restrictive. I find most of the answers very useful and I am happy I didn't exclude them. By higher results I meant measurable performance in comparable tests. Performance in standardized tests would be the most desired metric for me. Metrics like student perception and satisfaction, as well as the sign-up likelihood are out of the scope of this question. Knowledge/skills only. $\endgroup$ – Džuris Jan 1 at 21:39
4
$\begingroup$

What follows is a critique of the paper cited in another answer here: Setren, et. al., "Effects of the Flipped Classroom: Evidence from a Randomized Trial", Annenberg Institute at Brown University (2019). That paper doesn't, in fact, refute the value of the flipped classroom in any way and is an example of poor research design. In fact, it says nothing about a flipped classroom by any reasonable definition.

I'm adding this for the benefit of the O.P. who shouldn't be misled by "research" that purports to show that the flipped classroom is a poor educational practice, when it actually discusses something else entirely.

First, the design of the experiment in the paper by Setren et. al. (the authors) is fatally flawed. I would not have let any of my doctoral students continue with this research, though a number might have been interested in the topic. Fundamentally, if you can predict the outcome of a research project simply from the design, then it isn't really research. I don't doubt the sincerity of the authors but this seems like a situation of confirmation bias rather than a true search for truth regarding the flipped classroom.

The most serious issue with the paper, other than methodological failure, is an incorrect conception of the flipped classroom. The authors seem to have the mistaken impression that the "flipped classroom" is about where things take place. This is a serious misconception that prejudices any conclusion from the study.

In fact, the flipped classroom is about what takes place when the students and instructors are face to face. If they aren't different from the standard model, then nothing positive can be expected to happen. In a standard lecture the students are relatively passive, though they may take notes and, perhaps, ask questions. In a flipped classroom the students are active, typically in groups with collaborative educational activities intended to reinforce learning. Students come together in a flipped classroom to do things that can only be done face to face. Things that don't require interpersonal communication (not just broadcasting) are done otherwise.

In this experiment, the students were given videos to watch on their own, along with readings. No effective channel to ask questions is described, and neither are exercises based on that preparation. This alone is a flaw. It advantages people used to passively watching lectures, but it disadvantages those who need more help and have questions along the way. Separating the questions (next day in class) from the video is, itself a flaw. Students can, of course, make a written note of questions for answering later, and the best and most advantaged students are likely to do that, but only if they have enough experience with the format to recognize that it is essential. Students who try to rely on memory alone to bridge the gap between the video and the question period are disadvantaged. (See below for comments on lack of training.)

In most educational settings there is an expectation that students will spend twice as much time on a course outside the classroom as inside it. This seems to have been missing from the experimental setup as described. The videos were short, in particular (a good thing). But other preparation is also needed.

Also, active learning seems to have been actively discouraged. This again, advantages those students who don't need much instruction and are comfortable with relatively passive consumption of material. This is a poor way to generate insight, but most especially in those students who are otherwise disadvantaged by the educational system they have been faced with. The flipped classroom is actually an attempt to address this issue, hence group work. Group work is reported in the experiment, but mostly in the context of students solving problems, rather than more general learning and discussion. This again, seems to suggest a misconception. Students in a standard setting will, themselves, solve their exercises out of class partly in isolation and partly in a group context. But the flipped classroom, used properly, permits more opportunities since the instructor is present and can guide the activities.

The paper seems to suggest that video is a key to the flipped classroom. This is not actually true, though it is a common misconception. Students are given some way to prepare for class, but it needn't be videos. It could be text, such as PDFs prepared by the instructor. It could involve interactive simulations that let the students engage with the material. And it could (and should) involve some exercises based on that preparation.

The best flipped classrooms will also provide a way to get questions asked and answered before class, perhaps via a mailing list. Students might be required to write a one or two paragraph summary of the readings (or videos) as a sort of "ticket" for attendance in the subsequent face to face session. None of this was provided or expected in the experiment. With video alone, students are likely to be passive. But it need not be that way.

In a better course design, the flipped classroom will provide the instructor some feedback, prior to the class, that follows up on the out of class activities. If there is a mailing list, for example, the questions posed there are seen by other students and by the instructor. Even if they aren't answered on the list itself, the instructor can prepare the next class period in response to student concerns. The instructor, then, can spend less time and effort on things that the students already grasp well, and can focus on more essential concerns.

The paper seems to suggest that students work on exercises (alone or in groups) when in class with some chance for feedback, but I doubt that it is especially effective, just due to lack of time and lack of instructor resource. A sole instructor and a T.A. can only work with a few students at a time, leaving the other students bored and stuck. This is why pairing and group work is suggested, and normally required, for the flipped classroom. Students help one another with questions and explorations, not just exercises, freeing up instructors for the most difficult cases. Research has been done on this in the situation of paired programming in a communal space. The U.S.M.A. has strict ethical rules about joint work, though the paper doesn't say if these prohibitions against assistance were enforced here or not.

The faculty who actually carried out the experiment were all somewhat experienced in standard lecture formats and it was noted in the paper that they had little or no experience with other formats. Most of the instructors, however, were relatively inexperienced faculty members. As junior military officers, their own educational background was most likely on a standard model, with which they have excelled. Also, they had likely taught the target courses, or similar courses, in a standard format, similar to their own education. The experimental design doesn't describe that they were given any specific training in the flipped classroom. Nor were they permitted to deviate from the designed format. The paper does, however, admit that the students did better in those sections in which the instructor was more enthusiastic about the flipped classroom.

The paper also seems to admit that it may have been a flaw in the design that each instructor taught both a standard model section and a flipped section. Military officers follow orders, of course, but they don't have to accept the validity of what they are commanded to do. The whole process was, in fact, an imposition on them, as none were volunteers. This leads to uniformity of a certain kind, of course, but not to good educational outcomes for students.

Now, even if we somehow decide that all of this is to be forgiven, and that the flipped classroom model was well run (thus validating the data itself), there is still a fundamental flaw that would prevent us from drawing out generalized conclusions from the study: all of the subjects are from a highly advantaged subset of the US population. The U.S.M.A. is a highly selective undergraduate college in which students gain entry by proving that they excel in the standard educational process. They are used to lecture/exercise/exam based courses and have a history of success in such situations and most have little experience with alternate educational structures. Moreover, most of them come from dominant cultural and economic situations since admittance requires the recommendation of a member of Congress. They are not a cross section of students in any case, and especially, neither are their educational backgrounds.

Next, the experimental period was short, apparently between one and three weeks, with no one in the treatment group, student and instructor alike, given an opportunity to become comfortable or familiar with the process, while the control group had a long history of such experience with standard practice. Moreover, the authors report that students in the experimental group weren't very vigilant in doing the necessary preparation for class. This is likely due to their expectation from past experience that it was the classroom in which the material would be adequately "covered". But it was not. So, many seem to have entered the classroom unprepared.

It would be natural to expect a bit of chaos in the experimental group as the experiment was carried out in the middle of an otherwise standard course. All of a sudden, the rules of engagement are changed. This alone would lead one to suspect lower achievement over that lesson, as students have been dropped into a new world for a few weeks, not knowing how to act or what to expect, other than that they were supposed to "follow orders".

Given the experimental design, the outcome was foreordained. Put students into a new situation that might seem a bit chaotic, and they aren't likely to do very well.

Consider the following thought experiment in which I suspect that you can easily predict the outcome.

Suppose we had a large group of students who had spent their entire lives learning in a collaborative group environment in which they have access outside the classroom to a large variety of resources, but who spend their face to face time in collaborative learning. They can, and are encouraged to, prepare for class by reading, etc, and are also expected to bring questions with them. But there are also exercises they can do to test their own understanding prior to class. The face to face time is spent in reinforcement of the ideas and feedback on attempts. Students are encouraged to speak and offer ideas and other students are encouraged to comment.

Now, take the experiment as described in the paper, but flip the treatment and control groups. the course proceeds as normal for them (flipped) until we reach the start of the experiment. Otherwise run it as before, but in this case all of the students instructors are thoroughly familiar with the above described methods, but not so much with lecture to a relatively passive group, followed by exercises to be done at home.

What do you suppose the outcome of the new experiment would be. Isn't it pretty obvious?

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ -1 This is, of course, not an answer to the OP, and presents no references or studies as requested. $\endgroup$ – Daniel R. Collins Jan 1 at 17:54
  • 2
    $\begingroup$ @DanielR.Collins, the discussion I give of the flipped classroom is, however, based on about 20 years of teaching experience. $\endgroup$ – Buffy Jan 1 at 17:59
  • $\begingroup$ It appears that a method has to be taken to a more ridiculous side to produce obviously different results. Similarly one could "refute" the answer by Ben as it puts "lectures" on one side and everything else, including "lectures+homework" and "lectures+exercises" on the opposite (active). One could similarly predict from the setup that the latter will perform better. However, I still find those results meaningful, it's just that one has to be aware of the caveats. $\endgroup$ – Džuris Jan 1 at 21:48
  • $\begingroup$ "research that purports to show that the flipped classroom is a poor educational practice" I didn't see that claim in the paper. And what exactly is "fatally flawed" about the design of experiment? $\endgroup$ – Michael Bächtold Jan 2 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.