7
$\begingroup$

Suppose you're the instructor of a large university course split into several sections, each of which has a TA. (Think of a giant lecture-style calculus course with several hundred students that meet in smaller "recitations" on other days.) Homework is graded by the TAs. Presumably, each has their own method/style/harshness of grading.

What is an effective method to incorporate homework grades that is fair to the students?

Should this be addressed from a forward-looking perspective, by providing detailed grading rubrics to each TA? Should you follow up and oversee their grading somehow? Or, should this be addressed retroactively, by somehow analyzing the grades statistically and "normalizing" with some scheme? Or is there a feasible combination of these methods? Or something else entirely?

I am interested in what you have done in the past and how effective it was, especially if you have a good sense for how well your scheme worked (data analysis, student feedback, TA feedback, etc.). I am also interested if you have any statistical insight/research on this question. Mostly, I am interested in fairness to the students, as well as how to handle this problem efficiently (that is, getting maximum fairness for a reasonable time commitment, not just minimizing time).

$\endgroup$
4
$\begingroup$

As student I was TA in a first-year course with a total of around 600 students each time, divided into classes with around 40 to 60 students each, and weekly quizzes in 20 student groups, a different question for each of the 3 or 4 days recitation was held. No homework.

The exams were divided among teachers and grading assistants by question (each one graded one question only, for part of the students), and had very detailed grading instructions. The quizzes were graded by us TAs.

Overall this worked out pretty well, but required an enormous effort in setting up exams and quizzes, and in logistics.

$\endgroup$
4
$\begingroup$

In all the exercise groups where I was a TA or a instructor for other TAs, we made a lot of efford how questions were graded. The problem is, however, even if you spent even more time, the problems arise when you see the homework.

Students did some mistakes destroying the structure of the questions (when you said "When someone arrives at that certain point, he/she will get 2 points, etc. - And they did some alternative way with a nice idea you did not think about, but not comparable to other solutions). Or they did something else you didn't plan for. In these cases, TAs have to make up their minds and everyone will problably grad in different ways. Or students did some subtle error which can be overlooked by inexperienced TAs. Lets face reality: Some TAs correct also every deep (They take responsiblity of their work) and some correct sketchy (due to time or competence).

If you really want to maximize this, there are several possibilities:

  • You can divide the homework into questions and every TAs has to correct only one questions. If he/she e.g. overlooks some subtle error, then he/she it for everyone. The problem here is that students have to visit several different TAs if they have questions about the grading. If your TAs have in general a free weekend, you can also do the extreme way that one TAs is grading all the homework (this is possible in courses with at most 100-150, but not with the mentioned 700).
  • You can ask the TAs to write emails to everyone reflecting each equivalent class of solution. If there is some good responsible TAs, he/she will get most of the errors and suggest the grading for such a solution and the rest can follow. The big problem is that you have to describe and summarize solution and read maybe 50-100 emails every week. I have seen this sometimes when only a few TAs have to grad exams for a lot of students.
  • You can have extra grading sessions where all TAs meet and discuss directly if something appears. This is very effective, but you have to find a date where all TAs are free. Also, the problem is that maybe someone is faster than the others etc. If the groups are really small or the homework is not much, this is maybe the best solution.

However, I think you can never attain the maximum by this.

Our solution

Last time we had a bigger course (about 100-150 students, which is big in our department), there were not too few complains about grading although there was weekly very long meeting with all TAs and they were forced to come to me if anything is not clear, etc. We also saw this in statistics.

Luckily, the date of the exercise groups was the same for all TAs and we decided to rotate the TAs between the different groups. The message was: "Yes, the world is not fair and in case of doubt, some persons will give a better grade and other persons will give a worse grade. In order to have to same condition for every student, we switch TAs every X weeks." (In smaller courses you can try that every student and every TAs have met; in bigger courses you maybe have to find a out a good mixture of severe and slack TAs with the contraint of their timetable.)

In a nutshell, I think the method worked really well. Some problems occured:

  • For psychological reasons, you should communicate this at the beginning of the course. For some students, their feelings boiled over due to the "surprise".
  • The main focus of the groups should be to learn something, to ask questions, etc. -- If there is a regular change for the TAs, many students will not dare to ask questions. There should not be too many changes. I tried a intervall with 5 meetings (From a different viewpoint: If the students get some TAs they don't like, they can then count to 5 and get a new one).
  • After the first change, there were many complaints: Most of them were either from good students having problems with a inexperienced TAs or poor students now getting worse grades. The TAs should be strongly encouraged to take over their responsibility (and you may combine the "change-method" with some of the methods above) in order to avoid such situations.
  • Some students wanted to change the group in order to "travel" with the TAs. You should think about how to deal with such a situation before doing this "change-method".
$\endgroup$
3
$\begingroup$

I'll share my experience, as well.

Context: As a graduate student, I was the lead instructor for a 120-student lecture course with 4 recitation sections amongst 3 TAs. Lectures were MWF and recitations TuTh. I assigned weekly homework assignments collected and returned in recitation. This was our "intro to proofs" course, a 100-level course entitled "Concepts of Math".

Method: I wrote and distributed detailed grading rubrics for homework problems. I also assigned specific problems to each TA, rather than have them grade everything from their own section. (Logistically, this was accomplished via a locked filing cabinet in a designated room that everyone had access to. Homework was turned in on Th, TAs would visit over the next few days to grade, and papers were returned on Tu.) My goal was to standardize grades across the sections throughout the semester, rather than at the end via statistical normalizing. My advisor had always used the end-of-term statistical method, but I worried this might foster jealousy of other sections with easy graders / contempt for TAs with high standards. I also helped out by grading one problem per assignment.

I used the same procedure for exams. I wrote detailed rubrics, including provisions for multiple methods of solution, predictions of common mistakes and how to handle them, and tips for providing good feedback beyond just a score. I graded a couple of questions on each exam, and read through a bunch of them to get a sense for how the students were doing.

For homework grading questions/complaints, I told the students to talk to their TA first, who would then direct them to the TA (or me) who graded the question. If that could not be resolved, they could come to me.

For exams, I gave the students exactly one week after return in which they could write up a document to explain their issue, their understanding of the relevant problem, and how their demonstration of knowledge should deserve more points. I would then read each of these and, when appropriate, raise the grade; either way, I wrote a response back to help them understand the grading as it was.

Results: I honestly think this was a great success, and am very proud of how the logistics of this course were handled, looking back. I never had complaints from students about unfairness, and I spoke with some of them often enough on a personal level that I like to think I could have determined whether there was an undercurrent amongst them that was just unspoken. When I did receive grading complaints, they were often of the form "I don't understand why I lost points here, can you explain more?" and I would do so on a case-by-case basis. Sometimes, their complaints were valid (and, indeed, there was one grader who was sloppy and unfair on early assignments, so I had to have a long meeting with them and show them how to better; thereafter, they improved greatly). Sometimes, they were just grade-grubbing, but in those cases, I was glad I had those formal rubrics written out, so I could point to the problem and say, "This part was worth X points, and you didn't address issue Y. There's nothing I can do, in fairness to other students."

Thoughts: As much of a success as this was, it required a huge time commitment from me as the primary instructor. I spent a lot of time designing the rubrics, taking into account my experiences grading similar problems in the past as a TA myself. I spent a lot of time answering TA questions, in person and via email, about grading issues. I spent a lot of time talking to students about homework problems (and how they were graded). I spent a lot of time just outright grading. But that was my choice. Ultimately, I enjoyed it all and don't regret it, simply for the experience that I gained from it all. I was glad to have TAs who cared as much about grading fairness as I did and didn't see it as just a teaching job on the side.

However, this is grossly infeasible. This was the one course I was teaching at the time while finishing my dissertation and, even then, it was cutting into my life. Like I said, I enjoyed all of it, but it just doesn't scale upwards nicely at all.

On the positive side, I now have a cache of homework/exam problems as well as their solutions and grading rubrics, and could happily share these with future instructors. So, perhaps this doesn't have to be a perpetual problem.

Recommendations: Frankly, I would strive for this but make sacrifices where they are needed to save time/effort. In the future, if I have more than one course like this, I would first cut the actual grading of problems myself. Or, at the least, I would make sure one problem is completely trivial to grade, almost like a multiple-choice problem. By reading through those submissions, I could at least get some sense of how the students are doing. I would then cut time in the development of the rubrics. As it was, I spent much time considering varied approaches to problems and trying to nip grading considerations in the bud a priori. This isn't necessary, if you're willing to put trust in your TAs and potentially spend the time answering their email questions as they come up.

Ultimately, I feel strongly that the statistical method is flawed, and personally prefer a method that attempts to standardize grading across sections. Specifically, I actively promote the each person grades a problem, not a section method. I believe this is the only way to be truly fair, and I don't think it requires that much more effort on the part of the instructor/TAs to be implemented.

$\endgroup$
  • $\begingroup$ A quick upgrade to this answer seems to be to make the person in charge of grading problem 3 create their rubric for problem 3, then turn it back in to the lead instructor. Is there an easy counterargument to this idea that I don't see? $\endgroup$ – Chris Cunningham Mar 31 '14 at 18:45
  • $\begingroup$ @ChrisCunningham: In my context, it was quicker for me to make the rubrics because I already had past experience grading these exact (and other similar) problems, so I could anticipate many common errors. I think that the TAs I had would not have incorporated these as effectively into their rubrics. Although, in general, yes I think your suggestion is a good one. $\endgroup$ – Brendan W. Sullivan Apr 1 '14 at 20:54
3
$\begingroup$

Presumably at least the exams are graded in a standardized way such as "each TA grades a problem."

This means you can use exam grades to calibrate the students' homework grades. Split the students up by TA, and find the mean and standard deviation of the exam grades, and the mean and standard deviation of the homework grades for just that TA's students.

Check for significant differences in rank (i.e. TA X has the top homework grades but the second-worst exam grades), and if there is a massive difference, add homework points to certain sections.


This has clear downsides, but it has a place: it is only slightly more work than "standardize all sections so that the homework average is 80%," but it is substantially more fair. Of course many of the other answers here are significantly more fair, but they are also significantly more work!

$\endgroup$
  • $\begingroup$ The above procedure worked (I think) well in a many-sectioned calculus course I was in charge of. For various reasons, not necessarily connected to instructor ability, there has always been great variation in mean final exam grades between sections. (Final exams are team-graded.) And homework/test marking are too difficult to monitor effectively. $\endgroup$ – André Nicolas Mar 31 '14 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.