9
$\begingroup$

"[Myth] that exams are objectively graded. Daniel Stark and Edward Elliot sent two English essays to 200 high school teachers for grading. They got back 142 grades. For one paper, the grades ranged from 50 to 99; for the other, the grades went from 64 to 99. But English is not an "objective" subject, you say. Well, they did the same thing for an essay answer in mathematics and got back grades ranging from 28 to 95. Though most of the grades they received in both cases fell into the middle ground, it was evident that a good part of any grade was the result of who marked the exam and not of who took it"

Source: https://www.nyu.edu/projects/ollman/docs/why_exams.php

The study the author is referring to is: "Starch D, Elliott EC. Reliability of grading work in mathematics." http://www.jstor.org/stable/1076246?seq=1#page_scan_tab_contents

This is pretty consistent with my experience, both as a grader and a student. The cited experiment was done in 1913, however. I am curious about more recent, broader studies about testing the variance of grading - especially for math exams such as those we give in college algebra or calculus.

The same studies are referenced here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4041495/

The only recent reproduction I have found was for the Starch/Elliot experiment about english papers, not math exams: http://pareonline.net/getvn.asp?v=16&n=17

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ I don't have an answer to the actual question, but it would be highly surprising if the variation in scores was small. Even if we all agree on what counts as right and wrong, there are so very many ways to weight a mistake. Did anyone ever expect universally comparable grades? It seems like a strawman position. $\endgroup$
    – Adam
    Dec 29, 2017 at 5:40
  • 2
    $\begingroup$ Even if the grading were consistent among teachers, there is still tremendous variability in test questions. The important thing is that teachers grade their own test consistently and for that many people use a Rubric $\endgroup$
    – Amy B
    Dec 29, 2017 at 5:59
  • 2
    $\begingroup$ Multiple choice is the format the minimizes variance (to zero). It also is the least time intensive form of grading. But it has many other drawbacks. Because there is wide variation in how different people grade the same work, it's worthwhile to try to avoid having multiple people grade different problems or, if that is necessary, to use a common grade scale that attempts to reduce variance between graders as much as possible. $\endgroup$
    – Michael Joyce
    Dec 29, 2017 at 15:07
  • $\begingroup$ @Adam Regardless of what we expect (as educators), we treat grades as if they were universally comparable; for example because employers and educators look at GPA when considering candidates. (And because of the way they are used, students learn to see grades as universally comparable.) $\endgroup$
    – Areaperson
    Dec 29, 2017 at 18:32
  • 2
    $\begingroup$ @AreaMan I don't think that employers or educators do treat the same grade from different institutions the same. Within the same institution, perhaps. $\endgroup$
    – Adam
    Dec 29, 2017 at 19:49

1 Answer 1

Reset to default
3
$\begingroup$

Eductational Testing Service has done a lot of work on essay grading for AP exams and similar (SATs when they've had essays, etc.) They put a fair amount of work into coming up with a detailed key and having support mechanisms: essays are graded twice, large variances examined, essays graded together with a supervisor able to deal with new questions that are generated (e.g. unusual solution method, questions of the graders). They used to have a lot of methodology articles on their website.

From personal experience, having a detailed key with partial credit spelled out helps to drive better results. Maybe more easily justified for a monster exam with many graders (e.g. TA's for a college chemistry exam). But probably still good practice even for an individual teacher grading a 30 person section. [Like war planning, even if you don't anticipate everything, the effort to plan out a key helps when running the grading and dealing with questions that arise as you grade exams/fight the enemy.]

P.s. How about instead of (or at least in addition to) giving us the range, giving us the standard deviation or the 90% interval? While much of mathematics is logical and rigorous, much of EDUCATION is statistical in nature and even Bayesian (in the sense that we don't know all the variables or the actual pdf being sampled). Saying "OMG, what a range" is not even how to think about educational methodology, which is a practical art with cost benefit tradeoffs and uncertainties (not a Euclidean proof). INstead thinking about the typical difference (e.g. SD/mean) is a more intutive way to think about the scale of the issue.

Improve this answer
$\endgroup$
24
  • $\begingroup$ If you look at the original article ( the jstor article that I linked to ), you'll see much more detailed statistics. (The quote is part of a polemic, not a statistical analysis.) $\endgroup$
    – Areaperson
    Dec 29, 2017 at 18:35
  • $\begingroup$ Oh...that is a little better. So more of the issue is on the polemic than on you. But my critisism of framing the topic first with range remains. It's not the right framework to even start talking about this. (Sort of a student type error or maybe for adults reflects a mindset used to math rigor rather than science exploration.) And pedagogy research lives in the latter world, not the former. Normalized standard deviation should be an immediate first consideration. Shows you how tight the distribution is. Scales the problem. $\endgroup$
    – guest
    Dec 29, 2017 at 19:49
  • $\begingroup$ Frankly, I think range is the issue here. Attacking the objectivity of grading because of a 70% difference in scores is convincing, for most people anyway. We've all had many exams, and the difference that percentage difference makes doesn't need an explanation of scale. (Since it is the difference between failure and an A.) $\endgroup$
    – Areaperson
    Dec 29, 2017 at 20:08
  • 1
    $\begingroup$ Frankly, I think it is a flaw in how to think about problems in the real world. It's like on sports blogs when people cite Tom Brady as an example of how having a 1st round draft choice is not helpful for getting a good quarterback. There is a world that exists between certainty (same x always leads to same y) and perfect chance (coin flips). Citing a range is a common misconception and is "sound bite-y". Correlations, normalized standard deviations, ratios. This is how to think about social sciences. Pedadogy is a social science. One with lots of complex factors. It is NOT math. $\endgroup$
    – guest
    Dec 29, 2017 at 20:57
  • $\begingroup$ I apologize for being contentious. I am just repeating myself. Have a different point, will make in comments since not enough to be its own answer $\endgroup$
    – guest
    Dec 29, 2017 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.