Hypothesis Tests in Students' Lives

Question

Consider, in a basic statistics class, the difference between the following two examples:

A six-sided die is rolled twenty times. It comes up a "two" eight times, which seems unfair. The owner of the die claims that it is fair, namely that each face of the die is equally likely to come up. Complete a hypothesis test to assess this claim.

versus

A store manager assigns shifts to six different employees, including you. Twenty workdays into the job, you notice that you have been assigned the "bad shift" eight times, which seems unfair. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." Complete a hypothesis test to assess this claim.

The second one puts statistics directly into the real world, and seems (to me) to be more engaging.

Besides shift assignments from your boss, what other topics from students' lives are yield interesting statistical tests of claims?

Although it is not entirely accurate, for the purposes of this question, let's assume that the mean age of my students is 20 years, with a standard deviation of 1 year.

Answer 1

A) First, let's explore the problem in more detail:

The difficulty with both the presented scenarios is that they only decide to perform a test after noticing something about the data -- and then they test the hypothesis that the data led them to on the same data.

When this happens, the test no longer has its nominal properties.

To extend the more "boring" example (and I do it because the probability model is less ambiguous), imagine that ten thousand students each carry our the experiment "roll a six-sided die twenty times".

Further imagine that if a number comes up eight times or more, the students will says "that seems unfair" and so test it.

Let's ask ourselves two questions

(i) If the dice are all fair, what proportion of students will say "that seems unfair"?

(ii) For the students who carry out a test, what's their type I error rate? (i.e. what proportion will reject the null?)

For (i) I just did three sets of simulations of 10000 such students, and got 6.74%, 6.93% and 6.62% getting enough of one face to say "that doesn't seem fair".

For (ii), we have to figure out what test they'll apply and what the nominal significance level they conduct it at might be.

I presume the students are specifically testing for too many or too few of the particular face they noticed a problem with. The sample size is too small to use a normal approximation on a proportions test; if we apply a binomial test, a two tailed test can't be symmetric (equal probability in each tail), but we can get 2.6% in the lower tail and 1.1% in the upper tail by rejecting with 0 or 8+ on the face tested (even though none of the people who test will actually have a zero, since our scenario didn't have students with 0 for a face trying to test it). So the overall nominal significance level is 3.7%.

But what's the actual rejection rate for people who carry out the test (the true type I error rate)? Well, under our rejection rule they all reject ... so it's 100%!

--

I recall working on a real-world problem like your scenario for someone once; the person's employer conducted random drug testing of drivers, and the particular person had been tested very many times -- he was certain another employee was making trouble for him, leading him to be tested much more often than randomly. If you work out the a priori probability he would be tested that many times, it seemed like something odd was going on. But if you worked out the probability that some employee would be tested that many times, it turned out to be not at all surprising. And of course to that employee it would look strange, even though nothing need be going on. ... and of course the unluckiest of all the (>100) employees would be the one asking what was up.

---

B) So what can be done? The easiest way is to slightly modify the example. Instead of saying 'that doesn't seem fair, let's test that data', we instead say 'that doesn't seem fair, let's specify exactly what we want to test, and then get some new data to test it with'. Then your examples will work fine!

For example:

A store manager assigns shifts to six different employees, including you. After a month on the job, you get the impression that you're getting the "bad" shift way too often. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." ... so you decide to collect data for the next 20 work days, and you find you're assigned the bad shift eight time. Complete a hypothesis test to assess this claim.

A similar modification can be made to the dice scenario

Answer 2

An example which I like to use in class is Paul the Octopus (predicted 11/13 games in 2010 World Cup). Sports are useful sources of statistical examples in general, but I particularly like this one because it illustrates how using hypothesis testing in real life can lead to nonsense.

Answer 3

Students in the class could google their birth dates before class and come to class knowing the day of the week on which they were born (MTWRF). In class you then test the hypothesis that days of birth are equally distributed.

Answer 4

On the softer side:

Example: Murder Trial

In a murder trial, the null hypothesis (innocent until proven guilty) is given by:

The defendant did not murder the deceased ($H_0$)

The alternate hypothesis is

The defendant did murder the deceased ($H_A$)

Implicit in $H_0$ is a range of possibilities that are consistent with $H_0$. It is up to the prosecuting council to produce evidence which lies outside this range of possibilities (e.g. the probability of the defendant being innocent AND owning the gun is small). If the prosecuting council succeed in producing such evidence (similar to a sample mean outside the confidence interval), the jury reject $H_0$ and the man is convicted.

A Type I error here is convicting an innocent man — which is considered worse than a Type II: acquitting a guilty man.

Answer 5

The problem is, in almost all recurring human interactions, almost nothing exhibits "random" (i.e. equiprobable) behavior, especially at the micro-level (and at the macro-level it is more likely to observe behavior represented by other distributions rather than the Uniform one -Normal and Pareto come easily to mind). So even though the distinction "fair/not fair" is very powerful and indeed engaging compared to say "test that "so and so" follows or not a Pareto distribution", it will be rather hard to find many convincing real world examples.

One of my Statistics professors when I was an undergraduate narrated to us how he had an argument with his wife about whether two different and long urban itineraries from their house to some relative they would visit often, were the same in terms of travel time (using car)... and he conducted an "equality of means" hypothesis test to check it.

Since the "average" is a fundamental statistical concept which is also intuitively known to everybody even before he hears the word "Statistics" for the first time, perhaps building examples and task-assignments around "equality of means" could be engaging and educationally beneficial. It would also have the students collecting the data themselves which is very useful as an experience.

For example, there may be the impression that Sally "talks more on her cell phone" than her friend Sue. But is this statistically validated? Record number of phonecalls and duration of each call (both are provided by the cell phones), for say 7 consecutive days (to include possible daily seasonality over a week), and conduct "equality of means" test for number of phone calls, and another for duration of calls. This specific example serves also to bring in the concept of independence - we expect that Sally and Sue talk to each other over the cell phones, being friends - so the two samples will include common observations. Should they? Should we exclude them? Why? etc

Answer 6

Games!

Meet your students where they are. Many students in their late teens are going to be into various forms of gaming activities. Find out what games are currently popular and come up with questions that allow them to use their existing interest and knowledge of the game to their advantage.

• Dungeons & Dragons: Sam is playing a figher-classed half-orc, rolling a D20 to hit. Seven out of ten of Sam's last rolls were above 15. The Dungeon Master suspects that Sam's D20 is unbalanced. Develop a test to investigate this.

• Pokémon: Mary's Rapidash has been using Fury Attack, which has an 85% Accuracy. Mary attempted to use this move seven times today, and all seven attempts missed. Mary tells you that she thinks her little sister hacked her game cartridge to make her miss more often. Develop a statistical trial to determine if this is plausible.

For the students in your class that are not familiar with the game, provide definitions. For example, you may define D20 as "a twenty-sided die" and Accuracy as "the percentage likelihood to hit".

Answer 7

I have students gather data from amongst themselves, or also from other students, and we have looked at things like sleep (significantly less than average?), study time (less than the assumed norm), difference between genders in coffee consumption, correlation between hours studied and hours worked (no), etc.

I did not do this fully rigorously; due to limitations of time, we first gathered data (both numeric and category-based), then went through such statistical tools as confidence interval, hypothesis testing and correlation/regression and had students use the relevant tool somewhere in the data. At the end, in the most ambitious take, I had them write a report and present their findings.