# Hypothesis Tests in Students' Lives

Consider, in a basic statistics class, the difference between the following two examples:

A six-sided die is rolled twenty times. It comes up a "two" eight times, which seems unfair. The owner of the die claims that it is fair, namely that each face of the die is equally likely to come up. Complete a hypothesis test to assess this claim.

versus

A store manager assigns shifts to six different employees, including you. Twenty workdays into the job, you notice that you have been assigned the "bad shift" eight times, which seems unfair. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." Complete a hypothesis test to assess this claim.

The second one puts statistics directly into the real world, and seems (to me) to be more engaging.

Besides shift assignments from your boss, what other topics from students' lives are yield interesting statistical tests of claims?

Although it is not entirely accurate, for the purposes of this question, let's assume that the mean age of my students is 20 years, with a standard deviation of 1 year.

• That's an engaging example. But a real manager would probably say: "I'm not trying to be unfair to you; I'm balancing a complex combination of store needs and people's preferences. We'll see what you can do in the different shifts and you'll see how you can rearrange things outside of work to accommodate this more easily, and after a couple months things will get better."
– user173
Oct 28, 2014 at 11:33
• If you flip a coin enough, you'll see a run 10 heads or tails eventually. At what point do you determine the coin isn't fair? Oct 28, 2014 at 18:10
• "It comes up a "two" eight times, which seems unfair" and "Complete a hypothesis test to assess this claim" -- there's a problem: the hypothesis of unfairness and the wish to test it is generated based on seeing the data you're about to use to test the claim. This impacts the properties of the test (it will have an actual significance level of much more than the nominal level, for example). The same problem affects the second example. These examples will be giving students a serious misconception. Apr 16, 2015 at 11:17
• @Glen_b You should fix everything about what I've done, and let me award a bounty on your answer for improving my statistics instruction. Obviously my examples are not good. Apr 21, 2015 at 16:10
• Personally I found examples of the second kind disengaging as a high school student, since I wasn't interested at the time in working in a store at the time. You need to be careful with "real life" examples because they can put people off if they are not interested in the specific application. Apr 21, 2015 at 22:53

A) First, let's explore the problem in more detail:

The difficulty with both the presented scenarios is that they only decide to perform a test after noticing something about the data -- and then they test the hypothesis that the data led them to on the same data.

When this happens, the test no longer has its nominal properties.

To extend the more "boring" example (and I do it because the probability model is less ambiguous), imagine that ten thousand students each carry our the experiment "roll a six-sided die twenty times".

Further imagine that if a number comes up eight times or more, the students will says "that seems unfair" and so test it.

(i) If the dice are all fair, what proportion of students will say "that seems unfair"?

(ii) For the students who carry out a test, what's their type I error rate? (i.e. what proportion will reject the null?)

For (i) I just did three sets of simulations of 10000 such students, and got 6.74%, 6.93% and 6.62% getting enough of one face to say "that doesn't seem fair".

For (ii), we have to figure out what test they'll apply and what the nominal significance level they conduct it at might be.

I presume the students are specifically testing for too many or too few of the particular face they noticed a problem with. The sample size is too small to use a normal approximation on a proportions test; if we apply a binomial test, a two tailed test can't be symmetric (equal probability in each tail), but we can get 2.6% in the lower tail and 1.1% in the upper tail by rejecting with 0 or 8+ on the face tested (even though none of the people who test will actually have a zero, since our scenario didn't have students with 0 for a face trying to test it). So the overall nominal significance level is 3.7%.

But what's the actual rejection rate for people who carry out the test (the true type I error rate)? Well, under our rejection rule they all reject ... so it's 100%!

--

I recall working on a real-world problem like your scenario for someone once; the person's employer conducted random drug testing of drivers, and the particular person had been tested very many times -- he was certain another employee was making trouble for him, leading him to be tested much more often than randomly. If you work out the a priori probability he would be tested that many times, it seemed like something odd was going on. But if you worked out the probability that some employee would be tested that many times, it turned out to be not at all surprising. And of course to that employee it would look strange, even though nothing need be going on. ... and of course the unluckiest of all the (>100) employees would be the one asking what was up.

B) So what can be done? The easiest way is to slightly modify the example. Instead of saying 'that doesn't seem fair, let's test that data', we instead say 'that doesn't seem fair, let's specify exactly what we want to test, and then get some new data to test it with'. Then your examples will work fine!

For example:

A store manager assigns shifts to six different employees, including you. After a month on the job, you get the impression that you're getting the "bad" shift way too often. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." ... so you decide to collect data for the next 20 work days, and you find you're assigned the bad shift eight time. Complete a hypothesis test to assess this claim.

A similar modification can be made to the dice scenario

• Perfect. Thanks for your efforts. I know this kind of thing must really bother experts in statistics, and I appreciate the tone of your answer. Cheers! Apr 27, 2015 at 13:48

An example which I like to use in class is Paul the Octopus (predicted 11/13 games in 2010 World Cup). Sports are useful sources of statistical examples in general, but I particularly like this one because it illustrates how using hypothesis testing in real life can lead to nonsense.

Students in the class could google their birth dates before class and come to class knowing the day of the week on which they were born (MTWRF). In class you then test the hypothesis that days of birth are equally distributed.

• No Sunday or Saturday?
– JRN
Oct 28, 2014 at 5:20
• @JoelReyesNoche No one wants to do all of the work of giving birth on their day off! Oct 30, 2014 at 13:53

On the softer side:

Example: Murder Trial

In a murder trial, the null hypothesis (innocent until proven guilty) is given by:

The defendant did not murder the deceased ($H_0$)

The alternate hypothesis is

The defendant did murder the deceased ($H_A$)

Implicit in $H_0$ is a range of possibilities that are consistent with $H_0$. It is up to the prosecuting council to produce evidence which lies outside this range of possibilities (e.g. the probability of the defendant being innocent AND owning the gun is small). If the prosecuting council succeed in producing such evidence (similar to a sample mean outside the confidence interval), the jury reject $H_0$ and the man is convicted.

A Type I error here is convicting an innocent man — which is considered worse than a Type II: acquitting a guilty man.

The problem is, in almost all recurring human interactions, almost nothing exhibits "random" (i.e. equiprobable) behavior, especially at the micro-level (and at the macro-level it is more likely to observe behavior represented by other distributions rather than the Uniform one -Normal and Pareto come easily to mind). So even though the distinction "fair/not fair" is very powerful and indeed engaging compared to say "test that "so and so" follows or not a Pareto distribution", it will be rather hard to find many convincing real world examples.

One of my Statistics professors when I was an undergraduate narrated to us how he had an argument with his wife about whether two different and long urban itineraries from their house to some relative they would visit often, were the same in terms of travel time (using car)... and he conducted an "equality of means" hypothesis test to check it.

Since the "average" is a fundamental statistical concept which is also intuitively known to everybody even before he hears the word "Statistics" for the first time, perhaps building examples and task-assignments around "equality of means" could be engaging and educationally beneficial. It would also have the students collecting the data themselves which is very useful as an experience.

For example, there may be the impression that Sally "talks more on her cell phone" than her friend Sue. But is this statistically validated? Record number of phonecalls and duration of each call (both are provided by the cell phones), for say 7 consecutive days (to include possible daily seasonality over a week), and conduct "equality of means" test for number of phone calls, and another for duration of calls. This specific example serves also to bring in the concept of independence - we expect that Sally and Sue talk to each other over the cell phones, being friends - so the two samples will include common observations. Should they? Should we exclude them? Why? etc

• Equality of means is a good idea, but it's obscured here by paragraph 1, paragraph 2, and paragraph 4 before the comma. Without those parts, I'd like the answer more.
– user173
Oct 31, 2014 at 2:10
• @MattF. You are right about par 2 and I just merged it, and I understand your point. But in general, instead of just making a suggestion, I prefer, to first relate my answers to the OP's ideas (hence the first part about the fair/unfair set up), and also, to provide some underlying rationale for what I am proposing (hence the first part of now par 3). Oct 31, 2014 at 2:45

### Games!

Meet your students where they are. Many students in their late teens are going to be into various forms of gaming activities. Find out what games are currently popular and come up with questions that allow them to use their existing interest and knowledge of the game to their advantage.

• Dungeons & Dragons: Sam is playing a figher-classed half-orc, rolling a D20 to hit. Seven out of ten of Sam's last rolls were above 15. The Dungeon Master suspects that Sam's D20 is unbalanced. Develop a test to investigate this.

• Pokémon: Mary's Rapidash has been using Fury Attack, which has an 85% Accuracy. Mary attempted to use this move seven times today, and all seven attempts missed. Mary tells you that she thinks her little sister hacked her game cartridge to make her miss more often. Develop a statistical trial to determine if this is plausible.

For the students in your class that are not familiar with the game, provide definitions. For example, you may define D20 as "a twenty-sided die" and Accuracy as "the percentage likelihood to hit".

• I like it. Here is the one I ran into most recently :) Path of Exile: Each alteration orb independently has a 1/1800 chance of creating an excellent item. You used 3000 alteration orbs and you still did not get an excellent item. You start to wonder whether there is really a 1/1800 chance of creating an excellent item, or if you were mistaken from the beginning. Explain why you should not use these 3000 trials to test the hypothesis, then develop a test to investigate the issue. Jan 28, 2023 at 17:37

I have students gather data from amongst themselves, or also from other students, and we have looked at things like sleep (significantly less than average?), study time (less than the assumed norm), difference between genders in coffee consumption, correlation between hours studied and hours worked (no), etc.

I did not do this fully rigorously; due to limitations of time, we first gathered data (both numeric and category-based), then went through such statistical tools as confidence interval, hypothesis testing and correlation/regression and had students use the relevant tool somewhere in the data. At the end, in the most ambitious take, I had them write a report and present their findings.