# Hypothesis Tests in Students' Lives

Consider, in a basic statistics class, the difference between the following two examples:

A six-sided die is rolled twenty times. It comes up a "two" eight times, which seems unfair. The owner of the die claims that it is fair, namely that each face of the die is equally likely to come up. Complete a hypothesis test to assess this claim.

versus

A store manager assigns shifts to six different employees, including you. Twenty workdays into the job, you notice that you have been assigned the "bad shift" eight times, which seems unfair. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." Complete a hypothesis test to assess this claim.

The second one puts statistics directly into the real world, and seems (to me) to be more engaging.

Besides shift assignments from your boss, what other topics from students' lives are yield interesting statistical tests of claims?

Although it is not entirely accurate, for the purposes of this question, let's assume that the mean age of my students is 20 years, with a standard deviation of 1 year.

• Outside statistics lessons I have done exactly one test of significance, and that was to determine the fairness of dice. Several games (board, roleplaying, computer) use dice or equivalent random number generators, and a fair number of people play such games. – Tommi Oct 28 '14 at 8:03
• That's an engaging example. But a real manager would probably say: "I'm not trying to be unfair to you; I'm balancing a complex combination of store needs and people's preferences. We'll see what you can do in the different shifts and you'll see how you can rearrange things outside of work to accommodate this more easily, and after a couple months things will get better." – user173 Oct 28 '14 at 11:33
• @JoeTaxpayer I can't tell what your question has to do with mine. Is your comment an attempt at answering the question? – Chris Cunningham Oct 29 '14 at 0:38
• "It comes up a "two" eight times, which seems unfair" and "Complete a hypothesis test to assess this claim" -- there's a problem: the hypothesis of unfairness and the wish to test it is generated based on seeing the data you're about to use to test the claim. This impacts the properties of the test (it will have an actual significance level of much more than the nominal level, for example). The same problem affects the second example. These examples will be giving students a serious misconception. – Glen_b Apr 16 '15 at 11:17
• @Glen_b You should fix everything about what I've done, and let me award a bounty on your answer for improving my statistics instruction. Obviously my examples are not good. – Chris Cunningham Apr 21 '15 at 16:10

A) First, let's explore the problem in more detail:

The difficulty with both the presented scenarios is that they only decide to perform a test after noticing something about the data -- and then they test the hypothesis that the data led them to on the same data.

When this happens, the test no longer has its nominal properties.

To extend the more "boring" example (and I do it because the probability model is less ambiguous), imagine that ten thousand students each carry our the experiment "roll a six-sided die twenty times".

Further imagine that if a number comes up eight times or more, the students will says "that seems unfair" and so test it.

(i) If the dice are all fair, what proportion of students will say "that seems unfair"?

(ii) For the students who carry out a test, what's their type I error rate? (i.e. what proportion will reject the null?)

For (i) I just did three sets of simulations of 10000 such students, and got 6.74%, 6.93% and 6.62% getting enough of one face to say "that doesn't seem fair".

For (ii), we have to figure out what test they'll apply and what the nominal significance level they conduct it at might be.

I presume the students are specifically testing for too many or too few of the particular face they noticed a problem with. The sample size is too small to use a normal approximation on a proportions test; if we apply a binomial test, a two tailed test can't be symmetric (equal probability in each tail), but we can get 2.6% in the lower tail and 1.1% in the upper tail by rejecting with 0 or 8+ on the face tested (even though none of the people who test will actually have a zero, since our scenario didn't have students with 0 for a face trying to test it). So the overall nominal significance level is 3.7%.

But what's the actual rejection rate for people who carry out the test (the true type I error rate)? Well, under our rejection rule they all reject ... so it's 100%!

--

I recall working on a real-world problem like your scenario for someone once; the person's employer conducted random drug testing of drivers, and the particular person had been tested very many times -- he was certain another employee was making trouble for him, leading him to be tested much more often than randomly. If you work out the a priori probability he would be tested that many times, it seemed like something odd was going on. But if you worked out the probability that some employee would be tested that many times, it turned out to be not at all surprising. And of course to that employee it would look strange, even though nothing need be going on. ... and of course the unluckiest of all the (>100) employees would be the one asking what was up.

B) So what can be done? The easiest way is to slightly modify the example. Instead of saying 'that doesn't seem fair, let's test that data', we instead say 'that doesn't seem fair, let's specify exactly what we want to test, and then get some new data to test it with'. Then your examples will work fine!

For example:

A store manager assigns shifts to six different employees, including you. After a month on the job, you get the impression that you're getting the "bad" shift way too often. The manager claims that he assigns shifts randomly, namely that each employee has an equal chance of getting assigned the "bad shift." ... so you decide to collect data for the next 20 work days, and you find you're assigned the bad shift eight time. Complete a hypothesis test to assess this claim.

A similar modification can be made to the dice scenario

• Perfect. Thanks for your efforts. I know this kind of thing must really bother experts in statistics, and I appreciate the tone of your answer. Cheers! – Chris Cunningham Apr 27 '15 at 13:48

An example which I like to use in class is Paul the Octopus (predicted 11/13 games in 2010 World Cup). Sports are useful sources of statistical examples in general, but I particularly like this one because it illustrates how using hypothesis testing in real life can lead to nonsense.

Students in the class could google their birth dates before class and come to class knowing the day of the week on which they were born (MTWRF). In class you then test the hypothesis that days of birth are equally distributed.

• No Sunday or Saturday? – Joel Reyes Noche Oct 28 '14 at 5:20
• @JoelReyesNoche No one wants to do all of the work of giving birth on their day off! – Steven Gubkin Oct 30 '14 at 13:53

The problem is, in almost all recurring human interactions, almost nothing exhibits "random" (i.e. equiprobable) behavior, especially at the micro-level (and at the macro-level it is more likely to observe behavior represented by other distributions rather than the Uniform one -Normal and Pareto come easily to mind). So even though the distinction "fair/not fair" is very powerful and indeed engaging compared to say "test that "so and so" follows or not a Pareto distribution", it will be rather hard to find many convincing real world examples.

One of my Statistics professors when I was an undergraduate narrated to us how he had an argument with his wife about whether two different and long urban itineraries from their house to some relative they would visit often, were the same in terms of travel time (using car)... and he conducted an "equality of means" hypothesis test to check it.

Since the "average" is a fundamental statistical concept which is also intuitively known to everybody even before he hears the word "Statistics" for the first time, perhaps building examples and task-assignments around "equality of means" could be engaging and educationally beneficial. It would also have the students collecting the data themselves which is very useful as an experience.

For example, there may be the impression that Sally "talks more on her cell phone" than her friend Sue. But is this statistically validated? Record number of phonecalls and duration of each call (both are provided by the cell phones), for say 7 consecutive days (to include possible daily seasonality over a week), and conduct "equality of means" test for number of phone calls, and another for duration of calls. This specific example serves also to bring in the concept of independence - we expect that Sally and Sue talk to each other over the cell phones, being friends - so the two samples will include common observations. Should they? Should we exclude them? Why? etc

• Equality of means is a good idea, but it's obscured here by paragraph 1, paragraph 2, and paragraph 4 before the comma. Without those parts, I'd like the answer more. – user173 Oct 31 '14 at 2:10
• @MattF. You are right about par 2 and I just merged it, and I understand your point. But in general, instead of just making a suggestion, I prefer, to first relate my answers to the OP's ideas (hence the first part about the fair/unfair set up), and also, to provide some underlying rationale for what I am proposing (hence the first part of now par 3). – Alecos Papadopoulos Oct 31 '14 at 2:45

On the softer side:

Example: Murder Trial

In a murder trial, the null hypothesis (innocent until proven guilty) is given by:

The defendant did not murder the deceased ($H_0$)

The alternate hypothesis is

The defendant did murder the deceased ($H_A$)

Implicit in $H_0$ is a range of possibilities that are consistent with $H_0$. It is up to the prosecuting council to produce evidence which lies outside this range of possibilities (e.g. the probability of the defendant being innocent AND owning the gun is small). If the prosecuting council succeed in producing such evidence (similar to a sample mean outside the confidence interval), the jury reject $H_0$ and the man is convicted.

A Type I error here is convicting an innocent man — which is considered worse than a Type II: acquitting a guilty man.