Students in the basic statistics courses I teach often learn a little bit of probability and then learn hypothesis testing. The core concept that ties the course together is the p-value, but most students never adequately connect the p-value to the earlier topics in probability.
I would like my students to have an "aha moment" about the p-value by completing an assignment that has the following outline:
Question 1: Assume [Situation X]. Calculate the probability of [unlikely data Y] occurring.
Question 2: Complete a hypothesis test of the claim "We are in the situation [Situation X]," given the data [unlikely data Y].
In Question 1, the students would compute a probability. Then in Question 2, the students would find that the p-value is exactly the same thing as their answer to Question 1. This would illustrate what the p-value means precisely.
Here are some almost-answers to my question that do not satisfy me:
Situation X is "You have a fair coin." When students do the first question to compute a probability assuming a fair coin, they use the binomial distribution. But then inside a hypothesis test, students use the normal approximation to the binomial distribution, so the probabilities are not equal and the exercise does not work.
Situation X is "You have a normally distributed random variable with a known population mean and standard deviation." The problem here is that when you go to the second question, you have to awkwardly declare that you are absolutely sure about the distribution and the population standard deviation, and that the hypothesis test is only the test of a claim about the population mean. This seems really unrealistic to me.
Thanks for your insight!