In introductory statistics hypothesis testing, why not always use P-values?

Question

A test statistic is a measure of significance of a data set with respect to a claim.

My statistics textbook is insistent that the students should be able to interpret the test statistic in two different ways:

1. Convert the test statistic to a P-value, then decide whether that P-value is sufficiently low.
2. Find critical values, then decide whether the test statistic is more extreme than the critical values.

My understanding of how statistics is used "in the real world" is that researchers always convert their data and hypothesis into a P-value, then go from there. This is possible because we have technology that allows us to easily convert test statistics of all types into P-values.

I am tempted to stop discussing critical values entirely. If the students have access to the technology to turn everything into a P-value, why should we make them think about critical values? Does this topic have educational value, or is it simply a relic of a less-technological time, when tables of statistical values were the only way to operate?

Answer 1

When I learned statistics at university, the people teaching me made direct reference to p-values even when we were using critical values. The process of using critical values was something like this (for a significance level of 0.05):

1. Calculate the test statistic, suppose it's called $t$.
2. Look up the critical value of $t^\ast$ in the table. This is the test statistic that would have given me a p-value of exactly 0.05.
3. My test statistic is beyond $t^\ast$, so my p-value must be less than 0.05.

The key part was that final step. We would never just say "my test statistic is beyond $t^\ast$". We would always include the last part where we say "so p-value < 0.05". We never only compared our test statistics to critical values without also referring to p-values.

Some may argue that this is making it more confusing, because we are introducing an extra step in the reasoning. I disagree. I think it is better to have one more step in the reasoning than a whole separate approach which is listed as a separate approach. Indeed, students in my Maths Learning Centre are often confused by having two approaches and this explanation usually allows them to understand what's going on by unifying the two.

In fact, I was not technically taught to use the above approach at all, but instead to find the p-value as accurately as you can and then make your decision. If you were using the F-table you could usually only narrow your p-value down to "less than 0.05" or "more than 0.05". If you were using the t-table you could get a bit more accurate and get it to "between 0.05 and 0.01". If you were using the z-table you could be more accurate again. And if you used software you could get it to 15 decimal place accuracy.

So the one approach I use is this:

1. Calculate the test statistic.
2. Find the p-value as accurately as you can. If using tables, the best you can usually do is compare to the critical values to get a range for the size of the p-value.
3. Based on the size of the p-value relative to the significance level, make your decision.

It is worth noting that in some modern uses of statistics (eg testing for equivalence), the concept of p-value is difficult to tie down and so they talk about "rejection regions". Basically, if the test statistic is in a certain region then you will reject the null hypothesis.

In line with this, one course at my university does use two methods, but they use the "p-value method" and the "rejection region method". Instead of getting students to refer to simply refer to the size of the test statistic relative to the critical value, they get the students to define the rejection region, and then say whether the test statistic is in it.

I personally feel it is better for beginning students to only be expected to use one approach rather than two. It's possibly a good idea to mention the idea of rejection regions, but then always expect them to refer to p-values in your course.

Finally, you do still need the terminology of critical value for when you construct confidence intervals, but again I refer to the critical value for 0.05 as "the test statistic that would have given you a p-value of exactly 0.05". So it's still p-values really.

Answer 2

If we place value in understanding the meaning of a p-value, it seems almost essential to discuss the test statistic, and hence the corresponding critical value. (The p-value is the probability that the test statistic is as "extreme as it is" assuming the null hypothesis, etc.).

Could you do a good job of explaining to a student the meaning of a p-value without ever mentioning the test statistic? Once you relate the p-value and the test statistic, referring to the critical value is a natural part of the picture. I think the approach of "getting by" without understanding the meaning of a p-value is something more typical of a social science methods course.