# Impressive common misleading interpretations in statistics to make students aware of

Statistics are used everywhere; politicians, companies, etc. argue with the help of statistics. Since calculations are needed for the interpretation of statistics, such things should be taught in mathematics in school.

What are the most impressive common misleading interpretations of statistics that students should be aware of?

-
See also What are common statistical sins? on Cross Validated. –  Nick Stauner Apr 6 '14 at 6:59
several books on this see eg how to lie with statistics Huff –  vzn Apr 7 '14 at 4:00
I see this question is tagged secondary education, but it applies equally to tertiary education. –  J W Apr 7 '14 at 7:47
I protect this for now; please feel free to contact me if you prefer it undone. –  quid Apr 9 '14 at 14:22
I am not allowed to answer, but someone should mention the Will Rogers phenomenon; often, if you have two sets of numbers, you can increase the averages of both just by moving things from one to the other. This has relevance in medicine. Alternatively, open any newspaper and you can find an avalanche of bad statistics. For example, a recent news story claims that it's "shocking" that the number of women over-50 in the UK giving birth has doubled in four years... to 154. –  Flounderer Apr 10 '14 at 22:49

Here are two well known examples:

1. If someone tests positive for a rare disease (say its prevalence is 1 out of 100,000) with a test that has a 1% false positive rate, it is tempting to say that we are 99% sure they have that disease. This isn't true if you go through the numbers; they probably don't have that disease and are a false positive. (Bayes)

2. If you look at a list of cancer rates by county, you see that counties with the lowest rates of cancer tend to have a much lower population than average. Students will speculate all sorts of reasons for this - "healthier country living" etc. But you can also look at the counties with the highest cancer rates. You find they too are the least populated. If you show that to students first they will have all sorts of reasons why that makes sense. But what is really going on is that the standard error is larger for smaller samples. Standardized test results from schools show the same effect. I heard the Gates Foundation invested millions in small high performing schools before realizing that this effect was in play.

Here is a great article with a much better explanation of the first error and a lot of other examples of statistical confusion: http://web.mit.edu/5.95/readings/gigerenzer.pdf

EDIT: I recently discovered that p-values are a lot more subtle than I had thought. The Wikipedia page for p-values lists 6 common misconceptions, most of which I had. It references this article on twelve common misconceptions about p-values.

-
Putting the numbers for #1 here for others: the probability that someone has the disease and tests positively is $\frac{99}{100} \frac{1}{100000} = 0.0000099$. The probability that somoene doesn't have the disease and test positively is $\frac{1}{100} \frac{99999}{100000} = .0099999$. So, if someone tests positively, the probability that they actually have the disease is $.0989 \%$ –  Michael T Apr 6 '14 at 16:45
@MichaelT, these are wrong numbers. Noah said nothing about sensitivity of the test (fraction of times it is correct for those who have the disease), so you cannot compute the first probability at all here. Realistically, most cheap tests are designed to have a rather lousy sensitivity but high specificity: if somebody has a disease, the test MUST find it; more sophisticated tests may follow, but the initial screener should be designed in such a way as to never miss the case, at the expense of producing a lot of false positives. –  StasK Apr 7 '14 at 14:06

Anscombe's quartet is pretty good:

All four of these sets have almost identical mean and variance for both x and y coordinates, correlation, and best-fit linear regression. But they're obviously very different!

-
There are a number of misconceptions and counterexamples mentioned in this Cross Validated thread Datasets constructed for a purpose similar to that of Anscombe's quartet –  Glen_b Apr 18 '14 at 3:39

Sally Clark (http://en.wikipedia.org/wiki/Sally_Clark) was convicted in the UK of murdering both her infant sons, when in fact it is much more likely that they died of natural causes. The case against her was largely based on invalid statistical reasoning. The Royal Statistical Society made a statement about at at the time, which begins as follows:

In the recent highly-publicised case of R v. Sally Clark, a medical expert witness drew on published studies to obtain a figure for the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having some of the characteristics of the defendant's family. He went on to square this figure to obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This approach is, in general, statistically invalid. It would only be valid if SIDS cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong a priori reasons for supposing that the assumption will be false. There may well be unknown genetic or environmental factors that predispose families to SIDS, so that a second case within the family becomes much more likely.

After more than three years in prison Sally Clark was released following a second appeal, but she died of alcohol poisoning a few years later. This is a very sad but instructive story.

-

To summarize the Berkeley Admissions example: in 1973, 43% of men applying to graduate school at Berkeley were admitted, but only 35% of women. But, broken down across the six departments, women either did better than men, or the difference was not significant. The paradoxical result appeared because women were more likely than men to apply to the most competitive departments.

Benoît Kloeckner's answer mentions some other problems that arise from averaging percentages.

-
I wondered if this one was already mentioned. I have seen it used often to explain declines in, e.g., SAT scores. For example, see ets.org/research/policy_research_reports/publications/report/…. –  Benjamin Dickman Apr 6 '14 at 9:28
While this is a paradox to be sure, the example is not without statistical meaning. Berkeley admitted a smaller proportion of applicants to programs which attracted more female interest. Was the "competitiveness" inevitable or a consequence of bias? It may be a useful measurement of something, with proper care. –  Potatoswatter Apr 7 '14 at 6:56
An excellent illustration of the politically fashionable but logically invalid use of statistics to "prove" discrimination. –  OrbWeaver Apr 9 '14 at 15:31
An attractive interactive visualization of Simpson's Paradox can be found at vudlab.com/simpsons –  Mark S. Feb 28 at 22:55

A book I remember has the title "the egg-laying dog". The titular dog enters a room where we placed 10 sausages and 10 eggs. After a while the dog leaves the room, and we observe, that the percentage of eggs relative to the sausages increased, so we conclude that the dog must have produced eggs.

It's easy to spot the mistake in the above example, because the image of a dog laying eggs is absurd. However, consider the following case: a few decades ago a new medicine against heart diseases was developed. It worked well. However, 10 years later someone observed, that the rate of people dying of cancer was much higher among those who have been treated with the new medicine, in fact, the rate of cancer increased by a significant margin. Mass hysteria ensues: the new medicine causes cancer! Bans are being issued, companies are sued, etc. until a better look at the statistics showed that the situation was exactly the same as in the case of the egg-laying dog: people are not immortal, and sooner or later they tend to die of something. As fewer people died of heart diseases, they died of other causes years or decades later, and cancer, being a leading cause of death especially among older people, was one of them. The new medicine was not causing cancer at all, it just decreased the rate of another disease.

Other interesting, commonly occurring examples:

• Regression toward the mean: people, especially bosses tend to think that scolding people when they perform badly improves their effectiveness and complimenting them when they perform well decreases it. It's easy to see the problem. Take a 6-sided die and start throwing it. Every time you throw a 1, scold the die why did it give such a bad result. Observe, that after your scolding, in over 80% of the cases, the result of the next throw was better. Was it because of the scolding?
• Improper scaling in graphs. A common election tactic, you put two bar graphs next to each other, one very low for your opponent and one very high for yourself. What people tend to miss, is that the values don't start from zero. In fact you created 26857 new jobs, while your opponent only 26819. Not a big difference, but if you start the graph from 26800, it seems quite large.

While not strictly statistics-related, it's worth to mention the "fallacy fallacy". If someone uses a fallacy to prove or defend a statement, this fact alone is not a proof that the statement is wrong.

-
This is an incredibly high-quality first post to a stackexchange site. I see you're not new to the party, but welcome to this site anyway! –  Chris Cunningham Apr 9 '14 at 14:47

Percentages are a source of many, many, many common mistakes.

One that is very common is believing that percentages can be added. An example: one of our presidents increased its salary by 172%; the next president decreased the presidential salary by 30%. It was commented that compared to the salary before the raise, it was still a 142% increase.

Another one is to confuse 300% of a price and a raise by 300%. One of our former ministers made this mistake on twitter (while in office), sticking by it when corrected.

Another mistake, which is not really about percentage, is to invert roles in ratios when considering the correlation between two characters. E.g.: "30% of convicted criminals have purple hairs" is sometimes translated into " 30% of purple haired people are convicted criminals".

-
An even worse example with percentages. Quote of a danish politician some years ago, addressing Folketinget (apart from being translated, I have paraphrased a bit since I could not find the exact quote again, and the numbers might have been slightly different): "30% of danish men and 35% of danish women use the libraries. Now, usually we need to be careful when adding percentages, but in this case it is OK ["tør jeg godt"]. So this means that 65% of danes use the libraries, which is not so bad". The next speaker started with "Now, we are of course not here to tech each other math, but..." –  Tobias Kildetoft Apr 8 '14 at 7:31

Just two (now three, see below), to whet the appetite. Stating the mistakes:

1. "Correlation implies Causation": it doesn't. The finding of statistical correlation between two variables may strengthen a pre-existing theoretical/logical argument of existing causal links. But it may also reflect the existence of an underlying third variable that affects both and thus creates the correlation. When correlation is unexpectedly found, it indicates the possible existence of hitherto unknown causal links and should initiate a deeper (and not necessarily statistical) investigation -but it does not imply causation from the outset.

2. "If we take a larger and larger sample from a population, its distribution will tend to become normal (Gaussian) no matter what it is initially": it won't. The Central Limit Theorem, the misreading of which is the cause of this mistake, refers to the distribution of standardized sums of random variables as their number grows, not to the distribution of a collection of random variables. Alternative statement of the mistake: "Everything has a bell-shaped distribution" -let alone the fact that the normal distribution does not always look so bona fide bell-shaped.

The research paper Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education from Ana Elisa Castro Sotos et al., reviewing research papers on the matter can be downloaded from

I am adding a third one, which is really dangerous, since it relates in a more general sense to reasoning and inference, not necessarily statistical inference.

3."My sample is representative of the population": it isn't. Ok, it may be, but you need to try hard to achieve that (or to be lucky), so don't take it as a given. It may look an "ordinary" day to you, with nothing special, but this does not mean that it is a representative day. So counting during just this one day, the numbers of red and of blue cars passing outside your window, won't give you a reliable estimate of the average number of red and blue cars, or of the average proportion of red and blue cars, or of the probability that the cars will be red or they will be blue per day... This is sometimes called, sarcastically, "the law of small numbers" (but the Poisson distribution is sometimes also called that), and it points out the pitfalls of doing any kind of inference based on too little information, persuading yourself that this information is nevertheless "representative" of the whole picture, and so it suffices to reach valid conclusions. People do it all the time, even when statistics do not appear to be involved. Fundamentally, it has to do with the difficulty we have of understanding and accepting the phenomenon of random variability: it does not need a reason to occur, it just occurs (at least given our current state of knowledge).

-
Too often I see "XYZ linked with ABC disease" in the media, often making it seem a causation. –  Ramchandra Apte Apr 6 '14 at 8:10
@RamchandraApte Reminds me of this comic: phdcomics.com/comics.php?n=1174 –  Markus Klein Apr 6 '14 at 9:10
@MarkusKlein Very amusing comic! –  Alecos Papadopoulos Apr 6 '14 at 10:15
My favorite example for correlation doesn't imply causation: Ice cream consumption causes drowning accidents! The more ice cream is consumed in a month, the more drowning accidents happen. They must be linked! No, they are both independently linked to the weather. When the weather is hot, more ice cream is consumed and more people go swimming and drown. Both are influenced by the same third variable, but one doesn't influence the other. –  Philipp Apr 6 '14 at 19:37
Can't help: xkcd.com/552 ... –  mbork Apr 6 '14 at 19:52

Multiple hypothesis testing is a common one.

Let's say you run a study where you try to link some genetic marker to cancer rates. You look at perhaps 80 different genes and see if any of them have a correlation with occurrence of cancer. Lo and behold, one does! With p-value = 0.03! You conclude that there is a strong correlation (and seek to prove causation since you know the difference).

Seems pretty reasonable, but here's an analogous example: Let's say you want to find out if any of 80 people have the ability to predict the future. You ask each of them to predict 6 coin flips. And one predicts all 6 correctly! The p-value of this occurrence is 0.03 again! The stats seem to reject the null hypothesis that "John Doe cannot predict the future." But obviously this individual just got lucky. Your null hypothesis should be "someone can predict the future" but your p-values don't give any data about this.

There are a number of ways to adjust p-values because of this. The simplest (but not very powerful) of these is Bonferroni correction.

This is perhaps one of the most common violations of statistics in published academic papers.

-
xkcd.com/882 –  naught101 Apr 10 '14 at 6:24
It is also very hard if not impossible to detect on review. –  Richard Mar 24 at 20:33

Sometimes extreme sample bias. Here is an example (numbers made up, but realistic): In some country with a population of 100 million people, every year 100 people are bitten by poisonous snakes and 50 of these die. Every year 50 people are given treatment against snake bites, and 10 of these die (40 die without getting treatment).

Your chances of dying from snake bite if you are not given treatment is 40 in 100 million or 1 in 2.5 million. Your chance of dying from snake bit if you are given treatment is one in five. Clearly you should strongly refuse snake bite treatment.

Here the error is quite obvious, but there are medical situations where something similar happens. For some medical condition, there are two medications. One is more effective but may increase your blood pressure. The other is slightly less effective but won't increase your blood pressure. If your doctor sees that you have high blood pressure, you will be given the second medication to avoid a risk of the blood pressure getting too high. Now if you examine the statistics, people given the second medication will statistically end up with higher blood pressure, and totally wrong conclusions can be drawn.

-

Unfortunately, it is in German, but the book Angewandte Statistik: Eine Einführung für Wirtschaftswissenschaftler und Informatiker by Kröpfl, Peschek and Schneider contains many typical mistakes that you can make.

My favorite example is that you can show a strong geographic correlation in Germany between the number of stork nests and the number of newborn children.

-
Since you've mentioned a German book, there is also a German webpage www.unstatistik.de which deals with very specific examples from current newspaper storys (That's how the question came up, since I was looking for more common issues than on that site). –  Markus Klein Apr 5 '14 at 20:36

If a coin is biased to land heads with probability $p$ and $(a,b)$ is a $95\%$ confidence interval for $p$ then $p$ is in $(a,b)$ with probability $95\%$.

-
Even some scientists/doctors have this misconception, as I recently discovered. –  JAB Apr 8 '14 at 16:59
They're really the same thing, it's just a difference of philosophy, not results, and scientists and doctors are all about results. If you create multiple 95% CIs of a known p, then you will notice that p will be within about 95% of them. Hence, p is in (a,b) with probability 95%. –  Lukasz Apr 10 '14 at 12:30
Can you explain this more fully, and how it could lead to a problem? –  Richard Mar 24 at 21:02

I think, a very commong example are newspapers writing something like

The economic performance decreases: Last year's economic growth was 3%, now it is only 2.2%.

My interpretation: Although probably not knowing what a derivative is, they mix up decreasing of the first derivative (=economic growth) with the decreasing of the function (=economic performance).

-
At least here journalists have heard of derivatives (they talk about the "second deriviative" even, to mean indirect effects). I'm pretty sure they have no clue what they are about... –  vonbrand Apr 6 '14 at 2:29
@vonbrand No, the comment was from me as explonation how to think even that the normally don't know about derivatives. –  Markus Klein Apr 6 '14 at 14:21
+1 - I'm not sure how common it is, but here is another example of that error in this blog post. –  Andy W Apr 6 '14 at 19:20
I was under the impression that relative economic growth IS a measure of economic performance; even if absolute growth for one year is greater than for the previous, if the relative growth from one year to the next is not it counts as a decrease in economic performance as the previous growth is assumed to encourage future growth to some degree. (This does not consider whether or not such a measure is accurate or sustainable in the long term; that would be a separate, economically-focused topic for discussion.) –  JAB Apr 8 '14 at 16:58

Related to the answer of Mark, it is also a widely believed misconception (even among MDs) that if an AIDS test has 99% sensitivity, then someone testing positive is with 99% probability ill. Acually, the numbers can be extremely low... This is true for all illnesses which are erlatively rare in our society, see http://uhavax.hartford.edu/bugl/treat.htm#notions

-
@Noah already covered this. –  Nick Stauner Apr 6 '14 at 7:10

If you succumb to the temptation of ejecting, say, a 5-sigma outlier from a n=10 sample taken from what you believe to be a normally distributed source, then you are discarding 50% of the sample's information content. Not so harmless.

EDIT: I'll give it a go:

Low-probability events carry more information (a.k.a. surprisal) than high-probability events. E.g. "The building is on fire." carries more information than "The building is not on fire." This "information" is quantified by information theory, and appreciated by inductive inference. Solomonoff inductive inference is rooted in Bayesian statistics and is the optimal procedure for drawing strong-as-possible conclusions from available data, which is what frequentist statistical inference (as I understand its aims, vaguely) also tries to do, albeit less efficiently. Since information is a commodity valued by (Solomonoff) inductive inference, and since frequentist statistical inference seems to be a largely parallel pursuit of the same aims under a different theoretical framework, we would expect information to be a valuable commodity to statistical inference in general.

To get back to the point: Outliers are unlikely events (according to anyone tempted to dismiss them as fluke events, anyway) and therefore carry the most information within the sample (to that person) and therefore should be seen by that person as the most valuable members of a sample. The desire to "clean the data" and get rid of them is diametrically misguided (unless an explanation has been provided for them, in which case they would no longer carry much information anyway since under that explanation they are no longer such low-probability events).

-
Welcome to the site! Unfortunately I (and many other users of this site, and many people who teach introductory statistics) will be unable to parse that wikipedia page very easily. If you expanded your answer into something that was usable in a classroom setting, I think it would be extremely well-received! –  Chris Cunningham Apr 8 '14 at 21:40

The book The tiger that isn't is what you're looking for.

-
Would you mind adding an example from this book? –  András Bátkai Apr 7 '14 at 10:12
The random but nonrandom-looking clusters: books.google.es/… to books.google.es/…. –  Martín-Blas Pérez Pinilla Apr 7 '14 at 10:27

Years ago, there was a news story that coffee caused cancer. It was great, my opportunity to quickly tell everyone I ran into that I'd bet a year's pay this was a strong, but false, correlation. It was pretty obvious to me that for whatever reason, the coffee population had a higher smoking rate than the non-coffee drinkers. It took some time, but that was exactly what created the initial conclusion.

Similarly, TV watching has been correlated with a sedentary lifestyle. A fair correlation, but that shouldn't discourage one from mounting a TV on the wall in front of their treadmill.

-

Folks often think that an event having probability zero and being impossible are the same thing.

-
Care to clarify the difference? –  JoeTaxpayer Jun 14 '14 at 3:54
See the answers to this question. –  ncr Jun 14 '14 at 4:34
A typical situation is that one of infinitely many things must happen, but none of them has a non-zero probability. If you hit a dartboard, every single of the infinitely many points on the board has zero probability to be hit, but one of them must be hit. –  gnasher729 Jun 16 '14 at 10:40

## protected by quid♦Apr 9 '14 at 14:22

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.