How to Deceive with Statistics

Correlation Does Not Equal Causation edition

A while back, there was a striking graphic circulating showing the correlation between educational outcomes (meaning test scores) and parental income (graph available here):


This graph makes it clear that wealthy people manage to buy good education for their children, while the poor are continually left behind. That is, if you think because test scores are correlated with parental income, then tests scores are caused by parental income.

In general, correlation can only mean one of five possible things (for a more general expression, just replace "parental income" with "X" and "test scores" with "Y"):

(A) Differences in test scores are caused by differences parental income. (sounds plausible)
(B) Differences in parental income are caused by differences in test scores. (not that plausible in this case, but with "X" and "Y" just as likely as A)
(C) Differences in test scores and differences in parental income each influence the other. (again, not that plausible here, but in general a perfectly reasonable explanation, which statisticians call endogeneity)
(D) Both differences in test scores and differences in parental income are caused by some third thing we haven't looked at yet. (we'll talk about this one below)
(E) The two are correlated purely by chance, and in reality are not related at all. (typically not a problem if you have enough data; this, by the way, is why Katie says that the plural of 'anecdote' is not 'data.')

Sorting out between A, B, and C is usually either based on theory and some appeal to the obvious (such as in this case, or when we say farm prices are affected by the weather, and not the other way around) or through more complex techniques called Instrumental Variables (IV) and Simultaneous Equations. E is typically ruled out by standard test statistics, which can be used to identify the probability that the variables look related, but really aren't. D is in many ways the trickiest to rule out. If the third thing is something you also have data on, you can control for its possible influence using a technique called Multiple Regression. If the third thing isn't observable, though, other tricks must be used.

In the case of income and test scores, it is unclear whether the former causes the latter or whether they are both caused by some manner of inheritable talent, capability, intelligence, personality traits, or what have you. Although there are some measures of some of these (such as IQ scores for intelligence), without including full genetic information we cannot use Multiple Regression to rule out the influence of inheritable capacities.

Unless, that is, we can observe test score where some children share the genetic information of the parents, while others don't. The best trick to use here would be to study the effect of parental income on tests scores of adopted vs. non-adopted children, as is discussed here:


The study providing this graph also controls for a variety of other possible explanations. The main result is that there may be a small effect of income (as opposed to genetics) on test scores, but with the data available we can't confidently say the relationship is there.

So, let that be a lesson to you. Just because there is a correlation doesn't mean there is a causal link. Of course, if there's no correlation in the first place...