This Could Have Been Three Posts...

... if I had time to provide decent comments on each item, that is.  Instead of delving into my thoughts about the following links, though, I'll just put them up Marginal Revolution style.

Psychohistory, the sci-fi version of what I do for a living. For those with some knowledge of the history of macroeconomics, you'll notice that as early as 1942, Asimov saw the importance of the Lucas Critique, something economists didn't latch on to until 1976.

while one cannot foresee the actions of a particular individual, the laws of statistics as applied to large groups of people could predict the general flow of future events. [...] The character responsible for the science's creation, Hari Seldon, established two axioms:

• that the population whose behaviour was modeled should be sufficiently large
• that the population should remain in ignorance of the results of the application of psychohistorical analyses

That last bullet point is the one macroeconomists took nearly 35 years to figure out.

A mind-boggling "How to Deceive with Statistics" entry. The nature of averages suggests that your friends are cooler than you. The money quote:

Let’s imagine a small department offering three courses for the semester. One is a survey course with 80 students, one an upper-level course with 15 students, and one a seminar with five students. Now what is the average class size? Clearly, it is (80 + 15 + 5)/3, or 33.3 students. This is the number the department is likely to publicize.

But once again, let’s adopt the perspective of the average person and reexamine these numbers. Eighty of the 100 students find themselves in a class with 80 students, 15 find themselves in a class of 15 students, and five in a class of five students. Thus, the average student’s class size is (80 × 80 + 15 × 15 + 5 × 5)/­100, or 66.5 students. This number is less likely to be publicized by the department.

Did that just blow your mind? It did mine.

Theories of International Relations and Zombies. How is this *not* something I need to read?  A nice discussion by the author is available here. Max Fisher at the Atlantic Wire, capturing a lot of why I find zombies so fascinating, has this to say:

[...]the beauty of zombie theory is that it applies too [sic] all sorts of emerging trans-national security threats, including those we have yet to anticipate or imagine.

Yes. Zombies are really just a proxy for disasters and crises of a scale and nature we can't quite foresee.  Namely, the zombie apocalypse.  If someone wants to buy this book for me, I will put a detailed review up on this blog with a shout out / link to the website of your choice.

How to Deceive with Statistics

For this Labor Day post, I'm letting Alex Tabarrok of George Mason University handle this one for me:

Notice that nearly 30% of the smallest decile (10%) of schools were in the top 25 at some point during 1997-2000 but only 1.2% of the schools in the largest decile ever made the top 25.

Sounds like small schools are proven to be awesome! Oh wait, we've got a problem:

If for random reasons a few geniuses happen to enroll one year in a small school scores jump up and if a few extra dullards enroll the next year scores fall.

Thus, for purely random reasons we would expect small schools to be among the best performing schools in any givenyear.  Of course we would also expect small schools to be among the worst performing schools in any given year!  And in fact, once we look at all the data this is exactly what we see.

Tabarrok provides this really nice graph:

which shows that smaller schools tend to vary more, but on average they're not really any better than big ones in terms of test scores. We could argue that test scores are a bad metric of education quality, and that small schools are better for other reasons. But then we'd better not be pointing to test scores as the reason for supporting small schools.

This just goes to show that looking at extreme cases on one end of the spectrum only will almost always be misleading. This is one of the big reasons for the maxim "'Data' is not the plural of 'anecdote.'" Just remember that it's possible to reduce statistics to the level of important-sounding anecdotes if you do it wrong. Thanks to Marginal Revolution for the reminder.

How to Deceive with Statistics

Someone Else Does My Job Edition

I realize I haven't talked about statistical errors in a while, but nothing readily accessible to the layman has really come up. That is, until this thorough article discussed a recent article on the topic of children and cow's milk:

Mothers who feed their babies cow’s milk in the first 15 days of life may be protecting their children from dangerous allergies later on, says a new study.

"O RLY?" the author of the above article said upon reading the above sentence. After all:

[...] given they claim the exact amount is unknown, making a suggestion to "give a single bottle daily" is highly unusual; not least because it contradicts worldwide recommendations which are based on extensive evidence.

Not given to trusting Prof. Yitzhak Katz of Tel Aviv University’s Department of Pediatrics, Sackler Faculty of Medicine---at least not as a default position---she (I think the author is a she) decided to look up both Professor Katz and the original study.

On Dr. Katz, she found he has received funding from the Israel Dairy Board, which presumably could benefit from adding the 0-6 months demographic to their target audience. In other words, everything he says is probably a lie, right? Well, I don't recommend going as far as the author on this. Partially this is because while the Dairy Board might benefit from an increase in sales to newborns, inducing health concerns and possible allergies later on means they'd lose a lot of revenue to the soy folk, so I'm skeptical that they are really pushing junk science. Follow the Money is good advice, but it not only tells you when someone has an incentive to lie, it also says when they have an incentive to be careful about which lies they tell.

But I also think we should be careful about dismissing Dr. Katz for his Dairy Board connection because this is what the debate kids call an ad hominem attack: namely, we're complaining about Katz's credentials, but not dealing with his argument. In a very real sense, if the science works, the science works, regardless of who paid for it. Follow the Money lets us know when we should be extra perceptive to things like wording, but it doesn't give us license to just dismiss evidence. Fortunately, although the article's author takes a hard line in her rhetoric, her practice is solid in this area. I do so love the magic words "So I decided to dig out the study."

The article is a great example of giving a close reading to the original sources cited in a piece of potentially biased news. It also highlights how a close reading and attention to the wording of claims is important when dealing with statistics. Really, there's not much I can add to this case study. Do read the whole article, please.

The bottom line, given what the study actually says, is I think accurate:

To suggest infants be given a bottle of cow's milk on the basis of this one study, is not only irresponsible, but really quite scary!

In other words, don't do this. The bottom line from the "don't be deceived" angle is this: Be wary of any news item that tries to convince you with the phrase "says a new study." The study might not say that at all; even if it does, it might not lay claim to the level of confidence the article suggests; and even if it does that, the study might not be very well done in the first place!

Creating Families

I've read about the hormone Oxytocin before in If you haven't read it, I heartily recommend the fantastic book Unprotected, by Anonymous, MD. The tag line is "A Campus Psychiatrist Reveals How Political Correctness in Her Profession Endangers Every Student," and it was a real eye opener for me, and provided a lot of things to think about in terms of how the human mind and body work and how we're told they work. One of the most interesting things I learned about is the hormone Oxytocin. 

The book talks about how when you have sex (or breastfeed), your body releases chemicals that make you form an emotional bond with the other person present. There is a new study out involving oxytocin that got more specific about what sort of reactions the hormone produces. From the abstract:

Humans [...] self-sacrifice to contribute to in-group welfare and to aggress against competing out-groups. [...] Here, we have linked oxytocin, a neuropeptide produced in the hypothalamus, to the regulation of intergroup conflict. In three experiments using double-blind placebo-controlled designs, male participants self-administered oxytocin or placebo and made decisions with financial consequences to themselves, their in-group, and a competing out-group. Results showed that oxytocin drives a "tend and defend" response in that it promoted in-group trust and cooperation, and defensive, but not offensive, aggression toward competing out-groups.

I have not read the full study, but this really got me thinking about how our bodies are an indispensable part of who we are. They have so much power over our identities that they can make other people into family. Our bodies can do that. Sex is not just an enjoyable activity, it literally changes who the other person is to us at the neuro-psychological level. More than that, it changes who we are at the physical level by making us people who want to take care of those we have these bonds with, and to keep out those we don't. It doesn't make us attack outsiders, but it does make us shun.

To me, this makes the issue of sex and marriage much more serious. Every time you have sex with someone, you are making them into family. When you leave that person (whether because you got divorced, broke up, etc.), your family is destroyed. Every. Time. And to think the average youth starts doing this in their early teens. Over and over again, familial bonds are built and broken. Is it any wonder our culture is filled with people who don't know how to love anymore? That every affection is interpreted as sexual, and sexual affection is interpreted as casual?

My classroom isn't just filled with kids from broken homes. It's filled with kids from dozens of broken homes each. How does someone cope with that kind of trauma, a trauma they might not even be aware of? Students might even seek solace in more pain, making the wound deeper and harder to heal, like drinking vodka in the desert. What hope do we have when the thing that can comfort us, the thing we most want--love and affection--is the thing that is killing us?

He who testifies to these things says, "Surely I am coming soon." Amen. Come, Lord Jesus!¹

Losing LOST

Word is that tonight is the series finale of LOST. Now, I haven't been watching since season four, but I loved the first couple seasons so much that this still seems like a sad time. Therefore, in honor of the show, I have a few videos reminding all of us why the show was so great. Hit the jump for a tribute.

How NOT to Deceive with Statistics

Fox "News" Is Bad At Its Job Edition

First of all, the least you could do before running something on the air like this --


-- is have someone just look at the screen and tell you if the numbers add up to 100% (plus or minus a 20% margin of error, perhaps?).

Second, if you've made a mistake like the one above, you might want to double check and make sure you don't do this:




That's right, apparently Fox's source managed to poll 193% of Republicans. Priceless.

How to Deceive with Statistics

Nightly News Slight-of-Hand Edition

If your goal is to get people nervous about something specific like Swine Flu, one of the most effective ways to do it is to talk about how scientists have been attempting to identify how many people have been affected by it, then start quoting some statistics about how many people have been affected by any and all diseases:

Visit msnbc.com for Breaking News, World News, and News about the Economy

Now, even if the statistics are actually reporting about those suffering from flu-like symptoms, they're still palming the actual swine flu card and playing something different. Boo, NBC Nightly News. Boo.

In other news, sorry I haven't been posting much lately. I've been trying to help out with my daughter more. I've also been developing my dissertation and doing some curriculum development for a couple distance learning courses. So that hasn't left much time for blogging. The curriculum development is almost done now, so I'm hoping to be back to a more regular schedule soon. I've got a few more posts scheduled for the next couple of weeks, so hopefully that will last until I can put things up more regularly.

How to Deceive with Statistics

Correlation Does Not Equal Causation edition

A while back, there was a striking graphic circulating showing the correlation between educational outcomes (meaning test scores) and parental income (graph available here):


This graph makes it clear that wealthy people manage to buy good education for their children, while the poor are continually left behind. That is, if you think because test scores are correlated with parental income, then tests scores are caused by parental income.

In general, correlation can only mean one of five possible things (for a more general expression, just replace "parental income" with "X" and "test scores" with "Y"):

(A) Differences in test scores are caused by differences parental income. (sounds plausible)
(B) Differences in parental income are caused by differences in test scores. (not that plausible in this case, but with "X" and "Y" just as likely as A)
(C) Differences in test scores and differences in parental income each influence the other. (again, not that plausible here, but in general a perfectly reasonable explanation, which statisticians call endogeneity)
(D) Both differences in test scores and differences in parental income are caused by some third thing we haven't looked at yet. (we'll talk about this one below)
(E) The two are correlated purely by chance, and in reality are not related at all. (typically not a problem if you have enough data; this, by the way, is why Katie says that the plural of 'anecdote' is not 'data.')

Sorting out between A, B, and C is usually either based on theory and some appeal to the obvious (such as in this case, or when we say farm prices are affected by the weather, and not the other way around) or through more complex techniques called Instrumental Variables (IV) and Simultaneous Equations. E is typically ruled out by standard test statistics, which can be used to identify the probability that the variables look related, but really aren't. D is in many ways the trickiest to rule out. If the third thing is something you also have data on, you can control for its possible influence using a technique called Multiple Regression. If the third thing isn't observable, though, other tricks must be used.

In the case of income and test scores, it is unclear whether the former causes the latter or whether they are both caused by some manner of inheritable talent, capability, intelligence, personality traits, or what have you. Although there are some measures of some of these (such as IQ scores for intelligence), without including full genetic information we cannot use Multiple Regression to rule out the influence of inheritable capacities.

Unless, that is, we can observe test score where some children share the genetic information of the parents, while others don't. The best trick to use here would be to study the effect of parental income on tests scores of adopted vs. non-adopted children, as is discussed here:


The study providing this graph also controls for a variety of other possible explanations. The main result is that there may be a small effect of income (as opposed to genetics) on test scores, but with the data available we can't confidently say the relationship is there.

So, let that be a lesson to you. Just because there is a correlation doesn't mean there is a causal link. Of course, if there's no correlation in the first place...

How to Deceive with Statistics

Graphical comparisons of unemployment edition

Here we have a much cited graphic that made its way to the floor of the House of Representatives (discussed, among others, by Matt Yglesias):


The chart compares the number of unemployed (on the vertical axis) during a recession to the length of time into the recession (months on the horizontal axis). The three lines are the recession beginning in 1990 (blue), the recession beginning in 2001 (red), and the current recession. This makes it look like our current recession is dangerous and scary to an unprecedented degree.

However, the comparison made here is deceptive for a couple reasons. (1) We have more people in the labor force now than previously, so raw unemployment numbers are not really comparable; and (2) the US has faced many more recessions previously than these two relatively mild recessions, so omitting the others exaggerates how unusual this recession is.

A more accurate and meaningful comparison is provided at The Curious Capitalist:



The current recession is the light blue line that terminates after 12 months. As we can see, this recession is still bad, but hardly unprecedented. Based on this graphic, we might expect the current recession to be a lot like the recession beginning in July 1981, which actually ended faster than either of the recessions cited on the House floor.

If we decide to go back further to compare against all post-WWII recessions, we'll notice that the current recession is more mid-range in terms of severity and probable duration. The corresponding graphic is produced by William Polley, which I don't reproduce here since it's pretty hard to read (especially for us color-blind folk).

The point of all this is not to say how bad or not-so-bad the current recession is. The point, rather, is to emphasize a couple of key details in interpreting statistics.

Point #1: What you're measuring on your axes matters. This is why I always tell my students (even in theory classes) to LABEL.

Point #2: Beware picking and choosing data points. I'll do on data mining and pretending samples are random at some point, but for right now, just now that you should always ask why the presenter of statistics is showing you the details they are.

How to Deceive with Statistics

Here we have a key example of using statistics and figures in a way that will make people think what you want them to think: (Courtesy of Greg Mankiw's Blog, an excellent resource for any curious individuals out there)


As you can see, the author wants the reader to realize our current economic crisis isn't anywhere near the problem that the Great Depression was. It is much more in line with other recessions.

The problem becomes clear if you notice the dates, though. Each recession, including the current one, shows a decline over a single year, while the Depression data shows the decline over 4 whole years. These statistics are not really comparable. A more meaningful comparison would be to compare the 1-year recessions with (A) the average decline per year of over the Depression, or (B) the decline in the worst single year of the Depression. For (A), this would be a decline of real GDP of about -7.3%: still twice the current crisis, but not factor of 8 implied by the existing graph. Even if (B) showed a more substantial difference, it wouldn't be the deceptively large contrast we see here.

I'm thinking this may be an ongoing series on this blog, so I'm going to start a new trend and label this post. Crazy.