Lies, Damned Lies, and Statistics

Why you should always be willing to ask questions about what is presented to you.

I’m currently working on a post around Freakonomics’ latest podcast, “How Efficient is Energy Efficency?” I will have that post up sometime in the next few days, but in my research for the post I ran across an insight that I thought was important to share about how we present and challenge information. The podcast centers around the work of Arik Levinson, an Environmental Economist at Georgetown University. Two of his papers in recent years have focused on energy efficiency regulations for housing in California. The standard narrative is that because California enacted energy efficiency standards for all new houses starting in 1978, it has lead the nation in electricity reduction. This narrative is represented most clearly by the following graph:

Residential Electricity Per Capita. From Original Academic Article Here.

Residential Electricity Per Capita. From Original Academic Article Here.

Wow! Look at that difference and how dramatically California separated from the rest of the nation right in the 1970s!

People have gotten really excited about this graph. If you want to see how many groups have referenced it, take a look at the Google Image results for searching “California 1970s energy efficiency.” The National Resources Defense Council, Scientific American, and the World Bank (on page 215) are just a few. Each of these organizations has made California and their energy efficiency standards out as some major hero. I am not going to cast doubt on the accuracy of these numbers themselves and so, I believe, they display some useful truths about the nature of California’s energy consumption profile. However, no matter how much truth is represented in this graph, I believe that it serves to cloud the discussion rather than illuminate it. Think of all of the possible conclusions that are implied by this graph:

  • California is much more energy efficient than all of the other U.S. states.
  • California was identical to the rest of the U.S. in terms of energy consumption before about 1970.
  • California must have had some really effective policy changes in the 1970s to cause this change.
    • As a subset: There were some changes in energy efficiency policies for housing in 1978. That must be the culprit.
  • Finally, California is different from more than one other U.S. State in terms of per capita energy efficiency.

I love this last one, because the truth is, we can’t know any of those above conclusions, including it, based on the data presented in that graph. All we can know is this: California’s per capita energy use figures, since 1969, have been below the per capita energy efficiency use of the other 49 states. This is illustrated by the following graph from Arik Levinson:

Residential Electricity Per Capita with Top 5 Other Low-Growth States. From Original Academic Article Here.

This graph illustrates that while California is still in the top 5 of all states in electricity use, it is by no means an outlier and is not even leading the pack among U.S. states. California’s consumption is in fact above the average of its 5 other West-Coast brethren shown here. This graph tells us way more information than the original one (although still not the full story.) The original just serves as an illustration that in some situations, people can paint whatever picture they fancy out of a given data set. We should always be thinking about what questions are ignored by a certain graph or statistic and look at problems from more than one angle before drawing conclusions. I’d like to close with a fun exercise, where I have recreated the original graph (as closely as possible without the original data) with a hypothetical situation: Imagine a United States where per capita residential electricity consumption is broken up into two groups of states. California is joined by 45 other states with identical numbers to its current history over the past 50 years, but there are 4 rogue consumption states that have had record numbers for consumption in the last 50 years. It might look something like this:

Fictional Per Capita Residential Electricity Consumption by State

Fictional Per Capita Residential Electricity Consumption by State

But if we combine the 4 high polluting states with the 45 others which are equivalent to California into one average, we can produce the following graph:

Fictional Residential Per Capita Electricity Consumption, by State. Same data as previous graph.

Fictional Residential Per Capita Electricity Consumption, by State. Same Data as Previous Graph.

“Wow! Look how much better California is than the rest of the United States!” …I hope you’ve enjoyed this little exercise. Have a good weekend, folks.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s