The Reinhart-Rogoff Film Festival

You can observe a lot by watching.”  – Yogi Berra

 The biggest story in economics this year revolves around a study by Harvard economists Carmen Reinhart and Kenneth Rogoff (“RR”).  Their work has been credited (or blamed) for efforts around the world to reduce deficits and slow the growth in sovereign debt.  A recent critique of their work found a now-infamous Excel error and challenged RR’s methodology, prompting a furious debate between those advocating restraint in deficit spending and those calling for more spending to stimulate economic growth.  The spending doves have latched onto the real and perceived flaws in RR’s work to call for an end to “austerity” measures, while the deficit hawks argue that not enough has been done to curtail spending.

It seemed to me that this noise might be obscuring an opportunity to learn from the data behind the study.  What if we just let the data speak for itself, instead of using it as a weapon?  I have attempted to do that by animating the data. I think the results are interesting, if unlikely to change any minds.


RR’s 2010 paper, “Growth in the Time of Debt”, had been widely cited by policymakers for concluding that countries with a Debt/GDP ratio above the “threshold” of 90% experience dramatic declines in their growth rates.  There were several sections to the paper, but only one part has attracted controversy.  It concerns the experience, since World War II, of twenty advanced economies:

Australia (AU) Italy (IT)
Austria (AT) Japan (JP)
Belgium (BE) Netherlands (NE)
Canada (CA) New Zealand (NZ)
Denmark (DE) Norway (NO)
Finland (FI) Portugal (PT)
France (FR) Spain (ES)
Germany (DE) Sweden (SE)
Greece (GR) UK
Ireland (IE) US

For each of these countries and for each year from 1946 to 2009 (except for some gaps), RR compiled the ratio of sovereign debt to GDP, and the growth rate in real GDP.  There were nearly 1,200 such observations.

RR grouped the observations by Debt/GDP ratio:

  • 0-30%
  • 30-60%
  • 60-90%
  • Above 90%.

We will refer to the last category as “high debt”, and we will use “debt” as shorthand for the ratio of sovereign debt to GDP.

For each debt bucket, they found the mean and median rates of growth in real GDP.  Here are the mean calculations from RR’s Appendix Table 1:

Debt/GDP Ratio:





Average Real Growth Rate:





RR concluded that average growth falls considerably when the debt ratio exceeds a “threshold” of 90%.

This conclusion was recently challenged by graduate student Thomas Herndon,  and professors Michael Ash and Robert Pollin (“HAP”), all from the University of Massachusetts.  They disputed RR’s selection of data and RR’s method of weighting the data for average calculations.  Pretty dry stuff.  But HAP jolted economists, pundits, and spreadsheet users everywhere with their discovery that RR had made an embarrassing spreadsheet error.  RR had not included all of the countries in their average calculations.  This kind of error is common but easily prevented, as I wrote here.

The impact of the spreadsheet error depends on your perspective.  If you accept RR’s approach to selecting and averaging data, then the average growth for debt above 90% changes from -0.1% to +0.2%.  Not very troubling.  But HAP argue that when RR’s other mistakes (in HAP’s view) are corrected the average growth rate for high debt should have been 2.2% instead of -0.1%.  Now critics of austerity use the HAP paper to argue that high levels of debt should not be feared since they do not seem to dramatically slow growth rates.

In response, RR assert that the Excel error was minor and that their general conclusions remain valid.

Limited Vision

It seemed to me that this uproar has kicked up a cloud of dust, obscuring the reality behind the original study.  I wondered what could be learned by taking a close look at the actual data.

RR didn’t help matters by boiling down a thousand data points into a few buckets.  If the data were a distant galaxy, RR’s summary would be like a grainy speck seen through a weak telescope.

HAP deliver much higher resolution with a nice scatter chart in Figure 3 of their paper.  Here is my version of their chart (two points are missing because I limited the range of the y axis):


HAP call the observations “Country-Years.”  Each marker represents the debt ratio and growth rate for some observation.  But for any point, we can’t tell which country and which year.  Is it Ireland in 1950 or Finland in 2005?

I suspected that the country and the year actually matter.  Each country has its own culture, politics, resources, demographics, institutions and history.  On the other hand, each country is subject to the same laws of economics.  For example, hyperinflation has never ended well.

Different years may not be fully comparable.  The recovery from World War II might have been a fundamentally different era from our own.

Certainly, statistical analysis could help scrub away the noise of special circumstances if we had a sufficiently large data set.  To some critics, the data set in this case is inadequate.  There may be more than a thousand data points, but these correspond to only twenty countries and to even fewer business cycles.

Taking averages, as RR and HAP have done, is a way to wash out distortions caused by individual circumstances.  But perhaps it is useful to actually look at individual cases.  An anthropologist who visits real families might learn more than one who just studies an abstract family of 2.3 children and 0.6 dogs.

The scatter chart is like an overhead x-ray of a complex archaeological site.  Scattered shards of pottery from many different eras are jumbled together.  The x-ray is nice, but reassembling and categorizing the objects is better.

I decided to dig deeper into the data to compare the performance of the different countries.  I began by considering the data’s four dimensions: 

  • Country
  • Year
  • Debt/GDP
  • Real GDP Growth

It is possible to work all four dimensions into a single chart, but it is not very readable: 



Animating the data shows the four dimensions more clearly.  The following video shows the experience of all twenty countries from 1946 to 2009.


Like all great movies, there is too much to see in one viewing.  Here are some suggestions for watching the video:

  • Follow one or two countries at a time.
  • Watch for historic events that interest you.
  • Look for countries moving together.
  • Look for when changes in growth and debt seem to be related or unrelated.
  • Look for evidence that high debt causes slow growth or vice versa.
  • Look for evidence supporting or contradicting a threshold at 90% Debt/GDP.

To me, there is no obvious change in behavior when countries cross the 90% Debt/GDP line.  The “threshold” seems to be an artifact of how RR set up their debt categories.  On the other hand, much of the highest growth was experienced at low debt levels.

It is also clear that growth rates are often volatile and are affected by more factors than the debt level.

The RR Rogue’s Gallery

Now let’s look at the countries at the heart of the RR controversy.  RR identified eight countries that experienced more than 90% debt for at least one year in the 1946-2009 range (although Belgium was inadvertently excluded from their overall average due to the Excel error):


For three of these countries, the U.S., the U.K., and New Zealand, the only experience with high debt levels came after World War II.  All three pulled their debt down to more normal levels (although very slowly in Britain’s case).  I wonder if these cases of debt reduction can inform us about recent debt expansion around the world.  The old data are relevant only if a reversible process is involved.  Does leveraging look like deleveraging in reverse?  We will return to that question.

HAP pointed out that RR included only one year of New Zealand’s post war experience in their 90%+ calculation, distorting that country’s contribution to average growth.  HAP also objected to RR’s exclusion of Australia and Canada from the high debt group.  Apparently, the data for these two countries had not been available to RR when they wrote their paper.

After the War

The next animation highlights the countries that reduced their high debt levels after World War II.  This time we include Australia and Canada, for which fast growth seemed to helped them reduce their debt.  Only Belgium ever crossed back over the 90% level  (Belgium crossed repeatedly).  “Tails” show the previous five years for each country.


A Greek Tragedy

HAP objected to the way RR averaged the data.  RR averaged the growth rates for each country in each debt bucket, and then took the simple average of these country averages.  For example, the average growth rate for Greece’s nineteen years of high debt was 2.9%.  To RR, this was as important as the -2.0% growth rate for the U.S., which experienced only four years of high debt.

On the other hand, HAP contend that each Country-Year observation should carry equal weight.  In their view, Greece’s high debt growth rate is 4.75 times as important as the American rate.

HAP’s averaging method implicitly assumes that each Country-Year delivers the same amount of information.  To test this assumption, consider the next animation. It includes Greece and other countries whose debt levels were not the worst in the whole data set.


For the seven years from 2000 to 2006, Greece’s economy was remarkably consistent.  The Greek data points stayed within ranges that were small compared to the data set as a whole. 

Were these all truly independent observations that should be given equal weight in the analysis, like separate test tube experiments?  Is it just chance that these data points are clustered so closely together?  Or were they joined together in a single episode of economic history?    If so, treating each data point as an equally valuable source of information would be misguided.

It is ironic that HAP’s greater emphasis on the Greek experience would have made high levels of debt look more benign.  While it is true that Greece enjoyed some healthy growth along with high debt, no one would want to emulate Greece’s recent experience.

Japan in the English Channel

Finally, I want to share an observation that surprised me.  I noticed that as the U.K. recovered from its high debt load after World War II, its growth rate was almost always between 0% and 5%.  I also noticed that Japan’s growth was generally in the same range as its debt grew in the 2000s.  These are only two cases, but perhaps the processes of getting into debt and getting out of debt can be mirror images after all.


Former trader Bruce Krasting  argues that “Japan is all that the economists need to look for the evidence of the failure of More Debt is Better.”  I agree.

Summary of Observations

After studying these animations, my impressions are that:

  • There is no hard “threshold” of debt where economic performance suddenly changes.
  • The debt level isn’t always the strongest influence on the growth rate.
  • No method of weighting the data is without flaws.
  • Countries follow their own paths and great care should be taken in drawing conclusions from the experience of a few countries.
  • The shortcomings of Reinhart and Rogoff’s paper are not proof that high debt is benign.


I would like to thank Kirk Monteverde of and Michael Ash, one of the HAP coauthors, for making the RR data available to me in a form that I could use.

Copyright 2013.  All Rights Reserved.

This entry was posted in Bonds, Economics, Graphic Presentation, Visualization and tagged , , , , , . Bookmark the permalink.

3 Responses to The Reinhart-Rogoff Film Festival

  1. hobsonlane says:

    Really appreciate the difficult, tedious work it takes to create these animations and analyses. Only wish our leaders were as diligent.


  2. Pingback: The Reinhart-Rogoff Film Festival: A Deleted Scene | The Well-Tempered Spreadsheet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.