Cambridge Journal of Economics 2013 1 of 23doi101093cjebet075

copy The Author 2013 Published by Oxford University Press on behalf of the Cambridge Political Economy Society All rights reserved

Does high public debt consistently stifle economic growth A critique of Reinhart and Rogoff

Thomas Herndon Michael Ash and Robert Pollin

We replicate Reinhart and Rogoff (2010A and 2010B) and find that selective exclu-sion of available data coding errors and inappropriate weighting of summary sta-tistics lead to serious miscalculations that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies Over 1946ndash2009 countries with public debtGDP ratios above 90 averaged 22 real annual GDP growth not minus01 as published The published results for (i) median GDP growth rates for the 1946ndash2009 period and (ii) mean and median GDP growth figures over 1790ndash2009 are all distorted by similar methodological errors although the magnitudes of the distortions are somewhat smaller than with the mean figures for 1946ndash2009 Contrary to Reinhart and Rogoff rsquos broader contentions both mean and median GDP growth when public debt levels exceed 90 of GDP are not dramatically different from when the public debtGDP ratios are lower The rela-tionship between public debt and GDP growth varies significantly by period and country Our overall evidence refutes RRrsquos claim that public debtGDP ratios above 90 consistently reduce a countryrsquos GDP growth

Key words Public debt AusterityJEL classifications E60 E62 E65

1 Introduction

In lsquoGrowth in Time of Debtrsquo Reinhart and Rogoff (hereafter RR 2010A for their working paper version and 2010B for their published paper) present a set of what they characterise as lsquostylised factsrsquo concerning the relationship between public debt and GDP growth RR summarise the overarching results of these papers succinctly

whereas the link between growth and debt seems relatively weak at lsquonormalrsquo debt lev-els median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise average (mean) growth rates are several percent lower (RR 2010A p 573)

Manuscript received 9 October 2013 final version received 12 November 2013Address for correspondence Robert Pollin Department of Economics University of Massachusetts Amherst

MA 01002 USA email pollineconsumasseduDepartment of Economics (TH) Department of Economics and Center for Public Policy and

Administration (MA) and Department of Economics and Political Economy Research Institute (RP) University of Massachusetts Amherst We thank Arindrajit Dube Stephen A Marglin and four anonymous CJE referees for valuable comments We also thank Carmen Reinhart and Kenneth Rogoff for sending us their working spreadsheet on 4 April 2013 and for their constructive responses to our initial HAP working paper (originally posted on 15 April 2013)

Cambridge Journal of Economics Advance Access published December 24 2013 by guest on January 1 2014

httpcjeoxfordjournalsorgD

ownloaded from

Page 2 of 23 T Herndon M Ash and R Pollin

To build the case that they had established a new set of stylised facts regarding public debt levels and GDP growth RR stress the robustness of their overarching findings to a range of countries and time periods They also stress the robustness of these findings to alternative measurement techniques and ways to categorise data

This paper presents a critical analysis of lsquoGrowth in a Time of Debtrsquo both the work-ing paper and the subsequent published version in terms of the methods used by RR the ways they executed their research approach and their research results To be more specific our paper is a narrowly gauged critical replication As is standard for such crit-ical replication exercises we maintain our focus on their two versions of their lsquoGrowth in a Time of Debtrsquo paper Aside from a few brief comments we do not attempt to integrate our discussion on lsquoGrowth in a Time of Debtrsquo into a broader survey of the lit-erature on public indebtedness and GDP growth or other related topics We do briefly consider some policy implications that follow from the results of our critical replica-tion in particular regarding austerity policies as practised in our contemporary period in the USA and Europe But again given this paperrsquos deliberately narrow focus as a critical replication exercise we do not attempt an extended discussion on austerity policies or any other related policy issue

The methods employed by RR in lsquoGrowth in a Time of Debtrsquo are non-parametric and appealingly straightforward They organise data for a range of countries and dif-ferent time periods based on the ratios of public debt relative to GDP in each country They present four data categories in terms of public debtGDP ratios le30 30ndash60 60ndash90 and gt90 They then compare average real GDP growth rates across each of the public debtGDP groupings Organising data in this way they identify a major non-linearity in the relationship between countriesrsquo public debtGDP ratios and their corresponding rate of GDP growth That is they find that average GDP growth rates vary only to moderate degrees as long as the level of public debt remains lt90 of its GDP for the countries in their sample However they find that these countriesrsquo aver-age GDP growth performance falls off significantly when their public debtGDP ratios exceed 90

In Table 1 we present RRrsquos key results on mean real GDP growth in advanced econ-omies between 1946 and 2009 As the table shows real mean GDP growth ranges between about 3 and 4 as long as the ratio of public debtGDP falls within either their 0ndash30 30ndash60 or 60ndash90 public debtGDP category However mean GDP growth collapses to minus01 for their sample of countries when the public debtGDP ratio exceeds 901

A necessary condition for establishing a stylised fact is that the calculations on which such facts are based are accurate and that the results of such calculations are robust across alternative reasonable methods of calculation Through our replication exercise we conclude that their findings are neither accurate nor robust We rather show that (i) selective exclusion of available data (ii) coding errors and (iii) inappropriate methods for the weighting of summary statistics have generated serious measurement problems that produce inaccurate figures on the relationship between public debt and growth among these 20 advanced economies both over the period 1946ndash2009 as well as in

1 The figures we present in Table 1 are taken from the first line of Appendix 1 in RR (2010A) We note that the data shown in the bar charts (Figure 2 in both RR 2010A and 2010B) do not match up precisely with the data they present in Appendix 1 of the 2010A working paper version Despite these differences in the figures they themselves present the overall finding is still consistent across the two versions of the paper

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 3 of 23

their long historical time period 1790ndash2009 As one critical case in point referring to the data from RR that we show in Table 1 when properly calculated the average real GDP growth rate over 1946ndash2009 for countries carrying a public debtGDP ratio greater than 90 is actually positive 22 not negative 01 as RR claim

More generally contrary to RR we find no evidence for a dramatic drop-off in average GDP growth when countriesrsquo public debt levels rise above 90 of their GDP We correspondingly refute the claim by RR that there exists a lsquohistorical boundaryrsquo that is robust across countries and time periods in which economic growth consist-ently falls off in a non-linear pattern when the public debt levels exceed 90 of GDP In fact as we show there is a major non-linearity in the relationship between public debt and GDP growth but that non-linearity is between the lowest two public debtGDP categories 0ndash30 and 30ndash60 a range that is not relevant to current policy debate

For the purposes of this replication we follow RR in assuming that the direction of causation in the relationship between public debt levels is that high public debt levels produce declines in average GDP growth rates In other work (see eg RR 2011 Reinhart et al 2012) RR acknowledge the potential for reverse causation This would occur primarily as a result of recessions which would raise public indebtedness by both reducing tax revenue and increasing public expenditures through countercyclical interventions However in the papers that we are replicating (RR 2010A 2010B) RR make clear that their analysis is organised around the premise that the primary direc-tion of causation runs from high public debt to slower GDP growth

We originally posted a preliminary working paper version of this study in April 2013 (to which we refer hereafter as HAP 2013) Our working paper produced an intense global debate through much of the spring of 2013 This was an outgrowth of the major influence that had been exerted by the RR research in both academic and policy-making circles in particular in providing an intellectual foundation in support of austerity policies in the aftermath of the 2007ndash09 financial crisis and subsequent Great Recession2 We refer briefly to this background in what follows Of particular importance for this current paper is that RR twice responded formally to the HAP (2013) critique in two lengthy New York Times articles and an Errata memorandum (RR 2013A 2013B 2013C) These responses have enabled us to focus our present discussion more sharply than was possible in our original HAP working paper

Table 1 Duplication of RR (2010) findings real average annual GDP growth at various public debtGDP ratios for 20 advanced economies 1946ndash2009

Public debtGDP ratios GDP growth rates

le30 4130ndash60 2860ndash90 28gt90 minus01

Sources RR (2010A 2010B)

2 As we discuss further below it is an indisputable fact that lsquoGrowth in a Time of Debtrsquo has provided a critical intellectual underpinning on behalf of austerity policies This is the case regardless of whether it was RRrsquos intention to exert this type of influence We do not attempt to discern RRrsquos intentions on this matter

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 4 of 23 T Herndon M Ash and R Pollin

Our discussion proceeds as follows Section 2 describes the influence that the RR research has exercised since it was first published in 2010 Section 3 the main body of this paper describes in detail the three problems we identify with RRrsquos research in their 2010 papers their selective data exclusions coding errors and inappropriate weighting methodology Focusing initially on RRrsquos mean GDP growth calculations with their 1946ndash2009 data sample we also show how these problems interact to mag-nify the effects on the calculation of real GDP growth relative to the impact of each separate factor acting alone We then show how the problems with their data account-ing and methodology generate similar distortions with their calculations of both the mean GDP growth figures for 1790ndash2009 and the median figures for 1946ndash2009

Based on these findings we offer new evidence on RRrsquos claim to have identified a robust non-linearity in the pattern of GDP growth once public debt levels exceed 90 of GDP We produce a range of evidence demonstrating that in fact there is no such non-linearity at the 90 public debtGDP boundary We also show that there is a non-linearity in the relationship between GDP growth and public debtGDP ratios But this non-linearity occurs when public debt levels range between 0 and 30 of GDP

In Section 4 we conclude the paper by recognising first the issues we raised in HAP (2013) in which RR have subsequently acknowledged mistakes We then summarise the major areas where they have not retracted their findings and show that these areas of ongoing dispute are all matters of considerable significance about which this cur-rent paper seeks to provide greater clarity

2 Public impact and policy relevance

As we noted above the RR research has exerted a substantial influence throughout the globe in shaping macroeconomic policy debates since its publication in 2010 In particular this work has provided a major empirical foundation in support of austerity policies aimed at reducing the high public debt levels that emerged in the aftermath of the 2007ndash09 global financial crisis and the subsequent Great Recession

More specifically according to RRrsquos web site3 the findings reported in the two 2010 papers formed the basis for Reinhartrsquos 10 March 2011 testimony before the US House of Representatives Budget Committee lsquoLifting the Crushing Burden of Debtrsquo4 and a Financial Times opinion piece lsquoWhy we should expect low growth amid debtrsquo (RR 28 January 2010) The key tables and figures have been reprinted in additional Reinhart and Rogoff publications and presentations at the Centre for Economic Policy Research and the Peter G Peterson Institute for International Economics (RR 2011A 2011B) A Google Scholar search for the publication excluding pieces by the authors them-selves finds more than 500 results5

Their main findings have also been widely cited in the popular media RRrsquos own web site lists 76 high-profile features including The Economist Wall Street Journal New York

3 httpwwwreinhartandrogoffcomrelated-researchgrowth-in-a-time-of-debt-featured-in [date last accessed 7 April 2013]

4 The testimony for the house committee hearing chaired by Paul Ryan is not linked to on their web site but can be found at httpbudgethousegovuploadedfilesreinharttestimony3102011pdf [date last accessed 8 October 2013] Reinhart also testified before the US Senate Budget Committee on 9 February 2010

5 A search on [Reinhart Rogoff lsquoGrowth in a Time of Debtrsquo-authorrogoff -authorreinhart] yielded 538 Google Scholar results on 7 April 2013

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 5 of 23

Times Washington Post Fox News National Public Radio and MSNBC as well as many international publications and broadcasts

Further RR (2010B) was the only evidence cited on the consequences of high public debt on economic growth in the 2013 US Federal Budget plan proposed by Republican Paul Ryan which was passed in the House of Representatives Congressman Ryanrsquos lsquoPath to Prosperityrsquo proposal reports that RRrsquos research lsquofound conclusive empirical evidence that gross debt (meaning all debt that a government owes including debt held in government trust funds) exceeding 90 percent of the economy has a significant negative effect on economic growthrsquo (Ryan 2013 p 78) George Osborne the UK Chancellor of the Exchequer and Olli Rehn the leading economic official of the European Commission are other leading policy makers who have frequently cited the RR work as significantly influencing their thinking Indeed Paul Krugman observed in June 2013 that lsquoReinhartndashRogoff may have had more immediate influence on public debate than any previous paper in the history of eco-nomicsrsquo (Krugman 2013)

Krugman wrote this comment as part of the intense global response that followed from the initial posting of our April 2013 working paper (HAP 2013) The fact that our critique even while still in the form of a preliminary draft elicited such a high level of worldwide interest offers further evidence of the major influence exerted by the RR research We provide as relevant some brief comments on our subsequent public debate with RR6

3 Replication

RR examine three datasets 20 advanced economies over 1946ndash2009 the same 20 economies over the long historical period 1790ndash2009 and 20 emerging market econo-mies from 1970 to 2009 We replicate the results only from the first two samples We focus primarily on the 1946ndash2009 time period for the advanced economies since these figures are clearly the most relevant to ongoing US and European policy debates The more recent data are also the most reliable since they entailed much less splicing together of data by RR from multiple sources that frequently used different statistical methodologies We examine the results reported by RR in terms of both mean and median figures

On their web site RR provide public access to country historical data for public debt and GDP growth in spreadsheets with complete source documentation7 However these publicly available spreadsheets do not include information on the exact data series years and methods used in their paper As such we were unable to replicate the RR results from the data they posted on their web site

In response to our request of April 2013 RR did provide us with the working spreadsheet that they used in producing the RR papers Through using their working spreadsheet we were able to approximate closely the published RR results This was how we were able to identify the selective exclusion of available data coding errors and inappropriate methods for weighting summary statistics

6 Our contributions to the debate and rejoinders to RR include Ash and Pollin (2013) Pollin and Ash (2013A 2013B) and Herndon (2013) See Harding (2013) for a summary of the debate as of May 2013

7 See httpwwwreinhartandrogoffcomdatabrowse-by-topictopics9 and httpwwwreinhartandro-goffcomdatabrowse-by-topictopics16 [date last accessed 7 April 2013]

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 6 of 23 T Herndon M Ash and R Pollin

31 Data gaps and selective exclusion of available data

RR designate 1946ndash2009 as their period of analysis for the post-World War II advanced economies Most differences between countries in their period of coverage are due to the starting year of each countryrsquos relevant dataset For example the US data series extends back to 1946 Outside the USA the series for some countries do not begin until the 1950s and that for Greece is unavailable before 1970 Overall nine countries are available from 1946 onward 17 from 1951 and all countries but Greece enter the dataset by 1957

There are some gaps and oddities in this 1946ndash2009 dataset For example the pub-lic debtGDP ratio series is unavailable for France for 1973ndash77 real GDP growth is unavailable for Spain for 1959ndash80 Austria experienced 273 and 189 real GDP growth in 1948 and 1949 (with both years in lower public debt groups) respectively and Portugalrsquos debtGDP jumps by 25 percentage points from 1999 to 2000 when the countryrsquos currency and the denomination of the series changed from the escudo to the euro

The longer time series 1790ndash2009 includes available years of data for the same 20 now advanced economies There is substantial variation in the availability of data with this longer series For example complete public debt and GDP data begin for the USA in 1791 in 1831 for the UK 1880 for France Germany and several other countries 1925 for Canada and 1932 for New Zealand

There are also significant gaps within the available time series for many of the countries In notes to their Table 1 of both 2010A and 2010B RR report that lsquoThere are missing observations most notably during World War I and II yearsrsquo Some gaps in the data do correspond to the two World War periods For example Belgium lacks data for 1914ndash20 and 1940ndash46 but there are also other gaps that are longer and not explained by RR Thus Denmark lacks public debtGDP data for 1914ndash49 The series for France is incomplete from 1932 to 1948 Nine countries both neutral and non-neutral in World War I include data for 1914ndash18 Only for the USA (1941ndash44) and the UK (1940ndash45) are data for World War II explicitly excluded from the spreadsheet calculations of mean and median growth by public debtGDP category

In our replication exercises we do not pursue the implications of these data gaps described above However we do examine further RRrsquos data exclusions concern-ing three countries which significantly affect their overall conclusions These are for Australia (1946ndash50) New Zealand (1946ndash49) and Canada (1946ndash50) At no point do RR either explicitly explain they why they chose to make these data exclusions or even indicate that they had done so The closest they come to considering the issue in either the 2010A or 2010B versions of their paper is in the following passage

Of course there is considerable variation across the countries with some countries such as Australia and New Zealand experiencing no growth deterioration at very high debt levels It is noteworthy however that those high-growth high-debt observations are clustered in the years following World War II (RR 2010A p 11)

In other words RR appear to justify their selective data exclusions because as quoted above these data lsquoare clustered in the years following World War IIrsquo when economic growth was high However in contrast with this reasoning as applied to the cases of Australia Canada and New Zealand RR include all four of the immediate post-World War II observations in which the USA was in the gt90 public debtGDP category In

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 7 of 23

three of these four years the US economy was contracting at the same time as it was in the highest public debtGDP category RR do not provide an explanation of their reasoning behind the decision to exclude Australia Canada and New Zealand in these years while these economies were growing rapidly but to include the USA which was contracting in three of the four relevant years8

In the case of Canada all five omitted years were in the gt90 public debtGDP cat-egory Those years were also the only ones in which Canada was in this highest public debtGDP category Mean GDP growth in Canada for the excluded years was 30 while median Canadian GDP growth for these years was 22 For Australia as well all five excluded years were in the highest public debtGDP category They were also the only years in which Australia was in this highest public debtGDP category RR note that Australia experienced lsquono growth deteriorationrsquo (RR 2010A p 11) during these years that RR chose to exclude from their dataset

The 1946ndash49 exclusions for New Zealand are of particular significance This is because New Zealand was in the highest public debtGDP category in all four of these excluded years New Zealandrsquos real GDP growth rates in those years was +77 +119 minus99 and +108 After RR chose to exclude the 1946ndash49 New Zealand data New Zealand then contributes only one year 1951 to the highest public debtGDP category RR report New Zealandrsquos real GDP growth in 1951 as being minus76 As we discuss below these data choices by RR regarding New Zealand have a substan-tial impact on their overall findings

32 Spreadsheet coding error

In addition to these deliberate data exclusions by RR a coding error in the RR working spreadsheet also unintentionally excludes five countries entirely (Australia Austria Belgium Canada and Denmark) from all parts of the analysis9 The error appears in the calculations of both mean and median GDP growth with the 1946ndash2009 sample as well as with the mean and median GDP growth for the sample over the 220-year period 1790ndash2009 The omitted countries are selected alphabetically It is clear from the spreadsheet itself that these are random exclusions RR have since acknowledged this to be the case (RR 2013A 2013B 2013C)

33 Summarising all RR data exclusions for highest public debtGDP category

Table 2 lists all of the countries in RRrsquos 1946ndash2009 data sample that are included at any time in the gt90 public debtGDP category We also show the number of years

8 The US series includes very large declines in GDP growth associated with post-World War II demobili-sation while over this same period the US public debtGDP ratio is in the gt90 category due to the public debt build-up tied to the high levels of wartime borrowing and spending Thus in 1946 the US public debtGDP ratio was 1213 and GDP growth was negative 109 More generally over the full 1946ndash2009 time period the USA experienced only four years (1946ndash49) during which its public debtGDP ratio exceeded 90 GDP growth in these years was minus109 minus09 44 and minus05 See Irons and Bivens (2010) for a more detailed discussion

9 In their analysis with the 1946ndash2009 dataset RR calculated both means and medians of cells in lines 30ndash44 instead of lines 30ndash49 In their analysis with the 1790ndash2009 dataset RR calculated both means and medians for cells in lines 5ndash19 instead of lines 5ndash24 As such Australia Austria Belgium Canada and Denmark were excluded entirely from their calculations with both the 1946ndash2009 and 1790ndash2009 data samples

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 8 of 23 T Herndon M Ash and R Pollin

that each country is included in this highest public debtGDP category10 We can sum up the RR data exclusions in two ways summarising by lsquocountry-yearsrsquo in which each year of excluded data from any of the countries in the sample counts as one observa-tion or summarising by countries in which any number of years of data (one or more) for any given country is counted as one country exclusion

As we see from Table 2 the correct total in RRrsquos 1946ndash2009 data sample in which a country is in the gt90 public debtGDP category is 110 country-years With RRrsquos chosen data exclusions the total falls to 96 country-years With both their chosen exclusions and their spreadsheet errors the total number of country-years in the gt90 public debtGDP category falls to 71 The differences are due again to RRrsquos deliber-ate exclusions of Australia Canada and New Zealand and to the additional exclusions for Belgium

RR made the same spreadsheet coding error in calculating mean and median GDP growth by public debtGDP category for 20 advanced economies over the 220-year

10 Table A1 in the Appendix presents a full listing of all 20 countries in the RR dataset This table also shows the number of years in which the public debtGDP ratio for each of the countries fell within one of the four public debtGDP categories (ie le30 30ndash60 60ndash90 and gt90) and the average GDP growth rate experienced by each country within each of these public debtGDP categories As we report in Table A1 Austria and Denmark (ie two of the five countries RR excluded from their analysis due to their spreadsheet error) did not in fact experience any years over 1946ndash2009 in which their public debt exceeded 90 of their GDP As such RRrsquos inadvertent exclusion of these two countries from their analysis did not affect their estimates of average GDP growth when public debt exceeded 90 of GDP

Table 2 Accounting for years in which countriesrsquo public debtGDP ratio exceeds 90 for 1946ndash2009

Countries in which public debtGDP gt90 over 1946ndash2009

Years in which public debtGDP gt90

Total number of years with public debtGDP gt90

Correct total

RR own count through chosen exclusionsa

RR actual count through chosen exclusions plus spreadsheet errorsb

Australia 1946ndash50 5 0 0Belgium 1947 1984ndash2005

2008ndash0925 25 0

Canada 1946ndash50 5 0 0Greece 1991ndash2009 19 19 19Ireland 1983ndash89 7 7 7Italy 1993ndash2001 2009 10 10 10Japan 1999ndash2009 11 11 11New Zealand 1946ndash49 1951 5 1 1UK 1946ndash64 19 19 19USA 1946ndash49 4 4 4Totals for all

countriesCountry-years ndash 110 96 71Countries ndash 10 8 7

Notes aExclusions for Australia Canada and New ZealandbAdditional exclusions for Australia Austria Belgium Canada and DenmarkSource Authorsrsquo calculations from unpublished working spreadsheet provided by RR

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 9 of 23

period 1790ndash2009 By calculating the mean for cells in lines 5ndash19 instead of lines 5ndash24 the coding error entirely excludes the same five countries (Australia Austria Belgium Canada and Denmark) from the analysis

We examine the impact of all these data exclusions after we also consider next their inappropriate weighting methodology

34 Inappropriate weighting in calculating summary statistics

Throughout their analysis RR adopt a weighting methodology for measuring the cen-tral tendency of real GDP growth within each of their four public debtGDP categories that in our view is inappropriate for understanding the issues at hand

Their approach is as follows focusing first on their calculations of mean GDP fig-ures After assigning each country-year to one of the four public debtGDP categories RR calculate the mean real GDP growth for each country within the category ie a single average value for the country for all the years it appeared in the category In other words RR compute overall averages as means of country means Thus real GDP growth in the UK averaged 24 per year during the 19 years that the UK appeared in the gt90 public debtGDP category This mean GDP growth figure for the UKrsquos 19 years in this highest public debtGDP category then counts as one country obser-vation in generating the mean GDP growth figure for the category for all countries By the same token as we have discussed above according to RRrsquos (incorrect) accounting New Zealand was in the gt90 public debtGDP category for one year only 1951 In that one year New Zealandrsquos GDP growth was minus76 According to RRrsquos method-ology this one year experience for New Zealand counts equally with the 19 years in which the UK was in the highest public debtGDP category in calculating the mean GDP growth figure for all countries

In other words RR are generating mean values for GDP growth through averag-ing by country with each country counting as a single observation no matter how many years it appears in any given public debtGDP category 19 years for the UK or 1 year only for New Zealand Clearly the impact of RRrsquos approach is to greatly amplify the effects of short-term episodes with high public debt levels in calculating the overall impact of high public debt on GDP growth As we will show more sys-tematically below the impact of New Zealandrsquos one-year episode would be far more modest if it were counted as only one country-year observation within as we saw in Table 2 the total of 110 country-years in which any countryrsquos public debtGDP ratio was above 90

New Zealandrsquos one-year experience in 1951 is the most obvious case in point illus-trating the problem with RRrsquos weighting methodology But there are also other impor-tant examples Thus with respect to RRrsquos 1790ndash2009 data sample Norway spent only one year (1946) in the 60ndash90 public debtGDP category during the total 130 years (1880ndash2009) in which data for Norway are included in the sample Norwayrsquos eco-nomic growth in this one year was 102 (due to rapid recovery after occupation dur-ing World War II) This one extraordinary growth experience contributes fully 53 (one of 19 countries) of the weight for the mean GDP growth in the 60ndash90 public debt category even though it constitutes only 02 (one of 455 country-years) of the country-years in this category Indeed Norwayrsquos one year in the 60ndash90 public debt category receives a weight equal to for example 23 years for Canada 35 years for Austria 42 years for Italy and 47 years for Spain

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 10 of 23 T Herndon M Ash and R Pollin

Further RR utilise this same methodology in calculating medians The result is that what RR term lsquomedianrsquo GDP growth figures for each public debtGDP category is more precisely the median of each countryrsquos median GDP growth figure calculated over the total number of country-years in which each country is included in one of the four public debtGDP categories Clearly this approach contrasts sharply with calculating central tendencies through taking the means or medians of all country-years which can be calculated in straightforward ways The RR approach requires two decisions first how to aggregate annual data for each country into a single country summary measure and second how to aggregate country summary measures into a single summary measure of central tendency for the full data sample

RR need to explain and justify in detail their weighting methodology for generating means and medians yet at no point do they do so in either version of their 2010 paper11 As such their methodology appears arbitrary and unsupportable In fact it is possible that within-country serially correlated relationships could support an argument that not every additional country-year observation contributes a proportional amount of additional use-ful information Thus the existence of serial correlation could suggest that with the case of the UK for example 19 years of carrying a public debtGDP load greater than 90 and averaging 24 GDP growth over those years does not warrant 19 times the weight of New Zealandrsquos single year at minus76 GDP growth But RR do not themselves offer any argument as to why the one-year experience in New Zealand should have 19 times the influence as each year in which the UK economy operated with high public debt levels

35 Impact of RR exclusions errors and methodology

We can observe the impact of RRrsquos selective data exclusions spreadsheet errors and inappropriate weighting methodology from various perspectives To begin with in Table 3 we show how each countryrsquos experiences in the gt90 public debtGDP cat-egory are weighted using RRrsquos data sample and one-number-per-country weighting methodology versus a country-year weighting approach along with the correct inclu-sion of all countries in the sample As we see Australia Belgium and Canada are all dropped through RRrsquos accounting and thus carry zero weight In contrast through correctly including the relevant data in the sample and weighting by country-years Canada and Australia should be weighted at 45 each and Belgium should properly account for fully 227 of the total weight of country-years in the gt90 public debtGDP category

Using RRrsquos methodology the remaining countries are all weighted equally as one-seventh or 143 of the total number of observations in the high public debt category no matter how many years each country was carrying debt above 90 of its GDP In contrast with country-year weighting the overall weight for each of the countries ranges widely from 36 for the USA to the highest figure for Belgium of 227

In Table 4 we show the differences in the estimates of GDP growth for each country during the years in which the countries were in the gt90 public debtGDP category The table shows clearly how we move from a mean GDP growth rate of negative 01

11 They also have not offered a substantive defence of their methodology in response to the criticism we presented in HAP (2013) The closest they have come to such a defence is their assertion that lsquoIt is the accusation that our weighting procedure is unconventional that is itself unconventionalrsquo (RR 2013B) But this assertion is not followed by any substantive discussion as to why their methodology should be preferred relative to eg weighting observations by country-years as we have done both here and in HAP (2013)

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 11 of 23

through RRrsquos accounting and weighting methodology as well as their errors to a positive 22 mean under accurate accounting and country-year weighting There are three major factors at play as we have discussed

(i) The full exclusions of Australia Belgium and Canada from the highest public debt category The GDP growth rates for these three countries while in the highest public debt category averaged 38 26 and 30 respectively

(ii) The exclusions of 1946ndash49 data for New Zealand This meant that RR included only the one year 1951 in which New Zealand was counted as being in the gt90 public debtGDP category Given their weighting methodology this one year with ndash76 growth counted as 143 of the entire sample of observations for countries in the highest public debt category

(iii) There are large differences in weights among the countries that are fully included in their data sample In particular the four years (1946ndash49) in which the USA is in the highest public debt category and averaged minus20 GDP growth are weighted as 143 of all observations by RR as opposed to 36 of all observations through proper accounting and country-year weighting

In addition a fourth smaller factor affecting RRrsquos average GDP estimate is that they made a transcription error in transferring the country average figure from the country-specific spreadsheets to the summary spreadsheet This transcription error reduced New Zealandrsquos average growth in the gt90 public debtGDP category from minus76 to the figure they report minus79 Because only seven countries appear in RRrsquos gt90 highest public debt category with each country carrying a 143

Table 3 Alternative weighting of country observations for above 90 public debtGDP category for the 1946ndash2009 time period

Countries in which public debtGDP gt90 over 1946ndash2009 time period

RR country weighting Country-year weighting

Australia1946ndash50

0 45

Belgium1947 1984ndash2005 2008ndash09

0 227

Canada1946ndash50

0 45

Greece1991ndash2009

143 173

Ireland1983ndash89

143 64

Italy1993ndash2001 2009

143 91

Japan 1999ndash2009

143 100

New Zealand 1946ndash49 1951

143 45

UK1946ndash64

143 173

USA1946ndash49

143 36

Source Authorsrsquo calculations from working spreadsheet provided by RR

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 12 of 23 T Herndon M Ash and R Pollin

weight this transcription error reduces RRrsquos estimate of mean real GDP growth by 01 percentage point

Table 5 provides a full accounting of the impact of the data exclusions spreadsheet errors inappropriate weighting methodology and transcription error as they impact in combination calculations of mean GDP growth for all countries in the RR 1946ndash2009 sample These factors have strong interactive effects We see in Table 5 the effect of each possible interaction between the data exclusions spreadsheet errors RR weight-ing methodology and transcription error

As the table shows the combined effects of RRrsquos overall approach have relatively small effects on mean GDP growth in the lower three public debtGDP categories Thus for the 0ndash30 public debtGDP category average GDP growth remains consistently around 4 per year For the 30ndash60 and 60ndash90 public debtGDP categories average GDP growth is consistently around 3 per year with or without adjusting for the RR errors and methodological choices However we see how with the gt90 category we go from an appropriately calculated mean GDP growth figure of +22 to the RR estimate of minus01 In other words with their estimate that average GDP growth in the gt90 public debtGDP category is minus01 RR overstate the growth gap between the highest and next highest public debtGDP categories by a factor of nearly two-and-a-half

We can see the relationship between public debtGDP ratios and GDP growth more fully in Figure 1 This figure presents all of the country-year data as continuous

Table 4 Combined impact of RR data exclusions spreadsheet errors and weighting methodology on GDP growth for gt90 public debtGDP category for 1946ndash2009 time period

Countries in which public debtGDP gt90 over 1946ndash2009 time period

RR estimate Estimate with full data sample and country-year weighting

Australia1946ndash50

No years in sample 38(45 weight)

Belgium1947 1984ndash2005 2008ndash09

No years in sample 26(227 weight)

Canada1946ndash50

No years in sample 30(45 weight)

Greece1991ndash2009

29(143 weight)

29(173 weight)

Ireland1983ndash89

24(143 weight)

24(64 weight)

Italy1993ndash2001 2009

10(143 weight)

10(91 weight)

Japan1999ndash2009

07(143 weight)

07(10 weight)

New Zealand1946ndash49 1951

minus79(143 weight)

26(45 weight)

UK1946ndash64

24(143 weight)

24(173 weight)

USA1946ndash49

minus20(143 weight)

minus20(36 weight)

Average GDP growth for all countries in gt90 public debt GDP category

minus01 +22

Source Authorsrsquo calculations from working spreadsheet provided by RR

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 13 of 23

real GDP growth rates plotted against public debtGDP categories RR mean growth estimates are indicated by diamonds with the corrected growth estimates indicated by filled circles The substantial disparity between the RR estimate and our corrected figure for the gt90 public debtGDP category is evident in the plot as are the rela-tively inconsequential errors in the lower three categories The plot also shows large variation in real GDP growth in each public debtGDP category Finally the plot includes an empty square as the data point for New Zealand in 1951 which as we have discussed alone accounts for 143 of RRrsquos result for the highest public debtGDP category

36 Reassessing RRrsquos mean GDP calculations for 1790ndash2009

The three sets of problems (data exclusions spreadsheet errors and inappropriate weighting methodology) that distorted RRrsquos GDP growth estimates with the 1946ndash2009 data sample have a similar impact with their long time series We can observe this by reviewing the data presented in Table 6 As the first row of the table shows RR (ie incorporating the exclusions spreadsheet errors and weighting methodology) find that mean GDP growth is at its peak with the le30 public debtGDP category at 37 Mean growth then falls to 30 for the 30ndash60 category and rises to 34 for the

Table 5 HAP recalculated GDP growth rates with RR calculated figures (percentages) for 1946ndash2009 time period

Public debtGDP category

le30 30ndash60 60ndash90 gt90

Recalculated resultsAll data with country-year weighting 42 31 32 22

Replication elementsSeparate effects of RR calculationsSpreadsheet error only 42 30 32 19Selective years exclusion only 42 31 32 19Country weights only 40 30 30 19

Interactive effects of RR calculationsSpreadsheet error + selective years exclusion 42 30 32 17

Spreadsheet error + country weights 41 29 34 14

Selective years exclusion + country weights 40 30 30 03

Spreadsheet error + selective years exclusion + country weights

41 29 34 00

Spreadsheet error + selective years exclusion + country weights + transcription error

41 29 34 minus01

RR published resultsRR (2010A 2010B Figure 2) (approximated) 38 29 34 minus01RR (2010B Appendix Table 1) 41 28 28 minus01

Note Values from bar chart in RR (2010A Figure 2) are approximateSources Authorsrsquo calculations from working spreadsheet provided by RR (2010A 2010B)

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 14 of 23 T Herndon M Ash and R Pollin

60ndash90 category According to RRrsquos calculations mean GDP growth then falls in the gt90 category to 17 a steep drop-off of 17 percentage points in GDP growth

But these differences in mean growth for the higher public debt categories dimin-ish dramatically when correcting RRrsquos data exclusions and errors and weighting by country-years rather than their one-number-per-country approach As we see in the second row of Table 6 GDP growth falls off from 32 in the 30ndash60 category to 25 in the 60ndash90 category The subsequent drop in mean GDP growth in the gt90 category is to 21 Two important points emerge here (i) the growth decline in the gt90 public debtGDP category relative to the 60ndash90 category is a modest 04 percentage points and (ii) this growth drop-off is less than the decline that occurs

Table 6 Recalculation of RRrsquos mean GDP growth rates (percentages) with 1790ndash2009 data sample

Public debtGDP categories

le30 30ndash60 60ndash90 gt90

RR means in 2010 papers 37 30 34 17Recalculations correcting for exclusions

and errors country-year weighting37 32 25 21

Source Underlying data from working spreadsheet for RR (2010A 2010B)

Fig 1 Real GDP growth by public debtGDP categories (data are country-years 1946ndash2009)Note Our replication of RR published values for average real GDP growth within category are printed to the right Corrected values for average real GDP growth within category are printed to the leftSource Authorsrsquo calculations from working spreadsheet provided by RR

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 15 of 23

between the 30ndash60 and 60ndash90 public debtGDP categories where the decline is 07 percentage points

Briefly after correcting for RRrsquos data exclusions errors and inappropriate weight-ing method with the 1790ndash2009 dataset there is no longer evidence supporting RRrsquos contention that when countries reach a public debt level gt90 of GDP economic growth declines to a significant degree and in a pattern that does not occur with the lower public debtGDP categories

37 Reassessing RR median GDP growth calculations for 1946ndash2009

In their 25 April 2013 New York Times response to HAP (2013) RR emphasise that they had always accorded greater significance to their results based on calculating median GDP growth figures rather than means Their response includes the following two representative passages

Our paper gave significant weight to the median estimates precisely because they reduce the problem posed by data outliers a constant source of concern when doing archival research that reaches far back into economic history spanning several periods of war and economic crises

We have never used anything but the conservative median estimate in our public discussions where we stated that the difference between growth associated with debt under 90 percent of GDP and debt over 90 percent of GDP is about 1 percentage point

In fact as we noted above in both versions of their 2010 paper RR made the same mistakes with exclusions spreadsheet errors and weighting method in generating their median GDP growth figures12 In Table 7 we show the impact of correcting for these problems As we see in the first row of the table according to their initial published fig-ures median GDP growth falls from about 30 for the 30ndash60 and 60ndash90 public debtGDP categories to 16 in the gt90 category However once we include the full data sample and calculate the median GDP growth based on country-years rather than one number per country we see in the second row of Table 7 that the GDP growth decline is much smaller That is median GDP growth is at 29 for the 60ndash90 public debtGDP category and 23 for the gt90 category a 06 percentage point drop-off

Table 7 Recalculation of RRrsquos median GDP growth rates (percentages) with 1946ndash2009 data sample

Public debtGDP categories

le30 30ndash60 60ndash90 gt90

RR 2010 medians 42 30 29 16HAP recalculations correcting for exclusions

and errors country-year weighting41 31 29 23

RR recalculations correcting for exclusions and errors country weighting

42 30 29 25

Source Underlying data from working spreadsheet for RR (2010A 2010B) and from RR (2013C)

12 They also clearly did not recognise this when they published their New York Times response (RR 2013A 2013B) but only after we published our New York Times rejoinder on 29 April 2013 (Pollin and Ash 2013B) They posted online their Errata document with corrections for their median figures on 5 May 2013 (RR 2013C)

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 16 of 23 T Herndon M Ash and R Pollin

What is equally notable in calculating median GDP growth figures is that in RRrsquos own recalculation in their Errata (RR 2013C) they themselves show a still smaller GDP growth decline in their gt90 category In this calculation RR now properly include all the relevant figures from their spreadsheet while still calculating medians through their one-countryone-observation methodology Through this method as we see in the last row of Table 7 RR themselves find that GDP growth falls from 29 for the 60ndash90 category to only 25 for the gt90 category That is using medians as their preferred measure of central tendency as well as their preferred one-observation-per-country weighting methodology the GDP growth drop-off for the gt90 public debtGDP category is now only 04 percentage points

38 Non-linearity at historical boundary

Beyond the specifics of their estimates on the relationship between public debtGDP ratios and GDP growth RR also conclude more generally that they have identified a robust non-linear relationship between public debt levels and GDP growth ie the significant fall-off in GDP growth after the 90 threshold in the public debtGDP ratio is crossed They hold that this robust relationship operates consistently across time peri-ods and countries However our recalculation of both the mean and median values for GDP growth after adjusting for their data exclusions spreadsheet errors and inappro-priate weighting method casts doubt on these broader historical generalities as well We now examine the evidence on the existence of any such non-linearity in several ways

381 Adding an additional public debtGDP category Working with both the 1946ndash2009 and 1790ndash2009 data samples we first add the category 90ndash120 public debtGDP ratio which then also establishes a gt120 category as now being the highest public debtGDP grouping We show the results of adding this additional category in Figures 2 and 3 including each country-year data point from both datasets

Starting with Figure 2 showing the 1946ndash2009 time period mean GDP growth in the 90ndash120 category is 24 which is reasonably close to the 32 GDP growth fig-ure for the 60ndash90 category Mean GDP growth in the new category (gt120 public debtGDP) is lower at 16 but does not fall off a non-linear cliff

It is also evident from the individual observations in Figure 2 with each cross mark in the figure representing one observation that variation in GDP growth within each of the public debtGDP categories is large This is an important observation in its own right for assessing the validity of generalities regarding the existence of robust non-linearities at historical boundaries But RR do not examine this issue at all in their 2010 studies

The absence of a non-linearity boundary at the 90 public debtGDP threshold is even clearer with the long period (1790ndash2009) data sample as we see in Figure 3 Thus with this long time period data sample mean real GDP growth in the 90ndash120 public debtGDP cat-egory is 25 This GDP growth rate is identical to the figure for the 60ndash90 category Even with the new highest category of gt120 mean GDP growth is lower at 16 but GDP growth does not decline in a sharp non-linear way even at this highest public debtGDP level

382 Scatter plots with regression line In Figure 4 we present a scatter plot of the 1946ndash2009 data showing all of the country-years with continuous real GDP growth plotted against public debtGDP ratios The figure also includes a locally fitted

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 17 of 23

regression function13 As we see no particular boundary or non-linearity is evident in either dimension around the 90 figure for the public debtGDP ratio The data thin out gradually between 70 and 120 public debtGDP ratios as is clear from both the points in the scatter plot and the widening of the 95 confidence interval for mean GDP growth More generally the wide range of GDP growth at various public debt levels is evident

Figure 4 does suggest a non-linearity in the relationship between GDP growth and pub-lic debtGDP ratios but this occurs as the public debtGDP ratio rises from 0 to around 30 This pattern becomes more evident still in Figure 5 which is a close-up of Figure 4 focusing on the country-year observations in which GDP growth ranges between 0 and 7 and the public debtGDP ratio ranges up to 150 What we see clearly in Figure 5 is that at 0 public debtGDP ratio average GDP growth is almost 5 But when the public debtGDP ratio reaches 30 average GDP growth is only slightly greater than 3 ie average GDP growth drops off by nearly 2 percentage points as countriesrsquo public debtGDP ratios rise from 0 to 30 This pattern contradicts RRrsquos claim that lsquoit is evident that there is no obvious link between debt and growth until public debt reaches a threshold of 90 percentrsquo (RR 2010B p 575) At the same time as we can see in Figure 5 the relationship

Fig 2 Real GDP growth with expanded public debtGDP categories (data are country-years 1946-2009)

Note Mean real GDP growth within each public debtGDP category indicated with filled circlesSource Authorsrsquo calculations from RR working spreadsheet

13 The locally smoothed regression is estimated with the general additive model with integrated smooth-ness estimation using the mgcv package in R The smoothing parameter is selected with the default cross-validation method Alternative methods such as LOESS and smoothing parameters produce similar results

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 18 of 23 T Herndon M Ash and R Pollin

between average GDP growth and public debt levels is relatively flat over a wide domain of debtGDP values Specifically between public debtGDP ratios of 38ndash117 we cannot reject a null hypothesis that average real GDP growth is 3

This pattern contradicts RRrsquos claim that lsquothe nonlinear response of growth to debt as debt grows towards historical boundaries is reminiscent of the ldquodebt intolerancerdquo phe-nomenon developed in Reinhart Rogoff and Savastano (2003)rsquo (RR 2010B p 577) The concept of debt intolerance presented in the 2003 Reinhart Rogoff and Savastano paper (Reinhart et al 2003) refers to the propensity with developing countries for debt crises and default to result when a countryrsquos external debt approaches a context-spe-cific threshold According to this 2003 paper (p 1) lsquodebt intolerantrsquo countries undergo lsquoextreme duressrsquo ie debt crisis and serial default when they approach the threshold By suggesting a similarity to this lsquodebt intolerancersquo scenario in summarising the findings of their 2010 paper RR make it clear that they envision a public debtGDP threshold that leads to sharp reductions in GDP growth once a country crosses the historical threshold

383 Varying results by post-war subperiods We explore the historical specificity of the results by examining mean real GDP growth by public debt categories for subsample periods in the 1946ndash2009 data Table 8 presents results for 1950ndash2009 1960ndash2009 1970ndash2009 1980ndash2009 and 2000ndash09 We see in the table that the higher GDP growth rates for the 0ndash30 public debtGDP category erodes substantially in the shorter and more recent time periods Thus GDP growth for the 0ndash30 category was 41 per

Fig 3 Real GDP growth with expanded public debtGDP categories (data are country-years 1790ndash2009)

Note Mean real GDP growth within each public debtGDP category indicated with filled circlesSource Authorsrsquo calculations from RR working spreadsheet

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 19 of 23

year in the 1950ndash2009 sample but declines to only 25 in the 1980ndash2009 sample Growth in the middle two public debtGDP categories also declines noticeably espe-cially in the 2000ndash09 period where average GDP growth was 19 for the 30ndash60 category and 13 for the 60ndash90 category as opposed to the 30 and 31 aver-ages respectively for the middle two public debtGDP categories in the 1950ndash2009 sample In contrast average GDP growth in the highest public debtGDP category remains stable across all samples of years remaining within 03 percentage points of 2 per year throughout

It will be useful to focus further on the results for 2000ndash09 Only four countries appear in the gt90 public debtGDP category over these years They collectively con-tribute 31 country-years of data Belgium eight years Greece 10 years Italy three years and Japan 10 years As we see in the last row of Table 8 with this data sample for 2000ndash09 average real GDP growth for these four countries in the gt90 category is 17 It is notable that between 2000 and 2009 the growth trajectory for the gt90 category actually outperformed growth in the 60ndash90 category and is only 02 per-centage points below the 19 growth rate for the 30ndash60 category14

Fig 4 Real GDP growth and public debtGDP ratios (all country-year observations 1946-2009)Notes See footnote 9 for details on regression line calculationShaded region indicates 95 confidence interval for mean real GDP growthSource Authorsrsquo calculations from RR working spreadsheet

14 Table 8 reports standard errors as well as mean values for GDP growth We do this because we recog-nise that the number of observations for the 2000ndash09 years is significantly lower than for the various longer time periods presented in this table As such as is conveyed by the larger size of the standard errors as our sample size narrows our estimates for GDP growth over 2000ndash09 are less reliable than the figures for the longer time periods

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 20 of 23 T Herndon M Ash and R Pollin

These patterns suggest two important conclusions (i) even the apparent non-lin-earity between the 0ndash30 public debtGDP category and the higher categories is a historically specific pattern not a robust result across the full 1946ndash2009 time period

Table 8 Alternative post-war subperiods for real average annual GDP growth (percent) by public debtGDP categories

Public debtGDP category

le30 30ndash60 60ndash90 gt90

Subperiod1950ndash2009 41

(01)30

(01)31

(02)21

(03)1960ndash2009 39

(01)29

(01)28

(02)21

(02)1970ndash2009 31

(02)26

(01)26

(02)20

(03)1980ndash2009 25

(02)25

(01)24

(02)20

(03)1990ndash2009 27

(03)24

(02)25

(03)18

(03)2000ndash2009 27

(03)19

(03)13

(04)17

(05)

Note Standard errors are in parenthesesSource Authorsrsquo calculations from working spreadsheet provided by RR

Fig 5 Close-up of real GDP growth and public debtGDP ratios (showing only country-year observations between 0 and 7 GDP growth and 0ndash150 public debtGDP 1946ndash2009)

Notes See footnote 9 for details on regression line calculationShaded region indicates 95 confidence interval for mean real GDP growthSource Authorsrsquo calculations from RR working spreadsheet

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 21 of 23

and (ii) the relationship between public debt and GDP growth is weaker in more recent years relative to the earlier years of the sample

4 Conclusion

The influence of RRrsquos research came from their straightforward intuitive use of data to construct a set of stylised facts characterising the relationship between public debt levels and GDP growth for a range of national economies and a range of time peri-ods However this laudable effort at clarity notwithstanding RR made significant mistakes in reaching the conclusion that countries facing public debt levels in excess of 90 of GDP will experience a major decline in their GDP growth rate The key problems we have identified with RRrsquos work including exclusion of available data spreadsheet errors and an inappropriate weighting method significantly reduced the measured average GDP growth rate for countries in the gt90 public debtGDP cat-egory The full extent of their mistakes transforms the reality of modestly diminished average GDP growth rates for countries carrying high public debt levels into a false image that high public debt ratios inevitably entail sharp declines in GDP growth Moreover as we show there is a wide range of GDP growth performances at every level of public debt among the 20 advanced economies that RR survey

In the aftermath of the public debate generated by the posting of our April 2013 working paper RR did acknowledge their spreadsheet errors They also recognised that in fact there is no clear public debt threshold beyond which GDP growth will fall off sharply At the same time RR have not addressed other crucial problems that we identified with their papers These include the following

(i) RR have not addressed their decision to include data for the USA in the early post-World War II period while explicitly choosing to exclude data for Australia Canada and New Zealand for the same years The US figures for these years support their hypothesis while those from Australia Canada and New Zealand weaken their hypothesis

(ii) RR have not responded to our findings showing that the relationship between public debt levels and GDP growth varies substantially by country and over time Especially significant here is the pattern for the most recent decade in their post-war dataset ie 2000ndash09 As we have shown there is no evidence in these most recent years for any drop-off at all in GDP growth when public debt exceeds 90 of GDP relative to when the public debtGDP ratio ranges between 30 and 90 Relative to experiences from 60 or 200 years ago such recent patterns for GDP growth under high public debt levels are likely to be more informative for assessing present-day policy concerns

(iii) Considering only median figures the results that RR regard as more reliable the GDP growth drop-off falls only 04 percentage points with the gt90 public debtGDP category for the 1946ndash2009 dataset according to RRrsquos own recalculations in their Errata document The growth drop-off with the gt90 category is only 03 percentage points for the 1790ndash2009 sample (Table 1 Panel 4 RR 2013C) Their recalculations corrected for data exclusions and coding errors but still retained their preferred country weighting methodology Briefly RR themselves find no substantial GDP growth decline through their own recalculations but they have not acknowledged this result

(iv) Whatever happen to be RRrsquos preferences with respect to data exclusions and weight-ing methodology they do not acknowledge the need for their main findings to be

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Page 22 of 23 T Herndon M Ash and R Pollin

robust across reasonable alternative methodologies and data choices The most obvi-ous case in point is that their main finding on mean GDP growth for the gt90 public debtGDP category can swing by almost 2 percentage points of GDP growth based on the treatment of New Zealandrsquos early postwar years alone We strongly support what we take to be a consensus view of research standards that any major empirical conclusions need to hold up consistently when one moves from using one method of calculation to another RRrsquos findings do not meet this standard test for robustness

Beyond these strictly analytical considerations we also believe that the debate gener-ated by our critique of RR has produced some forward progress in the sphere of eco-nomic policy making In particular it has established that policy makers cannot defend austerity measures on the grounds that public debt levels greater than 90 of GDP will consistently produce sharp declines in economic growth

Bibliography

Ash M and Pollin R 2013 lsquoSupplemental Technical Critique of Reinhart and Rogoff ldquoGrowth in a Time of Debtrdquolsquo Political Economy Research Institute Research Brief April httpwwwperiumassedufileadminpdfworking_papersworking_papers_301-350PERI_TechnicalAppendix_April2013pdf [date last accessed 15 December 2013]

Harding R 2013 Reinhart and Rogoff publish formal correction Financial Times 8 May httpwwwftcomcmss0433778c4-b7e8-11e2-9f1a-00144feabdc0html [date last accessed 15 December 2013]

Herndon T 2013 The grad student who took down Reinhart and Rogoff explains why theyrsquore fundamentally wrong Business Insider 22 April httpwwwbusinessinsidercomherndon-responds-to-reinhart-rogoff-2013ndash4

Herndon T Ash M and Pollin R 2013 lsquoDoes High Public Debt Consistently Stifle Economic Growth A Critique of Reinhart and Rogoff rsquo PERI Working Paper no 322 httpwwwperiumassedufileadminpdfworking_papersworking_papers_301-350WP322pdf

Irons J and Bivens J 2010 lsquoGovernment Debt and Economic Growth Overreaching Claims of Debt ldquoThresholdrdquo Suffer from Theoretical and Empirical Flawsrsquo Briefing Paper no 271 Economic Policy Institute httpwwwepiorgpage-pdfBP271pdf

Krugman P 2013 How the case for austerity has crumbled New York Review of Books 6 June httpwwwnybookscomarticlesarchives2013jun06how-case-austerity-has-crumbled pagination=false

Pollin R and Ash M 2013A Austerity after Reinhart and Rogoff Financial Times 17 April httpwwwftcomintlcmss09e5107f8-a75c-11e2-9fbe-00144feabdc0html [date last accessed 15 December 2013]

Pollin R and Ash M 2013B Debt and growth a response to Reinhart and Rogoff New York Times 29 April httpwwwnytimescom20130430opiniondebt-and-growth-a-response-to-reinhart-and-rogoffhtml

Reinhart C Reinhart V R and Rogoff K 2012 Public debt overhangs advanced economies episodes since 1800 Journal of Economic Perspectives vol 26 no 3 69ndash86

Reinhart C and Rogoff K 2010A lsquoGrowth in a Time of Debtrsquo Working Paper no 15639 National Bureau of Economic Research httpwwwnberorgpapersw15639

Reinhart C and Rogoff K 2010B Growth in a Time of Debt American Economic Review vol 100 no 2 573ndash8

Reinhart C and Rogoff K 2011A lsquoA Decade of Debtrsquo Centre for Economic Policy Research Discussion Paper no 8310

Reinhart C and Rogoff K 2011B lsquoA Decade of Debtrsquo Policy Analyses in International Economics 95

Reinhart C and Rogoff K 2013A Debt growth and the austerity debate New York Times 25 April httpwwwnytimescom20130426opiniondebt-growth-and-the-austerity-debatehtmlpagewanted=all

Reinhart C and Rogoff K 2013B Reinhart and Rogoff responding to our critics New York Times 25 April httpwwwnytimescom20130426opinionreinhart-and-rogoff-respond-ing-to-our-criticshtmlref=opinion

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from

Critique of Reinhart and Rogoff Page 23 of 23

Reinhart C and Rogoff K 2013C Errata lsquoGrowth in a Time of Debtrsquo Harvard University 5 May httpwwwcarmenreinhartcomuser_uploadsdata36_datapdf

Reinhart C Rogoff K and Savastano M A 2003 Debt intolerance Brookings Papers on Economic Activity vol 34 no 1 1ndash74

Ryan P 2013 The Path to Prosperity A Blueprint for American Renewal Fiscal Year 2013 Budget Resolution House Budget Committee httpbudgethousegovuploadedfilespathtoprosper-ity2013pdf

Appendix

Table A1 Data on public debtGDP ratios and GDP growth rates for all countries in RR 1946ndash2009 dataset

Country le30 30ndash60 60ndash90 gt90

Australia Years in category 37 13 9 5GDP growth 32 49 40 38

Austria Years in category 34 27 1 0GDP growth 52 34 minus38

Belgium Years in category 0 17 21 25GDP growth ndash 42 31 26

Canada Years in category 3 42 14 5GDP growth 25 35 45 30

Denmark Years in category 23 16 17 0GDP growth 35 17 24 ndash

Finland Years in category 44 16 4 0GDP growth 38 24 55 ndash

France Years in category 24 21 10 0GDP growth 51 27 30 ndash

Germany Years in category 48 11 0 0GDP growth 39 09 ndash ndash

Greece Years in category 13 5 3 19GDP growth 40 03 27 31

Ireland Years in category 10 14 32 7GDP growth 42 45 40 24

Italy Years in category 26 6 17 10GDP growth 54 21 18 10

Japan Years in category 22 17 4 11GDP growth 73 40 10 07

Netherlands Years in category 17 34 2 0GDP growth 41 26 11 ndash

New Zealand Years in category 9 33 17 5GDP growth 25 29 39 26

Norway Years in category 51 12 1 0GDP growth 34 51 102 ndash

Portugal Years in category 42 9 7 0GDP growth 45 35 19 ndash

Spain Years in category 5 36 1 0GDP growth 15 34 42

Sweden Years in category 18 35 11 0GDP growth 36 29 27 ndash

UK Years in category 0 39 6 19GDP growth ndash 22 25 24

USA Years in category 0 37 23 4GDP growth ndash 34 33 minus20

Total country-years 426 439 200 110Total countries 17 20 19 10

Source Authorsrsquo calculations from working spreadsheet provided by RR

by guest on January 1 2014httpcjeoxfordjournalsorg

Dow

nloaded from