the qog institute qog survey: a new cross-national...
TRANSCRIPT
The QoG Institute QoG Survey:
A New Cross-National Dataset on the Structure of Public Administration
Jan Teorell
Department of Political Science
Lund University
The Quality of Government Institute, University of Gothenburg
In collaboration with:
Mette Anthonsen, Nicholas Charron, Stefan Dahlberg, Carl Dahlström, Sören Holmberg,
Staffan Kumlin, Victor Lapuente, Naghmeh Nasiritousi, Henric Oscarsson, Bo Rothstein,
Marcus Samanni and Helena Stensöta
The Quality of Government Institute, University of Gothenburg
WORK IN PROGRESS
PLEASE DO NOT CITE WITHOUT AUTHOR’S PERMISSION
Prepared for presentation at the Conference “New Public Management and the Quality of
Government”, University of Gothenburg, November 13-15, 2008.
2
Abstract: The aim of this paper is to provide a first overview of and some preliminary findings
from the Quality of Government Institute Quality of Government Survey, an ongoing web
survey of public administration experts from some 50 countries. The general study design –
how countries and country experts were selected, as well as how we fare on response rates
and country selection thus far – is presented, and the general questionnaire design and some
considerations that motivated this design is discussed. Moreover, the basic correlational
structure of some core dimensions of public administration is explored, together with some
descriptive cross-national comparisons.
3
Introduction
The last decade or so has seen a proliferation of expert polls on the quality of government
around the world. Various aspects such as degree of democracy, the provision of rule of law,
government effectiveness, bureaucratic quality and political or administrative corruption are
now included in regular expert surveys, which are then compiled and aggregated to cross-
national datasets such as the World Bank Institute Governance Indicators (Kaufman et al.
2006). There are however several problems with these extant measures. First, the most serious
and widely used measures academically are heavily geared towards properties of the input
side of the political system, such as democracy. In terms of the important distinction between
the access and exercise of public authority (Rothstein & Teorell 2008), that is, most measures
tap into the former at the expense of the latter. Second, with the sole exception of Rauch and
Evans’s (2000) data on bureaucratic structure, the measures that do dwell on the
output/exercise side are not well anchored in theories of public administration. Primarily
being produced by private consultant firms selling advice and investment ratings to business
firms, the theoretical rationale for extant data on the structure of public administration is
generally opaque. Third, most of these measures are oriented towards the “dark side” of
public administration, covering concepts such as red tape, corruption and state failure. What is
lacking (again, with the exception of Rauch and Evans 2000) is thus a normative theory of
what a public administration should look like when it works as it should.
The general purpose of the Quality of Government Institute Quality of Government Survey
(the QoG Survey for short) is to address these shortcomings. This survey is geared towards
measuring the structure and organization of public administration across countries, thus
ignoring issues of democracy and the input side of the system that we have established
measures for already. Moreover, the core conceptual basis of Rauch and Evans’s (2000) data
on Weberian bureaucracies is used and refined as a theoretical tool to guide data collection,
but other theoretical perspectives such as New Public Management have also informed our
questionnaire design. Finally, this survey provides the first direct cross-national measures of
the impartiality of government institutions, based on Rothstein and Teorell’s (2008)
normative theory of the quality of government.
The aim of this paper is to provide a first overview of and some preliminary findings from the
QoG survey. First, I will present the general study design: how countries and country experts
4
were selected, as well as how we fare on response rates and country selection thus far. I then
present the general questionnaire design and some considerations that motivated this design.
Moreover, the basic correlational structure of some core dimensions of public administration
will be explored, together with descriptive cross-national comparisons. It should be stressed at
the outset that since this is an ongoing project – data collection proceeds at the moment of
writing this – all findings and results in this paper are highly preliminary. I end by discussing
some possible uses and extensions of this work in the future.
Sampling Frame and Data Collection
After a pilot conducted in the Winter of 2007-2008, the survey has been administrated starting
in September as a web survey of public administration experts in a wide array of countries.
For the pilot, we opted for a very open format: we simply “advertised” for respondents on our
website, and anyone could then supply their responses for any country in the world, free to
their own choosing. In a couple of months time, this generated 83 respondents from 31
countries worldwide, but with a heavy concentration (not surprisingly) to Sweden and the US
(with 13 respondents each). In the main study, we wanted more control over the selection of
both countries and experts. Although the theoretical scope of the survey is global in principle,
we realized at this stage that there would be a trade-off between the number of countries we
could include in the study, particularly from the developing world, and the information we
could acquire on potential public administration experts.
The solution to this problem that we opted for was to select experts first, and then let the
experts, by themselves choosing the country for which they wanted to provide their responses,
determine the selection of countries. In practice, what we did was to assemble a list of persons
registered with four international networks for public administration scholars: The Network of
Institutes and Schools of Public Administration in Central and Eastern Europe (NISPACEE),
The European Group of Public Administration Scholars (EGPA), the European Institute of
Public Administration (EIPA), and the Structure and Organization of Government (SOG)
Research Committee at IPSA. The homepages of these scholarly networks provided the bulk
of names of public administration scholars that was sent the questionnaire, but we also did
some complementary searches on the internet, drew from personal contacts of scholars at the
QoG Institute, and used the list of experts recruited from the pilot survey. All in all, this
resulted in a sample of 1274 persons from some 50 countries. We contacted these persons by
5
email, including some background information on the survey, a request to take part, together
with a clickable link inside the email leading to the web-based questionnaire in English. The
only incentives presented to participants was access to the data, a first-hand report, and the
possibility of being invited to future conferences on the Quality of Government.
At present (more precisely, on October 30), after two reminders, 452 or 35.4 percent of these
experts have responded, providing responses for 53 countries. The average respondent is a
male (67 %), 46-year-old PhD (81 %). An overwhelming majority of respondents were either
born (91 %) or live (92 %) in the country for which they have provided their responses.
*** Table 1 about here ***
The distribution of experts and the response rate across countries is provided in Table 1.
While the number of respondents vary substantially, from only 1 for Brazil, China and
Mauritius to a maximum of 27 in the Czech Republic, on average 8.5 experts per country
have taken the time to respond to our survey. As should be expected from the sampling frame,
Western Europe and Northern America together with postcommunist Eastern Europe and the
former Soviet Union carry the weight of countries covered. Only three small European Union
member countries are not covered (Cyprus, Luxembourg and Malta). Only four non-Western
and non-postcommunist countries are covered by more than four respondents: India, Japan,
South Korea and Turkey, the last three of which are OECD members. By and large, then, our
sample of countries is heavily geared towards the developed world.
Questionnaire Design
The exact question wording and graphical layout of the questionnaire is provided in the
Appendix. As should be clear, the questionnaire is fairly short yet covers a variety of topics
relevant to the structure and functioning of the public administration, such as meritocratic
recruitment, internal promotion and career stability, salaries, impartiality, NPM reforms,
effectiveness/efficiency, and bureaucratic representation.
Three considerations motivating the questionnaire design deserve special mentioning. These
considerations were also, by far margin, the issues most intensively discussed among the team
designing the questionnaire. First, we ask about perceptions rather than about statements of
6
facts. In this regard, our questionnaire differs from Rauch and Evans’s (2000) and is more in
line with the general surge in expert polls on quality of government across the globe. Thus,
for example, whereas Rauch and Evans (2000, 56) ask their respondents to state
“approximately what proportion of the higher officials…enter the civil service via a formal
examination system”, with responses coded in percentages, we instead ask “Thinking about
the country you have chosen, how often would you say the following occurs today: Public
sector employees are hired via a formal examination system”, with responses ranging from
“hardly ever” to “almost always”. The difference between these two question formats should
not be exaggerated. At the end of the day, we believe most of our questions have a factual
basis in the sense that some answers for a given country are more correct than others. It would
for example at least in principle be possible to learn how many public sector employees
actually were hired in a country a certain year that had to pass a formal examination. Yet we
ask each respondent to translate this basic fact into a more subjectively oriented response
scale ranging from 1 (“hardly ever”) to 7 (“almost always”).
We do this for two reasons. First, this enables us to use the same response scale for a large
number of “factual” questions, rather than having to tailor the response categories uniquely
for each individual item in the questionnaire. The overarching rationale here is thus
efficiency: we save both space and response time by a more standardized question format.
Second, we believe that even the most knowledgeable country experts are rarely in a position
to correctly answer more than a handful of these questions with any precision. In other words,
even the factual question format used by Rauch and Evans (2000) evokes informed guesswork
on behalf of the experts. We make this guesswork more explicit from the outset by asking
about perceptions rather than “correct” answers.
This should of course not be interpreted as implying that we do not care about the
correspondence between our respondents’ perceptions and the actual workings of the public
administration systems they assess. We are not primarily interested in perceptions per se, but
in the reality that underlie these perceptions. By relying on more than one expert per country,
however, our idea in his regard is to rely on the convergence of different expert perceptions as
our point estimate for the actual workings of a country. In practice, this means relying on the
mean estimate per country, while taking the variation around that mean as an indication of the
uncertainty surrounding this estimate.
7
The second design issue concerns the time frame of the study. Whereas Evans and Rauch
(2000) asked about the state of affairs prevailing over a 20 year period (1970-1990), we opted
for another solution: to mostly ask about the current state of affairs (questions #2, #4, and #6-
#8), but to ask about perceived change over the last 10-year period for a selected set of items
(questions #3 and #5). With this retrospective approach we hope to at least being partially
able to address the perennial issue of endogeneity bias when these data are to be used for
explanatory purposes.
The third and most pressing design issue concerns how to label and select the dramatis
personae at center stage of our inquiry. More precisely, should we ask about the public
administration in general or about specific sectors or agencies? And what term (in English)
should we use to designate the persons working in the public administration in order to
convey an equivalent meaning across countries? We first opted for the terms “civil service”
and “civil servant”, before realizing that these terms do not even convey the same meaning in
English speaking countries across the Atlantic (in American English, civil servants include
political appointees; in British English, they do not). We also considered selecting a “core
agency” in the public administration, as did Rauch and Evans (2000), but as opposed to them
we could not agree on what then should be considered the “core”. Recall that Rauch and
Evans (2000) had a particular outcome in mind when designing their study: that of attaining
economic development (Evans and Rauch 1999). Our approach is more general. Apart from
studying outcomes such as growth or economic well-being, we hope to explore consequences
for public opinion such as generalized trust and subjective well-being. For this types of
outcomes the characteristics of street-level bureaucrats could arguably be as important as the
those of senior officials, and what specific sector or agency within the public administration
that should matter the most cannot be easily settled in advance (and might very well vary by
country). Thus, we opted for a holistic take on the public administration, trying to gauge
perceptions of its working in general (with one major exception: we explicitly exclude the
military).
After serious consideration and some pre-testing in the pilot, we then chose the term public
sector employee to designate – at the most general level – those persons within the public
administration we inquire into. This is of course a debatable solution. Most notably, there
might be large variation across different types of public sector employees in a country, and
the expert respondents might then run into difficulties when asked to provide one overall
8
judgment. To off-set this problem somewhat, we made the following clarification in the
opening page of the questionnaire:
When asking about public sector employees in this survey, we would like you to think about a typical
person employed by the public sector in your country, excluding the military. If you think there are large
discrepancies between branches of the public sector, between the national/federal and subnational/state
level, or between the core bureaucracy and employees working with public service delivery, please try to
average them out before stating your response.
This is of course more easily said than done. Only by exploring the consistency and face
validity of the data, and by closely scrutinizing the open-ended comments to the questionnaire
supplied by a large number of respondents, can we make any firm conclusions as to whether
this strategy has worked or not.
Four Dimensions of Public Administration
As an appetizer, and also to substantiate the intelligibility of these data in light of the
aforementioned problems, Table 2 reports the result from a country-level principal
components factor analysis of the bulk of questionnaire items pertaining to current affairs
(questions # 2, 4, 6 & 8). The purpose of this exercise is to ascertain whether a set of
underlying dimensions structure the differences in mean responses across countries. As a
simple take on the problem of handling countries with few expert observations, this analysis
excludes the 11 countries with less than 5 respondents. The first four factors (i.e., dimensions)
extracted account for 76 percent of the country-level variation.
*** Table 2 about here ***
The figures given in the table are the so-called factor loadings, which may be interpreted as
correlation coefficients for the relationships between each questionnaire item and the four
factors. Strong loadings are highlighted in bold to facilitate interpretation. The first factor thus
consists of the items tapping into the extent of meritocratic recruitment (q2_a), internal
recruitment of senior officials (q2_e), mechanisms to reprimand misconduct (q2_m),
impartiality (q4, q6_1), efficiency (q8_a), client-orientation (q8_c), and rule-following
(q8_d). With negative loadings indicating inverse relationships, this factor thus also consists
of items tapping into the opposite of political recruitment (q2_b, q2_d) and ideology-driven
9
implementation (q8_e), as well as into three clear examples of partial policy implementation:
kickbacks (q2_g), group discrimination (q2_h) and favoritism (q2_i). This factor, which on its
own mops up some 50 percent of the country-level response variation in the 23 items included
in the analysis, clearly indicates that in the realm of public administration most good things go
together. Rule-following does not clash with efficiency; efficiency is compatible with
impartiality; impartiality goes hand in hand with a help-oriented “ethics of care” (Rothstein
and Teorell 2008; cf. Stensöta 2004). And underlying these behavioral differences among
public administrators looms the old Weberian ideal of meritocratic recruitment and a non-
politicized civil service. I thus choose to call this first and dominant dimension the
patrimonial vs. Weberian public administration.
Figure 1 shows where the 42 countries in our restricted sample are located along this
dimension, with negative values indicating closeness to the patrimonial position and positive
values closeness to the Weberian position.1 Not very surprisingly, the most patrimonial
countries are located in the postcommunist region, most notably in Ukraine, Bosnia and
Herzegovina and Russia. The public administrations of certain Western European countries,
namely Greece and Italy, are however also closer to the patrimonial than to the Weberian end
of the scale. At the other end of the spectrum, Australia and New Zealand are ranked as
closest to the Weberian ideal type, closely followed by the Scandinavian countries and Japan.
India, interestingly, is located near Spain, at the middle of the scale.
*** Figure 1 about here ***
The second factor in Table 2, which by construction is uncorrelated with the patrimonial vs.
Weberianism dimension, taps into another aspect of the structure of public administration.
Items loading strongly on this factor are those extensively relying on formal examination
systems (q2_c), where tenure within the public administration is lifelong (q2_f), and
employment contracts in the public sector are regulated by special laws not pertaining to the
private sector (q8_f). This very much resembles the distinction between open. vs. closed civil
service systems (Bekke and Van der Meer 2000). In Figure 2, the countries’ position along
1 Figures 1-3 are based on the so-called factor scores for each factor. These scores are computed as an additive index of all items entered into the analysis, weighted by their respective factor loadings. By construction, the factors scores have zero means and unit standard deviation.
10
this dimension supports this intuition.2 Countries with a reputation of having highly closed
systems such as Japan, South Korea and countries in continental Europe are located far to the
right extreme, whereas Scandinavian and Anglo-Saxon countries of the more open variety are
located to the left. Since no clear expectations could be formed as to where the
postcommunist countries should be located along this dimension, it is quite interesting to note
that they are actually not concentrated to any of the end points. Apparently, both the closed
and the open systems are fairly well represented among them.
*** Figure 2 about here ***
The third factor in Table 2 comes closest to a set of policy reforms that may be grouped under
the heading New Public Management (or NPM, for short). This is clearest so with respect to
the prevalence of performance-related pay (q2_l), an element which according to Dahlström
and Lapuente (2008, 2) “represents both the main values of NPM as well as most of its
doctrinal components”. Interestingly, performance related pay seems to have been
implemented together with other “classical” NPM policies such as greater emphasis on
competition in public service delivery (q8_g) and privatization (q8_h). In particular the latter
item, however, covering the extent to which “the provision of public sector services is funded
by user fees and/or private insurances rather than taxes”, may more tap into the distinction
between the liberal vs. the universal welfare state than into the actual introduction of NPM
“reforms”. With this caveat in mind, Figure 3 should be interpreted with caution. The location
of most postcommunist countries closer to the right-hand end of the spectrum probably to a
large extent reflects the absence of tax financed public welfare in these countries. New
Zealand’s location toward the middle of the “NPM” dimension, moreover, starkly contrasts
with its position as the country with the most extensively implemented performance-related
pay system (i.e., the most clear-cut NPM item).
*** Figure 3 about here ***
2 Interestingly, closedness correlates negatively with the willingness of public sector employees to implement policies decided upon by the top political leadership (q8_b)
11
Finally, Figure 4 displays the relationship between the two remaining items, none of which
are systematically related to any of the previously presented three dimensions. The first is our
version of Rauch and Evans’s (2000) notion of competitive salaries (q2_k), the second is the
extent to which women are fairly represented among public sector employees (q8_i). I would
not claim that these two items make up for an autonomous, let alone interpretable, underlying
“dimension of public administration”. What is noteworthy, although not very surprising,
however, is the fact that these two items are negatively correlated (at –.31 among all 42
countries, but at –.42 if India, an outlier, is excluded). Countries where public sector salaries
are on par with the private sector, such as Japan and Ireland, also have the least women
employed in the public sector.
*** Figure 4 about here ***
Conclusion
There is a number of directions into which further analysis of these data may take us. A first
and more technical challenge will be to find a way to estimate confidence intervals around the
country means by using the within-country discrepancies in assessments among experts. Upon
completion of the data collection, the aim is to make the data publicly available. Apart from
descriptive exercises such as the one above, these data could then be used for two general
analytical purposes. The first is to gauge the consequences of the structure of public
administration, either writ large for society (such as for growth, armed conflict or public
opinion), or more internally for the performance of the public sector itself (on independent
outcomes such as output evaluations and corruption). The second, of course, is to explore the
causal roots of different dimensions of public administration.
References
Bekke, Hand and Frits Van der Meer (2000) Civil Service Systems in Western Europe. Cheltenham: Edward Elgar.
Dahlström, Carl and Victor Lapuente (2008) “New Public Management as Trust Problem: Explaining Cross-Country Differences in the Adoption of Performance-Related Pay in the Public Sector”, QoG Working Paper Series 2008:7, The Quality of Government Institute, University of Gothenburg.
12
Evans, Peter and James Rauch (1999) “Bureaucracy and Growth: A Cross-National Analysis of the Effects of ‘Weberian’ State Structures on Economic Growth”, American Sociological Review 64(5): 748–765.
Kaufman, Daniel, Aart Kray and Massimo Mastruzzi (2006) “Governance Matters V: Aggregate and Individual Governance indicators for 1996-2005”, The World Bank, 2006.
Rauch, James, and Peter Evans (2000) “Bureaucratic structure and bureaucratic performance in less developed countries”, Journal of Public Economics 75: 49-71.
Rothstein, Bo and Jan Teorell (2008) “What Is Quality of Government? A Theory of Impartial Government Institutions”, Governance 21(2): 165-190.
Stensöta, Helena (2004) Den empatiska staten: Jämställdhetens inverkan på daghem och polis. PhD diss., Department of Political Science, University of Gothenburg.
13
Appendix: The Questionnaire
14
15
16
17
18
19
20
21
Figure 1. Patrimonial vs. Weberian Public Administrations (country scores of factor 1)
-2 -1 0 1 2
Patrimonialism vs. Weberianism
New ZealandAustraliaDenmark
NorwaySweden
JapanCanadaFinland
SwitzerlandNetherlands
UKBelgiumAustriaIreland
USGermany
South KoreaSpain
CroatiaIndia
LithuaniaPoland
PortugalCzech Rep
TurkeyHungary
ItalySloveniaRomaniaSlovakiaBelarus
BulgariaGreece
ArmeniaAzerbaijan
GeorgiaKyrgyzstan
AlbaniaMacedonia
RussiaBosnia
Ukraine
22
Figure 2. Open vs. Closed Public Administrations (country scores of factor 2)
-3 -2 -1 0 1 2
Open vs. Closed
IndiaJapanSpain
IrelandBelgiumGreeceCroatiaTurkey
LithuaniaSouth Korea
MacedoniaRomaniaPortugal
ItalyGermanyArmeniaSlovenia
PolandAustria
HungaryCanadaBulgaria
UKNorwayBosnia
USAzerbaijan
FinlandAlbaniaUkraine
KyrgyzstanSwitzerlandCzech Rep
SwedenAustraliaSlovakia
NetherlandsRussia
DenmarkBelarus
New ZealandGeorgia
23
Figure 3. New Public Management Reforms (country scores of factor 3)
-2 -1 0 1 2
NPM Reforms
South KoreaKyrgyzstan
BulgariaRomania
MacedoniaUK
AustraliaUS
RussiaLithuania
CroatiaHungaryFinland
AzerbaijanCanadaAlbania
SwitzerlandIreland
New ZealandUkraineArmenia
NetherlandsGermanyBelgiumPortugalGeorgiaPolandJapan
SloveniaCzech Rep
NorwaySpain
SlovakiaTurkey
ItalyDenmark
GreeceAustria
IndiaBosnia
SwedenBelarus
24
Figure 4. Competitive Salaries vs. Female Representation (country means of f2_10, f8_9)
Albania
Armenia
Australia
Austria
Azerbaijan
Belarus
Belgium
Bosnia
Bulgaria
Canada
Croatia
Czech RepDenmark
Finland
Georgia
Germany
GreeceHungary
India
Ireland
Italy
Japan
Kyrgyzstan
Lithuania
Macedonia
Netherlands
New Zealand
NorwayPoland
Portugal
Romania
Russia
SlovakiaSlovenia
S. Korea
Spain
Sweden
Switzerland
Turkey
UK
US
Ukraine
23
45
6
Com
petit
ive
sala
ries
2 3 4 5 6
Female representation
25
Table 1. Number of Valid Responses by Country
Country Respondents Country Respondents
Albania 11 Switzerland 5 Armenia 15 Turkey 5 Australia 7 Ukraine 11 Austria 5 United Kingdom 9 Azerbaijan 6 United States 15 Belarus 8 Uzbekistan 3 Belgium 7 Bosnia & Herzegovina 7 Brazil 1 Bulgaria 20 Canada 12 China 1 Croatia 5 Czech Republic 27 Denmark 13 Estonia 4 Finland 9 France 3 Georgia 7 Germany 12 Greece 21 Hungary 15 India 7 Ireland 15 Italy 7 Japan 9 Kazakhstan 4 South Korea 6 Kyrgyzstan 6 Latvia 2 Lithuania 11 Macedonia 7 Mauritius 1 Mexico 3 Netherlands 11 New Zealand 12 Norway 12 Poland 10 Portugal 8 Romania 16 Russian Federation 5 Serbia and Montenegro 2 Slovakia 6 Slovenia 10 South Africa 3 Spain 6 Sweden 9
26
Table 2. Four Dimensions of Public Administration
Factor 1 Factor 2 Factor 3 Factor 4
Meritocratic recruitment (q2_a) .95 –.05 .06 –.08 Political recruitment (q2_b) –.92 –.02 –.04 .23 Formal examination system (q2_c) –.09 .89 .12 –.11 Political elite recruits senior officials (q2_d) –.70 –.18 .15 .42 Senior officials internally recruited (q2_e) .69 .14 –.20 –.30 Lifelong careers (q2_f) .27 .73 –.39 .02 Kickbacks give procurement contracts (q2_g) –.94 .07 .01 .09 Unfair implementation (q2_h) –.91 –.04 .01 –.07 Contacts important for firm licenses (q2_j) –.89 .22 .09 .11 Competitive salaries (q2_k) .17 –.12 .22 –.72 Performance related salaries (q2_l) .49 –.24 .65 –.01 Misconduct reprimanded (q2_m) .88 –.04 .26 –.04 Impartial implementation (q4) .93 .06 .12 –.06 Money to poor reaches the poor (q6_1) .87 –.07 .05 –.02 Strive for efficiency (q8_a) .88 –.17 .23 –.01 Strive to implement policies (q8_b) .53 –.51 .23 –.02 Strive to help clients (q8_c) .95 –.07 .09 .06 Strive to follow rules (q8_d) .84 .05 .06 .08 Strive to fulfill government ideology (q8_e) –.65 –.38 .01 .04 Special employment laws (q8_f) –.27 .67 .03 .31 Competition in public service (q8_g) .56 .02 .52 .14 Private financing of public services (q8_h) –.03 .03 .86 –.05 Women proportionally represented (q8_i) .03 –.04 .09 .81
Note: Entries are varimax rotated factors loadings for the first factors retained from a principal components factor analysis at the country level (n=42). Loadings >.5 or <–.5 are highlighted in bold, questionnaire items within parentheses.