A Concept-driven Approach to Measurement:

The Lexical Scale

John Gerring Department of Political Science

Boston University jgerring@bu.edu

Svend-Erik Skaaning Department of Political Science

Aarhus University skaaning@ps.au.dk

Paper prepared for presentation at the APSA annual meeting, Chicago, August 29-September 1, 2013.

2

ABSTRACT This paper introduces a concept-driven method of scale construction – the lexical scale – that is viable in situations where a concept can be meaningfully operationalized according to a series of necessary-and-sufficient conditions arrayed in an ordinal scale. We offer several examples of how lexical scales might be applied to key social science concepts, with a focus on “electoral democracy,” for which we provide a new dataset extending back to 1800. We proceed to contrast the lexical scale with the Mokken scale and conclude with an evaluation of the overall strengths and weaknesses of the proposed approach.

3

The theoretical burden of social science is carried by highly abstract concepts such as democracy, corruption, property rights, and rule of law. We require such concepts in order to articulate high-order theories. Yet, they are difficult to operationalize, even when agreement can be reached on a general definition for a term. A key obstacle is aggregation. Faced with a number of indicators that seem relevant to a concept the researcher must decide how to combine them into a single index. Factor analysis, structural equation models, and IRT models are three generic approaches to this problem (DeVellis 2011). Other approaches, such as those employed by Polity IV (Marshall, Jaggers 2007) and Freedom House (2007) to measure various facets of regimes, are more idiosyncratic but nonetheless influential. Where attributes co-vary in a predictable fashion the problem of aggregation is easily solved. But where they do not, as is generally the case with multivalent concepts, researchers are at pains to solve the aggregation problem in a non-arbitrary fashion. This paper introduces a novel approach to scale construction that builds on the properties of concepts. Any attempt at measurement must be oriented around a concept, the thing that is purportedly being measured. Indeed, problems of validity often stem from inattention to the core concept and a consequent lack of fit between the term, its definition, and the indicators employed to measure it (Adcock, Collier 2001). This much is generally recognized, even if not always achieved. In this study, we look to concepts to do more than identify relevant indicators. Specifically, we enlist the structure of concepts to solve the aggregation problem. This is accomplished by regarding conceptual attributes as necessary-and-sufficient conditions arrayed in an ordinal scale. Following Rawls (1971), we refer to this as a lexical scale. The first section reviews the concept-guided approach to measurement with a focus on binary scales. The following section lays out the construction of a lexical scale. In the third section, we propose a Lexical index for the key concept of electoral democracy, providing an original dataset including all countries and semisovereign entities from 1800 to 2008, which is then tentatively explored in the context of research on state repression. The fourth section situates the lexical scale in the measurement literature, with special reference to Mokken scales. The paper concludes with some general observations about the strengths and weaknesses of the lexical scale relative to other approaches to scale construction. A few notes on terminology will be helpful before we begin. Concept formation refers to the construction of a concept, including the choice of terms, defining characteristics, and referents. Defining properties of a concept refer to attributes that provide its formal definition. Associated properties are attributes that are thought to be associated with the defining properties, but not definitional. Measurement refers to concept operationalization, i.e., the instructions or instruments required to identify membership, or degrees of membership, in the extension of a concept. This involves the construction of an indicator or index (a group of indicators combined in some fashion). A scale is a generic type of indicator or index.

I. Concept-Driven Approaches Concepts are indispensable at a fundamental level; one cannot apprehend reality without verbal tools. However, in choosing how to operationalize a term social scientists often rely on the empirical properties of things “out there” to construct an index. Concept formation is thus often subordinated to measurement, as Sartori (1970: 1038) lamented some time ago.

4

This state of affairs is partly a product of the underdevelopment of concept-led approaches to measurement. With the exception of a small body of work on concept formation and typologies (Bailey 1994; Collier et al. 2012; Collier, Gerring 2009; Elman 2005; Gerring 2012: ch 5; Goertz 2006; Sartori 1984), and a few studies focused explicitly on the connection between conceptualization and measurement (e.g., Adcock, Collier 2001; Munck 2009), no literature addresses the question of how one might construct a scale based on the properties of a concept. Despite efforts to overcome the qualitative/quantitative gap (Brady, Collier 2010), the twin tasks of conceptualization and measurement seem far removed from one another. Insofar as a tradition of concept-driven measurement exists it is located in the binary scale, where a concept is operationalized as a series of 0’s and 1’s. Commonly, membership criteria consist of one or more necessary conditions, jointly understood as necessary-and-sufficient. Occasionally, sufficient conditions are invoked (Goertz 2006). In either case, binary scales are generally constructed with a view to represent ordinary meanings and/or important theoretical properties of a concept. This is central to the “classical” tradition of concept formation (Collier, Gerring 2009; Sartori 1984) and to set-theoretic approaches of social science (Goertz 2006; Goertz, Mahoney 2012; Schneider, Wagemann 2012). It is also implicit in experimental and quasi-experimental studies, where treatments are usually understood in a binary fashion and are derived from a priori research hypotheses (Shadish, Cook, Campbell 2002). In the study of democracy, one binary scale, known as Democracy-Dictatorship (DD), has come to play an especially influential role. According to Przeworski and colleagues (Alvarez et al. 1996; Cheibub et al. 2010: 69), a regime is a democracy if leaders are selected through contested elections. To operationalize this conception, they identify four criteria:

1. The chief executive must be chosen by popular election or by a body that was itself popularly elected.

2. The legislature must be popularly elected. 3. There must be more than one party competing in the elections. 4. An alternation in power under electoral rules identical to the ones that brought the

incumbent to office must have taken place.

Like many binary scales, the DD index adopts a minimal definition of the theoretical concept and operationalizes that concept with one or more necessary conditions (in this case, four), all of which must be satisfied in order to receive a score of 1 (=democracy). The main complaint of binary scales when imposed on complex concepts is that they reduce all aspects of that concept to two categories, converting a plethora of information into a series of 0’s and 1’s (Elkins 2000). Of course, this procedure is perfectly reasonable (a) if the associated properties of a concept are highly correlated with the binary division or (b) if the associated properties are randomly distributed across the two groups. However, with non-experimental data the likelihood of either (a) or (b) is slight. Associated properties of democracy are not likely to be correlated perfectly with the binary scale; nor are they likely to be distributed in a random fashion. Likely as not, they will be somewhere in between. This means that the resulting scale is difficult to interpret. It is neither a proxy for democracy at-large nor a uniform treatment, and is apt to be associated with confounders (other properties of democracy that don’t align with the binary index).1

1 Consider the DD index, as defined above, and consider an attribute that is probably associated with its defining features such as executive constraints (EC). Let us suppose that EC can be conceptualized in a binary fashion, and let us suppose that it correlates partially with the DD index: countries that are scored 1 on the DD index are more likely to score 1 on a binary index of EC, but the association is not perfect. It follows that any causal analysis that places DD on the right side of the model runs into a problem of omitted variable bias if EC is unmeasured and collinearity (and possible endogeneity) if it is included. (We are presuming that EC may be affected by DD, but is not entirely the product

5

Binary scales play a vital role in social science and also constitute an important link to the classical tradition of concept formation. However, they cannot carry all the freight that is sometimes assigned to them.

II. A Lexical Scale There may be a way to preserve the virtues of conceptually driven scales with the need for more differentiated scales. This, in brief, is the strategy of the lexical scale, which incorporates necessary-and-sufficient conditions as distinct levels of an ordinal scale. Before discussing the properties of the lexical scale and comparing it with other scales it will be helpful to sketch out the procedure envisioned for the formation of such a scale. This procedure begins by identifying relevant attributes of a concept. In order to make sure that all possible attributes are considered, and none arbitrarily excluded, it is important to survey definitions and usage patterns of a concept in ordinary language and in whatever specialized language region may be relevant to the research. The initial culling should be as comprehensive as possible, excluding only idiosyncratic features.2 Next, one must arrange these attributes so that each attribute serves as a necessary-and-sufficient condition within an ordered scale. That is, each successive level is comprised of an additional condition, which defines the scale in a cumulative fashion. Condition A is necessary and sufficient for L1; conditions A&B are necessary and sufficient for L2; and so forth, as illustrated in Table 1. If there are five levels to an index, five necessary conditions must be satisfied in order to justify a score of 5. This means that each level in a lexical scale is defined by a set of conditions that are both necessary and sufficient, fulfilling a goal of the classical concept. Note that the structure of a lexical scale presupposes that there is a true zero, representing phenomena that do not meet the first condition (~A). [Table 1 about here] In achieving these desiderata four criteria must be satisfied: (1) binary values, (2) unidimensionality, (3) qualitative differences, and (4) centrality or dependence. First, each level in the scale must be measurable in a binary fashion without recourse to arbitrary distinctions. It is either satisfied or it is not. To be sure, the construction of a binary condition may be the product of a set of necessary and/or sufficient conditions. Collectively, however, these conditions must be regarded as necessary and sufficient. Second, levels in a lexical scale must be understood as elements of a single latent (unobserved) concept. Empirical multidimensionality may persist, as discussed below. However, conceptual multidimensionality must be eliminated, either by dropping the offending attribute and/or by re-defining the concept in a clearer and perhaps more restrictive fashion. Third, each level must demarcate a distinct step or threshold in a concept, not simply a matter of degrees. Levels in a lexical concept identify qualitative differences. A “3” on a lexical scale is not simply a midway station between “2” and “4.” Indeed, each level may be viewed as a subtype of the larger concept. Note that these sub-types are defined by cumulative combinations of the attributes possessed by the full concept – A, A&B, A&B&C, and so forth – fulfilling the criterion of a classical concept.

of DD.) Thus, even when binary coding is transparent and consistent with a common reading of the concept, as with DD, the resulting index may engender confusion. 2 Examples of this sort of semantic surveying can be found in Collier, Gerring (2009) and Sartori (1984).

6

The most challenging aspect of lexical scale construction is the ordering of attributes, which follows a conceptual (rather than empirical) logic. One attribute may be considered prior to another if it is more central to the concept of theoretical interest (from some theoretical vantage point). This follows a constitutive approach to measurement, where attributes are the defining elements of a concept. Alternatively, one attribute may be considered prior if it is a logical, functional, or causal pre-requisite of another. The dependence of B on A is what mandates that A assume a lower level on a scale. Whether responding to considerations of centrality or dependence, the levels of a lexical scale bear an asymmetric relationship to each other; some are more fundamental than others. This is the most distinctive feature of a lexical scale, and clearly differentiates it from Mokken scales (discussed below). While traditional measurement models consider attributes in an independent and additive fashion, the lexical scale presumes that attributes can be understood only according to their inter-relationship with one another. A concept is thus defined by specific combinations of attributes.3 Where lexical ordering is unclear a priori (according to considerations of centrality and dependence) one is well-advised to consider the shape of the empirical universe. Specifically, if A is always (or almost always) present where B is present, there may be grounds for considering A as more central or more fundamental than B. However, any conclusions reached on the basis of an exploration of empirical properties must be justified as a matter of centrality or dependence. Thus, we regard the relative prevalence of attributes as a clue to asymmetric relationships among the properties of a concept, not as a desideratum. In constructing a lexical scale, deductive considerations trump data distributions. Deliberation and Disagreement

A product of deliberation, the lexical scale follows the procedure by which John Rawls (1971) orders the three core principles that establish his theory of justice – (1) the Liberty principle, (2) the Fair Equality of Opportunity principle, and (3) the Difference principle. These are arranged in order of lexical (short for lexicographical) priority. That is, one should not consider 2 or 3 until 1 has been fully satisfied, nor 3 until 1 and 2 are fully satisfied. Thus, each principle serves as a necessary condition of the next, creating a lexical scale with four levels. The force of this argument hinges, in large part, on a conceptual argument – that the core meaning of justice is reflected in this particular ordering of attributes, with category (1) understood as the most basic or essential (Moldau 1992). Of course, Rawls was interested in defining the terms by which institutions within a society could be established and justified. He was not interested in measuring the presence/absence or degrees of justice in a society, and it is not clear whether he would have applied the same rules to such a measurement instrument. Even so, his approach is remarkably similar to that which we envision for empirical concepts in the social sciences. The lexical scale depends upon reaching a reflective equilibrium with respect to the defining attributes of a concept and their lexical ordering.4 Insofar as this solution is persuasive, the scale will be useful. Insofar as it strains the meaning of a concept or theory it will seem arbitrary and forced, and is – on that account – unlikely to perform any useful function in social science. A lexical scale must resonate with everyday usage of a word as well as with considered judgments about what a concept should mean in a given theoretical context. Just as there are disagreements over the meaning of justice, so there will be disagreements over how to scale empirical concepts (concepts whose purpose is to capture something specific and measurable “out there”). These may be placed into three categories.

3 This echoes the approach of qualitative comparative analysis (QCA) to empirical relationships more generally (Schneider, Wagemann 2012). 4 On the concept of reflective equilibrium see Daniels (1996).

7

First, we must consider disagreements over the definition of a term. Evidently, many high-order concepts in the social-science lexicon are contested (Collier et al. 2006), and this sort of contestation necessarily affects a concept-driven approach to measurement (though it also affects any other approach to measurement). In particular, it may affect the attributes included and excluded as conditions in a scale. Yet, because the lexical scale is developed by reference to a concept’s core meaning we anticipate that disagreements over attribute inclusion are likely to affect positions on the periphery of the scale. As such, they will impact only those cases whose value is determined by features at the high end of the scale. For example, if two 7-point scales for the same concept differ in the chosen attributes these differences are most likely to be located at levels 6 and 7 and least likely to be located at levels 1 and 2. As such, only cases with the highest scores (i.e., those whose score is affected by the 6th or 7th conditions) will be affected. It follows that measurement error arising from errors in the hierarchy are more likely at the high end of the scale than at the low end. The second sort of disagreement concerns the number of levels assigned to a lexical scale. Evidently, scholars working on the same concept may produce scales of differing lengths. Indeed, the decision about when to aggregate and when to disaggregate attributes (to form conditions) is somewhat arbitrary, hinging on contextual matters such as the sort of data that is at-hand and the use envisioned for the scale. For example, a scale developed for use on the right side of a causal model (X) is likely to be more concise than a scale developed for use on the left side of a causal model (Y), as discrimination is vital in measuring outcomes whereas in constructing independent variables one usually strives to limit the number of treatments. In principle, there is no limit to the number of levels in a lexical scale. In practice, we anticipate that the number of levels will rarely surpass ten. This is because some concepts do not have a great many (truly distinct) attributes and because, even where a great many attributes are present, they are unlikely to follow the logic of lexicality. In any case, disagreement over scale length is not critical, as differently-sized scales will co-vary so long as the identifiable elements are ordered in the same fashion. Compare two hypothetical scales: I (A-B-C) and II (A1-A2-B1-B2-C1-C2), where the latter disaggregates each element of the former into two components. These alternate scales for the same concept are different insofar as one has more levels than the other; but they are not in conflict with each other. A third, more damaging, sort of disagreement concerns the lexical priority of different elements. If researchers cannot agree on how to prioritize the attributes of a concept there is little hope of arriving at a useful lexical scale – or, to put the matter differently, each scale will be useful only in a very narrow context and may seem idiosyncratic. This sort of basic-level disagreement is encountered whenever the attributes of a concept bear no apparent relationship – functional or logical – to each other or where multiple attributes are equally important to the core concept. While there is no simple solution to this situation, one strategy is to redefine the boundaries of the concept in a narrower fashion so as to exclude elements that cannot easily be integrated. This may be understood as a shift from a background concept to a systematized concept (Adcock, Collier 2001), and causes no damage so long as the re-definition is plausible (it does not strain the meaning of the core concept), or can be communicated by a compound noun that makes clear how the narrower concept relates to the parent concept. Accordingly, we eschew democracy in favor of a diminished subtype, electoral democracy. Examples in Brief

As with most things, it is easier to grasp the workings of a method when specific examples are brought into view. In this section we provide a cursory exploration of possible lexical indices for the

8

following well-known concepts: (1) civil liberty, (2) party strength, and (3) rule of law.5 Each discussion begins with a brief definition, clarifying how we understand the concept. This is followed by a proposed index, following the principles of the lexical scale (as laid out above). Civil liberty is a human right, as well as a key component of democracy. Here, we understand civil liberty as a property of a government, including parties, civil society groups, and paramilitary groups that are closely associated with that government. We intend to measure the extent to which governments respect civil liberties, not the extent to which civil liberties exist in a society.6 Another important caveat is that we are concerned with the actions of government relative to the citizens of a polity. Its actions towards non-citizens (foreign nationals) lie outside the boundaries of our concept. Finally, it should be clarified that we are concerned primarily with the relevance of civil liberty to citizens. That is, we order the components of the index according to their presumed importance to individuals living in that society.7 This is how we arrive at judgments of relative centrality across attributes. With these qualifications, the following Lexical index is proposed:

0. Political and extra-judicial killings.

1. No political and extra-judicial killings. The government does not organize or condone arbitrary killings or the killing of dissidents or of citizens based on their ascriptive characteristics (e.g., ethnic minorities).

2. No torture. The government does not organize or condone the torture of dissidents or of citizens based on their ascriptive characteristics (e.g., ethnic minorities).

3. Due process. The government does not arbitrarily arrest, imprison, or harass its citizens.

4. Free movement. The government does not restrict movement and residence within the polity.

5. Free discussion. The government does not restrict discussion in private arenas (among family, friends).

6. Free public speech. The government does not restrict speech in public arenas including the media.

7. Free association. The government does not restrict association, including political parties, labor unions, religious organizations, and other civil society organizations.

Political parties may be defined minimally as organizations that nominate officials for public office, a key function in most theories of representative democracy (Schumpeter 1950). Within this context, the relative strength of these organizations is often considered to be an important component of democracy and perhaps also of good governance (Ranney 1962; Schattschneider 1942). Party strength within an authoritarian context may also be important, e.g., for regime stability. However, measuring party strength in this context would require a different sort of scale since parties function quite differently in authoritarian contexts (Brownlee 2007). Thus, our focus is restricted to parties as they operate within minimally democratic settings. Party strength is understood here as the mean strength of all political parties that gain entrance into the legislature,

5 For further examples of concept operationalizations that seem to follow the logic of a lexical scale one might consider constitutionalism (Nino 1998: 3-4), human security (Tadjbakhsh & Chenoy 2007: Ch.2), peasants (Kurtz 2000: 96), and liberal democracy (Howard, Roessler 2006; Møller, Skaaning 2013). 6 Of course, one might argue that it is the responsibility of governments to protect civil liberties, regardless of who is infringing upon them. Nonetheless, it seems important to distinguish between the actions of governmental and non-governmental actors. 7 One might also consider the issue from the perspective of Rawls’ original position: which attributes would one consider most important in establishing a society de novo?

9

and is not to be confused with party system strength (the durability of a set of parties within a polity). With these clarifications, we propose the following Lexical index:8

0. Not allowed. Parties are not allowed to organize.

1. Allowed. Parties are allowed to organize. If the system is minimally democratic, the state may restrict entry to small parties judged to be hostile to democratic principles.

2. Independence. Parties are independent of the state (e.g., the bureaucracy, the military) and independent of each other (though naturally members of a coalition will be to some extent constrained by coalition agreements).

3. Defections rare. Party officials rarely leave their party voluntarily (to join another party or to continue their political career as an independent). Expulsions and retirements are not counted as defections.

4. Legislative cohesion. Members of a party usually vote together in the legislature.

5. Centralization. Parties do not have strong factions or regional strongholds with distinct organizational structures; important decisions over policy and candidate selection are made at the center, or can be overturned by central party leadership.

6. Programmatic. Parties publically embrace policies and ideologies that are relatively distinct.

The rule of law is a virtually universal political ideal which has in recent decades been identified as crucial for economic and human development (Tamanaha 2004: 1-4). Among the varying definitions of this concept, most of the attributes may be understood along a continuum from “thin” to “thick” conceptions of the concept (Bedner 2010; Møller, Skaaning 2012; Tamanaha 2004; Trebilcock, Daniels 2008: 12-13). In this spirit, we suggest the following Lexical index:

0. No rule by law.

1. Rule by law. Law is used as instrument for government action.

2. Formal legality. Laws are general, clear, prospective, certain, and equally applied.

3. Institutional checks. An institutionalized system of government characterized by checks and balances, including an independent judiciary and penalties for misconduct.

4. Civil liberties. Liberal (negative) rights in the form of physical integrity rights and First Amendment-type rights are safeguarded.

5. Democratic consent determines laws. The citizens, through their elected representatives, are the ultimate source of laws.

It should be clear that our purpose in offering this set of examples is to demonstrate the potential applicability of the lexical scale to a diverse array of concepts in the social sciences, not to make claims about particular indices. Evidently, a great deal more work would need to be done before promulgating the foregoing indices for general consumption, and especially in operationalizing each condition.

III. Electoral Democracy

8 The index does not include a consideration of party nationalization. In our view, parties may be strong while also being rooted in particular regions, as in the United Kingdom.

10

Having explored several concepts in a cursory fashion, we now turn to a more extended example. Our focus will be on electoral democracy, the idea that democracy is achieved through competition among leadership groups which vie for the electorate’s approval during periodic elections before a broad electorate. This view of our subject is sometimes referred to as polyarchy, contestation, or as the competitive, elite, minimal, realist, or Schumpeterian conception of democracy (Dahl 1956, 1971; Przeworski et al. 2000; Schumpeter 1950). In identifying attributes for possible inclusion we are mindful of the vast literature on this topic, with special attention to linguistic studies of the concept (e.g., Held 2006; Lively 1975; Naess 1956) and foundational works on the electoral conception of democracy (see above). From these attributes we tried to identify those that were most central to the concept, combining attributes where they seemed to express the same general idea and discarding attributes if they seemed peripheral to the electoral conception of democracy. Conveniently, most of these attributes can be understood in a binary fashion (as being present/absent). Equally important, it seemed plausible to regard them as cohering to a unidimensional latent variable, i.e., more/less electoral democracy. To order these attributes in a single index we attempted to gauge their relative centrality or interdependence. In this fashion, the existence of elections was judged fundamental since no other attributes commonly associated with electoral democracy make any sense outside of an electoral context. One could not say, for example, that Country A is more of an electoral democracy than Country B if neither polity holds elections, regardless of what other characteristics those polities might possess. Likewise, some of the attributes depended upon other attributes in a logical manner. For example, one cannot hold multi-party elections unless an electoral regime is in place. Finally, some of the attributes seem to depend for their meaning on other attributes in a functional manner. For example, the extent of suffrage allowed in an election is of little import unless elections count for something, i.e., unless they allow for multi-party competition and the most important policymaking offices are elective. Whether or not suffrage is restricted or universal in North Korea is not a question that is generally regarded as consequential. (North Korea would hardly be less democratic if it decided to restrict access to the ballot, while changing no other aspect of its totalitarian system.) After considerable deliberation, we arrived at a Lexical index with six conditions and seven levels, as follows:

0. No elections. Elections are not held for any national-level policymaking offices. This includes situations in which elections are postponed indefinitely or the constitutional timing of elections is violated in a more than marginal fashion.

1. Elections. There are regular national elections.

2. Multi-party elections. Opposition parties are allowed to participate in legislative elections and to take office.

3. Executive accountability. The chief executive is accountable either directly to the electorate or indirectly to an elected parliament.

4. Competitive elections. The chief executive offices and the seats in the effective legislative body are filled by elections characterized by uncertainty, meaning that the elections are, in principle, sufficiently free and fair to enable the opposition to gain power.

5. Male or female suffrage. Virtually all adult male or female citizens are allowed to vote in national elections.

11

6. Universal suffrage. Virtually all adult citizens are allowed to vote in national elections.

Several features of this index deserve clarification. First, our reading of the concept of electoral democracy is fairly narrow. We do not presume that the electorate is encompassing. Indeed, it may be highly restrictive – though it must be separable from, and much larger than, the group of officials it is charged with selecting. Thus, South Africa under Apartheid may be considered an electoral democracy for white South Africans, just as Ancient Athens functioned as a direct democracy for male citizens. Likewise, in measuring suffrage we take a juridical approach. Suffrage is achieved when constitutionally prescribed, even though local or informal practices may impede the achievement of this right (as in the American South prior to the Civil Rights movement). This is consistent with the usage of the concept by Schumpeter, Dahl (for “polyarchy”), Przeworski, and also with many extant indices such as the Polity2 index. Second, indirect elections do not qualify as elections unless the electors endorse specific candidates or parties, as in US presidential elections. In the absence of explicit endorsements, indirect elections usually serve to restrict competition, allowing the in-group to monopolize political power. Third, electoral democracy does not presume complete sovereignty. A polity may be constrained in its actions by other states, by imperial control (as over a colony), by international treaties, or by world markets. In sum, to say that a polity is an electoral democracy, according to our proposed index, is to say that it functions as such (a) for those who are allowed to vote and (b) for policies over which it enjoys decision-making power. To be sure, expansions of this rather minimalist approach to electoral democracy can be envisioned. One might, for example, include a seventh condition measuring high electoral integrity (i.e., only minor irregularities with respect to intimidation, disruption, violence, registration, and high respect for political liberties such as freedom of expression, assembly, and association). Lexical indices for a given concept can usually be crafted so to as to encompass a greater or lesser number of levels, as noted. For present purposes, we restrict ourselves to what might be considered the most basic or fundamental aspects of electoral democracy. (Extant sources do not provide sufficient guidance on the aforementioned factors to allow for historical coding, as set forth in the next section.) The reader should be aware that electoral democracy identifies one aspect, or dimension, of the parent concept. Other dimensions of democracy – sometimes articulated as liberal, deliberative, egalitarian, or participatory – are ignored. Presumably, lexical scales might also be developed for these concepts. In any case, the utility of this definition and operationalization of democracy (like all others) rests ultimately on how faithfully it explains the world around us. Specifically, the electoral interpretation of democracy rests on a theory about which aspect of democracy has the greatest impact on governance, wellbeing, and on other aspects of democracy (liberal, deliberative, et al.). Schumpeter, Przeworski, and many other writers were convinced that the electoral component of democracy is causally exogenous, and it is on this basis that we separate it out from other dimensions. Likewise, our electoral index is premised on an idea of which features of electoral democracy are likely to be most fundamental. It is on this basis that we included some attributes and excluded others, and arrived at a lexical ordering of those that were included. In the following sections, we offer a brief exploration of how the Lexical index might be applied to the world of nation-states. A New Dataset

To apply the Lexical index of electoral democracy in the broadest manner possible we code all sovereign or semisovereign countries from 1800 to 2008, generating a new dataset with 228

12

countries and 17,176 country-years.9 Crossnational coding sources include the Political Institutions and Political Events (PIPE) dataset (Przeworski et al. 2013) and the Dataset of Political Regimes (Boix et al. 2013). In addition, we consult numerous books, articles, and reports in order to code missing observations, to corroborate or revise codings suggested by the principal sources, and to create a new indicator of competitive elections. A detailed description of sources and coding procedures is contained in Appendix A. In order to get a feel for the application of this index we provide country scores for the median year in our sample, 1904, as shown in Table 2. At that time, there were fifty-three independent countries in the world. These were distributed fairly evenly across the 7 categories of the Lexical index, with the exception of the most democratic category (6), which has only one occupant. (Only Australia granted universal suffrage to both men and women, while satisfying the other criteria stipulated in the index.) [Table 2 about here] A frequency distribution of scores across the entire 1800-2008 period is provided in Table 3. It will be seen that the most populated categories are L0, L1, L3, and L6, while others (notably L5) have fewer occupants. Although a fairly high proportion of cases stack up at the two ends of the index the distribution of cases is not bimodal, a problem that affects ordinal/interval indices such as PR and Polity2 (as noted by Cheibub et al. 2010: 77; see also Treier, Jackman 2008). The distributions of cases changes over time, as one might expect. This feature may be portrayed in a stacked graph, as shown in Figure 1. Here, the width of each category represents the number of countries falling into that category for each year from 1800 to 2008. Note that our sample grows over time – from 27 in 1800 to 194 in 2008 – due to the appearance of newly sovereign states (e.g., in Africa) and the break-up of sovereign states (e.g., the Soviet Union). [Figure 1 about here] Figure 1 also illustrates a key feature of a lexical scale, namely the possibility of decomposing a concept into constituent parts – parts that are intrinsically meaningful because they represent qualitative (i.e., step or threshold) differences. Each lexical scale contains an implicit typology. This allows one to interpret category membership – and changes in membership over time – in a meaningful way. For example, one can evaluate the composition of political regimes at each point in time over the past two centuries. In 1800, polities were predominantly of type 0 (no elections). Later in the nineteenth century, we see the rise of types 1-5 and the concomitant decline of type 0. This is the most diverse period, when no single type is dominant, as illustrated by our snapshot of the world in 1904 (see Table 2). Over the course of the 20th century, we can see the extraordinary rise of type 1 (elections without multiparty competition), followed by a steep decline beginning in the 1980s and coincident with the Third Wave of democratization (Huntington 1991). Type 3 (multi-party executive and legislative elections without real competition), which had been a modestly sized category for a century, begins to grow in the late 20th century to the point where it constitutes the second-most dominant regime-type. It is worth noting that this regime-type is quite similar to polities described as competitive authoritarian (Levitsky, Way 2002) or limited multi-party (Hadenius, Teorell 2007). The other striking pattern over the past century is the rise of type 6, the highest level of our Lexical index, corresponding to polities that satisfy all six criteria. This category now comprises over half of all polities in the world. Figure 1 also offers insight into those periods in which electoral democracy advanced throughout the world (e.g., at the end of World War I, World War II, and the Cold War), as well as those periods in which it declined (e.g., the 1930s). By way of contrast, interval indices such as the Unified Democracy Scores (Pemstein et al. 2010) allow one to track the overall increase and decrease in democracy. Ordinal indices such as

9 In comparison, the coverage of the Polity2 variable for the same period is 189 countries and 15,823 country-years.

13

Political Rights (or, perhaps, the Polity2 scale) are similar in this respect, given that the levels in the index do not indicate categories that are qualitatively different.10 As such, extant indices are capable of indicating overall trends – more or less democracy – but they cannot indicate anything about the specific content (quality) of regimes, or about which regime-types expanded or contracted at different points in time. The latter information is both substantively important as well as useful for tracing causal mechanisms. As is to be expected, our Lexical index generally co-varies with other measures of the same general concept, as shown in Table A1. For example, it correlates with Polity2 at 0.79 and with the Political Rights index at 0.85 (Spearman’s rho). Dropping the highest scoring cases (Lexical=6), i.e., those codings likely to be least controversial, these correlations drop to 0.59 and 0.43, respectively. These are not especially high correlations, suggesting that empirical relationships with political regime type as the independent or dependent variable are likely to reveal different patterns when explored with our Lexical index, relative to extant (interval, ordinal, and binary) indices. In sum, the Lexical index allows us to represent more information than is possible in a binary scale (e.g., DD) but without the ambiguities of a composite scale whose composition is derived either from complex models (e.g., UDS) or a less formulaic but often opaque weightings across dimensions (e.g., Freedom House and Polity). The level affixed to a country in a particular year is immediately interpretable. The cost is that elements of democracy not associated with a narrow definition of electoral democracy are left aside. Empirical uses

One purpose of the Lexical index of electoral democracy is univariate and descriptive: to differentiate regime-types in the world (e.g., Table 2) and to portray changes over time (e.g., Figure 1). Another use is to show relationships between regime-type and other factors. Such inter-relationships may be descriptive, causal, or putatively causal (where causal inference is suspected but not entirely proven). Consider the vaunted democratic peace hypothesis (Brown et al. 1996). While a new scale of democracy will assuredly not solve this obdurate research question by itself, it does allow a more nuanced test of the thesis (at least as pertains to the electoral components of democracy). Specifically, we can explore whether there is a specific level in the Lexical index beyond which conflict between nations ceases to occur, and whether one or both members of the dyad must surpass that threshold. This is arguably more informative than a binary or ordinal/interval analysis of the problem.

10 The 7-point Political Rights index, constructed by Freedom House, is explained as follows: “1. Enjoy a wide range of political rights, including free and fair elections. Candidates who are elected actually rule, political parties are competitive, the opposition plays an important role and enjoys real power, and minority groups have reasonable self-government or can participate in the government through informal consensus. 2. Have slightly weaker political rights than those with a rating of 1 because of such factors as some political corruption, limits on the functioning of political parties and opposition groups, and foreign or military influence on politics. 3-5. Include those that moderately protect almost all political rights [and] those that more strongly protect some political rights while less strongly protecting others. The same factors that undermine freedom in countries with a rating of 2 may also weaken political rights in those with a rating of 3, 4, or 5, but to an increasingly greater extent at each successive rating. 6. Very restricted political rights. They are ruled by one-party or military dictatorships, religious hierarchies, or autocrats. They may allow a few political rights, such as some representation or autonomy for minority groups, and a few are traditional monarchies that tolerate political discussion and accept public petitions. 7. Few or no political rights because of severe government oppression, sometimes in combination with civil war. They may also lack an authoritative and functioning central government and suffer from extreme violence or warlord rule that dominates political power.” Downloaded 1/16/2013 from www.freedomhouse.org/report/freedom-world-2012/methodology.

14

As a second example, one might consider the equally contested relationship between development and democracy (Przeworski et al. 2000). With democracy on the left side of the model, one may investigate whether the empirical relationship of socio-economic development to electoral democracy is different at various points in the Lexical index. Do increases in per capita GDP have a greater impact on electoral democracy at certain thresholds? With democracy on the right side of the model, one might investigate whether different thresholds of electoral democracy have varying relationships to economic growth (Gerring, Skaaning 2013). For example, does the initial transition to multiparty elections have a different impact on growth performance than the transition to competitive elections? Another research agenda that lends itself to reassessment is the role of democracy in structuring state repression of personal integrity rights (Davenport, Armstrong 2004). Here, we shall explore the empirical relationship in order to demonstrate in greater detail how the Lexical index may be brought to bear on a (putatively) causal relationship. Democracies are expected to be less repressive than autocracies for a variety of reasons. A democratic framework is thought to promote tolerance; low respect for human rights may be punished by the electorate at the ballot box; and political participation and contestation provide an outlet for protests and secure legitimacy in the broad population, alleviating the extra-constitutional challenges that often spur violent government repression. Extant theory thus presents a strong prima facie case for political regime type as an influence on state repression. However, it is not clear what the precise empirical relationship might be. Extant work on the subject suggests three possible patterns. As summarized by Davenport and Armstrong (2004: 538-39) [hereafter D&A]: (1) “with every step toward democracy, the likelihood of state-related civil peace is enhanced”; (2) “human rights conditions are not only improved when full democracy exists but also when full autocracy is present”; or (3) “there may…be some threshold of domestic democratic peace, below which there is no effect of democracy on repression, but above which a negative influence can be found.” To investigate this question we adopt the empirical format employed by D&A, with some minor modifications to update the analysis to include all years until 2008.11 State repression is measured by the widely used Political Terror Scale (PTS), based on the State Department human rights country reports (Wood, Gibney 2010). Like D&A we enlist the OLS (TSCS) regression to assess the model, and we employ a similar battery of covariates: interstate armed conflict (UCDP, categories 1-3 collapsed), internal armed conflict (UCDP, categories 1-3 collapsed), military dictatorship (Cheibub et al. 2010), population (ln) (PWT), GDP/cap (ln) (PWT), and a one-period lag of the outcome. Following D&A, democracy is measured by the 10-point Polity Democracy index (scaled from 0 to 10) drawn from the Polity IV dataset. Our second measure of democracy is the Lexical index, with one coding change. Note that since data on state repression is available only from 1976 there is little variation in suffrage laws during the observed period.12 Distinctions across levels L4-L6 of the Lexical index are therefore rendered moot, prompting us to collapse L4-6 into a single category (L4). The resulting index has five levels – L0-L4, with roughly equal membership – rather than seven (L0-L6), and is otherwise identical to the index described above. But how does democracy impact state repression? In this discussion we shall leave aside the knotty problem of causal inference. This is not to gainsay the importance or the knottiness of the

11 Apart from Lexical, all data used is taken from the QoG standard dataset 2013. Downloaded 08/27/2013 from www.qog.pol.gu.se/data/datadownloads/qogstandarddata/ 12 Moreover, the analyses are limited to overlapping country-years in order to increase the comparability of the results.

15

problem. Rather, it is because issues of causal inference are extraneous to this comparison, whose purpose is to illustrate the value-added of a lexical approach to measurement. In Table 4, we begin by testing the possibility of a linear relationship between democracy and state repression. Polity (Model 1) and Lexical (Model 2) both indicate a negative relationship: more democracy is correlated with less repression, vindicating the general theory presented above. Next, we test the possibility of a curvilinear relationship by introducing a multiplicative term. The coefficients for Polity (Model 3) and Lexical (Model 4) are similar, though only Polity offers support for the notion that democracy’s impact on repression is nonlinear. Finally, again following the lead of D&A, we attempt to get inside the causal box by exploring each category of these indices separately through the use of dummy variables representing each level (with the first level omitted as a reference category). Results, shown in Models 5 and 6, are again broadly similar across the two indices, though there are some important differences. The coefficient for L1 in the Polity index is significantly more repressive than the reference category, L2-6 do not show results are statistically distinguishable from the null, and L7-10 show negative, and statistically significant coefficients. By contrast, L1, L2, and L4 in the Lexical index are statistically significant from the reference category, but not L3. Leaving aside for a moment the question of which index offers a truer representation of the relationship between democracy and repression, let us consider what might be learned from Models 5 and 6. D&A (2004: 548) conclude that “there are important differences between the political systems associated with the highest levels of the Polity measure…” – a reasonable conjecture. But they cannot follow this statement up with any speculation about what is distinctive about the higher levels of the Polity index or what might be driving the apparently curvilinear relationship between democracy and repression. This is because the levels of the Polity index are not individually interpretable. In this respect, ordinal indices of democracy such as Polity, PR, and CL function very much like interval indices. They inform us about quantities (more or less of some latent trait) but not about qualities (categorical differences across levels). By contrast, the Lexical index provides ample fodder for theorizing because each level defines a discrete category and each category is plausibly approached as a regime-type. Let us begin by reviewing the information contained in Model 6. Two levels in the Lexical index reveal higher levels of state repression, L0 (no elections) and L3 (multi-party elections but no executive accountability). This accounts for the curvilinear relationship discovered in Models 3-4. While it is unsurprising to discover that a nonelectoral state has high levels of repression (for all the reasons set forth in our initial theory), it is somewhat surprising to find that there is no (statistically significant) difference in levels of repression across L0 and L3. In other words, repression increases as a polity moves (hypothetically) from multiparty elections (L2) to multiparty elections plus executive accountability (L3). [Table 4 about here] An explanation may be found in the hybrid nature of this regime, which installs many of the constitutional forms of democracy without the crucial missing step in which elections are allowed to become competitive. That is, L3 polities look like they are democratic, and undoubtedly are portrayed by their leaders as democratic. But leaders have mechanisms to prevent the possibility of opposition parties gaining power (Schedler 2002). Opposition groups are free to organize and to participate in the political system but are not allowed to win. This setting may engender repression for several complementary reasons. First, the government is compelled to recognize the opposition; it cannot prevent them from organizing as it might in a one-party or no-party regime. Second, because the opposition is free to organize it is likely to pose a significant challenge to the government. And because it is not allowed to compete fairly in elections it is likely to pursue extraconstitutional measures, which in turn are likely to provoke government repression. Finally,

16

since the government cannot use constitutional means to maintain its power (for otherwise the opposition might win at the ballot box) it must resort to state repression. In short, both government and opposition have means and motive to engage in a cycle of protest and retaliation, a setting that is likely to feed high levels of state repression, as measured by the PTS. This short explanatory sketch is not intended to convince. In order to be fully convincing a causal explanation would need to be accompanied by a much longer theoretical discussion intended to make sense of case-based evidence and extant theorizing on this well-trodden subject (not to mention a battery of robustness tests). Our purpose is heuristic. We hope to have shown that a lexical approach to measurement provides a useful tool for gaining insight into (putatively) causal relationships and specifically into the causal mechanisms that may (or may not) be at work. Note that this information gained from categories on a lexical scale are useful for shedding light on why X might be a cause of Y and why it might not be a cause of Y. We are not proposing that a Lexical index has any claim to ontological priority over other sorts of indices, each of which represent certain aspects of reality and each of which has its uses. Sometimes, relationships are continuous (and hence best measured with an interval scale) and sometimes they have only one threshold (and hence best measured with a binary scale). By the same token, sometimes causal relationships are ordinal in character – or require an ordinal scale to test various threshold possibilities. In these settings, which surely apply to many theories about democratic development (as cause or effect) a lexical scale – where ordinal levels represent qualitatively different categories – may be appropriate.

IV. Situating the Lexical Scale Within the literature on concept formation, the lexical scale may be viewed as an attempt to reconcile minimal (thin) and maximal (thick, ideal-type) strategies (Coppedge 1999; Gerring 2012: ch 5). Note that the first condition (or first several conditions) establishes a minimal definition while the last condition in the scale completes what might be viewed as an ideal-type concept. Granted, ideal-type definitions are often more expansive than those envisioned by the lexical scale, primarily because the requirements of an ideal-type are less restrictive (anything that coheres with the concept is admissible). Even so, the lexical scale serves as a bridge between minimal and maximal concepts. The lexical scale is also closely linked to a long intellectual tradition focused on typologies (Bailey 1994; Collier et al. 2012; Elman 2005; Gerring 2012: ch 5; Lazarsfeld 1937; Lazarsfeld, Barton 1951). Note that each level of a lexical scale corresponds to a distinctive category or type, some of which may have recognizable names and identities. Within the literature on measurement the lexical scale is similar in structure to Guttman scales (Coppedge, Reinicke 1990; Guttman 1950) and Mokken scales, aka ordinal or nonparametric item response theory (Cingranelli, Richards 1999; Mokken 1971; Sitjtsma, Molenaar 2002; Sitjtsma, Debets, Molenaar 1990; van Schuur 2003, 2011). However, there are important differences. First, a Mokken scale typically attempts to integrate a large number of items, many of which are not qualitatively different from each other. In doing so, the resulting index has a better chance of identifying fine-grained differences among similar entities, at the cost of introducing levels in the scale that have no intrinsic or theoretical meaning. Second, in ordering these items the Mokken scale approach is inductive rather than deductive. Specifically, the most common attributes constitute the bottom rungs of the scale while the least common attributes constitute the top rungs.

17

Third, the inductive method of scaling is probabilistic, while the lexical method of scaling is deterministic. By this, we mean that a Mokken scale presumes that some cases will not fit neatly into the monotonicity presuppositions of the scale. (The degree of fit is captured in a statistic called Loevinger’s H, which is based on the frequency of Guttman errors compared to expected Guttman errors.) By contrast, the score for a lexical scale is deterministic because it arises from a series of necessary-and-sufficient conditions. Naturally, there is error associated with the coding of cases, but this error is at the level of brute data; sometimes, one does not have enough information at one’s disposal to code a particular case, or the coding criteria for a scale is too ambiguous to resolve coding decisions. Both Mokken and lexical scales are cumulative. However, all cells in the matrix are defined in a Mokken scale while only the “positive” elements of the scale are defined in a lexical scale. This may be seen by directly comparing lexical and Mokken scales with the same number of levels. (As we pointed out, this would be rare, as inductively derived indices generally seek to integrate many more items than the corresponding lexical scale.) Returning to Table 1, the reader will notice that all cells containing a term are defined while empty cells are undefined. By defined, we mean that they define the lexical scale in a deterministic fashion (successive necessary-and-sufficient conditions). If a case receives a score of 3 it must code positively on A, B, and C, and negatively on D. Condition E is undefined; whether it is satisfied or not will not affect the coding for that case. Table 5 presents a hypothetical Mokken scale, mirroring the features of Table 1 in all but one respect. Here, all cells are relevant, which means that the predictive capacity of the scale is much greater. Specifically, knowledge of a case’s score will allow one to predict values for all conditions in the scale, including both lower- and higher-level conditions. Of course, because the scale is probabilistic these predictions include error. This error should be interpreted as error in the fit between the model and reality, not measurement error in the usual sense (although the latter may be present as well). [Table 5 about here] Naturally, lexical and Mokken scales lead to different indices wherever the deductive properties of a concept do not line up neatly with its inductive properties. Consider what happens when we construct a hypothetical Mokken index of electoral democracy using the same seven elements we identified as components of our Lexical index. The ordering of elements in the Lexical index is presented in the first row of Table 6. The ordering of elements in a Mokken index – using the same sample of country-years and based on the relative frequency of each attribute – is located in the second row. The ordering of elements in two Mokken indices constructed from restricted samples is presented in the final rows. Note that not only are the Mokken scales quite different from the lexical scale; each of the three Mokken scales is also quite different from the others due to the fact that they draw on varying samples. [Table 6 about here] To say that lexical and Mokken scaling procedures sometimes arrive at different indices for the same general concept is not to say that one is necessarily superior to the other. Both surely have their uses. However, there are reasons to imagine that the solutions provided by Mokken/IRT models are less appropriate for some concepts than for others. In addressing this issue we shall compare concepts such as civil liberty, party strength, rule of law, and electoral democracy (which we explored in Section III) with concepts that have been central to the development of Mokken/IRT models in the fields of education and psychology such as intelligence and aptitude. Note that while the first class of concepts describes features of institutions, the second class of concepts describes features of individuals. (This is not the only aspect in which they differ, but it may be a critical one.) Four important differences across these two classes of concepts may be identified.

18

First, with individual-level concepts it is often possible to identify “outcome” indicators that measure the latent concept of interest. A subject’s performance on a test offers a good indication of their aptitude – if not of their overall intelligence at least of their knowledge in a subject area. With institutional concepts the outcome-based approach is often difficult to apply. Outcome measures of democracy might include the closeness of the vote between parties in an election or the frequency of turnover as the result an election (Gerring, Teorell, Zarecki 2013; Vanhanen 2000). While informative, such indicators obviously do not capture the entirety of the concept and can be misleading. (A large margin of victory, and infrequent turnover, may be a sign of citizen satisfaction rather than of authoritarian tendencies.) Thus, for institutional concepts the items on a scale are likely to consist of substantive attributes that define the concept of interest. In measuring electoral democracy, for example, an item might consist of the question “Are there competitive elections?” or “Is suffrage universal?” This is roughly equivalent to an aptitude test in geography that includes the question “Are you good at identifying place-names?”

Second, with individual-level concepts that can be measured with outcome indicators like answers to an aptitude test it is fair to assume that all questions (that relate in some way to the subject) are relevant and none are more important than others, except insofar as they might serve a more useful function in discriminating between subjects. A question’s utility derives from its function within the overall schedule of items on a test, and is easily judged by the extent to which it improves the overall capacity of that test to discriminate among subjects. By contrast, when institutional qualities are measured by the attributes that define them, it follows that some items will be more important than others (in defining the latent concept of interest). Consider the following two indicators of electoral democracy: “Are there competitive elections?” and “Are 18-21-year-olds allowed to vote?” Evidently, the first is more central to the concept (as normally understood) than the latter. Inductive methods of scale construction offer no way of sorting this out. (Granted, our lexical ordering of attributes results in an ordinal scale whose levels are probably not equidistant from each other. Nonetheless, the construction of the scale presumes that each level is qualitatively different and theoretically significant, guarding against the introduction of trivial conditions.)

Third, with individual-level concepts like intelligence it is often reasonable to construct a scale inductively by reference to the pervasiveness of different responses. This is because responses to items on a questionnaire often possess a cumulative quality. All subjects who answer a hard question correctly will also answer easier questions correctly. However, the empirical distribution of institutional concepts does not always follow this cumulative logic. For example, polities that are undemocratic may thwart the will of the people in different ways, meaning that they will not receive the same score on some questionnaire items even when they are equally authoritarian. IRT models do not provide a means for identifying functional substitutes (several individually sufficient conditions).

Finally, within individual-level concepts like intelligence the meaning of items on a scale are independent of each other. A subject’s answer to question #4 does not affect the meaning of her answer to question #7. However, in a survey purporting to measure democracy the meaning of “Is there universal suffrage?” changes depending upon the answer to the question “Are there competitive elections?” Specifically, it is not clear that universal suffrage has much significance to electoral democracy unless and until there are minimally competitive elections. This is not an empirical relationship; indeed, many countries hold elections without competition. It is, rather, about the meaning of the attributes relative to the underlying concept. This issue of non-independence does not arise in most individual-level concepts; accordingly, IRT models do not usually take into account relationships of logical necessity. (If they do, they are constructed in a highly deductive manner. Recall that our argument is not with technical methods of scale construction per se but rather with their use.)

19

Of course, some institutional concepts have properties that are conducive to Mokken/IRT scaling. And others can be scaled in different ways depending upon the goal of the researcher or one’s interpretation of a concept. Each scaling tradition has its uses. The point of our extended comparison is simply to indicate that, in some settings, the Mokken/IRT approach may result in problems of concept validity and, in these settings, more concept-driven approach to measurement may be warranted.

V. Discussion We shall now attempt to summarize our wide-ranging discussion pertaining to the strengths and weaknesses of the lexical scale. In principle, the recalcitrant aggregation problem is solved by treating defining attributes as necessary-and-sufficient conditions arrayed in an ordinal fashion.13 If the scale is true to its objectives, each level in the scale defines a stronger, more complete instantiation of the underlying concept. This is no mean feat, as composite indices are often plagued by problems of aggregation (Goertz 2006; Munck 2009). Indeed, wherever multiple indicators are combined in a single index there are usually multiple principles of aggregation that may be invoked, and rarely is there a definitive justification for choosing one over another. Because conceptualization is integrated into measurement there should, in principle, be less slippage between concept and indicator than is typically encountered with other methods of scale construction.14 However, if the analyst drops attributes of a concept from an index (because they cannot be meaningfully arrayed in an ordinal scale), forces continuous phenomena into an arbitrary binary coding, or prioritizes conditions without some underlying rationale, the resulting index will depart from ordinary meanings implied by the concept. The lexical scale is by no means immune to problems of concept/construct validity. Likewise, the strictures of the lexical scale are not universally applicable. They require that relevant attributes of a concept be coded in a binary fashion without undue distortion and that the chosen attributes be arrayed along a single dimension according to their centrality to the concept or relations of dependence. These are not easy requirements to satisfy. We have noted that the deductive properties of a lexical scale require many judgments on the part of the analyst. Accordingly, different analysts may arrive at different scales for the same concept. This, by itself, does not differentiate the lexical scale from scales constructed in a more inductive fashion. After all, there are many moving parts to any scale, particularly when one is attempting to operationalize a highly abstract concept. One must choose an indicator or set

13 This presumes, of course, that each condition can be accurately measured in a binary fashion without too much loss of information. 14 Authors’ choices of indicators to include in an index are often somewhat arbitrary (Goertz 2006; Haig, Borsboom 2008; Munck 2009). For example, the Worldwide Governance Indicator for “rule of law” (Kaufmann, Kraay, Mastruzzi 2007) primarily measures crime and property rights, downplaying or entirely excluding other attributes of the concept (Skaaning 2010). Likewise, the Freedom House Political Rights and Civil Liberties indices (a component of various latent-variable models of democracy) include indicators pertaining to corruption, civilian control of the police, the absence of widespread violent crime, willingness to grant political asylum, the right to buy and sell land, and the distribution of state enterprise profits (Freedom House 2007). Some observers might regard these features as elements of political rights and civil liberties; others might not. Since most abstract concepts can be defined in a variety of ways and do not possess sharp boundaries, it is no surprise to discover that one analyst’s bundle of indicators may be quite different from another’s, even when they purport to operationalize the same term.

20

indicators to represent a concept and, if more than one indicator is chosen, one must decide upon an aggregation technique(s) that combines those elements into a single scale. Accordingly, it is not the case that lexical scales are more “subjective” than other scales. Arguably, the assumptions employed in the construction of a lexical scale are more transparent than the assumptions used to construct many other composite indices, especially when a number of aggregation principles are embedded in a complex statistical model. On the other hand, because of the set-theoretic nature of the lexical scale it seems likely that alternative lexical scales (for the same concept) will be less highly correlated than varying inductive scales (for the same concept). A small change in an ordinal scale generally has greater consequences than a small change in an interval scale. Lexical scale construction is a highly deductive enterprise insofar as the resulting index is constructed to suit a priori requirements drawn from the concept rather than from the empirical distribution of the data. Yet, many concepts do not provide clear guidance with respect to the relative priority of their defining conditions, a limiting condition on the applicability of lexical scaling. Where applicable, however, the deductive properties of the scaling procedure offer certain advantages. Note that insofar as the distribution of data is allowed to govern the construction of an index the resulting index is sample-dependent. If key properties of a sample change (e.g., when drawn from different populations or when drawn non-randomly from a single population), so does the resulting scale. Sometimes, sample bias can be corrected with an IRT model; however, this presumes considerable knowledge about the larger population of interest. In many circumstances (e.g., where the population extends into the future), it is not possible to determine what the shape of a larger population looks like. In these situations, indices are biased – or, alternatively, they lack generalizability (they are sample-dependent). Note that the Mokken indices of electoral democracy that we constructed in Table 5 vary across each chosen sample period. Relatedly, a basic (and nearly universal) operating assumption of standard index construction is that one can combine information from observed variables by paying attention to their commonalities and discarding their differences (as error). This is a reasonable set of assumptions in many circumstances, especially when the commonalities are great and the remaining differences do not seem to represent anything of substantive significance (i.e., they do not compose an identifiable dimension). However, it involves a considerable simplification of reality, especially when intercorrelations are modest. In such circumstances, the lexical scale offers a viable alternative. With respect to discrimination, the lexical scale may be counted as modestly successful. It provides much more information than the classical concept, understood as a binary scale. It is on par with many ordinal indices, which generally incorporate a handful of levels. It is also on par with indices that purport to be interval scales but, in reality, are probably better understood as ordinal such as the Polity and Freedom House indices of democracy (Armstrong 2011; Cheibub et al. 2010; Pemstein et al. 2010; Treier and Jackman 2008). To be sure, a lexical scale will discriminate less successfully than a scale whose construction is geared to detect small differences (e.g., scales constructed with IRT models). While sensitivity to small differences is valuable, it is not the only factor of importance in constructing a scale. Note that some concepts in the social science universe are probably lumpy rather than continuous. This appears to be the case with electoral democracy. One is at pains to describe the difference between a regime with popular elections and one without (the first condition of our proposed Lexical index) as a matter of degrees. The same point might be made with reference to some of the other examples discussed in section III. Likewise, where a concept is being formulated as a right-side variable in a causal model it may be helpful to recognize distinct treatments, understood as a cumulative series of compound

21

treatments – A, A&B, A&B&C, et al. These can be tested with (a) pairwise comparisons and matching algorithms, (b) dummy variables in a regression model, (c) generalized additive models (GAM [Beck, Jackman 1998]), or (d) Bayesian shrinkage models (Alvarez, Bailey, Katz 2011). If used to achieve covariate balance in a matching analysis a categorical variable is generally more tractable than a continuous variable. In these respects, lexical scales are well-suited for causal inference. By contrast, inductively derived indices often function awkwardly on the right side of a causal model. A useful treatment is uniform, imposing the same condition on all those within the treatment group. However, indices usually include heterogeneous elements – a little bit of this and little bit of that, in portions that are difficult to account for. Typically, there are many ways to obtain a score of “3” along a continuous scale.15 Consequently, it is difficult to say what the treatment consists of, what causal mechanisms might be at work, and whether the resulting relationship should be interpreted as causal. Composite scales generally indicate differences of degree, but not of kind. A “4” on the Polity2 scale indicates that a regime is more democratic than a country receiving a “2.” But it offers no additional information about the qualities of these regimes. In this respect, the information contained in a standard composite index is “quantitative” (more/less) rather than “qualitative” (differences of type). Accordingly, a point on a composite index rarely has an obvious interpretation or meaning except in terms of standard deviations from the mean, and thresholds used to convert a continuous scale into a nominal or ordinal scale are apt to be highly arbitrary. This makes it difficult to evaluate concept validity, even if aggregation rules are perfectly transparent. And it makes it difficult to apply concepts to real-world situations, detracting from social science’s relevance to politics and policy. By contrast, a lexical scale is relatively transparent. Researchers and reviewers know exactly what a shift from “2” to “3” or “3” to “4” means because each level in the scale is achieved by only one additional criterion. This eases the burden of ex ante coding and ex post interpretation. Likewise, insofar as levels correspond to distinctive types, membership in each category of an ordinal scale is meaningful. Units coded as “3” share various characteristics, which may signal important theoretical properties (e.g., as inputs or outputs of a causal model). Qualitative differences are sometimes more informative than quantitative differences.

15 With respect to the Freedom House indices, Cheibub et al. (2010: 75) note: “for each of the ten categories in the political rights checklist and the 15 categories of the civil liberties checklist, coders assign ratings from zero to four and the points are added so that a country can obtain a maximum score of 40 in political rights and 60 in civil rights. With five alternatives for each of ten and 15 categories, there are 510 = 9,765,625 possible ways to obtain a sum of scores between zero and 40 in political rights, and 515 = 30,517,578,125 possible ways to obtain a sum of scores between zero and 60 in civil liberties. All of these possible combinations are then distilled into the two seven-point scales of political rights and civil liberties.”

22

VI. References Adcock, Robert, David Collier. 2001. “Measurement Validity: A Shared Standard for Qualitative and

Quantitative Research.” American Political Science Review 95:3 (September) 529-46. Alvarez, R. Michael, Jose A. Cheibub, Fernando Limongi, and Adam Przeworski. 1996. “Classifying

Political Regimes.” Studies in Comparative International Development 31(2): 3–36. Alvarez, R. Michael; Delia Bailey; Jonathan N. Katz. 2011. “An Empirical Bayes Approach to

Estimating Ordinal Treatment Effects.” Political Analysis 19:1, 20-31. Armstrong II, David A. 2011. “Stability and Change in the Freedom House Political Rights and Civil

Liberties Measures.” Journal of Peace Research 48(5) 653–62. Bailey, Kenneth D. 1994. Typologies and Taxonomies: An Introduction to Classification Techniques. Thousand

Oaks: Sage. Beck, Nathaniel; Simon Jackman. 1998. “Beyond Linearity by Default: Generalized Additive

Models.” American Journal of Political Science 42:596–627. Bedner, Adriaane. 2010. “An Elementary Approach to the Rule of Law.” Hauge Journal on the Rule of

Law 2(1): 48-74.

Beetham, David, ed. 1994. Defining and Measuring Democracy. London: Sage.

Berg-Schlosser, Dirk. 2004a. “Indicators of Democracy and Good Governance as Measures of the Quality of Democracy in Africa: A Critical Appraisal.” Acta Politica 39(3): 248–278.

Berg-Schlosser, Dirk. 2004b. “The Quality of Democracies in Europe as Measured by Current Indicators of Democratization and Good Governance.” Journal of Communist Studies and Transition Politics 20(1): 28–55.

Boix, Carles; Michael K. Miller, and Sebastian Rosato. 2013. “A Complete Dataset of Political Regimes, 1800-2007.” Comparative Political Studies.

Bollen, Kenneth A. 1993. “Liberal Democracy: Validity and Method Factors in Cross-National Measures.” American Journal of Political Science 37(4): 1207–1230.

Bollen, Kenneth A., and Pamela Paxton. 2000. “Subjective Measures of Liberal Democracy.” Comparative Political Studies 33(1): 58–86.

Bowman, Kirk, Fabrice Lehoucq, and James Mahoney. 2005. “Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America.” Comparative Political Studies 38(8): 939–970.

Brady, Henry E.; David Collier (eds). 2010. Rethinking Social Inquiry: Diverse Tools, Shared Standards, 2d ed. Lanham: Rowman & Littlefield.

Brown, Michael E.; Sean M. Lynn-Jones; Steven E. Miller (eds). 1996. Debating the Democratic Peace. Cambridge: MIT Press.

Brownlee, Jason. 2007. Authoritarianism in an Age of Democratization. Cambridge: Cambridge University Press.

Cheibub, Jose Antonio; Jennifer Gandhi; James Raymond Vreeland. 2010. “Democracy and Dictatorship Revisited.” Public Choice 143:1–2, 67–101.

Cingranelli, David L.; David L. Richards. 1999. “Measuring the Level, Pattern, and Sequence of Government Respect for Physical Integrity Rights.” International Studies Quarterly 43:2 (June) 407-17.

Collier, David: Jody LaPorte; Jason Seawright. 2012. “Putting Typologies to Work: Levels of Measurement, Concept Formation, and Analytic Rigor.” Political Research Quarterly 65:2, 217-32.

Collier, David; Fernando Daniel Hidalgo; Andra Olivia Maciuceanu. 2006. “Essentially Contested Concepts: Debates and Applications.” Journal of Political Ideologies 11:3 (October) 211-46.

23

Collier, David; John Gerring (eds). 2009. Concepts and Method in Social Science: The Tradition of Giovanni Sartori. London: Routledge.

Coppedge, Michael J. 1999. “Thickening Thin Concepts and Theories: Combining Large N and Small in Comparative Politics.” Comparative Politics 31:4 (July) 465-76.

Coppedge, Michael, Angel Alvarez, and Claudia Maldonado. 2008. “Two Persistent Dimensions of Democracy: Contestation and Inclusiveness.” Journal of Politics 70(3): 335–350.

Coppedge, Michael; John Gerring; Staffan I. Lindberg; Jan Teorell. 2013. “Global Standards, Local Knowledge: The Varieties of Democracy.” Unpublished manuscript.

Coppedge, Michael; Wolfgang H. Reinicke. 1990. “Measuring Polyarchy.” Studies in Comparative International Development 25, 51-72.

Correlates of War. 2011. State System Membership (v2011), http://www.correlatesofwar.org/COW2%20Data/SystemMembership/2011/System2011.html

Dahl, Robert A. 1956. A Preface to Democratic Theory. Chicago: University of Chicago Press. Dahl, Robert A. 1971. Polyarchy: Participation and Opposition. New Haven: Yale University Press. Daniels, Norman. 1996. Justice and Justification: Reflective Equilibrium in Theory and Practice. Cambridge:

Cambridge University Press. Davenport, Christian; David Armstrong. 2004. “Democracy and the Violation of Human Rights: A

Statistical Analysis from 1976-1996.” American Journal of Political Science 48:3, 538-54. DeVellis, Robert F. 2011. Scale Development: Theory and Applications. Thousand Oaks, CA: Sage. Elkins, Zachary. 2000. “Gradations of Democracy? Empirical Tests of Alternative

Conceptualizations.” American Journal of Political Science 44:2 (April) 287-94. Elman, Colin. 2005. “Explanatory Typologies in Qualitative Studies of International Politics.”

International Organization 59:2 (April) 293-326.

Foweraker, Joe, and Roman Krznaric. 2000. “Measuring Liberal Democratic Performance: An Empirical a Conceptual Critique.” Political Studies 48(4):759–787.

Freedom House. 2007. “Methodology,” Freedom in the World 2007. New York. (http://www.freedomhouse. org/template.cfm?page=351&ana_page=333&year=2007), accessed September 5, 2007.

Gerring, John. 2012. Social Science Methodology: A Unified Framework, 2d ed. Cambridge: Cambridge University Press.

Gerring, John; Jan Teorell; Dominic Zarecki. 2013. “Demography and Democracy: A Global, District-level Analysis of Electoral Contestation.” Unpublished manuscript, Department of Political Science, Boston University.

Gerring, John; Svend-Erik Skaaning. 2013. “Democracy and Growth: A Lexical Approach.” Unpublished manuscript, Department of Political Science, Boston University.

Gleditsch, Kristian S., and Michael D. Ward. 1997. “Double Take: A Re-examination of Democracy and Autocracy in Modern Polities.” Journal of Conflict Resolution 41: 361–83.

Gleditsch, Kristian. 2013. List of Independent States, http://privatewww.essex.ac.uk/~ksg/statelist.html

Goertz, Gary. 2006. Social Science Concepts: A User's Guide. Princeton: Princeton University Press. Goertz, Gary; James Mahoney. 2012. A Tale of Two Cultures: Qualitative and Quantitative Research in the

Social Sciences. Princeton: Princeton University Press. Guttman, Louis. 1950. “The Basis for Scalogram Analysis.” In Samuel A. Stouffer et al. (eds),

Measurement and Prediction: Studies in Social Psychology in World War II, vol. 4 (Princeton, NJ: Princeton University Press) 60–90.

Hadenius, Axel, and Jan Teorell. 2005a. “Assessing Alternative Indices of Democracy.” Committee on Concepts and Methods Working Paper Series, August.

24

Hadenius, Axel, and Jan Teorell. 2007. “Pathways from Authoritarianism.” Journal of Democracy 18.1, 143-156.

Haig, Brian D.; Denny Borsboom. 2008. “On the Conceptual Foundations of Psychological Measurement.” Measurement 6, 1-6.

Held, David. 2006. Models of Democracy, 3d ed. Stanford: Stanford University Press. Howard, Marc Morjé; Philip G. Roessler. 2006. “Liberalizing Electoral Outcomes in Competitive

Authoritarian Regimes.” American Journal of Political Science 50.2, 362-78. Huntington, Samuel P. 1991. The Third Wave: Democratization in the Late Twentieth Century. Norman,

OK: University of Oklahoma Press. Kaufmann, Daniel; Aart Kraay; Massimo Mastruzzi. 2007. “Governance Matters IV: Governance

Indicators for 1996-2006.” Washington: World Bank. Kurtz, Marcus. 2000. “Understanding Peasant Revolution: From Concept to Theory and Case.”

Theory and Society 29, 93-124. Lazarsfeld, Paul F. 1937. “Some Remarks on the Typological Procedures in Social

Research.” Zeitschrift für Sozialforschung 6, 119-39. Lazarsfeld, Paul F.; Allen H. Barton. 1951. “Qualitative Measurement in the Social Sciences:

Classification, Typologies, and Indices.” In Daniel Lerner and Harold D. Lasswell (eds), The Policy Sciences (Stanford: Stanford University Press) 155-92.

Levitsky, Steven; Lucan Way. 2002. “The Rise of Competitive Authoritarianism.” Journal of Democracy 13.2, 51-55.

Lively, Jack. 1975. Democracy. New York: St. Martin’s. Marshall, Monty G. and Keith Jaggers. 2007. “Polity IV Project: Political Regime Characteristics and

Transitions, 1800–2006.” (http://www.systemicpeace.org/ inscr/p4manualv2006.pdf ), accessed February 1, 2011.

McHenry, Dean E., Jr. 2000. “Quantitative Measures of Democracy in Africa: An Assessment.” Democratization 7(2): 168–185.

Mokken, R. J. 1971. A Theory and Procedure of Scale Analysis. With Applications in Political Research. Berlin: De Gruyter.

Moldau, Juan Hersztajn. 1992. “On the Lexical Ordering of Social States According To Rawls' Principles of Justice.” Economics and Philosophy 8, 141-48.

Møller, Jørgen; Svend-Erik Skaaning. 2012. “Systematizing Thin and Thick Conceptions of the Rule of Law.” Justice Systems Journal 33(2): 136-153

Møller, Jørgen; Svend-Erik Skaaning. 2013. “Disaggregating the Third Wave.” Journal of Democracy, 24.

Munck, Gerardo L. 2009. Measuring Democracy: A Bridge between Scholarship and Politics. Baltimore: John Hopkins University Press.

Munck, Gerardo L. 2009. Measuring Democracy: A Bridge between Scholarship and Politics. Baltimore: John Hopkins University Press.

Munck, Gerardo L., and Jay Verkuilen. 2002. “Conceptualizing and Measuring Democracy: Alternative Indices.” Comparative Political Studies 35(1): 5–34.

Munck, Gerardo L.; Jay Verkuilen. 2002. “Conceptualizing and Measuring Democracy: Alternative Indices.” Comparative Political Studies 35(1): 5–34.

Naess, Arne. 1956. Democracy, Ideology, and Objectivity. Blackwell. Nino, Carlos Santiago. 1998. The Constitution of Deliberative Democracy. New Haven: Yale University

Press. Papineau, David. 1976. “Ideal Types and Empirical Theories.” British Journal for the Philosophy of Science

27:2 (June) 137-46.

25

Pemstein, Daniel; Stephen Meserve; James Melton. 2010. “Democratic Compromise: A Latent Variable Analysis of Ten Measures of Regime Type.” Political Analysis 18:4, 426–49.

Przeworski, Adam et al. 2013. “Political Institutions and Political Events (PIPE) Data Set.” Department of Politics, New York University.

Przeworski, Adam, Michael Alvarez, Jose Antonio Cheibub, Fernando Limongi. 2000. Democracy and Development: Political Institutions and Material Well-Being in the World, 1950-1990. Cambridge: Cambridge University Press.

Ranney, Austin. 1962. The Doctrine of Responsible Party Government: Its Origins and Present State. Urbana: University of Illinois Press.

Rawls, John. 1971. A Theory of Justice. Cambridge: Harvard University Press. Sartori, Giovanni. 1970. “Concept Misformation in Comparative Politics.” American Political Science

Review 64:4 (December) 1033-46. Sartori, Giovanni. 1984. “Guidelines for Concept Analysis.” In Giovanni Sartori (ed), Social Science

Concepts: A Systematic Analysis (Beverly Hills: Sage) 15-48. Schattschneider, E. E. 1942. Party Government. New York: Rinehart. Schedler, Andreas. 2002. “The Menu of Manipulation.” Journal of Democracy 13:2 (April) 36-50. Schneider, Carsten Q.; Claudius Wagemann. 2012. Set-Theoretic Methods for the Social Sciences: A Guide to

Qualitative Comparative Analysis. Cambridge: Cambridge University Press. Schumpeter, Joseph A. 1950. Capitalism, Socialism and Democracy. New York: Harper & Bros. Shadish, William R.; Thomas D. Cook; Donald T. Campbell. 2002. Experimental and Quasi-experimental

Designs for Generalized Causal Inference. Boston: Houghton Mifflin. Sitjtsma, Klaas; Ivo W. Molenaar. 2002. Introduction to Nonparametric Item Response Theory. Thousand

Oaks, CA: Sage. Sitjtsma, Klaas; Pierre Debets; Ivo W. Molenaar. 1990. “Mokken Scale Analysis for Polychotomous

Items: Theory, A Computer Program and an Empirical Application.” Quality & Quantity 24, 173-88.

Skaaning, Svend-Erik. 2010. “Measuring the Rule of Law.” Political Research Quarterly 63(2): 449-460. Tadjbakhsh, Sharbanou; Anuradha Chenoy. 2007. Human Security: Concepts and Implications. London:

Routledge. Tamanaha, Brian Z. 2004. On the Rule of Law: History, Politics, Theory. Cambridge: Cambridge

University Press. Trebilcock, Michael; Ronald Daniels. 2008. Rule of Law Reform and Development. Cheltenham: Edward

Elgar. Treier, Shawn; Simon Jackman. 2008. “Democracy as a Latent Variable.” American Journal of Political

Science 52:1, 201–17. van Schuur, Wijbrandt H. 2003. “Mokken Scale Analysis: Between the Guttman Scale and

Parametric Item Response Theory.” Political Analysis 11, 139-63. van Schuur, Wijbrandt H. 2011. Ordinal Item Response Theory: Mokken Scale Analysis. Beverly Hills:

Sage. Vanhanen, Tatu. 2000. “A New Dataset for Measuring Democracy, 1810-1998.” Journal of Peace

Research 37, 251-65.

Vermillion, Jim. 2006. “Problems in the Measurement of Democracy.” Democracy at Large 3(1): 26–30.

Waldron, Jeremy. 2002. “Is the Rule of Law an Essentially Contested Concept (in Florida).” Law & Philosophy 21:137-64.

26

Table 1:

Generic Lexical scale

0. ~A

1. A ~B

2. A B ~C

3. A B C ~D

4. A B C D ~E

5. A B C D E

0-5 = ordinal scale. A-E = conditions that are satisfied. ~A-E = conditions that are not satisfied. Empty cells = undefined. Relationships are deterministic.

27

Table 2: Lexical Index of Democracy: The World in 1904

0 1 2 3 4 5 6

- National elections

+ National elections

+ Multi-party

elections

+ Executive

accountability

+ Competitive

elections

+ Male or

female suffrage

+ Universal suffrage

Afghanistan China Ethiopia Iran Korea Montenegro Nepal Oman Ottoman Emp Russia Thailand

Colombia Cuba Haiti Honduras Liberia Mexico Nicaragua Paraguay Venezuela

Austria-Hungary Bulgaria Germany Japan Portugal Romania Spain Sweden

Argentina Bolivia Brazil Dom. Rep. Ecuador El Salvador Guatemala Italy Peru Serbia

Chile Costa Rica Denmark Luxembourg Netherlands Panama United Kingdom Uruguay

Belgium Canada France Greece Switzerland United States

Australia

N=53

28

Table 3: Frequency Distribution of the Lexical Index of Electoral Democracy, 1800-2008

N % 0 4,501 26.20 1 2,920 17.00 2 1,507 8.77 3 2,594 15.10 4 863 5.02 5 503 2.93 6 4,291 24.98

29

Table 4: Electoral Democracy as a Predictor of State Repression: A Comparison of Indices

Linear Curvilinear Disaggregated

1 2 3 4 5 6

Polity -.027*** (.004)

.038** (.014)

Polity2 -.007*** (.001)

Polity, L1 .115* (.057)

Polity, L2 -.086 (.053)

Polity, L3 -.046 (.081)

Polity, L4 .028 (.074)

Polity, L5 -.053 (.067)

Polity, L6 -.010 (.040)

Polity, L7 -.093* (.047)

Polity, L8 -.148*** (.041)

Polity, L9 -.182*** (.050)

Polity, L10 -.394*** (.045)

Lexical -.041*** (.009)

.008 (.032)

Lexical2 -.011 (.007)

Lexical, L1 -.107** (.037)

Lexical, L2 -.167** (.061)

Lexical, L3 -.038 (.039)

Lexical, L4 -.226*** (.040)

R2 .752 .749 .755 .749 .756 .751

Sample period: 1976-2008. Countries: 158. Observations: 3582. Estimator: OLS (TSCS). Standard errors in parentheses. *<.1, **<.01, ***<.001 (two-tailed test). L=levels on an ordinal scale (not lags). Additional covariates (presented in the text) are not shown.

30

Table 5: Generic Mokken Scale

0. ~A ~B ~C ~D ~E

1. A ~B ~C ~D ~E

2. A B ~C ~D ~E

3. A B C ~D ~E

4. A B C D ~E

5. A B C D E

0-5 = ordinal scale. A-E = conditions that are satisfied. ~A-E = conditions that are not satisfied. Relationships are probabilistic.

31

Table 6: Lexical and Mokken Democracy Indices Compared

Indices Ordering of attributes

Lexical 0,1,2,3,4,5,6

Mokken (1800-2008) 0,1,3,5,2,6,4

Mokken (1800-1945) 0,1,2,3,5,4,6

Mokken (1946-2008) 0,5,6,1,3,2,4

0=No elections 1=National elections 2=Multi-party elections 3=Executive elections 4=Minimally competitive elections 5=Male or female adult suffrage 6=Universal adult suffrage. Full definitions are contained in the text. Mokken analyses reveal Loevinger’s H coefficients of 0.66-0.77 for the scales and 0.56-0.96 for the individual items, indicating rather strong scalability. (Note: Mokken coding for “0” presumes no national elections and no universal suffrage for males or females.)

32

Figure 1: Distribution of Countries

Across the Lexical index of Democracy, 1800-2008

0

20

40

60

80

100

120

140

160

180

200

0

1

2

3

4

5

6

33

Appendix A: The Lexical Index of Electoral Democracy

When introducing a new index of democracy to the world it is traditional to begin by critiquing extant indices. Because this critique has been issued so many times in recent years it seems gratuitous to revisit these issues here. Instead, we invite readers to peruse the voluminous literature on the subject (e.g., Beetham 1994; Berg-Schlosser 2004a, 2004b; Bollen 1993; Bollen and Paxton 2000; Bowman, Lehoucq, and Mahoney 2005; Coppedge et al. 2013; Coppedge, Alvarez, Maldonado 2008; Foweraker and Krznaric 2000; Gleditsch and Ward 1997; Hadenius and Teorell 2005; McHenry 2000; Munck 2009; Munck and Verkuilen 2002; Treier and Jackman 2008; Vermillion 2006).

Accordingly, we focus our discussion – in the text and in this appendix – on that which distinguishes the Lexical index from other indices. Occasionally, these comparisons will verge on critique. But the general point should be kept in mind: most indices of a concept (e.g., democracy) are useful for some purposes. It is exceedingly rare to discover an indicator that is inferior to another on all accounts. Thus, the principal goal in launching a new index should be on making its distinctive qualities clear, without derogating the alternatives.

We begin by sketching out in some detail the coding process entailed in the construction of the Lexical index, which encompasses all independent countries from 1800 to 2008. Next, we compare the Lexical index to other prominent democracy indices in a series of tables (some of which have already been referenced in the text). Coding

To code the Lexical index of electoral democracy we began by identifying independent countries that would serve as the units of analysis. Here, we rely on Gleditsch (2013) and Correlates of War (2011), supplemented for the years 1800-1815 by various country-specific sources. To operationalize the different levels of the Lexical index we make use of four indicators from the PIPE dataset (Przeworski et al. 2013): LEGSELEC, EXSELEC, OPPOSITION, and FRANCHISE, as described below. We also construct a new indicator, COMPETITION, also described below. Note that each of these indicators refer to the status of a country on the last day of the calendar year (31 December), and are not intended to reflect the mean value of an indicator across the previous 365 days. These five indicators are employed, as follows, in order to construct the Lexical index:

0. No elections. Elections are not held for any national-level policymaking offices. This includes situations in which elections are postponed indefinitely or the constitutional timing of elections is violated in a more than marginal fashion. LEGSELEC = 0 (the lower house of the legislature is not elected) & EXSELEC = 0 (the chief executive is not elected – whether directly or indirectly, i.e., by people who have been elected). Sources: PIPE, country-specific sources.

1. National elections. There are regular national elections. LEGSELEC =1 (the lower house of the legislature is at least partly elected) or EXSELEC =1 (the chief executive is either directly or indirectly elected, i.e., by people who have been elected). Sources: PIPE, country-specific sources.

34

2. Multi-party elections. Opposition parties are allowed to participate in legislative elections and to take office. OPPOSITION =1 (there is a legislature that is at least in part elected by voters facing more than one choice). Sources: PIPE, country-specific sources.

3. Executive accountability. The chief executive is accountable either directly to the electorate or indirectly to an elected parliament. EXSELEC = 1 (the chief executive is either directly or indirectly elected, i.e., by people who have been elected). Sources: PIPE, country-specific sources.

4. Competitive elections. The chief executive offices and the seats in the effective legislative body are – directly or indirectly – filled by elections characterized by uncertainty, meaning that the elections are, in principle, sufficiently free to enable the opposition to gain power. COMPETITION =1 (there is a positive probability that the opposition can win government power). Sources: Cheibub et al. (2010), Boix et al. (2013), country-specific sources.

5. Male or female adult suffrage. Virtually all adult male or female citizens are allowed to vote in national elections. (In no extant cases was universal female suffrage introduced before universal male suffrage, so in practice this level is reserved for countries with male (only) suffrage.) MALE SUFFRAGE = 1 (virtually universal male suffrage) or FEMALE SUFFRAGE = 1 (virtually universal female suffrage). Sources: PIPE (FRANCHISE), country-specific sources.

6. Universal suffrage. Virtually all adult citizens are allowed to vote in national elections. MALE SUFFRAGE = 1 (virtually universal male suffrage) & FEMALE SUFFRAGE = 1 (virtually universal female suffrage). Sources: PIPE (FRANCHISE), country-specific sources.

It is important to emphasize that although we employ PIPE as an initial source for coding

L0-3 and L5-6, we deviate from PIPE codings – based on our reading of country-specific sources – in several ways. First, with respect to executive elections, in the PIPE dataset “Prime ministers are always coded as elected if the legislature is open.” However, for our purposes we need an indicator that also takes into account whether the government is responsible to an elected parliament if the executive is not directly elected, which, for example, has not been the case in a number of monarchies in Europe before World War I and today in the Middle East. To illustrate, PIPE codes Denmark as having executive elections from 1849 to 1900 although the parliamentary principle was not established until 1901. Before then, the government was accountable to the king. Among the current cases with elected multi-party legislatures not fulfilling this condition, we find Jordan and Morocco. In order to achieve a higher level of concept-measure consistency, we have thus recoded all country-years (based on country-specific accounts) for this variable where our sources suggested doing so. Second, we have filled out all the missing values in the PIPE dataset, meaning that we have a complete dataset for all conditions for all independent countries of the world in the period 1800-2008. In general, except for the minor adjustment regarding executive elections mentioned above, we have followed the more specific coding rules laid out in the PIPE codebook. This work did not only imply that we used country-specific sources to fill the gaps but also that many years and a number of countries (such as the German principalities of the 19th century) were added. If we, in our coding of the missing values, came across information which suggested a recoding of any PIPE

35

scores, we did so. All values refer to the status of countries as of the end of a particular year. Whereas the numbers of observations for the employed PIPE indicators range between 14,465 and 15,302, the additional codings mean that our dataset has 17,179 observations for all indicators.

Third, in order to measure if elections are competitive, we have constructed a new variable to capture if there is a positive probability (see Przeworski et al. 2000: 16-17) that the opposition can win government power. Like Cheibub et al. (2010; see also Przeworski et al. 2000), we have considered instances of electoral incumbent turnover as a rather robust indicator of contested elections. However, like Boix et al. (2013), we have not considered electoral executive turnover to be either a necessary or a sufficient criterion for genuinely contested elections. Instead, we have also taken into account more general impressions of how free elections were according to country-specific sources. Doing so, we have followed Schumpeter (1950; see also Przeworki et al. 2000; Boix et al. 2013) by establishing a modest threshold, e.g., not insisting on an entirely level playing field or a high level of respect for civil liberties. Thus, elections are generally considered competitive if voters experience little or no systematic coercion in exercising their electoral choice, and electoral fraud is not determining who wins the elections. By and large, the coding decisions required for the Lexical index of democracy are factual in nature, resting on institutional features that require historical knowledge but not subjective judgments on the part of the coder. Uncertainties are introduced when source material for a country or era is weak. But we can probably assume that this sort of bias is random rather than systematic (as it might be if coder judgments involved questions of meaning and interpretation). In this respect, the Lexical index echoes a feature of the DD and BMR indices. Indeed, it is quite similar to these indices insofar as it relies on binary codings, which are combined to form the lexical scale. Comparisons

36

Table A1: Democracy Indices Compared

SCALE COVERAGE CORRELATION

Type Range Countries Years Obs

Full sample

Restricted sample

Lexical (authors) Lexical 0-6 228 1800-2008 $

DD (Cheibub et al) Binary 0-1 200 1946-2008 $ .84 (S) .36 (S)

BMR (Boix et al.) Binary 0-1 213 1800-2007 15,972 $ (S) $ (S)

Polity2 (Marshall, Jaggers) Ordinal -10-10 189 1800-2012 $ .79 (S) .59 (S)

PR (Freedom House) Ordinal 1-7 200 1972-2012 $ .85 (S) .43 (S)

CL (Freedom House) Ordinal 1-7 200 1972-2012 $ .79 (S) .31 (S)

Democracy Index (Vanhanen) Interval 0-100 187 1810-2000 $ $ (P) $ (P)

Contestation (Coppedge et al.) Interval -1.84-1.96 200 1950-2000 $ .91 (P) $ (P)

Inclusiveness (Coppedge et al.) Interval -3.04-1.91 200 1950-2000 $ .59 (P) .54 (P)

UDS(Pemstein et al.) Interval -2.10-2.12 200 1946-2008 $ .88 (P) .63 (P)

The final columns show the Spearman’s (S) or Pearson’s (P) correlation coefficient between the Lexical index and other indices of democracy in a full sample (all available country-years) and a restricted sample (Lexical<6).

Table A2: Crosstabulations

DD (Cheibub et al) | Lexical

| 0 1 2 3 4 5 6 | Total

-----------+-----------------------------------------------------------------------------+----------

0 | 3,162 2,548 1,119 1,882 418 62 130 | 9,321

1 | 11 13 11 80 351 432 2,387 | 3,285

-----------+-----------------------------------------------------------------------------+----------

Total | 3,173 2,561 1,130 1,962 769 494 2,517 | 12,606

PR (FH)

| Lexical

| 0 1 2 3 4 6 | Total

-----------+------------------------------------------------------------------+----------

1 | 0 0 14 1 0 1,307 | 1,322

2 | 3 0 13 27 0 805 | 848

3 | 5 15 13 73 0 318 | 424

4 | 24 55 45 211 4 142 | 481

5 | 150 214 69 192 16 24 | 665

6 | 330 505 35 228 1 1 | 1,100

7 | 385 478 21 61 0 0 | 945

-----------+------------------------------------------------------------------+----------

Total | 897 1,267 210 793 21 2,597 | 5,785

Polity2 (Polity IV)

| Lexical

| 0 1 2 3 4 5 6 | Total

-----------+-----------------------------------------------------------------------------+----------

-10 | 1,067 52 39 0 0 0 0 | 1,158

-9 | 256 618 114 128 0 0 4 | 1,120

-8 | 122 265 76 76 0 1 0 | 540

-7 | 629 798 224 82 0 0 0 | 1,733

-6 | 657 197 154 191 0 0 0 | 1,199

-5 | 140 158 39 191 20 0 10 | 558

38

-4 | 42 67 210 169 24 2 1 | 515

-3 | 241 265 83 410 45 42 14 | 1,100

-2 | 16 22 34 139 39 1 3 | 254

-1 | 49 132 51 225 6 4 21 | 488

0 | 107 51 10 126 8 9 19 | 330

1 | 37 31 76 101 8 7 3 | 263

2 | 17 35 43 177 66 21 20 | 379

3 | 4 26 10 77 76 14 30 | 237

4 | 87 10 24 119 113 8 51 | 412

5 | 2 26 11 84 50 19 119 | 311

6 | 5 3 25 15 53 22 264 | 387

7 | 5 13 0 36 66 36 281 | 437

8 | 6 12 3 20 50 32 457 | 580

9 | 0 1 1 4 50 56 363 | 475

10 | 4 2 0 10 93 225 1,801 | 2,135

-----------+-----------------------------------------------------------------------------+----------

Total | 3,493 2,784 1,227 2,380 767 499 3,461 | 14,611