the value of optimising quantitative instrument development via ... · and challenges of the...
TRANSCRIPT
The Value of Optimising Quantitative Instrument
Development via Qualitative Techniques in
Political Science Research
Stefania Kalogeraki1
Paper for the 6th ECPR General Conference
Reykjavík, Iceland, 25 - 27 August 2011
This paper is work in progress,
please do not cite it. Comments are more than welcome.
1 Stefania Kalogeraki is an elected lecturer in Quantitative Methods in Sociology in the Department ofSociology of the University of Crete (Greece). She has BSc in Statistics (Athens University inEconomic and Business, Greece), MA and PhD in Sociology (Reading University, UK). She hasparticipated in European and Greek research projects and published articles in international and Greekjournals. Her main research interests include quantitative methods in social research with a specialfocus on measurement issues and social demography.E-mail: [email protected]
2
AbstractThe optimization of the development of quantitative instruments is a key concern inany social science endeavour. Whilst instrument development has been for manyyears restricted to mono-method approaches, a new era has come that combinesqualitative techniques to enhance the development of quantitative instruments. Thisapproach constitutes one of the main rationales of conducting mixed method studieswhich in political science research, in contrast to the adjacent social sciencedisciplines, has been scarcely applied. The paper’s rationale is two-fold; a) to reviewthe three-phase exploratory design for the optimization of the development ofquantitative instruments via qualitative techniques and b) to present the main benefitsand challenges of the approach for political science research. Instrument developmentvia qualitative techniques applies an inductive-deductive and an emic-etic perspectivethat increases construct-related validity in cross-sectional political science studies andeliminates construct bias in cross-national and cross-cultural ones. The paper’s mainthesis is that the mixed method approach is not the panacea of all research inquiries.However, the optimization of quantitative instrument development via qualitativetechniques may adequately serve mixed method’s fundamental principal, i.e.,maximizing the potency and minimizing the weaknesses derived from theamalgamation of the two methods to enhance the validity of political studies’conclusions.
Introduction
Mixed method studies have become increasingly popular in social sciences providing
evidence that scholars utilize multiple perspectives in order to examine phenomena in
a more eclectic manner (Tashakkori & Creswell, 2008; Tashakkori & Teddlie, 2010).
The mixed method approach bridges across the schism between the qualitative and
quantitative research traditions (Teddlie & Tashakkori, 2003) by offering a variety of
choices, options and approaches (Johnson & Onwuegbuzie, 2004). Johnson,
Onwuegbuzie and Turner (2007) based on the analysis of 19 mixed methods’
definitions given from academic leaders provide the following definition:
Mixed methods research is the type of research in which a researcher or teamof researchers combines elements of qualitative and quantitative researchapproaches (e.g., use of qualitative and quantitative viewpoints, datacollection, analysis, inference techniques) for the broad purposes of breadthand depth of understanding and corroboration. (Johnson, Onwuegbuzie &Turner, 2007, p. 123)
The fundamental principle of mixed method research is based on the collection of
multiple data using different approaches and methods in such a way that the resulting
amalgamation is likely to yield a research outcome of complementary strengths and
3
non-overlapping weaknesses (Johnson & Turner, 2003). Under this framework, mixed
method research is likely to produce a research outcome that is superior to mono-
method studies (Johnson & Onwuegbuzie, 2004). Philosophically, it is the “third
methodological movement” (Tashakkori, & Teddlie, 2003, p. x) that involves the
pragmatic method and system of philosophy. It combines three logics of inquiry by
linking the deductive logic of traditional quantitative research (confirmation,
theory/hypothesis testing), the inductive logic of qualitative research (discovery,
exploration, theory/hypothesis generation) and the abductive one (generative, creative
problem solving) (Johnson & Onwuegbuzie, 2004). Pragmatism has emerged as a
common alternative to the either/or choice of positivism and constructivism of the
quantitative and qualitative approach, respectively offering an epistemological
approach that justifies the integration of methods in order for researchers to best
frame, address and provide logical and practical answers to their research questions.
Mixed method research is a relatively new approach, compared to the qualitative and
quantitative traditions, with a short history of approximately 20 years (Greene, 2008).
Its history is even shorter in the field of disciplines such as political science and
comparative politics that so far have not participated much in mixed method research
discussions (Harrits, 2011).
Tarrow’s (1995) work constitutes one of the first steps of bridging quantitative and
qualitative methods in the field of political science by providing some guidance on
how to blend the two traditions. For instance, he highlights the importance of
qualitative approach for process tracing1, for exploring the ‘tipping points’ that play a
critical role in shaping long-term processes of change and for providing deeper
insights into findings derived from quantitative studies. In turn, he emphasizes the
importance of quantitative methods to best frame and to generalize qualitative studies’
findings. In Tarrow’s (1995) view the most valuable interaction between qualitative
and quantitative methods takes place when scholars triangulate among alternative
methods and data in order to answer the same research question. Whilst Tarrow
(1995) provides a general framework of conducting mixed method studies, the
author’s overview is not exhaustive excluding among others the crucial role mixed
methods play in instrument development processes.
1 Process tracing refers to the application of qualitative techniques to investigate procedures of changewithin cases providing detailed evidence on causal mechanisms and causal processes that underliequantitative findings.
4
A few years later, Bennett and Braumoeller (2002) emphasized the role of combining
quantitative and qualitative techniques in political science research for theory testing
and theory development. In the former, the mixed method approach is applied to
determine whether the qualitative findings can be generalized to the wider population
or whether they can be used to uncover the causal mechanisms behind correlations. In
theory generating mixed method research, the outliers derived from quantitative
studies can be qualitatively analyzed to provide information for identifying omitted
variables. Bennett and Braumoeller’s (2002) arguments in theory generation mixed
method studies, imply that the specific approach may reveal potential measurement-
related problems in political science research by uncovering left-out variables through
qualitative analysis of outlier quantitative data.
The significance of the mixed method approach in enhancing the quality of
measurements in comparative political science research has been outlined in
Lieberman’s (2005) nested design which involves the specification of different paths
in the combination of statistical (large N analysis-LNA) and case study (small N
analysis-SNA) techniques. Lieberman (2005, p.436) argues that the main scope of the
specific design is to “improve the quality of conceptualization and measurement,
analysis of rival explanations, and overall confidence in the central findings of a
study”. The nested design allows the evaluation and/or development of a survey
instrument for the measurement of a large number of cases (LNA) through close
range analysis of one or a few cases (SNA) with the rationale of increasing the
explanatory power and the robustness of the former. Whilst Lieberman (2005)
advocates that the amalgamation of the advantages of SNA and LNA associated with
internal and external validity, respectively result in the enhancement of the quality of
the concepts and measurements, the nested design has been criticised by different
scholars.
For instance, Mastenbroek and Doorenspleet (2007) argue that Lieberman’s (2005)
nested strategy does not always result in the improvement of measurements unless it
is applied in a comparison of most similar systems, i.e., of cases with great similarities
(such as similar countries). In addition, Harrits (2010) develops some criticism against
the nested design by advocating that Lieberman’s (2005) thesis with respect to the
conceptualization of SNA, as a qualitative approach that focuses on the analysis of
one or a few intrinsically interesting entities, is inadequate for individual-level
5
behaviours. Hence, SNA of close range analysis is likely to be inappropriate for
evaluating and/or developing instruments measuring attitudes in large scale surveys.
Whilst optimizing instrument development and confronting measurement-related
issues via mixed method approaches has been an issue of concern in political science
research, the utilization of qualitative techniques to enhance the development of
quantitative instruments has not been fully investigated. However, the specific issue
has been adequately explored in the adjacent fields of social science research
providing typologies that underline that among the main rationales of conducting
mixed method studies is the optimization of instrument development. For instance,
Greene, Caracelli and Grahamet (1989) in their five-purpose typology include mixed
method studies with the rationale to use sequentially the results from one method to
help develop and inform the other. More specifically, they argue that “One method is
implemented first, and the results are used to help select the sample, develop the
instrument, or inform the analysis for the other method” (Greene, Caracelli &
Grahamet, 1989, p.267). Furthermore, Collins, Onwuegbuzie and Sutton (2006)
develop a four-fold typology of mixed method rationales involving among others
instrument fidelity. Instrument fidelity refers to the assessment of the appropriateness
and/or utility of existing instruments and the creation and improvement of new ones.
The development of quantitative instruments and their validation through multiple
research methods has a long history dated back to Campbell and Fiske’s (1959)
framework of multiple traits and multiple methods (MTMM). The approach involves
multiple operationalism by using at least three traits measured with three different
methods, leading to nine different observed variables for validation purposes. Whilst
some scholars advocate that Campbell and Fiske’s (1959) MTMM framework provide
the impetus for the concept of triangulation for validating purposes (Johnson,
Onwuegbuzie & Turner, 2007) others suggest that MTMM’s underling principal is
that quantitative methods are sufficient for the development of quantitative
instruments (Onwuegbuzie, Bustamante & Nelson, 2010).
Despite the debates on MTMM’s main rationale, the amalgamation of qualitative and
quantitative methods in instrument development is likely to produce a research
outcome that is superior to quantitative mono-method approaches (Onwuegbuzie,
Bustamante & Nelson, 2010), as the former optimize the conceptualization process
and enhance measurements’ internal validity. Such a mixed method approach
involves an inductive-deductive-abductive logic and an emic-etic perspective that
6
increases measurement-related validity in cross-sectional and cross-national/cross-
cultural research in social science in general and specifically in political science.
Under this framework, the main rationale of the paper is two-fold; to review one of
the mixed method designs applied to the optimization of the development of
quantitative instruments via qualitative techniques named sequential exploratory
three-phase design and to present its main benefits and challenges for political science
research.
Sequential exploratory three-phase design
Creswell (1999, p.460) highlights the fundamental role mixed methods play in
quantitative instrument development via qualitative techniques by advocating that
among the most crucial purposes of the mixed method approach is to develop
“Quantitative measures and instruments grounded in the views of subjects or
participants”. Such a mixed method approach is extremely valuable in cases where
there are no available measures, the existing ones are poorly conceptualized and when
the available measures do not represent the population being studied (Creswell, 1999;
Creswell, Fetters & Ivankova, 2004; Creswell & Plano Clark, 2011).
Creswell (1999) is among the first scholars to introduce a sequential exploratory
design with the rationale of instrument development. The design has been recently
enriched including three phases and a detailed framework of how to apply qualitative
techniques in order to construct rigorous quantitative instruments (Creswell & Plano
Clark, 2011). The initial phase of the design is a qualitative one as the most adequate
to explore a phenomenon (Creswell et al., 2003). The second phase represents the
point of interface in mixing, as mixed method researchers build on the result of the
qualitative phase to develop the quantitative instrument. The final phase includes the
testing and administration of the quantitative instrument (Creswell & Plano Clark,
2011).
Although the three-phase exploratory design seems quite straightforward it has its
own challenges. Researchers applying such a mixed method approach need to decide
the sampling procedures applied in each phase, the most appropriate data to use from
the qualitative phase to build the quantitative instrument and the processes of
designing an instrument with good psychometric properties. The sampling procedures
in the qualitative and quantitative approach of the sequential exploratory design are
different as the participants in the former are not the same as the ones in the
7
quantitative follow-up study which involves large sample sizes with the rationale of
inferences to the population studied (Creswell & Plano Clark, 2011).
With respect to the challenge of determining the qualitative data to be used for
instrument generation, mixed method scholars need to carefully identify and assess
useful quotes or sentences, coding segments of information and the grouping of codes
into broad themes (Morse & Niehaus, 2009). In this process, the broad themes derived
from the qualitative analysis can be applied for constructing the scales to be
measured; the individual codes within each theme can be used for the variables and
the individuals’ quotes for specific items or questions (Creswell & Plano Clark,
2011). Oppenheim (1992) suggests that at the initial stage of item development the
over inclusion of items derived from the qualitative analysis is essential to ensure that
all the aspects or dimensions of the topic are covered.
The most fundamental challenge in instrument building process is to generate an
instrument with psychometric stability and feasibility (Vogt, King & King, 2004).
Mixed method researchers need to employ rigorous scale development procedures
addressing validity and reliability considerations (DeVellis, 2003). The former
involve results that accurately reflect the concept being measured, whilst the latter
refer to the consistency of results derived from the same measure (Babbie, 2002).
In the process of scale development, the qualitative findings are used to generate an
item pool and the appropriate scales of measurement are determined which are further
reviewed by an expert panel (DeVellis, 2003). Expert consultation on the
specification of constructs and the review of scaling procedures is considered by
Messick (1995a) to be the sin qua non of measurements’ validity. At this stage, it
might be also useful to include validated items from other scales or instruments
(DeVellis, 2003). Following experts’ consultation, the instrument is administered to a
sample of individuals for further validation that allow the evaluation of the items with
quantitative techniques e.g., item-scale correlations, item variance, reliability (De
Vaus, 1990; Oppenheim, 1992; DeVellis, 2003). The results of the pre-test allow for
the optimization of the instrument and its scales’ length based on the item
performance and reliability checks (DeVellis, 2003). At the final stage of the
sequential exploratory design the instrument is administered to a large sample in order
for researchers to conduct statistical tests and potentially make claims about the
population in question.
8
Benefits of mixed method approach for instrument development
Munk and Verkuilen (2002, p.31) advocate that “the careful development of measures
constitutes the foundation for efforts at drawing causal inferences and is a critical task
in itself”. In the development of measures two processes are crucial; the
conceptualization and the operationalization. The former involves the specification of
observations and measurements that give concepts definite meaning for the rationale
of the study. In addition, conceptualization includes specifying concept’s indicators
and describing its dimensions. Operationalization is considered to be an extension of
conceptualization that denotes the exact procedures that will be used to measure the
variable’s attributes (Babbie, 2002). The translation of theoretical concepts into
related variables as part of the operationalization process is crucial as it bridges the
theoretical and conceptual level to the measurement level (Bergman, 2010). Any
mismatch between the conceptual and the operational definitions constitute a serious
threat to the validity of the measures and consequently to the study’s findings (Vogt,
King & King, 2004).
The assessment of measurement validity in an instrument development process
involves three types of validation; content validity, criterion-related validity
(including predictive and concurrent validity) and construct validity (Cronbach &
Meehl, 1955). Content validity assesses the degree to which the items are relevant to
and representative of the content being measured. According to Cronbach and Meehl
(1955), criterion-related validity assesses the extent to which scores produced by a
measure are correlated with scores of independent/external variables (called criterion
variables) which are considered to measure directly the phenomenon of concern
(concurrent validity) and are useful in predicting future scores representing
hypothetically related phenomena (predictive validity). Construct validity assesses the
extent to which an instrument adequately measures the constructs they are designed to
measure.
The triumvirate of content, criterion and construct validity does not represent three
distinct types of measurement validity but rather reflects a unitary concept that places
construct validity at the centre encompassing all types of measurement-related
validity evidence (Cronbach & Meehl, 1955; Messick, 1989a; 1989b; 1995b). Valid
measurements are derived when scores accurately reflect the intended concept; hence
a prerequisite of valid measurements is the refinement and clarification of the
concepts of concern (Adcock & Collier, 2001).
9
In social science research in general and specifically in political science investigators
are usually concerned with concepts that are more complex to handle. Northrop
(1947) distinguishes between two types of concepts; the concepts-by-intuition and the
concepts-by-postulation. The former involve simple concepts whose meaning is
immediately apparent and involve feelings, judgments, norms and behaviours which
are relatively clear what they mean. On the contrary, concepts-by-postulation or
constructs, as they are also called, involve less obvious concepts and hence require
explicit definitions to be properly understood (Saris & Gallhofer, 2007). Concepts-by-
postulation can be captured by using a set of items that represent simpler concepts,
i.e., concepts-by-intuition. Concepts-by-postulation have been also characterised as
thick concepts due to their multidimensional nature and their measurement through
multiple indicators. Thick concepts (such as “social trust”, “political trust”, “political
efficacy” etc) are rather prevalent in political science research (Coppedge, 1999).
Political democracy is a traditional example of a thick concept that has been
frequently applied in political science research. However, various scholars address the
challenges of political democracy’s measurement in terms of its conceptualization,
operationalization and aggregation specifically when the concept is applied across
diverse contexts (Bollen, 1980, 1990; Coppedge & Reinicke, 1990; Przeworski et al.,
2000; Munk & Verkuilen, 2002;Bowman, Lehoucq & Mahoney, 2005).
The qualitative techniques are more adequate than quantitative ones to measure thick
concepts as they allow for more careful conceptualization enhancing the internal
validity of measurements (Coppedge, 1999). Qualitative scholars are highly
concerned with conceptual validity hence their primary focus is on the development
of precise and clear conceptual definitions (Mahoney & Goertz, 2006; Bergman,
2010). On the contrary, scholars of the quantitative strand sacrifice definitional
complexity by applying thin but broad concepts, i.e., simplistic descriptions of cases
that allow testing generalizations in the larger population (Coppedge, 1999).
Furthermore, in the quantitative tradition the measurement detaches information from
its original ecological “real world” (Moghaddam, Walker, & Harre, 2003), a
phenomenon that has been characterised as decontextualization (Viruel-Fuentes,
2007). On the contrary, the qualitative approach investigates individuals holistically
within their natural environment allowing a fully contextualised approach of rich
accounts of human experiences examined within the original context in which
observations occur (Gelo et al., 2008).
10
The enhancement of quantitative measurements’ internal validity through the initial
qualitative phase of the sequential exploratory design is accomplished as the process
of defining the elements that are relevant to and representative of the construct under
study is grounded in real life situations and observations (Padgett, 1998; Rowan &
Wulff, 2007; Nassar-McMillan et al., 2010). The initial qualitative phase explores
respondents’ variations in the construction of concepts’ meaning informing the
processes of conceptualization and operationalization and yields items characterised
by high degrees of content validity (Haynes, Richard & Kubany, 1995; Vogt, King &
King, 2004; Bergman, 2010).
In addition, the combination of the qualitative and quantitative approaches in a
sequential exploratory design with the rationale of instrument development allows the
incorporation of insiders’ views (stemming from the participants involved in the
instrument development) and the outsiders’ views (stemming from extant theories,
researchers’ a priori assumptions and experts’ consultation) on the specification of the
construct. The insiders’ views involve the emic viewpoint of the group members
whilst the outsiders’ views capture the etic viewpoint (Pike, 1967; Currall & Towler,
2003). By mixing the two viewpoints with the rationale of instrument development an
inside-outside legitimation is accomplished enhancing measurement-related validity
(Onwuegbuzie & Johnson, 2006). Such a mixed method approach not only increases
the validation of the instrument but also reveals additional facets of the phenomenon
under study optimizing research outcomes. As Bergman (2010) argues the systematic
concept analyses through qualitative approaches:
…not only help in validating research instruments and scales, but may gofurther in that they could produce complementary subsets of results, whichwould enrich overall findings (Bergman, 2010, p.172)
Furthermore, instrument development via mixed method studies is particularly useful
when seeking to establish equivalence in the theoretical concepts and their operational
indicators as they are applied for comparisons across different cultural contexts
(Onwuegbuzie, Bustamante & Nelson, 2010) and over time. Equivalence implies that
concepts measure the same latent traits in a way that respondents’ scores on certain
items and scales can be compared in a direct and straightforward manner across
different contexts (Hui & Triandis, 1985; Dogan & Pelassy, 1990; Sartori 1994; Van
de Vijver & Leung , 1997; Van de Vijver, 1998).
11
The validation of an instrument in a specific context does not imply that the
theoretical concepts and their operational indicators can be applied in multiple
environments (MacIntyre, 1971; Messick, 1989b). Diverse understandings of a
concept are likely to result in different operationalizations (Adcock & Collier, 2001).
For instance, the concept of political participation is likely to mean very different
things across different contexts, such as voting in one country or mobilizing activists
against nuclear power in another. Moncagatta’s (2009) paradoxical findings with
respect to the high support of the democratic regime in Venezuela both by
respondents of great and no trust to president Chávez reflect in-equivalence problems
of the concept of “support to democracy” when applied in diverse cultural settings.
Scholars conducting comparisons across different contexts have to ensure that their
instrument measures the same underlying factors otherwise there is a serious threat of
construct in-equivalence and consequently of construct bias in the study’s findings
(Van de Vijver & Leung, 1997). Construct bias takes place when there is only partial
overlap in the definitions of the construct, there is incomplete coverage of the
construct’s facets and when constructs are associated with different behaviours or
characteristics across individuals in different nations, cultures and cultural groups
(Van de Vijver & Tanzer, 2004).
Scholars from comparative political research are highly concerned with the
application of constructs that are vulnerable to in-equivalence (Bunce, 1995; Munk &
Verkuilen, 2002). For instance, Bollen (1990) emphasises the need to optimize the
measurement of political democracy in comparative political research by further
clarifying its meaning and applying new measures. Similarly, Munk and Verkuilen
(2002) offer a comprehensive framework with the rationale of improving democracy’s
conceptualization and measurement in cross-national studies.
Instruments that are applied for comparisons over time are also susceptible to in-
equivalence, as the meaning of different constructs and the items used to measure
them might change as time passes. For instance, Baumgartner and Walker (1990)
emphasise the need of a new measurement of group membership in the United States
as the measures applied reflect conceptualizations that are appropriate for previous
generations. Furthermore, Elkins and Sides (2009) advocates that comparisons of
democracy over time potentially exhibit serious issues of non-equivalence.
Whilst there are statistical tools to diagnose in-equivalence problems and
consequently construct bias, these techniques are inadequate to trace and tackle the
12
sources of such bias (Elkins & Sides, 2009). Van de Vijver (2011) highlights the
pivotal role the mixed method approach plays on tracing and tackling the sources of
in-equivalence in specific items through the careful re-conceptualization and the
deeper knowledge of the context that a concept is investigated, i.e., processes that are
adopted in the initial qualitative phase of the sequential exploratory design. The same
author advocates that the qualitative approach in the development of an instrument
that is applied across different contexts helps researchers to evaluate the
appropriateness of the concept definition, identify missing concepts, employ
appropriate question wording and create suitable response formats (Van de Vijver,
2011).
Challenges of mixed method approach for instrument development
In the previous section it was advocated that instrument development via the
sequential exploratory design increases measurement’s internal validity as the initial
qualitative phase allows for more careful concept construction specifically of thick
multidimensional and complex concepts frequently applied in political science
research (Coppedge, 1999). However, such an increase in internal validity is done at
the cost of external validity of the measurement as the qualitative phase involves
results derived from a small number of cases that do not represent universally
applicable, comparable and generalizable concepts. Qualitative scholars in defense of
such criticism would advocate that their quantitative counterparts have to deal with
the opposite challenge. The increase of external validity through the application of
broad but thin concepts for testing generalizations in the larger population is done at
the cost of internal validity as the simplistic indicators applied in the quantitative
analysis do not capture all the relevant aspects of thick multidimensional concepts
(Coppedge, 1999; Munck & Verkuilen, 2002; Bowman, Lehoucq & Mahoney, 2005).
The diverse focus on internal and external validity of the qualitative and quantitative
strand respectively reflects the chasm that exists in the two traditions with respect to
measurement issues in social science research in general, and specifically in political
science (Mahoney & Goertz, 2006). On the one hand, qualitative scholars attempt to
minimize measurement error by concentrating on conceptual validity; on the other
hand quantitative scholars consider that measurement error occurs at the level of
indicators and not at the conceptual one. Hence, the latter focus on modeling
13
measurement errors and modifying indicators rather than developing processes
associated with concept revision.
The most crucial challenge in the application of qualitative techniques for optimizing
instrument development is to bridge the two traditions with the rationale of enhancing
simultaneously the internal and external validity of measurements. The three-phase
sequential exploratory design is likely to produce such an outcome as the initial
qualitative phase increases the internal validity of the measurement by refining the
concept of concern (Haynes, Richard & Kubany, 1995; Padgett, 1998; Vogt, King &
King, 2004; Rowan & Wulff, 2007; Bergman, 2010); whilst the second phase through
the expert panel consultation and the quantitative analysis of the pre-test results
optimizes the external validity of the measurement based on the item performance and
reliability checks (De Vaus, 1990; Oppenheim, 1992; Messick, 1995a; DeVellis,
2003).
The sequential exploratory design has been applied in numerous studies exploring
diverse social phenomena, verifying that the challenges of mixing the two traditions
with the rationale of instrument optimization can be adequately confronted in
empirical research (Meijer, Verloop & Beijaard, 2001; Mak & Marshall, 2004;
Durham, Tan & White, 2011). For instance, the specific design has been applied for
developing instruments that measure perspectives of academia held by school
psychology graduate students (Nagle et al., 2004), the nature of domestic violence
against women (Smith, Earp & DeVellis, 1995), the shopping motivations (Arnold &
Reynolds, 2003), the organizational assimilation in diverse industries (Myers &
Oetzel; 2003), the factors affecting changes in the size of graduate programs (Milton
et al., 2003), and the health promoting lifestyle behaviours among Japanese college
women (Tashiro, 2002).
Mixed method scholars in the process of instrument development for comparisons
across different contexts are likely to face additional challenges associated with the
conceptual stretching which arises when the concept of concern is applied to cases for
which, by relevant scholarly standards, it is inappropriate (Sartori, 1970; Collier &
Mahon, 1993). Sartori (1970) argues that concepts are formed at different levels of
abstraction (or generality) due to the trade off between concept’s “intension” (i.e., the
number of defining attributes) and “extension” (i.e., the number of cases it can be
applied). Qualitative scholars are more concerned with the more detailed contextual
analyses working with a high level of concept’s intension (i.e., incorporate relatively
14
complex conceptual definitions), at the cost though of its extension (i.e., involve a
limited number of cases). Meanwhile, quantitative scholars develop concepts of high
extension (i.e., involve relatively big sample sizes) applying comparable and
generalizable concepts; at the cost though of the concepts’ intension (i.e., apply
relatively simplistic indicators).
When concepts travel across different contexts, then scholars in order to avoid
conceptual stretching have to move up the ladder of abstraction (or generality), i.e., to
increase the extension of the concept by redefining it as something broader and more
applicable to different contexts (Sartori, 1970). Such a climbing, on the one hand,
ensures the concept’s comparability over time and space but on the other the concept
per se is likely to become very imprecise and abstract. In order to confront with
conceptual stretching, Sartori (1970) suggests that scholars may opt to make ‘bounded
generalizations’ for comparisons between relatively homogenous cases. Alternative
solutions to cope with conceptual stretching is to develop appropriate subtypes of the
concept of concern; for instance, researchers can develop appropriate subtypes of
democracy that simultaneously allow them to capture diverse forms of democracy and
avoid conceptual stretching (Collier & Levitsky, 1997).
An additional solution to conceptual stretching, although not so welcome by Sartori
and his followers (Lindberg, 2009), is to redefine the concept of interest by
eliminating some of its characteristics and adding new ones. Whilst it is feasible to
accomplish this alternative with the application of the sequential exploratory design,
recently more advanced mixed method designs have been developed for instruments
applied across diverse cultural settings. For instance, Onwuegbuzie, Bustamante and
Nelson (2010) present a ten-step mixed method process for instrument development
and construct validation through multiple data sources and crossover analyses for
cross-cultural comparisons.
Discussion
The optimization of instrument development is a key concern in any social science
endeavour. Quantitative tools that are valid in their indented purpose and measure the
concepts they are intended to measure possess the requisite psychometric properties to
provide valid conclusions and to lead to advances in theory and practice (Cronbach &
Meehl, 1955; Messick, 1989b; Messick, 1995a; DeVellis, 2003; Groves et al., 2009;
Bergman, 2010). Whilst there is a massive literature on quantitative measures,
15
indicators and scales, quantitative scholars insist on focusing whether the items
represent the construct of concern as measured by reliability and validity scores
disregarding the origins of questionnaire items (Rowan & Wulff, 2007; Nassar-
McMillan et al., 2010). The crucial process of item generation becomes the major
concern of qualitative scholars who focus on conceptual validity in contrast to their
quantitative counterparts who concentrate on measurement validity by modeling
measurement errors and modifying indicators (Mahoney & Goertz, 2006) as the
mathematical statistics applied are not designed to tackle measurement errors on the
conceptual level (Jacoby, 1991). However, the process of conceptualization is crucial
in instrument development, as Lazarsfeld and Barton (1965) advocated a few decades
ago
Before we can investigate the presence or absence of some attribute ... orbefore we can rank objects or measure them in terms of some variable, wemust form the concept of that variable. (Lazarsfeld & Barton, 1965, p.155)
Whilst the chasm between qualitative scholars’ concern for substantively valid
concepts and the quantitative scholars’ interest in good numerical measures seems
unbridgeable, the sequential three-phase exploratory design is likely to yield
instruments with high levels of internal and external validity. Such a design allows
mixed method scholars to shift from the constructivist principles of the initial
qualitative phase that involve multiple perspectives and deeper understanding of the
concept of concern to the positivist principles of quantitative analysis of measuring
variables and statistical trends (Creswell & Plano Clark, 2011). Hence, instead of
relying on deductive reasoning and general premises to reach specific conclusions or
inductive approaches that seek general conclusions based on specific premises an
abductive logic is adopted that simultaneously allows the generation of a practical
research solution and the integration of various theories and approaches (Tomiyamal
et al., 2003) with the rationale of developing valid and reliable measurements. The
sequential exploratory design employs a pragmatic approach that sidesteps the debates
between the existence of objective ‘truth’ and the value of subjective perceptions and
balances the etic viewpoint of outsiders’ views (i.e., researchers, expert panels) with
the emic one derived from real life observations. Such a mixed method approach is
likely to produce a rigorous instrument that is superior to using exclusively
quantitative techniques for instrument development (Onwuegbuzie, Bustamante &
Nelson, 2010).
16
Whilst mixed method approaches is not the panacea of all research inquiries the
sequential exploratory design with the rationale of instrument development
adequately serves mixed method’s fundamental principal i.e., maximizing the potency
and minimizing the weaknesses derived from the amalgamation of qualitative and
quantitative techniques (Johnson & Turner, 2003) to enhance validity in political
science research conclusions.
References
Adcock, R., & Collier, D. (2001). Measurement validity: A shared standard forqualitative and quantitative research. American Political Science Review, 95(3), 529-546.
Arnold, M., & Reynolds, K. (2003). Hedonic shopping motivations. Journal ofRetailing, 79(2), 77-95.
Babbie, E. (2002). The Basics of Social Research (2nd ed.). Belmont, CA:Wadsworth/Thomson Learning.
Baumgartner, F., & Walker, J. (1990). Response to Smith’s ‘Trends in voluntarygroup membership: Comments on Baumgartner and Walker’: Measurement validityand the continuity of research in survey research. American Journal of PoliticalScience, 34, 662-670.
Bennett, A., & Braumoeller, B. (2002). Where the model frequently meets the road:Combining statistical, formal, and case study methods. Paper presented at the AnnualMeeting of the American Political Science Association, 28 August, Boston,Massachusetts.
Bergman, M. (2010). On concepts and paradigms in mixed methods research.Journal of Mixed Methods Research, 4(3), 171-175.
Bollen, K. (1980). Issues in the comparative measurement of political democracy.American Sociological Review, 45(3), 370-390.
Bollen, K. (1990). Political democracy: Conceptual and measurement traps. Studies inComparative International Development, 25(1),7-24.
Bowman, K., Lehoucq, F., & Mahoney, J. (2005). Measuring political democracy:Case expertise, data adequacy, and Central America. Comparative Political Studies,38, 939-970.
Bunce, V. (1995). Should transitologists be grounded? Slavic Review, 54(1), 111-127.
Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by themultitrait multimethod matrix. Psychological Bulletin, 56, 81-105.
17
Collier, D., & Levitsky, S. (1997). Democracy with adjectives: Conceptual innovationin comparative research. World Politics, 49, 430-451.
Collier, D., & Mahon, J. (1993). Conceptual stretching revisited: Adapting categoriesin comparative analysis. American Political Science Review, 87, 845-855.
Collins, K., Onwuegbuzie, A., & Sutton, I. (2006). A model incorporating therationale and purpose for conducting mixed methods research in special education andbeyond. Learning Disabilities: A Contemporary Journal, 4, 67-100.
Coppedge, M. (1999). Thickening thin concepts and theories: Combining large-N andsmall in comparative politics. Comparative Politics, 31, 465-476.
Coppedge, M., & Reinicke, W. (1990). Measuring polyarchy. Studies in ComparativeInternational Development, 25(1), 51-72.
Creswell, J. (1999). Mixed-Method research: Introduction and application. In G.Cizek (Ed.), Handbook of Educational Policy, (pp. 455-472). San Diego: AcademicPress.
Creswell, J., Fetters M., & Ivankova N. (2004). Designing a mixed methods study inprimary care. Annals of Family Medicine, 2(1), 7-12.
Creswell, J., & Plano Clark, V. (2011). Designing and Conducting Mixed MethodsResearch (2nd ed.). Thousand Oaks, CA: Sage.
Creswell, J., Plano Clark, V., Guttman, M., & Hanson, W. (2003). Advanced mixedmethods research designs. In A. Tashakkori, & C. Teddlie, (Eds.), Handbook ofMixed Methods in Social and Behavioral Research, (pp.209-240). Thousand Oaks,CA: Sage.
Cronbach, L., & Meehl, P. (1955). Construct validity in psychological tests.Psychological Bulletin, 52(4), 281-302.
Currall, S., & Towler, A. (2003). Research methods in management andorganizational research: Toward integration of qualitative and quantitative techniques.In A. Tashakkori, & C. Teddlie (Eds.), Handbook of Mixed Methods in Social andBehavioral Research, (pp. 513-526). Thousand Oaks, CA: Sage.
De Vaus, D. (1990). Surveys in Social Research (2nd ed.). Sydney: Allen & Unwin.
DeVellis, R. (2003). Scale Development (2nd ed.). Thousand Oaks, CA: Sage.
Dogan, M., & Pelassy, D. (1990). How to Compare Nations: Strategies inComparative Politics (2nd ed.). Chatham, NJ: Chatham House.
Durham, J., Tan B.-K., & White, R. (2011).Utilizing mixed research methods todevelop a quantitative assessment tool: An example from explosive remnants of a warclearance program. Journal of Mixed Methods Research, 1-15.
18
Elkins, Z., & Sides J. (2009). The vodka is potent but the meat is rotten: Evaluatingmeasurement equivalence across contexts. Paper presented at the October 30 JuniorFaculty Research Colloquium, Department of Political Science, George WashingtonUniversity, Washington, DC. [Online] Available at:http://home.gwu.edu/~jsides/vodka.pdf (15 July 2011)
Gelo, O., Braakman, D., & Benetka, G. (2008). Quantitative and qualitative research:Beyond the debate. Integrative Psychological & Behavioral Science, 42, 266-290.
Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology?Journal of Mixed Methods Research, 2, 7-22.
Greene, J., Caracelli, V., & Graham, W. (1989). Toward a conceptual framework formixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11,255-274.
Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R.(2009). Survey Methodology (2nd edn). New York: John Wiley & Sons.
Harrits, G. (2010). Understanding and explaining political action. A mixed methodstrategy. Political Methodology - Concepts and Methods Working Papers Series, 17.Available at:http://www.concepts-methods.org/Files/WorkingPaper/PM%2027%20Harrits.pdf (29June 2011)
Harrits, G. (2011). More than method? A discussion of paradigm differences withinmixed methods research. Journal of Mixed Methods Research, 5(2) 150-166.
Haynes, S. Richard, D., & Kubany, E. (1995). Content validity in psychologicalassessment: A functional approach to concepts and methods. PsychologicalAssessment, 7, 238-247.
Hui, C., & Triandis, H. (1985). Measurement in cross-cultural psychology. A reviewand comparison of strategies. Journal of Cross-cultural Psychology, 16, 131-152.
Jacoby, W. (1991). Data Theory and Dimensional Analysis. Newbury Park, CA:Sage.
Johnson, R., & Onwuegbuzie, A. (2004). Mixed methods research: A researchparadigm whose time has come. Educational Researcher, 33(7), 14-26.
Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixedmethods research. Journal of Mixed Methods Research, 1(2), 112-133.
Johnson, R., & Turner, L. (2003). Data collection strategies in mixed methodsresearch. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods inSocial and Behavioral Research, (pp. 297-319). Thousand Oaks, CA: Sage
19
Lazarsfeld, P., & Barton, A. (1965). Qualitative measurement in the social sciences:Classification, typologies, and indices. In D. Lerner & H. Lasswell (Eds.), The PolicySciences (pp.155-192). Stanford, California: Stanford University Press.
Lieberman, E. (2005). Nested analysis as a mixed-method strategy for comparativeresearch. American Political Science Review, 99, 435-452.
Lindberg, S. (2009). Accountability :The core concept and its subtypes. Africa Powerand Politics Working Paper,1. London: Africa Power and Politics Programme(APPP). Available at: http://www.dfid.gov.uk/r4d/PDF/Outputs/APPP/APPP-WP1.pdf (5 July 2011)
MacIntyre, A. (1971). Is a science of comparative politics possible? In A. MacIntyre(Ed.) Against the Self-Images of the Age: Essays on Ideology and Philosophy, (pp.260-79). London: Duckworth
Mahoney J., & and Goertz G. (2006). A tale of two cultures: Contrasting quantitativeand qualitative research. Political Analysis, 14(3), 227-249.
Mastenbroek, E., & Doorenspleet, R. (2007). Mind the gap! On the possibilities andpitfalls of mixed methods research. Paper presented at the 4th ECPR GeneralConference, September 6-8, Pisa, Italy. Available at:http://www.essex.ac.uk/ecpr/events/generalconference/pisa/papers/PP1466.pdf (23May 2011)
Meijer, P. Verloop, N., & Beijaard, D. (2001). Similarities and differences inteachers’ practical knowledge about teaching reading comprehension. The Journal ofEducational Research, 94 (3), 171-184.
Mak, L., & Marshall, S. (2004). Perceived mattering in young adults' romanticrelationships. Journal of Social and Personal Relationships, 21, 469-486.
Messick, S. (1989a). Meaning and values in test validation: The science and ethics ofassessment. Educational Researcher, 18(2), 5-11.
Messick, S. (1989b). Validity. In R. Linn (ed.) Educational Measurement (pp.13-103), New York: Macmillan.
Messick, S. (1995a). Standards of validity and the validity of standards inperformance assessment. Educational Measurement: Issues and Practice, 14 (4), 5-8.
Messick, S. (1995b). Validity of psychological assessment: Validation of inferencesfrom persons’ responses and performances as scientific inquiry into score meaning.American Psychologist, 50(9), 741-749.
Milton, J., Watkins, K., Studdard, S., & Burch, M. (2003). The ever widening gyre:Factors affecting change in adult education graduate programs in the United States.Adult Education Quarterly, 54(1), 23-41.
20
Moghaddam, F., Walker, B., & Harre, R. (2003). Cultural distance, levels ofabstraction, and the advantages of mixed methods. In A. Tashakkori & C. Teddlie(Eds.), Handbook of Mixed Methods in Social & Behavioral Research, (pp. 111-134).Thousand Oaks, CA: Sage
Moncagatta, P. (2009). The paradox of support for democracy in Venezuela. RECSMWorking Paper 4. Available at: http://www.upf.edu/survey/_pdf/RECSM_wp004.pdf(13 June).
Morse, J., & Niehaus, L. (2009). Mixed Method Design: Principles and Procedures.Walnut Grove, CA: Left Coast Press.
Munk, G., & Verkuilen, J. (2002). Conceptualizing and measuring democracy:Evaluating alternative indices. Comparative Political Studies, 35(1),5-34.
Myers, K., & Oetzel, J. (2003). Exploring the dimensions of organizationalassimilation: Creating and validating a measure. Communication Quarterly, 51, 438-457.
Nagle, R., Suldo, S., Christenson, S., & Hansen, A. (2004). Graduate students’perspectives of academic positions in school psychology. School PsychologyQuarterly, 19, 311-326.
Nassar-McMillan, S., Wyer, M., Oliver-Hoyo, M., & Ryder-Burge, A. (2010). Usingfocus groups in preliminary instrument development: Expected and unexpectedlessons learned. The Qualitative Report, 15(6), 1621-1634.
Northrop, F. (1947). The Logic of the Sciences and the Humanities. New York: WorldPublishing Company.
Onwuegbuzie, A., Bustamante, R., & Nelson, J. (2010). Mixed research as a tool fordeveloping quantitative instruments. Journal of Mixed Methods Research, 4(1) 56-78.
Onwuegbuzie, A. & Johnson, R. (2006). The validity issue in mixed research.Research in the Schools, 13(1), 48-63.
Oppenheim, A. (1992). Questionnaire Design, Interviewing and AttitudeMeasurement (2nd ed.). London, UK: Pinter.
Sartori, G. (1970). Concept misformation in comparative politics. American PoliticalScience Review, 64, 1033-1053.
Padgett, D. (1998). Qualitative Methods in Social Work Research: Challenges andRewards. Thousand Oaks, CA: Sage.
Pike, K. (1967). Language in Relation to a Unified Theory of the Structure of HumanBehavior. The Hague, Netherlands: Mouton.
21
Przeworski, A., Alvarez, M., Cheibub, J., & Limongi, F. (2000). Democracy andDevelopment: Political Institutions and Well-being in the World, 1950-1990.Cambridge, UK: Cambridge University Press.
Rowan, N., & Wulff, D. (2007). Using qualitative methods to inform scaledevelopment. The Qualitative Report, 12(3), 450-466.
Saris, W. & Gallhofer, I. (2007). Design, Evaluation, and Analysis of Questionnairesfor Survey Research. Hoboken, NJ: Wiley.
Sartori, G. (1994). Compare why and how: Comparing, miscomparing and thecomparative method. In M. Dogan & A. Kazancigil (Eds.), Comparing Nations:Concepts, Strategies, Substance, (pp. 14-34). London: Basil Blackwell.
Smith, P., Earp, J., & DeVellis R. (1995). Measuring battering: Development of theWomen’s Experience with Battering (WEB) scale. Women’s Health: Research onGender, Behavior, and Policy, 1(4), 273-288.
Tarrow, S. (1995). Bridging the quantitative-qualitative divide in political science.American Political Science Review, 89(2), 471-474.
Tashakkori, A., & Creswell, J. (2008). Mixed methodology across disciplines.Journal of Mixed Methods Research, 2(1), 3-6.
Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of Mixed Methods in Socialand Behavioral Research. Thousand Oaks, CA: Sage.
Tashakkori, A., & Teddlie, C. (2010). Sage Handbook of Mixed Methods in Socialand Behavioral Research (2nd ed.). Thousand Oaks, CA: Sage.
Tashiro, J. (2002). Exploring health promoting lifestyle behaviors of Japanese collegewomen: Perceptions, practices, and issues. Health Care for Women International,23(1), 59-70.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use ofmixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie(Eds.), Handbook of Mixed Methods in Social & Behavioral Research, (pp. 3-50).Thousand Oaks, CA: Sage.
Tomiyamal, T., Takeda, H., Yoshioka, M., & Shimomura, Y. (2003). Abduction forCreative Design (AAAI Tech. Rep. No. SS-03-02). Menlo Park, Ca: AAAI Press.
Van de Vijver, F. (1998). Towards a theory of bias and equivalence. ZUMANachrichten Spezial, Cross-cultural Survey Equivalence, 3, 41-65.
Van de Vijver, F. (2011). Bias and real differences in cross-cultural differences:Neither friends nor foes. In F. van de Vijver, A. Chasiotis, & S. Breugelmans (Eds.),Fundamental Questions in Cross-cultural Psychology, (pp. 235-257). Cambridge:Cambridge University Press.
22
Van de Vijver, F., & Leung, K. (1997). Methods and Data Analysis for Cross-Cultural Research. Thousand Oaks, CA : Sage.
Van de Vijver, F., & Tanzer, N. (2004). Bias and equivalence in cross-culturalassessment: An overview. European Review of Applied Psychology, 54(2), 119-135.
Viruel-Fuentes, E. (2007). Beyond acculturation: Immigration, discrimination, andhealth research among Mexicans in the United States. Social Science & Medicine, 65,1524-1535.
Vogt, D., King, D., & King L. (2004). Focus groups in psychological assessment:Enhancing content validity by consulting members of the target population.Psychological Assessment, 16, 231-243.