current issues in individual, group, and organizational level measurement : strategic management...
TRANSCRIPT
Current Issues in Individual, Group, and Organizational Level
Measurement :Strategic Management
Prof. Brian BoydW. P. Carey School
Arizona State Universityhttp://www.briankboyd.com
RoadmapRoadmap
• Background
• State of the art
• Consequences of measurement problems
• Application
BackgroundBackground
• Most of my papers have a strong measurement component, e.g.,– SMJ 90: Environmental characteristics– JMS 91: Implications for planning performance– SMJ 94: Board control– AMJ 98: Discretion and executive pay– SMJ 98: Planning systems– SMJ 01: CEO social capital
• Reviewers often profoundly uninterested in measurement
• What is the ‘state of the art’ for construct measurement in strategy?
• What are the implications of this?
Strategy and Strategy and Paradigm Paradigm
DevelopmentDevelopment• In a Kuhnian sense, management is a
young discipline
• Strategy is one of the youngest subsets of management
• Empirics play a key role in advancing paradigm development
How well does strategy research How well does strategy research contribute to the goal of advancing contribute to the goal of advancing
the paradigm?the paradigm?
Where Were We Two Decades Where Were We Two Decades Ago?Ago?
• Frequent use of nominal and single-item scales
• Inadequate assessment of validity
• Inexplicit treatment of strategy as an organization-level construct
Source: Venkatraman & Grant (1986) AMR
Strategy & Methods:Strategy & Methods:An OverviewAn Overview
• Inattention to methodology and design (Bowen & Wiersma, 1999)
• Few replication studies (Hubbard et al., 1997)
• Insufficient power (Mone et al., 1996)
• Little emphasis on construct measurement (Hitt, Boyd & Li, 2004)
Overall, the rigor of much strategy Overall, the rigor of much strategy research is weak.research is weak.
We’ll focus on these issues
A Power PrimerA Power Primer
Type I errorType I error
Type II errorType II error
Example: Population correlation = .30, a = .05Example: Population correlation = .30, a = .05
N = 30, 50% chance of successful detection
N = 50, 70% chance
N = 100, 90+% chance
Power in Strategy ResearchPower in Strategy Research• Published strategy papers had only 4 in 10 chance
of successful detection – half of recommended level (Mazen, Hemmasi & Lewis, 1987)
• Review of one research stream – only 8% of studies had sufficient power (Ferguson & Ketchen, 1999)
• SMJ has lower power than other journals (Mone, Mueller & Mauland, 1996)
• Few studies report power statistics (Mone et al, 1996)
This is the This is the goodgood news… news…
Blalock’s (1979) Elements of Social Blalock’s (1979) Elements of Social ProcessesProcesses
1. A theoretical language explaining causal relations between constructs
2. An operational language for examining relationships between constructs using indicators
3. An integrative theory describing causal relationships between constructs and indicators
Quality of Strategy Measures?Quality of Strategy Measures?
Lack of consensus on Lack of consensus on measurement of basic measurement of basic strategy constructsstrategy constructs
• Strategic planning• Performance• Diversification• Board oversight• Environment• Discretion
Low emphasis on Low emphasis on measurementmeasurement
• Prevalence of single measures
• Sporadic reporting of reliability and validity
The Role of Measurement The Role of Measurement QualityQuality
ConstructConstructxx ConstructConstructyy Hypotheses Hypotheses framedframed
IndicatorIndicatorxx IndicatorIndicatoryy Hypotheses Hypotheses testedtested
Weak indicators lead to measurement Weak indicators lead to measurement error, and attenuation. Power error, and attenuation. Power calculations are based on fully accurate calculations are based on fully accurate constructs.constructs.
In other words, the statistics presented In other words, the statistics presented earlier earlier overestimateoverestimate the power of the power of strategy researchstrategy research
An ExampleAn Example
ConstructConstructxx ConstructConstructyy N = 150N = 150power = .80power = .80
0.300.30
IndicatorIndicatorxx IndicatorIndicatoryy N = 150N = 150power = .33power = .33
0.100.10
.60.60 .60.60
State of the Art & Consequences of State of the Art & Consequences of Measurement Problems in Strategy Measurement Problems in Strategy
ResearchResearch
Study 1: AssessmentStudy 1: Assessment• Content analysis of
several hundred published articles
• Multiple journals• Multiple years
Study 2: ConsequencesStudy 2: Consequences• Replication of a
prominent research question
• Large sample analysis of US firms
• Manually adjust levels of measurement precision
State State of the Artof the Art
AssessmentAssessment• Identified all empirical strategy papers in key
outlets: AMJ, ASQ, MS, and SMJ• Sampled articles published 1998 -- 2002• Content coded articles for:
– Sample size– Measurement schemes– Reliability
• Statistical test is unit of analysis: 625 tests in 196 articles
• High inter-rater reliability on selection & coding of studies
DescriptivesDescriptives
MeanMean Std. DevnStd. Devn
Sample size 2,559 12,908
IVs 5.66 5.09
Controls 8.62 30.26
Obs/Inds 236.12 1,333.82
Sample size limitations do not Sample size limitations do not appear to constrain measurementappear to constrain measurement
Good News…Bad NewsGood News…Bad News
0.7
0.72
0.74
0.76
0.78
0.8
0.82
0.84
Reliability
IV
DV
Control
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
No Data
IV
DV
ControlWhile reported reliabilities are While reported reliabilities are strong, this information is typically strong, this information is typically
notnot available in the article. available in the article.
Summary of Content AnalysisSummary of Content Analysis
• Most strategy studies rely on single indicators
• Limited efforts made to report reliability data, when multiple indicators are used
• Many studies report either partial or no correlation matrices, making post hoc asessment impossible
Is this a salient problem,Is this a salient problem, or a non-issue?or a non-issue?
ConsequencesConsequences
Why Do Firms Diversify?Why Do Firms Diversify?
• Amihud & Lev (1981): Diversification is a result of agency conditions.
• Lane, Cannella & Lubatkin (1998): No support for A&L agency model.
• Denis , Denis & Sarin (1999): Both studies contaminated by methodological flaws. Substituting improved measures led to stronger results.
““Though both sets of authors conduct Though both sets of authors conduct similar empirical tests on virtually similar empirical tests on virtually identical data, they arrive at completely identical data, they arrive at completely different conclusions.” Denis et al., 1999different conclusions.” Denis et al., 1999
Methods Issues in the Agency Methods Issues in the Agency ArgumentArgument
MeasurementMeasurement• Both A&L and LCL
studies used broad ownership categories
• Denis et al found better results when using ratio level indicators
• Only single indicator measures used
PowerPower• 3 IV regression, w/ a=.05
power = .80
N = 34 for large effect76 for moderate547 for small
• LCL N=309Denis et al N=933
• Bulk of these studies had insufficient power
Insiders
StockOwnership
BoardControl
Diversification
Firm Size
Duality
Owner Reps
Director Pay
Related
Unrelated
-.16 [2.2]
.11 [2.3]
(Palepu, 1985)
(Boyd, 1994)
Sales
Assets
Equity
Sample:Sample:640 640 Fortune Fortune firms, representing 50 firms, representing 50 2-digit SIC s, and 200 4-digit SICs2-digit SIC s, and 200 4-digit SICs
BoardControl
Diversification
Firm Size
-.15 [2.1]-.12 [1.7]
.11 [2.3]
Related
Unrelated
(Palepu, 1985)
(Boyd, 1994)
Sales
Assets
Equity
Now, Let’s Introduce Now, Let’s Introduce Some Shadows…Some Shadows…
Duality
Insiders
StockOwnership
Owner Reps
Director Pay
BoardControl
Diversification
Firm Size
.09 [2.2] .13 [2.9] -.07 [1.6]
-.05 [1.3] .05 [1.2] .06 [1.5]
.12 [2.7] -.06 [1.3] -.03 [0.7]
-.02 [0.5]
.11 [2.3]
Related
Unrelated
(Palepu, 1985)
(Boyd, 1994)
Sales
Assets
Equity
……And Some More…And Some More…Duality
Insiders
StockOwnership
Owner Reps
Director Pay
Quantifying Shadows…Quantifying Shadows…
(a) Single indicator DV(a) Single indicator DV(2 models)(2 models)
• Failure to detect effect in 1 of 2 models
• R2 drops by a 10X factor
• Path coefficient drops slightly
(b) Single indicator DV (b) Single indicator DV and IV (10 models)and IV (10 models)
• Failure to detect effect in 7 of 10 models
• R2 drops by a 10X factor
• Path coefficient cut in half
Bottom Line:Bottom Line:The A&L and LCL debate is The A&L and LCL debate is
entirely an artifact of measurement entirely an artifact of measurement error and attenuation.error and attenuation.
Do We Know What We Know?Do We Know What We Know?• Strategy research has placed
little emphasis on statistical power
• Construct measurement is not a priority of empirical research
• Many strategy constructs are broad
• Weak consensus regarding measurement of key concepts
How many valid research How many valid research questions have been cast questions have been cast aside due to these method aside due to these method limitations?limitations?
Sales Assets
Employees Other
Measuring Firm Size
What’s Next?What’s Next?
A Comparison: Marketing
Two decades agoTwo decades ago
• The field of marketing was criticized for failing to pay attention to construct validity and associated measurement issues
TodayToday
• Studies routinely provide reliabilities, factor models, and evidence of convergent and discriminant validity
Source: Jarvis, Mackenzie & Podsakoff (2005) JCR
Some Easy StepsSome Easy Steps
• Sneak measurement into substantive topics
• Less attention to knee-jerk criticism (why ROE?) and more critical review of published measures
• Collect multiple measures
• Pay attention to validity
• Hang out with the OB crowd
More Balance in Research FocusMore Balance in Research Focus
ConstructConstructxx ConstructConstructyy SubstantiveSubstantiveResearchResearch
IndicatorIndicatorxx IndicatorIndicatoryyConstructConstruct
MeasurementMeasurementResearchResearch
Factor Models for Better Factor Models for Better MeasurementMeasurement
X X X
A
X X X
C
X X X
B
(a) Common Approach(a) Common Approach
Factor Models for Better Factor Models for Better MeasurementMeasurement
X X X
A
X X X
C
X X X
B
(b) Convergent (b) Convergent andand discriminant validity discriminant validity
Factor Models for Better Factor Models for Better MeasurementMeasurement
X X X X X XX X X
A
(b) CMV test(b) CMV test
BoardControl
StockOwnership
Insiders
Duality
Owner Reps
Director Pay
Constructs: Cause or Effect?Constructs: Cause or Effect?
Insiders
StockOwnership
Duality
Owner Reps
Director Pay
BoardControl
Constructs: Cause or Effect?Constructs: Cause or Effect?
Cause or Effect? Deciding…Cause or Effect? Deciding…Item Reflective Formative
Indicators needed Representative pool Census
Directionality Construct drives indicator
Indicators drives construct
Indicators imperfect representation of construct
Yes Yes
Measurement error Indicator level Construct level
Reliabilities Good Good or bad
Sensitivity to change in indicators
Minimal Substantial
What About Board Control?What About Board Control?
• Substitution, portfolios, and balancing (Rediker & Seth ’85 Finkelstein & D’Aveni ’84) all suggest reflective model
• CED of CFA is 0.97• All indicators load at .001, with
some moderate loading• Supplementary model using a
sixth indicator (CEO tenure)– Main model unaffected– 8 of 12 subsets NS: 67% versus
70% NS in 5-factor tests
BoardControl
StockOwnership
Insiders
Duality
Owner Reps
Director Pay
EnvironmentDimensions
Objective Perceptual
Industry Industry
Firm Outsiders Simulation
Insiders
TMT
Middle ManagersSource: Boyd, Dess & Rasheed (1993) AMR
Slope of SalesMean of Sales
Munificence
Discretion
Levels of Levels of MeasurementMeasurement
Organizational Research MethodsOrganizational Research MethodsSpecial issue on
Strategic Management
Sample topics:Sample topics:• Pitfalls of using archival proxies• Evaluation of construct validity of key metrics• Providing insights into studying ‘unobservables’• Articles/notes and empirical/conceptual papers
welcome• Click here for Call for Papers• Proposals due March 1, 2006
Web Resources
• Boyd, Gove & Hitt (2005). Construct measurement in strategic management research, SMJ
• Boyd, Gove & Hitt (2005). Consequences of measurement problems in strategic management research, SMJ
• Manuscript draft, reviews, and reviewer replies to BGH 2005a/b
• Hitt, Boyd & Li (2004). The state of strategic management research, RMSM