1 group-level measurement katherine klein university of pennsylvania [email protected] carma...
Post on 15-Jan-2016
215 views
TRANSCRIPT
1
Group-Level Measurement
Katherine KleinUniversity of [email protected]
CARMA PresentationFebruary 2007
2
Why Group-Level Measurement?
• Burgeoning of multilevel theory and research in last 25 years
• Great progress in conceptualizing and measuring group-level constructs
– Especially shared constructs
• Continuing challenges and opportunities– Especially regarding configural constructs
3
A Few Terms and Assumptions
• I’ll refer to groups but much or all of what I say will apply as well to organizations, departments, stores, etc.
• I’ll focus on the creation and use of original survey measures to assess group constructs.
• I’ll address statistical issues in passing only.– But see past CARMA presenters including James
LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others
4
Roadmap Fundamentals: Theory First
Construct Types: Global, Shared, and Configural Constructs
Practicalities and Technicalities Survey Wording Sampling Qualitative Groundwork Single-source Bias Justifying Aggregation
Opportunities and Challenges The Configuration of Diversity Social Network Analysis
5
Fundamentals: Theory First
• Constructs are our building blocks in developing and in testing theory.
• High quality measures are construct valid.
• The development of construct valid measures thus begins with careful construct definition.
• Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): – Global, shared, or configural.
6
Global Constructs
• Relatively objective, easily observable, descriptive group characteristics.
• Originate and are manifest at the group level.
• Examples: – Group function, size, or location.
• No meaningful within-group variability.
• Measurement is generally straightforward.
7
Shared Constructs
• Group characteristics that are common to group members
• Originate in group members’ attitudes, perceptions, cognitions, or behaviors– Which converge as a function of attraction, selection,
socialization, leadership, shared experience, and interaction.
• Within-group variability predicted to be low.• Examples:
– Group climate, norms, leader style.
• Measurement challenges are well understood.
8
Configural Group-Level Constructs
• Group characteristics that describe the array, pattern, dispersion, or variability within a group.
• Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes)– But no assumption or prediction of convergence.
• Examples: – Rates, diversity, fault-lines, social networks, team mental
models, team star or weakest member.
• Measurement challenges are less well understood.
9
A Related Framework: Chan’s (1988) Composition Typology
• Shared Constructs– Direct consensus models (e.g., group norms) – Referent shift models (e.g., team efficacy)
• Configural Constructs– Dispersion model (e.g., climate strength)– Additive models (e.g., mean group member IQ)
• Multilevel, Homologous Models– Process model (e.g., efficacy-performance
relationship)
10
Construct Definition Complexities:An Example: Shared Leadership
• Shared leadership– “A dynamic, interactive influence process among
individuals in work groups in which the objective is to lead one another to the achievement of group goals… [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence”
– Conger & Pearce, 2003, p. 286
• Is this a shared construct, or a configural construct, or … ?
11
Construct Definition Complexities:An Example: Shared Leadership
• Well, how would you measure it?
– Shared team leadership as a shared construct• “Team members share in the leadership of this team.”• “Many team members provide guidance and direction for
other team members.”
– Shared team leadership as a configural construct (network density):
• “To what extent do you consider _____ an informal leader of the team?”
12
Construct Definition Complexities:An Example: Shared Leadership
• Calling it a “referent shift” construct is not the answer.– Referent shift is a measurement strategy, not a
construct type
• Shifting the referent in an unthinking manner can be quite problematic:– The members of my team…
• “Express confidence that we will achieve our goals” • “Will recommend that I am compensated more if I perform
well” • “Are friendly and approachable” • “Rule with an iron hand”
13
A Quick Recap
• Theory first: Define and explain the nature of your group-level constructs.
– Is it a clearly objective description of the group? • If yes, a global construct.
– Do you expect within-group agreement? • If yes, a shared construct.
– Does it describe the group in terms of the pattern or array of group members on a common attribute?
• If yes, a configural construct.
14
Now What?
• Having defined your constructs, the goal is to create measures that:– Are construct valid– Show homogeneity within (shared constructs)– Show variability between (all group-level constructs)
• Practicalities and technicalities– Survey wording– Sampling– Qualitative groundwork– Minimizing single-source bias– Testing for aggregation
15
Survey Wording:Global Constructs
• Draw attention to objective descriptions of each group.
• Gather data from experts and observers (SMEs) who can provide valid information about the groups in question.
• No need to gather data from individual respondents within groups
• Use language that fits your sample.
16
Survey Wording: Shared Constructs
• Draw attention to shared group characteristics
• Use a group referent rather than individual referent to enhance: – Within group agreement– Between group variability– Predictive validity
• Gather data from individual respondents so within-group agreement can be assessed.
• Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.
17
Survey Wording:Configural Constructs
• Draw attention to individual group member characteristics by using an individual referent.
• Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups.
• The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.
18
Sampling
• Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest
• Whether they are global, shared, or configural.
• Statistical power reflects both: – Group sample size (n of groups)– Within-group sample size
• When group size is large (number of respondents per group), measures of shared constructs are more reliable.
• More research needed on power in multilevel analyses.
19
Qualitative Groundwork
• The survey wording and sampling guidelines seem fairly obvious and easy, but …
• Check your assumptions in the field prior to survey data collection.– Are you measuring the right “groups”?
• Example: Grocery stores or departments?– Is there meaningful between-group variability?
• Example: Fast food chain– Are you measuring the right variables, and not too
many of them?• Beware the blob.
20
Single-Source Bias
• Group-level correlations between measures of shared group constructs may be disturbingly high.– Examples:
• Transformational and transactional leadership• Task, emotional, and procedural conflict
• Aggregation does not “average away” response biases.• Rather, group members may share response biases
– Halo, logical consistency, social desirability
• Response bias may be particularly influential when respondents must make subtle distinctions among constructs.
21
Single-Source Bias:Beating the Blob
• Survey measures– Choose and measure truly distinct constructs– Use different survey response formats
• Survey design– Keep survey items measuring distinct
constructs separate.• Help respondents recognize the distinction
between leadership types, or conflict types, for example.
22
Single-Source Bias:Beating the Blob
• Survey analysis– Randomly split the within-group sample of
respondents during data analysis.• All receive the same survey, but half provide IV and the other
half provide the DV for analyses
• Survey administration– Randomly split the within-group sample of
respondents during data administration. • Respondents receive distinctive surveys. Half receive the IV
survey and the other half receive the DV survey.
23
A Quick Recap
• Having– Defined our constructs– Written our survey items– Conducted qualitative groundwork– Sampled appropriately– Taken steps to reduce single source bias
• We’re almost ready for hypothesis testing
• But first: We need to justify aggregation
24
Justifying Aggregation
• Why is this essential?– In the case of shared constructs, our very construct
definitions rest on assumptions regarding within- and between-group variability.
– If our assumptions are wrong, our construct “theories,” our measures, and/or our sample are flawed and so are our conclusions.
• So, test both:– Within group agreement
• The construct is supposed to be shared, but is it really?– Between group variability (reliability)
• Groups are expected to differ significantly, but do they really?
25
Justifying Aggregation: rwg(j)
• Developed by James. Demaree, & Wolf (1984)• Assesses agreement in one group at a time.• Compares actual to expected variance.• Answers the question:
– How much do members of each group agree in their responses to this item (or this scale)?
• Highly negatively correlated with the within group standard deviation
• Valid values range from 0 to 1• Rule of thumb: rwg(j) of .70 or higher is
acceptable
26
Justifying Aggregation: rwg
• Common to report average or median rwg(j) for each group for each variable:– If rwg(j) is below .70 for one or more groups, check:
• Does the group have low rwg(j) values on several variables?
• Do many groups have low rwg(j) values on this variable?
• Remember: rwg(j) indicates within-group agreement, not between-group variability.
• Beware: When variance in a group exceeds expected variance, out of range rwg(j) result.
27
Justifying Aggregation: 2
• Assesses between-group variance relative to total variance, across the entire sample.
• Based on a one-way ANOVA• Answers the question:
– To what extent is variability in the measure predictable from group membership?
• The F-test provides a test of significance– The larger the sample of individuals, the more likely
eta2 is to be significant.• Beware: 2 may be inflated when group sizes
are small (under 25 individuals per group)– But, this is an easy way to begin tests of aggregation
28
Justifying Aggregation: ICC(1)
• Assesses between-group variance relative to total variance
• Based on a one-way ANOVA• Answers the question:
– To what extent is variability in the measure predictable from group membership?
• The F-test provides a test of significance• Based on 2 but controls for the number of
predictors relative to the total sample size, so ICC(1) is not biased by group size.
29
Justifying Aggregation: ICC(2)
• Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size.
• Answers the question: – How reliable are between-group differences on the
measure?
• Reflects ICC(1) and within-group sample size– Example: If ICC(1) = .20 and:
• Mean group size is 5, expected ICC(2) = .56• Mean group size is 20, expected ICC(2) = .71
30
Justifying Aggregation: An Example
2 ICC(1) ICC(2) Average rwg(j)
Financial Resource Availability
.28*** .21 .75 .65
Mgt. Support .31*** .19 .61 .78
Policies and Practices
.33*** .22 .74 .79
Implement. Climate
.23*** .15 .64 .81
31
A Quick Recap
• The hope is that we have successfully:– Defined our constructs.– Written our survey items.– Conducted qualitative groundwork.– Collected data from a large sample of groups.– Taken steps to reduce single source bias.– Justified aggregation.– And moved on to test our hypotheses.
• So, what remains?
32
Opportunities and Challenges:The Configuration of Diversity
• Configural constructs describe the array, pattern, dispersion, or variability within a group.– The easy example is diversity
• Demographic diversity• Climate strength
• But even the easy example isn’t so easy: What is the definition of diversity? And how should it be measured?
33
The Configuration of Diversity
• A starting definition of diversity:– The distribution of differences among the members of
a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay.
• Okay, but what’s maximum diversity? – Which team has maximum age diversity?
• 20, 20, 20, 70, 70, 70• 20, 30, 40, 50, 60, 70• 20, 20, 20, 20, 20, 70• 20, 70, 70, 70, 70, 70
34
The Configuration of Diversity
• Diversity isn’t one thing.
• It’s three things: Separation, Variety, or Disparity
• The three types differ in: – Meaning or substance – Pattern or shape – Likely consequences– Appropriate operationalization
• Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions
35
The Configuration of DiversityExample: Three Research Teams
• Team S – Members differ in their view of qualitative research.
• Half of the team members respect it, half don’t.
• Team V– Members differ in their discipline.
• 1 psychologist, 1 sociologist, 1 anthropologist, etc.
• Team D– Members differ in their rank
• 1 senior professor, others are incoming graduate students.
36
Diversity as Separation
• Differences in group members’ position, attitude, or opinion along a continuum
• Min: Every member has the same opinion
• Max: Two polarized extreme factions
• Theory: Similarity-attraction
• Operationalization: Standard deviation
37
Diversity as Variety
• Differences in kind or category
• Min: Every member is the same type
• Max: Each group member is a different type
• Theory: Requisite variety, cognitive resource heterogeneity
• Operationalization: Blau’s index of categorical differences
38
Diversity as Disparity
• Differences in concentration or proportion of valued assets or resources
• Min: Every member has an equal portion of the resource
• Max: One member is “rich” and all others are “impoverished”
• Note: Disparity is asymmetric
• Theory: Inequality, relative deprivation, tournament compensation
• Operationalization: Coefficient of variation (SD/Mean)
39
The Configuration of Diversity:A Recap
• Theory first– Separation is about position, attitude, or opinion
– At maximum: Polarized factions
– Variety is about knowledge or information.– At maximum: One of a kind
– Disparity is about resources or power. – At maximum: One towers over others
• Operationalize accordingly– The coefficient of variation is not a default or catch-all
40
Opportunities and Challenges: Social Network Analysis
• Multilevel analysis and social network analysis have developed along separate paths.
• Rich opportunities for cross-fertilization.
• Social network analysis provides a means to conceptualize and operationalize configural constructs.– Illuminating the pattern or array of interpersonal ties
within a group
41
Opportunities and Challenges: Social Network Analysis
• Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups.
• Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict
• But we know little about the configuration of interpersonal ties – the structures – that underlie our shared constructs and measures.
42
An Example: Social Network Analysis and Shared Team Conflict
• When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team?
• As a starting point:– How dense are positive (advice) ties? – How dense are negative (difficulty) ties?
43
An Example: Social Network Analysis and Shared Team Conflict
Task and emotional conflict: The blob r = .83
Advice density and negative tie density: More weakly correlated r = -.36
Task conflict (mean task and emotional conflict), advice density, and negative tie density Team Conflict and Advice Density: r = -.47 Team Conflict and Difficulty Density r = .40
44
Negative Tiesin a Low Conflict Team
45
Negative Ties in a High Conflict Team
46
Advice Ties in a High Conflict Team
47
Advice Ties in a Low Conflict Team
48
Social Network Analysis:A Recap
• Social network analysis illuminates the configuration of interpersonal ties in groups.– What network structures underlie our shared
constructs and measures? – Do network measures provide incremental validity?
• Not just density, but centralization, cliques, and more.
• What explains between-group differences in network structures?
49
In Conclusion
• Theory first. Define your constructs. – Are they global, shared, or configural?
• Measure constructs and collect data with care– Match item wording to the construct – Conduct qualitative groundwork– Sample appropriately– Take steps to reduce single source bias– Test for aggregation
• Studying configural constructs remains a challenge and an opportunity– Conceptualizing and measuring diversity– Integrating social network analysis within our arsenal
50
Some Helpful References
1. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
2. Borgatti, S. P. (2003). The network paradigm in organizational research: A review and typology. Journal of Management, 29, 991-1013.
3. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
4. Harrison, D. A. & Klein, K. J. (2007). What’s the difference? Diversity as separation, variety, or disparity in organizations. Academy of Management Review.
5. Harrison, D. A. & McLaughlin, M. E. (1996). Structural properties and psychometric qualities of organizational self-reports: Field tests of connections predicted by cognitive theory. Journal of Management, 22, 313-338.
6. James, Demaree, & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
51
Some Helpful References7. Klein, K. J., Conn, A. B., Smith, B., & Sorra, J. S. (2001). Is everyone in
agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
8. Klein, K. J., Conn, A. B. & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86, 3-16.
9. Kozlowski, S. W. J. & Klein, K. J. (2000). A multilevel approach to theory and research in organizations. In Klein, K. J. & Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3-90). San Francisco: Jossey-Bass.
10. Morgeson, F. P. & Hofmann, D. A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24, 249-265.
11. Ostroff, C., Kinicki, A. J., & Clark, M. A. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfactoin relationships. Journal of Applied Psychology, 87, 355-368.