1 measuring group-level psychological properties (a tribute to larry james) daniel a. newman...

1

Measuring Group-Level Psychological Properties

(A Tribute to Larry James)

Daniel A. Newman

University of Illinois

Daniel A. Newman, Ph.D.

22

OverviewGroup-Level Psychological Properties?

1.Psychological Climate Group-Level vs. Individual-Level Constructs

2.Aggregation Bias

3.Why we need rWG (Within-group agreement) Justifying Aggregation

4.rWG(J) for multi-item scales Agreement vs. Reliability

333


1.Psychological Climate James & Jones (1974), Jones & James (1979), James & Sells (1981),

James (1982), James et al. (1988), James & James (1989)

2.Aggregation Bias James (1982), James et al. (1980)

3.Why we need rWG (Within-group agreement) James (1982), James, Demaree, & Wolf (1984; 1993), George &

James (1993)

4.rWG(J) for multi-item scales James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005)

4444


1.Psychological Climate James & Jones (1974), Jones & James (1979), James & Sells (1981),

James (1982), James et al. (1988), James & James (1989)

2.Aggregation Bias James (1982), James et al. (1980)

3.Why we need rWG (Within-group agreement) James (1982), James, Demaree, & Wolf (1984; 1993), George &

James (1993)

4.rWG(J) for multi-item scales James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005)

555

Quotes & EquationsIn summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach:

1.Quotes

2.Equations

66

Quotes & EquationsIn summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach:

1.Quotes

2.Equations

)(1 22ExWG j

sr

7

Levels of Analysis• In social science, hypothetical constructs

reside at multiple levels of analysis (or levels of aggregation):– National Level: Culture– Organizational Level: Organizational Climate,

CEO personality, Strategy– Team Level: Team efficacy, Norms, Leader style– Individual Level: Attitude, Personality, Job

Performance, Psychological Climate

8

Levels of Analysis

Group

Organizational

Individual

9

Levels of Analysis• Individuals are nested within Groups• Groups are nested within Organizations

• One level can influence another – Group norms influence individual behavior– Individual behaviors aggregate to produce

group/team performance

1010

Psychological Climate• Psychological Climate – ‘the meaning an

individual attaches to a work environment’

• Organizational Climate – ‘the aggregated meaning; i.e., the typical, average, or usual way people in a setting [work environment] describe it’

• Schneider (1981, pp. 4-5), as cited by James (1982)

111111

Psychological Climate• Psychological Climate – individual level

construct

• Organizational Climate – group level construct

121212

Psychological Climate• “… perceptual agreement implies a shared

assignment of psychological meaning, from which it follows that an aggregate (mean) climate score provides the opportunity to describe an environment in psychological terms.”

• “Furthermore, given perceptual agreement, I submit that a climate construct at the aggregate level is defined in precisely the same manner as it is at the individual level.”

• James (1982, p. 221)

1313131313

Psychological ClimateRelationship between organizational climate and

psychological climate:

• PC = psychological climate perception of person in a group

• OC = organizational climate of the group

)(PCaverageOC

141414



• PCpg = psychological climate perception of person p in group g

• OC0g = organizational climate in group g

g

n

ppgg nPCOC

g

1

0

1515



• PCpg = psychological climate perception of person p in group g

• OC0g = organizational climate in group g • upg = deviation of person p’s individual psych.

climate perception from group g’s org. climate

pggpg uOCPC 0

161616

Psychological ClimateJames & Jones (1974), reviewed 3 approaches to conceptualize & measure org. climate:

1)Org.-Level Attribute, Multiple Measures2)Org.-Level Attribute, Perceptual Measures3)Indiv.-Level Attribute, Perceptual Measures*

* Introduced the term, “Psychological Climate”

171717

James & Jones (1974)• “Returning to the perceptual definition of organizational

climate, it would seem that the reliance on perceptual measurement may be interpreted as meaning that organizational climate includes not only descriptions of situational characteristics, but also individual differences in perceptions and attitudes. This is somewhat confusing if one wishes to employ organizational climate as an organizational attribute or main effect, since the use of perceptual measurement introduces variance which is a function of differences between individuals and is not necessarily descriptive of organizations or situations. Therefore, the accuracy and/or consensus of perception must be verified if accumulated perceptual organizational climate measures are used to describe organizational attributes (Guion, 1973).” (p. 1103)

18181818

Jones & James (1979)• “The [conceptual] argument for aggregating perceptually

based climate scores (i.e., psychological climate scores) appears to rest heavily on three basic assumptions: first, that psychological climate scores describe perceived situations; second, that individuals exposed to the same set of situational conditions will describe these conditions in similar ways; and third, that aggregation will emphasize perceptual similarities and minimize individual differences. Based on this logic, it is generally presumed that empirically demonstrated agreement among different perceivers implies that these perceivers have experienced common situational conditions (Guion, 1973; Insel & Moos, 1974; James & Jones, 1974; Schneider, 1975a),”

• (p. 206).

19191919

James & Jones (1974)• “Although this school of thought [from Schneider and others]

assumes that situational and individual characteristics interact to produce a third set of perceptual, intervening variables, such an assumption does not mean that perceived climate is different from an individual attribute. Rather, the intervening variables are individual attributes which provide a bridge between the situation and behavior.”

• (p. 1107)

• So … “Psychological Climate” is born!

2020202020

James (1982)• “current thinking in climate suggests that the unit of theory

for climate, including organizational climate, is the individual, and the appropriate unit to select for observation is the individual. This thinking is based on the view that climate involves a set of macro perceptions that reflect how environments are cognitively represented in terms of their psychological meaning and significance to the individual.”

• (p. 219)

• So … measuring organizational climate (an org.-level attribute) involves an individual-level true score (i.e., psychological climate).

212121212121

James et al. (1988)• “Shared assignment of meaning justifies aggregation to a

higher level of analysis (e.g., groups, subsystems, organizations) because it furnishes a way of relating a construct (PC) that is defined and operationalized at one level of analysis (the individual) to another form of the construct at a different level of analysis (e.g., group climate, subsystem climate, OC). Although the unit of analysis for the aggregate psychological variable is the situation (e.g., group, subsystem, organization), the definition and basic unit of theory remains psychological.”

• (p. 130, from Organizations Do Not Cognize)

222222222222

James & James (1989)General PC

Leader Support

Role Stress, Conflict,

Ambiguity

Job Autonomy, Challenge

Group Warmth

& Cooperat.

.85.86 .77 .81

- PC = Cognitive evaluation of work environment- See James & Sells (1981), Jones & James (1979)

23232323232323

Psychological Climate

Psych. Climate

Job Satisfaction

/Affect

- Reciprocal relationship between PC and Job Satis./Affect- James & Tetrick (1986), James & James (1992)

242424

Psychological ClimateSummary:•There is a group-level organizational reality (“the situation”)•That reality is reflected in individual-level, psychological perceptions•The individual-level psychological climate perceptions are a meaningful locus of theory•The individual perceptions can be aggregated to represent a group-level, psychological property [if perceptions are shared]

25

Aggregation Bias

Aggregation – combining micro-level data so it can represent the macro-level (typically, by taking an average of micro-level responses)

• The aggregate of individuals’ scores represents the group-level construct

2626

Levels of Analysis

Group

Organizational

Individual

27

Aggregation• Ecological fallacy – generalizing group-level

(aggregate) results to the individual level– Because we know group collectivism is related to group-

level cooperation, we inaccurately assume individual collectivism is related to individual cooperativeness.

• Atomistic fallacy – generalizing individual-level results to the group (aggregate) level– Because we know indiv. IQ is strongly related to indiv.-

level job performance, we inaccurately assume group IQ is strongly related to group performance.

28

Aggregation

The Truth about Aggregates:

• If the individual-level correlation between X and Y is rindiv. = .3, this does not imply that the group-level correlation between X and Y is rgroup = .3.

• Likewise, if the group-level correlation between X and Y is rgroup = .3, this does not imply that the individual-level correlation between X and Y is rindiv. = .3.

29

AggregationDirection of a correlation (+ or -) can change when we

move from the individual level to the group level. Within-Group Correlation Between-GroupCorrelation

Y

X

30

AggregationExample) Foreign birth & Illiteracy (Robinson, 1950).rindiv. = .12; rgroup(states) = -.53

Within-Group Correlation Between-GroupCorrelation

Y

X

31

AggregationTotal correlation is a combination of the individual-

level correlation and the group-level correlation. Within-Group Correlation Between-GroupCorrelation

Total

Correlation

Y

X

rtotal

rbetween

rwithin

32

Aggregation• Total correlation is a combination of the

individual-level (within) correlation and the group-level (between) correlation.

),( withinbetweentotal rrfr

33

Aggregation• Specifically,

• rtotal = overall X-Y correlation, ignoring group membership

• rbetween = between-groups X-Y correlation• rwithin = within-groups X-Y correlation

• (from ANOVA; DV= X, IV= group) [like R2; variance in X accounted for by group membership, then inflated by the

unreliability of group means; i.e., .]

)1)(1()( 22yxwithinyxbetweentotal rrr

totalbetweenx SSSS2

)2(/)1(2 ICCICCx

34

Aggregation• For example, suppose• rbetween = -.45 = between-groups X-Y correlation• rwithin = .20 = within-groups X-Y correlation

• = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group)Then …


2x2y

27.)81.1)(64.1(20.)81.64.(45. totalr

35

Aggregation• For example, suppose• rbetween = -.45 = between-groups X-Y correlation• rwithin = .20 = within-groups X-Y correlation

• = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group)Then …


2x2y

27.)81.1)(64.1(20.)81.64.(45. totalr

36

AggregationTotal correlation is a combination of the individual-

level correlation and the group-level correlation. Within-Group Correlation Between-GroupCorrelation

Total

Correlation

Y

X

rtotal

rbetween

rwithin

37

Aggregation

Implications:• Even if total correlation between X and Y

(rtotal) is statistically significant, – rwithin might not be– rbetween might not be

* Many studies in top journals report total relationships between variables, while ignoring nesting/ nonindependence (e.g., different groups, different jobs, different supervisors). Considering levels of analysis could potentially change the results!

3838

Aggregation

Implications:• So-called “aggregation bias” – when rbetween is

larger than rtotal

– Only occurs if rbetween happens to be larger than rwithin


39

Aggregation Bias

Implications:• Don’t look at rtotal to draw inferences about

rwithin!• Don’t look at rtotal to draw inferences about

rbetween!

• See James (1982) and James, Demaree, & Hater (1980), who applied similar formulae to estimate bias in both and corr.’s between aggregated situational (OC) and individual difference variables.


40404040

Aggregation BiasSummary:•When we aggregate individual-level measures (e.g., psychological climate) to represent organizational attributes (e.g., organizational climate), then all the theoretical and empirical relationships can change.•Aggregation of the same measures can create a different construct!

414141

Why We Need rWG

• Justifying Aggregation

• “… organizational climate is the overall meaning derived from the aggregation of individual perceptions of a work environment (i.e., the typical or average way people in an organization ascribe meaning to that organization) (James, 1982; Schneider, 1981). Thus, organizational climate can be viewed as the outcome of aggregating individuals’ psychological climates. The important caveat is that these psychological climates are shared in order to make the inference that an organizational climate exists.”

• James et al. (2008, pp. 15-16)

42

Why We Need rWG

Group-Level Consensus Constructs• In measuring group consensus constructs,

agreement and reliability are tools used to justify aggregation of individual-level responses to the group level

• Agreement and reliability help us gauge how well the average across individual responses represents the group.

43

Why We Need rWG

Group-Level Consensus Constructs

Organizational Climate

(average)

Psych. Climate, Person #1



44

Why We Need rWG

Overview• Aggregation/Composition Models

– Chan (1998)– Kozlowski & Klein (2000)

• Agreement– rWG family of indices

• Reliability– ICC(1)– ICC(2)

See Bliese, 2000

45

Why We Need rWG

• Aggregation/Composition Models– Chan (1998)– Kozlowski & Klein (2000)

• Both typologies include consensus models– Use the mean of individual responses to

represent the group-level construct– Assume isomorphism (James, 1982)

– Require high within-group agreement

46

Why We Need rWG

• Within-Group Agreement – degree to which ratings from individuals are interchangeable– Agreement-based tests reflect degree to which

raters provide essentially the same rating

– Three dominant indices designed to assess within-group agreement:

• James et al.’s (1984) rWG(J)

• Lindell et al.’s (1999)

• Burke, Finkelstein, & Dusig’s (1999) AD index

*)( JWGr

4747474747474747

George & James (1993)• “The key statistical test of the appropriateness of

aggregation to the group level of analysis is that there is within-group agreement on the variable in question. If there is agreement within groups on the theorized group-level variable, then the aggregate may be used in subsequent analyses.”

• … agreement within a group is not conditional on between-groups differences. For example, in a scenario that Yammarino and Markham portray, in which all members in each group have the same moderately high score, both agreement and aggregation may be justified provided that aggregation to the group level was theoretically based. However, there would be no group effect inasmuch as the group means do not vary under these conditions.”

• (p. 799)

48

Why We Need rWG

• Within-Group Agreement – For single items:

– = observed variance of single item

– = theoretical null variance (represents “zero agreement”)

– rWG = “1 - observed variance over expected variance”

)(1 22ExWG j

sr 2

jxs2E

4949494949

Why We Need rWG

Summary:•Under consensus composition models (with isomorphism across levels), within-group agreement is needed to justify aggregation.

•Within-group agreement is even more essential than ICC(1) and ICC(2), both of which depend upon between-group variance.

•Within-group agreement = shared psychological meaning!

•rWG is the key to measuring group-level psychological properties!

505050

rWG(J) for Multi-Item Scales

rWG(J) is NOT the same as rWG!

•rWG = for single items

•rWG(J) = for multiple-item climate scale

)(1 22ExWG j

sr

)()](1[

)](1[2222

22

)(

ExEx

Ex

JWG

jj

j

ssJ

sJr

51


• Within-Group Agreement (James et al., 1984)– For multiple

items:

– J = number of items

– = mean of observed item-level variances


† Can be derived without Spearman-Brown (LeBreton et al., 2005)

)()](1[

)](1[2222

22

)(

ExEx

Ex

JWG

jj

j

ssJ

sJr

2E

2

jxs

52


• Three Issues with James et al.’s (1984) rWG(J) :

1)J = number of items

(is rWG(J) an index of agreement, reliability, or both?)

2) = mean of observed item-level variances

3) = theoretical null variance (represents “zero agreement”)

(addressed by LeBreton & Senter, 2008)

2E

2

jxs

535353535353535353

James et al. (1993)• Describing whether rWG(J) is an index of agreement vs.

reliability:• “Kozlowski and Hattrup are also correct in stating that our

intention was to suggest a measure of agreement, and not consistency [reliability], and that rWG is an estimator of agreement. However, what cannot be done, at least not the way things are presently set up, is to follow Kozlowski and Hattrup's recommendation to sever all ties between interrater reliability and rWG and to treat rWG as strictly a measure of agreement with, in effect, no ties to classic measurement theory. It is not possible to follow this recommendation because rWG is currently derived in terms of classic measurement theory as an interchangeability (agreement) index of interrater reliability.”

• (p. 306)

5454


• Issues with James et al.’s (1984) rWG(J) :


• What happens to rWG(J) as number of items (J) increases?

)()](1[

)](1[2222

22

)(

EXEX

EX

JWG

jj

j

ssJ

sJr

555555




• What happens to rWG(J) as number of items (J) increases?

)()](1[

)](1[2222

22

)(

EXEX

EX

JWG

jj

j

ssJ

sJr

5656



0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 7 11 20

Number of Items (J)

Jam

es e

t al

.'s r

WG

(J)

mean_itemvar = 0.2

mean_itemvar = 0.6

mean_itemvar = 1

mean_itemvar = 1.4

mean_itemvar = 1.8

5757



0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 7 11 20

Number of Items (J)

Jam

es e

t al

.'s r

WG

(J)

mean_itemvar = 0.2

mean_itemvar = 0.6

mean_itemvar = 1

mean_itemvar = 1.4

mean_itemvar = 1.8

2

jxs

J

rWG(J) = .7

5858



– To get a large rWG(J) (James et al., 1984), simply add more items to your scale!!

– Even under near-maximal within-group variance,

[ = 1.8] rWG(J) = .7 when the scale has J = 20 items!2

jxs

59




• What is it?

• First calculate the within-group variance of each item,

• Then average these variances across items,

2

jxs

2

jxs2

jxs

60


• = mean of observed item-level variances

Compare vs. (scale score variance):

Scale score variance => • First calculate mean across items (i.e., scale score), • Then take the within-group variance of scale score,

• is almost always larger than

2

jxs2

jxs2xs

)]1([22 JJss xx j

2xs

x

2

jxs2xs

2xs

61


• Why is almost always larger than scale score variance ?

2

jxs2xs

Psych. Climate

PC Item 1

PC Item 2

PC Item 3

PC Item 4

True Score Variance

Item Unique Variance

62


• Why is almost always larger than scale score variance ?

2

jxs

2

jxs

2xs

2xs

Psych. Climate

PC Item 1

PC Item 2

PC Item 3

PC Item 4

True Score Variance

Item Unique Variance

63


• = mean of observed item-level variances

Compare vs. (scale score variance):

, Scale score variance => zooms in on true, construct-level variance within-groupsvs.

, Mean of observed item-level variances => includes true construct-level variance + item-specific variance

2

jxs2

jxs2xs

)]1([22 JJss xx j

2xs

2

jxs

64




• It would be much clearer to just base within-group agreement on the within-group variance in scale scores, rather than on the average of item-level within-group variances, .

2

jxs

2

jxs2

jxs

65




– E.g., Uniform null distribution– A = number of response options (e.g., A = 5 for a 5-

point Likert scale);

2E

12)1( 22 AEU

212)15( 22 EU

66


• Issues with James et al.’s (1984) rWG(J) : = theoretical null variance

– Can alternatively use a non-uniform expected null variance for rWG(J) (see James et al., 1984; LeBreton & Senter, 2008)

• Normal null dist.

• Skewed null dist.

• Maximum null dist. (Brown & Hauenstein, 2005)

2E

67


• Issues with James et al.’s (1984) rWG(J) : = theoretical null variance

– Can alternatively use an Average Deviation index (AD; average absolute value deviation from mean or median; Burke et al., 1999).

• Less vulnerable to outliers

• Still compared against arbitrary cutoff, AD < A/6

• Still includes item-specific variance (like )

2E

2

jxs

6868686868


Summary:•Whereas rWG is a great index of standardized within-group agreement,

rWG(J) reflects 3 sources of variance: a) within-group variance in psych. climate/latent

construct true scores (“shared meaning”), plusb) item-specific variance (in ), and c) number of items (J).

•It would be better to use an agreement index that homes in on (a) within-group variance in psych. climate/latent construct true scores (“shared psychological meaning”).

2

jxs

6969

Within-Group Agreement

• So what is the alternative?

70


• What if we still want to assess within-group agreement (“shared psychological meaning”) with a multi-item climate scale?

• First, conceptualize the degree of “shared psychological meaning” at the latent theoretical level (James, 1982; James et al., 1988), but use a format similar to rWG:

2

2)lim.(1

E

atecpsychWG

7171


• WG does not increase as you add items to the climate scale (i.e., it is a pure parameter of within-group agreement, not reliability)

2

2)lim.(1

E

atecpsychWG

727272


• How well does each of the following within-group agreement indices estimate WG? (“shared psychological meaning”)

1)James et al. (1984)

2)Lindell et al. (1999)

3)Simple index:

)()](1[

)](1[2222

22

)(

EXEX

EX

JWG

jj

j

ssJ

sJr

)(1 22*)( EXJWG j

sr

)(1 22)( ExWG sr

737373

Within-Group Agreement• Comparison of rWG(J), rWG(J)*, and rWG()

Newman & Sin, 2008

J =5 items, WG = .90

74747474


• Conclusions:

1)All within-group agreement indices are very strongly correlated.

2) rWG(J) can notably overestimate within group agreement, especially when rWG(J) > .7.

3)rWG() seems to offer a closer estimate of within group agreement (slight underestimate)

4)One could also directly estimate WG .

75757575


• How well does each of the following within-group agreement indices estimate WG? (“shared psychological meaning”)

1)When WG = .60:

rWG(J)= .75; rWG(J)*= .38, rWG() = .56

2)When WG = .65:

rWG(J)= .81; rWG(J)*= .46, rWG() = .61

3)When WG = .70:

rWG(J)= .85; rWG(J)*= .53, rWG() = .67

J =5 items, WG = .90

767676



2.Aggregation Bias



777777



2.Aggregation Bias



787878



2.Aggregation Bias



797979



2.Aggregation Bias



808080



2.Aggregation Bias



8181

Thank You Larry!OC

(average)

PC Person #1

PC Person #2

PC Person #3

)(1 22ExWG j

sr

1 measuring group-level psychological properties (a tribute to larry james) daniel a. newman...

Documents

james james

jones james

george james

psychological terms

aggregation bias james

climate construct

aggregate level

strategyteam level