Perspectival Diversity and Consensus Analysis

John Gatewood . . . . . . Lehigh University

John Lowe . . . . . . . Cultural Analysis Group

AAA Meetings, Philadelphia, Dec 5, 2009

Preview

• INTRODUCTORY REMARKS– Problem of “culture-sharing” (and non-sharing)– Basic patterns of inter-informant agreement … adding

“perspectival diversity” to the list

• OUR CURRENT STUDY– Assessing effects of different distributional patterns on

consensus analysis’s key indicators– Some initial findings

• CONCLUSIONS– Methodological “lessons” for researchers– Future directions

INTRODUCTORY REMARKS

Problem of “Culture-Sharing”

• By definition, culture is socially transmitted knowledge; hence, it must be “shared” … but sharing is always a matter of degree

• Hence, two related issues for any given cultural domain:1. How much knowledge is shared?

(the AVERAGE “cultural competence”)

2. How is the knowledge socially distributed?(the DISTRIBUTIONAL PATTERN)

• KEY INSIGHT = assess degree of culture-sharing by examining patterning of inter-informant agreement

Basic Patterns of Agreement

• Boster (1980, 1985) … four basic patterns of agreement (paraphrasing & expanding):

1. UNIFORM agreement – traditional view of culture

2. RANDOM agreement – free variation → no culture

3. EXPERTISE gradient – experts tend to agree with one another whereas non-experts deviate randomly

4. SUBCULTURAL variation – more than one ‘school of thought’a. Competing answer sets – different groups, different truths

b. Complementary knowledge – different groups systematically know different things

• Romney, Weller & Batchelder (1986) … cultural consensus theory & consensus analysis

Perspectival Diversity … a 5th Pattern

• Pilot study of credit union employees (Gatewood & Lowe 2006)

No consensus in sample *+

No identifiable subcultural groups

=> Fish-scale overlappings of partial knowledge… “perspectival diversity”

i.e., social interaction and knowledge among the employees was rather departmentalized … their understandings of ‘credit unions’ reflected what they needed to know to perform their own jobs, not necessarily what might be relevant to other people

_______________________ * Pilot study’s conclusion about “no consensus” turned out to be an artifact of our failure to

counter-balance items in the questionnaire form (see Gatewood & Lowe 2008) … but that’s another story

To generalize, perspectival diversity occurs when…

a) All individuals have limited knowledge with respect to a given domain and to approximately the same degree

b) Each individual’s range of knowledge only partially overlaps with the ranges known by others

And, consistent with this definition, different geometries of perspectival diversity are possible … e.g., circular pattern, linear pattern, taxonomic-hierarchical,

overlapping polygons on a surface, etc.

OUR CURRENT STUDY

RESEARCH QUESTION:• Ceteris paribus, do the different distributional patterns

affect the key indicators of consensus analysis?– As the average knowledge in a sample varies, do different

distributional patterns “show” consensus more readily than other patterns?

– Do some distributional patterns “mask” cultural consensus when other patterns “reveal” it?

• If NO … nothing to worry about [ yippee! ]• If YES distributional patterning has an independent

effect that needs to be taken into account when interpreting results of consensus analyses

ITEM FORMAT:• Counter-balanced Likert-style questions, i.e., 6-point

“strongly agree” to “strongly disagree” response scale … because these are so common in survey research

ANALYSES:• Such data can be analyzed two ways:

– INFORMAL MODEL of consensus analysis … i.e., input to factor analysis is a Resp x Resp correlation matrix (data treated as interval-scale)

– FORMAL MODEL of consensus analysis … i.e., input to factor analysis is a chance-corrected agreement matrix (data treated as nominal-scale, e.g., dichotomized responses)

Research Design … with average knowledge and distributional pattern as manipulated variables

Distributional Patterns

Key Indicators

[ variety of other

measures ]

Ratio of eigenvalues

Mean 1st factor

loading

Number of negative loadings

Uniform-to-Random ? ? ? ?

Expertise ? ? ? ?

Subcultures ? ? ? ?

Perspectival ? ? ? ?

“Theoretical” Predictions

Distributional Pattern Prediction

Uniform-to-Random None … serves as benchmark for other patterns

Expertise Gradient INCREASE consensus indicators

Subcultures DECREASE consensus indicators

Perspectival ???

Implementation

• How to “experimentally manipulate” key parameters for different distributional models while holding others constant ??

… computer simulation to the rescue !

See: Excel “data-generating” file & Excel “findings” file

SOME INITIAL FINDINGS

Key Findings

1. Distributional pattern has an independent effect with respect to consensus indicators– w/r/to RATIO OF EIGENVALUES

( compared to the Uniform-to-Random model )

• Expertise patterns INCREASE the ratio • Subcultural patterns DECREASE the ratio • Perspectival patterns DECREASE the ratio

– w/r/to MEAN 1st FACTOR LOADING• Distributional patterns have little effect on this indicator,

AND consensus analysis estimates actual competence very well … with one exception:

• Expertise (triangular) pattern INFLATES mean competence as well as the ratio of eigenvalues … (because it violates the “homogeneity of items” assumption?)

2. Expertise (rectangular) pattern– The range of expertise about the same average competence

also makes a difference:greater range larger ratio of eigenvalues

3. Subcultural patterns– As expected, systematic differences in sub-group knowledge

undermine consensus:• “By question” sub-groups may still show consensus overall,

with the groups showing up on the 2nd factor• Different “answer keys” just destroy consensus

4. “Formal consensus model” (on dichotomized data) and “informal consensus model” yield very similar results

CONCLUSIONS

“Lessons” for Researchers

1. Since the ratio of eigenvalues is particularly sensitive to the distributional pattern of knowledge, REPORT MORE than just the ratio

– Minimally, include:• Ratio of 1st to 2nd eigenvalues• Mean 1st factor loading (and st.dev. of those loadings)• Number of negative loadings

– And, comparable “guidelines” should be established for evaluating these additional measures: e.g., 0.500 for mean loading; fewer than ~5% negative loadings in sample

– These output statistics are necessary for more meaningful interpretations of one’s data

2. IF your data show a hefty mean 1st factor loading but a low ratio of eigenvalues… DO NOT leap to the conclusion that either (a) subcultures exist or (b) there is free variation in the domain

– You may be dealing with a case of PERSPECTIVAL DIVERSITY … which would warrant further investigation, such as examining the inter-person correlation matrix and the response-profiles of individuals one at a time to see if you can detect a subtle social patterning to who-knows-what

3. Try to formulate questions that are “EQUALLY DIFFICULT” ( and ask lots of questions )

– Violations of Assumption 3 will inflate both the obtained ratio of eigenvalues & the mean 1st factor loading

• e.g., Expertise (triangular) pattern INFLATES both indicators

– So … ex post facto…if you notice that some questions were “much easier” than others, then either: (a) use higher threshold criteria before claiming the data conform to the cultural consensus model, and/or (b) remove the very easy questions and re-analyze

Future Directions

• Developing additional ‘geometries’ of perspectival overlapping

• Analyzing relations between a variety of measures describing the initial Resp x Resp correlation matrix and the key indicators from consensus analysis

• Exploring different instantiations of “guessing” (binomial, truncated-normal, beta distributions)

• Exploring other possible measures from the factor analysis as predictors of culture-sharing, e.g., 1st eigenvalue divided by sample size

Thank you

… and we would be happy to continuetalking with interested folks

after the session

DEIGRAT

14121086420

EIG

EN

RA

T

14

12

10

8

6

4

2

0

DMFLOAD

.8.7.6.5.4.3.2

MF

LO

AD

.8

.7

.6

.5

.4

.3

.2

Scalar data analyzed via Informal Method (vertical axis)VS

Dichotomized data analyzed via Formal Method (horizontal axis)

Ratios of eigenvalues( r = .933 )

Mean 1st factor loadings( r = .969 )

B6

strongly agree

agree

slightly agree

slightly disagree

disagree

strongly disagree

Co

un

t200

100

0

Ck=2,p=.5

6.005.004.003.002.001.00

Co

un

t

20

10

0

POTENTIAL PROBLEM…

Frequency distributions of itemsfrom “real” surveys (top panels)are more graded than our“simulated” data (lower right)

something we’re trying to resolve,but not there yet …

“Real” Survey Item

Simulated Item

B2

agree

slightly agree

slightly disagree

disagree

strongly disagree

Co

un

t

200

100

0

“Real” Survey Item