casro 2013: battle of the scales

Examining Respondent Scale Usage across 10 countries

By Melanie Courtright, Kartik Pashupati,

and Annie Pettit

Battle of the Scales

2

Background

3


Background

Scales are a cornerstone of market research

They’re how we determine that:

• 49% of people like Coca-Cola and 42% of people like Pepsi

• How citizens feel about the government

• Men like watching sports more than women do

• Canadians like Shania Twain more than Brits… or do they??

4


Background

5


So Many Client Questions…

Is there a difference in the reliability of attitudinal scales when using 4-point, 5-point, 7-point, and 10-point scales?

Does excluding a neutral point impact the answers?

Does labeling each point vs. only the end points produce different results?

Can we replicate and extend the results of previous research on the impact of cultural factors on response styles?

Do scales with greater variance (e.g., 7-point and 10-point scales) reduce Extreme Response Style (ERS) compared to lesser variance (e.g., 4-point and 5-point scales)?

Conversely, do scales with greater variance (e.g., 7-point and 10-point scales) produce a greater incidence of Medium Response Style (MRS) compared to lesser variance (e.g., 4-point and 5-point scales)?

http://www.google.com/url?sa=i&rct=j&q=why+do+you+ask+so+many+questions&source=images&cd=&cad=rja&docid=GsB1_zXaIQV64M&tbnid=2K8iByX5lsJIkM:&ved=0CAUQjRw&url=http://www.sodahead.com/fun/do-you-think-i-ask-too-many-questions-or-not-enough/question-2333729/&ei=Wyk4UeTNKdTqqAHL5YGQAQ&bvm=bv.43287494,d.aWM&psig=AFQjCNGwEaJf7NGfzV6JpfnRJllS8fj9Zw&ust=1362721480673327

6


Research Plan

7


Research Plan

7-minute attitudinal survey

Globally relevant topics

Mix of positive and negative wording

Scales with published measures of reliability

Include behavioral statements that should correlate to attitudinal questions

Simultaneously field across ten countries

Age and gender quota sampling by country

Same sample source throughout

Research Now’s proprietary Valued Opinions Panel (VOP)

First study to

Simultaneously compare the effect of multiple response options (4-, 5-, 7- and 10-point scales)

Using a large, census-balanced, multi-country sample

8


Research Plan

Sample

Sizes

4 point

5 point LABELED

5 point UNLABELED

7 point

10 point

TOTAL

Brazil 500 250 250 500 500 2,000

China 500 250 250 500 500 2,000

France 500 250 250 500 500 2,000

Germany 500 250 250 500 500 2,000

India 500 250 250 500 500 2,000

Japan 500 250 250 500 500 2,000

Mexico 500 250 250 500 500 2,000

Russia 500 250 250 500 500 2,000

UK 500 250 250 500 500 2,000

US 500 250 250 500 500 2,000

TOTAL 5,000 2,500 2,500 5,000 5,000 19,886

Fieldwork conducted December 2012 and January 2013

All five scale options tested in each of ten countries

Note: For ease of reading, sample sizes have been rounded up or down by no more than 11.

9


Analysis Plan

Three indices:

Extreme Response Style Index (ERSI)

– Respondents who answered either extreme of scale were assigned a score of 1. Otherwise, they were assigned zero.

Acquiescence Response Style Index (ARSI)

– Respondents who strongly agreed with an item were assigned a score of 1. Otherwise, they were assigned zero.

Medium Response Style Index (MRSI)

– On scales with an odd number of options (i.e., 5 point, 7 point), respondents who answered exactly in the middle were assigned a medium response score of 1. Otherwise, they were assigned zero.

Possible summary values ranged from 0.0 to 1.

10


Use of Scale Results

Male Female

ERS .43 .43

ARS .32 .33

MRS .28 .28

Hypothesis 1: There will be no meaningful differences in ERS, MRS or ARS indices between male and female respondents.

Result 1 Confirmed: Men and women do not differ in their response patterns.

11



Hypothesis 2: There will be significant but not systematic differences in ERS, ARS and MRS Indices across the different age groups.

Results 2 partially confirmed: There were significant differences but the differences were systematic. ERS and MRS gradually increased with age until 55-64, and then declined. The pattern for ARS (yea-saying) was reversed.

.41 .41.43 .43 .44 .44

.41 .41

.46

.31.32

.33 .33.32 .32

.30.29 .28

.27 .27 .28.28 .29 .29 .28

.29

.25

.20

.25

.30

.35

.40

.45

.50

15-17 18-24 25-34 35-44 45-54 55-64 65-74 75-84 85+

ERS (Extreme)

ARS (Acquiescent)

MRS (Medium)

12



Hypothesis 3: There will be significant differences in ERS, ARS, and MRS across the different countries.

Results 3 confirmed: Respondents from Brazil and Mexico have the highest tendency to give extreme responses. Respondents from Japan have a significantly lower ERS Index, and significantly higher MRS (and ARS).

.53 .53.48

.45.41 .41 .40 .40

.37.31

.27 .26 .26 .27 .29 .28 .28 .29 .29

.33

.00

.10

.20

.30

.40

.50

.60

▌ERS (Extreme)

MRS (Acquiescent)

13



Hypothesis 4: Individualism (Hofstede) will correlate positively with ERS and negatively with MRS. Individualism will equate to stronger, and therefore more extreme, opinions.

Results 4 not confirmed: Differences could not be attributed to individualism. Analysis actually showed a negative correlation with ERS.

.49

.41.39

.38

.31

.28.27

.29 .28

.20

.25

.30

.35

.40

.45

.50

.55

Low Medium HighIndividualism

ERS (Extreme)

ARS (Acquiescent)

MRS (Medium)

14



Hypothesis 5: Respondents in countries that are higher in masculinity (Hofstede) would exhibit higher ERS.

Results 5 not confirmed: While masculinity did affect the differences, the results were not in the expected direction. Countries with lower masculinity demonstrated higher ERS indices.

.46

.41 .42

.34

.32.31

.28 .28.30

.25

.30

.35

.40

.45

.50

Low Medium High

Masculinity

ERS (Extreme)

ARS (Acquiescent)

MRS (Medium)

15



16


Number of Scale Points Results

Hypothesis 6: The number of scale points and scale labeling will affect ERS, MRS, and ARS.

Results 6 confirmed: 7- and 10-point scales saw fewer medium responses. ERS, MRS, and ARS were all lower for the 5-point labeled scale versus the unlabeled scale.

.45

.41

.45

.42.41

.34.32

.33.32 .31

.33.31

.33

.27

.20

.15

.20

.25

.30

.35

.40

.45

.50

4 point 5 point labeled

5 point unlabeled

7 point 10 point

ERS (Extreme)

ARS (Acquiescent)

MRS (Medium)

17


Number of Scale Points Results

Hypothesis 7: The number of scale points and scale labeling will have an impact on scale reliability.

Results 7 Not confirmed: There is no significant variation in the reliability of scales by number nor labeling of scale points.

Scale

# of Total Items

# of ReverseItems

Cronbach's alpha

4 point

5 point labeled

5 point unlabel

ed7

point10

point

Health Environment Sensitivity 8 1 0.8 0.8 0.8 0.8 0.8

Personal Health Responsibility 8 2 0.7 0.6 0.6 0.6 0.7

Motorcycle Helmet Mandate 2 1 0.7 0.7 0.7 0.7 0.7

Attitude toward helping others (AHO)

4 0 0.9 0.85 0.9 0.9 0.9

Material Values Scale (MVS 9) 9 2 0.8 0.75 0.8 0.8 0.8

Attitude toward Advertising in General (AAG)

7 4 0.7 0.7 0.7 0.7 0.7

Online privacy concern 2 0 0.5 0.5 0.5 0.6 0.55

Lie acceptability scale 8 4 0.8 0.8 0.8 0.8 0.8

Note: For ease of reading, alphas were rounded. See the paper for precise values.

18


Summary Findings

ERS, MRS, and ARS do not differ by gender, but do differ by age

Response styles vary by country– India, Mexico, Russia and Brazil are similar

– The US and UK are similar, as are France and Germany.

– Respondents from Japan are unique in terms of lower extreme and higher medium response styles

Reasons for country differences are not yet isolated– Need more research and

– Need scale norms that are available on a multi-country basis.

Varying the number of response options – Does affect MRS

– Does NOT impact scale reliability or ERS

Scale labeling did not impact scale reliability, but did impact ERS, MRS, and ARS

19


Closing Thoughts

READ THE PAPER!

20


Closing Thoughts

21


Closing Thoughts

What about mobile?

If number of options and labeling do impact results and screen space is a luxury on mobile devices…

22


Closing Thoughts

What about social media? HideousDisgustingAbhor

If number of options and labeling do impact results, what is the complementary impact on textual data…

HuhDunnoWhatevs

GoodNiceCool

AwesomeWickedBomb

CrapYuckDumb

23


More to Come!

casro 2013: battle of the scales

Education