1

Protocol & preliminary results from a Protocol & preliminary results from a transnational cognitive interviewing study: transnational cognitive interviewing study: Ensuring valid cross-cultural comparisonsEnsuring valid cross-cultural comparisons

INSTITUTO NACIONAL DE SALUD PÚBLICA MÉXICO

James F. Thrasher, Anne C. K. Quah, Ron Borland, Rahmat Awang,

Maizurah Omar, Buppha Sirirassamee, Marcelo Boado, Ashlee

Watts, Ana Dorantes Alonso

TTURC Planning Meeting,

Portland, OR, USAMarch 2, 2008

2

Purpose of the studyPurpose of the study

• Develop a protocol for assessing and reducing Develop a protocol for assessing and reducing systematic response error in multi-lingual and systematic response error in multi-lingual and multi-cultural surveysmulti-cultural surveys

• Focus on existing ITC survey questions:Focus on existing ITC survey questions:– Assess equivalence of comprehension and Assess equivalence of comprehension and

response error across 6 countries, 4 languages;response error across 6 countries, 4 languages;– Recommend alterations to questions to increase Recommend alterations to questions to increase

equivalence in later studies.equivalence in later studies.

• Increase data quality Increase data quality more confidence in more confidence in results from comparative data analysesresults from comparative data analyses

3

Questionnaire protocolQuestionnaire protocol

1. Ask participant survey question.

2. Passive recording of behaviors that indicate comprehension problems (i.e., behavioral coding*).

3. Ask follow-up questions that aim to assess response error across the steps involved in cognitive processing of survey questions (i.e., cognitive interviewing).

*Johnson, TP et al (2006). Cultural variability in the comprehension of health survey *Johnson, TP et al (2006). Cultural variability in the comprehension of health survey questions. questions. Annals of EpidemiologyAnnals of Epidemiology, , 16,16, 661-668. 661-668.

4

Study StepsStudy Steps

Identify candidate questions

Determine likely

biases

Develop probes to

assess biases

Develop translator

guide

Translate protocol

Train interviewers

Pilot protocol

Review pilot data

Adjust protocol

Implement protocol

Enter data

Translate responses to English

Review quantitative

data

Identify themes in qualitative

data

Determine evidence for systematic

bias

Suggest changes

5

Sample Characteristics*Sample Characteristics*

USA Australia Uruguay Mexico Malaysia Thailand

Male 65% 50% 50% 65% 100% 80%

Age 36 36 40 38 31 39

<HS 5% 30% 30% 20% 10% 85%

Edu HS 65% 30% 40% 55% 45% 10%

Uni 30% 40% 30% 25% 30% 5%Daily smoker 85% 90% 80% 85% 90% 85%

Average cigs/day 23 15 14 15 13 13

*n=20 for each country

6

Nicotine in cigarettes is the chemical Nicotine in cigarettes is the chemical that causes most of the cancer: Falsethat causes most of the cancer: False

60%65%

20% 20%25%

5%0%

20%

40%

60%

80%

100%

US Aust Urgy Mex Mal Thai

7

Behavioral codes indicate some Behavioral codes indicate some systematic biassystematic bias

Did the participant… US AUST URGY MEX MAL THAI p…need you to repeat the question?

1 1 0 0 1 1 1.00

…have difficulty with response options?

5 3 0 0 3 0 0.008

…ask for clarification or qualify answer?

4 1 0 1 0 0 0.047

Total 10 5 0 1 4 1

8

Can you tell me what you know Can you tell me what you know about nicotine?about nicotine?

Theme USA AUST URGY MEX MAL THAI

Addictive 80% 60% 20% 20% 15% 35%

Harmful 15% 15% 30% 30% 35% 30%

Chemical-drug 0% 30% 0% 15% 20% 20%

Stimulant 5% 5% 0% 10% 0% 0%

Discoloration 10% 5% 10% 0% 5% 5%

Don’t Know 0% 15% 10% 15% 5% 20%Haven’t heard of nicotine 0% 0% 20% 20% 5% 25%

9

Conclusions & RecommendationConclusions & Recommendation• Behavioral codes systematic bias

– Awkward wording?

• Acquiescence bias?– “don’t know” or haven’t heard of nicotine

• Order effect after health outcome questions?

Recommendation:

Clarify question intent:– Assess knowledge? Why does this matter?

Could first ask if they have heard of nicotine– ITC-SEA W3: Cigarettes contain nicotine

10

Tobacco is addictive: Tobacco is addictive: ““agree” or “strongly agree”agree” or “strongly agree”

80%

100%90% 90% 85% 85%

0%

20%

40%

60%

80%

100%

US Aust Urgy Mex Mal Thai

11

Behavioral Codes indicate no Behavioral Codes indicate no problem with “Tobacco is addictive”problem with “Tobacco is addictive”

Did the participant… USA AUST URGY MEX MAL THAI p

…need you to repeat the question?

0 0 0 0 0 1 1.00

…have difficulty with response options?

1 1 0 0 1 1 0.87

…ask for clarification or qualify answer?

1 0 0 0 0 0 1.00

Total 2 1 0 0 1 2

12

Other things that are addictiveOther things that are addictiveCategory USA AUST URG MEX MAL THAI

Alcohol 70% 70% 70% 50% 15% 15%

Other drinks 5% 5% 25% 25% 10% 0%

Illegal drugs 80% 70% 70% 55% 60% 80%

Legal drugs 35% 35% 0% 10% 25% 0%

Inhalants 0% 0% 0% 5% 20% 15%

Food 40% 20% 25% 5% 0% 0%

Sex 15% 10% 0% 20% 5% 0%

Gambling 5% 15% 10% 5% 0% 0%

Other 35% 15% 15% 0% 20% 5%

DK 5% 0% 5% 25% 15% 0%

13

What does it mean to say that What does it mean to say that something is addictive?something is addictive?

Themes USA AUST URG MEX MAL THAI

Control-General 50% 50% 55% 55% 45% 30%

Physiological 40% 25% 5% 15% 0% 0%

Control-Psych 10% 10% 0% 0% 30% 0%

Control-Quit 0% 10% 5% 10% 10% 15%

Freq-Quantity 5% 50% 40% 15% 20% 30%

Danger 5% 0% 0% 0% 0% 35%

Pleasure 5% 0% 5% 0% 5% 5%

It's got a grip of you and you can't let go. It's a must have,

you've gotta have it. 39 year old Australian male

The body gets tolerance for it and the body won't function

normally with out it.27 year old US female

If he doesn’t get that thing, he cannot do other things…His mind is always imagining that thing…if he doesn’t get it, he cannot sit still.

26 year old Malaysian male

14

Conclusions & RecommendationsConclusions & Recommendations

• Behavioral coding no clear bias

• Connotations of and associations with addiction appear to differ across countries

Recommendation:

• Ask question(s) that focus specifically on:– Control over smoking (e.g., Smokers cannot

go long without smoking, even when they don’t want to smoke)

– Control over quitting (e.g., Completely stopping smoking is extremely difficult)

15

Lights are less harmful than regular Lights are less harmful than regular cigarettes: “Disagree”/”Strongly disagree”cigarettes: “Disagree”/”Strongly disagree”

55%65%

35%

20%

35% 35%

0%

20%

40%

60%

80%

100%

US Aust Urgy Mex Mal Thai

16

Behavioral codes indicate no Behavioral codes indicate no systematic bias systematic bias

Did the participant… USA AUST URGY MEX MAL THAI p

…need you to repeat the question?

2 1 0 1 2 3 .377

…have difficulty with response options?

1 1 0 1 2 3 .343

…ask for clarification or qualify answer?

0 1 0 0 2 1 .279

Total 3 1 0 2 6 7

17

Other differences beween light and Other differences beween light and regular cigarettes?regular cigarettes?

themes USA AUST URGY MEX MAL THAI

Nicotine & Tar 65% 60% 5% 5% 15% 10%

Other Chemicals 30% 30% 15% 5% 15% 15%Filter & Tobacco Qty 35% 5% 5% 0% 5% 10%

Taste & Flavor 20% 15% 50% 30% 50% 60%Other sensory effects 10% 5% 0% 5% 35% 50%

other (danger, packaging) 10% 15% 20% 0% 15% 0%

18

Conclusions & RecommendationsConclusions & Recommendations• Behavioral coding no systematic bias• Perceptions of light vs. regular appear to differ

across countries– Focus on product characteristics in western

countries– Focus on sensory effects of product use in non-

western countries

Recommendation• Be aware of the possibility of the salience of

these characteristics when developing new questions about product risks and effects.

19

Noticed warning labels “often” or “very Noticed warning labels “often” or “very often” in last monthoften” in last month

35%

80%

65%75%

55% 60%

0%

20%

40%

60%

80%

100%

US Aust Urgy Mex Mal Thai

20

Behavioral codes indicate some Behavioral codes indicate some evidence of systematic biasevidence of systematic bias

Did the participant… USA AUST URGY MEX MAL THAI p

…need you to repeat the question?

0 0 3 1 1 1 .377

…have difficulty with response options?

3 1 0 1 1 0 .377

…ask for clarification or qualify answer?

4 4 0 1 0 0 0.011

Total 3 1 0 2 6 7

21

Follow up probes indicate Follow up probes indicate incorrect reference period and incorrect reference period and issues with the term “notice” issues with the term “notice”

USA AUST URGY MEX MAL THAI P

Correct reference period identified (last month)

15% 25% 15% 10% 5% 5% .437

Can you notice without looking closely?

55% 80% 70% 65% 35% 75% .049

Can you notice if you know what label says?

75% 75% 55% 90% 60% 80% 0.146

22

Conclusions & RecommendationsConclusions & Recommendations

• Behavioral coding indicates systematic bias (*problems with US & Australia)

• Consistently incorrect identification of reference period for the question

• Some indications of systematic differences in meanings of “notice”

Recommendation: – Ask about last time when “noticed”– Review terms used for translation of “notice”

23

In last 6 months, noticed ads on…In last 6 months, noticed ads on…Measure USA AUST URG MEX MAL THAI ALL p*

TV 15% 5% 25% 35% 30% 5% 19% 0.054

Dif to remember 25% 10% 50% 15% 60% 40% 33% 0.003

Radio 15% 0% 25% 20% 10% 0% 12% 0.041

Dif to remember 30% 5% 45% 25% 65% 50% 37% <.001

Papers/Mags 75% 0% 40% 50% 30% 10% 34% <.001

Dif to remember 25% 10% 30% 15% 45% 40% 28% 0.095

Billboards 60% 0% 60% 95% 35% 10% 43% <.001

Dif to remember 20% 25% 45% 35% 50% 40% 36% 0.332

In stores 95% 45% 70% 90% 70% 30% 67% <.001

Dif to remember 15% 35% 30% 20% 25% 35% 27% 0.660

Correct period (last 6 months)

0% 20% 30% 10% 5% 15% 5% 0.067

24

Conclusions & RecommendationsConclusions & Recommendations• Behavioral coding no systematic bias• Across channels, participants in all countries

found it difficult to remember whether they had seen an ad– Some indication of systematic bias in difficulty of

remembering exposure for TV and radio.

• Consistently incorrect identification of reference period for noticing

Recommendation:– Ask about last time “saw”, with options for “never”

and for “more than 6 months ago”

25

General ConclusionsGeneral Conclusions

• Cognitive interviewing and behavioral coding techniques can be implemented across linguistic and cultural settings

• Overcoming challenges of assessing equivalence across languages demands collaboration from multilingual partners.

• Comprehension of some questions appeared generally comparable whereas other questions appear to generate response error and need to be adjusted.

26

General RecommendationsGeneral Recommendations

• Any changes to harmonize item comprehension should be verified through another round of cognitive interviews.

• Interviewers should be better trained to clarify ambiguous or repetitive responses.

27

General RecommendationsGeneral Recommendations• Prepare translation notes (i.e., definitions of the

concept being measured & question intent) for new questions.

• Choose source language questions that use simple, translatable language– “Notice” vs. “saw”

• To avoid augmenting “translation loss”, questions should originate from one source.– NOT: English Spanish Malay

• Equivalence of comprehension may demand flexibility in “source” language item

28

General RecommendationsGeneral Recommendations

Systematic documentation of the evolution, translation, and meanings of ITC survey questions, key terms, and phrasing across countries. Translator’s notes May include possible translation equivalents in

the target languages Provides record of original survey item

wording Allows future researchers to understand

intentions behind survey items and any changes to them

International Tobacco Control Policy Evaluation Projecthttp: / / www.itcproject.org

http: / / www.roswelltturc.org

 

ITC Project Research Support

Core support provided by the U.S. National Cancer Institute to the Roswell Park TTURC

(P50 CA111236)

Additional major funding provided by the Canadian Institutes of

Health Research

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

30

Study StepsStudy Steps

Identify candidate questions

Determine likely

biases

Develop probes to

assess biases

Develop translator

guide

Translate protocol

Train interviewers

Pilot protocol

Review pilot data

Adjust protocol

Implement protocol

Enter data

Translate responses to English

Review quantitative

data

Identify themes in qualitative

data

Determine evidence for systematic

bias

Suggest changes