1

Using Qualitative Methods to Develop Culturally Adequate Surveys

Anna Nápoles-Springer, Ph.D.University of California San Francisco

Center for Aging in Diverse Communities

2

Overview

Integrating quantitative and qualitative methods to develop surveys

Focus groups Cognitive interview methods Using results of cognitive interviews to make

decisions about item revisions/deletions

3

Integrating Quantitative and Qualitative Methods

In diverse populations, qualitative work is necessary in addition to more traditional quantitative studies

Prior to quantitative: to develop concepts, items appropriate to culture

After quantitative: – to help identify reasons for items not performing well

quantitatively– to explore possible explanations for unexpected results

4

What are Qualitative Methods?

Data consist of words, not numbers Richly descriptive, open-ended Focus on inductive analytic approaches Many types: ethnography, participant-

observation, direct observation, interviews, focus groups

5

When are Qualitative Methods Useful?

To understand the meaning of participants’ events, situations, and actions

To understand contextual influences on participants’ actions

To identify unanticipated phenomena and influences (e.g., exploratory studies to design questionnaires and identify variables for study)

To understand the processes underlying observed relationships between variables

6

Combining Qualitative and Quantitative Methods

Qualitative Methods to Develop Quantitative Tools

Qualitative Methods to Explain Quantitative Results

Qualitative ResultsQuantitative

Quantitative Results

Qualitative

7

Combining Qualitative and Quantitative Methods cont.

Quantitative Methods to Expand on Qualitative Study

Quantitative and Qualitative Methods Equal

Qualitative Results

Quantitative

Qualitative Results Quantitative

(Tasjakkori A. Sage Publications, Inc. 1998)

8

Focus Groups can Identify Right Questions to Ask in Expanded Surveys

1996 US Census Bureau –little difference between rates of work force participation of low-income families of children with special needs and those with no special needs

Might conclude child care systems adequately meeting needs of special needs families-able to work

Focus groups-enormous strain, scarce programs, working nights, losing multiple jobs

(Sunhua L. 2002)

9

From Focus Groups to InterventionThe Subjective Culture of Smoking

Purposeful sampling stratified by comparison groups (by ethnicity)

Examine similarities and differences in attitudes, beliefs and behaviors (Triandis)

Antecedents and consequences of smoking

10

Focus Groups to Identify Themes

Themes Extracted from Focus Groups Family concerns - 2nd hand smoke Heightened health issues Importance of appearance and

interpersonal relations - simpatía Habitual use less important Addiction less of a concern

11

Integrate Results to Develop Intervention: Programa Latino Para Dejar de Fumar

Guia Para dejar de Fumar--self-help guide

Electronic media in Spanish

Emphasis on family and collective

orientation of culture

Immediate effects of smoking

Health effects in different contexts

12

Develop Structured Questionnaire

Scales developed based on themes/lit/theory 17 antecedents to smoking (habitual activities, social

activities, emotional states) 15 reasons to quit (family, appearance, health) 3 reasons to keep smoking (weight, nervousness,

concentration)

13

Triangulation of Focus Group Themes: Latinos’ Reasons to Quit

OR 95% CI

Criticized by family1.93 (1.26, 2.98)

Burn clothes 1.57 (1.02, 2.42)

Children's' health 1.67 (1.08, 2.57)

Bad breath 2.07 (1.40, 3.06)

Family pressure 1.69 (1.10, 2.60)

Good example 1.83 (1.21, 2.76)

Not to gain weight 0.38 (0.24, 0.59)

14

Integrating Qualitative and Quantitative Methods to Develop Surveys

Goal: develop survey items that mean the same thing across groups

Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns

Qualitative methods uncover cognitive processes respondents use to answer questions; disclose reasons for ethnic difference in response patterns

Iterative quantitative and qualitative research methods

15

Integrating Qualitative and Quantitative Methods to Develop Surveys

Example: – Item: In the past 12 months, how many times have

you been to see a doctor?– Test-retest reliability was .56-.65 for Chinese and

Vietnamese compared to .90 in Whites– Interviewer debriefing revealed problems with

interpretation of what constituted a visit to a doctor among Chinese and Vietnamese

Pasick RJ, et al. 2001 Public Health Rep;116:223-43

16

Open-ended Interviews to Develop Structured Questionnaire

Example: Constructing attitudinal scales Use open-ended interviews (individual or

focus groups) to obtain statements on topic of interest to identify relevant concepts

Organize statements into “item pool” Pretest closed-ended items Modify items based on pretest and administer

in structured interview

17

Goals of Purposeful Sampling

Achieve representative, typical settings, individuals, or activities

Adequately capture heterogeneity, sample for broadest range of variation rather than typical individuals

Set up contrasts to examine differences between settings or individuals

(Maxwell JA. Sage Publications, Inc., 1996)

18

Sampling in Qualitative Research

Quantitative research-2 types of sampling:

– Probability

– Convenience Qualitative research-3rd type of sampling:

– Purposeful»Deliberately select settings, persons or events

to best answer research questions

19

Qualitative Methods to Develop Surveys

Focus Groups– Open-ended guided group discussion with probing of

responses

Cognitive interviews– Open-ended probes to assess how items are interpreted and

adequacy of response choices

– Respondent is expert

Expert panels provide input on relevant concepts Interviewer debriefing

20

Focus Groups

Group discussion led by experienced moderator usually 1.5 - 2 hours

Purposeful sampling of 6-10 homogenous participants per group

Use open-ended questions with follow-up probes for clarification, usually covering 10-12 topics

Participants stimulate comments of others Audio-record and transcribe discussion

21

Focus Groups-Logistics

Moderator skills: listening, communication, negotiation, cultural similarity to participants

Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food)

Convenient and hospitable community setting In-person recruitment with telephone and mail

follow-up

22

Example: The Interpersonal Processes of Care

Examined similarities and differences in attitudes, beliefs and behaviors

Sampling stratified by comparison groups: Whites, African Americans and Latinos

Probed on aspects of communication, decision making and interpersonal style of physician

Used information to design structured questionnaire

23

Identify Themes: How Interpersonal Processes of Care Might Affect Health Outcomes

Poor communication - doctors ...– Use unclear language, jargon

– don’t listen carefully

– don’t make sure that the patient understands Decision-making did not involve patients Cultural insensitivity

– patients’ beliefs, preferences not taken into account

24

Identify Domains of Communication

General clarity Elicitation of and responsiveness to patient

problems and concerns Explanations and information

– about condition, prognosis

– about processes of care

– about what to do for themselves

25

Define Domains: Definition of General Clarity in Communication

Communicates clearly even when patient does not speak English well– Uses interpreters if needed

Easy to understand Speaks clearly and slowly enough Explains things well Makes sure patients understand

26

Develop Items Based on Definition: General Clarity

Over the past 12 months, how often did doctors…

• use words that were hard to understand• speak too fast• explain things well• asked if you understood what they told you

27

Develop Item Pool and Pretest

Develop 5 – 6 items per domain or content area of interest

Will end up with 3 – 4 that work Use pre-testing to identify best candidate items for

structured survey Types of pretests

– General debriefing on comprehension, flow

– Cognitive interviews

28

Cognitive Interviews

Derived from social and cognitive psychology to explore processes respondents use to answer survey questions

Diagnostic tool for pretesting survey questions

29

Cognitive Interviews Examine 4 Steps in Answering Questions

Interpret and understand the question– as intended by the researchers

Retrieve the information – various schemas used to access memory

Judgment formation - formulate an answer– calculate or judge the correct information

Edit response - decide what to report– is answer embarrassing, socially undesirable?

30

Purpose of Cognitive Interviews

To learn .. .. if respondents understand words and phrases as

intended (meaning) .. about the process of answering the questions .. whether items are unacceptable .. about the usefulness of response choices

– whether response choices are adequate– how they use the response choices

31

Two Types of Cognitive Interviews

Think aloud interviews– Respondent asked to think aloud as they answer

question Probe interviews

– Interviewer asks specific questions to elicit how respondent answered question

Think alouds - greater respondent burden

32

Steps Involved in Cognitive Interview Pretesting

Decide on a final item pool Develop interview “probes” for a subset of items Translate interview and probe questions Recruit sample for cognitive interviews Conduct cognitive interviews Analyze results Revise items based on results

33

Reducing Item Pool

Reduce items from item pool (subjective process among research team)

Criteria:– maintain breadth of concept, multiple items/concept– reduce redundancy (but OK to test alternate

versions of items)– eliminate items that are unclear, complex, require

high verbal skills, lack face validity, or will not translate well

34

Writing Probe Questions

From reduced set of items, select potentially problematic items for pretesting

Write open-ended probe questions– worded to reveal if suspected problem with

item is present

35

Types of Cognitive Probes

General– Tell me what you were thinking when you

answered that question– How easy or difficult was it to answer that

question? Why? Explore meaning of word or phrase

– I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?

36

Types of Cognitive Probes (cont.)

Retrieval– How did you remember that?

Judgment– Why did you pick that number for your answer?

Response– Do you think that most people answer this question

honestly?Collins D. Quality of Life Research 2003. 12:229-38.

37

Types of Cognitive Probes (cont.)

Redundancy– How is the phrase “give you advice about your diet

and exercise” different from the phrase “talk to you about your diet and exercise”?

Acceptability– When I asked you how often you felt discriminated

against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?

38

Types of Cognitive Probes (cont.)

Cultural appropriateness– I asked you how often doctors asked you about your

health beliefs? What does the term ‘health beliefs’ mean to you?

39

Sampling for Cognitive Interviews

Aim is not representative samples Include respondents from major segments of

population to be sampled for main survey Approximately 5-30 interviews/group

40

Recruiting for Cognitive Interviews

Explain how their help fits into larger study, process of creating questions

Explain their role clearly: – “help us learn how to ask better questions”– “help us make questions clearer for others” – “help us to identify problems with questions”

Pay subjects – interview is demanding If survey is long, pretest different sections on

different subjects

41

Conducting Cognitive Interviews

Individual face-to-face, in-depth interviews Standard administration of closed-ended

items Administer probe questions at the end Typically 1 to 1.5 hr interview Each interview audiotaped and transcribed

42

Analysis of Cognitive Interviews

Data preparation– Data are transcripts– Translate transcripts if necessary back to

English Two approaches

– Behavioral coding– Use qualitative data analysis software to

perform content analysis

43

Behavioral Coding

Review transcripts to identify problems with items– respondent and interviewer “problem” behaviors

Assign “problem behavior” codes to each item using pre-determined categories

Summarize results for each item:– proportion of interviews with each problematic

behavior for each item» e.g., 7/48 respondents requested clarification of

item 10

44

Examples of Behavioral Codes

Interviewer behavior Hard to read - interviewer experiences

difficulty reading question

Respondent behavior Repeat question - respondent asks to have the

question repeated

45

Analysis of Cognitive Interviews (cont.)

Using qualitative analysis software, review all dialogue that ensued during standard administration of closed-ended items and open-ended probes– can reveal source of problems– can help in deciding whether to keep, modify or

drop items Allows you to examine dialogue for each item

– within groups– across groups/languages

46

Example of Results: Interpreting the Question

Original item: How satisfied are you with the amount of stress or worries in your life?

Probe: Did you answer this question in terms of stress, worries, or both?

AA - tended to answer in terms of worries No term in Spanish for “stress” Item revised

– How satisfied are you with the amount of worries in your life?Warnecke RB, etal. JNCI Monographs No. 20, 1996; 29-38.

47

Example of Results: Unclear Phrase

Original item: Have you had any medical tests or procedures in past year?– 26% of respondents asked for clarification

Probe: What did you include as medical tests or procedures?

Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures)

Item revised to include examples:– Have you had any medical tests or procedures, such as blood

tests, x-rays, or cancer screening tests?

48

Example of Results: Information Retrieval

Original item: How satisfied are you with your ability to travel on vacations?

Probe: What do you think we meant by vacation? Neither concept of “vacation” nor “travel for pleasure”

had relevance to lifestyle of AA and MA; travel for family reasons

Dropped itemWarnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

49

Example of Results: Response Sets

Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar)

Problem: No direct Spanish translation for “unimportant”– bilinguals understood translation “sin importancia” as

equivalent to unimportant– Spanish monolinguals did not understand it as the

negative pole of the scale Solution: Changed English to unipolar scale “not at all

important” to “very important” to correspond to best Spanish

Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38

50

Example of Results: Redundancy of Items

Original items:– How often did doctors explain what was causing

your health problem?– How often did doctors explain your diagnosis?

Probe: What do the words health problem and diagnosis mean to you?

Respondents viewed them as the same Some respondents did not know meaning of diagnosis Dropped item with word diagnosis

51

Summary

Greatest problems are with question interpretation– usually due to need to write at lower level of verbal

comprehension

Sometimes English concepts not meaningful in other languages or are irrelevant for certain groups

Need to pretest response sets– Ethnic groups may use them differently

52

Advantages of Cognitive Interviews

Complement other survey field test methods (e.g. where problems are identified by missing or truncated answers)

Identifies where responses might be affected by cultural or group experiences

Suggests ways to revise items, response sets Improves validity of questions

53

Disadvantages of Cognitive Interviews

Flags problems, but significance of the problem remains matter of subjective judgment– at which point need to revise or drop items

Based on small number of respondents Time and labor intensive

54

Conclusions:Integrating Quantitative and Qualitative Methods

Focus groups, cognitive interviews complement and are necessary component of research in diverse groups

To identify relevant themes, constructs, items In context of developing theory (research

questions), structured surveys, interventions