1

Qualitative Methods to Develop and Pretest Surveys in Diverse Groups

Anna Nápoles-Springer, Ph.D.University of California San Francisco

Center for Aging in Diverse Communities

2

Overview

Integrating quantitative and qualitative methods to develop surveys

Focus groups Cognitive interview methods Using results of cognitive interviews to

make decisions about item revisions/deletions

3

When are Qualitative Methods Used?

Open-ended interviews typically used in new areas of study

Useful for in-depth knowledge about issues, especially in less studied groups

Especially critical in cross-cultural studies due to lack of information

4

What are Qualitative Methods?

Data consist of words, not numbers Richly descriptive, open-ended Focus on inductive analytic approaches Many types: ethnography, participant-

observation, direct observation, focus groups, in-depth interviews

5

When are Qualitative Methods Useful?

To understand the meaning of participants’ events, situations, and actions

To understand contextual influences on participants’ actions

To identify unanticipated phenomena and influences (e.g., exploratory studies to design questionnaires and identify variables for study)

To understand the processes underlying observed relationships between variables

6

Meaning of Events

Richness of Qualitative Data

Contextual Influences on

Behavior

Processes Underlying Observed

Relationships

7

Integrating Qualitative and Quantitative Methods to Develop Surveys

Goal: develop survey items that mean the same thing across groups

Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns

Qualitative methods uncover cognitive processes used to answer questions; conceptual relevance

Iterative quantitative and qualitative methods

8

Combining Qualitative and Quantitative Methods

Prior to quantitative: to develop concepts, framework, hypotheses, and content for structured survey items or interventions

After quantitative: – to help identify reasons for survey items not

performing well quantitatively– to explore possible explanations for

unexpected results

9

Applications

To develop: Concepts and their definitions Measures of these concepts Pre-test measures

10

Qualitative Methods to Develop Surveys

Focus Groups– Open-ended guided group discussion with probing of

responses

Cognitive interviews– Open-ended probes to assess how items are interpreted

and adequacy of response choices

– Respondent is expert

Expert panels provide input on relevant concepts Interviewer debriefing

11

Focus Groups to Develop Surveys

To assess the universality of concepts and measures – funnel approach, concept to term

Facilitate culturally sensitive models (e.g., how being AA shapes experiences of discrimination)

Use phrases from transcripts to develop items Pretest clarity and relevance of existing items

12

Focus Groups

Open-ended guided group discussion with probing of responses

Led by experienced moderator; usually 1 - 2 hours

Purposeful sampling of 6-10 homogenous participants per group

Participants stimulate comments of others Audio-record and transcribe discussion

13

Focus Groups-Logistics

Moderator skills: listening, communication, negotiation, cultural similarity to participants

Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food)

Convenient and hospitable community setting In-person recruitment works best with

telephone/mail reminders

14

Focus Groups

Advantages– group stimulates fruitful discussion

– spontaneity leads to discovery of new issues and factors

Disadvantages: scheduling, skilled moderator, group setting may be inappropriate

15

Sampling in Qualitative Research

Purposive, nonprobabilistic Deliberately select settings, persons or

events to best answer research questions Set up contrasts to examine differences

between settings or individuals

16

Example: Measures of Cultural Sensitivity of Clinicians

Conducted 19 focus groups with 163 participants

61 African Americans, 45 Latinos and 55 non-Latino Whites

Asked about the influence of cultural factors on the quality of encounters

17

Focus Group Questions

What does the word “culture” mean to you? What do or don’t your doctors understand about

your culture or health beliefs that might affect your visits?

Embedded in more general discussion of communication and decision making in medical encounters

18

Identify Themes (Codes): How Culture Might Affect Health Care and Outcomes

CAM Discrimination Doctor Culture Ethnicity of MD Family Immigration

Language Modesty Nutrition Patient

submissiveness Spirituality

19

Define Domains of Each Cultural Sensitivity Domain: CAM

Definition: MD’s knowledge and acceptance of non-Western, non-biomedical, holistic approaches to health or healing

“When I told her I was on estrogen, it was refreshing to hear suggestions about alternative types of herbal treatments.”

AA woman > 50

20

Develop Items Based on Definition and Wording Used by Participants

Over the past 12 months, how often did doctors…

..ask if you would be interested in hearing more about alternative types of herbal treatments? (CAM domain)

21

Cognitive Interviews

Derived from social and cognitive psychology to explore processes respondents use to answer survey questions

Diagnostic tool for pretesting survey questions

22

Cognitive Interviews Examine 4 Steps in Answering Questions

Comprehension of the question– as intended by the researchers

Retrieve the information – various strategies used to access memory

Judgment formation - formulate an answer– calculate or judge the correct information

Edit response - decide what to report– is answer embarrassing, socially undesirable?

23

Purpose of Cognitive Interviews

To learn .. .. if respondents understand words and phrases

as intended (meaning) .. about the process of answering the questions .. whether items are unacceptable .. about the usefulness of response choices

– whether response choices are adequate

– how they use the response choices

24

Two Types of Cognitive Interviews

Think aloud interviews– Respondent asked to think aloud as they answer

question Probe interviews

– Interviewer asks specific questions to elicit how respondent answered question

– Scripted and spontaneous probing Think alouds - greater respondent burden

25

Two Types of Probe Interviews

Concurrent probing– Ask probes immediately after respondent has given

answer to survey item– Advantage-information is fresh in respondent’s

mind Retrospective probing

– Ask probes after entire interview– Advantage-able to assess standard administration of

items

26

Steps Involved in Cognitive Interview Pretesting

Decide on a final item pool Develop scripted “probes” for a subset of items Translate interview and probe questions Recruit sample for cognitive interviews Conduct cognitive interviews Analyze results Revise items based on results Cognitive testing of revised items

27

Reducing Item Pool

Reduce items from item pool (subjective process among research team)

Criteria:– maintain breadth of concept, multiple items/concept– reduce redundancy (but OK to test alternate

versions of items)– eliminate items that are unclear, complex, require

high verbal skills, lack face validity, or will not translate well

28

Writing Probe Questions

From reduced set of items, select potentially problematic items for pretesting

Write open-ended scripted probe questions– worded to reveal if suspected problem

with a specific item is present

29

Types of Cognitive Probes

General– Tell me what you were thinking when you

answered that question– How easy or difficult was it to answer that

question? Why? Explore meaning of word or phrase

– I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?

30

Types of Cognitive Probes (cont.)

Retrieval– How did you remember that?

Judgment– Why did you pick that number for your answer?

Response– Do you think that most people answer this

question honestly?Collins D. Quality of Life Research 2003. 12:229-38.

31

Types of Cognitive Probes (cont.)

Redundancy– How is the phrase “give you advice about your diet

and exercise” different from the phrase “talk to you about your diet and exercise”?

Acceptability– When I asked you how often you felt discriminated

against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?

32

Types of Cognitive Probes (cont.)

Cultural appropriateness– I asked you how often doctors asked you about

your health beliefs? What does the term ‘health beliefs’ mean to you?

33

Sampling for Cognitive Interviews

Usually do not use representative samples Include respondents from major segments

of population to be sampled for main survey

Approximately 5-15 interviews/group, but may involve several rounds

34

Recruiting for Cognitive Interviews

Explain how their help fits into larger study, process of creating questions

Explain their role clearly: – “help us learn how to ask better questions”– “help us make questions clearer for others” – “help us to identify problems with questions”

Pay subjects $25 - $50, interview is demanding If survey is long, pretest different sections on

different subjects

35

Conducting Cognitive Interviews

Individual face-to-face, in-depth interviews

Standard administration of closed-ended items

Administer probe questions at the end (or concurrently)

Typically 1 hr interview Each interview audiotaped and transcribed

36

Data from Cognitive Interviews

Two general methods for compiling data Use electronic version of survey to enter

comments on each item directly under each question

OR Transcripts are data

37

Analysis of Cognitive Interviews

For method where entered comments under each question– Annotated questionnaire - aggregate item-by-item

comments over multiple interviews– Summary of most significant problems

Two approaches for transcripts– Behavioral coding– Use qualitative data analysis software to perform

content analysis

38

Behavioral Coding

Review transcripts to identify problems with standard administration of items– respondent and interviewer “problem” behaviors

Assign “problem behavior” codes to each item using pre-determined categories

Summarize results for each item:– proportion of interviews with each problematic behavior

for each item» e.g., 7/48 respondents requested clarification of item 10

39

Examples of Behavioral Codes

Interviewer behavior Hard to read - interviewer experiences

difficulty reading question

Respondent behavior Repeat question - respondent asks to have

the question repeated

40

Content Analysis of Items and Probes

Using qualitative analysis software, review all dialogue that ensued during standard administration of closed-ended items and open-ended probes– can reveal source of problems– can help in deciding whether to keep, modify or

drop items Allows you to examine dialogue for each item

– within groups– across groups/languages

41

Example of Results: Interpreting the Question

Original item: How satisfied are you with the amount of stress or worries in your life?

Probe: Did you answer this question in terms of stress, worries, or both?

AA - tended to answer in terms of worries No term in Spanish for “stress” Item revised

– How satisfied are you with the amount of worries in your life?Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

42

Example of Results: Unclear Phrase

Original item: Have you had any medical tests or procedures in past year?– 26% of respondents asked for clarification

Probe: What did you include as medical tests or procedures?

Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures)

Item revised to include examples:– Have you had any medical tests or procedures, such as blood

tests, x-rays, or cancer screening tests?

43

Example of Results: Information Retrieval

Original item: How satisfied are you with your ability to travel on vacations?

Probe: What do you think we meant by vacation? Neither concept of “vacation” nor “travel for

pleasure” had relevance to lifestyle of AA and MA; travel for family reasons

Dropped itemWarnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.

44

Example of Results: Response Sets

Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar)

Problem: No direct Spanish translation for “unimportant”– bilinguals understood translation “sin importancia” as

equivalent to unimportant– Spanish monolinguals did not understand it as the

negative pole of the scale Solution: Changed English to unipolar scale “not at all

important” to “very important” to correspond to best Spanish

Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38

45

Example of Results: Redundancy of Items

Original items:– How often did doctors explain what was causing

your health problem?– How often did doctors explain your diagnosis?

Probe: What do the words health problem and diagnosis mean to you?

Respondents viewed them as the same Some respondents did not know meaning of diagnosis Dropped item with word diagnosis

46

Example of Results: Cultural Differences

Original item:– How often did doctors ask you if you wanted to

include your family when making decisions?Probe: When would you include your family in making

decisions about your health care? AAs and WHs viewed question as irrelevant; only in

cases of genetic or terminal illness Latinos more likely to include family in less serious

cases Dropped item

47

Summary

Greatest problems are with question interpretation– usually due to need to write at lower level of verbal

comprehension Sometimes English concepts not meaningful in

other languages, are irrelevant for certain groups, or differ in meaning across groups

Need to pretest response sets– Ethnic groups may use them differently

48

Advantages of Cognitive Interviews Complement other field test methods (e.g.

where problems are identified by missing or truncated answers)

Identifies where responses might be affected by cultural or group experiences

Suggests ways to revise items, response sets Improves validity of questions

49

Disadvantages of Cognitive Interviews

Flags problems, but significance of the problem remains matter of subjective judgment– at which point need to revise or drop items

Based on small number of respondents Time and labor intensive

50

Conclusions: Usefulness of Qualitative Methods

Qualitative methods are necessary component of research in diverse groups

To identify relevant constructs, items, unanticipated mechanisms of disparities

To pretest items for conceptual adequacy Even limited cognitive interviewing can reveal

significant issues overlooked by survey developers

51

How To Resources

Ward H, Atkins J. From their lives: a manual on how to conduct focus groups of low-income parents. Institute for Child and Family Policy, Edmund S. Muskie School of Public Health, University of Southern Maine, 2002: http://muskie.usm.maine.edu/focusgroupmanual/manual.htm

Willis GB. Cognitive Interviewing: A “How To” Guide. Research Triangle Institute, 1999:http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf