![Page 1: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/1.jpg)
1
Qualitative Methods to Develop and Pretest Surveys in Diverse Groups
Anna Nápoles-Springer, Ph.D.University of California San Francisco
Center for Aging in Diverse Communities
![Page 2: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/2.jpg)
2
Overview
Integrating quantitative and qualitative methods to develop surveys
Focus groups Cognitive interview methods Using results of cognitive interviews to
make decisions about item revisions/deletions
![Page 3: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/3.jpg)
3
When are Qualitative Methods Used?
Open-ended interviews typically used in new areas of study
Useful for in-depth knowledge about issues, especially in less studied groups
Especially critical in cross-cultural studies due to lack of information
![Page 4: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/4.jpg)
4
What are Qualitative Methods?
Data consist of words, not numbers Richly descriptive, open-ended Focus on inductive analytic approaches Many types: ethnography, participant-
observation, direct observation, focus groups, in-depth interviews
![Page 5: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/5.jpg)
5
When are Qualitative Methods Useful?
To understand the meaning of participants’ events, situations, and actions
To understand contextual influences on participants’ actions
To identify unanticipated phenomena and influences (e.g., exploratory studies to design questionnaires and identify variables for study)
To understand the processes underlying observed relationships between variables
![Page 6: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/6.jpg)
6
Meaning of Events
Richness of Qualitative Data
Contextual Influences on
Behavior
Processes Underlying Observed
Relationships
![Page 7: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/7.jpg)
7
Integrating Qualitative and Quantitative Methods to Develop Surveys
Goal: develop survey items that mean the same thing across groups
Quantitative methods (descriptive statistics, reliability, validity, missing data) identify ethnic differences in response patterns
Qualitative methods uncover cognitive processes used to answer questions; conceptual relevance
Iterative quantitative and qualitative methods
![Page 8: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/8.jpg)
8
Combining Qualitative and Quantitative Methods
Prior to quantitative: to develop concepts, framework, hypotheses, and content for structured survey items or interventions
After quantitative: – to help identify reasons for survey items not
performing well quantitatively– to explore possible explanations for
unexpected results
![Page 9: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/9.jpg)
9
Applications
To develop: Concepts and their definitions Measures of these concepts Pre-test measures
![Page 10: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/10.jpg)
10
Qualitative Methods to Develop Surveys
Focus Groups– Open-ended guided group discussion with probing of
responses
Cognitive interviews– Open-ended probes to assess how items are interpreted
and adequacy of response choices
– Respondent is expert
Expert panels provide input on relevant concepts Interviewer debriefing
![Page 11: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/11.jpg)
11
Focus Groups to Develop Surveys
To assess the universality of concepts and measures – funnel approach, concept to term
Facilitate culturally sensitive models (e.g., how being AA shapes experiences of discrimination)
Use phrases from transcripts to develop items Pretest clarity and relevance of existing items
![Page 12: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/12.jpg)
12
Focus Groups
Open-ended guided group discussion with probing of responses
Led by experienced moderator; usually 1 - 2 hours
Purposeful sampling of 6-10 homogenous participants per group
Participants stimulate comments of others Audio-record and transcribe discussion
![Page 13: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/13.jpg)
13
Focus Groups-Logistics
Moderator skills: listening, communication, negotiation, cultural similarity to participants
Costs of group: $600 - $1000 per group (incentives, audio-taping, transcription, translation, food)
Convenient and hospitable community setting In-person recruitment works best with
telephone/mail reminders
![Page 14: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/14.jpg)
14
Focus Groups
Advantages– group stimulates fruitful discussion
– spontaneity leads to discovery of new issues and factors
Disadvantages: scheduling, skilled moderator, group setting may be inappropriate
![Page 15: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/15.jpg)
15
Sampling in Qualitative Research
Purposive, nonprobabilistic Deliberately select settings, persons or
events to best answer research questions Set up contrasts to examine differences
between settings or individuals
![Page 16: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/16.jpg)
16
Example: Measures of Cultural Sensitivity of Clinicians
Conducted 19 focus groups with 163 participants
61 African Americans, 45 Latinos and 55 non-Latino Whites
Asked about the influence of cultural factors on the quality of encounters
![Page 17: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/17.jpg)
17
Focus Group Questions
What does the word “culture” mean to you? What do or don’t your doctors understand about
your culture or health beliefs that might affect your visits?
Embedded in more general discussion of communication and decision making in medical encounters
![Page 18: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/18.jpg)
18
Identify Themes (Codes): How Culture Might Affect Health Care and Outcomes
CAM Discrimination Doctor Culture Ethnicity of MD Family Immigration
Language Modesty Nutrition Patient
submissiveness Spirituality
![Page 19: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/19.jpg)
19
Define Domains of Each Cultural Sensitivity Domain: CAM
Definition: MD’s knowledge and acceptance of non-Western, non-biomedical, holistic approaches to health or healing
“When I told her I was on estrogen, it was refreshing to hear suggestions about alternative types of herbal treatments.”
AA woman > 50
![Page 20: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/20.jpg)
20
Develop Items Based on Definition and Wording Used by Participants
Over the past 12 months, how often did doctors…
..ask if you would be interested in hearing more about alternative types of herbal treatments? (CAM domain)
![Page 21: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/21.jpg)
21
Cognitive Interviews
Derived from social and cognitive psychology to explore processes respondents use to answer survey questions
Diagnostic tool for pretesting survey questions
![Page 22: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/22.jpg)
22
Cognitive Interviews Examine 4 Steps in Answering Questions
Comprehension of the question– as intended by the researchers
Retrieve the information – various strategies used to access memory
Judgment formation - formulate an answer– calculate or judge the correct information
Edit response - decide what to report– is answer embarrassing, socially undesirable?
![Page 23: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/23.jpg)
23
Purpose of Cognitive Interviews
To learn .. .. if respondents understand words and phrases
as intended (meaning) .. about the process of answering the questions .. whether items are unacceptable .. about the usefulness of response choices
– whether response choices are adequate
– how they use the response choices
![Page 24: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/24.jpg)
24
Two Types of Cognitive Interviews
Think aloud interviews– Respondent asked to think aloud as they answer
question Probe interviews
– Interviewer asks specific questions to elicit how respondent answered question
– Scripted and spontaneous probing Think alouds - greater respondent burden
![Page 25: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/25.jpg)
25
Two Types of Probe Interviews
Concurrent probing– Ask probes immediately after respondent has given
answer to survey item– Advantage-information is fresh in respondent’s
mind Retrospective probing
– Ask probes after entire interview– Advantage-able to assess standard administration of
items
![Page 26: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/26.jpg)
26
Steps Involved in Cognitive Interview Pretesting
Decide on a final item pool Develop scripted “probes” for a subset of items Translate interview and probe questions Recruit sample for cognitive interviews Conduct cognitive interviews Analyze results Revise items based on results Cognitive testing of revised items
![Page 27: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/27.jpg)
27
Reducing Item Pool
Reduce items from item pool (subjective process among research team)
Criteria:– maintain breadth of concept, multiple items/concept– reduce redundancy (but OK to test alternate
versions of items)– eliminate items that are unclear, complex, require
high verbal skills, lack face validity, or will not translate well
![Page 28: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/28.jpg)
28
Writing Probe Questions
From reduced set of items, select potentially problematic items for pretesting
Write open-ended scripted probe questions– worded to reveal if suspected problem
with a specific item is present
![Page 29: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/29.jpg)
29
Types of Cognitive Probes
General– Tell me what you were thinking when you
answered that question– How easy or difficult was it to answer that
question? Why? Explore meaning of word or phrase
– I asked you how often doctors take a genuine interest in you. What does the phrase “genuine interest” mean to you?
![Page 30: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/30.jpg)
30
Types of Cognitive Probes (cont.)
Retrieval– How did you remember that?
Judgment– Why did you pick that number for your answer?
Response– Do you think that most people answer this
question honestly?Collins D. Quality of Life Research 2003. 12:229-38.
![Page 31: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/31.jpg)
31
Types of Cognitive Probes (cont.)
Redundancy– How is the phrase “give you advice about your diet
and exercise” different from the phrase “talk to you about your diet and exercise”?
Acceptability– When I asked you how often you felt discriminated
against by doctors because of your race or ethnicity, you answered (read answer given). Were you offended by this question?
![Page 32: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/32.jpg)
32
Types of Cognitive Probes (cont.)
Cultural appropriateness– I asked you how often doctors asked you about
your health beliefs? What does the term ‘health beliefs’ mean to you?
![Page 33: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/33.jpg)
33
Sampling for Cognitive Interviews
Usually do not use representative samples Include respondents from major segments
of population to be sampled for main survey
Approximately 5-15 interviews/group, but may involve several rounds
![Page 34: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/34.jpg)
34
Recruiting for Cognitive Interviews
Explain how their help fits into larger study, process of creating questions
Explain their role clearly: – “help us learn how to ask better questions”– “help us make questions clearer for others” – “help us to identify problems with questions”
Pay subjects $25 - $50, interview is demanding If survey is long, pretest different sections on
different subjects
![Page 35: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/35.jpg)
35
Conducting Cognitive Interviews
Individual face-to-face, in-depth interviews
Standard administration of closed-ended items
Administer probe questions at the end (or concurrently)
Typically 1 hr interview Each interview audiotaped and transcribed
![Page 36: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/36.jpg)
36
Data from Cognitive Interviews
Two general methods for compiling data Use electronic version of survey to enter
comments on each item directly under each question
OR Transcripts are data
![Page 37: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/37.jpg)
37
Analysis of Cognitive Interviews
For method where entered comments under each question– Annotated questionnaire - aggregate item-by-item
comments over multiple interviews– Summary of most significant problems
Two approaches for transcripts– Behavioral coding– Use qualitative data analysis software to perform
content analysis
![Page 38: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/38.jpg)
38
Behavioral Coding
Review transcripts to identify problems with standard administration of items– respondent and interviewer “problem” behaviors
Assign “problem behavior” codes to each item using pre-determined categories
Summarize results for each item:– proportion of interviews with each problematic behavior
for each item» e.g., 7/48 respondents requested clarification of item 10
![Page 39: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/39.jpg)
39
Examples of Behavioral Codes
Interviewer behavior Hard to read - interviewer experiences
difficulty reading question
Respondent behavior Repeat question - respondent asks to have
the question repeated
![Page 40: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/40.jpg)
40
Content Analysis of Items and Probes
Using qualitative analysis software, review all dialogue that ensued during standard administration of closed-ended items and open-ended probes– can reveal source of problems– can help in deciding whether to keep, modify or
drop items Allows you to examine dialogue for each item
– within groups– across groups/languages
![Page 41: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/41.jpg)
41
Example of Results: Interpreting the Question
Original item: How satisfied are you with the amount of stress or worries in your life?
Probe: Did you answer this question in terms of stress, worries, or both?
AA - tended to answer in terms of worries No term in Spanish for “stress” Item revised
– How satisfied are you with the amount of worries in your life?Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.
![Page 42: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/42.jpg)
42
Example of Results: Unclear Phrase
Original item: Have you had any medical tests or procedures in past year?– 26% of respondents asked for clarification
Probe: What did you include as medical tests or procedures?
Medical test or procedures unclear (e.g., asked if it included dental or cosmetic procedures)
Item revised to include examples:– Have you had any medical tests or procedures, such as blood
tests, x-rays, or cancer screening tests?
![Page 43: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/43.jpg)
43
Example of Results: Information Retrieval
Original item: How satisfied are you with your ability to travel on vacations?
Probe: What do you think we meant by vacation? Neither concept of “vacation” nor “travel for
pleasure” had relevance to lifestyle of AA and MA; travel for family reasons
Dropped itemWarnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38.
![Page 44: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/44.jpg)
44
Example of Results: Response Sets
Original scale: “Very unimportant to Very Important” on 0-100 scale (bipolar)
Problem: No direct Spanish translation for “unimportant”– bilinguals understood translation “sin importancia” as
equivalent to unimportant– Spanish monolinguals did not understand it as the
negative pole of the scale Solution: Changed English to unipolar scale “not at all
important” to “very important” to correspond to best Spanish
Warnecke RB, et al. JNCI Monographs No. 20, 1996; 29-38
![Page 45: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/45.jpg)
45
Example of Results: Redundancy of Items
Original items:– How often did doctors explain what was causing
your health problem?– How often did doctors explain your diagnosis?
Probe: What do the words health problem and diagnosis mean to you?
Respondents viewed them as the same Some respondents did not know meaning of diagnosis Dropped item with word diagnosis
![Page 46: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/46.jpg)
46
Example of Results: Cultural Differences
Original item:– How often did doctors ask you if you wanted to
include your family when making decisions?Probe: When would you include your family in making
decisions about your health care? AAs and WHs viewed question as irrelevant; only in
cases of genetic or terminal illness Latinos more likely to include family in less serious
cases Dropped item
![Page 47: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/47.jpg)
47
Summary
Greatest problems are with question interpretation– usually due to need to write at lower level of verbal
comprehension Sometimes English concepts not meaningful in
other languages, are irrelevant for certain groups, or differ in meaning across groups
Need to pretest response sets– Ethnic groups may use them differently
![Page 48: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/48.jpg)
48
Advantages of Cognitive Interviews Complement other field test methods (e.g.
where problems are identified by missing or truncated answers)
Identifies where responses might be affected by cultural or group experiences
Suggests ways to revise items, response sets Improves validity of questions
![Page 49: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/49.jpg)
49
Disadvantages of Cognitive Interviews
Flags problems, but significance of the problem remains matter of subjective judgment– at which point need to revise or drop items
Based on small number of respondents Time and labor intensive
![Page 50: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/50.jpg)
50
Conclusions: Usefulness of Qualitative Methods
Qualitative methods are necessary component of research in diverse groups
To identify relevant constructs, items, unanticipated mechanisms of disparities
To pretest items for conceptual adequacy Even limited cognitive interviewing can reveal
significant issues overlooked by survey developers
![Page 51: 1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for](https://reader030.vdocument.in/reader030/viewer/2022032805/56649edf5503460f94bef6e7/html5/thumbnails/51.jpg)
51
How To Resources
Ward H, Atkins J. From their lives: a manual on how to conduct focus groups of low-income parents. Institute for Child and Family Policy, Edmund S. Muskie School of Public Health, University of Southern Maine, 2002: http://muskie.usm.maine.edu/focusgroupmanual/manual.htm
Willis GB. Cognitive Interviewing: A “How To” Guide. Research Triangle Institute, 1999:http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf