1 class 9 interpreting pretest data, considerations in modifying or adapting measures november 13,...
TRANSCRIPT
![Page 1: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/1.jpg)
1
Class 9
Interpreting Pretest Data, Considerations in Modifying or Adapting Measures
November 13, 2008
Anita L. Stewart Institute for Health & Aging
University of California, San Francisco
![Page 2: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/2.jpg)
2
Overview of Class 9
Analyzing pretest data Modifying/adapting measures Keeping track of your study measures Creating and testing scales in your
sample
![Page 3: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/3.jpg)
3
Summarize Data on Pretest Interviews
Summarize problems and nature of problems for each item
Determine how important problems are Results become basis for possible
revisions/adaptations
![Page 4: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/4.jpg)
4
Methods of Analysis
Optimal: transcripts of all pretest interviews
For each item - summarize all problems Analyze dialogue (narrative) for clues to
solve problems
![Page 5: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/5.jpg)
5
Behavioral Coding
Systematic approach to identifying problems with items– “interviewer” and “respondent” problems
Can code problems based on:– Standard administration
– Responses to specific probes
![Page 6: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/6.jpg)
6
Examples of Interviewer “Behaviors” Indicating Problem Items
Question misread or altered– Slight change – meaning not affected
– Major change – alters meaning Question skipped
![Page 7: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/7.jpg)
7
Examples of Respondent “Behaviors” Indicating Problem Items
Asks for clarification or repeat of question Did not understand question Doesn’t know the answer Qualified answer (e.g., it depends) Indicates answer falls between existing
response choices Refusal
![Page 8: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/8.jpg)
8
Summarize Behavioral Coding For Each Item
Proportion of interviews (respondents) with each problematic behavior
# of occurrences of problem divided by N– 7/48 respondents requested clarification
![Page 9: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/9.jpg)
9
Behavioral Coding Summary Sheet: Standard Administration (N=20)
Item #
Interviewer: difficulty reading
Subject: asks to repeat Q
Subject: asks for
clarification
1 2/20 0 1/20
2 0 0 0
3 1/20 3/20 2/20
4 0 1/20 0
![Page 10: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/10.jpg)
10
Can Identify Problems Even When No Problem “Behaviors” Found
Respondents appear to answer question appropriately
Additional problems identified with probes– Probe on meaning: Response indicates lack of
understanding
– Probe on use of response options: Response indicates options are problematic
![Page 11: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/11.jpg)
11
Behavioral Coding of Probe Results
I asked you how often doctors asked you about your health beliefs. What does the term “health beliefs” mean to you?
Behavioral coding: # times response indicated lack of understanding as intended– e.g., 2/15 respondents did not understand meaning
based on response to probe
![Page 12: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/12.jpg)
12
Behavioral Coding Summary: Standard Administration (N=20) + Probes (N=10)
Item # Probe
Meaning unclear
Interviewer -difficulty
reading
Subject: asks to
repeat Q
Subject: asks for
clarification
1 10 2/10 2/20 0 1/20
2 0 0 0 0 0
3 10 4/15 1/20 3/20 2/20
4 10 0 0 1/20 0
![Page 13: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/13.jpg)
13
Interpret Behavioral Coding Results
Determine if problems are common– Items with only a few problems may be fine
Quantifying “common” problems– several types of problems (many row entries)
– several subjects experienced a problem» problem w/item identified in >15% of interviews
![Page 14: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/14.jpg)
14
Continue Analyzing Items with “Common” Problems
Identify “serious” common problems– Gross misunderstanding of the question– Yields completely erroneous answer– Couldn’t answer the question at all
Some less serious problems can be addressed by improved instructions or a slight modification
![Page 15: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/15.jpg)
15
Addressing More Serious Problems
Conduct content analysis of transcript – Use qualitative analysis software (e.g.,
NVIVO) For these items: review dialogue that
ensued during administration of item and probes– can reveal source of problems– can help in deciding whether to keep, modify
or drop items
![Page 16: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/16.jpg)
16
Results: Probing Meaning of Phrase I asked you how often doctors asked you
about your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine”
“.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?”
![Page 17: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/17.jpg)
17
Results: Probing Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
![Page 18: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/18.jpg)
18
Results: Probing Meaning of Phrase
On about how many of the past 7 days did you eat foods that are high in fiber, like whole grains, raw fruits, and raw vegetables? – Probe: what does the term “high fiber” mean to
you? Behavioral coding of item
– Over half of respondents exhibited a problem Review answers to probe
– Over ¼ did not understand the term
Blixt S et al., Proceedings of section on survey research methods,American Statistical Association, 1993:1442.
![Page 19: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/19.jpg)
19
Results: No Behavior Coding Issues but Probe Detected Problems
I seem to get sick a little easier than other people (definitely true, mostly true, mostly false, definitely false)
Behavioral coding of item– Very few problems
Review answers to probe– Almost 3/4 had comprehension problems– Most problems around term “mostly” (either its true
or its not)
Blixt S et al., Proceedings of section on survey research methods,American Statistical Association, 1993:1442.
![Page 20: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/20.jpg)
20
Results: Beck Depression Inventory (BDI) and Literacy
Cognitive interviews: older adults, oncology pts, and less educated adults
Administered REALM (reading literacy test) and some selected BDI items
Asked to paraphrase items
TL Sentell, Community Mental Health Journal, 2008;39:323
![Page 21: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/21.jpg)
21
Results: Beck Depression Inventory (BDI) and Literacy (cont)
For each item, from 0-62% correctly paraphrased item
Most misunderstandings: vocabulary confusion Phrase: I am critical of myself for my
weaknesses and mistakes– “Critical is when you’re very sick”– “I don’t know how to explain mistakes”
![Page 22: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/22.jpg)
22
Interpreting Pretest Results of Self-Administered Questionnaires
Missing data is a clue to problematic items– More missing data associated with unclear,
difficult, or irrelevant items
– Cognitive interviewing can help determine reasons for missing data
![Page 23: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/23.jpg)
23
How Missing Data Prevalence Helps
Items with large percent of responses missing – clue to problem
In H-CAHPS® pretest: Did hospital staff talk with you about whether
you would have the help you needed when you left the hospital?– 35% missing for Spanish group– 29% missing for English group
MP Hurtado et al. Health Serv Res, 2005;40-6, Part II:2140-2161
![Page 24: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/24.jpg)
24
Exploring Differences by Diverse Groups
Back to issue of “equivalence” of meaning across groups
All cognitive interview analyses can be done separately by group
![Page 25: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/25.jpg)
25
Results: Use of Response Scale
Do diverse groups use the response scale in similar ways?
Re questions about cultural competence of providers– Interviewers reported that Asian respondents
who were completely satisfied did not like to use the highest score on the rating scale
California Pan-Ethnic Health Network (CPEHN) Report, 2001
![Page 26: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/26.jpg)
26
Results: Use of Response Scale (cont)
Behavioral Risk Factor Surveillance Survey (BRFSS) pretesting
Found that Puerto Rican, Mexican American, and African American respondents more likely to choose extreme response categories than Whites.
RB Warnecke et al, Ann Epidemiol, 1997:7:334-342
![Page 27: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/27.jpg)
27
Differential Use of CAHPS® 0-10 Global Rating Scale
Compared Medicaid and commercially insured adults on use of scale
Medicaid enrollees more likely than commercial participants to use extreme ends of scale– All other things being equal
PC Damiano et al, Health Serv Outcomes Res Method, 2004:5:193-205
![Page 28: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/28.jpg)
28
Results: Probe on Difficulty:CES-D Item
“During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends”
Probe: Do you feel this is a question that people would or would not have difficulty understanding?– Latinos more likely than other groups to report
people would have difficulty
TP Johnson, Health Survey Research Methods, 1996
![Page 29: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/29.jpg)
29
Overview of Class 9
Analyzing pretest data Modifying/adapting measures Keeping track of your study measures Creating and testing scales in your
sample
![Page 30: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/30.jpg)
30
Now What!
Issues in adapting measures based on pretest results
Cognitive interview pretesting during development phases of measure– Can modify items and continue pretesting
Cognitive interview pretesting prior to using published survey:– More problematic
![Page 31: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/31.jpg)
31
Modification: Probing the Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
We changed the question to receptionist and appointment staff
![Page 32: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/32.jpg)
32
Results: Probing Meaning and Cultural Appropriateness
I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?” We changed the question to “personal beliefs
about your health
![Page 33: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/33.jpg)
33
Criteria for Whether or Not to Modify Measure
Contact author – May be open to modifications, working with you
Be sure your opinion is based on extensive pretests with consistent problems– Don’t rely on a few comments in a small pretest
Work with a measurement specialist to assure that proposed modifications are likely to solve problem
![Page 34: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/34.jpg)
34
Tradeoffs of Using Adapted Measures
Advantages Improve internal validity
Disadvantages Lose external validity Know less about modified measure Need to defend new measure
![Page 35: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/35.jpg)
35
Adding New (Modified) Items
One approach if you find serious problems with a standard measure– Write new items you think will be better (use same
format)– Retain original intact items and add modified items
Can test original scale and revised scale with modified items instead of original items
![Page 36: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/36.jpg)
36
Modifying response categories
If response choices are too few and/or coarse, can improve without compromising too much– Try adding levels within existing response
scale
![Page 37: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/37.jpg)
37
One Modification: Too Many Response Choices
SF36 version 1 1 - All of the time 2 - Most of the time 3 - A good bit of the time 4 - Some of the time 5 - A little of the time 6 - None of the time
SF36 version 2 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time
![Page 38: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/38.jpg)
38
Modification of Health Perceptions Response Choices for Thai Translation
Usual responses: 1 - Definitely true 2 - Mostly true 3 - Don’t know 4 - Mostly false 5 - Definitely false
Modified: 1 – Not at all true 2 – A little true 3 - Somewhat true 4 - Mostly true 5 – Definitely true
e.g., My health is excellent, I expect my health to get worse
![Page 39: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/39.jpg)
39
Modifying Item Stems
If item wording will not be clear to your population– Can add parenthetical phrases
Have you ever been told by a doctor that you have diabetes (high blood sugar)?
![Page 40: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/40.jpg)
40
Strategy for Modified Measures
Test measure in original and adapted form Choose measure that performs the best
![Page 41: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/41.jpg)
41
Analyzing New (Modified) Measure
Factor analysis – All original items– Original plus new items replacing original
Correlations with other variables– Does the new measure detect stronger associations?
Outcome measure– Does the new measure detect more change over
time?
![Page 42: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/42.jpg)
42
Analytic Strategy: CAHPS® 0-10 Global Rating Scale: Response
Usual classifications 0-9, 10 (dichotomy)
Proposed classification 0-8, 9-10
PC Damiano et al, Health Serv Outcomes Res Method, 2004:5:193-205
Can’t change the scale – part of standardized survey
![Page 43: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/43.jpg)
43
Overview of Class 9
Analyzing pretest data Modifying/adapting measures Keeping track of your study measures Creating and testing scales in your
sample
![Page 44: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/44.jpg)
44
Questionnaire Guides
Organizing your survey measures– Keep track of measurement decisions
Sample guide to measures (last week)– Documents sources of measures
– Any modifications, reason for modification
![Page 45: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/45.jpg)
45
“Sample Guide to Measures” Handout
Type of variable Concept Measure Data source Number of items/survey question numbers Number of scores or scales for each measure References
![Page 46: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/46.jpg)
46
Sample “Summary of Survey Variables..” Handout
Develop “codebook” of scoring rules Several purposes
– Variable list
– Meaning of scores (direction of high score)
– Special coding
– How missing data handled
– Type of variable (helps in analyses)
![Page 47: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/47.jpg)
47
Item Naming Conventions
Optimal coding is to assign raw items their questionnaire number – Can always link back to questionnaire easily
Some people assign a variable name to the questionnaire item– This will drive you crazy
![Page 48: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/48.jpg)
48
Variable Naming Conventions
Assigning variable names is an important step– make them as meaningful as possible– plan them for all questionnaires at the beginning
For study with more than one source of data, a suffix can indicate which point in time and which questionnaire– B for baseline, 6 for 6-month, Y for one year– M for medical history, L for lab tests
![Page 49: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/49.jpg)
49
Variable Naming Conventions (cont)
Medical History Questionnaire
HYPERTMB HYPERTM6
Baseline 6 months
![Page 50: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/50.jpg)
50
Variable Naming Conventions (cont)
A prefix can help sort variable groupings alphabetically– e.g., S for symptoms
SPAINB, SFATIGB, SSOBB
![Page 51: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/51.jpg)
51
Overview of Class 9
Analyzing pretest data Modifying/adapting measures Keeping track of your study measures Creating and testing scales in your
sample
![Page 52: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/52.jpg)
52
On to Your Field Test or Study
What to do once you have your baseline data
How to create summated scale scores
![Page 53: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/53.jpg)
53
Preparing Surveys for Data Entry: 4 Steps
Review surveys for data quality Reclaim missing and ambiguous data Address ambiguities in the questionnaire
prior to data entry Code open-ended items
![Page 54: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/54.jpg)
54
Review Surveys for Data Quality
Examine each survey in detail as soon as it is returned, and mark any..– Missing data
– Inconsistent or ambiguous answers
– Skip patterns that were not followed
![Page 55: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/55.jpg)
55
Reclaim Missing and Ambiguous Data
Go over problems with respondent– If survey returned in person, review then
– If mailed, call respondent ASAP, go over missing and ambiguous answers
– If you cannot reach by telephone, make a copy for your files and mail back the survey with request to clarify missing data
![Page 56: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/56.jpg)
56
Address Ambiguities in the Questionnaire Prior to Data Entry
When two choices are circled for one question, randomly choose one (flip a coin)
Clarify entries that might not be clear to data entry person
![Page 57: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/57.jpg)
57
Code Open-Ended Items
Open-ended responses have no numeric code– e.g., name of physician, reason for visiting
physician Goal of coding open-ended items
– create meaningful categories from variety of responses
– minimize number of categories for better interpretability
– Assign a numeric score for data entry
![Page 58: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/58.jpg)
58
Example of Open-Ended Responses
1.What things do you think are important for doctors at this clinic to do to give you high quality care?
Listen to your patients more often Pay more attention to the patient Not to wait so long Be more caring toward the patient Not to have so many people at one time Spend more time with the patients Be more understanding
![Page 59: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/59.jpg)
59
Process of Coding Open-Ended Data
Develop classification scheme– Review responses from 25 or more questionnaires – Begin a classification scheme– Assign unique numeric codes to each category– Maintain a list of codes and the verbatim answers
for each– Add new codes as new responses are identified
If a response cannot be classified, assign a unique code and address it later
![Page 60: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/60.jpg)
60
Example of Open-Ended Codes
Communication = 1 Listen to your patients more often = 1 Pay more attention to the patient = 1 Access to care = 2 Not to wait so long = 2 Not to have so many people at one time = 2Allow more time = 3 Spend more time with the patients = 3Emotional Support = 4 Be more understanding = 4 Be more caring toward the patient
![Page 61: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/61.jpg)
61
Verify Assigned Codes
Have a second person independently classify each response using final codes
Investigator can review a small subset of questionnaires to assure that coding assignment criteria are clear and are being followed
![Page 62: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/62.jpg)
62
Reliability of Open-Ended Codes
Depends on quality of question, of codes assigned, and the training and supervision of coders
Initial coder and second coder should be concordant in over 90% of cases
![Page 63: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/63.jpg)
63
Data Entry
Set up file Double entry of about 10% of surveys
– SAS or SPSS will compare two for accuracy» Acceptable 0-5% error» If 6% or greater – consider re-entering data
![Page 64: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/64.jpg)
64
Print Frequencies of Each Item and Review: Range Checks
Verify that responses for each item are within acceptable range– Out of range values can be checked on
original questionnaire» corrected or considered “missing”
– Sometimes out of range values mean that an item has been entered in the wrong column» a check on data entry quality
![Page 65: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/65.jpg)
65
Print Frequencies of Each Item and Review: Consistency Checking
Determine that skip patterns were followed Number of responses within a skip pattern
need to equal number who answered “skip in” question appropriately
![Page 66: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/66.jpg)
66
Print Frequencies of Each Item and Review: Consistency Checking (N=90)
1. Did your doctor prescribe any medications? (75 = yes, 15 = no)
1a. If yes, did your doctor explain the side effects of the medication? (80 responses)
Often will find that more people answered the second question than were supposed to
![Page 67: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/67.jpg)
67
Print Frequencies of Each Item and Review: Consistency Checking (cont.)
Go back to a questionnaires of those with problems – check whether initial “filter” item was
incorrectly answered or whether respondent inadvertently answered subset
– sometimes you won’t know which was correct Hopefully this was caught during initial
review of questionnaire and corrected by asking respondent
![Page 68: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/68.jpg)
68
Deriving Scale Scores
Create scores with computer algorithms in SAS, SPSS, or other program
Review scores to detect programming errors
Revise computer algorithms as needed Review final scores
![Page 69: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/69.jpg)
69
Creating Likert Scale Scores
Translate codebook scoring rules into program code (SAS, SPSS):– Reverse all items as specified
– Apply scoring rules
– Apply missing data rules Sample for SAS (see handout)
![Page 70: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/70.jpg)
70
Testing Scaling Properties and Reliability in Your Sample for Multi-Item Scales
Obtain item-scale correlations– Part of internal consistency reliability
program Calculate reliability in your sample
(regardless of known reliability in other studies) – internal-consistency for multi-item scales– test-retest if you obtained it
![Page 71: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/71.jpg)
71
SAS – Chapter 3: Assessing Reliability with Coefficient Alpha
Review statements and output How to test your scales for internal
consistency and appropriate item-scale correlations
![Page 72: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/72.jpg)
72
SAS/SPSS Both Make Item Convergence Analysis Easy
Reliability programs provide:– Item-scale correlations corrected for overlap
– Internal consistency reliability (coefficient alpha)
– Reliability with each item removed» To see effect of removing an item
![Page 73: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/73.jpg)
73
SAS – Obtaining Item-Scale Correlations and Coefficient Alpha
PROC CORR– DATA=data-set-name– ALPHA– NOMISS– VAR (list of variables)
Output:– Coefficient alpha– Item correlations– Item-scale correlations corrected for overlap
SAS Manual, Chapter 3: Assessing ScaleReliability with Coefficient Alpha
![Page 74: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/74.jpg)
74
SAS – Chapter 3: Assessing Scale Reliability with Coefficient Alpha
PROC CORR– DATA=data-set-name– ALPHA– NOMISS– VAR (list of variables)
Output:– Coefficient alpha– Item correlations– Item-scale correlations corrected for overlap
![Page 75: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/75.jpg)
75
Testing Reliability in STATA
www.stata.com/help.egi?alpha
Alpha varlist [if] [in] [, options]
SEE HANDOUT
![Page 76: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/76.jpg)
76
What if Reliability is Too Low?
Have to decide if you need to modify a scale New scales under development
– Modify using item-scale criteria Standard scales – cannot change
– Simply report problems as caveats in your analyses If problem is substantial
– Can create a modified scale and report results using standard and modified scale
![Page 77: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/77.jpg)
77
Value of Pretesting: Experts Say..
…evidence from our work suggests that many survey questions are seriously underevaluated
Evaluating items at final pretest phase is often too late in the process– Too late for extensive question redesign
A series of question evaluation steps is needed beginning well before the survey
FJ Fowler and CF Cannell. Using behavioral coding to identify problems with survey questions. In Answering Questions…, eds N Schwarz et al, Jossey-Bass, 1996
![Page 78: 1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University](https://reader035.vdocument.in/reader035/viewer/2022070403/56649f2f5503460f94c49a11/html5/thumbnails/78.jpg)
78
Homework for Class 10
Conduct 2 pretest interviews with individuals similar to your target population – Administer all questions– Administer your 4 probes
Summarize briefly your pretest results Indicate whether the measure appears to be
appropriate for the 2 pretest subjects– No inferences to broader sample needed