total survey error in disability assessments
DESCRIPTION
Total Survey Error in Disability Assessments. Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards and Tamara Bruce, Westat Presented at the International Total Survey Error Workshop Quebec, Canada June 2011. Overview. - PowerPoint PPT PresentationTRANSCRIPT
Total Survey Error in Disability Assessments
Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS)
Brad Edwards and Tamara Bruce, Westat
Presented at the International Total Survey Error Workshop Quebec, Canada
June 2011
Overview
NHATS and capacity measures Self report and performance NHATS assessments Strategies for error reduction
- design, training, standardization, parsing out nonresponse
Pretest results National experience Future research2
NHATS
Westat working with a team led by Johns Hopkins Bloomberg School of Public Health
Funded through cooperative agreement from NIA
CAPI panel study with 9,000 Medicare beneficiaries, annual interviews starting 2011
New and improved measures of disability for producing data on trends and trajectories; self-reports, mental and physical assessments, eventually biomarkers, links with admin data
3
Conceptual Framework
Blend of Nagi’s model and WHO perspective
Distinguish explicitly between– Capacity to do something – basic
building blocks
– What people actually do in actual environment – activities
Accommodations may fill gap
4
Capacity Is Critical
Measures of capacity over time are key elements in understanding individual patterns of progression to activity limitations. They allow us to track trends in function that are independent of environmental changes or accommodations, for understanding the disablement process, and as targets for interventions to prevent or slow disability.
5
Reducing Error in Capacity Measures
Questionnaire design Importance of performance measures Challenges for quality management
– New to surveys
– Complex
– Expect high level of interviewer variance
6
Physical Capacity: Upper Extremity
Self report
Able to– Put book on shelf/reach
overhead– Open jar/grasp small object
Performance
Grip strength
7
Physical Capacity: Lower Extremity
Self Report
Able to– Walk 6/3 blocks?– Kneel/bend over?– Lift and carry 20/10
pounds?– Walk up 20/10 stairs?
Performance
Walking speed Balance stands
– Side by side– Semi-tandem– Full-tandem– One leg eyes open– One leg eyes closed
Chair stands
8
Cognitive Capacity: Memory
Self report
At present time? Memory problems
interfere with activities? Memory compared to one
year ago?
Performance
10 word recall– Immediate– Delayed
9
Other Cognitive Performance Measures
Orientation– Day of week, date, naming President and
Vice President Overall cognitive screening/executive
function– Clock drawing
Attention & interference/executive function– Stroop test (computerized)
10
Standardizing Administration
11
Performance measures of capacity can be hard to standardize in large-scale surveys with lay interviewers
NHATS uses multiple methods to improve consistency & accuracy:– Activities Booklet design & use within CAPI framework
– Standardized presentation of cognitive tasks using flash displays
– Survey design & interviewer feedback using CARI coding
– Formal certification process using live respondents
– Follow-up web-based recertification
Training Approaches
Broader scope of demands on NHATS interviewers for performance measures– tests require using a variety of equipment
– navigation of unfamiliar environments
– potentially uncomfortable instructions to respondents
Integration of video components– recruitment
– in-person interviewer training modules such as walking course
– certification & re-certification processes
Walking Course Timing 1
Walking Course Timing 2
Walking Course Timing 3
15
Training Video Demo
16
Item Nonresponse
Performance measures used in depth set of exclusion criteria to minimize burden
High completion rate for both cognitive & performance tasks for those eligible
Comprehensive list of reasons why test was not conducted allow detailed analyses
Majority of unit nonresponse due to inability to complete easier task & safety concerns
Performance related to age, health rating, care setting, memory
17
Respondent Reactions
Avoiding “test” and “performance” Training interviewers how to respond
to requests (e.g., “How did I do?”) Physical activities: “7th inning stretch” Positive feedback at the end Interest in assessing respondent
satisfaction and rapport in relation to nonresponse in later rounds
18
Comparison of Performance and Self Report
High functioning performance group (attempting one-leg balance stands, meeting accuracy thresholds for Stroop) reported better self-reported health and memory and (for the Stroop) had higher performance on two memory tests
19
Comparison with Other Surveys
NLTCS screening questions in NHATS form a bridge from previous trend data to detailed NHATS self reports and assessments
Possibility of future bridge to 6 disability questions developed by NCHS and used on the American Community Survey
20
Conclusions
Reducing error in capacity measures based on performance assessment– sharpens understanding of individual
trajectories and accommodations– is expected to improve predictive ability
for health outcomes NHATS protocol
– standardizes administration– focuses on reasons for nonresponse
21
Conclusions (2)
As a new survey, NHATS has been able to give consideration to the implications of doing these types of assessments from the outset, starting with interviewer recruitment and training. Experience from pretesting has led to refinements of these procedures (e.g., use of videos in recruitment; certification procedures in training) and to the NHATS Activities Booklet. Training materials and data collection instruments will be available later this year at www.nhats.org.
22
Future Research
Interviewer variance study Examination of
– interviewer and respondent conditioning effects – “drift” trajectories– item nonresponse as predictor of unit nonresponse
Error comparison/tradeoffs between self-reports and performance Analysis of cognitive performance and data quality More development
– distance learning– SPC charts, integration with paradata for management – re-certification via WebEX, 2-way video
23
