burden and loss: the role of panel survey recordkeeping in self-report quality and nonresponse

Burden and Loss: The Role of Panel Survey

Recordkeeping in Self-report Quality and Nonresponse

ITSEW 2010

Ryan Hubbard and Brad Edwards

2

Record Usage, Perceived Respondent Burden, and Measurement

• Record use may increase perceived respondent burden. Respondents experiencing higher levels of perceived burden may be more likely to drop out of the study.

• The regular use of records should improve event measurement and subsequent matching to administrative data.

• Record usage may disproportionately affect certain subsamples.

Background

3

4

Medicare Current Beneficiary Survey

• Longitudinal Study for Centers for Medicare and Medicaid Services currently in Round 58

• 16,000 Medicare beneficiaries in a rotating panel design

• 12 round study including baseline and exit interview over 4 years

• CAPI interview on health, health care utilization and health care expenditures designed to augment Medicare administrative data on events and payments

Matches costs to health care events in an effort to enumerate payers

and payments.

Relies on respondent collection and interviewer entry of medical

insurance statements from multiple sources

Asks respondents to voluntarily track health care utilization on a

provided calendar.

Modeling Error in MCBS Record Usage

5

Simplified Model

6

Perceived Burden

Non-response

Record Usage

Measurement Error

Ratio of Event Costs Covered by a

Statement

Panel Attrition

Non-Reporting of Events in

Admin Records

Event and Cost Data Differing from Admin

Records

EventCalendar

IncreaseIncrease

Decrease

7

Predicting Attrition in Later WavesMain Predictors Other Predictor Variables in ModelRatio of costs covered by statement

Average health care payments per round

Any interviews conducted in Spanish

Highest degree obtained

Ratio of rounds a calendar used

SP ever on Medicaid

MCBS panel Ever use a proxy

Average experience of interviewer

Same interviewer throughout study

Household income < $25,000

Married at some point during study

Average interview length (min)

Age when entered study (categorical)

Eligible for VA benefits

Average number of events per round

Sex of sample person

Average of self reported health(1=excellent;5=poor)

8

Summary of Attrition Results

• Higher levels of statement usage reduces the likelihood for refusal-based attrition. This relationship is stronger than any other in the model.

• Greater use of the calendar, controlling for other factors, reduces the likelihood for refusal-based attrition.

• Statement usage, because it is relatively passive, and calendar usage, because it is voluntary and may improve interview flow may actually dissuade attrition through a reduction in perceived burden.

• The effects we see actually represent unmeasured motivations such as topic salience and altruism. Those invested in the study are more likely to comply with requests for record keeping and complete all rounds of the study.

Simplified Model

9

Perceived Burden

Non-response

Record Usage

Measurement Error

Ratio of Event Costs Covered by a

Statment

Panel Attrition

Non-Reporting of Events in

Admin Records

Event and Cost Data Differing from Admin

Records

EventCalendar

10

Research Questions

• Does statement usage reduce event measurement error?

• Does statement usage reduce cost measurement error?

• Are there other factors related to the interview and the respondent that outweigh the effects of record usage?

11

Analytical Approach

• Analysis uses the two most recently completed panels in the study.

• Explores differences in reporting for respondents who provide insurance statement records vs those who do not.

• Examines the role of other factors such as demographics and paradata in this process

• Final goal is to examine match rates for two groups with regard to: Event reporting Cost reporting

12

Preliminary Analyses and Findings

13

Differences in Event Reporting: Full Sample

With Statements No StatementsN 4597 2964

Average Number of Events 6.3 4.3

Total Number of Providers 16.9 7.8

Total Number of Costs 114.5 32.6

Total No Statement Costs 60.8 32.6

OLS: Average # of Events (N= 5690)

B Beta

Provided Statements 4.13 .26

Average Self Reported Health 1.66 .27

R2 = .14

Full Sample: Full Regression

14


B Beta



Used Calendar 4.88 .27

Highest Degree Obtained 3.08 .11

R2 = .24OLS: Average # of Events

(N= 5266) B Beta

Ratio of Costs Covered by Statement 111.37 .44




R2 = .36

15

Differences in Event Reporting: High Event Sample (50+ Events)

With Statements No StatementsN 2964 198

Average Number of Events 10.7 8.2

Total Number of Providers 21.0 13.9

Total Number of Costs 152.7 77.2

Total No Statement Costs 76.7 76.9


B Beta



R2 = .07

High Event Sample: Full Regression

16


B Beta




Highest Degree Obtained .273 .10

R2 = .12OLS: Average # of Events

(N=2958) B Beta

Ratio of Costs Covered by Statement 92.70 .34




R2 = .21

17

Modeling the Match

18

Forthcoming Data

• Events reported through survey data are matched with CMS administrative data (when available).

• Events are combined to form a comprehensive reporting of medical events for each sample person.

• Complex algorithm used for this process that results in a source code for each event.

• Source code (survey, admin, or both), event type, and event date can be matched back to sample person to help assess measurement error.

History of Matching on the MCBS

• Goal of match to produce data more complete than either survey or billing records (Eppig and Chulis,1996).

• Administrative records are not the gold standard but provide good evidence: That a health service occurred That Medicare was a payer The amount paid by Medicare

• Survey data provide: A good record of out-of-pocket and third party payments Record of events/services not covered by or submitted to

Medicare

20

Matching Details from an Earlier Panel

• 2008 match data not yet available

• For all medical practitioner, inpatient and outpatient events in 2006 a raw match indicates: 30% of events come from survey data only 43% of events come from administrative data only 27% of events match (can be found in both)

• Inpatient events: 55% match

• Medical practitioner events: 26% match

• Outpatient events: 26% match

• Dental Visits: 0.4% match

• Not a 1 to 1 correspondence

21

Workshop Issues

• Given the rich administrative data to be available, we are interested in advice regarding the best analytical approach for assessing measurement error in: Event reporting Event detail reporting Cost reporting

• How do we best utilize administrative data that is a good source for matching a subset of survey events

Ryan A. [email protected]

Brad [email protected]

23

Logistic Regression Results: Full ModelN=5193Variable

Standardized Coeff. [B]

Odds Ratio [EXP(B)]

Ratio of costs covered by statement -3.17 .04

Ratio rounds calendar used -.39 .68

Ever use a proxy -.47 .63

SP ever on Medicaid -.62 .54

Average of self reported health .11 1.12

Eligible for VA benefits -.57 .57

Age when entered study (categorical) .09 1.05

Any interviews conducted in Spanish -.56 .57

MCBS panel .27 1.31

Same interviewer throughout study

Average interview length (min) -.02 .98

Average number of events per round .17 1.18

Average years of interviewer experience -.05 .95

burden and loss: the role of panel survey recordkeeping in self-report quality and nonresponse

Documents