National Immunization Survey:

Data Quality and Public-Use Data Files

Meena Khare, National Center for Health Statistics

Michael P. Battaglia, Abt Associates Inc.

July 16, 2002

Outline

National Immunization Survey

Quality of the data

Confidentiality issues

Public Use Data Files (PUF)

Summary

2

National Immunization Survey (NIS) Large ongoing RDD survey, conducted by CDC since 1994

Measures vaccination coverage among children aged 19-35 months at National, State, and Urban area levels (78 IAP areas)

Monitors Healthy People 2000 and 2010 goals of immunization coverage >>90% Coverage: 4DTP, 3Polio, 1MMR, 3HepB, 90% Coverage: 4DTP, 3Polio, 1MMR, 3HepB,

3Hib, and 4:3:1:3 series3Hib, and 4:3:1:3 series

Monitors introduction of new vaccines

<4% of households in the U.S. contain child of age 19-35 months

3

NIS Operations

N o Co n se n t o r n o p rov id er in fo

4 Q T R E s tim a tes

In te rn a l A n a lytic F ileC h ild re n w ith ad e qu a te P ro vid e r D a ta o n ly

F in a l P UF

D R B /co nfid e n tia lity R e v iew

C h ild re n w ith com p le te d ho u seh o ld in te rv ie wsa n d a de qu ate p rov ide r da ta

Q C Ch e ck/ D a ta p roce ss in g / M a tch in g S h e e t re v iew

IH Q R e tu rn e d w ith d a ta IH Q N o n re sp o n sem iss in g p ro v id er d a ta

P ro v ide r R eco rd Ch e ck S tu dyIH Q M a iled

W ith C o nse n t a n d w ith p rov id er in fo

H o u se h o ld In te rv iewC o m ple te d in te rv ie ws

C A T I S c re e n er

N ISR D D S a m p le

4

NIS Data Collection

Household CATI Screener and Interview Household CATI Screener and Interview Parent/Guardian Parent/Guardian (most knowledgeable person)(most knowledgeable person) Socio-demographic information: Socio-demographic information: mother and mother and

childchild Shot card use or memory recall Shot card use or memory recall Vaccination dates (shot card only)Vaccination dates (shot card only) Provider’s contact information with consentProvider’s contact information with consent

Provider Record Check Study (mailed Provider Record Check Study (mailed IHQ)IHQ) Provider’s office (e.g., staff, nurse, manager)Provider’s office (e.g., staff, nurse, manager) Mail, Fax, Telephone Mail, Fax, Telephone Completed IHQ or copy of medical recordsCompleted IHQ or copy of medical records Matched on DOB, Gender, NameMatched on DOB, Gender, Name Vaccination historyVaccination history Provider’s informationProvider’s information

5

Goals of Quality Control in the NIS

Evaluate and improve quality of Evaluate and improve quality of the information collectedthe information collected

Reduce/eliminate discrepanciesReduce/eliminate discrepancies

Improve quality of the estimatesImprove quality of the estimates

Reduce potential bias in official Reduce potential bias in official estimates of vaccination estimates of vaccination coverage ratescoverage rates

7

Quality Control Procedures used in the NIS

Extensive monitoring and QC review of Extensive monitoring and QC review of the data collected from CATI and the data collected from CATI and mailed IHQsmailed IHQs

Automated edit software for processing Automated edit software for processing combined household and provider data combined household and provider data

Manual ‘Matching Sheet’ review of Manual ‘Matching Sheet’ review of discrepant cases discrepant cases

Resolution by using guidelines Resolution by using guidelines developed by expert immunization developed by expert immunization program managers program managers

8

Review of Data Errors in the IHQ

Data Entry ValidationData Entry Validation Double data entryDouble data entry Approximately 400 forms Approximately 400 forms

reviewed/quarterreviewed/quarter Error rates ranged from 0.23% - 0.67% Error rates ranged from 0.23% - 0.67%

(~30,000 fields with data, NIS Q4/1999-(~30,000 fields with data, NIS Q4/1999-Q4/2001)Q4/2001)

~350-1000 manual Matching Sheet ~350-1000 manual Matching Sheet reviews/quarter for discrepant datareviews/quarter for discrepant data

Most errors in dates are found in the Most errors in dates are found in the original provider-reported recordsoriginal provider-reported records

<1% children had data errors <1% children had data errors 9

Observed Discrepancies in Dates

DOBDOB Between household (HH) and Between household (HH) and

provider(s)provider(s) Between multiple providers for a Between multiple providers for a

childchild

Shot datesShot dates Between HH and provider(s)Between HH and provider(s) Within a single providerWithin a single provider Between multiple providers for a Between multiple providers for a

childchild

10

DOB and Shot datesDOB and Shot dates 122 children classified as age-ineligible122 children classified as age-ineligible 69 IHQs filled out for the ‘wrong’ child69 IHQs filled out for the ‘wrong’ child 21 children had changes in best DOB21 children had changes in best DOB 2163 children had shot dates before DOB 2163 children had shot dates before DOB

or vaccination interval < 1 month, or other or vaccination interval < 1 month, or other shot date editsshot date edits

Hepatitis B ‘birth’ doseHepatitis B ‘birth’ dose 361 children had missing provider ‘birth 361 children had missing provider ‘birth

dose' of Hepatitis B and providers had dose' of Hepatitis B and providers had checked the ‘given at birth’ box on IHQchecked the ‘given at birth’ box on IHQ

34 children had imputed dates from the household 34 children had imputed dates from the household shot cardshot card

327 children had imputed dates from the distribution 327 children had imputed dates from the distribution of provider-reported dates for the birth doseof provider-reported dates for the birth dose

Matching Sheet Review, 2000 NIS

11

Results of Quality Control Evaluation

No single data source is 100% complete and No single data source is 100% complete and accurateaccurate

Some discrepancies in DOB and vaccination Some discrepancies in DOB and vaccination dates/number of doses could not be verified and dates/number of doses could not be verified and remained after edits remained after edits

Number of discrepancies increased if more than Number of discrepancies increased if more than one immunization provider per child (1.35 one immunization provider per child (1.35 providers per child in 1999 NIS ) was identified providers per child in 1999 NIS ) was identified

Errors in Errors in Original Immunization records Original Immunization records Locating records for the correct child Locating records for the correct child Transcribing shot dates to the IHQTranscribing shot dates to the IHQ Illegible dates, transposed dates, missing day/month, Illegible dates, transposed dates, missing day/month,

incorrect dates (especially year)incorrect dates (especially year) Shot date interval< 1 month, or shot dates<DOBShot date interval< 1 month, or shot dates<DOB Missing ‘first’ Hepatitis B dose given at birthMissing ‘first’ Hepatitis B dose given at birth 12

‘Best’ Vaccination Value Research

Currently, coverage estimates from the Currently, coverage estimates from the providerprovider data are used as data are used as official official estimates estimates

The goal of ‘best’ value research is to The goal of ‘best’ value research is to obtain the most complete and accurate obtain the most complete and accurate immunization data possible from immunization data possible from twotwo sourcessources Supplement incomplete provider data with Supplement incomplete provider data with

household data for children who are UTD from household data for children who are UTD from ‘shot card’ ‘shot card’

Substitute completely missing provider data Substitute completely missing provider data with household data for children who are with household data for children who are 4:3:1:3 UTD from ‘shot card’ 4:3:1:3 UTD from ‘shot card’

13

NIS 2000 Sample

35,960 households with eligible 35,960 households with eligible children 19-35 monthschildren 19-35 months

33,477 (93.1%)of eligible households 33,477 (93.1%)of eligible households with completed interviewswith completed interviews

34,087 age-eligible children with 34,087 age-eligible children with completed interviewscompleted interviews

22,958 (67.4%) children with 22,958 (67.4%) children with completed interviews and ‘adequate’ completed interviews and ‘adequate’ provider dataprovider data

14

Sources of Immunization Histories, 2000 NIS

SourceSource Household data from Household data from Shot Card Shot Card

‘‘Adequate’ Adequate’ Provider Provider data data

YesYes NONO TotalTotal

nn %% nn %% nn %%

YesYes 11,911,96363

73.873.8 10,910,99595

61.561.5 22,922,95858

67.467.4

NoNo 4,244,2499

26.226.2 6,886,8800

38.538.5 11,111,12929

32.632.6

TotalTotal 16,216,21212

47.647.6**

17,817,87575

52.452.4**

34,034,08787

100.100.00

* row percents15

Assessment of Provider-Reported 4:3:1:3* Coverage Estimates among Children with

Household Data,2000 NIS

Source of HH reportsSource of HH reportsand measures of qualityand measures of quality

Shot CardShot Card ‘‘Memory’ Memory’ Recall**Recall**

Sample Size Sample Size 11,96311,963 5,5265,526

SensitivitySensitivity 67.167.1 44.844.8

SpecificitySpecificity 61.861.8 65.365.3

Misclassification (prov)Misclassification (prov)(HH UTD)/ Prov NUTD)(HH UTD)/ Prov NUTD)

7.47.4 8.78.7

Overall agreementOverall agreement 66.166.1 49.949.9

Overall disagreementOverall disagreement 33.933.9 50.150.1

Net Difference (HH Cov - Prov Net Difference (HH Cov - Prov cov)cov)

-19.1-19.1 -32.7-32.7

*4DTP/3Polio/1MCV/3Hib** Other 5,469 non-shot card children have missing 4:3:1:3 UTD status

16

Comparison of Vaccine-specific Coverage Rates by Source of

Immunization Histories, 2000 NIS

17

0

20

40

60

80

100

4+ DTP 3+ POLIO 1+ MCV 3+ HIB 3+ Hep B 1+ VRC 4:3:1 4:3:1:3

Type of Vaccine

Co

vera

ge

rate

(%

)

Household Provider Best Value Revised Best Value

Best Value: Combined Provider and household ‘shot card’ data

Revised Best Value: After adjusting Best Value for missing provider data among non-shot card children

NIS Data Files and Estimates

In-house analytic filesIn-house analytic filesPublic-use data files (PUF )Public-use data files (PUF )National, State, and Urban area National, State, and Urban area

level estimates of vaccination level estimates of vaccination coveragecoverage Internet: Internet: www.cdc.gov/nip/coveragewww.cdc.gov/nip/coverage MMWR MMWR Journal ArticlesJournal Articles Articles in ProceedingsArticles in Proceedings Presentations at National and State Presentations at National and State

ConferencesConferences Publications are listed on NIS Web sitePublications are listed on NIS Web site

18

Confidentiality Issues Legal obligation to protect confidentiality of Legal obligation to protect confidentiality of

respondents and reduce risk of disclosurerespondents and reduce risk of disclosure Sample size (4 quarters)Sample size (4 quarters)

~ 440 children per IAP with interview data~ 440 children per IAP with interview data ~ 290 children per IAP with provider data ~ 290 children per IAP with provider data

Only telephone households (90% HHs with Only telephone households (90% HHs with telephone)telephone)

Rare population (children 19-35 months)Rare population (children 19-35 months) Geographic identifiersGeographic identifiers

State, Urban AreasState, Urban Areas Census RegionCensus Region Smaller geographic areas (e.g., Zip code, county)Smaller geographic areas (e.g., Zip code, county)

Socio-demographic detailsSocio-demographic details Analytic cells and cell sizes (within IAP area)Analytic cells and cell sizes (within IAP area)

Age, gender, race/ethnicityAge, gender, race/ethnicity WIC participation, Poverty statusWIC participation, Poverty status Minimum population sizeMinimum population size>5 in unique cells5 in unique cells

Date of birth, shot dates, interview datesDate of birth, shot dates, interview dates Clearance by the NCHS Disclosure Review Clearance by the NCHS Disclosure Review

Board (DRB)Board (DRB) 19

The NIS PUF

Public-use data files (1995-2000): Child-Public-use data files (1995-2000): Child-level records with 78 IAP area (state and level records with 78 IAP area (state and urban) identifiers urban) identifiers (2001 NIS PUF by Fall 2002)(2001 NIS PUF by Fall 2002)

Approximately 35,000 age-eligible children Approximately 35,000 age-eligible children with household interview datawith household interview data

Approximately 23,000 children with Approximately 23,000 children with household interview household interview andand ‘adequate’ ‘adequate’ provider data; on average 295 children per provider data; on average 295 children per IAP area IAP area

PUFs released on the Internet and CD-PUFs released on the Internet and CD-ROMsROMs Internet : Internet : www.www.cdccdc..govgov//nisnis (1995-2000) (1995-2000) CD_ROMs No 1-5CD_ROMs No 1-5

20

Contents of The NIS PUF

Household Interview data itemsHousehold Interview data items Unique sequence ID, age at interview Unique sequence ID, age at interview

(months), gender, race/ethnicity, …(months), gender, race/ethnicity, … Mother’s age group, education, marital status, Mother’s age group, education, marital status,

…… WIC participation, income, poverty status,…WIC participation, income, poverty status,… Vaccine specific immunization history Vaccine specific immunization history

(#doses, up-to-date status) (#doses, up-to-date status) IAP area and State identifiers, Census Region IAP area and State identifiers, Census Region

Provider data itemsProvider data items Vaccine-specific Immunization History: Vaccine-specific Immunization History: age at vaccination(days/months), number of age at vaccination(days/months), number of

doses, up-to-date status (UTD)doses, up-to-date status (UTD) Providers’ information (e.g., facility type, VFC Providers’ information (e.g., facility type, VFC

participation)participation)

21

Contents of NIS PUF (cont.)

Composite vaccine-specific and vaccine Composite vaccine-specific and vaccine series flags with up-to-date statusseries flags with up-to-date status

Sampling weights and key analytic Sampling weights and key analytic variablesvariables Sampling weights: Sampling weights: Children with household interview (HY_WGT)Children with household interview (HY_WGT) Children with Provider data(W0)Children with Provider data(W0) Stratum + PSU identification variablesStratum + PSU identification variables

22

SummaryComplicated Childhood Immunization Complicated Childhood Immunization

Schedule Schedule

No single source is 100% complete or accurateNo single source is 100% complete or accurate

<1% children were observed with <1% children were observed with discrepancies in household and/or provider discrepancies in household and/or provider reported datareported data

Data reported from written ‘shot card’ are of Data reported from written ‘shot card’ are of high quality; Shot card use varies by State/IAP high quality; Shot card use varies by State/IAP areaarea

High vaccine-specific agreement in the UTD High vaccine-specific agreement in the UTD status between HH ‘shot card’ and provider status between HH ‘shot card’ and provider report(s) whenreport(s) when HH reported UTD HH reported UTD

23

Substantial underreporting in vaccination Substantial underreporting in vaccination coverage rates from the household coverage rates from the household reportsreports

Strongly supports the use of provider-Strongly supports the use of provider-reported immunization histories in reported immunization histories in computing vaccination coverage computing vaccination coverage estimatesestimates

Official estimates from provider reports Official estimates from provider reports may slightly underestimate coverage may slightly underestimate coverage rates due to missing provider data; rates due to missing provider data; however, provider data produce however, provider data produce comparable estimates across IAP areascomparable estimates across IAP areas

24

Summary

Visit the Web Sites

For information on the NIS For information on the NIS PUF and coverage estimates PUF and coverage estimates http://www.cdc.gov/nishttp://www.cdc.gov/nishttp://www.cdc.gov/nip/coverahttp://www.cdc.gov/nip/covera

gege