1 incorporating statistical process control and statistical quality control techniques into a...
TRANSCRIPT
1
Incorporating Statistical Process Control and Statistical Quality Control Techniques
into a Quality Assurance Program
Robyn Sirkis
U.S. Census Bureau
Purpose Incorporate SPC and SQC methods into quality
assurance program
Monitor and improve interviewer performance to ensure the collection of high quality data in real time
Examine different types of quality indicators
2
Outline
Background Cluster Analysis Statistical Process Control (SPC) and Statistical
Quality Control (SQC) Methodology SPC and SQC Examples Next Steps Conclusion
3
Data for Analysis National Health Interview Survey 2008 to 2010 data
Quality indicators: include item don’t know and refusals responses
- Item nonresponse rates for the current asthma estimate, family income question, and pneumonia shot question
Quality indicators: exclude don’t know, refusals and missing responses
- Current asthma estimate: proportion of “no” responses- Family income question: average and standard deviation- Pneumonia shot question: proportion of “yes” responses
4
Cluster Analysis Compare interviewers working in similar census tracts with
similar workloads
Choose clustering variables (variable reduction)• Census 2000 Planning Database• Nine variables chosen
Create clusters within Regional Office (RO)• Group census tracts into clusters• 4 to 8 clusters within each RO
5
SPC and SQC Terminology Statistical Process Control (SPC)
• Graphical tool to measure and analyze variation in a process over time
• Control charts: use control limits and time component• Multivariate charts: reduces amount of charts for correlated
measures
Statistical Quality Control (SQC)• Broad array of statistical tools to measure and improve
operational procedures• Analysis of Means (ANOM) charts: use decision limits• ANOM charts: interviewer is an example of a rational subgroup
6
SPC and SQC Methodology Construct cluster-level control chart including
historical months data • Historical months: January 2008 to November 2010
Construct the trial control limits by removing observations outside the limits
Construct final cluster-level control chart including historical and current months data• Current month: December 2010
7
SPC and SQC Methodology Continued
Construct Analysis of Means (ANOM) charts for the current month (December 2010)
Construct interviewer-level control charts if overall average or proportion outside decision limits in ANOM chart
8
Family Income Example
Average and standard deviation of family income
Family income item nonresponse rate
SPC and SQC techniques used to improve interviewer performance in real time• Current month: December 2010
9
Analysis of Means Chart December 2010
(RO = 1, Cluster = 4)
10
UDL
LDL
1 2 3 4 5 6 7 8 9 10 11
Mea
n
Interviewer
=.1 Limits:
Group Sizes: Min n = 2 Max n = 10
X
Process Not in Control (RO = 1, Cluster = 4, Interviewer = 3)
11
UCL
LCL
1 1
S
UCL
LCLMar
2008
Apr
2008
Sep
2008
Dec
2008
Mar
2009
Apr
2009
Sep
2009
Dec
2009
Mar
2010
Apr
2010
Jun
2010
Sep
2010
Dec
2010
Ave
rage
Sta
ndar
d D
evia
tion
Month
3 Limits:
Subgroup Sizes: Min n = 2 Max n = 10
X
Current Asthma Estimate Example
Current asthma estimate uses responses from three questions
Proportion of “no” responses for current asthma estimate
Current asthma estimate item nonresponse rate
12
Analysis of Means Chart December 2010
(RO = 2, Cluster = 3)
13
=.88
UDL
LDL
1 2 3 4 5 6 7 8 9 10 11 12 13 14
0.2
0.4
0.6
0.8
1.0
Rat
e
Interviewer
=.1 Limits:
Group Sizes: Min n = 2 Max n = 13
P
Process Not in Control (RO = 2, Cluster = 3, Interviewer = 13)
14
=.89
UCL
LCL
1
Feb
2008
Mar
2008
Apr
2008
May
2008
Jun
2008
Jul
2008
Aug
2008
Sep
2008
Nov
2008
Dec
2008
Mar
2009
Apr
2009
May
2009
Jun
2009
Jul
2009
Aug
2009
Sep
2009
Oct
2009
Dec
2009
Mar
2010
Apr
2010
May
2010
Jun
2010
Jul
2010
Aug
2010
Sep
2010
Oct
2010
Nov
2010
Dec
2010
0.2
0.4
0.6
0.8
1.0
Rat
e
Month
3 Limits:
Subgroup Sizes: Min n = 2 Max n = 24
P
Summary of Charts Current Asthma Estimate
- Process not in control in interviewer-level control chart
Family Income- Average and standard deviation exceed the control
limits in interviewer-level control chart
Interviewers did not have item nonresponse rates significantly different from RO and cluster for December 2010
15
Implementation of SPC and SQC Techniques
Choose quality indicators• Multivariate charts minimize the number of charts to
monitor by combining correlated measures
Select interviewers to follow-up• Most quality indicators where process is not in control• Highest difference between control limits and quality
indicator for current month
16
Next Steps Incorporate SPC and SQC techniques into current
quality assurance program
Current quality assurance program: conduct second interview to determine if interviewers conducting interview in accordance with established procedures
Current sampling method: random and supplemental• Stratified random sampling• Supplemental sampling: additional cases and interviewers
identified by Headquarters and the Regional Office
17
Next Steps Continued
Add targeted sampling method• Target interview cases which bear hallmarks of suspected
falsification
Choose quality indicators to select cases for targeted sampling method• Use statistical models to choose indicators which predict
potential data quality issues• Includes using digit frequency and length of field techniques• Includes indicators from PANDA system such as interview times
and contact attempts
18
Next Steps Continued
Choose methods for targeted sampling method• SPC and SQC techniques• Statistical tests such as chi-square test to compare distributions• Unlikely response pattern analysis
Use weighting equations to leverage best available methods and measures to optimize detection of potential data quality issues
Select additional interviewers just using SPC and SQC techniques and then other interviewers using different methods
19
Conclusion Crucial to monitor both item nonresponse rates and
other quality indicators
Develop method to identify interviewers whose work should be investigated given the available amount of resources for follow-up purposes
20
