incorporating statistical process control and statistical ... analysis ... spc and sqc examples next...

21
1 Incorporating Statistical Process Control and Statistical Quality Control Techniques into a Quality Assurance Program Robyn Sirkis U.S. Census Bureau

Upload: dinhxuyen

Post on 10-Mar-2018

235 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

1

Incorporating Statistical Process Control

and Statistical Quality Control Techniques into a Quality Assurance Program

Robyn Sirkis U.S. Census Bureau

Page 2: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Purpose Incorporate SPC and SQC methods into quality

assurance program

Monitor and improve interviewer performance to ensure the collection of high quality data in real time

Examine different types of quality indicators

2

Page 3: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Outline

Background Cluster Analysis Statistical Process Control (SPC) and Statistical

Quality Control (SQC) Methodology SPC and SQC Examples Next Steps Conclusion

3

Page 4: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Data for Analysis National Health Interview Survey 2008 to 2010 data Quality indicators: include item don’t know and refusals

responses - Item nonresponse rates for the current asthma estimate,

family income question, and pneumonia shot question

Quality indicators: exclude don’t know, refusals and missing responses

- Current asthma estimate: proportion of “no” responses - Family income question: average and standard deviation - Pneumonia shot question: proportion of “yes” responses

4

Page 5: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Cluster Analysis Compare interviewers working in similar census tracts with

similar workloads Choose clustering variables (variable reduction)

• Census 2000 Planning Database • Nine variables chosen

Create clusters within Regional Office (RO) • Group census tracts into clusters • 4 to 8 clusters within each RO

5

Page 6: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

SPC and SQC Terminology Statistical Process Control (SPC)

• Graphical tool to measure and analyze variation in a process over time

• Control charts: use control limits and time component • Multivariate charts: reduces amount of charts for correlated

measures

Statistical Quality Control (SQC) • Broad array of statistical tools to measure and improve

operational procedures • Analysis of Means (ANOM) charts: use decision limits • ANOM charts: interviewer is an example of a rational subgroup

6

Page 7: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

SPC and SQC Methodology

Construct cluster-level control chart including historical months data • Historical months: January 2008 to November 2010

Construct the trial control limits by removing observations outside the limits

Construct final cluster-level control chart

including historical and current months data • Current month: December 2010

7

Page 8: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

SPC and SQC Methodology Continued

Construct Analysis of Means (ANOM) charts for the

current month (December 2010)

Construct interviewer-level control charts if overall average or proportion outside decision limits in ANOM chart

8

Page 9: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Family Income Example

Average and standard deviation of family income Family income item nonresponse rate

SPC and SQC techniques used to improve

interviewer performance in real time • Current month: December 2010

9

Page 10: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Analysis of Means Chart

December 2010 (RO = 1, Cluster = 4)

10

UDL

LDL

1 2 3 4 5 6 7 8 9 10 11

Mea

n

Interviewer

=.1 Limits:

Group Sizes: Min n = 2 Max n = 10

X

α

Page 11: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Process Not in Control

(RO = 1, Cluster = 4, Interviewer = 3)

11

UCL

LCL

1 1

S UCL

LCL M a r 2 0 0 8

A p r 2 0 0 8

S e p 2 0 0 8

D e c 2 0 0 8

M a r 2 0 0 9

A p r 2 0 0 9

S e p 2 0 0 9

D e c 2 0 0 9

M a r 2 0 1 0

A p r 2 0 1 0

J u n 2 0 1 0

S e p 2 0 1 0

D e c 2 0 1 0

Aver

age

Stan

dard

Dev

iatio

n

Month

3 Limits:

Subgroup Sizes: Min n = 2 Max n = 10

X

σ

Page 12: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Current Asthma Estimate Example

Current asthma estimate uses responses from three questions

Proportion of “no” responses for current asthma

estimate Current asthma estimate item nonresponse rate

12

Page 13: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Analysis of Means Chart

December 2010 (RO = 2, Cluster = 3)

13

=.88

UDL

LDL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 0.2

0.4

0.6

0.8

1.0

Rat

e

Interviewer

=.1 Limits:

Group Sizes: Min n = 2 Max n = 13

P

α

Page 14: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Process Not in Control

(RO = 2, Cluster = 3, Interviewer = 13)

14

=.89

UCL

LCL 1

F e b

2 0 0 8

M a r

2 0 0 8

A p r

2 0 0 8

M a y

2 0 0 8

J u n

2 0 0 8

J u l

2 0 0 8

A u g

2 0 0 8

S e p

2 0 0 8

N o v

2 0 0 8

D e c

2 0 0 8

M a r

2 0 0 9

A p r

2 0 0 9

M a y

2 0 0 9

J u n

2 0 0 9

J u l

2 0 0 9

A u g

2 0 0 9

S e p

2 0 0 9

O c t

2 0 0 9

D e c

2 0 0 9

M a r

2 0 1 0

A p r

2 0 1 0

M a y

2 0 1 0

J u n

2 0 1 0

J u l

2 0 1 0

A u g

2 0 1 0

S e p

2 0 1 0

O c t

2 0 1 0

N o v

2 0 1 0

D e c

2 0 1 0

0.2

0.4

0.6

0.8

1.0

Rat

e

Month

3 Limits:

Subgroup Sizes: Min n = 2 Max n = 24

P

σ

Page 15: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Summary of Charts Current Asthma Estimate

- Process not in control in interviewer-level control chart

Family Income - Average and standard deviation exceed the control

limits in interviewer-level control chart

Interviewers did not have item nonresponse rates significantly different from RO and cluster for December 2010

15

Page 16: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Implementation of SPC and SQC Techniques

Choose quality indicators • Multivariate charts minimize the number of charts to

monitor by combining correlated measures

Select interviewers to follow-up • Most quality indicators where process is not in control • Highest difference between control limits and quality

indicator for current month

16

Page 17: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Next Steps

Incorporate SPC and SQC techniques into current quality assurance program

Current quality assurance program: conduct second interview to determine if interviewers conducting interview in accordance with established procedures

Current sampling method: random and supplemental • Stratified random sampling • Supplemental sampling: additional cases and interviewers

identified by Headquarters and the Regional Office

17

Page 18: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Next Steps Continued

Add targeted sampling method • Target interview cases which bear hallmarks of suspected

falsification

Choose quality indicators to select cases for targeted sampling method • Use statistical models to choose indicators which predict

potential data quality issues • Includes using digit frequency and length of field techniques • Includes indicators from PANDA system such as interview times

and contact attempts

18

Page 19: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Next Steps Continued

Choose methods for targeted sampling method • SPC and SQC techniques • Statistical tests such as chi-square test to compare distributions • Unlikely response pattern analysis

Use weighting equations to leverage best available

methods and measures to optimize detection of potential data quality issues

Select additional interviewers just using SPC and SQC techniques and then other interviewers using different methods

19

Page 20: Incorporating Statistical Process Control and Statistical ... Analysis ... SPC and SQC Examples Next Steps ... Incorporating Statistical Process Control and Statistical Quality Control

Conclusion Crucial to monitor both item nonresponse rates and

other quality indicators Develop method to identify interviewers whose work

should be investigated given the available amount of resources for follow-up purposes

20