Controlling Control Charts Interpreting p-valuesIntermediate Statistics for ICPs
Ona Montgomery RN, BSN, MSHA, CICAnne Denison, RN, BSN, MSTexas Society of Infection Control PractitionersOctober 2006
Objectives• Discuss the basic principles of
epidemiology as they apply to monitoring of infections in a health care setting.
• Differentiate between common cause variation and special cause variation.
• Interpret statistical significance through the use of control charts and p-values.
• Relate the use of statistical analysis to action plans and intervention strategies.
Epidemiology Basic Principles
Why Epidemiology?
• Systematic way to look at health problems
• The focus is on defining a problem in order to focus on prevention– Understand the cause of disease– Plan interventions– Evaluate preventive measures
• Applies to diverse situations, flexible
Epidemiology Key Points
• ‘distribution’ – frequency of disease in a specific population
• ‘determinants’ – factors or events associated with changes in health outcomes
• ‘population’ – a group of people rather than an individual
Epidemiological Perspective• Prevention of osteoporotic hip
fracture– Medical / Clinical model
• Target persons at risk based on low BMD• Intervention: pharmacotherapy
– Epidemiological model• Target populations at risk• Intervention: broad based changes in
health behaviors of populations,
exercise, smoking
Infectious Disease Process
• The chain of infection– Causative agent– Reservoir– Portal of exit– Mode of transmission– Portal of entry– Susceptible host
Why determine causality?
• Identify points where disease process can be interrupted
• Develop prevention and control efforts that decrease the outcome measured
• Identify the natural history of disease
Numerators and Denominators
Numerator - the outcome or process you will “count”
Denominator - “out of how many”
Must be clearly defined (and written down)
Case Definitions• Internally created
– Especially useful in outbreak situation where the organism has not been identified – Eg. Gastroenteritis or respiratory illness in a nursing home
• Surveillance definitions ≠ clinical diagnoses
• Clearly define both the numerator AND denominator
• Choose carefully, difficult to modify mid-stream
Case Definition• National Nosocomial Infection Surveillance (NNIS)• http://www.cdc.gov/ncidod/dhqp/nnis_pubs.html• Standardized criteria and case definitions
– Clinical signs and symptoms– Laboratory results– Physician actions/diagnoses
• Applies to inpatient hospitals• McGeer et al.
– Long term care setting– Rehabilitation setting
CDC Acute Care Definition: Symptomatic Urinary Tract Infection
Criterion 1: Patient must have
– 1 of the following: • fever, urgency, frequency, dysuria or
suprapubic tenderness - AND -– Positive urine culture with > 100,000
col/ml with < 2 species
UTI definition continued
Criterion 2: Patient must have– 2 of the following:
• fever, urgency, frequency, dysuria or suprapubic tenderness
- AND -– 1 of the following:
• positive dipstick test • pyuria (>10 WBC/cc) • organisms seen on gram stain of unspun
urine• 2 urine cultures, same organism, > 10,000
col/ml, in non voided specimens• Physician’s diagnosis/ initiation of
antimicrobial therapy
Device Associated Rates
• Numerator = events• Denominator = device days
– Catheter associated UTIs per 1000 Foley days
• 4 UTIs / 235 days x 1000 = 17 CUTI per 1000 Foley days
• CL-BSI per 1000 CL days• VAP per 1000 ventilator days
http://www.cdc.gov/ncidod/hip/nhsn/members/PSProtocolsMay06.pdf
Apply Risk Stratification Methods
Risk stratification simply means subdividing (stratifying) your surveillance population into groups at similar levels of infection risk prior to performing any analyses or comparisons.
To ensure comparing “apples to apples”
CDC NNIS Risk Index for SSI Surveillance
Patient-specific Risk Score Total 0-3 points
Wound class class III or IV 1 point
ASA score 3, 4, 5 1 point
Duration of surgery > cutpoint 1 point
SSI – Wound Class vs NNIS Class
Wound Class All NNIS 0 NNIS 1 NNIS 2 NNIS 3
Clean 2.1% 1.0% 2.3% 5.4% N/A Cl /Contam 3.3% 2.1% 4.0% 9.5% N/AContaminated 6.4% N/A 3.4% 6.8% 13.2%Dirty infected 7.1% N/A 3.1% 8.1% 12.8%
All 2.8% 1.5% 2.9% 6.8% 13.0%
NNIS. CDC. Am J Infect Control. 2001;29:404-421.
CENTERS FOR DISEASE CONTROL
AND PREVENTION
Determining the NNIS Risk IndexCategory in 3 Patients
Determining the NNIS Risk IndexCategory in 3 Patients
Elements of theNNIS Risk Index
Operation >t hours
Wound class
ASA class
NNIS Risk Category
Patient 1
Yes
Dirty
4
Patient 2
No
Clean
2
Patient 3
Yes
Clean-contaminated
2
Infection Rate And Timing of Prophylactic Antibiotics
0
0.5
1
1.5
2
2.5
3
3.5
4
<3 -2 -1 0 1 2 3 4 5
14/369
5/699
5/1009
2/180
1/81
1/41 1/47
14/441
Classen DC et al, NEJM, 1992
CDC NNIS Risk Stratification for High Risk Nursery (HRN)
Surveillance
Stratification by Birthweight Categories:
• </= 1000 grams
• 1001-1500 grams
• 1501-2500 grams
• >2500 grams
Interpretation of this surveillance data
• Compare to NISS report risk class by strata
• example
On the UP side, you are the healthiest patient in ICU
BSI Rates in MICU and SICU
0
2
4
6
8
10
12
14
16
18
MICUNNIS MICU MedianSICUNNIS SICU Median
See: Am J Infect Control 2002;30:458-75.
NNIS 90th percentile
NNIS 10th percentile
BSI Rates in MICU and SICU
0
2
4
6
8
10
12
14
16
18
MICUNNIS MICU MedianSICUNNIS SICU Median
See: Am J Infect Control 2002;30:458-75.
NNIS 90th percentile
NNIS 10th percentile
Surgical site rate analysis
EXAMPLE OF CALCULATING RISK-ADJUSTED RATES
Colon surgery
# Risk Factors #SSI #Operations Rate %
0 2 48 4.17%1 5 77 6.49%2 4 39 10.26%3 1 5 20.00%TOTAL 12 169 --------
Pro
cedu
re
Risk In
dex
Categ
ory
10% 25% 50% 75% 90%
Ho
spital X
YZ
Spinal Fusion 0 0 0 0.7 1.4 2.5 1.2
Spinal Fusion 1 0 0.8 2.2 3.5 4.7 2.6
Spinal Fusion 2,3 0 2.3 4.8 7.3 10.2 18.8
• Ratios• Proportions• Crude Rates• Adjusted
Rates• Incidence• Prevalence• Attack Rates• Mean • Median
Control Chart Theory
• Looking at a system / population
• Individual measurements are unpredictable
• BUT, if all observations are from a stable common system, as an aggregate they will follow a predictable pattern of distribution
Gaussian Distribution Function‘Normal curve’
Normal pattern of common cause variation
Standard Deviations
Variation
Types of variation
• Common cause variation– Part of the natural process– Always present– Partially unknown– Difficult to control
Types of variation
• Special cause variation– Larger variation– Special or non-typical event– Easier to pinpoint in time– Sentinel event
Sources of Variation
• People
• Machines
• Materials
• Methods
• Bias
Both kinds of variation are important in a health care setting
• Monitor common cause variation – Look for non random patterns that
may indicate positive or negative trends
• Monitor special cause variation – Identify critical system errors and
analyze to prevent recurrence
Control Charts, Components
• Data in the form of a line graph
• Statistical parameters– Mean– Upper Control Limit– Lower Control Limit
Control Chart
UCL
MEAN
LCL
Standard deviation= square root of the variance
Or let Excel do it for you!!Insert
Function ‘Select a function’ = STDEV
Art and Science
• Surveillance, tracking and plotting the data = Science
• Interpreting the data for effective and appropriate response = Art
• Achieve a balance / flexibility– Avoid tampering – React appropriately
Tampering • Identify a ‘trend’ where there is none
• Try to explain natural variation as a special event
• Blame or credit people for processes they have no control over
• Makes it difficult to understand past processes
• Makes it difficult to plan future priorities and interventions
Spotting Special Cause Variation
• Non-random patterns – Points more that 3 SDs from mean– Two of 3 successive points more than 2
SDs from mean– Four of 5 successive points more than 1
SDs from mean– Eight successive points on one side of the
center line – Six successive points increasing or
decreasing (trend)
P-Charts, G- Charts
• P-Charts track percentage data– Surgical infection rates into control chart
• G-Charts track days between events– Number of days between on the job
accidents – Good for tracking rare occurrences
(like VRE infections – I hope)– Real time – ‘Early warning system’
0
50
100
150
200
250
300
350
03/0
1/01
06/0
6/01
08/2
1/01
12/2
0/01
05/0
6/02
07/1
8/02
08/2
3/02
10/2
1/02
11/0
5/02
02/1
9/03
04/0
2/03
06/1
0/03
09/0
4/03
04/2
1/04
09/2
7/04
11/1
7/04
12/0
2/04
12/0
8/04
01/1
1/05
03/0
9/05
08/0
9/05
08/1
6/05
08/2
4/05
09/1
4/05
01/0
4/06
01/1
1/06
02/2
2/06
Date of Surgery
# Ca
ses
betw
een
Mean = 86
Post-op PneumoniaTrend Analysis
mean calc based on FY01 - FY04
GOOD
• Data are the raw materials
• Statistical analysis provides the tools
• Use the data analysis to drive intervention
Relate the use of statistical analysis to action plans and intervention strategies
Only You Can Prevent Pneumonia
Walk, Walk, Walk Use your incentive spirometry (deep
breathing device)
Breathe deeply, turn and cough
P-Values
• P stands for PROBABILITY• It is a proportion of times a particular
event will occur in a series of repeated trials
• Quantifies degree of uncertainty about our data
• Range is from 0 to 1– 0 is no probability– 1 is 100% probability
Hypothesis testing
• Looking for a difference between 2 populations
• Question: Does eating ice cream cause heart attacks/– Null hypothesis = Any difference between ice
cream eater and non-ice cream eaters is a result of chance
– Alternative hypothesis = People who eat ice cream have more heart attacks
Significance
• If the p-value is very small you have strong evidence against the null hypothesis
• A result that is statistically significant is one that has a very small probability of happening just by chance or coincidence
Significance ParametersHow much difference is enough?
• Types of Error – Type I error (α) (significance level)
• ‘False Alarm’ • Found a difference but there really was none
– Type II error (β) • ‘Missed detection’ • Found no difference where one really existed• May be a result of too small of a sample size
Interpretation of the p-value
• Determine that the difference is statistically significant (reject the null hypothesis) if the p-value is smaller or equal to the significance level (α) – [Usually 0.05]
• If statistic is greater than the significance level, conclude there is no statistically significant difference between the two populations
Significance ≠ Proof
• Take any statistically significant results with a grain of salt
• How was the sample selected?
• How big was the sample?
• Were there confounding effects
Table A.4 Health behavior determinants of osteoporotic hip fracture: Univariate comparison of cases and controls
Health Behavior Characteristics
Sample N
Sample %
Control %
Case %
p-valu
e
Smoke now? 0.147
Yes 45 13.04 11.21 16.81
No 300 86.96 88.79 83.19
Exercise more than 2 times per week? <0.001
Yes 143 42.31 50.88 24.55
No 195 57.69 49.12 75.45
Drink any alcohol? 0.012
Yes 86 25.44 29.65 16.96
No 252 74.56 70.35 83.04
Confidence Interval• Margin of error in your data
• Usually set at 95%
• 95% likelihood that the true value of a statistic lies within the upper and lower 95%CI
• Wide is bad, narrow is good
• Factors that influence CI– Sample size– Amount of variability in the data
95% Confidence interval
The Odds Ratio (OR) for an infant dying because of Sudden Infant Death Syndrome was found to be 2.5 for households where the mother smoked cigarettes. The p value was < .05. Three studies found same OR but different CI.
• Study #1 showed an OR of 2.5
–95% CI from 2.4-2.6
• Study #2 showed an OR of 2.5
–95% CI from 1.4-3.6
• Study #3 showed an OR of 2.5
–95% CI from 0.4-5.6
Table A.8 Multiple logistic regression analysis of health behavior determinants of osteoporotic hip fracture (n=315)
Health Behavior Characteristics
Adjusted OR* (95%CI)
Current smoker
No 1.00
Yes2.99 (1.30,
6.86)
Exercise more than 2 times per week?
No 1.00
Yes0.64 (0.35,
1.20)
Drink any alcohol
No 1.00
Yes0.71 (0.35,
1.44)
* adjusted for age/fragility
P-values are just tools
• They help you assess the ‘merchandise’– You would not buy an avocado without
squeezing it to see if it is ripe
• Not all data are collected and analyzed correctly and fairly
• Look beyond P-values– What was the confidence interval?– What was the N?– Random sample or Convenience sample
Good Data
• Reliable– Reproducible results
• Unbiased– No systematic errors in study design and
sample selection
• Valid– The data measure what they are supposed
to measure
Top Ten Statistical Mistakes• #1 Misleading Graphs
84
85
86
87
88
89
90
Unit A Unit B Unit C Unit D
per
cen
t
0
20
40
60
80
100
Unit A Unit B Unit C Unit D
per
cen
t
Percent of Patient Care Employees with Influenza Vaccination, 2006
• #2 Biased Data– Instrument not calibrated properly– Participants influenced by the way questions
are asked– Sample does not represent the total
population of interest– The researcher is not objective
Top Ten Statistical Mistakes
Top Ten Statistical Mistakes
• #3 No margin of error reported– 10 + 1.2
• Range 8.8 to 11.2
– 10 + 6.4• Range 3.6 to 16.4
• #4 Non-Random Samples
• #5 Missing Sample sizes– What’s the N?
Top Ten Statistical Mistakes
• #6 Misinterpreted correlation– Correlation does not automatically
mean cause and effect
• #7 Confounding variables• #8 Botched Numbers• #9 Selectively reporting results
– Data fishing
• #10 Anecdote– Show me the data!’