Quality Control – Part II
Tim Hanley
EPA Office of Air Quality Planning and Standards
2
Error
…the difference between your answer and the “truth”
3
Continuous PM QC
Instrument Stability Checks (EPA calls this general QA in their 1995 memo; see Attachment A in the TAMS template QAPPs)
Precision (wiggle)
Bias (jump)
4
Instrument Stability
Internal diagnostics conducted with each measurement (no error codes)TEOM uses standard filter kit available from dealer BAM uses standard foil or internal sourceUse kits more frequently at beginning of program, thereafter as necessary and at least twice a year
5
Results of Stability Checks
Use dealer’s guidelines
Within 2.5% for TEOM standard filter
Within 7% for BAM foil
Plot results on a control chart (can also plot other parameters on a control chart)
6
Verifying the accuracy of your Transfer Standard
Option 1: send standard to manufacturer, who has NIST-traceable equipment; they send you back your standard with certificate of traceability
Option 2: use natural standards, such as freezing ice water for temperature, carefully document your calibration according to a standard method
7
Single (One) Point Verifications
#Verification occurs when you conduct check using one flow rate or concentration or value
#Span is an example of one-point check, meaning spanning from zero up to a range
#Verifications DO NOT involve any changes to your equipment
8
Multi-Point Verifications
Check of response of instrument to more than one flow rate, concentration, or value
Usually zero, a low level, and at the upper end of the range expected to be measured
9
Example of Verification
Perform check of equipmentFlow rate for PM
Concentration for gas
Internal mass for TEOM
This may be single point or multi-point
If within specifications, record and continue
10
Next steps of QC
If you are outside limitsReview procedures & logs to identify problem
Go back and review data—you may have to throw out data to last good check
Fix, document and report (when significant)
11
Who can conduct verifications?
Routine instrument-stability checks can be conducted by site operator
Periodic assessments (may be every 6 months, may be every year) should be conducted by someone OTHER than site operator
QAPP specifies how often checks are conducted, by whom, and what to do if results are off
12
Calibrations
If results NOT within specs, instrument must be adjustedThis adjustment means that response of instrument is changed, which is defined as a CALIBRATIONUsually involves a zero and a spanAfter calibrating, check again at a single point
13
Precision Error
Unavoidable
Sometimes up, sometimes down–“random”
Estimated by measuring the same thing several times
Minimized by carefully following procedures
14
Precision
Collocated analyzers are too expensive, so use repeated measurements of same thing to estimate precision
Flow rate is most important parameter for PM
Flow rate changes with time—variance is estimate of precision error
15
Estimating Precision in the Field
Somehow have to estimate random “wiggle” errorCan be done with side-by-side instruments (same sample of air) as with filter-based FRM instrumentsCan be done with repeated measurements (same sample of air)Verify results are within limits
16
Two Ways to Estimate Precision of Flow Rate
At beginning of program, use your external flow rate transfer standard in field, compare flow rate transfer standard’s result with that shown (or set) on analyzer (one-point check)
Conduct this check at least every 2 weeks
17
During the First Months
Ensure that your external flow rate transfer standard is checked against another flow rate transfer standard at least once (every six months if you want to use second, easier way to assess precision)
Record every check of analyzer’s flow rate, verify it is within 4% of external flow rate transfer standard’s flow rate for 3 consecutive biweekly checks
18
Easier Way to Estimate Precision After Several Months of Stable Operation
Use “alternative procedure”: Record analyzer’s set point flow rate, assume this is same flow rate that would have been measured by external flow rate transfer standardRecord analyzer’s indicated flow rateCalculate relative percent difference, use value as precision error of flow rate (use relative percent difference calculation; see cells B5 – G28 in the Tribal Data Analysis spreadsheet, pg “PM10 Automated”)
19
Calculating Precision Error
Because we don’t know which device is better, no “truth,” so use average as “truth”
X = one sampler
Y = the other sampler
then precision error = ( )x−yx+y
2
20
Coefficient of Variation (COV) is another statistic to represent imprecision
COV = coefficient of variation
Where s = the sample standard deviation, or STDEV in Excel
average d { } { }COV= sor COV=
average 2
21
Bias Error
Minimized by calibrating equipment against a standard
Make sure standard is certified against NIST-traceable standard (keep certificate!)
Compare this transfer standard to field equipment
Measures any bias in your equipment
22
Quantifying Bias Error
T = the standard’s result, or “truth”
Y = your equipment’s result
Your equipment’s bias = T - Y
T
23
Definitions
EPA calls “accuracy” what others call “total error”
Accuracy, or total error, combines both precision and bias errors
For continuous PM analyzers, difference between assessing precision and assessing accuracy is that to assess accuracy, flow rate transfer standard used must not be your routine field transfer standard
24
Continuous PM Accuracy
Can estimate only accuracy of flow rate, but since this is most important parameter, it’s okay
Use external flow rate transfer standard FROM A DIFFERENT CALIBRATION SET to compare indicated (or set) flow rate of analyzer with that shown by transfer standard
25
Accuracy (cont.)
Flow rate transfer standard must not be same used to calibrate analyzer (from different calibration set)Audit may be conducted by someone from your tribe, just not routine site operatorConduct this flow rate audit at least once per year for each analyzer
26
Accuracy Calculations
Relative percent difference between audit flow rate transfer standard and your analyzer’s flow rate is estimated accuracy of your flow rate
Since there is no other (inexpensive) way to estimate accuracy, this is your estimated accuracy, or total error
27
Control Charts
• Try to keep something steady, but it naturally varies• 95% of time within 2 s of average • 99.8% of time within 3 s of average
red lines within 2 s of average
S = sample standard deviation
28
Control chart of monthly single-point flow rate checksdata:
date indicated FRmeasured FR%differenceflag if >4% warning limits set at +-4% warning limitwarning limit
4/1/2002 16.67 16.63 0.2 4/1/2002 4 -4
5/3/2002 17.67 16.67 6.0FR 5/3/2002 4 -4
6/1/2002 16.68 16.66 0.1 6/1/2002 4 -4
7/5/2002 17.04 16.67 2.2 7/5/2002 4 -4
8/1/2002 16.67 16.99 -1.9 8/1/2002 4 -4
9/6/2002 16.7 16.67 0.2 9/6/2002 4 -40.2
6.0
0.1
2.2
-1.9
0.2
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
8.0
Apr-02 May-02 Jun-02 Jul-02 Aug-02 Sep-02 month
% difference
% difference between FRtransfer standard and sampler'sindicated FR
warning limit=4%
warning limit=4%
Example Control Chart (see example QAPPs) on Disk