Quality Control – Part II

Tim Hanley

EPA Office of Air Quality Planning and Standards

2

Error

…the difference between your answer and the “truth”

3

Continuous PM QC

Instrument Stability Checks (EPA calls this general QA in their 1995 memo; see Attachment A in the TAMS template QAPPs)

Precision (wiggle)

Bias (jump)

4

Instrument Stability

Internal diagnostics conducted with each measurement (no error codes)TEOM uses standard filter kit available from dealer BAM uses standard foil or internal sourceUse kits more frequently at beginning of program, thereafter as necessary and at least twice a year

5

Results of Stability Checks

Use dealer’s guidelines

Within 2.5% for TEOM standard filter

Within 7% for BAM foil

Plot results on a control chart (can also plot other parameters on a control chart)

6

Verifying the accuracy of your Transfer Standard

Option 1: send standard to manufacturer, who has NIST-traceable equipment; they send you back your standard with certificate of traceability

Option 2: use natural standards, such as freezing ice water for temperature, carefully document your calibration according to a standard method

7

Single (One) Point Verifications

#Verification occurs when you conduct check using one flow rate or concentration or value

#Span is an example of one-point check, meaning spanning from zero up to a range

#Verifications DO NOT involve any changes to your equipment

8

Multi-Point Verifications

Check of response of instrument to more than one flow rate, concentration, or value

Usually zero, a low level, and at the upper end of the range expected to be measured

9

Example of Verification

Perform check of equipmentFlow rate for PM

Concentration for gas

Internal mass for TEOM

This may be single point or multi-point

If within specifications, record and continue

10

Next steps of QC

If you are outside limitsReview procedures & logs to identify problem

Go back and review data—you may have to throw out data to last good check

Fix, document and report (when significant)

11

Who can conduct verifications?

Routine instrument-stability checks can be conducted by site operator

Periodic assessments (may be every 6 months, may be every year) should be conducted by someone OTHER than site operator

QAPP specifies how often checks are conducted, by whom, and what to do if results are off

12

Calibrations

If results NOT within specs, instrument must be adjustedThis adjustment means that response of instrument is changed, which is defined as a CALIBRATIONUsually involves a zero and a spanAfter calibrating, check again at a single point

13

Precision Error

Unavoidable

Sometimes up, sometimes down–“random”

Estimated by measuring the same thing several times

Minimized by carefully following procedures

14

Precision

Collocated analyzers are too expensive, so use repeated measurements of same thing to estimate precision

Flow rate is most important parameter for PM

Flow rate changes with time—variance is estimate of precision error

15

Estimating Precision in the Field

Somehow have to estimate random “wiggle” errorCan be done with side-by-side instruments (same sample of air) as with filter-based FRM instrumentsCan be done with repeated measurements (same sample of air)Verify results are within limits

16

Two Ways to Estimate Precision of Flow Rate

At beginning of program, use your external flow rate transfer standard in field, compare flow rate transfer standard’s result with that shown (or set) on analyzer (one-point check)

Conduct this check at least every 2 weeks

17

During the First Months

Ensure that your external flow rate transfer standard is checked against another flow rate transfer standard at least once (every six months if you want to use second, easier way to assess precision)

Record every check of analyzer’s flow rate, verify it is within 4% of external flow rate transfer standard’s flow rate for 3 consecutive biweekly checks

18

Easier Way to Estimate Precision After Several Months of Stable Operation

Use “alternative procedure”: Record analyzer’s set point flow rate, assume this is same flow rate that would have been measured by external flow rate transfer standardRecord analyzer’s indicated flow rateCalculate relative percent difference, use value as precision error of flow rate (use relative percent difference calculation; see cells B5 – G28 in the Tribal Data Analysis spreadsheet, pg “PM10 Automated”)

19

Calculating Precision Error

Because we don’t know which device is better, no “truth,” so use average as “truth”

X = one sampler

Y = the other sampler

then precision error = ( )x−yx+y

2

20

Coefficient of Variation (COV) is another statistic to represent imprecision

COV = coefficient of variation

Where s = the sample standard deviation, or STDEV in Excel

average d { } { }COV= sor COV=

average 2

21

Bias Error

Minimized by calibrating equipment against a standard

Make sure standard is certified against NIST-traceable standard (keep certificate!)

Compare this transfer standard to field equipment

Measures any bias in your equipment

22

Quantifying Bias Error

T = the standard’s result, or “truth”

Y = your equipment’s result

Your equipment’s bias = T - Y

T

23

Definitions

EPA calls “accuracy” what others call “total error”

Accuracy, or total error, combines both precision and bias errors

For continuous PM analyzers, difference between assessing precision and assessing accuracy is that to assess accuracy, flow rate transfer standard used must not be your routine field transfer standard

24

Continuous PM Accuracy

Can estimate only accuracy of flow rate, but since this is most important parameter, it’s okay

Use external flow rate transfer standard FROM A DIFFERENT CALIBRATION SET to compare indicated (or set) flow rate of analyzer with that shown by transfer standard

25

Accuracy (cont.)

Flow rate transfer standard must not be same used to calibrate analyzer (from different calibration set)Audit may be conducted by someone from your tribe, just not routine site operatorConduct this flow rate audit at least once per year for each analyzer

26

Accuracy Calculations

Relative percent difference between audit flow rate transfer standard and your analyzer’s flow rate is estimated accuracy of your flow rate

Since there is no other (inexpensive) way to estimate accuracy, this is your estimated accuracy, or total error

27

Control Charts

• Try to keep something steady, but it naturally varies• 95% of time within 2 s of average • 99.8% of time within 3 s of average

red lines within 2 s of average

S = sample standard deviation

28

Control chart of monthly single-point flow rate checksdata:

date indicated FRmeasured FR%differenceflag if >4% warning limits set at +-4% warning limitwarning limit

4/1/2002 16.67 16.63 0.2 4/1/2002 4 -4

5/3/2002 17.67 16.67 6.0FR 5/3/2002 4 -4

6/1/2002 16.68 16.66 0.1 6/1/2002 4 -4

7/5/2002 17.04 16.67 2.2 7/5/2002 4 -4

8/1/2002 16.67 16.99 -1.9 8/1/2002 4 -4

9/6/2002 16.7 16.67 0.2 9/6/2002 4 -40.2

6.0

0.1

2.2

-1.9

0.2

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

Apr-02 May-02 Jun-02 Jul-02 Aug-02 Sep-02 month

% difference

% difference between FRtransfer standard and sampler'sindicated FR

warning limit=4%

warning limit=4%

Example Control Chart (see example QAPPs) on Disk