qa strategy workgroup ambient air quality system training ambient air monitoring quality systems...

QA Strategy Workgroup

Ambient Air Quality System Training

Ambient Air Monitoring Quality Systems Requirements

Mike Papp- OAQPS

24-Hour, 5-minute Flow Rates

15

15.5

16

16.5

17

17.5

18

1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141

5-min data

Flo

w R

ate

(L

/min

)



2

• CAA- Clean Air Act

“(2) Establishment of a national network to monitor, collect, and compile data with quantification of uncertainty in the status and trends of air emissions, deposition, air quality, surface water quality, forest condition, and visibility impairment and to ensure the comparability of air quality data collected in different States and obtained from different nations.”



3

Major 40 CFR Pt. 58 App. A QA Requirements

• Section 1– General Info– Applicability– Definitions– Differences SLAMs/PSD

• Section 2– QMPs/QAPPs*– Independent quality management

function*– DQOs- (EPA)

• PM2.5, O3, PM10-2.5

– NPAP/PEP– TSAs-(EPA Regions)– NIST Traceable Standards

• Section 3 – PQAO definition– Quality Control and Assessment

• 1-point QC• Annual PE• Flow Rate Verification• Flow Rate Audit• Collocated Sampling• PEP

• Section 4 – QA Stats

• Section 5-– Reporting Req.

• List- Site ID/PQAO by pollutant• Quarterly submission of QA per

58.16 reporting requirements• Calculation of QA (EPA) once

annual submission occurs

} Gaseous pollutants

}PM/Pb

* EPA QA Policy

There is so much more to QA beyond Appendix A



4

The Important CFR QA Requirements

• QMPs/QAPPs/SOPs a must

• Independent quality management

• Establishment of a Primary Quality Assurance Organization

• Use of NIST Traceable Standards

• Use of FRM/FEMs at the appropriate operating ranges



5

More Important CFR Gaseous QA Requirements

• One Point QC Checks- Biweekly• Annual Performance Evaluation at 3 consecutive

audit ranges • NPAP Through the Probe Audits- 20% of Sites• Precision and Bias data reporting to AQS • Data assessments with the correct statistics

– Data Toobox, DASC, AMP255



6

PlanningQMPs, QAPPS and SOPs

QMP- All programs

QAPP

SOP

QMP - Organization Specific•Describes organizations quality system•Establishes capability

QAPP - Project Specific•Identifies the reasons for collecting data and for collecting it in a specific way•Documents how the data are collected and how quality is maintained

SOP - Instrument/Method Specific•Ensures consistency

- From day to day - From one person to the next



7

All QAPPs Are Not Created Equal- the Graded Approach

The level of detail in each QA Project Plan will vary according to the nature of the work being performed and the intended use of the data.

High Road- Big important decisions with little uncertainty

Low Road – Small studies without major consequences or an allowance of greater uncertainty

OAQPS has developed a graded approach for ambient air monitoring

http://www.epa.gov/ttn/amtic/geninfo.html



8

TURBO-QAPP



9



10

SOPs

• They can never be too detailed– Remember those coming after you

• They are easy to write!• Look at the reference methods in CFR for the

minimum requirements• Don’t reinvent the wheel –Share good SOPs-

Ask vendors for electronic versions• Use document control • Use quality bulletins for changes

Army Field Manual



11

Independent Quality Management

• who functions independently of direct environmental data generation,

• who reports on quality issues to the senior manager having executive leadership authority for the organization; and

• who has sufficient technical and management expertise and authority to conduct independent oversight of and assure the implementation of the organization’s quality system in the environmental programs of the organization.

EPA QA Policy Requires: A quality assurance manager (QAM), or person/persons assigned to an equivalent position:



12

Establishment of Primary Quality Assurance Organizations

Common factors of PQAOs• Operation by a common team of field operators

according to a common set of procedures;• Use of a common QAPP or standard operating

procedures;• Common calibration facilities and standards;• Oversight by a common quality assurance

organization; and• Support by a common management, laboratory or

headquarters.



13

Possible PQAO Consolidations

• Consolidation of Tribes within a State

• Consolidation of Tribes across States within an EPA Region

• Consolidation of Tribes with State PQAO

• Consolidation of Tribes across EPA Regions



14

Use of NIST Traceable Standards(CFR Part 58 App A Sec 2.6.1)

“Gaseous pollutant concentration standards must be traceable to either a National Institute of Standards and Technology (NIST) Traceable Reference Material (NTRM) or a NIST-certified Gas Manufacturer’s Internal Standard (GMIS), …

Vendors advertising certification…. must participate in the EPA Protocol Gas Verification Program or not use ‘‘EPA’’ in any form of advertising.”



15

Certifications

Testifies to the accuracy of an instrument/standard/gas as

it compares to a NIST traceable standard.

•Make sure you are aware of how long a cylinder is certified for. It’s different for different gasses and concentrations

•Create a tracking mechanism

•Follow guidance for your MQOs in QAPP



16

Use of Federal Reference Methods (FRM) anmd Federal Equivalent Methods (FEMs)

• All monitors used for comparison to NAAQS must be FRM/FEM (40 CFR Part 53)

• Non-NAAQS monitoring has more flexibility• EPA National Environmental Research

Laboratory (NERL) develops an FRM for each NAAQS

• NERL approves each FRM/FEM– Posted on AMTIC

• Each has acceptable operating range(s)– Pick the range that matches your need.

http://www.epa.gov/ttn/amtic/criteria.html



17

PerformanceCharacteristics of Gaseous FRM/FEM Methods

• Acceptable levels of – detection– precision– bias– drift

• Relatively unaffected by interfering compounds

• Sensitive at typical ambient concentrations

• Commercially viable• Supports reliable and relatively

maintenance-free operation• Easily “QC-able”



18

Quality Control and Evaluation

• Quality Control Checks

• Performance Evaluations

• Technical Systems Audits



19

Understanding and Controlling Uncertainty in Order to Minimize Decision Errors

Uncertainty = Natural Variability + Measurement

2.Precision*3.Bias*4. Completeness*5. Comparability6. Detectability

MQOs

Preparation Field

Laboratory

DQO

DQA

}1. Representativeness

Data Quality IndicatorsSpatial/Temporal



20

Data Quality Indicators & MQOs

CO/ NO2 / SO2 O3

Representativeness # sites/freq # sites/freq

Comparability FRM/FEM/ FRM/FEM

Completeness 75% 75%

Sensitivity (Detectability)1.0 / 0.01 / 0.01 0.003 ppm

Precision 10%* 7%*

Bias +10% * +7%**Based on 3 years of data so any individual value can be higher than MQO gaseous P & B estimates at site level



21

QC Checks

QC Checks have a variety of purposes• To estimate our errors and keep within limits

(internal program use, stay ahead of auditors)• To troubleshoot performance of equipment

(lamp intensity, pump efficiency)• To identify drift over time• To keep system stable (instrument checks,

shelter conditions)• To report to EPA for their use

A QC Check is essentially an audit on yourself...



22

QC Checks• One Point QC Check

– Biweekly- Used to estimate precision and bias at the site level.

– Select QC Check concentration at around routine concentrations.

• Annual Performance – More thorough independent check– Independent standards (and possibly the person)– 3 consecutive ranges (some flexibility to concentration

levels in CFR



23

NPAP-TTP PEP

NATTS PT &ORIA Round Robins

SRP PAMS Cylinders

Protocol Gas

Performance Evaluation-What and Why

Performance evaluations (PEs) are a type of audit in which the quantitative data generated in a measurement system are obtained independently and compared with routinely obtained data to evaluate the proficiency of an analyst, or a laboratory



24

3 NPAP TTP Mobile Platforms



25



26

Technical Systems Audits

Two types of Audits:

Internal: Conducted by your own agency

QA group. This group is independent of the monitoring group

External: Conducted by your Regional EPA

staff or another agency. This group is independent of your agency completely



27

Acceptance Criteria for Calibrations, QC Checks, and Instrument Audits

Look for these tables

All the ranges and parameters you will need are neatly laid

out here



Statistical Data Assessments

100

audit

auditmeasdi



29

Another Way of Looking at It…#

of O

bser

vatio

ns

% Difference

0 +10-10

Ideal: Precise w/o Bias

Precise w Negative Bias Precise w Positive Bias

Low Precision w/o Bias

Low Precision w Negative Bias Low Precision w Positive Bias



30

Precision

Precision is more conservative when evaluated atthe 90% one-sided upper confidence level

Defines the “Width” of the

Bell Curve

Gaseous Estimate

21 1

22

1

)1(

)(_Pr

n

n

i

n

iii n

nn

ddnEstimateecision

truth

90th Percentile of a Chi-Squared Distribution

More Conservative



31

truthBias

n

iidn

AB1

||1 Central

Tendency

95th Quartile of a Student-t Distribution

w/ n-1 df

Standard Error

)1(

|)|(||1 1

22

nn

ddnAS

n

i

n

iii

n

AStABABEstimateBias n 1,95.0||_



32

DASC ToolHolds Up

to 500 Pairs of Values

Graphs & Stats

Automatically Updated

References Equations in Appendix A



33

DASC Tool (Cont’d)

Easy to Use Navigation to Find

the Stats You Need



34

AMP255 Features

Statistics Shown on a per Site

and Per PQAO Level as

Appropriate



35

Box and Whisker Plots

*

*

*

+

Lower FenceQ1 – (1.5*IQR)

Upper FenceQ3 + (1.5*IQR)

InterquartileRange (IQR)

Outliers

Whiskermin

Whiskermax

Q1 (25th Percentile)

Q2 (50th Percentile)Q3 (75th Percentile)

Mean



36

Box and Whisker Plots

Posted Box and Whisker Plots are Available at:www.epa.gov/ttn/amtic/parslist.html

Remember Me?



Don’t ForgetThe

It contains a number of the QC Checks Also!!!



38

What Data is in AQS?

• Collocated Data

• Flow Data

• Annual Performance Audits

• NPAP Data

• Technical Systems Audit Information???



Guidance On All This!



40

OR

GA

NIZ

AT

ION

/PR

OG

RA

M

PR

OJE

CT

PO

LIC

Y/R

EG

UL

AT

ION

S

Defensible Products and Decisions

EPA Program &Regional Policy

External PoliciesContracts - 48 CFR 46

Assistance Agreements - 40 CFR 30, 31, and 35

Internal EPA PoliciesEPA Order 5360.1EPA Manual 5360

Consensus StandardsANSI/ASQC E4ISO 9000 Series

Annual Review and Planning

(e.g., QAARWP)

SystemsAssessments(e.g., QSAs)

Quality SystemDocumentation

(e.g., QMP)

Training/Communication(e.g., Training Plan,

Conferences)

Supporting System Elements(e.g., Procurements,

Computer Hardware/Software)

TechnicalAssessments

IMPLEMENTATIONPLANNING ASSESSMENT

Conduct Study/Experiment

QA Project Plan

SystematicPlanning

(e.g., DQO Process)

StandardOperating

Procedures

Data QualityAssessment

Data Verification& Validation

Ambient AirCAA, CFR 50,53, 58

Regional Policy

OR

GA

NIZ

AT

ION

/PR

OG

RA

M

PR

OJE

CT

PO

LIC

Y/R

EG

UL

AT

ION

S

Defensible Products and Decisions

EPA Program &Regional Policy

External PoliciesContracts - 48 CFR 46

Assistance Agreements - 40 CFR 30, 31, and 35

Internal EPA PoliciesEPA Order 5360.1EPA Manual 5360

Consensus StandardsANSI/ASQC E4ISO 9000 Series

Annual Review and Planning

(e.g., QAARWP)

SystemsAssessments(e.g., QSAs)

Quality SystemDocumentation

(e.g., QMP)

Training/Communication(e.g., Training Plan,

Conferences)

Supporting System Elements(e.g., Procurements,

Computer Hardware/Software)

TechnicalAssessments

IMPLEMENTATIONPLANNING ASSESSMENT

Conduct Study/Experiment

QA Project Plan

SystematicPlanning

(e.g., DQO Process)

StandardOperating

Procedures

Data QualityAssessment

Data Verification& Validation

Ambient AirCAA, CFR 50,53, 58

Regional Policy

G-4

R5-G-5

EPA Guidance for planning, implementation, and assessment activities.

G-6

http://www.epa.gov/quality1/qa_docs.html

G-9RG-9S

G-8

G-3G-7



41



42

Excellent Air Monitoring Resource

REDBOOK!

Light reading for the

air monitoring newbies

and grizzled veterans

http://www.epa.gov/ttn/amtic/qabook.html