PM Model Performance Goals and Criteria

James W. BoylanGeorgia Department of Natural Resources

- VISTAS

National RPO Modeling MeetingDenver, CO

May 26, 2004

Outline

• Standard Bias and Error Calculations

• Proposed PM Model Performance Goals and Criteria

• Evaluation of Eight PM Modeling Studies Using Proposed Goals and Criteria

• Discussion: Should EPA recommend PM Model Performance Goals and Criteria in PM Modeling Guidance Document?

PM Model Evaluations

• Air Quality Modeling and Ambient Measurements are two different ways to estimate actual ambient concentrations of pollutants in the atmosphere

• Both modeling and measurements have some degree of uncertainty– Measurements should not be considered the

absolute truth– Large differences between monitoring networks

due to sampling and analysis techniques

• Normalized bias and error calculations should not be normalized by observations, but rather the average of the model and the observations.

Performance Metrics Equation

Mean Bias (g/m3)

Mean Error (g/m3)

Mean Normalized Bias (%) (-100% to +) Mean Normalized Error (%) (0% to +)

Normalized Mean Bias (%) (-100% to +) Normalized Mean Error (%) (0% to +)

Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)

N

iom CC

NMB

1

1

N

i mo

om

CC

CC

NMFE

1

2

1

N

i mo

om

CCCC

NMFB

1

2

1

N

io

N

iom

C

CCNME

1

1

N

io

N

iom

C

CCNMB

1

1

N

i o

om

C

CC

NMNE

1

1

N

i o

om

C

CC

NMNB

1

1

N

iom CC

NME

1

1

Performance Metrics

• Mean Normalized Bias and Error – Usually associated with observation-based minimum

threshold• Some components of PM can be very small making it

difficult to set a reasonable minimum threshold value without excluding a majority of the data points

– Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small

• A few data points can dominate the metric

– Overestimations are weighted more than equivalent underestimations

– Assumes observations are absolute truth

Performance Metrics

• Normalized Mean Bias and Error – Biased towards overestimations– Assumes observations are absolute truth

• Mean Fractional Bias and Error– Bounds maximum bias and error – Symmetric: gives equal weight to underestimations

and overestimations– Normalized by average of observation and model

Example Calculations

• Mean Normalized Bias and Error– Most biased and least useful of the three metrics

• Normalized Mean Bias and Error • Mean Fractional Bias and Error

– Least biased and most useful of the three metrics

Model g/m3

)

Obs. g/m3

)

MBg/m3)

NMB (%)

MNB (%)

MFB (%)

ME g/m3

)

NME (%)

MNE (%)

MFE (%)

0.05 1.0 -0.95 -95 -180.95 +0.95 +95 +180.95

1.0 0.05 +0.95 +1900 +180.95 +0.95 +1900 +180.95

1.0 0.01 +0.99 +9900 +196.04 +0.99 +9900 +196.04

0.04 0.05 -0.01 -20 -22.22 +0.01 +20 +22.22

0.5225 0.2775 +0.245 +88.3 +2921.3 +43.5 0.725 261.3 2978.8 145.0

PM Goals and Criteria• Performance Goals: Level of accuracy that

is considered to be close to the best a model can be expected to achieve.

• Performance Criteria: Level of accuracy that is considered to be acceptable for regulatory applications.

• It has been suggested that we need different performance goals and criteria for:– Different Species– Different Seasons– Different Parts of the Country– 20% Haziest and 20% Cleanest Days

• Answer: performance goals and criteria that vary as a function of concentration

PM Modeling Studies Used for Performance Benchmarks

• SAMI (GT)– July 1995 (URM/IMPROVE/variable grid)– July 1991 (URM /IMPROVE /variable grid)– May 1995 (URM /IMPROVE /variable grid)– May 1993 (URM /IMPROVE /variable grid)– March 1993 (URM /IMPROVE /variable grid)– February 1994 (URM /IMPROVE /variable grid)

• VISTAS (UCR/AG/Environ)– July 1999 (CMAQ/IMPROVE/36 km)– July 1999 (CMAQ /IMPROVE/12 km)– July 2001 (CMAQ /IMPROVE/36 km)– July 2001 (CMAQ /IMPROVE/12 km)– January 2002 (CMAQ /IMPROVE/36 km)– January 2002 (CMAQ /IMPROVE/12 km)

PM Modeling Studies Used for Performance Benchmarks

• WRAP 309 (UCR/CEP/Environ)– January 1996 (CMAQ/IMPROVE/36 km)– February 1996 (CMAQ/IMPROVE/36 km)– March 1996 (CMAQ/IMPROVE/36 km)– April 1996 (CMAQ/IMPROVE/36 km)– May 1996 (CMAQ/IMPROVE/36 km)– June 1996 (CMAQ/IMPROVE/36 km)– July 1996 (CMAQ/IMPROVE/36 km)– August 1996 (CMAQ/IMPROVE/36 km)– September 1996 (CMAQ/IMPROVE/36 km)– October 1996 (CMAQ/IMPROVE/36 km)– November 1996 (CMAQ/IMPROVE/36 km)– December 1996 (CMAQ/IMPROVE/36 km)

PM Modeling Studies Used for Performance Benchmarks

• WRAP 308 (UCR/CEP/Environ)– Summer 2002 (CMAQ/IMPROVE/36 km/WRAP)– Summer 2002 (CMAQ/IMPROVE/36 km/US)– Winter 2002 (CMAQ/IMPROVE/36 km/WRAP)– Winter 2002 (CMAQ/IMPROVE/36 km/US)

• EPA (Clear Skies)– Fall 1996 (REMSAD/IMPROVE/36 km)– Spring 1996 (REMSAD/IMPROVE/36 km)– Summer 1996 (REMSAD/IMPROVE/36 km)– Winter 1996 (REMSAD/IMPROVE/36 km)

PM Modeling Studies Used for Performance Benchmarks

• MANE-VU (GT)– July 2001 (CMAQ/IMPROVE/36 km)– July 2001 (CMAQ/SEARCH/36 km)– January 2002 (CMAQ/IMPROVE/36 km)– January 2002 (CMAQ/ SEARCH /36 km)

• Midwest RPO– August 1999 (CMAQ/IMPROVE/36 km)– August 1999 (CAMx/IMPROVE/36 km)– August 1999 (REMSAD/IMPROVE/36 km)– January 2000 (CMAQ/IMPROVE/36 km)– January 2000(CAMx/IMPROVE/36 km)– January 2000(REMSAD/IMPROVE/36 km)

PM Modeling Studies Used for Performance Benchmarks

• EPRI (AER/TVA/Environ)– July 1999 (CMAQ/IMPROVE/32 km)– July 1999 (CMAQ/IMPROVE/8 km)– July 1999 (MADRID/IMPROVE/32 km)– July 1999 (MADRID /IMPROVE/8 km)– July 1999 (CAMx/IMPROVE/32 km)

Mean Fractional Error

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

Mean Fractional Bias

Proposed PM Goals and Criteria

• Based on MFE and MFB calculations• Vary as a function of species

concentrations– Goals: MFE +50% and MFB ±30%– Criteria: MFE +75% and MFB ±60%– Less abundant species should have less stringent

performance goals and criteria

• Continuous functions with the features of:– Asymptotically approaching proposed goals and

criteria when the mean of the observed and modeled concentrations are greater than 2.5 g/m3

– Approaching +200% MFE and ±200% MFB when the mean of the observed and modeled concentrations are extremely small

Proposed Goals and Criteria

• Proposed PM Performance Goals

• Proposed PM Performance Criteria

301703/5.0

)(5.0

mg

CC mo

eMFB

501503/75.0

)(5.0

mg

CC mo

eMFE

751253/75.0

)(5.0

mg

CC mo

eMFE

601403/5.0

)(5.0

mg

CC mo

eMFB

MFE Goals and Criteria

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Goal

Criteria

Zone 1

Zone 3

Zone 2

MFB Goals and Criteria

-200

-150

-100

-50

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

(+) Goal

(-) Goal

(+) Criteria

(-) CriteriaZone 1

Zone 3

Zone 3

Zone 2

Zone 2

Model Performance Zones

• Zone I – Good Model Performance– Level I Diagnostic Evaluation (Minimal)

• Zone II– Average Model Performance– Level II Diagnostic Evaluation (Standard)

• Zone III– Poor Model Performance– Level III Diagnostic Evaluation (Extended)

and Sensitivity Testing

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Goal

Criteria

Sulfate Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Sulfate Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Nitrate

Goal

Criteria

Nitrate Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Nitrate

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Nitrate Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Ammonium

Ammonium Bi

Goal

Criteria

Ammonium Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Ammonium

Ammonium Bi

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Ammonium Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0 6.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Organics

Goal

Criteria

Organics Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0 5.0 6.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Organics

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Organics Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

EC

Goal

Criteria

EC Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

EC

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

EC Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Soils

Goal

Criteria

Soils Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 1.0 2.0 3.0 4.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Soils

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Soils Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

PM2.5

Goal

Criteria

PM2.5 Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

PM2.5

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

PM2.5 Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

PM10

Goal

Criteria

PM10 Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

PM10

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

PM10 Mean Fractional Bias

Combined Modeling Studies

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

CM

Goal

Criteria

CM Mean Fractional Error

Combined Modeling Studies

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

CM Mean Fractional Bias

SAMI: May 1995

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

SAMI Mean Fractional Error

SAMI: May 1995

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

SAMI Mean Fractional Bias

SAMI: 6 Episodes

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

SAMI Mean Fractional Error

SAMI: 6 Episodes

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Ammonium Bi

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

SAMI Mean Fractional Bias

EPA Mean Fractional Error

EPA: 4 Seasons (1996)

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Organics

EC

Soils

PM2.5

Goal

Criteria

EPA Mean Fractional Bias

EPA: 4 Seasons (1996)

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Organics

EC

Soils

PM2.5

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

VISTAS: 3 Episodes (2 Grids)

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

VISTAS Mean Fractional Error

VISTAS: 3 Episodes (2 Grids)

-200

-150

-100

-50

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

VISTAS Mean Fractional Bias

MANE-VU Mean Fractional Error

MANE-VU: 2 Episodes (IMPROVE + SEARCH)

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

PM2.5

Goal

Criteria

MANE-VU Mean Fractional Bias

MANE-VU: 2 Episodes (IMPROVE + SEARCH)

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 4.0 8.0 12.0 16.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

PM2.5

(+) Bias Goal

(-) Bias Goal

(+) Criteria

(-) Criteria

WRAP (309) Mean Fractional Error

WRAP: 12 Months (1996)

0

50

100

150

200

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

WRAP (309) Mean Fractional Bias

WRAP: 12 Months (1996)

-200

-150

-100

-50

0

50

100

150

200

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

WRAP (308) Mean Fractional Error

WRAP: 2 Seasons (WRAP & US Domains)

0.00

50.00

100.00

150.00

200.00

0.0 3.0 6.0 9.0 12.0 15.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

WRAP (308) Mean Fractional Bias

WRAP: 2 Seasons (WRAP & US Domains)

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 3.0 6.0 9.0 12.0 15.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

MRPO Mean Fractional Error

MRPO: 2 Episodes (CMAQ/CAMx/REMSAD)

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0 10.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

CM

Goal

Criteria

MRPO Mean Fractional Bias

MRPO: 2 Episodes (CMAQ/CAMx/REMSAD)

-200.00

-150.00

-100.00

-50.00

0.00

50.00

100.00

150.00

200.00

0.0 2.0 4.0 6.0 8.0 10.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

CM

(+) Bias Goal

(-) Bias Goal

(+) Criteria

(-) Criteria

EPRI Mean Fractional Error

EPRI: July 1999 (3 Models)

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

Goal

Criteria

EPRI Mean Fractional Bias

EPRI: July 1999 (3 Models)

-200

-150

-100

-500

50

100

150

200

0.0 5.0 10.0 15.0 20.0

Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Sulfate

Nitrate

Ammonium

Organics

EC

Soils

PM2.5

PM10

CM

(+) Goal

(-) Goal

(+) Criteria

(-) Criteria

Concluding Remarks• Performance evaluation should be done on

an episode-by-episode basis or on a month-by-month basis for annual modeling.

• Recommended performance goals and criteria should be used to help identify areas that can be improved upon in future modeling – Failure to meet proposed performance criteria

should not necessarily prohibit the modeling from being used for regulatory purposes • Need to perform extended diagnostic evaluation

and sensitivity tests to address poor performance

Concluding Remarks (cont.)• As models mature, performance goals can be

made more restrictive by simply adjusting the coefficients in the performance goals and criteria equations (MFE and MFB)

• Performance goals and criteria for measurements with longer averaging times (e.g., weekly) should be more restrictive and those with a shorter averaging times (e.g., hourly) should be less restrictive.

• Discussion Questions:– Should EPA recommend PM model performance goals

and criteria in PM Modeling Guidance Document?– Is there a need for performance goals for gaseous

precursors and/or wet deposition species?