NYS Department of Environmental Conservation

Dirk Felton RTP, NC February 12-13, 2008

Air Quality Data Summit: Session: Inventory of Data Systems

Data provider perspectives

The NY State Dept of Environmental Conservation SubmitsAir Quality data to the following Databases:

AQS: Nearly 80 sites, hundreds of parameters, since the 1970’s- Submit, Valid, Certified, Flagged or null

AirNow: selected Ozone, PM-2.5 and PM-10- Submit selected un-validated, modified data (NY’s Network is too dense in urban area: NYC)

VIEWS: 2 urban IMPROVE sites- IMPROVE submits and validates

NADP/MDN: 2 urban wet deposition and Continuous Hg sites- NADP submits and validates

NARSTO: 5 Yr collaboration with NY Supersite- PI/NYSDEC submit Valid and Flagged data

AQS:

Input Issues: Difficult format, parameter and method codes limit detail, not suitable for short term studies, no ability to calculate: flows (std vs vol)

Retrieval issues: Flags not included automatically (should be), Public, EPA Regional offices and many consultants have difficulty finding data

Meta data issues: site data not updated, system does not accept accessory and diagnostic data, scale data not by parameter or complete, no analysis detail or history of changes

Correction/certification issues: Should have a method to inform users of changes to datasets or at least institute a version or customer tracking system. Data certification not necessarily meaningful, Blanks not incl.

Frustration: downloaded data by parameter creates data sets that are not internally compatible (FRM, TEOM, STN mass), 120 days not long enough for data requiring lab analysis, FEMs not always equal.

DataCollection

PrimaryStorage

Aggregation, Integration, Processing,

Modeling

Analysis, Visualization,

Reporting

Decision Support

EndUsers

Draft Data Value Chain Diagram

Analysis, Visualization and ReportingApplications

Decision SupportApplications

Group A2: Data

Access

Group B2:Integration

Consolidation

Group C2:Data Visualization and

Analysis

Group A2: Data

Access

DataCollection

PrimaryStorage

Aggregation, Integration, Processing,

Modeling

Analysis, Visualization,

Reporting

Decision Support

EndUsers

Draft Data Value Chain Diagram

Analysis, Visualization and ReportingApplications

Decision SupportApplications

Group A2: Data

Access

Group B2:Integration

Consolidation

Group C2:Data Visualization and

Analysis

Group A2: Data

Access

AQS Frustration Continued: (This is what normally happens)

Data Provider

Public &Consultants

AirNow:

Input Issues: “FRM-like” data not consistently modified. (Should there be a test?) Need hourly connectivity to monitoring site, Some older data systems limited to Ozone and PM-2.5.

Retrieval issues: Limited, Access to un-validated data should be limited.

Meta data issues: Access to FRM like adjustments is provided but is rarely used by data users. (may be password protected)

Correction/certification issues: Some “error catching” can be implemented by provider but this is difficult in a live system. Buddy system checking should be provided by database. Data not likely to match AQS data.

Frustration: Invalid data gets out to the public more often than we would like. Max data thresholds can limit bad data but in extreme conditions (fires) can also limit good data when the public really needs it.

VIEWS:

Input Issues: Data input by administrator not data providers.

Retrieval issues: Very long delay before data is available, Data sometimes partially available or in draft before it is finalized

Meta data issues: Helpful documents provided but cannot be uploaded by

data provider. QA data and QA program not sufficient.

Correction/certification issues: No data provider input

Frustration: Data delays of more than 1 year are common.

NADP/MDN

Input Issues: Data input by administrator not provider.

Retrieval issues: Limited to public, More availability to providers.

Meta data issues: Limited but not usually a problem for a consistent network.

Correction/certification issues: Tekran data will be a challenge

Frustration: We’ll see, The Mercury program is still being designed.

NARSTO

Input Issues: File format is archaic, cumbersome and does not provide enough options.

Retrieval issues: ?? I don’t know how.

Meta data issues: Some information is included in the input file format.

Correction/certification issues: Utilizes Data QC flags (0,1,2)

Frustration: Seems like a black hole.

*DATA EXCHANGE STANDARD VERSION NARSTO 2001/10/31 (2.213)*QUALITY CONTROL LEVEL 2 (a complete, externally consistent data set of specified quality that consists of research products that have undergone interpretative and diagnostic analyses by the project staff or user community.)*DATE THIS FILE GENERATED/ARCHIVE VERSION NUMBER 9/27/2001 1*ORGANIZATION ACRONYM ENVCAN*ORGANIZATION NAME Environment Canada, Meteorological Service of Canada*STUDY OR NETWORK ACRONYM CAPMoN*STUDY OR NETWORK NAME Canadian Air and Precipitation Monitoring Network*FILE CONTENTS DESCRIPTION--SHORT/LONG air-filter-meas Air filter measurements*PRINCIPAL INVESTIGATOR NAME--LAST/FIRST MacTavish Dave*PRINCIPAL INVESTIGATOR AFFILIATION Environment Canada, Meteorological Service of Canada*CO-INVESTIGATOR NAME--LAST/FIRST None*CO-INVESTIGATOR AFFILIATION None*COUNTRY CODE CA*STATE OR PROVINCE CODE ON*SAMPLING INTERVAL AS REPORTED IN MAIN TABLE 24 hour*SAMPLING FREQUENCY OF DATA IN MAIN TABLE Same as sampling interval*PRINCIPAL INVESTIGATOR CONTACT INFORMATION Environment Canada, Meteorological Service of Canada, 4905 Dufferin St., Toronto, Ont. M3H 5T4*DATA USAGE ACKNOWLEDGEMENT Environment Canada, Meteorological Service of Canada, 4905 Dufferin St., Toronto, Ont. M3H 5T4*NAME AND AFFILIATION OF PERSON WHO GENERATED THIS FILEBill Sukloff, Environment Canada (MSC)*DATE OF LAST MODIFICATION TO DATA IN MAIN TABLE 3/27/2001*NAME AND VERSION OF SOFTWARE USED TO CREATE THIS FILEMS Excel/97*STANDARD CHARACTERS !#$%&'()*,-./0123456789:;<>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ\^_`abcdefghijklmnopqrstuvwxyz{}~*COMPANION FILE NAME/FORMAT AND VERSION None

*TABLE NAME NARSTO standard flags*TABLE FOCUS Metadata*TABLE COLUMN NAME Flag: NARSTO Description*TABLE COLUMN UNITS None None*TABLE COLUMN FORMAT TYPE Char Char*TABLE COLUMN FORMAT FOR DISPLAY 2 120*TABLE BEGINS

V0 Valid valueV1 Valid value but comprised wholly or partially of below detection limit dataM1 Missing value because no value is availableM2 Missing value because invalidated by Data Originator

*TABLE ENDS

*TABLE NAME Site information*TABLE FOCUS Metadata*TABLE COLUMN NAME Site ID: standard Site abbreviation: standard Description Country code State or province code Latitude: decimal degreesLongitude: decimal degreesLat/lon reference datum*TABLE COLUMN UNITS None None None None None Decimal degrees Decimal degrees None*TABLE COLUMN FORMAT TYPE Char Char Char Char Char Decimal Decimal Char*TABLE COLUMN FORMAT FOR DISPLAY 12 3 50 50 20 10.5 10.5 20*TABLE COLUMN MISSING CODE None None None None None -999.99999 -999.99999 None*TABLE BEGINS

CAPMCAONEGB_ EGB_ Egbert CA ON 44.233 79.781 NAD83*TABLE ENDS

*TABLE NAME air_filter_meas*TABLE FOCUS Surface--fixed *TABLE EXPLANATION OF ZERO OR NEGATIVE VALUESNo zero values or negative values appear in the data in this file.*TABLE USER NOTE This is an example file only. The full file would have many additional lines of data.*TABLE KEY FIELD NAMES Site ID: standard Co-location ID Date start: local time Time start: local time*TABLE COLUMN NAME Site ID: standard Instrument co-location ID Date start: local time Time start: local time Date end: local time Time end: local time Time zone: local Date start: UTC*TABLE COLUMN NAME TYPE Variable Variable Variable Variable Variable Variable Variable Variable*TABLE COLUMN CAS IDENTIFIER None None None None None None None None*TABLE COLUMN USER NOTE None None None None None None None None*TABLE COLUMN USER NOTE2 None None None None None None None None*TABLE COLUMN UNITS None None yyyy/mm/dd hh:mm yyyy/mm/dd hh:mm None yyyy/mm/dd*TABLE COLUMN FORMAT TYPE Char Char Date Time Date Time Char Date*TABLE COLUMN FORMAT FOR DISPLAY 12 2 10 5 10 5 3 10*TABLE COLUMN MISSING CODE None None None None None None None None*TABLE COLUMN LOOKUP TABLE NAME Site information Instrument co-location None None None None None None*TABLE COLUMN OBSERVATION TYPE Supplementary data Supplementary data Supplementary data Supplementary data Supplementary data Supplementary data Supplementary dataSupplementary data*TABLE COLUMN FIELD SAMPLING OR MEASUREMENT PRINCIPLENot applicable Not applicable Data logger Data logger Data logger Data logger Data logger Not applicable*TABLE COLUMN PARTICLE DIAMETER--LOWER BOUND (UM)Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN PARTICLE DIAMETER--UPPER BOUND (UM)Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN PARTICLE DIAMETER--MEDIAN (UM) Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN MEDIUM Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN COATING OR ABSORBING SOLUTION/MEDIANot applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN WAVELENGTH (NM) Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN SAMPLING HEIGHT (M AGL) Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN INLET TYPE Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN SAMPLING HUMIDITY OR TEMPERATURE CONTROLNot applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN LABORATORY ANALYTICAL METHOD Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN SAMPLE PREPARATION Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN BLANK CORRECTION Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN VOLUME STANDARDIZATION Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE COLUMN INSTRUMENT NAME AND MODEL NUMBERNot applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable Not applicable*TABLE BEGINS

CAPMCAONEGB_ P 8/1/1995 8:00 8/2/1995 8:00 EST 8/1/1995*TABLE ENDS

NARSTO: Format for a single data entry

What Data Providers Need:

Accurate: verified datasets:

State and Locals have to provide very accurate data for use in attainment designations.

OzonePM-2.5 Annual and Daily (98%tile)

PM-10New Lead Standard? (Monthly)

2 -What Data Providers Need:

Long term datasets utilizing consistent methods and locations

Health StudiesSIP/Trend Analysis Haze monitoring

3-What Data Providers Need:

Real-Time Public Access: The public, the press and Government officials demand information. This data does not have to be verified.

In the past data turnaround was so slow, monitoring Agencies were not even asked unless the inquiry was retrospective.

New York Speciation Trends PM-2.5 Mass vs PM-2.5 FRM Mass (Linear Relationships)All Data Available April 2001 through July 2002 (Average R Squared = .94)

0

5

10

15

20

0 2 4 6 8 10 12 14 16 18 20

FRM Mass ug/m3

ST

N M

ass

ug

/m3

Buffalo IS 52 South Bronx NYBG North Bronx Pinnacle

Queens College Rochester Whiteface Mt Unity

Annual Standard Min Max

STN Mass is not the same as mass from the FRM

4- What Data Providers Need: A method to explain data inconsistencies that do not make the Public distrust the provider

This is harder to explain when it effects individual components preferentially: The lower flow rate of the SuperSass sampler makes it more efficient for the

collection of volatile species.

PM-2.5 FRM, FDMS and TEOM in NYC

0

20

40

60

80

100

5/8/04 5/9/04 5/10/04 5/11/04 5/12/04 5/13/04 5/14/04 5/15/04 5/16/04

ug

/m3

FDMS TEOM FRM

The FDMS data indicates that the FRM and the 50 Deg C TEOM did not capture a substantial fraction of PM-2.5 during this pollution episode.

5- What Data Providers Need: An easier method to communicate dataset issues that compromise data quality: Sodium Filter Contamination (After more than 2 Yrs of complaining the Sodium Data was Flagged in AQS)

New York State PM-2.5 Sodium DataSpeciation Trends Network Quarterly Averages (Smoothed by Location)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

2000-2Qtr 2000-3Qtr 2000-4Qtr 2001-1Qtr 2001-2Qtr 2001-3Qtr 2001-4Qtr 2002-1Qtr 2002-2Qtr

ug/m3

Pinnacle State Park Buffalo Rochester IS 52 South Bronx

NYBG North Bronx Queens College Whiteface Mt

This data is compromised but it has not been flagged because the Lab Denies that Ammonium concentrations were effected by Na contamination (This data is still in AQS)

New York State PM-2.5 Ammonium DataSpeciation Trends Network Quarterly Averages (Smoothed by Location)

0

0.5

1

1.5

2

2.5

3

2000-2Qtr 2000-3Qtr 2000-4Qtr 2001-1Qtr 2001-2Qtr 2001-3Qtr 2001-4Qtr 2002-1Qtr 2002-2Qtr

ug/m3

Pinnacle State Park Buffalo Rochester IS 52 South Bronx

NYBG North Bronx Queens College Whiteface Mt

Summary: Needs of Data Providers:

Access to accurate, validated data (including from other States)

Access to long data records (20+ years)

Access for the Public for near real-time reports and warnings

Easy input of short term special purpose, special study data

Method to communicate meta data and nuances to users: flags, inconsistent methods, different sampler type, different lab analyses, errors

Method to communicate changes or updates to data sets. NY is currently changing 2006 FRM data to fix error in null reporting

End of Presentation