1

Alberta Agriculture and Food (AF)

Surface Meteorological Stations and Data Quality Control Procedures

2

Presentation Overview

• Existing and proposed (AF) network

• Data QA/QC• Parameter list

• Quality states

• QA/QC checks

• Data filling

• Conclusions

3

Meteorological Station Expansion

• 67 N-R-T scalable station platforms• ►all season ppt (GEONOR) ►temperature ►humidity

• ► GOES platform ► 2M wind speed

• ► Campbell Cr10x-2m loggers

• Additional sensors can be added later

• Data will be freely accessible and sensors can be added by any one with dollars, with the caveat that all data would be public domain.

• Currently 44 are installed and operational

• 23 more will be operational by May 1, 2008

4

AF Stations:(N = 113)

• Common Elements• ►All season ppt (GEONOR) ►Temperature ►Humidity• ►Wind speed 2 m ►GOES platform ► Campbell Cr10x-2m loggers

• 36 Drought Net Stations (AGDM)• Incoming short-wave solar radiation (26)• Net solar radiation (3)• Wind speed and direction at 10 m (36)• Soil moisture and temperature at 5, 20, 50, 100 cm (30)

• 10 IMCIN Stations• Incoming short-wave solar radiation (10)• Wind speed and direction at 10 m (10)

• 67 Agriculture Climate Monitoring Stations (AGCM)• Wind direction at 2m (15)• Incoming Short-wave solar radiation(15)

5

6

Existing and proposed stations in Alberta’s Near-Real-Time Network

AGDM (AF)

ACGM (AF)

IMCIN (AF)

Other (AENV AES SRD)

20 km buffer

7

A QA/QC and Data-Filling Decision Support System for

Near Real-Time Climate DataProviding computer-assisted quality assurance, quality control and data

filling

8

Parameter List (Hourly)

• Temperature

• Humidity

• Solar Radiation

• Wind Speed

• Wind Direction

• Precipitation (hourly and 6 hourly)

• Soil Moisture

• Soil Temperature

9

Quality States

• Valid• Not needed to be checked by a human

• Suspect• Needs to be checked by a human and validated or filled

• Invalid• Needs to be checked by a human and filled

• Missing • Needs to be checked by a human and filled

10

QA/QC Checks• Range

• within a reasonable range

• Step • maximum allowable change

• Persistence • minimum allowable change

• Like Sensor • similar value to similar sensors

• Spatial • similar value to neighboring stations (parameter dependent)

11

Methodology for Defining QA/QC checks

• We used the hourly period of record supplied by Environment Canada that contains >25 million records from 250 stations in and around Alberta

• An adjustable trigger point for the “suspect” occurrences was set at 0.01% (1:10,000) for each test• Arbitrary and adjustable (default or station specific)

• For 200 stations examine 50hourly values per day

12

Range Checks

• Three range checks

1. Valid

2. Suspect

3. Invalid

• If the data falls within the inner range then it will be marked Valid If it falls in between the outer range and the inner range it will be marked Suspect

• If data falls outside the outer range it will be marked as Invalid

• If the data is missing it will be marked Missing and then filled

Invalid

Suspect

Valid

Hourly Parameter of Interest

Invalid

Suspect

Valid

Hourly Parameter of Interest

13

Range Checks:Solar Radiation

-0 950

Invalid

Suspect

Valid

Hourly Solar Radiation (W m-2)

• Use wave function to define when day light occurs: Daylight = f(latitude, Julian day)

14

Range Checks:Temperature:

Monthly Max./Min. Extreme (0.01%) Air Temperature Readings

-60

-40

-20

0

20

40

60

1 2 3 4 5 6 7 8 9 10 11 12

Month

Te

mp

era

ture Max. Temp. Obs. Max. Temp. Est.

Min. Temp. Obs Min. Temp. Est.

15

Data Filling

• Temporal filling

• Spatial filling (IDW)

• Spatial-temporal filling (IDW+)

• Manual filling

In every parameter’s daily rollup you know how many records were filled so you can judge the validity of the daily value

16

Conclusions

• Relatively dense high quality and scalable network in the Agricultural area of Alberta

• We have a state of the art QA/QC process that is both flexible and data driven

• Reduces man power

• Capable of generating error logs for maintenance checks

17

18

Persistence Check

valid

valid

susp.susp. susp.

valid

Difference of Maximum and Minimum over n steps must begreater than y

Persistance Checks

19

Step

valid

valid

susp.

valid validvalid

Difference of maximum and minimum over n steps must beat most y

Step Checks

20

Other Tests

• Like Sensors• Relating wind speed 2M to wind speed 10M

• Relating occurrence of precipitation to humidity

• Nearest Neighbors

21

Temporal Filling:for most parameters

One value missing either sideSimply average of two values adjacent values

If more than X consecutive values are missing use spatial interpolation

Data Filling

• 3 for most parameters

• 6 for Soil Temperature

• 12 for Soil Moisture

Missing or Invalid

M M M M

Up to X values missing linearly interpolate missing values from valid end points

22

Spatial filling:Inverse Distance Weighting

Barnwell

Vauxhall

Enchant

Wrentham

Lethbridge

Iron Springs

42.5 km

30.3 km

35.9 km

33.9 km

40.5 km

• Adjustable parameter dependent radius

• Max 8 neighbors

• Rainfall = 70 km radius

• Other = 120 km radius

• Else use nearest station if within X radius

• Else use nearest station and mark as suspect

Data Filling

23

Spatial-Temporal Filling:Precipitation

Total ppt. at Barnwell using IDW = 16.4

Station 12-Oct 13-Oct 14-Oct 15-Oct 16-Oct 17-Oct 18-Oct Distance TotalBarnwell

Vauxhall 0.0 0.0 2.3 1.8 0.0 0.0 1.1 30.3 5.2

Iron Springs 0.0 0.0 11.8 1.0 0.2 0.0 3.0 33.9 16.0

Wrentham 0.0 0.0 15.4 0.4 1.2 2.7 1.4 35.9 21.1

Lethbridge 0.0 0.5 28.5 4.0 1.5 1.0 2.5 40.5 38.0

Enchant 0.0 0.0 6.6 0.0 0.6 0.0 1.2 42.5 8.4

0.44*16.4 0.35*16.4 0.21*16.4

Station 12-Oct 13-Oct 14-Oct 15-Oct 16-Oct 17-Oct 18-OctVauxhall 0% 0% 44% 35% 0% 0% 21%

Distribution of total ppt. by day at nearest station

Station 12-Oct 13-Oct 14-Oct 15-Oct 16-Oct 17-Oct 18-Oct TotalBarnwell 0.0 0.0 7.2 5.7 0.0 0.0 3.5 16.4

Estimated ppt at Barnwell

Data Filling