Archived Data Management System

Study Advisory Committee MeetingMay 14, 2003

Preliminary Data Analysis

2002 AADT Estimation TRIMARC ARTIMIS

Procedure

1. Data extraction and screeningApplied quality check criteria used in mobility monitoring study by TTI

Problem encountered:The rounding of occupancy in

the segment file created problems.

Potential solution:Eliminating the vehicle length criteria

Procedure (cont’d)

2. Summarize 15min volume to hourly volume

If there are at least 2 15min records, calculate hourly volume by adding or extrapolating;

Otherwise, mark hourly volume as missing/null.

Procedure (cont’d)

3. Summarize hourly volume to daily volume

If there are 24 hourly volume records for a day, add them.

If there are 18-23 hourly volume records for a day, impute the hourly volumes for those “missing” hours based on hourly distribution at the same site throughout the year, and then add them.

Otherwise, mark the day as missing/null.

Procedure (cont’d)

For each site, screen out daily volumes on those days of week based on ±1.5σ from the day-of-week mean.

Calculate day-of-week volume distribution using yearly average for each day of week.

Procedure (cont’d)6. Calculate AADT using the AASHTO formulation (I)

where:

VOL = daily traffic for day k, of day-of-week i, and month j i = day of the week j = month of the year k = 1 when the day is the first occurrence of that day of the week

in a month, 4 when it is the fourth day of the week. n = the number of days of that day of the week during that month

(usually between 1 and 5, depending on the number of missing data).

Quality Control CriteriaQuality Control Test and Description Sample Code with Threshold Values Action

Controller error codesSpecial numeric codes that indicate that controller or system software has detected an error or a function has been disabled.

If VOLUME={code} or OCC={code} or SPEED={code} where {code} typically equals “-1” or “255”

Set values with error codes to missing/null, assign missing value flag/code.

No vehicles presentSpeed values of zero when no vehicles presentIndicates that no vehicles passed the detection zone during the detection time period.

If SPEED=0 and VOLUME=0 (and OCC=0) Set SPEED to missing/null, assign missing value codeNo vehicles passed the detection zone during the time period.

Consistency of elapsed time between recordsPolling period length may drift or controllers may accumulate data if polling cycle is missed.Data collection server may not have stable or fixed communication time with field controllers.

Elapsed time between consecutive records exceeds a predefined limit or is not consistent

Action varies. If polling period length is inconsistent, volume-based QC rules should use a volume flow rate, not absolute counts.

Duplicate recordsCaused by errors in data archiving logic or software process.

Detector and date/time stamp are identical. Remove/delete duplicate records.

QC1-QC3: Logical consistency testsTypically used for date, time and location.Caused by various types of failures.

If DATE={valid date value} (QC1)If TIME={valid time value} (QC2)If DET_ID={valid detector location value} (QC3)

Write to off-line database and/or remove records with invalid date, time or location values.

QC4: Maximum volumeTraffic flow theory suggests a maximum traffic capacity.

If VOLUME > 17 (20 sec.)If VOLUME > 25 (30 sec.)If VOLUME > 250 (5 min.)If VPHPL > 3000 (any time period length)

Assign QC flag to VOLUME, write failed record to off-line database, set VOLUME to missing/null.

QC5: Consecutive identical volume valuesResearch and statistical probability indicates that consecutive runs of identical data values are suspect.Typically caused by hardware failures.

No more than 8 consecutive identical volume values Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

Quality Control Criteria (cont’d)Quality Control Test and Description Sample Code with Threshold Values Action

QC6: Maximum occupancyEmpirical evidence suggests that all data values at high occupancy levels are suspect.Caused by detectors that may be “stuck on.”

If OCC > 95% (20 to 30 sec.)If OCC > 80% (1 to 5 min.)

Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

QC7: Minimum speedEmpirical evidence suggests that actual speed values at low speed levels are inaccurate.

If SPEED < 5 mph Assign QC flag to SPEED, write failed record to off-line database, set SPEED value to missing/null

QC8: Maximum speedEmpirical evidence suggests that actual speed values at high speed levels are suspect.

If SPEED > 100 mph (20 to 30 sec.)If SPEED > 80 mph (1 to 5 min.)

Assign QC flag to SPEED, write failed record to off-line database, set SPEED value to missing/null

Maximum reduction in speedEmpirical evidence suggests that speed reductions greater than some maximum value are suspect.

If SPEEDn+1 < (0.45 SPEEDn) Assign QC flag to SPEED, write failed record to off-line database, set SPEED value to missing/null

QC9: Multi-variate consistencyZero speed values when volume (and occupancy) are non-zeroSpeed trap not functioning properly

If SPEED = 0 and VOLUME > 0 (and OCC > 0) Assign QC flag to SPEED, write failed record to off-line database, set SPEED value to missing/null

QC10: Multi-variate consistencyZero volume values when speed is non-zero.Unknown cause.

If VOLUME = 0 and SPEED > 0 Assign QC flag to VOLUME, write failed record to off-line database, set VOLUME to missing/null

Quality Control Criteria (cont’d)Quality Control Test and Description Sample Code with Threshold Values Action

QC11: Multi-variate consistencyZero speed and volume values when occupancy is non-zero.Unknown cause.

If SPEED = 0 and VOLUME = 0 and OCC > 0 Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

QC12: Truncated occupancy values of zeroCaused when software truncates or rounds to integer valueCalculate maximum possible volume (MAXVOL) for an occupancy value of “1”:

If OCC = 0 and VOLUME > MAXVOL whereMAXVOL=(2.932*ELAPTIME*SPEED)/600

Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

QC13: Minimum average effective vehicle length (AEVL)This test computes an avg. vehicle length using volume, occupancy and speed. The test then assumes a minimum avg. length.

If AEVL < 9 ft. where AEVL=(SPEEDOCC52.8ELAPTIME)/(VOLUME*3600)

Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

QC14: Maximum average effective vehicle length (AEVL) This test computes an avg. vehicle length using volume, occupancy and speed. The test then assumes a minimum avg. length.

If AEVL > 60 ft. where: AEVL=(SPEEDOCC52.8ELAPTIME)/(VOLUME*3600)

Assign QC flag to VOLUME, OCCUPANCY and SPEED; write failed record to off-line database; set VOLUME, OCCUPANCY and SPEED to missing/null

Data Screening Summary2002 Volume Data Analysis TRIMARC ARTIMIS

Total number of segments within KY 53 82

Total number of EXPECTED 15min data records 1,857,120 2,873,280

Total # of 15min records 1,727,455 544920

Completeness percentage 93.0% 19.0%

Quality Check Controller error codes 0 0

No vehicle present 19,265 69,221

Consistency of elapsed time between records 0 0

Duplicate record 5,792 75

Invalid date, time 1,242 0

Maximum volume (750) 10 15

8 Consecutive identical volume 27480 183,472

Maximum occupancy (80%) 385 20

Multi-variate consistency (Vol = 0,Speed > 0) 1,551 0

Multi-variate consistency (Speed = 0,Vol=0,OCC<>0) 0 0

Truncated occupancy values of zero 11 3

Minimum average effective length (9ft) 156,164 N/A

Maximum average effective length (60ft) 33,711 N/A

Total flagged 218593 183,522

Total # of valid records AFTER QUALITY CHECK 1,503,070 361,323

% passing quality check (compared to raw data) 87.0% 66.3%

Data Screening Summary (cont’d)2002 Volume Data Analysis TRIMARC ARTIMIS

15min records used for hourly volume calculation 1,492,768 361,323

Total number of hourly records factored from 15min data 391,442 82,917

Average # of hours per segment 7,386 1,011

% of 15min records used to calculate hourly volume 86.4% 66.3%

15min records used for daily volume calculation 1,392,101 352,682

% of 15min records used to calculate daily volume 80.6% 64.7%

Hourly records used for daily volume calculation 359,793 80,715

Total number of daily records grouped from hourly data 15,453 3,361

Average # of days per segment 292 41

TRIMARC Data Quality2002 TRIMARC Overall Volume Data Quality

93.0%87.4% 86.4%

80.6%

Completeness percentage % passing quality check(compared to raw data)

% of 15min records used tocalculate hourly volume

% of 15min records used tocalculate daily volume

ARTIMIS Data Quality2002 ARTIMIS Overall Volume Data Quality

19.0%

66.3% 66.3% 64.7%

Completeness percentage % passing quality check(compared to raw data)

% of 15min records used tocalculate hourly volume

% of 15min records used tocalculate daily volume

Alternative Methods Calculate Monthly ADT (MADT) and multiply

it with the monthly factor to get AADT (II)

Estimate MADT using two-week’s of “good” data (with minimum 15min records marked as missing/null) and multiply it with the monthly factor to get AADT (III)

7

1 1

1

7

1

i

n

kijkj VOL

nMADT

2003 Monthly Factor (Division of Planning, KYTC)

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

1 - Rural Interstate All Week 1.23 1.14 1.03 0.99 0.97 0.91 0.91 0.92 0.99 0.98 0.99 1.08

Weekend 1.34 1.17 1.01 0.99 0.98 0.89 0.90 0.89 0.99 0.97 0.99 1.10

Weekday 1.17 1.12 1.04 0.99 0.97 0.93 0.93 0.95 0.98 0.98 1.00 1.07

2 - Rural General All Week 1.18 1.07 1.03 0.97 0.94 0.94 0.96 0.94 0.98 0.98 1.02 1.13

Weekend 1.23 1.07 1.01 0.98 0.94 0.93 0.96 0.93 0.99 0.99 1.00 1.11

Weekday 1.16 1.08 1.03 0.97 0.95 0.95 0.96 0.96 0.98 0.98 1.04 1.13

3 - Urban General All Week 1.10 1.02 0.99 0.97 0.95 0.96 0.99 0.97 1.00 1.00 1.03 1.07

Weekend 1.14 1.03 0.98 0.97 0.96 0.95 1.00 0.96 1.01 1.01 1.01 1.06

Weekday 1.07 1.02 1.00 0.96 0.95 0.97 0.99 0.98 1.00 1.00 1.04 1.08

4 - Rural Recreation All Week 1.35 1.21 1.10 0.96 0.87 0.79 0.76 0.84 0.96 1.06 1.22 1.44

Weekend 1.50 1.26 1.14 0.98 0.83 0.76 0.75 0.78 0.94 1.06 1.26 1.55

Weekday 1.24 1.17 1.07 0.95 0.90 0.83 0.76 0.89 0.99 1.06 1.19 1.36

5 - Urban Interstate All Week 1.09 1.03 0.99 0.97 0.97 0.96 0.99 0.97 1.03 1.00 1.02 1.07

Weekend 1.15 1.05 0.97 0.98 0.98 0.94 0.98 0.95 1.04 1.01 1.01 1.04

Weekday 1.06 1.02 0.99 0.97 0.97 0.97 0.99 0.98 1.01 1.00 1.03 1.08

TRIMARC Summary (Method I)

TRIMARC Summary (Method II)

TRIMARC Summary (Method II) (cont’d)

TRIMARC Summary (Method II) (cont’d)

TRIMARC Summary (Method II) (cont’d)

TRIMARC Summary (Method II) (cont’d)

ARTIMIS Summary (Method I)

ARTIMIS Summary (Method II)

ARTIMIS Summary (Method II) (cont’d)

ARTIMIS Summary (Method II) (cont’d)

ARTIMIS Summary (Method II) (cont’d)

AADT Estimates at Sample Sites

TRIMARC ARTIMIS

Segment SKYI71001S SEGK715035

Device WBR039 LFC076(N)

MP I-71 0.0 I-71/75 183.1

State AADT (one direction) 35450 % of State # 86500 % of State #

AADT Estimates

IDetector data only

29724 83.8% 77129 89.2%

II MADT + MF 31850Sep1.03

89.8% 75835Jan1.09

87.7%

IIITwo week + MF (all week)

32378Sep1.03

91.3% 76952Jan1.09

89.0%

Two week + MF (weekday)

35335Sep1.01

99.7% 80453Jan1.06

93.0%

Two week + MF (weekend)

24641Sep1.04

69.5% 64208Jan1.15

74.2%

Observations

Data availability varied by sites in 2002.

The rounding of occupancy data may have caused large amount of data being screened out by vehicle length criteria, which were hence dropped when processing ARTIMIS data.

Observations (cont’d)

Generally, the AADT estimates obtained from method II are closer to the State figure when more valid days are present for a month.

Significant differences exist between AADT estimates obtained using monthly factors for all week, weekday, and weekend (method III).