update on the collaborative radar acquisition field test (craft): planning for the next steps

Update on the Collaborative Update on the Collaborative Radar Acquisition Field Radar Acquisition Field

Test (CRAFT): Test (CRAFT): Planning for the Next StepsPlanning for the Next Steps

Kelvin K. DroegemeierKelvin K. DroegemeierUniversity of OklahomaUniversity of Oklahoma

Presented toPresented toNEXRAD Technical Advisory CommitteeNEXRAD Technical Advisory Committee

3 December 20013 December 2001

NCDC

http://www.nsf.gov/start.htm

http://www.roc.noaa.gov/

s fn

The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test

(CRAFT)(CRAFT) A Collaborative project to demonstrate the real A Collaborative project to demonstrate the real

time compression and Internet-based time compression and Internet-based transmission of WSR-88D base (Level II) datatransmission of WSR-88D base (Level II) data

Motivated by Motivated by – Storm-scale NWP research at CAPS (now for WRF)Storm-scale NWP research at CAPS (now for WRF)– Decision support system development (NSSL)Decision support system development (NSSL)– Desire to improve long-term archive rate (NCDC)Desire to improve long-term archive rate (NCDC)

Philosophy: Leverage existing infrastructures Philosophy: Leverage existing infrastructures (Internet), public-domain software (Unidata (Internet), public-domain software (Unidata LDM), and low-cost services (56K phone lines)LDM), and low-cost services (56K phone lines)

CRAFT was established by the OU Center CRAFT was established by the OU Center for Analysis and Prediction of Storms for Analysis and Prediction of Storms (CAPS) in collaboration with the (CAPS) in collaboration with the – UCAR Unidata Program OfficeUCAR Unidata Program Office– NOAA WSR-88D Operational Support Facility NOAA WSR-88D Operational Support Facility

(now Radar Operations Center)(now Radar Operations Center)– University of WashingtonUniversity of Washington– National Severe Storms LaboratoryNational Severe Storms Laboratory

Now coordinated by CAPS, NSSL, ROC, Now coordinated by CAPS, NSSL, ROC, Unidata, and NCDC Unidata, and NCDC

s fn

ParticipantsParticipants

s fn

Current Technical StrategyCurrent Technical Strategy

Repeater Hub

RIDDS Linux PC + LDM + Compression

($1000)

LDM Server

Dedicated 56K line or ($2000 - $6000/year)

or NWS LANOptional Router

($2000)

WSR-88D

At the radar site

Users

Internet or Abilene

BZIP2 off-the-shelf compression algorithmBZIP2 off-the-shelf compression algorithm– maximum 16:1 in clear air; 8:1 in the worst case; maximum 16:1 in clear air; 8:1 in the worst case;

average of about 12:1average of about 12:1 56K lines found to be inadequate in some cases 56K lines found to be inadequate in some cases

(squall lines) – maybe 10% of time(squall lines) – maybe 10% of time– Exceed available bandwidth by 10-20%Exceed available bandwidth by 10-20%– Can create latencies of 10s of minutesCan create latencies of 10s of minutes– Up to 26% improvement in compression possible in Up to 26% improvement in compression possible in

BZIP2, but requires changing code at user endBZIP2, but requires changing code at user end– A pre-processing algorithm (NSSL) provides 10% A pre-processing algorithm (NSSL) provides 10%

improvement – no changes in BZIP requiredimprovement – no changes in BZIP required– 56K line overrun may not be an issue – used for initial 56K line overrun may not be an issue – used for initial

testing because of cost and securitytesting because of cost and security– New scan strategies and availability of cheaper, faster New scan strategies and availability of cheaper, faster

links (even cable modems) may render 56K lines mootlinks (even cable modems) may render 56K lines moot

s fn

Technical HighlightsTechnical Highlights

Radars Now Delivering Real Radars Now Delivering Real Time Level II Data to NCDC/OUTime Level II Data to NCDC/OU

s fn

Cedar City

TulsaLas Vegas

Flagstaff

Tucson

Amarillo

Lubbock

Oklahoma CityFort Smith

Dallas/Fort Worth

WilmingtonYuma Phoenix

Seattle

Huntsville

Norfolk

Charleston

Morehead City

Columbia

Raleigh

Denver

2121

Courtesy Steve DelGreco (NCDC)

Old WSR-88D Level II Data Processing at Old WSR-88D Level II Data Processing at NCDCNCDC

WSR-88DWSR-88D

Abilene T3Abilene T3

Compressed Level II DataCompressed Level II DataFrom 21 RadarsFrom 21 Radars

LDM HDSS

MassMass Storage Storage

At NCDC TodayAt NCDC Today

Modified from Steve DelGreco (NCDC)

via FTPvia FTP

Processing/Processing/RecompressionRecompression

s fn

WSR-88D Level II Archival at WSR-88D Level II Archival at NCDCNCDC% Rcpts % Rcpts #Sites#Sites

Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212

Total 88D NetworkTotal 88D Network 55.755.7158158

s fn

WSR-88D Level II Archival at WSR-88D Level II Archival at NCDCNCDC

% Rcpts % Rcpts #Sites#Sites

Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212

Total 88D NetworkTotal 88D Network 55.755.7158158

CRAFT EFFICIENCY IS ABOVE 95%CRAFT EFFICIENCY IS ABOVE 95%

University of Iowa requested 19.4 gigabytes University of Iowa requested 19.4 gigabytes (uncompressed) of Level II data(uncompressed) of Level II data

The NCDC was able to retrieve it from the The NCDC was able to retrieve it from the HDSS, compress it, and stage it to FTP in HDSS, compress it, and stage it to FTP in 2 2 hours and 5 minuteshours and 5 minutes

Data was then moved over the Internet in a Data was then moved over the Internet in a matter of minutesmatter of minutes

The old 8 mm tape system would have The old 8 mm tape system would have required required – 58 tapes58 tapes– 9 workstations each running for 26 hours9 workstations each running for 26 hours– A total of A total of 232 hours232 hours

Improvement is more than a factor of 100 in Improvement is more than a factor of 100 in time, not including personnel coststime, not including personnel costs

s fn

A Notable NCDC Success A Notable NCDC Success StoryStory

s fn

0

20

40

60

80

100

120

0 5 10 15 20 25 30

Radar Data LatencySEAAMACMBFLGFTWSRPTULLBBFSMOKCTCS

Hour

PreliminaryPreliminary Latency DataLatency Data

ESDIM is fundingESDIM is funding– Analysis of latency, bandwidth, etcAnalysis of latency, bandwidth, etc– Network simulation (reliability, Network simulation (reliability,

redundancy)redundancy) Additional analysis being conducted in Additional analysis being conducted in

collaboration with the private sectorcollaboration with the private sector

s fn

Networking R&DNetworking R&D

Additions by End of CY 2001Additions by End of CY 2001

s fn

Cedar City

TulsaLas Vegas

Flagstaff

Tucson

Amarillo

Lubbock


Dallas/Fort Worth


Seattle

Huntsville

Norfolk

Charleston

Morehead City

Columbia

Raleigh

Denver

Rapid City

Melbourne

Buffalo

Chicago

Pittsburgh

Cleveland

Cincinnati

Detroit

N. Indiana

21+ 21+ 1111

Cheyenne

Goodland

s fn

Cedar City

TulsaLas Vegas

Flagstaff

Tucson

Amarillo

Lubbock


Dallas/Fort Worth


Seattle

Huntsville

Norfolk

Charleston

Morehead City

Columbia

Raleigh

Denver

Rapid City

Melbourne

Buffalo

Chicago

Pittsburgh

Cleveland

Cincinnati

Detroit

N. IndianaCheyenne

Goodland

Dodge CityWichita Springfield

Little Rock

Shreveport

Kansas City

Additions by Spring, 2002Additions by Spring, 2002

21+ 21+ 11 + 11 + 66

IHOP Field ProgramIHOP Field ProgramPlus CAPS FAAPlus CAPS FAAResearchResearch

s fn

Cedar City

TulsaLas Vegas

Flagstaff

Tucson

Amarillo

Lubbock


Dallas/Fort Worth


Seattle

Huntsville

Norfolk

Charleston

Morehead City

Columbia

Raleigh

Denver

Rapid City

Melbourne

Buffalo

ChicagoPittsburgh

Cleveland

Cincinnati

Detroit

N. Indiana


Little Rock

Shreveport

Kansas City

Cheyenne

Goodland

Enid

Altus

Military RadarsMilitary Radars

21+21+11+11+66+2+2

s fn

Cedar City

TulsaLas Vegas

Flagstaff

Tucson

Amarillo

Lubbock


Dallas/Fort Worth


Seattle

Huntsville

Norfolk

Charleston

Morehead City

Columbia

Raleigh

Denver

Rapid City

Melbourne

Buffalo

ChicagoPittsburgh

Cleveland

Cincinnati

Detroit

N. Indiana


Little Rock

Shreveport

Kansas City

Cheyenne

Goodland

Enid

Altus

FAA-MIT/LL Additions FY 2002FAA-MIT/LL Additions FY 2002

Central Ill

Milwaukee

Quad Cities

St. Louis

Paducah

Louisville

Charleston

Baltimore

Central PAPhiladelphia

New York City

AlbanyBoston

Binghamton

4040+14=+14=5454

s fn

Current Technical StrategyCurrent Technical Strategy

Repeater Hub

RIDDS Linux PC + LDM + Compression

($1000)

Server

Dedicated 56K line or ($2000 - $6000/year)

or NWS LANOptional Router

($2000)

WSR-88D

At the radar site

Users

Internet or Abilene

s fn

Interim BDDSInterim BDDS

Repeater Hub

BDDS Linux PC + LDM + Compression

($1000)

Server

Dedicated 56K lines or or NWS LAN

Optional Router ($2000)

WSR-88D

At the radar site

Users

Internet or Abilene

s fn

Final BDDSFinal BDDS

Repeater Hub

BDDS+LDM/Compression

ServerDedicated 56K lines or or

NWS LAN

WSR-88D

At the radar site

Users

Internet or Abilene

How Does This How Does This Experiment Experiment Become an Become an Operational Operational

System?System?

s fn

Abilene NetworkAbilene NetworkJanuary 1999 January 1999

Cleve

land

New York

Atlanta

Indianapolis

Kansas City

Houston

Denver

Los Angeles

Sacramento

Seattle

Abilene Router Node

Abilene Access Node

Operational January 1999

Planned 1999

One Possible StrategyOne Possible Strategy

s fn

NOAA Facility

via phone linesor commodity

Internet

Silver SpringSilver Spring

s fn


Internet

Abilene NetworkNOAA Facility

NOAA Facility NOAA Facility


NOAA Facility

NormanNorman

BoulderBoulder

Silver SpringSilver Spring SeattleSeattle

MiamiMiami

AtlantaAtlanta


s fn


Internet




NOAA Facility

NormanNorman

BoulderBoulder

Silver SpringSilver Spring SeattleSeattle

MiamiMiami

AtlantaAtlanta

““Super Sites”: Each LDM on Abilene Super Sites”: Each LDM on Abilene ““bus” Carries all 88D data -- redundancybus” Carries all 88D data -- redundancy


s fn




NOAA Facility

s fn

Abilene Network

Commodity InternetCommodity Internet

Commodity InternetCommodity InternetCommodity InternetCommodity Internet

Commodity InternetCommodity Internet

Com

mod

ity

Inte

rnet

Com

mod

ity

Inte

rnetC

omm

odity Internet

Com

modity In

ternet

NOAA Facility



NOAA Facility


s fn




NOAA Facility

Private Sector

Academia

s fn




NOAA Facility

Private Sector

Academia

Private Sector

Academia

Private Sector

Academia

Private Sector

Academia

Private Sector

Academia

Private Sector

Academia

Meeting all of the NeedsMeeting all of the Needs

88D 88D 88D 88D 88D

Abilene-based network of (aggregating & relaying) LDMs, each carrying all 88D data, with redundant interconnections

Private 3

Academia 2Private 3

Academia 1

Private 4

Private 1

NOAA 3

NOAA 2

NOAA 1

Courtesy D. Fulker, UCAR

– NOAA runs its operational ingest system but links to NOAA runs its operational ingest system but links to private sector and academic systems, each providing private sector and academic systems, each providing redundancy for the otherredundancy for the other

– Entire community benefits – builds upon the Entire community benefits – builds upon the collaborative spirit of CRAFTcollaborative spirit of CRAFT

– Configuration completely scalableConfiguration completely scalable– Leverages existing infrastructure (LDM sites, Abilene, Leverages existing infrastructure (LDM sites, Abilene,

WFO connection to commodity Internet)WFO connection to commodity Internet)– Significant capacity for future growth (dual-pol, Significant capacity for future growth (dual-pol,

phased array)phased array)

Benefits of This ConceptBenefits of This Concept

s fn

s fn

Weber (2000)

Longer Term: Addition of FAA Longer Term: Addition of FAA RadarsRadars

update on the collaborative radar acquisition field test (craft): planning for the next steps

Documents