e-vlbi: connecting the world’s radio telescopes with high-speed networks alan r. whitney mit...

e-VLBI: Connecting the World’s Radio Telescopes

with High-Speed Networks

Alan R. WhitneyMIT Haystack Observatory

Westford, Massachusetts, USA

Traditional VLBIThe Very-Long Baseline Interferometry (VLBI) Technique(with traditional data recording on magnetic tape or disk)

The Global VLBI Array(up to ~20 stations can be used simultaneously)

• Science Missions– Development of the very early universe

– Formation and evolution of galaxies

– Black holes in galactic cores

– Stellar nurseries

– Evolution and formation of elements

VLBI for Astronomy

• Combines data from global telescope array to synthesize a very large, distributed antenna

• Highest resolution technique available to astronomers – tens of microarcseconds (corresponds to resolving dimples on a golf ball at distance 5000 km!)

• Allows detailed studies of the most distant objects in the Universe

• Data is derived from national and international array of large radio telescopes

• Disadvantage: Very large amount of data and huge amount of computing required to make a single image

VLBI for Astronomy

VLBI astronomy example


Galaxy NGC6251

• Distance: ~500Mlight-yrs

• 1 parsec = 3.262 light-yrs = ~0.1arcseconds

• Allows observations to focus in on energetic core

Chautauqua 2001

Quasars, hotspots, polarization

Resolution: ~10 marcsec


• Fundamental VLBI measurement is time-of-arrival difference of signals between telescopes in array; typical single-measurement precision is <10 picosec

• Highest precision (few mm) technique available for global tectonic measurements

• Highest precision Earth Orientation and Earth Rotation measurements

• Earth-rotation measurements are important for military/civilian navigation

• Stable reference frame formed by distant quasars• Fundamental calibration for GPS constellation within Celestial

Reference Frame• Highest spatial and time resolution of Earth’s motion in

space for the study of Earth’s interior• International observing program includes ~35 stations

around the world, coordinated by International VLBI Service

• In South American, stations in Concepcion, Chile and Fortaleza, Brazil are regular contributors

VLBI for Geodesy


Plate Tectonic Motion from VLBI measurements


Correlation between Earth Rotation and Atmospheric Angular Momemtum

Conclusion: Primary driver of variations in Earth rotation rate is weather!

Disadvantages of Traditional ‘Record & Ship’ VLBI

• Long interval between data collection and results (typically weeks to months)

• Uncertainty of proper equipment operation during experiment

• Expensive media (tape, disks) sometimes lost or damaged in transit

• Limited bandwidth limits sensitivity (few Gbps per station is practical maximum)

A Little History: Mark 4 VLBI data system

16-station VLBI correlator at JIVE in The Netherlands

Enter ‘e-VLBI’ –

Electronic transmission of VLBI data over high-speed global networks!

e-VLBI is not new!

• 1979 – OVRO-Haystack 1 Mb/station; 2400-baud modems (Mark III)• 1980’s – JPL DSN for EOP; 500 kb/s sample rate buffered to disc,

56 kbps transfer over land lines • 1995 – Japanese Keystone project; 256 Mbps over dedicated fiber• 1999 – Europe; 1 Mb/station over Internet using ftp• 2000 - Japan develops 1 Gbps e-VLBI system within Japan over

dedicated links• 1977 – NRAO-Algonquin – 20 Mb/sec real-time satellite link

But the advent in global high-speed networks is completing changing the game!

Scientific Advantages of e-VLBI

• Bandwidth growth potential for higher sensitivity– VLBI sensitivity (SNR) proportional to square root of Bandwidth

resulting in a large increase in number of observable objects(only alternative is bigger antennas – hugely expensive)

– e-VLBI bandwidth potential growth far exceeds recording capability(practical recordable data rate limited to few Gbps)

• Rapid processing turnaround– Astronomy

• Ability to study transient phenomena with feedback to steer observations

– Geodesy• Higher-precision measurements for geophysical investigations

• Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

Practical Advantages of e-VLBI

• Increased Reliability– remove recording equipment out of field

– remote performance monitor & control capability in near real-time

• Lower Cost– Automated Operation Possible

• eliminates manual handling and shipping of storage media

– Real-time or near-real-time Processing• forestalls growth of storage-capacity requirements with bandwidth growth

– Elimination of recording-media pool (millions of $’s!)

• Avoid unexpected media-shipping interruptions and losses

Elements of e-VLBI Development

• Develop e-VLBI-compatible data system– Mark 5 system developed at MIT Haystack Observatory with support from several

national and international institutions

• Demonstrate e-VLBI in feasibility experiment– ~700 km link between Haystack Observatory and NASA/GSFC

• Develop specialized e-VLBI data-transport formats and protocols– Develop international standard for e-VLBI data format– New IP-based protocol tailored to operate in shared-network ‘background’ to

efficiently using available bandwidth

• Extend e-VLBI to national and global VLBI community


• Develop e-VLBI-compatible data system

• Demonstrate e-VLBI in feasibility experiment

• Develop specialized e-VLBI data-transport formats and protocols


Mark 5 VLBI Disk-Based Data System

• 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks• Developed at MIT Haystack Observatory with multi-institutional support• Mostly COTS components• Two removable ‘8-pack’ disk modules in single 5U chassis• With currently available 250GB disks – capacity of single ‘8-pack’ 2.0TB; expected to increase to 3.0TB by early 2005 at cost of ~$0.5/GB• GigE connection for real-time and quasi-real-time e-VLBI operations• Inexpensive: <$20K• ~75 Mark 5 systems now installed at stations and correlators

Haystack

Westford

NASA/GSFCUSNO

(correlator)

(correlator)

Bossnet 1 Gbps e-VLBI demonstration experiment

(Fall 2002)

• 788 Mbps e-VLBI transfer achieved over shared IP infrastructure, but took much tuning

• Full report at www.haystack.edu/e-vlbi

Initial experiment

Future ~700

km

Westford-GGAO e-VLBI results

• First near-real-time e-VLBI experiment demonstrated on 6 Oct 02– Recorded data at 1152 Mbps on Westford-GGAO baseline

– GGAO disk-to-disk transfer at average 788 Mbps transfer rate

• Direct data transfer experiment demonstrated on 24 Oct 02– Direct transfer of GGAO data to disk at Haystack at 256 Mbps

– Immediate correlation with Westford data

– Nominal fringes observed

• Conclusion– e-VLBI at near Gbps speeds over ordinary shared networks is possible

but still difficult

VSI-E

• VSI-E = VLBI Standard Interface for e-VLBI

• Follows on heels of VSI-H and VSI-S specifications over last 3 years

• Goal is to allow compatible interchangeable of data between heterogeneous VLBI data sytems

• VSI-E currently under development by international VSI committee

• RTP protocol has been chosen

• Draft specification currently under discussion.

• Goal: Complete VSI-E specification by mid-2004

• Prototype software will be available June 04

RTP Capabilities• RTP provides an Internet-standard format for:

– Transmission of sampled analog data– Dissemination of session information– Monitoring of network and end system performance (by participants and third parties)– Adaptation to varying network capability / performance– Message Sequencing / reordering– Multi-cast distribution of statistics, control and data

• RTP allows the reuse of many standard monitoring / analysis tools• RTP seen as internet-friendly by the network community:

– attention to efficiency • protocol designed to have minimum overhead for in-band data

– attention to resource constraints • won't use up all your bandwidth with control information

– attention to scaling issues

RTCP Capabilities• Monitors network’s real-time data delivery performance

• Statistics collected from receivers

• Information delivered to–Senders (adapt to prevailing conditions)–Network management (identifies faults, provisioning problems)

• Adaptive, bandwidth-limited design

Possible VSI-E Topologies

New Application-Layer Protocols for e-VLBI

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed which are tailored to e-VLBI characteristics; for example:– Can tolerate some loss of data (perhaps 1% or so) in many cases

– Can tolerate delay in transmission of data in many cases

• ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy being developed at Haystack Observatory under 3-year NSF grant:– Will ‘scavenge’ and use ‘secondary’ bandwidth

– ‘Less than best effort’ service will not interfere with high-priority users

– Dr. David Lapsley has joined Haystack staff to lead this effort

Typical bit-rate statistics on Abilene network

Conclusion: Average network usage is only a few % of capacity

Usage >20Mbps less than 1% of the time

100 500 Mbps

0.1

1.0

0.01

0.001

Typical distribution of heavy traffic on Abilene

Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes

secs200 400 1000

1.0

0.9

0.8

0.7

<10% of ‘bulk’ transfers exceed ~100 secs

New Transport Protocols for e-VLBI

• TCP is very inefficient is there are virtually any losses on a network due either to network sharing or due to physical packet losses– e-VLBI is particularly sensitive due to typical long RTT’s

• UDP can used, but is sometimes ‘unfriendly’ to other users

• New transmission protocols are being developed which are much more aggressive, but fair – some examples:– FAST (Caltech)

– Tsunami (Indiana University)

– UDT (UDP-based Data Transfer; Univ. of Illinois)

– High-speed TCP (Sally Floyd)

– XCP (Explicit Control Protocol; Dina Katabi, MIT)

Acceptable Latencies for e-VBLI

• Latency delays must not exceed the ability of the correlator to buffer the data

• Near real-time operation

– At 1 Gbps, disk buffers of large size can be used for; in this case latency is not a real issue.

– At 10 Gbps, disk buffers are not economically feasible and will require real-time correlation

• Real-time operation

– At 1 Gbps, a buffer size of ~0.5GB is currently available from each station;a latency delay of up to ~2 seconds is acceptable

– At 10 Gbps, it is likely that larger buffers will need to be built to accommodate latency delays up to at least 1 second

• Optically switched networks with dedicated wavelengths may eventually put the latency issue to rest as high-performance applications are able to use dedicated-lambda facilities.

Westford-to-Kashima e-VLBI experiments

• First Westford/Kashima experiment conducted on 15 Oct 02

– Data recorded on K5 at Kashima and Mark 5 at Westford at 256 Mbps

– Files exchanged over Abilene/GEMnet networks

• Nominal speed expected to be ~20 Mbps, but achieved <2 Mbps for unknown reasons - investigating

– File formats software translated

– Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima

– Nominal fringes obtained

• Kashima data is now regularly transmitted to Haystack for processing

Westford to Sweden ‘Real-time’ experiments

• ‘Real-time’ experiments conducted in March/April 2004

– Data transmitted and processed in real-time (i.e. no disk buffering); data transmitted directly from stations to correlator

– First experiment at 32 Mbps due to temporary low-speed connection to Haystack

– Plan to extend these experiments to at least 512 Mbps with GigE links

UT1 ‘Intensive’ e-VLBI• Daily ~1 hour VLBI sessions between

Kokee Park, Hawaii and Wettzell, Germany are used for UT1 measurements

• Data are time sensitive since they are used for predicting UT1

• Currently requires ~4 day turnaround shipping media

• These measurements are an ideal candidate for routine e-VLBI

– Short daily session collect <100 GB of data– Even 100 Mbps will allow transfer in a few

hours

• Work now in progress to make necessary connections

– Network being organized from Kokee Park to USNO;connection speed OC-3

– Data from Mark 5 system in Wettzell will be carried over dedicated fiber at ~30Mbps to Univ. of Regensberg, then to GEANT

First e-VLBI to South America!

• First e-VLBI data transmitted from TIGO in Concepion, Chile to Haystack Observatory in April 2004

• Two scans of ~1.5GB each were transmitted at an average rate of 1.0-1.5 Mbps• Hope to continue this effort and upgrade the link!

A sample of international connections• Possibilities for international connections

– Surfnet – U.S. to Europe at 10 Gbps

– TransPAC/APAN – U.S. to Japan at 2.5 Gbps x2

– GEMnet – U.S. to Japan at 2.5 Gbps (privately operated by NEC)

– AMPATH – connections to telescopes in Chile and Brazil

– AARNET – Recently announced 2x10 Gbps connections to Hawaii and U.S.

– IEEAF –Europe/U.S./Japan link at 10 Gbps

– A sample of others under construction• GLORIAD – connecting China and Russis

• TEIN – Paris to Seoul

• EUMEDCONNECT – Europe to Mediterranean

• NeDAP – Europe to Russia

• ALIS – Europe to Latin and South America

– And many others………..

622 Mbps +10 Gbps

Transoceanic donations to IEEAF (in red)Credit: IEEAF

TransPAC Network(planned upgrade in 2004 to 2xGigE plus OC-48)

Credit: J. Williams, IU

AMPATH: Research and Education Network and International Exchange Point for the Americas

Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide

Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico

Provides connectivity to US research programs in the region

AMPATH is a project of FIU and the National Science Foundation’s Advanced Networking Infrastructure & Research (ANIR) Division

Note: VLBI telescopes currently in Chile and Brazil Credit: Julio Ibarra, FIU

0

UKUKUKUK

FRFRFRFR

CHCHCHCH

SESESESE

ITITITIT RORORORO

HRHRHRHR

EEEEEEEE LVLVLVLV LTLTLTLT

IEIEIEIENLNLNLNL

BEBEBEBE

DEDEDEDE

PLPLPLPL

CZCZCZCZ

HUHUHUHUATATATAT

SISISISI

SKSKSKSK

LULULULU

ESESESES

PTPTPTPT

GRGRGRGR

BGBGBGBG

CYCYCYCY ILILILIL622622 3434

3434

4545

622622

622622

155155155155

155155

155155

3434

3434

155155622622

155155 3434 4545

10 Gbps10 Gbps 2.5 Gbps2.5 Gbps

SE - PoP for Nordunet

GÉANT:The connectivity at 10 Gbps

Aarnet: SXTransport Project in 2004 Connect Major Australian Universities to 10 Gbps Backbone Two 10 Gbps Research Links to the US Aarnet/USLIC Collaboration on Net R&D Starting Now

Credit: George McLaughlin, AARNET

GLORIAD: Global Optical Ring (US-Ru-Cn)“Little Gloriad” (OC3) Launched January 12; to OC192 in 2004

Abilene - Upgrade Completed!

Credit: Internet2

National Light Rail Footprint

15808 Terminal, Regen or OADM siteFiber route

NLRStarting Up NowInitially 4x10 Gb

WavelengthsFuture: to 40x10Gb Waves

Transition beginning now to optical, multi-wavelength R&E networks. Also Note: XWIN (Germany); IEEAF/GEO plan for dark fiber in Europe

PITPIT

PORPOR

FREFRE

RALRAL

WALWAL

NASNASPHOPHO

OLGOLGATLATL

CHICHI

CLECLE

KANKAN

OGDOGD

SACSAC BOSBOSNYCNYC

WDCWDC

STRSTR

DALDAL

DENDEN

LAXLAX

SVLSVL

SEASEA

SDGSDG

JACJAC

What is the future of e-VLBI?

• Global connectivity is increasing at a very rapid rate

• e-VLBI is being aggressively developed in the U.S., Europe and Japan and will likely become standard procedure within the next decade.

• In South America, the ALMA project in Chile will be a large driver for high-bandwidth communications to the rest of the world.

Bandwidth Growth of Int’l Bandwidth Growth of Int’l HENP Networks (US-CERN HENP Networks (US-CERN

Example)Example)

Bandwidth Growth of Int’l Bandwidth Growth of Int’l HENP Networks (US-CERN HENP Networks (US-CERN

Example)Example) Rate of Progress >> Moore’s Law. Rate of Progress >> Moore’s Law. (US-CERN Example)(US-CERN Example)

9.6 kbps Analog9.6 kbps Analog (1985) (1985) 64-256 kbps Digital (1989 - 1994) [X 7 – 27]64-256 kbps Digital (1989 - 1994) [X 7 – 27] 1.5 Mbps Shared (1990-3; IBM) [X 160]1.5 Mbps Shared (1990-3; IBM) [X 160] 2 -4 Mbps2 -4 Mbps (1996-1998) [X 200-400] (1996-1998) [X 200-400] 12-20 Mbps (1999-2000) [X 1.2k-2k]12-20 Mbps (1999-2000) [X 1.2k-2k] 155-310 Mbps155-310 Mbps (2001-2) [X 16k – 32k] (2001-2) [X 16k – 32k] 622 Mbps622 Mbps (2002-3) [X 65k] (2002-3) [X 65k] 2.5 Gbps 2.5 Gbps (2003-4) [X 250k] (2003-4) [X 250k] 10 Gbps 10 Gbps (2005) [X 1M] (2005) [X 1M]

A factor of ~1M over a period of 1985-2005 A factor of ~1M over a period of 1985-2005 (a factor of ~5k during 1995-2005)(a factor of ~5k during 1995-2005) HENP has become a leading applications driver, HENP has become a leading applications driver, and also a co-developer of global networks; and also a co-developer of global networks;

Credit: Harvey Newman, Caltech

What are the problems?

• Biggest problem: ‘Last-mile’ connection of telescopes– Most telescopes are deliberately built in remote locations

– Biggest single obstacle is physical cost of laying fiber

– Cost of terminal equipment is rapidly diminishing, even for 10 Gbps

– A concerted international effort must be made to connect every major telescope in the world to high-speed network

• Some countries require buying services from service providers, which is very expensive; lighting dark fiber is far cheaper, if possible

e-VLBI is one among many applications

• HENP – high-energy physics community is currently heaviest user of international networks, mainly in dissemination of very large data files from major HENP facilities in Europe and U.S.

• Astronomy in general – optical telescopes are now sending high-resolution images in real-time to remote observers around the world;NVO will be among the world’s largest distributed database

• Education – tremendous opportunities here– Remote interactive learning from world experts in all fields

– Demonstration of advanced surgical procedures

– Easy international collaboration for all disciplines

– Master classes in the performing arts, with remote students having access to the world’s great performers and teachers

– Access to global databases and information of all kinds will help level the playing field between rich and poor nations.

Summary of Impact of e-VLBI Program

• Opens new doors for national and international astronomical and geophysical research.

• Represents an excellent match between modern Information Technology and a real science need.

• Motivates the development of a new shared-network protocols that will benefit other similar applications.

• Drives an innovative IT research application and fosters a strong international science collaboration.

The End

Thank you!

e-vlbi: connecting the world’s radio telescopes with high-speed networks alan r. whitney mit...

Documents

vlbi data system

gbps evlbi system

diskthe global vlbi

station vlbi correlator

data collection

astronomycombines data

traditional data recording

highspeed global networks