5-2 research and developments for e-vlbi utilizing global

131KOYAMA Yasuhiro et al.

1 Introduction

Very Long Baseline Interferometry(VLBI) observation technology applies theprinciples of interferometry to celestial radiosignals received by multiple radio telescopes,enabling high-resolution imaging of celestialradio sources received by multiple radio tele-scopes and high-precision determination of thetime delay between received signals. Figure 1shows a photograph of a radio telescope withan aperture of 34 meters, located at theKashima Space Research Center. Large radiotelescopes of similar dimensions locatedaround the world are linked to carry outinternational VLBI observations for a variety

of purposes. Although each radio telescopefeatures relatively coarse resolution, limitingits ability to obtain a brightness distributionfor a given celestial radio source, the forma-tion of an array of multiple telescopes spacedfar apart can provide resolution correspondingto a telescope with an aperture comparable tothe distance between two telescopes in thearray. This allows for imaging of celestialbodies at resolutions surpassing those of theHubble Space Telescope or large ground-based optical telescopes.

Furthermore, it is possible to construct aterrestrial reference frame and a celestial ref-erence frame and to make high-precision mea-surements of earth orientation parameters by

5-2 Research and Developments for e-VLBIUtilizing Global High Speed NetworkConnections

KOYAMA Yasuhiro, KONDO Tetsuro, HIRABARU Masaki, KIMURA Moritaka, and TAKEUCHI Hiroshi

Rapid developments of the Information and Communications Technology have been soremarkable and as the results it is becoming possible to transfer enormous amount of data overlong distances which could not be considered before. It is not only causing many and large inno-vations in social life styles and economic activities, but also various and variety of applicationsare expected to be realized in the field of scientific research and developments. The main themeof this paper, e-VLBI, can be said as a typical example of such applications which requires highspeed network. To realize e-VLBI, it is necessary to transfer enormous amount of data betweenmany sites around the world, and it requires effective usage of the network and calculationresources by using multi-cast and distributed computing to realize high speed computing uponhigh volume of digital data arising from multiple places in the global scale. Therefore, e-VLBI isconsidered as a unique and quite suitable theme for network technology research and develop-ments. Solving problems and fulfilling unique requirements which e-VLBI presents, we areexpecting that it will accelerate further progress for the network technology.

KeywordsVery Long Baseline Interferometry, e-VLBI, Space geodesy, Radio astronomy, High speednetwork application

132 Journal of the National Institute of Information and Communications Technology Vol.52 Nos.3/4 2005

analyzing VLBI observation data, since thetime delay between the received signals areexpressed as functions of the relative positionsof the reference points of each radio telescope,the position of the celestial radio source, andthe earth orientation parameters whichdescribe the direction of the earth’s rotationalaxis and variations in rotation velocity. Theseparameters include precession and nutation,which represent the direction of the earth’srotational axis in the celestial reference frame;polar motion, which represents the inclinationof the earth’s rotational axis in the terrestrialreference frame, and universal time (UT1),defined by the earth’s rotation (Fig. 2). Theseparameters constantly display irregular varia-tions resulting from the gravity of bodies inthe solar system acting on the earth, themotions of the atmosphere and ocean, and thefluid motions in the earth’s interior. In contrastto the terrestrial reference frame constructedby combining the data obtained by space geo-detic techniques such as the VLBI, GPS(Global Positioning System), and SLR (Satel-lite Laser Ranging), the celestial referenceframe is constructed by combining the posi-tions of celestial radio sources measured byVLBI. Therefore, VLBI is the only choice fordirect high-precision measurements of allearth orientation parameters describing therotation between the terrestrial and celestialreference frames. However, in past interna-

tional VLBI observations, observation datahad to be recorded on magnetic tapes andtransported to facilities (correlators) for corre-lation processing of the observation datastored in magnetic-tape format. Thus, severaldays to weeks were required before the resultsof the processing and analysis of observationdata became available, and applicationsrequiring real-time earth orientation parame-ters had to use values predicted by extrapola-tion of past observation data, which led toinevitable errors in measurement.

The term e-VLBI has recently arisen todescribe a system that uses high-speed net-work connections to transmit observation dataelectronically to correlators, reducing the timerequired for data processing and analysis rela-tive to existing VLBI observations. With thenear-real-time processing of VLBI observationdata enabled by e-VLBI, high-precision esti-mation of the irregular variations of earth ori-entation parameters becomes theoreticallypossible, which in turn can lead to improvedprecision in tracking deep-space probes andmore precise determination of the satelliteorbital information required for high-precisionGPS measurements. These developments are

Fig.1 Radio telescope with an apertureof 34 meters in diameter, locatedat the Kashima Space ResearchCenter (Kashima City, Ibaraki Pre-fecture)

Fig.2 Earth orientation parameters


expected to make a significant overall contri-bution to the fields of space exploration andgeodesy. With VLBI, it is necessary toenhance the signal-to-noise ratio by employ-ing as broad a frequency band as possible inorder to detect extremely weak signals. Thus,the observed analog data must be convertedinto digital data and processed as rapidly aspossible. Increasing the data-sampling rate isexpected to prove particularly effective forVLBI observations in radio astronomy, sincethis will enable observations of weak radiobodies previously inaccessible to researchersdue to the limited data-recording rate of mag-netic data recorders used in conventionalVLBI observations. Recent R&D in networktechnology have resulted in an environment inwhich the network data-transmission rate farexceeds the data-recording rate of magnetictape recorders (typically 1,024 Mbps). There-fore, significant improvement in the sensitivi-ty of VLBI observation can be expected ifreal-time correlation processing of observationdata can be performed without the magnetictape-recording procedure; this in turn woulddrastically reduce the minimum value ofobservable radio-source intensity. Such a leapin sensitivity would enable the observation ofweak celestial radio sources (such as the ther-mal radio emissions of stars) otherwise impos-sible with existing VLBI technology; in short,these developments would together represent abreakthrough in radio astronomy.

Nevertheless, technical problems remainin high-speed transmission of massive vol-umes of data over the Internet—for example,the data collected by the international VLBIexperiment carried out in March 2004, whichtotaled approximately 18 TBytes for fiveobservation stations situated in different coun-tries throughout the world. Some of theseproblems form interesting themes for R&D innetwork technology, with topics includingmaximizing use of available data-transmissioncapacity in the presence of other types of traf-fic or effective congestion control in the eventof significant network transmission delay.Accordingly, numerous researchers are cur-

rently focusing concerted efforts on these andsimilar challenges.

2 History of e-VLBI

If the term e-VLBI is not limited strictly to aform using high-speed networks but instead isbroadened to include all systems for the elec-tronic transmission of data observed by theVLBI, then the first experiment in e-VLBI canbe dated back to 1976. In that experiment, VLBIobservations at a data rate of 20 Mbps were car-ried out at the Algonquin station in Canada andthe Greenbank station in the U.S. through link-age of these facilities via satellite［1］. In Japan,real-time VLBI observation was carried out in1977 by transmitting data at a rate of 4.096Mbps through a microwave link between theHiraiso Branch of the Radio Research Labora-tories (RRL) (presently the Hiraiso Solar Ter-restrial Research Center of the National Insti-tute of Information and CommunicationsTechnology (NICT)), and the Kashima Branchof the RRL (presently the Kashima SpaceResearch Center of the NICT)［2］. Althoughthese two experiments were revolutionary atthe time, the lack of available high-speedsatellite communication links and the rapidprogress in digital data recording technologieson magnetic tape that followed made it gener-ally more favorable to record observation dataon magnetic tape and to perform the correla-tion processing after transportation to a corre-lator facility. Since electronic data transmis-sion technologies were limited to those usingsatellite communications and public telephonelines offering limited data-transmission rates,electronic transmission was used only to vali-date observation data in preliminary process-ing; this approach continued until the recentwide availability of high-speed networks［3］.The Tokyo Metropolitan Wide Area CrustalDeformation Monitoring Project (also knownas the Keystone Project, or KSP) launched bythe Communications Research Laboratory (theCRL; presently the NICT) represented the firstefforts at full-fledged introduction of the e-VLBI. System development for the project


had as its goal high-precision, high-frequencymeasurements of the relative positions of fourVLBI stations, and was structured to overridethe existing framework of VLBI observationsystems at several points［4］. One such innova-tion involved the realization of real-timeVLBI observation data processing for thefour-station, six-baseline array via an ATM(Asynchronous Transfer Mode) network. Inthis system, the interface for recording data onmagnetic tapes was used as is, and the digitaldata output from the interface was convertedinto ATM cells and transmitted from eachobservation station to a data receptor installedat the Koganei Correlator. On the receivingside, the data was fed to the correlator via abuffer unit to absorb the difference in networktransmission delay among the stations, fol-lowed by real-time correlation processing.This system, developed under the auspices ofcollaboration between CRL and NTT Commu-nications, Inc., dramatically reduced the timerequired for data processing relative to exist-ing systems, which relied on magnetic taperecording. Further, this was the first system ofits kind to be completely automated—fromobservation to data processing and analysis—with the results of analysis of nearly non-stopVLBI observation automatically available tothe public on the Internet［5］. The developmentof this system proved that e-VLBI technologycould enable near-real-time data processingwith virtually no delay between observationand processing.

3 Development of the K5 obser-vation and processing system

In the KSP, a real-time VLBI observationprocessing system was developed using anATM network featuring a data-transmissionrate of 2.4 Gbps; continuous real-time VLBIobservation was realized in 1998 for the four-station, six-baseline array. However, thisobservation and processing system dedicated-ly used an ATM network. As such, the systemcould not be used to connect multiple VLBIstations throughout the world since it was not

feasible to construct an international dedicatedATM network just for e-VLBI. To increase theversatility of this system, development of theK5 observation and processing system beganin 2000, eventually resolving earlier problemsby enabling data transmission via IP (InternetProtocol), under Internet network conditionsinvolving the presence of other types of traf-fic. Figure 3 shows the data sampling boarddeveloped for the K5 system and Fig. 4 showsa photograph of the K5 observation and pro-cessing system now being put to initial use ina variety of observations.

The prototype K5 system for geodeticVLBI observations consists of four PC unitsoperating on FreeBSD or LINUX; each PCsystem is equipped with a data-samplingboard that can sample signals in four channels.The sampled data is recorded on the internalhard disk and may also be transmitted in realtime via IP network interface. The K5 systemis being developed to process data stored onthe hard disk as well as data received in realtime through the network, such that it shouldeventually be possible to form a systemaccommodating both near-real-time VLBIdata processing (in which observed data istransmitted before processing) and real-timeVLBI data processing. The data-samplingboard uses a highly stable reference frequencysignal and a time-pulse signal per second; bothof the signals are provided by a hydrogen

Fig.3 Data sampling board developedfor the K5 system


maser, recording a precise time stamp in theheader of the data file and maintaining accura-cy of phase information corresponding to thereceived signal. One of the features of thisboard is that the sampling rate (from 16 MHzto 20 kHz) and the quantization number(selectable from 1, 2, 4, and 8 bits) may beselected according to the target and purpose ofobservation. Since existing VLBI systemsattempted to improve the S/N ratio by maxi-mizing the data rate of the magnetic tape-recording process, the quantization numberwas limited to 1 bit or to a choice between 1and 2 bits, while the K5 system allows multi-bit quantization of up to 8 bits, enabling moreaccurate recording of waveforms. This featurerenders the K5 system suitable not only for e-VLBI applications but also in a range of fieldsof scientific instrumentation requiring sam-pling based on an accurate and stable timesystem.

In contrast to the real-time correlation-pro-cessing unit developed for the KSP, which

realized high-speed digital data processingthrough the use of the FPGA (Field Program-mable Gate Array), a software correlator hasbeen developed for the K5 system to performdistributed processing using a PC running on aversatile operating system. While a hardwarecorrelator lacks flexibility due to the extendedrequired development time, a software corre-lator may be modified flexibly to add newfunctions or to revise processing modes. Inaddition, development is also underway ofsoftware required to enable distributed pro-cessing using available computer resources(consisting of multiple CPUs) to the maxi-mum extent, in order to provide the neededprocessing capacity for data collected atnumerous VLBI stations in the context oflarge-scale VLBI experiments (Fig. 5).

4 UT1 estimation experiment

Using the K5 observation and processingsystem described in the previous section, testobservations began in 2003 for rapid estimationof UT1 using the Kashima 34-m station and theWestford 18-m station at the Haystack Obser-vatory of the Massachusetts Institute of Tech-nology［6］. Unlike other earth orientation para-meters, UT1 can be estimated from observa-tions as brief as one hour in duration. Further,as only one baseline is required for estimation,this method is extremely suitable for obtainingrapid analysis results via e-VLBI. Observationat the Westford station was performed using theMark-V observation system［7］developedmainly by the Haystack Observatory, while theKashima station employed the K5 system. Likethe K5 system, the Mark-V system is designedto store observation data on hard disks using acommodity PC and can also perform real-timedata transmission via the Internet. The input forthe system is output from a unit referred to asthe Mark-IV formatter, which allows observa-tion data to be divided and stored on multiplehard disks. In an experiment conducted on June30, 2004, multiple data files collected over thecourse of one hour of observation from 4:00 to5:00 a.m. (JST) at the Westford station were

Fig.4 K5 VLBI observation and processingsystem, currently in use in variousobservations


sent to the Kashima Space Research Center viaa high-speed research-dedicated Internet net-work. Figures 6 and 7 show the network con-nections used for file transmission. Inside theU.S., the Abilene network managed by Inter-net2 was used as the backbone, and the connec-tion from the Westford station to Abilene wasperformed through two networks, BOSSNETand GLOWNET. The Kashima station estab-lished connections to the Otemachi access pointusing the JGNⅡ, which began operations inApril 2004, and from there was connected tothe Abilene network in the U.S. via the JGNⅡ/TransPAC network.

Immediately after observations were com-plete, transmission of approximately 13.5GBytes of data was initiated, ending 1 hourand 15 minutes later. Thus, the average trans-mission rate was calculated at approximately24 Mbps. The transmitted data file was con-verted into K5-system data-file format andcorrelation processing was performed by dis-tributed correlation-processing software with-in the K5 system using a total of 21 CPUs.After correlation processing, database fileswere created and the data were analyzed usingthe geodetic VLBI data analysis software pro-grams SOLVE and CALC. As a result of these

operations, we succeeded in estimating theUT1 approximately 4 hours and 30 minutesafter observation, as opposed to the minimumof one-week period previously required,demonstrating that the use of a high-speedInternet can dramatically reduce the timerequired to obtain data-processing results.

5 Conclusions

The UT1 estimation experiment conductedby the Westford and Kashima stations resultedin the gradual reduction of the time requiredfor UT1 estimation; this was accomplished byimproving the processing rate of the softwareprogram for correlation processing, develop-ing software for efficient distributed process-ing, and automation of the proceduresinvolved. In the future we plan to continuewith similar test observations and to developrelated software, in addition to increasedefforts at reducing processing time—by inves-tigating effective methods to use the availablebandwidth of the network under various con-ditions of the shared network and furtherautomation of the processing system. We arealso pursuing methods of establishing a stan-dard data-transmission format, which will

Fig.5 Scheme for distributed processing (right) and a control screen for distributed processingunder development (left)


reduce the time required to convert the variousfile formats of the different observation sys-tems. Our goal is to enable estimation of allearth orientation parameters, not only UT1,instantaneously after observation through thedevelopment of a real-time correlator systemoffering increased processing speed, allowing

observation data to be transmitted in real time,and through correlation processing that doesnot require prior hard-disk storage. To accom-plish this goal, technologies must be estab-lished for high-speed transmission of wide-band e-VLBI observation data over extremelylong distances between points featuring large

Fig.6 Structure of the JGNⅡ network (from the website of JGNⅡ)

Fig.7 Network connections from the Abilene network to the Westford station


time delays. Thus, it is important to promoteresearch on the application of a research-dedi-cated high-speed Internet to studies on scien-tific instrumentation such as advanced conges-tion-control technologies, and dynamic controltechnologies to ensure efficient use of trans-mission bandwidth in these applications with-out inhibiting concurrent network traffic.Additional associated themes for future R&Dinclude the development of a system for dis-tributed and time-sharing processing of obser-vation data to maximize use of available com-puter resources as well as the development ofa distributed multicast processing systemcapable of sending identical sets of data tomultiple CPUs for distributed processing, inorder to perform large-scale e-VLBI experi-ments (where multiple stations perform obser-vations at the same time). Such R&D effortsare ultimately expected to produce new tech-nologies not only useful in e-VBLI develop-ment, but also in the application of research-dedicated high-speed networks to variousfields of scientific instrumentation and furtherareas of R&D.

Acknowledgements

The e-VLBI research and developmentprogram for near real-time UT1 estimation isbeing conducted jointly with a group ofresearchers from the Haystack Observatory ofthe Massachusetts Institute of Technology,including Dr. Alan Whitney, Dr. David Laps-ley (presently at BBN Technologies), Dr.Kevin Dudevoir, Dr. Jason SooHoo, and Dr.Chester Ruszczyk. In Japan, the NationalInstitute of Information and CommunicationsTechnology, the National Astronomical Obser-vatory of Japan, the Geographical SurveyInstitute, the Japan Aerospace ExplorationAgency, Gifu University, and Yamaguchi Uni-versity are collaborating partners in e-VLBIresearch and development. We are deeplygrateful for the support of the NTT Laborato-ries, KDDI R&D Laboratories, NTT Commu-nications, Inc. in the use of the research-dedi-cated high-speed network, and to the staffmembers managing the research networks ofthe JGNⅡ, TransPAC2, Internet2, and SuperSINET.

References01 Yen, J. L., K. I. Kellermann, B. Rayhrer, N. W. Broten, D. N. Fort, S. H. Knowles, W. B. Waltman, and G.

W. Swenson, Jr., “Real-Time, Very Long Baseline Interferometry Based on the Use of a Communications

Satellite”, Science, 198, pp. 289-291, 1977.

02 Kawano, N., “Real-time correlation system”, Rev. Research Laboratory, 42, pp.550-554, 1977.

03 Kondo, T., J. Amagai, H. Kiuchi, and M. Tokumaru, “Cross-correlation Processing in a Computer for

VLBI Fringe Tests”, J. Commun. Res. Lab., 38, pp. 503-512, 1991.

04 Yoshino, T., “Overview of the Key Stone Project”, Special Issue of the J. Comm. Res. Lab., 46, pp.3-6,

1999.

05 Koyama, Y., N. Kurihara, T. Kondo, M. Sekido, Y. Takahashi, H. Kiuchi, and K. Heki, “Automated geo-

detic very long baseline interferometry observation and data analysis system”, Earth Planets Space, 50,

pp. 709-722, 1998.

06 Koyama, Y., T. Kondo, H. Osaki, A. R. Whitney and K. A. Dudevoir, “Rapid Turn Around EOP Measure-

ments by VLBI Over the Internet”, Proc. the XXIIIrd. General Assembly of the International Union of Geo-

desy and Geophysics, Jul. 2003, Sapporo, Japan, ed. by E. Sanso, pp. 119-124, 2003.

07 Whitney, A. R., “The Mark 5 VLBI Data System and e-VLBI Development”, in 2002 Annual Report,

International VLBI Service for Geodesy and Astrometry, by N. R. Vandenberg and K. D. Baver (eds.),

NASA/TP-2003-211619, pp. 22-33, 2003.


KOYAMA Yasuhiro, Ph.D.

Group Leader, Radio Astronomy Appli-cations Group, Applied Research andStandards Department

Geodesy, Astronomy, Radio Science

HIRABARU Masaki, Dr. Eng.

Senior Researcher, Internet Architec-ture Group, Information and NetworkSystems Department

Internet Technology

KONDO Tetsuro, Dr. Sci.

Research Center Supervisor, KashimaSpace Research Center, Wireless Com-munications Department

Space Geodesy, Geophysics

KIMURA Moritaka

Radio Astronomy Applications Group,Applied Research and StandardsDepartment

Astronomy

TAKEUCHI Hiroshi, Dr. Sci.

Expert Researcher, Radio AstronomyApplications Group, Applied Researchand Standards Department

Radio Astronomy, Radio Engineering

5-2 research and developments for e-vlbi utilizing global

Documents