FPGA Cluster Computing in the ETA Radio Telescope

C. Patterson, B. Martin, S. EllingsoDept. of Electrical & Computer Engine

Virginia TechBlacksburg VA 24061

{cdp, bm92, ellingson}@vt.edu

Abstract

Array-based, direct-sampling radio telescopes havecomputational and communication requirements unsuitedto conventional computer and cluster architectures. Syn-chronization must be strictly maintained across a largenumber of parallel data streams, from AID conversion,through operations such as beamforming, to dataset record-ing. FPGAs supporting multi-gigabit serial I/O are ide-ally suited to this application. We describe a recently-constructed radio telescope called ETA having all-sky ob-serving capability for detecting low frequency pulses fromtransient events such as gamma ray bursts and explodingprimordial black holes. Signalsfrom 24 dipole antennas areprocessed by a tiered arrangement of28 commercial FPGAboards and 4 PCs with FPGA-based data acquisition cards,connected with custom I/O adapter boards supporting In-finiBand and LVDS physical links. ETA is designed forunattended operation, allowing configuration and record-ing to be controlled remotely.

1 Introduction

The Eight-meter-wavelength Transient Array (ETA) isa new radio telescope designed to observe a variety ofpostulated but as-yet undetected astrophysical phenomenawhich are suspected to produce single pulses detectableat relatively long wavelengths. Potential sources for thesepulses include the self-annihilation of primordial blackholes (PBHs) and prompt emission associated with gammaray bursts (GRBs). PBHs are postulated to arise from den-sity fluctuations in the early universe rather than the grav-itational collapse of a star, and may be of any size. Ifblack holes evaporate as suggested by specific combina-tions of general relativity and quantum mechanics, thosewith masses below about 1014 g should be approachinga state of runaway evaporation, terminating in a massiveburst of radiation [18]. A number of models have been pro-

J. Simonetti, S. CutchinDept. of PhysicsVirginia Tech

Blacksburg VA 24061{jhs, scutchin} @vt.edu

posed to explain short-duration GRBs, including the mergerof closely-separated compact objects such as neutron starsand/or black holes. The formation of a single black holewould release an immense amount of energy over a fewseconds [14]. Prompt radio emission from GRBs would bevery useful in pinning down the physics of the bursts, thenature of the progenitor object, and possibly the mediumin which it occurs. Recently, a dispersed pulse of duration< 5 ms was detected during the analysis of archival pulsarsurvey data [5]. The brightness (30-Jy) and singularity ofthe burst suggest the source was not a rotating neutron star.Although pulses can be quite strong by astronomical stan-dards, they are difficult to detect due to their transient andunpredictable nature, and the narrow field of view providedby existing telescopes.

ETA, in contrast, is designed to provide roughly uni-form (albeit very modest) sensitivity over most of the visiblesky, all the time. The complete array consists of 12 dual-polarized dipole-like elements (i.e., 24 radio frequency in-puts) at the Pisgah Astronomical Research Institute (PARI),located in a rural mountainous region of Western North Car-olina (350 11.98' N, 820 52.28' W). Each dipole is indi-vidually instrumented and digitized. The digital signals arecombined to form fixed "patrol beams" which cover the sky,and the output of each beam is searched for the unique time-frequency signature expected from short pulses which havebeen dispersed by the ionized interstellar medium. ETA hasthe computational resources and communication bandwidthto implement up to eight beamforming datapaths across allantennas, permitting it to operate as eight independent tele-scopes simultaneously recording data from different regionsof the sky. A patrol beam can be quickly reoriented by up-dating its beamforming coefficients. Additional informa-tion about ETA and its science objectives are available atthe project web site [10].

This paper is organized as follows. Section 2 describesthe ETA architecture, with particular emphasis on the digitalback-end. Section 3 discusses system validation issues suchas determining bit error rates and ensuring channel synchro-

1-4244-1472-5/07/$25.00 cO 2007 IEEE FT20FT 200n -7

Figure 1. ETA's ten-element core array. Oneantenna stand is in the center, and the re-maining nine stands form a circle of radius8 m.

nization. Current status and conclusions are summarized inSections 4 and 5.

2 System Design

ETA is designed to operate in the range 29-47 MHz,which is a response to a number of factors. First, some as-trophysical theories suggest the possibility of strong emis-sion by the sources of interest in the HF and lower VHFbands, limited at the low end by the increasing opacity ofthe ionosphere to wavelengths longer than about 20 m (15MHz). Useable spectrum is further limited by the pres-ence of strong interfering man-made signals below about 30MHz (e.g., international shortwave broadcasting and Cit-izens Band (CB) radio) and above about 50 MHz (e.g.,broadcast television), which make it difficult to observe pro-ductively outside this range. At these frequencies, however,the ubiquitous Galactic synchrotron emission is extraordi-narily strong and can be the dominant source of noise in theobservation [17].

The ETA system receives RF input from 24 dipole-likeantennas mounted on 12 stands. Figure 1 shows the ten-stand core array; not shown are a pair of outrigger stands lo-cated several hundred feet away to the north and east. Eachstand supports two orthogonally-polarized, dipole-like ele-ments shown in Figure 2. Figure 3 is a snapshot of typ-ical spectrum seen by the antenna. Buried coaxial cableconnects antenna outputs to analog receivers that amplifyand filter the signals. The digital signal processing and datarecording are the focus of this paper, and are shown in Fig-ure 4.

The architecture consists of three tiers: receiver nodes,FPGA cluster nodes, and data acquisition nodes. At eachlevel, data must be synchronized and processed in paral-lel. The system receives antenna signals through the re-ceiver nodes consisting of twelve Altera Stratix DSP de-velopment boards, referred to as S25s [7]. Each S25 boardperforms A/D conversion and digital filtering for two dipoleinputs, and provides a source-synchronous stream of data toa cluster node. The cluster nodes are Xilinx ML310 FPGAboards, referred to as ML31 Os [12], and form twelve outer

2.1 Receiver Nodes

The main functions of the receiver nodes are to digi-tize, downconvert and channelize (filter to a narrower band-width) the analog antenna input. Since designing customhardware to perform this function would be costly and timeconsuming, this functionality was implemented on commer-cially available Altera S25 boards. This board was chosenbecause it contains the necessary hardware elements for thedesign, and was familiar to the design team. Each S25 boardcontains two 12-bit 125 MSPS A/Ds, allowing all 24 analoginput signals to be digitized with 12 boards.

To maintain synchronization across all receiver nodes,one board generates the clock and reset signals for the otherS25 boards. A counter, synchronous across all S25 boards,is encoded with each time sample. This counter is used inthe cluster nodes to align data streams from different S25s.Two antenna streams are combined into a single stream andtransmitted to a cluster outer node at 30 MB/s. The com-bined stream consists of a 60 MHz clock, four bits of sourcesynchronous data, and a counter bit. These LVDS signalsare transmitted over Precision Interconnect's medical grade"Blue Ribbon" brand coaxial cable [3] with MICTOR con-nectors to help maximize noise immunity and minimize ra-dio frequency interference (RFI) generation.

Receiv/er Nodes

.....................................................

-H

........

AntenraIrputs

RecEDT N

"ilium = uuoconnections

................. -- W-

Serial Network

LVDS

Connecti s ML31O cluster Control PC

cordinglodes

w

GBEtherret

Figure 4. ETA's digital front-end and back-

1 iOal v - 300 kHz

Figure 5. Inside ETA's equipment hut. The

16-node FPGA cluster occupies the middlerack. To the left is the PC cluster and tapedrives, and to the right are the receiver nodesmounted inside a cabinet.

ner nodes collect data from two outer nodes, combine andformat the input streams for storage on the PC cluster. Inthis mode each of the four PCs receives and store data at 60MB/s, giving the system an aggregate recording rate of 240MB/s. Data collections usually occur for roughly one hour,generating an 800 GB dataset. After offloading the data to400 GB LTO-3 tapes [11], another acquisition can begin.

The basic beamforming mode uses all 12 outer nodes,combining multiple streams of data to form patrol beams forimproved sensitivity. Although the sky could be tessellatedwith up to 12 independent beams, only 8 beams are imple-mented due to hardware resource limits, and poor sensitiv-ity (because of antenna pattern roll-off) near the horizon.Each outer node again receives data from two antennas at a

rate of 30 MB/s, or 360 MB/s aggregate. The outer nodesmultiply each antenna input stream with the correspondingbeamforming coefficients and sum the results to produceeight single-pol beams. Four of the beams are sent to one

Figure 6. ML310 board and adapters. TheML310's form factor allows mounting insidePC ATX motherboard cases. This innernode has six InfiniBand cables connected tothe adapter board on the bottom left of theML310. The adapter board to the right pro-vides a MICTOR connector to a 3 m "BlueRibbon" brand coaxial cable coming from anS25 receiver node, and an Amphenol 80-pinright angle connector to a 7 ft black cablegoing to an EDT PCI CDa data acquisitioncard in a PC node. Between the two adapterboards is a copper bar conducting heat fromthe FPGA to the bottom of the case.

inner node and the other beams are sent to an adjacent innernode as illustrated in Figure 8. At this stage each Auroraconnection is transmitting 120 MB/s for an aggregate band-width of 2.88 GB/s to the inner nodes. Each connection hasan available bandwidth of 240 MB/s (5.76 GB/s aggregate),increasable to 360 MB/s (8.64 GB/s aggregate) by select-ing a faster RocketIO reference clock although the bit errorrate (BER) will increase. Each inner node combines a four-beam input from six outer nodes. The six input streams aresynchronized, with corresponding samples summed, shiftedand rounded (to avoid a DC bias) before transmission to thePCs. This reduction allows each PC to record four single-pol beams at 60 MB/s, or an aggregate 240 MB/s. For eachpolarization, one PC contains data for beams 1 to 4 and theother PC contains data for beams 5 to 8.

Since propagation speed through the ionized interstellarmedium varies with the frequency of the wave, pulse dis-persion or smearing results. In order to dedisperse the sig-nal, the observing band is split into narrow channels and thedetected signal from each channel is delayed by a differ-ent amount before summation to obtain the total power sig-

Aurora Parallel LVDS tion, correspConnections Connections transmission

mode. A uscbins can besystem to rec

6 viewing time.... <- ,;As shown

PI

S25node are to s

S25ipUtS 1 = combine theiInputs METhe streams

7/ W start of a vetors are sumr

6 A 32-bit valAcquisition ther one or t

Inner Nodes IS mode. TwoPC. This allcther four antc

RCC Outer Node Hardware Implementation of sixteen aneight dual-pc

Output Select to be obsermand Encode The outef

S25_ ...... R=iW. _the logic elciInpmut 1Decode *X Datts Aurora Interface of the multip

Tr§t 1366M 28 (S%) ofA.L 124 Output to nodes use Si

Inner ML31Os necessitatin|1366_ dg cluding the

while beamfGMAC/s.

Control PC 2.3 PC

Figure 8. ML310 outer node implementation. Four coStEEnterprise LiSC430 is eq

nal. Even on a fast workstation, dedisnersion recuires over Drocessor, 1

RCC Computation for a Single Polarization

Antennainputs 1 bforone *11 11111

polarization FFT

16-bit complex, 1024 point input;36-bit complex output; 1024 cycletransform time, 1024 word burstevery 2048 cycles.

ol ,, to

urNo es -1 0.

11hed116fc

up to 68 MB/s, and the tape automatically speeds up or

slows down to match the incoming data rate. LTO-4 tape

drives are now available, which can record 800 GB (uncom-

pressed) at up to 120 MB/s.

A Fedora Linux-based control PC is accessible through

the Internet and allows external control and observation of

the FPGA and PC clusters. A Gigabit Ethernet switch con-

nects the control PC and the SC430s. The control PC ac-

cesses each ML3 10 through two Lava Link Octopus-550 se-

rial PCI cards [15]. Each card has eight RS-232 ports giving

a total of 16 serial connections. Scripts run on the control

PC start and terminate acquisitions, download new config-

urations and coefficients to the M3 10 nodes, and record

rnpff

input width: 840 bits; output width: 28bits; 4 cycle latency, 1024 word burstevery 2048 cycles; 4 level tree.

Adde Outputs toTree ~Acquisition

32-bit x 36-bit complex input; 70-bitcomplex output; 3 cycle latency; usescipnt~ 4 18x18 multiplier blocks.

channels.

system status information. Test modes are also available tocheck system integrity.

3 System Validation

To enable BER testing, synthetic data can be sourced ei-ther from the S25 or ML3 10 nodes. The data (typicallycounter or numerically-controlled oscillator output) are ver-ified before and after every communication link in the sys-tem, and any errors are tallied on each ML3 10. The serialconnection between each ML3 10 and the control PC allowsfor internal state to be read or written. C programs convertuser-generated text files to a bit string defining the oper-ating mode and data such as beamforming coefficients, orfrom a bit string to text files describing the internal state.Readback also returns the status of the Aurora links, indi-cates whether buffers have ever overflowed, and confirmsthe beamforming coefficients were written correctly. Errorcounts may be queried after a test to facilitate on-site orremote error rate testing and diagnosis, or after a data ac-

quisition to check that the integrity of the data collected. Inaddition, synthetic data tests may be invoked at the end of adata collection script. Through the detailed information re-ceived from the ML3 IOs, problems can be quickly isolatedto specific connections, cables or boards. These scan pathsallowed each half of the FPGA cluster to be connected andtested in less than a day.

ETA's systolic architecture makes data retransmissiondifficult, and error-correcting codes would be a significantoverhead given the number of communication links. How-ever, digital system errors are unacceptable even though

RFI is a much larger concern. All data errors originatingin the ETA system have occurred in the 2.5 Gbps Aurora 28 -j IIIIllchannels. Synchronization errors (manifesting as detectablebuffer overflows) have never been observed. Careful de- 26sign and signal integrity analysis of the adapter boards, and 24 -

transmitter pre-emphasis and differential voltage swing ad- , 22-lllljustments, have resulted in observed BER less than the In- 'finiBand maximum of 10 12 In a 200 GB dataset, the num- m 20ber of data bit errors is generally 0, with a maximum of 2 . 18 \Xobserved. The main source of errors are the connections -16between a cable and an adapter board. Outer nodes trans-mit only two channels and make each channel available on 14three of the adapter board connectors to facilitate bypassing 12a connection generating errors. Inner nodes use all six inputchannels, and errors are usually remedied by reseating ca- 10 0 5 10 15 20ble connections. Enabling the RocketIO transceiver's CRC LST [h]feature and checking if any CRC errors have occurreWV.rti Result of experiment to confirm diurnal variation in a 5 MHz bandwidth centered at 45 MHz. Blue line:

ing an observation should address inter-board data integriIitdicted poweR Il11i. Diu(n a1riNea15 H bancIe array dipole. Redisatter: Measured74oer from a dipole on the eas igge iboutt50 m dtsnt. E4ch "blob" is actuallyconcerns. 100 contiguou WJofnteredAt 4*5 lat eaI OtW UIIMnLS between dipoles is

attributable to f6di't6o6wer due to galactic noise, while

4 Current Status

ETA development began in August 2005, with a demon-stration of direct sampling from the first four dipoles threemonths later. One commissioning test was confirmation ofthe diurnal variation shown in Figure 11. The first 200 GBraw data acquisition through S25 and ML3 10 nodes to a PCoccurred in April 2006. A lightning strike in July 2006 dam-aged preamplifiers, but most have since been repaired. Over30 hours (6 TB) of raw data has been collected by the endof February 2007. FFT beamforming mode was added inJuly 2007. Two independent dedispersion software work-benches have been developed, and we see a role for un-dergraduate students in applying these tools to the datasetarchives. An analysis of how frequently pulses are detectedand their dispersion measures should provide valuable in-sights to the distribution of the progenitor objects, some ofwhich may be a component of dark matter [2].

Mitigation of RFI is an important consideration for anyinstrument operating at low frequencies. Although dedis-persion tends to suppress narrowband RFI, and FFT binsmay be selectively blanked to reject noise in certain fre-quency ranges, an additional approach to RFI rejection isanticoincidence. A portable second instrument (ETA2) isbeing assembled at a site more than 300 km from PARI.ETA2 still sees the same section of the sky as ETA butis not affected by ETA's local RFI. Although each stationrecords time accurately via GPS, there is no beamformingacross stations since data collection is not synchronized dur-ing the observation. During dataset post-processing, pulsesdetected simultaneously at both ETA and ETA2 would ruleout local RFI sources.

the black and gray scatters are measuredpower from a core array dipole and the eastoutrigger respectively.

5 Conclusions

FPGAs with integrated multi-gigabit serial transceiversare ideal platforms for the signal processing requirementsof radio telescope arrays. Synchronous streaming computa-tions may be implemented over many boards without re-quiring system-wide clock distribution. As in ETA, thephysical communication topology can be tailored to thedataflow requirements, simplifying the interfaces and de-velopment effort. In contrast, software processors and theircommunication protocols are difficult to use in this environ-ment due to variable latencies.

Data recording is the bottleneck for the ETA back-end.One of the biggest design challenges was finding a low-costPC interface and configuration to record data continuouslyat 60 MB/s for one hour. The first attempt used a GigabitEthernet link between each ML310 and PC, but PCI driveroverheads overwhelm the ML310 FPGA's 300 MHz Pow-erPC processor. Positive developments for data streamingand recording performance are multi-core processors allow-ing better overlap of computation and I/O, and high capacitysolid state disks that avoid seek latencies.

Our goal of obtaining results within the first year neces-sitated the use of commercial-off-the-shelf (COTS) FPGAboards. This approach distinguishes ETA from many otherradio telescope arrays using custom boards. The design ofcomplex, high speed boards can squander FPGA develop-ment time and cost advantages. Although a single board

integrating the receiver and beamforming functions is de-sirable, the effort required to design a custom PCB wouldbe excessive even if no respins were required. The low-cost evaluation boards meet the ETA system specificationsremarkably well, and all interfacing is accomplished withstandard cables and simple adapter boards containing onlyconnectors and traces.An additional benefit of COTS is straightforward up-

grades to new FPGA families. ETA2 can use a newer gen-eration of FPGA evaluation boards: the ML410 contains aVirtex-4 FX60 while retaining the PC ATX form factor andpersonality module interfaces [13], and the Stratix-II DSPDevelopment Kit contains a 2S60 device and preserves theMICTOR interface [8]. Evaluation boards further extendthe low cost and rapid development virtues of FPGAs, withthe new boards costing roughly the same as their predeces-sors despite having significantly improved logic capacityand performance. The university price for these boards usu-ally reflects a donation or discount of the FPGA and config-uration components.ETA invites comparisons with BEE2, a general-purpose,

scalable, FPGA-based system for large-scale emulation andDSP [4]. Both suit stream-based computations, although aBEE2 board is significantly more powerful with 5 2VP70FPGAs, 20 GB of DDR2 DRAM, and 18 10-Gbps serialinterfaces allowing connections to other BEE2 boards, to anInfiniBand or 10 Gbps Ethernet packet-switched network,or to analog interfaces. On the other hand, ETA's use ofevaluation boards is cost effective for the size of the system,and also allows quicker migration to new FPGA families.A larger scale version of ETA might justify more powerfulcompute platforms such as BEE2, or servers with FPGAmodules plugged into some of the processor slots [6]. Itmay also be feasible to have FPGAs stream data directly toa parallel set of SATA disks. Regardless of whether FPGAsand/or multicore processors are used, built-in interfaces forhigh speed serial protocols are as essential as computationalpower.

References

[1] http://www.xilinx.com/aurora/aurora technology.htm.[2] D. Blais, C. Kiefer, and D. Polarski. Can primordial black

holes be a significant part of dark matter? Physics Letters B,535:11, 2002.

[3] http://precisionint.com.[4] C. Chang, J. Wawrzynek, and R. W. Brodersen. BEE2: a

high-end reconfigurable computing system. IEEE Des. Test.Comput., 22(2):114-125, Mar. 2005.

[5] D.R. Lorimer, M. Bailes, M.A. McLaughlin, D.J. Narkevic,and F. Crawford. A bright millisecond radio burst of extra-galactic origin. To appear in Science, 2007. Preprint avail-able at http://www.arxiv.org/abs/0709.430.

[6] http://drccomputer.com/.[7] http://altera.com/products/devkits/altera/kit-

dsp stratix.html.[8] http://altera.com/products/devkits/altera/kit-dsp-2S60.html.[9] http://www.edt.com/pcicda.html.

[10] ETA project website: http://www.ece.vt.edu/swe/eta.[11] http://www.lto-technology.com.[12] http://www.xilinx.com/products/boards/ml3 10/current/.[13] http://www.xilinx.com/ml410-p/.[14] R. Narayan, B. Paczynski, and T. Piran. Gamma-ray bursts

as the death throes of massive binary stars. AstrophysicalJournal, 395:L83-L86, Aug. 1992.

[15] http://www.lavalink.com/index.php?id=232.[16] http://www.xilinx.com/publications/books/serialio/serialio-

book.pdf.[17] S.W. Ellingson, J.H. Simonetti, and C.D. Patterson. De-

sign and evaluation of an active antenna for a 29-47 MHzradio telescope array. IEEE Trans. Antennas Propagat.,55(3):826-831, Mar. 2007.

[18] S.W. Hawking. Black hole explosions? Nature, 248:30-31,1974.

6 Acknowledgments

This work was supported by the National Science Foun-dation under Grant No. AST-0504677, a SCHEV ETFgrant, and by the Pisgah Astronomical Research Institute.Graduate student Vivek Venugopal performed data stream-ing tests on the Aurora links and data acquisition PCs, andMichael Kavic is investigating theoretical connections be-tween PBH explosions and large extra dimensions. PARIpersonnel constructed the antenna masts and undergroundcoaxial cable system. We are grateful for early access tothe ML3 10 board and the assistance of Vince Eck, RickLeBlanc, Punit Kalra and Saeid Mousavi in Xilinx's Sys-tems Engineering Group.