An FX software correlator for VLBI

Adam DellerSwinburne University

Australia Telescope National Facility (ATNF)

Outline

• Function of a correlator

• Reasons for software correlation

• Code and platforms

• Progress and current performance

• Immediate and near future plans

• Facilitation of real-time gigabit eVLBI

Correlator functionality

• Averaging the product of signals at two telescopes gives visibility

Correlator functionality

• Averaging the product of signals at two telescopes gives visibility

• To get frequency information, average additional multiplications by lagged signal, then FFT (or reverse for FX)

• What does it all boil down to? Gigabit interferometry, 1024 channels with

6 telescopes, FFT cost ~ 200 Gflops (FX)

Why software correlate?

• Flexibility Handle disparate file formats and

bands High spectral resolution + high b/w Real time fringe checking

• Rapid development• For Australian LBA

8 x bandwidth, using disk-based recorders instead of S2 tape system

Correlator code structure

• Developed in C++

• Disk read/write threaded to hide latency

• Currently using IPP for vector mathematics

fxcorr

Telescope Baseline

Mode InputStream

Platform options

• Initial development to run on Swinburne cluster: 200+ 3GHz Pentium 4 machines

• Cray XD-1 chassis: 12 AMD Opteron processors, 6 Xilinx FPGAs

Hybrid Architecture

• PC flexibility with FPGA power

• FPGA used for unpacking/fringe rotating and FFT

• Possibly incorporate cross-multiply• FPGA code compiled into library,

available as C subroutine call• Cray contributing via development

of FPGA subroutine library

Milestones to date

• ‘Basic’ and ‘complex’ FX algorithms implemented and verified

• Fortran driver for CALC ported to C for delay modeling

• Used in search for first trans-Tasman fringes to 6m antenna in NZ

• Range of support/analysis packages created

Correlator output

Current performance

• LBA: Complex correlation, 6 telescopes, 2x 16MHz bands (128 Mb/s), 4 products/band, one node of cluster: 512 channels: 21x real time 2048 channels: 25x real time 32786 channels: 32x real time

• 15 telescopes, 128 Mb/s, 4 products 512 channels: 88x real time

Immediate applications

• Improved LBA sensitivity: parallax of double pulsar J0737-3039

• High time and frequency resolution + high bandwidth: Wide field imaging

• Masers, SNRs, AGNs…NGC253 image credit:Emil Lenc (Swinburne)

Real-time gigabit eVLBI?

• All ATNF telescopes of LBA capable of 1 Gb/s, others 512 Mb/s

• Technical/cabling issues to overcome: disk-based recorders

• Software correlator ready now• Simulated Gb/s data, 512 channels

<160x real time Current system has disk transfer

limitation

In the near future…

• Fibre links overcome disk limitations?• Investigate alternate architecture:

‘Cell’ processor array at VPAC• Code improvements:

Sophisticated pulsar gating Polyphase filterbank alternative to FFT Automated configuration from schedule,

GUI frontend, output FITS compatible…

Integration into the LBA

• Available for use from now on a limited basis

• Final release planned before March 2006 observing session

• Will incorporate automatic configuration, GUI frontend etc

• Feedback on desired features encouraged

Applications outside the LBA

• Potentially useful for niche applications anywhere

• Applicable to connected-element as well as VLBI• “Piggy-backing” on existing interferometers - only need facility to get data to disk, and some processors

Existing correlator limited + disk space

available + processors = SOFTWARE CORRELATE

Conclusions

• Fast, flexible new FX software correlator for generic clusters developed

• Hybrid architecture will be explored• Will provide significant increase in

LBA sensitivity until next upgrade• Real-time gigabit eVLBI using the

software correlator is feasible on Swinburne cluster

• Final release planned in early 2006

Questions??

XD-1 Specifications

• 12x 2.2GHz Opteron processors - 106 Gflops total

• 1-8 GB RAM per processor, 12.8GB/sec• 4-8 GB/s interconnect (between nodes)• 6 Xilinx Virtex 4 FPGAs• 4 PCI-X slots for ethernet/fibre channel• Max 1.5 TB local storage