DiFX: Software Correlation at

Swinburne for the LBA

DiFX: Software Correlation at

Swinburne for the LBA Adam Deller

Swinburne University/CSIRO Australia Telescope National Facility

Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

9/6/2007 VSOP-2 meeting

Outline

• History• Correlator architecture• Capabilities• Computing environment and resources

• Usage inside and outside LBA• Future improvements

9/6/2007 VSOP-2 meeting

Correlator History• Commenced with my PhD thesis (2005)• Preliminary work (XF) undertaken in 2004 by Craig West

• First science late 2005• Conversion to distributed system 2006

• Verification with LBA and VLBA correlators, code released late 2006

http://astronomy.swin.edu.au/~adeller/software/difx/

9/6/2007 VSOP-2 meeting

Correlator Architecture

Written in C++, MPI for message passing, Intel IPP for vector arithmetic

9/6/2007 VSOP-2 meeting

Capabilities• Supports LBA, Mk5* (K5->LBA converter)• Pulsar gating, binning or matched filter

• Arbitrary time/frequency resolution• Presently, visibilities in RPFITS format

• eVLBI: data can be streamed from network socket directly into correlator

• Also used for real-time fringe checks*export to linux filesystem

9/6/2007 VSOP-2 meeting

The Swinburne supercomputer• Swinburne supercomputer presently consists of ~300 P4 processors

• Typically “reserve” ~30 processors for VLBI after observing run

• Next month: complete replacement of cluster with ~140 dual CPU, quad core nodes, >1100 cores total

9/6/2007 VSOP-2 meeting

Computing requirements

• DiFX requires little RAM (except extreme cases): processor speed, cache and SIMD support essential

• Performance scales linearly with aggregate bandwidth, near-linear with number of stations

• Realtime example: Max sensitivity LBA (6 x 1 Gbps) requires ~25 new nodes

9/6/2007 VSOP-2 meeting

Typical LBA usage

• Required to correlate ~5 days of 256 Mbps before next session (2 months)

• For each experiment, automatically select and correlate fringe-finders, solve clocks, followed by full correlation.

• Using ~30 (old) machines, high data rate experiments take 2x observe time

9/6/2007 VSOP-2 meeting

Non-LBA applications

• New geodetic array in Australia/NZ to use software correlator: modifications to control, output format

• VLBA is trialling software correlator to run in parallel with hardware correlator: experiments with specific requirements, testing high data rate

9/6/2007 VSOP-2 meeting

Future improvements

• Current version is production: future improvements will be incremental

• Direct read from MkV• Alternative output formats: AIPS++ MS, FITS-IDI, UVFITS

• Support in AIPS: tasks like CVEL currently confuse with LBA S2 (XF)

9/6/2007 VSOP-2 meeting

9/6/2007 VSOP-2 meeting

Interferometry & Correlators

RA

Dec

Interferometry: delay signals from two dishes to common reference and multiply

Common signal, independent noise: average improves S/N

Earth rotates, interferometer samples visibility: Fourier transform of sky brightness

9/6/2007 VSOP-2 meeting

Interferometry & Correlators

u ()

v ()

Finite bandwidth, not monochromatic, therefore visibility varies across band

To get v(), XF correlator accumulates lags, then FFTs

FX correlator FFTs segments of baseband data, and cross-multiply/accumulates

9/6/2007 VSOP-2 meeting

Hardware vs software• Software correlator: program running on a computer cluster/supercomputer

• Hardware correlator: ASIC boards, specialised data transport

• Software is unclocked, could be faster or slower than real-time

• No channel/integration time restrictions

• Floating pt vs int calculations

9/6/2007 VSOP-2 meeting

Why software?• Flexibility - you can do things that are impossible with a hardware correlator

• Rapid (and cheap) development• Add-ons MUCH easier in software • Compatibility• Expandability• For me: Allow disk-based correlation, and improved pulsar binning (sensitivity)

9/6/2007 VSOP-2 meeting

DiFX (Distributed FX)

Master Node

Core 1DataStream 1

DataStream 2

DataStream N

Core 2

Core M

… …

Timerange, destination

Baseband data

Visibilities

Source dataSource data

MPI is used for inter-process communications

9/6/2007 VSOP-2 meeting

DiFX (Distributed FX)

• Configured by text files (like jobscripts)

• Delay modelling - CALC 9 (separate)• Output: RPFITS (built on-the-fly)• Arbitrary time/frequency resolution• Arbitrary pulsar binning (incoherent dedispersion) - allows weighted bin sum

• Real time LBA @ 1 Gbps: 100-200 CPUs

9/6/2007 VSOP-2 meeting

DiFX (Distributed FX)

• Verification: recent successful comparisons with LBA and VLBA

9/6/2007 VSOP-2 meeting

LBA Science

• All require one or more of flexibility, high time/frequency resolution, or sensitivity• Wide field VLBI (Lenc & Tingay)• Masers (Horiuchi)• CDF radio counterparts (Norris et al.)

• RRATs (Kramer et al.)• eVLBI (Phillips et al.)• Pulsar parallax (me)

9/6/2007 VSOP-2 meeting

Worldwide science• So far, motivated by very high frequency resolution, or flexibility and minimum effort for new system• Pulsar scintillation (Brisken) requires extreme frequency resolution (244 Hz channels over 32 MHz bandwidth)

• Geodesy (MPIfR, Bonn, Germany)

• Geodesy, new Australian array (NCRIS)

t

9/6/2007 VSOP-2 meeting

Status: Correlator code• “Correlation” code completed and verified - now finishing GUI and packaging for public release

• VLBA, MPIfR and others keen to continue trialling code once released

• Should be online within a month• PASP paper submitted simultaneously

9/6/2007 VSOP-2 meeting

Status: Observing and reduction• Four sessions of eight allocated, two observed - applications for 2007/2008 year (four more sessions) soon

• Both observed sessions correlated and verified

• Starting to work on atmospheric/ ionospheric compensation and pipeline, crucial for efficient, accurate astrometry

9/6/2007 VSOP-2 meeting

Roadmap to submission

1 month DiFX available online

3 months First four sessions complete

6 months Reduction techniques tested, pipeline started

12 months Pipeline complete, draft begun

18 months Data collection, reduction and thesis draft complete

20 months Thesis submission

9/6/2007 VSOP-2 meeting

Conclusions• DiFX is a general purpose software correlator, publicly available soon

• Already used with success in Australia by the wider VLBI community, and generating interest internationally

• Pulsar parallax program is underway, with the bulk of observations to come

• Transitioning from code to science

9/6/2007 VSOP-2 meeting

9/6/2007 VSOP-2 meeting

Interferometry & Correlators• Major operations performed by FX software correlator:• Delay• Unpack quantized data to float• Fringe rotate• FFT• Correct fractional sample error• Cross multiply and accumulate A B

Datastream

Per station N

Per station N2