Distributed FX software correlation

Distributed FX software correlation

Adam DellerSwinburne University/CSIRO Australia Telescope National

Facility

Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

Adam DellerSwinburne University/CSIRO Australia Telescope National

Facility

Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

28/09/200628/09/2006 Torun, PolandTorun, Poland

OutlineOutline

• Software vs hardware correlators• Why software (what it does best,

and the price you pay)• Current applications• The DiFX architecture• Performance and current status• Conclusions

• Software vs hardware correlators• Why software (what it does best,

and the price you pay)• Current applications• The DiFX architecture• Performance and current status• Conclusions

28/09/200628/09/2006 Torun, PolandTorun, Poland

Software vs HardwareSoftware vs Hardware

• “Software correlator”: program running on a supercomputer/cluster

• Software is unclocked, could be faster or slower than real-time

• No channel/integration time restrictions

• Floating pt vs int calculations

• “Software correlator”: program running on a supercomputer/cluster

• Software is unclocked, could be faster or slower than real-time

• No channel/integration time restrictions

• Floating pt vs int calculations

28/09/200628/09/2006 Torun, PolandTorun, Poland

Why softwareWhy software

• Flexibility - can do science that is impossible with hardware correlators

• Rapid (and cheap) development• Add-ons MUCH easier in software• Compatibility and expandability• In general, less approximations due

to the use of floating point

• Flexibility - can do science that is impossible with hardware correlators

• Rapid (and cheap) development• Add-ons MUCH easier in software• Compatibility and expandability• In general, less approximations due

to the use of floating point

28/09/200628/09/2006 Torun, PolandTorun, Poland

The price you payThe price you pay

• Because software & hardware not specifically tuned, less computation per $$ hardware

• Therefore not useful for EVLA/SKA scale correlators

• Running costs (electricity, aircon) may be higher

• Because software & hardware not specifically tuned, less computation per $$ hardware

• Therefore not useful for EVLA/SKA scale correlators

• Running costs (electricity, aircon) may be higher

28/09/200628/09/2006 Torun, PolandTorun, Poland

Current applications/results

Current applications/results

• New science the software correlator has enabled:– Disk-based LBA (greater bandwidth)– Adding global disk-based antennas to

LBA experiments– High time and frequency resolution

allowing wide-field imaging– VERY high frequency resolution for

pulsar scintillation studies

• New science the software correlator has enabled:– Disk-based LBA (greater bandwidth)– Adding global disk-based antennas to

LBA experiments– High time and frequency resolution

allowing wide-field imaging– VERY high frequency resolution for

pulsar scintillation studies

28/09/200628/09/2006 Torun, PolandTorun, Poland

The DiFX architectureThe DiFX architecture

Master NodeMaster Node

Core 1Core 1DataStream 1DataStream 1

DataStream 2DataStream 2

DataStream NDataStream N

Core 2Core 2

Core MCore M

…… ……

Timerange, destinationTimerange, destination

Baseband dataBaseband data

VisibilitiesVisibilities

Source dataSource data

MPI is used for inter-process communicationsMPI is used for inter-process communications

28/09/200628/09/2006 Torun, PolandTorun, Poland

Usage in the LBAUsage in the LBA

• PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas)

• Data is recorded directly to Linux-formatted storage (Apple Xraid)

• Disks shipped to Swinburne• Correlated at the Swinburne

supercomputer (~300 P4 machines)• Ftp fringe tests and 256 Mbps eVLBI

• PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas)

• Data is recorded directly to Linux-formatted storage (Apple Xraid)

• Disks shipped to Swinburne• Correlated at the Swinburne

supercomputer (~300 P4 machines)• Ftp fringe tests and 256 Mbps eVLBI

28/09/200628/09/2006 Torun, PolandTorun, Poland

PerformancePerformance

Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s)

Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s)

Cores required for real-time, 256 Mbps, 256 channels, 1 second integrations

0

50

100

150

200

250

300

0 5 10 15 20

Antennas

Nod

es

28/09/200628/09/2006 Torun, PolandTorun, Poland

Current statusCurrent status

• Verification via correlator comparison with VLBA completed

• Used in production capacity with LBA - complete switch as soon as sufficient disks available

• Further development (graphical frontend, FITS-IDI etc) ongoing

• Exploring potential for geodesy

• Verification via correlator comparison with VLBA completed

• Used in production capacity with LBA - complete switch as soon as sufficient disks available

• Further development (graphical frontend, FITS-IDI etc) ongoing

• Exploring potential for geodesy

28/09/200628/09/2006 Torun, PolandTorun, Poland

ConclusionsConclusions

• Software correlators can be rapidly developed and enable science that cannot be done elsewhere

• Perfect for niche/part-time experiments and feasibility tests

• Used successfully with the LBA• Early science results encouraging!

• Software correlators can be rapidly developed and enable science that cannot be done elsewhere

• Perfect for niche/part-time experiments and feasibility tests

• Used successfully with the LBA• Early science results encouraging!

28/09/200628/09/2006 Torun, PolandTorun, Poland