A A real-timereal-time software backend for the software backend for the

GMRT : towards hybrid backendsGMRT : towards hybrid backends

CASPER meeting Capetown 30th September 2009

Collaborators : Collaborators :

Jayanta Roy (NCRA)Jayanta Roy (NCRA)

Yashwant Gupta (NCRA)Yashwant Gupta (NCRA)

Ue-Li Pen (CITA)Ue-Li Pen (CITA)

Jeff Peterson (CMU)Jeff Peterson (CMU)

Jitendra Kodilkar (NCRA)Jitendra Kodilkar (NCRA)

Sanjay Kudale (NCRA)Sanjay Kudale (NCRA)

Existing GMRT backend Existing GMRT backend

30 antennas, 32 MHz, dual pol, 30 antennas, 32 MHz, dual pol, correlator + beam former correlator + beam former

Based on ASICs and DSPs ; Based on ASICs and DSPs ; designed in early 1990s ; running designed in early 1990s ; running for more than 10 yrs nowfor more than 10 yrs now

Limitations :Limitations : Lack of flexibility in time and Lack of flexibility in time and

frequency resolution frequency resolution Limited number of bits Limited number of bits Obsolescence; difficult to Obsolescence; difficult to

maintain maintain

New, desirable features :New, desirable features : Capability for recording raw Capability for recording raw

voltage signalsvoltage signals Facility to add new algorithms Facility to add new algorithms

e.g. RFI mitigation e.g. RFI mitigation Ability to form multiple beams Ability to form multiple beams

within the primary beam within the primary beam

Software backends Software backends

Software based back-ends :Software based back-ends : Few made to order hardware Few made to order hardware

components ; mostly off-the-shelf components ; mostly off-the-shelf items items

Easier to program ; more flexible Easier to program ; more flexible

GMRT Software Back-end (GSB) GMRT Software Back-end (GSB) :: 32 antennas32 antennas 32 MHz bandwidth, dual pol32 MHz bandwidth, dual pol Net input data rate : 2 Net input data rate : 2

Gsamples/sec Gsamples/sec FX correlator + beam former FX correlator + beam former Uses off-the-shelf ADC cards, CPUs Uses off-the-shelf ADC cards, CPUs

and switches to implement a and switches to implement a fully fully real-time back-endreal-time back-end

Raw voltage recording to disks, for Raw voltage recording to disks, for all antennas; off-line read back & all antennas; off-line read back & analysis analysis

Currently status : completed and Currently status : completed and ready to release as observatory ready to release as observatory facility (from 15facility (from 15thth October) October)

The GMRT software backend : The GMRT software backend : block diagram block diagram

The ADC Card The ADC Card

PCI-X compliant ADC card PCI-X compliant ADC card (Acquisition Logic)(Acquisition Logic)

4 channels, 8 bits each4 channels, 8 bits each

External trigger (from GPS pulse)External trigger (from GPS pulse)

External reference clock (from Rb)External reference clock (from Rb)

Several operating modes :Several operating modes : 33 / 66 MHz sampling, 4 33 / 66 MHz sampling, 4

channels (8 / 4 bits)channels (8 / 4 bits) 100 MHz, 2 channels (4 bits)100 MHz, 2 channels (4 bits) 200 MHz, 1 channel (2 bits)200 MHz, 1 channel (2 bits)

Data transfer from ADC card to Data transfer from ADC card to memory using DMA @ 145 MB/smemory using DMA @ 145 MB/s

Software flow : real-time mode Software flow : real-time mode

PAPABeamBeam

IAIABeamBeamADCADC

16 MHz16 MHzor or

32 MHz32 MHz

(with (with AGC)AGC)

Int Int Delay Delay CorrectCorrect

Filter Filter ++

DesampDesamp

FFTFFT++

FSTC FSTC & &

FringeFringeMACMAC

BeamBeamformerformer

visibilitiesvisibilities

64 analog64 analogInputsInputs

(32 ants, (32 ants, 2 pols)2 pols)

Software flow : real-time mode Software flow : real-time mode

Software flow : raw voltage mode Software flow : raw voltage mode

Performance Optimisation Performance Optimisation

Network transfer optimisation : jumbo packets Network transfer optimisation : jumbo packets

Computation optimisation : Computation optimisation : Intel IPP routines (for FFT)Intel IPP routines (for FFT) Vectorised operationsVectorised operations Cache optimisationCache optimisation Multi-threading load balancingMulti-threading load balancing

Performance specs :Performance specs : Better than 85% compute efficiencyBetter than 85% compute efficiency $190 / baseline ; 250 Mflops / W$190 / baseline ; 250 Mflops / W

Sample Results : Imaging Sample Results : Imaging

J1609+266 calibrator J1609+266 calibrator field at 1280 MHzfield at 1280 MHz

8.5 hrs synthesis image8.5 hrs synthesis image

Central source : 4.83 JyCentral source : 4.83 Jy

Noise level at HPBW : Noise level at HPBW : 34 microJy34 microJy

Dynamic range achieve : Dynamic range achieve : ~ 1.5 x10~ 1.5 x1055

Sample Results : Beamforming Sample Results : Beamforming

Phasing the array using a point Phasing the array using a point source calibratorsource calibrator

Single pulses from PSR B0329+54Single pulses from PSR B0329+54

Sample Results : Sample Results : Coherent Dedispersion Coherent Dedispersion

Real-time Data MonitoringReal-time Data Monitoring

New Capabilities : RFI mitigation New Capabilities : RFI mitigation

MAD filtering on raw time resolution data to eliminate bursty, MAD filtering on raw time resolution data to eliminate bursty, time domain RFI : works very nicelytime domain RFI : works very nicely

Future Plans : Multiple BeamformingFuture Plans : Multiple Beamforming

Offline processing mode of software back-end : beam-forming to produce Offline processing mode of software back-end : beam-forming to produce multiple multiple phased array beams covering the primary beamphased array beams covering the primary beam – for pulsar and transient science – for pulsar and transient science

Multiple Beamforming Multiple Beamforming

Can get full phased array sensitivity over the entire field of view Can get full phased array sensitivity over the entire field of view

Useful for searching for pulsars in extended targets (e.g. SNRs, GCs) Useful for searching for pulsars in extended targets (e.g. SNRs, GCs)

New Plans : Transient New Plans : Transient Search PipelineSearch Pipeline

Collaborative effort of NCRA and Swinburne University of Technology Running in piggy-back mode simultaneously with other observations Object of interest : Fast transients -- nanosecond to 100's of millisecond Event detection : based on the sensitivity of 8 antennae incoherent array

beamover 32 MHz Search in Dispersion measure space : Discriminate out fast

radio transient from RFI. Coincidence or anti-coincidence filter : Multiple sub-array

multiple beam coincidence filter reduces the false triggers due to direction-location dependent RFI.

Efficient RFI removal technique

New Plans : Transient Search New Plans : Transient Search Pipeline Pipeline

Multiple incoherent beam Multiple incoherent beam formation for transient pipeline formation for transient pipeline

3 incoherent array beams : PSR B0329+54

Detection of RFI by coincidence filter

RFI ?

Beam-1

Beam-2

Beam-3

Future Possibilities : Towards Future Possibilities : Towards Hybrid Designs Hybrid Designs

Upgraded GMRT : 32 stations, Upgraded GMRT : 32 stations, 400 MHz BW, dual pols 400 MHz BW, dual pols ~ 200 nodes (8 cores each)~ 200 nodes (8 cores each)

GPU accelaration feasibleGPU accelaration feasible

Mix and Match of FPGAs and Mix and Match of FPGAs and CPUs/GPUsCPUs/GPUs

Software backend for special Software backend for special applications :applications :

Spectral zoom modes Spectral zoom modes Raw voltage recording Raw voltage recording

modesmodes Debugging mode for new Debugging mode for new

arrays arrays

Thank YouThank You