aa correlator system concept description · wp2‐040.040.010‐td‐001 revision : 1...

Name Designation Affiliation Date Signature

Additional Authors

Submitted by:

A. Faulkner UCAM 2011‐03‐26

Approved by:

W. Turner Signal Processing Domain Specialist

SPDO 2011‐03‐26

AA CORRELATOR SYSTEM CONCEPT DESCRIPTION

Document number .................................................................. WP2‐040.040.010‐TD‐001

Revision ........................................................................................................................... 1

Author .................................................................................................... Andrew Faulkner

Date ................................................................................................................. 2011‐03‐29

Status ............................................................................................... Approved for release

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

DOCUMENT HISTORY

Revision Date Of Issue Engineering Change

Number

Comments

A ‐ ‐ First draft release for internal review

1 29th March 2011 First Issue

DOCUMENT SOFTWARE

Package Version Filename

Wordprocessor MsWord Word 2003 AA_ Correlator_CoDR_2011

Block diagrams

Other

ORGANISATION DETAILS

Name SKA Program Development Office

Physical/Postal

Address

Jodrell Bank Centre for Astrophysics

Alan Turing Building

The University of Manchester

Oxford Road

Manchester, UK

M13 9PL

Fax. +44 (0)161 275 4049

Website www.skatelescope.org

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

TABLE OF CONTENTS

1 INTRODUCTION ............................................................................................. 7

1.1 Purpose of the document ....................................................................................................... 7

2 REFERENCES ................................................................................................ 7

3 OVERVIEW .................................................................................................. 8

4 AA CORRELATOR IMPLEMENTATION ................................................................ 10

4.1 Central Processing design ..................................................................................................... 10

4.1.1 Correlator/beamformer – C/B ...................................................................................... 10

4.1.2 Switch for visibility routing and dish corner turning ..................................................... 12

5 POWER REQUIREMENTS ................................................................................ 13

6 AA‐CORRELATOR COST ................................................................................. 14

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

LIST OF FIGURES

Figure 1: SKA Phase 2 overall system diagram from SKADS White Paper .............................................. 9

Figure 2: Central processing architectures ........................................................................................... 10

Figure 3: A possible physical implementation of AA sub‐correlator shelf ............................................ 12

LIST OF TABLES

Table 1: SKADS‐SKA implementation ...................................................................................................... 8

Table 2: Power requirement per correlator board ............................................................................... 13

Table 3: Accumulated correlator power ............................................................................................... 14

Table 4: Estimated costs of an AA Correlator for SKA Phase 2 ............................................................. 14

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

LIST OF ABBREVIATIONS

AA .................................. Aperture Array

ADC ............................... Analogue-to-Digital Converter

AI ................................... Arithmetic Intensity

ASIC .............................. Application-Specific Integrated Circuit

CoDR ............................. Conceptual Design Review

CMAC ............................ Complex Multiplication and ACcumulation

CPU ............................... Central Processing Unit

CUDATM ......................... Compute Unified Device Architecture (NVIDIA 2009)

DFT ............................... Discrete Fourier Transform

DiFX .............................. Distributed FX correlator (Deller et al. 2007)

DRAM ............................ Dynamic Random Access Memory

DRM .............................. Design Reference Mission

DSP ............................... Digital Signal Processing

eMERLIN ....................... extended Multi-Element Radio-Linked Interferometer Network

eVLA .............................. Extended Very Large Array

FFT ................................ Fast Fourier Transform

FLOPS ........................... Floating Point Operations per second

FPGA ............................. Field Programmable Gate Array

FoV ................................ Field of View

GMRT ............................ Giant Meter-wave Radio Telescope

GPGPU ......................... General-Purpose Graphics Processing Unit

GPU ............................... Graphics Processing Unit

HPC ............................... High-Performance Computing

IF ................................... Intermediate Frequency

LO .................................. Local Oscillator

LOFAR .......................... LOw-Frequency ARray

MPI ................................ Message Passing Interface (MPI Forum 2009)

MWA .............................. Murchison Widefield Array

NRE ............................... Non-Recurring Engineering

PEP ............................... Project Execution Plan

PrepSKA........................ Preparatory Phase for the SKA

RF .................................. Radio Frequency

SEMP ............................ Systems Engineering Management Plan

SRS ............................... Systems Requirement Specification

SIMD ............................. Single Instruction Multiple Data

SKA ............................... Square Kilometre Array

SKADS .......................... SKA Design Studies

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

SPDO ............................ SKA Program Development Office

TBD ............................... To be decided

VLBA ............................. Very Long Baseline Array

WIDAR .......................... Widefield Interferometric Digital ARchitecture (correlator implementation)

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

1 Introduction

This paper is a description of a potential correlator structure for an aperture array system. It is not a

full design of a correlator; it makes assumptions about the availability processing devices in the

2018+ timeframe. The design discussed here is structured to support the full AA system for SKA

phase 2; it can readily be scaled back for the correlator in phase 1 if necessary.

The architecture of an AA correlator is very different from the SKA dish correlator: there are only

expected 250 arrays, but each has a very large data output rate of ~16Tb/s. This makes a highly

modular and integrated design essential. Many of the issues are related to routing of the large

numbers of fibre feeds from each of the arrays.

1.1 Purpose of the document

The purpose of this document is to provide a concept description as part of a larger document set in

support of the SKA Signal Processing concept design review (CoDR). It provides the concept of how

an aperture array correlator could be constructed.

System parameters for SKA Phase 2 have been drawn from Schilizzi et al. (2007) [1] and the SKA

Design Reference Mission (DRM) [2]. These have been incorporated into the SKADS White Paper,

Faulkner et al. [3], much of the design of the correlator has been described in this final report.

2 References

[1] Schilizzi, R.T., et al. (2007), Preliminary Specifications for the Square Kilometre Array, Memo

100

[2] SKA Science Working Group, (2010), The Square Kilometre Array Design Reference Mission:

SKA‐mid and SKA‐lo, report, v1.0, February 2010. [3] Faulkner A.J., et al. (2010), Aperture Arrays for the SKA: the SKADS White Paper, Memo 122

[4] “Square Kilometre Array Design Studies, SKADS”, www.skads‐eu.org

[5] Garrett, M.A., et al. (2010), A Concept Design for SKA Phase 1 (SKA1), Memo 125

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

3 Overview

This paper considers the aperture array, AA, correlation requirements for SKA Phase 2. The putative

implementation considered is based on the work in the SKA Design Studies, SKADS, [4]. This heavily

uses AAs for all frequencies up to 1.4GHz and a substantial dish array from 1.2GHz up to 10 GHz as

described in [3]. The outline SKADS‐SKA implementation is shown in Table 1. The system

implementation is shown in Figure 1.

Table 1: SKADS‐SKA implementation

Freq. Range Collector Sensitivity Number / size Distribution

70 MHz to 450 MHz

Aperture array (AA-low)

4,000 m2/K at 100 MHz

250 arrays, Diameter 180 m

66% within core 5 km diameter, rest along 5 spiral arms out to 180 km radius

400 MHz to 1.4 GHz

Aperture array (AA-mid)

10,000 m2/K at 800 MHz

250 arrays, Diameter 56 m

1.2 GHz to 10 GHz

Dishes with wide-band single pixel feed (SD-WBSPF)

5,000 m2/K at 1.4 GHz

1,200 dishes Diameter 15 m

50% within core 5 km diameter, 25% between the core and 180 km, 25% between 180 km and 3,000 km radius.

The requirements of correlation are described elsewhere in this CoDR, the essential difference of the

AA correlator becomes clear from examination of an SKA Phase 2 system diagram shown in Figure 1.

The correlators receive all the high data rate signal data from all the collectors, cross correlate each

baseline of a collector type, integrate and pass the resulting uv data via a large data switch to be

buffered at the start of the post processing systems. The data rates from the SKA collectors means

that it is impractical to put the correlator anywhere other than local to the core. Assuming that the

data rates are reduced sufficiently through the correlator, then it is likely that the uv data will be

transported to the nearest major city for processing.

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

Figure 1: SKA Phase 2 overall system diagram from SKADS White Paper

..

Sparse AA

Dense AA

..

Central Processing Facility ‐ CPF

User interfacevia Internet

...

To 250 AA Stations

DSP

...DSP

To 1200 Dishes

...15m Dishes

16 Tb/s

10 Gb/s

Data

Time

Control

70-450 MHzWide FoV

0.4-1.4 GHzWide FoV

1.2-10 GHzWB-Single Pixel feeds

Tile &Station

Processing

OpticalData links

... AA slice

... AA slice

... AA slice

...Dish

& AA+D

ish Correlatio

n

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

Data sw

itch ......

Data Archive

Science

Processo

rs

Tb/s Gb/s Gb/s

...

...Time

Standard

Imaging P

rocessors

Control Processors & User interface

Pb/s

Correlator UV Processors Image formation Archive

Aperture Array Station

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

4 AA Correlator implementation

The design of the AA correlator and beamformer cannot at this stage be definitive, since there are

too many unknowns before 2018 on, however, the general principles can be discussed and the

following is a worked example of a potential implementation.

4.1 Central Processing design

Note: The figures here consider the implementation of 2400 dishes. This may be revised for 1200.

The implementation of the central processor follows from the discussion in the SKADS White Paper

[3] discussing imaging and non‐imaging requirements. The central processing configurations by

function are show side‐by‐side in Figure 2.

Imaging Non-imaging

Figure 2: Central processing architectures

The similarity of requirements for the different stages of processing for both systems imply that

ensuring that a configuration or alternative programming that can support imaging and non‐imaging

observations will be highly cost effective. Using a unified central processing system avoids extensive

data switching for raw collector data; provides opportunities for commensurate observations of

imaging and non‐imaging science experiments; enables innovative new observing techniques to be

used; avoids duplicating development effort and is an easier maintenance environment.

4.1.1 Correlator/beamformer – C/B

It is assumed that the correlators are FX type and that the frequency division has been done to the

appropriate level prior to the beam data arriving at the C/B. This can readily be achieved by the AAs

in the station processors.

Correlation and beamforming both perform a very large quantity of simple operations, which can be

efficiently implemented using integer arithmetic; there is also a great deal of communications I/O.

The processing is dominated by complex MACs.

Whilst being the same process, there are major structural differences between the correlation of the

AA signals and the dish signals. For the AA there are relatively few, ~250, stations each forming very

many beams, >1000, over 1600 10Gb/s data links or equivalent. Whereas, the dishes are providing

one beam from up to 2400 collectors over 8 10Gb/s links. This inherently implies that there are two

...

AA slice

...

AA slice

...

AA slice

...D

ish &

AA

+D

ish Co

rrelation

......

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

Data sw

itch ......

AA

Station

sD

ishes

Data Archive

ScienceProcessors

Correlator UV Processors

Image formation

Science analysis, user interface & archive

Beams Visibilities UV data Images

250 x 16Tb/s

2400 x 80Gb/s

Tb/s Gb/s Gb/sPb/s

...

...

Imaging P

rocessors

...

AA beamformer

...

AA beamformer

...

AA beamformer

...D

ish & A

A+

Dish

Beam

form

er

......ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

ProcessorBuffer

Data sw

itch ......

AA

Stations

Dishes

Data Archive

ScienceProcessors

BeamformingDe-dispersion, retiming, spectral separation and

profiling

Pulsar Identification

Science analysis, user interface & archive

CollectorBeams SKA-Beams

Candidates & Spectra Profiles

250 x 16Tb/s max

2400 x 80Gb/s

Tb/s Gb/s Gb/sPb/s

...

...

An

alysis Processing

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

C/Bs which confers a number of advantages: all the collectors can be used concurrently; there is no

need for large amounts of switching of raw beam data and mass production can be used efficiently

for the AA C/B.

The number of complex operations, Nop‐cor, for a correlator operating on two polarisations, providing

cross‐polarisation and auto‐correlation products, for N antenna, is given by:

4 ∆

It is convenient to re‐state this in terms of incoming digital sample rate, data rate G1 using Nbit

samples, which covers any mixture of number of beams and bandwidth as with an AA:

1

The AA correlator lends itself to a highly modular implementation split by beams or matched

incoming data channels.

It is assumed here that the communications are using 10Gb/s (8Gb/s actual data rate with 8:10

encoding) channels, the most cost effective data rate currently and fits conveniently with the

available data rates for optical and copper communications. A similar scheme can be built using

100Gb/s channels and splitting the data into channels using a simple switch. Eight of these 10Gb/s

channels are colour multiplexed onto a fibre, delivering 80Gb/s per fibre. The AA correlator can then

be designed as 200 “shelves” each with eight identical sub‐correlators.

Each sub‐correlator can be characterized as having:

250 stations, N

4‐bits per sample, Nbit

Incoming data rate per collector of 8Gb/s, G1

Hence, the processing rate required per sub‐correlator is: 63x1012 complex MACs or ~250TMACs.

An outline physical design is shown in Figure 3. This is aimed to make construction and interconnect

straightforward. It is constructed as a double‐sided shelf in a rack, where a multiplexed fibre from

each of 250 AA stations is connected using sixteen input cards, each with 16 inputs. It is assumed

here that each fibre carries 8 off 10Gb/s channels. The demultiplexed optical signals from the

demultiplexer are ordered to match each of the sub‐processor cards to minimise signal distribution.

A 10Gb/s channel from each station is presented to a sub‐correlator card via a mid‐plane, which

routes the incoming signals appropriately. An alternative connector is the “cross‐connector” which

links boards orientated at 90° to each other. There are therefore eight sub‐correlators per shelf. The

visibilities are output via local fibre from the sub‐correlator boards to a data switch for routing to the

appropriate UV processor.

The maximum correlator output data rate for a continuum survey using 180km baselines and a 100

deg2 FoV is approximately 14Tb/s [3]. This is a very considerable reduction from the incoming raw

data rate of 4Pb/s.

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

Each sub‐correlator needs to be capable of processing 250TMACs. In this layout, this could be

provided by an array of fifteen of the 20TMAC multi‐core processing chips, as used in the AA‐

processing.

The full AA‐correlator would use 200 of these shelves; at three to a rack the entire system is 70

racks. Improvements in processing performance or communications throughout this sub‐system

could reduce the power or number of boards involved. There are a total of 1600 sub‐correlator

boards in the system.

Figure 3: A possible physical implementation of AA sub‐correlator shelf

The beamforming requirement for the AAs covering 3 deg2 is a total of 1.25x1015 complex MACS.

This is a beamforming load of <1T complex MAC per sub‐correlator; this is much less than the 63T

complex MACs required for correlation. The total output data rate from the beamformer of ~3Tb/s

for 10,000 beams, 100μS sampling of 8‐bits, 2048 channels, and 2 polarisations, each beam is about

330Mb/s. This is rather less than the maximum imaging data rate of 14Tb/s, discussed above, hence

the hardware is able to support both beamforming and correlation in the same system.

4.1.2 Switch for visibility routing and dish corner turning

The use of commercial data switch technology is a vital element in this design. The total data rates

for the “visibility” switch and number of ports needed could be required is very large. However,

major commercial data centres have similar requirements; hence switches are available with very

high performance. In many respects the architecture used here makes the system more

straightforward to build: it can be defined at design time what the range of switching that may be

required is and the system can then be broken down into a tiered arrangement.

The switch is show very simplistically in Figure 2 and in principle can connect any correlator output

to any processing blade. This is undoubtedly unnecessary, although the details of the precise

flexibility are the subject of substantial development. For example, the correlators could be treated

Optical beam inputs16 cards each: 16inputs of 8x10Gb/s

8 AA slices 8 cards each: 256 inputs of 10Gb/s

........

256 Fibres.. 1 per station.. 16 per card.. 8 x 10Gb/s ea

Optical 1:8 demux Optical Rx Midplane

AA Slices: Correlator/Beamformer

Data Rx & “corner turner”Beams

Visibilities/

Timeseries

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

completely independently, so that the AA correlator operates only with a defined set of uv processor

blades, similarly for the dish correlator. Indeed, the dish correlator splits naturally into a number of

frequency bands, which could be independent. These arrangements would be unnecessarily

restrictive and careful design should enable data to be piped flexibly between the different sub‐

systems.

For this paper it is worth considering the performances available from current equipment. A major

determinant is the “bisection bandwidth”, which is the total data rate that can be moved from one

half of the switch to the other. If the bisection bandwidth exceeds the total input data rate then it is

likely to be completely non‐blocking and hence completely flexible. Switches exceeding 50Tb/s1 are

available now, using Infiniband protocols. This compares very favourably with the maximum data

rates from the dish correlator of a few hundred Tb/s across the whole band with 2400 dishes and

long baselines. The data rate from the AA correlator is significantly lower at ~14Tb/s maximum [3].

The data rate from the ports can be split down to 10Gb/s to match the processing speed of the uv

processors.

The power requirements are relatively modest with only 8kW being required to fully switch ~50Tb/s.

This is currently available; the performance will improve over time with increasing data switching

requirements in the commercial sector.

5 Power requirements

The total processor cards needed for the correlator is:

Aperture Array. 200 sub‐correlator shelves at eight processing boards per shelf. A total of

1600 correlator cards.

The power for the correlator is dominated by the processing chips and their associated

communications. In [3], the expectation of a processor of 20TMACs with 128 digital inputs and

outputs is discussed in section 7. The nature of the processing device is not specified here, other

than it needs to be efficient at the task of correlation and beamforming and be programmable; this

is really a discussion on the physical implementation of the system. By using this device the X part of

this correlator can have its power requirements estimated.

Table 2: Power requirement per correlator board

# Function Powerea. (W)

Total(W)

Remarks

15 Processing devices 25 375 Processing only. Implemented using the same device as AA DSP

1000 10Gb/s I/O channel 0.1 100 256 overall I/O channels and on board chip interconnect

Electrical power used 475

Electrical power supplied @ 85% efficiency

560

Cooling power at 25% 140

Board Total 700 Total power used per correlator board

1 The Mellanox IS5600. http://www.mellanox.com

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

The total correlator power required is shown in Table 3. The estimate, not including switch, is ~1.2MW, which will improve over time with improved devices.

Table 3: Accumulated correlator power

Type # Powerea. (W)

Total(kW)

Remarks

AA 1600 Processing boards 700 1,120 Board power and interconnect

400,000 Fibre optical receivers 0.1 40 1600, 10Gb/s channels for each 250 AA stations

Total AA ‘X’ correlator power 1,160

6 AA‐correlator cost

The cost of the AA correlator at this stage has to be an estimate with relatively wide errors.

However, it is clear that if the design is simplified to this level the minimum cost envelope can be

achieved. The programmability of the processing device and its reuse elsewhere in the system avoids

the substantial NRE requirements of developing a specialist ASIC. There are, however, significant

development costs that are not itemised here; this is the capital cost of implementation.

A conservative estimate of the costs is shown in Table 1, which shows a total cost, including fibre

receivers, of €45M. This is for the full SKA Phase 2 using 250 AA stations and 16Tb/s per station

output data rate.

Table 4: Estimated costs of an AA Correlator for SKA Phase 2

Item Quantity Cost each Total Comments

Shelf Cost

Correlator boards 8 €8,000 €64,000 Includes cooling, assembly and test

Input boards 16 €9,000 €144,000 Includes the fibre receivers

Backplane 1 €2,000 €2,000

Mechanics 1 €1,000 €1,000

Sub-total shelf €211,000

Rack Cost

Shelf 3 €211,000 €633,000

Rack mechanics 1 €1,000 €1,000

Power supplies 1 €10,000 €10,000

Sub-total Rack €643,000

AA-Correlator

Rack 70 €643,000 €45,010,000

WP2‐040.040.010‐TD‐001

Revision : 1

2011‐03‐29 of 15

The cost of a rack at ~€600,000 is similar to other high complex processors, giving a simple check. It is

interesting to note that the processing itself does not dominate the cost of the correlator, in this

implementation with a large number of fibre receivers it is this component which make the biggest cost

impact; which may well change with higher speed links.

aa correlator system concept description · wp2‐040.040.010‐td‐001 revision : 1...

Documents