Galfa Spectrometer

Jeff Mock, Dan WerthimerJeff Mock, Dan WerthimerStudents: Henry Chen, Pavel Monat, Aaron Parsons, Wonsop SimStudents: Henry Chen, Pavel Monat, Aaron Parsons, Wonsop Sim

Science Advisors: Carl Heiles, Snezana StanimirovicScience Advisors: Carl Heiles, Snezana Stanimirovic

http://seti.berkeley.edu/galfa

Image RejectionImage Rejection

• 45 dB at +- 10 MHz45 dB at +- 10 MHz

• 40 dB at +-20 MHz 40 dB at +-20 MHz

• 35 db at +-30 MHz35 db at +-30 MHz

Gain ControlGain Control

• 0 to 40 dB gain – each of 16 channels0 to 40 dB gain – each of 16 channels– (2 spare IF to baseband converters)(2 spare IF to baseband converters)

• Controlled by RS232 serial portControlled by RS232 serial port

• Overflow/Saturation detectionOverflow/Saturation detection

• Remote Operation (Displays as well)Remote Operation (Displays as well)

GALFA SpectrometerGALFA Spectrometer

Multi-Purpose FPGA-Multi-Purpose FPGA-Based Spectrometer Based Spectrometer (NSF, A. Parsons)(NSF, A. Parsons)

XilinxVirtex-II 6000

FPGA

XilinxVirtex-II

1000FPGA

256 MB DRAM

AreciboFeed

Array

200 MhzADC

Compact PCIBackplane

Software

200 MhzADC

200 MhzADC

200 MhzADC

I

I

Q

Q

Pol. 1

Pol. 2

{

{

200 Aux. I/O

compactPCIcompactPCI•PowerPower•CoolingCooling•Robust/ReliableRobust/Reliable•InexpensiveInexpensive

FPGA-Based FPGA-Based SpectrometerSpectrometer

SETI ApplicationsSETI Applications• ALFA Sky Survey (300 MHz x 7 beams)ALFA Sky Survey (300 MHz x 7 beams)

• Parkes Southern SERENDIPParkes Southern SERENDIP

• JPL/UCB/SI Survey (20 GHz Bandwidth)JPL/UCB/SI Survey (20 GHz Bandwidth)

• SETI Italia (Bologna)SETI Italia (Bologna)

• SETI@homeSETI@home

Astronomy ApplicationsAstronomy Applications• GALFA Spectrometer – Arecibo Multibeam Hydrogen SurveyGALFA Spectrometer – Arecibo Multibeam Hydrogen Survey

• Astronomy Signal Processor – ASP – Backer/Stairs (pulsars)Astronomy Signal Processor – ASP – Backer/Stairs (pulsars)

– GreenBank and Arecibo, soon Nancay GreenBank and Arecibo, soon Nancay

• ATA4 Correlator F Engine ATA4 Correlator F Engine

• Two Reionization Experiments Two Reionization Experiments

(Backer/Demorest/Peeks (UCB), Chippendale/Green/Ekers (ATNF)) (Backer/Demorest/Peeks (UCB), Chippendale/Green/Ekers (ATNF))

GALFA SpectrometerGALFA Spectrometer

sin

cos

LPF

LPF

100 MHz

-50 to +50 MHz

sin

cos

LPF

LPF

100 MHz

-50 to +50 MHz

QuadratureDownconverter

Board

IF Pol. 1

IF Pol. 2

Biplex256 pnt.

PFB

e^-it

e^-it

FIRLPF

FIRLPF

Digital LO

Decimateby 14

Decimateby 14

Biplex8192 pnt.

PFB

Detect

Detect

cPCIBackplan

eto

CPU

Multipurpose Spectrometer Board

PFB vs. FFTPFB vs. FFT

PFB vs. FFTPFB vs. FFT

PFB vs. FFTPFB vs. FFT

GALFA Lowpass FilterGALFA Lowpass Filter

GALFA Lowpass FilterGALFA Lowpass Filter

GALFA 2v6000 FPGA Resource Budget

Blockrams MultipliersBiplex 8k PFB

4x overlap FIR 16Coefficient tables 16delays 56

8k pt decimating FFT 3Twiddle tables 2delays 26

Biplex 256 PFB4x overlap FIR 16

Coefficient tables 4delays

256 pt FFT 18Twiddle tables 2delays

Dual Complex Mixer 8Dual lowpass filter 49Dual 256 point (x41) accumulator 4Dual 8k point (x32) accumulator 32Dual wideband power 4Dual narrowband power 4

Total 142 118

Chip resource 144 144

GALFA DiagnosticsGALFA Diagnostics

GALFA DiagnosticsGALFA Diagnostics

GALFA DiagnosticsGALFA Diagnostics

Integration TimeIntegration Time

• 988 mS fixed (99% duty cycle)988 mS fixed (99% duty cycle)

• Spectra read out every secondSpectra read out every second

• Integration starts 6 mS after 1 PPSIntegration starts 6 mS after 1 PPS

• Integration stops 6 mS before 1 PPSIntegration stops 6 mS before 1 PPS

• Timing accuracy +-10 nS wrt observatoryTiming accuracy +-10 nS wrt observatory

Data OutputData Output

• Galfa.yyyymmdd.projname.sequence_number.FITSGalfa.yyyymmdd.projname.sequence_number.FITS

• file written every 15 seconds (programmable)file written every 15 seconds (programmable)

• 0.5 Mbyte/sec (8 MByte files)0.5 Mbyte/sec (8 MByte files)

• Files written over NFS to file serverFiles written over NFS to file server

• 2*8K + 2*256 32 bit powers (42 bit 2*8K + 2*256 32 bit powers (42 bit accumulators)accumulators)

• Time stamped (NTP), plus telescope dataTime stamped (NTP), plus telescope data

Worries, WeaknessesWorries, Weaknesses

• ADC on same board with Digital Electronics ADC on same board with Digital Electronics

(spurs at +- 25.000000 MHz…)(spurs at +- 25.000000 MHz…)

• Dynamic Range Problems Dynamic Range Problems (Overflow/Saturation) (Overflow/Saturation)

(prog. gain/shifting, overflow detection) (prog. gain/shifting, overflow detection)

• Unknown LO phase shift (0 or 180 degrees) Unknown LO phase shift (0 or 180 degrees) between channels after power cyclebetween channels after power cycle– (LO is divided by two in downconverter)(LO is divided by two in downconverter)

Spectrometer Spectrometer DocumentationDocumentation• Schematics, Cabling DiagramsSchematics, Cabling Diagrams

• Software (Source code)Software (Source code)

• Users ManualUsers Manual

• DiagnosticsDiagnostics

• About 100 Pages of stuffAbout 100 Pages of stuff

• http://seti.berkeley.edu/galfahttp://seti.berkeley.edu/galfa

Short Term PlansShort Term Plans

• Tonight: Jeff Mock ArrivesTonight: Jeff Mock Arrives

• Monday – Thursday: Integration/TestMonday – Thursday: Integration/Test

• Friday – Sunday: ObserveFriday – Sunday: Observe

• This week: come and visit for demoThis week: come and visit for demo

feel, touch, and get to know your spectrometerfeel, touch, and get to know your spectrometer

Long Term PlansLong Term Plans

• Find ETFind ET

Future SETI Future SETI SpectrometersSpectrometers2015 4 THz 400 beams

10 GHz each2020 128 THz 12,800 beams

2025 4000 THz 40,000 beams

2030 128,000 THz 1M beams

SERENDIP V

PolyphaseFilter Bank

Serverw/ EDT card

GbESwitch

PC

Serverw/ EDT card

Serverw/ EDT card

Serverw/ EDT card

PCPCPC

PCPC

GbESwitch

PCPC

PCPC

PCPC

GbESwitch

PCPC

PCPC

PCPC

GbESwitch

PCPC

PCPC

PCPC

100 MHz

100 MHz

Pol. 1

Pol. 2

Astronomy Signal Processor – Don Backer

SERENDIP V SpectrometerSERENDIP V Spectrometer

Biplex Pipelined FFTBiplex Pipelined FFTExample: 8 pnt. FFTExample: 8 pnt. FFT

-4Z

-2Z

Switch

Delay by NCoefficientMultiply

add

sub

Unity

-2Z

-1Z

add

sub

2nd Roots of Unity

-1Z

-0Z

add

sub

4th Roots of Unity

-4Z

Pol. 1

Pol. 2

Time Domain

Frequency Domain

JFFT FFT controlsJFFT FFT controls• Any length transformAny length transform

• Input width and output widthInput width and output width

• twiddle coefficient widthtwiddle coefficient width

• 3 or 4 multiplier complex multiply3 or 4 multiplier complex multiply

• Specify downshift or programmable downshift Specify downshift or programmable downshift (optional rounding)(optional rounding)

• Decimate FFT optionDecimate FFT option

• Blockram / CLB memory threshold optionBlockram / CLB memory threshold option

• Overflow detectionOverflow detection

JFFT additional PFB controlsJFFT additional PFB controls• Filter overlapFilter overlap

• Width of filter coefficientsWidth of filter coefficients

• Window function for filter (hamming, hann, etc.) Window function for filter (hamming, hann, etc.)

• Import filter coefficients for custom filter performanceImport filter coefficients for custom filter performance

Moore’s Law in FPGA Moore’s Law in FPGA worldworldComputational Density Comparison

1000

10000

100000

1000000

10000000

10/28/1995

3/11/1997

7/24/1998

12/6/1999

4/19/2001

9/1/2002 1/14/2004

Release Date

(MO

PS

/MH

z)*l

am

da^

2 Processor Peak

FPGA 32-bit int MAC

FPGA maximum sustained performance

1

10

100

1000

10000

100000

12/1/1996

6/19/1997

1/5/1998

7/24/1998

2/9/1999

8/28/1999

3/15/2000

10/1/2000

4/19/2001

11/5/2001

5/24/2002

Release date

MO

PS

(3

2 b

it M

AC

)

100X More efficient100X More efficientthan micro-processors!than micro-processors!

3X improvement3X improvementper year!per year!

Next Generation BoardNext Generation BoardBEE2 (2004/5) – Chen BEE2 (2004/5) – Chen ChangChang• 5 Xilinx XC2VP70 5 Xilinx XC2VP70

• 40 GB RAM (8 GB each chip, 13Gbit/sec/chip)40 GB RAM (8 GB each chip, 13Gbit/sec/chip)

• 18 10Gbit/sec infiniband ports18 10Gbit/sec infiniband ports

• 50 boards per rack, Tbit/sec infiniband switch50 boards per rack, Tbit/sec infiniband switch

• Applications:Applications:– 1 GHz, 1 Gchannel spectrometer (single board)1 GHz, 1 Gchannel spectrometer (single board)

– Next Generation ATA backends (ata32 = 2 boards)Next Generation ATA backends (ata32 = 2 boards)

– SKA imagingSKA imaging

B2 Module: board layoutB2 Module: board layout• 5 compute

elements on a board

• Up to 400 billion CMAC/s performance

• communication bandwidth: – 240 Gbps on-

board 360 Gbps off-board

Global InterconnectsGlobal Interconnects

• Commercial Infiniband switch from Mellanox, Voltaire, etc.– Packet switched, non-

blocking– 24 ~ 144 ports (4X) per

chassis– 480Gbps ~ 2.88Tbps

full duplex constant cross section bandwidth

– <$400 per port

ComputeNode

#N

ComputeNode

#1

Infiniband Crossbar Switch

Ethernet Switch

19” 48RU Rack 19” 48RU Rack CabinCapacityCabinCapacity

• 40 compute nodes in 5 chassis (8U) per rack

• Up to 16 trillion CMac/s performance per rack

• 250 Watt AC/DC power supply to each blade

• 12.5 Kwatt total power

Unified Digital Processing Unified Digital Processing Architecture Architecture

• Distributed per antenna spectral channel processing• Multiple reconfigurable backend application processing• Commercial packet switched interconnect• Backend data pulling through remote DMA access

PolyphaseFilter

Banks

ChannelReorderBuffer

Infin

iban

d S

wtic

h

XMAC imaging

Spectrometer

Pulsar SearchingB

eam

form

ing

PolyphaseFilter

Banks

ChannelReorderBuffer

An #1

An #N

Infin

iban

d

Sw

tich

Happy Birthday CarlHappy Birthday Carl