rf distribution over white rabbit. wr2rf - a replacement ......• each white rabbit switch will...

RF distribution over White Rabbit. 

  WR2RF - a replacement

board for the BOBR.

CERN, 3 December 2018 - John Gill

Thanks and acknowledgements

G. Hagmann, A. Spierer - BE-RFM. Gonzalez Berges, T. Levens - BE-BI

M. Rizzi, J. Serrano, T. Włostowski - BE-CO

Collaboration

Thanks and acknowledgements

Motivation

• Motivation - BST + BOBR

• Core technology - WR + DDS

• Design considerations - WR2RF card

• RFoWR systems - status + timeline

Motivation - What is the BOBR - overview within the TTC network

• BOBR (Beam OBseRvation) is a timing receiver.

• Forms part of the Beam Synchronous Timing (BST) system.

• Receives BST information via a TTC network.

BST Master

TTC network

BST Receivers (BOBR)

Motivation - What is the BST - overview

Motivation - What is the BOBR - main functions

• The BOBR card produces local copies of the bunch and turn clocks.

• Distribute triggers and produces pulses that are beam synchronous.

• Distributes LHC telegram information.

BST-Master BOBR TTC

LHC-

B2

LH

C-B1

S

PS

Motivation - What is the BOBR - what are its problems

• TTC network subject to diurnal and seasonal changes in signal delay.

• TTCrx PLL cannot operate with FSK that is utilised in SPS ion fills. PLLs unlock, SPS BST uses average of FSK RF signals.

• Ions experience large accelerations in the SPS and their time of flight is not compensated.

All of the above results in the bunch clock, and the bunch, no longer being synchronous with each other. This makes beam monitoring (position and loss) difficult.

Motivation - What are we trying to achieve.

• Replay an RF signal anywhere within the accelerator complex.

• Regenerate bunch and turn clocks.

• Automatic adjustment for signal propagation delays.

• Correct particle time of flight variations.

• Support complex RF schemes, for example, ion acceleration in the SPS.

• Achieve < 1 ps RMS jitter for RF signal reproduction.

Core technology





Core technology - White Rabbit - course corrections

Core technology - White Rabbit - fine corrections

Core technology - White Rabbit summary

• Utilises PTP protocol to achieve a coarse delay adjustment that will synchronise clocks between different receivers.

• The master monitors the differences between its clock and a slave's clock to produce a fine delay adjustment to the timing.

• Together, these adjustments result in a synchronised clock scheme (at 125 MHz) throughout an ethernet network. With < 1 ns accuracy and precision < 10 ps RMS jitter.

Core technology - Direct Digital Synthesis overview

• Utilises a Numerically Controlled Oscillator (NCO) to accumulate Frequency Tuning Words (FTW).

• The output from the NCO is used as an input address into a Look Up Table (LUT) that contains the signal (function) you want to produce.

• Finally a DDS uses a Digital to Analog Converter (DAC) to reproduce the RF signal you require.

Phase to amplitude conversion

LUT (CORDIC)

DAC

Starting Phase

FTWPhase

Accumulator +

Core technology - Direct Digital Synthesis - phase to amplitude

Time

Time

Addr

ess

of L

UT

FTW

PHASE ACCUMULATOR

PHASE TO AMPLITUDE CONVERSION

LUT

Valu

es

FTW

Output frequency of a DDS can be described as function of the phase accumulator size, frequency tuning word and the clock frequency of the DDS (e.g. WR frequency @ 125 MHz).

OUTPUT FREQUENCY

Core technology - Direct Digital Synthesis and White Rabbit

+

ACCUMULATOR

FTW

LUT

DAC

FPGA

WR Timing ensures and allows us to maintain all the Numerically Controlled Oscillators (NCO) with the same values everywhere.

NCOs without WR timing

NCOs with WR timing

Then it follows that the DACs are being driven identically to recreate the same RF signal.

Design considerations





Design considerations - top-level proposal

BST Master

TTC network

BST Receivers (BOBR)

FTW

FTW

WR2RF receivers

WR switch

network

FTW will be sourced directly from the

LLRF - SPS LS2 LLFR upgrade.

Design considerations - We plan to re-use existing designs.

• Targeting this RFoWR receiver card to be an hybrid version of the eTRM-14/15 cards.

• Plan to use kintex 7 FPGA as on the eTRM-14.

• These cards are already part of the planned SPS LLRF system, so reusing elements of their logic and design should lower risk.

Design considerations - RF clock recovery

WR cores receive packets and forward the FTWs to the NCO block.

FTWs are applied after a fixed latency has elapsed, packet delay unequal.

Every NCO "plays" the same FTWs and phase accumulator values.

RF Signal - 1

DAC DDS

FPGA - Kintex 7

Osc 10 MHz

Osc 1 GHz

SFP

WR Network

WR Core

Configurable NCOs + PACs

Clean-up PLL

Clock Multiplier

RF Signal - 2

DAC DDS

OCXO

Clk 125 MHz

FTWFTW FTW

External high quality oscillator and PLLs to provide White Rabbit in-phase clocks for the FPGA and DDS.

Multiplex RF distribution.

FTW

WR2RF Card

Design considerations - Expected phase noise.

• Prototype boards developed for the eRTM-15 project have shown:

• White Rabbit clocks are generated with 70 fs RMS jitter (from 100 Hz to 5 MHz).

• Additional noise from the DDS increases the jitter to 100 fs RMS jitter (from 100 Hz to 5 MHz).

• We anticipate that the noise for the whole system will be < 1 ps RMS jitter (100 Hz to 5 MHz).

Design considerations - bunch and turn clock generation

• Clocking FPGA logic with the recovered RF signal provides a method to create beam synchronous logic.

• Generating an in-phase bunch clock requires using the regenerated RF signal as a clock.

• Once we have a counter in the RF clock domain, we can coordinate triggers + pulses throughout a WR network.

• RFtime.

+

ACCUMULATOR

FTW

LUT

DAC14b

FPGA

+

RFtime Counter

WR Clock Domain

RF Clock Domain

Bunch Clock

÷ m

RFtime - As a Coordinate System for Synchrotrons Design considerations - RFtime - RF Synchronous Events and Triggers

Flat Top

Flat Bottom

Ramp

Time

Freq

uenc

y The input RF is not necessarily constant. The frequency has to change as particles are accelerated.

The RF at injection (flat bottom) into the LHC is ~87 Hz (protons) or ~550 Hz (ions) less the frequency reached at the flat top.

Bunch clock with varying frequency

Bunch clock with FSK

0 1 2 3 4 5

... 69 70 71 72 73 74 75 76

By expressing events and triggers in RFtime, we remove the need to express events in TAI time and avoid complicated calculations for each receiver. RFtime

RFtime = turn# x RF harmonic + bunch#

Design considerations - SPS RF is tricky

• SPS RF is dynamic. This makes it difficult to utilise simple NCO driving a DAC approach.

• The BE-RF have developed an approach using a multi-level NCO that compensates for:

• Slip-stacking.

• Frequency Shift Keying (FSK) for ion fills.

Design considerations - time of flight compensation

+

ACCUMULATOR

FTW

LUT

DAC+

LUT

Valu

esAd

dres

s of

LU

T

• Protons and particularly ions in the SPS accelerator are subject to changes in their Time of Flight (ToF).

• Correcting the RF (and bunch clock) for ToF should be straightforward as it can be expressed as a phase offset to be applied to the output of the NCO.

Design considerations - Motivation revisited

• Replay the RF signal anywhere within the complex. We can recreate RF anywhere in the WR network by transmitting FTW to control DACs.

• Regenerate bunch and turn clocks. Use the regenerated RF signal as an FPGA clock to acquire beam synchronous logic.

• Correct signal propagation delays. Constant monitoring and corrections with WR and PTP.

• Correct particle time of flight variations. Straightforward as an additional phase offset component before the PAC stage.

• Support complex RF schemes, for example, ion acceleration in the SPS.

Ability to insert custom NCO designs.

• Achieve < 1 ps RMS jitter for RF signal reproduction. High performance external clocking scheme aims to clean-up clock.

RFoWR systems





RF over White Rabbit is not new. On the master node the DDS is used as a VCXO in a PLL subsystem to lock-to and track the incoming RF signal. https://www.ohwr.org/projects/wr-d3s/wiki

A RFoWR system based on this technology is deployed in at the ESRF facility in Grenoble. REFURBISHMENT OF THE ESRF ACCELERATOR SYNCHRONISATION SYSTEM USING WHITE RABBIT http://inspirehep.net/record/1656131/files/tucpl01.pdf?version=1

RFoWR systems - status

Achieves 15 ps RMS jitter (10 Hz to 5 MHz) for RF signals between 10 - 70 MHz.

https://www.ohwr.org/projects/fmc-dac-600m-12b-1cha-dds/wiki

MASTER/ENCODER SLAVE/RECEIVER

RF in

PRODUCT AVAILABILITY MASTER SLAVEFREQUENCY

RANGEPHASE NOISE (RMS JITTER: 100 Hz - 5 MHz)

v2.0 FMC-DDS +

SPEC/SVECNOW

INPUT: 10 - 500 MHz

OUTPUT: 10 - 70 MHz~15 ps

v3.0 FMC-DDS +

SPEC/SVEC

Q3 2019

INPUT: 10 - 500 MHz

OUTPUT: 10 - 70 MHz< 15 ps

WR2RF

VME RECEIVER

Q3 2020

INPUT: N/A

OUTPUT: 10 - 400 MHz< 1 ps

RFoWR systems - Timeline

Q4 2018

Q1 2019

Q2 2019

Q3 2019

Q4 2019

Q1 2020

Q2 2020

Q3 2020

Release of DDS-DEMO, using v2.0 FMC-DAC600.

Complete schematics and layout of v3.0 FMC-DAC600.

Release a RFoWR starter kit. Based on the DDS-DEMO and v3.0 FMC-DAC600.

Prototype RFoWR high performance receivers available for test.

Production RFoWR HP receiver complete.

Thank you for listening!

Backup slides

Design considerations - RF Frequency flexibility

An external mixer approach provides maximum frequency flexibility, but comes with additional cost and requires the design of a bandpass filter.

WR NetworkFTW

FTW

FPGA - Kintex 7

Osc 10 MHz

Osc 1 GHz

RF

SFPWR Core

Configurable NCO + PAC

Clean-up PLL

Clock Multiplier

DAC DDS

OCXO

Clk 125 MHz

FTW

LO

IFBPF

Design considerations - WR data capacity

FIELD IPG Preamble dMAC sMAC Type IP hdr UDP hdr TCP hdr CRC

BYTES 12 8 6 6 2/4 20 8 20 4

Minimum UDP size ~ 66 bytes, Minimum TCP size ~ 78 bytes, Minimum Time ~ 0.5 us

Packetisation of data, coupled with orbit periods can be limiting with orbit periods of 1 to 2 us. Table below illustrates the number of payload bytes

delivered as a function of orbit period and number of packets.

Orbit Period

Number of packets sent per orbit

1 2 3 4 5 6 7 8

1 us 47 0 0 0 0 0 0 0

2 us 172 94 16 0 0 0 0 0

4 us 422 344 266 188 110 32 0 0

8 us 922 844 766 688 610 532 454 376

16 us 1922 1844 1766 1688 1610 1532 1454 1376

23 us 2797 2719 2641 2563 2485 2407 2329 2251

89 us 11047 10969 10891 10813 10735 10657 10579 10501

LHC telegram - Data requirements

Note update frequencies • TURN • CHANGE • 1 Hz - LHC Telegram

GPS Timestamp can be deleted WR provides this.

SPS Message

LHC Message

PLUS BI SPECIFIC DATA

Design considerations - FPGA design - reuse existing cores.

• IP cores for the White Rabbit.

• Currently LLRF transmits data via WR Streamers. These blocks guarantee fixed message latency delivery.

• BE-RF have created the NCOs and modulation scheme that support the SPS's dynamic RF.

• BE-RF have developed a Trigger Unit (TU) that we may may also incorporate into the design to generate the bunch and turn clocks. Evaluate TU versus RFtime.

• Reuse existing VME cores.

Design considerations - Latency

• Each White Rabbit switch will contribute up to 10 µs of additional latency to the output signal. In some respects, the switch appears as a long cable (up to 2 km).

• A reasonable estimate of this latency is assumed to be 100 µs, or approximately 4 orbit turns of the SPS.

• The interaction of a dynamic RF and latency in reproducing that RF will result in a systematic and random errors in the recovered RF signal.

• Smaller latency is better.

Design considerations - BE-BI triggers and pulses, specific to P0 VME bus.

• The bunch and turn clocks output onto the P0 VME backplane.

• The original BOBR card outputs x16 triggers onto the P0 backplane these are selectable from the BST message.

• A bunch selector RAM outputs x8 signals onto the backplane. This enables injection of triggers and pulses associated with a range or specific bunch.

rf distribution over white rabbit. wr2rf - a replacement ......• each white rabbit switch will...

Documents