cms phase ii precision clock distribution studies€¦ · htpd meeting cern 03.12.2018. Ö. sahin,...

CMS Phase II Precision Clock Distribution Studies

P.-A. Bausson, M. Besançon, F. Couderc, M. Dejardin, J.-L. Faure, F. Ferri, S. Ganjour, P. Gras, I. Mandjavidze, Ö. Şahin, M. Titov CEA Saclay-Irfu

HTPD meetingCERN03.12.2018

Ö. Sahin, P.-A. Bausson

Motivation: need for low jitter clock distribution

2

LHC clock 40.078 MHz

Sensor+TDCLHC Clock distribution

• The clock distribution may significantly degrade the precise timing information obtained from the CMS Phase-II detectors.

• A 15 ps random jitter on the clock may have an impact of 3.5 ps (assuming 30 ps [Sensor+TDC] uncertainty).

• The impact will be significantly larger for higher jitter values.

• We need a low-jitter clock distribution not to jeopardise the timing performance.

• Over 10000 channels, FE under harsh radiation conditions.

Pierre-Anne Bausson, Özgür Şahin

Outline

3

• Phase I final full clock distribution chain performances

• Temperature effects

• Phase II hardware measurements

• Pure clock distributions studies

• Clock distribution networks with cascaded PLLs

• Outlook and conclusion


Last year reminder

4

• Overview of the last year:

• Different clock distribution schemes

• PLL loop bandwidth effect on the reference clock jitter for GBT-FPGA.

• Front-End (GBTx chip based) clock performances

• And we introduced studies outlook on :

• Back-End (MicroTCA standard based) clock performances

• Full chain (Back-End to Front-End) clock performances

• Effect of temperature on clock jitter


Phase I final clock chain performances

5

MM OM3 (10 m)

4.8 Gbps

Back-end

Clock source

Front-endClock

generator CG635

BNCSMA 80.156 MHz

HDMI 40.078 MHz

MM OM3 (10 m) 80.156MHz

Jitter cleaner Si5344

Oscilloscope Lecroy820zi-b

VLDBe-link

VTRx

HDMI-SMA

HDMI-SMA

SMA - BNC

SMA - BNC

HDMI 40.078 MHz

Jitter cleaner Si5344

Clk Inj

Clk Inj

SMA

SMA

VLDBVTRx

e-link

µTCA crate

AMC

13

FC7

FMC

µTCA crate

AMC

13 FC7

FMC

MM OM3 (10 m) 80.156MHz

Backplane 40.078 MHz

Backplane 40.078 MHz

eLink to eLink resultsRJ 11.5 psDJ 50.7 psRMS 27.8 ps

3ps RJ increase and 30ps DJ increase are observed with respect to the simpler BE clock distribution (see GBTx tests).


Temperature effects

6

• The impact of temperature is investigated at Saclay PCD lab:

• 20 GHz BW 80 GS/s oscilloscope

• [-40; +180] °C climate chamber

• Setup is as shown below :

Clock source

Back-end

Clock generator CG635 BNC

SMA 40.078 MHz

HDMI 40.078 MHz

Jitter cleaner Si5344 PLL

Xilinx KCU105

SFP+

SFP+

VLDB

VLDB

Oscilloscope SDA 820Zi-B

VTRx

VTRx

e-link

e-link

HDMI-SMA

HDMI-SMA

SMA - BNC

SMA - BNC

SMA 120.234

MHz

HDMI 40.078 MHz

MM OM3 (10 m)

4.8 Gbps

(Climate chamber Front-end)

[-30; +10] °C


Temperature effects

7

• We observed a constant improvement in the clock quality with the decrease in the temperature in the absolute measurements (wrt. scope’s PLL).

• Similarly, we obtained the cleanest channel-to-channel differential clock at the lowest die temperature.

• A consistent 2 ps decrease in the RMS jitter is observed from 20ºC climate chamber temperature to -30ºC.


Clock distribution schemes

8

TCDS2 DAQ crate ReadoutASIClpGBT(s)

~100 m MM Fiber

Encoded @ 2.5 Gbps

RF clock

TCDS2 DAQ crate ReadoutASIC

Rad Hard Clock buffer

Pure clock160 MHz

RF clock

Pure clock master

Pure clock node

ReadoutASIC

Rad Hard Clock bufferRF clock

Pure clock160 MHz

Fiber?@ 320 MHz?

BASELINE

HYBRID

PURE



9

lpGBT performances (ASIC+FPGA)

Full system redundancy

Back-End clock distribution

DAQ board as the clock

distribution + Monitoring

PCD system + MonitoringMonitoring PCD system +

Monitoring

<10ps >10ps

YesNo >10ps<10ps



10

lpGBT performance (asic+fpga)

Full system redundancy

Backend clock distribution

DAQ board as the clock

distribution + Monitoring

PCD system+ MonitoringMonitoring PCD system +

Monitoring

<10ps >10ps

YesNo >10ps<10psBaseli

ne Pure

Pure+

BaselineHybrid


Phase II hardware measurements

11

• The Serenity platform is an ATCA leaf board designed for the Phase-II backend DAQ applications and it is made by a collaboration led by Imperial College.

• We designed daughter cards dedicated to the platform to perform clock jitter measurements

• Views of the 2 daughter cards :


Phase II hardware measurements

12

ATCABack-End Front-End

Interposer

Interposer

Firefly

Firefly

Si53345 Clock buffer

Si5345 PLL

Si5345 PLL

Si5345 PLL

Si5345 PLL

[email protected] MHz - Si5345 9 outputs (North) 1.3 M at 20 MS Acquisition window @ 20GS/s— Reference: clock from the south interposer. The clock (RMS = 1.3 ps) is fed externally.RMS: 2.8 ps, RJ: 1.8 ps, DJ: 4.2 ps

Roger Rusack et al.

Pure clock distribution study

Roger Rusack et al.

Results


U. Minnesota clock distribution tests

16

• University of Minnesota (R. Rusack et al) designed and presented a simple but effective passive clock distribution system to the collaboration.

• They demonstrated that with using modern fast integrated circuits that it is a possible to construct a clock distribution network with a precision of 200 fs between different branches of the network.

• At CEA Saclay will also welcome their system in our lab and tests the system with our test stand.

• The tests are currently planned to start next Monday (10th of December).

Eduardo Mendes, Özgür Şahin

TCDS

• Together with the CERN HPTD group, we have performed tests evaluating the performance of clock distribution networks with cascaded PLL configurations.

• Particularly important for the Phase II clock distribution networks.

Cascaded PLL measurements

DAQ

SMA-SMA 320.625

MHz

Clock source

SourceSi5345revD

EVB

Si5344revD EVB (L1)

MicroSemi SSA

Si5344revD EVB (L1)

320.625MHz

Si5344revD EVB (L2)

SMA-SMA 320.625

MHz Si5344revD EVB (L2)

320.625MHz


• The PLL cascades not only provides a very clean clock distribution network, but it also cleans the (already very clean) input clock as well (with RMS of 0.75 ps).


1

2

3

4

5

6

RM

S jit

ter [

ps]

6.29 4.74 4.50 4.67

6.14 3.61 2.81 2.97

4.96 3.26 1.08 0.87

4.37 2.82 0.87 0.32

Si5345D loopBW 4000 Hz

100 400 1000 4000loopBW L1 [Hz]

100

400

1000

4000

loop

BW L

2 [H

z]

Si5345D loopBW 4000 Hz

1 10 210 310 410 510 610Offset freq. [Hz]

140−

120−

100−

80−

60−

40−

Phas

e no

ise

[dBc

]

RMS: 6.29 psRMS: 6.14 psRMS: 4.96 psRMS: 4.37 psRMS: 4.74 psRMS: 3.61 psRMS: 3.26 psRMS: 2.82 ps


loopBW- L1: 100 Hz - L2: 100Hz


loopBW- L1: 100 Hz - L2: 1000Hz

loopBW- L1: 100 Hz - L2: 4000Hz



loopBW- L1: 400 Hz - L2: 1000Hz

loopBW- L1: 400 Hz - L2: 4000Hz

loopBW- L1: 1000 Hz - L2: 100Hz

loopBW- L1: 1000 Hz - L2: 400Hz

loopBW- L1: 1000 Hz - L2: 1000Hz

loopBW- L1: 1000 Hz - L2: 4000Hz

loopBW- L1: 4000 Hz - L2: 100Hz

loopBW- L1: 4000 Hz - L2: 400Hz

loopBW- L1: 4000 Hz - L2: 1000Hz

loopBW- L1: 4000 Hz - L2: 4000Hz

Si5345lbw4000Hz


• Encouraged with the results, we introduced additional jitter into one of the the clock distribution channels.

• The jitter functions are chosen as square waves (~70 ps RMS) and sinusoidal (~60 ps RMS) waves with different frequencies.


SMA-SMA 320.625

MHz

Source

Si5344revD EVB (L1)

MicroSemi SSA

Si5344revD EVB (L1)

Si5344revD EVB (L2)

SMA-SMA 320.625

MHz Si5344revD EVB (L2)

320.625MHz

Noise


1 10 210 310 410 510 610 710Offset freq. [Hz]

160−

140−

120−

100−

80−

60−

40−

Phas

e no

ise

[dBc

]





loopBW- L1: 100 Hz - L2: 1000Hz

loopBW- L1: 100 Hz - L2: 4000Hz



loopBW- L1: 400 Hz - L2: 1000Hz

loopBW- L1: 400 Hz - L2: 4000Hz

loopBW- L1: 1000 Hz - L2: 100Hz

loopBW- L1: 1000 Hz - L2: 400Hz

loopBW- L1: 1000 Hz - L2: 1000Hz

loopBW- L1: 1000 Hz - L2: 4000Hz

loopBW- L1: 4000 Hz - L2: 100Hz

loopBW- L1: 4000 Hz - L2: 400Hz

loopBW- L1: 4000 Hz - L2: 1000Hz

loopBW- L1: 4000 Hz - L2: 4000Hz

[email protected]

10

20

30

40

50

60

RM

S jit

ter [

ps]

3.89 2.47 2.50 2.84

3.05 5.34 9.33 16.48

2.86 8.98 17.24 32.97

3.22 16.96 33.38 61.80

[email protected]

100 400 1000 4000loopBW L1 [Hz]

100

400

1000

4000

loop

BW L

2 [H

z]

[email protected]

• The initial findings show that if correctly tuned, the cascaded PLLs do not degrade the clock distribution network.

• It can clean possible noise introduced by the network components.



Outlook

21

• Most Phase II are still under design or manufacturing :

• Our ATCA crate and power supply are setup at Saclay

• ATCA DTH and other leaf boards currently in manufacturing, expected Q2 2019

• VTRx+ prototypes and lpGBT engineering samples, expected early 2019

• Phase II baseline full clock chain tests will follow as soon as we have them in hand

• We are trying to identify the system need for a rad-hard clock fan-out chip

• We started to work on possible solutions to monitor slow clock phase drifts

• We will also welcome and tests the passive clock distribution system from U. Minn.

• Our short-term main goals are to study various phase monitoring possibilities and to identify the system requirements for an eventual rad-hard clock fan-out chip

• This year we presented our work to PM2018, TWEPP and ISPCS conferences. A CMS note is also in preparation.

https://agenda.infn.it/conferenceDisplay.py?confId=13450

https://kuleuvencongres.be/twepp2018

http://2018.ispcs.org/


Conclusion

22

! Temperature effects on clock jitter is encouraging given the operating conditions of HgCal at -30 °C

! Our short-term main goal is to study various phase monitoring possibilities and

identify the system requirements for an eventual rad-hard clock fan-out chip

Phase I hardware (MicroTCA+VTRx+GBTx) clock distribution performances are presented "

Phase II hardware (ATCA+VTRx+lpGBT) clock distribution measurements started. Serenity platform seem to perform well and is suitable for clock distribution "

Still waiting components availability. Our test stand is ready to welcome them at

Saclay for clock distribution measurements.


Backup

23


Clock gen blocks

PLL320.625

MHzFS725 Rb.

10MHz

320.625MHz

81134A clock gen

FS725 Rb.

10MHz

TG5011A noise gen

Withnoise

320.625MHz


• The simulation of 30 ps RMS (sensor) convoluted with 11.5 ps RJ and 50.7 ps DJ assuming the dual-dirac jitter model. RMS of the resulting distribution is 40.9 ps.

• DJ becomes significant; it is important to keep (if possible to minimalize) the deterministic jitter under control.

Simulating the clock distribution jitter

25

t [ps]


• AMC13(XG) is distributing the TTC clock through FCLKA pin of the MicroTCA crate.

• The input clock is 2xLHC clock frequency

• The distributed clock is 40.078 MHz

• In our tests we used FC7 board to emulate the BE clock distribution node.

µTCA Clock distribution

26

In the CMS topology one of the MCH slotsare occupied by the AMC13(XG) board.


Rad-hard clock buffer

• 2 to n clock fan-out architecture: output characteristics should be similar to eLink clock.

• It needs to be phase aligned with eLink clock OR

• It needs to forward and align the fast control signals as well

27

Rad hard clock buffer

1

2

…

n

lpGBT eLink clock

Pure clock

Fast control


• Digital Dual Mixer Time Difference circuit aims to measure the ‘time difference’ i.e. ‘latency’ between two clock inputs with a digital circuit.

• Using a slightly different frequency clock, we amplify the clock difference!✴ Waveform is taken from P. Moreira’s thesis

DDMTD

28

cms phase ii precision clock distribution studies€¦ · htpd meeting cern 03.12.2018. Ö. sahin,...

Documents