the cern lhc central timing a vertical slice pablo alvarez jean-claude bau stephane deghaye ioan...
TRANSCRIPT
The CERN LHC central timing
A vertical slicePablo Alvarez
Jean-Claude BauStephane Deghaye
Ioan KozsarJulian Lewis
Javier Serrano
Talk layout
• LHC Injector Chain Timing• Timing Distribution Overview• Central Timing Overview• Reflective Memory• UTC Time and GPS• The Multitask Timing Generator• The Controls Timing Receiver Module• Safe Machine Parameters Hardware
Checking• LHC Software Architecture and Event Tables
R1LHC
CNGS
Linac PSB CPS SPS
D3Dump
TI8Dump
TI2DumpSPS
Dump
R2LHC
TCLP
The LHC Proton Injector Chain
Strongly time coupled
CERN accelerator networksequenced by central timing
generator
….…. ….….
PSB
CPS
SPS
LHC
Experimentalarea
Experimentalarea
ExperimentalArea
CBCM Sequence Manager
PSB1 PSB1 PSB2PSB2 PSB3PSB3 PSB4 PSB4
CPS Batch 1 CPS Batch 2 CPS Batch 3 CPS Batch 4
SPS Cycle for the LHC
SPS injection plateaux
The LHC Beam
LSA Beam request:RF bucketRingCPS batches
ExtractionForewarning
LHC Injection plateaux
Injection Injection
ExtractionExtraction
The LHC timing is onlycoupled by extraction
start-rampevent
ExtractionForewarning
Timing Distribution Overview
• RS485 drop nets distribute timing around the CERN accelerator complex
• Long distance transmission over optical fibers
• One timing network for each accelerator• Hundreds/Thousands of timing receiver
modules distributed around the complex• One timing generator drives one timing
network
Timing Frames
• Timing frames are Manchester encoded at 500kbit using a 1MHz clock.
• Each frame carries 32 data bits, parity, start and stop bits.• One frame transmitted each 125us, 8 per millisecond.• Frames are broken into bit fields
– 4 Bits Accelerator [A]– 4 Bits frame Type [T]– 8 Bits Code [CODE]– 16 Bits Payload [PAYLOAD]
• Some frame are recognized by the hardware and cause special treatment– Two UTC frames carry the time of day in their payload– Millisecond frames are always sent in phase with the PPS– Telegram frames are stored in double buffers– Event frames cause counters to be loaded and triggered and
may produce bus interrupts
MillisecondMillisecond
A T CODE PAYLOAD
7 Other frames
LHCCentral Timing
CBCM Controlling
Injector Chain
CS-CCR-CTMGWFESA API
LSASequencer
“LSA”
SPS destination request R1,R2
CPS Batch Request 1,2,3,4
LIC Timing
HIX.FW1K
LHC TimingInhibitsRequestsInterlocks
SIS TI8/TI2& SPS DumpInhibits
LHC Fill Requests:BucketRingBatches
MasterON/OFF
CTR
LHC Fill Requests:BucketRingBatches
ReflectiveMemoryLink
CBCM SequenceManager
LIC Sequence
TI8/TI2Dump
RequestTI8/TI2Dump
LHCUser
RequestLHC User
NormalSpare
LSA changes Allowed
LSA Master
SEX.FW1KSPS.COMLN.LSA_ALW
Central TimingOverview
LHC GatewayImplements FESA API
2Gbit/SToken ringVMIACC-5595
Single ModeHub
64MbyteVMIPMC-5565Reflective memory
LHC Timing generator A LHC Timing generator B
Reflective Memory
Reflective memory:
A and B must always bein the same state.If no restrictions forswitch over
Protects token ring
IdenticalexceptID event
GPSOne pulse per Second
GPS
SymmetricomXLI
PLLOne pulse
per Second
Phase locked10MHz
Basic Period1200/900/600 ms
Advanced (100us)One pulseper Second
Synchronized 1KHz(slow timing clock)
Phase locked10MHz
Phase looked40 MHz Eventencoding clock
40MHz PLL
Synchronizationmodule in eachtiming generatorcrate
RS485Timing
MTTMultitaskTimingGeneratorMTT
UTC time (NTP or GPS)
Eventtables
External events
UTC Time and GPS
CERN UTC Time
Set once on startup & on Leap Seconds
RS485Timing
CTR
PPS
10 MHz
1 KHz
40MHz
Delay
Control SystemCERN UTC Time
25ns steps
Timing receiver
SymmetricomCS4000portable
Atomic Clock
Multitask Timing Generator MTT
• Hardware multitasking for 16 tasks – 32 local registers per task– 218 Global registers– 6 Memory mapped IO registers
• Timing frame out register• VME P2 in register …
• Host processor access to all registers
• Implements general purpose CPU– Op-codes are triadic: AND SrcREG,SrcREG,DstREG – AND 0x7,VMEP2,TMP
• Arithmetic and logical• Move indexed, literal, register• Wait value, relative• Conditional branch • Interrupt host
• Tasks defined from host via Task Control Block– PC– PC Offset– Processor Status Word
• Command and status registers allow host access to running tasks
MTT hardware module
FPGA (Spartan2 600E)
CPU
VME
Interface
UTC SERIALIZER
40MH
z
PP
S
SY
NC
Timing Signal
VME
P2
External Events
VME BUS
See: The LHC central timing hardware implementationP. Alvarez, J. Lewis, J. Serrano CERN, Geneva, Switzerland This conference
strp: % Start program and interrupt survey taskmovv TskStsRUNNING LRegTASK_STATUS % Say we are runningmovv ConsNOT_SET LRegParRUN_COUNT % Run foreverint 2 % Notify survey we are running
cont: % Wait for the VME P2 bits and send out events accordinglyworv ConsVMEP2_BITS VMEP2 % Wait for VME P2 bitsmovr VMEP2 RegVmeP2 % Copy reg and clear bits
tdmp1: % Test for dump ring 1andv ConsHX_DMPD1_BIT RegVmeP2 LRegTEMP % Test dump 1 bitbeq tdmp2 % Go check for dump 2 bitmovv ConsHX_DMPD1 EVOUT % Send dump 1movv ConsHX_ENBPM1 EVOUT % Re-enable after a dump
tdmp2: % Test for dump ring 2andv ConsHX_DMPD2_BIT RegVmeP2 LRegTEMP % Test dump 2 bitbeq tinj % Go injection warning bitmovv ConsHX_DMPD2 EVOUT % Send dump 2movv ConsHX_ENBPM2 EVOUT % Re-enable after a dump
tinj: % Test for LHC injectionandv ConsHIX_FW_BIT RegVmeP2 LRegTEMP % Test inject bitbeq tpm1 % Go check for PM ring 1movv ConsHIX_FW EVOUT % Send injection forewarning
tpm1: % Test for post mortem bit 1andv ConsHX_PM1_BIT RegVmeP2 LRegTEMP % Test PM ring 1 bitbeq tpm2 % Go check for PM ring 2movv ConsHX_PM1 EVOUT % Send PM-1 trigger
tpm2: % Test for post mortem bit 2andv ConsHX_PM2_BIT RegVmeP2 LRegTEMP % Test PM ring 2 bitbeq cont % Go check for PM ring 2movv ConsHX_PM1 EVOUT % Send PM-1 trigger (not PM-2)
jmp cont % Go wait for next P2 interrupt
MTT External Events Task
Controls Timing Receiver CTR
HybridPLL
Delay
FrameDecoder
TGM
TGMUTC
Content AddressedMemory - Triggering
CounterConfigurationsLoader
22Bit 50MHzCounters
BusInterrupt
Output
Timing
40MHz
LookupLoad
Trigger
•Start•Previous•External•Event
•Clock•Previous•40MHz harmonic•External
•Modes•Once•Multi-pulse•Burst
•Trigger•Event frame•Wildcard•Telegram
•Action•Output•Bus interrupt
•Output•TTL/TTL bar•Pulse width
CTRV – CTRI – CTRP formats
Reads thresholds
@1kHz-24Bit 108
Safe Machine Parameters Controller
for LHC
Energy A
I_beam1 & 2
Reads status
Energy B
BEM
BCT “A”
BEM
ManagementCritical
Settings
(CTRV)
CTRV
CTRx
LSA
BCT “B”
EXPLine
driver
LHCTiming
Generator
SMP @ 10Hz 16Bit 1010
(Flags, E & Int.)
BLMCTRVCTRV BLM BLM
CTRVCTRV
BIS BIS
BIS
CTRVCTRVCTRV
Kickers Kickers
Timing Network
Events, UTC, & Telegrams (including SMP)
Safe Machine Parameters Distribution
BIS
CTRxCTRxCTRx
EXP
Flags TTLHw Output
If length > 5m
LSA and FESA
• The FESA API is implemented on the LHC timing gateway
• Accesses timing generators across reflective memory
• Implements – Load or Unload event table– Get running tables list– Set event table run count and synchronization
event– Stop or Abort event table– Set telegram parameters– Send event– Read status of tasks and MTT module
Event tables
• Event tables are ASCII strings sent from LSA to the LHC timing gateway
• Each entry in the table contains– The name of the event to send “HX.START-RAMP”– The relative time to wait in seconds before sending
the event “1.01”– The payload the event will carry “0x0005”– A carriage-return <CRTN> terminates the entry
• The table run count and synchronization event will be filled in later when the table is to be executed
Garbage CollectLoad objectInitialize TCBRun task
Assembletask
Translateand Merge
HX.Start-Ramp 1.01 0x5HX.Start-Freq 0 0x05
waitr MSFR,1001movv 0x14020005 EVOUTmovv 0x14030005 EVOUT
Event Table Processing
FESA API
LSAEventTable
Table Compiler TranslationEvent Table
template
Table Task
Assembler
Object CodeReflectiveMemory
TaskLoader
MTTProgramMemory
Hot Standby
Data
Process
Conclusion
– The LHC machine processes such as squeezing, physics… may occur at any time and run independently of each other, they share no common basic time period. This has led to a new approach in which machine processes are modeled as independent timing event tasks that can be started, stopped at any time.
• Reuse of existing timing boards was facilitated by using FPGAs and writing new VHDL
– The LHC timing distributes machine safety critical data around the control system. This has led to the development of hardware to monitor its performance and dump the beam when necessary.
– Reflective memory has greatly simplified the implementation of the hot-standby and the timing generators themselves which are no longer required communicate directly with the rest of the control system. This has increased their reliability considerably.