changes in powering interlocks i.romera , te-mpe

Changes in Powering InterlocksI.Romera, TE-MPE

MPP Workshop – 11th-13th March 2013

Thanks to: P.Dahlen, S.Gunther, R.Mompo, S.Ravat, H.Thiesen, M.Zerlauth

I.Romera - TE/MPEMPP Workshop March 2013

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System

Quench Protection System

Power Converters

Cryogenics Auxiliary Controllers

Warm Magnets

Experiments

Access System

Beam Loss Monitors (Arc)

Collimation System

Radio Frequency System

Injection Systems

Vacuum System

Access System

Beam Interlock System

Control System

Essential Controllers

General Emergency Stop

Uninterruptible Supplies

Discharge Circuits

Beam Loss Monitors (Aperture)

Beam Position Monitor

Beam Lifetime Monitor

Fast Magnet Current Changes

Beam Television

Control Room

Software Interlock System

TimingSystemPost Mortem

Safe Machine Parameters

HTS temperature interlock

Access vs Powering

Outline

Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room





Access vs Powering

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room





Access vs Powering


Sensitivity to electrical perturbations• Electrical network glitches are one of the main dump causes from magnet powering systems• 24 FMCM triggers last year due to glitches in the electrical network distribution• A large fraction do not result in equipment trips and only seen by FMCMs!• High sensitivity (<3x10e-4) on FMCMs is required to avoid effects on circulating beams• Being RD1 and RD34 most sensitive circuits which tripped in most of the cases• RD1.LR5 is about 40% more sensitive than RD1.LR1 despite similar input amplitude


Sensitivity to electrical perturbationsProposed mitigations

2012

FMCM dump cause Systems affected by EL perturbationRD1 RD34 RQ5 RQ4 RBXWTV RMSD

0%

10%

20%

30%

40%

50%

60%

70% 63%56%

15%10%

13% 13%

Likelihood to trip


Mitigations put in place in 2012• FMCM review carried out to study possible mitigations to the sensitivity problem [1]• Thresholds relaxed by factor 3 on RBXWTV and RD34 still maintaining specification values[2]• Simulations on RD1 shown that there is no margin for threshold change• No significant availability improvement seen after mitigations as always in the shadow of RD1• Other mitigations were studied: Improving regulation characteristics of the PC, changing circuit

impedance by adding SC solenoid and replacing thyristor-based by switch mode PC…


Sensitivity to electrical perturbationsMitigations in place in 2012Post L1 operation

Courtesy of T.Baer

Effect of 100mA current change on orbit in the arc


Post LS1 operation• EPC will soon initiate a collaboration with EPFL to better understand the effects of such glitches• After tests started in 2012, EPC is confident to provide a better rejection of elec. perturbations

by only changing the regulation characteristics of the RD1 and RD34 power converters• We expect that… ~40% of minor perturbation trips will be avoided in the future…


Sensitivity to electrical perturbationsMitigations in place in 2012Post LS1 operation

23-1005-0924-0823-0804-0808-0730-0608-0603-0620-0507-0507-0512-0412-040.0

5.0

10.0

15.0

20.0

25.0

RD1.LR1RD1.LR5RD34.LR3RD34.LR7RQ5.LR3RQ4.LR3

EL perturbations where only FMCM tripped (2012)

Exce

ed fa

ctor

in F

MC

Ms


~7% beam loss in B2,almost nothing in B1

I_MEAS

V_MEAS

Trip of power converter

400um in B1H

200um in B1V

200um in B2H

100um in B2V

Late dump by CMS solenoid trip - 10.08.2012• CMS solenoid trip due to a cooling problem which had an effect on beams [3]• Beams dumped only when I_MEAS at 40% of nominal value (3 minutes after solenoid trip)• Orbit deviation up to 400um seen by interlock BPMs• Very slow and distributed losses not reaching BLM thresholds• Beams dumped on high voltage BLM interlock (via the SIS)


Experimental magnetsLHC Power Converters60A orbit correctors


Late dump by LHCb trip - 19.08.2012• LHCb trip led to slight beam orbit deviations before beam dump [3]• Orbit changed by 20um during the last 200 turns• Magnet Safety System (MSS) takes 30ms to send request to Beam Interlock System




Foreseen changes to Magnet Safety System• MSS upgrade to MSS2: NI cRIO FPGA platform to replace obsolete fuse programmable FPGA• Interlock will be provided in case of a Fast Discharge (CMS trip…)• Optocoupler will replace the output safety relay and time to interlock reduced to 16ms



Power Converter Fault

Optocouplers

cRIO FPGA Processing

Optocouplers

CIBU

13 ms

1 ms

1 ms

< 1 ms

MSS2

Total delay ≈ 16ms

Power Converter Fault

Optocouplers

FPGA Processing

Safety relays

CIBU

13 ms

1 ms

15 ms

< 1 ms

MSS

Total delay ≈ 30msCourtesy of S.Ravat


Late dumps by LHC Power Converters• 2 events last year on inner triplet power converters • On 15.06.2012 beams dumped due to high beam losses in SR7 [4]• A broken diode inside RTQX2.L2 provoked voltage oscillations which put in conduction the

FWD in parallel with Q3 magnet • Large circulating currents between the 3 converters and RTQX1.L2 tripped by I_OUT_OVER

CURRENT only 300ms after first current excursions observed• Better sensitivity achieved by reducing I_ERR and I_POS limits in the FGC



RQX[8kA/8V]

RTQX2[6kA/8V]

RTQX1[±600A/±10V] FWDFWT

KEK MQXA6450A/91mH

Q2a Q2b Q3Q1

Fermilab MQXB11390A/19mH

Fermilab MQXB11390A/19mH

KEK MQXA6450A/91mH

++

+-

-

-Courant d’oscillation

RTQX1.L2

I_OUT OVER CURRENT

I_AI_MEAS

I_REFV_REF


Late dumps by LHC Power Converters• On 07.09.2012 beams were dumped on beam losses due to trip of RTQX2.R5• Interlock was not sent to the PIC due to radiation induced latch-up• PIC interlock only received ~2 seconds after the dump by VS_FAULT on RTQX1• During LS1 the 2s FGC watchdog will be removed as part of FGC2 CPLD upgrade!




Late dump by loss of 60 Powering Permit • No hardware signals between Power Converters and interlock system• Magnet protection ensured by Power Converters and interlocks just to avoid

unnecessary quenches of magnets and current leads and to help operations• On 25.10.2012 communication problem with PIC => BLMs triggered in SR7 [5]

FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS

Experimental magnetsLHC Power Converter interlocks60A orbit corrector interlocks

Cryogenics

Powering Interlock

Controllers

PICPVSS LHC Timing

CRYO_OK_PP60A(PVSS -> PVSS)

POW_SUB_OFFCOMM ALIVE

PC_PERMIT_60A(CMW)

EPC gateway(s)

LHC Timing cable

Telegram groupPP60A

Cryogenics

Powering Interlock

Controllers

PICPVSS LHC Timing

CRYO_OK_PP60A(PVSS -> PVSS)

POW_SUB_OFFCOMM ALIVE

PC_PERMIT_60A(CMW)

EPC gateway(s)

LHC Timing cable

Telegram groupPP60A

~ 16:20:38 ~ 3min

~ 1sec


Late dump by loss of 60 Powering Permit • PC_PERMIT_60A would have never been lost with the correct PVSS logic due to a

communication problem affecting a single PIC…• PVSS logic will be corrected during LS1

Logic implemented…What we saw…


Experimental magnetsLHC Power Converter interlocks60A orbit corrector interlocks


Renovation of interlocks in SPS• Last June a problem with ring-line interlocks in SPS caused 16 hours of downtime! [6]• Existing magnet protection system relies on 3 interlock systems (dating from 70’s)• Spares for ring-line and auxiliary lines but no spares available for main interlocks• Consolidation works done during 2012 and full renovation will be done during LS1 [7]• Electromagnetic solution to be replaced by standard WIC interlock system


Renovation of SPS interlocksChanges in Access PoweringOperational improvements


Courtesy of L.Ponce – see Chamonix 2010

Changes in Access vs Powering• Software-based solution put in place after the incident in September 2008• A more dependable implementation will replace the existing one during LS1 [8]• New PIC-PLC to be installed by the LHC Access Control System (LACS) in the CCR• Communication between LACS and PIC-PLC via hardwired signals• PIC-PLC will publish access status to CMW and SIS will subscribe to the data



Courtesy of J.Wenninger


Operational improvements• Protection mechanisms to avoid quench propagation to neighboring magnets• During HWC campaigns such global protection is a bottleneck for testing!• Proposal is to make the Global protection mechanisms maskable via PVSS• Masking status clearly visible from PIC SCADA system




Operational improvements• To allow for some flexibility, not all circuit trips will directly request beam dumps:

• RCD, RCO, ROD, ROF, RQS, RSS… do NOT directly dump the beam• Review of critical circuits for operation in view of new ATS optics… • In 2012, quadrupole correctors RQSX3 were included in maskable configuration




And some additional changes…

• PICs:• 25% of the controllers to be relocated within R2E project (UL14/16 &UL557) [9]• Re-commissioning of UPS links following UPS renovation (see V.Chareyre talk)• HTS room temperature interlocks in RR53 routed through the PIC

• FMCM:• FMCM in UJ56 relocated to USC55 within R2E project [10]• Controls renovation in SPS transfer lines (removal of obsolete ROCS-MUGEF)

WIC:• Ti2/Ti8 - Installation of 800 interlock boxes

to allow remote testing of magnet interlocks




Summary• High sensitivity of FMCMs against electrical network perturbations:

• Thresholds relaxed on RBXWTV and RD34 during 2012 • Voltage regulation improvements on the power converters during LS1

• Redundant protection to the BLMs will be restored after LS1:• PC_PERMIT_60A logic will be corrected in PVSS• Upgrade on Magnet Safety System• FGC2 CPLD upgrade to remove the 2s watchdog in case of FGC crash

• No SEUs observed on PICs last year, thanks to temporary relocation in 2011

• SPS magnet interlocks were consolidated and will be fully renovated during LS1

• More dependable approach for the Access Powering interlocks



Thank you for your attention!• References:

• [1] FMCM review, TE-Technical meeting• [2] ECR-Change of Protection Threshold for FMCMs installed on the ALICE

compensators and the twin aperture warm separation circuits, EDMS 1213337• [3] 67 - MPP meeting, Minutes • [4] 63 - MPP meeting, Minutes• [5] 75 - MPP meeting , Minutes• [6] Problem with SPS interlock crate, MPE-COMS-167• [7] TE-MPE-Technical meeting, Slides• [8] ECR-Change of the Interlocking of Powering and Access systems, EDMS 1246780 • [9] ECR-Activités R2E au point 1 durant LS1, EDMS 1263553• [10] ECR-Activités R2E au point 5 durant LS1, EDMS 1263595


https://edms.cern.ch/file/1211444/1/TE-TM_03_04_2012.pdf

https://edms.cern.ch/document/1213337/1

http://lhc-mpwg.web.cern.ch/lhc-mpwg/MPP-Meetings/No67-14-09-2012/20120914_Minutes_MPP67.docx



http://issues/browse/TE-MPE-COMS-167

http://indico.cern.ch/conferenceDisplay.py?confId=203150

https://edms.cern.ch/document/1246780

https://edms.cern.ch/document/1246780

https://edms.cern.ch/comment/1263553/0.1

https://edms.cern.ch/comment/1263595/0.1

changes in powering interlocks i.romera , te-mpe

Documents

cern authentication

authentication sign

cern credentials

cern account reminder

cern windows

federation account

institution account

htw chur university