TE

-MP

E-E

P, R

D,

26-A

ug-2

015

1

Workshop on QPS Software LayerHardware / Agents

R. Denz, TE-MPE-EP

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

2

QPS supervision - basic architecture

nQPS

IPD, IPQ, IT600 A Leads main circuits EE

QPS supervision provides data for:• Operator screens (WinCC OA) and expert consoles• Software interlocks (QPS_OK signal)• LHC logging database• Post mortem servers, viewers and automatic analysis• Warning generation (SMS, email) for pre-defined faults states

(e.g. loss of a quench heater power supply)• QPS configuration management (LSA)

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

3

QPS supervision user requirements over the years …

2001: • Main scope is the supervision of QPS/EE

detection and protection devices.• The initially limited diagnostic capabilities

for the superconducting circuits have been regarded as fully sufficient at the time.

2014: • Main scope is now the enhanced

diagnostics of the superconducting circuits and elements exceeding in many cases the capabilities of the magnet test benches by far (e.g. splices, quench heaters).

• The supervision of the significantly more complex detection systems required as well some substantial upgrades.

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

4

QPS low level hardware (field-bus couplers)

Fieldbus (WorldFip™) controlled data acquisition system

– “Classic” architecture combining a 8052 compatible μ-controller (ADuC831™) with the notorious MicroFip™ ASIC

• Developed at the beginning of the century

– Synchronized to accelerator time (∆t = 1 ms) via the field-bus

• Absolute time stamping done by the field device

– Up to 8 analog input channels and up to 80 digital I/O channels

• Latest QPS crate designs use dedicated circuit boards for the sampling of analogue signals (e.g. enhanced quench heater supervision)

– Interfaces to associated equipment by local busses (SPI or I2C)

• Up to 24 active clients

2516 devices in LHC (~2300 exposed to ionizing radiation)

– Required radiation tolerance restricts functionality of field devices e.g. for data buffering & processing

2002 - 2005

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

5

QPS low level hardware – device firmware

Core of the field-bus coupler is the 8052 compatible ADuC831™ chip

– Running @ 11.059 MHz with 12 clocks / instruction

– Code is executed from up to 62 kBytes program flash/EE memory

– Chip is sufficiently radiation tolerant for LHC operation (run 1 experience, tests)

Firmware written in C for 8052 using Keil™ compiler

– OS free embedded C application

– Synchronization to LHC controls and time by external interrupt created by MicroFip™

– PM event timestamping by polling the DAQ trigger lines with ∆t = 1 ms

– Controller manages as well the local crate communication (as software master!)

– Base application and corresponding API date from 2002 with many incremental upgrades over the years

• Based on the gained experience the code and the API could be simplified significantly

• Requires a major revision (also for gateway RT application)

• Preferably to be combined with possible hardware upgrade during LS3

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

6

QPS low level hardware – field-bus

Fieldbus of the WorldFIP™ standard (1 Mbit/s)

– Macro-cycle length:

• tmacro = 100 ms for tunnel segments (agents type DQAMC, DQAMGS)

• tmacro = 200 ms for underground areas (agents type DQAMG, DQAMS)

Clients are based on the MicroFIP™ ASIC (different versions and makes)

– Only periodic communication, no messages

– Consumed variables TIME (global) and COMMAND, 8 Bytes each

– Produced variables DATA0, 1, 2, 3 24 Bytes each

• Maximum data transmission rate per client: 960 Byte/s

– The modest amount of 96 Bytes may translate into 160 supervision signals!

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

7

QPS low level hardware limitations

Rather limited size of MicroFip™ memory (120 Bytes)

– Limits maximum amount of data transferable during one macro-cycle to 96 Bytes

– Cannot transfer data from redundant devices within one cycle (independent of macro- cycle length) must toggle between A/B

Limited number of hardware trigger lines

– Typically one hardware DAQ trigger per physical controller

– Can cause problems with PM time stamping, identification of trigger sources, independent operation of circuits, especially during commissioning

• Affects in particular the 600 A multi-circuit protection crates

Remote firmware upgrades

– Controller hardware supports remote firmware upgrades in most of the cases

• Implementation would require however a new controller firmware and a major upgrade of the gateway RT application (e.g. allowing on the fly change of macro cycle etc.)

– Remote upgrade of detection board firmware only supported by some hardware ( SPI/I2C programmable devices only, no JTAG chain)

• Rarely needed, potential safety issue

• The original hardware design dating back to 2002 revealed during the many years of operation some limitations

• Not all of the constraints can be overcome easily and some may require major upgrades

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

8

QPS low level hardware – R2E related problems

Radiation tolerance of the MicroFip™

– In some cases the device gets stuck by radiation induced effects loss of field-bus communication power cycle for recovering

• Automatic for the MB supervision systems (LS1 upgrade)

• By remote power cycle for the 600 A supervision installed in the RR

• By access for the rest

• 12 events in 2012 (9 x DQAMCMB, 3 x DQAMCMQ)

• Upgrade for MQ supervision in preparation (could be deployed during TS)

Stalled DAQ triggers due to SEU for DQAMCMB and DQAMCMQ controllers

– “ISO150 problem” mitigated by firmware (199 events in 2012)

• The blocked DAQ trigger can be remotely unlatched (DQAMCMB only)

• Creates a “fake” PM, interpreted by the analysis tools as failed heater discharge

• As the controller communicates this particular faults state the analysis tools should take notice

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

9

QPS low level hardware – upgrades during LS1 I

Re-configuration of QPS field-bus networks (for details see Julien’s talk)

– Eases possible upgrades as less agents per segment

– Doubles the data transmission rate in the LHC tunnel segments

• Original user request for reducing transmission time of large PM data blocks, especially for the enhanced quench heater supervision

• now everybody wants it …

Visibility of redundant circuit boards

– Simultaneous transmission during normal operation (=logging) not feasible due to field-bus limitations A/B toggling required

• Some issues in high level supervision still to be fixed (loss of QPS_OK)

– Successfully implemented for PM data

Transfer of analogue data to logging database

– Suppression of on change data recording, differentiation between slow and fast data

– Essential basis for many tools like signal integrity checkers etc.

– Very successful, smooth and absolutely essential upgrade!

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

10

QPS low level hardware – upgrades during LS1 II

Remote recovery of stalled communication busses

– Implemented for SPI bus based systems (MB and MQ protection)

• Allows bus recovery without power cycle or access

• Meanwhile integrated into Swiss tool

• To be extended to I2C based systems

– As the bus recovery process does not create triggers the implementation of a fully automatic transparent solution can be envisaged

Introduction of configuration management

– Additional level of system security by checking systematically system parameters

• Critical QPS parameters are already very well protected by the detection system firmware

– Implementation starting currently with main circuits followed by IPQ …

– Automatic change/correction of device parameters not yet implemented

• No urgent need for the time being

• Could be used in some cases to ease system maintenance

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

11

QPS low level hardware – future developments

Complete LS1 upgrades

– Concerns 600 A systems in the RR, can profit to fix some PM issues

– Complete implementation of configuration management

Additional test benches allowing system level tests

– Simulation of complete field-bus segments, tests of complete software layer etc.

– To be installed in the new TE-MPE test areas in 272

Additional hardware DAQ triggers for IPQ and 600 A systems

– Technically solved, implementation depends only on available specialist resources

Radiation exposed systems (tunnel + RR73/RR77)

– Keep CERNFip as standard field-bus

– Migrate from QPS clients μFIP to NanoFip

• Requires complete new development due to lack of backward compatibility

• Some QPS functionality to be transferred to RT application

Systems installed in radiation free areas (LHC)

– Possible change to modern field-bus, e.g. Ethercat™

TE

-MP

E-E

P, R

D,

26-A

ug-2

015

12

Summary & conclusions

QPS supervision is an essential part of an important machine protection system and the key to the diagnostics of the superconducting circuits

– QPS supervision low level hardware is based custom made electronics taking into account the rather specific constraints of the LHC tunnel (radiation, access, EMC …)

Upgrades of the low level hardware and the corresponding firmware are tedious, require specialist knowledge and must be planned carefully and well ahead in time

– Future developments will try minimizing the data processing in the field and move functionality to the gateways

Further evolution of RT application needs to be discussed

– The current application is working very well but the LS1 experience and future upgrades of the QPS supervision ask for changes

• Future developments will require better control of the application code by a team of experts familiar with the QPS field constraints

• Effective data processing prior to the transfer to the end user application can solve the majority of the problems encountered during the LS1 hardware commissioning

– The QPS team prefers clearly insourcing the RT application in the medium term