blm 2008 audit follow up

B.Dehning 1

BLM 2008 Audit Follow Up

B. DehningCERN BE/BI

B.Dehning 2

System Overview

Tunnel: Detector: ionisation chamber + Secondary emission monitor Signal digitalisation and transmission by CFC card

Surface VME cards: Signal reception and threshold comparison Combination of beam permit signals and surveys tasks

Database and Software: Detailed system information in MTF Detailed installation description in Layout

Reference settings for hardware are in LSA Status of System:

after the 19 September event about 500 monitors have been dismounted re-installation is ongoing

Upgrade of frontend electronics was done (reset card and change in cable HV network) Upgrades on data bases, check procedures and management software

Beam energy

detector LBDSBICCombinerCFC elec. Threshold comp.

Tunnel Surface

Beam energy

B.Dehning 3

Auditors Report Content

Recommendations by the Auditors

5.1 Determination and Management of Thresholds

5.1.1 Simulations of Loss Signals and Response Functions (6 items) (Barbara and Mariusz)

5.1.2 Management and Storage of Threshold Values (10 items) (Barbara and Mariusz.

16+17 Bernd)

5.2 Monitors & Electronics (Bernd)

5.2.1 Ionisation Chambers (2 items) (Bernd)

5.2.2 PCBs and Choice of Components (9 items) (Bernd)

5.2.3 FPGA Programming (5 items) (Bernd)

5.3 Environmental Aspects (5 items) (Bernd)

5.3.1 Electromagnetic Compatibility (EMC) (3 items) (Bernd)

5.4 Commissioning, Testing, and Documentation (6 items) (Bernd, last Barbara)

B.Dehning 4

Answers to the Audit Report (I)

15+16. A “service level agreement” and or “Memorandum of understanding” has been

discussed, but proposal has been rejected (it is not expected that it will be useful BE-CO).

18. It is believed that a save beam permit inhibit is every time possible with the present

tunnel cable network (enough charge in capacitors). Due to observation with one bunch

losses, the decision was taken to separate IC from SEM and also ARC supply from SSS

supply. And in addition in point 3 and 7 also the ICs of LSS had been grouped in high and

medium losses. The effect is only visible in the chambers connected to the same HV box.

19. SEM are calibrated in the beam lines H6 and H4 (see next transparency).

20. Components and PCBs for 100 Spare cards are ordered and production is launched. We

have 400 FPGA spares (CFC, tunnel card).

21. Not done for complete card. Will be done later.

22. Mezzanine card for reset was developed and implemented on all the CFC-Cards, the

reset can be provoked with a reset bottom, WorldFip (card wise, but not implemented yet)

and the HV (point wise, operational).

23. Not done. Will be done later.

B.Dehning 5

Secondary Electrons per Protons on target

• From histogram:– Mean ratio =

3.40

– Standard deviation = 0.92

• From Geant4– Ratio = 3.95

– Stdev = 0.17

-50 0 50 100 150 200 250 300 350 4000

2

4

6

8SEM/beam ratio, expect 3.95+-0.17

SEM ID#

QS

EM

/Qbe

am

0 1 2 3 4 5 6 70

2

4

6

8

10SEM/beam ratio Mean: 3.40 sigma: 0.92

QSEM

/Qbeam

B.Dehning 6

Answers to the Audit Report (I)

15+16. A “service level agreement” and or “Memorandum of understanding” has been

discussed, but proposal has been rejected (it is not expected that it will be useful BE-CO).

18. It is believed that a save beam permit inhibit is every time possible with the present

tunnel cable network (enough charge in capacitors). Due to observation with one bunch

losses, the decision was taken to separate IC from SEM and also ARC supply from SSS

supply. And in addition in point 3 and 7 also the ICs of LSS had been grouped in high and

medium losses. The effect is only visible in the chambers connected to the same HV box.

19. SEM are calibrated in the beam lines H6 and H4 (see next transparency).

20. Components and PCBs for 100 Spare cards are ordered and production is launched. We

have 400 FPGA spares (CFC, tunnel card).

21. Not done for complete card. Will be done later.

22. Mezzanine card for reset was developed and implemented on all the CFC-Cards, the

reset can be provoked with a reset bottom, WorldFip (card wise, but not implemented yet)

and the HV (point wise, operational).


B.Dehning 7

Answers to the Audit Report (II)


25. + 26. PC; 3A max current of the regulator, nominal current are up to 350mA; 250mA;

800mA. Redundant PC will be installed when the test result support this decision.

27. HV redundant PC exist .

28. The filter is used for a pre integration, no signal is lost but stored in the capacitor and

then discharged in the integrator. Test have been made with short pulses. Over voltage and

over current has been tested 1500V@100us and 10A@100us. MTBF calculations have been

done. We do not expect a increased failure rate, no action.

29. FPGA synchronisation of asynchronous inputs. We are in a advanced state of defining

and documenting FPGA design rules to be used for future projects in BI and CO. In addition,

the mentioned example has been corrected.

30. CFC card, remote control is added, which will reset the state machine if in a non defined

state.

31. We have invested in the construction of the vertical slice test system and a set of tests

necessary to pass before releasing a new firmware (see next five transparencies).

32. Unfortunately it is not still possible for this kind of review as the firmware for the BLETC

and BLECS are needed to continue changing and adapting to the rest of the controls

infrastructure. The complexity requires optimisations across the complete hierarchy. It will

be done.

B.Dehning 8

BLM Acquisition Chain and CFC Simulator

B.Dehning 9

Vertical Slice test system

Tests foreseen• Drive packets similar to those

sent by the CFCs• Verify the response of the TC

in case of correct inputo Are the running sums

produced correctly?o Are threshold comparisons

done correctly?

• Verify the response of the TC in case of erroneous input

o CRC errorso Differences between the

redundant optical channelsMezzanine PCB in production

B.Dehning 10

Test of Beam Permit Inhibit

B.Dehning 11

Beam Permit Inhibit Tests (I)

• Compiled C code running on the Front End

Computer (PowerPC architecture)

• The threshold table of a given Threshold

Comparator is repetitively modified to

generate beam permit inhibit

• The responses of the TC and the Combiner

card are evaluated in every iteration

• Complete run: test every possible

combination of

o 16 channels

o 12 running sums

o 32 energy levels

= 6’144 iterations, about 18 hours running

time

• One test in each of the 8 IPs can be run in

parallel

B.Dehning 12

Beam Permit Inhibit Tests (II)

Tests runs A total of 10 complete tests on 9 different cards in 5 crates: 61’440 accesses

Error rates Combiner (direct connection to the Beam Interlock): no errors Threshold Comparator ≤ 1% of dump requests not logged

max. 62 out of the total of 6’144 No errors writing the thresholds into the memory (iterative testing)

Possible reasons for errors Bugs in the logging part of the Threshold Comparator firmware Bugs in LynxOS, the operating system of the FECs Unhandled bus errors

Tests run to investigate errors sources A total of over 750’000 accesses have been done in 5 cards in 5 crates Test code: TC and Combiner dump response read out and evaluated over

numerous iterations in unchanged conditions Error rates

Combiner (direct connection to the Beam Interlock): no errors Threshold Comparator ≤ 0.01% of dump requests not logged

max. a dozen out of 200’000

B.Dehning 13



25. + 26. PC; 3A max current of the regulator, nominal current are up to 350mA; 250mA;

800mA. Redundant PC will be installed when the test result support this decision.

27. HV redundant PC exist .

28. The filter is used for a pre integration, no signal is lost but stored in the capacitor and

then discharged in the integrator. Test have been made with short pulses. Over voltage and

over current has been tested 1500V@100us and 10A@100us. MTBF calculations have been

done. We do not expect a increased failure rate, no action.

29. FPGA synchronisation of asynchronous inputs. We are in a advanced state of defining

and documenting FPGA design rules to be used for future projects in BI and CO. In addition,

the mentioned example has been corrected.

30. CFC card, remote control is added, which will reset the state machine if in a non defined

state.

31. We have invested in the construction of the vertical slice test system and a set of tests

necessary to pass before releasing a new firmware (see next five transparencies).

32. Unfortunately it is not still possible for this kind of review as the firmware for the BLETC

and BLECS are needed to continue changing and adapting to the rest of the controls

infrastructure. The complexity requires optimisations across the complete hierarchy. It will

be done.

B.Dehning 14

Answers to the Audit Report (III)

33. All sub-projects have been placed in a centralised BI-BL folder on SVN. In addition, common guidelines for the information to be included and the structure for its storage has been defined and followed for all sub-projects. Currently an effort is done to extend this also in the rest of the section and include the scripts for data analysis, threshold generation, GEANT4 simulations etc.

34. The walkie-talkie not done yet, will be done later. 35. Noise rejection of the CFC part often tested during operation. 36. Magnetic fields no influence observed, CFC card has been tested up to 3000 Gauss. 37 to 40. radiation issues of electronic discussed before, see next two transparencies. 41. The tests with the source are essential, because it tests the whole chain at a very low

currents 70 pA (10 times lower as any quench level). Batteries are only used in case of a non-functioning of the source tests (see next transparency). The likelihood of a failure after a disconnection is regarded as too high compared to a test at higher currents (every card has a LAB test record).

42. Done last September (see next transparency). 43. Not done yet, many channels are not re-installed yet will be done as soon as possible. 44. Done, often tested, also foreseen in the test procedure before start-up. 45. Vertical slice bench. On the way of its implementation. The installation was instead chosen to

be in the LHC; this will provide the ability of direct comparison of the output signal of the vertical slice with the operational system.

B.Dehning 15

Radiation Tests and Error Rates

Error rates due to tested

radiation effects are

negligible except for the

ARC equipment.

ARC CFC digital part has

redundant signal chain

ARC PC accelerated

aging tests to be done

Equipment total#

criticl#

location Particle particle energy [MeV]

TID particle [kGy]

Fluence (test)[1/ cm2]

h > 20MeV

cross section[cm2]

ARC PS 700 340 below quad p 60 1 1.30E+11 7.69E-12CFC 700 340 below quad p 60 0.66 8.58E+10 1.17E-11

CFC mezzanine 700 700 p 60 0.66 8.58E+10 1.17E-11PS 5 V (with load) 74 8 site tunnel p 63 0.33 4.29E+10 2.33E-11

" " 8 " n+g 1 8.00E+11 1.25E-12" " 8 " mixed CNGS broad 0.031 2.20E+11 4.55E-12

PS 2.5 V (with load) 37 4 site tunnel p 63 0.32 4.16E+10 2.40E-11" " 4 " n+g 1 2.80E+12 3.57E-13" " 4 " mixed CNGS broad 3.60E+11 2.78E-12

Radiation tests

Status 10.06.2009

Doseperyear[Gy]

Total Fluence in year

[1/ cm2/ year]

Expected operation

time [year/ PS]

# of failures

peryear

Comments

2.00E+01 8.00E+10 1.63E+00 2.09E+02 One year of operation, no failure2.00E+01 8.00E+10 1.07E+00 3.17E+02 double optical link redundant2.00E+01 8.00E+10 1.07E+00 6.53E+02 new

1.00E+09 4.29E+01 1.86E-01 max fluence UA63/UA67 (8 critical) 1.00E+09 8.00E+02 1.00E-02 working up to 5E12 n/cm21.00E+09 2.20E+02 3.64E-02 20 % increase1.00E+09 4.16E+01 9.62E-02 max fluence UA63/UA681.00E+09 2.80E+03 1.43E-03 working up to 5E12 n/cm21.00E+09 3.60E+02 1.11E-02 2.6 % increase

LHC

New radiation tests

CNGS mixed field

Prospero 1MeV

neurons

PSI 60 MeV

protons

B.Dehning 16

1 MeV Neutron Tests

In every (p and n) tests smooth change of output voltage observed. No SEE effect observed

B.Dehning 17







B.Dehning 18

Source Measurements

Left top: Signal, Offset and Difference vs monitor number (right of IP1)

Right top: offset signal distribution

Bottom left: source signal distribution:

relative 0.064 spread

B.Dehning 19







B.Dehning 20

Noise Level Distributions

Procedure:

require data

during nominal

operation

conditions of LHC

Choose most

sensitive

integration

intervals

Set histogram

max value to

lowest quench

threshold level

(MB-magnet)

Overflows are

interpreted as

false signals

B.Dehning 21

Noise Signal Distribution of LHC 12. September 2008

Checked:

15 hours in the

nights between 12 to

16 September 2008

Results:

19 channels would

have given false

signals located on 4

electronics cards

Conclusion:

It is assumed that

this channels could

have been

repaired

2009 checks are

foreseen after

reinstallation of

BLM system

Integration time 80 us Integration time 1.3 s

B.Dehning 22



34. The walkie-talkie not done yet, will be done later. 35. Noise rejection of the CFC part often tested during operation. 36. Magnetic fields no influence observed, CFC card has been tested up to 3000 Gauss. 37 to 40. radiation issues of electronic discussed before, see next two transparencies. 41. The tests with the source are essential, because in it test the hole chain at a very low

current 70 pA, 10 times lower as any quench level). Batteries are only used in case of a non-functioning of the source tests (see next transparency).

42. Done last September (see next transparency). 43. Not done yet, many channels are not re-installed yet will be done as soon as possible. 44. Done often tested, also foreseen in the test procedure before start-up. 45. Vertical slice bench. On the way of its implementation. The installation was instead

chosen to be in the LHC; this will provide the ability of direct comparison of the output signal of the vertical slice with the operational system.

B.Dehning 23

Conclusion

Several of the recommendations are implemented

Several will be followed up and decisions will be taken later

(accelerated aging testing of PC and redundant

implementations)

Documentation will be made available at:

www.cern.ch/blm

blm 2008 audit follow up

Documents

items bernd5

items barbara

reset card

tunnel card

monitors electronics

mezzanine card

sss supply

complete card