Pasquale D’Angelo – INFN Milano - Bicocca

LV-workshop during CMS Electronics weekLV-workshop during CMS Electronics week

During a recent Workshop, Sergei Lusin reported that in CMS was observed an OVERHEATING IN SEVERAL POWER CONNECTORS.

This is a potential “Fire hazard” that affect all Subdetectors using Power Supplys which use the 55A “powerclaw contact” Specifically the CAEN A3486 – (MAO)

18/06/2010

Pasquale D’Angelo – INFN Milano - Bicocca

Short and Long term suggestion plans

It was generally agreed upon that something needed to be done to allow long term lifetime of the system.

Short term plan Use softer cables less capable of transferring twist,(which does not involve extensive recabling)Followed by the addition of a screwed bracket.

Long term plan The ultimate solution will require the modification of the connector assemblies on the power supplies, whichwill involve discussions with the vendor, and the development of a retrofit plan.

28/06/2010

Voltage drop measurement between MAO and Easy Crates by installation of sense wires.( The option will be tested by the HCAL)

Option

Pasquale D’Angelo – INFN Milano - Bicocca

Following the “Short term solution” we decided the replacement of the short cables links between the AC-DC Converters (MAOs) and the Easy crates 4000, using more flexible cables.More in details for each racks we introduced two new short extensions related to the upper MAO, and we replaced three other extensions related to others MAOs.This will allow the spring mechanism inside the connectors shells to align the contacts properly.

Again, to prevent any progressive “walking – out” of the plug connectorswe added a screwed bracket between the cable connectors and the panel.

Partial recabling on PIXELs racks X2F36/X2F37

Power Supplies cables replacement made at P5 (17February2010)

38/06/2010

LV- workshop during CMS electronics week

General discussion ......& answers

Do you have in-line fuses on the service Power Connectionsthat are fed directly by MAOs? .......................................................................YES

Have you observed any strange turn-off issues of the DC outputsIn the event of interruption of primary or service power?.............................ANY

How do you fix bad modules (LV, HV, any others) ?...............Via a Pixels supportactivities performed at TIF, & via CERN pool.

How do you keep records?.......................................................Via elogs- etc.

Remarks: To control all the devices used at P5 has been created a link on TWIKI twiki.cern.ch/twiki/bin/view/CMS/PixelCaenSpareComponents with an UPDATE MAP of the Power System Parts installed. You can find also a MAP of the DEVICES out of the system......(spare parts)

8/06/2010 Pasquale D’Angelo – INFN Milano Bicocca 4

LV- workshop during CMS electronics week

General discussion ......& answers

8/06/2010 Pasquale D’Angelo – INFN Milano Bicocca 5

What hardware interlocks do you use on the LV and HV Systems?................................Built –in caen system According to our request a new firmware for the A4603 has been implemented, so now the HV can ramp down to V0 using a signal generated by the LHC and sent by the BRM directly to the Pixel CAEN system ( V sel input ) The LV is unaffected.

Who’s in charge of keeping the LV & HV systems working?...........................Gino Bolla, Mauro Dinardo, Pasquale D’Angelo, Bill Gabella, Lalith Pereira

And the interface with DCS? (CAEN system, and Branch drivers).................Gino Bolla, Mauro Dinardo, Pasquale D’Angelo, Bill Gabella, Lalith Pereira.

PIXELSPIXELS SUPPORT ACTIVITIES PERFORMED AT TIF

In addition to the PILOT DETECTOR, In addition to the PILOT DETECTOR, an improved an improved TEST FACILITYTEST FACILITY can perform varius technical functions. can perform varius technical functions.Here the most important:Here the most important:

Control all the spare devices ,and make sure they are ready for replacement.

Quickly test the devices malfunctioned.

Long term test on devices with suspected faults.

Test the functionality of the new Firmwares.

Run all the functional test on prototypes devices for next CMS-UPGRADE.

Activete a very simple set-up (with Panel + Adapter Board + Flex Cable + Port Card + Aoh) on which we can easily operate, to try to understand and address problems, in attempt to avoid future occurences.

1)

2)

6)

5)

4)

3)

Help to maintain under control all the PIXELS devices used at P5.7)

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 6

Pasquale D’Angelo – INFN Milano - Bicocca 78/06/2010

IMPROVEMENTS ON THE..........IMPROVEMENTS ON THE..........

Power Supplys Test Facility Setup

Spare crates & modules

Modules under test

MSC-long cables (43m)

Dummy loads

A3486 (mao)

Same configuration used at P5A3486 MAO

BUS CONNECTIONSBUS CONNECTIONS . . . . . . . . . . . .

The A1676 Branch Controller handles a chainwich includes both: EASY crate and A3486 (MAO)

Easy crate panel(INPUT connector)

A3486 (MAO) (INPUT connector)

A3486 (MAO)(INPUT connector)

A1676 Branch Controller(OUTPUT connector)

Easy crate overview

The crate, which is suitable to host P.S. Modules A4602 & A4603is handled by one of six branchesof the A1676 Branch Controller.

The crate panel is provided with I/O flat connectors of the Easy BusRemote control and the 48V StatusInput connector, wich allow thePS boards microcontrollers to detectfailures in supply voltages andconsequently to enable PS channelsramping down.

Branch Controller overview

The Branch Controller must be linkedto the EASY crate usually placedin the “ostile area”The A1676A is the interface betweenthe Mainframe and the remote boardsin the Easy crate.The channels of the Easy boardswill operate as channels of the A1676A.

Up to six Easy crates can be controlledby one A1676A.

Mainframe

Easy crate

2x2KW/1x4KW 48Vdc Power source

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 8

CAEN devices – Firmwares overviewCAEN devices – Firmwares overview

Device Firmware Version Issued on Currently used at P5Currently used at P5

Power SupplyPower Supply A4603A4603

1.04 Nov.2009 1.04 Nov.2009 1.041.041.03 Jul. 20091.03 Jul. 20091.02 May 2008 1.02 May 2008 1.01 Dec. 2007 1.01 Dec. 2007

Power SupplyPower Supply A4602A4602

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010

1.03 May 2008 1.03 May 2008 1.031.031.02 Jan. 2008 1.02 Jan. 2008

48 Vdc Converter (MAO)-A3486

2.04 May 2008 2.042.03 Dec. 2007 1.10 Apr. 2007

9

CAEN devices – Firmwares overviewCAEN devices – Firmwares overview

Device Firmware Version Issued on Currently used at P5Currently used at P5

MainframeMainframe SY1527SY1527

3.01.05 Nov. 2009 3.01.05 Nov. 2009 3.01.053.01.053.01.02 Mar. 2008 3.01.02 Mar. 2008 3.01.01 Feb. 20083.01.01 Feb. 20083.01.00 Jan. 20083.01.00 Jan. 20083.00.00 Dec. 20083.00.00 Dec. 20082.01.02 Sep. 20062.01.02 Sep. 20062.01.01 Jul. 20062.01.01 Jul. 20062.01.00 Jul. 20052.01.00 Jul. 20052.00.02 Feb. 20042.00.02 Feb. 2004

Branch ControllerBranch Controller A1676AA1676A

3.01 Dec. 2009 3.022.12 Nov. 20082.11 Jun. 20082.10 May. 20082.09 Sep. 2006

2.04 Jul. 2005

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 10

New features of the FIRMWARES used by PIXEL CommunitiesNew features of the FIRMWARES used by PIXEL Communities

Branch Controller A1676A

In the new Firmware (rel. 3.01) the Vsel bug has now been corrected Bug fixed : Internal memory corruption caused by the use of the Vsel parameters.

Power Supply A4603

According to our request with the new Firmware (rel. 1.04) the H.V. can ramp down to V0. The L.V. unaffected

As a result of these new features will be possible use a signal generated by the LHC and sent by the BRM directly to the pixel caen system ( Vsel input )The A4603s will then ramp down from V1 to V0. The A4603 L.V. Is unaffected.

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 11

Simple and Easy Simple and Easy TEST STANDTEST STAND act to understand and address problems. act to understand and address problems.

We can use the same TEST FACILITYused for the PILOT DETECTOR

Port card + CCU + AOH

Panel + Adapter Board + Flet cable

Test Stand details

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 12

CMS – Pixels DEVICES

To control all the DEVICES used at P5 has been created a link on TWIKIWhere you can find an UPDATE MAP of the Power System Parts installed.

You can find also a MAP of the DEVICES Out of the System.…( Spare Parts.)

Our idea is to improve this very useful tool, extending it to other devices …( FED…FEC…etc. )and adding a history of any inconvenient occurred on that device.

This would help us to TRACE the history of the Channel.…Layer….Sector.

twiki.cern.ch/twiki/bin/view/CMS/PixelCaenSpareComponents

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 14

SUSPECTED MALFUNCTIONS……………..SUSPECTED MALFUNCTIONS……………..

Sometimes happens that a power unit ( A4602, A4603 ) goes bad, so suspectingthat the module has problems, we replace it.

The swapping process fixes the problems, but the replaced unit ….tested at TIF…. is OK

So the problem is solved but still…… we don’t know why.

We had several technical interview with CAEN Engineers to understand in detail how the modules works.HERE ARE THE ANSWER.

_

_

_

The Firmware for A 4602 and A4603 are stored in a flash Eeprom inside a Micro Controllers which handled the module.

There is NO “Firmware Load” at Power ON.

Therefore a corruption of the Firmware can NOT be recovered by a Power Cycle.

Each time a module is removed and re-inserted into the crate,( which is equivalent to power cycle the crate ) there is an integrity check of the Firmware.If the integrity check is not passed, then the module can no longer be accessed, and the Firmware must be reloaded into the Eeprom.

The RESET button on the front panel acts directly on the microcontroller, and performa reset of all channel for that module.

Through Telnet can be performed a so called HW Reset, which will Reset all the modules in one crate.

The Recovery-command through Telnet, resets only the communication. The channels remain on.

_

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 13

Summary

All the activities currently being held at TIFplay a support function mainly for the PIXEL COMMUNITY

Here are been mentioned the most importantOthers activities will arise in the light of the coming CMS - UPGRADE

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 15

Pasquale D’Angelo – INFN Milano - Bicocca8/06/2010 16

FINAL REMARKS and CONCLUSION

For what concern pixels ( Barrel & Forward ) we can said :

- The Power Supply failure rates is extremely low.

-Are already several months that we don’t observe failure on our devices.

My personal opinion:

-This very high efficiency is mainly due to the big effort of everybody- done during the commisioning, and the preliminary test made in the past.-And also thanks to the “CONSTANT & PERMANENT...............- Pixels support activities performed at TIF.