CMS ME CSC HV systemCMS ME CSC HV systemproduction readiness reviewproduction readiness review

Alex MadorskyUniversity of Florida

July 2004, PNPI 2

UF/PNPI HV system architectureUF/PNPI HV system architecture

Card 72

Counting Room Detector Area

Long Distance HV Cables ~ 100 m long

SHV Connectors on both ends

Primary HV Supply

Multiwire HV Cable ~ 12 m long

LEMO REDEL Connectors on both ends

Three Main Units: Primary HV Supply, Master Distribution Card, Remote Distribution Card One Primary HV Supply per up to 9 Master Distribution Cards Master Distribution Card: 1 Input, 8 Independent Outputs One Master Distribution Card per 8 Remote Distribution Cards Remote Distribution Card Type 1: 1 Input, 30 Independent Outputs

(One Card per one ME 23/2 Chamber) Remote Distribution Card Type 2: 1 Input, 36 Independent Outputs

(One Card per two ME1 Chambers)

Card 1

Card 9

Card 1

Master Distribution

Card

Remote Distribution

Card Type 1

Remote Distribution

Card Type 2

Multiwire HV cables, 100 m, one per 18 distribution boards

•Primary HV power supplies: off the shelf, one per distribution rack, 8 total

•Master board: One output per distribution board. Regulates voltage 0-4KV (VMAX), measures current on each output.

•Remote Distribution board: powers one large or two small chambers (36 outputs max). Regulates voltage 1KV down from VMAX, measures current on each output. Each output can be disconnected from HV if necessary.

•Total number of individually controlled HV channels: 8856

Master boards Distribution

board 36

Distribution board 30

July 2004, PNPI 3

HV system grounding diagramHV system grounding diagram

~ 100 m

Opticalcoupler

HV return wire

Notes:

- HV return wire connects to building ground only through chamber shell.

- In all modules there are safety connections from HV return wire to building ground

- Rack enclosures, chamber shells and cable shields formFaraday cage that extends from the primary HV power supply down to chambers, and completely encloses all HV leads, control cables and HV return wire.

- There is individual HV return wire for eachdistribution rack, to break ground loops between disks and racks.

CMS ME CSC HV system grounding

Opticalcoupler

Z load(multiple)

Primary HV power supply (floating output)

Distributionboard

Chamber

30 or 18 HVoutputs

To other distribution boardsin the rackMaster board

Control room Disk

To other master boards in the same rack

HV rack

Chamber shell connects to disk

0-4KVregulators

1KV downregulators

15 m (average) cable

Building ground

Board / module / rack enclosure

Safety connection. Does not allow the HVreturn wire float more than a few voltsfrom the building ground

HV distribution rack

Control computer

July 2004, PNPI 4

Distribution boardsDistribution boards

Two types of distribution boards:

• 36 channels (two small chambers) – 139 boards

• 30 channels (one large chamber) – 159 boards

• Parts procurement, SMT assembly: UF

• Final assembly and tests: PNPI

Each channel:

• Controls voltage 1 KV down from input voltage

• Measures output current

Control connector

Power connectors

HV input

HV outputs

July 2004, PNPI 5

Master boardMaster board

Master board:

• Each output drives one distribution board

• Controls HV from 0 to 4KV on 8 outputs

• Measures current on each output

• Provides low negative voltage for fuse control circuit

Input HV

8 output HV connectors

Control and LV power connector

• 46 boards to produce

• Parts procurement, SMT assembly: UF

• Final assembly and tests: PNPI

July 2004, PNPI 6

Host cardHost card

Host card:

• Sits in PCI bus of the control computer

• Able to control up to 16 Distribution or Master boards

Xilinx JTAG connector, for FPGA configuration

Control cable connector.

PCI connector

• 20 boards to produce

• Parts procurement, assembly: UF

• Tests: PNPI

July 2004, PNPI 7

Scope of the productionScope of the production

Component Quantity (ME4/1 included)

Pre-production quantity

UF PNPI

1KV regulator boards 8856 + 1216 spares = 10,072

360 X X

4KV regulator boards 304 + 96 spares = 400 16 X X

Relay boards 304 + 96 spares = 400 16 X X

Distribution boards – 30 channels (DB-30) 144 + 15 spares = 159 0 X X

Distribution boards – 36 channels (DB-36) 126 + 13 spares = 139 10 X X

Master boards (MB) 38 + 8 spares = 46 2 X X

Host Cards 28 + 3 spares = 31 2 X X

Distribution crates 30 + 3 spares = 33 1 X

Master crates 8 + 1 spare = 9 1 X

Patch-panels 16 + 2 spares = 18 0 X

Long HV cables 18 + 2 spares = 20 1 X

Long control cables 20 + 3 spares = 23 0 X

Miscellaneous and off-the-shelf components As required + 10% spares As required X

Total number of individually controlled HV channels: 8856 (spares not included, ME4/1 included)

Production items (fan-out cables not included):

July 2004, PNPI 8

Overall production organizationOverall production organization

Project leader Project engineer Chamber group liaison

Guenakh Mitselmakher Alex Madorsky Andrey Korytov

UF Production coordinator

Alex Madorsky

See UF production diagram on page 10

PNPI Production coordinator

Victor Golovtsov

See PNPI production diagram in V.Golovtsov’s presentation

July 2004, PNPI 9

UF responsibilityUF responsibility

Development and production managementPre-production prototype construction and testingTest stands constructionTest procedures verification, instructionsOff-the-shelf components procurementBare PCBs manufacturingAutomated SMT assemblyUS labor and components contingency

July 2004, PNPI 10

UF production organizationUF production organization

Production coordinator

Alex Madorsky

Technician:

Arkadi Averbukh

Student assistants

• Quality assurance (tests)

• Repairs

• Database maintenance

• Shipping and handling

Distributors

Assembly houses

Machine shops

Software engineer:

Victor Barashko

Test software maintenance

July 2004, PNPI 11

UF Quality AssuranceUF Quality AssuranceBoards tested:

Distribution boardsMaster boards

Visual test:

Presence of all componentsValuesOrientation

Electrical test:

Control bus interfaceDACADCs ID register

Results of all tests are inserted into production database

July 2004, PNPI 12

UF production statusUF production status

All parts for Distribution, Master and Host boards are purchased PNPI received all parts and equipment necessary to begin production Remaining test stand equipment and all production components shipped to

PNPI Distribution boards:

68 boards SMT-assembled 68 SMT-assembled boards shipped to PNPI

Master boards: 20 boards SMT-assembled 4 SMT-assembled boards shipped to PNPI

Host cards: all boards assembled

Distribution crate: prototype assembled and tested Master crate: prototype assembled and tested Test stand software development finished for Distribution and Master boards

Binary release and step-by-step instructions are available via web-page (along with other information):

http://www.phys.ufl.edu/cms/emu/hv/ Access to source code via CVS Production database similar to chamber database is kept in text files

July 2004, PNPI 13

PNPI responsibilityPNPI responsibility

Simple mechanical components manufacturedPre-production and production manual assemblyPre-production and production testingPNPI labor and space contingency

July 2004, PNPI 14

PNPI test stand typesPNPI test stand typesRegulator board test stand

Uncoated boards at 1KV Coated boards at 4KV

Assembled Distribution and Master board test stand Full parametric testing of assembled boards Calibration

Voltage setting Voltage measurement Current measurement

Long term test stand Burn-in Parameter verification after burn-in

Total 11 tests for each board. Results of all tests are inserted into production database.

Most of the test stand components provided by UF.More details on test stands in the presentation by V.Golovtsov.

July 2004, PNPI 15

PNPI production statusPNPI production status

Completely assembled and tested:10 Distribution boards 2 Master boards

PNPI production organization details to be shown in the presentation by Victor Golovtsov.

July 2004, PNPI 16

Additional documentsAdditional documents

Filename (hyperlink). To open files, start presentation (F5) and click hyperlinks.

Description

DS001.txt Database file for Distribution board #001 (see example on page 17)

T01_M001.log.txt Log file for test #01 for Master board #01 (see example on page 18)

uf_visual_instructions.doc UF visual test instructions

uf_electrical_instructions.doc UF electrical test instructions

HV_Tests_User_Manual.doc PNPI test instructions (in Russian)

July 2004, PNPI 17

Database file for Distribution board (fragment)Database file for Distribution board (fragment)

File name: DS001

Date : April 21 2004Time : 10:00Name : L.SergeevType : Assembly_fullComments : R00001,R00002,R00003,R00004,R00005,R00006,R00007,R00008,R00009,R00010,R00011,R00012,R00013,R00014,R00015,R00016,R00017,R00018,R00019,R00020,R00021,R00022,R00023,R00024,R00025,R00026,R00027,R00028,R00029,R00030,R00031,R00032,R00033,R00034,R00035,R00036,V001

Date : April 22 2004Time : 10:00Name : N.IsaevType : TestTest : 1Result : FailComment : Bad Test1

Date : April 22 2004Time : 10:00Name : S.VolkovType : DiagnosisReference : U32Partnumber : ADS7818EBComment : Bad chip

Date : April 22 2004Time : 10:00Name : S.VolkovType : SolutionReference : U32Partnumber : ADS7818EBComment : Replace chip

• All records are in fixed format• Records are easy to import into production database• Production database is being developed

July 2004, PNPI 18

Test log file example (fragment)Test log file example (fragment)

File name: T01_M001.log

========================================================

Test1 results 04-02-2004 18:33:20Board ID001 30 channelsHV Tests Software Ver.1.0

+5V 2493 - O'k-5V 1306 - O'kInterlock - On

Point1- No data

Point2Voltages: 552 549 547 553 549 553 552 550 551 550 552 553 551 550 551 550 548 546 536 545 551 549 542 549 552 551 553 551 547 547Channels: 1 4 6 7 9 11 12 13 15 21 25 26 27 28Out of range 530 +/- 20

Currents: 2506 2514 2555 2552 2539 2546 2569 2539 2540 2517 2534 2537 2475 2513 2532 2520 2490 2558 2532 2561 2536 2524 2603 2461 2511 2540 2509 2545 2512 2551

Point3Voltages: 801 798 797 801 798 801 800 798 801 798 800 802 799 798 799 799 797 795 785 794 799 797 791 795 800 798 801 800 796 795Channels: 1 4 6 9 12 27Out of range 780 +/- 20

Currents: 2502 2504 2544 2540 2531 2535 2558 2533 2537 2511 2530 2533 2473 2511 2529 2521 2489 2558 2531 2561 2535 2521 2599 2460 2511 2540 2509 2544 2512 2550

• Each test generates a log file• All relevant information is inserted into log file• Log file is appended if this test is run multiple times

July 2004, PNPI 19

Pre-production system testsPre-production system tests

Minimal system segmentOne primary power supplyOne Master board8 Distribution boards288 channels8 chambers connected to Distribution boardsOne of the chambers is connected to FAST DAQ

Parametric testsCheck if the system matches all specificationsSee results on few next slides

Tests with chamberGround & Power configuration as close as possible to real

detectorResults are in the presentation by A. Denisov

July 2004, PNPI 20

Pre-production system testsPre-production system testsWiring diagramWiring diagram

Primary HV source

Masterboard

Distributionboards

FASTDAQ

SHV cables

Regular HVfanout cables(Kerpen)

July 2004, PNPI 21

Pre-production system testsPre-production system testsGrounding diagramGrounding diagram

Opticalcoupler

Control and HV cable length ~ 3 m

Opticalcoupler

HV return wire

CMS ME CSC HV system groundingPNPI test setup

Notes:

- HV return wire connects to building ground only through chamber shell.

- In all modules there are safety connections from HV return wire to building ground

- There is individual HV return wire for eachdistribution rack, to break ground loops between disks and racks.

Z load(multiple)

ChamberDistributionboard

Primary HV power supply (floating output)

30 or 18 HVoutputs

To other distribution boardsin the rackMaster board

DiskSimulation

Control roomSimulation

HV rack

To other master boards in the same rack

0-4KVregulators

Regular 15 m fan-out cable

1KV downregulators

Building ground

Safety connection. Does not allow the HVreturn wire float more than a few voltsfrom the building ground

Board / module / rack enclosure

HV distribution rack

Control computer

8 SHV cables. HV return wire is connected to the braid of these cables

Ripple generator

Compare with page 3

July 2004, PNPI 22

Test of specification requirementsTest of specification requirements

Parameter Requred Provided

Maximum output voltage, Vmax 4000V 4000V

Voltage regulation, each channel Vmax-500V Vmax-1000V

Voltage regulation resolution, each channel

<= 50V ~1V

Channel to channel voltage difference 20V max 17V

Voltage output Floating Yes

Maximum output current per channel 100 uA 100 uA

Maximum output current of the board 40uA*number of outputs

Yes

Individual output trip speed <= 1s <=1s

July 2004, PNPI 23

Test of specification requirementsTest of specification requirementsParameter Requred Provided

Trip level, programmable 1 to 100 uA 1 to 100uA

Trip level setting resolution 1uA 1uA

Hardwired maximum current per channel

100uA ~350uA

Mutual influence of channels (during trips)

No trips on other channels

OK

Voltage measurement resolution, per output

10V 1V

Current measurement resolution 100nA 100nA, 10nA possible*

Channel to channel measured current difference

10% of measured value

10% of measured value

Rate of voltage change, programmable

2 to 50 V/s 2 to 1000 V/s

July 2004, PNPI 24

Test of specification requirementsTest of specification requirementsTest result conclusions:

Pre-production system works fine in normal conditions. In the conditions of very high humidity (above 80%) increased

leakages were observed in some channels. Steps are taken to improve the system performance in high humidity conditions:

Coating process is modified to provide better coating quality

Triple coating Slots are made in critical high-voltage gaps on PCBs Quality assurance will include humidity chamber tests

Maximum current per channel without software trip: about 350 uA. Not a problem, according to Andrey Korytov.

July 2004, PNPI 25

Pre-production system tests with chambersPre-production system tests with chambers

Control room simulation

HV Distributionrack simulation

Disk with chamberssimulation

FAST site

July 2004, PNPI 26

ScheduleScheduleMilestone Details

DB-XX – Distribution board with XX outputsMB – Master board

Date Respon-sible

Pre-production system finished and tested 10 DB-36 + 2 MB (360 channels) September 25 ’04 UF, PNPI

External Production Readiness Review September 30 ’04 UF, PNPI, review committee

Production start October 4 ’04 UF, PNPI

Shipment of SMT-assembled boards to PNPI, pre-tested 76 DB23 MB

October 10 ’04 UF

First shipment to CERN 18 DB-368 DB-304 MB

November 30 ’04 PNPI

Shipment of SMT-assembled boards to PNPI, pre-tested 76 DB23 MB

January 15 ‘05 UF

25% of the boards shipped to CERN 48 DB-3012 DB-3612 MB

January 30 ‘05 PNPI

50% of the boards shipped to CERN 48 DB-3012 DB-3612 MB

Apr 15 ’05 PNPI

Shipment of SMT-assembled boards to PNPI, pre-tested 90 DB March 15 ’05 UF

75% of the boards shipped to CERN 48 DB-3012 DB-3612 MB

July 1 ’05 PNPI

100% of the boards shipped to CERNEnd of production

6 DB-3084 DB-366 MB

September 1 ’05 PNPI

July 2004, PNPI 27

Development in progressDevelopment in progressMiscellaneous components are being developed in UFNot critical for production

Patch panels for HV and control cables See “Scope of production” slide for quantities Finish by January ‘05

Cables from Master boards to Distribution boards Length to be defined, about 100 meters HV cables

KERPEN halogen-free cables, same as Distribution-to-chamber cables One prototype is assembled (120 m) Production in CERN

Control cables Amphenol Spectra-Strip halogen-free round-n-flat

See “Scope of production” slide for quantities Finish by January ‘05

Pending approval from “Cable committee” chaired by Gerard Faber

Internal Distribution and Master rack cables HV cables – coaxial with SHV connectors, ~3 m (800 pcs) Control cables – Flat twisted pair, ~3 m (60 pcs) Finish by January ‘05

July 2004, PNPI 28

Low Voltage Requirements for Low Voltage Requirements for Remote Distribution CardsRemote Distribution Cards

Parameter Min Max

Positive voltage 7 V 8 V

Negative voltage -8 V -7 V

Current on both channels

300 mA

Power per distribution board

4.2 W 4.8 W

Ripple/noise 100 mV

• Low voltage power will be provided by CMS AC/DC LV system

July 2004, PNPI 29

SoftwareSoftware

Linux driver for HostCard (V. Barashko, UF) Multiple Host Cards Multiple Distribution and Master boards Main functionality finished

DIM server (V. Barashko, UF) Works with Linux driver Compatible with CERN slow control software (PVSS) Main functionality finished

Chamber database (V. Barashko, UF) In development

Operator interface program (V. Barashko, UF) Simple text-mode version ready GUI conceptual design underway

PVSS shell for slow control (Marina Giunta (CERN), Valery Sytnik (UC Riverside)) Specification is in preparation in UF

July 2004, PNPI 30

HV system software structureHV system software structure

Host cards

PCI driver

Slave DIM server

Control computer #2

Host cards

PCI driver

Main DIM server

Control computer #1

Host cards

PCI driver

Slave DIM server

Control computer #3

Chamberdatabase

Operator’s GUI PVSS application

July 2004, PNPI 31

DIM server command exampleDIM server command example

Field name Example Note

Chamber name ME2/3-12 Real chamber name, as marked

Segment number 3 Segment number in the chamber

Action 23 Action code (set voltage, set trip current, set ramp up speed, etc.)

Data field … Action-specific data in high-level units (voltage in volts, current in amperes, etc.).

July 2004, PNPI 32

HV system software reliabilityHV system software reliabilityCritical pieces of software:

PCI driver DIM server

Critical software-dependent functionality: Interlock Trip

Software may crash for the variety of reasons: Bugs Power failure Computer failure Operator mistake …

How to detect any possible software crush: Hardware “watchdog” Such devices are available for about $130

What to do on software crush: Kill primary HV power supply / supplies that are controlled by the crashed computer Reset computer by turning power off and on

July 2004, PNPI 33

Operator shell screenshotOperator shell screenshotPCI:0:6, Card:0 dataslot:0, ioaddr:0xE2D8E000Module:0xd: state:OFF, type:RDB 36, ID:0, Chans:36, ILock:0Ch# Vmon Imon Vset Vcur RmpUp RmpDn Vmax Imax Vtrip Itrip Relay Fuse State Status V mkA V V V V V mkA V mkA---------------------------------------------------------------------------------------01| 0 0.0 2752 0 18 18 4000 100 0 0 0 0 OFF OFF02| 0 0.0 2745 0 18 18 4000 100 0 0 0 0 OFF OFF03| 0 0.0 2745 0 18 18 4000 100 0 0 0 0 OFF OFF04| 0 0.0 2761 0 18 18 4000 100 0 0 0 0 OFF OFF05| 0 0.0 2757 0 18 18 4000 100 0 0 0 0 OFF OFF06| 0 0.0 2756 0 18 18 4000 100 0 0 0 0 OFF OFF07| 0 0.0 2756 0 18 18 4000 100 0 0 0 0 OFF OFF08| 0 0.0 2749 0 18 18 4000 100 0 0 0 0 OFF OFF09| 0 0.0 2754 0 18 18 4000 100 0 0 0 0 OFF OFF10| 0 0.0 2735 0 18 18 4000 100 0 0 0 0 OFF OFF11| 0 0.0 2748 0 18 18 4000 100 0 0 0 0 OFF OFF12| 0 0.0 2748 0 18 18 4000 100 0 0 0 0 OFF OFF13| 0 0.0 2742 0 18 18 4000 100 0 0 0 0 OFF OFF14| 0 0.0 2756 0 18 18 4000 100 0 0 0 0 OFF OFF15| 0 0.0 2761 0 18 18 4000 100 0 0 0 0 OFF OFF16| 0 0.0 2739 0 18 18 4000 100 0 0 0 0 OFF OFF17| 0 0.0 2748 0 18 18 4000 100 0 0 0 0 OFF OFF18| 0 0.0 2741 0 18 18 4000 100 0 0 0 0 OFF OFF

July 2004, PNPI 34

Distribution boardDistribution board

• 36 channels (two small chambers) – 139 boards

• 30 channels (one large chamber) – 159 boards

July 2004, PNPI 35

Master BoardMaster Board

• 46 boards to produce

July 2004, PNPI 36

Host CardHost Card

• 20 boards to produce

July 2004, PNPI 37

1KV regulator1KV regulator

• 10,072 boards to produce

July 2004, PNPI 38

4 KV regulator4 KV regulator

• 400 boards to produce

July 2004, PNPI 39

HV Relay boardHV Relay board

• 400 boards to produce

July 2004, PNPI 40

Distribution crateDistribution crate

• Made from standard ELMA parts, except side panels (custom length)• Houses 9 distribution boards• No backplane

• 30 crates to make• Parts procurement: UF• Assembly: CERN

July 2004, PNPI 41

HV Distribution CrateHV Distribution Crate

• 30 crates to produce

July 2004, PNPI 42

Distribution RackDistribution Rack

Fan unit & heat exchanger

Distribution crate

Distribution boards

HV and control cables patch panel

Output HV cables to chambers

Need from CMS:

1. Racks

2. Fan units & heat exchangers

3. Strain reliefs

4. Space in front and behind the racks

5. Low Voltage power for distribution boards

July 2004, PNPI 43

Pre-production results Pre-production results Problem Proposed solution Person

responsible for correction

Some wires in HV tails are ripped out

HV wire routing must be done with routing fixture, allowing some slack

Volkov, Astashin (PNPI)

At least one wire in HV tail is fixed in-house

Do not try to repair HV tails – it is a complicated procedure. Please use the other one. Damaged HV tails must be shipped to UF for repairs.

Volkov, Astashin (PNPI)

Incorrect types of screws are used in mechanical assembly

Use screws in accordance with the assembly drawing.

Volkov, Astashin (PNPI)

Holes in side bars are not in-line, which makes installing them difficult

Use conductor to drill holes in side bars Astashin (PNPI)

Side bars don’t have any finish Drawing requires anodizing, but we can consider other finish, such as brushing

Madorsky (UF), Astashin (PNPI)

Thread in some holes of the side bars is not deep enough

Make thread in accordance with drawing Astashin (PNPI)

Material for side bars is too soft (?)

Purchase material locally Golovtsov, Astashin (PNPI)UF responsible for payment

July 2004, PNPI 44

Pre-production results Pre-production results

Problem Proposed solution Person responsible for corrections

Nylon stand-offs are not tightened well enough

Tighten nylon stand-offs with torque-controlled screwdriver

Volkov, Astashin (PNPI)

Plastic handle on the front panel is attached upside-down

Attach plastic handle in accordance with drawing

Volkov, Astashin (PNPI)

On Master boards, LEMO connectors don’t have heat shrink insulator

Put heat shrink insulator in accordance with drawing.

Volkov, Astashin (PNPI)

Jumpers on the boards are not set up as required

Add the “shipping” jumper positions to the test instructions (UF), follow that instructions (PNPI)

Madorsky (UF), Volkov (PNPI)

GND socket is connected to the board with wire

Wire is not shown on assembly drawing, since GND socket is already in contact with board’s case.

Volkov (PNPI)

The spring in some fuses does not jump out as expected when fuse is blown

Modification of fuse soldering procedure required. Manual cleaning required to remove coating bridges.

Volkov, Astashin (PNPI)

DAC amplifier “rings” on some boards

3pF capacitor is required in feedback. Madorsky (UF), Volkov (PNPI)

July 2004, PNPI 45

Pre-production resultsPre-production results

Procedure modification Person responsible for corrections

Distribution board: R62, JP52 – remove Volkov (PNPI)

Master board: R52 - remove Volkov (PNPI)

Remove ventilation holes on Side bars (Distribution and Master boards) Madorsky (UF), Astashin (PNPI)

Software modification for Burn-in test Madorsky, Barashko (UF)

Additional equipment Price Purchased by

Payment source

Ultrasonic cleaner $1.5K PNPI UF

Drying hood $2.5K PNPI UF

Thermostat $1.2K PNPI UF

Assembly workstation $1.2K PNPI UF

Computer for additional test stand $0.8K PNPI UF

Aluminum for side bars $1.5K PNPI UF

Humidity chamber $16K UF UF