lcls maintenance personnel and procedures paul bellomo

bellomo@slac.stanford.eduOctober 31, 20061

LCLS Maintenance Personnel and Procedures

Paul Bellomo

bellomo@slac.stanford.eduOctober 31, 20062

Topics

Scope and Perspective

Why we know we are ready

Department descriptions

ARTEMIS

The maintenance process

Examples

bellomo@slac.stanford.eduOctober 31, 20063

Scope and Perspective

Accelerator equipment

New electrical, mechanical, electro-mechanical and electronic equipment from the injector gun through the end of the tune-up dump TD-11.

New control room in Building 5

New substations K10B, S520, MCC S20, panel-boards

Operations Directorate

12 Departments, only 5 directly involved

Controls, CEF, Klystron/Microwave, Mechanical Fabrication, Power Conversion

Perspective

Maintenance departments

bellomo@slac.stanford.eduOctober 31, 20064

Why We Know We Are ReadyAll departments

LCLS equipment possesses some new features, but is not totally unlike the equipment that SLAC currently services and maintains

Designers are the persons/groups that will maintain the equipment

Maintenance personnel are qualified and have been involved in the LCLS design from the start. Training requirements are modest and provided by LCLS, Department, or outside agency

Staff quantity is adequate to handle increased LCLS workload. The documentation (detailed later) needed to support the systems exist

Procedures (detailed later) needed to maintain the systems exist

bellomo@slac.stanford.eduOctober 31, 20065

Department Descriptions

Department Head – Hamid Shoaee, staff of 75

Software and hardware sections

Software Section

Write low-level applications, EPICS drivers, displays

Maintain MATLAB infrastructure (licenses), ensure compatibility with all control room platforms

Provide network hardware and software

Provide control room infrastructure (alarms, displays, servers and workstations)

Development and maintenance are the same people

bellomo@slac.stanford.eduOctober 31, 20066

Department Description - Controls (Continued)

Hardware SectionBeam Containment System (BCS)

Prevents radiation from 'escaping' the shielding enclosure

BSOICs, BTMs, LIONS, toroids and current monitors

Machine Protection System (MPS)Protect equipment from beam damage

VME processor-based protection ion chambers (PICs), RTDs, digital monitors

Personnel Protection System (PPS)Protect personnel from radiation exposure, laser and electrical hazards

PLC driven systems

Vacuum - PLC based gages, valves, pumps and interlocks

Movers - Tables, mirrors, cameras, wire scanners

bellomo@slac.stanford.eduOctober 31, 20067

Department Head – John Weisend, staff > 150

CEF Operations - Bernie Romero, staff of 28

Electrical maintenance of substations and electrical distribution equipment

Beam-line manifolds and plumbing

Documentation

One-line diagrams

Panel-board schedules

Manufacturer drawings and instruction manuals

Beam line assembly and layout drawings

bellomo@slac.stanford.eduOctober 31, 20068

Klystron/Microwave DepartmentDepartment Head - Chris Pearson, staff of 80

Microwave Engineering and Maintenance/AMRF, staff of 15

Positron Source, Damping Rings, LINAC, LCLS LINAC and SLC RF systems and 2856MHz pulsed klystrons

PEP (HER and LER) and SPEAR 3 RF systems and 1.2 MW, 476MHz CW klystrons

Documentation

Rack front views and Captar rack profile drawings

Electrical interconnect diagrams, cable database

Electronic chassis diagrams

Manufacturer drawings and instruction manuals

ELPs, EWPs, bench test, hi-pot procedures

bellomo@slac.stanford.eduOctober 31, 20069

Mechanical Fabrication Department (MFD)

Department Head – Karen Fant, staff of 80

Mechanical fabrication/maintenance – no design

Vacuum gages and pumps on the vacuum side, pneumatic valves, beam pipe and other components designed by others (LCLS staff)

Magnet cooling water hoses, disassemble magnets to find magnet problems and ground faults

On-call support of Operations

Documentation

Beam line assembly and layout drawings

Vacuum pump-down procedures

bellomo@slac.stanford.eduOctober 31, 200610

Department Head – Paul Bellomo, staff of 75

DC magnet power systems and RF modulators

Cables, cable tray and raceways

Dedicated maintenance staff of 31 people, 24/7, 3-shift operation

Participate in power system commissioning

Documentation

Rack front views and Captar rack profile drawings

Electrical interconnect diagrams, cable database

Electronic chassis diagrams

Manufacturer drawings and instruction manuals

ELPs, EWPs, bench test, hi-pot procedures

bellomo@slac.stanford.eduOctober 31, 200611

ARTEMIS

Accelerator REMEDY Trouble Entry & Maintenance Information System

Web-based problem reporting, work scheduling and work status program

Report and track both hardware and software problems

Incorporates ISMS core functions

Five statesNew Entry – report problems or schedule maintenance

In Progress – Jobs in progress or have more work to do

Scheduled Jobs – Input from the New Entry state

Review To Close – All known work completed, ready for management review

Closed – trouble report entered into archive

bellomo@slac.stanford.eduOctober 31, 200612

ARTEMIS

bellomo@slac.stanford.eduOctober 31, 200613

The Process

bellomo@slac.stanford.eduOctober 31, 200614

Example S20 Power Distribution Layout

bellomo@slac.stanford.eduOctober 31, 200615

Example One-Line Diagram

Excerpt from ID-950-075-03

Substation K10B 13.8kV:480V to S20 MCC to 6 Power Supply Racks

Portion of S20 MCC shown

bellomo@slac.stanford.eduOctober 31, 200616

Example Rack/Equipment Profile

bellomo@slac.stanford.eduOctober 31, 200617

Example Electrical Interconnect (EI/ELP) Diagram

bellomo@slac.stanford.eduOctober 31, 200618

The PCD Process – Example 1

LCLS bend magnet system fails unexpectedly at 3AM. The indications are that a power supply has failed.

LCLS Operations files an ARTEMIS and also notifies the PCD owl shift supervisor/maintenance manager (MM). Ops classifies the problem as critical, because the machine is down without this particular magnet.

MM evaluates the system, reviews past ARTEMIS reports and chooses appropriate technicians to troubleshoot the problem.

The MM, the technicians then assess the situation, from the technical, logistics and safety standpoints. The system is 480VAC input, 40VDC output.

Appropriate test equipment and PPE (Category 0 with Class 0 gloves)

The group notifies Ops that it is responding to the call. Ops turns the system over to PCD. PCD applies LOTO. ARTEMIS upgraded to “in process”

bellomo@slac.stanford.eduOctober 31, 200619

The PCD Process – Example 1 Continued

Work progresses power system will not turn on. After several hours, the solution is not readily apparent.

Ops escalates the problem to the designated system expert or Department Head. After 3 hours without resolution the Department Head and LCLS manager are notified. Work continues

The technical expert finds and solves a short in the power supply rectifier. Parts are field-replaced.

Ops is notified that the system is running. The system is returned to Ops.

Technicians file “review to close” ARTEMIS report, file e-log detailing the problem, the solution and parts used.

The PCD MM checks the parts inventory and if below a pre-determined critical level, orders replacement parts.

PCD closes the job internally

bellomo@slac.stanford.eduOctober 31, 200620

The PCD/MFD Process – Example 2

A ground fault is discovered in a magnet system. Ops files ARTEMIS and notifies PCD.

The fault keeps the system off. It must be repaired immediately

PCD locks out the system. They troubleshoot the system and determine that the ground fault is in the magnet. PCD asks Ops to notify MFD. The ARTEMIS is annotated and responsibility re-assigned.

PCD replaces personal LOTO with group LOTO. The PCD MM is now responsible for the system.

MFD is called. MFD responds and confers with PCD. PCD demonstrates that the power supply cannot be turned on.

MFD apply personal locks over the PCD group locks. MFD apply other LOTO, verify magnet ground fault existence and perform work necessary

to remove the ground fault.

bellomo@slac.stanford.eduOctober 31, 200621

The PCD/MFD Process - Example 2 Continued

The ground fault is found and repaired. The magnet is re-assembled and hi- potted.

MFD personnel remove their locks. PCD is re-called.

PCD replaces Group with personal locks. PCD hi-pots the system, remove their locks, turn the system on and release it to Ops.

PCD technicians file a “review to close” the ARTEMIS and file e-log entries

PCD MM conducts safety de-briefing and closes the job

bellomo@slac.stanford.eduOctober 31, 200622

Last Slide

The SLAC maintenance groups are ready and eager to safely and professionally support the LCLS project