improving the reliability of accelerator magnets learning from our failures
DESCRIPTION
Improving the Reliability of Accelerator Magnets Learning from our failures. Presented at the Accelerator Reliability Workshop, Cape Town, 14 th April 2011 Cherrill M. Spencer SLAC National Accelerator Laboratory, USA. Overview of this talk. Decision to hold a magnet reliability survey - PowerPoint PPT PresentationTRANSCRIPT
Improving the Reliability of Accelerator MagnetsLearning from our failures
Presented at the Accelerator Reliability Workshop,
Cape Town, 14th April 2011 Cherrill M. Spencer
SLAC National Accelerator Laboratory, USA
Overview of this talk• Decision to hold a magnet reliability survey• Description of on-line survey• Example questions• Who took the survey• Kinds of failures magnets suffer from- ranked• Correlate materials & practices with failures• Advice based on survey responses• Continue as open discussion • Raffle for bottle of SA wine- for survey takers
Improving Magnet Reliability-Spencer 2
Decision to hold an on-line magnet reliability survey-to date a 12 year story
• Anticipating designing and building a 20 Km long e+e- linear collider (ILC) with over 13,000 magnets of about 100 different styles:– Estimated the MTBF & MTTR of typical SLAC magnets
from historical records– Calculated that to reach desired availability could not
continue to make magnets in same way– Did a Failure Modes & Effects Analysis (FMEA) of a
typical water cooled electromagnet and saw there were MANY ways such magnets could fail, at all life stages
– Decided to consult with & learn from other magnet engineers at other particle accelerators
Decision to hold an on-line magnet reliability survey- could learn from each others’ problems
• Through interacting with magnet engineers from other labs at PAC and Magnet Technology conferences realized would be useful to all of us to– Collect detailed technical information about
electromagnets in use at accelerators all over the world– Find out what materials and fabrication techniques are
being used– How these magnets are operated and the ways they fail
• Do this through a web-based survey, March 2011– Send out an e-mail to over 150 accelerator engineers &
physicists inviting them, or their magnet colleagues, to fill in the survey for their institution’s magnets
Improving Magnet Reliability-Spencer 4
Goals of the web-based survey- a work in progress
• Use a web-based survey program available to all countries (chose “Survey Monkey”)
• Analyze the survey data to look for correlations amongst failure modes & rates, fabrication materials & techniques, operating conditions and maintenance practices
• Goals : to identify the• more reliable materials (e.g. insulating tapes, epoxies, LCW
hoses) ,• more reliable fabrication techniques (e.g. coil winding methods)• more reliable operating conditions (e.g. properties of the LCW)• more reliable maintenance practices• seek other experts’ opinions and advice on these topics
• Publish the analysis results for all to learn from Improving Magnet Reliability-Spencer 5
Summary description of the survey• Total of 64 questions, of which 9 had to be answered• 6 identifying questions, e.g. institution name,
accelerator/beam line name, age of magnets• 12 questions about magnet design standards in
effect when these magnets were designed• 15 questions about materials used & associated
problems, e.g. epoxy resin & fillers; cracks in potted coils• 4 questions about types & frequencies of failures • 21 questions about their LCW system supplying LCW• 6 questions about other failure types & their advice
Improving Magnet Reliability-Spencer 6
Example of material used question
Improving Magnet Reliability-Spencer 7
Many questions had extra windows for free-form answers that didn’t fit with the given list of answers
Example of material question plus problem and their solution info
Improving Magnet Reliability-Spencer 8
List of countries represented in the 28 survey takers. Some institutions answered for multiple machines
Countries where accelerators situated No. of Accelerators No. of InstitutionsAustralia 1 1
Canada 3 3 China 1 1 France 2 2 Germany 2 2 Japan 2 1 Russia 1 1 South Africa 1 1 Sweden 1 1 Switzerland 3 1 United Kingdom 1 1 USA 10 6TOTALS 12 countries 28 accelerators 21 institutions
Improving Magnet Reliability-Spencer 9
Main uses of these 28 machines
Main use of the particle beams Number• High Energy Particle Physics Research: 10• Synchrotron Radiation Light Source: 7 • Nuclear Physics Research
6• Proton Therapy & Radioisotope Prod. 4 • Educational
1
Improving Magnet Reliability-Spencer 10
Surveys completed for these age ranges of sets of magnets (28 surveys)
Improving Magnet Reliability-Spencer 11
3.6%
28.6%
46.4%
21.4%
SURVEY COMPLETED FOR MAGNETS OF THIS AGE
More than 50 years old30-50 years old10 to 29 years old
No. 1 813 6
Surveys completed for magnets in 3 different running modes
Improving Magnet Reliability-Spencer 12
67.9%
28.6%
3.6%
Choose the mode the magnets run in
DCRamping up and down all the timePulsed
Number 19
8
1
Nine common magnet failure types: rank how frequently your magnets suffer each type
Improving Magnet Reliability-Spencer 13
Failure Type 1st 2nd 3rd 4th 5th 6th 7th 8th 9th Survey Count
Water leaks at braze joints 4 6 4 2 3 0 0 1 5 25Water leaks from LCW hoses and their fittings 14 4 2 1 0 0 0 0 7 28Water leaks from copper conductor 0 3 0 0 4 2 2 1 11 23Insulation problems leading to shorted turns 0 2 3 2 1 3 3 3 5 22Insulation problems leading to ground faults 1 2 1 4 3 3 1 1 7 23Power connections: e.g. cable lug to terminal 3 6 3 1 0 3 1 1 5 23Human error: e.g. LCW hoses hooked up incorrectly 3 4 5 2 2 1 0 1 4 22Blocked water passages (e.g. from CuO) 2 2 5 2 0 2 2 0 8 23Failures caused by poor design 0 0 1 2 0 2 0 1 13 19
Rank your magnets by how frequently they suffer each failure type
Based on my SLAC experiences I categorized nine types of magnet failures.Survey taker had to find out how often their magnets suffered such failures and then RANK them for their frequency. Chose 1st for the failure type that happened the most over past 4 years, 2nd for the second-most etc. Chose “9th” for all remaining types they’ve had at least once in past 4 years.
Another view of failures ranked by frequency data
Improving Magnet Reliability-Spencer 14
1st rank2nd rank
3rd rank0
2
4
6
8
10
12
14
Num
ber o
f sur
veys
1st rank
2nd rank
3rd rank
%surveys mention this type
89.29100.0082.1478.5782.1482.1478.5782.1467.86
Most labs have <5 magnet failures a year.
Analyze survey data associated with LCW hose & fittings failures. Part 1
MOST POPULAR FAILURE TYPE:• 20 machines ranked failures of LCW hoses
or fittings as 1st, 2nd or 3rd in frequency. • But 7 machines ranked them as 9th = hardly
any hose or fittings failures compared to eight other types. What about their hose make?– Filter data to look at just these 7 machines and
their inner hose material & fitting types- NOT conclusive, except none used nylon inner hose.
Improving Magnet Reliability-Spencer 15
Analyze survey data associated with LCW hose & fittings failures. Part 2
• Compared nylon and EPDM inner hose tubes for hose failures and blocked passages:– Nylon: 5 machines ranked 1st and 2nd for hose leaks– EPDM: 2 machines ranked 1st and 2nd for hose leaks– Nylon: a 2nd, 3rd & 4th rank for blocked passages– EPDM: zero ranks for blocked passages
• My conclusion: LCW hoses with EPDM innermost tube are better than nylon for avoiding hose failures & blocked passages
Improving Magnet Reliability-Spencer 16
Analyze survey data associated with LCW hose & fittings failures. Part 3
Improving Magnet Reliability-Spencer 17
Compared permanently crimped hose couplings with “other methods” with respect to hose & fittings leaks-No conclusion as to which is best; they had equal numbers of 1st & 2nd ranked failures
Analyze survey data associated with LCW hose & fittings failures. Part 4
• Q56: Do you only replace hoses after they or their fitting has leaked? Yes or No.
• Filter data on Q56 to answer :Does replacing hoses before they fail help reduce the frequency of leaking-hose failures?
• 15 machines replace hoses before they leak, 6 of them ranked hose leaks as 9th (recall only 7 machines with 9th)
• 9 machines only replace hoses after they have leaked, just 1 of them ranked hose leaks as 9th
• My conclusion: One should replace LCW hoses on a schedule even if they have not yet leaked.
Improving Magnet Reliability-Spencer 18
Do newer magnets have longer MTBFs?
Improving Magnet Reliability-Spencer 19
Water cooled magnet MTBFs vary widely, 13 surveys reported from 220,000 to 12,600,000 hours. Chart shows ages of magnets .Conclude: cannot say newer magnets (<30 yrs) will fail less often than old magnets (>30 yrs) [in last 4 years]
0
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13
Num
ber o
f acc
eler
ator
s w
ith th
e M
TBF
MTBF in millions hours of various accelerators' magnets
Over 50 yrs30-50 yrs10-29 yrsUnder 10 yrs
Why have magnet design standards?• Q6: Did your institution have a written set of
magnet design standards/rules when these magnets were designed [e.g. limiting velocity of LCW flowing in coils; not allowing internal braze joints in coils] ?
• Second-most popular failure type: water leaks at braze joints (14 machines ranked 1st , 2nd or 3rd)
• A leaking internal braze joint has more severe consequences than an external one: turn to turn short
• Can one see a difference in the frequency of braze leaks correlated with an institution having a design standard forbidding internal brazes or not?Improving Magnet Reliability-Spencer 20
Do magnets at machines with a rule forbidding internal brazes have less braze leaks?
Filter on Q17- either have a rule or don’t, and look at the frequency rankings of water leaks at braze joints:
Rule? 1st 2nd 3rd 4th 5th 6th 7th 8th 9th Tot #No rule 3 4 0 1 1 0 0 0 2 11Yes 1 2 4 0 2 0 0 1 3 13My conclusion: having rule helps reduce
frequency of braze leaks [look at 1st & 2nd ranks numbers]
Improving Magnet Reliability-Spencer 21
Third most popular failure type: those caused by human error
• Q60: Root cause of many magnet failures is human error. What are you lab’s strategies for minimizing human errors; describe strategies used at design , fabrication and operation stages?
• This was one of the 9 questions you had to answer in order to complete the survey (and win a prize). Some people wrote very brief answer for only one stage, others wrote long paragraphs and even added “after maintenance period” strategies.
• On next few slides: summary of strategies
Improving Magnet Reliability-Spencer 22
Strategies at DESIGN stage for minimizing human errors
• Have written magnet design standards• Specify standard set of materials: Cu, insulation
tape, epoxy components. Insist vendors use them.• Refer to past experience.• Hold design reviews with “external” reviewers• Do computer modeling• Design in safety factors for field strength, delta T• Build a prototype and measure it
Improving Magnet Reliability-Spencer 23
Strategies at FABRICATION stage for minimizing human errors
• Monitor commercial vendor’s operations closely• Hipot (“megger”) coils to steel core• Do hydrostatic pressure tests of coils• Measure LCW flow at operating pressure• Design & use a traveler: step-by-step instructions
for every fabrication task with space for pertinent test results to be written in and signatures of tech.
• Train technicians to braze hollow-core conductor into the terminal block; wind coils
• Do full magnetic measurement of every magnetImproving Magnet Reliability-Spencer 24
Strategies at OPERATION stage for minimizing human errors
• Put thermal switches on every LCW return cond.• Double check power connections before power up• Check magnetic polarity with gauss meter at few A• Software limits on current allowed• Flow switch alarms to indicate low LCW flow [keep
checking switches are functioning]• Train technicians in installation tasks• Use male & female power connections• Use bayonet power connections [completely
avoiding lugs needing tightening]Improving Magnet Reliability-Spencer 25
Strategies after a maintenance period for minimizing human errors
• Go through a check list and inspect mechanical items like “LCW turned back on”
• Use micro-switches on all water valves, read out by control system
• Run magnets at intermediate power, turn off, enter tunnel with infra-red thermal camera to look for hot spots on power terminals
• Train technicians, give them time to do the job right
Improving Magnet Reliability-Spencer 26
Conclusions for talk for Accelerator Reliability Workshop
• No institution has failure-free magnets• Very old magnets can continue to function• The actual numbers of magnet failures that bring
down the beam are quite small, but each one can be costly
• The most common failures are associated with the LCW cooling the magnets;
• The survey will be re-opened for more labs to complete it and further analysis of the survey data will be made, to be presented at MT 22 [I hope]
Improving Magnet Reliability-Spencer 27
Open discussion of magnet reliability
• The characteristics of the Low Conductivity Water (LCW) flowing through our magnets have impact on the operation of the magnet and it appears from my survey that a large fraction of labs do not appreciate this.
• I had 13 questions about LCW properties and the components of an LCW system and this section generated the most “don’t knows”
Improving Magnet Reliability-Spencer 28
Summary of survey responses to some LCW system questions page 1
Improving Magnet Reliability-Spencer 29
Summary of survey responses to some LCW system questions page 2
Improving Magnet Reliability-Spencer 30
Summary of survey responses to some LCW system questions page 3
Improving Magnet Reliability-Spencer 31
Summary of survey responses to some LCW system questions page 4
Improving Magnet Reliability-Spencer 32
Studied the LCW answers of the 2 (a) surveys and can see they have too much dissolved oxygen in their LCW
Useful Articles on Effects of LCW in Copper & Optimum Parameters
• “Corrosion of Copper by De-Ionized Cooling Water” by H.Scholer and H. Euteneur, Institute fur Kernphysik, Mainz FRG. EPAC 1988 paper
• “Accelerator Magnet Plugging by Metal Oxides: A theoretical investigation, remediation and preliminary results” . By W.W.Rust, Jefferson Labs, USA. Published in PAC2003
• “Low Conductivity Water Systems for Accelerators” by R. Dortwegt, APS, Argonne National Lab, USA
Published in PAC 2003Improving Magnet Reliability-Spencer 33
34
Availability: Average ratio of the time that the system or component is usable to the total amount of time that is needed.
MTBF (Mean Time Between Failure): MTBF is a basic measure of reliability for repairable items. It can be described as the number of hours that pass before a component, assembly, or system fails.
MTTR (Mean Time To Repair): MTTR is the average time required to perform corrective repair on the removable items in a product or system.
Maximum value for A = 1 (or 100%)Overall availability of system of 3 dis-similar devices = A1 x A2 x A3
Overall A of N identical devices= A1 xA1 x A1 ….x AN = AN
MTTRMTBFMTBF
DowntimeUptimeUptimeAtyAvailabili
)(
Availability of N magnets = (Availability of one magnet) N
AVAILABILITY: DEFINITIONS for 1 & N devices
Improving Magnet Reliability-Spencer
Improving Magnet Reliability-Spencer 35
Ensuring/improving magnet availability: Various things must be done. What, depends on stage of a project you are in. If you have
existing magnets- different approach
Reliability engineering must be started as early as the conceptual design stage of a project and involve all engineering disciplines and accelerator physicists.
If large numbers of new magnets are needed- multiple commercial vendors to provide, so
• Magnet design standards will be imposed – to ensure reliable magnets by staying within well-recognized limits on magnet
parameters– to ensure reliable magnets by using high quality materials
• Common review, QC and maintenance processes will be followed to ensure reliability
• But in order to have a significant impact on the MTBF and MTTR values of your magnets I recommend you carry out a detailed Failure Modes and Effects Analysis (FMEA) to learn how to revise your future magnet designs, fabrication techniques and operating conditions to make more reliable magnets and reduce their repair time.
• MANUAL ON HOW TO DO A FMEA ON AN ELECTROMAGNET:• http://www.slac.stanford.edu/spires/find/hep/www?r=SLAC-TN-09-001