CERN Availability Working Group & Accelerator Fault Tracker Availability Working Group & Accelerator Fault Tracker - Where do we go from here? B. Todd, A. Apollonio, L. Ponce, C. Roderick + input from R. Schmidt, M. Zerlauth, TE/MPE, TE/EPC, BE/CO, BE/OP and others LHC Performance Workshop CERN Availability Working Group & Accelerator Fault Tracker 3 Introduction Availability is the only means to increase integrated luminosity once a machine is levelled. LHC Availability Working Group (AWG) launched in : objective view of availability not possible = weaknesses in data captured : AWG proposed the Accelerator Fault Tracker to solve data issues 2014: Accelerator Fault Tracker launched by BE/CO, BE/OP and TE/MPE 2015: Accelerator Fault Tracker was extensively used for availability data analysis Where do we go from here? CERN Availability Working Group & Accelerator Fault Tracker 4 Availability Working Group integrated luminosity is the real Key Performance Indicator coherent & objective information capture is primary concern biggest challenge of AWG CERN Availability Working Group & Accelerator Fault Tracker 5 Important Concepts from AWG Coherent & objective = viewpoint from both operations & equipment Cardiogram Availability Matrix Operations & Equipment = Hybrid Pareto CERN Availability Working Group & Accelerator Fault Tracker 6 Cardiogram = Operations Viewpoint KPI = inverse femtobarn Increase physics performance in stable beams Increase stable beams duration Decrease turnaround time Decrease fault time CERN Availability Working Group & Accelerator Fault Tracker 7 Availability Matrix = Equipment Viewpoint Typical KPI optimised by equipment groups = Mean Time Between Failures Remove or mitigate the failure mode entirely Make the failure less likely (increase reliability, ) Make the failure have a lower impact (decrease repair time, decrease diagnostics time, ) CERN Availability Working Group & Accelerator Fault Tracker 8 Hybrid Pareto = Both Viewpoints equipment fault time operations unavailable time Combined Viewpoint: equipment fault time longer than operational unavailability equipment fault time shorter than operational unavailability equipment fault time zero Correlation of all of these is the only way to really see availability Equipment Group optimisation of MTBF does not mean LHC optimises inverse femtobarn shadow / parallel faults, pre-cycles, beam events, operational errors, CERN Availability Working Group & Accelerator Fault Tracker 2010 Manual report creation carried out once per year Subjective approach Opaque process no correlation operations vs equipment strategic conclusions impossible to make lag data capture to report generation 0.2 FTE data processing (3 x STAFF & DOCT) = proposal for the Accelerator Fault Tracker tool (AFT) CERN Availability Working Group & Accelerator Fault Tracker 10 Accelerator Fault Tracker AFT 1.0 ( ) infrastructure to collect operations view-point data produce cardiogram structure foreseen to fold in equipment data LS1 = AFT launched as BE/CO, BE/OP and TE/MPE initiative Led by C. Roderick, proposed in three releases: AFT 2.0 ( ) Capture data from equipment groups Produce combined equipment and operations viewpoints AFT 3.0 (2018+) Connect to other data services at CERN (INFOR EAM, IMPACT, LAYOUT) fully integrated transverse view CERN Availability Working Group & Accelerator Fault Tracker March November 2015 possibility for automation of parts of this feedback direct eLogbook extraction One-click cardiogram Before AFT = 2 months to cardiogram 2010 data few seconds to get it now correlation operations vs equipment operations viewpoint = weekly review equipment viewpoint = annual review objective view possible no lag data capture to report generation 0830 meetings using previous days cardiograms transparent approach 1.5 FTE tool (BE/CO) 0.25 FTE tool feedback (AWG) 0.5 FTE data entry (AWG) CERN Availability Working Group & Accelerator Fault Tracker One-Click (ish) 12 CERN Availability Working Group & Accelerator Fault Tracker 13 Where Do We Go From Here? [1/2] AFT 1.0 AFT 2.0 Add some equipment group information TE/MPE and TE/EPC already started work New Analysis Requests E.g. what is the influence of energy / intensity on availability? E.g. why was the ion run having such high availability? Fold in more information from more sources to have better analyses and correlations AFT & AWG in the Injectors It is possible to propagate the AFT tool, At the same time it is an LHC AWG How to approach this? CERN Availability Working Group & Accelerator Fault Tracker 14 Where Do We Go From Here? [2/2] Day to Day data validation and continuous improvement of the AFT remains a core aspect of the AWG effort to maintain fault data increased x3 used to be a part-time job dedicated resources? Strategic View information created for the LHC can be exploited for HL-LHC, FCC, new and existing machines are being designed facing availability as a primary deliverable. AFT information should be able to be used to create generic models Modelling and strategic aspects are becoming more detailed and heavier to manage, with several parties involved. The mandate of the AWG only covered the LHC. centralised modelling & strategy into a different dedicated (sub-) working group. CERN Availability Working Group & Accelerator Fault Tracker 2012 Availability Working Group Established Thank you! Questions?