FaultDetectionusingAdvancedAnalyticsatCERN'sLargeHadronCollider

AntonioRomeroMarínManuelMartinMarquez

USA- 27/01/2016 BIWA16 1

What’sCERN

USA- 27/01/2016 BIWA16 2

What’sCERN• EuropeanLaboratoryforParticlePhysics• FundamentalResearch

• WorldwideInternationalCollaboration• USisanimportantcontributortoCERNandtheexperiments

• Education&Training• PushFrontiersofTechnology

USA- 27/01/2016 BIWA16 3

CERNopenlab• Public-privatepartnershipbetweenCERNandleadingICTcompaniesandresearchinstitutes• Acceleratecutting-edgesolutionsfortheworldwideLHCcommunityandwiderscientificresearch.• Designedtocreateanddisseminateknowledge• Publicationofreportsandarticles• Workshopsorseminars• CERNopenlab StudentProgramme

USA- 27/01/2016 BIWA16 4

CERNAcceleratorComplex

USA- 27/01/2016 BIWA16 5

DataAnalyticsChallenges• Somefaultscannotbeavoided

• Decreasetheavailabilityforrunning physics

• Preventivemaintenanceisnotenough• Doesnottakeintoaccounttheconditionoftheequipment

• LHCupgradeswillfurtherincreaseluminosity• Computing resourcesneedswillbehigher• Datageneratedwillincreasedrastically

USA- 27/01/2016 BIWA16 6

DataAnalyticsObjective

USA- 27/01/2016 BIWA16 7

Areasofinvestigation• Predictivemaintenanceandsystemoptimization• Dataextraction,transformationandloading(ETL)• DataVisualizationandDiscovery

USA- 27/01/2016 BIWA16 8

Rfordataanalytics• Languageforstatisticalcomputingandgraphics• Powerfulintegratedsuiteofsoftwarefacilities

• Datamanipulation• Calculation• Graphicaldisplay,plotting

• FreeandOpenSource• Extensible

• Over7800CRANpackagesextendingbasefunctionality• Interactiveandprogrammaticenvironment• Greatusercommunity

• Activedevelopment,frequentlyupdated

USA- 27/01/2016 BIWA16 9

ParallelProcessingApproachesinR• ParallelprocessingwithR• Foreach• Snow• Rmpi• BatchExperiments package(BatchJobs)

• SparkR• PackagedinSparkfrom1.4.0

• OracleREnterprise(ORE)

USA- 27/01/2016 BIWA16 10

WhyORE?- OREbenefits• Adatabase-centricenvironmentforanalyticalprocessesinR• AllowstousethedatabaseservertorunRscripts(scalability&performance)• EliminatememoryconstraintofclientRengine• Rworkingondatadirectlyinthedatabase,nodatatransfer• Integration withtheSQLlanguage• Allowsin-databasedataanalysis• Providedataparallelism

• TransparencyLayer• TransparentlyanalyzeandusedatainOracleDatabasethroughR• NoneedtoknowSQL

USA- 27/01/2016 BIWA16 11

CERNDatabaseactivity

USA- 27/01/2016 BIWA16 12

October2012 December2015Max size ACCLOG 136TB ACCLOG352TB

Maxredo ACCMEAS27TB/month QPSR115TB/month

OREdeployment

USA- 27/01/2016 BIWA16 13

4xXeonE7-8895v2(15coreseach)2TBRAM

4.8TBFlash+6x1.2TB10KHDD

R ENTERPRISE

RAC– RealApplicationCluster

+

-EquipmentExperts-Operators

OracleDatabase

Engine

OracleDatabase

Engine

CERNControlSystems• IoT andControlSystem

• Cryogenics• Vacuum• MachineProtection• PowerConverters• QPS

• AcceleratorLoggingService• ~275GB/day• Storingmorethan50TB/year• Dataacquisition

• CERNacceleratorcomplex• Relatedsubsystems• Experiments

• Around1millionsignals

USA- 27/01/2016 BIWA16 14

LargestCryogenicsInstallation• 50kI/O,11kactuators, ~5k control loops• Control:

• ~100PLCs(Siemens,Schneider)• ~40FECs(industrial PCs)

• Supervision:26 SCADAservers

USA- 27/01/2016 BIWA16 15

Instrument/Actuators TotalTemperature[1.6– 300 K]Pressure[0– 20bar]LevelFlowControl valvesOn/Off valvesManualvalvesVirtual flowmetersControllers (PID)

10361230092326333692183519163254833

UseCase:AnomalyDetectiononBeamScreen

USA- 27/01/2016 BIWA16 16

the resistive wall power losses dissipated by the beam and it intercepts syn-chrotron radiation and molecules in order to increase the quality of the beamvacuum [8]. Each beam screen is cooled by conduction through two smallcooling pipes, supplied by supercritical helium coming from the CryogenicDistribution Line of the LHC (QRL) at 3 bar and 4.6 K to avoid two-phaseflow regime. One beam screen is installed on each beam pipe in a half-cell ofthe LHC corresponding to three superconducting dipoles and one supercon-ducting quadrupole for a total length of 53 m.

Beam Screen

Cooling tube

Beam Pipe

Figure 2: LHC beam screen cooled by its two small cooling tubes

The initial cryogenic library provides only 0-D pipe models or very simpli-fied 1-D models where the ideal gas law is used. Hence, to model this specificsupercritical helium flow cooling the LHC beam screen, a new model had tobe developed reusing the facilities of the Ecosimpro cryogenic library. Thiskind of approach was already used previously to simulate a heat wave prop-agation along the LHC cryogenic distribution line after a quench [9] but inthis last paper, the model was based on simplifications for very low pressurehelium flows which are not valid for supercritical helium flows.

The following section describes the model of a supercritical helium flowthough small cooling pipes which will be then embedded in a larger systemwith other components from the cryogenic library in order to simulate thecomplete beam screen cooling circuits.

4.1. Modeling of a supercritical helium flow

A one dimensional compressible flow according to the x axis has to beconsidered as the isothermal compressibility is very high for supercritical

5

Source:EN-ICE(BenjaminBradu,EnriqueBlanco)

• PIDoutput(timeseries)segmentation• Characterization+Featureextraction• Featuresbasedclassification

UseCase:AnomalyDetectiononBeamScreen

USA- 27/01/2016 BIWA16 17

STABLEBEAM BEAMDUMPSETUPINJECTBEAM

UseCase:FaultyCryogenicsValveDetection• Whatistheobjective?

• Predictfaultyvalvesbeforetheyactuallyfail

• How?• Valvereceiveanapertureordervalue(apertureorder)• Effectiveaperturerealizedbythevalve(aperturemeasured)• Analyzing thedifferencebetweenboth(S =apertureorder- aperturemeasured)

13/08/2015 openlabSummerStudentsLecture13:DataMining,DataAnalyticsandBigData

18

UseCase:FaultyCryogenicsValveDetection• Signalsused:

• S =apertureorder- aperturemeasured• FeaturesextractionsbasedonS

• Variance• Percentile99.9• Ropedistance– R(S)• NoiseBand – B(S)

• AutomaticFaultyValvesDetectionSystem• SVM- SupportVectorMachine

• Thelearningset44valves• apertureorder(%)• aperturemeasured (%)

USA- 27/01/2016 BIWA16 19

Cryo Valves– ExtractFeaturesinORE

• Valvesdatasetwith6features• 10.5Mrows• 42Mrows• 168Mrows

USA- 27/01/2016 BIWA16 20

Ourexperience• Movethedataisveryexpensive• ItisfastertoprocessitintheDBwithOREusingtheappropriatedegreeofparallelism

• DBnodesalreadypreparedfortheworkload• Memoryisaproblemforsmallclients• Simplifiestheinfrastructure

• Write/adaptRcodeisstraightforward• ThankstotransparencylayerandembeddedRexecution

USA- 27/01/2016 BIWA16 21

Nextsteps• TestwithDBin-memoryfeatures• Benefitfromthehybridecosystem• GetthebestfromRDBMS+Hadoop

• Don’tforgetaboutcurrentapplicationlayerworkingwiththeDB• Deployinstreaming

• PredictiveModelMarkupLanguage(PMML)• Compatibleacrossmultiplelanguages(R,Python,Spark,Java,etc.)

USA- 27/01/2016 BIWA16 22

Conclusions• RecosystemisverygoodforDataAnalytics• OREoffersagoodsolutiontogetmorefromyourDBdeployment• Nopainadaptingcode• FollowsRstandards

• Avoidmovingdatawhenpossible• Hybridsystemslookquitepromisingtocovermultiplescenarios

USA- 27/01/2016 BIWA16 23