CASE STUDY

Intel®Xeon®Processor E5 Family

High-Performance Computing

A Physicist’s Best Friend

The discovery of the Higgs boson marked an important step towards understanding the fundamentallaws which underpin the universe. This elusive particle, thought to be at the very root of the existenceof mass, has been the focus of experiments carried out at the Large Hadron Collider (LHC) at CERN onthe French-Swiss border.

Challenges• Crunch numbers. Process, filter and analyze petabytes of complex LHC data every year

• Needles in haystacks. Empower scientists to identify the 20 or 30 data points they need to find inmillions – using advanced algorithms

Solutions • High energy physics computing. The 9,000 servers at the heart of CERN’s WLCG are powered byroughly 90,000 cores, the majority of which are powered by Intel® technology, with more than 12,000belonging to the Intel® Xeon® processor E5 family

• Well connected. As part of the equipment used for the interconnects, the CERN part of the computinggrid uses Intel® 10 Gigabit Ethernet Network Interface Cards 82599EB to connect disparate serversand support a variety of specialized experiments

Impact • World-class research. Scientists use the computing platform to explore mysteries surrounding theHiggs boson and the Big Bang

• More to come. Ambitious development plans for the LHC will continue to need cutting edge processingtechnology

CERN uses Intel®Xeon®processor E5 family to find the Higgs boson

“A solid relationship with an in-

dustry leader like Intel is important,

not only so we can make the most

of the resources currently available

to us, but also in terms of future

development.”

Ian Bird, Project Leader for the Worldwide

LHC Computing Grid (WLCG) at CERN

Cracking nature’s code The scientists at CERN carry out high-energy particlephysics research, studying the interactions betweenparticles accelerated around the 27km-long LHCring and analyzing the immense quantity of datacaptured by sophisticated sensors. These sensorscollect an extremely large volume of information,which needs to be processed by CERN’s computeinfrastructure at incredibly fast rates. After mas-sive capturing and filtering efforts, CERN storesmore than 25 petabytes (or two million double-layer DVDs-worth) of data from the LHC and itsfour main experimental facilities each year.However, even this volume of information is smallcompared to the total amount of insignificant datathat the physicists first must identify, usingmassive processing power, and then discard. Thedecision of what to keep and what to discard mustbe made in fractions of a second.

The critical role of algorithmsAs two proton beams speed around the LHC, theycollide with each other, creating a split second ofchaos in which traces of the Higgs boson can beidentified. Catching a glimpse of these fleetinginteractions requires highly complex calculations,and the physicists must have the right informationat their fingertips.

The first step in making these calculations is cap-turing the proton-proton interaction data throughthe sensors of LHC experiments. Specialized algo-rithms filter this data in real time, reducing thenumber of events from 40 million per second toaround 100,000. However, this still leaves around1.6 gigabytes of data per second to be reconstructedand analyzed before it is fed into WLCG, a net-work of over 160 data centers worldwide thatcarries out further refined analysis with improvedcalibration using the grid’s vast capacity of distrib-uted computing resources.

Image © CERN

“The nucleus of this network, representing onesixth of the overall WLCG computational power, ismade up of 9,000 servers housed in our Genevadata center,” explain Wayne Salter, computingfacilities group leader in CERN’s IT Departmentand Olof Bärring, his deputy responsible for facilityplanning and procurement. “Some of these serversare powered by more than 12,000 cores of high-performance Intel Xeon processors E5 family.” Thecalculations used at this stage determine whichinformation should be stored for further analysis.

David Francis is one of the scientists in charge ofcapturing data generated by the ATLAS* (AToroidal LHC ApparatuS*) experiment, involvedin the search for the Higgs boson. He explainsthat the complexity of the various ATLAS detectionsystems requires a multi-phase data-capture process.“During the initial decision-making phase we workwith customized electronics, built with a mixtureof custom application-specific integrated circuits(ASICs) and commercially available componentsthat receive data directly from the detector sen-sors that record about 40 million proton beamcrossings per second,” he says. “However, only300 to 400 of these interactions deserve ATLASphysicists’ attention.”

The huge volume of data to be processed at eachstage for multiple experiments is similar to seek-ing needles in haystacks. This means advancedcomputing capacity is essential for CERN’s re-searchers.

A forward-looking team for top-levelusersThe unique and highly skilled scientific team, aswell as the use of the latest technologies, are keyelements of the computational magic performedat CERN. Another is the use of high-performance,multi-core processors such as the Intel Xeonprocessor E5 family. The third essential elementis the equally well-trained and responsive supportteam that ensures technology and scientists workin perfect harmony. Intel offers this type of sup-port through CERN openlab, the research and de-

velopment framework that CERN created in 2001with leading ICT companies.

“CERN openlab was conceived for the very reasonthat information technology has become funda-mental in allowing science to progress even whenthe necessary experiments cannot be conductedin the visible world,” explains Sverre Jarp, CERNopenlab chief technology officer. “The IT industryis constantly evolving, particularly in the area ofresearch, and in order to test these innovationsand implement them at the right time, CERNphysicists need a laboratory that focuses on thefuture, while also evaluating fundamental issuessuch as the cost of computing infrastructure andenergy efficiency.”

Planning ahead in such a way is a vital resourcefor the analytical scientists, since they require ex-traordinary computational capacity and significantIT resource planning. “For this reason,” stressesDavid Foster, deputy head of CERN’s IT depart-ment, “Intel’s most important contribution to oursuccess is really based on Moore’s Law. It allowsus to predict the potential of the computing in-struments we’ll have access to in the future witha decent level of accuracy.”

One of the fundamental roles of Intel® technologyin WLCG is to support the grid’s interconnectivity.“The 10-Gigabit backbones in production are nowcommonplace,” states Jarp. “They weren’t 10 yearsago, during the preparatory phases of the LHCexperiments. Thanks to CERN openlab, we wereable to try out the technology much earlier.” Thesame thing is happening today in the computingarchitecture’s transition from petascale – with aprocessing capacity measured in petaflops (athousand trillion floating point operations persecond) – to the future exascale architecturewith performance a thousand times higher.

As an ideal test bed for extreme-scale computingCERN is playing an active role in this crucial devel-opment. Following the major advances in physicsthat the laboratory achieved during the LHC’s firstrun, including the discovery of the Higgs boson,the team has now shut down the accelerator to

World-leading research institution uses Intel®technology to push scientific boundaries

“The IT industry is constantly

evolving, particularly in the area

of research, and in order to test

these innovations and implement

them at the right time, CERN

physicists need a laboratory that

focuses on the future, while also

evaluating fundamental issues

such as the cost of computing infra-

structure and energy efficiency.”

Sverre Jarp, CERN openlab Chief Technology Officer

Lessons learned

In the world of scientific breakthroughs, there’sno time to stand still. As data gets bigger andcalculations get more complex, it’s essential tohave the most powerful and reliable computingresources available to keep up. CERN has createda dedicated team in CERN openlab to plan ahead,making sure the technologies deployed will meetever-increasing demands and drive further ground-breaking discoveries like that of the Higgs boson.

develop it further. Upgrades and consolidationwork performed over two years will enable futureincreases in energy. When it resumes running in2015, the LHC will be capable of operating at itsdesign energy of 7 TeV per beam.

The computer that changes scienceThe advanced computing resources that CERNopenlab and Intel have made possible are driving awide range of experiments and scientific progressbeyond the search for the Higgs boson.

Frans Meijers coordinates the complex task ofcapturing data produced by the CMS experiment,which together with Atlas discovered the Higgsboson, determined its mass, and analyzes its sub-sequent decay into other particles. “In capturingour data we apply two rounds of selection. Oneround uses our own custom-built electronics com-ponents, and the other uses software algorithmspowered by commercially available processors,such as the Intel Xeon processor E5 family,” heexplains. Now that physicists have obtained thefirst approximate value of the mass of the Higgsboson, the future increase in the accelerator’sluminosity will be key in tackling the challengingtask of determining the properties of the Higgsboson more precisely and doing searches of newparticles at the higher energy after the LHC ma-chine upgrade. The new Intel® Xeon Phi™ coproces-sors (Intel® Many Integrated Core Architecture)could play a fundamental role here,” he adds. “I’meager to put them to the test.”

For Andrzej Nowak, a researcher working along-side Jarp in CERN openlab, the relationship betweenCERN and Intel has a unique impact on his research.“We have the opportunity to access Intel’s exclu-sive know-how and, in turn, Intel can achieve adeeper and more direct relationship with an environ-ment that is very demanding from a technologicaland computing standpoint, where innovative ideascirculate,” he says. One of the main lines of Intelresearch that CERN openlab is exploring is theintegration of silicon and photonics. This newtechnology promises to increase the applicability

of optical systems in chips, ultimately providing aqualitative jump in the transfer of data – like thatgenerated by the sensors of LHC experiments – tothe digital systems that have to filter and analyzeit. In 2010, Intel introduced a photonic componentfor connections of 50 gigabits per second andnow it is on the brink of reaching transfer speedsmeasured in terabits. CERN’s interest in this tech-nology has also inspired it to collaborate withIntel in offering advanced training to young scien-tists through programs such as the Intel-CERNEuropean Doctorate Industrial Program (ICE-DIP).This is integrated with FP7, the Seventh Frame-work Programme for Research in Europe, estab-lished by the European Commission.

ICE-DIP trainees will focus on the techniquesneeded for acquiring and processing many ter-abits per second, using and expanding the mostinnovative concepts available in the informationand communications industry today. The programwill explore new, untested ideas such as siliconphotonics for network links in harsh operationalconditions, tight integration of reconfigurable logicwith standard computing, and new approaches todata acquisition.

The mysteries of the universe tackledusing 260,000 coresIan Bird, who works with CERN’s IT Departmentas a project leader for the WLCG, says he’s con-vinced of the need to work even more closelywith CERN openlab and Intel in the future to opti-mize the code that runs on the 260,000 combinedcores of this immense distributed architecture.“A solid relationship with an industry leader likeIntel is important,” stresses Bird, “not only so wecan make the most of the resources currentlyavailable to us, but also in terms of future devel-opment.”

In agreement with Bird is Niko Neufeld, who workson the online team for the Large Hadron Colliderbeauty (LHCb) experiment, which is investigatingthe mystery of antimatter. He explains: “The BigBang Theory hypothesizes that when the universe

Copyright © 2013, Intel Corporation. All rights reserved. Intel, the Intel logo, Intel Xeon and Xeon inside are trademarks of Intel Corporation in the U.S. and other countries.This document and the information given are for the convenience of Intel’s customer base and are provided “AS IS” WITH NO WARRANTIES WHATSOEVER, EXPRESS OR IMPLIED,INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF INTELLECTUAL PROPERTY RIGHTS. Receipt orpossession of this document does not grant any license to any of the intellectual property described, displayed, or contained herein. Intel® products are not intended for use inmedical, lifesaving, life-sustaining, critical control, or safety systems, or in nuclear facility applications.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark,are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You shouldconsult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined withother products. For more information go to http://www.intel.com/performance.

*Other names and brands may be claimed as the property of others. 1013/JNW/RLC/XX/PDF 329733-001EN

was formed, the quantity of matter created shouldhave been equal to that of antimatter. However,these two incompatible states did not cancel eachother out, as the creation of the universe goes toprove. So where did the antimatter go?” To answerthis question, the LHCb researchers require com-putational efforts ranging across a vast numberof compute nodes able to work simultaneously.For this reason, Neufeld’s team of researchersworks extensively on software parallelization,studying the opportunities provided by Intel XeonPhi coprocessor devices and their Infiniband* con-nectivity for the interconnection of the computenodes. They see these technologies as essentialsupport for the research project’s most complexcalculations.

Like Lewis Carroll’s famous character, the A LargeIon Collider Experiment (ALICE) passes throughthe looking glass of visible matter to get back toconditions of density and temperature very similarto those of the Big Bang. Where the other exper-iments concentrate on observing the interactionsamong individual protons, ALICE uses lead nuclei,which are much heavier. These nuclei collide andform a plasma, a gas made up of quarks and gluons,the particles that make up protons and neutrons.“Observing this plasma in the ALICE detector,”explains Pierre Vande Vyvre, ALICE project leaderfor data acquisition, “generates an enormousvolume of data to analyze – up to 16 gigabytes asecond.” This is a significant computational challengewhich will be even more grueling after the accel-erator’s second upgrade scheduled for 2018, requir-ing up to 100 times more events to be capturedin real time. The ALICE researchers worked withthe CERN openlab team on software optimizationto maximize the potential benefits coming fromcode parallelization and vectorization technology

“CERN is at the forefront of IT

industry with its thought leader-

ship in using the latest computing

technologies to achieve scientific

breakthroughs.”

Steve Pawlowski, Intel Senior Fellow, Datacenter and Connected Systems Group,Chief Technology Officer and GeneralManager, Pathfinding

to be tested with the new Intel Xeon Phi co-processor. In these investigations into the originof the universe, computational power is crucial,but the role of the scientists – as indispensiblemediators between the theoretical aspects andthe analysis of experimental events and mathe-matical models – is, of course, paramount.

“CERN is at the forefront of IT industry with itsthought leadership in using the latest computingtechnologies to achieve scientific breakthroughs,”says Steve Pawlowski, Intel senior fellow, Data-center and Connected Systems Group, chief tech-nology officer and general manager, Pathfinding.“Its need for more performance and increasinglyefficient high-performance computers reflectsthe hunger to reach further in solving some ofthe world’s biggest mysteries like the existenceof the Higgs boson. CERN’s insatiable hunger formore compute capacity took every advantage ofMoore’s Law and new generations of processingtechnology that Intel has created. Going forward,as we see some significant changes in how wearchitect the next generation of high-performancecomputer systems, the relationship with CERNopenlab will be even more important for us inmaking sure the technology we develop deliversas expected.”

By contributing both technology and expertiseto support the team at CERN, Intel is continuingto pursue its goals of overcoming the physicaland technological limits of supercomputing.

Find the solution that’s right for your organization.Contact your Intel representative, visit Intel’s Busi-ness Success Stories for IT Managers(www.intel.co.uk/Itcasestudies) or explore theIntel.co.uk IT Center (www.intel.co.uk/itcenter).