petabyte-scale computing challenges of the lhcb experiment uk e-science all hands meeting 2008,...
TRANSCRIPT
Petabyte-scale computing challenges of the LHCb experiment
UK e-Science All Hands Meeting 2008, Edinburgh,
9th September 2008
Y. Y. Li on behalf of the LHCb collaboration
Outline
The questions…The LHC – the experiments taking up the
challengeThe LHCb experiment at the LHCLHCb computing model
Data flow, processing requirements
Distributive computing in LHCb Architecture and functionality Performance
The questions…
The Standard Model of particle physics explains much of the interactions between the fundamental particles that form the Universe
all experiments so far confirming its predictionsBUT many questions still remain …
How does gravity fit into the model? Where does all the mass come from? Why do we have a Universe made up of matter? Does dark matter exist and how much?
Search for phenomena beyond our current understanding
Go back to the 1st billionth of a second after the BIG BANG…
The Large Hadron Collider
100m below surface on Swiss/French border 14 TeV proton-proton collider, 7x higher than previous machines 1,232 superconducting magnets chilled to -271.3ºc 4 experiments/detectors
France
Swiss
CMSAlice
LHCb
Atlas
27Km
After ~25 years since its first proposal…
1st circulating beam tomorrow!
1st collisions in October 2008.
LHCb
p p
VErtex LOcator – b decay vertex
Operates only ~5mm from the beam
Ring Imaging CHerenkov detector – particle ID
Human eye – 100 photos/s
RICH – 40million photos/s
LHC beauty experiment Special purpose detector to search for:
New physics in very rare b quark decays Investigates particle-antiparticle asymmetry
~ 1Trillion bb pairs per year!
Data flow
Five main LHCb applications (C++ : Gauss, Boole, Brunel, DaVinci Python: Bender)
Gauss
Event Generation
Detector Simulation
Boole
Digitization
Brunel
Reconstruction
DaVinci
Analysis
Bender
Sim
DST
DSTStatistics
RAWRAWData flow
from detector
Production Job
Detector calibrations
Analysis Job
Sim – Simulation data formatDST – Data Storage Tape
CPU times
Gauss
Event Generation
Detector Simulation
Analysis
2,000 interesting events selected per second = 50MB/s data transferred and stored
40 million collisions (events) per second
DST
Offline full reconstruction, 150MB processed per second of running
Full simulation reconstruction, 100MB / event
500KB / event
962 physicist, 56 institutes in 4 continents
Full simulation DST 80s / event (2.8GHz Xeon processor)~100 years for 1 CPU to simulate 1s of real data!
107s data taking / year + simulation ~O(PB) data per year
LHCb computing structure
CERN
RAL, UK
PIC, Spain
IN2P3, France
GridKA, Germany
NIKHEF, Netherlands
CNAF, Italy
Detector RAW data transfer
10MB/s
Simulation data transfer
1MB/s Tier 0 CERN Raw data, ~3K cpu
Tier 1 Large centres Reconstruction and
analysis, ~15K cpu
Tier 2 Universities (~34)
Simulations, ~19K cpu
Tier 3 / 4 Laptops, desktops etc…
Simulations
Needs distributed computing
LHCb Grid Middleware - DIRAC
LHCb’s grid middleware: Distributed Infrastructure with Remote Agent Control
PythonMulti-platform (Linux, Windows)Built with common grid tools
GSI (Grid Security Infrastructure) authenticationPulls together all resources, shared with other
experiments Uses experimental wide CPU fair share
Optimises CPU usage with Long, steady simulation jobs by production managers Chaotic analysis usage by individual users
DIRAC architecture
Service orientated architecture4 parts
User interface Services Agents Resources
Uses a pull strategy for assigning CPU’s Free, stable CPU’s request for jobs from main
server Useful in masking instability of resources from
users
Linux based
Multi-platform
Multi-platform
Combination of DIRAC
services and non-DIRAC
services
Web monitoring
Security and data access
DISET, DIRAC SEcuriTy module Uses openssl and modified pyopenssl Allows for proxy support for secure access
DISET portal used to facilitate secure access on various platforms when authentication process is OS dependent
Platform binaries shipped with DIRAC, version is determined during installation
Various data access protocols supported SRM, GridFTP, .NetGridFTP on Windows etc …
Data Services operates on main server Each file is assigned a logical file name that matches
to the physical file name(s)
Compute element resources
Other grids, e.g. WLCG (Worldwide LHC Computing Grid) Linux machines
Local batch systems, CondorStand alone, desktops, laptops etc …Windows
3 sites so far ~100 CPU Windows Server, Windows Compute Cluster Windows XP
~90% of the World’s computers are Windows
Pilot agents
Used to access other grid resources, e.g. WLCG via gLite
User job triggers pilot agent submission by DIRAC as a ‘grid job’ to reserve CPU time
Pilot on WN checks environment before retrieving the user job from DIRAC WMS
Advantages Easy control of CPU quota for shared resources Several pilot agents can be deployed for the same job if failure on WN occurs If full reserved CPU time is not used another
job can also be retrieved from the DIRAC WMS
Agents on Windows
Windows resources – CPU scavenging
Non-LHC dedicated CPU’s Spare CPU’s at
Universities, private home computers etc…
Agent launch would be triggered by e.g. screen saver
CPU resource contribution determined by owner during DIRAC installation
Head Node
DIRA
C A
PID
IRAC
API
DIRAC WMS
Job ManagementService
SandboxService
Job Matcher
LFC Service
Job MonitoringService
SoftwareRepository
DISETLocal
SE
DIRACAgent
Watch-dog
DIRAC
WrapperJob
ProxyServer
Windows Compute Cluster external communication
Windows Compute Cluster internal communication
Head NodeHead Node
DIRA
C A
PID
IRAC
API
DIRAC WMSDIRAC WMS
Job ManagementService
SandboxService
Job Matcher
Job ManagementService
SandboxService
Job Matcher
LFC Service
Job MonitoringService
SoftwareRepositorySoftware
Repository
DISETDISETLocal
SELocal
SE
DIRACAgent
Watch-dog
Watch-dog
DIRAC
WrapperJobJob
ProxyServerProxyServer
Windows Compute Cluster external communication
Windows Compute Cluster internal communication
Windows Compute Cluster Shared single DIRAC installation Job Wrapper submits retrieved jobs via Windows CC submission calls Local job scheduling determined by the Windows CC scheduling service
Cross-platform submissions
Submissions made with valid grid proxy
Three Ways JDL (Job Description Language) DIRAC API Ganga job management system
Built on DIRAC API commands
Full porting to Windows under process
SoftwarePackages = { “DaVinci.v19r12" };InputSandbox = { “DaVinci.opts” };InputData = { "LFN:/lhcb/production/DC06/v2/00980000/DST/Presel_00980000_00001212.dst" };JobName = “DaVinci_1";Owner = "yingying";StdOutput = "std.out";StdError = "std.err";OutputSandbox = { "std.out", "std.err", “DaVinci_v19r12.log” “DVhbook.root” };JobType = "user";
JDLimport DIRACfrom DIRAC.Client.Dirac import *
dirac = Dirac()job = Job()
job.setApplication(‘DaVinci', 'v19r12')job.setInputSandbox(['DaVinci.opts’])job.setInputData(['LFN:/lhcb/production/DC06/v2/00980000/DST/Presel_00980000_00001212.dst'])job.setOutputSandbox([‘DaVinci_v19r12.log’, ‘DVhbook.root’])
dirac.submit(job)
API
User pre-compiled binaries can also be shipped
Jobs are then bound to be processed on the same platform
Successfully used in full selection and background analysis studies (User – Windows, resources – Windows and Linux)
Performance
Successful processing of data challenges since 2004 Latest data challenge
record of >10,000 simultaneous jobs (analysis and production) 700M events simulated in 475 days, ~1700 years of CPU time
Windows SitesLinux Sites
Total Running Jobs: 9715
Conclusions
LHC will expect O(PBytes) of data per year per experiment
Data to be analysed by 1,000s physicists on 4 continents
LHCb distributed computing structure is in place, pulling together a total of ~40K CPU’s from across the World
The DIRAC system has been fine tuned on the experiences from the past 4 years of intensive testing
We now eagerly await for the LHC switch on and the true test!
1st beams tomorrow morning!!!