cern - it department ch-1211 genève 23 switzerland t relational databases for the lhc computing...
Post on 18-Dec-2015
214 views
TRANSCRIPT
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Relational Databases for the LHC Computing Grid
The LCG Distributed Database Deployment (3D)and Conditions Database (COOL) Projects
Andrea Valassi (CERN IT-PSS-DP)
NEC2007, Varna, Bulgaria
14th September 2007
Physics Services Support
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 2 Andrea Valassi – NEC 2007, Varna
Acknowledgements
• Several people have ‘lent’ me their slides or contributed useful suggestions for this talk – Dirk Duellmann and the 3D team– Maria Girone and the CERN Physics DB team– The COOL and CORAL teams – Several users in the experiments
• Many thanks to all of them!
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 3 Andrea Valassi – NEC 2007, Varna
Outline
• Relational databases for LHC computing– Reliable services at CERN and other LCG sites– The 3D project: distributed database deployment
• COOL and other conditions data– COOL development and deployment status
• Conclusions
• Relational databases for LHC computing– Reliable services at CERN and other LCG sites– The 3D project: distributed database deployment
• COOL and other conditions data– COOL development and deployment status
• Conclusions
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 4 Andrea Valassi – NEC 2007, Varna
Relational databases for LHC
• In LHC computing, relational databases will be crucial to store metadata of both physics applications and grid services– Detector conditions (calibration, geometry…)– Experiment data production bookkeeping – Core grid services for cataloguing, monitoring
and distributing LHC data (e.g. LFC file catalog)
• Key features of relational db services– High availability, backup and recovery,
performance and scalability, security…
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 5 Andrea Valassi – NEC 2007, Varna
The 3D Project
• Distributed Database Deployment– The LCG provided initially tools for distributed
access and replication of file-based data– The aim of the 3D project is to provide a similar
infrastructure for data stored in RDBMS services• Experience in running RDBMS services at CERN and
several other LCG sites already since a long time
• Goals of the 3D project as part of LCG– Increase database availability and scalability– Allow applications to access databases in a
consistent and location-independent way– Provide database replication between sites– Coordinate the setup and deployment of the
database and replication infrastructure
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 6 Andrea Valassi – NEC 2007, Varna
3D Service Architecture
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 7 Andrea Valassi – NEC 2007, Varna
Building block – db cluster
• CERN db services use Oracle 10g RAC– High availability – redundant storage and network– Scalability – for CPUs and storage independently– Cost reduction – commodity hardware on Linux
• Homogeneous h/w and s/w setup for all physics DBs– Similar setup is used by most T1 sites as well
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 8 Andrea Valassi – NEC 2007, Varna
Physics DB services at CERN
• Size of Oracle services for physics– 110 mid-range servers, 110 disk arrays
• i.e. 220 CPUs, 440 GB RAM, 300 TB disk space
• Several production clusters– One offline RAC per LHC experiment (up to 8
nodes), Atlas online RAC, COMPASS RAC– In addition: development and validation services
• Development and validation services too– Application release cycle
Development service Validation service Production service
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 9 Andrea Valassi – NEC 2007, Varna
Frontier and CMS
• Read-only access to Oracle data via http– Oracle server at T0– Tomcat server at T0– Squid web cache at
T0/T1/T2
• Frontier used in CMS– Under evaluation in Atlas
(integrated in Coral/Cool)– Successfully tested in
CMS CSA’06, many improvements in 2007
– CMS are confident that they have ways to avoid stale-cache issues
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 10 Andrea Valassi – NEC 2007, Varna
Replication – Oracle Streams(Capture, Propagation, Apply)
Barbara Martelli, INFN T1/T2 Workshop, Nov. 2006
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 11 Andrea Valassi – NEC 2007, Varna
Replication – T0 to T1
• CERN data are replicated to ten T1 sites– Streams used by Atlas (10 T1) and LHCb (6 T1)
• More details in the slides about COOL deployment– The present setup can sustain 2 GB/day to T1
• This is the Atlas requirement for COOL user data
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 12 Andrea Valassi – NEC 2007, Varna
Streams downstream capture
• This technology provides isolation of the source database against problems with the network or with the destination databases
• In 3D, this shields the CERN T0 services from problems in the replication to T1 sites– The redo log retention on the downstream
database is optimized (e.g. 5 days) to allow for re-synchronisation without recall from tape
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 13 Andrea Valassi – NEC 2007, Varna
Replication – online to offline
• Streams used by Atlas, LHCb and CMS– For LHCb offline to online too (see COOL slides)
• Work in progress with Atlas to test replication of the full PVSS archive– Allow detector expert analysis without impacting
the performance of the online production server– Data rates (6 GB/day) much higher than COOL
• Tests over the last two months are promising
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 14 Andrea Valassi – NEC 2007, Varna
3D service operation
• DB service level according to WLCG MoU– At T0: piquet service being set up to replace
current 24x7 best-effort operation • Streams interventions 8x5 for now
– At T1: need more experience to confirm coverage
• Some policies proposed by CERN T0 have been accepted also by the T1 sites– Backup and recovery (Oracle RMAN)– Security patch application (frequency, procedure)– Database and Streams monitoring, usage reports
• Integration with WLCG procedures– GGUS tickets, intervention announcement
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 15 Andrea Valassi – NEC 2007, Varna
Outline
• Relational databases for LHC computing– Reliable services at CERN and other LCG sites– The 3D project: distributed database deployment
• COOL and other conditions data– COOL development and deployment status
• Conclusions
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 16 Andrea Valassi – NEC 2007, Varna
What are conditions data?
• Non-event detector data that vary with time– And may also exist in different versions
• Data produced both online and offline– Geometry, detector control, alignment, calibration...
• Data used for event processing and more– Detector experts– Alignment and calibration – Event reconstruction and analysis
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 17 Andrea Valassi – NEC 2007, Varna
• ALICE– Alice-specific software for time/version handling– ROOT files with AliEn file catalog
• ALICE-managed deployment (AliEn MySQL at T0)
• CMS– CMS-specific software for time/version handling– Oracle (via POOL-ORA) with Frontier web cache
• 3D/CMS deployment: Oracle/Frontier (T0), Squid (T1/T2)
• ATLAS and LHCb– COOL common software for time/version handling
• Common development of Atlas, LHCb and CERN IT– Oracle, MySQL, SQLite, Frontier (via COOL API)
• 3D/Atlas/LHCb deployment: Oracle (T0/T1) with Streams
• ALICE– Alice-specific software for time/version handling– ROOT files with AliEn file catalog
• ALICE-managed deployment (AliEn MySQL at T0)
• CMS– CMS-specific software for time/version handling– Oracle (via POOL-ORA) with Frontier web cache
• 3D/CMS deployment: Oracle/Frontier (T0), Squid (T1/T2)
• ATLAS and LHCb– COOL common software for time/version handling
• Common development of Atlas, LHCb and CERN IT– Oracle, MySQL, SQLite, Frontier (via COOL API)
• 3D/Atlas/LHCb deployment: Oracle (T0/T1) with Streams
CondDB in the 4 experiments
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 18 Andrea Valassi – NEC 2007, Varna
COOL software overview
• Consistent approach to many use cases– Single-version (DCS) and multi-version (calib/align)
• Technology-neutral C++ API – API is not relational - no direct SQL user access– Same user code can be used on all backends
• Maximize reuse of other LCG AA software– CORAL and SEAL for C++ implementation– ROOT/Reflex for python bindings (PyCool)
• Single relational implementation via Coral– Same code for Oracle, MySQL, SQLite, Frontier– Same relational schema for all backends
• Emphasis on read and write performance– Best practices (bulk operations, bind variables)– Detailed performance studies and optimizations
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 19 Andrea Valassi – NEC 2007, Varna
COOL relational implementation
• Modeling of condition data “objects”– System-managed common “metadata”
• Data items: many tables, each with many “channels”• Interval of validity - IOV: since, until• Versioning information with handling of interval overlaps
– User-defined schema for “data payload” • Support for simple C++ types
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 20 Andrea Valassi – NEC 2007, Varna
Development summary
• Milestones– COOL 1.0 released in April 2005
• Basic functionality (development started in Nov. 2004)
– COOL 2.0 released in January 2007• Major backward-incompatible API and schema changes
• Current focus is performance optimization– Separate optimizations for different use cases
• Several performance issues solved in 2007• Feedback from and for Atlas/LHCb stress tests
– Work in progress also on support for new platforms and a few functional enhancements
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 21 Andrea Valassi – NEC 2007, Varna
COOL data distribution
• Replication at the database backend level– Oracle Streams (see next slides)– Cross-technology replication is possible (same
schema for all backends), not really attempted yet
• Oracle remote access via Frontier– Under evaluation in Atlas
• Replication tools based on the COOL API– Static (copy once) or dynamic (copy then update)
• Data slicing/selection is also possible
– Cross-technology replication is possible • Many use cases for SQLite files in Atlas and LHCb
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 22 Andrea Valassi – NEC 2007, Varna
Deployment in LHCb• Computing model
– Reconstruction at T0/T1– Only MC prod at T2
• COOL stores only conditions data for event reconstruction– Oracle at PIT, T0, T1 with
replication via Streams– Geometry and conditions
for MC sent to T2 as SQLite file
• Online db master at PIT– Replicated forward to T0
and T1 via Streams– Data from PVSS processes
• Offline db master at T0– Replicated back to PIT and
forward to T1 via Streams– Data computed in offline
calibration/alignment jobs
(Marco Clemencic, COOL meeting 3 July 2006)
COOL
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 23 Andrea Valassi – NEC 2007, Varna
Deployment in Atlas
• Largest COOL data set comes from DCS– Via the PVSS2COOL data transfer (1.5 GB/day)
• From the online RAC in the T0 computer centre– For offline reconstruction and detector experts
• Many options open for T2 replication– Many use cases (simulation, calibration, analysis)– Static/dynamic replication to sqlite/mysql, Frontier
(Florbela Viegas, CHEP 2007)
3D and COOL - 23
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 24 Andrea Valassi – NEC 2007, Varna
COOL deployment status
• The T0 setup is (almost) complete– The LHCb online server is being set up these days
• Atlas and LHCb T1 sites are all connected– SARA, RAL, PIC, IN2P3, Gridka, CNAF (both)– Plus Nordugrid, Triumf, BNL, Taiwan (Atlas only)– Distributed tests underway in both experiments
COOL Status - 24
Much larger data rates in ATLAS!
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 25 Andrea Valassi – NEC 2007, Varna
Atlas scalability tests (1)
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 26 Andrea Valassi – NEC 2007, Varna
Atlas scalability tests (2)
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 27 Andrea Valassi – NEC 2007, Varna
Outline
• Relational databases for LHC computing– Reliable services at CERN and other LCG sites– The 3D project: distributed database deployment
• COOL and other conditions data– COOL development and deployment status
• Conclusions
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 28 Andrea Valassi – NEC 2007, Varna
Conclusions
• The 3D project has set up a world-wide distributed database infrastructure for LHC– This is one of the largest distributed deployments
of the Oracle database worldwide (over 100 nodes at CERN and a few nodes at each of ten T1 sites)
– T0/T1 are ready for ramp-up to LHC production
• The COOL software is used by both Atlas and LHCb to store their conditions data– COOL deployment is one of the largest users of 3D– First results from Atlas scalability tests confirm that
resources allocated should match required #jobs/h
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it 3D and COOL - 29 Andrea Valassi – NEC 2007, Varna
For more information
• Physics database services at CERN– http://cern.ch/phydb
• The 3D project– https://twiki.cern.ch/twiki/bin/view/PSSGroup/LCG3DWiki
• The COOL project– http://cern.ch/cool
• The CORAL project– http://pool.cern.ch/coral