Oracle for Physics

Jamie.Shiers@cern.ch

Oracle & CERN:25 years of collaboration

The Evolution of Databases in HEPA Time-Traveller's Tale

Jamie Shiers, CERN

~ ~ ~…, DD-CO, DD-US, CN-AS, CN-ASD, IT-ASD, IT-DB, IT-GD, IT-GD’, IT-GS, ¿¿-??, …

The Year 2000…

• A watershed year in the history of HEP & Oracle• The true numerical end to the 2nd millienium• The final run of CERN’s Large Electron-Positron collider

(LEP)

• The story of LEP is closely linked to that of CERN-Oracle collaboration – but it left us some major challenges!

• How do you store data forever?• Short-ish term archiving of bulk scientific data

• Also the point at which “Grid” enter our joint worlds…

What led these two universes to collide? What have we learnt? What are the messages for the future?

3

How long is “forever”?

• I’ve been waiting (in this traffic, at this airport, 4U) forever

• I’ve been in this job forever…

• Q: “How long have you been married?”

• Well, “forever” is actually much longer than any of these…

• (See picture from CSO!)

4

5http://www.damtp.cam.ac.uk/user/gr/public/bb_history.html

Standard Cosmology

Good model from 0.01 secafter Big Bang

Supported by considerable observational evidence

Elementary Particle Physics

From the Standard Model into theunknown: towards energies of1 TeV and beyond: the Terascale

Towards Quantum Gravity

From the unknown into the unknown...

In terms of Oracle releases…

Oracle 5,000,000,000

6

Oracle 5,000,000,002g

UNESCO Information Preservation debate, April 2007 - Jamie.Shiers@cern.ch7

Persistent Archives: Motivation

• Climate data: in an era when climate change is ‘hotly’ debated, the motivations appear clear…

• Medical data: important for understanding issues such as historical pandemics, cross-species diseases etc. Avian flu, HIV, …

• Cosmological data: plays a vital role in our evolving understanding of the Universe – astrophysics community has an explicit policy (data is made public after 1 year – data volume doubles each year)

• Particle Physics data: Similar arguments – will we ever be able to build similar accelerators to those of today? If we ‘lose’ this data, what of our scientific heritage? Need to look at old data for a signal that should have been seen (has happened several times)

UNESCO Information Preservation debate, April 2007 - Jamie.Shiers@cern.ch8

Understanding the bits

• In the mid-1990s, a successful re-analysis of 10-year old data from the JADE collaboration at the PETRA accelerator at DESY was made

• A sub-set of the data was found abandoned in an office corner. The programs to read the data were in an obsolete language and were unusable. The data format was proprietary (but de-codable).

This provided valuable input into the LEP data archive(s)

• Data format: will this be readable in 5 / 10 / 100 years? 1000?• Programs: languages / operating systems / hardware platforms

have very short life-spans wrt an archive• Metadata: essential to understand what the data means

The best solution to date is a so-called ‘Museum system’, but this is still a very short term solution wrt even Einstein, let alone Tyco Brahe, Kepler and Newton…

Way back when… (ECFA report) - 1983

• Sub-group 11: “Databases and Book-keeping for HEP experiments”, ECFA/83/78 (September 83):

To provide a guide to the database and bookkeeping packages used at present by HEP groups;

To find out what future requirements (would) be; To make recommendations as to how these (could) best

be met.

• Definition of a “data base”:

”A collection of stored operational data used by the application system of some particular enterprise.”

9

(Some) ECFA recommendations

1. There would be many advantages in using commercially available DBMSs in HEP to reduce the amount of work required to obtain a database or bookkeeping system tailored to the needs of a particular experiment. They will clearly have a place in HEP computing in the future and should be used for LEP experiments in place of complex user-written systems;

4. Greater awareness is needed within the HEP community of what DBMSs offer. Pilot projects should be set up so that some experience can be obtained as soon as possible;

5. There is an immediate need for the major HEP computing centres, especially CERN, to make suitable relational DBMSs (e.g. SQL/DS or Oracle) available to users;

6. Simple HEP database packages will continue to be needed, especially in the short term.

10

Computing at CERN in the 1990s…

Green book: where DBs were in use (or would make sense):

Collaboration address lists; Electronic mail addresses; Experiment bookkeeping; Online databases; Detector geometry description databases; Calibration constants; Event data; Bookkeeping of program versions; Histograms and other physics results; Software documentation; Publication lists;

Other applications(?)

The report also made concrete recommendations about training and support (2 pages – not 2 words)12

Databases for HEP Panel at CHEP 1992

1. Should we buy or build database systems for our calibration and book-keeping needs?

2. Will database technology advance sufficiently in the next 8 to 10 years to be able to provide byte-level access to petabytes of SSC/LHC data?

Drew Baden, University of Maryland (PASS Project);

B. Linder, Oracle Corporation;Richard Mount, Caltech;Jamie Shiers, CERN.

13

Buy or build? – Calibration & Bookkeeping

1. Is it technically possible to use a commercial system?

2. Would it be manageable administratively and financially?

Questions already investigated during LEP planning phase

• Computer Physics Communications 45 (1987) 299-310

• “Possibility” in 1984 (technically but not Q2);

• “Probability” in 1992 (technically and conceivably also Q2).

14

Arguments - 1992

• A significant amount of HEP-specific code would need to be added – roughly comparable in size (within a factor of 2) of existing home-grown solutions

• Commercial systems require well trained support staff at every site. (Home-grown too, but experience shows that this is typically much less than for commercial solutions.)

• Licensing and support for large, diverse HEP collaborations clearly a concern

We shall revisit these questions later in the show…

15

2000: The Story So Far…

• Early 1980s: value of using an RDBMS for physics(-related) data identified

• Mid-1980s: first use of Oracle for such applications at CERN

• Early 1990s: proposals to use databases for all data

• Mid-1990s: successful use of (O)DBMS at >>TB level -> PB

• Late 1990s: decision to migrate away from ODBMS: alternative?

16

15. Observations – IT “Eloise” Retreat 2000

• Large volume event data storage and retrieval is a complex problem that the particle physics community has had to face for decades.

• The LHC data presents a particularly acute problem in the cataloguing and sparse retrieval domains, as the number of recorded events is very large and the signal to background ratios are very small. All currently proposed solutions involve the use of a database in one way or another.

• …• Major relational database vendors have announced support for

Object-Relational databases, including C++ bindings.• Potentially this could fulfil the requirements for physics data

persistency using a mainstream product from an established company.

CERN already runs a large Oracle relational database service

17

Eloise Recommendation

• The conclusion of the Espresso project, that a HEP-developed object database solution for the storage of event data would require more resources than available, should be announced to the user community.

The possibility of a joint project between Oracle and CERN should be explored to allow participation in the Oracle 9i ß test with the goals of evaluating this product as a potential fallback solution and providing timely feedback on physics-style requirements. Non-staff human resources should be identified such that there is no impact on current production services for Oracle and Objectivity.

18Fellow, later also openlab resources

UNESCO Information Preservation debate, April 2007 - Jamie.Shiers@cern.ch19

2000: Preserving the bits

• Lifetimes of Particle Physics experiments are extremely long! Currently measured in decades…

• Ironically, one of the solutions proposed for the LEP data archive (the then-current proposal for the LHC) was later abandoned (technical / commercial reasons)

• This necessitated a ‘triple migration’: Of 300TB of data between storage media; Of the same data from one data format to another; Of the accompanying processing codes.

• In the end, the exercise took around 2 months per 100TB of data migrated, as well as a significant amount of effort (~1 FTE / 100TB) and hardware resources

Enter The Grid

The Solution to LHC ComputingThe Solution to LHC Computing

2121Джейми Шиерс Ноябрь 2004 г.Научные и корпоративные grid-

инфраструктуры

Физические изыскания - перспективы

• Реинжениринг всех сервисов СУБД для физической науки на базе

Oracle 10g RACOracle 10g RAC

• Цели:– Изолирование – ‘сервисы’ 10g и / или физическое разделение– Масштабируемость - как для вычислительной мощности

для обработки БД, так и для устройств хранения– Надежность – автоматический обход сбоя в случае проблем– Управляемость – упрощение процессов администрирования

• Вернемся к этому вопросу позже, в разделе ‘Enterprise Grids’ …

Physics Activities – Futures (2004)

• Re-engineering all DB services for Physics on

Oracle 10g RACOracle 10g RAC

• Goals are:• Isolation – 10g ‘services’ and / or physical separation• Scalability – in both database processing power and storage • Reliability – automatic failover in case of problems• Manageability – significantly easier to administer than

now

• Will revisit this under ‘Enterprise Grids’ later…

22Oracle Grid Tech. day, Moscow

CERN & Oracle• Share a common vision regarding the future of high

performance computing• Wide spread use of commodity dual processor PCs running Linux;• Focus on Grid computing

• CERN has managed to influence Oracle product

Oracle 10g features:

• Support for native IEEE floats & doubles;

Support for “Ultra large” Databases (ULDB); 16 bit fields again!

• Cross-platform transportable tablespaces;

• Instant-client developer etc.

Oracle Grid Tech. day, Moscow

What are Grids all about?• Grids are about sharing and pooling of resources

• We all know that when we can and do work together we achieve much more than if we work alone• CERN is a classic example of this at a world-wide scale!

Two other examples (from CHEP ’04 in Interlaken, CH)

1. Resilience• Security of valuable data• Continuity in case of major disruption

2. Expedience• Access to additional resources• Engagement of distributed communities

The Grid – Disruptive Technology?

• From OracleWorld San Francisco press panel Sep 2003:

[ Work on LHC Computing started around 1992 ]

• “What would happen if something came along that would change everything? Like the Web. We would simply have to take it into account”

• “We believe that thing has come along, and that thing is the grid.”

• “We are actively involved in making it happen, and it is the underlying cornerstone of our computing model”

26

Grids & Robust Services

• 100% of my Grid experience relates to the deployment and delivery of Production Services

• This started already in the days of EDG with the Replica Location Service and its deployment at CERN and some key (WLCG) Tier1 sites

• In the then-current WLCG Computing Model, the EDG-RLS was a critical component which, if unavailable, meant:

Running jobs could not access existing data Scheduling of jobs at sites where the needed data was

located was not possible

The Grid – if not down – was at least seriously impaired…

• This was taken into account when designing the service deployment strategy & procedures – a taste of things to come!

And the EGI Added Value?

• In order to be both attractive and maintainable, Grids need to have the following attributes:1. Low cost of entry;2. Low cost of ownership.both in terms operations as well as application and user support

• The basic principles of reliability and usability must be designed in from the start – adding them later is not consistent with the goals of low cost of ownership.

• The “win-win” situation of low cost of entry & ownership is a convincing argument for funding bodies, application communities and Grid service providers alike

27

EGI Preparation Meeting, Munich, March 19 2007 - Jamie.Shiers@cern.ch28

Advantages of Robust Services

• The advantages of such an approach are simply huge!

The users see a greatly improved service

Service providers have significantly more flexibility in scheduling interventions

The service provider – user relationship is enhanced

Everyone’s stress levels plummet!

But it must be supported by the middleware…

Experiments’ Definitions of “Critical”

• All of the services ranked by the experiments as “most critical” rely on databases directly or indirectly (data / storage management services)

• 30’ downtime can only be addressed by careful design and extensive automation – and powerful techniques!

29

Experiment Down Seriously Degraded

Perturbed

ALICE 2 hours 8 hours 12 hours

ATLAS As text As text As text

CMS 30’ 8 hours 24 hours (72)

LHCb 30’ 8 hours 24 hours (72)

The Techniques DNS load balancing Oracle “Real Application Clusters” & DataGuard (next)• H/A Linux (less recommended… because its not really H/A…)

Murphy’s law of Grid Computing!

• Standard operations procedures:• Contact name(s); basic monitoring & alarms; procedures; hardware

matching requirements;

No free lunch! Work must be done right from the start (design) through to operations (much harder to retrofit…)

• Reliable services take less effort(!) to run than unreliable ones!30

Problem Description

• More and more data centers run Oracle databases on commodity hardware relying on:• Software solutions for high availability (RAC, ASM)• Hardware redundancy

• Using commodity hardware may impose relatively frequent hardware changes due to:• Short hardware lifetime• Short support period

Replacing database hardware without significantlycompromising service availability becomes a challengeas database systems grow larger and larger.

31

And the experience…

• DataGuard based migration procedure has been used this year at CERN:

We migrated all production and validation databases ~15 systems in total

We moved from RHEL 3 to RHEL 4 at the same time We also enlarged all production clusters

Downtime associated with the migration did not exceed 1 hour per database

32

Summary• The ECFA and “Green book” reports of the mid-late 1980s

have turned out to be remarkably prescient in terms of the value of databases for the HEP community (but it took some time!)

By and large, all recommendations have been implemented in some form or other

This is all the more remarkable as most people probably didn’t even know these documents existed!

The recommendations concerning training and support have been particularly important!

• From relatively small beginnings in the 1980s, databases now underpin all of the critical services for the LHC experiments

• May collisions commence…33

Th

e E

volu

tion

of

Dat

abas

es in

HE

P

If you want to know more…

Visit http://hepdb.blogspot.com/

And also:

http://wwwasd.web.cern.ch/wwwasd/cernlib/rd45/ http://wwwasd.web.cern.ch/wwwasd/lhc++/indexold.html http://hep-proj-database.web.cern.ch/hep-proj-database/