chep2000 padova, italy

The GIOD Project(Globally Interconnected Object Databases)

For High Energy Physics

Harvey Newman, Julian Bunn, Koen Holtman and Richard Wilkinson

A Joint Project between Caltech (HEP and CACR), CERN and Hewlett Packard

http://pcbunn.cacr.caltech.edu/

CHEP2000 Padova, Italy

7 February 2000 CHEP 2000 - Harvey Newman - GIOD 2

The GIOD Project - Overview

GIOD Project began 1997, a joint effort of Caltech and CERN with funding from Hewlett Packard for two years

with collaboration from FNAL, SDSC

Leveraging existing facilities at Caltech’s Center for Advanced Computing Research (CACR) Exemplar SPP2000, HPSS system, high speed WAN, CACR expertise

Build a prototype LHC data processing and analysis Center using: Object Oriented software, tools and ODBMS

Large scale data storage equipment and software

High bandwidth LAN (campus) and WAN (regional, national, transoceanic) connections

Measure, evaluate and tune the components of the center for LHC data analysis and physics

Confirm the viability of the LHC Computing Models


Components of the GIOD Infrastructure

Supercomputer facilities at CACR

Large pool of fully simulated multi-jet events in CMS

Experienced large-scale systems engineers at CACR

Connections at T3- >OC3 in the Local and Wide Area Networks; Fiberoptic links Caltech HEP/CACR

Strong collaborative ties with CMS, RD45, Fermilab and San Diego Supercomputer Center;CERN, CALREN-2 and Internet2 Network Teams


Generation of CMS multi-jet events

Made possible by 1998, 1999 (NSF-sponsored) NPACI Exemplar allocations

Produced ~1,000,000 fully-simulated multi-jet QCD events since May 98; selected from 2 X 109 pre-selected generated events Directly study Higgs backgrounds for first time Computing power of the HP-Exemplar SPP 2000

(~0.2 TIPs) made this attainable Events used to populate a GIOD Object Database

system “Tag” database implemented and kept separately;

Can be quickly replicated to client machines

In 2000: Proposal to NPACI requesting 25% of the Exemplar has been grantedTargeted at event simulation for ORCA (CMS)

Replicas of this database were installed at FNAL and Padua/INFN (Italy)

Simple “Tag” class


Scalability tests using the Exemplar Caltech Exemplar used as a relatively

convenient testbed for multiple client tests with Objectivity

Two main thrusts: Using simple fixed object data Using simulated LHC events

Results gave support to the viability of the ODBMS system for LHC data

CMS 100 MB/sec milestonemet (170 MB/sec achieved)

0

25

50

75

100

125

150

175

200

0 10 20 30 40 50

Number of Database Clients

Ag

gre

gat

e T

hro

ug

hp

ut

(MB

ytes

/sec

on

d)

Writing to 1 Container Writing to 6 Containers

Up to 240 clients reading simple objects from the database

> 170 MB/sec writing LHC raw event data to the database


Attaches to the GIOD database: allows to scan all events in the database, at multiple detail levels

Demonstrated at the Internet2 meetings in 1998 and 1999, and at SuperComputing’98 in Florida at the iGrid, NPACI and CACR stands

Java 3D Applet to view GIOD events

HCAL towers

ECAL crystals

Reconstructed Tracks

Reconstructed Jets

Java2 GUI

Tracker geometry and hitmap

Run/event selection widget


Other ODBMS tests

0

2 0 0 0

4 0 0 0

6 0 0 0

8 0 0 0

1 0 0 0 0

1 2 0 0 0

1 4 0 0 0

1 6 0 0 0

1 8 0 0 0

2 0 0 0 0

0 5 0 1 0 0 1 5 0 2 0 0 2 5 0

U p d a t e N u m b e r ( T i m e o f D a y )

mil

ise

co

nd

s

c r e a t e L A Nc r e a t e W A Nc o m m i t L A Nc o m m i t W A N

S a t u r a t e d h o u r s ~ 1 0 k b i t s / s e c o n d U n s a t u r a t e d ~ 1 M b i t s / s e c o n d

DRO WAN Tests with CERN

Production on CERN’s PCSF

and file movement to

Caltech

Objectivity/DB Creation of 32000

database federation

Tests with Versant(fallback ODBMS)


Tests with Objy/Java binding and JAS

Java2D Tracker viewer

Java Track Fitter

Objy DIM and analysis using Java Analysis

Studio


WAN tests: Caltech SDSC,FNAL Client tests between SDSC/CACR,

CACR/FNAL and CACR/HEP ftp, LHC event reconstruction,

event analysis, event scanning

Investigated network throughput dependence on: TCP window size, MSS, round trip

time (RTT), etc. payload (ftp, Objy, Web, telnet

etc.)

Objectivity Schema transfer8 kB DB Pages

Flattened by staggering client

startups

Simple ftp traffic


WAN tests Caltech SDSC,FNAL

Using “out of the box” single-stream ftp, achieved ~7 MB/sec over LAN ATM @ OC3 ~3 MB/sec over WAN @ OC3

Expect to ramp up capability by use of Tuned ftp (buffer, packet and window sizes) Jumbo frames New IP implementations or other protocols

Predict ~1 GB/sec in WAN by LHC 2005 using parallel streams

Measurements to be used as a basis for model parametersin further MONARC simulations


Using the Globus Tools Tests with “gsiftp”, a modified ftp

server/client that allows control of the TCP buffer size

Transfers of Objy database files from the Exemplar to Itself An O2K at Argonne (via

CalREN2 and Abilene) A Linux machine at INFN (via

US-CERN Transatlantic link)

Target /dev/null in multiple streams (1 to 16 parallel gsiftp sesssions).

Aggregate throughput as a function of number of streams and send/receive buffer sizes

gsiftp rate as a function of Buffer Size (single stream over HiPPI)

0

5000

10000

15000

20000

25000

30000

0 500 1000 1500 2000 2500

Buffer Size (kBytes)

Ra

te (

kB

yte

s/s

ec

on

d)

gsiftp rate as a function of Buffer Size (single stream to Argonne)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 500 1000 1500 2000 2500

Buffer Size (kBytes)

Ra

te (

kB

yte

s/s

ec

on

d)

gsiftp Aggregate rate to Argonne as function of the number of parallel streams

0

500

1000

1500

2000

2500

0 2 4 6 8 10 12 14 16 18

Number of parallel streams

Rat

e (k

Byt

es/s

ec)

~25 MB/sec on HiPPI loop-back

~4MB/sec to Argonne by tuning TCP window size

Saturating available B/W to

Argonne


GIOD - Summary

GIOD investigated

Usability, scalability, portability of Object Oriented LHC codes

In a hierarchy of large-servers, and medium/small client machines

With fast LAN and WAN connections

Using realistic raw and reconstructed LHC event data

GIOD has

Constructed a large set of fully simulated events and used these to create a large OO database

Learned how to create large database federations

Developed prototype reconstruction and analysis codes that work with persistent objects

Deployed facilities and database federations as testbeds for Computing Model studies


Associated Projects MONARC - Models Of Networked Analysis at Regional Centers (CERN)

Caltech, CERN, FNAL, Heidelberg, INFN, KEK, Marseilles, Munich, Orsay, Oxford, Tufts, …

Specify candidate model’s performance: throughputs, latencies Find feasible models for LHC matched to network capacity and data handling Develop “Baseline Models” in the “feasible” category

PPDG - Particle Physics Data Grid (DoE Next Generation Internet) Argonne Natl. Lab., Caltech, Lawrence Berkeley Lab., Stanford Linear

Accelerator Center, Thomas Jefferson National Accelerator Facility, University of Wisconsin, Brookhaven Natl. Lab., Fermi Natl. Lab., San Diego Supercomputer Center

Delivery of infrastructure for widely distributed analysis of particle physics data at multi-PetaByte scales by 100s to 1000s of physicists

Acceleration of development of network and middleware infrastructure aimed at data-intensive collaborative science.

ALDAP - Accessing Large Data Archives in Astronomy and Particle Physics (NSF Knowledge Discovery Initiative) Caltech, Johns Hopkins University, FNAL Explore data structures, physical data storage hierarchies for archival of next

generation astronomy and particle physics data Develop spatial indexes, novel data organisations, distribution and delivery

strategies. Create prototype data query execution systems using autonomous agent

workers


Future Directions: GIOD II

Review the advantages of ODBMS vs. (O)RDBMS for persistent LHC data;in light of recent (e.g. Web-enabled) RDBMS developments, for HEPand other scientific fields

Fast traversal of complex class hierarchies ?

Global (“federation”) schema and transparent access ?

Impedance match between the database and the OO code ?

What are the scalability and use issues associated with implementing a traditional RDBMS as a persistent object store for LHC data?

What benefits would the use of an RDBMS bring, if any ?

Which RDBMS systems, if any, are capable of supporting, or projected to support, the size, distribution and access patterns of the LHC data ?


GIOD II : Other New Investigations

What are the implications/benefits for the Globally-distributed LHC computing systems of: Having Web-like object caching and delivery mechanisms

(distributed content delivery, distributed cache management)

The use of Autonomous Agent query systems

Organizing the data and resources in an N-tiered hierarchy

Choosing (de facto) standard Grid tools as middleware

How can data migration flexibility be built in ? Schema/data to XML conversion (Wisdom, Goblin) ?

Data interchange using JDBC or ODBC

Known format binary files for bulk data interchange:for simple and efficient transport across WANs


GIOD II and ALDAP Optimizing performance of Objectivity for LHC/SDSS data

Use of Self Organizing Maps (e.g. Kohonen) to recluster frequently accessed data into collections in contiguous storage

Use of Autonomous Agents to carry queries and data in WAN distributed database system

Identify known performance issues: get them fixed by the vendor

Example 1: 11,000 cycles cf. 300 cycles overhead to open an object Example 2: Selection speeds with simple cuts on Tag objects

Make new performance comparisons between Objectivity and ER database (SQLServer)

on identical platforms, with identical data, with identical queries, with all recommended “tweaks”, with all recommended coding tricks

We have begun tests with SDSS sky objects, and with GIOD”Tag” objects


GIOD II and PPDG Distributed Analysis of ORCA data

Using Grid middleware (notably gsiftp, SRB) to move database files across the WAN

Custom tools to select subset of database files required in local “replica” federations, and attach them once copied

Making “compact” data collections

Remote requests from clients for sets of DB files Simple staging schemes that asynchronously make data available,

and give ETA for delivery, and migrate “cool” files to tertiary storage

Marshalling of distributed resources to achieve production task goals Complementary ORCA DB files in Caltech, FNAL and CERN replicas Full pass analysis involves distributing task to all three sites

Move/compute cost decisionTask and results carried by Autonomous Agents between sites

(work in ALDAP)

chep2000 padova, italy

Documents

giod database

giod eventsattaches

lhc data analysis

lhc datacms

prototype lhc data processing

analysis center

lhc computing modelschep

simulated multijet events