Metadata: ARDA Implementation

PrototypeValidate our ideas and expose a concrete example to interested partiesMultiple back endsCurrently: Oracle, PostgreSQL, SQLite, MySQLDual front endsTCP Streaming Chosen for performanceSOAPFormal requirement of EGEECompare SOAP with TCP StreamingAlso implemented as standalone Python libraryData stored on the file system

Massimo Lamanna / CERN

Python Interpreter

Metadata Python API

Client

filesystem

Database

MDServer

Metadata Server

SOAP

TCP Streaming

Oracle

PostgreSQL

SQLite

Client

Client

INFSO-RI-508833

NA4 biomedical gLite tests on the PPSPPS components tested: Weekly testing of WMS, FiReMan, security related components (ACLs for file access), R-GMAOther gLite testingSite installation (v1.0, 1.1, 1.2 and 1.3)Advanced data management on prototype testbedMedical ImagerTrigger:Retrieve DICOM files from imager.Register file in FiremangLite EDS client for data encryptionRegister Metadata in AMGAClient Library:Lookup file through Metadata (AMGA)Use gLite EDS client for data decryption Serve it up to the application

INFSO-RI-508833

All gLite Services are available and tested on GILDAgLite Services

INFSO-RI-508833

2. The capture of requirements, and the assessment of middlewareThe status at PM18

A database of requirements has been constructed by the Project Technical Forum(PTF)This database already contains >400 requirementsThis database is populated by the scientific communities deploying applications on EGEE and by the FP6 projects (eg Diligent, Seegrid,...)

The capture of requirements from FP5 projects

Several FP5 projects were granted access to the database (Grace, Mammogrid)But it has proved to be difficult to collect their requirementsNo formalization of requirementsProjects finishedRequirements were collected mainly from the scientific communities (Crossgrid, Datagrid)

The assessment of middleware w.r.t requirements

There is on-going work with JRA1 within the framework of the PTF and the TCG

INFSO-RI-508833

3. The clarifification of the true motivation of usersAn improved Memorandum of Understanding(MOU) between EGEE and the scientific communities has been designed and implemented

Detailed SA1 questionnaire to evaluate the required resourcesRegular phone conferences with the Operation Advisory Group

An improved application integration process is under design

Creation of lightweight recognised VOs vs supported Vos

Only supported VOs will be asked to write an MOU

Decentralised integration vs centralised EGAAP-like approach

Initial deployment will be at regional level

INFSO-RI-508833

The status of the MoUs with the generic supported applications

INFSO-RI-508833

The status of generic applications deployment on EGEEApplications accepted by the Project Executive Board(PEB)

Earth Science Research (Earth Observation, Hydrology, Climate)Geophysics (Industry)Computational ChemistryAstrophysics (MAGIC and Planck collaborations) Finance (EGRID)

New Applications recommended to the PEB by the EGEE Generic Applications Advisory Panel (EGAAP)

Fusion (ITER)ArchaeologyEC projects (EELA, EUMEDGRID, EUCHINAGRID, BIOINFOGRID)

INFSO-RI-508833

4. The evaluation of user satisfactionFirst user survey at PM1534 users filled a questionnaire available onlineResults available in DNA4.3.2User satisfaction is good for user supportUser satisfaction on grid services is intermediateUsers would like improvements in documentation (with the exception of system manuals which are well appreciated)Action: creation of the User information Group to address issues related to documentation in the project Next user survey at PM21

INFSO-RI-508833

5. Demonstration of the scientific benefits of the EGEE gridThe number of users in the VOs related to NA4 activity has continued to grow

from ~500 at PM9 to ~1000 at PM18 More than 20 applications are deployed on the production infrastructure

WE HAVE DEMONSTRATED

added value in terms of deployment scale

The very large scale production of simulated data for HEP experiments

In particular this has been essential for the commissioning of the LHC experiments

added value in terms of response time

The computation of the seismic epicentre by ESRThe search for new drugs: large scale in silico docking (WISDOM) - access to large scale resources gives much improved turnaround

added value in terms of data distribution

The distribution of LHC data on the grid, and the large users activity (~150k jobs/month in the CMS case)

INFSO-RI-508833

Large Scale Production on EGEE by several HEP experiments Fundamental activity in the preparation of the LHC experiments start up

Physics and detector studiesComputing systems commissioning

Examples:

ATLAS: Ran on 3 grids (2/3 of work on EGEE) Peaks of 10,000 jobs per day on EGEE200 CPU years in 2005 ; 45 TB dataComprehensive analysis: see S.Campana et al., e-Science 2005, Melbourne, Australia

LHCb:Peaks of 5,500 jobs/day on EGEE700 CPU/years in 2005 ; 77 TB of data

A lot of activity within non-LHC experiments (BaBar,CDF, D, ZEUS,H1)

More details in DNA4.3.2ATLAS production evolution 2004-2005

INFSO-RI-508833

First biomedical data challenge: World-wide In Silico Docking On Malaria (WISDOM)

Significant biological parameterstwo different molecular docking applications (Autodock and FlexX)about one million virtual ligands selectedtarget proteins from the parasite responsible for malariaSignificant numbers Total of about 46 million ligands docked in 6 weeks1TB of data produced Up 1000 computers in 15 countries used simultaneously corresponding to about 80 CPU yearsAverage crunching factor ~600WISDOM open dayDecember 16th, 2005, Bonn (Germany)

Discuss Data Challenge resultsPrepare next steps towards a malaria Grid (EGEE-II, Embrace, Bioinfogrid)Information: http://wisdom.eu-egee.fr

Graph2

597

1072

383

715

5122

7580

2004

263

3687

3356

1877

337

218

827

8106

Domain distribution of Flexx run jobs

flexx

flexx

479

62

56

659

55

283

1072

662

129

151

383

1877

189

56

1759

766

874

1014

811

612

2488

2600

1361

382

133

74

460

97

125

282

644

9

102

2690

1347

4940

3

337

666

715

56

263

827

380

2763

218

233

nb of run jobs

CE distribution for Flexx run jobs

autodock

479

62

56

659

55

283

1072

662

129

151

383

1877

189

56

1759

766

874

1014

811

612

5088

1361

382

133

74

460

97

125

282

644

9

102

2690

1347

4940

3

337

666

715

56

263

827

380

2763

218

233

Site distribution for DC run jobs (Flexx)

paysFlexx

paysFlexx

229

14

286

9

17

174

301

603

757

141

230

1082

143

13

699

108

418

708

436

255

912

1136

445

201

483

5

45

107

9

324

165

59

703

1080

585

1716

414

119

470

128

35

250

465

221

854

17

83

842

nb of run jobs

CE distribution for Autodock run jobs

paysautodock

229

14

286

9

17

174

301

603

757

141

230

1082

143

13

699

108

418

708

436

255

2048

445

201

483

5

45

107

9

324

165

59

703

1080

585

1716

414

119

470

128

35

250

465

221

854

17

83

842

Site distribution of DC Biomed run jobs (Autodock)

data

paysnb

bg479

bg62

bg56

Total bg597bg597

com1072com1072

Total com1072com1072

cy383cy383

Total cy383cy383

de715de715

Total de715de715

es283es283

es662es662

es151es151

es874es874

es9es9

es380es380

es2763es2763

Total es5122es5122

fr55fr55

fr1014fr1014

fr811fr811

fr612fr612

fr2488fr2488

fr2600fr2600

Total fr7580fr7580

gr189gr189

gr56gr56

gr1759gr1759

Total gr2004gr2004

il263il263

Total il263il263

it129it129

it1361it1361

it382it382

it133it133

it74it74

it460it460

it97it97

it125it125

it282it282

it644it644

Total it3687it3687

nl2690nl2690

nl666nl666

Total nl3356nl3356

pl1877pl1877

Total pl1877pl1877

ro337ro337

Total ro337ro337

ru218ru218

Total ru218ru218

tw827tw827

Total tw827tw827

uk659uk659

uk766uk766

uk102uk102

uk1347uk1347

uk4940uk4940

uk3uk3

uk56uk56

uk233uk233

Total uk8106uk8106

Total36144

data

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Domain distribution of Flexx run jobs

AUTODOCKpaysnb

bg229

bg14

Total bg243bg243

com301com301

Total com301com301

cy230cy230

Total cy230cy230

de128de128

Total de128de128

es174es174

es603es603

es141es141

es418es418

es59es59

es221es221

es854es854

Total es2470es2470

fr17fr17

fr708fr708

fr436fr436

fr255fr255

fr912fr912

fr1136fr1136

Total fr3464fr3464

gr143gr143

gr13gr13

gr699gr699

Total gr855gr855

il250il250

Total il250il250

it757it757

it445it445

it201it201

it483it483

it5it5

it45it45

it107it107

it9it9

it324it324

it165it165

Total it2541it2541

nl1080nl1080

nl470nl470

Total nl1550nl1550

pl1082pl1082

Total pl1082pl1082

ro119ro119

Total ro119ro119

ru17ru17

ru83ru83

Total ru100ru100

tw465tw465

Total tw465tw465

uk286uk286

uk9uk9

uk108uk108

uk703uk703

uk585uk585

uk1716uk1716

uk414uk414

uk35uk35

uk842uk842

Total uk4698uk4698

Total18496

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Domain distribution of Autodock run jobs

SiteCEdomainrun jobs

-1BG01-IPPce001.grid.bas.bgbg229AUTODOCKBG01-IPP229

-1BG02-IMce001.imbm.bas.bgbg14BG02-IM14

-1BHAM-LCG2epgce1.ph.bham.ac.ukuk286BHAM-LCG2286

-1CAVENDISH-LCG2serv03.hep.phy.cam.ac.ukuk9CAVENDISH-LCG29

-1CEA-DAPNIA-SACLAYnode03.datagrid.cea.frfr17CEA-DAPNIA-SACLAY17

-1CESGA-EGEEce2.egee.cesga.eses174CESGA-EGEE174

-1CGG-LCG2ce1.egee.fr.cgg.comcom301CGG-LCG2301

-1CIEMAT-LCG2lcg02.ciemat.eses603CIEMAT-LCG2603

-1CNAF-T1ce01-lcg.cr.cnaf.infn.itit757CNAF-T1757

-1CNB-LCG2mallarme.cnb.uam.eses141CNB-LCG2141

-1CY01-LCG2ce101.grid.ucy.ac.cycy230CY01-LCG2230

-1CYFRONET-LCG2zeus02.cyf-kr.edu.plpl1082CYFRONET-LCG21082

-1GR-01-AUTHnode001.grid.auth.grgr143GR-01-AUTH143

-1GR-04-FORTH-ICSgrid001.ics.forth.grgr13GR-04-FORTH-ICS13

-1HG-01-GRNETce01.isabella.grnet.grgr699HG-01-GRNET699

-1IC-LCG2gw39.hep.ph.ic.ac.ukuk108IC-LCG2108

-1IFCA-LCG2egeece.ifca.org.eses418IFCA-LCG2418

-1IN2P3-CCcclcgceli01.in2p3.frfr708IN2P3-CC708

-1IN2P3-CPPMmarseillece01.mrs.grid.cnrs.frfr436IN2P3-CPPM436

-1IN2P3-LALgrid10.lal.in2p3.frfr255IN2P3-LAL255

-1IN2P3-LPCclrce01.in2p3.frfr912IN2P3-LPC2048

-1IN2P3-LPCclrce02.in2p3.frfr1136INFN-BARI445

-1INFN-BARIgridba2.ba.infn.itit445INFN-CAGLIARI201

-1INFN-CAGLIARIgrid002.ca.infn.itit201INFN-CATANIA483

-1INFN-CATANIAgrid012.ct.infn.itit483INFN-CNAF5

-1INFN-CNAFgridit-ce-001.cnaf.infn.itit5INFN-FERRARA45

-1INFN-FERRARAgrid0.fe.infn.itit45INFN-NAPOLI107

-1INFN-NAPOLIgriditce01.na.infn.itit107INFN-PADOVA9

-1INFN-PADOVAprod-ce-01.pd.infn.itit9INFN-ROMA1324

-1INFN-ROMA1t2-ce-01.roma1.infn.itit324INFN-ROMA1-VIRGO165

-1INFN-ROMA1-VIRGOvirgo-ce.roma1.infn.itit165INTA-CAB59

-1INTA-CABce00.inta.eses59ManHEP-LCG2703

-1ManHEP-LCG2bohr0001.tier2.hep.man.ac.ukuk703NIKHEF-ELPROD1080

-1NIKHEF-ELPRODtbn20.nikhef.nlnl1080OXFORD-01-LCG2585

-1OXFORD-01-LCG2t2ce02.physics.ox.ac.ukuk585RAL-LCG21716

-1RAL-LCG2lcgce01.gridpp.rl.ac.ukuk1716RALPP-LCG414

-1RALPP-LCGheplnx201.pp.rl.ac.ukuk414ROGRID-ICI119

-1ROGRID-ICItestbed001.grid.ici.roro119SARA-LCG2470

-1SARA-LCG2mu6.matrix.sara.nlnl470SCAI128

-1SCAIscaicl0.scai.fraunhofer.dede128ScotGRID-Edinburgh35

-1ScotGRID-Edinburghce.epcc.ed.ac.ukuk35TAU-LCG2250

-1TAU-LCG2lcfgng.cs.tau.ac.ilil250Taiwan-LCG2465

-1Taiwan-LCG2lcg00125.grid.sinica.edu.twtw465UPV-GRyCAP221

-1UPV-GRyCAPramses.dsic.upv.eses221pic854

-1picce01.pic.eses854ru-Moscow-SINP-LCG217

-1ru-Moscow-SINP-LCG2lcg06.sinp.msu.ruru17ru-PSN-LCG283

-1ru-PSN-LCG2lcgce.psn.ruru83scotgrid-gla842

-1scotgrid-glace1-gla.scotgrid.ac.ukuk842

1BG01-IPPce001.grid.bas.bgbg479FLEXXBG01-IPP479

1BG02-IMce001.imbm.bas.bgbg62BG02-IM62

1BG04-ACADce01.grid.acad.bgbg56BG04-ACAD56

1BHAM-LCG2epgce1.ph.bham.ac.ukuk659BHAM-LCG2659

1CEA-DAPNIA-SACLAYnode03.datagrid.cea.frfr55CEA-DAPNIA-SACLAY55

1CESGA-EGEEce2.egee.cesga.eses283CESGA-EGEE283

1CGG-LCG2ce1.egee.fr.cgg.comcom1072CGG-LCG21072

1CIEMAT-LCG2lcg02.ciemat.eses662CIEMAT-LCG2662

1CNAF-T1ce01-lcg.cr.cnaf.infn.itit129CNAF-T1129

1CNB-LCG2mallarme.cnb.uam.eses151CNB-LCG2151

1CY01-LCG2ce101.grid.ucy.ac.cycy383CY01-LCG2383

1CYFRONET-LCG2zeus02.cyf-kr.edu.plpl1877CYFRONET-LCG21877

1GR-01-AUTHnode001.grid.auth.grgr189GR-01-AUTH189

1GR-04-FORTH-ICSgrid001.ics.forth.grgr56GR-04-FORTH-ICS56

1HG-01-GRNETce01.isabella.grnet.grgr1759HG-01-GRNET1759

1IC-LCG2gw39.hep.ph.ic.ac.ukuk766IC-LCG2766

1IFCA-LCG2egeece.ifca.org.eses874IFCA-LCG2874

1IN2P3-CCcclcgceli01.in2p3.frfr1014IN2P3-CC1014

1IN2P3-CPPMmarseillece01.mrs.grid.cnrs.frfr811IN2P3-CPPM811

1IN2P3-LALgrid10.lal.in2p3.frfr612IN2P3-LAL612

1IN2P3-LPCclrce01.in2p3.frfr2488IN2P3-LPC50885088

1IN2P3-LPCclrce02.in2p3.frfr2600INFN-BARI1361

1INFN-BARIgridba2.ba.infn.itit1361INFN-CAGLIARI382

1INFN-CAGLIARIgrid002.ca.infn.itit382INFN-CATANIA133

1INFN-CATANIAgrid012.ct.infn.itit133INFN-CNAF74

1INFN-CNAFgridit-ce-001.cnaf.infn.itit74INFN-FERRARA460

1INFN-FERRARAgrid0.fe.infn.itit460INFN-NAPOLI97

1INFN-NAPOLIgriditce01.na.infn.itit97INFN-PADOVA125

1INFN-PADOVAprod-ce-01.pd.infn.itit125INFN-ROMA1282

1INFN-ROMA1t2-ce-01.roma1.infn.itit282INFN-ROMA1-VIRGO644

1INFN-ROMA1-VIRGOvirgo-ce.roma1.infn.itit644INTA-CAB9

1INTA-CABce00.inta.eses9ManHEP-LCG2102

1ManHEP-LCG2bohr0001.tier2.hep.man.ac.ukuk102NIKHEF-ELPROD2690

1NIKHEF-ELPRODtbn20.nikhef.nlnl2690OXFORD-01-LCG21347

1OXFORD-01-LCG2t2ce02.physics.ox.ac.ukuk1347RAL-LCG24940

1RAL-LCG2lcgce01.gridpp.rl.ac.ukuk4940RALPP-LCG3

1RALPP-LCGheplnx201.pp.rl.ac.ukuk3ROGRID-ICI337

1ROGRID-ICItestbed001.grid.ici.roro337SARA-LCG2666

1SARA-LCG2mu6.matrix.sara.nlnl666SCAI715

1SCAIscaicl0.scai.fraunhofer.dede715ScotGRID-Edinburgh56

1ScotGRID-Edinburghce.epcc.ed.ac.ukuk56TAU-LCG2263

1TAU-LCG2lcfgng.cs.tau.ac.ilil263Taiwan-LCG2827

1Taiwan-LCG2lcg00125.grid.sinica.edu.twtw827UPV-GRyCAP380

1UPV-GRyCAPramses.dsic.upv.eses380pic2763

1picce01.pic.eses2763ru-PSN-LCG2218

1ru-PSN-LCG2lcgce.psn.ruru218scotgrid-gla233

1scotgrid-glace1-gla.scotgrid.ac.ukuk233

INFSO-RI-508833

Example of a generic applications use of EGEE (ESR) OzonemapsEarthquakes epicentre determination

INFSO-RI-508833

Conclusions and a forward lookAchievements

The number of users in the VOs related to NA4 activity has doubled

The usage of the grid by the pilot applications has significantly evolved into large scale production

EGAAP applications have also achieved production status on EGEE (ESR,MAGIC)

The migration of several existing applications to gLite has been achieved

A detailed MOU procedure has been defined and executed for several selected applications

The first User Survey has been performed Issues being actively addressed

Reduction of the amount of skilled effort required by the VOs for production use of the grid (EGEE is improving middleware, monitoring, user interfaces and user support)

The availability, accessibility and quality of user documentation (an inter-activity group is working on this)

Looking forwards : User Forum, EGEE-II

INFSO-RI-508833

EGEE first User ForumDates: March 1-3 2006Location: CERN, SwitzerlandTarget attendance: 150 participantsGoalsObtain a consistent understanding across the EGEE related projects of their expectations, present status and possible evolutionLearn how to improve EGEE quality of servicePromote cross-application fertilisationPrepare EGEE-IIParticipation open to external projects and EGEE membersFormat: 3-day workshopPresentations by thematic areas selected by invitation and through a call for contributionsEGEE presentations (integration of new applications, access to resources, status of middleware,...)With a lot of time for discussionURL:http://egee-intranet.web.cern.ch/egee-intranet/User-Forum/index.html

Montrer un slide sur le user survey (evaluation of user satisfaction)Mieux expliciter la user documentation qui manque, Expliquer ce qui est en cours pour repondre au besoin

****ACLs for file access -> which componentChanger ce slide pour mettre les tests sur le PPSMake it explicit why egeode MoU is not finished Adding Diligent

LHCb list of texts to eraseImprove resolution, too heavy, too many images, bad slide

Expliquer pourquoi on a besoin Speak about EGEE-II