edgi brings desktop grids to distributed computing interoperability
Post on 13-Jan-2016
31 Views
Preview:
DESCRIPTION
TRANSCRIPT
The EDGI project receives Community research funding 1
EDGI Brings Desktop Grids To Distributed Computing
Interoperability
Etienne URBAH urbah@lal.in2p3.fr OGF29 Chicago
LAL, Univ Paris-Sud, IN2P3/CNRS, Orsay, France
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 2
• EDGI project : Context– EGEE = Service Grid focused on Secure Data Sharing– DG = Desktop Grid = Loose opportunistic grid using
idle resources– EDGeS project :
Presentation, 3G Bridge, Infrastructure, Gflop/s
• EDGI project : Goal = Grid + Cloud– Bridging Security and Info– Useful Official and De facto Standards– Interfaces : Diagrams – Scope of JSDL + BES
Summary
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 3
EDGI project : Context
Academic Clouds
OGF PGI
EDGeS
EDGI
EDGI
Cluster based service grids (SGs)(EGEE, NDGF, ...)
Supercomputer based SGs(DEISA, ...)
Desktop grids (DGs)(Volunteer DGs – Home computers,
Organizational DGs – Institutional desktops)
Achievements of EDGeS
• Modular design of 3G Bridge with Application Repository, permitting integration between various DGs and SGs
• 3G Bridges in full production for SGs implementing gLite :
– BOINC and XWHEP <--> EGEE,
– OurGrid <--> EELA2.
• ATTIC = Proof of Concept of Data Distribution
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 4
EGEE = Service Grid focused on Secure Data Sharing
Grid User
X509 proxy
VOMS Extensions
VOMS Admin
Manages VO
Site Admin
Manages Site
Accesses Data with VOMS proxy
VOMS Server
SiteStorage Resource
VOMS Proxy =X509 proxy with
VOMS extensions
Trust Anchor = IGTF
Publishes the Root CA Certificates
Interoperability with another Grid Infrastructure is possible only if it uses
IGTF as Trust Anchor
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 5
EGEE = Service Grid focused on Secure Data Sharing
Grid User
X509 proxy
VOMS Extensions
Submits Job with VOMS proxy
Publishes available Resources
Pushes Job
Log
Log
VOMS Admin
Manages VO
Site Admin
Manages Site
Accesses Data with VOMS proxy
Accesses Data with VOMS proxy
Gives Job Status
Gives Accounting and Auditing
VOMS Server
AccountingLogging & Bookkeeping
Meta-scheduler(WMS)
SiteComputing Resource
SiteStorage Resource
Grid Admin
Sends back Output Sandbox Sends back
Output Sandbox
VOMS Proxy =X509 proxy with
VOMS extensions
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 6
DG = Desktop Grid = Loose opportunistic grid using idle resources
WU = Work Unit = Application + Input DataWU = Work Unit = Application + Input Data
Grid User
Submits input data for an application
Requests WU
Sends WU
Application Manager
Certifies Application
Resource Owner(often volunteer)
Owns Resource
Sends back results
Accepts or Refuses an application on his resource
Grid Server withApplicationRepository
Computing Resource
(often Desktop Computer) Sends back results
Currently, for BOINC, both roles of ‘Application Manager’ and ‘Grid User’ are fulfilled by ‘BOINC Project Owners’.
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 8
EDGeS project : Presentation
FP7 project started in 2008 and just ended
• Integrate Service Grids and Desktop Grids
• Enable very large number of computing resources (100K-1M processors)
• Attract new scientific communities
• Provide a Grid application development environment
• Provide application repository and bridges for the execution in the SG-DG system
WLCG (CERN)
EDGeS
gLite(EGEE, EELA,
GridPP)
ARC(NDGF)
Boinc(Berkeley)
XWHEP(INRIA/IN2P3)
OurGrid(Brazil)
Unicore(DEISA)
VDT(OSG)
Operational
Future
Prototype with BES
GT4(NGS)
OperationalFuture
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 9
EDGeS 3G Bridge : Modular Design
EDGeS 3G Bridge
EGEE
EGEE
Sou
rce
Grid
H
andl
er
Inte
rfac
e
JobDatabase
Queue Manager
Tar
get
Grid
P
lugi
n In
terf
ace
BOINC Plugins
(DC-API)
EGEEPlugins
Scheduler
Control path WU Job
BOINCHandlers
EGEEHandlerLCG-CE
for EDGeS
gLite WMS
BOINC Server
UserJob with
VOMS proxy
Work Unit
BOINC Server
gLite WMS
WU
Job
Control path Job WU
Job
WU
XWHEP Plugins XWHEP
Server
BESPlugins
NDGF
DEISA
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 10
EDGeS Infrastructure : Principle of Operation
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 11
EDGeS operational EGEE DG infrastructure
EDGeS VO of EGEE
SZTAKI CE
16 cpus
CNRS / IN2P3 CE
1.600 cpus
CIEMAT CE
20 cpus
BDII VOMS
MyProxy
WMS LB
EGEE Users
EDGeS
BOINC EGEE bridge
EGEE XWHEP bridge
BOINC-based Desktop Grids
SZDG (public)80.000 PCs
IberCivis (public)28.000 PCs
AlmereGrid (public)1.800 PCs
UoW (local)3.100 PCs
EDGeS@ home (public)3.300 PCs
BOINC Project Owners
XWHEP-based Desktop Grids
LAL (local)80 PCs
IN2P3 (public)30 PCs
AlmereGrid (local)10 PCs
XWHEP Users
EGEE BOINC bridge
Application Repository
XWHEP Agent
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 12
EDGeS@Home – GFLOP/s (7 days)
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 13
EDGI project : Goals
Academic Clouds
OGF PGI
EDGeS
EDGI
EDGI
Cluster based service grids (SGs)(EGEE, NDGF, ...)
Supercomputer based SGs(DEISA, ...)
Desktop grids (DGs)(Volunteer DGs – Home computers,
Organizational DGs – Institutional desktops)
• On the model of IberCivis, creation of a self-sustainable EuroCivis
• Integration between DGs and SGs implementing BES (ARC, Unicore)
• Bridge to Academic Clouds for Jobs needing QoS
• Long term support of the EDGI bridges by the NGIs
• Secure Data Distribution using ATTIC
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 14
BOINC EGEE Bridging Security
EGEE
WMS
EDGeS 3G bridge
EGEE Plugin
1 for each (BOINC Project Owner, EGEE VO) pair
Queue Manager & Job DB
BOINC Handler1 for each (BOINC server,
BOINC Project Owner, EGEE VO) triple
WUi+1
WUi+2
WUi+3
Job
Jobi+1
Jobi+2
BOINC Server
Work Unit
BOINC Project Owner
Submission
MyProxy trusting EDGeS
3G bridgeMedium term X509 proxy
Config. file
DN of X509 proxy
Short term X509 proxy
VOMS Server
VOMS extensions
Sou
rce
G
rid
Han
dler
In
terf
ace
Tar
get
Grid
P
lugi
n In
terf
ace
BOINC jobwrapper client (simulating
a large BOINC computing resource)
3G job-wrapper
3G job-wrapper
VOMS proxy Retriever
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 15
BOINC Server
EGEE
LCG-CE
for EDGeS
EDGeS3G bridge
EGEE BOINC Bridging Info
Gets EXE
Reports resourcesand performance
Checks EXE
Adds jobWatches
job
Gets VOMS proxy
BOINC
Computing
Resource
BOINC plugin (DC-API)
EDGeS
Application
Repository
Information
provider
GRAM Job
Manager
for EDGeSEGEE VOMS
EGEE User
Queue Manager
& Job DB
Generic Job WS Handler
BOINC Service
Watches
Pushes jobSubmits Job
Logs events
Logs events
EGEE
BDII
gLite
WMS
EGEE LB
Sends output
Gets output
Gets Infos
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 16
Grid + Cloud : Useful Official and De facto Standards
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 17
Interfaces for Grid + Cloud : Abstract Diagram
EDGIEDGI
Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability
Author : E. Urbah v1.0 18
Interfaces for Grid + Cloud : Detailed Diagram – Scope of JSDL + BES
top related