the glite middleware distribution
Post on 18-Mar-2016
50 Views
Preview:
DESCRIPTION
TRANSCRIPT
EGEE-III INFSO-RI-222667
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite are registered trademarks
The gLite middleware distributionOliver KeebleCERN
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 2
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Overview
• Why is gLite?• What is gLite?• Services and status• Roadmap• Release process• The future
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 3
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Middleware
• What’s so great about being in the middle anyway?
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 4
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Why is gLite?• So we need software to manage the infrastructure, but
why does EGEE produce a distribution?– Simplify availability of services and updates
– Provide integrated 'services‘ from components
– In the absence of standards, promote interoperability
– Identify endorsed projects at certified versions
– Provide an entry point for the uninitiated
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 5
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Where does gLite begin... and end?
• gLite is not just somewhere we stick stuff which might be useful for grids– Adding something to gLite has a cost for the project
• Core services• Relationship with
– Applications
– RESPECT
– Operational Tools
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 6
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
How is gLite put together?
• gLite is…– A distribution
– With its own eponymous development project
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 7
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
gLite 3.1
• The current release is gLite 3.1• It is updated almost every week• Its purpose is to provide a stable platform for production
grid usage• Some highlights
– Data Management
– Workload Management
– Information System
– Security and Authorisation
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 8
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
What’s in there?
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 9
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
What’s in there… continued
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 10
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Running jobs
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 11
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Workload Management
UI
VOMS
CE
WN
BDIIWMS
MyProxy
TORQUE
• Just getting a job started involves a lot of services
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 12
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Data Management
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 13
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
The future• The EGEE/gLite development activity is JRA1• Most of their effort is earmarked for maintenance
– Logging, monitoring, docs
– Address the bugs as found on the EGEE Production and Pre-Production Systems
– Provide support
– Address the short/medium term requests of the applications as decided by the TMB
– Improve the manageability of the middleware in terms of deployability, reliability and usability.
– Provide the internal unit testing of new or modified software
– Participate in the definition of the gLite releases together with the SA3 activity
– Maintain a web page with the relevant information needed by middleware users
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 14
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Authorisation service
• The single biggest chunk of development anticipated for EGEE-III
• What will this bring? Consistency across services Distributed Policy Management Auditing and Logging
• Most components will have to be adapted
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 15
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Authorisation Service
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 16
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
CREAM• The infrastructure currently runs with the lcg-CE• CREAM is the scheduled replacement
– Computing Resource Execution And Management service
– Scalability
– Parameter passing
– Direct submission WS interface job management (inc proxy renewal)
– Improved service administration
– A BES/JSDL interface is planned
• A first release is ready now for production
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 17
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
WMS/ICE
• Job management tasks are delegated to the gLite Workload Management Service
• Can now sustain 30k jobs per day under normal operations
Direct JobSubmission
WMS
Submissionthrough the WMS
CREAM CREAM CREAM
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 18
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Support for pilot jobs• A pilot job is a shell which pulls in the ‘real’ job when its
execution starts• This is now a common way for VOs to manage their
workloads• As it runs something on behalf of another user, the
mapping to local accounts has to be redone; glexec• This mapping must be consistent across a site to
prevent two different users sharing an account; SCAS
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 19
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
glexec and SCAS
SAML2-XACML2 Query Protocol
Site
gLExec-on-WN
LCMAPS
Query for Authorisation decisions on behalf of users
GUMS & SAZ SCAS
LCAS & LCMAPS
Site Central AuthZ Service
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 20
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Encrypted Data Storage
• gLite provides solutions for managing sensitive data• One application is medical data management, where
DICOM is a standard for image acquisition and analysis• gLite will release new services for this
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 21
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The great thing about standards…
PBS/Torque
LSF
Condor
Load Leveler
Sun Grid EngineGRAMv2
ARCCREAM
NAREGI
UnicoreOSG
GRAMv4
Nordugrid
Naregi
DEISA
EGEE
Teragrid
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 22
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Some standards relevant to gLite• Glue2
– Encapsulates the ‘accumulated wisdom’ of a number of years of grid operations Fixes problems
Better model
• e.g. relationships between main entities in compute model have been improved
Buy-in
• BES/JSDL– Proposed as solutions by OGF
– Viability has not been established, investigations are ongoing
• SRM2– The Storage Resource Manager (SRM) is a middleware component whose
function is to provide dynamic space allocation and file management on shared storage components on the Grid.
– The DPM supports SRM2.2
• IGTF / x509
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 23
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Glue 2
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 24
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
How gLite is put together• We allow independent things to remain independent• We make a gLite release almost every week!
Release Day
time
C
Update1
B
Update2
AC
Update3
B
Integration CertificationBuild
Regular release interval
Component A
Component B
Component C
Illustration of
in a component based release process
Update4
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 25
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Patch States
Ready For Certification
In CertificationCertifiedIn Pre Production
In Production ObsoleteRejected
Open
SA1
SA3
JRA1
In Configuration
In PPS-PDT
With Provider
Configured
[yes][no]
Ready forIntegration
AcceptanceCriteria
ConfigRequired
PassPassPass
OtherUpdate
Available
] Closed States[
[yes]
[yes]
[yes][yes]
[yes]
[no]
[no]
[no][no]
[no]
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 26
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Testing infrastructure• Framework
– SAM, Nagios, Manual Testing• Regression Tests• Extensive use of virtualisation• ETICS
– Move testing as far upstream as possible– Deployment tests– Regression tests– Unit tests
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 27
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
A gLite 3.1 update announcement
This indicates to an administrator which services they should update
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 28
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Multiplatform Support
• gLite currently runs on SL3 and SL4 ‘generations’, including selected services on 64bit
• A release of the WN on SL5 is nearly ready– More to come
• Work continues on Debian 4• Now profiting from earlier work to make the codebase
more portable• Our build programme is being overhauled to be more
multiplatform friendly
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 29
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Challenges for gLite• Release mechanisms
– What is the best way to deliver the software?
• IPv6• Response to virtualisation
– Now the resource can adapt to the application, not the other way around– Distribution of VMs for services or applications– Support for running jobs in VMs
• Retirement and end-of-life for services and platforms• Fight the Complexity
Partially a result of heterogeneity
• Improve Agility
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 30
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
gLite Consortium• EGI as currently proposed does not directly cover
middleware development• The EGEE PMB recommended the construction of a new
organisation capable of guaranteeing the support and evolution of gLite in the EGI timeframe
• Would allow gLite to be properly represented within EGI or a UMD project
• A proposal for the gLite consortium is currently under discussion
The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN 31
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Summary
• gLite is critical in the realisation of one of the principal goals of EGEE; Maintain, enhance and simplify the use of the production quality computing infrastructure for an increasing range of researchers in diverse scientific fields
• gLite is enabling grids for e-science in Europe, across a multitude of scientific disciplines
• While there are plenty of new ideas in the pipeline, the overarching goal is that stated above, and the primary focus is now on consolidation
top related