crossgrid wp3 task 3.3 grid monitoring trinity college dublin (tcd, ac14 - cr11) brian coghlan,...
Post on 15-Jan-2016
213 views
TRANSCRIPT
CrossGrid CrossGrid WP3WP3
Task 3.3Task 3.3Grid Grid MonitoringMonitoring
Trinity College Dublin (TCD, AC14 - CR11)Trinity College Dublin (TCD, AC14 - CR11)
Brian CoghlanBrian Coghlan, , Stuart KennyStuart Kenny
CYFRONET Academic Computer Centre, Krakow (CYFRONET Academic Computer Centre, Krakow ( CYFRO, CYFRO, CO1)CO1)
Xxx YyyXxx Yyy
ICM, University of Warsaw (ICM, University of Warsaw (ICM, ICM, AC2 - C01)AC2 - C01)
Xxx YyyXxx Yyy
Poznan 28-JAN-2002
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3Task 3.3 Presentation Presentation
Will concentrate on:Will concentrate on:GoalsGoalsTechnology to be developedTechnology to be developedTechnology requiredTechnology required
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3Task 3.3 Target Audience Target Audience
Application-level:Application-level: ProgramsProgramsToolsToolsServicesServices
Poznan 28-JAN-2002
WP3Portals(3.1)
Roaming Access(3.1)
Grid Resource Management
(3.2)
Grid Monitoring
(3.3)
Optimisation of Data Access
(3.4)
Tests and Integration
(3.5)
ApplicationsWP1
End Users
WP1, WP2, WP5TestbedWP4
Performance evaluation tools
(2.4)
WP3WP3
CrossGridCrossGridTechnicalTechnical
AnnexAnnexFig. WP3-1Fig. WP3-1
Poznan 28-JAN-2002 CrossGrid Task 3.3
WP2WP2
Application
source code
Grid Monitoring
(3.3)
Performance
measurement
MPI verification
(2.2)
Benchmarks
(2.3)
Automatic
analysis
Analytical
model
Visualization
Applications (WP1)executing on
Grid testbed (WP4)
Performance analysis (2.4)
CrossGridCrossGridTechnicalTechnical
AnnexAnnexFig. WP2-1Fig. WP2-1
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3Task 3.3
We MUST get it RIGHTWe MUST get it RIGHT
Very central functionVery central function
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3 MilestonesTask 3.3 MilestonesPM 1-3PM 1-3 Definition of requirements Definition of requirements
PM3 : Deliverable D3.1 [ALL]PM3 : Deliverable D3.1 [ALL]
PM 4-6PM 4-6 Designing of architecture, interfaces and security issue Designing of architecture, interfaces and security issue
PM6 : Deliverable D3.2 (report) [ALL]PM6 : Deliverable D3.2 (report) [ALL]
PM 6PM 6 First testbed set-up on selected sites First testbed set-up on selected sites
PM6 : Deliverable D3.2 [ALL]PM6 : Deliverable D3.2 [ALL]
PM 6-12PM 6-12 Implementation of 1 Implementation of 1stst prototype (running on local grid) prototype (running on local grid)
PM12 : Deliverables D3.3 (prototype and report)PM12 : Deliverables D3.3 (prototype and report)
PM 13-24PM 13-24 Second development phase Second development phase
PM13 (beginning) full testbed prototypePM13 (beginning) full testbed prototype
PM 13-15PM 13-15 Testing, feedback collecting Testing, feedback collecting
PM13 : Extraction of raw performance data locally for WP2.4PM13 : Extraction of raw performance data locally for WP2.4
PM 16-18PM 16-18 Refinement, implementation of WP3.3 Refinement, implementation of WP3.3
PM18 : Deliverable D3.5 Extraction of raw performance data on Grid for WP2.4PM18 : Deliverable D3.5 Extraction of raw performance data on Grid for WP2.4
PM 18-24PM 18-24 Refinement, implementation of WP3.3. Refinement, implementation of WP3.3.
??????
PM 16-24 PM 16-24 Refinement, implementation of other tasks within WP3Refinement, implementation of other tasks within WP3
PM24 : Deliverable D3.6 (prototype and report)PM24 : Deliverable D3.6 (prototype and report)
PM 25-33PM 25-33 3 3rdrd development phase development phase
PM33 (end) : Milestone M3.5 Final version of the programming environmentPM33 (end) : Milestone M3.5 Final version of the programming environment
… … and so onand so on
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3Task 3.3 Philosophy Philosophy
‘‘A good Engineer is a person A good Engineer is a person who makes a design that works who makes a design that works
with as few original ideas as with as few original ideas as possible’possible’
Freeman DysonFreeman Dyson[via Yannis Perros][via Yannis Perros]
So, wSo, where possible, reuse ideashere possible, reuse ideas from from::GlobusGlobusGGFGGFDataGridDataGridElsewhereElsewhere
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3Task 3.3 Technology to be Developed Technology to be Developed
OMISOMISInterfaceInterface
RGMARGMA OMISOMISInterfaceInterface
LocalLocalMonitorMonitor
PerformancePerformanceData StorageData Storage
RGMARGMAProducerProducer
ServiceServiceManagerManager
RGMARGMAConsumerConsumer
PerformancePerformanceToolsTools
ApplicationApplication ApplicationApplicationMonitorMonitor
Non-Non-invasiveinvasive
MonitoringMonitoring
Invasive MonitoringInvasive Monitoring
Non-invasive MonitoringNon-invasive Monitoring
fromfromTechnicalTechnical
AnnexAnnex
Poznan 28-JAN-2002 CrossGrid Task 3.3
Invasive MonitoringCopyright
S.M.Fisher / RAL
Poznan 28-JAN-2002 CrossGrid Task 3.3
Non-invasive MonitoringNon-invasive Monitoring
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3 Task 3.3 Technology Required: ChoicesTechnology Required: Choices
HomebrewHomebrew
Globus Sockets – current technologyGlobus Sockets – current technology
Globus:Globus:
MDS - current technologyMDS - current technology
GGF:GGF:
GMA - guidelines onlyGMA - guidelines only
DataGrid:DataGrid:
R-GMA - evolving technologyR-GMA - evolving technology
Elsewhere:Elsewhere:
Jiro - new technologyJiro - new technology
Poznan 28-JAN-2002 CrossGrid Task 3.3
Homebrew: Globus SocketsHomebrew: Globus Sockets
Supported by Globus toolsetSupported by Globus toolset
Includes GSI securityIncludes GSI security
http://www.globus.org/http://www.globus.org/
Quick solutionQuick solution
Does not give access to grid information systemDoes not give access to grid information system
Poznan 28-JAN-2002 CrossGrid Task 3.3
Globus: Meta-Directory Service (MDS)Globus: Meta-Directory Service (MDS)
Hierarchical SchemaHierarchical Schema
Queries constrained to schemaQueries constrained to schema
Can have parallel schemasCan have parallel schemas
Based on Lightweight Directory Access Protocol (LDAP)Based on Lightweight Directory Access Protocol (LDAP)
Good for static dataGood for static data
Less good for dynamic dataLess good for dynamic data
Now in 2Now in 2ndnd generation (MDS2 released OCT-2001) generation (MDS2 released OCT-2001)
Now uses LDAP v3Now uses LDAP v3
Poznan 28-JAN-2002 CrossGrid Task 3.3
MDS: DataGridMDS: DataGrid
DataGrid WP3 has delivered:Coordination of GRIS/GIIS work
OpenLDAP caching backend (FTREE)
GRIS/GIIS infrastructure maintain as necessary
Little effort anticipated
FTREE maintain as necessary
Little effort anticipated
Copyright
S.M.Fisher / RAL
Poznan 28-JAN-2002 CrossGrid Task 3.3
MDS: MDS: LDAPLDAP
LDAP reflects hierarchical database technologyLDAP reflects hierarchical database technology
LDAP is LDAP is efficientefficient if you know the query in advance if you know the query in advance::
you can build your database to you can build your database to respondrespond very rapidly very rapidly
For other queFor other queriesries, it could be , it could be veryvery inefficient:inefficient:
LDAP query language cannot give results based on computation LDAP query language cannot give results based on computation on two different objects in the structure on two different objects in the structure
Relational databaseRelational database technology technology proposedproposed by Codd by Codd ((30yrs ago30yrs ago) as) as solution to inflexibility of hierarchical solution to inflexibility of hierarchical && network data bases network data bases
Poznan 28-JAN-2002 CrossGrid Task 3.3
GGF: Grid Monitoring Architecture (GMA)GGF: Grid Monitoring Architecture (GMA)
Open schemaOpen schema
Distributed architectureDistributed architecture
Allows for dynamic dataAllows for dynamic data
GGF is a ForumGGF is a Forum
GMA is a set of guidelinesGMA is a set of guidelines
GGF will not ‘approve’ anythingGGF will not ‘approve’ anything
Several prototype implementationsSeveral prototype implementations
Poznan 28-JAN-2002 CrossGrid Task 3.3
GMAGMA: Architecture: Architecture
Consumers can themselves be ProducersConsumers can themselves be Producers
ConsumerConsumer
ProducerProducer
RegistryRegistry
registerregister
lookuplookup
Poznan 28-JAN-2002 CrossGrid Task 3.3
GMA: DataGMA: Data MModelodel / Protocols / Protocols
GGF GGF dodo NOTNOT specifyspecify any data model any data model
GGF GGF dodo NOTNOT specif specifyy protocol protocolss between consumer between consumerss && producer producerss
An implementer MUST:An implementer MUST:
FirstFirst choose a suitable data model choose a suitable data model
TThenhen select suitable protocols select suitable protocols
The chosen data model The chosen data model MUSTMUST have the power to support have the power to support ALLALL required required queries queries
Poznan 28-JAN-2002 CrossGrid Task 3.3
DataGrid: Relational GMA (R-GMA)DataGrid: Relational GMA (R-GMA)
Trying to Trying to do “better” than MDSdo “better” than MDS
An implementation of GMAAn implementation of GMA
Relational schemaRelational schema
Queries less constrained than MDSQueries less constrained than MDS
Designed to handle dynamic dataDesigned to handle dynamic data
11stst prototype released in NOV-2001 prototype released in NOV-2001
Based on servlet technologyBased on servlet technology
HTTP queries / XML result-setsHTTP queries / XML result-sets
Can be set up to access MDS dataCan be set up to access MDS data
Poznan 28-JAN-2002 CrossGrid Task 3.3
R-GMA: Data ModelR-GMA: Data Model
DataGridDataGrid have chosen a have chosen a RELATIONALRELATIONAL data modeldata model
Not general distributed RDBMS system, but a way to use relational Not general distributed RDBMS system, but a way to use relational model in model in a a distributed environment distributed environment where ACID (Atomicity, where ACID (Atomicity, Consistency, Isolation and Durability) properties not Consistency, Isolation and Durability) properties not importantimportantProducersProducers announce:announce: SQL “CREATE TABLE”SQL “CREATE TABLE”
publish:publish: SQL “INSERT”SQL “INSERT”
ConsumersConsumers collect:collect: SQL “SELECT”SQL “SELECT”
Viewed asViewed as one huge logical data base one huge logical data base,, partitioned according to partitioned according to certain criteriacertain criteria (specified by WHERE clause as a predicate) (specified by WHERE clause as a predicate)
Poznan 28-JAN-2002 CrossGrid Task 3.3
R-GMA: Data QualityR-GMA: Data Quality
Difficult to knowDifficult to know when to get rid of archived data when to get rid of archived data
Information may no longer be Information may no longer be ‘‘up to dateup to date’’
BUTBUT
ForFor historical historical analysisanalysis this is this is not a criteria for pruningnot a criteria for pruning
Only the Only the coconsumernsumer of data of data can devise a suitable strategycan devise a suitable strategy
Only theyOnly they know know why why theythey areare collecting the data collecting the data
ProducerProducer of data is no judge of its continued worth of data is no judge of its continued worth
Time-to-Live (Time-to-Live (TTLTTL)) of of questionablequestionable value value
Poznan 28-JAN-2002 CrossGrid Task 3.3
R-GMA: Philosophy
Information vs. Monitoring
From the user’s point of view there there is little or no difference between “plain” information and monitoring information.• Arguments about rates of change [slowly=Information whereas
rapidly=Monitoring] are unconvincing.
• Any measurement or fact represented as a tuple.
• Add a time stamp to the tuple and the information can be stored for later analysis as monitoring information.
• At most the difference is 1 field - the time stamp.
Time is the common element
DatagridDatagrid use use R-GMAR-GMA not only for monitoring but also as the basis not only for monitoring but also as the basis of an information systemof an information system
Copyright
S.M.Fisher / RAL
Poznan 28-JAN-2002 CrossGrid Task 3.3
R-GMA: Implementation
Consumer Servlet
RegistryAPI
Consumer Servlet
RegistryAPI
Consumer Servlet
RegistryAPI
Consumer ServletConsumer Servletand Mediatorand Mediator
RegistryAPI
Sensor CodeSensor Code
ProducerAPI
Application Code Application Code
ConsumerAPI
ProducerServlet ProducerServlet
RegistryAPI
Registry ServletRegistry Servlet
SchemaAPI
Schema ServletSchema Servlet
Event Dictionary
DBProducer
DBProducerDBProducerServletServlet
Application CodeApplication Code
ArchiverAPI
Archiver ServletArchiver Servlet
ConsumerAPIConsumer
APIConsumerAPIConsumer
API
Copyright
S.M.Fisher / RAL
lookuplookup
subscribesubscribe
Poznan 28-JAN-2002 CrossGrid Task 3.3
R-GMA: Progress
API in both JAVA and C++
Basic sensors (i.e. cpuload) for producing information
Integration with MDS:R-GMA sensor queries MDS GRIS and republishes information
Some basic displays to consume data
Demo code to get people started
Released into Testbed1 of EU-DataGrid
Copyright
S.M.Fisher / RAL
Poznan 28-JAN-2002 CrossGrid Task 3.3
JiroJiroTMTM Technology Technology
New technologyNew technology
http://www.jiro.com/
For distributed resource managementFor distributed resource management
Part of ‘Federated Management Architecture’Part of ‘Federated Management Architecture’
ICM will discussICM will discuss
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3: SubtasksTask 3.3: Subtasks
Task 3.3 (CYFRO / TCD / ICM)Task 3.3 (CYFRO / TCD / ICM)
Requirements SpecificationRequirements Specification
Task 3.3.1 (CYFRO - 38 MM)Task 3.3.1 (CYFRO - 38 MM)
Invasive Monitoring:Invasive Monitoring:‘‘Autonomous monitoring system for on-line and automaticAutonomous monitoring system for on-line and automatic
performance analysisperformance analysis ‘‘
Task 3.3.2 (TCD – 32 MM)Task 3.3.2 (TCD – 32 MM)
Non-invasive Monitoring:Non-invasive Monitoring:‘‘SQL-query-based tool support and interfaces to GridSQL-query-based tool support and interfaces to Grid
application programming environmentapplication programming environment’’
Task 3.3.3 (ICM - 28 MM)Task 3.3.3 (ICM - 28 MM)
Jiro:Jiro:‘‘Grid Jiro-based services for Grid infrastructure monitoGrid Jiro-based services for Grid infrastructure monitoring’ring’
Poznan 28-JAN-2002 CrossGrid Task 3.3
Task 3.3: To Be DiscussedTask 3.3: To Be Discussed
Source Code PortabilitySource Code Portability
C / Java / Perl / Python ?C / Java / Perl / Python ?
API definitionAPI definition
For Task 3.3.1For Task 3.3.1
Open Source LicensingOpen Source Licensing
GPL ?GPL ?
Poznan 28-JAN-2002 CrossGrid Task 3.3
WP3 Task 3.3WP3 Task 3.3
THE ENDTHE END