wlcg-rus an extensible solution to grid accounting & usage monitoring egee 3 rd user forum x....
TRANSCRIPT
![Page 1: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/1.jpg)
WLCG-RUS
An Extensible Solution to Grid Accounting & Usage Monitoring
EGEE 3rd User ForumX. Chen, A. KhanBrunel University
![Page 2: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/2.jpg)
Who am I?
• PhD Candidate
• Current Research Scope– Grid Computing, Accounting;– Middleware Solutions;
• OGF– Co-chair of OGF UR working group;– Member of OGF RUS working group;
![Page 3: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/3.jpg)
Outline
• WLCG-RUS Overview
• Component Architecture
• Design Status
• Conclusion
![Page 4: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/4.jpg)
Overview
• RUS– Resource Usage Service– Resource Management in the context of
OGSA;– Aims at providing Grid resource logging and
tracking interfaces;– OGF RUS and OGF UR spec.– Mainly used to enable Grid accounting &
Usage Monitoring
![Page 5: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/5.jpg)
Grid Accounting
• Many Production Grids provide resource usage logging and tracking facilities– e.g. NorduGrid (SGAS being accepted as one of key
service in Globus), EGEE/WLCG (DGAS), and OSG (Gratia);
• Issues– Heterogeneous and Grid-specific solution;– XML:DB persistence only– Gaps between Custom usage schema and OGF UR;– Lacking of interoperability in multi-Grid environment
![Page 6: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/6.jpg)
Accounting In WLCG
• Multi-Grid Environment– Resources from three Grid peers (OSG, NorduGrid, and EGEE) – Most of EGEE services are reused;– WLCG Resource Broker;
• Interoperability
– Get usage data from individual accounting service;
– Get usage data from heterogeneous usage storages (mostly relational DB);
![Page 7: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/7.jpg)
WLCG-RUS• Standardization
– Full OGF RUS compatible– OGF UR version 1.0 as uniform data representation as message
level;
• Customization– at persistence level– Heterogeneous data persistence (Relational, file and XML);– Flexibly data provision (DGAS, Gratia, SGAS & UR generators);
• Advanced Features– Summarization: aggregate UR in summary format (e.g. Total usage
of a specific VO)
![Page 8: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/8.jpg)
Framework
• Based on JISC funded Project:– “Review of Grid Accounting and Usage Monitoring”
(Manchester & Brunel)– http://www.jisc.ac.uk/media/documents/programmes/einfrast
ructure/jisc_aum_final_report_wth.pdf
• Proposed Framework
Client Side
UR Generator
RUS Client
RUS Service
Access Control
Configuration Manager
UR Mapping Session Management
RUS Operation Logics
SOAP
![Page 9: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/9.jpg)
WLCG-RUS Architecture• Layered Architecture
Providing RUS-compatible SOAP Message and Aggregation Extensions
Delegate client-side requests to appropriate command
RUS operation interfaces allowing custom implementations
Property file-based configuration information for individual components
Provide Access Control interface and default XACML implementation
Enable flexible search, update dialects (e.g. SQL, Xupdate, and Xquery)
Aggregator interface for custom algorithms for summarisation
Data Access Object that allows various usage persistence access and Mapping to Usage Records
![Page 10: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/10.jpg)
Messaging• RUS Core WS-I Rendering (draft)
– https://forge.gridforum.org/sf/go/artf6015– http://forge.ogf.org/sf/go/artf6090
• Extensions– Grouping
• Allowing summarisation by grouping criteria;
• An alternative and easier search criteria;
– Sorting• Sorting usage retrieval results
![Page 11: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/11.jpg)
Example: Extract Request Message• Extract and summarise Disk Usage & CPU Duration where
“VO=cms” and “VO=altas” on this month, sorted by Disk Usage;
![Page 12: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/12.jpg)
• Alternative, usage selection can be specified with RUS request body as defined within RUS core spec.
Example: Extract Request Message (Cont.)
![Page 13: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/13.jpg)
Example: Extract Response Message• Response message should return a single usage record that
represented as a summary usage record
• An extension, <urf:Resource description=“NumberOfRecordAggregated” /> is used to indicated total amount of records taken effect on this aggregation.
![Page 14: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/14.jpg)
UR Modelling• UR Modelling
– Derived From OGF-UR 1.0 for those use custom usage record representation;
– Entity Type• Usage properties are shared by reference;• Nine entities
– Value Type• Usage properties are shared by value;• All other usage properties defined in OGF-UR 1.0
![Page 15: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/15.jpg)
Entity Relationship
![Page 16: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/16.jpg)
Default Mapping Strategies
• To bring relational usage storage into RUS• XRM
– XML-Relational Mapping
– Based on JAXB and Hibernate
– Automatic creation of default relational representation of OGF-UR schema;
– Hibernate file-based configuration for custom relational representation
![Page 17: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/17.jpg)
An Example
• Disk Usage Table– Association to a single record;– i.e. One-to-Many record-disk association– The disk_usage table is generated as
following:
![Page 18: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/18.jpg)
Data Access Framework• Generic Data Access Interface;
– Extensive access to relational DB, XML:DB, file system, other service (OGSA-DAI, DGAS, Gratia, and etc.)
– Default DAO implementation provides access through Hibernate engine;– DAO pattern– Abstraction while customization, flexibility and polymorphism.
![Page 19: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/19.jpg)
Security
• Authentication– TSL and mutual-authentication
• Access Control– Coarse-grain Access Control on invocation
of RUS operations (insert, update, extract, delete and auditing)
– Fine-grain Access control on per usage record basis.
![Page 20: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/20.jpg)
Access Control• Role Based Access Control
– Roles are defined according to URF entities (see session1: WLCG-RUS XOM)
– Four default Roles derived from RUS Entities• User
(UserEntity/urf:UserIdentity/ds:KeyInfo/ds:X509Data/ds:X509SubjectName);
• Machine Manager(MachineEntity/urf:MachineName);
• Host Manager(HostEntity/urf:Host);
• Project Manager (ProjectEntity/urf:ProjectName);
– One Role for administration• RUS administrator (for all permissions)
– Roles derived from URF extension framework– e.g. Site Manager (urf:Resource/@description)
![Page 21: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/21.jpg)
Access Control Rules• RULE 1: A user is only allowed to “extract”
usage records on his/her name – either
/urf:Usage/urf:UserIdentity/urf:GlobalUserName/text()
– or /urf:Usage/urf:UserIdentity/ds:KeyInfo/ds:X509Data/ds:X509SubjectName/text()
– depending on deployment environment
![Page 22: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/22.jpg)
Access Control Rules (cont..) RULE 2: Administrator has full permission on ALL RUS
operations upon ALL persistent usage records
RULE 3: Other roles (project manager, host manager, machine manager, and extensive managers) have permissions on “insert”, “extract” and “audit” operations ONLY upon relevant usage records; e.g. A CMS project manager can only execute “insert”,
“extract” and “audit” operations on usage records where “//urf:ProjectName/text()” value is equal to “CMS”;
e.g. The Brunel Tier 2 site manager can only execute “insert”, “extract” and “audit” operations on usage records where“//urf:Resource/@description” value is equal to “Brunel-Tier2”;
![Page 23: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/23.jpg)
Role Configuration• WLCG-RUS schema;• An example
![Page 24: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/24.jpg)
Authorization Framework• Composed of two abstract components:
– Authorizer• The component that performs access control check;
• For both coarse- and fine-grain access control;
– Authorizer Factory• the factory component that creates an instance of authorizer
implementation;
![Page 25: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/25.jpg)
Default Implementation• Default Authorizer
– Based on XACML RBAC 1.0 profile;
– Configurable for custom role definition according to URF extension framework;
– Auto-generation of XACML policy sets;
– Supporting policy persistence;
– Simplified configuration (ease writing verbose XACML policy files)
– High performance (through XACML policy evaluation ONLY when fine-grained access control);
• Default Authorizer Factory– Creation of default authorizer;
![Page 26: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/26.jpg)
Configuration• Runtime Configuration
– Property-based configuration;– Parameters includes DAO factory, Aggregator factory,
Filter factory;
• UR Mapping Configuration– Default UR mapping is based on hibernate-mapping
configuration file (XML-based)
• Deployment Configuration– WS-Management configuration (see implementation
slide)
![Page 27: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/27.jpg)
Runtime: Insertion Example
![Page 28: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/28.jpg)
Accounting with WLCG-RUS
Interoperability
![Page 29: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/29.jpg)
Implementation
• Based on Sun WS-Man platform– Web Service for Management;– Usage Record as a management resource;– Enabling enumeration for large number of query
results;– Extensible through WS-Management interfaces;
• Hibernate– For Default UR Mapping to relational usage
representation;
![Page 30: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/30.jpg)
Development Status
• Version 1.0– Requirement Analysis (done)– System Design (done)– Implementation (debugging)– First alpha release (expected at Feb. Or
March)– Test (Planned to deployed at Tier-2 site,
Brunel-Tier2?)
![Page 31: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/31.jpg)
Conclusion & Future Work
• WLCG-RUS– Provides extensible solution to RUS;– A development framework;– Default implementation for relational usage
persistence (milestone);– Dynamic XML-Relational mapping;– Aggregation;
• Future work– Version 2 with evolvement of UR 2.0
![Page 32: WLCG-RUS An Extensible Solution to Grid Accounting & Usage Monitoring EGEE 3 rd User Forum X. Chen, A. Khan Brunel University](https://reader035.vdocument.in/reader035/viewer/2022081511/5697bfc51a28abf838ca6e80/html5/thumbnails/32.jpg)
The End
• Thanks