grid - korisnicki pristup i razvoj aplikacija
DESCRIPTION
Grid - korisnicki pristup i razvoj aplikacija. Meteo VO Training, Belgrade 24. June 2008. Branko Marovic RCUB - UoB. SEE-GRID-2 Applications Adaptation & Support. Application lifecycle defined in “Gridification Guidelines” document. - PowerPoint PPT PresentationTRANSCRIPT
www.see-grid-sci.eu
SEE-GRID-SCI
The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338
Grid - korisnicki pristup i razvoj aplikacija
Branko MarovicRCUB - UoB
Meteo VO Training, Belgrade24. June 2008
Application Gridification 2/26
SEE-GRID-2 Applications Adaptation & Support
Application lifecycle defined in “Gridification Guidelines” document.SG Gridification Guide (wiki pages) structured and contributed by developers.Policy: Applications initially supported at the country level.Application Support Group (ASG) created for gridification support.Application Review Committee(ARC) created to assess existing and new candidate applications.Application selection criteria defined by ARC.MoU Between Applications and SEEGRID-2.
Application Gridification 3/26
SEE-GRID-2 Strategy for New Applications
There is “Gridification Guideline” document to provide initial help to developers. (application life-cycle, policy and methodology to run and support an application on Grid)There is “SG Gridification Guide” on SEE-GRID-2 Wiki to provide comprehensive help to developers (porting an existing application to the Grid)ARC (Application Review Committee) recommendations push applications towards collaborations with other applications and projects.Applications are required to provide update on their established user community and collaborations within their periodic Application Progress Report.
Application Gridification 4/26
Application Gridification 5/26
SEE-GRID2 Applications
Application Gridification 6/26
Support the deployed applications
Application support group (ASG) created to provide fast and continuous gridification support to developers. Experienced developers & admins ASG team members assigned for each application. National level application support SEE-GRID - global level application support
SG Gridification Wiki online.SEE-GRID helpdesk available and used for problems. Useful Helpdesk Tickets for Developers. SEE-GRID Helpdesk “Applications Support” knowledge base. GOODs responsible to dump useful tickets of the week.
Work in close collaboration with WP5 (training) and WP3 (software requirements, maintenance of performance)
Application Gridification 7/26
Application Lifecycle
Application Gridification 8/26
SEE-GRID-2Gridification Guides
An initial guide for the developer including the application life-cycle, VO and site policies, gridification methodology: Gridification Guideline http://www.see-grid.eu/content/modules/
downloads/SEEGRID2-WP4-TR-008-Gridification_Guideline-n-2006-11-22.doc
Detailed wiki pages with illustrations for gridification issues: SG Gridification Guide http://wiki.egee-see.org/index.php/
SG_Gridification_Guide
Application Gridification 9/26
SEE-GRIDGridification Guide
Relevant topics for application developers identified trough online questionnaire systemSome investigation areas identified as well – candidates for future GG topicsGridification guide will provide information on these topicsWiki as collaboration medium
Application Gridification 10/26
SEE-GRIDGridification Guide
1 Job Management 1.1 Running Jobs (T1) 1.2 Software Installation Management (T3)
2 File Management and Access 2.1 Data Management (T2) 2.2 Data Access (T4) 2.3 Logical Files (T6) 2.4 Advanced Replica Management (T2, T6 related investigation area)
3 Inter-Job Communication 3.1 Workflows (T7) 3.2 MPI (T9) 3.3 Interactive & Pilot Jobs (T1 related investigation area) 3.4 Java networking
4 Information Services 4.1 Application-Level Event Logging & Performance (T5)
4.1.1 Applications and Systems monitoring tools 4.1.2 Benchmarking
4.2 Metadata Catalogues 5 Development Environments and Portals
5.1 Web Authentication and Authorization (T8) 5.2 P-GRADE Portal (T1-T9 related investigation area) 5.3 P-GRADE (T1-T9 related investigation area)
Application Gridification 11/26
Application Gridification 12/26
VO Resource Allocation Policies
Seegrid VO is used for all applications: Utilization of groups and roles:
/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=production/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=software
For gridification testing phase, no role or group is used, all sites support all
applications.
For deployed applications, sites are encouraged to support as many
applications as possible.
MoU template between SEE-GRID-2 Project and the Application created.
Application Gridification 13/26
SEE-GRID-SCI
Project administrative and technical management
Adapt Grid applications
Support the deployed
applications
NGI support and international collaboration
Dissemination and Training
JRA1NA4User communities support
NA1
NA2
NA3
Application extensions deployment and assessment
Implementation of application-specific
services
Assess applications Grid usage and take-up
Development of application-level services and operational tools
SA1eInfrastructure operations
Application requirements analysis
Design, implementation and assessment
of the application-specific
monitoring and control tools
Operational tools analysis
Resource Centre Service Level Agreement monitoring and enforcement
Network Resource Provision
Implementation of the advanced SEE-GRID infrastructure
Study the Grid applications
Application Gridification 14/26
Meteorology VO
Members National Observatory of Athens (coordinator) South Environment and Weather Agency of Serbia Hydrometeorological Institute of Montenegro RBI of Zagreb University of Zagreb Federal Hydro-Meteorological Institute of Bosnia and
Herzegovina Republic Hydrometeorogical Institute, Banjaluka
Two sets of applications for initial deployment: Regional scale Multi-model, Multi-analysis ensemble
forecasting system Study of the interaction of airflow with complex terrain
Application Gridification 15/26
Developer Resources
Grid environment is constantly evolving, but Useful features persist New are constantly being added Bugs are being fixed Gained knowledge remains relevant, must be updated Applications can be easily migrated to new/updated APIs
gLite User Guide https://edms.cern.ch/file/722398/1.2/gLite-3-UserGuide.pdf
WMS user guides https://edms.cern.ch/document/572489/1
SEE-GRID Gridification Guide http://wiki.egee-see.org/index.php/SG_Gridification_Guide
SEEGRID Wiki http://wiki.see-grid.eu/
gLite documentation http://glite.web.cern.ch/glite/documentation/
Application Gridification 16/26
Grids: a foundation for e-Research
Enabling a whole-system approachCollaborative research / engineering / public service …
sensor nets
Shared data archives
computers
software
colleagues
instruments
Grid
Diagram derived fromIan Foster’s slide
Application Gridification 17/26
Načini upotrebe grida
Komandna linija, JDL, input/output sandbox i/ili ručna manipulacija podacimaPrimena skriptova u generisanu poslova, obradi i manipulaciji podacimaKorišćenje postojećih Grid portalaPrimena grid API-ja i servisa iz koda koji se izvršava na griduPravljenje aplikativno specifičnih pristupnih tačaka-servisa
Application Gridification 18/26
Complexities of grid applications
1. Simple jobs – submitted to WMS to run in batch mode2. Job invokes grid services
• To read & write files on SE• Monitoring• For outbound connectivity (interactive jobs)• To manage metadata• …
3. Complex jobs• An environment controls multiple jobs on users’ behalf
• High-level services• Portals with workflow • Software written for the VO (or by the user)• …
Application Gridification 19/26
Invocation of applications
From the UI Command Line Interfaces / Scripts APIs Higher level tools
From desktop Windows applications Use Grids without awareness of them! But gLite not (yet) fully supporting Windows (more info: http://jessica.trigrid.it/grid2win/)
From portals For recurring tasks: “core grid services” as well as application layer Accessible from any browser Tailored to applications Different portal solutions, and wide range of capabilities. Second part
of this course: P-GRADE Portal
Application Gridification 20/26
Characteristics of VOs
What is being shared? resources of storage and/or compute cycles software and/or data
Distinct groups of developers and of users? Some VOs have distinct groups of developers and users…
Biomedical applications used by clinicians,…. …. Some don’t
Physics application developers who share data but write own analyses
Effect: need to hide complexity from the 1st type of VOs expose functionality to 2nd type of Vos many security issues
Application Gridification 21/26
Challenges
Research software is often Created for one user:
the developer Familiarity makes it
useable Short-term goals: Used
until papers are written and then discarded
Grid applications are often used by a VO Without support from
developer In new contexts and
workflows
Need expertise in:
• software engineering
• application domain
• grid computing
Grid application developers are In a research
environment Yet their s/w must
have: Stability Documentation Usability Extendability
i.e. Production quality
Application Gridification 22/26
Consequences
Team work!Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions.From research to production software: ~5 times the effort. “80% of the time for last 10% of the functionality & reliability”
Standardisation is key For re-use, for dynamic configuration of services,.. Both for middleware and domain specific
Need to follow a deliberate development process Waterfall? Rapid prototyping? Requirements engineering, design, implementation, validation,
deployment Engaged with the user community
Application Gridification 23/26
Basic tasks while Porting applications to the Grid
1. Developing a non-grid application (or inheriting and updating an ancient one);
2. Go/no-Go decision about gridification• Is it suitable for the Grid environment?• “Cost/profit” analysis / feasibility study
Typical Questions Groups: Current structure of the application Dependencies of the application Available resources (manpower, knowledge, etc.) Requirements for the gridified application Expected impact of gridification Requirements for the grid infrastructure
More info: Application Description Template http://www.lpds.sztaki.hu/gasuc/?m=4
Application Gridification 24/26
Basic tasks while Porting applications to the Grid (contd.)
3. Grid environment access• Requesting Certificates / VO membership• Accessing Grid environment
• Appropriate VO UI machine account for command line• Portal GUI account;
4. Executing, Testing and Debugging the application;• Testing the non-grid application (debugging in Grid
environment is a hard task), creating use cases for single (non-grid) runs;
5. Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files;
6. Submitting the job to the Grid as small-scale pilot application;
Application Gridification 25/26
Basic tasks while Porting applications to the Grid (contd.)
7. Executing, Testing and Debugging the pilot application;8. IF something goes wrong
THEN GOTO 4;
9. IF everything seems to workTHEN increase the scale of the application (increase problem size, amount of used resources);
10. Optimizing the grid application;