introduction to grid computing with high performance computing
TRANSCRIPT
• Objectives• Training Units• Introduction to Grid Computing• Registration with the White Rose Grid• Information Sources
Outline
Learning Outcomes
• Develop a High Performance computing (HPC) application.• Develop and manage applications for Computational grids• Manage the execution of HPC applications over a grid
computing system.• Discover resources, applications and data on distributed
systems.• Demonstrate an appreciation of evolving Grid
Technologies.
Sections
• Introduction to the White Rose Grid and grid operating systems.
• Application development for computational grids.
• Development tools and techniques for distributed high performance computing applications.
• Grid technologies and example grid projects
Assessment
• 3 Sets of assessment problems• A mini project on grid computing with a 3000
word report• Marks for the course are broken down as
follows– 40% from assessed problems– 60% from mini project
Grid Computing Mini Project
• Grid Applications for Specific Disciplines– Design of a grid application– Implementation of a prototype– An essay on the grid technologies that will influence
different disciplines and how this will be achieved.
Introduction to the White Rose Grid and Grid Operating Systems• Introduction to Grids, Registration and Access• Application management and development
using the UNIX operating system.• Review of White Rose Grid architecture and
applications.• Job Management and Scheduling.• Middleware for grid computing.
Application Development for Computational Grids
• Using and developing matlab applications for the grid.• Development of HPC applications using the C and
Fortran• Review of object orientation techniques in the
development of HPC applications.• Application Development using C++• Application development environments for the White
Rose Grid
Development tools and techniques for distributed high performance computing applications.• Libraries for HPC Application Development on
the WRG.• Performance Assessment Tools• Distributed Application Development Using MPI• Distributed Application Development Using
Open MP
Grid technologies and example grid projects• Introduction to Grid Services and the Open
Grid Services Architecture.• Development of Grid Portals• Implementing a project as a grid service, The
DAME project.• Cluster building and Integration with Grids• From semantic web to semantic grids
– Distributed searching of textual databases
Grid Computing is New Technology• Grid computing is in the
pioneering age!• As grids and grid
technology evolves it is to be expected that the contents of course material will change.
Introduction to Grid Computing
• Grid Technologies• Grid Definition• Types of Grid• Grid Architecture
• Large scale multi site data mining and data fusion• Shared virtual reality• Interactive collaboration• Real-time access to remote resources.
Grid Technologies
Definition of the Grid Problem
The “Grid Problem” is defined as the flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions and resources.
From “Anatomy of the Grid (Enabling Scalable Virtual
Organisations)”
I.Foster et al. Intl. J. Supercomputer Applications 2001.
Characteristics of Grids
•Global collections of resources connected with high speed networks
•Supercomputers, databases, storage, instruments, immersive environments
•Next generation internet•Emerging technology•Cost effective management of high end compute and data resources
Evolution of Networked Computing
Architecture
Client Server
Distributed
Web Applications
Web Services
Grid Applications
Grid Services
Description
Network is computer
Objects The web Computer is Network
Virtual Organisations
Globally Extended Virtual Organisations
Peak Late 80’s
Early 90’s Late 90’s Early 2k Late 2k?? ??
Protocol X X HTTP, JVM XML, LDAPSOAPWSDL
GRIP, GRAM, GFTP,LDAP
MDS,UDDI
Web Services
Web service are OK but…
They can only run on the computerWhere they are put.
The computer or dataavailability may not be fit for the task.
Types of Grid
Computational Grid Distributed Supercomputing
Grid aggregates computational resources to compute large complex problems
High throughput computing
Grid Schedules large numbers of loosely coupled or independent tasks
On Demand Computing Meet short term requirements for projects that cannot be cost-effectively or conveniently computed locally
Data Grid Data Intensive Computing Grid Synthesises from data managed on geographically distributed repositories
Access Grid Collaborative Computing Enhancing human-to-human interactions, e.g. collaborative design and virtual worlds
Grid ArchitectureAPPLICATION Molecular Modelling, Aerospace, Graphics and
Visualisation
APPLICATION TOOLKITS Data Grid, Remote Computation, Portals, Collaborative Visualisation, Sensors
GRID SERVICES Protocols, Authentication, Policy, Resource Management, Instrumentation
GRID FABRIC Storage, Computers, Networks, Sensors, Devices, Experiments
Grid Examples
•NASA’s Information Power Grid•computing infrastructure connecting the resources of several of its R&D Laboratoriesforming the Information Power Grid or IPG
•High Energy Physics E Data Grid
•The DataGrid Project is providing thesolution for storing and processing data generated by the LHC at CERN.
Grid and Web Service Demonstrations• SETI@home• The Dame Portal
– Requires valid e-Science Certificate
• Virtual Observatory• European Data Grid Demonstrators• OpenGIS Web Service• Matlab data Mining Service
Statistics for SETI at Home (16/10/2003)
Total Last 24 Hours
Users 4710399 1534
Results received 1073252223 1377476
Total CPU time 1655366.998 years
1347.953 years
Floating PointOperations
3.769869e+21 5.372156e+18(62.18 TeraFLOPs/sec)
Average CPU timeper work unit
13 hr 30 min 40.6 sec
8 hr 34 min 20.0 sec
The DAME Portal
• Distributed Aircraft Maintenance Environment• Portal runs engine vibration analysis tools on
selected sets of engine performance data.• http://iri02.leeds.ac.uk:9080/damexto/
damexto
OpenGIS Web Service
• http://www.opengis.org/resources/?page=demos• Standards based• Data Connections
– Forestry information– Satellite data Landsat 7, ESA Global imagery data– Atlas data, Roads, boundaries, populations, water
areas..
• Based on Web services enables easy integration with OGSA based Grid Services
e-Science Projects
•Distributed Aircraft Maintenance Environment•Distributed Collaborative Visualisation and problem solving environments•Clinical and Biomedical projects
• A co-operative clinical e-Science Framework•Biosim GRID•Decision making in the health sector
Lots of projects Here are a few!!
White Rose Grid Registration
• Conditions of Use• Becoming a White Rose Grid user• Obtaining an e-Science certificate
Conditions of Use
• White Rose Grid users must abide by the conditions pertaining to the individual systems at York, Leeds and Sheffield.– http://www.shef.ac.uk/calendar/compregs.html– http://www.leeds.ac.uk/iss/rules/ISShtml– http://www.wrg.york.ac.uk
Becoming a White Rose Grid User1. Register for an account on your local system2. Apply for White Rose Grid resources3. Obtain an e-Science certificate
Why obtain an x509v3 Certificate• Enables secure single sign on to the White
Rose Grid• Use portals e.g. the DAME portal• Use Globus to access WRG compute nodes
Register for an Account on Your Local System
Institution Registration DetailsThe University of Sheffield,Titania
Email: [email protected] Web: http://www.shef.ac.uk/cics/support/userreg/registration.html
The University of Leeds,Maxima,Snowdon
http://www.leeds.ac.uk/iss/helpdesk/ usernames.html
The University of York, Pascali
http://www.wrg.york.ac.uk/access.html
Application for White Rose Grid Resources• Details at http://www.shef.ac.uk/wrgrid/access• Complete the Application form “Application for a
Username and Resources on the White Rose Grid”
• Forward application to local member of the White Rose Grid executive and obtain Authorisation.
• Await confirmation of registration
Accessing the White Rose Grid
• From a UNIX account node access using ssh or XServer application (e.g. Exceed
• Using Globus with e-Science Certificate• White Rose Grid Portal (requires e-Science
certificate)
Accessing the White Rose Grid
• Resources– titan00.shef.ac.uk (10 Sun V880 Sun Fire Servers)– pascali.york.ac.uk (1 Sun V880 Sun Fire Server)– maxima.leeds.ac.uk (5 Sun V880 Sun Fire Servers)– snowdon.leeds.ac.uk (256 node Intel Beowulf cluster)
• From a UNIX account node access using ssh– ssh –l wrsmyname –X [node name].ac.uk
• Allows X applications from maxima and pascali
References
• The White Rose Grid http://www.wrgrid.org.uk• http://www.shef.ac.uk/wrgrid/trainingresources• http://www.shef.ac.uk/wrgrid/access• Grid Support Centre http://www.grid-support.ac.uk• National e-Science Centre http://umbriel.dcs.gla.ac.uk/NeSC• UK DTI e-Science programme http://www.escience-grid.org• Global grid forum http://www.ggf.org
Registration
• Register with the White Rose Grid• Obtain an e-Science Certificate
Go to the linkhttp://www.shef.ac.uk/wrgrid/access/index.html
Grid Computing References
• The Grid: Computing Without Bounds– Ian Foster, Scientific American, April 2003.
• “The Anatomy of the Grid”– http://www.globus.org/research/papers/anatomy.pdf
• Grid Services – “The Physiology of the Grid”– http://www.gridforum.org/ogsi-wg/drafts/
ogsa_draft2.9_2002-06-22.pdf• Research Agenda for the Semantic Grid
– http://www.semanticgrid.org/v1.9/semgrid.pdf