service-oriented science: scaling escience impact
DESCRIPTION
Service-Oriented Science: Scaling eScience Impact. Ian Foster Computation Institute Argonne National Lab & University of Chicago. Acknowledgements. Carl Kesselman, with whom I developed many ideas (& slides) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/1.jpg)
Ian FosterComputation Institute
Argonne National Lab & University of Chicago
Service-Oriented Science: Scaling eScience Impact
![Page 2: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/2.jpg)
2
Acknowledgements Carl Kesselman, with whom I developed many
ideas (& slides) Bill Allcock, Charlie Catlett, Kate Keahey, Jennifer
Schopf, Frank Siebenlist, Mike Wilde @ ANL/UC Ann Chervenak, Ewa Deelman, Laura Pearlman
@ USC/ISI Karl Czajkowski, Steve Tuecke @ Univa Numerous other fine colleagues in NESC, EGEE,
OSG, TeraGrid, etc. NSF & DOE for research support
![Page 3: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/3.jpg)
3
Context:System-Level Science
Problems too large &/or complex to tackle alone …
![Page 4: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/4.jpg)
4
Two Perspectives on System-Level Science
System-level problems require integration Of expertise Of data sources (“data deluge”) Of component models Of experimental modalities Of computing systems
Internet enables decomposition “When the network is as fast as the computer's
internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder)
![Page 5: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/5.jpg)
5
Integration & Decomposition:A Two-Dimensional Problem
Decompose across network
Clients integrate dynamically Select & compose services Select “best of breed” providers Publish result as new services
Decouple resource & service providers
Function
Resource
Data Archives
Analysis tools
Discovery toolsUsers
Fig: S. G. Djorgovski
![Page 6: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/6.jpg)
6
A Unifying Concept:The Grid
“ Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
1. Enable integration of distributed resources
2. Using general-purpose protocols & infrastructure
3. To deliver required quality of service“ The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001
![Page 7: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/7.jpg)
FacilitiesComputersStorageNetworksServicesSoftwarePeople
Implementation
System-LevelProblem
Grid technology
Decomposition
U. Colorado Experimental
Model
NCSA
Computational Model
COORD.COORD.
UIUC Experimental
Model
![Page 8: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/8.jpg)
8
Provisioning
Service-Oriented Systems:Applications vs. Infrastructure
Service-oriented Gridinfrastructure Provision physical
resources to support application workloads
ApplnService
ApplnService
Users
Workflows
Composition
Invocation
Service-oriented applications Wrap applications as
services Compose applications
into workflows
“ The Many Faces of IT as Service”, ACM Queue, Foster, Tuecke, 2005
![Page 9: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/9.jpg)
9
Scaling eScience:Forming & Operating Communities
Define membership & roles; enforce laws & community standards I.e., policy for service-oriented architecture Addressing dynamic membership & policy
Build, buy, operate, & share infrastructure Decouple consumer & provider For data, programs, services, computing,
storage, instruments Address dynamics of community demand
![Page 10: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/10.jpg)
10
Defining Community: Membership and Laws
Identify VO participants and roles For people and services
Specify and control actions of members Empower members delegation Enforce restrictions federate policy
A
1 2
B
1 2
A B
1
10
1
10
1
16
Access granted by community
to user
Site admission-
control policies
EffectiveAccess
Policy of site to
community
![Page 11: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/11.jpg)
11
Evolution of Grid Security & Policy
1) Grid security infrastructure Public key authentication & delegation Access control lists (“gridmap” files)
Limited set of policies can be expressed
2) Utilities to simplify operational use, e.g. MyProxy: online credential repository VOMS, ACL/gridmap management
Broader set of policies, but still ad-hoc
3) General, standards-based framework for authorization & attribute management
![Page 12: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/12.jpg)
12
Core Security Mechanisms Attribute Assertions
C asserts that S has attribute A with value V Authentication and digital signature
Allows signer to assert attributes Delegation
C asserts that S can perform O on behalf of C Attribute mapping
{A1, A2… An}vo1 {A’1, A’2… A’m}vo2 Policy
Entity with attributes A asserted by C may perform operation O on resource R
![Page 13: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/13.jpg)
13
Security Services for VO Policy Attribute Authority (ATA)
Issue signed attribute assertions (incl. identity, delegation & mapping)
Authorization Authority (AZA) Decisions based on assertions & policy
VO AService
VOATA
VOAZA
VOUser A
VOUser B
![Page 14: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/14.jpg)
14
Security Services for VO Policy Attribute Authority (ATA)
Issue signed attribute assertions (incl. identity, delegation & mapping)
Authorization Authority (AZA) Decisions based on assertions & policy
VO AService
VOATA
VOAZA
VOUser A
Delegation AssertionUser B can use Service A
VOUser B
Resource AdminAttribute
![Page 15: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/15.jpg)
15
Security Services for VO Policy Attribute Authority (ATA)
Issue signed attribute assertions (incl. identity, delegation & mapping)
Authorization Authority (AZA) Decisions based on assertions & policy
VO AService
VOATA
VOAZA
VOUser A
Delegation AssertionUser B can use Service A
VOUser B
Resource AdminAttribute
VO MemberAttribute
VO Member Attribute
![Page 16: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/16.jpg)
16
Security Services for VO Policy Attribute Authority (ATA)
Issue signed attribute assertions (incl. identity, delegation & mapping)
Authorization Authority (AZA) Decisions based on assertions & policy
VO AService
VOATA
VOAZA
MappingATA
VO BService
VOUser A
Delegation AssertionUser B can use Service A
VO-A Attr VO-B Attr
VOUser B
Resource AdminAttribute
VO MemberAttribute
VO Member Attribute
![Page 17: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/17.jpg)
17
Closing the Loop:GT4 Security Toolkit
VO
RightsUsers
Rights’
ComputeCenter
Access
Services (runningon user’s behalf)
Rights
Local policyon VO identityor attributeauthority
CAS or VOMSissuing SAMLor X.509 ACs
SSL/WS-Securitywith ProxyCertificates
Authz Callout:SAML, XACML
KCA
MyProxy
Shib
![Page 18: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/18.jpg)
18
Security Needn’t Be Hard:Earth System Grid
Purpose Access to large data
Policies Per-collection control Different user classes
Implementation (GT) Portal-based User Registration
Service PKI, SAML assertions
Experience >2000 users >100 TB downloaded
PURSE User Registration
Optionalreview
www.earthsystemgrid.org
See also:GAMA (SDSC),Dorian (OSU)
![Page 19: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/19.jpg)
19
Scaling eScience:Forming & Operating Communities
Define membership & roles; enforce laws & community standards I.e., policy for service-oriented architecture Addressing dynamics of membership & policy
Build, buy, operate, & share infrastructure Decouple consumer & provider For data, programs, services, computing,
storage, instruments Address dynamics of community demand
![Page 20: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/20.jpg)
20
Community
Services Provider
Content
Services
Capacity
Bootstrapping a VOby Assembling Services
1) Integrate services from other sources Virtualize external services as VO services
2) Coordinate & compose Create new services from existing ones
Capacity Provider
“ Service-Oriented Science”, Science, 2005
![Page 21: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/21.jpg)
21
Providing VO Services:(1) Integration from Other Sources
Negotiate servicelevel agreements
Delegate and deploy capabilities/services
Provision to deliver defined capability
Configure environment Host layered functions
CommunityA
CommunityZ…
![Page 22: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/22.jpg)
22
Virtualizing Existing Services into a VO
Establish service agreement with service E.g., WS-Agreement
Delegate use to VO user
UserA
VO Admin
UserBVO User
ExistingServices
![Page 23: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/23.jpg)
23
Deploying New Services
Policy
Client
Environment
Activity
Allocate/provisionConfigure
Initiate activityMonitor activityControl activity
Interface Resource provider
WSRF (or WS-Transfer/WS-Man, etc.), Globus GRAM, Virtual Workspaces
![Page 24: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/24.jpg)
24
Available in High-Quality Open Source Software …
Data Mgmt
SecurityCommonRuntime
Execution Mgmt
Info Services
GridFTPAuthenticationAuthorization
ReliableFile
Transfer
Data Access& Integration
Grid ResourceAllocation &
ManagementIndex
CommunityAuthorization
DataReplication
CommunitySchedulingFramework
Delegation
ReplicaLocation
Trigger
Java Runtime
C Runtime
Python Runtime
WebMDS
WorkspaceManagement
Grid Telecontrol
Protocol
Globus Toolkit v4www.globus.org
CredentialMgmt
Globus Toolkit Version 4: Software for Service-Oriented Systems, LNCS 3779, 2-13, 2005
![Page 25: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/25.jpg)
25
http://dev.globus.org
Guidelines(Apache)
Infrastructure(CVS, email,
bugzilla, Wiki)
ProjectsInclude
…
![Page 26: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/26.jpg)
26
Virtual Workspaces(Kate Keahey et al.)
GT4 service for the creation, monitoring, & management of virtual workspaces
High-level workspace description Web Services interfaces for monitoring &
managing Multiple implementations
Dynamic accounts Xen virtual machines (VMware virtual machines)
Virtual clusters as a higher-level construct
![Page 27: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/27.jpg)
27
deploy, suspend
How do Grids and VMs Play Together?
C
lient
request
VM EPR
inspect & manage
use existing VM image Create VM image
VM Factory
VM Repository
VM Manager
create new VM image
ResourceVM
start program
![Page 28: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/28.jpg)
28
Virtual OSG Clusters
OSG cluster
Xen hypervisors
TeraGrid cluster
OSG
“ Virtual Clusters for Grid Communities,” Zhang et al., CCGrid 2006
![Page 29: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/29.jpg)
29
Dynamic Service Deployment(Argonne + China Grid)
Interface Upload-push Upload-pull Deploy Undeploy Reload
“ HAND: Highly Available Dynamic Deployment Infrastructure for GT4,”
Li Qi et al., 2006
![Page 30: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/30.jpg)
30
Providing VO Services:(2) Coordination & Composition
Take a set of provisioned services …
… & compose to synthesize new behaviors
This is traditional service composition But must also be concerned with emergent
behaviors, autonomous interactions See the work of the agent & PlanetLab
communities
“ Brain vs. Brawn: Why Grids and Agents Need Each Other," Foster, Kesselman, Jennings, 2004.
![Page 31: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/31.jpg)
31
Birmingham•
The Globus-BasedLIGO Data Grid
Replicating >1 Terabyte/day to 8 sites>40 million replicas so farMTBF = 1 month
LIGO Gravitational Wave Observatory
www.globus.org/solutions
Cardiff
AEI/Golm
![Page 32: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/32.jpg)
32
Pull “missing” files to a storage system
GridFTPReliable
File Transfer Service
GridFTP
Data Replication Service
“ Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005
Data Movement
![Page 33: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/33.jpg)
33
Pull “missing” files to a storage system
GridFTPLocal
ReplicaCatalog
ReplicaLocation
IndexReliable File
Transfer Service Local
ReplicaCatalog
GridFTP
Data Replication Service
“ Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005
ReplicaLocation
Index
Data MovementData Location
![Page 34: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/34.jpg)
34
Pull “missing” files to a storage system
List of required
Files
GridFTPLocal
ReplicaCatalog
ReplicaLocation
Index
Data Replication
Service
Reliable File
Transfer Service Local
ReplicaCatalog
GridFTP
Data Replication Service
“ Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005
ReplicaLocation
Index
Data MovementData Location
Data Replication
![Page 35: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/35.jpg)
35
Hypervisor/OS Deploy hypervisor/OS
Composing Resources …Composing Services
Physical machineProcure hardware
VM Deploy virtual machine
State exposed & access uniformly at all levelsProvisioning, management, and monitoring at all levels
JVM Deploy container
Deploy service GridFTP
![Page 36: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/36.jpg)
36
Hypervisor/OS Deploy hypervisor/OS
Composing Resources …Composing Services
Physical machineProcure hardware
VM VM Deploy virtual machine
State exposed & access uniformly at all levelsProvisioning, management, and monitoring at all levels
JVM Deploy container
DRS Deploy service GridFTP LRC
VO Services
GridFTP
![Page 37: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/37.jpg)
37
Decomposition EnablesSeparation of Concerns & Roles
User
ServiceProvider
“ Provide access to data D at S1, S2, S3 with performance P”
ResourceProvider
“ Provide storage with performance P1, network with P2, …”
D
S1
S2
S3
D
S1
S2
S3Replica catalog,User-level multicast, …
D
S1
S2
S3
![Page 38: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/38.jpg)
38
Another Example:Astro Portal Stacking Service
Purpose On-demand “stacks” of
random locations within ~10TB dataset
Challenge Rapid access to 10-10K
“random” files Time-varying load
Solution Dynamic acquisition of
compute, storage
++++++
=
+
S4 SloanDataWeb page
or Web Service
![Page 39: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/39.jpg)
39
Astro Portal Stacking Performance (LAN GPFS)
![Page 40: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/40.jpg)
40
Summary Community based science will be the norm
Requires collaborations across sciences— including computer science
Many different types of communities Differ in coupling, membership, lifetime, size
Must think beyond science stovepipes Community infrastructure will increasingly become the
scientific observatory Scaling requires a separation of concerns
Providers of resources, services, content Small set of fundamental mechanisms required to build
communities
![Page 41: Service-Oriented Science: Scaling eScience Impact](https://reader033.vdocument.in/reader033/viewer/2022051218/568157c8550346895dc5506e/html5/thumbnails/41.jpg)
41
For More Information Globus Alliance
www.globus.org Dev.Globus
dev.globus.org Open Science Grid
www.opensciencegrid.org TeraGrid
www.teragrid.org Background
www.mcs.anl.gov/~foster
2nd Editionwww.mkp.com/grid2