the earth system grid (esg) computer science and technologies doe scidac esg project review argonne...
Post on 14-Jan-2016
222 Views
Preview:
TRANSCRIPT
The Earth System Grid (ESG)
Computer Scienceand Technologies
DOE SciDAC ESG Project ReviewArgonne National Laboratory, Illinois
May 8-9, 2003
May 8, 2003 Earth System Grid 2
Computer Science Perspective:Why is ESG Important?
• Application needs help formulate new frameworks and information technologies– Scientific apps good indicator of future trends– Climate community leading IT consumer
• Experimentation key to (computer) science– Needs robust instantiation of new technology– Needs an engaged community of consumers
• Multi disciplinary (intra-CS and CS-apps) teams are key to IT advances
May 8, 2003 Earth System Grid 3
We’re Particularly Interested in the Following Aspects of ESG
• “Enable [a community of] researchers to understand and make effective use of large, distributed climate datasets”– Dataset federation—physical and semantic– Security: who can do how much of what– Efficient analysis: distribution and placement
of computation and data
• Within the context of real data centers, real data, real analyses, and real users
May 8, 2003 Earth System Grid 4
The Computer Science Team
• ESG engages CS people at every institution• Four groups act as Grid technology providers
– Argonne National Laboratory (Globus Toolkit, etc.)– Lawrence Berkeley National Laboratory (SRM)– USC Information Sciences Institute (Globus Toolkit, etc.)– Oak Ridge National Laboratory (monitoring)
• Two groups act as climate data analysis tech providers– NCAR (data delivery and analysis software)– PCMDI (data delivery and analysis software)
• Integration, application, experimentation are highly collaborative activities
May 8, 2003 Earth System Grid 5
Our Hammers …
• Storage Resource Managers, Multiple File Transfer service
• Grid Security Infrastructure, Community Authorization Service
• GRAM Job Management • GridFTP data movement, Reliable File
Transfer service, • Metadata and replica management• Monitoring technologies
May 8, 2003 Earth System Grid 6
ESG CS Mission
• Work closely with application groups to– Integate these (and other) components to
provide end-to-end application solutions– Identify and, if possible, develop missing
pieces– Evaluate what happens when real users apply
our “solution” at scale
• Iterate to improve both Grid technologies and climate solutions
May 8, 2003 Earth System Grid 7
Contribution to IT
• New functionality: New features have been added to Globus Toolkit & SRM to meet climate community needs
• Robustness: “Production” deployment exposed limitations (functional and/or scale) and bugs in software tools
• Research: ESG requirements limitations exposed by deployment triggered new IT research directions
May 8, 2003 Earth System Grid 8
ESG Achievements
• Real value has been delivered to users– Mike Wehner, LLNL: “This has changed my life”
• Significant buy-in from climate scientists• Middleware is more robust & easier to use• Real interdisciplinary CS-climate scientist
teams established• National and international visibility for, and
interest in, our work
May 8, 2003 Earth System Grid 9
Observations
• We are building a middleware and people infrastructure w/o long term commitment– How do we persuade the community to
engage?
• Scope of the demand for ESG solutions is enormous, we can easily be overwhelmed– What is needed is an international
environmental sciences Grid– How can ESG contribute to its realization, via
leadership and technology development?
The Earth System Grid (ESG)
Architecture
DOE SciDAC ESG Project ReviewArgonne National Laboratory, Illinois
May 8-9, 2003
May 8, 2003 Earth System Grid 11
May 8, 2003 Earth System Grid 12
ESG Architecture
Metadata Catalog
Replica Catalog
Tape Library
Disk Cache
Attribute Specification
Logical Collection and Logical File Name
Disk Array Disk Cache
Application
Replica Selection
Multiple Locations
NWSSelectedReplica
gsiftp commands PerformanceInformation andPredictions
Replica Location 1 Replica Location 2 Replica Location 3
MDS
May 8, 2003 Earth System Grid 13
Metadata Catalog
Replica Catalog
Tape Library
Disk Cache
Attribute Specification
Logical Collection and Logical File Name
Disk Array Disk Cache
Application
Replica Selection
Multiple Locations
NWSSelectedReplica
gsiftp commands PerformanceInformation andPredictions
Replica Location 1 Replica Location 2 Replica Location 3
MDS
ESG Architecture
Remote Data
Tookit
Remote Calc.
Toolkit
Remote Viz
Toolkit
GenericApps
Grid Infrastructure
Brokers Info Schedule Data Monitor Security
Grid Application Toolkit (Middleware)
User
Adm.
PortalsApplications Generic
U.S. Users
CDAT Users Ferret Users
U.K. UsersClimate Community
Commercial Users
Community OutreachUniversity Users
Sponsors
Networks
ESG GridU.K. NERC DataGrid
CEOS GridOther Grids
May 8, 2003 Earth System Grid 14
NCAR
LBNL
LLNL
ISI
ANL
ORNL
GSI
GSIGSIGSI
GSI
GSI CAS server
CAS client
CAS client
CAS client
MyProxy client MyProxy server
TOMCAT
SECURITY services
GRAM
METADATA services
FRAMEWORK services
Auth metadata
RLS
RLS
RLS
RLS
NCARMSS
ORNLHPSS
DATA storage
The Earth System Grid
THREDDS catalogs
OGSA-DAISMCS
TRANSPORT services
gridFTP server/client
gridFTP server/client
gridFTP server/clientgridFTP server/client
TRM+DRM
TRM+DRM
TRM+DRMDRM
openDAPg server
openDAPg server
ANALYSIS & VIZ services
NCL openDAPg client LAS server
CDAT openDAPg client
MONITORING services
SLAMON daemon
SLAMON daemon
TOMCAT
AXIS
NERSCHPSS
DISK
DISK
DISK
DISK
mySQL
mySQL
mySQLxindice
mySQL xindicemySQL
May 8, 2003 Earth System Grid 15
Typical Application
Data(local)
netCDF lib
Application
Data(remote)
OPeNDAP Client
Application
OPeNDAPViahttp
Big Data(remote)
ESG client
Application
ESG+
DODS
OPeNDAP Server ESG Server
Distributed Application
dataOPeNDAP
ViaGrid
Distributed Data Access Protocols
Gridded Application
May 8, 2003 Earth System Grid 16
Data Movement
Data(local)
Data(remote)
Big Data(remote)
ESG client
Application
ESG+
DODS
ESG Server
Distributed Analysis
OPeNDAPViaGrid
Additional Scenarios
Big Data(remote)
ESG Server
May 8, 2003 Earth System Grid 17
Grid and NetworkInfrastructure
Grid-enabledstorage systems
Computationalresources
? RCAS
ESG services: information, replica,metadata, community authorization
M
Data consumers
Data producers
ESG: Collaboration Network
top related