Migrating from Grid to Cloud:Migrating from Grid to Cloud:Migrating from Grid to Cloud:Migrating from Grid to Cloud:Case Study from GEO GridCase Study from GEO Grid
Kyoung-Sook Kim
Data Science Research GroupInformation Technology Research Institute
Interoperability and IntegrationInteroperability and Integration
Satellite imageGeology
Geo* Contents Applications
Individual interfaces & protocols for information exchangeSatellite image for information exchange
It is not easy to• Access and (re)use• Manage and control
GIS data Environment & Energy• Manage and control• Integrate geospatial contents and services
Cooperate with different
Sensor data
http://www.geofabrik.de/data/shapefiles.html
Disaster Response
• Cooperate with different organizations
High cost of application Sensor data High cost of application development and management
Geospatial World Forum 2014 @ Geneve, Switzerland2014/5/15 2
GEO Grid: OverviewGEO Grid: Overview*Global Earth Observation
ee--Science infrastructureScience infrastructure for global earth observation systems with heterogeneous geospatial datasets
GeologyGeo* Contents Applications
StandardStandard -based interoperability & IntegrationSatellite image
Geology
- Ease-of-use interfaces (search, access, process data)- Low integration efforts
StandardStandard-based interoperability & Integration
WMS WFSWeb Web
GIS dataEnvironment & EnergyWPS
WMS WFS
SOSWCS CS-W
Web Web Services Services
Sensor data
http://www.geofabrik.de/data/shapefiles.html
Disaster Response• Security managementSensor data
Disaster Response• Security management• Scalability of resources
(computing and storage)
Geospatial World Forum 2014 @ Geneve, Switzerland2014/5/15 3
GEO Grid: GEO Grid: OverviewOverviewee--Science infrastructureScience infrastructure for global earth observation systems
with heterogeneous geospatial datasets
GeologyGeo* Contents Applications
StandardStandard -based interoperability & IntegrationSatellite image
Geology
- Ease-of-use interfaces (search, access, process data)- Low integration efforts
StandardStandard-based interoperability & Integration
WMS WFSWeb Web
GIS dataEnvironment & EnergyWPS
WMS WFS
SOSWCS CS-W
Web Web Services Services
Grid Computing Technologies &Sensor data
http://www.geofabrik.de/data/shapefiles.html
Disaster ResponseGrid Computing Technologies &
StandardsSensor data
Disaster Response
for sharing of geographically distributed resources
Geospatial World Forum 2014 @ Geneve, Switzerland2014/5/15 4
for sharing of geographically distributed resources and controlling resource sharing rules
GEO Grid: GEO Grid: OverviewOverviewee--Science infrastructureScience infrastructure for global earth observation systems
with heterogeneous geospatial datasets
GeologyGeo* Contents Applications
StandardStandard -based interoperability & IntegrationSatellite image
Geology
- Ease-of-use interfaces (search, access, process data)- Low integration efforts
StandardStandard-based interoperability & Integration
WMS WFSWeb Web
GIS dataEnvironment & EnergyWPS
WMS WFS
SOSWCS CS-W
Web Web Services Services
Grid Computing Technologies &Sensor data
http://www.geofabrik.de/data/shapefiles.html
Disaster ResponseGrid Computing Technologies &Sensor data
Disaster Response
Distributed Distributed and Parallel Processing
Metadata Management(LOD/RDF)
Security and User Management
Storage Grids
Heterogeneous Heterogeneous and Distributed Database Federation
Geospatial World Forum 2014 @ Geneve, Switzerland2014/5/15 5
ProcessingProcessing(LOD/RDF)
Grid and Cloud InfrastructureGrid and Cloud Infrastructure
ManagementFederationFederation
GEO GEO Grid:Grid: System ArchitectureSystem ArchitectureOutcomesOutcomesOutcomesOutcomes
Decision MakingSupport
Decision MakingSupport
Sci-TechR&D
Sci-TechR&D
BusinessBusinessShare
Disaster Information
Share Disaster
Information SupportSupportInformationInformation
Application Application ServicesServices
ASTERDEM
QuiQuakeQuiQuake Flood Simulation
Flux Monitoring
HotspotDetection
Others
Science DCP
ServicesServices
OGC standard web services OGC standard web services
Heterogeneous
WMS WCS CS-W WPS SOS WFS
Grid and CloudGrid and CloudInfrastructureInfrastructure
GeoSpatialGeoSpatial ResourcesResources
Storage GridsSecurity and User Management
Heterogeneous and Distributed Database Federation
Computing Grid and Cloud
Our HPC clusterGeoSpatialGeoSpatial ResourcesResources Our HPC clusterData Archive Services
Satellite Data
Geological Data
40m
20m
10m
30m
-0.01-0.05-0.1
-0.2
-0.5
40m
20m
10m
30m
-0.01-0.05-0.1
-0.2
-0.5
40m
20m
10m
30m
-0.01-0.05-0.1
-0.2
-0.5
sensors
ASTER satellite archive~= 2million scenes~= 1PB
MODIS, JERS-1PALSAR,LANDSAT
Geological Map of Japan
CfreeVG10,SRTM, Other geospatial info
2014/5/15 6Geospatial World Forum 2014 @ Geneve, Switzerland
sensorsPALSAR,LANDSAT
geospatial info
resources
An An Example Example of GEO Grid of GEO Grid Public ServicesPublic Services
• QuiQuake (https:/ /gbank.gsj.jp/QuiQuake/ index.en.html)
• Quick estimation system for • Quick estimation system for earthquake maps triggered by observation records
• Provides wide-ranging and detailed (250m-grid) strong ground motion maps for quick disaster response.
• Produces the results soon after the occurrence of an earthquake.
• Archiving all data of past earthquake occurrences.
2014/5/15 7Geospatial World Forum 2014 @ Geneve, Switzerland
QuiQuakeQuiQuake Realization Realization in GEO Gridin GEO Grid
Share Disaster
Information
Share Disaster
Information
Sensor Data Decision MakingSupport
Decision MakingSupport
InformationInformation
High Performance
Provide the results as OGC services and formats
Geospatial Data
Data Integration
PerformanceComputing
OutcomeOutcomeApplication Application
ServicesServicesGrid and CloudGrid and CloudInfrastructureInfrastructure
GeoSpatialGeoSpatial ResourcesResources
2014/5/15 8Geospatial World Forum 2014 @ Geneve, Switzerland
GEO Grid SecurityGEO Grid Security• GSI (Grid Security Infrastructure)
– Secure communication (authenticated and confidential) between Grid elements– Security across organizational boundaries– Security across organizational boundaries– ”Single sign-on" for users of the Grid in multiple resources and/or sites
• VO (Virtual Organization) • VO (Virtual Organization) ManagementA dynamic group of individuals, groups, or organizations who define organizations who define the conditions and rules for sharing resources
2014/5/15 9Geospatial World Forum 2014 @ Geneve, Switzerland
Migration from Grid to CloudMigration from Grid to Cloud
2011 Tohoku Earthquake
GEO Grid at AIST (~200km away) was damaged by the earthquake.Cannot operate GEO Grid for 2 months by recovery works & subsequent power Cannot operate GEO Grid for 2 months by recovery works & subsequent power restriction request.
BUT satellite imageswere very important to understand situations and provide useful information for helping the disaster response activities.
2014/5/15 10Geospatial World Forum 2014 @ Geneve, Switzerland
provide useful information for helping the disaster response activities.
ASTER Comparison Before and After TsunamiASTER Comparison Before and After Tsunami
• ASTER: – Japanese sensor (optical, 15m) on NASA Terra
satellitesatellite– Can create 3D model by making stereo-matching – AIST can produce true-color images.
11
Mouth of Kitakami-river
GEO Grid Disaster GEO Grid Disaster Responses Responses
• Rapid Evacuation :– Moved our minimum environment to oversea volunteer servers
• Using VM/Cloud technology
– Changed our daily network (data transfer) workflow to use these servers– Changed our daily network (data transfer) workflow to use these servers• To get/receive latest images from the satellite(ASTER)ALOS/PALSAR
TDRSUntil March 11
- Data providing- Portal
70 GB/day (ASTER)
NASA
AIST
ERSDAC
JAXA360 GB/day (PALSAR)
- Archive (tape, B-ray) - Archive (on-Disk)- Processing- WMS
• ASTER data: NASA ERSDAC AIST
2014/5/15 12Geospatial World Forum 2014 @ Geneve, Switzerland
• ASTER data: NASA ERSDAC AIST• PALSAR data: JAXA ERSDAC AIST (AIST: processing, WMS, portal site, and data archive)
What We DIDWhat We DIDTerra/ASTERALOS/PALSAR
TDRSData Flow and Services
from March 11 to April 20
- Portal
NASA
(AIST)
ERSDAC
JAXA
Orkney
- Processing- WMS- WMS
• ASTER data: NASA ERSDAC (AIST) (Orkney: processing and WMS, Google: portal site)
2014/5/15 13Geospatial World Forum 2014 @ Geneve, Switzerland
• ASTER data: NASA ERSDAC (AIST)• PALSAR data: JAXA ERSDAC (AIST)
(Orkney: processing and WMS, Google: portal site)
What We DIDWhat We DID
TDRS
Terra/ASTERALOS/PALSARData Flow and Services
From April 21
- Portal- Processing
NASA
(AIST)
ERSDAC
JAXA
UCSD
OCC
UCSD
- Processing
NCHC
WMS Server- Processing- WMS
• ASTER data: NASA ERSDAC (AIST) NCHC@TW, SDSC@USA, and OCC@USA: processing
WMS Server
QuiQuakeHigh performance (long-running) computing using oversea servers for creating scientific information
2014/5/15 14Geospatial World Forum 2014 @ Geneve, Switzerland
• ASTER data: NASA ERSDAC (AIST)• PALSAR data: JAXA ERSDAC (AIST)
NCHC@TW, SDSC@USA, and OCC@USA: processingNCHC@TW: WMS Google: portal site
TimeTime--line of our Activityline of our Activity
Date Events and Activities DescriptionsMar 11 Occurrence of Earthq. GEO Grid services stop
Mar 14 GEO Grid disaster TF Establishment Support by private companies, research inst., universities worldwideworldwide
Mar 15 Start of data transport and migration of processing codes
ASTER and PALSAR from ERSDACComputer and WMS in OrkneyPortal site in Google site
Mar 19 HP open and WMS start ASTER and PALSAR, damage interpretation, QuiQuake(manual)Mar 19 HP open and WMS start ASTER and PALSAR, damage interpretation, QuiQuake(manual)
Mar 21 Start of data transport Formosat-2 by NSPO (via JAXA)
Mar 25 Geological maps in WMS Seamless geological map, active fault, and geochemical map
Mar 31 Value-added-ASTER Natural color, orthorectification, automation
Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public1 month to recover the service using oversea sites2 months to recover/resume our environment
Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public1 month to recover the service using oversea sites2 months to recover/resume our environment Mar 31 Value-added-ASTER Natural color, orthorectification, automation
Mar 31 CS-W deployment WMS list
Apr 1 QuiQuake open QuiQuake (automatic)
Apr 20 Value-added-PALSAR Crustal deformation by InSAR
2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites
Apr 20 Value-added-PALSAR Crustal deformation by InSAR
Apr 21 Services move abroad NCHC (NARL-Taiwan), SDSC (UCSD), OCC (Univ.Chicago)
Apr 28 GEO Grid cluster resume
May 24 ASTER/AIST service resume Data publication to research users
May 27 QuiQuake/AIST resume
2014/5/15 15Geospatial World Forum 2014 @ Geneve, Switzerland
May 27 QuiQuake/AIST resume
June 30 PALSAR/AIST service resume
Towards Cloud ComputingTowards Cloud Computing
• Fortunately, we had VM images of satellite data processing– Ready to provisioning in a cloud environment– Connected to HPC international grid testbed (PRAGMA)– Connected to HPC international grid testbed (PRAGMA)
• Sharing our VM images in PRAGMA VM repository• We can boot our application VMs at any site by any PRAGMA colleagues
• However,• However,– Manual deployment at each site one by one – Lots of manual configuration due to heterogeneity and for security– Lots of manual configuration due to heterogeneity and for security– Tightly coupled data and computing servers– Not real-time– Heavy data formats and integration (OGC standards)– Heavy data formats and integration (OGC standards)– …
2014/5/15 16Geospatial World Forum 2014 @ Geneve, Switzerland
GEO Grid for Cloud ComputingGEO Grid for Cloud Computing• Virtualization as a Practical Mechanism
– Supporting Multiple VM Infrastructures (Xen, KVM, OpenNebula, CloudStack, Rocks, EC2)
• Data Intensity
• On-demand self-service• Broad network access• Resource poolingCloud
[NIST definition for using computing resources]
• Data Intensity– Data services to support data discovery, access,
processing, and delivery on demand with minimal transmission (ex., radiation monitoring service)
• Resource pooling• Rapid elasticity• Measured service
Cloud Characteristics
transmission (ex., radiation monitoring service)
• Complex Workflows– Machine learning platform based on Hadoop and
MapReduce– Lavatube (visual workflow engine)
• Software as a Service• Platform as a Service• Infrastructure as a
Service
Service Models
– Lavatube (visual workflow engine)– Real-time analytics
• High-performance Database for Linked Data– Distributed and parallel LOD processing
• Public• Private• Hybrid
Deployment Models – Distributed and parallel LOD processing
• Cloud Security– OpenID/OAuth for AuthN/AuthZ
• GeoSocial Media
• Hybrid• Community
Models
The NIST Definition of Cloud Computing, http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf • GeoSocial Media
– New data integration method
2014/5/15 17Geospatial World Forum 2014 @ Geneve, Switzerland
Global Earth Observation GridGlobal Earth Observation GridGlobal Earth Observation GridGlobal Earth Observation Gridhttp:/ /www.geogrid.org/http:/ /www.geogrid.org/