escience challenges in levees monitoring - lessons from "flood" projects marian bubak...
TRANSCRIPT
![Page 1: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/1.jpg)
eScience challenges in levees monitoring - lessons from "flood" projects
Marian BubakDepartment of Computer Science
AGH University of Science and TechnologyKraków, Poland
http://dice.cyfronet.pl/
eScience 2015, Munich, August 31 – September 4, 2015
![Page 2: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/2.jpg)
• Bartosz Balis• Daniel Harezlak• Maciej Malawski• Piotr Nowakowski• Bartosz Wilk
• Tomasz Gubala• Marek Kasztelnik• Jan Meizner• Maciej Pawlik• ...
And colleagues from CrossGrid, K-WfGrid, UrbanFlood, ISMOP
Thanks to
![Page 3: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/3.jpg)
Outline
• Motivation• Interactive system (person in a loop)• Exploitation of knowledge• Building early warming systems• IT support for levees monitoring• Summary
![Page 4: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/4.jpg)
Motivation: our area of research
• Investigation of methods for complex scientific collaborative applications• Elaboration of environments and tools for eScience• Integration of large-scale distributed computing infrastructures• Knowledge-based approach to services, components, and their composition
![Page 5: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/5.jpg)
Motivation: Krakow, May 2010
![Page 6: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/6.jpg)
Flood - CrossGrid (2002- 5)
CrossGrid: Development of Grid Environment for Interactive Applicationshttp://cordis.europa.eu/project/rcn/63588_en.html ftp://ftp.cordis.europa.eu/pub/ist/docs/grids/crossgrid_achievement.pdf
L. Hluchy, V. D. Tran, O. Habala, B. Simo, E. Gatial, J. Astalos, M. Dobrucky: Flood Forecasting in CrossGrid Project, in Marios D. Dikaiakos (Eds): Grid Computing Second European AcrossGrids Conference, AxGrids 2004, Nicosia, Cyprus, January 28-30, 2004. Revised Papers, LNCS 3165, 51-60, 2004
This paper presents a prototype of flood forecasting system based on Grid technologies. The system consists of workflow system for executing simulation cascade of meteorological, hydrological and hydraulic models, data management system for storing and accessing different computed and measured data, and web portals as user interfaces. The whole system is tied together by Grid technology and is used to support a virtual organization of experts, developers and users.
![Page 7: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/7.jpg)
Flood - K-WfGrid (2004-7)
K-WfGrid: Knowledge-based workflow system for Grid applications
http://cordis.europa.eu/publication/rcn/9410_en.htmlftp://ftp.cordis.europa.eu/pub/ist/docs/grids/k-wf-grid-interim-sheet_en.pdf
Ladislav Hluchý, Ondrej Habala, Martin Maliska, Branislav Simo, Viet D. Tran, Ján Astalos, Marian Babik: Grid Based Flood Prediction Virtual Organization. e-Science 2006, 4-6 December 2006, Amsterdam
This paper describes evolution of one such system -- a flood prediction application. The application consists of a set of simulation models, visualization tools, and various support components. During past six years it has evolved from a simple hydraulic modeling scenario into a sophisticated cascade of simulations, using state-of-the art grid, workflow and knowledge management technologies, and is one of the first applications of the SOKU [1]concept in the field of computer simulations.
![Page 8: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/8.jpg)
From IJdijk to UrbanFlood (2008)
The IJkdijk consortium turns to 7FP to organize research on the development of • GeoSensing technology• Sensor network
telecommunication systems
• Sensor data processing facilities
• Smartness in sensors (sensor plug and play, data acquisition).
Robert Meijer, TNO ICT Groningenand University of Amsterdam
http://www.floodcontrolijkdijk.nl/en/
![Page 9: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/9.jpg)
Smart levees
![Page 10: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/10.jpg)
Stand-by mode
• Monitoring data collection (low frequency)
• Initial on-line analysis (trends, deviations in sensor readings)
• Presentation of external info: weather prediction, flood wave prediction, etc.
Threat assessment mode
• Increased frequency of sensor data collection
• Resource-intensive threat level evaluation
Monitoring and decisions
![Page 11: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/11.jpg)
Monitor (AI)Decision SupportScenario ComputationVisualization
SSS
Control CentreSSS
Control Centre
AmsterdamRhine, De
Boston UK
Internet
AuthoritiesScience
Public
CISCISCIS
EWS1EWS2
EWS3
SoftwareHardware
UrbanFlood -Early Warning System
![Page 12: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/12.jpg)
A platform facilitating development, deployment and execution of EWSs• EWS development
– EWS reference model– EWS development framework
• EWS deployment– EWS blueprints– EWS-factory-as-a-service
• EWS execution– CIS runtime services for resource allocation, self-monitoring,
self-healing, mission-critical operation, and urgent computing
Common Information Space
![Page 13: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/13.jpg)
Flood simulation with CIS
![Page 14: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/14.jpg)
Domain resources exposed as Basic ServicesData, sensors, apps wrapped as appliances and deployed
onto clouds, …
Composite Services (Parts)Building blocks for EWSs
Orchestrate domain resources towards complex application scenarios (e.g.
area flood simulation)
Early Warning SystemA number of Parts deployed,
connected, and configured for a specific setting (e.g. a dike
section)
Common Information Space
![Page 15: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/15.jpg)
Flood EWS with CIS
![Page 16: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/16.jpg)
CIS as a system factory
• On-demand resource provisioning (local resources, clouds)
• Horizontal scaling of infrastructure (more instances)
• Load balancing with lazy evaluation• On-line availability monitoring• Notifications about problems• Automatic restart of failed components
![Page 17: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/17.jpg)
ISMOP: towards a levee monitoring system
Investigations on monitoring and assessment of levees:• Construction of an artificial levee• New sensors for levee instrumentation• Design and development of a sensor
communication infrastructure– Optimal collection and transmission of sensor
data• Levee modeling and simulation
– Comparison of simulated and real levee behavior
• Central System: software platforms for execution management, data management, visualization and decision support
![Page 18: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/18.jpg)
ISMOP: Consortium
• Department of Computer Science AGH• Department of Hydrogeology and Engineering
Geology AGH• Department of Geoinformatics and Applied
Computer Science AGH• NeoSentio, Kraków• Sweco Hydroprojekt Krakówin collaboration with the Czernichów Community
http://www.ismop.edu.pl/
![Page 19: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/19.jpg)
ISMOP central system use cases
• Support for experiments on the artificial levee– Controlled flooding of the artificial levee and on-
line data collection– Validation of models of levees
• Elaboration of a decision support system– Continuous monitoring of levees – Automation data-driven and model-driven analyses– Prediction of breaches
![Page 20: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/20.jpg)
Experimental levee 14.08.2015 (1/4)
![Page 21: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/21.jpg)
Experimental levee 14.08.2015 (2/4)
![Page 22: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/22.jpg)
Experimental levee 14.08.2015 (3/4)
![Page 23: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/23.jpg)
Experimental levee 14.08.2015 (4/4)
![Page 24: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/24.jpg)
Assessment of levee breach threat via scenario matching
![Page 25: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/25.jpg)
ISMOP Decision Suport System
![Page 26: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/26.jpg)
•Solution•Leveraging open standards (OGC, INSPIRE) for data & metadata models
Interoperability with external systems (e.g.
ISOK, regional flood protection agencies)
•Solution: research in progress…
Visualization of relevant information to
effectively support the decision making process
•Solution•Open domain-agnostic design (metadata and public APIs design are crucial)
Adaptability to other domains (e.g. monitoring
of communication infrastructure)
Challenges: visualization and decision support
![Page 27: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/27.jpg)
Challenges: execution management
•Solution•Monitored area divided into sections •Managed by multiple instances of a Monitoring Application, dynamically deployed on-demand
Scale up to 100s-1000s
kilometers of levees
•Solution•Dynamic provisioning of resources from private or public clouds•Autoscaling algorithms and policies
Highly variable resource demands:
from very low in standby mode to
high in threat assessment mode
![Page 28: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/28.jpg)
Challenges: data management
•Solution•Multiple data stores and models to address diverse needs
Diverse data sets (spatial, time series, binary,
metadata) and data usage patterns
•Solution•Big data infrastructure•Map-Reduce data search
Data-intensive processing
Threat level evaluation scenario: up to 130 GB of data to search per
1km of a levee
![Page 29: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/29.jpg)
Urgent computing scenario
Goal: Assess flood risk for a large set levees by a specified deadlineSolution: dynamic provisioning of cloud resources
• A user:– Target area for flood threat assessment– Time window size for current measurements– Deadline to get results
• The system:– Generates workflow representing all required computations and data
dependencies– Plans the workflow execution so as to meet the deadline– Runs the workflow– Monitors its execution and reconfigures resource allocation if needed
![Page 30: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/30.jpg)
Levee breach threat assessment
![Page 31: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/31.jpg)
Implementation of urgent computing
![Page 32: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/32.jpg)
Resource provisioning model
• Bag-of-tasks model– Selection of dominating tasks– Uniform task runtimes
• Performance model: T = f (v, d, s, …)– T – total computing time– v – number of VMs– d – time window in days– s – number of tasks (sections)
![Page 33: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/33.jpg)
Resource provisioning model
(1)
T – total computing timev – number of VMsd – time window in dayss – number of tasks (sections)
Parameters a, b, c to be determined experimentally
Solve eq. (1) to compute v (# of VMs) given T (deadline)
![Page 34: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/34.jpg)
Simulations
• Setup: private cloud infrastructure– a node with 8 cores (Xeon E5-2650)– virtual machines (1VCPU, 512MB RAM)– data for simulated scenarios (244MB total) on local
disks• Simulations:
– 1-1024 sections– 1-16 VMs– 1-7 days time window
• Warmup tasks:
![Page 35: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/35.jpg)
Resource provisioning - results
• Warmup tasks clearly separated as outliers
• Linear functions• Parameters a, b, c determined using non-
linear fit• The model fits well to the data
War
mup{
a = 6.53b = 9.41c = 31.71
1024 sections
128 sections
![Page 36: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/36.jpg)
Clouds for urgent computing (1/2)
• Elasticity– On-demand provisioning of VMs – Job prioritization and preemption
• Reliability– Public cloud services are specifically designed to support systems with
high availability demands – Amazon: only five major outages in the years 2010-2013 (only one for
more than 6 h)• Safety
– Serious natural disaster may damage a local computing infrastructure– Public clouds as an emergency computing infrastructure– Data safety: public clouds as a reliable storage infrastructure for
important but not sensitive data (example: pre-simulated scenarios data sets)
![Page 37: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/37.jpg)
Clouds for urgent computing (2/2)
• Cost-effectiveness– Decision support systems for natural disasters generate
‘spiky’ workloads: perfect cloud use case – Cheaper than maintenance of a dedicated infrastructure – Day-to-day operation can be handled by a relatively small,
low-cost on-premises infrastructure• Performance?
– Bag-of-tasks applications such as scenario identification perfectly fit the cloud
– What about CPU- and communication-intensive tightly-coupled simulations? HPC-in-the-Cloud is an emerging trend.
![Page 38: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/38.jpg)
Summary (1/2)
• Environmental models results in complex applications – collaborative – multi-scale, multi-domain, – time-critical– With data and resource intensive scenarios
• We have contributed to – methods and tools for environmental computing – advanced problem solving environments, virtual
laboratories– compositions of resources into complex scenarios– ccommodation to ”spiky” behavior (variable workload)
![Page 39: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/39.jpg)
Summary (2/2)
• We have addressed– Complex distributed systems– Coordination of execution (workflow)– Monitoring and management of services– allocation of resources to services– fault tolerance– provenance tracking
• Sustainability issue - supporting technologies
![Page 40: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/40.jpg)
More at
http://www.urbanflood.eu
http://www.ismop.edu.pl
http://dice.cyfronet.pl
![Page 41: EScience challenges in levees monitoring - lessons from "flood" projects Marian Bubak Department of Computer Science AGH University of Science and Technology](https://reader035.vdocument.in/reader035/viewer/2022070408/56649e695503460f94b66e81/html5/thumbnails/41.jpg)
Acknowledgements
This research was supported by the National Centre for Research and
Development (NCBiR) under Grant No. PBS1/B9/18/2013.