Łukasz kokoszkiewicz. envirogrids project overview swat explanation swat gridification assumptions...
TRANSCRIPT
SWAT GRIDIFICATIONSHORT SUMMARY
Łukasz Kokoszkiewicz
CONTENTS
EnviroGRIDS project overview SWAT explanation
SWAT gridification assumptionsParallelization results SWAT model parallelization LH-OAT uncertainty analysis parallelization SWAT-CUP sufi2 parallelization
EnviroGRIDS VOMonitoring Tools
DHMO
o Project timeframe: April 2009 – March 2013o Total budget of €8.1 mio. (€6.2 mio. EC contribution) and 1367
man-months.o The EnviroGRIDS Project Team includes 27 partners from 15
countries, representing also several European (CERN, EEA) and United Nations organizations (UNEP, UNESCO).
o Among these partners, 22 belong partially or entirely to the Black Sea Catchment.
o Eight partners belong to an International Cooperation Partner Countries (Ukraine, Georgia and Russian Federation) and five to Associated Countries (Switzerland and Turkey).
o Five partners can be defined as Small and Medium Enterprises (SME).
TECHNICAL OBJECTIVES
data gap analysis modeling of large scale, high resolution distributed hydrologic
processes developing a grid-enabled Spatial Data Infrastructure (SDI) spatially explicit regional scenarios of development develop access to real time data from sensors and satellites develop early warning and decision support tools at regional,
national and local levels Build capacities in the Black Sea region to enhance data
interoperability, by promoting on OGC and open standards (ISO 19115/19119/19139; web services WMS, WFS, WCS)
SWAT
SWAT (Soil and Water Assessment Tool) is a river basin, or watershed, scale model.It predicts the impact of land management practices on water, sediment and agricultural chemical yields in large complex watersheds with varying soils, land use and management conditions over long periods of time.
Models
SWAT
Hydrological modeling
GRIDIFICATION ASSUMPTIONS
SWAT Gridification
Creating EnviroGRIDS VO
Research which use cases are
possible to utilize on the
GRID
Creating easy to use monitoring
tools for the community
Envirigrids VO exists for few months now
Easy to use client application called hBrowse framework was created and
based on it Diane and Ganga monitoring tools
were created
3 use cases, for the SWAT gridification, were found
SWAT MODEL GRIDIFICATION
Sub-basin based model splitting followed by routingThe main idea is to split model itself based on sub-basins.Pre and post processing of the model is required to complete splitting and merging steps.After splitting, each sub-model can be run separately (in parallel) on the grid.Subbasin based parallelization is very interesting for large models.
DANUBE RIVER 15K HRU'S MODELGRID RUNS
As was observed SWAT model sub-basin based grid executions show some promises but to be fully utilized it requires much bigger jobs. Ideal situation would be when single sub-basin execution time takes more than half an hour, in this kind of situations the grid overhead would be negligible.
Tim
e [s
]
Simulation length [years]
LH-OAT SENSITIVITY ANALYSISALGHORITM PARALLELIZATION
LH-OAT is an uncertainty analysis method.In uncertainty analysis we are sorting out parameters sensitive to selected real world physical process (eg. nitrogen transport).Then an effective model calibration can be applied.The mechanism behind LH-OAT method parallel run is very similar to SWAT-CUP parallelization.We simply run many SWAT runs with a different parameters.
SWAT Model
LH-OAT Splitter
SWAT Model, params set 1
SWAT Model, params set 2
SWAT Model, params set n
swat swat swat
Output Output Output
LH-OAT Merger
GRID
SWAT-CUP SUFI2 ALGORITHM PARALLELIZATION
SWAT-CUP is an application for calibration of SWAT models. Calibration of SWAT models requires running SWAT simulations hundreds of times with different parameters setThe idea here is to run them separately or in small independent groups and resolve the best parameter configuration that gives simulation effects most corresponding to realityWe have “ready to go” example script set, to run swat sufi2 calibration efficiently from command line
Parameters
New SWAT Inputs
SWAT Outputs
swEdit.exe
swExtract.exe
swat
Output
SUFI-2
MCMC
ParaSol
GLUE
GRID
SWAT-CUP algorithms
ENVIROGRIDS VO
EnviroGRIDS VO is up and running since Autumn 2010Resources from Romania were added shortly after the VO were started and are being used since CE - ce01.mosigrid.utcluj.ro:8443/cream-pbs-envir – 960CPUs SE - se01.mosigrid.utcluj.ro - 11.739489 terabytes
Registration https://lcg-voms.cern.ch:8443/vo/envirogrids.vo.eu-egee.org/vomrs Approval is automatic for a listed members of the EG project People that are not members should be “recommended” by a listed member
VO configuration https://voms.cern.ch:8443/voms/envirogrids.vo.eu-egee.org/
Configuration.do voms113.cern.ch needs to be replaced by voms.cern.ch inside the VOMSES
string CIC VO Identity card: https://cic.egi.eu/index.php?
section=vo&vo=envirogrids.vo.eu-egee.org
MONITORING
hBrowse framework hBrowse.net It’s a common, easy to configure, client side, tool for creating job
monitoring systems It can visualise any hierarchical data up to 4 levels deep in a neat,
table based environment Google charts and highcharts can serve as renderer in creating
visually attractive charts
Based on hBrowse framewrok many monitoring tools was created including: http://dashb-atlas-job.cern.ch/templates/client/index.html http://dashb-atlas-job.cern.ch/templates/prodsys/index.html http://gangamon.cern.ch/ganga/ http://dianemon.cern.ch/diane/
ActiveMQ messages
MONITORING
The main system component is a JavaScript client which serves as a presentation layer.Different server layers can be applied.Server layer serves only as a data source.Client and server communicates using JSON data transfer format.JSON in this case is a perfect solution because of its syntax: in most cases it is the same as in
JavaScript and Python
HBROWSE SYSTEM ARCHITECTURE
CLIENT
JavaScriptAjaxjQuery
DATABASEWEB SERVER
SERVER
JSON
HTTP REQUEST
FINAL THOUGHTS
SWAT Gridification state The state of the SWAT gridification is that we can
run swat tasks on the grid With monitoring tools we can easily monitor our
simulations/calibrations
SWAT Gridifications final requirements Support of the linux executables (preferably both
32-bit and 64-bit) from SWAT developers is critically required
THANK YOU