HPCN modeling at Vah river site
HPCN modeling at Vah river site
Institute of Informatics, SASWater Research Institute
Vah River Authority
OutlinesOutlines
Input data available at Vah pilot site
Modeling and simulation at Vah pilot site using SMS/FESWMS
HPCN approach for SMS/FESWMS
Data available at Vah pilot site
Data available at Vah pilot site
What we have yet:What we have yet:Two sets (43 and 141) of cross-sections
positives: hi-precision data, two independent sources negatives: problems with fitting to coordinate system usage: modeling, calibrating models
Vector 1:50 000 ARC GIS positives: global information, base database negatives: accuracy 1:50 000 usage: help interpret the area from its economic viewpoint
Raster 1:10 000 positives: good accuracy for interpretation of area negatives: raster format, the manual processing is needed usage: interpret the area from economic viewpoint and provide some parameters for modeling
LANDSAT IMAGES positives: actual (1999) multi-spectral data negatives: low accuracy usage: interpret the area from its economic viewpoint and provide some parameters for modeling of
landcover features
ORTHOPHOTOMAP positives: actual (2000) data from land negatives: 2D dimension usage: interpret the area from its economic viewpoint and provide parameters for modeling
What we are waiting for:What we are waiting for:
LIDAR data – contract has been signed– Data will be available due the end of June
Digital Surface Model DSM with the following features: grid width: 2 m height accuracy: RMS = +/- 0.15m data coding: 16 Bit or 8 Bit raster data data format ASCII
Digital Terrain model DTM (describing the ground surface) with
the following features: grid width: 2 m height accuracy: RMS = +/- 0.15m data coding: 16 Bit or 8 Bit raster data data format ASCII
GIS vector data in 1:50 000 scaleGIS vector data in 1:50 000 scale
ArcView format (with single database)
Focus to downstream part of pilot site (Povazska Bystrica town)
Focus to downstream part of pilot site (Povazska Bystrica town)
The same place in raster maps1: 10 000 scale
The same place in raster maps1: 10 000 scale
… and how it really looks(orthophotomap)… and how it really looks(orthophotomap)
Set of 43 cross-sectionsSet of 43 cross-sections
road
railway
cross-sections
Power canal
Vah river channel
43 cross-sections placed on satellite image43 cross-sections placed on satellite image
Modeling and simulation using SMS at Vah river
Modeling and simulation using SMS at Vah river
Input data: 6 cross-sections from VahInput data: 6 cross-sections from Vah
Mesh generated by SMSMesh generated by SMSThe right bank of river has denser mesh because there
is larger change in elevations and roughness
Mesh in detailsMesh in detailsThe black points represent nodes of the mesh. At this
time, elevations of the nodes are interpolated from the measured cross-sections
(not very accurate). LIDAR data will give more accurate
elevations for every node
The red points represent the measured
cross-sections
Map elevations (interpolated from cross-sections)
Map elevations (interpolated from cross-sections)
Material (roughness)Material (roughness)The main river channel has smaller roughness
The floodplains have bigger roughness
Experimental parametersExperimental parameters
Inflow: 3000 m3s-1 (return period more than 100 years) steady state
Number of elements: 7800 Number of nodes: 15750 Average distance between two neighbor
nodes: ~ 5m Number of equations: 35500 Computation time: 0:30:56 (on Pentium III,
550Mhz, 512MB RAM)
Results: water surface elevationsResults: water surface elevations
Results: water depthsResults: water depths
Results: flow velocity magnitudes and vectorsResults: flow velocity magnitudes and vectors
Results: trace flow animationResults: trace flow animation
HPCN approach for SMSHPCN approach for SMS
Complexity of SMS/FESWMSComplexity of SMS/FESWMS
If the simulated area increases 2 times in every dimension (or the distances between two neighbor nodes decreases 2 times for better accuracy), then:
– Number of nodes increases 4 times (O(N2))– Number of equations increase 4 times (O(N2))– Length of fronts in FESWMS increases 2 times (O(N))– Total memory requirement increases 8 times (O(N3))– Computation time increases 16 times (O(N4)) !!!
Computational time increases very fast with data size. Without using high performance platforms, users can not achieve reliable simulation results for large areas in reasonable time. HPCN implementation is necessary.
Pre-processing
Post-processing
140 NELL = NELL + 1 IF (NELL .GT. NE) GO TO 470 N = NFIXH(NELL)C C IF (IMAT(N) .LE. 0) GO TO 140 NM = IMAT(N) IF (NM .LT. 901) THEN IF (ORT(NM,1) .EQ. 0.) GO TO 140 END IF C =BPD= CALL SECOND(TA1) IF (NCORN(N) .LT. 6 .OR. NM .GT. 900) THEN CALL COEF1(N, 1) ELSE OMEGA = OMEGAS(N) CALL COEFS(N, 1) END IF C =BPD= CALL SECOND(TA2)C =BPD= SELT=SELT+TA2-TA1 NBN = NCN * NDF DO 150 LK = 1, NBN LDEST(LK) = 0 NK(LK) = 0 150 CONTINUE KC = 0 DO 170 J = 1, NCN I = NOP(N,J) DO 160 L = 1, NDF LL = NBC(I,L) KC = KC + 1 NK(KC) = LL IF (LL .EQ. 0) GO TO 160 IF (NLSTEL(LL) .EQ. N) NK(KC) = - LL 160 CONTINUE 170 CONTINUEC C ... SET UP HEADING VECTORSC DO 220 LK = 1, NBN NODE = NK(LK) IF (NODE .EQ. 0) GO TO 220 IF (LCOL .EQ. 0) GO TO 190 DO 180 L = 1, LCOL LL = L IF (IABS(NODE) .EQ. IABS(LHED(L))) GO TO 200 180 CONTINUE 190 LCOL = LCOL + 1 LDEST(LK) = LCOL LHED(LCOL) = NODE GO TO 210 200 LDEST(LK) = LL LHED(LL) = NODE 210 CONTINUE 220 CONTINUE IF (LCOL .GT. LCMAX) LCMAX = LCOL IF (LCOL .LE. MFWX) GO TO 240 PRINT 225, MFWX PRINT 230, NFIXH(NELL) IF (IOUT.GT.0) WRITE (IOUT,225) MFWX 225 FORMAT (/ ' Fatal ERROR in Subroutine FRONT.', //, * ' The Parameter variable MFW is presently=',I6,/
Program structuresProgram structures
Input files
Processing input data
Save solutions
Computational
kernel
Output files
Save solution to output files
•Read solutions from output files•Visualization, animation, analysis, statistics …•Export solutions
Graphical user interface (SMS) for pre- and post-processing• is not interesting for research• is absolutely necessary for end-users
•Import terrain maps (TIFF, XYZ, ArcView shapfile, …)•Define mesh, boundary conditions, …•Define simulation parameters•Generate input files
•Read input data•Check if input parameters are valid
Numerical simulation(FEM, FDM)•The main focus of research
Computational module (FESWMS)• is the focus of research• is not interesting for end-users
HPCN approachHPCN approach
Pre-processing
Post-processing
Processing input data
Save solutions
Parallelcomputational
kernel
Output files
Input files
Parallel computational module will run on HPCN platform
(supercomputers, clusters of workstations)
The main work of HPCN solution is to implement parallel computational kernel
The existing code in I/O parts of computational module is reused guarantee compatibility with existing module and save development time
No modification is required for GUI environment. Users will not notice any changes and use the program as normally.
GUI (SMS) will run on PC terminals
Communication between GUI (SMS) and computational module (FESWMS) can be done via any standard protocols (FTP, CORBA, HTTP,…)
Detailed FESWMS structuresDetailed FESWMS structures
Input files
Preliminarycomputations
Nonlinearsolver
write solutionto the file
Solution file
Checksolution
Prepare next solution
Linear solver
OK
Nonlinear solverSolution schema Newton iteration is used to solve nonlinear equations.
Linear solver is the computational kernel of
FESWMS and is the most CPU-time consuming part. Therefore, it is the focus of
parallelization
Parallel direct solversParallel direct solvers
Frontal method: the main direct solver which is especially designed for FEM. SMS uses this algorithm in FESWMS computational module.
Multi-frontal method: the parallel version of the frontal method. It is based on partitioning the finite-element domain into sub-domains and applying the frontal method to each sub-domain.
Existing libraries with direct solvers: MUMPS/PARASOL (developed in ESPRIT IV projects), SuperLU, SPARSE,…
Parallel iterative solversParallel iterative solvers
Conjugate Gradient (CG): the most powerful iterative solver which contains only vector and matrix operations is trivially parallelized.
Existing libraries with iterative solvers: PINEAPL (developed in ESPRIT IV projects), PETSC, Aztec, …
Advantages (in comparison with direct solvers):– less expensive (in terms of memory and CPU time)– higher parallelism, easier to parallelize
Disadvantages– does not guarantee to converge (direct solvers
always do)
Preliminary experimental resultsPreliminary experimental results
Computation time of original sequential version (using frontal method): 0:30:56
Computation time of current version (using BiCGSTAB iterative solver from PETSC) on single processor: 0:02:57
Computational time of current version on PC cluster of eight processors connected by 100Mb Ethernet0:01:13
Speedup on eight processors: 2.4
WP 3.5 HPCN implementation started 01/01/2001 and will last 24 months. The speedup will be improved in the final version.
The original version is too slow due the fact that direct solvers like frontal
method need a lot of memory. As there is not enough physical memory on a
single computer, part of data has to be swapped on hard disks
ConclusionConclusion There are several data sets of Vah river ready for
modeling. LIDAR data will improve the accuracy of input data.
SMS/FESWMS is good environment for modeling and simulation. Experiments have been done at Vah river with real input data.
HPCN version of FESWMS not only reduces computation times but also allows the simulation of large scale problems and consequently provides more reliable results.
Using HPCN will affect only computational kernel. End-users will not notice any changes except for the performance.
Parallel solvers are available. The preliminary experimental results show good speedup achieved on PC clusters.