uncertainty and sensitivity analysis using hpc and htc

Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto

OPENSEES DAYS PORTUGAL 2014

UNCERTAINTY AND SENSITIVITY ANALYSIS

USING HPC AND HTC

André R. Barbosa

(1) Assistant Professor, School of Civil and Construction Engineering, Oregon State University

(1)

[email protected]

Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 2

Design Alternatives

Hazard Analysis

Structural Analysis

Damage Analysis

Loss Analysis

Decision Making

L,D [ ]P IM | X,D

[ ]IMν

[ ]P EDP | IM

[ ]EDPν

[ ]P DM| EDP

[ ]DMν

[ ]P DV | DM

[ ]DVν L,D

Intensity Measure

L: Location D: Design

Engineering Demand Par.

Damage Measure

Decision Variable

Select

q  Parametric sensitivity studies / optimization / design (Luis Celorrio-‐Barrague)

q  Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis (Filipe Ribeiro)

Introduction


Design Alternatives

Hazard Analysis

Structural Analysis

Damage Analysis

Loss Analysis

Decision Making

L,D [ ]P IM | X,D

[ ]IMν

[ ]P EDP | IM

[ ]EDPν

[ ]P DM| EDP

[ ]DMν

[ ]P DV | DM

[ ]DVν L,D

Intensity Measure



Damage Measure

Decision Variable

Select

q  Parametric sensitivity studies q  Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis

Introduction

Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto ( )1IM Sa T=

Probabilistic Seismic Hazard Analysis

M-‐R deaggrega8on Seismic hazard curve

Fault j Fault i

Fault k

Site

RIM

Mm0 mu

f M(m)

R

f R(r)

( ) ( ) ( )1

,flt

i i

i i

N

IM i i i M Ri R M

im P IM im M m R r f m f r dm drν ν=

= ⎡ > = = ⎤⎣ ⎦∑ ∫ ∫

R

IM

Mm0 mu

f M(m)

R

f R(r)

R

IM

Mm0 mu

f M(m)

Rf R(r)

AAenua8on rela8ons Magnitude Source-‐to-‐site distance

( )IM imν


XLB XM XUB

Response estimation accounting for modeling uncertainty q PSDA equa9on accoun9ng for model parameter uncertainty:

5

( ) [ ] ( ) ( )| ,EDP IMIM

edp P EDP edp IM d imf dν νΘ ⋅Θ Θ Θ= >∫

EDPLB EDPM EDPUB

NLTH ANALYSIS INPUT OUPUT

q Response es9ma9on: { }1, ,| , ,...,k l kP EDP edp IM im θ θ⎡ ⎤> = Θ =⎣ ⎦

µθ + aσθ


Parameter uncertainty progagation

INPUT Probability Distribution of RV X

X LB X UB X M

3D NL FE MODEL TIME HISTORY ANALYSIS

Uncertainty in ground motion Intensity Measure (IM) Ground motion profile (GM)

Uncertainty in structural properties Mass Viscous damping Strength Stiffness

OUTPUT

EDP(X LB ) EDP(X UB ) EDP(X M )

Probability Distribution of EDP j

Local EDPs Member: Curvature Strains: Reinforcing Steel

Concrete

Global EDPs U : Max Roof Displacement A : Max Floor Acceleration. IDR : Max Interstory Drift Ratio

Faggella , Barbosa, Conte, Spacone, Restrepo, 2013



INPUT Probability Distribu9on of Variable X

X LB X UB X M

OUTPUT

EDP(X LB ) EDP(X UB ) EDP(X M )

Probability Distribu9on of EDP j

TORNADO x10 , x50 , x90

FOSM (First Order Second Moments)

xm-as , xm , xm+as

TORNADO (swing) EDP(x10) – EDP( x90)

FOSM

mEDP , sEDP

MEAN and STD

Parameter uncertainty progagation


Swing =

EDP(x10) – EDP(x90)

TORNADO x10 , x50 , x90

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

EDP

Em

piric

al C

DF

11th value Median GM

Tornado sensitivity analysis

Procedure 1.  Perform Monte Carlo

Simulation using all ground motions (GM), fixing all other variables at their best estimates (median values)

(e.g. GM = 20)

2.  For each EDP, determine Median GM, and perturbe all other variables one at a time about their median value

XLB XM XUB


Sa

GM

Damping

Mass

Fy

Fc

Es

Ec


; , 1, 2, ,ij i j i j nθ ρ σ σ⎡ ⎤Σ = =⎣ ⎦ K[ ]T1 2 n, , ,= µ µ µKθµq  Mean values q  Variance-covariance matrix

q  Taylor series expansion of the response EDP

( ) ( ) ( ) ( )linr r r rθ

θ θ θθ µθ θ µ θ µ=

≈ = + ⋅ −∇

( ) ( )( )

2

i

i i i i

i i

i a

r rr

θ

µ θ µ θθθ θθ σ

+Δ − −Δ∂ =∂ Δ

Δ =

First Order Second Moment (FOSM) sensitivity analysis

XLB XM XUB

µθ + aσθ

EDPLB EDPM EDPUB

2 12 2

1 1 1

2i i j i j

n n i

ri i ji i j

r r rθ θ θ θ θσ ρ σ σ

θ θ θ

−

= = =

⎛ ⎞⎛ ⎞ ⎛ ⎞∂ ∂ ∂Σ = ⋅ + ⋅ ⋅⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟∂ ∂ ∂⎝ ⎠ ⎝ ⎠⎝ ⎠∑ ∑∑

Ø  Sensitivity

Ø  Covariance matrix of the response

9


0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10

Number of FE runs for TORNADO or FOSM analyses

EZ_e

rzi

KB_k

obj

LP_c

or

LP_g

av

LP_g

ilb

LP_l

ex1

LP

_lgp

c LP

_srtg

TO_t

tr007

TO_t

trh02

CL_

clyd

CL_

gil6

LV_f

gnr

LV_m

gnp

MH

_and

d

MH

_cly

d

MH

_hal

l

PF_c

s05

PF_c

s08

PF_t

emb

EDP

1

EDP

2

GM 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 201 med MONTE CARLO2 IMLB TORNADO3 dLB TORNADO4 mLB TORNADO5 fyLB TORNADO6 fcLB TORNADO7 EsLB TORNADO8 EcLB TORNADO9 IMUB TORNADO10 dUB TORNADO11 mUB TORNADO12 fyUB TORNADO13 fcUB TORNADO14 EsUB TORNADO15 EcUB TORNADO

Number of FE runs:

TORNADO

Swing = EDP(x10) – EDP(x90)

EDP

Em

piric

al C

DF

11th value Median GM

Sa

GM

Damping

Mass

Fy

Fc

Es

Ec

runsn GM 2 RV EDP= + ⋅ ⋅( )runse.g., n 20 2 7 10 160= + × × =


Parallelization of the analyses using XSEDE

OpenSees Mul9ple Parallel Interpreter (McKenna and Fenves 2007) hVp://opensees.berkeley.edu/OpenSees/parallel/TNParallelProcessing.pdf

Parallel Computer -0.4

-0.2

0

0.2

0.4

0 5 10 15 20Time (sec )

Accele

ration

(g)

GM 1, Par j

-0.4

-0.2

0

0.2

0.4

0 5 10 15 20Time (sec )

Accele

ration

(g)

…

-0.4

-0.2

0

0.2

0.4

0 5 10 15 20Time (sec )

Accele

ration

(g)

SUPERCOMPUTERS

GM 2, Par j

GM N, Par j

…


Case study: Bonefro 4 story building

Molise 2002 earthquake, Italy

Severe damage to first story infills and columns

Example 1: Bonefro Italy Faggella et al. 2008


Model (class) uncertainty Variation of the response under different modeling assumptions

Bare Frame Diaphragms (2x2)Stairs

NL Shear columnsNL Infills NL Inf. Bare 1st story


Model uncertainty

12

Variation of the response under different modeling assumptions

0

500

1000

1500

2000

0 50 100 150 200Top floor displacement (mm)

Base

She

ar (K

N)

bare framestairs

shell 2x2part. infilled

infilled

ADRS Demand SpectrumCapacity Spectra

0.89

0.15

0.4

2

1.251.090.83

0.71

0

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2Sde (m)

Se/g

, F

* /gm

*

bare frame

stairsshell 2x2

part. infilledinfilled

T C

0 50 100 150 2000

1

2

3

4

Displacements (mm)

Floo

r

0 0.5 1 1.5 20

1

2

3

4

Floo

r

Drift %

Bare FrameDiaph.2x2StairsNL Inf. Bare1NL InfillsNLshear col.

TH Average

TH Average


5

Ground motion and structural random variables

GM IM=Sa(T1)(g)

Damping(%)

Mass(ton/m2)

Fy(MPa)

Fc(MPa)

Es(GPa)

Ec(GPa)

Distrib. MCS Logn. Norm. Norm. Logn. Norm. Norm. Norm.

XM On EDP 0.2931 0.03 0.87 451 25 210 28

COV % // 84 40 10 10 6.4 3.3 8

Probability Functions based on• Seismic hazard• Values adopted in the literature• Experimental samples (material testing)

Uncertainty in ground motion• Intensity Measure (IM)• Ground motion profile (GM)

Uncertainty in structural properties• Mass• Viscous damping• Strength• Stiffness

Parameter uncertainty


25

3D Response Engineering Demand Parameters (EDPs)

X

Y

Rz

V

G

Μ, Χ

σ , ε SteelConcrete coreConcrete unconf.

U : Max Roof Displacement A : Max Floor Acceleration.

Member Sections CurvatureMember Sections Moment

GLOBAL

LOCAL

IDR : Max Interstory Drift Ratio

1001

1008

2001

2008

3001

3008

400140

08

121

122

R

Outputs (EDPs)


26

Results of MCS and TORNADO analysis

X

Y

Rz

V

G3D EDPsFloor DOFs

Monte Carlo using 20 ground motionsall other variables at medians

Tornado for MGM, all other variables perturbed one at a time about the median

Median MGM (11° value)

R

Outputs (EDPs)


!

25


X

Y

Rz

V

G

Μ, Χ




GLOBAL

LOCAL


1001

1008

2001

2008

3001

3008

400140

08

121

122

R

Outputs (EDPs)



X

Y

Rz

V

G

Μ, Χ




GLOBAL

LOCAL


1001

1008

2001

2008

3001

3008

400140

08

121

122

R

Outputs (EDPs)


PEER PBEE Methodology

Design Alternatives

Hazard Analysis

Structural Analysis

Damage Analysis

Loss Analysis

Decision Making

L,D [ ]P IM | X,D

[ ]IMν

[ ]P EDP | IM

[ ]EDPν

[ ]P DM| EDP

[ ]DMν

[ ]P DV | DM

[ ]DVν L,D

Intensity Measure



Damage Measure

Decision Variable

Select

q  Parametric sensitivity studies q  Probabilistic seismic demand analysis Ø Cloud Method


gu&&

Ø  Walls: Nonlinear truss modeling approach Ø  Columns and beams: Force-based beam-column elements Ø  Diaphragms: Flexible diaphragms allowing for plastic hinge

elongation

Example 2: NEHRP Building Modeling Approach

21

q  Rigid-end zone modeling for beam-column joints

(ASCE41-06)

REZ

NL

NL NL

NL NL

q Comprehensive/significant valida8on at system level ? … q Comprehensive/significant valida8on at component level


EW: 0.44 %NS: 2.93 %

N

Observed computational building behavior

22

(%)


q  NGA database (total 3551 records) Ø  Mechanism: Strike-slip (1004 records) Ø  Magnitude range: 5.5 to 8 (772 records) Ø  Distance: 0 – 40 kms (203 records) Ø  Vs30: C/D range (90 records)

“Cloud method”: Selection of earthquake records

23

0

5

10

15

20

25

30

35

40

5.5 6.0 6.5 7.0 7.5 8.0

Sou

rce-

to-s

ite d

ista

nce

Rru

p

Magnitude Mw

Non-pulse

Pulse

q  90 ground mo8on records selected from 14 earthquakes

0

5

10

15

20

25

30

35

40

5.5 6.0 6.5 7.0 7.5 8.0

Sou

rce-

to-s

ite d

ista

nce

Rru

p

Magnitude Mw

Non-pulse

Pulse


Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs.

q  Motivation

q  Application Example/Production campaign 1 (1)   Probabilistic seismic demand hazard analysis using the “cloud method”

q  Some numbers for this application example

Number of NLTH analyses 180

Average duration of NLTH analysis 12 hours

Average size of output data (compressed) 1.4 GB

Estimated clock time on a desktop computer (180x12)

2,160 hours 90 days

Estimated size of output data (180x1.4) 250 GB

24

OpenSees and Large Number of Runs

GM1

GM2

GM180

. . . 1. OpenSeesMP + Xsede? 2. Local Cluster? 3. Other options?


q OpenSeesMP + MPICH2 – useful for Domain Decomposition + Parameter Studies (addressed by other talks in this meeting)

q Condor + OpenSees Sequential – Parameter Studies

Possible Parallelization Options


https://twiki.grid.iu.edu/bin/view/Engagement/EngageOpenSeesProductionDemo

q  HTCondor (http://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computational-intensive jobs.

Schedd

(2) Central Manager

Collector

Negotiator

Startd

(1) Submit Node

(3) Worker Node

Submit job

Get results

HTCondor

Ø  Project started in 1988, directed at users with large computing needs and environments with heterogeneous distributed resources.

Ø  HTCondor is composed of 3 parts:

Startd

Worker Node

Worker Node

GM1

…GM180


Oregon State University: HTCondor + OpenSees q “Opportunistic” computing resources:

q Student computer labs (used by students mainly during the day, and during the term …)

q  Instruction computer labs (used during the term only during classes …)

q College of Engineering at OSU: 16 computer labs (~1500 cores)

http://monhost.engr.orst.edu/labs/


1

(1) Submit Node (3) Worker Nodes

(2) Central Manager

…

Implementation of HTCondor at Oregon State University

•  8 core Intel i7 •  Windows Server •  16 GB RAM •  SSD drive •  2 TB HDD 15K •  20 TB NAS

•  Windows 7 Premium

•  8 GB RAM •  2 x 1GB cards •  1 TB 7.2 K

Communication w/ IT, Dealing w/ Job recovery, W/O speed, data transfers, …?

The good news: ~ 1500cores


q  Some numbers for this application example

Number of NLTH analyses 180


Average size of output data 1.4 GB

Estimated clock time on a desktop computer (180x12)

2,160 hours 90 days

Estimated size of output data (180x1.4) 250 GB

29

OpenSees and Large Number of Runs

Clock time 36 hours !!

Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs.

q  Motivation

q  Application Example/Production campaign 1 (1)   Probabilistic seismic demand hazard analysis using the “cloud method”


(a) (b) (c)

Median2.5- and 97.5-percIndividual Ekqe

30

OpenSees and Parameters Studies

PFD – peak floor displacement; PIDR – peak interstory drift ratio; PFA – peak floor absolute acceleration


HTCondor and Open Science Grid

q  Open Science Grid is a national, distributed computing grid for data-intensive research.

Ø  Consortium of approx. 80 national laboratories and universities.

Ø  Version of Condor for the grid

Ø  Opportunistic resource usage: resources are sized for peak needs of large experiments (Atlas, CMS, etc.), OSG allows for non-paying organizations to use their resources.

q  HTCondor (hAp://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computa9onal-‐intensive jobs. Ø  Project started in 1988, directed at users with large compu9ng needs and

environments with heterogeneous distributed resources.

q  NEES and Open Science Grid have been active partners in creating the tools and infrastructures for making use of opportunistic resources


XLB XM XUB

32

EDPLB EDPM EDPUB

NLTH ANALYSIS INPUT OUPUT

Uncertainty in structural properties •  Mass •  Viscous damping •  Strength •  Stiffness

Engineering demand parameters •  Roof drift ratio •  Peak floor accelerations •  Shear demand in walls •  Residual deformatios..

µθ + aσθ

GM Damping (%)

Mass fy (ksi)

*fc (ksi)

Es (ksi)

*Ec (ksi)

XM MCS 0.02 68.7 6.84 29000 4714

COV % // 40 10 10 10 3.3 8

µθ

Response estimation accounting for parameter uncertainty


(1)  Probabilistic seismic demand hazard analysis using the cloud method (2)  Sensitivity of probabilistic seismic demand hazard to FE model parameters

q  Some numbers for production campaign 2 (99% complete)

Number of NLTH analyses per parameter set realization

180


Average size of output data 1.4 GB

Parameters considered 6

Perturbations considered 4

Estimated clock time on a desktop computer (180x12x[(6x4x2)+1])

105,840 hours 12.1 years

Estimated size of output (compressed) data (180x1.4x[(6x4x2)+1])

12 TB Clock time 30 days !!

Using Open Science Grid: Production Campaign 2 q  Production campaign

33


OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG)

34

Wall clock time in HTCondor / OSG

12 clusters of 180 jobs “Desktop”: 26,000 hours

OSG: 60,000 hours

25,000

20,000

15,000

10,000

5,000

0

30,000

Wal

l Tim

e (h

ours

)

(job preemp9on)


OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG)

120,000

80,000

40,000

0

160,000

Wal

l Tim

e (h

ours

) Wall clock time in HTCondor / OSG


Comparison Between Parallelization Options OpenSeesMP HTCondor Straight forward implementation of Domain Decomposition through OpenSees framework with parallel solving algorithm like MUMPS

No ready built solution for large problems, OpenSees sequential does not have parallel solvers for large problems

MPICH2 networking setup is relatively easier Job management easier

Condor pool setup requires some learning Condor requires maintenance and administration

Very active user support through OpenSees user community, most attractive aspect of using OpenSeesMP

There is no specific user community as such.

Limited tests show 190 % Speed up from one processor to two processor

Limited tests show 153 % Speed up from one processor to two processor

Main complication is compilation of OpenSeesMP, really really tough!! But once over it OpenSeesMP is really powerfull!!!

Global implementation, if want to connect to other grid systems. Steep learning curve , knowledge of networking (Computer science)

Khaled Mashfiq, MS – La Sapienza, Rome

Conclusions

37

ü  A workflow for running parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with large number of parameters and OpenSees runs has been developed for using NEEShub, Xsede, and Open Science Grid.

ü  HTCondor

ü  Pegassus (see Frank Mckenna’s presentation)

q  Where and what to store?

q  Post-processing? Data compression algorithms?

q  User interfaces for submitting jobs, receiving results

q  Data visualization

ü  OpenSees + Condor

ü  Management and Analysis of Large Research Data Sets


[email protected]

uncertainty and sensitivity analysis using hpc and htc

Engineering

response edp lin r r

ni r r

sensitivityr r r

im p im im

edp p edp edp im f d

construction engineering

university of porto5

university of porto9