high-performance computing (hpc) is transforming seismology

High-Performance Computing (HPC)IS

Transforming Seismology

Southern San Andreas Earthquake• M 7.7, scaled Denali slip• SCEC CVM3 (600 km x 300 km x 80 km)• 3000 x 1500 x 400 = 1.8 G nodes (200 m)• 20,000 time steps (0.01 s)• 19,000 SU per run• 47 TB of simulation data (150,000 files) per run

TeraShake 1 (Olsen et al. 2006) 1012 flops

Energy Funneling Effect (Olsen et al., 2006)

Blue: data

Red: synthetic

16 Jun 2005, ML4.9, Yucaipa earthquake

Data

Synthetic

Reference model: SCEC Community Velocity Model 3.0

HPC makes seismic wave propagation simulations more realistic and more accurate, opens up the possibility for physics-based, deterministic, seismic hazard analysis.

Let’s watch a video made by SCEC.

Two Problem Areas1. Develop simulation capability for physics-based seismic hazard and risk analysis

- TeraShake platform- CyberShake project

2. Improve physical models for SHA- Inversion of large data sets for Unified Structural Representation

SCEC computational pathways

AWM: Anelastic Wave ModelFSM: Fault-system ModelRDM: Rupture Dynamics ModelSRM: Site-response Model

Realistic 3D Earth Structure Model (CVM)

+ High-Performance Computing

(HPC)=

CyberShake

€

G(x i;x r )

€

G(x r;x i) = [G(x i;x r)]T

Receiver Green Tensor (RGT)• Obtain Green tensors from a receiver to all grid points by finite difference simulations (3 runs for 3 orthogonal forces at receiver).

• Reciprocity states that the Green tensors from all the grid points to the receiver is the transpose of the RGT obtained above.

• Synthetic seismograms due to an arbitrary point source s at receiver r and their gradients with respect to source locations can be retrieved from the RGT database.

3D Earth Structural Model

rS(l-1, m, n)

(l, m, n)(l+1, m, n)

h(l, m+1, n)

(l, m-1, n)

(l, m, n-1)

(l, m, n+1)

Red dash line: synthetics from RGT and reciprocity

Blue solid line: synthetics from forward wave propagation

Confirm Reciprocity

Numerical differentiation to get receiver strain Green tensor

Yorba Linda Earthquake to basin station BRE

Physics-based Seismic Hazard Analysis

(CyberShake)

€

u(x r, t) = dV (x s) dts∫ ∫ M(x s,ts) :∇ sGT (x s,t − ts;x r)€

∇sGT (x s,t − ts;x r )

€

M(x s,ts)

€

u(x r, t)

Callaghan et al. (2006)

Red: empirical ground motion model (Abrahamson & Silva 1997)

Black: CyberShake (Callaghan 2006)

Two Problem Areas1. Develop simulation capability for physics-based seismic hazard and risk analysis

- TeraShake platform- CyberShake project

2. Improve physical models for SHA- Inversion of large data sets for Unified Structural Representation

SCEC computational pathways

AWM: Anelastic Wave ModelFSM: Fault-system ModelRDM: Rupture Dynamics ModelSRM: Site-response Model

1 2 3 4 5 6 7 8Magnitude

Centroid Moment Tensor (CMT)

Finite Moment Tensor (FMT)

Isotropic Point

Source (IPS)

FaultSlip

Distribution (FSD)

(5) (8-10) (13-20) (>100)

Number of parameters

Seismic Source Parameter Inversion

€

u(x r, t) = dV (x s) dts∫ ∫ M(x s, ts) :∇ sGT (x s, t − ts;x r)

Numerical tests to verify inversion algorithm

Waveform inversion using 3D RGT synthetics .vs. first-motion focal mechanisms

Rapid CMT Inversion Using Waveforms computed in a 3D Earth Structural Model

Yorba Linda Cluster

Fontana Trend

A new left-lateral fault?

Resolving Fault-plane-ambiguity for Small Earthquakes

A new representation of finite moment tensor

Fréchet Kernel for Full-wave Tomography

Born Approximation:

Born Kernels

Seismogram perturbation kernel:

Data functional:

Fréchet kernel:

Receiver Green Tensor

δtp= -0.4 (s)

∫⊕

= rrr dKt mpp )()( δαδ

LAB Inversion Computational Cost For One GN Iteration

F3DT for Southern California (TERA3D)

• Target frequency: 1.0 Hz for body-waves and 0.5 Hz for surface waves

• Starting model: 3D SCEC CVM4

• Grid-spacing 200m, spatial grid points 1871M

• 150 stations, 200 earthquakes, 650 simulations, 5.2M CPU-Hrs

• Octree-based data compression, 895TB storage