an hpc implementation of the finite element methodjohn-rugis/pdf/ernz2016.pdf · 2016. 2. 10. ·...

New Zealand eScience Infrastructure

New Zealand eScience Infrastructure

1

An HPC Implementation of the Finite Element Method John Rugis

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

David Yule Physiology

James Sneyd, Shawn Means, Di Zhu Mathematics

John Rugis Computer Science

Project funding:

2

Interdisciplinary research group:

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Physically accurate modeling, simulation and visualisation of biological cell function.

Scalable!

3

The science goal:

•  High Performance Computing

•  An Object Oriented Approach to Biological Cell Modeling

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

4

Salivary glands

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

•  Modeling equations (How does the system work?)

•  Mesh generation (What does the system look like?)

•  Discretisation (How can this be implemented on a computer?)

•  Run simulation (What does the system do?)

•  Evaluate results (What exactly happened?)

5

A Finite Element Method workflow overview

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Start with a conceptual model…

6

The Modeling Equations

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Calcium and IP3 dynamics Partial differential equations model the cell calcium dynamics.

Reaction-Diffusion:

7

The Modeling Equations

1 Generic PDE Model 01 v10.11.15

class II, negative feedback, closed cell, three variables

@c

@t

= Dcr2c+ (JIPR + Jleak)(ce � c)� Jserca (1)

@p

@t

= Dpr2p+ VPLC(~x)� Vdeg

✓c

2

K

23K + c

2

◆p (2)

@h

@t

=

h1 � h

⌧

(3)

Jserca = Vsc

2

K

2s + c

2(4)

JIPR = kIPR(~x)PO (5)

PO = �c�ph (6)

�c =c

3

K

3a + c

3(7)

�p =

p

4

K

4p + p

4(8)

h1 =

K

2i

K

2i + c

2(9)

ce = (ct � c)/� (10)

c 0.06(init) µM cytosolic Ca

2+concentration

p 0.26(init) µM IP3 concentration

h 0.334(init) - IPR modelling variable

� 0.185 - ratio of ER volume to cystolic volume

ct 5.0 µM total Ca

2+concentration

ce µM ER Ca

2+concentration

t s time

JIPR s

�1calcium from ER

kIPR(~x) 7.4(max) s

�1parameter (highest near apical region)

PO - open probability of IPR (range: 0.0 - 1.0)

�c - function of Ca

2+concentration

Ka 0.3 µM parameter

�p - function of IP3 concentration

Kp 0.5 µM parameter

h1 - function of Ca

2+concentration

Ki 0.06 µM parameter

⌧ 0.5 s

�1parameter

Dc 5 µm

2s

�1Ca

2+di↵usion coe�cient

Jserca µM s

�1calcium flux into ER

Vs 0.25 µM s

�1parameter

Ks 0.1 µM parameter

Jleak 0.00148 s

�1calcium from ER (to balance Jserca at rest)

VPLC(~x) 0.012(max) µM s

�1parameter (highest near basal membrane)

Vdeg 0.16 s

�1parameter

K3K 0.4 µM parameter

Dp 283 µm

2s

�1IP3 di↵usion coe�cient

1

1 Generic PDE Model 01 v10.11.15

class II, negative feedback, closed cell, three variables

@c

@t

= Dcr2c+ (JIPR + Jleak)(ce � c)� Jserca (1)

@p

@t

= Dpr2p+ VPLC(~x)� Vdeg

✓c

2

K

23K + c

2

◆p (2)

@h

@t

=

h1 � h

⌧

(3)

Jserca = Vsc

2

K

2s + c

2(4)

JIPR = kIPR(~x)PO (5)

PO = �c�ph (6)

�c =c

3

K

3a + c

3(7)

�p =

p

4

K

4p + p

4(8)

h1 =

K

2i

K

2i + c

2(9)

ce = (ct � c)/� (10)

c 0.06(init) µM cytosolic Ca

2+concentration

p 0.26(init) µM IP3 concentration

h 0.334(init) - IPR modelling variable

� 0.185 - ratio of ER volume to cystolic volume

ct 5.0 µM total Ca

2+concentration

ce µM ER Ca

2+concentration

t s time

JIPR s

�1calcium from ER

kIPR(~x) 7.4(max) s

�1parameter (highest near apical region)

PO - open probability of IPR (range: 0.0 - 1.0)

�c - function of Ca

2+concentration

Ka 0.3 µM parameter

�p - function of IP3 concentration

Kp 0.5 µM parameter

h1 - function of Ca

2+concentration

Ki 0.06 µM parameter

⌧ 0.5 s

�1parameter

Dc 5 µm

2s

�1Ca

2+di↵usion coe�cient

Jserca µM s

�1calcium flux into ER

Vs 0.25 µM s

�1parameter

Ks 0.1 µM parameter

Jleak 0.00148 s

�1calcium from ER (to balance Jserca at rest)

VPLC(~x) 0.012(max) µM s

�1parameter (highest near basal membrane)

Vdeg 0.16 s

�1parameter

K3K 0.4 µM parameter

Dp 283 µm

2s

�1IP3 di↵usion coe�cient

1

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

2D image stacks 32 slices in Z direction.

Cells & Lumen red: Na-KATPase green: Cl Channel

Real dimensions used: 70.7µm2, 2.2µm spacing

TIFF files: 1024x1024

8

Mesh Generation - Data Acquisition Confocal microscopy

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Cells & Lumen

The cells are grouped in tight clusters.

The lumen has a tree-like branching structure.

9

Mesh Generation - 3D Physical Model

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

10

Cells & Lumen

Each cell is held by a lumen “claw”.

The lumen has a central trunk.

Mesh Generation - 3D Physical Model

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

11

Mesh Generation - for FEM

Cells

Solid volumetric meshing with tetrahedrons.

Tetrahedrons have vertices and edges.

Element = tetrahedron

Node = vertex

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Facilitates (almost direct) translation to computer code.

12

Discretisation

•  Build “mass” and “load” matrices.

•  Construct (sparse) system matrix: ~ 400,000,000 elements!

NOTE: Most of the simulation run-time is spent solving the sparse system matrix.

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Tool selection

Sparse matrix solver: PETSc Portable, Extensible Toolkit for Scientific Computation http://www.mcs.anl.gov/petsc

Vector and matrix library: Eigen Eigen is a C++ template library for linear algebra http://eigen.tuxfamily.org

13

Computer Code

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Object oriented!

Language: C++

Once the behaviour of a cell “object” is defined, we can instantiate clusters of cells that interact with each other and their environment.

14

Computer Code

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

15

Computer Code Object definition

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

HPC – How can we scale up?

•  One cell object per compute node

•  Scale “out” (i.e. use more compute nodes)

•  MPI for cell interactions (light weight!)

•  Scale “in” (i.e. use more cores per compute node)

•  Threads for utility functions

•  Accelerators for the main solver (i.e. GPU’s, Intel Phi) 16

Computer Code

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

HPC - Auckland Pan Cluster

•  Job submission scripting (shell scripting and python)

•  Multiple parameter sweeps

•  File and directory structures facilitate reproducibility!

17

Simulation

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

18

Simulation Job submission script

(extract)

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Cell Nodal view

Precomputed values.

Imprint of the lumenal “claw”.

19

Evaluate Results

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

20

Cell Nodal view

Static distributions.

IPR in red.

PLC in blue.

Evaluate Results 100% lumen 60% lumen

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

Calcium waves Wave-fronts propagate from the apical end to the basal end the cell.

21

FEM Simulation Results

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

•  Run simulations with coupled cells (i.e. gap junctions)

•  Include fluid flow in lumen (computational fluid dynamics)

•  Higher resolution digitisations

•  Stay tuned, more to come…

22

What’s next?

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

23

Questions & Answers?

Tuesday 11:15 (Queenstown)

NeSI Update Nick Jones

Tuesday 11:45 (Remarkables)

Growing NZ’s Researcher’s Computing Capability

Georgina Rae & John Rugis

Tuesday 14:30 (Queenstown)

An HPC Implementation of the Finite Element Method

John Rugis

Tuesday 16:00 (Queenstown)

The NeSI National Platform Framework

Michael Uddstrom

Tuesday 16:30 (Queenstown)

Early Experiences with Cloud Bursting

Jordi Blasco

Tuesday 16:30 (Remarkables)

NeSI – NZGL Alliance Dan Sun & Nic Mair

Tuesday 17:00 (Queenstown)

Acceleration Made Easy Wolfgang Hayek