an hpc implementation of the finite element methodjohn-rugis/pdf/ernz2016.pdf · 2016. 2. 10. ·...

New Zealand eScience Infrastructure

New Zealand eScience Infrastructure

1

An HPC Implementation of the Finite Element Method John Rugis

New Zealand eScience Infrastructure New Zealand eScience Infrastructure

David Yule Physiology

James Sneyd, Shawn Means, Di Zhu Mathematics

John Rugis Computer Science

Project funding:

2

Interdisciplinary research group:


Physically accurate modeling, simulation and visualisation of biological cell function.

Scalable!

3

The science goal:

•  High Performance Computing

•  An Object Oriented Approach to Biological Cell Modeling


4

Salivary glands


•  Modeling equations (How does the system work?)

•  Mesh generation (What does the system look like?)

•  Discretisation (How can this be implemented on a computer?)

•  Run simulation (What does the system do?)

•  Evaluate results (What exactly happened?)

5

A Finite Element Method workflow overview


Start with a conceptual model…

6

The Modeling Equations


Calcium and IP3 dynamics Partial differential equations model the cell calcium dynamics.

Reaction-Diffusion:

7

The Modeling Equations

1 Generic PDE Model 01 v10.11.15

class II, negative feedback, closed cell, three variables

@c

@t

= Dcr2c+ (JIPR + Jleak)(ce � c)� Jserca (1)

@p

@t

= Dpr2p+ VPLC(~x)� Vdeg

✓c

2

K

23K + c

2

◆p (2)

@h

@t

=

h1 � h

⌧

(3)

Jserca = Vsc

2

K

2s + c

2(4)

JIPR = kIPR(~x)PO (5)

PO = �c�ph (6)

�c =c

3

K

3a + c

3(7)

�p =

p

4

K

4p + p

4(8)

h1 =

K

2i

K

2i + c

2(9)

ce = (ct � c)/� (10)

c 0.06(init) µM cytosolic Ca

2+concentration

p 0.26(init) µM IP3 concentration

h 0.334(init) - IPR modelling variable

� 0.185 - ratio of ER volume to cystolic volume

ct 5.0 µM total Ca

2+concentration

ce µM ER Ca

2+concentration

t s time

JIPR s

�1calcium from ER

kIPR(~x) 7.4(max) s

�1parameter (highest near apical region)

PO - open probability of IPR (range: 0.0 - 1.0)

�c - function of Ca

2+concentration

Ka 0.3 µM parameter

�p - function of IP3 concentration

Kp 0.5 µM parameter

h1 - function of Ca

2+concentration

Ki 0.06 µM parameter

⌧ 0.5 s

�1parameter

Dc 5 µm

2s

�1Ca

2+di↵usion coe�cient

Jserca µM s

�1calcium flux into ER

Vs 0.25 µM s

�1parameter

Ks 0.1 µM parameter

Jleak 0.00148 s

�1calcium from ER (to balance Jserca at rest)

VPLC(~x) 0.012(max) µM s

�1parameter (highest near basal membrane)

Vdeg 0.16 s

�1parameter

K3K 0.4 µM parameter

Dp 283 µm

2s

�1IP3 di↵usion coe�cient

1

1 Generic PDE Model 01 v10.11.15

class II, negative feedback, closed cell, three variables

@c

@t

= Dcr2c+ (JIPR + Jleak)(ce � c)� Jserca (1)

@p

@t

= Dpr2p+ VPLC(~x)� Vdeg

✓c

2

K

23K + c

2

◆p (2)

@h

@t

=

h1 � h

⌧

(3)

Jserca = Vsc

2

K

2s + c

2(4)

JIPR = kIPR(~x)PO (5)

PO = �c�ph (6)

�c =c

3

K

3a + c

3(7)

�p =

p

4

K

4p + p

4(8)

h1 =

K

2i

K

2i + c

2(9)

ce = (ct � c)/� (10)

c 0.06(init) µM cytosolic Ca

2+concentration

p 0.26(init) µM IP3 concentration

h 0.334(init) - IPR modelling variable

� 0.185 - ratio of ER volume to cystolic volume

ct 5.0 µM total Ca

2+concentration

ce µM ER Ca

2+concentration

t s time

JIPR s

�1calcium from ER

kIPR(~x) 7.4(max) s

�1parameter (highest near apical region)

PO - open probability of IPR (range: 0.0 - 1.0)

�c - function of Ca

2+concentration

Ka 0.3 µM parameter

�p - function of IP3 concentration

Kp 0.5 µM parameter

h1 - function of Ca

2+concentration

Ki 0.06 µM parameter

⌧ 0.5 s

�1parameter

Dc 5 µm

2s

�1Ca

2+di↵usion coe�cient

Jserca µM s

�1calcium flux into ER

Vs 0.25 µM s

�1parameter

Ks 0.1 µM parameter

Jleak 0.00148 s

�1calcium from ER (to balance Jserca at rest)

VPLC(~x) 0.012(max) µM s

�1parameter (highest near basal membrane)

Vdeg 0.16 s

�1parameter

K3K 0.4 µM parameter

Dp 283 µm

2s

�1IP3 di↵usion coe�cient

1


2D image stacks 32 slices in Z direction.

Cells & Lumen red: Na-KATPase green: Cl Channel

Real dimensions used: 70.7µm2, 2.2µm spacing

TIFF files: 1024x1024

8

Mesh Generation - Data Acquisition Confocal microscopy


Cells & Lumen

The cells are grouped in tight clusters.

The lumen has a tree-like branching structure.

9

Mesh Generation - 3D Physical Model


10

Cells & Lumen

Each cell is held by a lumen “claw”.

The lumen has a central trunk.

Mesh Generation - 3D Physical Model


11

Mesh Generation - for FEM

Cells

Solid volumetric meshing with tetrahedrons.

Tetrahedrons have vertices and edges.

Element = tetrahedron

Node = vertex


Facilitates (almost direct) translation to computer code.

12

Discretisation

•  Build “mass” and “load” matrices.

•  Construct (sparse) system matrix: ~ 400,000,000 elements!

NOTE: Most of the simulation run-time is spent solving the sparse system matrix.


Tool selection

Sparse matrix solver: PETSc Portable, Extensible Toolkit for Scientific Computation http://www.mcs.anl.gov/petsc

Vector and matrix library: Eigen Eigen is a C++ template library for linear algebra http://eigen.tuxfamily.org

13

Computer Code


Object oriented!

Language: C++

Once the behaviour of a cell “object” is defined, we can instantiate clusters of cells that interact with each other and their environment.

14

Computer Code


15

Computer Code Object definition


HPC – How can we scale up?

•  One cell object per compute node

•  Scale “out” (i.e. use more compute nodes)

•  MPI for cell interactions (light weight!)

•  Scale “in” (i.e. use more cores per compute node)

•  Threads for utility functions

•  Accelerators for the main solver (i.e. GPU’s, Intel Phi) 16

Computer Code


HPC - Auckland Pan Cluster

•  Job submission scripting (shell scripting and python)

•  Multiple parameter sweeps

•  File and directory structures facilitate reproducibility!

17

Simulation


18

Simulation Job submission script

(extract)


Cell Nodal view

Precomputed values.

Imprint of the lumenal “claw”.

19

Evaluate Results


20

Cell Nodal view

Static distributions.

IPR in red.

PLC in blue.

Evaluate Results 100% lumen 60% lumen


Calcium waves Wave-fronts propagate from the apical end to the basal end the cell.

21

FEM Simulation Results


•  Run simulations with coupled cells (i.e. gap junctions)

•  Include fluid flow in lumen (computational fluid dynamics)

•  Higher resolution digitisations

•  Stay tuned, more to come…

22

What’s next?


23

Questions & Answers?

Tuesday 11:15 (Queenstown)

NeSI Update Nick Jones

Tuesday 11:45 (Remarkables)

Growing NZ’s Researcher’s Computing Capability

Georgina Rae & John Rugis


An HPC Implementation of the Finite Element Method

John Rugis


The NeSI National Platform Framework

Michael Uddstrom


Early Experiences with Cloud Bursting

Jordi Blasco

Tuesday 16:30 (Remarkables)

NeSI – NZGL Alliance Dan Sun & Nic Mair


Acceleration Made Easy Wolfgang Hayek

an hpc implementation of the finite element methodjohn-rugis/pdf/ernz2016.pdf · 2016. 2. 10. ·...

Documents