partitioned multi-physics on distributed data via precice · universitat stuttgart, technische...
TRANSCRIPT
![Page 1: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/1.jpg)
Universitat Stuttgart, Technische Universitat Munchen
SPPEXA APM 2016
Partitioned Multi-Physics on Distributed Data viapreCICE
Hans-Joachim Bungartz, Florian Lindner, Miriam Mehl,Klaudius Scheufele, Alexander Shukaev, Benjamin Uekermann
Universitat Stuttgart, Technische Universitat Munchen
January 25, 2016
Uekermann et al.: Partitioned Multi-Physics on Distributed Data via preCICE
SPPEXA APM 2016, January 25, 2016 1
![Page 2: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/2.jpg)
Multi-Physics and Exa-Scale
I many exciting applications need multi-physics
I more compute power ⇒ more physics?
I many sophisticated, scalable,single-physics, legacy codes
I our goal: minimal invasive coupling,no deterioration of scalability
![Page 3: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/3.jpg)
Our Example: Fluid-Structure-Acoustic Interaction
Structure
Fluid
Acoustic
I FEAP - FASTEST - Ateles
I OpenFOAM - OpenFOAM - Ateles
I glue-software: preCICE
![Page 4: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/4.jpg)
Our Example: Fluid-Structure-Acoustic Interaction
FS
AFSFSFSFSFS
A A A A A A A AFS
I implicit or explicit coupling
I subcycling
![Page 5: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/5.jpg)
Agenda
This talk: glue-software preCICENext talk (Verena Krupp): application perspective
1. Very brief introduction to preCICE
2. Realization on Distributed Data
3. Performance on Distributed Data
![Page 6: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/6.jpg)
Agenda
This talk: glue-software preCICENext talk (Verena Krupp): application perspective
1. Very brief introduction to preCICE
2. Realization on Distributed Data
3. Performance on Distributed Data
![Page 7: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/7.jpg)
preCICE
I precise Code Interaction Coupling Environment
I developed in Munich and Stuttgart
I library approach, minimal invasive
I high-level API in C++, C, Fortran77/90/95,Fortran2003
I once adapter written ⇒ plug and play
I https://github.com/precice/precice
![Page 8: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/8.jpg)
Big Picture
SOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
![Page 9: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/9.jpg)
Coupled Solvers
Ateles (APES) CF, A in-house (U Siegen)
Alya System IF, S in-house (BSC)
Calculix S open-source (A*STAR)
CARAT S in-house (TUM STATIK)
COMSOL S commercial
EFD IF in-house (TUM SCCS)
FASTEST IF in-house (TU Darmstadt)
Fluent IF commercial
OpenFOAM CF, IF, S open-source (TU Delft)
Peano 1 IF in-house (TUM SCCS)
SU2 CF open-source
![Page 10: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/10.jpg)
preCICE API
turn on()
while time loop 6= done do
solve timestep()
end whileturn off()
![Page 11: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/11.jpg)
preCICE API
turn on()precice create(“FLUID”, “precice config.xml”, index, size)
precice initialize()while time loop 6= done do
solve timestep()
end whileturn off()precice finalize()
![Page 12: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/12.jpg)
preCICE API
turn on()precice create(“FLUID”, “precice config.xml”, index, size)
precice initialize()while time loop 6= done dowhile precice action required(readCheckPoint) do
solve timestep()precice advance()
end whileend whileturn off()precice finalize()
![Page 13: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/13.jpg)
preCICE API
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done dowhile precice action required(readCheckPoint) do
solve timestep()precice advance()
end whileend whileturn off()precice finalize()
![Page 14: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/14.jpg)
preCICE API
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done dowhile precice action required(readCheckPoint) do
precice write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()precice read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize()
![Page 15: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/15.jpg)
preCICE Config
![Page 16: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/16.jpg)
Communication on Distributed Data
coupling surface
solver A
solver B
![Page 17: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/17.jpg)
Communication on Distributed Data
A acceptor B requestor
2
1
0
4
3
2
1
0
I kernel: 1:N communication based on either TCP/IP or MPI Ports⇒ no deadlocks at initialization (independent of order at B)
I asynchronous communication (preferred over threads)⇒ no deadlocks at communication
![Page 18: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/18.jpg)
Interpolation on Distributed Data
Projection-based Interpolation
I first or second order
I example: consistent mapping from B to A
I parallelization: almost trivial
B
A
B
A
nearest neighbour mapping nearest projection mapping
Radial Basis Function Interpolation
I higher order
I parallelization: far from trivial (realized with PETSc)
![Page 19: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/19.jpg)
Interpolation on Distributed Data
Projection-based Interpolation
I first or second order
I example: consistent mapping from B to A
I parallelization: almost trivial
B
A
B
A
nearest neighbour mapping nearest projection mapping
Radial Basis Function Interpolation
I higher order
I parallelization: far from trivial (realized with PETSc)
![Page 20: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/20.jpg)
Fixed-Point Acceleration on Distributed Data
Anderson Acceleration (IQN-ILS)
Find x ∈ D ⊂ Rn : H(x) = x , H : D → Rn
initial value x0
x0 = H(x0) and R0 = x0 − x0
x1 = x0 + 0.1 · R0
for k = 1 . . . doxk = H(xk) and Rk = xk − xk
V k = [∆Rk0 , . . . ,∆Rk
k−1] with ∆Rki = R i − Rk
Wk = [∆xk0 , . . . ,∆xkk−1] with ∆xki = x i − xk
decompose V k = QkUk
solve the first k lines of Ukα = −QkTRk
∆x = Wαxk+1 = xk + ∆xk
end for
![Page 21: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/21.jpg)
Fixed-Point Acceleration on Distributed Data
= v
r
0 = V
R
V Q
R
Q^
^ ^
^
Insert Column
In: Q ∈ Rn×m, R ∈ Rm×m, v ∈ Rn, Out: Q ∈ Rn×(m+1), R ∈ R(m+1)×(m+1)
for j = 1 . . .m dor(j) = Q(:, j)T vv = v − r(j) · Q(:, j)
end forr(m + 1) = ‖v‖2
Q(:,m + 1) = r(m + 1)−1 · v
R =
[r ,
(R0
)]Given’s rotations Gi,j s.t. R = G1,2 . . .Gm,m+1R upper triangleQ = QGm,m+1 . . .G1,2
![Page 22: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/22.jpg)
Performance Tests, Initialization
#DOFs at interface: 2.6 · 105, strong scaling
I II III IV V com ILS NN
Tim
e [m
s]
10 0
10 1
10 2
10 3
10 4
10 5
p=128p=256p=512p=1024p=2048
![Page 23: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/23.jpg)
Performance Tests, Work per Timestep
#DOFs at interface: 2.6 · 105, strong scaling
com ILS NN
Tim
e [m
s]
1
2
3
4
5
6
7
8
9
10p=128p=256p=512p=1024p=2048
![Page 24: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/24.jpg)
Performance Tests, Traveling Pulse
DG solver Ateles, Euler equations,#DOFs: total: 5.9 · 109, at interface: 1.1 · 107,NN mapping and communicationstrong scaling from 128 to 16384 processors per participant.
Ateles Left
Ateles Right
preC
ICE
xy
Z
Joint work with V. Krupp et al. (Universitat Siegen)
![Page 25: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/25.jpg)
Performance Tests, Work per Timestep
27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)
![Page 26: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/26.jpg)
Summary
Structure
Fluid
AcousticSOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done do
while precice action required(readCheckPoint) dopre write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()pre read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize() 27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)
![Page 27: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/27.jpg)
Summary
Structure
Fluid
Acoustic
SOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done do
while precice action required(readCheckPoint) dopre write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()pre read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize() 27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)
![Page 28: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/28.jpg)
Summary
Structure
Fluid
AcousticSOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done do
while precice action required(readCheckPoint) dopre write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()pre read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize() 27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)
![Page 29: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/29.jpg)
Summary
Structure
Fluid
AcousticSOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done do
while precice action required(readCheckPoint) dopre write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()pre read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize()
27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)
![Page 30: Partitioned Multi-Physics on Distributed Data via preCICE · Universitat Stuttgart, Technische Universit¨ at M¨ ¨unchen SPPEXA APM 2016 Partitioned Multi-Physics on Distributed](https://reader033.vdocument.in/reader033/viewer/2022052105/60407c5c603ee3749076258b/html5/thumbnails/30.jpg)
Summary
Structure
Fluid
AcousticSOLVER A.1
ADAPTER
STEERING
EQUATION COUPLING
COMMUNICATION
DATA MAPPING
MASTER
SOLVER A.2
ADAPTER
SLAVE
SOLVER A.N
ADAPTER
SLAVE
SOLVER B.1
ADAPTER
STEERING
MASTER
SOLVER B.2
SLAVE
SOLVER B.M
SLAVE
ADAPTER
ADAPTER
turn on()precice create(“FLUID”, “precice config.xml”, index, size)precice set vertices(meshID, N, pos(dim*N), vertIDs(N))precice initialize()while time loop 6= done do
while precice action required(readCheckPoint) dopre write bvdata(stresID,N,vertIDs,stres(dim*N))solve timestep()precice advance()pre read bvdata(displID,N,vertIDs,displ(dim*N))
end whileend whileturn off()precice finalize() 27 28 29 210 211 212 213 214100
101
102
103
104
105
106
6830335590
183958389
40701954
1002544606
175
2859
15 20 18
5
Number of Processes per Participant (NPP)
Tim
e[m
s]
Compute (Ateles)
Advance (preCICE)