anemos parallel mesh partitioning and adaptationnkonga/anemos/lachat_janvier2013.pdf · 2013. 4....
TRANSCRIPT
Cedric Lachat & Francois Pellegrini & Cecile Dobrzynski January 31, 2013
ANEMOS
PaMPA :Parallel MeshPartitioning and Adaptation
Common needs of solvers regarding meshes
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 2
Common needs of solvers regarding meshes
Common needs of solvers regarding meshes
I Handling of mesh structuresI Distribution of meshes across the processors of a parallel
architectureI Handling of load balance
I Data exchange across neighboring entitiesI Iteration on mesh entities
I Entities of any kind: e.g. elements, faces, edges, nodes, . . .I Entity sub-classes: e.g. regular or boundary faces, . . .I Inner or frontier entities with respect to neighboring processorsI Maximization of cache effects thanks to proper data reordering
I Dynamic modification of mesh structureI Dynamic redistribution
I Adaptive remeshing
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 3
What is PaMPA
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 4
What is PaMPA
What is PaMPA
I PaMPA: “Parallel Mesh Partitioning and Adaptation”I Middleware library managing the parallel repartitioning and
remeshing of unstructured meshes modeled as interconnectedvaluated entities
I The user can focus on his/her “core business”:I SolverI Sequential remesher
I Coupling with MMG3D provided for tetrahedra
Physics, solver
Remeshing and redistribution
PT-Scotch
Seq. Qual.Measurement
Sequentialremesher
MMG3D
PaMPA
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 5
What is PaMPA
Features of version 0.2
I Overlap greater than 1I Parallel I/OI Parallel partitioningI Parallel mesh adaptation based on sequential remesher
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 6
Data structures
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 7
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Top-level mesh entityMay bear some data (volume,
pressure, etc.)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
May bear some data (geometry,etc.)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
May bear some data (flux, etc.)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Regular mesh edge
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Boundary mesh edge
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
What all entities are in fact. . .
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Subset of edges between verticesbelonging to prescribed entity types
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Subset of vertices bearing the samedata
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Subset of entity vertices that maybear additional specific data
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Definitions
I Mesh:I ElementI NodeI Edge
I InternalI Boundary
I PaMPA Mesh:I VertexI RelationI EntityI Sub-entityI Enriched graph
Whole set of vertices and relationsEvery vertex belongs to one and only
one entity (and sub-entity)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 8
Data structures
Global vue
I All vertices have a global unique number
baseval
enttglbnbr
proccnttab
procvrttab
1
3
3 4 3
1 4 8 11
5
1
2 9
7
4
3 6 8
10
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 9
Data structures
Local vision of process 0
I All local and ghost vertices have a compact local indexI Per-entity numbering
vertgstnbr
vertlocnbr
edgelocnbr
ventloctab
edgeloctab
vertloctab
vendloctab
6
3
7
3 1
1
1 1
2 3 8
1 2 3 4 2
3
4
1
2
3
1 2
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 10
Data structures
Local vision of process 1
I All local and ghost vertices have a compact local indexI Per-entity numbering
vertgstnbr
vertlocnbr
edgelocnbr
ventloctab
edgeloctab
vertloctab
vendloctab
7
4
3 13 2
13
1
2
3 6
1 2 1 3 2 1 2 1 3 1
1412
3 3
23
1
1
2 1 3
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 11
Data structures
Linking values to entities
I Multiple value types can be associated with each entityI Value data structures can be split to improve cache usage
I Entities without values serve as iterators
5
1
2 9
7
4
3 6 8
10
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 12
Example: Laplacian equation using P1 finite element method
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 13
Example: Laplacian equation using P1 finite element method
Definition
I Solving 2D Poisson equation:I ∆u (x , y) = f (x , y)I g (x , y) = u (x , y) on the
boundary Γ
I Test case:I f (x , y) = −2∗cos (x)∗cos (y)
in the domain ΩI g (x , y) = cos (x) ∗ cos (y) on
the boundary ΓI u (x , y) = cos (x) ∗ cos (y)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 14
Example: Laplacian equation using P1 finite element method
Mesh properties
I Entities:I ElementsI NodesI Boundary edges
I Relations:I Element to elementI Element to nodeI Element to boundary edgeI Node to node
I Overlap of size 1I Values:
I Coordinates and solution on nodesI Type on boundary edgesI Area, volume on elements
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 15
Example: Laplacian equation using P1 finite element method
All steps
! On a l l p r o c e s s o r s :CALL D i s t r i b u t e d M e s h ( ) ! B u i l d PaMPA d i s t r i b u t e d mesh :! 1− Read i n p a r a l l e l a c e n t r a l i z e d mesh! 2− C a l l PaMPA mesh p a r t i o n e r! 3− R e d i s t r i b u t e d i s t r i b u t e meshCALL ElementVolume ( )CALL I n i t i a l i z e M a t r i x C S R ( )
! S o l u t i o n computat ion!−−−−−−−−−−−−−−−−−−−−−
CALL I n i t S o l ( )CALL F i l l M a t r i x ( )CALL So lveSystem ( )CALL W r i t e D i s t r i b u t e d M e s h A n d S o l F i l e s ( )
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 16
Example: Laplacian equation using P1 finite element method
FillMatrix
RHS = 0 .CALL PAMPAF dmeshIt In i tStart (dm, ENTITY ELEM , PAMPAF VERT ANY, i t v r t , i e r r )CALL PAMPAF dmeshItInit (dm, ENTITY ELEM , ENTITY NODE , i t n g b , i e r r )DO WHILE ( PAMPAF itHasMore ( i t v r t ) )
j t = PAMPAF itCurEnttVertNum ( i t v r t )V o l t = V o l E l ( j t )ngb = 0CALL PAMPAF itStart ( i t n g b , j t , i e r r )DO WHILE ( PAMPAF itHasMore ( i t n g b ) )
ngb = ngb + 1i s = PAMPAF itCurEnttVertNum ( i t n g b )NuElemt ( ngb ) = i sCoorElemt ( : , ngb ) = Coor ( : , i s )PAMPAF itNext ( i t n g b )
END DOCALL GradPhi ( CoorElemt ( : , 1 ) , CoorElemt ( : , 2 ) , CoorElemt ( : , 3 ) , GrdPhi )DO i = 1 , Nsmplx
i s = NuElemt ( i )DO j = 1 , Nsmplx
j s = NuElemt ( j )JJac = V o l t ∗ Sum ( GrdPhi ( : , i ) ∗ GrdPhi ( : , j ) )CALL assembly addCSR ( JJac , i s , j s )
END DO ! l oop on jRHS( i s ) = RHS( i s ) − V o l t ∗ SourceTerm ( Coor ( 1 , i s ) , Coor ( 2 , i s ) ) / Nsmplx
END DO ! l oop on iPAMPAF itNext ( i t v r t )
END DO
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 17
Example: Laplacian equation using P1 finite element method
Solve system: Jacobi (1/2)
UaPrec = 0 . ! Suppose A = L + D + U, system to s o l v e : A x = bCALL PAMPAF dmeshItInit (dm, ENTITY NODE , ENTITY NODE , i t n g b , i e r r )DO i r e l a x = 1 , N r e l a x
r e s = 0 .CALL PAMPAF dmeshIt In i tStart (dm, ENTITY NODE , PAMPAF VERT BOUNDARY, i t v r t , i e r r )DO WHILE ( PAMPAF itHasMore ( i t v r t ) )
i s = PAMPAF itCurEnttVertNum ( i t v r t )CALL PAMPAF dmeshMatLineData (dm, ENTITY NODE , i s , I1 , I 1 F i n , i e r r )CALL PAMPAF itStart ( i t n g b , i s , i e r r )r e s 0 = RHS( i s ) ! r e s 0 = b
i v = i 1DO WHILE ( PAMPAF itHasMore ( i t n g b ) )
j s = PAMPAF itCurEnttVertNum ( i t n g b )PAMPAF itNext ( i t n g b )r e s 0 = r e s 0 − MatCSR%V a l s ( i v ) ∗ UaPrec ( j s ) ! r e s 0 = b − ( L + U) xˆni v = i v + 1
END DOUa( i s ) = r e s 0 / MatCSR%Diag ( i s ) ! xˆn+1 = ( b − ( L + U) xˆn )/DPAMPAF itNext ( i t v r t )
END DO
CALL PAMPAF dmeshHaloValueAsync (dm, ENTITY NODE , PAMPA TAG SOL , req , i e r r )
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 18
Example: Laplacian equation using P1 finite element method
Solve system: Jacobi (2/2)
CALL PAMPAF dmeshIt In i tStart (dm, ENTITY NODE , PAMPAF VERT INTERNAL , i t v r t , i e r r )DO WHILE ( PAMPAF itHasMore ( i t v r t ) )
i s = PAMPAF itCurEnttVertNum ( i t v r t )CALL PAMPAF dmeshMatLineData (dm, ENTITY NODE , i s , I1 , I 1 F i n , i e r r )CALL PAMPAF itStart ( i t n g b , i s , i e r r )r e s 0 = RHS( i s ) ! r e s 0 = b
i v = i 1DO WHILE ( PAMPAF itHasMore ( i t n g b ) )
j s = PAMPAF itCurEnttVertNum ( i t n g b )PAMPAF itNext ( i t n g b )r e s 0 = r e s 0 − MatCSR%V a l s ( i v ) ∗ UaPrec ( j s ) ! r e s 0 = b − ( L + U) xˆni v = i v + 1
END DOUa( i s ) = r e s 0 / MatCSR%Diag ( i s ) ! xˆn+1 = ( b − ( L + U) xˆn )/DPAMPAF itNext ( i t v r t )
END DO
CALL PAMPAF dmeshHaloWait ( req , i e r r )
UaPrec = UaEND DO ! end loop on i r e l a x
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 19
Some results
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 20
Some results
First results
I Adapt in parallel a mesh: 2M to 190M tetrahedra (on 64 to256 processors)
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 21
Work in progress
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 22
Work in progress
Work in progress
I Release of version 0.2I Available soon from Inria GforgeI Licensed under GPL
I Quality of parallel adapted meshesI Periodic meshes
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 23
Upcoming features
Contents
Common needs of solvers regarding meshes
What is PaMPA
Data structures
Example: Laplacian equation using P1 finite element method
Some results
Work in progress
Upcoming features
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 24
Upcoming features
Upcoming features
I Mesh definition with a grammarI Face orientation and displacementI Unbreakable relations
I Partitioner will not cut these edgesI E.g. to implement DG methods
I Multi-grid meshesI Parallel I/O with HDF5I Parallel mesh adaptation scalability
ANEMOS - C. Lachat & F. Pellegrini & C. Dobrzynski - PaMPA January 31, 2013- 25
Upcoming features
Thank you
Franois Pellegrini January 31, 2013