integrating adios into openfoam for disk i/oopenfoam.com/documentation/files/adios-201610.pdf ·...

www.esi-group.com

Copyright © ESI Group, 2016. All rights reserved.

1www.esi-group.com


Integrating ADIOS into OpenFOAM for Disk I/OAn update and outlook

Karl Meredith, Mark Olesen, Norbert Podhorszki

FM Global, ESI-OpenCFD, Oak Ridge National Laboratory

www.esi-group.com


2

What is the problem?

• I/O performance issues with larger parallel cases

• Too many (small) files.

Overwhelms filer.

Quota issues.

• Simple example:

9 fields, 240 processor= 2160 files per checkpoint.

With 1000, 2000 processors...?

• Add into OpenFOAM?

No centralized I/O, but the object-registry is often used.

Background

www.esi-group.com


3

FM Global using ORNL resources

• Norbert:I/O bottleneck?Sounds familiar – let's try ADIOS.

• ADIOS = Adaptable Input/Output System

• Meta-layer for coordinated I/O from parallel simulations

• Has a lean 'BP' (binary packed) format.

Self-describing.

Contents entirely user-defined.

• Supports various parallel file-systems.

• Why not HDF5?

Better scaling, better throughput.

Simple API. Flexible content. Good traction.

Background

Already: ZFP supportUpcoming: ADIOS2 in C++11

www.esi-group.com


4

Transport Methods

• MPI

All processes write to a single file.

Good for small scale (1000 processors).

• POSIX

Each process writes a file independently.

Scales well until the meta-data limits on file-system(several thousand processors).

Works well on Luster.

• MPI-Aggregate

Data are aggregated for a group of processes.

Works well for 10's thousands of processors.

• Others...

ADIOS

www.esi-group.com


5

Transport Methods

• MPI

N procs -> 1 file

• POSIX

N procs -> N files

• MPI-Aggregate

N procs -> M files

M = number of aggregators

• Can write any number of time-steps into the file.

• For OpenFOAM, makes sense to write per time-step

Restart, removal of results etc.

ADIOS

www.esi-group.com


6

How/where to start?

• Function Object

Minimally invasive.

Rapid prototyping.

Ensure all details work well.

• Guiding Idea

Alternative for some OpenFOAM I/O.

Don't replace everything.

• Concentrate on the big offenders:

Meshes, Volume fields, Lagrangian.

• Should be relatively simple to use.

Implementation

www.esi-group.com


7

File Content / Layout Separated per time-step.

• ADIOS is very flexible.

Attributes are common for all processes.

Variables are data containers (byte, float,…)

• Internal Naming

Accommodate multi-region, multi-cloud, moving mesh ...

Enough description to document content, allow reuse ...

• Content

Meshes can use primitives (int, double).

Fields use raw byte content

▪ Parses identically to standard OpenFOAMbinary output stream.

Implementation

www.esi-group.com


8

File Content Layout

• Lagrangian

Much trickier.

Various cloud / thermodynamics types.

Variable length records

▪ Collisions, multiple-species.

Implementation

www.esi-group.com


9

General Adapter Layer

• Output

Stream through a counter.

Stream to an internal buffer.

Pass to ADIOS.

• Input

ADIOS -> data buffer.

Istream adapter on buffer -> OpenFOAM.

• Lagrangian

Encoding/decoding layer just added for downstream consumers.

Implementation

www.esi-group.com


10

Output Behaviour

• Query object-registry for AUTO_WRITE

Meshes.

Volume/surface fields.

Clouds.

• Optionally

Specify additional fields.

Restrict to explicitly named fields.

Blacklist fields (eg, reduced output for post-pro)

• Output file contains the mesh, or time values for the last points / topology written.

Function Object Control

www.esi-group.com


11

Restart Behaviour

• The boot-strapping problem.

• Use hybrid fast-forward approach.

Normal mechanisms for initial mesh, initial fields.

ADIOS data to jump forward in time.

Move points (currently w/o top change).

Update fields.

Recreate clouds.

Function Object Control

www.esi-group.com


12

Using interFoam

• 20.3M cell VOF case

• Intel(R) Xeon(R) E5-2630 v3 @ 2.4 GHz

• NFS file system

• 24 processors + 240 processors

Demonstration Case

www.esi-group.com


13

24 Processors Performance

• Test study

100 time steps.

Writing each time step.

• OpenFOAM

1080 s: no write

1647 s: write

• ADIOS transport

1251 s: null (no write)

1681 s: MPI

1670 s: POSIX

Demonstration Case

www.esi-group.com


14

240 Processors Performance

• Test study

100 time steps.

Writing each time step.

• OpenFOAM

178 s: no write

3426 s: write

• ADIOS transport

226 s: null (no write)

1614 s: MPI

1318 s: POSIX

Demonstration Case

www.esi-group.com


15

Output size and count

• Native output

2.2 GB

2160 files per checkpoint (9 x 240)

• ADIOS MPI

2.4 GB

1 bp file per checkpoint

• ADIOS POSIX

2.4 GB

240 bp files per checkpoint

Demonstration Case

www.esi-group.com


16

Performance / Potential Development

• Performance advantages shown

Better parallel scaling.

I/O bottleneck removed == 3x wall-clock speedup (not many processes).

Further optimizations possible for much larger benefit.

• Checkpoint restart possible

Bootstrap using native OpenFOAMmechanisms.

• Demonstrated feasibility of ADIOS adapter.

Possible as a modular/incremental change.

• Function Object

Still represents a good path forward

• Still needs some work to improve handling

Interim Balance

www.esi-group.com


17

Concrete General

• More Testing

Design, Performance, Scaling.

• Lagrangian

Finish handling more types.

More testing.

• Restart Improvements

Topology changes

• Code

Modularity / Structure

Decompose/reconstruct for ADIOS.

• Help support 3rd-party ADIOS/OpenFOAMreader.

Potential reuse of Takuya OSHIMA's vtk reader.

Slightly easier to parse than native files (no #include, regex etc)

• Many other topics

Ongoing / Future Work

www.esi-group.com


18

THANK YOU

[email protected]

integrating adios into openfoam for disk i/oopenfoam.com/documentation/files/adios-201610.pdf ·...

Documents