overview of the exascale additive manufacturing …...overview of the exascale additive...

Overview of the Exascale Additive Manufacturing Project (ExaAM) One of 15 Applications in the US DOE Exascale Computing Project

John A. Turner Oak Ridge National Laboratory Group Leader: Computational Engineering and Energy Sciences Chief Computational Scientist: Consortium for Advanced Simulation of Light Water Reactors (CASL) Principle Investigator: Transforming Additive Manufacturing Through Exascale Simulation (ExaAM)

Numerous others on the ExaAM team (incomplete list): Jim Belak (co-PI, LLNL), Andy Anderson (LLNL), Suresh Babu (UTK), Mark Berrill (ORNL), Curt Bronkhorst (LANL), Neil Carlson (LANL), Ondrej Certik (LLNL), Jean-Luc Fattebert (LLNL), Neil Hodge (LLNL), Wayne King (LLNL), Lyle Levine (NIST), Chris Newman (LANL), B. Radhakrishnan (ORNL), Adrian Sabau (ORNL), Srdjan Simunovic (ORNL)

HPC User Forum Santa Fe, NM 17-19 Apr 2017

www.ExascaleProject.org

http://www.ExascaleProject.org

2 Exascale Computing Project

Outline

• Additive Manufacturing • Exascale Computing Program • ExaAM Project


Slide from my presentation at the April 2014 HPC User Forum Meeting

(also in Santa Fe)


Slide from my presentation at the April 2014 HPC User Forum Meeting

(also in Santa Fe)

• Computer-Aided Engineering for Batteries Program (DOE / EERE / VTO)

• Battery Crashworthiness (DOT / NHTSA)

A lot has happened in the last three

years.


I assume most are aware of additive manufacturing, a.k.a. 3D printing, and that it is being used for metal as well as polymers

21.1 g 12.1 g 14.4 g


Test Stand at NASA Marshall Space Flight Center (Huntsville, AL)

7 Exascale Computing Project Powder Bed Technologies

Design

Material Feedstock

In-situ Process Control

Material µm-nm

Structure

Static and Dynamic

Mechanical Properties

Plasma (wire)

E-beam (wire)

Laser (wire)

Large Melt Pool Technologies

Laser (powder)

Direct Metal Deposition

Laser (powder)

E-beam (powder)

There are multiple metal additive manufacturing technologies Physical processes are similar • Energy Deposition • Melting & Powder Addition • Evaporation & Condensation • Heat & Mass Transfer • Solidification • Solid-State Phase Transformation • Repeated Heating and Cooling • Complex Geometries


Multiple computational challenges must be addressed for AM

• 1 m3 ~ 1012 particles ~ 109 m of “weld” line (assuming 50µm particles) and build times of hours

• Large temperature gradients, rapid heating and cooling – necessary / sufficient coupling between thermomechanics and melt/solidification

• Heterogeneous and multi-scale – resolution of energy sources and effective properties of powder for continuum simulations

• Path optimization • Large number of parameters and incomplete understanding

– key uncertainties and propagation of those uncertainties

• Validation is difficult as characterization is limited


Overview of electron beam Additive Manufacturing (Arcam®)

http://www.arcam.com/technology/electron-beam-melting/hardware/

3D CAD Model

Thin 2D Layers

To Machine

Nth Layer

Preheating Melting (N+1)th layer

Final Part

Conventional raster melt sequence

Microstructure manipulation of IN718 via additive

manufacturing is not well understood. Always results in

columnar grains oriented along the build direction (001)

• Microstructure plays significant role in determining mechanical properties of final part

• Directional vs. Isotropic properties • Feasibility of site specific microstructure control?


Mechanical anisotropy and poor properties are observed in z-direction

Kobryn and Semiatin (2001)

• Anisotropy is a function of material thermal path. • Thermal path of deposit material is non-uniform • HIP is not feasible for all additive deposits • This poses a challenge in part qualification

lacking fundamental understanding of process-structure-property-performance relationships

trial and error optimization is incredibly inefficient P. A. Kobryn and S. L. Semiatin, “The laser additive manufacture of Ti-6Al-4V,” JOM, vol. 53, no. 9, pp. 40–42, Sep. 2001. doi:10.1007/s11837-001-0068-x.


A more complete understanding of the linkage between process, structure, properties, and performance is needed

Courtesy of Wayne King, Director of the Accelerated Certification of Additively Manufactured Metals Initiative at LLNL


What is the Exascale Computing Project (ECP)? • Created in support of President Obama’s National Strategic

Computing initiative (NSCI) • A collaborative effort of two US Dept of Energy (DOE) offices:

– Office of Science (DOE-SC) – National Nuclear Security Administration (NNSA)

• A 10-year project to accelerate the development of a capable exascale ecosystem – 50x the performance of today’s 20 PF/s systems – Operates in a power envelope of 20–30 MW – Is sufficiently resilient (average fault rate: ≤1/week) – Includes a software stack that meets the needs of a broad

spectrum of applications and workloads – Led by DOE laboratories – Executed in collaboration with academia and industry

A capable exascale computing system will have a well-balanced ecosystem (software,

hardware, applications)


Application Development Software Technology

Hardware Technology

Exascale Systems

Scalable software stack

Science and mission applications

Hardware technology elements

Integrated exascale supercomputers

ECP has formulated a holistic approach that uses co-design and integration to achieve capable exascale

Correctness Visualization Data Analysis

Applications Co-Design

Programming models, development environment, and

runtimes Tools Math libraries and

Frameworks

System Software, resource management threading,

scheduling, monitoring, and control

Memory and Burst buffer

Data management I/O and file

system Node OS, runtimes

Res

ilienc

e

Wor

kflo

ws

Hardware interface


ExaAM is one of 15 initial ECP application development projects

Advanced Manufacturing Gaps and Opportunities • Improve quality, reliability, and application breadth of

additive manufacturing (AM) • Accelerate innovation in clean energy manufacturing

institutes (NNMIs) • Capture emerging manufacturing markets Simulation Challenge Problems • Continuum level predictions of non-uniform

microstructure and its relationship to process parameters

• Predictive mesoscale models for dendritic solidification scale-bridged to continuum

Prospective Outcomes and Impact • Routine qualification of AM parts via process-aware

design specs and reproducibility through process control • Fabrication of metal parts with unique properties such

as light weight strength and failure-proof joints and welds


Models and Code(s) • Physical Models: fluid flow, heat transfer,

phase change (melting/solidification and solid-solid), nucleation, microstructure formation and evolution, residual stress

• Codes: • Continuum: ALE3D, Diablo, Truchas • Mesoscale: AMPE, MEUMAPPS, Tusas

• Motifs: Sparse Linear Algebra, Dense Linear Algebra, Spectral Methods, Unstructured Grids, Dynamical Programs, Particles

Transforming Additive Manufacturing through Exascale Simulation (ExaAM)

PI: John Turner (ORNL), co-PI: Jim Belak (LLNL)

Goal and Approach • Accelerate the widespread adoption of

additive manufacturing (AM) by enabling fabrication of qualifiable metal parts with minimal trial-and-error iteration and realization of location-specific properties • Coupling of high-fidelity sub-grid simulations

within a continuum process simulation to determine microstructure and properties at each time-step using local conditions

Software and Numerical Library Dependencies • C++, Fortran • MPI, OpenMP, OpenACC, CUDA • Kokkos, Raja, Charm++ • Hypre, Trilinos, P3DFFT,

SAMRAI, Sundials, Boost • DTK, netCDF, HDF5, ADIOS,

Metis, Silo • GitHub, GitLab, CMake, CDash,

Jira, Eclipse ICE

Critical Needs Currently Outside the Scope of ExaAM • modeling of powder properties and spreading • shape and topology optimization • post-build processing, e.g. hot isostatic

pressing (HIP) • data analytics and machine learning of

process / build data • reduced-order models


Additive Manufacturing Physics / Process Workflow


Quick survey of selected ExaAM application codes (components)


ExaAM codes and attributes

Code(s) Area(s) Physical Models Computational Motifs Prog. Lang.

(Model)

Numerical Library

Depenencies

Proxy App

Diablo Process (part scale), Performance

solid mechanics, heat & mass trans, contact, implicit time integration

Lagrangian FEM, nonlinear physics, staggered & monolithic solvers, adaptive h-refinement

Fortran (MPI)

Hypre, HDF, Metis, Silo TBD

Truchas Process (melt pool to part scale)

free-surface flow, heat transfer, phase change, species diffusion

FVM, unstructured mesh, implicit mimetic finite difference, linear & nonlinear solvers

Fortran (MPI)

Hypre, HDF5, netCDF Pececillo

ALE3D Process (melt pool scale), Properties

implicit and explicit hydro, heat trans, phase change

FEM, unstructured mesh, advection, linear & nonlinear solvers

C++ (MPI) Hypre LULESH

MEUMAPPS Microstructure phase-field Fourier spectral method Fortran (MPI) P3DFFT N/A

AMPE Microstructure phase-field implicit FVM, linear & nonlinear

solvers, AMR C++ (MPI) Hypre, SAMRAI, Sundials

AMG2013

Tusas Microstructure phase-field implicit FEM, preconditioned

JFNK, unstructured 2D and 3D C++, (MPI, OpenMP)

Trilinos, netCDF, HDF5, Boost

N/A

Cont

inuu

m sc

ale

Mes

osca

le


ALE3D (LLNL) has been used to study details of the beam-powder interaction and melt pool dynamics

Khairallah, S.A., Anderson, A., 2014. Mesoscopic Simulation Model of Selective Laser Melting of Stainless Steel Powder. Journal of Materials Processing Technology 214, 2627-2636 DOI:10.1016/j.jmatprotec.2014.06.001.

Laser Thin Powder Layer

Thick Powder Layer

Bridge area

a

Yadroitsev, I., Gusarov, A., Yadroitsava, I., Smurov, I., 2010. Single track formation in selective laser melting of metal powders. Journal of Materials Processing Technology 210, 1624-1631.

Movie replaced by still image for distribution


Diablo (LLNL) simulation of residual stress during build

Hodge, N.E., Ferencz, R.M., Vignes, R.M., 2016. Experimental Comparison of Residual Stresses for a Thermomechanical Model for the Simulation of Selective Laser Melting. Additive Manufacturing DOI. http://dx.doi.org/10.1016/j.addma.2016.05.011.



• J. D. Hunt, “Steady state columnar and equiaxed growth of dendrites and eutectic,” Mater. Sci. Eng., vol. 65, no. 1, pp. 75–83, 1984. • M. Gäumann, C. Bezençon, P. Canalis, and W. Kurz, “Single-crystal laser deposition of superalloys: Processing-microstructure maps,” Acta

Mater., vol. 49, no. 6, pp. 1051–1062, 2001.

Simulation helped enable local control of grain structure in AM parts

• Given G and R, can calculate volume fraction of equiaxed grains at any location • G is temperature gradient, • R is velocity of liquid-solid interface, • No is nucleation density, • Φ is volume fraction of equiaxed grains

(probability of stray grain formation) • n and a are alloy constants

Lee, Y., Nordin, M., Babu, S. S., & Farson, D. F. (2014). Effect of Fluid Convection on Dendrite Arm Spacing in Laser Deposition. Metallurgical and Materials Transactions B, 45(4), 1520-1529.

• Columnar-to-Equiaxed Transition (CET) in rapid solidification processes primarily controlled by: – Thermal gradient at the liquid solid interface (G) – Velocity or growth rate of liquid-solid interface (R)

• Difficult to measure experimentally – Spatial resolution (microns) – Temporal resolution required (milliseconds) – Thermal imaging camera cannot capture 3D data


Truchas provides: • Thermal gradient at the liquid solid interface

• Velocity of liquid-solid interface

Truchas metal casting code (LANL) can determine G and R

Conventional Raster Pattern

Spot Melt Pattern along the contour “DOE”

Temperature gradient and melt pool isotherm

• Dehoff, R. R., Kirka, M. M., Sames, W. J., Bilheux, H., Tremsin, A. S., Lowe, L. E., & Babu, S. S. (2015). Site specific control of crystallographic grain orientation through electron beam additive manufacturing. Materials Science and Technology, 31(8), 931-938.

• N. Raghavan, R. Dehoff, S. Pannala, S. Simunovic, M. Kirka, J. Turner, N. Carlson, and S. S. Babu, “Numerical modeling of heat-transfer and the influence of process parameters on tailoring the grain morphology of IN718 in electron beam additive manufacturing,” Acta Materialia, vol. 112, pp. 303–314, Jun. 2016. doi:10.1016/j.actamat.2016.03.063.



Using MEUMAPPS (ORNL phase field code), nucleation rate has been identified as the main factor in formation of colony grain structure

Crucial Findings • Low nucleation rate promotes colony when a new nucleus

sees well developed strain field from a nearby variant • High nucleation rate promotes basket weave when all nuclei

see complex strain field due to multiple, evolving nuclei

N=0.5 s-1

Colony structure

N=5.0 s-1

Basket weave structure

950K

1000K B. Radhakrishnan, S. Gorti, and S. S. Babu, “Phase Field Simulations of Autocatalytic Formation of Alpha Lamellar Colonies in Ti-6Al-4V,” Metallurgical and Materials Transactions A, vol. 47, no. 12, pp. 6577–6592, Dec. 2016. doi:10.1007/s11661-016-3746-6.

Parametric studies performed using phase field simulations • Two levels of thermodynamic driving force: low

(1000K) and high: 950K • Two levels of nucleation rate: low (0.5 s-1) and

high (5 s-1) Auto-catalytic colony nucleation



Objective: Utilize exascale concurrency and locality to dynamically bridge continuum and mesoscale physics • Task-based embedded Scale-Bridging

escapes the traditional synchronous SPMD paradigm and exploits the heterogeneity expected in exascale hardware.

• To achieve this, we are developing a UQ-driven adaptive physics refinement approach.

• Coarse-scale simulations dynamically spawn tightly coupled and self-consistent fine-scale simulations as needed.

• This task-based approach naturally maps to exascale heterogeneity, concurrency, and resiliency issues.

24


Ultimately, ExaAM will deliver and deploy a new integrated simulation environment for AM


Questions? e-mail: [email protected] The research and activities described in this presentation were performed using the resources at Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC0500OR22725.

This research was supported by the Exascale Computing Project (http://www.exascaleproject.org), a joint U.S. Department of Energy and National Nuclear Security Administration project responsible for delivering a capable exascale ecosystem, including software, applications, hardware, and early testbed platforms, to support the nation’s exascale computing imperative.

overview of the exascale additive manufacturing …...overview of the exascale additive...

Documents