Simulating materials with atomic detail at IBM: From biophysical to high-tech

applications Application TeamGlenn J. Martyna, Physical Sciences Division, IBM ResearchDennis M. Newns, Physical Sciences Division, IBM ResearchBruce Elmegren, Physical Sciences Division, IBM ResearchXiao Liu, Silicon Technology, IBM ResearchLia Krusin-Elbaum, Physical Sciences Division, IBM ResearchJason Crain, School of Physics, Edinburgh UniversityRaphael Troitzsch, School of Physics, Edinburgh UniversityMartin Mueser, Applied Math, University of Western OntarioZak Hughes, Applied Math, University of Western OntarioRazvan Nistory, Applied Math, University of Western OntarioDimitry Shakhvorostov, Applied Math, University of Western Ontario

Methods/Software Development TeamGlenn J. Martyna, Physical Sciences Division, IBM ResearchMark E. Tuckerman, Department of Chemistry, NYUPeter Minary, Computer Science/Bioinformatics, Stanford University.Laxmikant Kale, Computer Science Department, UIUCRamkumar Vadali, Computer Science Department, UIUCSameer Kumar, Computer Science, IBM ResearchEric Bohm, Computer Science Department, UIUCAbhinav Bhatele, Computer Science Department, UIUC

Funding : NSF, IBM Research, DOE

IBM’s Blue Gene/L network torus supercomputer

Our fancy apparatus.

Goal : The accurate treatment of complex heterogeneous systems to gain physical insight.

Characteristics of current models

Empirical Models: Fixed charge, non-polarizable, pair dispersion.

Ab Initio Models: GGA-DFT, Self interaction present, Dispersion absent.

Problems with current models (empirical)

Dipole Polarizability : Including dipole polarizability changes solvation shells of ions and drives them to the surface.

Higher Polarizabilities: Quadrupolar and octapolar polarizabilities are NOT SMALL.

All Manybody Dispersion terms: Surface tensions and bulk properties determined using accurate pair potentials are incorrect. Surface tensions and bulk properties are both recovered using manybody dispersion and an accurate pair potential. An effective pair potential destroys surface properties but reproduces the bulk.Force fields cannot treat chemical reactions:

Problems with current models (DFT)• Incorrect treatment of self-interaction/exchange: Errors in electron affinity, band gaps …

• Incorrect treatment of correlation: Problematic treatment of spin states. The ground state of transition metals (Ti, V, Co) and spin splitting in Ni are in error. Ni oxide incorrectly predicted to be metallic when magnetic long-range order is absent.

• Incorrect treatment of dispersion : Both exchange and correlation contribute.

• KS states are NOT physical objects : The bands of the exact DFT are problematic. TDDFT with a frequency dependent functional (exact) is required to treat excitations even within the Born-Oppenheimer approximation.

Conclusion : Current Models

• Simulations are likely to provide semi-quantitative accuracy/agreement with experiment.

• Simulations are best used to obtain insight and examine physics .e.g. to promote understanding.

Nonetheless, in order to provide truthful solutions of themodels, simulations must be performed to long time scales!

Goal : The accurate treatment of complex heterogeneous systems to gain physical insight.

Evolving the model systems in time:

•Classical Molecular Dynamics : Solve Newton's equations or a modified set numerically to yield averages in alternative ensembles (NVT or NPT as opposed to NVE) on an empirical parameterized potential surface.•Path Integral Molecular Dynamics : Solve a set of equations of motion numerically on an empirical potential surface that yields canonical averages of a classical ring polymer

system isomorphic to a finite temperature quantum system. •Ab Initio Molecular Dynamics : Solve Newton's equations or a modified set numerically to yield averages in alternative ensembles (NVT or NPT as opposed to NVE) on a potential surface obtained from an ab initio calculation.•Path Integral ab initio Molecular Dynamics : Solve a set of equations of motion numerically on an ab initio potential surface to yield canonical averages of a classical ring polymer system isomorphic to a finite temperature quantum system.

Recent Improvements :•Increasing the time step to 100x from 1fs to 100fs for numerical solvers of empirical model MD simulations.

•Reducing the scaling of non-local pseudopotential computations in plane wave DFT from N3 to N2.

•Overcoming multiple barriers in empirical model simulations.

•Treat long range interactions for surfaces, clusters and wires in order Nlog N computational time.

•A unified formalism for including manybody polarization and dispersion in molecular simulation

Phys. Rev. Lett. 93, 150201 (2004).Chem. Phys. Phys. Chem. 6, 1827 (2005).J. Chem. Phys.. 126, 074104 (2007).J. Chem. Phys. 118, 2527 (2003). Chem. Phys. Lett. 424, 409 (2006), PRB (2009)Siam J. Sci Comp. (2008).

Unified Treatment of Long Range Forces:Point Particles and Continuous Charge Densities.

Clusters : Wires:

Surfaces: 3D-Solids/Liquids:

J. Chem. Phys. (2001-2004)

Open Question : Can we `tweak’ current models or are more powerful formulations required?

Exp.

Quantum Drude

Chem. Phys. Lett. 424, 409 (2006) , PRB (2009) J. Chem. Phys.. 126, 074104 (2007)

Dynamic Contact Staging REPSWA

1) The order O(N) Reference Potential Spatial Warping method removes 47 torsional barriers simultaneously whilst preserving the partition fnc.2) Dynamic contact staging treats intramolecular collisions.

Sampling the C50 alkane:

Siam J. Sci. Comp. (2008)

Exciting large scale domain motion

1) Standard HMC (red) cannot sample minor groove narrowing rapidly. 2) The O(N), partition function preserving Rouse Mode Projection HMC guarantees positive modes allowing fast low frequency mode sampling.

Submitted PNAS

Limitations of Limitations of ab initioab initio MDMD ((despite our despite our efforts/improvements!)efforts/improvements!)Limited to small systems (100-1000 Limited to small systems (100-1000 atoms)atoms)**..

Limited to short time dynamics and/or Limited to short time dynamics and/or sampling times.sampling times.

Parallel scaling only achieved for Parallel scaling only achieved for # processors <= # electronic # processors <= # electronic statesstates

until recent efforts by ourselves and until recent efforts by ourselves and others.others.

*The methodology employed herein scales as O(N3) with system size due to the orthogonality constraint, only.

Solution: Fine grained parallelization of Solution: Fine grained parallelization of CPAIMD.CPAIMD. Scale small systems to 10Scale small systems to 1055 processors!!processors!! Study long time scale Study long time scale phenomena!!phenomena!!

(The charm++ QM/MM application is work in progress.)

IBM’s Blue Gene/L network torus supercomputer

The worlds fastest supercomputer!Its low power architecture requires fine grain parallel algorithms/software to achieve optimal performance.

Density Functional Theory : Density Functional Theory : DFTDFT

Electronic states/orbitals of water

Removed by introducing a non-local electron-ion interaction.

Plane Wave Basis Plane Wave Basis Set:Set:

The # of states/orbital ~ N where N is # of atoms. The # of pts in g-space ~N.

Plane Wave Basis Set: Plane Wave Basis Set: Two Spherical cutoffs in G-Two Spherical cutoffs in G-spacespace

gxgy

gz (g)

(g) : radius gcut

gxgy

n(g)

n(g) : radius 2gcut

g-space is a discrete regular grid due to finite size of system!!

gz

Plane Wave Basis Set: Plane Wave Basis Set: The dense discrete real space The dense discrete real space mesh.mesh.

xy

z (r)

(r) = 3D-FFT{ (g)}

xy

n(r)

n(r) = k|k(r)|2

n(g) = 3D-IFFT{n(r)} exactly!

z

Although r-space is a discrete dense mesh, n(g) is generated exactly!

Simple Flow Chart : Scalar Simple Flow Chart : Scalar OpsOps

Object : Comp : MemStates : N2 log N : N2

Density : N log N : NOrthonormality : N3 : N2.33

Memory penalty

Flow Chart : Data Flow Chart : Data StructuresStructures

Parallelization under Parallelization under charm++charm++

Transpose

Transpose

Transpose Transpose

RhoR

Transpose

Effective Parallel Strategy:Effective Parallel Strategy:

The problem must be finely discretized.The problem must be finely discretized.The discretizations must be deftly chosen toThe discretizations must be deftly chosen to

–Minimize the communication between Minimize the communication between processors.processors.

–Maximize the computational load on the Maximize the computational load on the processors.processors.

NOTE , PROCESSOR AND DISCRETIZATION NOTE , PROCESSOR AND DISCRETIZATION ARE ARE

SEPARATE CONCEPTS!!!!SEPARATE CONCEPTS!!!!

Ineffective Parallel StrategyIneffective Parallel Strategy

The discretization size is controlled by the The discretization size is controlled by the number of physical processors.number of physical processors.

The size of data to be communicated at a The size of data to be communicated at a given step is controlled by the number of given step is controlled by the number of physical processors.physical processors.

For the above paradigm : For the above paradigm : –Parallel scaling is limited to # Parallel scaling is limited to # processors=coarse grained parameter in processors=coarse grained parameter in the model.the model.

THIS APPROACH IS TOO LIMITED TO THIS APPROACH IS TOO LIMITED TO ACHIEVE FINE GRAINED PARALLEL ACHIEVE FINE GRAINED PARALLEL SCALING.SCALING.

Virtualization and Charm++Virtualization and Charm++Discretize the problem into a large number Discretize the problem into a large number of very fine grained parts.of very fine grained parts.

Each discretization is associated with some Each discretization is associated with some amount of computational work and amount of computational work and communication.communication.

Each discretization is assigned to a light Each discretization is assigned to a light weight thread or a ``virtual processor'' weight thread or a ``virtual processor'' (VP).(VP).

VPs are rolled into and out of physical VPs are rolled into and out of physical processors as physical processors become processors as physical processors become available (Interleaving!)available (Interleaving!)

Mapping of VPs to processors can be Mapping of VPs to processors can be changed easily.changed easily.

The Charm++ middleware The Charm++ middleware provides the provides the data structures and controls required to data structures and controls required to choreograph this complex dance.choreograph this complex dance.

Parallelization by over partitioning of parallel objects : The charm++ middleware choreography!

Decomposition of work

Available Processors

On a torus architecture, load balance is not enough!The choreography must be ``topologically aware’’.

Charm++ middleware maps work to processors dynamically

Challenges to scaling:Challenges to scaling:Multiple concurrent 3D-FFTs to generate the states in real space require AllToAll communication patterns. Communicate N2 data pts. Reduction of states (~N2 data pts) to the density (~N data pts) in real space.

Multicast of the KS potential computed from the density (~N pts) back to the states in real space (~N copies to make N2 data).

Applying the orthogonality constraint requires N3 operations.

Mapping the chare arrays/VPs to BG/L processors in a topologically aware fashion.

Scaling bottlenecks due to non-local and local electron-ion interactions removed by the introduction of new methods!

Topologically aware mapping for CPAIMD

•The states are confined to rectangular prisms cut from the torus to minimize 3D-FFT communication. •The density placement is optimized to reduced its 3D-FFT communication and the multicast/reduction operations.

~N1/2

(~N1/3)

States ~N1/2

Gspace

Density N1/12

Topologically aware mapping for CPAIMD : Details

Improvements wrought by topological aware mappingon the network torus architecture

Density (R) reduction and multicast to State (R) improved.State (G) communication to/from orthogonality chares improved.‘’Operator calculus for parallel computing”, Martyna and Deng (2009) in preparation.

Parallel scaling of liquid water* as a function of system size on the Blue Gene/L installation at YKT:

•Weak scaling is observed!•Strong scaling on processor numbers up to ~60x the number of states!

*Liquid water has 4 states per molecule.

Scaling Water on Blue Gene/L

Software : SummarySoftware : Summary

Fine grained parallelizationFine grained parallelization of the Car-of the Car-Parrinello Parrinello ab initioab initio MD method demonstrated MD method demonstrated on thousands of processors : on thousands of processors :

# processors >> # electronic states# processors >> # electronic states..

Long time simulations of small systems are Long time simulations of small systems are now possible on large massively parallel now possible on large massively parallel supercomputers.supercomputers.

Goal : The accurate treatment of complex heterogeneous systems to gain physical insight.

Novel Switches by Phase Change Materials

The phases are interconverted by heat!

Novel Switches by Phase Change Materials

Ge0.15Te0.85

or Eutectic GT

Novel Switches by Phase Change Materials

GeSb2Te4 or GST-124

Piezoelectrically driven memories?

• A pressure driven mechanism would be fast and scalable.• Studying pressure can lead to new insights into mechanism.• Can we find suitable materials using a combined exp/theor approach?

GST-225 undergoes pressure induced “amorphization” both experimentally and theoretically ….

but the process is notreversible!

Eutectic GS (Ge0.15Sb0.85 ) undergoes an amorphous to crystalline transformation under pressure, experimentally!

Is the process amenable to reversible switching as in the thermal approach???

Amorphous converts to crystal and never recovers

Amorphous Crystal

Utilize tensile load to approach the spinodal and cause pressure induced amorphization: Eutectic GS alloy

Schematic of a potential device based on pressure switching

P~1.5 GPa

P~ -1.5 GPa

CPAIMD spinodal line!

Spinodal decomposition under tensile load

Solid is stable at ambient pressure

Postulate : Ge/Sb switch to 4-coordinate state in amorphous:

Umbrella-flip model

Alexander V. Kolobov, Paul Fons, Anatoly I. Frenkel, Alexei L. Ankudinov, Junji Tominaga & Tomoya UrugaNature Materials 3, 703 - 708 (2004)

Crystalline

Amorphous

Gap driven picture of the transition

The electronic energy, total energy, gap and conductivity track the 3-coordinate to 4-coordinate species conversion.

Near half-filling pseudogap opens into gap

maximally lowering electronic energy, driving the structural change.

Gap driven structural relaxation:

IBM’s Piezoelectric Memory

We are investigating other materials, better device designs andgeneralizing the principle to produce a fast low power architecture.

Conclusions

• Important physical insight can be gleaned from high quality, large scale computer simulation studies.

• The parallel algorithm development required necessitates cutting edge computer science.

• New methods must be developed hand-in-hand with new parallel paradigms.

• Using clever hardware with better methods and parallel algorithms shows great promise to impact both science and technology.