MLMOD Package: Machine Learning Methods forData-Driven Modeling in LAMMPS

Paul J. Atzberger 1,2,+*

1 Department of Mathematics, University of California Santa Barbara (UCSB).2 Department of Mechanical Engineering, University of California Santa Barbara (UCSB).

+ atzberg@gmail.comhttp://atzberger.org/

Abstract

We discuss a software package for incorporating into sim-ulations models obtained from machine learning methods.These can be used for (i) modeling dynamics and time-stepintegration, (ii) modeling interactions between system com-ponents, and (iii) computing quantities of interest character-izing system state. The package allows for use of machinelearning methods with general model classes including Neu-ral Networks, Gaussian Process Regression, Kernel Models,and other approaches. We discuss in this whitepaper our pro-totype C++ package, aims, and example usage.

IntroductionRecent advances in machine learning, optimization, andavailable computational resources are presenting new oppor-tunities for data-driven modeling and simulation in the natu-ral sciences and engineering. Empirical successes in deeplearning suggest promising techniques for learning non-linear mappings and for identifying features of underlyingstructure [12, 10]. Scientific computations and associateddynamical systems present a unique set of challenges foremploying recent machine learning methods often motivatedby image analysis and natural language processing [2, 7, 27].However, for scientific and engineering applications thereare often important constraints required to obtain plausi-ble models and a need for results to be more interpretable.In large-scale scientific computations modeling efforts of-ten aim to start with first principles, or more detailed con-cepts, from which computation is used to obtain insights intolarger-scale emergent system behaviors. Examples includethe rheological responses of soft materials and complex flu-ids from microstructure interactions [1, 4, 18, 15], molec-ular dynamics modeling of protein structure and functionaldomains from atomic level interactions [20, 14, 6, 23], andprediction of weather and climate phenomena from detailedphysical models and measurements [25, 3].

Obtaining observables and quantities of interest (QoI)from simulations of such high fidelity detailed models caninvolve significant computational resources [19, 26, 31, 22,

*Work supported by grants DOE Grant ASCR PHILMS DE-SC0019246 and NSF Grant DMS-1616353.Copyright © 2021for this paper by its authors.

21, 30]. However, many observables are not sensitive tomany of the system behaviors and are expected to only de-pend on a subset of underlying factors. If these key featuresof the system could be identified for classes of observables,this would present opportunities to formulate more simpli-fied models and make predictions which are less computa-tionally expensive to simulate and could provide further in-sights into underlying mechanisms yielding system behav-iors.

Recent machine learning approaches provide promisingdata-driven approaches for learning features and models forsystem behaviors from high fidelity simulation data. This in-cludes learning data-driven models for (i) dynamics of thesystem at larger spatial-temporal scales (ii) interactions be-tween system components, and (iii) features yielding coarserdegrees of freedom or new quantities of interest character-izing system behaviors. The models obtained from learn-ing can take many forms, including Deep Neural Networks(DNNs) [10], Kernel Regression Models (KRM) [28], Gaus-sian Process Regression (GPR) [24], and others [11].

A practical challenge is the effort often required to in-corporate the contributions of such learned system featuresto augment existing models and simulations. Our packageMLMOD aims to address this aspect of data-driven model-ing by providing a general interface for incorporating MLmodels using standardized representations and by leverag-ing existing simulation frameworks such as LAMMPS. TheMLMOD package provides hooks which are triggered dur-ing key parts of simulation calculations. In this way stan-dard machine learning frameworks can be used to train MLmodels, such as PyTorch and TensorFlow, with the result-ing models more amenable to being translated into practicalsimulations.

Data-Driven ModelingData-driven modeling can take many forms. As a specificmotivation for the package and our initial prototype imple-mentations, we discuss a rather particular example case indetail, but we aim for our overall package also to be usefulin other settings. Consider the case of a detailed molecu-lar dynamics simulation of relatively large colloidal parti-cles within a bath of much smaller solvent particles. It isoften of interest to try to infer the interaction law between

arX

iv:2

107.

1436

2v1

[cs

.LG

] 2

9 Ju

l 202

1

the colloidal particles given the type of solution, charge, andother physical conditions. While there is extensive theoreti-cal literature on colloidal interactions and approximate mod-els [8, 9, 13], which have had many successes, these canalso have limited accuracy and be challenging to apply inmany practical settings [13, 29]. Computational simulationsare widely used and allow for investigations of phenomenawhere detailed modeling can be employed to control for con-tributing physical effects.

While providing predictions related to the principles builtinto the detailed models, such computational simulationscan be expensive given the many degrees of freedom re-quired to represent the solvent in the entire spatial domainand from small time-scales associated with solvent-solventinteractions. However, given the size contrast of a colloidand the solvent it is expected that in many circumstancesthe colloidal dynamics and interactions are governed primar-ily through temporal averaging of the solvent-solvent andsolvent-colloid interactions.

colloidal system

deep neural network

kernel regression

...

ml model

QoI

Dynamics

Interactions

Figure 1: Data-driven modeling from detailed molecularsimulations can be used to train machine learning (ML)models for performing simulations at larger spatial-temporalscales. This can include models for the dynamics, interac-tions, or for computing quantities of interest (QoI) charac-terizing the system state. The colloidal system for examplecould be modeled by dynamics at a larger scale using equa-tion 1 withM obtained from training. In the MLMOD pack-age, the ML models can be represented by Deep Neural Net-works, Kernel Regression Models, or other model classes.

.

Relative to the detailed molecular dynamics simulation,this motivates a simplified model for the effective colloiddynamics

dX

dt= MF + kBT∇X ·M + Fthm

〈Fthm(s)Fthm(t)T 〉 = 2kBTMδ(t− s). (1)

The X ∈ R3n refers to the collective configuration of alln colloids. Here the main component to determine for thesimplified model is the mobility tensor M = M(X), whichcaptures the effective coupling between the colloidal parti-cles.

In principle, this could be tabulated for each X by per-forming a sequence of computational simulations over allconfigurations and force combinations. However, in the gen-eral case this is inefficient in practice unless there are known

symmetries or other physical structures. For example, in-teractions only occurring pairwise or translational and ro-tational invariances / equivariances. In the case of pairwiseinteractions, translational invariance, and rotational equiv-ariance, the mobility can be reduced effectively to a onedimensional dependence on the configuration. In many cir-cumstances such symmetries of the system may not be im-mediately apparent and even when symmetries are knowntabulation can present drawbacks for interpolation and stor-age.

Machine learning methods provide data-driven ap-proaches for learning representations and features for suchmodeling. Optimization frameworks with judicious choicesof loss functions and training protocols can be used to iden-tify system features underlying interactions, symmetries,and other structures. In machine learning methods these arerepresented in a compressed form over some model class al-lowing for efficient interpolation, and even extrapolation insome cases, especially when using explicit low dimensionallatent spaces or when imposing other inductive biases.

As further simplification for the colloidal example, ifwe assume the interactions occur to a good approxima-tion only pairwise the problem can be reduced to a modelM = M(X1,X2) depending on six dimensions. This canbe further constrained to learn only symmetric positive semi-definite tensors, for example by learning L = L(X1,X2) togenerate M = LLT .

There are many ways we can obtain the modelM . This in-cludes learning directly using machine learning approachesor a multi-stage approach using first a tabulation approachfrom more traditional estimators from statistical mechan-ics followed by compression to an ML model. For exam-ple, a common way to estimate mobility in fluid mechan-ics is to apply active forces F and compute the velocityresponse 〈V〉 = 〈dX/dt〉 ≈ τ−1〈∆τX(t)〉 ≈ MF. The∆τX(t) = X(t + τ) − X(t) for τ chosen carefully. Forlarge enough forces F, the thermal fluctuations can be av-eraged away readily by repeating this measurement andtaking the mean. In statistical mechanics, another estima-tor is obtained when F = 0 by using the passive fluctua-tions of system. A moment-based estimator commonly usedis M(X) ≈ (2kBTτ)−1〈∆τX(t)∆τX(t)T 〉 for τ chosencarefully. While theoretically each of these estimators giveinformation on M , in practice there can be subtleties suchas a good choice for τ , magnitude for F, and role of fluc-tuations. Even for these more traditional estimators, it couldstill be useful for storage efficiency and convenience to trainan ML model to provide a compressed representation andfor interpolation for evaluating M(X).

Machine learning methods also could be used to trainmore directly from simulation data for sampled colloid tra-jectories X(t). The training would select an ML model Mθ

over some class parameterized by θ, such as the weights andbiases of a Deep Neural Network. For instance, this could bedone by Maximum Likelihood Estimation (MLE) from thetrajectory data X(t) by optimizing the objectiveMθ = arg min

Mθ∈M− log ρθ (X(t1),X(t2), . . . ,X(tm)) . (2)

The ρθ denotes the probability density when M = Mθ

is used in equation 1 for observing the configurationsX(ti). To obtain tractable and robust training algorithmsfurther approximations and regularizations may be requiredto the MLE problem in equation 2. This could include us-ing variational inference approaches, further restrictions onthe model architectures, or introducing priors [17, 16, 5].Combining such approximations with further regularizationscould help facilitate learning possible symmetries and otherfeatures shaping the learned model M(X) = Mθ.

The MLMOD package provides a way for transferringsuch learned models from the data into components of prac-tical simulations in LAMMPS. We discussed here one exam-ple of a basic data-driven modeling approach. The MLMODpackage can be used more generally and support more broadclasses of models for incorporating machine learning resultsinto simulation components. Components can include thedynamics, interactions, and for computing quantities of in-terest as system state information or characterizing systembehaviors. The initial prototype implementation we presentin this whitepaper supports the basic mobility modelingframework as a proof-of-concept, with longer-term aims tosupport more general classes of reduced dynamics and inter-actions in future releases.

Structure of the Package ComponentsThe package is organized as a module within LAMMPS thatis called each time-step and has the potential to serve mul-tiple functions in simulations. This includes (i) serving as atime-step integrator updating the configuration of the systembased on a specified learned model, (ii) evaluating interac-tions between system components to compute energy andforces, and (iii) compute quantities of interest (QoI) that canbe used as state information during simulations or comput-ing statistics. The package is controlled by external XMLfiles that specify the mode of operation and source for pre-trained models and other information, see the schematic inFigure 2.

Figure 2: The MLMOD Package is structured modularlywith subcomponents for providing ML models in simula-tions for the dynamics, interactions, and computing quan-tities of interest (QoI) characterizing the system state. Thepackage makes use of standardized data formats such asXML for inputs and export ML model formats from ma-chine learning frameworks..

.

The MLMOD Package is incorporated into a simulationby either using the LAMMPS scripting language or thepython interface. This is done using the ”fix” command inLAMMPS, with this terminology historically motivated byalgorithms for ”fixing” molecular bonds as rigid each time-step. For our package the command to set up the triggers forour algorithms is fix 1 mlmod all filename.mlmod params.This specifies the tag ”1” for this fix, particle groups con-trolled by the package as ”all”, and the XML file of param-eters. The XML file filename.mlmod params specifies theMLMOD simulation mode and where to find the associatedexported ML models. An example and more details are dis-cussed below in the section on package usage. The MLMODPackage can evaluate machine learning models using frame-works such as C++ PyTorch API. While this would also al-low for the possibility of doing on-the-fly learning, we an-ticipate the most common use scenarios would be to trainin advance to learn models and then incorporate them intosimulations through evaluation.

A common scenario we anticipate is for a data-drivenmodel to be obtained from a machine learning frameworksuch as PyTorch by training from trajectory data from de-tailed high fidelity simulations. Once the model is trained,it can be exported to a portable format such as Torch. TheMLMOD package would import these pre-trained modelsfrom Torch files such as ”trained model.pt”. This allows forthese models to then be invoked by MLMOD to provideelements for (i) performing time-step integration to modeldynamics, (ii) computing interactions between system com-ponents, and (iii) computing quantities of interest (QoI) forfurther computations or as statistics. This provides a mod-ular and general way for data-driven models obtained fromtraining with machine learning methods to be used to governLAMMPS simulations.

Example Usage of the PackageWe give one basic example usage of the package in thecase for modeling colloids using a mobility tensor M . TheMLMOD package is invoked by using the ”fix” command inLAMMPS, with this terminology historically motivated byalgorithms for ”fixing” molecular bonds as rigid each time-step. To set up the triggers for the MLMOD package duringLAMMPS simulations a typical command would look like

fix m1 c_group mlmod model.mlmod_params

The m1 gives the tag for the fix, c group specifies the labelfor the group of particles controlled by this instance of theMLMOD package, the mlmod specifies to use the MLMODpackage, and model.mlmod params gives the XML file withparameters controlling the mode to run MLMOD packageand associated exported ML models.

Multiple instances of MLMOD package are permitted tocontrol different groups of particles. The package is de-signed with modularity so a mode is first defined in a param-eter file and then different sets of algorithms and parametersare used based on that information. For the mobility examplediscussed in the introduction, a basic prototype implementa-tion is given by the MLMOD simulation mode dX MF ML1.

For this modeling mode, a typical parameter file would looklike

<?xml version="1.0" encoding="UTF-8"?><MLMOD><model_data type="dX_MF_ML1"><M_ii_filename value="M_ii_torch.pt"/><M_ij_filename value="M_ij_torch.pt"/></model_data></MLMOD>

This specifies for an assumed mobility tensor of pairwiseinteractions the models for the self-mobility responses Mii

and the pairwise mobility response Mij = Mji. For ex-ample, a hydrodynamic model for interactions when thetwo colloids of radius a are not too close together is touse the Oseen Tensors Mii = (6πηa)−1I and Mij =(8πηr)−1

(I + r−2rrT

). The η is the fluid viscosity, r =

Xi(t) − Xj(t) with r = ‖r‖ give the particle separation.The responses are V` = M`mFm with `,m ∈ {1, 2} andsummation notation.

The dX MF ML1 mode allows for this type of model andreplacing the interactions with ML models learned fromsimulation data. Related modes can also be implemented toextend models to capture more complicated interactions ornear-field effects. For example, to allow for localized many-body interactions with ML models giving contributions tomobility M(X). In this way MLMOD can be used for hy-brid modeling combining ML models with more traditionalmodeling approaches within a unified framework.

The models used by MLMOD in principle can be of anyexported format from machine learning frameworks. Cur-rently, the implementation uses PyTorch and the export for-mat based on torch script with .pt files. This allows for avariety of models to be used ranging from those based onDeep Neural Networks, Kernel Regression Models, and oth-ers. We describe here our early work on a prototype imple-mentation, so some aspects of the interface may change infuture releases. For examples, updates, and additional in-formation please check the MLMOD package website athttp://software.atzberger.org/.

ConclusionThe package provides capabilities in LAMMPS for incor-porating into simulations data-driven models for dynam-ics and interactions obtained from training with machinelearning methods. The package supports representations us-ing Neural Networks, Gaussian Process Regression, Ker-nel Models, and other classes. Discussed here is our earlyprototype implementation of the MLMOD package. Pleasecheck the MLMOD website for updates and future releasesat http://software.atzberger.org/.

AcknowledgmentsAuthors research supported by grants DOE Grant ASCR PHILMSDE-SC0019246 and NSF Grant DMS-1616353. Authors also ac-knowledge UCSB Center for Scientific Computing NSF MR-SEC (DMR1121053) and UCSB MRL NSF CNS-1725797. P.J.A.would also like to acknowledge a hardware grant from Nvidia.

References[1] Atzberger, P. J. 2013. Incorporating shear into stochas-

tic Eulerian-Lagrangian methods for rheological stud-ies of complex fluids and soft materials. Phys-ica D: Nonlinear Phenomena 265: 57–70. ISSN0167-2789. doi:https://doi.org/10.1016/j.physd.2013.09.002. URL https://www.sciencedirect.com/science/article/pii/S016727891300256X.

[2] Atzberger, P. J. 2018. Importance of the MathematicalFoundations of Machine Learning Methods for Scien-tific and Engineering Applications. SciML2018 Work-shop, position paper URL https://arxiv.org/abs/1808.02213.

[3] Bauer, P.; Thorpe, A.; and Brunet, G. 2015. The quietrevolution of numerical weather prediction. Nature525(7567): 47–55. ISSN 1476-4687. URL https://doi.org/10.1038/nature14956.

[4] Bird, R.B., C. C. F. A. R. C. H. O. 1987. Dynamics ofPolymeric Liquids : Volume II Kinetic Theory. Wiley-Interscience.

[5] Blei, D. M.; Kucukelbir, A.; and McAuliffe, J. D. 2017.Variational Inference: A Review for Statisticians. Jour-nal of the American Statistical Association 112(518):859–877. doi:10.1080/01621459.2017.1285773. URLhttps://doi.org/10.1080/01621459.2017.1285773.

[6] Brooks, B. R.; Bruccoleri, R. E.; Olafson, B. D.; States,D. J.; Swaminathan, S. a.; and Karplus, M. 1983.CHARMM: a program for macromolecular energy,minimization, and dynamics calculations. Journal ofcomputational chemistry 4(2): 187–217.

[7] Brunton, S. L.; Proctor, J. L.; and Kutz, J. N. 2016.Discovering governing equations from data by sparseidentification of nonlinear dynamical systems. Pro-ceedings of the National Academy of Sciences 113(15):3932–3937. ISSN 0027-8424. doi:10.1073/pnas.1517384113. URL https://www.pnas.org/content/113/15/3932.

[8] Derjaguin, B.; Landau, L. 1941. Theory of the stabil-ity of strongly charged lyophobic sols and of the adhe-sion of strongly charged particles in solutions of elec-trolytes. Acta Physico Chemica URSS 633(14).

[9] Doi, M. 2013. Soft matter physics. Oxford UniversityPress.

[10] Goodfellow, I.; Bengio, Y.; and Courville, A. 2016.Deep Learning. The MIT Press. ISBN 0262035618.URL https://www.deeplearningbook.org/.

[11] Hastie, T.; Tibshirani, R.; and Friedman, J. 2001. Ele-ments of Statistical Learning. Springer Series in Statis-tics. New York, NY, USA: Springer New York Inc.

[12] Hinton, G.; and Salakhutdinov, R. 2006. Reducing theDimensionality of Data with Neural Networks. Science313(5786): 504–507. doi:10.1126/science.1127647.URL https://www.cs.toronto.edu/∼hinton/science.pdf.

[13] Jones, R.; Jones, R.; and R Jones, P. 2002. Soft Con-densed Matter. Oxford Master Series in Physics. OUP

Oxford. ISBN 9780198505891. URL https://books.google.com/books?id=Hl HBPUvoNsC.

[14] Karplus, M.; and McCammon, J. A. 2002. Moleculardynamics simulations of biomolecules. Nature Struc-tural Biology 9(9): 646–652. ISSN 1545-9985. URLhttps://doi.org/10.1038/nsb0902-646.

[15] Kimura, Y. 2009. Microrheology of Soft Matter. J.Phys. Soc. Jpn. 78(4): 8–. ISSN 0031-9015.

[16] Kingma, D. P.; and Welling, M. 2014. Auto-EncodingVariational Bayes. In 2nd International Conferenceon Learning Representations, ICLR 2014, Banff, AB,Canada, April 14-16, 2014, Conference Track Pro-ceedings. URL http://arxiv.org/abs/1312.6114.

[17] Lopez, R.; and Atzberger, P. J. 2020. Variational Au-toencoders for Learning Nonlinear Dynamics of Phys-ical Systems URL https://arxiv.org/abs/2012.03448.

[18] Lubensky, T. C. 1997. Soft condensed matterphysics. Solid State Communications 102(2-3): 187–197–. ISSN 0038-1098.

[19] Lusk, M. T.; and Mattsson, A. E. 2011. High-performance computing for materials design to ad-vance energy science. MRS Bulletin 36(3): 169–174.ISSN 1938-1425. URL https://doi.org/10.1557/mrs.2011.30.

[20] McCammon, J.; and Harvey, S. 1988. Dynamics ofProteins and Nucleic Acids. Cambridge UniversityPress. ISBN 9780521356527. URL https://books.google.com/books?id=OXDS3Oq2QYIC.

[21] Murr, L. E. 2016. Computer Simulations in MaterialsScience and Engineering, 1–15. Cham: Springer Inter-national Publishing. ISBN 978-3-319-01905-5. doi:10.1007/978-3-319-01905-5 60-2. URL https://doi.org/10.1007/978-3-319-01905-5 60-2.

[22] Pan, J. 2021. Scaling up system size in materialssimulation. Nature Computational Science 1(2): 95–95. ISSN 2662-8457. URL https://doi.org/10.1038/s43588-021-00034-x.

[23] Plimpton, S. 1995. Fast Parallel Algorithms forShort-Range Molecular Dynamics. Journal of Com-putational Physics 117(1): 1–19. ISSN 0021-9991.URL http://www.sciencedirect.com/science/article/pii/S002199918571039X.

[24] Rasmussen, C. E. 2004. Gaussian Processes in Ma-chine Learning, 63–71. Berlin, Heidelberg: SpringerBerlin Heidelberg. ISBN 978-3-540-28650-9. doi:10.1007/978-3-540-28650-9 4. URL https://doi.org/10.1007/978-3-540-28650-9 4.

[25] Richardson, L. F. 2007. Weather prediction by numer-ical process. Cambridge university press.

[26] Sanbonmatsu, K.; and Tung, C.-S. 2007. High perfor-mance computing in biology: multimillion atom sim-ulations of nanoscale systems. Journal of structuralbiology 157(3): 470–480.

[27] Schmidt, M.; and Lipson, H. 2009. Distilling Free-Form Natural Laws from Experimental Data 324: 81–85. ISSN 0036-8075. doi:10.1126/science.1165893.

[28] Scholkopf, B.; and Smola, A. J. 2001. Learningwith Kernels: Support Vector Machines, Regulariza-tion, Optimization, and Beyond. Cambridge, MA,USA: MIT Press. ISBN 0262194759.

[29] Sidhu, I.; Frischknecht, A. L.; and Atzberger, P. J.2018. Electrostatics of Nanoparticle-Wall Interactionswithin Nanochannels: Role of Double-Layer Structureand Ion-Ion Correlations. ACS Omega 3(9): 11340–11353. ISSN 2470-1343. doi:10.1021/acsomega.8b01393. URL https://doi.org/10.1021/acsomega.8b01393.

[30] van der Giessen, E.; Schultz, P. A.; Bertin, N.; Bula-tov, V. V.; Cai, W.; Csanyi, G.; Foiles, S. M.; Geers,M. G. D.; Gonzalez, C.; Hutter, M.; Kim, W. K.;Kochmann, D. M.; LLorca, J.; Mattsson, A. E.; Rot-tler, J.; Shluger, A.; Sills, R. B.; Steinbach, I.; Stra-chan, A.; and Tadmor, E. B. 2020. Roadmap on mul-tiscale materials modeling. Modelling and Simulationin Materials Science and Engineering 28(4): 043001.doi:10.1088/1361-651x/ab7150. URL https://doi.org/10.1088/1361-651x/ab7150.

[31] Washington, W. M.; Buja, L.; and Craig, A. 2009.The computational future for climate and Earthsystem models: on the path to petaflop and be-yond. Philosophical Transactions of the Royal Soci-ety A: Mathematical, Physical and Engineering Sci-ences 367(1890): 833–846. doi:10.1098/rsta.2008.0219. URL https://royalsocietypublishing.org/doi/abs/10.1098/rsta.2008.0219.