IMA Thematic Year on Mathematics of Materials and Mathematics of Materials and

MacromoleculesMacromolecules

Thanks to Local Organizers: Mitch Luskin, Maria Calderer,

Dick James

Effective Theories for Materials Effective Theories for Materials and Macromoleculesand Macromolecules

Sloppy Models: Universality in Sloppy Models: Universality in Data FittingData Fitting

Kevin S. Brown, JPS, Rick Cerione, Chris Myers, Kelvin Lee, Josh Waterfall, Fergal Casey, Ryan Gutenkunst, Søren Frederiksen,

Karsten Jacobsen, Colin Hill, Guillermo Calero

+NGF

Error Bars for Interatomic Potentials

Cell Dynamics

Fitting Exponentials, Polynomials

Ensemble: Extrapolation

Ensemble:Interpolation

Fitting Decaying ExponentialsFitting Decaying Exponentials

ttt eAeAeAt 321321)(

Classic Ill-Posed Inverse Problem

Given Geiger counter measurements from a

radioactive pile, can we recover the identity of the elements and/or

predict future radioactivity? Good fits with bad decay rates!

Fit

DN

i i

iyyC

12

2))(()(

P, S, I3532 125

6 Parameter Fit

PC12 DifferentiationPC12 DifferentiationE

RK

*

Time

10’

ER

K*

Time

10’

EGFREGFR NGFRNGFR

RasRas

SosSos

ERK1/2ERK1/2

MEK1/2MEK1/2

Raf-1Raf-1

+NGF+EGF

Pumps up signal (Mek)Tunes down

signal (Raf-1)

Biologists study which proteins talk to which. Modeling?

48 Parameter Fit

‘‘Sloppy Model’ Errors for AtomsSloppy Model’ Errors for AtomsBayesian Ensemble Approach to Error Estimation of Interatomic Potentials

Søren Frederiksen, Karsten W. Jacobsen, Kevin Brown, JPS

Atomistic potential820,000 Mo atoms(Jacobsen, Schiøtz)

Quantum Electronic

Structure (Si)90 atoms (Mo)

(Arias)

Interatomic Potentials V(r1,r2,…)• Fast to compute• Limit me/M → 0 justified• Guess functional form Pair potential V(ri-rj) poor Bond angle dependence Coordination dependence• Fit to experiment (old)• Fit to forces from electronic structure calculations (new)

17 Parameter Fit

Why the Name Sloppy Model?Why the Name Sloppy Model?Huge Fluctuations around Best Fit

eigen

parameters

bare parameters

Best Fit

Hessian ∂2C/∂2 at Best FitSloppy Directions Small Eigenvalues

Eigenvalues Span Huge Range

Each eigenvalue ~three times next

Ill-conditioned

Stiff 1cm Sloppy~meters,km

Local Collinearity of Parameters

Many alternative fits just as good

Huge ranges of allowed parameters

Eig

enva

lue

Tyson

Brown

Kholodenko

Stiff Sloppy

Sloppy Model EigenvaluesSloppy Model EigenvaluesMany fitting problems are sloppy

Molybdenum Interatomic Potential

Cell Dynamics Lessons:• Sloppy Due to Insufficient Data?

No: Perfect Data Sloppy Too• Survives Anharmonicity? Yes: Principle Component Analysis

Signal Transduction

Polynomial Fitting

Anharmonic

Perfect (Fake) Data

H

Ensemble of ModelsEnsemble of ModelsWe want to consider not just minimum cost fits, but all

parameter sets consistent with the available data. New level of abstraction: statistical mechanics in model space.

Generate an ensemble of states with Boltzmann weights exp(-C/T) and compute for an observable:

222

1

)()(

)(1

OO

ON

O

O

N

ii

E

E

O is chemical concentration, or rate constant …

TVVH

bare

eigen

Don’t trust predictions that vary

48 Parameter “Fit” to Data48 Parameter “Fit” to Data

ER

K*

Time

10’

ER

K*

Time

10’

+EGF

+NGF

DN

i i

iyyC

12

2))((

2

1)(

bare

eigen

Cost is Energy

Ensemble of Fits Gives Error Bars

Error Bars from Data Uncertainty

Does the Erk Model Does the Erk Model Predict Predict

Experiments?Experiments?

Model predicts that the left branch isn’t important

Bro

wn’

s E

xper

imen

tM

odel

Pre

dict

ion

Predictive Despite Sloppy Fluctuations!

Which Rate Constants are in the Stiffest Eigenvector?Which Rate Constants are in the Stiffest Eigenvector?

**

*

*

*

stiffest **

* *

2nd stiffest

Eigenvector components along

the bare parameters reveal which ones are most important

for a given eigenvector.

Ras

Raf1

Oncogenes

Interatomic Potential Error BarsInteratomic Potential Error Bars

Best fit is sloppy: ensemble of fits that aren’t much

worse than best fit. Ensemble in

Model Space!T0 set by

equipartition energy = best cost

Error Bars from quality of

best fit

Ensemble of Acceptable Fits to Data Not transferable Unknown errors• 3% elastic constant• 10% forces• 100% fcc-bcc, dislocation core

Green = DFT, Red = Fits

T0

Sloppy Molybdenum: Does it Work?Sloppy Molybdenum: Does it Work?Comparing Predicted and Actual Errors

Sloppy model error i gives total error if ratio r = errori/i

distributed as a Gaussian: cumulative distribution P(r)=Erf(r/√2)

Three potentials• Force errors• Elastic moduli• Surfaces• Structural• Dislocation core• 7% < i < 200%

Note: tails…MEAM errors underestimated by ~ factor of 2

“Sloppy model” systematic

error most of total

~2 << 200%/7%

Fitting Polynomials: HilbertFitting Polynomials: HilbertWhat is Sloppiness?

Sloppiness as Perverse, Skewed Choice of

Preferred Basis(Human or Biological)

Polynomial fit: L2 norm

• Hessian = 1/(i+j+1) = Hilbert matrix(Classic ill-conditioned matrix)• Monomial coefficients n sloppy.• Orthonormal shifted Legendre

• Coefficients an not sloppy

1

0

2))(()( dxxfxC nn

)(xPaModel nn

Exploring Parameter SpaceExploring Parameter SpaceRugged? More like Grand Canyon (Josh)

Glasses: Rugged LandscapeMetastable Local ValleysTransition State Passes

Optimization Hell: Golf CourseSloppy Models

Minima: 5 stiff, N-5 sloppySearch: Flat planes with cliffs

Ensemble Fluctuations Along Ensemble Fluctuations Along EigendirectionsEigendirections

log e

flu

ctua

tions

alo

ng

eige

ndire

ctio

n

stiff

sloppy

3x previous

Monte Carlo Fluctuations Suppressed in Soft Directions: Anharmonicity or Convergence?

Work In Progress

Error BarsError BarsStochastic versus Sensitivity

Sensitivity Analysis = Harmonic Approximation for Errors• Yields Much Larger Prediction Fluctuations• Anharmonicity Constrains Soft Modes• Mimic w/ modest prior (fluctuations < 106, one )• Sensitivity w/Prior Fluctuations Now Close to Monte Carlo

Work In Progress

Sloppy Model Universality?Sloppy Model Universality?Why are all these problems so similar?

Work In Progress

Random Matrix GOE Ensemble: many different NxN random symmetric matrices have level repulsion, universal ~Wigner-Dyson spacings as N→ Product ensemble: equally spaced logs! stronger level repulsionFitting exponentials: very strong level repulsion!New random matrix ensemble?

Fitting exponentials: stiffest minus second

Strong Level

Repulsion