Automati Di�erentiationGeorge F. Corliss, Marquette University, MilwaukeeCollaborators:Bis hof, Carle, Griewank, Hovland, Rall, Chang, ...Outline of the talk:� Need derivatives?� Fun tionality of automati di�erentiation� Challenges - mathemati al� Challenges - omputer s ien e� Survey �eld, not tool user manualA ompilation of available automati di�erentiationsoftware: www.m s.anl.gov/Proje ts/autodiff/AD_Tools/Fields Industrial Optimization Seminar,University of Toronto, De ember 7, 2004

1

Abstra tAutomati di�erentiation is a te hnique forproviding fast, a urate values of derivativeobje ts (gradients, Hessians, Taylor series)required by modern tools for optimization,nonlinear systems, di�erential equations,or sensitivity analysis. We outline some needsand appli ations for derivatives, survey thefun tionality of AD in forward and reverse modes, and dis uss some ofthe mathemati al and omputer s ien e hallenges of AD.This talk should be a essible after �rst semester al ulus and aprogramming ourse in data stru tures. It should be interesting toresear hers in symboli omputation or in s ienti� or engineeringappli ations requiring almost any numeri al analysis te hniquerequiring derivatives.

Thunderstorm image: www.theweather lub1.homestead. om/tpi s.html2

Need derivatives?Top 10 indi ators that AD may be for you:1. Code requires derivatives 6. Sensitivity2. Using �nite di�eren es 7. Adjoints3. Newton's method 8. Optimization4. Interval remainder terms 9. Sti� ODE's5. Nonlinear equations 10. DAE'sIf anyof these sound familiar, you are in dangerof �nding automati di�erentiation useful� Mi rosoft Ex el (via Frontline Systems)� AMPL (www.ampl. om)� NEOS (www-neos.m s.anl.gov)all use ADSee IMSL libraryPhotography by Joaquim Martins, mdolab.utias.utoronto. a/ gi-bin/ids/index. gi?mode=album&album=./Aeronauti s 3

Sensitivities?

Cloud near Melbourn, FL. From www.theweather lub1.homestead. om/tpi s.html 4

Ten minutes later

5

Di�erentiationAutomati /Computational:� Propagates values� A urate up to round-o�Symboli :� Propagates expressions� Bran hes?� Large ode?Numeri al:� Stable?� h = ?Photography by Joaquim Martins,mdolab.utias.utoronto. a/ gi-bin/ids/index. gi?mode=album&album=./Aeronauti s 6

AD Fun tionalityGiven a program for the omputation of a fun tion, AD produ es aprogram for evaluating the derivative exa tly by applying the hainrule to the elementary instru tionsProgram for f# ADProgram for f'

Sour e ode transformation: ADIFOR, TAMC, Odyss�ee, Padre2, ...Operator overloading: ADOL{C, ADOL{F, Lohner's AWA, ...7

AD Fun tionalityAD implements the hain ruleFor f = f(x1; : : : ; xm), its gradient ve torrf = " �f�x1; : : : ; �f�xm# ;is required, for example, by most algorithms for optimization.Let u and v be fun tions whose gradients ru and rv are known or arepreviously omputed, we ompute rf using the rulesr(u� v) = ru�rv;r(uv) = urv+ vru;r(u=v) = (ru� (u=v)rv)=v; v 6= 0;for the arithmeti operations and the hain ruler�(u) = �0(u)ru;for di�erentiable fun tions � (su h as the standard fun tions) withknown derivatives 8

AD Fun tionalityAssignment statementf(x; y) = (xy+ sinx+4)(3y2+6);exe uted as unary & binary operators ( ode list, omputational graph):t1 = x t6 = t5+4t2 = y t7 = t22t3 = t1t2 t8 = 3t7t4 = sin t1 t9 = t8+6t5 = t3+ t4 t10 = t6t9Derivatives may be evaluated in� Forward mode� Reverse mode9

Computational Graph: f(x; y) = (xy+ sinx+4)(3y2+6)

uE c)�uE c)�

x

+ +

e + x S

t�? x � 2

))

+it�*|G

W?_iTi?_i?|tG

10

AD - Forward ModeCompute � intermedate� independentWith every variable u asso iate a derivative obje t r�uIn our example, start with r�x and r�y and ompute r�f :t1 = x rt1 = [1; 0℄t2 = y rt2 = [0; 1℄t3 = t1t2 rt3 = t1rt2+ t2rt1 yields [t2; t1℄t4 = sin t1 rt4 = ( os t1)rt1 yields [ os t1; 0℄t5 = t3+ t4 rt5 = rt3+rt4 yields [t2+ os t1; t1℄t6 = t5+4 rt6 = rt5 yields [t2+ os t1; t1℄t7 = t22 rt7 = 2t2rt2 yields [0; 2t2℄t8 = 3t7 rt8 = 3rt7 yields [0; 6t2℄t9 = t8+6 rt9 = rt8 yields [0; 6t2℄t10 = t6t9 rt10 = t6rt9+ t9rt6 yields [t9(t2+ os t1);6t2t6+ t1t9℄Code for F : s alar operationsCode for �F�x : ve tor operations{ Parallelize/ve torize 11

AD - Reverse ModeCompute � dependent� intermedateWith every variable u asso iate an ADJOINT obje t ubar = �w�uKey rule for the reverse mode:s = f(t; u) ) tbar += sbar * �f�tubar += sbar * �f�uReverse mode requires the ability to reverse the ow of the omputation

12

AD - Reverse ModeAssignment: f(x; y) = (xy+ sinx+4)(3y2+6)In our example, start with �f�f = 1 and ompute r�f :�t10�t10 = 1�t10�t9 = t6�t10�t8 = �t10�t9 �t9�t8 yields t6 � 1 = t6�t10�t7 = �t10�t8 �t8�t7 yields t6 � 3 = 3t6�t10�t6 = t9�t10�t5 = �t10�t6 �t6�t5 yields t9 � 1 = t9

13

AD - Reverse Mode ( ont.)

�t10�t4 = �t10�t5 �t5�t4 yields t9 � 1 = t9�t10�t3 = �t10�t5 �t5�t3 yields t9 � 1 = t9�t10�t2 = �t10�t7 �t7�t2 + �t10�t3 �t3�t2 yields (3t6) � (2t2) + t9 � t1 = 6t2t6+ t1t9�t10�t1 = �t10�t4 �t4�t1 + �t10�t3 �t3�t1 yields t9 � os t1+ t9 � t2 = t9( os t1+ t2)

rf = h�t10�t1 ; �t10�t2 i = [t9( os t1+ t2); 6t2t6+ t1t9℄r� f = xbar * r�x + ybar * r�y{ mostly s alar operations{ one ve tor parallel loop

14

Has o�et: Adjoining a Data-Dependen e Graph

x = 2*a*b

T(i) = ... x

v = ... x

x = c + 2*d

b += 2*a*xa += 2*b*xx = 0

... += ... T(i) ...x += ... T(i) ...... += ... T(i) ...T(i) = 0

x += ... v ...... += ... v ...v = 0

d += 2*xc += xx = 0

execution orderex

ecut

ion

ord

er15

AD - Hybrid Mode\Prea umulation of Lo al Derivatives"If w = f(y; z), r�w = �w�y � r�y+ �w�z � r�zCompute \lo al"derivatives �w�y and �w�z in the reverse modePropagate \global"derivatives r�w in the forward modeGeneralizes to se ond derivatives

Navajo Bridge, Marble Canyon, Arizona, National Information Servi e for EarthquakeEngineering nisee.berkeley.edu/images/servlet/BrowseGodden?group=GoddenD27.1-316

Implementation - Sour e TransformationE.g., Adifor 2.0 for Fortran [Bis hof, Carle, et al.℄Identify top level subroutine:C File: squareroot.fC Referen e: ADIFOR 2.0 User's Guide, se tion 2.6C Figure 2.4. A Very Simple Subroutine.subroutine squareroot (x, y)real x, yy = sqrt (x)end

17

Implementation - ADIFOR-generated ode:subroutine g_squareroot(g_p_, x, g_x, ldg_x, y, g_y, ldg_y)real x, yinteger g_pmax_parameter (g_pmax_ = 1)integer g_i_, g_p_, ldg_y, ldg_xreal r1_p, r2_v, g_y(ldg_y), g_x(ldg_x)integer g_ehfiddata g_ehfid /0/ all ehsfid(g_ehfid, 'squareroot','g_squareroot.f')if (g_p_ .gt. g_pmax_) thenprint *, 'Parameter g_p_ is greater than g_pmax_'stopendifr2_v = sqrt(x)if ( x .gt. 0.0e0 ) thenr1_p = 1.0e0 / (2.0e0 * r2_v)else all ehufSO (9,x, r2_v, r1_p, g_ehfid, 42)endifdo g_i_ = 1, g_p_g_y(g_i_) = r1_p * g_x(g_i_)enddoy = r2_vend 18

Implementation - Sour e TransformationNew main program (or modi�ed appli ations algorithm):� allo ate variable storage� all ADIFOR-generated ode� use the derivativeprogram driverreal x, yreal g_x(1), g_y(1)print *, ' Enter x: 'read *, xg_x(1) = 1.0 all g_squareroot (1, x, g_x, 1, y, g_y, 1) all ehrptprint *, yprint *, g_y(1)end

19

Implementation - Operator OverloadingE.g., ADOL-C [Griewank, et al.℄ lass adouble f ... gFeatures in lude� Forward or reverse mode� Partial derivatives or Taylor series� Re ord omputations on a \tape"� Ja obians with stru ture� Impli it fun tions� ODE tools { impli it Taylor series/Pad�e� \Che kpointing"

Spiral bevel gears, Emerson Power Transmission Company,s ien e.howstuffworks. om/gear4.htm

20

Math - Not di�erentiable?f is de�ned, but f 0 does not exist?� E.g., f(x) = kxk; f(x) = maxfg(x); h(x)g� Very important { norms, s aling

��������

��������

~~~Automati di�erentiation di�erentiated the ode:if x < 0 then f = �x; f 0 = �x0else f = x ; f 0 = x0Yields k0k0 = 1 � x0Elementary fun tions are supportedIssue here is how to handle user's odeWarning: You should know what you are doing!

21

Math - Spe ial ases .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

............................................................................~

f isde�ned by a spe ial ase? E.g., f(x) = x2 [Fis her℄if x = 1 then f = 1 ; f 0 = 0else f = x2; f 0 = 2 � x � x0You would never ode it that way???Many odes have hard- oded spe ial ases

Automati di�erentiation di�erentiates your ode,NOT your intentionsIf Interior (Closure (Domain)) 6= ;, then derivatives omputed = limits22

Math - Impli it fun tions ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

.................................................................................................................................................................................................................................Original algorithm: Fun tion is de�ned impli itly:F (z; x) = 0 de�nes z�(x) and z0�(x)Pre ondition: Pk � F(z; x) = 0Iterate: zk+1 = zk � Pk � F(z; x)Converges when jjI � Pk � Fzjj � � < 1e.g., Newton's methodStrategy 1: Bla k boxApply ADUsually worksMay need 1 or more extra iterationsCan fail

23

Math - Impli it fun tionsStrategy 2: White boxIterate fun tion z(x) to onvergen e{ Tests based on F(z; x)Then turn on AD and iterate for z0(x){ Tests based on Fzz0+ Fx{ Often, one iteration suÆ esDo not need to di�erentiate the entire iterationExample: NASA{ Input: 6 - 100 wing shape parameters{ Generate initial 3-D �nite element grid{ Iterate to steady-state pressure distribution solving PDE withmulti-grid{ Vary parameters to optimize design (uses Newton){ f 0 ost only a few % additional!

24

Math - Impli it fun tionsStrategy 3: Use some math - Derivative z0�Di�erentiate to get: Fzz0+ Fx = 0Pre ondition: Pk � (Fzz0+ Fx) = 0Iterate: z0k+1 = z0k � Pk � (Fzz0+ Fx)Converges when jjI � Pk � Fzjj � � < 1Same ondition!A little math beats a lot of omputingWork of Gilbert, Griewank, Christianson, et al.

25

Computer S ien e - Compiler te hnologiesSour e transformation and operator overloading stret h ompilersAdifor was built on Ken Kennedy, et al. (Ri e) Paras ope tools forHigh Performan e Fortran� Analysis and annotation of omputational graph� Interpro edural dependen y analysis� Common subexpression removal� Code pruning (unused ode)� Parallel s heduling?Sun and NAG have let it be known that their Fortran ompilers willsoon have an -AD option

26

Computer S ien e - Che kpointingReverse mode (or multiple passes of forward) re ords exe ution tra e(potentially enormous)� Re ord: Operation, operands [result℄� Implies storage proportional to run timeSolution: Che kpointing, as used for exe ution re overyRequires memory to store 1 he kpoints (this example)Requires 2 forward passes of same ode (this example)Requires 1/2 as mu h tape storageVs. \Dry-run" (Has o�et)time

dry-run fragment{27

Computer S ien e - ComplexityFinite di�eren e approximation to r�f requires n+1 evaluations of fForward mode AD osts about 2-3 n � Cost(f)� Why? f = u � v ) f 0 = u � v0+ u0 � v has two �Reverse mode AD osts about 5 � Cost(f) independent of nf : Rn �! Rm rules of thumb:� n >> m (e.g., m = 1) prefer reverse� n << m (e.g., n= 1) prefer forwardSee omputational graph node elimination

28

Computer S ien e - ComplexityTadjoudine, Forth, and Pry e:Roe's numeri al ux ( ow �elds with moderate to strong sho ks)

Method time (rF) time (rF)time (F) Deviation vsADIFORFinite di�eren e, 1-sided 0.12 10.64 4.34E-07ADIFOR 0.15 13.37 |TAMC (Forward) 0.13 11.94 4.66E-15TAMC (Reverse) 0.11 10.28 5.77E-15AD01 (Forward) 1.55 134.68 7.99E-15AD01 (Reverse) 0.95 82.90 4.88E-15

CPU timings (in se onds) on SGI IRIX64 IP27

29

Computer S ien e - Graph node elimination

-1 0

1

2

3

4 5

ab

c

ed

fg

h1 = x1 � x2h2 = exp(sin(h1))y1 = h1 � h2y2 = os(h2)ForwardReverseHybrid? Minimize number of *Markowitz [Griewank, Naumann℄

30

Computer S ien e - Graph node elimination

6543

-1 0

a b

c

d

ef g h

2

1

a b c

f g

e h

k

-3 -2 -1 0

1 2

4 5 6 7

3

j

i l

d

Naumann: \Lion" and \bat" graphsNeither vertex nor edge elimination solves the general \OptimalJa obian A umulation" problem

31

Elimination sequen eDi�erent strategies applied to Roe0 50 100 150 200 250

0

50

100

150

200

250

Elimination Step

Nod

e E

limin

ated

Markowitz

0 50 100 150 200 2500

50

100

150

200

250

Elimination Step

Nod

e E

limin

ated

VLR

0 50 100 150 200 2500

50

100

150

200

250

Elimination Step

Nod

e E

limin

ated

Markowitz (reverse bias)

0 50 100 150 200 2500

50

100

150

200

250

Elimination Step

Nod

e E

limin

ated

VLR (reverse bias)

32

Appli ationsSee Computational Di�erentiation, SIAM, 1996; and Automati Di�erentiation, Springer, 2001a ousti wave equation aerofoil optimization hemi al rea tion ir uit simulation omputational aerodynami s omputational uid dynami s omputational quantum hemistry ooling system oupled penduladata assimilation design optimization disordered ondensed matterdiurnal kineti s adve tion-di�usion elasti -plasti torsion ele tromagneti problemele tron paramagneti resonan e Euler equations gear tooth onta thypoid bevel gears low thrust orbit transfer material modellingminimum time orbit transfer model �tting model stru ture validationMOS transistor model multibody dynami s Navier-Stokes solvernear-earth asteroids neutron s attering nonlinear least squaresnonlinear regression nonlinear solver North Sea herringobserver design ODE solver optimal ontrolparameter identi� ation parametri un ertainty analysis partial di�erential equationsperiodi fun tions power plant model ra e ar performan eradiation di�usion rational approximation sea i esensitivity analysis stru tural analysis stru tural me hani sterrain modelling thermal-hydrauli vehi le dynami s33

Con lusionsDerivatives an be omputed{ eÆ iently{ a urately{ easilyUsers an be expe ted to provide for library odesAlgorithm designers need not avoid derivativesBla k box users{ Minimize programmer and user e�ort{ Gets the right answer ompetitively with FDWizards{ Willing to apply knowledge of problem{ Get the right answer FASTGeorge Corliss, George.Corliss�Marquette.eduAD dragon - logo for AD2000, Automati Di�erentiation: From Simulation toOptimization, Ni e, Fran e, June 2000 34