Identifying Active Variables to Improve the Performance of

Operator Overloading Automatic Differentiation

By: Drew WickeJointly with:

Sri Hari Krishna Narayanan and Paul Hovland

What is Automatic Differentiation?

A means of computing the derivative of a function within a computer program

Operator Overloading AD – Utilizes features of the programming language to alter the meaning of mathematical operators to compute the derivative

What is Sacado?

Operator overloaded version of ADMade by Sandia National LaboratoriesA package in the Trilinos framework

http://trilinos.sandia.gov/packages/sacado/index.html

DERIV_TYPE operator* (DERIV_TYPE other) {this->val = this->val() * other.val();this->dx = this->val() * other.dx() + other->val() * this->dx();return *this;

}

Example Code

//Function where x is the independent f// is the dependentvoid foo(double *x, double *f){

double theConst = 3.14 * 2 – 6;double div = ((*x) / 2);double squarePoly = (*x) * (*x) + 45;if (squarePoly >= 100) {

(*f) = div * theConst;}else{

double altInput = (*x) * theConst;

(*f) = div * altInput + theConst;}

}int main(){

double x, f; foo(&x, &f);

return 0;}

Sacado Exampletypedef Sacado::Fad::DFad<double> DERIV_TYPE_double;void foo(DERIV_TYPE_double x, DERIV_TYPE_double f);int main(){

DERIV_TYPE_double x, f;// independent and dependent vars

// init value of x and f is set to zero and deriv// value is set to zerox = 2.0; // set the initialx.diff(0, 1); // set that x is independentfoo(&x, &f); // call foo with x and f as args// print the value of x and the value of f// by calling the method val which returns// the non-derivative value printf("f(%f) = %f\n", x.val(), f.val());// print the derivative by calling the dx method// and pass zero as the arg to dx to specify what // independent variable we want the derivative// with respect toprintf("df/dx = %f\n", f.dx(0));return 0;

}

//Function where x is the input f is the outputvoid foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){

DERIV_TYPE_double theConst = 3.14 * 2 – 6;DERIV_TYPE_double div = ((*x) / 2);DERIV_TYPE_double squarePoly = (*x) * (*x) + 45;

if (squarePoly >= 100) {(*f) = div * theConst;

} else{

DERIV_TYPE_double altInput = (*x) * theConst;(*f) = div * altInput + theConst;

}

}

Example Continued

Why That Code is Not OptimalVary variables are those whose value is

computed using an independent variable.Useful variables are used to compute the

value of the dependent variableActive variables are both vary and useful.

void foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){

DERIV_TYPE_double squarePoly = (*x) * (*x) + 45;DERIV_TYPE_double theConst = 3.14 * 2 – 6;DERIV_TYPE_double div = ((*x) / 2);

if (squarePoly >= 100) (*f) = div * theConst;

else{DERIV_TYPE_double altInput = (*x) *

theConst;(*f) = div * altInput + theConst;

}}

What Does Our Tool Do?

Replaces the manual process, which is slow and overestimates the number active variables.void foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){

double squarePoly = (*x) * (*x) + 45; // varydouble theConst = 3.14 * 2 – 6; // usefulDERIV_TYPE_double div = ((*x) / 2); // activeif (squarePoly >= 100) {

(*f) = div * theConst;} else{

DERIV_TYPE_double altInput = (*x) * theConst;(*f) = div * altInput + theConst;

}}

int main(){

double val;val =

4.3;…

return 0;}

Activity Analysis Tool Flow

int main(){DERIV_TYPE_double val;val = 4.3;…return 0;

}

Input code

ROSE AST OpenAnalysis ICFG

Useful AnalysisVary AnalysisActivity Analysis

Change typeof active variables

Outputcode

convert convert

Challenges in Type Changing

Typedefs – must extract base type and all other types such as pointers to set to the derivative type.

Structs – must copy struct and change all variables with the body of the struct to the Sacado derivative type.

// after:DERIV_TYPE_double **test;

struct mine { double mdou; };struct DERIV_CLASS_mine {

DERIV_TYPE_double mdou;}typedef struct DERIV_CLASS_mine DERIV_TYPE_mine;DERIV_TYPE_mine smin;// (after)

typedef double* fir;typedef fir* sec;sec test;//active (before)

struct mine { double mdou; };struct mine smin; // active (before)

Initial Code

//Function where x is the independent f is the dependentvoid foo(double *x, double *f){# pragma $adic_indep,x# pragma $adic_dep,f// do some math

double theConst = 3.14 * 2 – 6; double div = ((*x) / 2);double squarePoly = (*x) * (*x) + 45;if (squarePoly >= 100) {

(*f) = div * theConst;}else{

double altInput = (*x) * theConst;(*f) = div * altInput + theConst;

}}

Resulting Code//Function where x is the input f is the outputvoid foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){# pragma $adic_indep,x# pragma $adic_dep,f // do math

double theConst = 3.14 * 2 – 6; // .28DERIV_TYPE_double div = ((*x) / 2);double squarePoly = (*x) * (*x) + 45;// ERROR must downcast!if (squarePoly >= 100) {

(*f) = div * theConst;}else{

DERIV_TYPE_double altInput = (*x) * theConst;(*f) = div * altInput + theConst;

}}

Add a downcast if the left hand side is vary and the right hand side has an active variable.

Resulting Code With Downcast//Function where x is the input f is the outputvoid foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){# pragma $adic_indep,x# pragma $adic_dep,f // do math

double theConst = 3.14 * 2 – 6; // .28DERIV_TYPE_double div = ((*x) / 2);double squarePoly = ADValue((*x) * (*x) + 45); // Problem solvedif (squarePoly >= 100) {

(*f) = div * theConst;}else{

DERIV_TYPE_double altInput = (*x) * theConst;(*f) = div * altInput + theConst;

}}

Tool Limitations

Typedefed structs: typedef struct{…} myStruct;

C++ structs, classes, templatesFunction pointers: double (*func)(double, double)

Array parameters due to not being made active: void foo(double myParam[]);

Conclusion and Future Work

Can be used with any operator overloading AD package

Benchmarking the toolAdd support for C++

Thank You

How Does the Tool Work?

• Creates Abstract Syntax Tree (AST) of code using ROSE

int main(){double val;val = 4.3;return 0;

}

• Converts AST to anOpenAnalysis form

definition

Creates Interprocedural Control Flow Graph (ICFG)

Images from:M. J. Harrold, G. Rothermel, and S. Sinha, “Computation of Interprocedural Control Dependence”, Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ‘98), pp. 11-20, March 1998

What Analyses are Used?

Vary Analysis – Identifies variables that vary with the independent variable, but are not used in the output

Useful Analysis – Identifies variables that are not vary but contribute to the output

Activity Analysis – Identifies active variables, those that depend on the value of an input variable to compute the output variable. The intersection of vary and useful variables.

What Does Our Tool Do?

Replaces the manual process that is slow and overestimates, with our tool.

Identifies active variables using the source code analysis toolkit OpenAnalysis

Changes the type of active variables to a typedef type using the source transformation tool ROSE void foo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){

double theConst = 3.14 * 2 – 6;DERIV_TYPE_double div = ((*x) / 2);double squarePoly = (*x) * (*x) + 45;if (squarePoly >= 100) {

(*f) = div * theConst;} else{

DERIV_TYPE_double altInput = (*x) * theConst;

(*f) = div * altInput + theConst; }

}