branch detection and sparsity estimation in matlab euroad workshop - marina... · branch detection...

Branch Detection and Sparsity Estimation in Matlab

Marina Menshikova & Shaun Forth

Engineering Systems DepartmentCranfield University (DCMT Shrivenham)

Shrivenham, Swindon SN6 8LA, U.K.email: [email protected]/[email protected]

6th European Workshop on Automatic DifferentiationINRIA Sophia-Antipolis, France, November 15-16 2007

1/ 28 Branch Detection and Sparsity Estimation in Matlab

Plan

1 Branch Detection - MotivationSparsity estimation for JacobianReverse mode AD implemented via tapingUncertainty propagation using Taylor series

2 Branch detection principlesMain ideaImplementation

3 Examples

4 Sparsity EstimationSparsity Detection in MatlabThe sparsemad ClassWhy do we Propagate Entries (Value Sparsity) and DerivativeSparsity?Test Cases

5 Conclusions


Branch Detection - Motivation

Branch Detection

Branch Detection - detection of data-dependant control flow.

Some examples illustrate necessity of this.


Sparsity estimation for Jacobian

function y = y(x(1),x(2),x(3))

if x(1) > 0y = [x(1); x(2)*x(3)];

elsey = [x(1) + x(2); x(3)];

end

J =

[

1 0 00 x(3) x(2)

], x(1) > 0;[

1 1 00 0 1

], x(1) ≤ 0.

SJ =

[

1 0 00 1 1

], x(1) > 0;[

1 1 00 0 1

], x(1) ≤ 0.


Reverse mode AD implemented via taping

function y = y(x(1),x(2),x(3))

if x(1) > 0y = x(1) + x(2)*x(3);

elsey = x(1) + x(2) + x(3);

end

Tapes of operations:* x(2) x(3) v+ v x(1) y

, x(1) > 0;

+ x(1) x(2) v+ v x(3) y

, x(1) ≤ 0.


Uncertainty propagation using Taylor series

Taylor series for function y with single input x is

y(x) = y(µx ) +∂y

∂x(x − µx ) +

1

2!

∂2y

∂x2(x − µx )2

+1

3!

∂3y

∂x3(x − µx )3 + O

((x − µx )4

)For x distributed with mean µx , variance σ2

x , skewness S(x) and KurtosisK (x), then expectation and variance are approximated by

µy = y(µx ) +1

2

∂2y

∂x2σ2

x + O(σ3x )

σ2y =

(∂y

∂xσx

)2

+∂2y

∂x2

∂y

∂xS(x)σ3

x +1

4

(∂2y

∂x2σ2

x

)2 (K (x)− 1

)+

1

3

∂3y

∂x3

∂y

∂xK (x)σ4

x + O(σ5x )

provided Taylor series valid for range of x - no change of control-flow.


Main idea

Overloading comparison functions in fmad== >= > <= < ∼=eq.m ge.m gt.m le.m lt.m ne.m

Use of global MADBRANCHES with components.switch.new.info{:, 1:5}.old

function y=eq(a,b)% overloaded equality test for one or more fmad variables

y=deactivate(a)==deactivate(b);MADStoreBranches(a,b,y,’eq’);


function MADStoreBranches: part 1

global MADBRANCHES% Check if branch detection is requiredif isfield(MADBRANCHES, ’switch’) && MADBRANCHES.switch

. . . && (isactive(a) || isactive(b))MADBRANCHES.new = [MADBRANCHES.new; y(:)];ST = dbstack(’-completenames’);% Get call stackchange = false;i = MADBRANCHES.i;

for j = 2:i % See if present branch has been located beforeif strcmp(MADBRANCHES.info{j,1},ST(3).file) &&. . . MADBRANCHES.info{j,3} == ST(3).line

MADBRANCHES.info{j,5} = MADBRANCHES.info{j,5} + 1;change = true;break;

endend =⇒ . . .


function MADStoreBranches: part 2

% Check if this location has been seen before.if change

i = i + 1;% Record the information about new locationMADBRANCHES.info{i,1} = ST(3).file;MADBRANCHES.info{i,2} = ST(3).name;MADBRANCHES.info{i,3} = ST(3).line;MADBRANCHES.info{i,4} = op;MADBRANCHES.info{i,5} = 1;MADBRANCHES.i = i;

end


Example

1 function y = func(a,b,c)

2 v = a + b;3 if v > 24 y = 2*c;5 else6 y = v.*c;7 end

>> a = fmad(2, [1 0 0]); b = fmad(1, [0 1 0]); c =fmad(-3, [0 0 1]);>> MADSetBranches(’BranchDetection’,’ON’);The branch detection is switched on.>> func(a,b,c);>> branch = MADGetBranches(’Branches’)branch =

1


Example

1 function y = func(a,b,c)

2 v = a + b;3 if v > 24 y = 2*c;5 else6 y = v.*c;7 end

>> MADSetBranches(’BranchDetection’,’AGAIN’);The result of previous branch detection is recorded.>> func(c,a,b);>> branch = MADGetBranches(’AllBranches’)branch =

[1] [0]


Example

>> branch = MADGetBranches(’HTML’)fid=fopen(’MADBRANCHES.html’,’w’);fprintf(fid,’%s\n’,’<HTML>’);. . .fprintf(fid,’%s\n’,’</HTML>’);fclose(fid);branches = web([’file:///’ which(’MADBRANCHES.html’)]);


Example

1 function y = func2(a,b,c)

2 y = 0;3 while abs(a) <= 104 y = y + 1;5 a = a + 1;6 end

>> MADSetBranches(’BranchDetection’,’ON’);The branch detection is switched on.>> func2(a,b,c);>> MADGetBranches(’HTML’);


Example

1 function y = func2(a,b,c)

2 y = 0;3 while abs(a) <= 104 y = y + 1;5 a = a + 1;6 end


Sparsity Estimation

Compression Techniques [Gri00, Chap. 7] can greatly improve efficiency forcalculating sparse Jacobians or Hessians.


Example (Tridiagonal Jacobian)

1 Suppose Jf(x) is tridiagonalJf(x) =

b1 c1

a2 b2 c2

a3 b3 c3

a4 b4 c4

a5 b5

.2 Choose 5× 3 seed matrix

S =

1 0 00 1 00 0 11 0 00 1 0

.

3 Then calculating Jf(x) · S byforward mode AD (or FDapprox.) gives,

Jf(x) ·S =

b1 c1 0a2 b2 c2

c3 a3 b3

b4 c4 a4

a5 b5 0

,i.e., all the Jacobian entries.

4 Tridiagonal Jacobianevaluated using 3Jacobian-vector productsnot n.


Compression Process

1 Obtain the sparsity pattern - the (i , j) locations of all possiblynon-zero entries in the Jacobian.

2 Construct a seed matrix (or matrices) to initiate forward (and/orreverse) mode [Gri00, Chap. 7], [CV98], [GMP05].

3 Calculate Jacobian products with seed(s) and extract all Jacobianentries.


Compression Process

1 Obtain the sparsity pattern - the (i , j) locations of all possiblynon-zero entries in the Jacobian.

2 Construct a seed matrix (or matrices) to initiate forward (and/orreverse) mode [Gri00, Chap. 7], [CV98], [GMP05].

3 Calculate Jacobian products with seed(s) and extract all Jacobianentries.

Here we concentrate on step 1 - sparsity estimation.


AD and Sparsity Estimation

ADIFOR’s SparselinC package [BKBC96] allows for sparse storage ofderivatives - can be used to give Jacobian sparsity for a givenindependent x.

Griewank and Mitev have considered using Bayesian update strategiesto make use of Jacobian-vector and vector-Jacobian products toiteratively deduce sparsity patterns - probing.

Giering and Kaminski [GK06] - source-transformation implementationusing bit-vectors to indicate dependence of active variable on eachindependent (or dependent) in a pure (no values) calculation.

ADOL-C features overloaded bit-vector propagation.


Sparsity Detection in Matlab

More recent versions of Matlab do support some bit operations(bitand, bitor and bitxor) on unsigned integers - not investigatedto date since MAD tool [For06] relies on use of high-level (sparse)matrix operations.

Matlab also offers sparse logical matrices - can perform all thehigh-level matrix operations of the MAD package on them.


The sparsemad Class

Three components:

value - stores the numerical value of the object.

entries - logical array identifying the (nonzero) entries inthe value component.

sparsity - a derivvec object holding the sparsity patternfor each component of the value.


Initialising the sparsemad object

x=sparsemad([1 2 3 4 5],speye(5)>0)

value = 1 2 3 4 5entries = 1 1 1 1 1derivvec object sparsitySize = 1 5No. of derivs = 5directional derivative 1d = (1,1) 1directional derivative 2d = (1,2) 1:directional derivative 5d = (1,5) 1


The plus(+) Operation

function z=plus(x,y)if isa(x,’sparsemad’)

z=x;if isa(y,’sparsemad’)

z.value=z.value+y.value;z.sparsity=z.sparsity|y.sparsity;% | = logical orz.entries=z.entries|y.entries;% derivvec logical or

elsez.value=x.value+y;

endelse

z=y;z.value=x+y.value;

end


Why do we Propagate Entries (Value Sparsity) andDerivative Sparsity?

Consider,

x=[1 2 3 4]’; % column vectory=diag(x); % 4x4 diagonal matrixz=y*x; % matrix multiplication

z’s sparsity pattern is the 4× 4 identity.

Key here is sparsity propagation for matrix-multiplication,

z.sparsity = y.entries*x.sparsity + y.sparsity*x.entries

with above applied over each of the 4 directions for sparsity patterns.

If y’s entries are 4× 4 identity then sparsity pattern is correct.


Why do we Propagate Entries (Value Sparsity) andDerivative Sparsity? (ctd.)

If we didn’t store the entries then for y we could either:

1 Assume y’s entries were the non-zeros of its value component,

z.sparsity= (y.value~=0)*x.sparsity ...+ y.sparsity*(x.value~=0)

this fails (underestimates sparsity) if a component of x happens to bezero.

2 Assume all x’s and y’s entries are full,

z.sparsity= ones(size(y.value))*x.sparsity ...+ y.sparsity*ones(size(x.value))

which overestimates sparsity and would indicate that Jacobian wasfull.


Brusselator Problem

Semi-discretised ODE - 4 non-zeros per row of sparse Jacobian.

cpu(Jf)/cpu(f) with problem size n10 20 40 80 160 320

numjac(sparse) 7.5 7.4 8.1 8.8 9.6 9.9fmadsparse 96.8 87.6 78.5 68.5 57.5 48.6fmadcmp 88.6 78.0 67.1 55.0 39.6 25.5

cpu(Sf)/cpu(f) with problem size n10 20 40 80 160 320

hand-coded 2.0 2.2 2.9 3.7 4.8 5.0sparsemad 93.5 82.3 75.7 60.8 46.4 33.9


Finite Element ODE Problem

Finite-element semi-discretised of PDE - f (t, y) = Ay with A tri-diagonal.

cpu(Jf)/cpu(f) with problem size n10 20 40 80 160 320

numjac(sparse) 11.0 10.4 10.9 11.6 13.6 17.7fmadsparse 11.0 10.1 9.9 10.0 10.0 10.0fmadcmp 11.6 11.1 10.5 10.8 11.2 11.9

cpu(Sf)/cpu(f) with problem size n10 20 40 80 160 320

hand-coded 3.8 3.9 4.0 4.6 6.0 8.5sparsemad 11.6 11.3 11.2 11.2 11.4 11.3

Linearity of f (t, y) detected - a global MADsparsemadNonlinear set totrue if nonlinear function encountered by sparsemad class.


Conclusions

Branch detection allows us to check if sparsity pattern, tape or PDFTaylor series is safe to use for all inputs x.

Sparsity Detection allows us to determine Jacobian sparsity patterns.

Future Work

Discontinuous derivative testing to ensure validity of Taylor series.

Can we estimate how much of a pdf passes along currentcontrol-flow?

Adaptation of Taylor order for accurate statistical moments.

Sparsity detection based on bit-wise operations - can it beat thecurrent sparse matrix implementation?


References

Christian H. Bischof, Peyvand M. Khademi, Ali Bouaricha, and Alan Carle.

Efficient computations of gradients and Jacobians by dynamic exploitation of sparsity in automatic differentiation.Optimization Methods and Software, 7:1–39, 1996.

Thomas F. Coleman and Arun Verma.

The efficient computation of sparse Jacobian matrices using automatic differentiation.SIAM J. Sci. Comput., 19(4):1210–1233, 1998.

Shaun A. Forth.

An efficient overloaded implementation of forward mode automatic differentiation in MATLAB.ACM Trans. Math. Softw., 32(2):195–222, June 2006.DOI: http://doi.acm.org/10.1145/1141885.1141888.

Ralf Giering and Thomas Kaminski.

Automatic sparsity detection implemented as a source-to-source transformation.In Vassil N. Alexandrov, Geert Dick van Albada, Peter M. A. Sloot, and Jack Dongarra, editors, Computational Science –ICCS 2006, volume 3994 of Lecture Notes in Computer Science, pages 591–598, Heidelberg, 2006. Springer.dx.doi.org/10.1007/11758549_81.

Assefaw Hadish Gebremedhin, Fredrik Manne, and Alex Pothen.

What color is your Jacobian? Graph coloring for computing derivatives.SIAM Review, 47(4):629–705, 2005.

Andreas Griewank.

Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation.Number 19 in Frontiers in Applied Mathematics. SIAM, Philadelphia, PA, 2000.


http://doi.acm.org/10.1145/1141885.1141888

dx.doi.org/10.1007/11758549_81

branch detection and sparsity estimation in matlab euroad workshop - marina... · branch detection...

Documents