© 2013 ibm corporation a. craik, c black, p doyle 4-nov-14 mincer: a distributed automated problem...

© 2013 IBM Corporation

A. Craik, C Black, P Doyle 4-Nov-14

Mincer: a distributed automated problem determination tool


Debugging Optimizing Compilers

Nearly every optimizing transformation is optional

Adopt the scientific method – run experiments varying compilation control parameters to find failure

Problems

Time consuming & tedious to configure & run

Resource constraints limit number of experiments

Experience needed to choose the best experiments

Environment may cause problem to occur intermittently

Resources wasted repeatedly running the same experiments


Objectives

Tool to run tests in a distributed machine farm

Distributed experiment selection and execution logic for scalability

Configurability & extensibility to support new transformations and new debugging techniques

Reduce resource waste – gain knowledge from every experiment run

Tolerate false-positive & false-negative results and handle variable failure rates


Problem Conceptualization

A problem determination tactic aims to isolate a problem by varying an experimental parameter and observing the effect

Problem has two discrete components:– Deciding which problem determination tactics to use– Applying chosen problem determination tactics to failure

Each experimental parameter can be mapped back to an integer search space

Mathematically model the selection of experiments and the results they produce


Mincer Architecture

mincer.pl

DataLayer

Sequential Subset

LastOptIndexOptLevel

TacticExecutionEngine

GatherLogs

Solvers

Tactics

Results DB

LastOptSubIndexLimitFile


Experiment Selection Logic


The Numerical Solvers

Solvers use Bayesian Inference to update the underlying hyperparameters of the experiments

Solvers use Information Theory to make optimal test parameter choices

Solve to a user-defined confidence level (-c option)

Currently using 2 solvers: - sequential (contiguous subranges from 1 to some n)

- subset (unordered subsets)

Solvers also track failure rate to guide experiment selection


Sequential Solver


Current Status

Mincer being used to debug problems in production

We have implemented tactics for several different experimental parameters on our Java JIT

Working on Infinite Sequential Solver to allow us to isolate intermittent failures to specific changesets


Q & A


Sequential Solver

TheoryHave a finite, ordered set, T

Random Variable P, support on TRandom Variable f, support on [0, 1]Random Variables {Xi}, binary


Sequential Solver

TheoryEach Xi is associated with an index ni in TPr(Xi passing | f, P) = (1-f) if ni ≥ PPr(Xi passing | f, P) = 1 if ni < P

Choose index to associate with next Xi optimally


Subset Solver

TheoryEach Xi is associated with a subset Ni of TPr(Xi passing | f, P) = (1-f) if P ϵ Ni

Pr(Xi passing | f, P) = 1 otherwise

Choose next subset optimally

Faster convergence than possible with Sequential Solver

© 2013 ibm corporation a. craik, c black, p doyle 4-nov-14 mincer: a distributed automated problem...

Documents