tao: turing test as objective function

26 May 2013, RAISE Workshop

TAOismShin Yoo/Centre for Research on Evolution Search and Testing, UCL, UK

My World

My World

SoftwareEngineering

My World

Meta-heuristicOptimisation

SoftwareEngineering

My World

SBSEMeta-heuristicOptimisation

SoftwareEngineering

My World

SBSEMeta-heuristicOptimisation

SoftwareEngineering

SBSE = Representation + Objective Function + Optimisation

Test Data Generation Software Modularisation

Test Data Generation

“I want the value of x to makethe following predicate true”

if(x == 3) {...}

Software Modularisation



if(x == 3) {...}

Objective Function

f(x) = |3 - x|




if(x == 3) {...}

Objective Function

f(x) = |3 - x|

This is concrete: the problemdefines the objective function

directly and concretely.




if(x == 3) {...}

Objective Function

f(x) = |3 - x|




“I want my 892 classesto be properly modularised”

(insert your ugliest classes here)



if(x == 3) {...}

Objective Function

f(x) = |3 - x|






Objective Function

?



if(x == 3) {...}

Objective Function

f(x) = |3 - x|






Objective FunctionMin. coupling, Max. cohesion



if(x == 3) {...}

Objective Function

f(x) = |3 - x|






Objective Function

This is surrogate: the objective function is something we hope to

correlate with some abstract property.

Min. coupling, Max. cohesion

How do we solve problems with surrogate objectives better?

TAOism

TAOism

Turing-test As Objective function

TAO Hypothesis

✤ Surrogate objective functions are better learnt then defined.

A la Brooks...

✤ Having a predefined representation of the problem (objective function) may not produce an intelligent solution

✤ It is better to use the real world (i.e. the real SE problem) as its own representational model

✤ Learner should be given a complete freedom over from which to learn the objective function

New Problem Instances

Turing Test

HistoricalBest Practice

All Observables

LearntObjective

TAO Challenge

TAO Challenge

✤ Software engineers moan about realistic empirical evaluation

TAO Challenge


✤ We interact with open source community very statically

TAO Challenge



✤ Why don’t we just commit our AI/SBSE/$#@% generated feature/bug patch/test case/*#$%@ directly to open source projects?

TAO Challenge



✤ Why don’t we just commit our AI/SBSE/$#@% generated feature/bug patch/test case/*#$%@ directly to open source projects?

✤ Will they notice? Will they condemn us, or welcome us?

Summary

✤ Some SE objective functions are merely surrogates

✤ These are better learnt then defined; humans are the final judges

✤ If we are to use AI techniques to solve SE problems, our goal should be Turing Test

tao: turing test as objective function

Technology