oliver krauss msc dynamic fitness functions for genetic ... · genetic improvement in compilers and...

Oliver Krauss MSc

Dynamic Fitness Functions forGenetic Improvement in Compilersand InterpretersKyoto 16.7.2018

Advisors: Prof. Dr. Dr. h.c. Hanspeter Mössenböck,Prof. (FH) Priv.-Doz. Dipl.-Ing. Dr. Michael Affenzeller

AbstractGenetic Improvement on Interpreter / Compiler AbstractSyntax Tree (AST)

– Improve non-functional software features (performance)– Deal with large fitness landscapes– Dynamic fitness functions ...

– split test suites by the complexity of the cases– Sequential or Parallel

– ... don't perform well– ... show promise

Page 1 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 2 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 3 | 26

Search Space Size

∗

∗ ∗

∗ ∗

∗ ∗

depth

width

– n node types (MiniC -> 158)– depth 5, width 5

– All Options -> n3901

– MiniC -> 466.638– depth 10, width 10

– All Options -> n11.111.111.101

– MiniC -> 80.814.253.363

Page 4 | 26

Search Space Size

∗

∗ ∗

∗ ∗

∗ ∗

depth

width

– n node types (MiniC -> 158)– depth 5, width 5

– All Options -> n3901

– MiniC -> 466.638

– depth 10, width 10– All Options -> n11.111.111.101

– MiniC -> 80.814.253.363

Page 4 | 26

Search Space Size

∗

∗ ∗

∗ ∗

∗ ∗

depth

width

– n node types (MiniC -> 158)– depth 5, width 5

– All Options -> n3901

– MiniC -> 466.638– depth 10, width 10

– All Options -> n11.111.111.101

– MiniC -> 80.814.253.363

Page 4 | 26

Search Space Size

∗

∗ ∗

∗ ∗

∗ ∗

depth

width

– n node types (MiniC -> 158)– depth 5, width 5

– All Options -> n3901

– MiniC -> 466.638– depth 10, width 10

– All Options -> n11.111.111.101

– MiniC -> 80.814.253.363

Page 4 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 5 | 26

JIT Compilation with GI ICode

void main() {int i,

n,now ,prev ,next;

prev = 0;now = 1;i = 0;n = 30;while (i

< n){if (i

==0 )print (0);

elseif(i==1)print (1);

else {next

=now+prev;

prev=now;

now=next;

print(next);}i = i

+1;

}}

GI Truffle Graal

Coco/R Optimize Compile

Figure: Code Interpretation and Compilation with GI (Truffle and Graalimages from [1])

Page 6 | 26

JIT Compilation with GI II

MiniC– Subset of C11 [2]– Coco/R -> Creates Scanner & Parser in Java– 342 node classes (103 operators, 55 operands)

Page 7 | 26

JIT Compilation with GI IIITruffle [3, 1]

– Interpreter based on AST– Generalized and Spezialized AST - Nodes– Self optimizing by AST-Rewriting– Can run any Guest Language on the JVM– Python, Ruby, JavaScript, ...

Page 8 | 26

JIT Compilation with GI IV

Graal [4, 5, 6]

– Just-in-time (JIT) compiler– Agressive optimizations– Part of OpenJDK

Page 9 | 26

JIT Compilation with GI VAST from Coco/R Select subtree

for Optimization Run in TruffleCollect TestdataIN OUT

0 01 12 13 24 35 5

Create Stub

GI Population Evaluate Populationwith Truffle Hand best solution

to Truffle

clone & build

Profile Get Testdata clone AST

evaluate

cross & mutate

after x Generations

Figure: GP Optimization in depthPage 10 | 26

Architectural Overview

Figure: The architecture of the optimization framework in combinationwith Truffle and Graal

Page 11 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 12 | 26

Node Visitationfib

{}read

n if ret

ret||

==

n 1

==

0nn

+

fib

−

n 1

fib

−

n 2

Visited Previously VisitedSpezialisation

Goal: visit all nodes– n = 0 [v=14,s=0]

– n = 1 [v=11,s=0]– n = 2 [v=23,s=6]

Page 13 | 26

Node Visitationfib

{}read

n if ret

ret||

==

n 1

==

0nn

+

fib

−

n 1

fib

−

n 2

Visited Previously VisitedSpezialisation

Goal: visit all nodes– n = 0 [v=14,s=0]– n = 1 [v=11,s=0]

– n = 2 [v=23,s=6]

Page 13 | 26

Node Visitationfib

{}read

n if ret

ret||

==

n 1

==

0nn

+

fib

−

n 1

fib

−

n 2

Visited Previously VisitedSpezialisation

Goal: visit all nodes– n = 0 [v=14,s=0]– n = 1 [v=11,s=0]– n = 2 [v=23,s=6]

Page 13 | 26

Test Metrics ICorrectness - Percentage of successful tests

correctness(AST,tests) =∑tests

n=1 succeeded(AST,test)∑tests (1)

Complexity - Percentage of nodes visited by a testcomplexity(AST,test) =

∑nodesn=1 visited(AST,test)∑

nodes(AST) (2)

Page 14 | 26

Test Metrics IIOverlap - Percentage of nodes visited by two tests

overlap(A,B) =

∑nodesn=1 visited(A) ∩ visited(B)

max(∑nodes

n=1 visited(A),∑nodes

n=1 visited(B))(3)

Confidence - Percentage of nodes visited by test suiteconfidence(testsuite,AST) =

∑nodesn=1 visited(testsuite)∑

nodes(AST) (4)

Page 15 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 16 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Sequential Fitness Function

Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

Parallel Fitness Function

Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached

Page 18 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 19 | 26

Test SetUpGA vs. Sequential FFN vs. Parallel FFN

– Same configuration for all– No grafting– Comparison of solved test cases

Table: Use Case statisticsUse Case No. Nodes Tests Nodes visitedx*2 8 5 5, 5, 5, 5, 5Sqrt 60 5 14, 42, 42, 42, 53Fibonacci 32 5 11, 15, 30, 30, 30Constructed 31 5 11, 15, 20, 20, 21

Page 20 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)Constructed 3 of 5 tests

solved

Fitness Functions

GA BaselineSequential FFN Solved more

testsParallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2

Fibonacci only first 2(Sequential FFN)

Constructed 3 of 5 testssolved

Fitness Functions

GA BaselineSequential FFN Solved more

testsParallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)

Constructed 3 of 5 testssolved

Fitness Functions

GA BaselineSequential FFN Solved more

testsParallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)Constructed 3 of 5 tests

solved

Fitness Functions

GA BaselineSequential FFN Solved more

testsParallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)Constructed 3 of 5 tests

solved

Fitness FunctionsGA Baseline

Sequential FFN Solved moretests

Parallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)Constructed 3 of 5 tests

solved

Fitness FunctionsGA Baseline

Sequential FFN Solved moretests

Parallel FFN Total failure

Page 21 | 26

ResultsUseCases

x*2 always solved,except ParallelFFN

Sqrt return 2Fibonacci only first 2

(Sequential FFN)Constructed 3 of 5 tests

solved

Fitness FunctionsGA Baseline

Sequential FFN Solved moretests

Parallel FFN Total failure

Page 21 | 26

OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook

Page 22 | 26

Discussion IConcepts show promise

– Sequential FFN outperformed GA– Known why all solutions failed– Can be improved

Stuck in local optima– Sequential - solutions dominate from previous

– Additive mutator– Parallel - one previous repeat dominates

– Merge crossover

Page 23 | 26

Discussion IICorrectness measurement is binary 0|1

– Increase amount of tests– Switch correctness measure from 0|1 to 0..1

Search Space Size may be too big– Use indicators from original AST

– Operators / Operands– Types

– Restrict size with loop/branch counting

Page 24 | 26

Outlook

– Sequential FFN -> Age Layered Population Structure (ALPS)– Parallel FFN -> Island GA– Implement Mutator & Crossover Operators– Update FFN to also include hot path– Your Suggestion Here

Page 25 | 26

Contact

Oliver Kraussoliver.krauss@fh-hagenberg.at

+43 (0)50804-27195

Page 26 | 26

Bibliography I[1] T. Würthinger, C. Wimmer, A. Wöß, L. Stadler, G. Duboscq, C. Humer,

G. Richards, D. Simon, and M. Wolczko, “One vm to rule them all”, inProceedings of the 2013 ACM International Symposium on New Ideas, NewParadigms, and Reflections on Programming & Software, ser. Onward! 2013,Indianapolis, Indiana, USA: ACM, 2013, pp. 187–204. [Online]. Available:http://doi.acm.org/10.1145/2509578.2509581.

[2] ISO, ISO/IEC 9899:2011 Information technology — Programming languages— C. Geneva, Switzerland: International Organization for Standardization,Dec. 2011, 683 (est.) [Online]. Available: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=57853.

Page 1 | 7

Bibliography II[3] C. Wimmer and T. Würthinger, “Truffle: A self-optimizing runtime system”,

in Proceedings of the 3rd Annual Conference on Systems, Programming,and Applications: Software for Humanity, ser. SPLASH ’12, Tucson, Arizona,USA: ACM, 2012, pp. 13–14. [Online]. Available:http://doi.acm.org/10.1145/2384716.2384723.

[4] OpenJDK. (2018), Graal project, Last Accessed - 2018-04-23, [Online].Available: http://openjdk.java.net/projects/graal/.

[5] D. Simon, C. Wimmer, B. Urban, G. Duboscq, L. Stadler, and T. Würthinger,“Snippets: Taking the high road to a low level”, ACM Trans. Archit. CodeOptim., vol. 12, no. 2, 20:20:1–20:20:25, Jun. 2015. [Online]. Available:http://doi.acm.org/10.1145/2764907.

Page 2 | 7

Bibliography III[6] L. Stadler, G. Duboscq, H. Mössenböck, and T. Würthinger, “Compilation

queuing and graph caching for dynamic compilers”, in Proceedings of theSixth ACM Workshop on Virtual Machines and Intermediate Languages,ser. VMIL ’12, Tucson, Arizona, USA: ACM, 2012, pp. 49–58. [Online].Available: http://doi.acm.org/10.1145/2414740.2414750.

[7] R. Olsen and B. Ostman, Using fitness landscapes to visualize evolution inaction, https://www.youtube.com/watch?v=4pdiAneMMhU, [LastAccessed 14.07.2018], 2014.

Page 3 | 7

Appendix A - Run ConfigurationCrossover A single point crossover was selected

Mutator A single point mutator was selected. The point to bemutated is selected randomly by the mutator.

Mutation Probability was set to 57%Selector crossover and mutator use a tournament selection

PopulationSize was set to 200 individualsGenerations each run was 20 generations. GA, and the sequential

FFN took 80 generations over 4 restarts. Parallel FFNtook 60 generations initially and 20 generations afterrestart = 80 generations.

Elitism per generation the 3 best individuals were preservedRestart-Elitism 10 best individuals. Parallel FFN had 30 individuals

carried over (10 from each group).Operators and Operands all AST nodes of the MiniC language were

selected as operators and operands for each test case.Page 4 | 7

Appendix B - Constructed Use CaseListing 1: Constructed Use Case

int constructed(int n) {if (n == 0) {

return 1;}if (n == 1) {

return 2;}if (n == 2) {

return n + 1;}return n * n;

}

Page 5 | 7

Appendix C - Threats to Validity

Performance Measurements - Graal Specific– Runs in same JVM influence each following run– 200.000 runs for measurement due to warm up phase

Page 6 | 7

Appendix D - Fitness Landscape Vi-sualization

Figure: Regular Static FitnessLandscape [7]

Figure: Test-based FitnessLandscape

Page 7 | 7