Slide 1

Human-Tool, Tool-Tool, and Human-Human Cooperations to Get the Job Done Tao Xie North Carolina State University Raleigh, NC, USA

Slide 2

Slide 3

IBM's Deep Blue defeated chess champion Garry Kasparov in 1997 IBM Watson defeated top human Jeopardy! players in 2011

Slide 4

"Completely Automated Public Turing test to tell Computers and Humans Apart"

Slide 5

Movie: Minority Report CNN News iPad

Slide 6

Slide 7

2010 Dagstuhl Seminar 10111 Practical Software Testing : Tool Automation and Human Factors http://www.dagstuhl.de/programm/kalender/semhp/?semnr=1011

Slide 8

2010 Dagstuhl Seminar 10111 Practical Software Testing : Tool Automation and Human Factors Human Factors http://www.dagstuhl.de/programm/kalender/semhp/?semnr=1011

Slide 9

9 Recent advanced technique: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Example tool: Pex from Microsoft Research (for.NET programs) Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: directed automated random testing. In Proc. PLDI 2005 Koushik Sen, Darko Marinov, and Gul Agha. CUTE: a concolic unit testing engine for C. In Proc. ESEC/FSE 2005 Nikolai Tillmann and Jonathan de Halleux. Pex - White Box Test Generation for.NET. In Proc. TAP 2008

Slide 10 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123} a==null a.Length>0 a[0]==123 T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition">

Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123} a==null a.Length>0 a[0]==123 T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

Slide 11

Method sequences MSeqGen/Seeker [Thummalapenta et al. OOSPLA 11, ESEC/FSE 09], Covana [Xiao et al. ICSE 2011], OCAT [Jaygarl et al. ISSTA 10], Evacon [Inkumsah et al. ASE 08], Symclat [d'Amorim et al. ASE 06] Environments e.g., db, file systems, network, DBApp Testing [Taneja et al. ESEC/FSE 11], [Pan et al. ASE 11] CloudApp Testing [Zhang et al. IEEE Soft 12] Loops Fitnex [Xie et al. DSN 09] Code evolution eXpress [Taneja et al. ISSTA 11] @NCSU ASE

Slide 12

Download counts (20 months) (Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388 http://research.microsoft.com/projects/pex/

Slide 13

http://pexase.codeplex.com/ Publications: http://research.microsoft.com/en-us/projects/pex/community.aspx#publicationshttp://research.microsoft.com/en-us/projects/pex/community.aspx#publications

Slide 14

Running Symbolic PathFinder... ===================================== ================= results no errors detected ===================================== ================= statistics elapsed time: 0:00:02 states: new=4, visited=0, backtracked=4, end=2 search: maxDepth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81MB loaded code: classes=71, methods=884 14

Slide 15

object-creation problems (OCP) - 65% external-method call problems (EMCP) 27% Total block coverage achieved is 50%, lowest coverage 16%. 15 Example: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Challenge: path explosion

Slide 16 0) { // B4 25: isComputed = true; 26: foreach (Edge e in graph.GetEdges()) { 27:... // B5 28: } 29: } } } 16 [Thummalapenta et al. OOPSLA 11]">

16 A graph example from QuickGraph library Includes two classes Graph DFSAlgorithm Graph AddVertex AddEdge: requires both vertices to be in graph 00: class Graph : IVEListGraph { 03: public void AddVertex (IVertex v) { 04: vertices.Add(v); // B1 } 06: public Edge AddEdge (IVertex v1, IVertex v2) { 07: if (!vertices.Contains(v1)) 08: throw new VNotFoundException(""); 09: // B2 10: if (!vertices.Contains(v2)) 11: throw new VNotFoundException(""); 12: // B3 14: Edge e = new Edge(v1, v2); 15: edges.Add(e); } } //DFS:DepthFirstSearch 18: class DFSAlgorithm { 23: public void Compute (IVertex s) {... 24: if (graph.GetEdges().Size() > 0) { // B4 25: isComputed = true; 26: foreach (Edge e in graph.GetEdges()) { 27:... // B5 28: } 29: } } } 16 [Thummalapenta et al. OOPSLA 11]

Slide 17 0) { // B4 25: isComputed = true; 26: foreach (Edge e in graph.GetEdges()) { 27:... // B5 28: } 29: } } } [Thummalapenta et al. OOPSLA 11]">

17 Test target: Cover true branch (B4) of Line 24 Desired object state: graph should include at least one edge Target sequence: Graph ag = new Graph(); Vertex v1 = new Vertex(0); Vertex v2 = new Vertex(1); ag.AddVertex(v1); ag.AddVertex(v2); ag.AddEdge(v1, v2); DFSAlgorithm algo = new DFSAlgorithm(ag); algo.Compute(v1); 17 00: class Graph : IVEListGraph { 03: public void AddVertex (IVertex v) { 04: vertices.Add(v); // B1 } 06: public Edge AddEdge (IVertex v1, IVertex v2) { 07: if (!vertices.Contains(v1)) 08: throw new VNotFoundException(""); 09: // B2 10: if (!vertices.Contains(v2)) 11: throw new VNotFoundException(""); 12: // B3 14: Edge e = new Edge(v1, v2); 15: edges.Add(e); } } //DFS:DepthFirstSearch 18: class DFSAlgorithm { 23: public void Compute (IVertex s) {... 24: if (graph.GetEdges().Size() > 0) { // B4 25: isComputed = true; 26: foreach (Edge e in graph.GetEdges()) { 27:... // B5 28: } 29: } } } [Thummalapenta et al. OOPSLA 11]

Slide 18

object-creation problems (OCP) - 65% external-method call problems (EMCP) 27% Total block coverage achieved is 50%, lowest coverage 16%. 18 Example: Dynamic Symbolic Execution/Concolic (Pex) Instrument code to explore feasible paths Challenge: path explosion

Slide 19

Example 1: File.Exists has data dependencies on program input Subsequent branch at Line 1 using the return value of File.Exists. Example 2: Path.GetFullPath has data dependencies on program input Path.GetFullPath throws exceptions. Example 3: String.Format do not cause any problem 19 1 2 3

Slide 20

Tackle object-creation problems with Factory Methods 20

Slide 21

Tackle external-method call problems with Mock Methods or Method Instrumentation Mocking System.IO.File.ReadAllText 21

Slide 22

Running Symbolic PathFinder... ===================================== ================= results no errors detected ===================================== ================= statistics elapsed time: 0:00:02 states: new=4, visited=0, backtracked=4, end=2 search: maxDepth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81MB loaded code: classes=71, methods=884 Tools Typically Dont Communicate Challenges Faced by Them to Enable Cooperation between Tools and Users 22

Slide 23

Machine is better at task set A Mechanical, tedious, repetitive tasks, Ex. solving constraints along a long path Human is better at task set B Intelligence, human intent, abstraction, domain knowledge, Ex. local reasoning after a loop, recognizing naming semantics = A U B 23

Slide 24

Human-Assisted Computing Driver: tool Helper: human Ex. Covana [Xiao et al. ICSE 2011] Human-Centric Computing Driver: human Helper: tool Ex. Coding duels @Pex for Fun Interfaces are important. Contents are important too! 24

Slide 25

Motivation Tools are often not powerful enough Human is good at some aspects that tools are not What difficulties does the tool face? How to communicate info to the user to get help? How does the user help the tool based on the info? 25 Iterations to form Feedback Loop

Slide 26

Motivation Tools are often not powerful enough Human is good at some aspects that tools are not What difficulties does the tool face? How to communicate info to the user to get help? How does the user help the tool based on the info? 26 Iterations to form Feedback Loop

Slide 27

external-method call problems (EMCP) object-creation problems (OCP) 27

Slide 28

Existing solution identify all executed external-method calls report all object types of program inputs and fields Limitations the number is often high some identified problem are irrelevant for achieving higher structural coverage 28

Slide 29

Real EMCPs: 0 Real OCPs: 5 Reported EMCPs: 44 Reported OCPs: 18 vs. 29

Slide 30

Goal: Precisely identify problems faced by tools when achieving structural coverage Insight: Partially-Covered Statements have data dependency on real problem candidates 30 [Xiao et al. ICSE 11] Xusheng Xiao, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Precise Identification of Problems for Structural Test Generation. In Proc. ICSE 2011

Slide 31

Data Dependence Analysis Forward Symbolic Execution Problem Candidates Problem Candidate Identification Runtime Information Identified Problems Coverage Program Generated Test Inputs Runtime Events 31

Slide 32

Data Dependencies 32 External-method calls whose arguments have data dependencies on program inputs

Slide 33

Symbolic Expression: return(File.Exists) == true Element of EMCP Candidate: return(File.Exists) Branch Statement Line 1 has data dependency on File.Exists at Line 1 33 Partially-covered branch statements have data dependencies on EMCP candidates for return values

Slide 34

Subjects: xUnit: unit testing framework for.NET 223 classes and interfaces with 11.4 KLOC QuickGraph: C# graph library 165 classes and interfaces with 8.3 KLOC Evaluation setup: Apply Pex to generate tests for program under test Feed the program and generated tests to Covana Compare existing solution and Covana 34

Slide 35

RQ1: How effective is Covana in identifying the two main types of problems, EMCPs and OCPs? RQ2: How effective is Covana in pruning irrelevant problem candidates of EMCPs and OCPs? 35

Slide 36

Covana identifies 43 EMCPs with only 1 false positive and 2 false negatives 155 OCPs with 20 false positives and 30 false negatives. 36

Slide 37

Covana prunes 97% (1567 in 1610) EMCP candidates with 1 false positive and 2 false negatives 66% (296 in 451) OCP candidates with 20 false positives and 30 false negatives 37

Slide 38

Human-Assisted Computing Driver: tool Helper: human Ex. Covana [Xiao et al. ICSE 2011] Human-Centric Computing Driver: human Helper: tool Ex. Coding duels @Pex for Fun Interfaces are important. Contents are important too! 38

Slide 39

1,126,136 clicked 'Ask Pex!' www.pexforfun.com The contributed concept of Coding Duel games as major game type of Pex for Fun since Summer 2010 39 N. Tillmann, J. De Halleux, T. Xie, S. Gulwani and J. Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In Proc. ICSE 2013, Software Engineering Education (SEE), 2013.

Slide 40

Secret Implementation class Secret { public static int Puzzle(int x) { if (x