efficient instrumentation for code coverage testing
DESCRIPTION
Efficient Instrumentation for Code Coverage Testing. Mustafa M. Tikir Jeffrey K. Hollingsworth. Evaluation of Code Coverage. Measuring code coverage is important to Identify unexecuted program statements Verify that each path is taken at least once Requires extensive instrumentation - PowerPoint PPT PresentationTRANSCRIPT
University of Maryland
Efficient Instrumentation forCode Coverage Testing
Mustafa M. TikirJeffrey K. Hollingsworth
University of Maryland
Evaluation of Code Coverage
Measuring code coverage is important to– Identify unexecuted program statements– Verify that each path is taken at least once
Requires extensive instrumentation– Must determine if every statement is
executed
University of Maryland
Instrumentation for Code Coverage
Traditional Approach– Static instrumentation using counters – Instrumentation code remains for entire
execution– Conservative instrumentation of all possibly
needed instrumentation code for all functions
Useless instrumentation wastes time– especially for long running programs such as
servers and enterprise software
Full instrumentation increases setup time
University of Maryland
Our Approach
Insert instrumentation code dynamically– Pre-Instrument all functions at program start– On-Demand instrument at first function call
Periodically remove instrumentation Use dominator trees
– Reduces instrumentation within a function
Use incremental function instrumentation– Insertion of instrumentation on first call– Reduced which function gets instrumented
University of Maryland
Dyninst API
Modify code in a running program Implementations available for
– Alpha,Sparc,Power,Mips and x86 architectures A mutator program
– Generates machine code from high-level code– Transfers machine code to a running mutatee program
Mutatee is the application being instrumented
Base-trampoline contains
– relocated instructions– slots for calling code
Mini-trampoline stores the inserted code
Program
Pre
Post
Relocated Instruction
Base Tramp Mini Tramp
Save Registers
Setup Args
Code SnippetRestore
Registers
University of Maryland
Using Dominator Trees
Definitions:– A dom B if all paths from entry to basic
block B goes through basic block A– A idom B if, for all C, (C != A) and (C dom
B) implies (C dom A)
Fact:– If a basic block, B, is executed
• all basic blocks along the path from B to the root of dominator tree also execute
C
B
A
University of Maryland
Leaf Node Instrumentation
Leaf nodes in dominator tree are instrumented– Coverage of internal nodes will be inferred
Coverage information propagated from leaf nodes to entry
Dominator TreeControl Flow Graph
Entry
13
25
4
Exit
4
2
1
3
Exit
5
Entry
University of Maryland
Non-Leaf Node Instrumentation Leaf node instrumentation is necessary but not sufficient
– Control flow might cause cross edges in dominator tree
Control Flow Graph
Entry
1
32
Exit
Dominator Tree
Entry
2 1
3
We also instrument basic block, A, if A has at least one outgoing edge to basic block, B,
and A does not dominate B
Exit
University of Maryland
Code Coverage Algorithm
At Start– Create CFG and dominator
trees for all functions– Instrument the basic blocks
selected
During Execution– Stop the execution at fixed
time intervals– Delete the executed
instrumentation code
At Termination– Propagate and collect the
coverage information
At Start– Insert breakpoints at each
function entry During Execution
– On breakpoint• Identify the function• Create CFG and
dominator tree• Instrument the basic
blocks selected– At fixed time intervals
delete the executed code At Termination
– Propagate and collect the coverage information
Pre-Instrumentation On-Demand Instrumentation
University of Maryland
Reduction In Instrumentation Points
34-49% with pre-instrumentation 42-79% with on-demand instrumentation
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Ratio
all blocks all blocks OD dominator dominator OD
University of Maryland
SPEC/compress Coverage Curve
Covers 76% in the first 18% of the execution– Most of the basic blocks that will execute are covered
at the beginning of the program– Rest of the run is re-executions
0
10
20
3040
50
60
70
80
0 20 40 60 80 100 120 140 160 180 200Time Interval(sec)
Per
cent
age(
sour
ce li
ne)
University of Maryland
SPEC/compress Execution Time
Purecov is a commercial state of art code coverage tool Our code coverage tools outperform purecov Significant reduction when dynamic code deletion is enabled
– Most of the instrumentation code is deleted at the beginning– Setup time and deletion overhead is insignificant
• A few number of basic blocks to instrument and check
0
1
2
3
4
5
6
7
8
9
10
0 1 2 5 10
15
25
50 0 1 2 5 10
15
25
50 0 1 2 5 10
15
25
50 0 1 2 5 10
15
25
50
PC all blocks all blocks OD dominator dominator OD
Slo
wdow
n R
atio
University of Maryland
PostgreSQL Coverage Curve
Wisconsin benchmark queries– Measure the performance of database systems
• Executes select/join queries repeatedly
0
5
10
15
20
0 10 20 30 40 50 60 70 80Time Interval(sec)
Perc
enta
ge(s
ourc
e lin
e)
University of Maryland
PostgreSQL Execution Time
Using on-demand instrumentation– Our coverage tools outperform purecov almost always
On-demand instrumentation outperforms pre-instrumentation
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0 1 2 5
10
15
25
50 0 1 2 5
10
15
25
50 0 1 2 5
10
15
25
50 0 1 2 5
10
15
25
50
PC all blocks all blocks OD dominator dominator OD
Slo
wd
ow
n R
atio
Execution Time Setup, Instrumentation and Deletion Time
University of Maryland
Instrumentation Execution Frequency
Overhead of our dynamic code coverage system is bursty Running previously unexecuted code results in the execution
of a significant amount of instrumentation Running previously executed code does not result in the
execution of any instrumentation
PostgreSQL with Wisconsin Benchmark
0
5
10
15
20
25
0 10 20 30 40 50 60 70 80Time(sec)
Covera
ge P
erc
enta
ge
0
1
2
3
4
5
6
7
Execution F
requency
(log10)
University of Maryland
Overall Slowdown Results for our code coverage tool are for 2-second deletion Slowdown using purecov ranges from 1.83 to 19.78 Slowdown using our code coverage tools range
– From 1.002 to 2.6 for on-demand instrumentation– From 1.002 to 4.96 for pre-instrumentation
0
4
8
12
16
20
Slo
wdow
n R
atio
Dominator On-Demand Purecov
University of Maryland
Dyninst Coverage Tool
University of Maryland
Conclusions
Dominator trees– Reduce instrumentation points by 34-49%– Plus on-demand instrumentation reduce
instrumentation points by 42-79%
Combining dominator trees and on-demand instrumentation reduces– Setup time and deletion interval overhead– Runtime overhead by 38-90% compared to
purecov
Dynamic deletion of instrumentation– Computes coverage information faster
University of Maryland
Conclusions (cont)
Code Coverage overhead reduced to about 10%
Code coverage can now be included as part of production code– Information about the execution of
extremely infrequent error cases will be provided
– Reduced overhead for residual testing
Dyninst library + Dyninst Coverage tools – http://www.dyninst.org