dynamic analysis - cs.sfu.cawsumner/teaching/745/08-dynamic-analysis.pdf · dynamic binary...

Dynamic Analysis

CMPT 745Software Engineering

Nick [email protected]

2

Dynamic Analysis

● Sometimes we want to study or adapt the behavior of executions of a program

3

Dynamic Analysis


– Did my program ever …?

4

Dynamic Analysis


– Did my program ever …?– Why/how did … happen?

5

Dynamic Analysis


– Did my program ever …?– Why/how did … happen?– Where am I spending time?

6

Dynamic Analysis


– Did my program ever …?– Why/how did … happen?– Where am I spending time?– Where might I parallelize?

7

Dynamic Analysis


– Did my program ever …?– Why/how did … happen?– Where am I spending time?– Where might I parallelize?– Tolerate errors

8

Dynamic Analysis


– Did my program ever …?– Why/how did … happen?– Where am I spending time?– Where might I parallelize?– Tolerate errors– Manage memory / resources.

9

e.g. Reverse Engineering

Static CFG (from e.g. Apple Fairplay):

This is the result of a control flow flattening obfuscaton.[http://tigress.cs.arizona.edu/transformPage/docs/flatten/]

http://tigress.cs.arizona.edu/transformPage/docs/flatten/

10

e.g. Reverse Engineering

Static CFG (from e.g. Apple Fairplay):

Dynamically Simplified CFG:

This is the result of a control flow flattening obfuscaton.[http://tigress.cs.arizona.edu/transformPage/docs/flatten/]

http://tigress.cs.arizona.edu/transformPage/docs/flatten/

11

How?

● Can record the execution

12

How?

● Can record the execution– Record to a trace– Analyze post mortem / offline– Scalability issues: need enough space to store it

13

How?


● Can perform analysis online

14

How?


● Can perform analysis online– Instrument the program– Modified program invokes code to 'analyze' itself

15

How?



● Can do both!– Lightweight recording– Instrument a replayed instance of the execution

16

How?



● Can do both!– Lightweight recording– Instrument a replayed instance of the execution

Some analyses only make sense online.Why?

17

Simple Idea: Basic Block Profiling

Knowing where we are spending time is useful:

● Goal: Which basic blocks execute most frequently?

18



● Goal: Which basic blocks execute most frequently?

Profiling is a common dynamic analysis!

19



● Goal: Which basic blocks execute most frequently?● How can we modify our program to find this?

20




BB:0

BB:1 BB:2

BB:3

?

21




BB:0

BB:1 BB:2

BB:3

count[2] += 1

x = foo()y = bar()...

22




BB:0

BB:1 BB:2

BB:3

count[2] += 1

x = foo()y = bar()...

for i in BBs: count[i] = 0

for i in BBs: print(count[i])Start: End:

23




BB:0

BB:1 BB:2

BB:3

count[2] += 1

x = foo()y = bar()...

for i in BBs: count[i] = 0

for i in BBs: print(count[i])Start: End:

24


● Big concern: How efficient is it?– The more overhead added, the less practical the tool

25



count[0] += 1…

count[1] += 1…

count[5] += 1…

count[6] += 1…

count[4] += 1…

count[2] += 1…

count[3] += 1…

26



– Can we do better?

count[0] += 1…

count[1] += 1…

count[5] += 1…

count[6] += 1…

count[4] += 1…

count[2] += 1…

count[3] += 1…

27




count[0] += 1…

count[1] += 1…

count[5] += 1…

count[6] += 1…

count[4] += 1…

count[2] += 1…

count[3] += 1…

count[1] = count[4] = count[2] + count[3]

28




count[0] += 1…

count[1] += 1…

count[5] += 1…

count[6] += 1…

count[4] += 1…

count[2] += 1…

count[3] += 1…

count[1] = count[4] = count[2] + count[3]count[0] = count[6] = count[1] + count[5]

29




count[0] += 1…

count[1] += 1…

count[5] += 1…

count[6] += 1…

count[4] += 1…

count[2] += 1…

count[3] += 1…

count[1] = count[4] = count[2] + count[3]count[0] = count[6] = count[1] + count[5]

30

Efficiency Tactics

● Abstraction

31

Efficiency Tactics

● Abstraction

● Identify & avoid redundant information

32

Efficiency Tactics

● Abstraction


● Sampling

33

Efficiency Tactics

● Abstraction


● Sampling

● Compression / encoding

34

Efficiency Tactics

● Abstraction


● Sampling


● Profile guided instrumentation

35

Efficiency Tactics

● Abstraction


● Sampling



● Thread local analysis

36

Efficiency Tactics

● Abstraction


● Sampling



● Thread local analysis

● Inference

37

How / When to Instrument

● Source / IR Instrumentation– LLVM, CIL, Soot, Wala– During (re)compilation– Requires an analysis dedicated build

38



● Static Binary Rewriting– Diablo, DynamoRIO, SecondWrite, – Applies to arbitrary binaries– Imprecise IR info, but more complete binary behavior

39



● Static Binary Rewriting– Diablo, DynamoRIO, SecondWrite, – Applies to arbitrary binaries– Imprecise IR info, but more complete binary behavior

● Dynamic Binary Instrumentation– Valgrind, Pin, Qemu (& other Vms)– Can adapt at runtime, but less info than IR

40

Phases of Dynamic Analysis

In general, 2-3 phases occur:

1) Instrumentation– Add code to the program for data collection/analysis

41




2) Execution– Run the program an analyze its actual behavior

42





3) (Optional) Postmortem Analysis– Perform any analysis that can be deferred after termination

43





3) (Optional) Postmortem Analysis– Perform any analysis that can be deferred after termination

Very, very common mistake to mix 1 & 2.

44

Static Instrumentation

1) Compile whole program to IR

foo.cbar.cbaz.c prog.ll

45



2) Instrument / add code directly to the IR

foo.cbar.cbaz.c prog.ll prog’.ll

46




3) Generate new program that performs analysis


47




3) Generate new program that performs analysis

4) Execute


Test Cases

Results

48

Dynamic Binary Instrumentation (DBI)

1) Compile program as usual

2) Run program under analysis framework

(Valgrind, PIN, Qemu, etc)

3) Instrument & execute in same command:– Fetch & instrument each basic block individually– Execute each basic block

valgrind --tool=memcheck ./myBuggyProgram

Example: Test Case Reduction

50

Testing and Dynamic Analysis

● In some cases, just running a program with different inputs is enough

51


● In some cases, just running a program with different inputs is enough– Carefully selected inputs can target the analysis– The result of running the program reveals coarse information about its

behavior

52



behavior

● Intuitively, even just testing is a dynamic analysis– It requires no transformation– The result is just the success or failure of tests

53



behavior

● Intuitively, even just testing is a dynamic analysis– It requires no transformation– The result is just the success or failure of tests

● But even that is interesting to consider....

54

Bug reports are problematic

● Failing inputs can be large and complex

a r h w l n y e u m g k o w h > ` p

MB? GB?

55




MB? GB? What is relevant and essential to the bug?

56




a b cBug 2

a b cBug 3

a b cBug 1

57




a b cBug 2

a b cBug 3

a b cBug 1

1) Are these reports the same bug?2) Can we make it easier to reproduce?

58




a b cBug 2

a b cBug 3

a b cBug 1

a b cBug

59




a b cBug 2

a b cBug 3

a b cBug 1

a b cBug

1) Same? Yes!2) Easier? Yes! And easier to understand!

60




a b cBug 2

a b cBug 3

a b cBug 1

a b cBug

Test Case Reduction: finding smaller test cases that reproduce a failure

61

Classically – Delta Debugging

<SELECT NAME="priority" MULTIPLE SIZE=7>

http://en.wikipedia.org/wiki/File:Netscape_2_logo.gif

print

62


<SELECT NAME="priority" MULTIPLE SIZE=7>

http://en.wikipedia.org/wiki/File:Netscape_2_logo.gif

print

63


<SELECT NAME="priority" MULTIPLE SIZE=7>Intuition: trial and error

64


<SELECT NAME="priority" MULTIPLE SIZE=7> = cIntuition: trial and error1) Start w/ a failing text configuration c

65


<SELECT NAME="priority" MULTIPLE SIZE=7>Intuition: trial and error1) Start w/ a failing text configuration c2) Try removing subsets (Δ) of input elements ({δ}})

66


<SELECT NAME="priority" MULTIPLE SIZE=7>Intuition: trial and error1) Start w/ a failing text configuration c2) Try removing subsets (Δ) of input elements ({δ}})3) Failure still exists → new input is “better”

67


<SELECT NAME="priority" MULTIPLE SIZE=7>Intuition: trial and error1) Start w/ a failing text configuration c2) Try removing subsets (Δ) of input elements ({δ}})3) Failure still exists → new input is “better”4) Repeat on the new input

68



When do we stop? / What is our goal?● Global Minimum: c : ∀ |c'|<|c|, c'

69




Smallest subset of the originalinput reproducing the failure

70




Smallest subset of the originalinput reproducing the failure

Completely impractical! Why?

71



When do we stop? / What is our goal?● Global Minimum: c : ∀ |c'|<|c|, c'● Local Minimum: c : ∀ c'⊂c, c'

72




No subset of the result canreproduce the failure.

73




No subset of the result canreproduce the failure.

How does this differ from a global minimum?Is it still problematic?

74



When do we stop? / What is our goal?● Global Minimum: c : ∀ |c'|<|c|, c'● Local Minimum: c : ∀ c'⊂c, c'● 1-Minimal: c: ∀ δ ∈ c, (c-{δ})

75



When do we stop? / What is our goal?● Global Minimum: c : ∀ |c'|<|c|, c'● Local Minimum: c : ∀ c'⊂c, c'● 1-Minimal: c: ∀ δ ∈ c, (c-{δ})

No one element can be removedand still reproduce the failure

76

Classically – Delta Debugging1 2 3 4 5 6 7 8

Does binary search work?

77

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 8

78

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

So what should we do?

79

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

So what should we do?

We refine the granularity

80

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

81

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

82

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

83

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

84

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

And now check complements

85

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

86

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 8

87

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 8

88

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 8

89

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 8

What's clever about how we recurse?

90

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 8

91

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 7 81 2 7 8 So close! How many more?

92

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 7 81 2 7 8

1 2 7 81 2 7 81 2 7 81 2 7 81 2 7 81 2 7 8

Done?

93

Classically – Delta Debugging1 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 3 4 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 5 6 7 81 2 7 81 2 7 8

1 2 7 81 2 7 81 2 7 81 2 7 81 2 7 81 2 7 81 7 81 7 81 7 81 7 81 7 81 7 8

Done?

94


1) Test case to minimize

1 2 3 4 5 6 7 8c =

ddmin(c) = ddmin2(c, 2)

95


1 2 3 4 5 6 7 8c =

1) Test case to minimize2) Granularity (|Δ|=|c|/n)


96



1 2 3 4 5 6 7 8c = Δ = 4

Δ1 Δ2 Δ3 Δ4


97



1 2 3 4 5 6 7 8c = Δ = 4

Δ1 Δ2 Δ3 Δ4

∇1


98



ddmin2(c, 2)=


99


ddmin(c) = ddmin2(c, 2) 1) Test case to minimize2) Granularity (|Δ|=|c|/n)

ddmin2(c, 2)=

ddmin2(Δi, 2) If ... (a)

Δi = {3,4} (a) 1 2 3 4 5 6 7 8

100



ddmin2(c, 2)=

ddmin2(Δi, 2)

ddmin2(∇i, max(n-1,2))

1 2 3 4 5 6 7 8

If ... (a)If ... (b)

Δi = {3,4} (a)(b)

1 2 3 4 5 6 7 8

101



ddmin2(c, 2)=

ddmin2(Δi, 2)

ddmin2(∇i, max(n-1,2))

1 2 3 4 5 6 7 8

If ... (a)If ... (b)

Δi = {3,4} (a)(b)

1 2 3 4 5 6 7 8

102



ddmin2(c, 2)=

ddmin2(Δi, 2)

ddmin2(∇i, max(n-1,2))ddmin2(c, min(|c|,2n))

1 2 3 4 5 6 7 8

If ... (a)If ... (b)If ... (c)

Δi = {3,4} (a)(b)(c) n < |c|

1 2 3 4 5 6 7 8

103



ddmin2(c, 2)=

ddmin2(Δi, 2)

ddmin2(∇i, max(n-1,2))ddmin2(c, min(|c|,2n))

1 2 3 4 5 6 7 8

If ... (a)If ... (b)If ... (c)

Δi = {3,4} (a)(b)(c) n < |c|

1 2 3 4 5 6 7 8

104



ddmin2(c, 2)=

ddmin2(Δi, 2)

ddmin2(∇i, max(n-1,2))ddmin2(c, min(|c|,2n))c

1 2 3 4 5 6 7 8

If ... (a)If ... (b)If ... (c)otherwise

Δi = {3,4} (a)(b)(c) n < |c|

1 2 3 4 5 6 7 8

105


● Worst Case: |c|2 + 3|c| tests– All tests unresolved until maximum granularity– Testing complement succeeds

106



● Best Case: # tests ≤ 2log2(|c|)– Falling back to binary search!

107




● Minimality– When will it only be locally minimal?– When will it only be 1-minimal?

108




● Minimality– When will it only be locally minimal?– When will it only be 1-minimal? – Does formal minimality even matter?

109


● Observation:In practice DD may revisit elements in order to guarantee minimality

110



ddmin2(∇i, max(n-1,2))1 2 3 4 5 6 7 8

111



1 2 3 4 5 6 7 8

1 2 5 6 7 8

...ddmin2(∇i, max(n-1,2))

1 2 3 4 5 6 7 8

112



● If guaranteeing 1-minimality does not matter the algorithm can drop to linear time!– In practice this can be effective for what developers may care about

1 2 3 4 5 6 7 8

1 2 5 6 7 8


1 2 3 4 5 6 7 8

113



● If guaranteeing 1-minimality does not matter the algorithm can drop to linear time!– In practice this can be effective for what developers may care about

1 2 3 4 5 6 7 8

1 2 5 6 7 8


1 2 3 4 5 6 7 8

Don’t get bogged down by formalismwhen it doesn’t serve you!

114

Test Case Reduction in Practice

● Most problems do not use DD directly for TCR.– It provides guidance, but frequently behaves poorly

115



● What are the possible causes of problems?

1 2 3 41 2 3 41 2 3 4

Monotonicitymatters

116




1 2 3 41 2 3 41 2 3 4

Monotonicitymatters

1 2 3 41 2 3 4

Determinismmatters

117




1 2 3 41 2 3 41 2 3 4

Monotonicitymatters

1 2 3 41 2 3 4

Determinismmatters Structure

matters

for

i range =

5 10 [] *

a i i 5

118

Test Case Reduction for Compilers

● Programs are highly structured, so TCR for compilers faces challenges

119



● What structures could we use to guide the process?

120




for

i range =

5 10 [] *

a i i 5

121




● What challenges still remain?

for

i range =

5 10 [] *

a i i 5

Example: Memory Safety Bugs

123

Example: Finding memory safety bugs

● Memory safety bugs are one of the most common sources ofsecurity vulnerabilities

124



● Effects may only be visible long after invalid behavior– This complicates comprehension & debugging

125




● Two main types of issues:– Spatial – Out of bounds stack/heap/global accesses– Temporal – Use after free

126




● Two main types of issues:– Spatial – Out of bounds stack/heap/global accesses– Temporal – Use after free

● We would like to automatically identify & provide assistancewith high precision and low overhead– Suitable for testing & sometimes maybe deployment!

127


● Most common approach – track which regions of memory are valid– During execution!– Updated when new memory is allocated– Checked when pointers are accessed– With low overhead

128



● Common implementations– Valgrind – DBI based– AddressSanitizer – static instrumentation based

129



● Common implementations– Valgrind – DBI based– AddressSanitizer – static instrumentation based

Note, the implementation style affects which bugs can be recognized!

Why?

130

AddressSanitizer

● Need to track which memory is valid & check efficiently...

● Big Picture:– Replace calls to malloc & free

131

AddressSanitizer


● Big Picture:– Replace calls to malloc & free– Poison memory: (create red zones)

132

AddressSanitizer



1) around malloced chunks

ptr = malloc(sizeof(MyStruct));

ptr

133

AddressSanitizer



1) around malloced chunks2) when it is freed

free(ptr);

ptr

134

AddressSanitizer



1) around malloced chunks2) when it is freed3) around buffers and local variables

void foo() { int buffer[5]; ...}

buffer[0]

buffer[6]

135

AddressSanitizer




– Access of poisoned memory causes an error

136

AddressSanitizer





*address = ...;

instrumentation ?

137

AddressSanitizer





*address = ...; If (IsPoisoned(address, size)) { ReportError(address, size, isWrite);}*address = ...

instrumentation

138

AddressSanitizer





● The tricky part is tracking & efficiently checking redzones.

139

AddressSanitizer





● The tricky part is tracking & efficiently checking redzones.– Instrumenting every memory access is costly!

140

AddressSanitizer





● The tricky part is tracking & efficiently checking redzones.– Instrumenting every memory access is costly!– We must track all memory ... inside that same memory!

141

AddressSanitizer





● The tricky part is tracking & efficiently checking redzones.– Instrumenting every memory access is costly!– We must track all memory ... inside that same memory!

This kind of issue is common in dynamic analyses.

142

AddressSanitizer – Shadow Memory

Application Memory

Need to know whether any byte of application memory is poisoned.

143


Application Memory Shadow Memory

● We maintain 2 views on memory

144




Shadow memory of theshadow memory!

145




● Shadow Memory is pervasive in dynamic analysis– Can be thought of as a map containing analysis data

146




● Shadow Memory is pervasive in dynamic analysis– Can be thought of as a map containing analysis data– For every bit/byte/word/chunk/allocation/page,

maintain information in a compact table

147





maintain information in a compact table

Where have you encountered this before?

148





maintain information in a compact table– Common in runtime support, e.g. page tables

149


● Designing efficient analyses (& shadow memory) often requires a careful domain insight

Encoding & abstraction efficiency strategies

150



● NOTE: Heap allocated regions are N byte aligned (N usually 8)

151



● NOTE: Heap allocated regions are N byte aligned (N usually 8)– In an N byte region, only the first k may be addressable

k

152



● NOTE: Heap allocated regions are N byte aligned (N usually 8)– In an N byte region, only the first k may be addressable– Every N bytes has only N+1 possible states

153



● NOTE: Heap allocated regions are N byte aligned (N usually 8)– In an N byte region, only the first k may be addressable– Every N bytes has only N+1 possible states– Map every N bytes to 1 shadow byte encoding state as a number

4

6

7

0

5

3

-1

1

2

154




All good = 0 All bad = -1 Partly good = # good

4

6

7

0

5

3

-1

1

2

155




● What does accessing shadow memory for an address look like? (N=8)

156




● What does accessing shadow memory for an address look like? (N=8)– Preallocate a large table

157




● What does accessing shadow memory for an address look like? (N=8)– Preallocate a large table– Shadow = (address >> 3) + Offset

158




● What does accessing shadow memory for an address look like? (N=8)– Preallocate a large table– Shadow = (address >> 3) + Offset– With PIE, Shadow = (address >> 3)

159





if (*(address>>3)) { ReportError(...);}*address = ...

160





if (*(address>>3)) { ReportError(...);}*address = ...

Now you can also see the reason for the numerical encoding....

161


shadow = address >> 3state = *shadowif (state != 0 && (state < (address & 7) + size)) { ReportError(...);}*address = ...

● Handling accesses of size < N (N=8)

162


shadow = address >> 3state = *shadowif (state != 0 && (state < (address & 7) + size)) { ReportError(...);}*address = ...

● Handling accesses of size < N (N=8)

Careful construction of states can make runtime checks efficient

163

AddressSanitizer - Evaluating

● In dynamic analyses, we care about both overheads & result quality

164



● Overheads– Need to determine what resources are being consumed

165



● Overheads– Need to determine what resources are being consumed– Memory –

Shadow memory capacity is cheap, but accessed shadows matter

166




Shadow memory capacity is cheap, but accessed shadows matter– Running time –

Can effectively be free for I/O bound projectsUp to 25x overheads on some benchmarks

167




Shadow memory capacity is cheap, but accessed shadows matter– Running time –

Can effectively be free for I/O bound projectsUp to 25x overheads on some benchmarks

● Quality– Precision & recall matter

Where will it miss bugs?Where will it raise false alarms?

168


● False negatives– Computed pointers that are accidentally in bounds

169


● False negatives– Computed pointers that are accidentally in bounds– Unaligned accesses that are partially out of bounds

170


● False negatives– Computed pointers that are accidentally in bounds– Unaligned accesses that are partially out of bounds– Use after frees with significant churn

Example: Comparing Executions

172

Why compare traces or executions?

● Understanding the differences between two executions(& how some differences cause others)can help explain program behavior

173



● Several tasks could be made simpler by trace comparison

174



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new

175



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new– Validating patches – old vs new

176



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new– Validating patches – old vs new– Understanding automated repair – old vs new

177



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new– Validating patches – old vs new– Understanding automated repair – old vs new– Debugging with concurrency – buggy vs nonbuggy schedules

178



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new– Validating patches – old vs new– Understanding automated repair – old vs new– Debugging with concurrency – buggy vs nonbuggy schedules– Malware analysis – malicious vs nonmalicious run

179



● Several tasks could be made simpler by trace comparison– Debugging regressions – old vs new– Validating patches – old vs new– Understanding automated repair – old vs new– Debugging with concurrency – buggy vs nonbuggy schedules– Malware analysis – malicious vs nonmalicious run– Reverse engineering – desired behavior vs undesirable

180

How it might look

Correct Buggy

181

How it might look

x was 5 instead of 3

Correct Buggy

182

How it might look


So y was 2 instead of 7

Correct Buggy

183

How it might look



So the TRUE branch executedinstead of the FALSE branch

Correct Buggy

184

How it might look



So the TRUE branch executedinstead of the FALSE branchSo the update of z was skipped

Correct Buggy

185

How it might look




So the incorrect value of z was printed

Correct Buggy

186

How it might look





Correct Buggy What do we need?● locations● state● flow● causation

187

How it might look






188

How it might look






189

How it might look






190

How it might look






191

How it might look






We can construct this backwardfrom a point of failure/difference

192

So why not just...

● Traces can be viewed as sequences....– Why not just do LCS based sequence alignment?

193

So why not just...


def foo(int c): if c: while bar(): ...

foo(...)baz(...)foo(...)

194

So why not just...



foo(...)baz(...)foo(...)

foo()

baz()

foo()

195

So why not just...



foo(...)baz(...)foo(...)

foo()

baz()

foo()

foo()

baz()

foo()

196

So why not just...



foo(...)baz(...)foo(...)

foo()

baz()

foo()

foo()

baz()

foo()

What is marked as different?

197

foo()

foo()

So why not just...



foo(...)baz(...)foo(...)

foo()

baz()

baz()

foo()


198

baz()

baz()

So why not just...



foo(...)baz(...)foo(...)

foo()

foo()

foo()

foo()


What is intuitively different?

199

baz()

baz()

So why not just...



foo(...)baz(...)foo(...)

foo()

foo()

foo()

foo()



200

baz()

baz()

So why not just...



foo(...)baz(...)foo(...)

foo()

foo()

foo()

foo()



Execution comparison mustaccount for what a program

means and does!

201

The big picture

● Fundamentally, execution comparison needs to account for

202

The big picture

● Fundamentally, execution comparison needs to account for– Structure – How is a program organized?

203

The big picture

● Fundamentally, execution comparison needs to account for– Structure – How is a program organized?– Value – What are the values in the different executions?

204

The big picture

● Fundamentally, execution comparison needs to account for– Structure – How is a program organized?– Value – What are the values in the different executions?– Semantics – How is the meaning of the program affected by the differences?

205

The big picture

● Fundamentally, execution comparison needs to account for– Structure– Value– Semantics

● We can attack these through

206

The big picture


● We can attack these through– Temporal alignment

● What parts of the trace correspond?

207

The big picture




– Spatial alignment● What variables & values correspond across traces?

208

The big picture





– Slicing● How do differences transitively flow through a program?

209

The big picture





– Slicing● How do differences transitively flow through a program?

– Causality testing● Which differences actually induce difference behavior?

210

Temporal Alignment

● Given i1 in T1 and i2 in T2,– when should we say that they correspond? [Xin, PLDI 2008][Sumner, ASE 2013]

– how can we compute such relations?

i2

i1

?

211

Temporal Alignment



● In the simplest case T1 and T2 may follow the same path[Mellor-Crummey, ASPLOS 1989]

i2

i1

?

212

Temporal Alignment




foo()

i2

i1

?

213

Temporal Alignment




foo()

i2

i1

?

214

Temporal Alignment




foo()

i2

i1

?

215

Temporal Alignment




foo()

i2

i1

?

216

Temporal Alignment




foo()

i2

i1

?

217

Temporal Alignment




foo()Position along a path can be maintained via a counter

i2

i1

?

218

Temporal Alignment




foo()Position along a path can be maintained via a counter

Only need to increment along1) back edges2) function callsi

2

i1

?

219

Temporal Alignment




● Suppose that we know the programs are acyclic?

i2

i1

?

220

Temporal Alignment





i2

i1

?

221

Temporal Alignment





The position in the DAG relates the paths

i2

i1

?

222

Temporal Alignment





● Now consider the case where cycles can occur... [Sumner, ASE 2013]

How can we extend the acyclic case?

i2

i1

?

223

Temporal Alignment







We can unwind the loop to make it logically acyclic

i2

i1

?

224

Temporal Alignment







i2

i1

?

225

Temporal Alignment







i2

i1

?

226

Temporal Alignment







...

i2

i1

?

227

Temporal Alignment







...

These are different iterations of one loop.A counter for each active loop suffices (mostly).

i2

i1

?

228

Temporal Alignment







...

1 counter per active loop+ the call stack disambiguates!

i2

i1

?

229

Temporal Alignment






– Can we efficiently represent this?

i2

i1

?

230

Temporal Alignment







Call stack/context Iteration stack Instruction IDi2

i1

?

231

Temporal Alignment







Call stack/context Iteration stack Instruction ID

Can be encoded/inferred Can be inferred

i2

i1

?

232

Spatial Alignment

● We must also ask what it means to compare program state across executions

233

Spatial Alignment

● We must also ask what it means to compare program state across executions– How can we compare two integers X and Y?

3 != 5

234

Spatial Alignment

● We must also ask what it means to compare program state across executions– How can we compare two integers X and Y?– How can we compare two pointers A and B?

0xdeadbeef in T1 = 0xcafef00d in T2?

235

Spatial Alignment

● We must also ask what it means to compare program state across executions– How can we compare two integers X and Y?– How can we compare two pointers A and B?

0xdeadbeef in T1 = 0xcafef00d in T2?

If you allocated other stuff in only one run,this can be true even without ASLR!

236

Spatial Alignment

● We must also ask what it means to compare program state across executions– How can we compare two integers X and Y?– How can we compare two pointers A and B?– How can we compare allocated regions on the heap?

Should they even be compared?!?

237

Spatial Alignment



● In practice, comparing state across executions requires comparing memory graphs– We need a way to identify corresponding nodes (state elements)

238

Spatial Alignment




A B C

A BT1

T2

239

Spatial Alignment




A B C

A BT1

T2

What are the differences?

240

Spatial Alignment




A B C

A BT1

T2


1) list.append(value++)2) if c:3) list.append(value++)4) list.append(value++)

241

Spatial Alignment




A B C

A BT1

T2



1

1

242

Spatial Alignment




A B C

A BT1

T2



1

1 3

243

Spatial Alignment




A B C

A BT1

T2



1

1 3

4

4

244

Spatial Alignment




A B C

A BT1

T2



1

1 3

4

4

245

Spatial Alignment




A B C

A BT1

T2



1

1 3

4

4

246

Spatial Alignment




● Again, the semantics of the program dictate the solution– Identify heap allocations by the aligned time of allocation

A B C

A BT1

T2

1

1 3

4

4

247

Dual Slicing

● Now we can– Identify corresponding times across executions– Identify & compare corresponding state at those times

248

Dual Slicing


● We can use these to enhance dynamic slicing by being aware of differences! (called dual slicing)

249

Dual Slicing


● We can use these to enhance dynamic slicing by being aware of differences! (called dual slicing)– Based on classic dynamic slicing– Include transitive dependencies that differ or exist in only 1 execution

250

Dual Slicing



1)x 1← 1

2)y 1← 1

3)print(x+y)

1)x 0← 1

2)y 1← 1

3)print(x+y)3

2

1

3

2

1

0

1 1

1

251

Dual Slicing



1)x 1← 1

2)y 1← 1

3)print(x+y)

1)x 0← 1

2)y 1← 1

3)print(x+y)3

2

1

3

2

1

0 1 1

1

252

Dual Slicing



1)x 1← 1

2)y 1← 1

3)print(x+y)

1)x 0← 1

2)y 1← 1

3)print(x+y)3

2

1

3

2

1

3

2

1

0

1 1

10 1

253

Dual Slicing

● The differences in dependencies capture multiple kinds of information

254

Dual Slicing

● The differences in dependencies capture multiple kinds of information– Value-only differences

255

Dual Slicing

● The differences in dependencies capture multiple kinds of information– Value-only differences– Provenance/Source differences

256

Dual Slicing

● The differences in dependencies capture multiple kinds of information– Value-only differences– Provenance/Source differences– Missing/Extra behavior

257

Dual Slicing


● Recall: Dynamic slicing could not handle execution omission,but dual slicing can!

258

Dual Slicing


● Recall: Dynamic slicing could not handle execution omission,but dual slicing can!

● Dual slices can be effective for concurrent debugging & exploit analysis[Weeratunge, ISSTA 2010][Johnson, S&P 2011]

259

Adding Causation

● Now we can produce explanations exactly like our example!– Can answer “Why” and “Why not” questions about behavior & differences

[Ko, ICSE 2008]

Correct Buggy

260

Adding Causation


[Ko, ICSE 2008]

– But they may still contain extra information/noise...

261

Adding Causation


[Ko, ICSE 2008]


1) x = ...2) y = ...3) if x + y > 0:4) z = 05) else:6) z = 17) print(z)

262

Adding Causation


[Ko, ICSE 2008]



Correct

x = 10y = -1Truez = 0

“0”

263

Adding Causation


[Ko, ICSE 2008]



Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

264

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

265

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

266

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

267

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

268

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

269

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

BuggyDual slicing captures differences, not causes.

What does that mean here?

270

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

271

Adding Causation


[Ko, ICSE 2008]



11

33

77

4

Correct Buggy

6

2 2

Correct

x = 10 x = 0y = -1 y = -2True Falsez = 0

z = 1“0” “1”

Buggy

The cost of these extra edges is high in practice!All transitive dependencies...

272

Adding Causation

● Causation is often framed via “alternate worlds” & “what if” questions...– We can answer these causality questions by running experiments!

What Should We Blame?

Recall:

?

Trial


Recall: x = 5y = 4z = 3

x = 5y = 4z = 1

y = 4

?

Trial


Recall: x = 5y = 4z = 3

x = 5y = 4z = 1

y = 4

?

Trial

What does this patched run even mean?

x ← 1 1y ← 1 3z ← 1 6if False:

else: y ← 1 4print(4)

Example – Altered Meaning

1)x ← 1 input()2)y ← 1 input()3)z ← 1 input()4)if y+z > 10:5) y ← 1 56)else: y ← 1 y+17)print(y)

x ← 1 0y ← 1 7z ← 1 3if False:


CorrectBuggy


What should we blame here?


x ← 1 0y ← 1 7z ← 1 3if False:


x ← 1 1y ← 1 3z ← 1 6if False:


CorrectBuggy

x ← 1 0y ← 1 7z ← 1 3if False:




x ← 1 1y ← 1 3z ← 1 6if False:


CorrectBuggy Trial

x ← 1 0y ← 1 7z ← 1 3if False:


y ← 1 7



x ← 1 1y ← 1 7z ← 1 6if False:


CorrectBuggy Trial

x ← 1 0y ← 1 7z ← 1 3if False:


x ← 1 1y ← 1 7z ← 1 6



x ← 1 1y ← 1 7z ← 1 6if False:


CorrectBuggy Trial

x ← 1 0y ← 1 7z ← 1 3if False:


x ← 1 1y ← 1 7z ← 1 6if True: y ← 1 5

print(5)



x ← 1 1y ← 1 7z ← 1 6if False:


CorrectBuggy Trial

x ← 1 1y ← 1 7z ← 1 6if True: y ← 1 5

print(5)



● New control flow unlike original runs

● Occurs in large portion of real bugs

x ← 1 0y ← 1 7z ← 1 3if False:


x ← 1 1y ← 1 7z ← 1 6if False:


CorrectBuggy Trial

Dual Slicing


76

2

2)y ← 1 input()6)y ← 1 y+17)print(y)

Extract

x ← 1 0y ← 1 7z ← 1 3if False:


x ← 1 1y ← 1 3z ← 1 6if False:


CorrectBuggy

y ← 1 3y ← 1 4print(4)

Example – Extracted Meaning

y ← 1 7y ← 1 8print(8)

2)y ← 1 input()6)y ← 1 y+17)print(y)

CorrectBuggy Trial

y ← 1 7 y ← 1 7y ← 1 4print(4)


y ← 1 7y ← 1 8print(8)

2)y ← 1 input()6)y ← 1 y+17)print(y)

CorrectBuggy Trial

y ← 1 8print(8)

y ← 1 7 y ← 1 7y ← 1 4print(4)


y ← 1 7y ← 1 8print(8)

2)y ← 1 input()6)y ← 1 y+17)print(y)

Trial can now correctly blame y

CorrectBuggy Trial

289

Adding Causation

● Causation is often framed via “alternate worlds” & “what if” questions...– We can answer these causality questions by running experiments!

● We perform these causality tests in both directions in order to collect symmetric information

Summing Up

291

Key Challenges

● Identifying the information you care about– Dynamic dependence? Valid memory? Just the execution outcome?

292

Key Challenges


● Collecting that information efficiently– abstraction, encoding, compression, sampling, ...

293

Key Challenges



● Selecting which executions to analyze– Existing test suite? Always on runtime? Directed test generation?– How does underapproximation affect your conclusions?– Can you still achieve your objective in spite of it?

294

Key Challenges



● Selecting which executions to analyze– Existing test suite? Always on runtime? Directed test generation?– How does underapproximation affect your conclusions?– Can you still achieve your objective in spite of it?

● Doing some of the work ahead of time– What can you precompute to improve all of the above?

295

Summary

● Analyze the actual/observed behaviors of a program

296

Summary


● Modify or use the program’s behavior to collect information

297

Summary



● Analyze the information online or offline

298

Summary



● Analyze the information online or offline

● The precise configuration must be tailored to the objectives & insights– Compiled vs DBI– Online vs Postmortem– Compressed, Encoded, Samples, ...– ...

dynamic analysis - cs.sfu.cawsumner/teaching/745/08-dynamic-analysis.pdf · dynamic binary...

Documents