matryoshka: strengthening software protection via nested virtual … · 2019. 5. 3. · •...
TRANSCRIPT
![Page 1: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/1.jpg)
Matryoshka: Strengthening Software Protection via Nested Virtual Machines
Sudeep Ghosh (Microsoft Research) Jason D. Hiser (University of Virginia)
Jack W. Davidson (University of Virginia)
![Page 2: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/2.jpg)
The Problem
2
0101110 00111001010 001 01011001000110001110110 0101110110110010111010 101101000101101111111 100010101101100111110 01010111001110010111 1 11111111111111110
• Disassembler • Debuggers • Static Analyzers
• Coverage tools • Simulators
Software is now used to perform critical functionality.
• Banks • Power grids • Medical software • Transportation systems • Internet of Things Researchers must continually upgrade and
enhance software protection approaches
![Page 3: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/3.jpg)
Threat Model
n Software is created in a secure environment. n White-box attack model
n Adversary has access to multiple tools including debuggers, simulators and emulators.
n They can modify the OS to return inaccurate information. n As such, the adversary can inspect, modify or forge any
information. n Given enough time and resources, the adversary can succeed in
manually inspecting and modifying programs. n Boaz Barak , Oded Goldreich , Russell Impagliazzo , Steven Rudich , Amit
Sahai , Salil P. Vadhan , Ke Yang: On the (Im)possibility of Obfuscating Programs, Proceedings of the 21st Annual International Cryptology Conference on Advances in Cryptology, 2001.
n However, most attacks use algorithmic solutions to disable security features in programs.
3
![Page 4: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/4.jpg)
Virtualization
Virtualizing Software (e.g. Xen)
Operating System
Hardware
Applications
System-level VM
Hardware
Process-level VM
Applicationn Application1
Operating System
Virtualizing Software
(e.g. Strata)
Virtualizing Software
(e.g. Strata)
4
![Page 5: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/5.jpg)
Strata
n Infrastructure designed for building process-level virtualization systems
n Designed with extensibility, portability, and application-independence in mind
n Implement new systems by customizing the VM
n Binary only n No source code required n Can be applied to any
application regardless of source language, compiler used, libraries used, etc.
n Provides common services necessary for software dynamic translation
5
![Page 6: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/6.jpg)
Process-level Virtualization
Context Capture
Context Switch Next PC
Translate Decode Fetch
New Fragment
Finished?
Dynamic Translator
Cached? New PC
Application Binary
System Start (first PC)
F$
Direct Conditional branch Trampoline Non-control instruction
PC
7
See: Hiser et al., Fragment Cache Construction Policies for Software Dynamic Translation Systems, VEE 2006.
![Page 7: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/7.jpg)
Strata
0
0.5
1
1.5
2
2.5
Ove
rhea
d N
orm
aliz
ed t
o N
ativ
e R
un
DynamoRIO
PIN
HDTrans
Strata
8
![Page 8: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/8.jpg)
Strata Related Work n SDT
n Evaluating Indirect Branch Handling Mechanisms in Software Dynamic Translation Systems. CGO 2007.
n Evaluating Fragment Construction Policies for Software Dynamic Translation Systems. VEE 2006.
n Retargetable and Reconfigurable Software Dynamic Translation. CGO 2003. n Obfuscation and anti-tamper
n What’s the PointISA? IH 2014. n Replacement Attacks against VM-protected Applications. VEE 2006. n A Secure and Robust Approach to Software Tamper Resistance. IH 2010
n Security n ILR: Where did my gadgets go? Oakland 2012. n Secure and Practical Defense Against Code-injection Attacks Using Software
Dynamic Translation. VEE 2006. n Safe Virtual Execution Using Software Dynamic Translation, ACSAC 2002.
9
![Page 9: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/9.jpg)
Matryoshka: Nested PVMs
10
APPLICATION
PVM2 PVM4
dPVM1 PVM3
![Page 10: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/10.jpg)
G1
G2
G3
G4
Software Protection via Virtualization
Application
Protection Scheme (Guards,
encryption)
Builder (Diablo)
Protected Binary
11
PVMs
PVM1 PVM2 PVM2
![Page 11: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/11.jpg)
Nested PVMs Disk Image
12
F$ (SC1)
Strata1
Strata2
Translated Application Code
Memory
X86 (Hardware)
F$ (SC2)
Translated Application Code
Translated Application Code
![Page 12: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/12.jpg)
Evaluation
F$ Diversity
13
Software Cache Addresses App
Strata2
n Use compression ratio as a proxy for diversity and obfuscation.
n Single PVM: 149; N-PVM: 15.63
![Page 13: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/13.jpg)
Evaluation
Cyclomatic Complexity n Developed by McCabe in 1976 as a measure
of software complexity (TSE Vol. 2, No. 4) n M = E – N + 2P
14
Benchmark CC for PVM CC for N-PVM Increase
176.gcc 1,604 80,109 49X
181.mcf 351 9828 27X
256.perlbmk 803 32,903 40X
179.Art 181 5,130 27X
![Page 14: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/14.jpg)
Evaluation
Run-time Overhead n With a nesting level of two, the base run-time
overhead was 35X. n The problem is trampoline patching (i.e., self-
modifying code), which causes excessive F$ flushes.
15
![Page 15: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/15.jpg)
Nested PVMs Disk Image
17
F$ (SC1)
Strata!
Strata2
Translated Application Code
Memory
X86 (Hardware)
F$ (SC2)
Translated Application Code Translated Application Code
Translated Application Code
Translated Application Code
![Page 16: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/16.jpg)
Super Patching
n When Stratan patches a trampoline, the patch information is sent to Stratan-1
n When a patched (in F$2) target block is translated to F$n-1 by Stratan-1, Stratan-1 patches its F$ (F$n-1), thereby avoid the F$ flush
18
![Page 17: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/17.jpg)
Super Patching Overhead
19
0
0.5
1
1.5
2
2.5
3
3.5
4
164
.gzip
175
.vpr
176
.gcc
181
.mcf
197
.par
ser
253
.per
lbm
k
256
.bzip
2
300.
twol
f
177
.mes
a
179
.art
183
.equ
ake
188
.am
mp AV
G
Strata N-PVM with Super-patching
Perf
orm
ance
Ove
rhea
d (n
orm
alize
d to
nat
ive)
![Page 18: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/18.jpg)
Related Work
n Collberg and Nagra [Pearson 2006] provide an excellent overview of the area
n Anckaert et al [DRM 2006] showed the promise of virtualization for software protection
n Themida [1] and VMProtect [2] use interpreted virtual machines for software protection. Nested VMs apply to them as well.
n Anckaert et al [QoP 2007] discuss metrics for metrics for software protection
20
![Page 19: Matryoshka: Strengthening Software Protection via Nested Virtual … · 2019. 5. 3. · • Disassembler • Debuggers • Static Analyzers • Coverage tools • Simulators Software](https://reader034.vdocument.in/reader034/viewer/2022051904/5ff4d6ab48b4a8329a5cdcf8/html5/thumbnails/19.jpg)
Summary
n Nested PVMs can significantly increase the complexity of software that is the target of crackers
n More research is needed to determine when and how to apply nested VMs to software to balance run-time performance and the strength of the protection provided
21