hot-patching a web server: a case study of asap code repair · 3 1' 2' 3' kernel....
TRANSCRIPT
Hot-Patching a Web Server: a Case Study of ASAP Code Repair
Mathias Payer*, Thomas R. Gross
Department of Computer Science
ETH Zurich
* now at UC Berkeley
Security Dilemma
Integrity and availability threatened by vulnerabilities
Two remedies: update or sandboxing• Security updates fix known vulnerabilities but
require service restart
• Sandboxes protect from unknown exploits but stop the service when an attack is detected
DynSec in 1 Minute
Key insight: both sandboxes and dynamic update mechanisms rely on some form of virtualization
Binary Translation (BT) provides virtualization• Sandbox protects integrity
• Dynamic update mechanism protects availability
DynSec in 2 Minutes
Binary Translation
ApplicationDynSec
Kernel
Patches
Loader
Patch extractionand
distribution
Hot-Patching a Web Server
Analyze all security patches of Apache 2.2• From Dec 1st 2005 to Feb. 18th 2013
• Total of 49 security bugs, most are simple
• Many different classes of bugs
All vulnerabilities
Sandbox protection
Software patches
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation
Conclusion
Code Translation
● Translates individual basic blocks● Weave patches into translated code● Protect from security exploits
Original Code Translated Code
Binary Translator
1
2
43
1'
2'
3'
Kernel
Patch Classes
Simple patch• Only few instructions change, directly patched
Patches building on DSO:• New import patch: additional library function used
• New function patch: additional function
• Additional call patch: calls to existing functions
• New String patch: new static string used
Other patches• Type change, code refactoring, heavyweight changes
Patch Distribution
Most Linux distributions provide dynamic update service; piggy pack on this distribution service
• Automatically generate a dynamic patch when new package is generated
• Systems download packages and install dynamic patches to running services
• System administrators update binaries during next maintenance window
Implementation
DynSec builds on TRuE/libdetox [IEEE S&P’12, ACM VEE’11]
• Patching thread injected in BT layer
• Implemented in <2000 LoC
• 48 LoC changed in TRuE to add DynSec hooks
• Supports unmodified, unaware, multi-threaded x86 applications on Linux
Outline
Motivation
Patching architecture & distribution
Apache case-study• Vulnerability classes
• Distribution
Evaluation
Conclusion
Apache: Vulnerability Classes
DoS XSS IL EXE HBOF lPE
AIH lDoS CSRF ACI IOF
Low 23Moderate 19Important 7
DynSec Coverage
Most (45/49) vulnerabilities are hot patchable• All 7 important vulnerabilities
• 18 (out of 19) moderate vulnerabilities
• 20 (out of 23) low vulnerabilities
Patch complexity• Important patches: 4 simple, 3 DSO patches
• Moderate patches: 6 simple, 12 DSO patches
• Low patches: 10 simple, 10 DSO patches
DynSec: Uncovered exploits
CVE-2007-3304 (lDoS, mod): signals to arbitrary PIDs• Heavy code refactoring
CVE-2008-0005 (XSS, low): missing UTF-7 encoding• Additional types, new functions
CVE-2012-0031 (DoS, low): scoreboard parent DoS• Type change, new functions
CVE-2012-0883 (DoS, low): insecure variable in script• Not applicable to DynSec (start-up script only)
DynSec: Uncovered exploits
CVE-2007-3304 (lDoS, mod): signals to arbitrary PIDs• Heavy code refactoring
CVE-2008-0005 (XSS, low): missing UTF-7 encoding• Additional types, new functions
CVE-2012-0031 (DoS, low): scoreboard parent DoS• Type change, new functions
CVE-2012-0883 (DoS, low): insecure variable in script• Not applicable to DynSec (start-up script only)
Possibility for 4 year stride without restart
Sandbox Coverage
Protects from all code-based exploits• Code injection
• Control-Flow redirection (ROP/partial JOP)
• System call policies
Unpatched protection for 11 (of 49) bugs• Two important vulnerabilities (out of 7)
• 5 moderate vulnerabilities (out of 20)
• 4 low vulnerabilities (out of 21)
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation• SPEC CPU 2006 performance
• Apache performance
Conclusion
Evaluation
DynSec evaluated using SPEC CPU2006• CPU: Intel Core2 Quad Q6600 @ 2.64GHz, 8GB RAM
• Ubuntu 11.04, Linux 2.6.38
• Used GCC 4.5.1 with –O2
Benchmark configurations• Native
• Sandboxing (use TRuE w/ shadow stack and checks)
• DynSec (with different patches)
SPEC CPU2006: Performance
0
0.5
1
1.5
2
2.5
40
0.p
erlb
ench
40
1.b
zip
2
40
3.gc
c
42
9.m
cf
44
5.g
ob
mk
45
6.h
mm
er
45
8.s
jen
g
46
2.li
bq
uan
tum
46
4.h
26
4re
f
47
1.o
mn
etp
p
47
3.as
tar
41
0.b
wav
es
41
6.g
ames
s
43
3.m
ilc
43
4.z
eusm
p
43
5.gr
om
acs
43
6.ca
ctu
sAD
M
43
7.le
slie
3d
44
4.n
amd
45
0.s
op
lex
45
3.p
ovr
ay
45
4.ca
lcu
lix
45
9.G
emsF
DTD
46
5.to
nto
47
0.lb
m
48
2.s
ph
inx3
Mea
n
Sandbox DynSec
SPEC CPU2006: Performance
0
0.5
1
1.5
2
2.5
40
0.p
erlb
ench
40
1.b
zip
2
40
3.gc
c
42
9.m
cf
44
5.g
ob
mk
45
6.h
mm
er
45
8.s
jen
g
46
2.li
bq
uan
tum
46
4.h
26
4re
f
47
1.o
mn
etp
p
47
3.as
tar
41
0.b
wav
es
41
6.g
ames
s
43
3.m
ilc
43
4.z
eusm
p
43
5.gr
om
acs
43
6.ca
ctu
sAD
M
43
7.le
slie
3d
44
4.n
amd
45
0.s
op
lex
45
3.p
ovr
ay
45
4.ca
lcu
lix
45
9.G
emsF
DTD
46
5.to
nto
47
0.lb
m
48
2.s
ph
inx3
Mea
n
Sandbox DynSec
Low performance overhead (~11%)
Apache: “large” files (~250kb)
0
100
200
300
400
500
600
700
800
900
1000
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: picture.png
Native TRUE DynSec DynSec-50 DynSec-100
Apache: “large” files (~250kb)
0
100
200
300
400
500
600
700
800
900
1000
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: picture.png
Native TRUE DynSec DynSec-50 DynSec-100
Less than 7% slowdown
Apache: small (tiny) files (~50b)
0
0.5
1
1.5
2
2.5
3
3.5
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: index.html
Native TRUE DynSec DynSec-50 DynSec-100
Apache: small (tiny) files (~50b)
0
0.5
1
1.5
2
2.5
3
3.5
100 1000 10000
Thro
ugh
pu
t [M
B/s
]
Total connections
Performance impact: index.html
Native TRUE DynSec DynSec-50 DynSec-100
Low performance cost for large connection counts
Outline
Motivation
Patching architecture & distribution
Apache case-study
Evaluation
Conclusion
Conclusion
Virtualization enables on-the-fly code rewriting and repair for unmodified applications
• Sandbox protects integrity
• Patches provide availability
Study shows that protecting large, long-running, and modular applications like Apache is feasible
• High coverage: 45 of 49 Apache bugs patchable
• Low performance impact: 7% for Apache 2.2
Patching Architecture
DynSec thread waits for incoming patches
Patch application happens in 3 steps:• Signal all application threads to stop
• Flush all code caches
• Restart application threads
Patch is applied indirectly when code is retranslated• BT checks for every instruction if a patch is available
Patch Format
The focus of DynSec is on security patches• Most security patches are only few lines of code
• Type changes and code refactoring out of scope
Patches are sets of changed instructions
Each patch may specify additional shared library for more heavyweight changes
Patch Extraction
Build patched application with current toolchain
Extract instruction differences between patched and unpatched version of the binary (per function)
• Changed instructions are added to patch
• Check differences in static read-only data
• Manually ensure integrity of patch (no type changes, no data changes)