efficient vm introspection in kvm and performance comparison with xen kenichi kourai kousuke...
TRANSCRIPT
![Page 1: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/1.jpg)
Efficient VM Introspection in KVM and
Performance Comparison with Xen
Kenichi KouraiKousuke Nakamura
Kyushu Institute of Technology
![Page 2: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/2.jpg)
Intrusion Detection System (IDS)IDSes detect attacks against serversMonitor the systems and networks of serversAlert to administrators
Recently, attackers attempt to disable IDSesBefore they are detectedThis is easy because IDSes are running in
servers IDS
server
intrudedetect
![Page 3: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/3.jpg)
IDS OffloadingOffloading IDSes using virtual machines
(VMs)Run a server in a VMExecute IDSes outside the VM
Prevent IDSes from being compromisedCan be provided as a cloud service
Cloud providers can protect users' VMs
VM
IDS offloading
IDS
In-VM monitoring
IDSmonitorVM
![Page 4: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/4.jpg)
VM Introspection (VMI)A technique for monitoring VMs from the
outsideMemory introspection
Obtain raw memory contents and extract OS data
Disk introspectionObtain raw disk data and interpret a filesystem
Network introspectionObtain packets only from/to VMs
???IDS
VM
disk
??? memory
packets network
![Page 5: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/5.jpg)
Performance of VMIPerformance has not been reported in
detailNo performance comparison
E.g., VMwatcher [Jiang+ CCS'07]
Implemented in Xen, QEMU, VMware, and UMLReported only for UML
E.g., EXTERIOR [Fu+ VEE'13]
Implemented in KVM and QEMUNo difference due to using memory dump
Performance data is importantFor user's selection of virtualization software
![Page 6: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/6.jpg)
The Purpose of This WorkPerformance comparison among
virtualization software in terms of VMITarget: Xen and KVM
Widely used open source virtualization software
System architecture is different
hypervisor
VM VM
OS
VM
process
Xen KVM
![Page 7: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/7.jpg)
Implementation for KVMNo efficient implementation of VMI for KVMSeveral studies have been done for KVM
The implementation details are unclearLibVMI [Payne+ '11] supports VMI for both Xen
and KVMThe performance of memory introspection is
too low in KVMOptimized for Xen
![Page 8: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/8.jpg)
KVMonitorWe have developed an efficient VMI tool
for KVMExecute an IDS as a process of the host OSProvide functions for introspecting memory,
disks, and NICs in QEMU
host OS
diskQEMU
memory
NIC
KVM module
VM
IDS
KVMonitor
offload
monitor
![Page 9: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/9.jpg)
Memory Introspection (1/2)Difficult to efficiently introspect QEMU's
memoryLibVMI obtains memory contents from QEMU
KVMonitor shares VM's physical memory with QEMU via a memory fileAccess As a memory-mapped fileEnable direct memory introspection
IDS
VM's physical memory
VM
VM's physical memory
memory file
VM's physical memory
KVMonitor QEMU
![Page 10: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/10.jpg)
Memory Introspection (2/2)IDSes usually access OS data using virtual
addressesKVMonitor translates virtual addresses into
physical addressesLook up the page table for address
translationIntrospect the CR3 register using QMP
IDS VM
VM's physical memory
memory file
VM's physical memory
page table
KVMonitor QEMUCR3
![Page 11: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/11.jpg)
Disk/Network IntrospectionKVMonitor introspects VM's disks via the
network block device (NBD)Interpret the qcow2 format in the NBD
serverInterpret the filesystem in the host OS
KVMonitor captures packets from a tap device
host OS
VM
QEMU
IDS
KVMonitor
tap
diskimage
fileNBD
server
NBD
network
![Page 12: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/12.jpg)
Transcall with KVMonitorWe have ported Transcall [Iida+ '11] for Xen
to KVMEnable offloading legacy IDSes without any
modificationsConsist of a system call emulator and a
shadow filesystemIncluding the proc filesystemAnalyze OS data by memory introspection
QEMU
IDS
KVMonitor
VMTranscall
analyze
![Page 13: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/13.jpg)
ExperimentsWe examined that KVMonitor achievedEfficient memory introspectionNo impact on memory performance of a VMEffective IDS offloading
CPU: Intel Xeon E5630 (12 MB L3 cache)Memory: 6 GB DDR3 PC3-8500HDD: 250 GB SATANIC: gigabit EthernetHypervisor: KVM 1.1.2Host OS: Linux 3.2.0
CPU: 1Memory: 512 MBDisk: 20 GB (ext3)Guest OS: Linux 2.6.27
PC VM
![Page 14: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/14.jpg)
KVMonitor vs. LibVMIWe measured the performance of memory
introspectionCopy VM's physical memory
by 4KB
KVMonitor was32x faster than LibVMI
Series10
2
4
6
8
10
12
9.6
0.3
KVMonitor LibVMI
read
(G
B/s
) fast
![Page 15: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/15.jpg)
memoryfile
Why is LibVMI so slow?LibVMI has to issue a QMP command for
each memory accessMemory contents are transferred from QEMU
to LibVMI
QEMU
LibVMI
QMPIDS
LibVMI
KVMonitor
IDS
VM's memory
VM's memory
VM's memory
QEMUKVMonit
or
VM VM
![Page 16: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/16.jpg)
In-VM Memory PerformanceDoesn't using a memory file affect
memory performance of a VM?
Using a memory file wasas efficient as malloc
read write0123456789
108.6
6.6
8.5
6.3
memory file malloc
thro
ug
hp
ut
(GB
/s)
memoryfile VM's
memory
QEMU
VM
QEMU
VM
VM's memory
memory file malloc
![Page 17: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/17.jpg)
KVMonitor vs. In-VM AccessKVMonitor was faster than
in-VM memory accessDue to virtualization
overhead
Series10
2
4
6
8
10 9.68.6
KVMonitor In-VM
read
(G
B/s
)memory
file
IDS
VM's memory
VM's memory
QEMUKVMonit
or
VM
fast
![Page 18: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/18.jpg)
Offloading Legacy IDSes (1/3)TripwireCheck filesystem integrity in disks
We added, deleted, and modified filesOffloaded Tripwire detected changed files
Rule Name ... Added Removed ModifiedMonitor Filesystems 1 1 1Total Objects scanned: 67082Total violations found: 3
Tripwire
DB disk
VM
![Page 19: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/19.jpg)
Offloading Legacy IDSes (2/3)SnortInspect network packets
We performed portscans from another hostOffloaded Snort detected portscans[**] [1:1421:11] SNMP AgentX/tcp request [**][Classification: Attempted Information Leak] ...01/28-10:47:13.406931 192.168.0.68:47962 -> 192.168.0.81:705
VMSnort
rule sets
packetsportscan
![Page 20: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/20.jpg)
Offloading Legacy IDSes (3/3)ChkrootkitDetect rootkits using ps, netstat, and file
inspectionWe tampered with ps and netstat in a VMOffloaded chkrootkit detected tampered
commandsROOTDOR is ’/’Checking ’ps’...INFECTEDChecking ’netstat’...INFECTED :
chkrootkitdisk
VM
ps netstatps netstat
...
execute
![Page 21: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/21.jpg)
Cross-view Diff (1/2)A technique for detecting hidden malwareCompare the results of VMI and in-VM
monitoringThe difference means the existence of
hidden malware
VM
A B D ...A B C D ...
cross-view diff
engine
C is hidden
monitorIDS IDS
![Page 22: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/22.jpg)
Cross-view Diff (2/2)We tampered with ps in a VMA hidden process was detected as malicious
We tampered with netstat in a VMA hidden port was detected as a backdoor
PID TTY TIME CMD 1 ? 00:00:00 init 2 ? 00:00:00 kthreadd :
PID TTY TIME CMD 2 ? 00:00:00 kthreadd :
Proto ... Local Address ...tcp 0.0.0.0:22 :
Proto ... Local Address ...tcp 0.0.0.0:5900tcp 0.0.0.0:22 :
results from offloaded commands results from in-VM commands
ps
netstat
![Page 23: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/23.jpg)
KVMonitor vs. XenWe compared the performance of VMI
between KVM and XenUsing a VMI tool for Xen
Memory: standard libraryDisk: loopback mountNetwork: tap device
hypervisor
Dom0 (VM)
Hypervisor: Xen 4.1.3Dom0 OS: Linux 3.2.0VM: fully virtualized
IDSlibxenct
rl
tap
VM
disk imagefile
![Page 24: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/24.jpg)
Memory IntrospectionWe measured read throughputCopy VM's physical memory
by 4KB
KVMonitor was48x faster than Xen
VMI0
2
4
6
8
10
12
9.6
0.2
KVM Xen
read
(G
B/s
) fast
![Page 25: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/25.jpg)
Why is Xen so slow?Xen has to map each memory pageIt cannot map all the pages in advanceIt takes time proportional to the number of
pagesKVMonitor can read a pre-mapped file
KVMonitor
IDS
Xen
VM's memory
IDS
map
VM
memoryfile
KVMonitor
libxenctrl
![Page 26: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/26.jpg)
Kernel Integrity CheckingWe measured the execution time of the
kernel integrity checkerRead the code areaTranslate virtual to
physical addresses
KVMonitor was118x faster than Xen
0
50
100
150
200
250
1.9
224
KVM Xen
tim
e (
ms)
fast
![Page 27: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/27.jpg)
Why is the speedup so larger?The speedup in the real IDS was much
larger48x (simple benchmark)118x (kernel checker)
Due to address translationIn Xen, the access cost of the page table is
highOnly 8 bytes are read after memory mapping
simple benchmark
IDS map &read
VM
real kernel checker
IDS map &read
VM
libxenctrl libxenctrl
![Page 28: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/28.jpg)
Disk IntrospectionWe measured the execution time of
TripwireFor two formats of disks
raw and qcow2
KVMonitor wasComparable to Xen
The difference betweenformats was largerRaw was faster than qcow2
raw qcow20
2
4
6
8
10
7.5
9.4
7.5
9.2
KVM Xen
tim
e (
min
)fast
![Page 29: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/29.jpg)
Network IntrospectionWe measured the packet loss rate in SnortSend many packets as
fast as possible
KVMonitor wasmore lightweight
than XenDom0 suffered from
virtualization overhead
Series10
2
4
6
8
10
12
6.2
10.4
KVM Xen
packet
loss r
ate
(%
)fast
![Page 30: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/30.jpg)
ChkrootkitWe measured the execution time of
chkrootkit
KVMonitor was1.6x faster than Xen
Efficient memoryintrospection
No virtualizationoverhead
2x slower than in-VMDue to system call traps Offloading in-VM
0
10
20
30
40
50
60
35
18
55
21
KVM Xen
tim
e (
sec)
fast
![Page 31: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/31.jpg)
Related WorkVMI toolsLivewire [Garfinkel+ NDSS'03] for VMwareXenAccess [Payne+ ACSAC'07] for Xen
Shm-snapshot for LibVMI [Xu+ PDL'13]
Take a VM's memory snapshot in shared memory
It takes 1.4 seconds for 3 GBVolatility [Walters '07]
A memory forensics frameworkVMI for KVM is enabled by a Python adapter,
PyVMI from LibVMI
![Page 32: Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology](https://reader035.vdocument.in/reader035/viewer/2022062407/56649d055503460f949d919a/html5/thumbnails/32.jpg)
ConclusionKVMonitorAchieve efficient VM introspection (VMI) in
KVM32x faster than existing LibVMI
Performance comparison with Xen118x faster at maximum
Chkrootkit was 1.6x faster
Future workComparison with other virtualization
softwareIntegration with LibVMI