research overview: virtualization-based malware defense · n.c. state university 4/21/2009 xuxian...
TRANSCRIPT
Research Overview: Virtualization-Based Malware Defense
Assistant ProfessorDepartment of Computer Science
N.C. State University4/21/2009
Xuxian Jiang
Outline
q Motivations and research overviewq Virtualization-based malware defenseQ New virtualization mechanism: OBSERVQ New capabilities enabledv Invisible system loggingv Stealth malware detectionv OS Kernel integrity protection
q Future workq Summary
q Internet malware remains a top threatQ Malware: viruses, worms, rootkits, spyware, bots…
Motivations
Research Goals
q Research goalsQ Malware-free Cyberspace (long-term)Q Gaining the upper hand over malware (short-term)
Honeyfarm
Virtualization Technology
Research Agenda
Malware PlaygroundMalware Profiling &
Protocol Reverse Engineering
Malware Contamination Tracking*
Past Present Future
USENIX Sec’04, NDSS’06, JPDC’06
TPDS’07, ICDCS’06
RAID’05 NDSS’08, RAID’08,WORM’06
Kernel/HypervisorRootkit Defense
Botnet Defense*
Other Applications
OBSERV Mechanism & Applications (CCS’07, RAID’07)*
q State-of-the-art malware defenseQ Running anti-malware software inside the monitored
systemv Advantage: They can see everything (e.g., files,
processes…)v Disadvantage:
VirusScanFirefoxIE
OS Kernel
…
Why OBSERV?
They may not see anything!
Why OBSERV?q Current approach fundamentally flawedQ Malware running in the same system space with anti-
malware software at the same privileged levelQ No clear winner in the arms race between them
q Solution: Going out of the box
FirefoxIE
OS Kernel
…
VirusScan
Virtual Machine Monitor (VMM)
The “Semantic-Gap” Challenge
q What we can observe:Q Low-level states v Memory pages, disk blocks…
Q Low-level eventsv Privileged instructions,v Interrupts, I/O…
q What we want to observe:Q High-level semantic statesv Files, processes…
Q high-level semantic eventsv System calls, context switches…
Virtual Machine Monitor (e.g., VMware, Xen)
Guest OSSemantic Gap
VirusScan
Our Solution: OBSERVq OBSERV: “Out-of-the-Box” with SEmantically
Reconstructed ViewQ A new mechanism missing in all current VMMs
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV
New Capabilities
Capability II: Malware detection by
view comparison
Capability I: Invisible system logging
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV
protection
Capability III: OS kernel integrity
protection
OBSERV View In-the-boxView Diff
OBSERV: Bridging the Semantic Gap
q Step 1: Procuring low-level VM states and eventsQ Disk blocks, memory pages, registers…Q Traps, interrupts…
q Step 2: Reconstructing high-level semantic viewQ Files, directories, processes, and kernel modules…Q System calls, context switches…
VM Introspection
Guest View Casting
Step 1: VM Introspection
Raw VMM Observations
Virtual Machines (VMs)
VMware Academic Program
VM disk image
VM hardware state (e.g., registers)
VM physical memory
VM-related low-level events (e.g., interrupts)
Step 2: Guest View Casting
Virtual Machine Monitor (VMM)
Guest OS
Key observation: The guest OS provides all semantic “templates” of data structures and functions to reconstruct VM’s semantic view
OBSERVSemantic Gap
Guest View Casting
Raw VMM Observations Casted Guest Functions & Data Structures
Reconstructed Semantic View
Device drivers, file system drivers
Memory translation,task_struct, mm_struct
CR3, MSR_SYSENTER_CS,MSR_SYSENTER_EIP/ESP
Event semanticsSyscalls,
context switches, ....
Event-specific arguments…
VM disk image
VM hardware state (e.g., registers)
VM physical memory
VM-related low-level events (e.g., interrupts)
Guest View Casting on Memory StateProcess List
Process Memory Layout
Guest Memory Addressing
q Traditional memory addressingQ MMU translates VA to PAQ OS image mapped to known PAv Linux: VA 0xc0000000 == PA 0x0v Windows: VA 0x80000000 == PA 0x0
q VM complicates the translation
Q Guest virtual -> guest physical
Q Guest physical -> host physicalVM IntrospectionReverse Address Translation
Emulated Address Translation
Kernel Symbols
Trap
Guest View Casting on System Calls
q System call instructionsQ int 0x80; sysenter
q System call conventionQ EAX, EBX, ECX, EDX, ESI, EDI, EBP, …
Instr Handler
1. int 0x80, sysenter
2. Trap Generation
5. Continue the Execution
Trap Handler
3. Trap and Emulate
Guest
VMM
User
Kernel
4. Emulate Instruction
Related Workq Virtual machine introspection (Livewire[Garfinkel03],
IntroVirt[Joshi05], HyperSpector[Kourai05])Q Focusing on targeted attacks for specialized
IDSesq Secure monitors (CoPilot[Petroni04], Terra[Garfinkel03],
sHype[Sailer05], SecVisor[Perrig07])Q Missing a basic mechanism similar to OBSERV
Outline
q Motivations and research overviewq Virtualization-based malware defenseQ New VMM mechanism: OBSERVQ New capabilities enabledv Invisible system loggingv Stealth malware detectionv OS Kernel integrity protection
q Future workq Summary
New Capability I: Invisible System Loggingq Trusted logging: an essential function for
honeypotsq Two current approaches Q External (e.g., tcpdump, ethereal, etc)v Only monitoring network traffic
Q Internal (e.g., sebek, syslog, etc)v Can be compromised!
Internal
External
Tamper-Resistance
Deep Inspection
q Sebek: de-facto honeypot logging toolq Can be detected, disabled, or bypassed by
NoSEBrEaK [Holz+, BlackHat’04/Defcon 12]
Invisible System Logging
q Demo Clip (2.5 minutes):Q http://www.cs.ncsu.edu/faculty/jiang/research/vms
cope/sebek.swf
Invisible System Logging
[Holz+, Blackhat’04/Defcon 12]
OBSERV-based logging
Sebek-based logging
Invisible System Logging
Opera profile
Firefox profile
New Capabilities II & III
Capability II: Malware detection by
view comparison
Capability I: Invisible system logging
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERVprotection
Capability III: OS kernel integrity
protection
OBSERV View In-the-boxView Diff
q Experiment setupQ Guest VM: Windows XP (SP2)v Windows Fu rootkit
Q Host OS: Scientific Linux 4.4Q VMM: VMware Server 1.0.1
View Comparison on Volatile Memory State
“In-the-box” viewOBSERV view
Diff
q Experiment setupQ Guest VM: A Redhat 7.2-based honeypotv Linux SHv4 rootkit
Q Host OS: Windows XP (SP2)Q VMM: VMware Server 1.0.1
View Comparison on Persistent Disk State
“In-the-box” viewOBSERV view
Diff
Symantec AntiVirusSymantec AntiVirus
External Run of COTS Anti-Malware Softwareq Experiment setupQ Both guest OS and host OS run Windows XP (SP2)Q VMM: VMware Server 1.0.1
q Running Symantec AntiVirus twiceQ InsideQ Outside
Hacker Defender
NTRootkit
External Scanning Result
Internal Scanning Result
Diff
OBSERV Capability III: OS Kernel Integrity Protection
q High-assurance OS kernel Q No malicious kernel codeQ No kernel rootkit attacks
q Two main tasks:Q Tracking run-time kernel code layoutQ Enforcing the following propertiesv Only loading authenticated kernel codev Only executing authenticated kernel code
R. Riley, X. Jiang, D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing", RAID’08, Boston, MA, September 2008
OBSERV
NICKLE: “No Instruction Creeping into Kernel Level Executed”
NICKLE
Standard memory
Kernel Code
Shadow memory
VMM
Guest OS
q Step 1: Create two memory spacesQ Standard memoryQ Shadow memory
q Step 2: Authenticate and copy kernel code to shadow memory
q Step 3: Memory access dispatchQ Kernel code fetch ->
shadow memoryQ All other accesses ->
standard memoryKernel Code
Demonstration of Effectiveness
Successfully preventing 23 real-world kernel rootkits!
Other OBSERV-enabled Capabilities
q Tamper-resistant malware profiling and analysisQ Contamination tracking [TPDS’07, ICDCS’06]Q Protocol reverse engineering [NDSS’08, WORM’06]
q “Out-of-the-box” policy enforcement [SACMAT’07]
q Other opportunities
Future Work
Honeyfarm
Virtualization Technology
Malware PlaygroundMalware Profiling &
Protocol Reverse Engineering
Malware Contamination Tracking*
Past Present Future
USENIX Sec’04, NDSS’06, JPDC’06
TPDS’07, ICDCS’06
RAID’05 NDSS’08, RAID’08, WORM’06
Kernel/Hypervisor Rootkit Defense
Botnet Defense*
Other Applications
OBSERV Mechanism and Applications (CCS’07, RAID’07)*
Rootkit Defenseq Reality: rampant rootkits
Source: McAfee Avert Lab Report (April 2006)
400% growth400% growth
Q1 of 2005
700% growth700% growth
Viruses/worms/bots, …
Rootkit Defense -- Challenges
q A fundamental questionQ How to grab the upper hand?
q ChallengesQ How to secure the lowest level access?v Rethinking VMM design
Q How to defeat rootkit infection?v Rethinking OS kernel design (e.g., NX protection)
Q How to balance protection and performance?v Rethinking guest OS-VMM interactions
Summaryq OBSERV enables “out-of-the-box” malware defenseQ Eliminating semantic gapQ Enabling new malware defense capabilitiesQ A step towards malware-free Cyberspace
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV OK kernel integrity protection
Invisible system logging
Malware detection by view comparison
