security applications for emulation
TRANSCRIPT
Speaker details
An independent researcher. Presented a number of vulnerabilities at the first Ruxcon after auditing the
opensource kernels (FreeBSD, NetBSD, Linux, OpenBSD) Also interested in Reverse Engineering, speaking at CanSecWest on
Linux malware.
Outline
A Presentation examining public research, and the results of my own research, on the topic of emulation applied to security.
Technology review Security applications for emulation
Reverse engineering Cisco IOS Heap Management Tracing and evaluating the capabilities of binaries Dynamic Taint Analysis Automated unpacking Symbolic Execution Detecting Runtime Errors in Programs
And introducing a new tool for the detecting out of bounds heap access in the Linux Kernel
Virtualization
Different technologies all sharing similar themes Virtualization Emulation Dynamic Binary Translation
Different types of virtualization Full Virtualization provides a simulation of the underlying hardware
Host performs native execution of the guest as much as possible. Not an emulator, so aiming for near native speeds. In i386, if there isn't full virtualization hardware support,
privileged code is translated Eg VMWare, VirtualBox
Virtualization is an important technology, but this presentation focuses on the host being able to intercept and emulate each individual instruction in the guest. This is in contrast to virtualization, which executes guest code natively as much as possible, with little general host interception.
Emulation and Dynamic Binary Translation
Emulation Emulator Fetches, Decodes and Executes instruction by instruction Different types of emulators: whole system emulators capable of
running unmodified guest operating systems, or emulators only capable of running applications on specific systems.
Guest state is maintained in software, including the CPU, system memory, and for whole system emulators, hardware devices.
Eg Bochs Used in the open source automated unpacker, Pandora's Bochs.
Dynamic Binary Translation A faster form of emulation Caches blocks of decoded and translated instruction Eg QEMU
Used in Argos, a system for capturing 0day* Used in my MemCheck tool for detecting Linux kernel heap
access bugs*.
Dynamic Analysis and Emulation
An emulator can be used to implement dynamic analysis. Dynamic Analysis means running a program and seeing whats going on as
it executes, eg as in a debugger It can mean identifying specific behaviors in the program, such as how
the program accesses memory, transfers execution control, or treats network data.
Dynamic analysis using a debugger is prone to anti-debugging tricks, and is very cumbersome when applied in a kernel context.
A robust solution is to perform dynamic analysis from inside an emulator. Hooks are added in the fetch/decode/execute loop of an emulator. When modifying a dynamic binary translator generally,
instrumentation or callbacks are added to the translated code blocks. All the applications for emulation presented, are related to or applications
of dynamic analysis.
Part i)
Reverse Engineering Cisco IOS's Heap Management
Reverse Engineering Cisco IOSwith Dynamips
Dynamips is an open source emulator and binary translator of Cisco hardware running PPC/MIPS IOS images.
Potential future development environment for IOS exploits. Dynamic analysis of IOS*
My experience is with IOS on MIPS IOS MIPS images use an invalid ELF e_machine field. Some IDA (5.2) bugs with MIPS (turn off macros to workaround).
Dynamic analysis, can identify heap management functions in IOS and provide a means to potentially implement Valgrind style heap checkers.
It can also be used to reverse engineer other components of IOS. Dynamic analysis is different to the static approach, and has some
advantages Can be completely automated Since the behavior of the IOS implementation is relatively constant
this method can work across different IOS images, providing new or obsolete features aren't being examined
IOS Heap Management Basics
Well documented public research in developing heap based buffer overflow exploits describes general heap layout.
IOS heap allocated buffers have a header appearing directly before the buffer, and a trailer that follows the buffer.
These 'chunks' form a doubly linked list. Chunk header begins with a known constant
This fact is used later in the analysis.
Dynamic Analysis Approach
Knowing the header constant of a malloc chunk enables us to track memory allocations by intercepting writes to memory of that particular constant.
Heap management is slightly different in a kernel but a kernel or user mode alloc/free still has a set of expected semantics and prototypes.
An alloc(ation) function returns a pointer to an allocated buffer. But don't expect there only to be one argument of the allocation size,
eg kmalloc in Linux has multiple arguments including flags. Free might have multiple arguments also, but one of those arguments
is certainly a pointer to an allocated buffer. By tracking allocations, and checking the behavior of functions, we can
infer the locations of malloc and free.
Identifying Functions with Dynamic Analysis
Finding malloc Track writes to memory that write the constant that identifies a malloc
chunk. Track procedures exits, checking the return value for a pointer to a
known allocated buffer. This return value is the chunk location + chunk header length.
First function to return allocated buffer is malloc, but sample a number of times to be sure.
Finding free Find two malloc calls that return the same memory Free must have occurred between mallocs since logically, allocated
buffers can't overlap. Track procedure calls with an argument matching freed memory, eg
free(ptr) Sample large enough set, common function among samples is free.
Testing the results with a double free and overlapping allocation checker. How can we determine if malloc and free are the only heap management
functions. The solution is to trace those functions while running IOS, building our
own representation of the heap, all the while checking for consistency in our representation.
Certain conditions should always be true in a well managed heap. If any assertions fail catastrophically, our model of the heap is incorrect.
Only allocated memory can be freed. Allocated memory can not overlap.
This results in a checker that can be used to detect double free bugs in IOS, as they happen, much like Valgrind. But IOS checks the consistency of the heap regularly and also during free, so the checker is probably only useful for automated analysis.
Detecting IOS 0-day
Another type of IOS checker could potentially be made to detect 0-day attacks.
IOS exploitation uses corrupted malloc chunks that are subsequently freed.
Freeing the corrupt chunk causes an arbitrary write to memory. The checker could confirm the consistency of header attributes such as the
size of each chunk through the interception of free calls. For more complete coverage, the chunk header could be retrieved and
stored after every malloc, subsequently being verified before free. In a roll-out, honeypots could automatically detect mass 0-day
exploitation and raise alarms of the attack.
Reference Counting.
Tracing malloc and free, shows us conditions where we are freeing the same memory twice, or performing a double free.
Potentially this could indicate a bug in IOS but there are simply too many alerts to be meaningful.
In fact, it turns out that as suspected by other researchers, allocated buffers are reference counted
Before the two double frees is a call to increment the reference count (IncRefCnt) of the buffer, thus causing the first free to simply decrement the count without actually freeing the memory.
MIPS has an atomic addition instruction, used only for incrementing the malloc chunk refcnt.
Any procedure that uses this instruction on a malloc chunk is IncRefCnt.
For other architectures, the refcnt field in the malloc chunk is at a fixed offset, and writes to this address may also indicate the location of IncRefCnt.
MallocLite
Tracing also reveals the appearance of overlapping memory allocations. In later versions of IOS, 'MallocLite' implementation is used. A 64k allocation is used which is subsequently subdivided for use in
allocations <= 128 bytes. This feature may affect the writing of heap exploits and should be taken
into account. If malloc recursively calls itself, requesting 64k of memory, then
MallocLite is allocating this larger block of memory. For tracing, ignoring recursive allocations works.
Cisco IOS TODO
The malloc tracer could potentially be used to implement a Valgrind style MemCheck tool to detect out of bounds heap access.
This could be used alongside fuzzing to provide more accurate detection of vulnerabilities when they happen.
Easy to implement, but the initial attempt resulted in too many false positives.
Problem: There are other functions that have direct access to internal heap structures besides malloc, free and IncRefCnt, eg CheckHeaps.
More reversing is required. If Cisco gave me access to the source, I'm pretty sure I could whack
this out in a week ;-) The MemCheck concept was later successfully implemented for the Linux
Kernel as source code is openly available.
Cisco IOS Summary
By modifying the open source Cisco emulator, dynamips, dynamic analysis of IOS is possible.
Dynamic Analysis of IOS can aid in reverse engineering. Potentially one day we will have Valgrind style IOS memory checking
tool, or in the near future a 0-day detection tool.
Part ii)
Tracing execution and evaluating the capabilities of binaries and potential malware
Tracing and evaluating the capabilities of binaries
Running binary inside a sandboxed environment logging events of interest.
System calls, registry changes, files accessed, process management, services started or stopped etc.
Public websites offer free online services to evaluate binaries and potential malware.
Trace useful for quickly determining what a binary is doing. May help in determining if binary is malicious.
A non emulated approach is to trace the binary using a debugger based tool from userspace within a VM.
Malware almost certain to use anti debugging tricks which may make tracing problematic.
Another approach is to perform the execution inside an emulator. Emulated approach very resistant to modern anti-debugging tricks.
TTAnalyze
TTAnalyze: A Masters thesis that presented a closed source fork of QEMU that logged windows system calls.
Important as other techniques such as automated unpacking are based on similar methods and the thesis clearly describes the implementation.
Windows XP running as a guest, emulated by a fork of QEMU in the host. Host uploads binary to guest using virtual network created by VM. Binary is executed in guest environment. Host monitors execution and logs events of interest.
TTAnalyze concepts
Host emulator intercepts every instruction. It identifies instructions that belong to the process being monitored.
How to know what code is part of the process we wish to monitor? CR3 register (the page directory base address) is unique for each
process. Kernel maintains a process list (EPROCESS) with these addresses. Given a specific process instruction, it may be executing either kernel
code or user code. For our target process, kernel code is when EIP > 0x80000000.
For the target process, it checks EIP, and if it points to a Windows API call it logs the event.
It also logs returning from Windows API calls. To know the addresses of each Windows API call, it uses the PEB
from the target process used to eventually retrieve a list of all loaded DLL's.
The library calls in each DLL is parsed, and their addresses noted.
TTAnalyze Implementation
A component that executes inside the guest system Kernel driver to parse kernel EPROCESS list, to obtain the page
directory address (CR3), and PEB of the target process. RPC mechanism to control guest operations from host
uploading executables to guest Controlling execution of the target process, which is initially started in
a suspended state to allow querying. Querying the pdb/CR3 and PEB kernel driver.
QEMU modifications Identifying the process of interest using the CR3 result from the guest
kernel driver. The PEB is used to established a list of addresses for each windows
API call in a DLL* Identifying entering and leaving windows API calls in the guest, based
on intercepting each instruction and checking EIP.
TTAnalyze Implementation Challenges
Arguments for system calls which reside in virtual memory might be paged out.
QEMU page fault handler detects condition then alters guest code to access target memory, paging it in.
Malware can use the Native API directly. Understanding this requires unofficial documentation of API. Trap native calls by checking each instruction for a OS trap (int 2e or
sysenter).
TTAnalyze Attacks
Malware might evade detection of Windows API calls which is dependant on exact EIP matching.
Vulnerable if malware doesn't jump to the very beginning of a function, eg Caller might implement callee prologue
Malware might detect guest changes. Communication channel between host and guest. Kernel driver component. See Pandora's Bochs (An automated unpacker) implementation with
no guest changes. Malware might detect system emulators
CPU Bugs (in errata) generally not implemented Model Specific Registers implementation different for different CPU
vendors.
Binary Tracing Summary
Existing software that traces binaries using a userland style debugger based tool in a VM, vulnerable to many anti-debugging tricks.
An emulator can present a solution to that problem.
Part iii)
Using emulation for dynamic taint analysis
Dynamic Taint Analysis
A technique used to analyze the the flow of data in a program. Has applications in identifying vulnerabilities as they happen, eg Argos. Has also been used to identify spyware, eg, BitBlaze. Is a general concept that can be used in a number of applications,
including symbolic execution. Traces the flow of data, instruction by instruction, from a source that
generates 'tainted' data, to sinks where the data is used. Variables, registers and memory are tagged as being tainted or clean. Destination operand in instruction becomes tainted when a source operand is
tainted. Sometimes its useful that data can become untainted by certain operations.
Dynamic Taint Analysis in Vulnerability Detection
Dynamic Taint Analysis has been applied for vulnerability detection such as SQL injection, or incorrect use of the Unix exec*() or system() calls which run executables.
Source of user input, that is untrusted data, taints the data. Flow of untrusted data followed by taint analysis. If untrusted data checked in a condition, then input validation deemed to
have occurred, so untaint data. At site of exec*(), system(), or even mysql_query, check that argument is
non tainted. If tainted, then untrusted data assumed to have reached privileged code
and vulnerability has occurred.
Argos: A tool for detecting 0day attacks Uses dynamic taint analysis to detect 0day attacks. An open source fork of QEMU. Detects exploits as they are happening and automatically generates
vulnerability signatures. Vision is of an automatic worm defense system.
Honeypots detect 0day attacks. Generates and delivers vulnerability signatures to intrusion prevention
systems Argos works by dynamic taint analysis of network data which is considered
untrusted. Taints data returned from QEMU emulated network driver.
Exploits detected when their is code redirection under attacker control. If EIP becomes tainted (under the control of the attacker) If EIP points to tainted data. Execve system calls checked for tainted arguments.
Dyanamic Taint Analysis Summary
Dynamic Taint Analysis is a technique used to track the flow of data. Important because it can be used as a general technique in more applied
topics. Has applications including vulnerability detection and is used in places
like symbolic Execution.
Part iv)
Automated Unpacking
Packers
A packer rewrites an executable, wrapping a new layer of code around the original program.
Essentially becomes an executable inside an executable. A packer is used to compress, obfuscate or encrypt the original executable
Today almost all malware is packed. Packers originally used for compression I remember packers (or crunchers) from the early 90's, and had 2
floppy disks full of them, for the Commodore 64! The resulting packed executable consists of a runtime unpacking layer and
a binary blob of the compressed or obfuscated original program. At runtime, the unpacking layer, decompresses the blob writing to
memory the original executable. It then transfers execution back to the original code.
Not all packers follow this behavior. Some packers convert the original executable to PCODE. At runtime the packed executable acts as a VM.
Unpacking
Unpacking is the process of extracting the original executable from a packed image.
The manual approach is to run the packed executable in a debugger, skipping the unpacking stub which writes to memory the original image, and breaking (in the debugger) when execution transfers to the now unpacked image.
A dump of memory, but rebuild the image so its a valid executable again. Requires fixing the Import Address Table. ImpRec can do this.
Debugger scripts can automate the process on specific unpackers by identifying instruction sequences that indicate which stage the unpacking stub is in.
Automated Unpacking
Unpacking can be automated. Run packed executable. Track all memory writes by executable. If execution transfers to a priorly written to memory location, then
unpacking deemed to have occurred. May be necessary to repeat as multiple layers may exist. Public automated unpackers available from Offensive Computing, and
also Pandora's Bochs.
Automated Unpacking Implementation Approaches
Multiple approaches in implementation Use hardware page protection in OS to track writes and execution. Eg
Offensive Computing. This results in high performance. If running inside a virtualized environment like VMWare, VM
might be detected. Offensive Computing recommend using a real goat machine.
Dynamic Instrumentation or complete emulation of packed program to track memory writes and execution.
Offensive Computing use instrumentation approach with Intel PIN framework.
Pandoras Bochs uses the Bochs emulator.
Automated Unpacking using an Emulator Emulation is a mature closed source technology used by AntiVirus
Original usage of emulation was to detect polymorphic virus, but now used for unpacking also.
Typical AntiVirus emulator emulates both the instruction set and parts of the operating system.
This is how I wrote my own automated unpacker and emulator. There are no software licensing problems since the emulator is only a
regular piece of software. Another approach is to use a whole system emulator such as Bochs or
QEMU running an installed OS. Non emulated approaches are more likely to be detected or be suspect to anti-
debugging tricks employed by malware.
Using an AV style Emulator as a CPU checker
While developing my AV style emulator, a need arose to verify the emulation.
I Implemented a program tracer to trace programs in parallel to emulation Tracer needed to automatically evade anti-debugging tricks
Instructions needed to be emulated that would indicate the program was being debugged. (eg, EFlags popf, rdtsc, or software int1 being confused with single stepping)
Library calls also (eg, Process32* which shows debuger in process list, and IsDebuggerPresent)
For each traced instruction, the emulator executes the same instruction. The CPU state from the tracer is verified against the state of the emulator,
and checked for consistency. Some instructions produced differences between emulation and tracing,
not due to a fault of the emulator or tracer. CPU Bugs. Some Instructions not following Intel specifications.
Not setting/clearing processor status flags
Automated Unpacking using an Emulator implementation
Changes to an emulator required involve modifying the software MMU to track memory writes, and checking each instruction to see if the EIP matches any addresses where memory writes have occurred.
Similar problems as TTAnalyze are present in determining what code is part of the target process.
The Renovo unpacker from the BitBlaze project follows the TTAnalyze approach in starting the executable in a suspended state, and then using a kernel driver in the guest to find the page directory base address of the process.
Pandora's Bochs uses an unmodified guest system and instead watches for changes in the CR3 register to identify the target process.
To determine the value of CR3 it takes into account that in kernel mode windows uses the fs register to reference a known structure leading to the EPROCESS list which like TTAnalyze, contains the page directory base address (CR3) of each process.
Attacks against Automated Unpackers and Emulators
Malware might make use of unimplemented emulation of the architecture, instruction set or operating system
For AV emulators, use of obscure libraries. For whole system emulators, detection of the emulator. Malware
might check existence of known CPU errata. Having malware require activation (eg, using the Internet), or only
occasionally activating.
Attacks (cont): Virtual Machine Packers
Packer translates executable into PCODE. At runtime, PCODE is decoded and executed in the style of a virtual
machine. PCODE can be polymorphic. This type of packer doesn't follow the 'write to memory then execute'
algorithm. Eg, TheMida, but fortunately these packers are not as common in current
malware. No automated method of unpacking against an unknown packer of this
type.
Automated Unpacking Summary
Automated unpacking works on a theory of intercepting execution on priorly written to memory addresses.
Multiple approaches to implementation; emulation has some advantages. Automated unpacking doesn't work on VM based unpackers.
Part v)
Using emulation to design and implement symbolic execution
Symbolic Execution
A technique used to analyze programs. For unknown input to a program, it maintain generalized information on
program state, systematically exploring program paths. Really a definition for mixed symbolic execution.
Execution occurs, by emulating instructions and using symbolic formula instead of concrete data for user defined input.
Example symbolic data can be network packet contents, program arguments, file contents etc
Symbolic formula contain information on all program states on that program path for arbitrary user input, that is, all the values the data can possibly hold as held true by the symbolic formula.
Bug finding is equivalent to solving the equations. Eg, Is this pointer being dereference ever equal to 0, given arbitrary
user input. And if so, what is the user input that generates that bug.
SMT Based Constraint Solvers
Symbolic equations are generated for instructions that have symbolic arguments.
Conditional instructions generate equations which are constraints (eg, x < 10)
Equations handled by Satisfiability over Modulo Theory (SMT) Solvers. Efficient SMT based solvers are a relatively new achievement in the past
decade. Annual SMT competition pits solvers against each other. Microsoft has their own solver which is free to use, but not open
source. A number of open source solvers available.
SMT Solver can be queried, given a set of equations and constraints, to see if certain queried constraints are true.
Can easily determine if symbolic pointer is null.. SMT solvers can also generate concrete solutions from symbolic
equations
Applications of Symbolic Execution
As a Bug checker Dawson Englers closed source C checker ExE which could detect
buffer overflows, null pointer dereferences and divisions by zero. The open source Catchconv – which doesn't explore program paths,
but checks assertions on a given set of input using symbolic execution to find signedness bugs.
Intelligent fuzzing Symbolic Execution can automatically enumerate the paths and data in
a program that fuzzing normally misses, aiming towards complete automated code coverage.
Eg, closed source Microsoft Sage research Tracing and evaluating the capabilities of binaries
The closed source Bitblaze projects implements BitScope which is in a similar vein to TTAnalyze except it symbolically explores the many program paths in potential malware to find its capabilities.
Symbolic Execution Implementation
Emulator runs program, instruction by instruction, generating symbolic equations for instructions when a source operand is symbolic, such as the symbolic equation ebx=eax + 10.
In an instruction, if a source operand is symbolic, destination becomes symbolic.
This is implemented using Dynamic Taint Analysis At conditional instructions, two possible equations, the condition being
true, or the condition being false. Symbolic Execution explores each path separately.
A symbolic constraint representing the conditions truth is given to each path, eg (x > 10 and x <= 10).
Feasibility, that is if an equation can be satisfied as true, of each path is determined by SMT solvers.
Symbolic Execution Challenges
Symbolic Execution may never terminate in the presence of loops, so loops must be simplified, typically through unrolling.
Symbolic Execution therefore is not complete. Path Explosion: Dealing with functions like strcmp with symbolic input,
has many possible paths; an exponential number of paths for the size of the string.
BitBlaze approach: Hard code 'function summaries' to deal with common library functions.
Dealing with symbolic pointers. Dynamic taint analysis has trouble determining the target memory that
becomes tainted if a pointer is symbolic. Requires SMT solver to determine concrete solutions of pointer.
SMT solver support used for target architecture may not be complete No public solvers support floating point.
Symbolic Execution Summary
Symbolic execution is a relatively new method to analyze programs. Applications include bug checkers, smart fuzzers, and binary evaluation. I believe symbolic execution has a big part in the future of automated
analysis.
Part vi)
Detecting Runtime Errors in Programs
Valgrind
Valgrind is a heavyweight dynamic binary instrumentation framework. Most well known for the MemCheck checker. Memcheck used as a bug checker for incorrect heap use or access. Also detects uninitialized variable use.
Translates machine code to IR, then allows instrumentation, with modules that implement runtime checkers.
Valgrind's Memcheck can detect out of bounds or invalid heap access and tracks what addresses can be accessed by maintaining a 'shadow memory' mirroring allocations on the heap.
For each address in shadow memory, also stores weather its initialized or not.
Then checks all guest memory references belong to the shadow memory using IR instrumentation.
Valgrind's MemCheck with uninitialized variables
Uninitialized variable checker implemented using dynamic taint analysis. Newly allocated memory and new stack frames considered tainted. Initializing data untaints it.
Alert when using tainted/uninitialized data. Naive implementation causes false positives.
Memcpy of padded structures or memcpy of structures with uninitialized members causes false positives.
Fixed by warning only when using uninitialized variables in system calls, conditions or being dereferenced as a pointer.
Detecting Runtime Heap Errors in the Linux Kernel
Tools that have similar designs or aims to detect some classes of heap errors in the Linux Kernel.
KEFence (Linux) / MemGuard (FreeBSD) Detects overflows (and underflows for KEFence, but not both at the
same time) of heap buffers. Allocates a guard page next to the allocated buffer that page faults on
any access. Only detects overflows, not arbitary invalid access.
KmemCheck (Linux) Used to Detect uninitialized variable bugs. Maintains a shadow memory indicating state of data being initialized
or not. Page faults on all heap access, then checks shadow memory against
access. UML + Valgrind
Doesn't seem active, and source unavailable :(
Linux Kernel MemCheck
My own runtime checker that detects out of bounds heap access in the Linux Kernel.
Not Valgrind's MemCheck – I named it poorly I know. Tested under Linux 2.6.26 using a Windows Vista Cygwin host. Implemented as a C++ fork of QEMU. Dumps kernel stack trace on guest access violation Only reports when a memory access violation occurs, much like Valgrind.
Not a static analysis tool. Host maintains 'shadow memory' of guest Linux Kernel heap that
identifies valid heap addresses. The shadow memory is created by intercepting the heap management
functions in the Linux kernel and building a representation of the guest heap.
MemCheck validates all memory access against this shadow memory (like Valgrind).
Except in heap management functions like kmalloc, kfree etc.
Linux Kernel Heap Management
Linux has had several memory allocators, the latest Linux kernels now using the “slub” allocator.
MemCheck only supports the latest “slub” allocator. There are also three internal allocators in Linux that use the heap.
The Page Allocator, using the buddy allocator internally, which only handles allocations of sizes being a predetermined multiple of the page size.
The page allocator can be called directly or indirectly from the slub allocator.
The Slub Allocator which handles allocations of varying sizes by dividing up a “slab” that originates from the page allocator.
The BootMem Allocator which uses a simpler algorithm than the other allocators during boot time only.
Linux Kernel Heap Tracing and Guest Linux Implementation
MemCheck must trace the kernel allocator functions to properly create its shadow memory.
However tracing an unmodified Linux guest presents problems. The Page Allocator does not always return the address of the allocated
page contents, but returns a structure of the page description instead. The Slub Allocator defines kmalloc as an inline function which can't
be intercepted using a compile time symbol address. Following internal logic can be difficult, such as kmalloc using the
page allocator internally. The solution is to use a modified guest Linux Kernel that uses
instrumentation of the allocators that MemCheck can easily intercept
MemCheck QEMU implementation
QEMU was modified to implement MemCheck. MemCheck is written in C++ running in a Windows host, so I ported
QEMU 0.9.1 to compile under g++. In hindsight, porting was not necessary and not worth the effort. I also backported some patches that cause 0.9.1 to fail in windows.
QEMU has an optimization of merging basic blocks in a translation block. I needed basic block granularity to correctly intercept the beginning of functions so this QEMU optimization was turned off.
A tracer was implemented to track functions using a callback interface on function entry or exit.
By tracing the heap management code, a simple shadow memory was constructed using C++ STL maps for the implementation.
The software MMU in QEMU was modified to check the memory access was a valid address in the shadow memory.
MemChecking the Linux Kernel
The Linux Test Project (LTP) contains 3000+ tests for the Linux Kernel which exercise much of the core kernel code.
Ran the default test suite on Linux 2.6.26.3 using MemCheck. MemCheck is slow, but still allows for interactive sessions. Fedora Linux takes 30+ minutes to boot. Let the testsuite to run overnight
No out of bounds access detected. Reran the testsuite again using slub debugging which in combination to
MemCheck, may result in more bugs being detected. Again, no out of bounds access detected.
While no immediate bugs were identified in 2.6.26.3, MemCheck may be used against future kernel releases, possibly as part of an automated test suite, or used to aid kernel debugging and development.
MemCheck Limitations
Because MemCheck is based on QEMU, very little hardware is emulated so most of the Linux driver code is not tested.
Buffer overflows don't necessarily result in memory access using invalid heap addresses.
A slab based allocator fits heap allocations next to each other, so buffers overflow into adjacent and valid heap allocations.
A solution is to boot Linux using the slub_debug kernel option which separates heap objects using a redzone.
If MemCheck generates a report from a vulnerable kernel module, only kernel addresses are given in the stack trace no symbolic names are used.
MemCheck TODO
A solution to the adjacent buffer problem is to associate every heap access with its original allocation by tracking heap pointers using dynamic taint analysis.
This use of dynamic taint analysis could also be applied in userland, as a Valgrind checker.
Dynamic taint analysis can also be the basis of tracking uninitialized variable usage without the false positives currently associated with kmemcheck.
Dynamic taint analysis could also be used to implement garbage collection, which could be used to identify memory leaks at the exact location of each leak.
Symbol names for addresses in kernel modules!
MemCheck Packages
http://silvio.cesare.googlepages.com/ For the package http://silviocesare.wordpress.com/ For commentary on some of
MemCheck's internals.
Runtime Error Detection Summary
Existing tools for runtime error detection include Valgrind which detects userland heap bugs.
Tools for the kernel exist such as kmemcheck which detects uninitialized variables.
MemCheck is a new tool to detect heap bugs in the Linux Kernel, and operates similar to Valgrind.
