1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Root Kit Protection and Detection

Shane CanonOctober 23 2003

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Outline

● Protection – St Michael– What it does– How it works– Strengths and weaknesses

● Detection– Statistical detection of modified system calls– Strengths and weaknesses

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

St Michael

● Source Forge Project– http://sourceforge.net/projects/stjude– Extension of the St Jude Project

● Attempts to protect kernel from root kitting through various means– MD5 several critical memory regions– Makes backup of kernel text– Timers run periodic checks– Checks automatically run during module insertion

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Monitoring Critical Regions

● MD5 are calculated for several critical, non-volatile regions– Kernel text– System call table

● Timers run periodic calculations of the current MD5's and compare them to the saved values

● Several calls automatically trigger a check– Exit– Module create, delete, and init

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Recovering from attempts

● Backups are made of kernel text● Backup is encrypted through simple means● If changes are detected, the module will attempt

to decrypt the backup and restore it.● If too many attempts are detected, the system

automatically reboots

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Other Measures

● Module goes to some lengths to conceal itself– Decrypts several backups– During module initialization, some data structures

and text are erased– The module is “cloaked” by removing it from the

linked list of loaded modules● Implements checks of some file system data to

detect changes (possibly by writing to the raw device).

● Implements immutable files to prevent modification of files like init, kernels, insmod, etc.

● Protects kmem

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

StMichael – Strengths and Weaknesses

● Strengths

– Significantly raises the bar to trying to insert malicious code into the kernel.

– Protection of kmem makes it more difficult to insert malicious code

– Hashes detect typical changes performed by a root kit

– Any failed attempt will trigger warnings and log messages● Weakness

– If attacker knows StMichael is loaded, they can use a kernel module to try and determine locations of key SM data structures and disable it.

– As new root kit methods are designed, St Michael has to be updated to protect against those.

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Root Kit Detection

● Phrack Article– Volume 11, Issue 59, Article 10 - Execution path

analysis: finding kernel based rootkits by Jan K. Rutkowski

● Describes method for detecting root kits based on number of operations required to perform standard systems calls

● Proof of concept – PatchFinder implements the methods JR describes

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Root Kit Detection – Intro

● Current Kernel based root kits typically use modified system calls to hide malicious files, network connections, and processes.

● By their very nature these modified system calls perform various checks before calling the unmodified system call.

● Although these checks are usually simple, they do add additional instructions.

● If these additional instructions can be statistically distinguished, this can be used to detect a modified kernel.

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Detection – Instruction Counting

● If the processor allows, the instructions counted for performing various system calls can be counted.

● Intel ia32 supports a single step mode. However, the trace flag is automatically cleared when the processor receives an interrupt or exception.

● PatchFinder uses ptrace flag in processor descriptor to determine if the CPU trace flag should be enabled in kernel mode.

● Modifies system call entry point to test ptrace flag.

● Modifies sys_ptrace to retrieve the number of instructions.

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Detection – Testing

● Once the module is loaded, a base line can be created for a given system and kernel.

● Testing done with some published kernel root kits (adore, SucKIT, etc) are detected one way or another– Most root kits will show instruction counts that are

significantly different than the baseline.– SucKIT is detected because no results are returned

since the modified sys_ptrace is never called.

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Detection – Strengths and Weaknesses

● Strengths– Once again it makes it much harder to implement

a rock solid root kit● Weaknesses

– The malicious code is already loaded. If patch finder (or a similar method) is expected, the malicious code can attempt to return the expected values. This might involve testing the trace flag. However, this would be pretty tricky and tedious and any mistakes would trigger an alarm.

1

Future

Technologies

Group

Shane Canon, canon at nersc dot gov Summer Linux Kernel Class

Conclusions

● StMichael and PatchFinder can significantly raise the bar for implementing a fool proof kernel root kit.

● However, root kits are moving targets, so a static approach is unlikely to work forever.