intrusion detection using sequences of system calls
Post on 02-Jan-2016
37 Views
Preview:
DESCRIPTION
TRANSCRIPT
Overview
Focus: privileged processesDiscriminator: system call sequencesBuilding a database: defining “normal” Detecting anomalies: how to measureResults: promising numbersConcerns: remaining doubtsExtensions of research: Jones, Li & Lin
Inspiration
Human immune systemRecognition of self Rejection of nonself
How would we describe “self” for a software system, or a program?
Focus and Motivation
Focus on privileged processes Exploitation can give a user root access They provide a natural boundary
e.g. telnet daemon, login daemon
Privileged processes are easier to trackSpecific, limited functionStable over timeContrast with the diversity of user actions
Where do we look?
Need to distinguish when: Privileged process runs normally Privileged process exhibits an anomaly
The discriminator is the observable entity used to distinguish between these two
Use sequences of system calls as the discriminator, the signature
How much detail?
Discriminator is sequences of system calls Simple temporal ordering is chosen Ignore parameters Ignore specific timing information Ignore everything else!
Why? As much as possible, work with simple assumptions
Is it “enough”?
Is it enough detail?
Does the discriminator include enough detail for this hypothesis to hold? Answer seems to be yes !
Extra complication: due to the variability in configuration and use of individual systems, the set of “normal” sequences of system calls will be different on different systems
Design Decisions
Remember temporal ordering of calls Not total sequence, but sequences of length k
What size should k be? Long enough to detect anomalies, short as
possible Empirical observation: length 6 to 10 is
sufficientSo “self” is a database of (unordered) short
call sequences
Building the “normal” database
Synthetic Assurance that the normal database
contains no intrusions; reproducible But does not reflect any particular real
user activityActual use
Necessary to generate from actual use in order to have a unique “self”
How long to accumulate? Is it clean?
The normal database
Database of normal sequences does not contain all legal sequences If it did, anomalies would not be
detected Some rare sequences will not be used
during database initializationDatabase is stored as a forest to
save space
Signature Database Structure (length 3)
fopen fread strcmp
fread strcmp strcmp
strcmp strcmp fopen
strcmp fopen fread
fopen fread strcmp strcmp fopen fread strcmp
fopen
fread
strcmp
fread
strcmp
strcmp
strcmp
fopen
strcmp
fopen
fread
Derive Robust Signature Database
Robust Signature Database
0
100
200
300
400
500
600
0 2000 4000 6000 8000 10000
Total Seqences Scanned
Dat
abas
e S
ize
Detecting anomalies
A call sequence not in the database is an anomalous sequence
Strength of that anomalous sequence is measured by “Hamming distance” to the closest normal sequence (called dmin)
Any call trace with an anomalous sequence is an anomalous trace
Detecting anomalies
Strength of an anomalous trace is the maximum dmin of the trace normalized for the value of k (length of sequences in the database): ŜA = max{dmin values for the trace} / k Value is between 0 and 1
By adjusting the threshold value for ŜA, false positives can be reduced
Efficiency
Complexity of computing dmin O(k(RAN + 1))
k is sequence length, RA is ratio of anomalous to normal sequences, N is the number of sequences in the database
dmin is calculated after every system call The constant associated with this algorithm
is very important Not yet running in real time
Results (synthetic)Sanity test: If different programs are not
distinguishable, anomalies within one program will certainly not be either
Easy to distinguish between programs; mismatches on well more than 50% of the instruction sequences (and ŜA >= 0.6)
All intrusions (both attempted & successful) produced anomalies of varying strengths
Results (real environment)
The conjecture of unique normal databases Experiments in two configurations (at
UNM and MIT) had very different databases for the same program (lpr)
Is this typical?
Closing concerns
False positives vs false negatives If forced to choose, UNM prefers to have
false negatives because layering can mitigate
Saw 1 per 100 print jobs (lpr) Due to system problems
Is ŜA a good measure? It could help generate false positives Single extra system call might make ŜA = 0.5
Signature Length Has Little Effect
Illustrated by two attacks on Apache
Varied sequence length from 2 to 30
We chose length 10 to have margin of error
0
0.2
0.4
0.6
0.8
1
1.2
0 10 20 30 40Sequence Length
Nor
mal
ized
Ano
mal
y Si
gnal
Effectiveness: Buffer Overflow
Successfully detected buffer overflow attacks against wu-ftpd
Work well because attacker code adds new sequences of library calls
#Mismatches
%Mismatches
Normalized Anomaly Signal
Stack Overwrite 467 3.5 0.7
Realpath Vulnerability
569 2.7 0.6
High normalized anomaly signals indicate attacks
Effectiveness: Denial of Service
Simulated DOS attack that uses up all available memory
As attack progresses, library calls requesting memory return abnormally and are re-issued
DOS attack caused application to invoke new library call, fsync
Program - vi #Mismatches
%Mismatches
Normalized Anomaly Signal
Normal Run 0 0 0
DOS Attack 101 2.6 0.6
No intrusion detected
High normalized anomaly signal indicates attack
top related