network intrusion detection applications and research like zhang

Network Intrusion DetectionNetwork Intrusion DetectionApplications and ResearchApplications and Research

Like ZhangLike Zhang

OutlineOutline

• Recent debate over NIDS

• Introduction to NIDS

• A survey of current NIDS products

• Research on anomaly NIDS

• Conclusion

Is NIDS dead?Is NIDS dead?

“Hype Cycle for Information Security”

Gartner Report, 2003

• False positives and negatives

• Requiring full-time monitoring (24 hours a day, seven days a week, 365 days a year)

• Market failure

• Will be obsolete by 2005

Current SituationCurrent Situation

• Intrusion Detection evolves into Intrusion Prevention

• New types of IDS come into play (distributed IDS, application-based IDS,etc.)

• NIDS is applied to firewall, anti-virus system, optional plug-in for server-side program, or deployed as a standalone product

NIDS TechniquesNIDS Techniques

• Signature-based

• Anomaly-based

• Stateful detection

• Application-level detection

Signature-base NIDSSignature-base NIDS

Similar to the traditional anti-virus applicationsExample:Martin Overton, “Anti-Malware Tools: Intrusion Detection Systems”,

European Institute for Computer Anti-Virus Research (EICAR), 2005

Signature found at W32.Netsky.p binary sample

Rules for Snort:

Anomaly DetectionAnomaly Detection

• Already used by industry--Protocol Anomaly

--Statistical/Threshold based

• In Research--Data mining

Protocol Anomaly DetectionProtocol Anomaly Detection

Based on the well established RFCs

Focus on the packet headerExample:--All SMTP commands have a fixed maximum size. If the size exceeds

the limit, it could be a buffer overflow or malicious code inserting

attack

--SYN flood attack: attacker sends SYN with fake source address

--Teardrop attack: fragmented IP packets with overlapped offset

Threshold basedThreshold based

Using training data to generate a statistical

model, then select proper thresholds for

network environment (traffic volume, TCP

packet count, IP fragments count, etc.)

-- usually used as an complementary tool

Stateful IDSStateful IDS

• No practical Solutions

• Very simple implementing

Example:

Snort uses patter matching in continuous Packets.

Traditional signature rules: “pattern1” “pattern1 || pattern2”

The rule now can be defined as: “pattern1.*pattern2”

Application-level IDSApplication-level IDS

Focus on specific services or programs

(Web Server, Database, etc.)

Example

--Monitoring all invocation for Microsoft RPCs

--Analyze HTTP request for malicious query strings

Products:

--mod_security: an optional IDS component for Apache

Web Server

IDS TodayIDS TodayProducts and ApplicationsProducts and Applications

• Snort

• McAfee Intrushield

• ISS RealSecure

• Cisco IPS

• Symantec IDS

SnortSnort

• Open Source, since 1998

• Used by many major network security products

• Signature-based (more than 3000)

• Simple IP header protocol anomaly detection

• Simple stateful pattern matching

McAfeeMcAfee

• Profile-based anomaly detection--Manually create profile

--Create profile by self-learning through a training period

• Using profile plus threshold for defending against DOS and DDOS

• Inspect encrypted traffic by collecting the server side private keys

ISS RealSecureISS RealSecure

• About 2000 signatures• Application-based approach

--identifying any possible exploit to the published vulnerabilities of MS RPC, IIS, Apache, Lotus, etc.

• Additional support for P2P,Instant Messengers• Virtual Prevention System

--a virtual environment to examine the execution of a file in order to find any possible malicious behaviors

• Support for IPv6--Detect possible backdoors which enable the IPv6 of a system (usually off)

Cisco IPS produtcsCisco IPS produtcs

Protocol decoding

Threshold based property checking

Signature matching

Protocol Anomaly Detection

Checking file behaviors by intercepting all calls to the system resources

SymantecSymantec

• Multi-steps (protocol, vulnerability, signature, DOS, traffic, evasion check)

• Unique feature: evasion check

e.g. request “/index.html” can be replace with “/%69nd%65x.html” to evade the signature matching

Summary of Current ProductsSummary of Current Products

Snort McAfee Intrushield

ISS RealSecure

Cisco IDS Symantec IMUNE

Signature General x x x x x

Application based x

Anomaly Detection

Profile-based x

Vulnerability-based x x

Statistical-based x x x

Protocol-based x x x

Self-learning x

Application specific x x

Stateful x x

Behavior x x

Encrypted Traffic Detection x

IPv6 Support x

Challenges for NIDSChallenges for NIDS

• High false positives-- FP of 0.1% means a normal packet will be misclassified as an alert for every 1000 normal packets, which is about one error alert per minute on a 100M network

• Zero day attack (unknown attack)--Most current products rely on signature-based detection, difficult to detect new attacks.

• Poor at automatically preventing ability--Human interaction is required when attack is detected

Research on Intrusion DetectionResearch on Intrusion Detection

• Columbia University--Data mining based (since 1997)

• University of California at Santa Barbara--Service Specific (HTTP)--Stateful IDS

• Florida Institute of Technology--Protocol Anomaly (Statistical based)

• University of Minnesota--MIND (Minnesota Intrusion Detection System)

Columbia Univ. IDSColumbia Univ. IDS

• 1997, Applied RIPPER rule learning algorithm on UNIX system calls monitoring for malicious events detection

• 1998, Applied the algorithm on off-line network traffic data (clean training data)

• 2000, Applied EM and clustering algorithm for dealing with noisy dataset

• 2001, Developed an complete experiment NIDS based on those algorithms.

• 2004, New approach towards payload anomaly detection

Implementing ProcedureImplementing Procedure

Pre-Processing

Feature construction

Apply RIPPER algorithm

Process raw packet data

Create statistic features

Rule learning

Wenke Lee, Sal Stolfo, and Kui Mok., “A Data Mining Framework for Building Intrusion Detection Models”, Proceedings of the 1999 IEEE Symposium on Security and Privacy, Oakland, CA, May 1999

Pre ProcessingPre Processing

SYN flood attack

Feature ConstructionFeature Construction

(service=http, flag=S0, dst_host=victim),

(service=http, flag=S0, dst_host=victim)

-> (service=http, flag=S0, dst_host=victim)

[0.93, 0.03, 2]

93% of the time, after two http connections with S0

flag are made to host victim, within 2 seconds from

the first of these two, the third similar connection is

made, and this pattern occurs in 3% of the data

RIPPLE RulesRIPPLE Rules

smurf :- service=ecr_i, host_count >= 5,

host_srv_count>=5( if the service is icmp echo request, and connections with the same

destination host are at least 5, and connections with the same service

are at least 5,then it is a smurf/DOS attack)

satan :- host_REJ_%>=83%, host_diff_srv_% >=

87%( for connections with the same destination host, if the rejection rate is at least

83%, and the percentage of different services is at least 87%, then it is a

santa/PROBING attack)

Experiment ResultsExperiment Results

Applied on DARPA’98 Intrusion Detection Evaluation Data Set

Payload based ApproachPayload based Approach

K. Wang, S. J. Stolfo, “Anomalous Payload-based Network

Intrusion Detection”, RAID 2004• Construct the statistical model for all bytes in the header• Use Mahananobis distance to measure the difference

Problems:• Clean training data is required• False positive (unacceptable)

Service Specific IDS by UCSBService Specific IDS by UCSB

V.Giovanni et al at University of California at Santa Barbara

Since 2002

• Application level

• Focuses on HTTP request

• HTTP request analyzing

• Constructing models for important fields in the request instead of all bytes of the payload (Columbia payload approach)

Sample RequestSample Request

RequestGET /scripts/access.pl?user=johndoe&cred=admin

Properties for DetectionRequest Type: e.g. GET

Request Length: e.g. Length(“GET /scripts/access.pl?user=johndoe&cred=admin”)

Payload Distribution

Request TypeRequest Type

Assumption: If a rare used request type was found, it is very possible it

will initiate malicious activity

Anomaly Score:AStype=-log2(p[type])

P[type] stands for the probability of a certain type

Request LengthRequest Length

Assumption:The request length should not vary much of a certain type.

Otherwise, it is probably caused by some attacks

(e.g. overflow)

Anomaly Score:ASlen=1.5(1-)/(2.5*)

P[type] stands for the probability of a certain type

Characters DistributionCharacters Distribution

256 ASCII Characters

e.g. “passwd” -> “112 97 115 115 119 100”

Distributions: {0.33, 0.17, 0.17, 0.17, 0.17}

2=f(Oi, Ei) (i corresponds from segment 0 to 5)

Aspd= 2*(15/L) (L stands for the payload length)

Segment 0 1 2 3 4 5

ASCII Value 0 1-3 4-6 7-11 12-15 16-255

Final Anomaly ScoreFinal Anomaly Score

AS=0.3*AStype + 0.3*ASlen+0.4*ASpd

Later Research at UCSBLater Research at UCSB

Structure Inference with Markov Model

Other Properties UsedOther Properties Used

• Token Finderif the query parameter is drawn from known candidates

• Attribute Presence or absencemalicious crafted request usually ignore the order of parameters

• Access Frequency

• Invocation order

• Request time interval

Experiment ResultsExperiment Results

• Tested at UCSB campus network and Google

• False positive 0.06%

Major cons:

Limited to HTTP service

Packet Header Anomaly DetectionPacket Header Anomaly Detection

Packet Header Anomaly Detection (PHAD)developed by Florida Institute of Technology since 2001

Basic Assumption:If an event x happened n times with r different results in the

training period, the probability of a novel data is r/n

ImplementingImplementing

Step 1:Assign the novel data probability to important fields

of the packet header (protocol type, flags, etc.)

Step 2:Adding all the novel data probability together as a

threshold

MINDSMINDS

MINDS (Minnesota Intrusion Detection System)

Statistic outlier-based anomaly detection

Compared 5 outlier-based scheme:• K-th nearest neighbor• Nearest neighbor• Mahalanobis-distance based• Local Outlier Factor (LOF)• Unsupervised SVMs

Comparison ResultComparison ResultA. Lazarevic, et al, “A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection”, Proceedings of the 3rd SIAM Conference on Data Mining, San Francisco, 2003

Some Emerging ApproachesSome Emerging Approaches

• SVMs(unsupervised and supervised)

• PCA

• PCA + SVMs

• Neural Network

ConclusionConclusion

• Signature based approaches still play the major part in practical IDS

• Anomaly detection has only very limited success• New approaches are proposed everyday, but

false positive and detection rate are still the major problem

• Various mechanisms should work together for maximum success