intrusion detection & network forensics

1

Intrusion Detection&

Network Forensics

Lucius L. Millinder Jr.

security@secureitconsulting.usChief Technology Officer

Secure-IT Consulting, Inc.

2

An ounce of prevention is worth a pound of detection

3

Why Talk about IDS?

• Emerging new technology– Very interesting

...but...– About to be over-hyped

• Being informed is the best weapon in the security analyst’s arsenal– It also helps keep vendors honest!

4

What is an Intrusion?!

• Difficult to define– Not everyone agrees– This is a big problem

• How about someone telneting your system?– And trying to log in as “root”?

• What about a ping sweep?• What about them running an ISS scan?• What about them trying phf on your webserver?

– What about succeeding with phf and logging in?

5

What is IDS?

• The ideal Intrusion Detection System will notify the system/network manager of a successful attack in progress:– With 100% accuracy– Promptly (in under a minute)– With complete diagnosis of the attack– With recommendations on how to block it

…Too bad it doesn’t exist!!

6

Objectives: 100% Accuracy and 0% False Positives

• A False Positive is when a system raises an incorrect alert– “The boy who cried ‘wolf!’” syndrome

• 0% false positives is the goal– It’s easy to achieve this: simply detect

nothing

• 0% false negatives is another goal: don’t let an attack pass undetected

7

Objectives: Prompt Notification

• To be maximally accurate the system may need to “sit on” information for a while until all the details come in– e.g.: Slow-scan attacks may not be

detected for hours– This has important implications for how

“real-time” IDS can be!– IDS should notify user as to detection lag

8

Objectives: Prompt Notification (cont)

• Notification channel must be protected– What if attacker is able to sever/block

notification mechanism?– An IDS that uses E-mail to notify you is

going to have problems notifying you that your E-mail server is under a denial of service attack!

9

Objectives: Diagnosis

• Ideally, an IDS will categorize/identify the attack– Few network managers have the time to

know intimately how many network attacks are performed

• This is a difficult thing to do– Especially with things that “look weird” and

don’t match well-known attacks

10

Objectives: Recommendation

• The ultimate IDS would not only identify an attack, it would:– Assess the target’s vulnerability– If the target is vulnerable it would notify the

administrator– If the vulnerability has a known “fix” it

would include directions for applying the fix

• This requires huge, detailed knowledge

11

IDS: Pros

• A reasonably effective IDS can identify– Internal hacking– External hacking attempts

• Allows the system administrator to quantify the level of attack the site is under

• May act as a backstop if a firewall or other security measures fail

12

IDS: Cons

• IDS’ don’t typically act to prevent or block attacks– They don’t replace firewalls, routers, etc.

• If the IDS detects trouble on your interior network what are you going to do?– By definition it is already too late

13

Paradigms for Deploying IDS

• Attack Detection

• Intrusion Detection

14

InternalNetworkInternet

Routerw/somescreening

Firewall

DMZNetwork

WWWServer

Desktop

Attack Detection

IDS detects (and counts) attacks againstthe Web Server and firewall

IDS

15

Attack Detection

• Placing an IDS outside of the security perimeter records attack level– Presumably if the perimeter is well designed

the attacks should not affect it!– Still useful information for management (“we

have been attacked 3,201 times this month…)

– Prediction: AD Will generate a lot of noise and be ignored quickly

16

InternalNetworkInternet

Routerw/somescreening

Firewall

DMZNetwork

WWWServer

Desktop

Intrusion Detection

IDS detects hacking activity WITHINthe protected network, incoming or outgoing IDS

17

Intrusion Detection

• Placing an IDS within the perimeter will detect instances of clearly improper behavior– Hacks via backdoors– Hacks from staff against other sites– Hacks that got through the firewall

• When the IDS alarm goes off, it’s a red alert

18

Attack vs Intrusion Detection

• Ideally do both

• Realistically, do ID first then AD– Or, deploy AD to justify security effort to

management, then deploy ID (more of a political problem than a technical one)

• The real question here is one of staffing costs to deal with alerts generated by AD systems

19

IDS Data Source Paradigms

• Host Based

• Network Based

20

Host Based IDS

• Collect data usually from within the operating system– C2 audit logs– System logs– Application logs

• Data collected in very compact form– But application / system specific

21

Host Based: Pro

• Quality of information is very high– Software can “tune” what information it

needs (e.g.: C2 logs are configurable)– Kernel logs “know” who user is

• Density of information is very high– Often logs contain pre-processed

information (e.g.: “badsu” in syslog)

22

Host Based: Con

• Capture is often highly system specific– Usually only 1, 2 or 3 platforms are

supported (“you can detect intrusions on any platform you like as long as it’s Solaris or NT!”)

• Performance is a wild-card– To unload computation from host logs are

usually sent to an external processor system

23

Host Based: Con (cont)

• Hosts are often the target of attack– If they are compromised their logs may be

subverted– Data sent to the IDS may be corrupted– If the IDS runs on the host itself it may be

subverted

24

Host Based IDS

• Signature log analysis– application and system

• File integrity checking– MD5 checksums

• Enhanced Kernel Security– API access control– Stack security

• Network Monitoring Hybrids

25

Host Based IDS Limitations

• Places load on system

• Disabling system logging

• Kernel modifications to avoid file integrity checking (and other stuff)

• Management overhead

• Network IDS Limitations

26

messages

xfer

access_log

secure

sendmail

27

messages

xfer

access_log

secure

sendmail

OneSecurity

Log

28

Network IDS• Searches for patterns in packets• Searches for patterns of packets• Searches for packets that shouldn't be there• May ‘understand’ a protocol for effective

pattern searching and anomaly detection• May passively log, alert with SMTP/SNMP

or have real-time GUI

29

Network IDS Limitations

• Obtaining packets - topology & encryption

• Number of signatures

• Quality of signatures

• Performance

• Network session integrity

• Understanding the observed protocol

• Disk storage

30

/cgi-bin/phf

Jane usedthe PHFattack!

31

NMAP

Jane dida portsweep!

32

Network Based IDS

• Collect data from the network or a hub / switch– Reassemble packets– Look at headers

• Try to determine what is happening from the contents of the network traffic– User identities, etc inferred from actions

33

Network Based: Pro

• No performance impact

• More tamper resistant

• No management impact on platforms

• Works across O/S’

• Can derive information that host based logs might not provide (packet fragmenting, port scanning, etc.)

34

Network Based: Con

• May lose packets on flooded networks

• May mis-reassemble packets

• May not understand O/S specific application protocols (e.g.: SMB)

• May not understand obsolete network protocols (e.g.: anything non-IP)

• Does not handle encrypted data

35

IDS Paradigms

• Anomaly Detection - the AI approach

• Misuse Detection - simple and easy

• Burglar Alarms - policy based detection

• Honey Pots - lure the hackers in

• Hybrids - a bit of this and that

36

Anomaly Detection

• Goals:– Analyse the network or system and infer

what is normal– Apply statistical or heuristic measures to

subsequent events and determine if they match the model/statistic of “normal”

– If events are outside of a probability window of “normal” generate an alert (tuneable control of false positives)

37

Anomaly Detection (cont)

• Typical anomaly detection approaches:– Neural networks - probability-based pattern

recognition– Statistical analysis - modelling behavior of

users and looking for deviations from the norm

– State change analysis - modelling system’s state and looking for deviations from the norm

38

Anomaly Detection: Pro

• If it works it could conceivably catch any possible attack

• If it works it could conceivably catch attacks that we haven’t seen before– Or close variants to previously-known

attacks

• Best of all it won’t require constantly keeping up on hacking technique

39

Anomaly Detection: Con

• Current implementations don’t work very well– Too many false positives/negatives

• Cannot categorize attacks very well– “Something looks abnormal”– Requires expertise to figure out what

triggered the alert– Ex: Neural nets can’t say why they trigger

40

Anomaly Detection: Examples

• Most of the research is in anomaly detection– Because it’s a harder problem– Because it’s a more interesting problem

• There are many examples, these are just a few– Most are at the proof of concept stage

41

Misuse Detection

• Goals:– Know what constitutes an attack– Detect it

42

Misuse Detection (cont)

• Typical misuse detection approaches:– “Network grep” - look for strings in network

connections which might indicate an attack in progress

– Pattern matching - encode series of states that are passed through during the course of an attack

• e.g.: “change ownership of /etc/passwd” -> “open /etc/passwd for write” -> alert

43

Misuse Detection: Pro

• Easy to implement

• Easy to deploy

• Easy to update

• Easy to understand

• Low false positives

• Fast

44

Misuse Detection: Con

• Cannot detect something previously unknown

• Constantly needs to be updated with new rules

• Easier to fool

45

Burglar Alarms

• A burglar alarm is a misuse detection system that is carefully targeted– You may not care about people port-

scanning your firewall from the outside– You may care profoundly about people port-

scanning your mainframe from the inside– Set up a misuse detector to watch for

misuses violating site policy

46

Burglar Alarms (cont)

• Goals:– Based on site policy alert administrator to

policy violations– Detect events that may not be “security”

events which may indicate a policy violation

• New routers• New subnets• New web servers

47

Burglar Alarms (cont)

• Trivial burglar alarms can be built with tcpdump and perl

• Netlog and NFR are useful event recorders which may be used to trigger alarmshttp://www.nswc.navy.mil/ISSEC/Docs/loggingproject.html

ftp://coast.cs.purdue.edu/pub/tools/unix/netlog/

http://www.nfr.net/download

48

Burglar Alarms (cont)

• The ideal burglar alarm will be situated so that it fires when an attacker performs an action that they normally would try once they have successfully broken in– Adding a userid– Zapping a log file– Making a program setuid root

49

Burglar Alarms (cont)

• Burglar alarms are a big win for the network manager:– Leverage local knowledge of the local

network layout– Leverage knowledge of commonly used

hacker tricks

50

Burglar Alarms: Pro

• Reliable

• Predictable

• Easy to implement

• Easy to understand

• Generate next to no false positives

• Can (sometimes) detect previously unknown attacks

51

Burglar Alarms: Con

• Policy-directed– Requires knowledge about your network– Requires a certain amount of stability

within your network

• Requires care not to trigger them yourself

52

Honey Pots

• A honey pot is a system that is deliberately named and configured so as to invite attack– swift-terminal.bigbank.com– www-transact.site.com– source-r-us.company.com– admincenter.noc.company.net

53

Honey Pots (cont)

• Goals:– Make it look inviting– Make it look weak and easy to crack– Instrument every piece of the system– Monitor all traffic going in or out– Alert administrator whenever someone

accesses the system

54

Honey Pots (cont)

• Trivial honey pots can be built using tools like:– tcpwrapper– Burglar alarm tools (see “burglar alarms”)– restricted/logging shells (sudo, adminshell)– C2 security features (ugh!)

• See Cheswick’s paper “An evening with Berferd” for examples

55

Honey Pots: Pro

• Easy to implement

• Easy to understand

• Reliable

• No performance cost

56

Honey Pots: Con

• Assumes hackers are really stupid– They aren’t

57

Firewalls as an IDS

• Excellent source of network probe, attack and misuse information

• Detect policy deviations based on access control lists

• Some have “NIDS” capabilities

58

Network Honeypots

• Sacrificial system(s) or sophisticated simulations

• Any traffic to the honeypot is considered suspicious

• If a scanner bypassed the NIDS, HIDS and firewalls, they still may not know that a Honeypot has been deployed

59

honeypot HTTP DNS

Firewall

60

Hybrid IDS

• The current crop of commercial IDS are mostly hybrids– Misuse detection (signatures or simple

patterns)– Expert logic (network-based inference of

common attacks)– Statistical anomaly detection (values that

are out of bounds)

61

Hybrid IDS (cont)

• At present, the hybrids’ main strength appears to be the misuse detection capability– Statistical anomaly detection is useful more

as backfill information in the case of something going wrong

– Too many false positives - many sites turn anomaly detection off

62

Hybrid IDS (cont)

• The ultimate hybrid IDS would incorporate logic from vulnerability scanners*– Build maps of existing vulnerabilities into

its logic of where to watch for attacks

• Backfeed statistical information into misuse detection via a user interface

* Presumably, a clueful networkadmin would just fix the vulnerabilty

63

Books

• Internet Security and Firewalls: Repelling the Wily Hacker, by Bill Cheswick and Steve Bellovin, from Addison Wesley

• Internet Firewalls, by Brent Chapman and Elizabeth Zwicky

64

URLs

• Hacker sites: the fringe– http://www.2600.com– http://www.digicrime.com– http://www.zone-h.org/defaced/

2003/01/30/www.defensivethinking.com/hacked.html

– http://www.website.to/hacker

65

Addresses

• CERT– cert@cert.org

• Firewalls mailing list– firewall-wizards-request@honor.icsalabs.com:

subscribe firewalls

• Web security mailing list– listserv@lehigh.edu: subscribe www-

security