what about scanning?
DESCRIPTION
What About Scanning?. Analyzing Scan Data as part of a “Defense in Depth” Solution to the High Bandwidth Intrusion Detection Problem Douglas Cress. The Way Ahead. Introduction and Motivation Description of Scanning Analyzing NIDS Alerts Experiment Description Conclusions and Future Work. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/1.jpg)
What About Scanning?
Analyzing Scan Data as part of a “Defense in Depth”
Solution to the High Bandwidth Intrusion Detection
ProblemDouglas Cress
![Page 2: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/2.jpg)
M.S. Thesis Defense 8/6/03 2
The Way Ahead
Introduction and Motivation Description of Scanning Analyzing NIDS Alerts Experiment Description Conclusions and Future Work
![Page 3: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/3.jpg)
M.S. Thesis Defense 8/6/03 3
High Bandwidth Intrusion Analysis Challenges
Class A networks have 16 million hosts, Class B networks have 65,535 hosts Both class sizes require bandwidth in the
Multiple T3 (45 Mb/s ~ 486 GB/day) to OC-3 (155 Mb/s ~ 1.67 TB/day)
Detecting Intrusions at line rate is basically impossible
Most NIDS only sample the data stream at such high bandwidths
![Page 4: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/4.jpg)
M.S. Thesis Defense 8/6/03 4
High Bandwidth Intrusion Analysis Challenges
Small number of defenders vs. overwhelming force of attackers Global Information Assurance Certification
(GIAC) has certified only 643 people since 2000! Constantly changing vulnerability
landscape 2,572 unique entries in the Common
Vulnerability and Exposures (CVE) database Ever increasing rise of non-mission
essential software P2P, Chat, Warez etc.
![Page 5: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/5.jpg)
M.S. Thesis Defense 8/6/03 5
High Bandwidth Intrusion Analysis Challenges
Poor tools Visualizations break down because of
massive amount of data Meta-data like CISCO NetFlow isn’t
sufficient to prove an intrusion Even Network Intrusion Detection
Systems (NIDS), if poorly configured, can output more false alarms than true
![Page 6: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/6.jpg)
M.S. Thesis Defense 8/6/03 6
Hacker Methodology
1. Information gathering – Scanning2. Initial penetration – Buffer overflow3. Privilege escalation – Password
cracking 4. Various Activities – Data extraction5. Attack Relay – Violate trust
relationships
![Page 7: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/7.jpg)
M.S. Thesis Defense 8/6/03 7
High Bandwidth Intrusion Analyst Solutions
Defense in Depth Physical Devices
routers, firewalls, NIDS etc. Organization security policies
Fair-use, virus scanning, etc. Analysis methods
Real-Time, Trend, Area Of Responsibility (AOR), etc.
![Page 8: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/8.jpg)
M.S. Thesis Defense 8/6/03 8
Defense in Depth
RouterFirewall
NIDS
HIDS
![Page 9: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/9.jpg)
M.S. Thesis Defense 8/6/03 9
Thesis Synopsis
Reduce wasted analyst time by identifying most likely true-positive NIDS alerts based on related previous scanning
Using UMBC as a testing ground for theories
Novelty and Significance of work
![Page 10: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/10.jpg)
M.S. Thesis Defense 8/6/03 10
Background TCP/IP
TCP, UDP, and ICMP are all susceptible to scanning TCP has the three way handshake
SYN, SYN-ACK, ACK UDP provides auto-response for
available services ICMP provides challenge and
response functionality
![Page 11: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/11.jpg)
M.S. Thesis Defense 8/6/03 11
Types of Scans
Scanning is not illegal Moulton vs. VC3, 2000
Half-open scan (aka SYN scan) Null-host scan OS scan Packaged scan and attack tool
![Page 12: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/12.jpg)
M.S. Thesis Defense 8/6/03 12
Scan Tools
NMAP (Network MAPer) Most famous, most options
Nessus One of many vulnerability scanners
Grim’s Ping FTP – Warez emplacement tool
![Page 13: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/13.jpg)
M.S. Thesis Defense 8/6/03 13
Generic NIDS Description
Network appliance designed to examine all passing traffic for embedded intrusions
Produces alarms / alerts for an analyst to review
Anomaly-based vs. Signature-based Common Vendors include – ISS’s
RealSecure, Cisco’s IDS, Enterasys’s Dragon, and SNORT
![Page 14: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/14.jpg)
M.S. Thesis Defense 8/6/03 14
Brief Description of SNORT
Open source – libpcap based 3 parts
Packet decoder Detection engine Alert / logging system
SNORT pre-processors stream4, conversation, and portscan2
![Page 15: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/15.jpg)
M.S. Thesis Defense 8/6/03 15
Parsing Logs
UMBC has over 15 million alerts a day Use PERL to quickly parse logs to mine
the most important information Figure out who is involved in scanning
(both source and destination IP) Look for alerts either from or to IPs
related to previously detected scanning
![Page 16: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/16.jpg)
M.S. Thesis Defense 8/6/03 16
Predictive Analysis / Attack Forecasting
Data mining techniques are good for trend analysis
Type of scan should indicate skill level of attacker SYN-scan perpetrated by worm or
script-kiddie Null-host scan wielded by skilled
attacker
![Page 17: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/17.jpg)
M.S. Thesis Defense 8/6/03 17
UMBC’s fitness as a Testing Ground
Class B address space (130.85.0.0/16)
Varied users and missions Students, administrators, researchers
High bandwidth Multiple T3’s
Small intrusion analysis group
![Page 18: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/18.jpg)
M.S. Thesis Defense 8/6/03 18
Long-Term / Trend Analysis
Process of examining intrusion events over a long time period to determine both future events and missed past events
Difficult to perform Massive amount of data to process
and store Urgency of the now often crowds out
long-term view
![Page 19: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/19.jpg)
M.S. Thesis Defense 8/6/03 19
November 2002 Raw Alerts
![Page 20: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/20.jpg)
M.S. Thesis Defense 8/6/03 20
November 2002 Alert Types
![Page 21: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/21.jpg)
M.S. Thesis Defense 8/6/03 21
November Top 5 per Day
![Page 22: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/22.jpg)
M.S. Thesis Defense 8/6/03 22
Attack vs. Scan AlertsNovember 2002 Scans and Attack Alert Comparison
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
11/1
/200
2
11/3
/200
2
11/5
/200
2
11/7
/200
2
11/9
/200
2
11/1
1/20
02
11/1
3/20
02
11/1
5/20
02
11/1
7/20
02
11/1
9/20
02
11/2
1/20
02
11/2
3/20
02
11/2
5/20
02
11/2
7/20
02
11/2
9/20
02
Date
Co
un
t (M
illi
on
s)
Alerts
Scans
![Page 23: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/23.jpg)
M.S. Thesis Defense 8/6/03 23
Analysis Process
Execute scanTop10.pl against SNORT scan alerts
Execute checkAlerts2.pl to find SNORT attack alerts relating to the top ten scanning parties
Execute checkAlerts2_to_excel.pl to format the data for easy spreadsheet viewing
![Page 24: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/24.jpg)
M.S. Thesis Defense 8/6/03 24
November 1st Top 10 Source Scanners
55%
17%
13%
4%
2%1%
130.85.178.42
130.85.83.146
130.85.70.176
130.85.104.155
130.85.150.220
130.85.150.213
130.85.111.213
130.85.91.240
130.85.114.88
130.85.168.49
Nov 1 Top 10 Source Scanners
![Page 25: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/25.jpg)
M.S. Thesis Defense 8/6/03 25
Nov 1 Top 10 Scan VictimsNovember 1st Top 10 Scan Victims
59%
18%
8%
8%
2%2% 2% 1%
64.231.48.85
64.231.48.103
209.91.161.131
216.104.117.52
64.231.49.234
209.91.176.79
64.231.48.134
130.85.140.2
204.183.84.240
80.141.108.40
![Page 26: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/26.jpg)
M.S. Thesis Defense 8/6/03 26
11/01/02 Correlated Scans To Attacks for November 2002
0
500
1000
1500
2000
2500
3000
11
/1/0
2
11
/2/0
2
11
/3/0
2
11
/4/0
2
11
/5/0
2
11
/6/0
2
11
/7/0
2
11
/8/0
2
11
/9/0
2
11
/10
/02
11
/11
/02
11
/12
/02
11
/13
/02
11
/14
/02
11
/15
/02
11
/16
/02
11
/17
/02
11
/18
/02
11
/19
/02
11
/20
/02
11
/21
/02
11
/22
/02
11
/23
/02
11
/24
/02
11
/25
/02
11
/26
/02
11
/27
/02
11
/28
/02
11
/29
/02
11
/30
/02
Date
Ala
rm C
ou
nt
204.183.84.240209.91.161.131209.91.176.79216.104.117.5264.231.48.10364.231.48.13464.231.48.8564.231.49.23480.141.108.40MY.NET.104.155MY.NET.111.213MY.NET.114.88MY.NET.140.2MY.NET.150.213MY.NET.150.220MY.NET.168.49MY.NET.178.42MY.NET.70.176MY.NET.83.146MY.NET.91.240
Nov 1 Scans vs. Month
![Page 27: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/27.jpg)
M.S. Thesis Defense 8/6/03 27
Term Analysis for November
MY.NET.114.88 => ucommons-114-88.pooled.umbc.edu
MY.NET.170.176 => phaser.ucs.umbc.edu
MY.NET.150.213 => libpc11.lib.umbc.edu
MY.NET.150.220 => paladin.lib.umbc.edu
![Page 28: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/28.jpg)
M.S. Thesis Defense 8/6/03 28
Term Analysis for November
Analysis focus for hosts involved in scanning and later attacking Red Worm Alerts x86 setuid exploit alarms null scans
![Page 29: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/29.jpg)
M.S. Thesis Defense 8/6/03 29
Four types of hosts
ucommons – Dynamically assigned Could be anybody with a laptop
libpc11 – General use lab computer Rotating user set
paladin – Personal use computer Probably hacked
phaser – SA owned machine Embarrassingly hacked?
![Page 30: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/30.jpg)
M.S. Thesis Defense 8/6/03 30
Mar 1 Scans vs. MonthMar1 Scans to Rest Alerts
0
10
20
30
40
50
60
70
80
3/1/
2003
3/3/
2003
3/5/
2003
3/7/
2003
3/9/
2003
3/11
/200
3
3/13
/200
3
3/15
/200
3
3/17
/200
3
3/19
/200
3
3/21
/200
3
3/23
/200
3
3/25
/200
3
3/27
/200
3
3/29
/200
3
3/31
/200
3
Date
Ale
rt C
ou
nt
12.223.210.92
129.89.177.104
142.166.101.40
192.26.92.30
192.5.6.30
200.69.241.141
208.180.107.153
24.122.34.47
62.245.82.59
67.33.105.181
MY.NET.1.200
MY.NET.196.55
MY.NET.202.194
MY.NET.249.194
MY.NET.97.104
MY.NET.97.124
MY.NET.97.148
MY.NET.97.188
MY.NET.97.29
MY.NET.98.43
![Page 31: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/31.jpg)
M.S. Thesis Defense 8/6/03 31
Term Analysis for March
MY.NET.97.29 => ppp-29.dialup.umbc.edu
MY.NET.97.124 => ppp-124.dialup.umbc.edu
MY.NET.97.148 => ppp-148.dialup.umbc.edu
MY.NET.1.200 => Unresolved
![Page 32: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/32.jpg)
M.S. Thesis Defense 8/6/03 32
Term Analysis for March
MY.NET.1.200 Scanned with NMAP Windows SMB attacks Watch-listed host attempted access
Three Dial-up addresses all involved in IIS (Internet Information Server) attacks
![Page 33: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/33.jpg)
M.S. Thesis Defense 8/6/03 33
Real-Time Illustration
November 11, 2002 1.2 million scans Over 74,000 alerts
Boiled down to two hosts worth investigating
Discovered in less than five minutes
![Page 34: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/34.jpg)
M.S. Thesis Defense 8/6/03 34
Nov 11th Scan & Attack Alerts
November 11th Scans and Attacks
0
0.2
0.4
0.6
0.8
1
1.2
11/1/
2002
11/3/
2002
11/5/
2002
11/7/
2002
11/9/
2002
11/11
/200
2
11/13
/200
2
11/15
/200
2
11/17
/200
2
11/19
/200
2
11/21
/200
2
11/23
/200
2
11/25
/200
2
11/27
/200
2
11/29
/200
2
Date
Co
un
t (M
illi
on
s)
Alerts
Scans
![Page 35: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/35.jpg)
M.S. Thesis Defense 8/6/03 35
Nov 11th Scans correlated to Attacks
Nov 11 - 11 Correlated Alerts
0
200
400
600
800
1000
1200
1400
MY.N
ET.114
.25
MY.N
ET.88.
168
MY.N
ET.70.
200
MY.N
ET.83.
146
MY.N
ET.70.
176
MY.N
ET.150
.220
MY.N
ET.150
.213
MY.N
ET.139
.10
IP Addresses
Ale
rt C
ou
nt
Correlated Alerts
![Page 36: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/36.jpg)
M.S. Thesis Defense 8/6/03 36
Real-Time Analysis Nov 11th
MY.NET.150.220 => paladin.lib.umbc.edu Accessed over 1000 times by Dutch
registered host IIS overflow attempt Possible Red Worm related activity
![Page 37: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/37.jpg)
M.S. Thesis Defense 8/6/03 37
Real-Time Analysis Nov 11th
MY.NET.83.146 => aciv-83-146.pooled.umbc.edu Probably wireless host 250 Access attempts from different
Dutch registered host Further scanning against the UMBC
host from a third Dutch host
![Page 38: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/38.jpg)
M.S. Thesis Defense 8/6/03 38
Tools Created for Analysis
scanTop10.pl – examines SNORT scan logs and calculates the top 10 scanning offenders and victims
checkAlerts2.pl – compares the output of scanTop10.pl to a SNORT attack alert log
fit_checkAlerts2_to_excel.pl – formats the output from checkAlerts2.pl for absorption into a spreadsheet
![Page 39: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/39.jpg)
M.S. Thesis Defense 8/6/03 39
Conclusions
My novel analysis method would help a small group of intrusion analysts tackle a large network’s NIDS logs
The analysis method is simple to perform and rapid in execution
![Page 40: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/40.jpg)
M.S. Thesis Defense 8/6/03 40
Future Work
Integration of my analysis process into a SNORT Post-Processor would help reduce false-positives
SNORT already exports alerts in XML, is it possible to extend this feature to export alerts in RDFS or DAML+OIL to then be reasoned over in order to reduce false positives?
![Page 41: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/41.jpg)
M.S. Thesis Defense 8/6/03 41
Future Work
Trend analysis is difficult because of the massive amount of data that must be stored.
Usually this data is stored in a compressed format which is then un-compressed during each search
![Page 42: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/42.jpg)
M.S. Thesis Defense 8/6/03 42
Future Work
Perhaps storing a meta-rule version of the alerts which could then be reasoned over to provide a pointer into exactly the compressed file where the important events are located, would speed the information retrieval process
![Page 43: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/43.jpg)
??? Questions ???
![Page 44: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/44.jpg)
M.S. Thesis Defense 8/6/03 44
Selected Bibliography
S. Axelsson, “The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection.” In Proc. Of the 6th ACM Conference on Computer and Communications Security, 1999.
R. Bace, P. Mell, “Intrusion Detection Systems,” NIST Special Publication, Nov 2001, Available HTTP: http://csrc.nist.gov/publications/nistpubs/800-31/sp800-31.pdf
Honeynet Project, “Know Your Enemy: Statistics, Analyzing the past … predicting the future,” [Online Document], Jul 2001, [ cited 2003 Jun 25], Available HTTP: http://www.honeynet.org/papers/stats/
![Page 45: What About Scanning?](https://reader036.vdocument.in/reader036/viewer/2022062322/56814908550346895db640c3/html5/thumbnails/45.jpg)
M.S. Thesis Defense 8/6/03 45
Special Thanks
Dr. Nicholas for his help and mentoring
Andy Johnston for providing the SNORT logs and some background on UMBC
Paul Cress for his editing help