Streamlining IP Registrant Identification, Notification, and Blocking for Threats in

the Wild

Tom Jagatic and Merri Beth Lavagnino

Indiana University

Copyright Tom Jagatic and Merri Beth Lavagnino, 2005. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational

purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by

permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Overview

I. Who We AreII. Motivation and PreparationIII. Recurrent Incident Management

System (RIMS) DescriptionIV. Future WorkV. Questions?

I. Who We Are

Indiana University

• Indiana University has eight campuses: – the original campus in Bloomington, which

is a residential campus (38,589 students); – an urban campus in Indianapolis, which

also includes the IU Medical Center (29,860 students);

– and six regional campuses in the Indiana cities of Gary, South Bend, Fort Wayne, Kokomo, Richmond, and New Albany.

• Total students: More than 98,000

University IT Policy Office (ITPO)

Reporting to the VP for IT, responsible for…• Information technology policy development,

dissemination, interpretation, and education. • Computer accounts management. • Coordinating response to incidents of abuse,

misuse, or inappropriate use of Indiana University information technology resources.

• Interacting with computer security engineers to develop and administer security education and awareness programs, and to assist with investigation of computer security incidents.

• Administration of the University IT Security Office (ITSO).

University IT Security Office (ITSO)

• Highly capable in specific technologies• Detection (netflow, etc.)• Creates auto-processes that provide

vulnerable or likely compromised host lists daily to ITPO Incident Response

• Strategic prevention (firewall, border filters, etc.)

• Consults with central computing dept or departmental technicians on security issues and options

• Works with the central computing department on infrastructure security (security CDs, device registration, etc.)

Vice President for Information Technology and ResearchMichael McRobbie

Vice President for Information Technology and ResearchMichael McRobbie

Chief IT Security and Policy OfficerMark Bruhn

Chief IT Security and Policy OfficerMark Bruhn

Admin. AssistantL. McNabb

Admin. AssistantL. McNabb

Disaster RecoveryMarge Abels, Program Coordinator

Disaster RecoveryMarge Abels, Program Coordinator

IT Security OfficeIT Security Office

IT Security Officer Tom Davis

IT Security Officer Tom Davis

M. AbelsD. GreenbergN. JohnsonA. KortyS. KrulewitchD. Monnier M. RaineyJ. Sweeny

M. AbelsD. GreenbergN. JohnsonA. KortyS. KrulewitchD. Monnier M. RaineyJ. Sweeny

REN-ISACDoug Pearson

REN-ISACDoug Pearson

Center SupportStacie Wiegand

Center SupportStacie Wiegand

IT Policy OfficeIT Policy Office

Deputy IT Policy Officer Merri Beth Lavagnino

Deputy IT Policy Officer Merri Beth Lavagnino

Accounts AdministrationLaura Klein, Manager

Accounts AdministrationLaura Klein, Manager Incident ResponseIncident Response

IUBC. ConklinT. GrubbR. Hasty

IUBC. ConklinT. GrubbR. Hasty

IUPUIB. HanesIUPUIB. Hanes

IUBT. JagaticIUBT. Jagatic

IUPUIJ. AbelsR. Whitt

IUPUIJ. AbelsR. Whitt

Data/PolicyAdministration

Data/PolicyAdministration

S. WiegandS. Wiegand

Incident Response

• Works in web-based incident response application and database (RT -- Request Tracker)

• Works to locate reported compromised devices• Administers tactical filters (dhcp lease blocks,

disabling data jacks and usernames, etc.)• Interacts with department technicians and

individual users about issues with specific devices

• Reviews and works through lists from ITSO• Coordinates large responses with computing

dept units and department technicians• Works to identify specific misbehaving

individuals

Incident Response Statistics

0

2,000

4,000

6,000

8,000

10,000

12,000

2000/01 2001/02 2002/03 2003/04 2004/05

(partial)

Incident Response Staffing

• Three dedicated FTE:– Information Security Technician (Tier 1)– Coordinator/Senior Security Analyst (Tier 2

Technical)– Digital Copyright Compliance Coordinator

• Plus Manager:– Deputy IT Policy Officer (Tier 2 Policy)

• Plus others as needed:– Security Engineers in the ITSO (Tier 3 Technical)– Chief IT Security and Policy Officer (Tier 3 Policy)– Other central computing staff (network,

messaging, support units, communications staff, etc.)

II. Motivation and Preparation

We Needed Something to Deal with…

• Increase in “mass” incidents vs. individual incidents

• Increase in ITSO’s capability to detect malicious activity

• Lack of interoperability of existing tools• Difficulty to track repeat compromises/

multiple compromises of same host• Manual isolation processes no longer fast

enough and scalable to effectively manage volume and contain threats in the wild

Individual Incident Reports:NOT to be automated

• From outside or inside sources• Concerning one IP number• Two kinds:

– NON-BEHAVIORAL incidents are things that machines are doing that need to be fixed, such as a virus, worm, spam bot, etc.

– BEHAVIORAL incidents are people-initiated, such as chain mail, account misuse, malicious activity, copyright violation, etc.

“Mass” Incident Reports:Wanted to automate

• From security engineer proactive detection work

• Generally as a result of current vulnerabilities, viruses, worms, etc. or degradation of service

• Blaster, Nachi, Sobig, Spam bots, Gaobot, Sasser, IRC bots, etc.

Preparation

• Little by little we’ve been addressing pieces of the problem…

Building Blocks

• Central tracking system for all incidents of abuse or misuse of IU IT resources– All emails coming in to abuse, security, itpo,

itso, it-incident, copyright, etc. addresses

• Log host (syslogged data)• Network tools

– Datajacks database– DHCP registration and administration tool

DHCP Registration Tool

• A first attempt to connect to the network on specific subnets redirects to the DHCP registration web page

• Enter netid and password to authenticate

• Enter support provider contact• Agree to Appropriate Use statements• Automatically grabs MAC Address• Creates database of information

Recurrent Incident Management System (RIMS)

• After several basic building blocks were in place, we began working on RIMS to automate the manual processes

• We realize RIMS will phase out as newer solutions are implemented!!

• But this is a heck of a lot better today than it was three years ago…

III. RIMS Description(Recurrent Incident

Management System)

What is RIMS?

• A system to effectively manage recurrent incidents—identification, notification, blocking, and reporting

• RIMS is a symbiotic system with our overall incident response tracking system RT (Request Tracker, Best Practical)

• This approach has the following advantages:– One common repository to perform raw-text

incident searches (statistics are coarse)– If granular data/statistics are needed, RIMS is not

cluttered with unrelated (or potentially misclassified) incidents. The data sent into RIMS is very clean.

Overview

• Incident data sent into RIMS from various sources throughout the day– Trusted sources– Intrusion detection sensors

• Near real-time (every 3 hours) updates repository of registrant data (60 days)

• Upon loading of incident data into RIMS, registrant data is associated by date, time, and IP

Overview cont’d

• At a scheduled time (typically in the AM upon full receipt of IDS sensor data) sends summary reports into RT (Incident Tracking System)

• Incidents are evaluated and notifications sent

• Blocking occurs• Notifications and blocking have the

potential to be fully automated (presently human review is performed)

Sources of Registrant Data

• DHCP: full dhcpd handshake log– Focus on the DHCPACKs

• Dialup: radius/tacacs records• VPN: radius records

– Make some assumptions about orphaned records

• These three record types comprise a vast majority of incidents

• It should be emphasized, this system is designed to associate date, time, ip user (not the other way around). The above conditions don’t apply in reverse.

Identifying a registrant

• The process of identifying a user (DHCP)– Date, time, and ip MAC Address user– Interface with the MAC address registration

database

• Dialup and VPN– Much cleaner– Network IDs are explicitly logged– Information such as calling number, duration,

client IP (needed for remote VPN), …

• Static IPs (not yet, but coming soon)– Effort underway—IP Security and Accountability

Project

VPN subtlety

• There are two classes of VPN users– VPN wireless– VPN remote

• Users on the IU wireless network require use of the VPN

• These users of course, use DHCP. We treat VPN wireless users, just as any other DHCP host

• VPN remote users have their VPN account disabled

• Better to block machines than accounts

Gathering person info

• Connecting to an LDAP directory of people data, the following information is collected– Email address– Full name– Affiliation (faculty, staff, student)– Campus– Department

The lookup process• I’ve got an ip, where do I look?• Simply given an IP address, there’s no clear indication which

records contain registrant info• Changing network configurations make classifying IPs by

record source difficult• Our approach doesn’t attempt to make an initial classification

• Seven passes– Dialup records– VPN records– DHCP records (querying registrant databases from two

campuses)– Network names from DHCP records– Identify a campus (needed to know where to block)– If all else fails, perform a hostname lookup (sometimes DNS

names are helpful)– Query LDAP people directory

IDS Sensor detectsDate: Thu, 02 Dec 2004 00:20:00 ESTTo: [removed]From: [removed]Subject: [removed]

From [removed] Thu Dec 2 00: 20:03 2004Return-Path: [removed]X-Original-To: [removed]Delivered-To: [removed]

* compromised* korgo* Snort sensor* 445/tcp with payload "/x.exe"* Hi* block: yes ASAP* [removed]

Date-Time IP12/01-10:53:31.181928 10.1.72.14412/01-13:34:05.393195 10.1.81.24712/01-15:53:04.607407 10.1.82.2112/01-17:04:52.154014 10.1.73.1212/01-21:28:44.173896 10.1.74.177

Date: Thu, 02 Dec 2004 03:20:06 ESTTo: [removed]From: [removed]Subject: Korgo hosts

From [removed] Thu Dec 2 03: 20:09 2004Return-Path: [removed]X-Original-To: [removed]Delivered-To: [removed]

* compromised* Korgo worm* Snort Sensor* 445/tcp with payload of "/x.exe"* Hi* Block ASAP* [removed]

Date Time IP2004-12-01 00:00:33 10.2.231.442004-12-01 00:03:42 10.2.232.1122004-12-01 11:21:49 10.2.233.362004-12-01 15:18:56 10.3.224.412004-12-01 16:06:03 10.1.82.212004-12-01 17:08:56 10.1.73.122004-12-01 19:59:28 10.2.231.142

Notification

Blocking (e.g. DHCP)

Ad-hoc registrant lookups

Ad-hoc query pretty printed% event-query.pl | event-pp.pl 2005-04-03 12:00:00 10.1.15.6

***** DHCP Host Summary *****

Date of activity : 2005-04-03 12:00:00 Assigned IP : 10.1.15.6

MAC address : 1e:a7:de:ad:be:ef Date assigned : 2005-04-02 21:10:08 Computer name : FOOBAR Network : WCC-staff

Registrant : Tom Jagatic Email : tjagatic@indiana.edu Status : STUDENT Dept code : n/a Campus code : BL

Performance

• When database loads (to populate the registrant records), performance takes a hit

• Lookup performance, overall is acceptable• A lookup of 10,000 IPs

– 22.480 user, 1.030 system, 1:23.04 elapsed– This example: 120 lookups per second– Detailed profiling hasn’t been performed, yet

latency with other systems undoubtedly a factor

Volume of registrant records

• Kept for 60 days (our retention practice)• Upon every load, a purge for old records also

occurs• DHCP (IUB/IUPUI):

– 19,640,856 dhcp ACKs in 60 days– 40,350 distinct MAC addresses on an average day

• VPN (IUB/IUPUI): – 463,104 records– 16,478 distinct users

• Dialup (IUB/IUPUI):– 1,102,933 records– 12,636 distinct users

Statistics/Reporting

• Sensors - 1 detect per IP for a given day

• Fewer users than IPs identified• Most typical where IP turnover rapid

– VPN– Dialup

• Counting users is more appropriate than counting IPs

MS04011 LSASS Exploit v. Time2004-09-20 to 2005-04-01

0

20

40

60

80

100

120

140

160

180

200

9/20

/200

4

9/27

/200

4

10/4

/200

4

10/1

1/20

04

10/1

8/20

04

10/2

5/20

04

11/1

/200

4

11/8

/200

4

11/1

5/20

04

11/2

2/20

04

11/2

9/20

04

12/6

/200

4

12/1

3/20

04

12/2

0/20

04

12/2

7/20

04

1/3/

2005

1/10

/200

5

1/17

/200

5

1/24

/200

5

1/31

/200

5

2/7/

2005

2/14

/200

5

2/21

/200

5

2/28

/200

5

3/7/

2005

3/14

/200

5

3/21

/200

5

3/28

/200

5

Date

Cou

nt

IP

Users

Network

0

5

10

15

20

25

30

35

40

45

GREEK-1-w

ireles

s

GREEK-2-w

ireles

s

GREEK-3-w

ireles

s

GREEK-4-w

ireles

s

GREEK-5-w

ireles

s

GREEK-6-w

ireles

s

GREEK-7-w

ireles

s

GREEK-8-w

ireles

s

GREEK-A

GREEK-B

HALLS-a

shton

-004

HALLS-b

risco

e

HALLS-c

ampu

sview

HALLS-e

igenm

ann

HALLS-fo

rest

HALLS-fo

ster

HALLS-m

cnut

t

HALLS-re

ad

HALLS-S

HB-03

HALLS-S

HB-04

HALLS-te

ter

HALLS-tu

lip-tr

ee

HALLS-w

illkie

HALLS-w

right

Networks

Co

un

t

IP

User

LSASS Exploit Detects v. Networks

2004-09-20 to 2005-04-01

LSASS Exploit Repeat Offenders:

2004-09-20 to 2005-04-01User

Record Type

Num Days Observed

Days Diff

User1 dialup 2 12

User2 dialup 2 21

User3 dialup 3 91

User4 vpn 3 12

User5 vpn 3 24

User6 dialup 3 127

User7 vpn 3 55

User8 vpn 3 16

User9 dialup 3 11

User10 dialup 3 51

User11 vpn 3 26

User12 vpn 4 121

User13 dhcp 4 16

User14 dialup 4 71

User15 dialup 4 40

User16 vpn 4 8

User17 dialup 4 193

User18 vpn 4 14

User19 vpn 4 16

User20 dialup 4 7

User21 vpn 4 16

User22 vpn 4 10

User23 dialup 4 62

User24 vpn 4 10

Design

• Objected oriented Perl• Classes

– IRTools::Config - global configuration – IRTools::Log - abstraction for DHCP, Dialup, and VPN

lookups – IRTools::Log::Import - used for formatting data prior to

loading into DB– IRTools::RI - profile, report, event, block, notify record

manipulation – IRTools::Web - future UI class for web

• MySQL

Recurrent Incident Tables

• Profile– Creation time, description, data source, data coverage,

purpose, review, active, warning message, block message, LSP message

• Report– Creation time, timezone, inclusion criteria, thresholds

used, confidence level, Incident Response Action (Block/Notify), User Action (Rebuild/Clean), priority, reporting office, sender

• Event– Date and Time, record source (campus), record type

(dhcp,dialup, vpn, other), record begin, record end, client IP, IP, hostname, network, network id, email, full name, title, dept, campus, MAC address, NetBIOS name, calling number, false positive (Y/N)

IV. Future Work

Further work

• Admin Web Interface• Self-Service Web Interface

– Users can unblock themselves, provided they are not repeaters

• Reporting for Local Service Providers– Hinges on another project (IPSAP)

• Incorporation of authentication logs– Ability to associate Network id ip

Further work cont’d.

• Boolean logic for notification– If detect for profile A and B, only then notify

• Further enhancements to IDS data interface (SMTP not ideal)

• API (likely SOAP)• DMCA copyright

identification/notification• Ad hoc and standardized reporting

interface

V. Questions?

Tom Jagatictjagatic at iu.edu

Merri Beth Lavagninombl at iu.edu

Indiana University IT Policy Office

http://www.itpo.iu.edu/