Analysts International

Analyzing Production Windows Servers

2

Introductions• Mark Lachniet from Analysts International,

Sequoia Services Group (Lansing & Detroit)• Member of the HTCIA• Not in law enforcement or a lawyer• Senior Security Engineer and Security

Services technical lead• Frequent presenter and trainer• Certified Information Systems Security

Professional (CISSP)• Microsoft MCSE, Novell Master CNE, Linux

LPI Certified LPIC-1, Check Point Certified CCSE, TruSecure TICSA, etc.

3

Agenda• Where a technical security engineer fits

into an investigation

• Frequent types of incidents

• Types of investigation

• Capturing volatile information on a Windows box (tools and techniques)

• Expanding the scope of investigation

• Real life examples

4

Where the Security Engineer Fits• Dedicated security consultants can help in both

prevention and response• In prevention – designing and maintaining secure

technological and organizational systems (not just technology!) Too bad people don’t do this!

• In response – the topic at hand– For specific tools and technical expertise for a variety

of systems (servers, workstations, network devices)– To investigate an incident before deciding whether or

not to prosecute– To help weigh costs and benefits of various courses of

action – how to investigate, how to secure– To assist in prosecution by thoroughly researching

and documenting findings without the constrictions that law enforcement would have (?)

5

Frequent Security Incidents• The vast majority of calls I get are in regard

to a “hacking incident” • Almost all of these incidents are on Internet-

connected machines, most are Windows• Most incidents are precipitated by:

– An external complaint (your mail server is sending me a lot of spam e-mail)

– A change in the system (the hard drive is full, strange new programs are running, tape backups are taking a lot longer)

– The Internet is “slow” or we see strange activity– A threat from an insider – usually a network

administrator making casual statements about how they could “take them out” if they ever got fired

6

Types of Investigation• Once a call comes in requesting help in investigation,

the engineer is dispatched on-site• The first (and perhaps most important) step is discuss

the situation with the victim before doing any work• There are basically three ways we approach an

investigation:– “Pull the Plug” – don’t touch the machine– “Limited Investigation” – tread lightly– “Extensive Investigation” – heavy footprint, honeypot?

• Each of these approaches have advantages and disadvantages, depending on your goals

• The most important question to ask is how strongly the customer feels about trying to prosecute

• The second most important question to ask is how much $$ they have to spend

7

Analyzing Volatile Information• WARNING: This may not be the best way to analyze

a production system. I am not advocating that you adopt this methodology, or that you should even attempt to analyze a production server. I am not a lawyer! This hasn’t been defended in court!

• Obviously, simply shutting down a computer that is running so you can image the hard drive can result in the loss of lots of interesting information

• We might also trip a “bomb” by not unplugging, but we take our changes – hackers aren’t usually smart

• Interesting volatile information:– Running processes (and the specific executable)– System users (logged on and not logged on)– Network connections (incoming and outgoing)– Open files (which processes are using what files)

8

Getting Started• At this point, we have already had discussions

with the victim and have the “go ahead”• If it is an Internet-connected computer, perform

a vulnerability assessment against the machine (most hackers won’t be particularly bothered by another hacker scanning the box from an unknown IP, so its not necessarily a sign that someone is on to them)

• Identify any obvious holes in the box that could be the means of entry (sometimes hackers will fix the holes once they are in)

• This may lead you to focus on a particular service (such as a Trojan) or a particular log file (such as IIS)

9

Getting Started• Examine the server as a hacker would –if it is an

FTP server, is there any warez or pr0n there?• Do a google on the DNS name and IP address to

see if it is a known open proxy server (this sometimes works!) Also e-mail blacklist services

• Create an updated version of the forensics CD (updated virus signatures and anti-trojan software)

• Create and label floppy disks• Create a worklog to record on• Start up a “known good” command interpreter

from the CD and start working, get date & time• Results are normally saved to floppy

10

Using ‘pslist’• d:\pstools\pslist -t > a:\pslistt.txt• Gives you a “tree” view of processes

11

Using ‘pslist’• d:\pstools\pslist -x > a:\pslistx.txt• Gives you a “full” view of processes

12

Using ‘psloggedon’• d:\pstools\psloggedon > a:\psloggedon.txt• Gives you a listing of logged on users, both

locally and remotely via ‘net use’ mapping• Useful for finding unauthorized users, people

accessing the system through terminal server

13

Using ‘netstatp’• d:\netstatp\release\netstatp -a -n > a:\netstatp.txt

• Gives you a listing of listening network servers *and* current TCP and UDP connections• A basic netstat comes with windows, but this may be replaced by a “fixed” or Trojaned

version that will specifically ignore a traffic pattern they want to hide so don’t use it• Useful for finding illegal users of the systems and their IP addresses to track down later• Does *not* map specific programs to the ports that are listening or established

14

Using ‘netstatp’

SMTP e-mail is listening

Web Server is listening

Windows NetBIOS is listening

An established session between our computer and

another computer

15

Enough Evidence?

• Based on the last few slides, you already have enough information to determine at least one of the attacks on the server

• Can anyone say what it is?

16

Using ‘fport-NG’• d:\fportng\fport > a:\fport.txt • Allows you to map listening services to specific executables – a *very* useful

17

Using ‘handle’• d:\handle\handle > a:\handle.txt • Allows you to view current file “handles” (or

references) to files and objects in use

18

Grab MAC Times• d:\perl\perl.exe \sfile\sfile.pl -d c:\ | \unix\gzip > a:\sfile.gz

• Refers to file system Modify, Access, Create• This is old news to a forensic examiner, but

what if you can’t image the machine?• Its too much data to fit well on a floppy, so

gzip it up using a UNIX command line tool• Exports in a handy comma-delimited format

for import into databases, spreadsheets and other analysis tools

• Note: Date conversion may take some work• Now you can recreate events by date, or

figure out when a specific file was created

19

View MAC Times in Excel

20

Other Miscellaneous Items• There are various other things to look at

including:– Scheduled jobs (using the ‘at’ command)– Netbios info (using the ‘nbtstat –c’ command)– Net info (using ‘net accounts’, ‘net users’, etc.)– Create MD5 hashes of files (esp. system files)– Back up large files to a forensic laptop over the

network:• IIS Logs (WWW, FTP, SMTP) from c:\winnt\system32\

LogFiles\*.*• Windows Event Logs (if possible – not on open files)• Log files from applications, temp files, browser caches• Copies of suspect binaries for inspection• Any warez or pr0n that you need to analyze later• Etc.

21

Do Your Research!• At this point, you should more or less know all of the

running programs and network connections on the box

• Dig in and identify each of them to make sure they are legitimate – this is very time consuming!

• Use the Internet, google, etc. to identify unknown components and figure out what they do

• Verify “known” components actually match the MD5 signatures of their counterparts on a known-good system (if available) or compare file sizes and dates

• Verify that each “known” component is actually using a port that it is expected to use

• Some trojans name themselves the same as other legitimate executables (e.g. services.exe), but you’ll find them in a different directory with fport-NG

22

Our Test Case• Pretend we exhaustively verified each running

process, using google searches such as server.exe “port 139” until we get to patch.exe

23

We have identified a trojan!• In this case, it’s the “Net Bus” Trojan• Admittedly, we did this the hard way• We can now research the Trojan, learn what it does,

and how it works• In this case we can tell the MAC times of the

executable• We can correlate these times with other server logs

(such as firewall, web and FTP, and Windows Event Logs) to try to pin the activity to a user

• We can also look for the traffic pattern to see if we can find out who was using it. In this case, we did collect this volatile state data back on the ‘netstatp’ slide and know the IP address of the trojan user (192.168.109.1)

• Maybe we can find out more about the attacker from that IP address

24

Additional Analysis: f-prot• Run a virus scan on the server (we use f-prot)• Either map a drive to the server and run it across the network, or

run it directly from the forensic CD• There may be more than one naughty file to find

Here is our culprit

25

Additional Analysis: Anti-Trojan• Run an anti-trojan scan on the server• Either map a drive to the server and run it across the

network, or install it and leave a big footprint• It can also check via portscan and registry scan

26

Additional Analysis: DumpReg• Leaves a foot print on the server• Shows registry keys *AND* the time of last modification (a little-

known ability)• Useful for tracking software installations and modifications over time• Allows for export to comma delimited for Excel

27

Additional Analysis: Misc • There are plenty of other tools you can use, mainly to

monitor ongoing system activity• This may be useful to find out the depth of the attack,

gather more evidence, or trick them into revealing something useful

• Some realtime tools:– Filemon: Monitors file access– Regmon: Monitors registry access– Tdimod: Monitors TCP/IP activity– Snort: Intrusion Detection System to identify attack

signatures in use on the network (especially useful if you are being used as a stooge to launch attacks)

– Ethereal: Free protocol analyzer to capture and analyze network traffic – may have a useful “payload” like IRC or IM messages, usernames, email addresses, etc.

28

Analyze Other Log Sources• In the networked world, no machine is an island• If systems have been appropriately designed and

implemented, which isn’t that often, there will be useful information in a variety of places

• The investigator must expand the scope from the “victim system” and look elsewhere

• Additional evidence can be found in many places:– Network and security devices on location– Internet Service Providers (AOL, DSL providers, etc)– Other servers on the network– Client workstations (especially if an insider is

suspected)– Authentication systems– The attacker’s workstation

29

Expanding the Scope of Investigation

Internal Network(Protected Machines)

DMZ Network(Internet Accessible Machines)

The Internet

Bad Person

Good Person

Company Firewall

Exchange e-Mail

ACME Corp Network

Internet Router

Web Server

User Laptop Printer

File Server User Workstation

!!

!!

!

! !

!

Best Source for Logging

30

Analyzing Router/Firewall logs• Some of the best information for figuring out how an

attack occurred and subsequent activity is by examining the logs of network devices such as routers and firewalls

• Unfortunately, many people don’t collect this data and store it, or even know that its possible

• Network device logs can provide a detail of what type of information traveled between network systems:– Determine how the system was profiled (reconnaissance)– Determine how the system was attacked (vulnerability)– Determine what happened after the attack – did the hacker

use your system to store files? Attack other systems?– Determine if multiple parties were involved (hackers tend to

run in packs in different parts of the world)– Note: you need to compare time to victim system and “real”

time to that of all the other log sources

31

Create a Deliverable Document• Once you have as much information as possible, you

need to document all of the data you have collected and provide an analysis of the raw data

• This document should attempt to summarize:– What happened (chronological sequence of events)– How it happened (what vulnerability was used)– Problem areas (what couldn’t be done / analyzed)– Next steps (both short term recovery and long term security

steps that should be taken)– Full appendix of collected data

• All of this information needs to be thoroughly explained so that non-technical people can understand the scope and impact of the incident and make decisions

• This document can be given to law enforcement to save time –hopefully we didn’t ruin any evidence!

32

Next Steps• The decision to prosecute is not an easy one to make

because there are many implications:– What will be the cost of prosecuting, in terms of legal

expenses, time spent, interruption to operations, etc.– What is the likelihood of success?– What is to be gained by prosecuting?– What are the implications to public image? Nobody

wants to be in the newspaper, nobody wants to be exposed as having poor security

– There is no guarantee that you will even be able to prosecute if you want to. What if the perpetrator lives in a developing country with no computer laws?

• Unless it was an insider job, or a specifically targeted attack, most people consider it a “learning experience” and hopefully secure their systems

33

Examples: The Warez Server• For this presentation, I did a little experiment, and

set up a “honeypot” server on the Internet• This server was a standard Windows 2000 server,

and was fully up to date (no vulnerabilities)• The only change made from the default

configuration was a single (confusing) checkbox that said to allow write access on the File Transfer Protocol (FTP) server – an easy mistake to make

• I put the machine on the Internet to see how long it would take for hackers to find it and abuse it

• The answer is: 3 days. Within 3 days, hackers had found the server, and discovered that it was possible to store files there anonymously

34

Examples: The Warez Server• Within a week, a “tag” had been placed

(hacker lingo for claiming the server – there is honor among thieves)

• A few days later, a huge number of “hidden” directories were created on the server, and software was uploaded to it.

• A few days after that, people from the Internet were downloading the illicit content, and I pulled the plug

• I’m still not sure what they uploaded, but most of the time its porn

• The lesson here is that they WILL find you, and quickly at that

35

Examples: Manufacturing• A manufacturing company was getting

complaints from people claiming that spam was coming from their mail server

• Their ISP shut them down due to abuse calls• They had investigated internally and couldn’t

figure out what was happening• Analysis of the server found that they were

directly connected to the Internet without a firewall or other protection

• Further analysis found several problems:– An open mail relay (allows spam)– An open proxy server (allows anonymous web access)– An open socks server (allows full Internet access)

36

Examples: Manufacturing• Analysis of log files showed that people from all

over the world had been relaying connections through their server

• Abuse included people looking at pornographic web sites, sending spam

• A search of the Internet found that the company server had been listed on multiple hacker sites as being an “open” relay

• Thus, not only are the hackers who find you going to abuse you, but they are going to share their good fortune with others

• What are the legal liabilities of being a third party to this type of activity?

37

Examples: Marketing• A marketing firm calls with concerns because

the network administrator found a remote-control program on the server (very bad)

• The server was connected to the Internet without a firewall

• Additional user ID’s had been created and granted administrative access

• Client suspected internal involvement• Logging on the server was turned off, so no

good data was collected• Logging on the network devices was also

turned off, so there was no data there either

38

Examples: Marketing• Examination of the server turned up some

evidence, such as the time and date that the remote control software was installed, and evidence that there was a hack but not much!

• However, because there was no logging, there was no sure way to know if the attack was internal or external

• Also because there was no logging, there was no way to track to an offending workstation by IP address

• The only real option was to clean up the damage, and start recommending some security services to stop it from happening again

39

Examples: K12 District• School district in Michigan with a fast

connection to the Internet• No problems were known• The district contracted with us to have a

managed firewall installed• As soon as we turned it on and started

analyzing traffic, it was obvious that they were currently being abused

• Investigation showed that they were unknowingly hosting child pornography – not a good thing for a school

• Many other people have found existing problems just by logging

40

Prevention and Response• None of the previous incidents made it to the

legal system, it just wasn’t worth it for them• None the less, it was an expensive, emotional and

painful experience for them• Much of that pain could have been minimized

through prevention instead of response• Unfortunately, computer security is a somewhat

like the wild west – its somewhat lawless, although serious crimes can be pursued its usually not worth it

• We use the metaphor of the neighborhood when describing computer security – the best approach is to make your own home hard enough to break into that they go to your neighbor instead

41

Discussion

Mark Lachniet, Sr. Security EngineerCISSP, MCNE, MCSE, CCSE, LPIC-1, TICSAAnalysts International - Sequoia Services

3101 Technology Blvd. Suite ALansing, MI 48910

phone: 517.336.1004fax: 517.336.1004