4/7/16

1

LoggingforIR

PlanningCentralLogging

4/7/16

2

• Failures of distributed logging• Pros and cons of centralized logging• Centralized logging considerations

• Possible solutions • A Sample solution

Objectives

• Logging on each system is great• An administrator can pull the logs and search

for troubles• Can research back in time to see what

occurred on a system• Logging into 10, 100, 1000, 10,000 or

100,000 different machines is not practical• Miss system wide events

Distributedlogging

4/7/16

3

• Security– Logs reside apart from their originals

• Protection from loss• Comparison to find changes

– Complicates attackers activities• More Difficult to cover tracks• Needs access to more than one machine

– Major source of events• New Users/groups/membership• New software• Scanning and password attacks

CentralizedLoggingAdvantages

• Visibility– Logs in one place

• No longer necessary to visit each machine• Can search for patterns across many machines• Single log patterns makes for easy searching

– Move logs off of systems quickly• If a system is malfunctioning it can be difficult to retrieve logs• Logs can be lost altogether if a malfunction is large enough

– Access by non IT people• Not all logs are for IT• Sales, Accounting, and HR may want to see logs of certain

systems

CentralizedLoggingAdvantages

4/7/16

4

• Proactive Actions– Resource exhaustion– Errors that show future failure– Poor network performance

• Spare local resources– Takes space to store logs – centralizing means

space is centralized– Processing power – logs must be filtered and

rotated, keeping them small which limits cpu time

CentralizedLoggingAdvantages

• Why collect logs if no one will look at them?• There will be thousands, possibly millions of events• It will be someone’s job to sort and draw conclusions from the

data– Some can be automated– Will require significant configuration– Someone with knowledge of the system– Will require ongoing time and effort if it is to be relevant

• Not normally considered when implementing a logging solution

• Main Cause of failure – Management must plan for this!

Considerations:HumanResources

4/7/16

5

• Log storage takes up space• Amount will be massively dependent on

– What is logged– Size of logs– Number of events per second (EPS)

• Averages– Each log is about 34 bytes (based on cisco logging)– If each machine produced 1000 logs per day and you

had 100 machines that is about a year’s worth of logs per gig of storage

Considerations:PhysicalResources

• Text files (syslog)– Easily compressible– Easy to read files– Difficult to aggregate

• EVT files (Windows uses compressed xml)– Space efficient– Need Windows event

viewer to view

Considerations:StorageFormats

• Database– SQL, MySQL,

PostGreSQL– Great for search ability– Cost and complexity

• Other formats– Non-schema database

(ElasticSearch)– Great search ability– Complexity

4/7/16

6

• Related closely to space needs• Retained logs do not need to be easily searchable

(at first)• Longer retention allows for more diagnosis

– Attacks could last weeks or months– If logs are destroyed the original breach may be lost– Court cases may last years!

• Company council may wish to see records deleted• Industry standards may dictate storage length

Considerations:Retentionneeds

• Server– Treasure trove of information about network– Logs reveal usernames (sometimes passwords)– Attacker MUST have access to cover tracks– Therefore -- log servers are huge targets– Log server lockdown

• Single purpose• Limit open ports• Limit network access to and from them (most communication is in)

• Logs Themselves– Encrypted drives may be necessary– Encryption for removable media may also be indicated

Considerations:LogSecurity

4/7/16

7

• Reliable transport of events to central log server– TCP guarantees delivery (slower and can be overwhelmed)– UDP faster but based on best effort (can be ok if program takes

care of reliability issues)• Prevent unauthorized messages

– Can cause message loss by DOS– Make finding legitimate messages more difficult

• Prevent unauthorized message viewing– Some messages contain important system data– Attackers that can intercept log messages may gain knowledge

about network

Considerations:EventTransport

• Are all machines participating in logging?• Keep lists of machine names

– Make sure every machine sends at least one event per day (even if you have to create it yourself)

– Check any machine that fails to report• Scan periodically for new devices when

possible

Considerations:LogAuditing

4/7/16

8

• An event is basically a message and a time stamp• The time stamp is generated by the system who created the

event• When combining events from many different machines it is

best if they AGREE as to what time it is• Log correlation is nearly impossible when the clocks are

wrong• Network Time Protocol (NTP) best option for maintaining

clocks– Setup beyond scope of this class– Linux tool– Microsoft maintains a similar tool (necessary for Kerberos)

Considerations:Timing

• Syslog– Oct1708:59:24peradam.cs.colorado.edusendmail[21601]:

e9HExOW21601:SYSERR(root): Can'tcreate transcriptfile./xfe9HExOW21601:Permissiondenied

• Apache log– 127.0.0.1- - [28/Jul/2006:10:22:04-0300]"GET/HTTP/1.0" 2002216

• PFSenseLog– Jan1107:28:30141.102.4.254pf:000145rule141/0(match):blockinon

bge0:(tos0x0,ttl128,id58078,offset0,flags[none], protoUDP(17),length1052)141.102.12.99.1137>188.40.123.111.24460:UDP,length1024

• Eachsystemhasitsownunique wayoflogging ifthere istobeacommonsearch,each logmustbeparsed

• The loggingsolution willneed toprovide amechinism forit

Considerations:MultipleFormats

4/7/16

9

• Windowscomeswithasolutionforlogging• WindowsEventcollectorservices

– DatastoredaseventlogsonspecifiedWindowsserver– Sentviawebservicesmanagement– InstalledinWindows8and2k8andabove– Canbeaddedtoxpandolder(nolongersupported)– SameserviceMicrosoftSystemsOperationsManageruses

• MOM willscanforpatterns (somegivenbutcanbeusergenerated)

• Willgeneratealertsbasedon findings

CentralLoggingSolutions:Windows

WindowsLogSubscription

4/7/16

10

• MOMcostssomethousands• Designedforwindowsonly• Canreceivesyslogentriestoo

– Theyareaccepted byport514udp– Nocontrol(withinsoftware) forwhatserverscansendevents

– Noguaranteeddelivery

CentralLoggingSolutions:Windows

• Splunk– Widelyconsidered best– Considered bymany tobeexpensive

• Microsinstalls(<500MBfree)• From.5– 10GB$4500pergig• From10–50GB$2500pergig• Estimationdifficultbut

– ~3.5gigper1000users– 4500*4=$18K

– Getwhatyoupay for• Easyinstall• EasyConfiguration• Manyalertrules

CentralLoggingSolutions:Thirdparty

4/7/16

11

• Others– Arcsite

• Fullsystemforlookingatlogs• Allowsforsearchingand alertingonlogs• Processesmanydifferent typesoflogs includingeventlog• Licensed

– LogRythm• Logsearches• Alerting• Processesmanydifferent typesoflogs includingeventlog• Licensed

– ELKStack (ElasticSearchLogstashandKibana)• Logretrieval• Robust searchcapability• Limited alerting• Open Source

CentralLoggingSolutions:Thirdparty

• Convertalllogstosyslogandlogtheresults– KIWISyslogserver (www.solarwinds.com)– Snare (www.intersectalliance.com)– Adicson WinSyslog (www.adiscon.com)– OSSECOpen sourcesoftwareusingagentstopull logdata from

Windowsmachines (morelater)

CentralLoggingSolutions:Thirdparty

4/7/16

12

• Sometimesnecessarytopulllogsmanually• Linuxiseasy-- simplysshinandcopyoffthelogs• Windows takes a little more

– Powershell• Get-winevent can query local and remote logs• Export-csv will export any input to text files

– I.E. get-winevent application | export-csv output.csv• Get-wmiobject (wmi objects include a wide array of

items)

CentralLoggingSolutions:DIY

• Wmiscripting– Find examples online on msdn– Free script by sans

• @http://www.sans.org/windows-security/• Dumpeventlogs.vbs• Will dump clear and sort logs

CentralLoggingSolutions:DIY

4/7/16

13

AnOpenSourceSolution

• Preparation– We need a tool that aggregates logs to one spot– It needs to translate the different type of logs into a format that

can be searched– The data needs to be transmitted in such a way that they can’t

be intercepted– The system needs to audit which systems can transmit data to

prevent intentional overload• Discovery

– The logs must be examined for malicious patterns or other oddities

– The system should then alert the administrator of anything suspicious

Whatwewanttodo

4/7/16

14

• Containment– It should then allow the administrator to search the

logs for further evidence of the incident– Discovering the extent of the breach– If the evidence is convincing enough it should take

preventative actions• Eradication

– Deep inspection of the logs will tell investigators what occurred on the system

– This will allow them to clean up the mess

Whatwewanttodo

• Recovery– The ability to add rules to closely monitor servers

put back on the network• Lessons learned

– The ability to produce graphs and data to show how the attack progressed and was stopped

– Allow Investigators to see what went well and what did not

Whatwewanttodo

4/7/16

15

• Combine 4 pieces of software by two different companies– Trend Micro supports Open Source SECurity

or OSSEC– Elastic Supports ELK

• ElasticSearch• Logstash• Kibana

OneSolutionUsingOpenSourceTools

• More than a log aggregator full host based intrusion detection system– Rootkit checker– File integrity checker– Registry auditing– Active response

OSSEC

4/7/16

16

• Agents are installed on as many systems as possible

• Agents will collect data and send it to the OSSEC server for analysis

• Transmissions are guaranteed and encrypted

• Endpoints are authenticated to prevent unauthorized messages

OSSEC

• Alerts are generated and placed into the alerts.log file

• Alerts can also be output in json format (latest build only)

• All logs can be archived (if activated) in the archive folder

OSSEC

4/7/16

17

• Alerts based on rules• Several hundred rules

are included• New rules added

regularly• Rules can be generated

by administrators• Administrator rules can

override package rules

OSSEC

• Alerts have a severity rating – 1-3 information– 4-10 Possible bad action– 11-15 High chance of foul

play• Alerts can trigger automated

response– Lock down firewall– Kill connections

• Alerts can also generate e-mails or texts

OSSEC

4/7/16

18

• The OSSEC project does include a web interface

• It is poor• .3 is the current

stable beta• .8 is newer and fixes

many issues

OSSEC

• ELK is three products that work together to provide a robust log search tool

• Logstash is the front end– It takes log information from various systems– It sorts it and splits it up into index able fields– It then stores the data in ElasticSearch

• ElasticSearch is a schema-less database engine– It stores data in a free form manner– The data can then be indexed and searched

• Kibana is a web interface– It interfaces with ElasticSearch– Allows the data to be searched, sorted, and displayed– All kinds of abilities to create dashboards

ELK

4/7/16

19

• Pros– This solution works– Meets all criterion– Free to implement– Requires only human resources and servers– Server requirements relatively low

• Cons– Setup not Trivial– Nearly all Configuration via text files– Different software requires different types of syntax– Will require significant tweaking (true of all solutions)

OSSolutionPros/Cons

• Note the full implementation is beyond the scope of this course

• A quick write up provided in the appendix of the labs

• Even if you follow the instructions it will require some modification to integrate into your environment

• All the pieces of software are documented on the internet and have paid support if you would like to purchase it

OSSolutionPros/Cons

4/7/16

20

FinallabExploringtheOpensourcesolution

• Windows logging– We discussed what could be logged– Showed how to log deep details

• Linux logging– Spoke about the various logs automatically generated– Added a few logs that could help

• Central logging– Spoke about the advantages and disadvantages– Discussed some of the products– Showed an open source solution

Conclusion