how*duke*university* uses*splunk*to* improve*security*and ... · aboutduke*! 14,600*students*...
TRANSCRIPT
![Page 1: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Duke University Jeremy Hopkins, Phillip BaDon, & Eric Hope
How Duke University Uses Splunk to Improve Security and Reduce Fraud
![Page 2: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/2.jpg)
Disclaimer
2
During the course of this presentaLon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauLon you that such statements reflect our current expectaLons and
esLmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaLon are being made as of the Lme and date of its live presentaLon. If reviewed aUer its live presentaLon, this presentaLon may not contain current or accurate informaLon. We do not assume any obligaLon to update any forward-‐looking statements we may make. In addiLon, any informaLon about our roadmap outlines our general product direcLon and is subject to change at any Lme without noLce. It is for informaLonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaLon either to develop the features or funcLonality described or to
include any such feature or funcLonality in a future release.
![Page 3: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/3.jpg)
About the Presenters
! Jeremy Hopkins – Sr. IT Analyst, Enterprise Internet Services ! Phillip BaDon – Sr. Security Analyst, University IT Security Office ! Eric Hope – Security Analyst, University IT Security Office
3
![Page 4: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/4.jpg)
About Duke ! 14,600 Students ! 3,340 Faculty ! 35,998 Staff ! Total of 68,000+ AcLve Users
! University and Medical Center ! Worldwide Presence
4
![Page 5: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/5.jpg)
Splunk @ Duke University 250GB License
200+ Indices & Sourcetypes • Syslog • OS (Win & *nix) • Web • Network • IPS/IDS/Firewall • Shibboleth • Mail • LDAP • VPN • Many More
Infrastructure: • 2 Central search heads • 4 Central indexers • 2 Central deployment servers
– (1 Win & 1 Linux) • 9 Departmental pairs
– (1 Search Head + 1 Indexer) • Over 2500 hosts
Departments & Uses • IT Security Office • Systems Admin • Messaging • Network • Database • Emergency noLficaLon
Tracking • Departmental IT groups • Many uses by many
groups
5
![Page 6: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/6.jpg)
Agenda
! IniLal IncepLon of Splunk @ Duke – OperaLon Find a Phish
! Using Splunk to Bridge FuncLonal Teams – Phishing ADacks Lead to Paycheck TheU/Fraud
! Splunk and Security – CorrelaLons, EvaluaLons, and Risk Scoring
6
![Page 7: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/7.jpg)
So it Began… Phishing to Fraud • Phish targeLng 600+ faculty/staff • Typical style emails (nothing overly
sophisLcated) – Pay raise, login verificaLon, etc. – Cloned login page
• Compromised accounts used to access HR/Payroll sites
• AUer successful login, bank rouLng numbers for direct deposit changed
• Reports of monthly salaries not received
Source: The News & Observer
7
![Page 8: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/8.jpg)
Before We Were Friends…
“Can you tell me who has logged in the most in the last hour?” – Manager ! Your quesLons are my distracLons…
‒ Where are the users logging in from? ‒ Where is SPAM coming from? ‒ Where is legit mail coming from?
! Log analysis from a shell prompt using the ancient sysadmin combo of: grep | awk | sort | uniq grep sasl_username logfile | awk '{print $9}' | sort | uniq -‐c | sort -‐n | tail -‐5
7 [email protected] 7 [email protected] 8 [email protected] 10 [email protected] 58 [email protected]
8
![Page 9: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/9.jpg)
hDps://xkcd.com/208/
9
![Page 10: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/10.jpg)
Take NULL locaLons and puts a custom label on them
Use of event type makes this applicaLon independent
Creates a LocaLon label such as “Las Vegas, NV” instead of columns for each value.
Top SMTP Logins
evenDype=smtpauth | iplocaLon client_ip | eval LocaLon= if(CountryCode == "US",if(City=="",if(Region=="","Unknown LocaLon, "+Country,Region),if(Region=="",City+", ?? "+Country,City+", "+Region)),if(City=="",Country,City+", "+Country)) | eval LocaLon=
if(isnotnull(LocaLon), LocaLon, if(cidrmatch(”10.0.0.0/8",client_ip), "Duke -‐ Private”, client_ip ) )
| stats values(LocaLon) count(neLd) by neLd | rename … | table neLd Count LocaLon | sort -‐Count
Finds SMTP logins and builds a table with username, login count, and locaLon
10
![Page 11: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/11.jpg)
Empowering Others Top 10 SMTP Logins using previous search example
11
![Page 12: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/12.jpg)
Advanced XML to Map Mail Sources Example based on Sophos PureMessage scan engine
The hidden search to find inbound and delivered messages that are NOT originaLng from our IP space.
Title of the panel on the dashboard
Module to build and display the map
<module name="HiddenSearch" layoutPanel="panel_row4_col1" group="Source of Accepted Inbound Mail (last 5 minutes)" autoRun="True"> <param name="search"> index=mail host=mail* inbound ac@on=deliver (fur!=10.0.0.0/8 AND fur!=152.3.0.0/16) | geoip fur </param> <param name="earliest">-‐5m@m</param>
<module name="GoogleMaps"> <param name="mapType">splunk</param> <param name="scrollwheel">off</param> <param name="zoomLevel">3</param>
<param name="overlay">clusters</param> <param name="drilldown">true</param> <param name="drilldown_field">client_ip</param>
</module> </module>
12
![Page 13: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/13.jpg)
Fun With Maps Source of Accepted Mail using Advanced XML in previous slide
13
![Page 14: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/14.jpg)
Guess Where our Spam Originates…
14
![Page 15: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/15.jpg)
evenDypes Log example from pos�ix mail server
hostname process
Client IP
Login String
! Event types allow us to search common events with ease ! What does a login “look” like?
– Raw log data: Sep 2 13:03:19 mail12.oit.duke.edu pos�ix/smtpd[74209]: 272863911E8_405C017F: client=unknown[152.16.52.172], sasl_method=LOGIN, [email protected]
– All SMTP authenLcaLons have a few things in common: ê Hostname of mail*.oit.duke.edu ê Process of pos�ix/smtpd ê Client IP ê User login string of sasl_username=*@*duke.edu
15
![Page 16: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/16.jpg)
evenDype=smtpauth Based on Pos�ix mail server and Cisco AnyConnect VPN
! SMTP Auth Converted to a Splunk evenDype – index=mail host=mail* process=pos�ix/smtpd sasl_username=*
! Duke created event types for various login events from various sources: – VPN – evenDype=vpnlogin
index=network sourcetype=vpn vpn_user=* vpn_inner_ipv4=* vpn_source_ip=*
! Shibboleth (single sign on) -‐ evenDype=shiblogin – index=idms_shib sourcetype=idp-‐process (shib_success=“[password]” OR
SSO=“true”)
16
![Page 17: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/17.jpg)
Share With Others ! In theory, we can join the various login types together to find all login events – evenDype=dukelogin is defined with the following search
ê evenDype=vpnlogin OR evenDype=smtpauth OR evenDype=shiblogin
! Event types allow others to use your logs without knowing the specifics of your applicaLon
17
![Page 18: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/18.jpg)
Cool, But So What ! Plo�ng mail sources provided needed proof to management to allow implementaLon of geographically isolated mail flow and acceptance rules
! Dashboards and evenDypes allowed the ITSO to quickly see account abuse and provided the groundwork for collaboraLon with other funcLonal teams
! Prior to Splunk it would take the Messaging team several hours, at Lmes, to pull a list of recipients of a phishing email. Now it’s a maDer of minutes
18
![Page 19: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/19.jpg)
Using Splunk to Bridge FuncLonal Teams Phishing ADacks Lead to Paycheck TheU/Fraud
![Page 20: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/20.jpg)
Example of Phishing Email Received
Clicking here leads to URL on next slide
A pay rise… interesLng.
20
![Page 21: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/21.jpg)
Link from Phish in Previous Slide
Believe it or not, Duke does not own
nl-‐tour.ru
21
![Page 22: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/22.jpg)
Gone Phishin’ Example of Dashboard to Record Email into Phish Tracking Lookup Table
Your DUKE Pay Increase
Pay Increase -‐ 20141006
Search DuraLon
Actual Subject of Email
How we idenLfy a parLcular campaign
Adds to PhishList lookup table
22
![Page 24: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/24.jpg)
Duke PhishList Lookup Table Custom lookup table to record historical informaLon on phishing emails
Macro to find message by Subject
Arbitrary name of Phish Map email to user
Pull user info (name, department, etc)
Append the output to our PhishList lookup table
`message_by_subject($subject$,"inbound acLon=deliver")` | rename … | eval PhishName=$phishname$ | lookup mailmap email as To | lookup NeLdLookup neLd OUTPUT eppa as affiliaLon | table PhishTime PhishName To From Subject SendingIP affiliaLon neLd | outputlookup append=true PhishList.csv
24
![Page 25: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/25.jpg)
Phishing Dashboard Aggregate of PhishList Data with Search Panel
25
![Page 26: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/26.jpg)
1. What data points do we have? a. Compromised accounts (Names & NetIDs) b. Target sites (Duke HR/Payroll)
2. What log sources have that data? a. Shibboleth IdP Process
3. Any correlaLon between accounts? a. Phishing aDacks?
(Direct Deposit ModificaLon NoLficaLon)
26
And So the Security InvesLgaLon Begins
![Page 27: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/27.jpg)
Analysis (grep) 2013-‐11-‐24:15:09:29.492 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site1), 2013-‐11-‐24:15:09:46.143 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site2),
2013-‐11-‐24:15:39:32.285 IP_address=BadActorIP1, User=NetID#2, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:15:43:30.986 IP_address=BadActorIP1, User=NetID#3, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:15:54:22.372 IP_address=BadActorIP1, User=NetID#4, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:16:00:00.158 IP_address=BadActorIP1, User=NetID#5, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:16:45:19.111 IP_address=BadActorIP1, User=NetID#3, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:16:53:04.594 IP_address=BadActorIP1, User=NetID#6, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:23:54:12.213 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐24:23:54:41.797 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site2),
2013-‐11-‐24:23:58:06.624 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐28:01:15:10.517 IP_address=BadActorIP2, User=NetID#2, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐28:01:16:29.581 IP_address=BadActorIP2, User=NetID#2, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐28:01:19:39.436 IP_address=BadActorIP2, User=NetID#2, enLtyId=hDps://duke.edu(site2),
2013-‐11-‐29:16:39:10.052 IP_address=BadActorIP3, User=NetID#7, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐29:16:53:51.329 IP_address=BadActorIP3, User=NetID#8, enLtyId=hDps://duke.edu(site1),
2013-‐11-‐29:16:54:18.033 IP_address=BadActorIP3, User=NetID#8, enLtyId=hDps://duke.edu(site2),
27
![Page 28: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/28.jpg)
Shibboleth (Single Sign On) Logs
! Shibboleth = open source middleware for idenLty management based on SAML
! IdP Process Logs – Example: Lmestamp – INFO [IdP AuthN Provider] – IP Address, User,
EnLtyId, Success/Failure Message
1. IP Address = Source IP 2. User = NetID 3. EnLtyId = Target Website 4. Success = [password] OR [SSO]
28
![Page 29: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/29.jpg)
Props.conf (Shib)
29
![Page 30: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/30.jpg)
Let the Splunking Begin
evenDype=shiblogin User=$User$ | iplocaLon shib_src | ldapfilter domain=Duke search=“(uid=$User$)” aDrs=“Name,Affilia*on,Dept,Title,Phone” | table _Lme, IP, City, Region, Country, Site, NetID, Name, Dept, Title, AffiliaLon, Phone | sort NetID, -‐_Lme
30
(Form – Enter UserID, Seach Shib for AuthN Logs)
![Page 31: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/31.jpg)
And the IP Address…
evenDype=shiblogin IP=“IP address” | iplocaLon IP | ldapfilter domain=Duke search=“(uid=$User$)” aDrs=“Name,Affilia*on,Dept,Title,Phone” | table _Lme, IP, City, Region, Country, Site, NetID, Name, Dept, Title, AffiliaLon, Phone | sort NetID, -‐_Lme
EssenLally the same search as the previous slide, modified to query for IP rather than UserID
31
![Page 32: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/32.jpg)
Once the Bleeding Stops…
! How do we uLlize Splunk to become more proacLve (Shibboleth)? 1. Look for “non-‐Duke” IPs with mulLple user logins to HR/Payroll 2. Non-‐Duke IPs with mulLple user logins regardless of desLnaLon 3. Query the number of ciLes an account has logged in from (24hrs) 4. Query the number of countries an account has logged in from
(24hrs)
32
![Page 33: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/33.jpg)
HR/Payroll Logins – MulLple Users, Single IP evenDype=shiblogin (site=“DukeHR” OR site=“DukePayroll”) NOT (IP=“DukePublic” OR IP=“DukePrivate”) | iplocaLon IP
| where state!=NC 0R where country!=US (opLonal) | stats dc(NetID) AS “User_Count”, values(NetID) by IP | where User_Count > 1 | table IP User_Count NetID | sort –User_Count
! Easy to take out HR/Payroll specific search to look for trends regardless of desLnaLon
33
![Page 34: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/34.jpg)
Single User, MulLple CiLes (24hrs) evenDype=shiblogin | geoip IP | stats dc(geo_info) AS “Number_of_CiLes”, values(geo_info) AS CiLes by NetID | where Number_of_CiLes > 2 | table NetID Number_of_CiLes CiLes | sort –Number_of_CiLes
! Similarly, tweak the search to look for countries rather than ciLes
34
![Page 35: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/35.jpg)
What Else?
! Phishing recipients from Messaging – | inputlookup DukePhish.csv
! Direct deposit modificaLons – | inputlookup DDtransacLons.csv
! How do we incorporate these two and begin reporLng?
35
![Page 36: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/36.jpg)
High Risk Report (Incorporate Phish & DD lists to look for access from outside the
US OR IPs with mulLple UserIDs authenLcaLng)
! evenDype=shiblogin [| inputlookup DirectDeposits.csv | fields NetID]
| iplocaLon IP | eval foreign=if(Country!="United States","true","false")
| fields …
| join NetID [| inputlookup PhishList.csv]
|where (PhishTime <= DDChanged) AND Site=DukeSite.edu | stats dc(NetID) AS User_Count values(ShibLoginTime) by NetID values(Site) values(IP) values(City) values(Region) values(Country) values(PhishReceived) values(DepositChanged)
| eval mulL=if(User_Count>1,"true","false")
| where (foreign="true" OR mulL="true")
36
![Page 37: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/37.jpg)
The Fight ConLnues… ! MulLple IteraLons of DD Phish ! Team moves from Shib-‐only invesLgaLon > Web Hits ! Web log info to see IP info of “aDackers” + vicLms ! Chum Accounts
This type of visibility into logs just did not exist for the ITSO prior to Splunk It becomes organizaLonal, and not just the responsibility of security
! App developments ! MFA
37
![Page 38: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/38.jpg)
Because They Won’t Stop
38
![Page 39: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/39.jpg)
Splunk and Security CorrelaLons, EvaluaLons, and Risk Scoring
![Page 40: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/40.jpg)
When to Catch Direct Deposit Phishing?
! Before the users are Phished ! AUer the users submit their credenLals but before the aDackers try and use them
! AUer the aDackers change the direct deposit accounts but before payday
! AUer payday when users contact us about missing paychecks
40
![Page 41: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/41.jpg)
List of ADributes of Direct Deposit Changes ! Which countries ! How many countries ! Which regions (States) ! How many regions ! Coming from campus IP address ! How many IP addresses ! How many users came from the same IP address
! How many direct deposit changes
! Were the users phished ! How many Lmes were they phished
! Were the users phished before any of the direct deposit changes
! Does the user have mulL-‐factor AuthenLcaLon enabled
! Number of web hits
41
![Page 42: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/42.jpg)
No single aDribute/data point indicates the direct deposit informaLon has been modified
Except when the user calls to say they did not get paid
42
![Page 43: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/43.jpg)
Assigning numeric values to aDributes
WeighLng the numeric values
Filtering based on weight
Example of WeighLng
<Initial search> | iplocation clientip | stats dc(Country) as countrycnt dc(Region) as regioncnt dc(clientip) as ipcnt by user | eval risk=0 | eval risk=risk + ((countrycnt - 1 ) * 10) | eval risk=risk + ((regioncount - 1 ) * 5 ) | eval risk=risk + ((ipcnt - 1 ) * 3 ) | where risk > 15 | sort -risk
43
![Page 44: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/44.jpg)
User Name DD change
DD count
Phish Dates
Phish Count
Countries Country Risk
IP Count Max Users /
IP
Total Risk
juser1 Jane User
09/05 09/20 09/21
3 07/24 08/15 09/21
3 Nigeria US
Russia
21 14 12 25
juser2 Joseph User
09/12 09/21
2 09/20 1 US Germany
13 8 5 17
bprof5 Robert Prof
09/15 1 0 Cuba 17 12 1 11
Weighted Dashboard
44
![Page 45: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/45.jpg)
User AcLvity Data Sources
! Single sign on ! AuthenLcated web hits ! VPN access ! SSH logins ! Phishing emails received
! Outbound email ! Windows authenLcaLon ! Account locks and unlocks ! LDAP informaLon ! Direct deposit changes
45
![Page 46: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/46.jpg)
( index=sso username=$user$ ) OR ( index=web user=$user$ (status=200 OR status=3*) ) OR ( index=linux process=sshd user=$user$ (action=”Accepted” OR action=”Failed”) ) OR ( index=mail sasl=$user$ ) OR ( etc )
First ADempt at a Combined Login Form
46
![Page 47: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/47.jpg)
Example Using Radio BuDons
47
![Page 48: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/48.jpg)
Example Using Radio BuDons <fieldset> <input type="text" token="search_netid"> <label>NetID</label> <suffix/> </input> <input type=”radio” token=”mail_button”> <label>Mail Logins</label> <default>YES</default> <choice value=”index=mail”>YES</choice> <choice value=”index=NONESUCH”>NO</choice> </input> <input type="time"> <default>Last 7 days</default> <label>Time</label> </input> <fieldset> ~ <searchString>$mail_button$ host=”mail-gw-*” user=$search_netid$</searchString>
48
![Page 49: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/49.jpg)
User InvesLgaLon Form
49
![Page 50: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/50.jpg)
Example of Combining Details Using Case
| eval combined_event_type=case( isnotnull( vpn_login ), “VPN Login”, isnotnull( sasl_username ), “EMAIL”, match( event_action, “lock” ), “Account Lock”, match( event_action, “unlock” ), “Account UNLock”, isnotnull( clientip ) AND isnotnull( user ), “WEB HIT”, 1=1, “-”
)
50
![Page 51: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/51.jpg)
Further Case Example
| eval combined_notes1=case( isnotnull( vpn_login ), “username = “.vpn_username, isnotnull( sasl_username ), “mail client = “.client match( event_action, “lock” ), “locked by = “.done_by.”, reason = “.lckrsn, isnotnull( clientip ) AND isnotnull( user ), “host = “.host 1=1, “-”
)
51
![Page 52: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/52.jpg)
Complete User Login Form
52
![Page 53: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/53.jpg)
Weighted Dashboard
User Name DD change
DD count
Phish Dates
Phish Count
Countries Country Risk
IP Count
Max Users /
IP
Total Risk
juser1 Jane User
09/05 09/20 09/21
3 07/24 08/15 09/21
3 Nigeria US
Russia
21 14 12 25
juser2 Joseph User
09/12 09/21
2 09/20 1 US Germany
13 8 5 17
bprof5 Robert Prof
09/15 1 0 Cuba 17 12 1 11
53
![Page 54: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/54.jpg)
InvesLgaLng juser1's Logins
54
![Page 55: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/55.jpg)
Jane User's Logins
_time event_type user src_ip host location notes1 notes2 2014-09-21 15:23:44
SSO Login juser1 152.3.14.90 Dhcp-152.3.14.90
Durham-NC-United States
EntityID=https://hr.duke.edu/
success=mfa
2014-09-15 06:14:00
Mail Login juser1 91.236.24.90 Host90.msu.ru Moscow—Russia - -
2014-09-14 05:56:14
VPN Login juser1 91.236.24.95 Host95.msu.ru Moscow--Russia
inner_vpn_ip = 192.168.140.8 -
2014-09-05 10:21:21
SSO Login juser1 152.3.14.88 Dhcp-152.3.14.88
Durham-NC-United States
EntityID=https://hr.duke.edu/
success=mfa
55
![Page 56: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/56.jpg)
Joe User's Logins
_time event_type user src_ip host location notes1 notes2 2014-09-21 13:44:00
SSO Login juser2 192.168.14.50 Internal1450.duke.edu -- EntityID=traini
ng.duke.edu success=password
2014-09-21 13:21:15
SSO Login juser2 85.17.10.131 - Munich--Germany
EntityID=hr.duke.edu
success=password
2014-09-21 13:17:22
SSH Login juser2 85.17.10.131 - Munich--Germany
Server=login-01.duke.edu -
2014-09-21 13:11:55
Mail Login juser2 192.168.14.50 Internal1450.duke.edu -- - -
56
![Page 57: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/57.jpg)
Benefits of Splunk to Duke ! Splunk allowed Duke to begin leveraging mulLple data sources almost immediately with very liDle ramp-‐up Lme
! Detailed informaLon on recipients of phishing messages is now available to Security in minutes instead of hours
! Splunk has allowed Duke to more than double the number of compromised accounts we detect and lock each month
! Splunk provided the ability to create a custom SIEM like soluLon tailored to Duke’s needs
57
![Page 58: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/58.jpg)
Key Takeaways 1. Use Splunk to bridge the gap between teams and log knowledge 2. Use event types, macros, and saved searches to make your long
crazy searches usable by others 3. Decide what is suspicious for your use case 4. Use the eval command to create custom weighLng algorithms 5. ConLnue to educate your users about Phishing
58
![Page 59: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/59.jpg)
Q&A
![Page 60: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/60.jpg)
60
Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday Geek out, share ideas with Enterprise Security developers
Red Team / Blue Team -‐ Challenge your skills and learn new tricks Mon-‐Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM
Learn, share and hack
Birds of a feather-‐ Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room
![Page 61: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/61.jpg)
THANK YOU
![Page 62: How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staff* Total*of*68,000+AcLve*Users* University*and*Medical*Center*](https://reader033.vdocument.in/reader033/viewer/2022050206/5f594c7d71a81460094b97af/html5/thumbnails/62.jpg)
Macro: message_by_subject(2) index=mail S="*$subject$" $vars$ | makemv delim="> t=<" pmx_to | rename … | mvexpand to | rex field=f "\<(?<from>.+)\>" | rex field=S "\??q?\??(?<subject>.+)" | eval subject=replace(subject,"_"," ")
62