threat hunting with splunk
TRANSCRIPT
ThreatHuntingwithSplunkPresenter:KenWestin,M.Sc,OSCPSplunk,SecurityMarketSpecialist
Agenda
• ThreatHuting Basics
• ThreatHuntingDataSources
• Sysmon EndpointData
• CyberKillChain
• WalkthroughofAttackScenarioUsingCoreSplunk (handson)
• EnterpriseSecurityWalkthrough
• ApplyingMachineLearningandDataSciencetoSecurity
LogInCredentialsJanuary,February&March https://od-how-calgary2.splunkoxygen.com
April,May&June https://od-how-calgary3.splunkoxygen.com
July,August&September https://od-how-calgary4.splunkoxygen.com
October,November&December https://od-how-calgary5.splunkoxygen.com
User:hunterPass:pr3dator
Thesewon’twork…
AmIintherightplace?
Somefamiliaritywith…
● CSIRT/SOCOperations
● GeneralunderstandingofThreatIntelligence
● GeneralunderstandingofDNS,Proxy,andEndpointtypesofdata
5
Thisisahands-onsession.
Theoverviewslidesareimportantforbuildingyour“hunt”methodology
10minutes- Seriously.
HowZeusCybercrimeWorks
ThreatHuntingwithSplunk
8
Vs.
SANSThreatHuntingMaturity
9
AdHocSearch
StatisticalAnalysis
VisualizationTechniques
Aggregation MachineLearning/DataScience
85%55%50%48%32%
Source:SANSIR&ThreatHuntingSummit2016
HuntingTools:InternalData
10
• IPAddresses:threatintelligence,blacklist,whitelist,reputationmonitoringTools:Firewalls,proxies,Splunk Stream,Bro,IDS
• NetworkArtifactsandPatterns:networkflow,packetcapture,activenetworkconnections,historicnetworkconnections,portsandservicesTools:Splunk Stream,BroIDS,FPC,Netflow
• DNS:activity,queriesandresponses,zonetransferactivityTools:Splunk Stream,BroIDS,OpenDNS
• Endpoint– HostArtifactsandPatterns:users,processes,services,drivers,files,registry,hardware,memory,diskactivity,filemonitoring:hashvalues,integritycheckingandalerts,creationordeletionTools:Windows/Linux,CarbonBlack,Tanium,Tripwire,ActiveDirectory
• VulnerabilityManagementDataTools:TripwireIP360,Qualys,Nessus
• UserBehaviorAnalytics:TTPs,usermonitoring,timeofdaylocation,HRwatchlistSplunk UBA,(Alloftheabove)
LogInCredentialsJanuary,February&March https://od-how-calgary2.splunkoxygen.com
April,May&June https://od-how-calgary3.splunkoxygen.com
July,August&September https://od-how-calgary4.splunkoxygen.com
October,November&December https://od-how-calgary5.splunkoxygen.com
User:hunterPass:pr3dator
Endpoint:MicrosoftSysmonPrimer
12
● TAAvailableontheAppStore
● GreatBlogPosttogetyoustarted
● IncreasesthefidelityofMicrosoftLogging
BlogPost:http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/
LogInCredentialsJanuary,February&March https://od-how-calgary2.splunkoxygen.com
April,May&June https://od-how-calgary3.splunkoxygen.com
July,August&September https://od-how-calgary4.splunkoxygen.com
October,November&December https://od-how-calgary5.splunkoxygen.com
User:hunterPass:pr3dator
SysmonEventTags
14
MapsNetworkCommtoprocess_id
Process_idcreationandmappingtoparentprocess_id
sourcetype=X*|searchtag=communicate
15
sourcetype=X*|deduptag|searchtag=process
16
DataSourceMapping
DemoStory- KillChainFrameworkSuccessfulbruteforce– downloadsensitivepdfdocument
WeaponizethepdffilewithZeusMalware
Convincingemailsentwithweaponizedpdf
Vulnerablepdfreaderexploitedbymalware.Droppercreatedonmachine
Dropperretrievesandinstallsthemalware
Persistenceviaregularoutboundcomm
DataExfiltration
Source:LockheedMartin
Servers
Storage
DesktopsEmail Web
TransactionRecords
NetworkFlows
DHCP/DNS
HypervisorCustomApps
PhysicalAccess
Badges
ThreatIntelligence
Mobile
CMDB
IntrusionDetection
Firewall
DataLossPrevention
Anti-Malware
VulnerabilityScans
Traditional
Authentication
StreamInvestigations– chooseyourdatawisely
19
20
Let’sdigin!
Please,raisethathandifyouneedustohitthepausebutton
APTTransactionFlowAcrossDataSources
21
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
Svchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
OurInvestigationbeginsbydetectinghighriskcommunicationsthroughtheproxy,attheendpoint,andevenaDNScall.
Tobeginourinvestigation,wewillstartwithaquicksearchtofamiliarizeourselveswiththedatasources.
Inthisdemoenvironment,wehaveavarietyofsecurityrelevantdataincluding…
WebDNSProxyFirewallEndpointEmail
Takealookattheendpointdatasource.WeareusingtheMicrosoftSysmon TA.
Wehaveendpointvisibilityintoallnetworkcommunicationandcanmapeachconnectionbacktoaprocess.
}Wealsohavedetailedinfooneachprocessandcanmapitbacktotheuserandparentprocess.}
Letsgetourdaystartedbylookingusingthreatintel toprioritizeoureffortsandfocusoncommunicationwithknown highriskentities.
WehavemultiplesourceIPscommunicatingtohighriskentitiesidentifiedbythese2threatsources.
Weareseeinghighriskcommunicationfrommultipledatasources.
Weseemultiplethreatintel relatedeventsacrossmultiplesourcetypesassociatedwiththeIPAddressofChrisGilbert.Let’stakecloserlookattheIPAddress.
Wecannowseetheownerofthesystem(ChrisGilbert)andthatitisn’taPIIorPCIrelatedasset,sotherearenoimmediatebusinessimplicationsthatwouldrequireinformingagenciesorexternalcustomerswithinacertaintimeframe.
Thisdashboardisbasedoneventdatathatcontainsathreatintelbasedindicatormatch(IPAddress,domain,etc.).ThedataisfurtherenrichedwithCMDBbasedAsset/identityinformation.
Wearenowlookingatonlythreatintel relatedactivityfortheIPAddressassociatedwithChrisGilbertandseeactivityspanningendpoint,proxy,andDNSdatasources.
Thesetrendlinestellaveryinterestingvisualstory.ItappearsthattheassetmakesaDNSqueryinvolvingathreatintel relateddomainorIPAddress.
ScrollDo
wn
Scrolldownthedashboardtoexaminethesethreatintel eventsassociatedwiththeIPAddress.
Wethenseethreatintel relatedendpointandproxyeventsoccurringperiodicallyandlikelycommunicatingwithaknownZeusbotnetbasedonthethreatintelsource(zeus_c2s).
It’sworthmentioningthatatthispointyoucouldcreateatickettohavesomeonere-imagethemachinetopreventfurtherdamageaswecontinueourinvestigationwithinSplunk.
Withinthesamedashboard,wehaveaccesstoveryhighfidelityendpointdatathatallowsananalysttocontinuetheinvestigationinaveryefficientmanner.Itisimportanttonotethatnearreal-timeaccesstothistypeofendpointdataisnotnotcommonwithinthetraditionalSOC.
Theinitialgoaloftheinvestigationistodeterminewhetherthiscommunicationismaliciousorapotentialfalsepositive.Expandtheendpointeventtocontinuetheinvestigation.
Proxyrelatedthreatintel matchesareimportantforhelpingustoprioritizeoureffortstowardinitiatinganinvestigation.Furtherinvestigationintotheendpointisoftenverytimeconsumingandofteninvolvesmultipleinternalhand-offstootherteamsorneedingtoaccessadditionalsystems.Thisencryptedproxytrafficisconcerningbecauseofthelargeamountofdata(~1.5MB)beingtransferredwhichiscommonwhendataisbeingexfiltrated.
Exfiltrationofdataisaseriousconcernandoutboundcommunicationtoexternalentitythathasaknownthreatintelindicator,especiallywhenitisencryptedasinthiscase.
Letscontinuetheinvestigation.
Anotherclue.Wealsoseethatsvchost.exe shouldbelocatedinaWindowssystemdirectorybutthisisbeingrunintheuserspace.Notgood.
Weimmediatelyseetheoutboundcommunicationwith115.29.46.99viahttpsisassociatedwiththesvchost.exeprocessonthewindowsendpoint.Theprocessidis4768.ThereisagreatdealmoreinformationfromtheendpointasyouscrolldownsuchastheuserIDthatstartedtheprocessandtheassociatedCMDBenrichmentinformation.
WehaveaworkflowactionthatwilllinkustoaProcessExplorerdashboardandpopulateitwiththeprocessidextractedfromtheevent(4768).
ThisisastandardWindowsapp,butnotinitsusualdirectory,tellingusthatthemalwarehasagainspoofedacommonfilename.
Wealsocanseethattheparentprocessthatcreatedthissuspicuous svchost.exe processiscalledcalc.exe.
ThishasbroughtustotheProcessExplorerdashboardwhichletsusviewWindowsSysmon endpointdata.
SuspectedMalware
Letscontinuetheinvestigationbyexaminingtheparentprocessasthisisalmostcertainlyagenuinethreatandwearenowworkingtowardarootcause.
ThisisveryconsistentwithZeusbehavior.TheinitialexploitationgenerallycreatesadownloaderordropperthatwillthendownloadtheZeusmalware.Itseemslikecalc.exemaybethatdownloader/dropper.
SuspectedDownloader/Dropper
Thisprocesscallsitself“svchost.exe,”acommonWindowsprocess,butthepathisnotthenormalpathforsvchost.exe.
…whichisacommontraitofmalwareattemptingtoevadedetection.WealsoseeitmakingaDNSquery(port53)thencommunicatingviaport443.
TheParentProcessofoursuspecteddownloader/dropperisthelegitimatePDFReaderprogram.Thiswilllikelyturnouttobethevulnerableappthatwasexploitedinthisattack.
SuspectedDownloader/Dropper
SuspectedVulnerableAppWehaveveryquicklymovedfromthreatintel relatednetworkandendpointactivitytothelikelyexploitationofavulnerableapp.Clickontheparentprocesstokeepinvestigating.
WecanseethatthePDFReaderprocesshasnoidentifiedparentandistherootoftheinfection.
ScrollDo
wn
ScrolldownthedashboardtoexamineactivityrelatedtothePDFreaderprocess.
Chrisopened2nd_qtr_2014_report.pdfwhichwasanattachmenttoanemail!
Wehaveourrootcause!Chrisopenedaweaponized .pdf filewhichcontainedtheZeusmalware.Itappearstohavebeendeliveredviaemailandwehaveaccesstoouremaillogsasoneofourimportantdatasources.Letscopythefilename2nd_qtr_2014_report.pdfandsearchabitfurthertodeterminethescopeofthiscompromise.
Letsdigalittlefurtherinto2nd_qtr_2014_report.pdftodeterminethescopeofthiscompromise.
Letssearchthoughmultipledatasourcestoquicklygetasenseforwhoelsemayhavehavebeenexposedtothisfile.
Wewillcomebacktothewebactivitythatcontainsreferencetothepdf filebutletsfirstlookattheemaileventtodeterminethescopeofthisapparentphishingattack.
Wehaveaccesstotheemailbodyandcanseewhythiswassuchaconvincingattack.Thesenderapparentlyhadaccesstosensitiveinsiderknowledgeandhintedatquarterlyresults.
Thereisourattachment.
HoldOn!That’snotourDomainName!Thespellingisclosebutit’smissinga“t”.TheattackerlikelyregisteredadomainnamethatisveryclosetothecompanydomainhopingChriswouldnotnotice.
Thislookstobeaverytargetedspearphishingattackasitwassenttoonlyoneemployee(Chris).
RootCauseRecap
36
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
.pdfSvchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
Weutilizedthreatintel todetectcommunicationwithknownhighriskindicatorsandkickoffourinvestigationthenworkedbackwardthroughthekillchaintowardarootcause.
Keytothisinvestigativeprocessistheabilitytoassociatenetworkcommunicationswithendpointprocessdata.
ThishighvalueandveryrelevantabilitytoworkamalwarerelatedinvestigationthroughtorootcausetranslatesintoaverystreamlinedinvestigativeprocesscomparedtothelegacySIEMbasedapproach.
37
Letsrevisitthesearchforadditionalinformationonthe2nd_qtr_2014-_report.pdffile.
Weunderstandthatthefilewasdeliveredviaemailandopenedattheendpoint.Whydoweseeareferencetothefileintheaccess_combined (webserver)logs?
Selecttheaccess_combinedsourcetype toinvestigatefurther.
38
Theresultsshow54.211.114.134hasaccessedthisfilefromthewebportalofbuttergames.com.
ThereisalsoaknownthreatintelassociationwiththesourceIPAddressdownloading(HTTPGET)thefile.
39
SelecttheIPAddress,left-click,thenselect“Newsearch”.WewouldliketounderstandwhatelsethisIPAddresshasaccessedintheenvironment.
40
That’sanabnormallylargenumberofrequestssourcedfromasingleIPAddressina~90minutewindow.
Thislookslikeascriptedactiongiventheconstanthighrateofrequestsoverthebelowwindow.
ScrollDo
wn
Scrolldownthedashboardtoexamineotherinterestingfieldstofurtherinvestigate.
NoticetheGooglebotuseragent string whichisanotherattempttoavoidraisingattention..
41
Therequestsfrom52.211.114.134aredominatedbyrequeststotheloginpage(wp-login.php).It’sclearlynotpossibletoattemptaloginthismanytimesinashortperiodoftime– thisisclearlyascriptedbruteforceattack.
Aftersuccessfullygainingaccesstoourwebsite,theattackerdownloadedthepdf file,weaponized itwiththezeusmalware,thendeliveredittoChrisGilbertasaphishingemail.
Theattackerisalsoaccessingadminpageswhichmaybeanattempttoestablishpersistenceviaabackdoorintothewebsite.
KillChainAnalysisAcrossDataSources
42
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
Svchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
Wecontinuedtheinvestigationbypivotingintotheendpointdatasourceandusedaworkflowactiontodeterminewhichprocessontheendpointwasresponsiblefortheoutboundcommunication.
WeBeganbyreviewingthreatintel relatedeventsforaparticularIPaddressandobservedDNS,Proxy,andEndpointeventsforauserinSales.
Investigationcomplete!LetsgetthisturnedovertoIncidentReponse team.
Wetracedthesvchost.exeZeusmalwarebacktoit’sparentprocessIDwhichwasthecalc.exedownloader/dropper.
Onceourrootcauseanalysiswascomplete,weshiftedoutfocusintotheweblogstodeterminethatthesensitivepdffilewasobtainedviaabruteforceattackagainstthecompanywebsite.
Wewereabletoseewhichfilewasopenedbythevulnerableappanddeterminedthatthemaliciousfilewasdeliveredtotheuserviaemail.
Aquicksearchintothemaillogsrevealedthedetailsbehindthephishingattackandrevealedthatthescopeofthecompromisewaslimitedtojusttheoneuser.
Wetracedcalc.exe backtothevulnerableapplicationPDFReader.
WanttoFollowAlong?Optional– ForMachineLearningComponent
● DownloadSplunkhttp://www.splunk.com/en_us/download-21.html
● InstalltheMachineLearningToolkithttp://tiny.cc/splunkmlapp
● InstallthePythonforScientificComputingapphttps://splunkbase.splunk.com/app/2881/ (Mac)https://splunkbase.splunk.com/app/2883/ (Windows)
Break!
Splunk EnterpriseSecurity
SANSThreatHuntingMaturity
46
AdHocSearch
StatisticalAnalysis
VisualizationTechniques
Aggregation MachineLearning/DataScience
85%55%50%48%32%
Source:SANSIR&ThreatHuntingSummit2016
SplunkistheSecurityNerveCenter
AppServers
Network
ThreatIntelligence
Firewall
WebProxy
InternalNetworkSecurity
Identity
Endpoints
GartnerMagicQuadrantforSIEM
48
IncidentResponse
UserMonitoring
Data&AppMonitoring
AdvancedAnalytics
BasicSecurityMonitoring
AdvancedThreatDetection
Forensic&IncidentResponse
Real-TimeMonitoring
AdvancedThreatDefense
BusinessContext&SecurityIntelligence
Deployment&SupportSimplicity
Homework
OtherItemsToNote
ItemstoNote
Navigation- HowtoGetHere
Descriptionofwhattoclickon
Click
KeySecurityIndicators(buildyourown!)
Sparklines
Editable
Variouswaystofilterdata
Malware-SpecificKSIsandReports
SecurityDomains->Endpoint->MalwareCenter
Filterable
KSIsspecifictoRisk
Riskassignedtosystem,userorother
UnderAdvancedThreat,selectRiskAnalysis
(ScrollDown)
RecentRiskActivity
UnderAdvancedThreat,selectRiskAnalysis
Filterable,downtoIoC
KSIsspecifictoThreat
Mostactivethreatsource
Scrolldown… Scroll
UnderAdvancedThreat,selectThreatActivity
Specificsaboutrecentthreatmatches
UnderAdvancedThreat,selectThreatActivity
Toaddthreatintelgoto:Configure->DataEnrichment->ThreatIntelligenceDownloads
Click
Click“ThreatArtifacts”Under“AdvancedThreat”
Click
ArtifactCategories–clickdifferenttabs…
STIXfeed
Customfeed
UnderAdvancedThreat,selectThreatArtifacts
ReviewtheAdvancedThreatcontent
Click
DatafromassetframeworkConfigurableSwimlanes
Darker=moreevents
AllhappenedaroundsametimeChangeto“Today”ifneeded
AssetInvestigator,enter“192.168.56.102”
DataScience&MachineLearningInSecurity
62
EvolutionofSecurityCorrelationtoAdvancedAnalytics
63
Two-Dimensional- Correlation:Useofregexandpatternmatchingforstrings.Usedinanti-malware,IDS/IPS,DLPandbasicSIEM.Useofstringmatchingtosearchabinaryfiletoidentifytypeofthreat.Enhancedcapabilitytoidentifypreviouslyknownthreatsandhostenumerationwithinanenvironment.Multi-Dimensional- Analytics:Hybridmodeldevelopedasadversarieslearnedtocircumventbasiccorrelationandtoreducefalsepositives.Thresholdsandcombinationsofrulesdeveloped.Startingtocreatebehavioralmodels,statisticalanalysisandpatternidentificationnotjustbasedonsignatures.
One-Dimensional- Correlation:Fastandefficientbasicmatchingofdomains,IPaddresses,user-agent,MD5filehashes.UseofBooleanoperatorstoidentifyifsignatureisonablack/whitelist.CommonusageinmostfirewallandIDStools.
N-DimensionalAdvanced- Analytics:Shiftawayfromheavymanualtaggingandrulebuildingalone,leveragesadvancedandpredictiveanalytics,machinelearning,graphanalysisandelementsofdatasciencetoenhancetheanalysttoidentifypreviouslyunknownthreats,shiftfromcorrelationtocausation.
SANSThreatHuntingMaturity
64
AdHocSearch
StatisticalAnalysis
VisualizationTechniques
Aggregation MachineLearning/DataScience
85%55%50%48%32%
Source:SANSIR&ThreatHuntingSummit2016
Disclaimer:Iamnotadatascientist
TypesofMachineLearningSupervised Learning:generalizingfromlabeled data
SupervisedMachineLearning
67
DomainName TotalCnt RiskFactor AGD SessionTime RefEntropy NullUa Outcome
yyfaimjmocdu.com 144 6.05 1 1 0 0 Maliciousjjeyd2u37an30.com 6192 5.05 0 1 0 0 Maliciouscdn4s.steelhousemedia.com 107 3 0 0 0 0 Benignlog.tagcade.com 111 2 0 1 0 0 Benigngo.vidprocess.com 170 2 0 0 0 0 Benignstatse.webtrendslive.com 310 2 0 1 0 0 Benigncdn4s.steelhousemedia.com 107 1 0 0 0 0 Benignlog.tagcade.com 111 1 0 1 0 0 Benign
Unsupervised Learning:generalizingfromunlabeled data
UnsupervisedMachineLearning
• Notuning
• Programmaticallyfindstrends
• UBAisprimarilyunsupervised
• Rigorouslytestedforfit
69
AlgorithmRawSecurityData AutomatedClustering
70
MLToolkit&Showcase• SplunkSupportedframeworkforbuildingMLApps
– Getitforfree:http://tiny.cc/splunkmlapp
• LeveragesPythonforScientificComputing (PSC)add-on:– Open-sourcePythondatascienceecosystem– NumPy,SciPy,scitkit-learn,pandas,statsmodels
• Showcaseusecases:PredictHardDriveFailure,ServerPowerConsumption,ApplicationUsage,CustomerChurn&more
• Standardalgorithms outofthebox:– Supervised:LogisticRegression,SVM,LinearRegression,RandomForest,etc.– Unsupervised: KMeans,DBSCAN,SpectralClustering,PCA,KernelPCA,etc.
• Implementoneof300+algorithmsbyeditingPythonscripts
MachineLearningToolkitDemo
72
Splunk UBA
Splunk UBAUseCases
ACCOUNTTAKEOVER• Privilegedaccountcompromise• Dataexfiltration
LATERALMOVEMENT• Pass-the-hashkillchain• Privilegeescalation
SUSPICIOUSACTIVITY• Misuseofcredentials• Geo-locationanomalies
MALWAREATTACKS• Hiddenmalwareactivity
BOTNET,COMMAND&CONTROL• Malwarebeaconing• Dataleakage
USER&ENTITYBEHAVIORANALYTICS• Suspiciousbehaviorbyaccountsor
devices
EXTERNALTHREATSINSIDERTHREATS
SplunkUserBehaviorAnalytics(UBA)• ~100%ofbreachesinvolvevalidcredentials(Mandiant Report)• Needtounderstandnormal&anomalousbehaviorsforALLusers• UBAdetectsAdvancedCyberattacks andMaliciousInsiderThreats• LotsofMLunderthehood:
– BehaviorBaselining&Modeling– AnomalyDetection(30+models)– AdvancedThreatDetection
• E.g.,DataExfil Threat:– “Sawthisstrangelogin&datatransferfor userkwestin
at3aminChina…”– SurfacethreattoSOCAnalysts
Workflow
Raw Events
1
Statistical methods
Security semantics
2Threat Models
Lateralmovement
ML
Patterns
Sequences
Beaconing
Land-speedviolation
Threats
Kill chain sequence
5
Supporting evidence
Threat scoring
Graph Mining
4
Con
tinuo
us s
elf-l
earn
ing
Anomalies graph
Entity relationship graph
3
Anomalies
Splunk UBADemo
78
SecurityWorkshops
● SecurityReadinessAssessments(CSC20)● Splunk UBADataScienceWorkshop● EnterpriseSecurityBenchmarkAssessment● InsiderThreat
SecurityWorkshopSurvey
https://www.surveymonkey.com/r/KFVLF37