threat hunting workshop

74
Threat Hunting with Splunk Presenter: Ken Westin, M.Sc, OSCP Splunk, Security Market Specialist

Upload: splunk

Post on 16-Apr-2017

1.379 views

Category:

Technology


0 download

TRANSCRIPT

ThreatHuntingwithSplunkPresenter: KenWestin,M.Sc,OSCPSplunk,SecurityMarketSpecialist

Agenda

• ThreatHutingBasics

• ThreatHuntingDataSources

• Sysmon EndpointData

• CyberKillChain

• WalkthroughofAttackScenarioUsingCoreSplunk (handson)

• EnterpriseSecurityWalkthrough

• ApplyingMachineLearningandDataSciencetoSecurity

LogInCredentials

January,February&March https://od-norcal-2.splunkoxygen.comApril,May&June https://od-norcal-3.splunkoxygen.comJuly,August&September https://od-norcal-4.splunkoxygen.comOctober,November&December https://od-norcal-5.splunkoxygen.com

User:hunterPass:pr3dator

BirthMonth

Thesewon’twork…

AmIintherightplace?

Somefamiliaritywith…

● CSIRT/SOCOperations

● GeneralunderstandingofThreatIntelligence

● GeneralunderstandingofDNS,Proxy,andEndpointtypesofdata

5

Thisisahands-onsession.

Theoverviewslidesareimportantforbuildingyour“hunt”methodology

10minutes- Seriously.

ThreatHuntingwithSplunk

7

Vs.

SANSThreatHuntingMaturity

8

AdHocSearch

StatisticalAnalysis

VisualizationTechniques

Aggregation Machine Learning/DataScience

85%55%50%48%32%

Source:SANSIR&ThreatHuntingSummit2016

HuntingTools:InternalData

9

• IPAddresses:threatintelligence,blacklist,whitelist, reputationmonitoringTools:Firewalls, proxies, Splunk Stream,Bro,IDS

• NetworkArtifactsandPatterns:networkflow,packetcapture,activenetworkconnections, historicnetworkconnections, portsandservicesTools:Splunk Stream,BroIDS,FPC,Netflow

• DNS:activity,queries andresponses, zonetransferactivityTools:Splunk Stream,BroIDS,OpenDNS

• Endpoint– HostArtifactsandPatterns:users,processes, services, drivers, files, registry,hardware,memory, disk activity,filemonitoring: hashvalues, integritycheckingandalerts,creationordeletionTools:Windows/Linux, CarbonBlack,Tanium, Tripwire,ActiveDirectory

• VulnerabilityManagementDataTools:TripwireIP360,Qualys, Nessus

• UserBehaviorAnalytics:TTPs,usermonitoring, timeofdaylocation,HRwatchlistSplunk UBA,(All oftheabove)

LogInCredentialsJanuary,February&March https://od-norcal-2.splunkoxygen.com

April,May&June https://od-norcal-3.splunkoxygen.com

July,August&September https://od-norcal-4.splunkoxygen.com

October,November&December https://od-norcal-5.splunkoxygen.com

User:hunterPass:pr3dator

Endpoint:MicrosoftSysmonPrimer

11

● TAAvailable ontheAppStore

● GreatBlogPosttogetyoustarted

● Increases thefidelity ofMicrosoftLogging

BlogPost:http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/

Showapptoclickon

12

January,February&March https://od-norcal-2.splunkoxygen.comApril,May&June https://od-norcal-3.splunkoxygen.comJuly,August&September https://od-norcal-4.splunkoxygen.comOctober,November&December https://od-norcal-5.splunkoxygen.com

User:hunterPass:pr3dator

SysmonEventTags

14

MapsNetworkCommtoprocess_id

Process_id creation andmapping toparentprocess_id

sourcetype=X*|searchtag=communicate

15

sourcetype=X*|deduptag|searchtag=process

16

DataSourceMapping

DemoStory- KillChainFrameworkSuccessfulbruteforce– downloadsensitivepdfdocument

Weaponize thepdffilewithZeusMalware

Convincingemailsentwithweaponizedpdf

Vulnerablepdfreaderexploitedbymalware.Dropper createdonmachine

Dropper retrievesandinstallsthemalware

Persistence viaregularoutboundcomm

DataExfiltration

Source:LockheedMartin

Servers

Storage

DesktopsEmail Web

TransactionRecords

NetworkFlows

DHCP/DNS

HypervisorCustomApps

T

PhysicalAccess

Badges

ThreatIntelligence

Mobile

CMDB

IntrusionDetection

Firewall

DataLossPrevention

Anti-Malware

VulnerabilityScans

Traditional

Authentication

StreamInvestigations– chooseyourdatawisely

19

20

Let’sdigin!

Please,raisethathandifyouneedustohitthepausebutton

APTTransactionFlowAcrossDataSources

21

http(proxy)sessiontocommand&controlserver

RemotecontrolStealdataPersistincompanyRentasbotnet

Proxy

ConductBusiness

Createadditionalenvironment

GainAccesstosystemTransaction

ThreatIntelligence

Endpoint

NetworkEmail,Proxy,DNS,andWeb

DataSources

.pdf

.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs

Svchost.exe(malware)

Calc.exe(dropper)

AttackerhackswebsiteSteals.pdf files

WebPortal

.pdf

Attackercreatesmalware,embed in.pdf,

emailstothetarget

MAIL

Reademail,openattachment

OurInvestigationbeginsbydetectinghighriskcommunicationsthroughtheproxy,attheendpoint, andevenaDNScall.

Tobeginourinvestigation,wewillstartwithaquicksearchtofamiliarizeourselveswiththedatasources.

Inthisdemoenvironment,wehaveavarietyofsecurityrelevantdataincluding…

WebDNSProxyFirewallEndpointEmail

Takealookattheendpointdatasource. WeareusingtheMicrosoftSysmon TA.

Wehaveendpointvisibilityintoallnetworkcommunicationandcanmapeachconnectionbacktoaprocess.

}Wealsohavedetailedinfooneachprocessandcanmapitbacktotheuserandparentprocess.}

Letsgetourdaystartedbylookingusingthreatintel toprioritizeoureffortsandfocusoncommunicationwithknown highriskentities.

WehavemultiplesourceIPscommunicatingtohighriskentitiesidentifiedbythese2threatsources.

Weareseeinghighriskcommunicationfrommultipledatasources.

Weseemultiplethreatintel relatedeventsacrossmultiplesourcetypesassociatedwiththeIPAddressofChrisGilbert.Let’stakecloserlookattheIPAddress.

Wecannowseetheownerofthesystem(ChrisGilbert)andthatitisn’taPIIorPCIrelatedasset,sotherearenoimmediatebusinessimplicationsthatwouldrequireinformingagencies orexternal customerswithinacertaintimeframe.

Thisdashboardisbasedoneventdatathatcontainsathreatintelbasedindicatormatch(IPAddress,domain,etc.).ThedataisfurtherenrichedwithCMDBbasedAsset/identityinformation.

Wearenowlookingatonlythreatintel related activityfortheIPAddressassociatedwithChrisGilbertandseeactivityspanningendpoint,proxy,andDNSdatasources.

These trendlinestellaveryinterestingvisualstory.ItappearsthattheassetmakesaDNSqueryinvolvingathreatintel relateddomainorIPAddress.

ScrollDo

wn

Scrolldownthedashboardtoexamine thesethreatintel eventsassociatedwiththeIPAddress.

Wethenseethreatintel relatedendpointandproxyeventsoccurringperiodicallyandlikelycommunicatingwithaknownZeusbotnetbasedonthethreatintelsource(zeus_c2s).

It’sworthmentioningthatatthispointyoucouldcreateatickettohavesomeonere-image themachinetopreventfurtherdamageaswecontinueourinvestigationwithinSplunk.

Withinthesamedashboard,wehaveaccesstoveryhighfidelityendpointdatathatallowsananalysttocontinuetheinvestigationinaveryefficientmanner.Itisimportanttonotethatnearreal-timeaccesstothistypeofendpointdataisnotnotcommonwithinthetraditionalSOC.

Theinitialgoaloftheinvestigationistodeterminewhetherthiscommunicationismaliciousorapotentialfalsepositive.Expandtheendpointeventtocontinuetheinvestigation.

Proxyrelated threatintel matchesareimportantforhelpingustoprioritizeoureffortstowardinitiatinganinvestigation.Furtherinvestigationintotheendpointisoftenverytimeconsumingandofteninvolvesmultipleinternalhand-offstootherteamsorneeding toaccess additionalsystems.Thisencryptedproxytrafficisconcerningbecauseofthelargeamountofdata(~1.5MB)beingtransferredwhichiscommonwhendataisbeingexfiltrated.

Exfiltrationofdataisaseriousconcern andoutboundcommunicationtoexternal entitythathasaknownthreatintelindicator,especiallywhenitisencrypted asinthiscase.

Letscontinuetheinvestigation.

Anotherclue.Wealsoseethatsvchost.exe shouldbelocatedinaWindowssystemdirectorybutthisisbeingrunintheuserspace.Notgood.

Weimmediatelyseetheoutboundcommunicationwith115.29.46.99viahttpsisassociatedwiththesvchost.exeprocessonthewindowsendpoint. Theprocessidis4768.There isagreatdealmoreinformationfromtheendpointasyouscrolldownsuchastheuserIDthatstartedtheprocessandtheassociatedCMDBenrichment information.

WehaveaworkflowactionthatwilllinkustoaProcessExplorerdashboardandpopulateitwiththeprocessidextracted fromtheevent(4768).

ThisisastandardWindowsapp,butnotinitsusualdirectory, tellingusthatthemalwarehasagainspoofedacommonfilename.

Wealsocanseethattheparentprocessthatcreated thissuspicuous svchost.exe processiscalledcalc.exe.

ThishasbroughtustotheProcessExplorer dashboardwhichletsusviewWindowsSysmon endpointdata.

SuspectedMalware

Letscontinuetheinvestigationbyexaminingtheparentprocessasthisisalmostcertainlyagenuine threatandwearenowworkingtowardarootcause.

ThisisveryconsistentwithZeusbehavior.Theinitialexploitationgenerally createsadownloaderordropperthatwillthendownloadtheZeusmalware.Itseemslikecalc.exemaybethatdownloader/dropper.

SuspectedDownloader/Dropper

Thisprocesscallsitself“svchost.exe,”acommonWindowsprocess, butthepathisnotthenormalpathforsvchost.exe.

…whichisacommontraitofmalwareattemptingtoevadedetection.WealsoseeitmakingaDNSquery(port53)thencommunicatingviaport443.

TheParentProcessofoursuspecteddownloader/dropper isthelegitimatePDFReaderprogram.Thiswilllikelyturnouttobethevulnerable appthatwasexploitedinthisattack.

SuspectedDownloader/Dropper

SuspectedVulnerableAppWehaveveryquicklymovedfromthreatintel relatednetwork andendpointactivitytothelikelyexploitationofavulnerableapp.Clickontheparentprocesstokeepinvestigating.

WecanseethatthePDFReaderprocesshasnoidentifiedparentandistherootoftheinfection.

ScrollDo

wn

Scrolldownthedashboardtoexamine activityrelated tothePDFreaderprocess.

Chrisopened2nd_qtr_2014_report.pdfwhichwasanattachmenttoanemail!

Wehaveourrootcause!Chrisopened aweaponized .pdf filewhichcontained theZeusmalware. Itappearstohavebeendelivered viaemailandwehaveaccesstoouremaillogsasoneofourimportantdatasources.Letscopythefilename2nd_qtr_2014_report.pdf andsearchabitfurthertodeterminethescopeofthiscompromise.

Letsdigalittlefurtherinto2nd_qtr_2014_report.pdf todeterminethescopeofthiscompromise.

Letssearchthoughmultipledatasourcestoquicklygetasenseforwhoelsemayhavehavebeenexposed tothisfile.

Wewillcomebacktothewebactivitythatcontainsreference tothepdf filebutletsfirstlookattheemaileventtodetermine thescopeofthisapparentphishingattack.

Wehaveaccesstotheemailbodyandcanseewhythiswassuchaconvincingattack.Thesenderapparentlyhadaccesstosensitiveinsiderknowledge andhintedatquarterlyresults.

There isourattachment.

HoldOn!That’snotourDomainName!Thespellingisclosebutit’smissinga“t”.Theattackerlikelyregistered adomainnamethatisveryclosetothecompanydomainhopingChriswouldnotnotice.

Thislookstobeaverytargetedspearphishingattackasitwassenttoonlyoneemployee (Chris).

RootCauseRecap

36

DataSources

.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs

http(proxy)sessiontocommand&controlserver

RemotecontrolStealdataPersistincompanyRentasbotnet

Proxy

ConductBusiness

Createadditionalenvironment

GainAccesstosystemTransaction

ThreatIntelligence

Endpoint

NetworkEmail,Proxy,DNS,andWeb

.pdfSvchost.exe(malware)

Calc.exe(dropper)

AttackerhackswebsiteSteals.pdf files

WebPortal

.pdf

Attackercreatesmalware,embed in.pdf,

emailstothetarget

MAIL

Reademail,openattachment

Weutilizedthreatintel todetectcommunicationwithknownhighriskindicatorsandkickoffourinvestigationthenworkedbackwardthroughthekillchaintowardarootcause.

Keytothisinvestigativeprocessistheabilitytoassociatenetworkcommunicationswithendpointprocessdata.

Thishighvalueandveryrelevantabilitytoworkamalwarerelated investigationthroughtorootcausetranslatesintoaverystreamlinedinvestigativeprocesscomparedtothelegacy SIEMbasedapproach.

37

Letsrevisitthesearchforadditionalinformationonthe2nd_qtr_2014-_report.pdf file.

Weunderstandthatthefilewasdeliveredviaemailandopenedattheendpoint.Whydoweseeareference tothefileintheaccess_combined (webserver) logs?

Selecttheaccess_combinedsourcetype toinvestigatefurther.

38

Theresultsshow54.211.114.134hasaccessed thisfilefromthewebportalofbuttergames.com.

There isalsoaknownthreatintelassociationwiththesourceIPAddressdownloading (HTTPGET)thefile.

39

SelecttheIPAddress,left-click,thenselect“New search”.WewouldliketounderstandwhatelsethisIPAddresshasaccessed intheenvironment.

40

That’sanabnormallylargenumberofrequestssourcedfromasingleIPAddressina~90minutewindow.

Thislookslikeascriptedactiongiventheconstanthighrateofrequestsoverthebelowwindow.

ScrollDo

wn

Scrolldownthedashboardtoexamineotherinterestingfieldstofurtherinvestigate.

NoticetheGooglebotuseragent string whichisanotherattempttoavoidraisingattention..

41

Therequestsfrom52.211.114.134aredominatedbyrequeststotheloginpage(wp-login.php). It’sclearlynotpossibletoattemptaloginthismanytimesinashortperiodoftime– thisisclearlyascriptedbruteforceattack.

Aftersuccessfullygainingaccesstoourwebsite, theattackerdownloaded thepdf file,weaponized itwiththezeusmalware, thendelivered ittoChrisGilbertasaphishingemail.

Theattackerisalsoaccessingadminpageswhichmaybeanattempttoestablishpersistence viaabackdoorintotheweb site.

KillChainAnalysisAcrossDataSources

42

http(proxy)sessiontocommand&controlserver

RemotecontrolStealdataPersistincompanyRentasbotnet

Proxy

ConductBusiness

Createadditionalenvironment

GainAccesstosystemTransaction

ThreatIntelligence

Endpoint

NetworkEmail,Proxy,DNS,andWeb

DataSources

.pdf

.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs

Svchost.exe(malware)

Calc.exe(dropper)

AttackerhackswebsiteSteals.pdf files

WebPortal

.pdf

Attackercreatesmalware,embed in.pdf,

emailstothetarget

MAIL

Reademail,openattachment

Wecontinuedtheinvestigationbypivotingintotheendpointdatasourceandusedaworkflowactiontodeterminewhichprocessontheendpointwasresponsible fortheoutboundcommunication.

WeBeganbyreviewingthreatintel relatedeventsforaparticularIPaddressandobservedDNS,Proxy,andEndpointeventsforauserinSales.

Investigationcomplete!LetsgetthisturnedovertoIncident Reponse team.

Wetracedthesvchost.exeZeusmalwarebacktoit’sparentprocessIDwhichwasthecalc.exedownloader/dropper.

Onceourrootcauseanalysiswascomplete,weshiftedoutfocusintotheweblogstodetermine thatthesensitivepdffilewasobtainedviaabruteforceattackagainstthecompanywebsite.

Wewere abletoseewhichfilewasopenedbythevulnerable appanddetermined thatthemaliciousfilewasdeliveredtotheuserviaemail.

Aquicksearchintothemaillogsrevealed thedetailsbehindthephishingattackandrevealed thatthescopeofthecompromisewaslimitedtojusttheoneuser.

Wetracedcalc.exe backtothevulnerable applicationPDFReader.

WanttoFollowAlong?

● DownloadSplunk6.4.2http://www.splunk.com/en_us/download-21.html

● Download&InstalltheMachineLearningToolkithttp://tiny.cc/splunkmlapp

Break!

Splunk EnterpriseSecurity

OtherItemsToNote

ItemstoNote

Navigation- HowtoGetHere

Descriptionofwhattoclickon

Click

KeySecurityIndicators(buildyourown!)

Sparklines

Editable

Variouswaystofilterdata

Malware-Specific KSIsandReports

SecurityDomains->Endpoint->MalwareCenter

Filterable

KSIsspecifictoRisk

Riskassignedtosystem,userorother

UnderAdvancedThreat,selectRiskAnalysis

(ScrollDown)

RecentRiskActivity

UnderAdvancedThreat,selectRiskAnalysis

Filterable, downtoIoC

KSIsspecifictoThreat

Mostactivethreatsource

Scrolldown… Scroll

UnderAdvancedThreat,selectThreatActivity

Specificsaboutrecentthreatmatches

UnderAdvancedThreat,selectThreatActivity

Toaddthreatintelgoto:Configure ->DataEnrichment ->ThreatIntelligence Downloads

Click

Click“ThreatArtifacts”Under“AdvancedThreat”

Click

ArtifactCategories –clickdifferent tabs…

STIXfeed

Customfeed

UnderAdvancedThreat,selectThreatArtifacts

Review theAdvancedThreatcontent

Click

DatafromassetframeworkConfigurable Swimlanes

Darker=more events

AllhappenedaroundsametimeChangeto“Today”ifneeded

AssetInvestigator,enter“192.168.56.102”

DataScience&MachineLearningInSecurity

58

Disclaimer:Iamnotadatascientist

TypesofMachineLearningSupervised Learning:generalizingfromlabeled data

Supervised MachineLearning

61

DomainName TotalCnt RiskFactor AGD SessionTime RefEntropy NullUa Outcome

yyfaimjmocdu.com 144 6.05 1 1 0 0 Maliciousjjeyd2u37an30.com 6192 5.05 0 1 0 0 Maliciouscdn4s.steelhousemedia.com 107 3 0 0 0 0 Benignlog.tagcade.com 111 2 0 1 0 0 Benigngo.vidprocess.com 170 2 0 0 0 0 Benignstatse.webtrendslive.com 310 2 0 1 0 0 Benigncdn4s.steelhousemedia.com 107 1 0 0 0 0 Benignlog.tagcade.com 111 1 0 1 0 0 Benign

Unsupervised Learning:generalizingfromunlabeled data

Unsupervised MachineLearning

• Notuning

• Programmaticallyfindstrends

• UBAisprimarilyunsupervised

• Rigorouslytestedforfit

63

AlgorithmRawSecurityData AutomatedClustering

64

MLToolkit&Showcase• SplunkSupportedframeworkforbuildingMLApps

– Getitforfree:http://tiny.cc/splunkmlapp

• LeveragesPythonforScientificComputing (PSC)add-on:– Open-sourcePythondatascienceecosystem– NumPy,SciPy,scitkit-learn,pandas,statsmodels

• Showcaseusecases:PredictHardDriveFailure,ServerPowerConsumption,ApplicationUsage,CustomerChurn&more

• Standardalgorithmsoutofthebox:– Supervised:LogisticRegression,SVM,LinearRegression,RandomForest,etc.– Unsupervised: KMeans,DBSCAN,SpectralClustering,PCA,KernelPCA,etc.

• Implementoneof300+algorithmsbyeditingPythonscripts

MachineLearningToolkitDemo

66

Splunk UBA

Splunk UBAUseCases

ACCOUNTTAKEOVER• Privilegedaccountcompromise• Dataexfiltration

LATERALMOVEMENT• Pass-the-hashkillchain• Privilegeescalation

SUSPICIOUS ACTIVITY• Misuseofcredentials• Geo-locationanomalies

MALWARE ATTACKS• Hiddenmalwareactivity

BOTNET,COMMAND&CONTROL• Malwarebeaconing• Dataleakage

USER&ENTITY BEHAVIORANALYTICS• Suspiciousbehaviorbyaccountsor

devices

EXTERNAL THREATSINSIDERTHREATS

SplunkUserBehaviorAnalytics(UBA)• ~100%ofbreachesinvolvevalidcredentials(Mandiant Report)• Needtounderstandnormal&anomalousbehaviorsforALLusers• UBAdetectsAdvancedCyberattacks andMaliciousInsiderThreats• LotsofMLunderthehood:

– BehaviorBaselining&Modeling– AnomalyDetection(30+models)– AdvancedThreatDetection

• E.g.,DataExfil Threat:– “Sawthisstrangelogin&datatransferfor userkwestin

at3aminChina…”– SurfacethreattoSOCAnalysts

Workflow

Raw Events

1

Statistical methods

Security semantics

2 Threat ModelsLateralmovement

ML

Patterns

Sequences

Beaconing

Land-speedviolation

Threats

Kill chain sequence

5

Supporting evidence

Threat scoring

Graph Mining

4

Con

tinuo

us s

elf-l

earn

ing

Anomalies graph

Entity relationship graph

3

Anomalies

Splunk UBADemo

72

SecurityWorkshops

● SecurityReadinessAssessments● Splunk UBADataScienceWorkshop● EnterpriseSecurityBenchmarkAssessment

SecurityWorkshopSurvey

https://www.surveymonkey.com/r/8BCWHSF