machine data 101
TRANSCRIPT
![Page 1: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/1.jpg)
Copyright©2014SplunkInc.
MachineData101:TurningDataintoInsight
AudienceVersion
![Page 2: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/2.jpg)
Agenda
§ Non-TraditionalDataSources
§ DataEnrichment
§ LevelUponSearchandReportingCommands
§ DataModelsandPivot
§ AdvancedVisualizationsandtheWebFramework
2
![Page 3: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/3.jpg)
Non-TraditionalDataSources
![Page 4: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/4.jpg)
Non-TraditionalDataSources
§ NetworkInputs
§ HTTPEventCollector
§ LogEventAlertAction
§ SplunkAppforStream
§ ScriptedInputs
§ DatabaseInputs
§ SplunkODBCDriver
§ ModularInputs
§ zLinux Forwarder
§ MINT
§ Non-SplunkDatastores
4
![Page 5: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/5.jpg)
TraditionalDataSources§ Captureseventsfromlogfilesinrealtime
§ Runsscriptstogathersystemmetrics,connecttoAPIsanddatabases
§ Listenstosyslog andgathersWindowsevents
§ Universallyindexesanydataformatsoitdoesn’tneedadapters
5
Windows• Registry• Eventlogs• Filesystem• sysinternals
Linux/Unix• Configurations• Syslog• Filesystem• Ps,iostat,top
Virtualization• Hypervisor• GuestOS• GuestApps
Applications• Weblogs• Log4J,JMS,JMX• .NETevents• Codeandscripts
Databases• Configurations• Audit/querylogs• Tables• Schemas
Network• Configurations• syslog• SNMP• netflow
![Page 6: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/6.jpg)
NetworkInputs
§ CollectdataoveranyUDPorTCPport§ Somedevicesonlysenddataoveranetworkport
§ BestPractice:usesyslog-ng orrsyslog§ Offerspersistence§ Categorizesdatabyhost
6
![Page 7: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/7.jpg)
HTTPEventCollector(HEC)
§ CollectdataoverHTTPorHTTPSdirectlytoSplunk§ ApplicationDeveloperfocus– fewlinesofcodeinapp
tosenddata§ HECFeaturesInclude:
§ Token-based,notcredentialbased§ IndexerAcknowledgements– guaranteesdataindexing§ RawandJSONformattedeventpayloads§ SSL,CORS(CrossOrigion access),andNetworkRestrictions
7
![Page 8: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/8.jpg)
LogEventAlertAction
§ UseSplunkalertingtoindexacustomlogevent§ Splunksearchableindexofcustomalertevents
§ ConfigurableFeaturesInclude:§ Host§ Source§ Sourcetype§ Index§ Eventtext– constructtheexactsyntaxofthelogevent,
includinganytext,tokens,orotherinformation
8
![Page 9: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/9.jpg)
TheSplunkAppforStream
WireDataEnhancesthePlatformforOperationalIntelligence
Efficient,Cloud-readyWireDataCollection
SimpleDeploymentSupportsFastTimetoValue
9
![Page 10: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/10.jpg)
Stream=BetterInsightsfor*
SolutionArea ContextualData WireData Enriched View
ApplicationManagement
applicationlogs,monitoringdata,metrics,events
protocolconversationsondatabaseperformance,DNSlookups,clientdata,businesstransactionpaths…
Measureapplicationresponsetimes,deeperinsightsforroot-causediagnostics,tracetxpaths,establishbaselines…
IT Operations applicationlogs,monitoringdata,metrics,events
payloaddataincludingprocesstimes,errors,transactiontraces,ICAlatency,SQLstatements,DNSrecords…
Analyzetrafficvolume,speedandpacketstoidentifyinfrastructureperformanceissues,capacityconstraints,changes;establishbaselines…
10
![Page 11: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/11.jpg)
Stream=BetterInsightsfor*SolutionArea ContextualData WireData Enriched View
Security app+infralogs,monitoringdata,events
protocolidentification,protocolheaders,contentandpayloadinformation,flowrecords
Buildanalyticsandcontextforincidentresponse,threatdetection,monitoringandcompliance
DigitalIntelligence
websiteactivity,clickstreamdata,metrics
browser-levelcustomerinteractions
CustomerExperience – analyzewebsiteandapplicationbottleneckstoimprovecustomerexperienceandonlinerevenues
CustomerSupport(online,callcenter)– fasterrootcauseanalysisandresolutionofcustomerissueswithwebsiteorapps
11
![Page 12: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/12.jpg)
ScriptedInputs
12
§ SenddatatoSplunkviaacustomscript§ Splunkindexesanythingwrittentostdout§ Splunkhandlesscheduling§ Supportsshell,Pythonscripts,WINbatch,PowerShell§ Anyotherutilitythatcanformatandstreamdata
StreamingMode§ Splunkexecutesscriptandindexesstdout
§ Checksforanyrunninginstances
WritetoFileMode§ Splunklaunchesscriptwhichproducesoutputfile,noneedforexternalscheduler
§ Splunkmonitorsoutputfile
![Page 13: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/13.jpg)
UseCasesforScriptedInputs
13
§ Alternativetofile-baseornetwork-basedinputs§ Streamdatafromcommand-linetools,suchasvmstat andiostat§ Pollawebservice,APIordatabaseandprocesstheresults§ Reformatcomplexorbinarydataforeasierparsingintoeventsandfields§ Maintaindatasourceswithsloworresource-intensivestartup
procedures§ Providespecialorcomplexhandlingfortransientorunstableinputs§ Scriptsthatmanagepasswordsandcredentials§ Wrapperscriptsforcommandlineinputsthatcontainspecialcharacters
![Page 14: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/14.jpg)
DatabaseInputs
§ Createvaluewithstructureddata§ Enrichsearchresultswithadditionalbusinesscontext
§ Easilyimportdatafordeeperanalysis§ IntegratemultipleDBsconcurrently§ Simpleset-up,non-invasiveandsecure
DBConnectprovidesreliable,scalable,real-timeintegrationbetweenSplunkandtraditionalrelationaldatabases
14
![Page 15: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/15.jpg)
ConfigureDatabaseInputs
15
§ DBConnectApp§ Real-time,scalableintegrationwithrelationalDBs§ Browseandnavigateschemasandtablesbeforedataimport§ Reliablescheduledimport§ SeamlessinstallationandUIconfiguration§ Supportsconnectionpoolingandcaching
§ “Tail”tablesorimportentiretables§ Detectandimportnew/updatedrowsusingtimestampsoruniqueIDs
§ SupportsmanyRDBMSflavors§ AWSRDSAurora,AWSRedShift,IBMDB2forLinux,Informix,MemSQL,MSSQL,MySQL,
Oracle,PostgreSQL,SAPSQLAnywhere(akaSybaseSA),SybaseASEandIQ,Teradata
![Page 16: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/16.jpg)
SplunkODBCDriver
16
§ Interactwith,manipulateandvisualizemachinedatainSplunkEnterpriseusingbusinesssoftwaretools
§ LeverageanalyticsfromSplunkalongsideMicrosoftExcel,TableauDesktoporMicrostrategy AnalyticsDesktop
§ Industry-standardconnectivitytoSplunkEnterprise§ Empowersbusinessuserswithdirectandsecureaccesstomachinedata
§ Combinemachinedatawithstructureddataforbetteroperationalcontext
![Page 17: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/17.jpg)
ODBC:HowitWorks
17
![Page 18: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/18.jpg)
ModularInputs
18
§ Createyourowncustominputs§ Scriptedinputwithstructureandintelligence§ FirstclasscitizenintheSplunkmanagementinterface§ AppearsunderSettings>DataInputs
§ Benefitsoversimplescriptedinput§ Instancecontrol:launchasingleormultipleinstances§ Inputvalidation§ Supportmultipleplatforms§ StreamdataastextorXML§ SecureaccesstomodinputscriptsviaRESTendpoints
![Page 19: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/19.jpg)
ExampleModularInputs
19
Twitter§ StreamJSONdatafromaTwittersourcetoSplunkusingTweepy
AmazonS3OnlineStorage§ IndexdatafromtheAmazonS3onlinestoragewebservice
JavaMessagingService(JMS)§ PollmessagequeuesandtopicsthroughJMSMessagingAPI§ Talkstomultipleproviders:MQSeries (Websphere MQ),ActiveMQ,TibcoEMS,HornetQ,RabbitMQ,NativeJMS,WebLogic JMS,SonicMQ
SplunkWindowsInputs§ RetrieveWINeventlogs,registrykeys,perfmon counters
![Page 20: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/20.jpg)
MoreModularInputs
20
![Page 21: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/21.jpg)
zLinux Forwarder
21
§ EasilycollectandindexdataonIBMmainframes
§ Collectapplicationandplatformdata
§ DownloadasnewForwarderdistributionfors390xLinux
![Page 22: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/22.jpg)
ExtendOperationalIntelligencetoMobileApps
22
DeliverBetterPerforming,MoreReliableApps
DeliverReal-TimeOmni-Channel
Analytics
End-to-EndPerformanceandCapacityInsights
![Page 23: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/23.jpg)
MonitorAppUsageandPerformance
• Improveuserretentionbyquicklyidentifyingcrashesandperformanceissues
• Establishwhetherissuesarecausedbyanapporthenetwork(s)
• Correlateapp,OSanddevicetypetodiagnosecrashandnetworkperformanceissues
23
![Page 24: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/24.jpg)
IntegratedAnalyticsPlatformforDiverseDataStoresFull-featured,IntegratedProduct
FastInsightsforEveryone
WorkswithWhatYouHaveToday
Explore Visualize Dashboards
ShareAnalyze
HadoopClusters NoSQLandOtherDataStores
Hadoop ClientLibraries StreamingResourceLibraries
Bi-directionalIntegrationwithHadoop
![Page 25: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/25.jpg)
ConnecttoNoSQLandOtherDataStores
• Buildcustomstreamingresourcelibraries
• SearchandanalyzedatafromotherdatastoresinHunk
• InpartnershipwithleadingNoSQLvendors
• UseinconjunctionwithDBConnectforrelationaldatabaselookups
![Page 26: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/26.jpg)
VirtualIndexes
§ EnablesseamlessuseofalmosttheentireSplunkstackondata
§ AutomaticallyhandlesMapReduce
§ Technologyispatentpending
![Page 27: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/27.jpg)
DataEnrichment
![Page 28: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/28.jpg)
Agenda
§ Tags – categorizeandaddmeaningtodata
§ FieldAliases – simplifysearchandcorrelation
§ CalculatedFields – shortcutcomplex/repetitivecomputations
§ EventTypes – groupcommoneventsandshareknowledge
§ Lookups – augmentdatawithadditionalexternalfields
28
![Page 29: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/29.jpg)
§ Addsinlinemeaning/context/specificitytorawdata
§ Usedtonormalizemetadataorrawdata
§ Simplifiescorrelationofmultipledatasources
§ CreatedinSplunk
§ Transferredfromexternalsources
WhatisDataEnrichment?
29
![Page 30: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/30.jpg)
§ Addmeaning/context/specificitytorawdata
§ Labelsdescribingteam,category,platform,geography
§ Appliedtofield-valuecombination
§ Multipletagscanbeappliedforeachfield-value
§ Casesensitive
Tags
30
![Page 31: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/31.jpg)
CreateTags
31
![Page 32: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/32.jpg)
§ Searcheventswithtaginanyfield
§ Searcheventswithtaginaspecificfield
§ Searcheventswithtagusingwildcards
FindtheWebServersTagsinAction
32
tag=webserver
tag::host=webserver
tag=web*
§ Tagthehostaswebserver
§ Tagthesourcetypeasweb
1
2
3
4
5
![Page 33: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/33.jpg)
§ Normalizefieldlabelstosimplifysearchandcorrelation§ Applymultiplealiasestoasinglefield
§ Example:Username|cs_username |Userà user§ Example:c_ip |client|client_ipà clientip
§ Processedafterfieldextractions+beforelookups
§ Canapplytolookups
§ Aliasesappearalongsideoriginalfields
FieldAliases
33
![Page 34: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/34.jpg)
Re-LabelFieldtoIntuitiveNameCreateFieldAlias
34
1
2
3
![Page 35: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/35.jpg)
§ Createfieldaliasofclientip=customer
§ Searcheventsinlast15minutes,findcustomerfield
§ Fieldalias(customer)andoriginalfield(clientip)arebothdisplayed
SearchusinganIntuitiveFieldNameFieldAliasinAction
35
1
3
2
sourcetype=access_combined
![Page 36: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/36.jpg)
§ Shortcutforperformingrepetitive/long/complextransformationsusingevalcommand
§ Basedonextractedordiscoveredfieldsonly
§ Donotapplytolookuporgeneratedfields
CalculatedFields
36
![Page 37: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/37.jpg)
ComputeKilobytesfromBytesCreateCalculatedField
37
1
21
2
3
![Page 38: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/38.jpg)
§ Createkilobytes=bytes/1024
§ Searcheventsinlast15minutesforkilobytesandbytes
SearchUsingKilobytesinsteadofBytesCalculatedFieldsinAction
38
1
2
sourcetype=access_combined
![Page 39: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/39.jpg)
§ Classifyandgroupcommonevents
§ Captureandshareknowledge
§ Basedonsearch
§ Useincombinationwithfieldsandtagstodefineeventtopography
EventTypes
39
![Page 40: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/40.jpg)
§ BestPractice:Usepunctfield§ Defaultmetadatafielddescribingeventstructure§ Builtoninterestingcharacters:",;-#$%&+./:=?@\\'|*\n\r\"(){}<>[]^! »§ Canusewildcards
CreateEventTypes
40
event punct
####<Jun3,20145:38:22PMMDT><Notice><WebLogicServer><bea03><asiAdminServer><WrapperStartStopAppMain><>WLSKernel<><><BEA-000360><ServerstartedinRUNNINGmode>
####<_,__::__>_<>_<>_<>_<>_<>_
172.26.34.223- - [01/Jul/2005:12:05:27-0700]"GET/trade/app?action=logoutHTTP/1.1"2002953
..._-_-_[:::_-]_\"_?=_/.\"__
![Page 41: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/41.jpg)
§ Showpunctforsourcetype=access_combined
§ Pickapunct,thenwildcarditafterthetimestamp
§ AddNOTstatus=200
§ Saveas“bad”eventtype+Color:red+Priority:1(shiftreloadinbrowsertoshowcoloring)
ClassifyEventsasKnownBadCreateEventType
41
eventtype=bad
sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200
1
2
3
4
![Page 42: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/42.jpg)
LookupstoEnrichRawData
LDAPAD
WatchLists
CRM/ERP
CMDB
ExternalDataSources
Insightcomesout
DatagoesinCreateadditionalfieldsfromtherawdatawithalookuptoanexternaldatasource
![Page 43: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/43.jpg)
§ Augmentraweventswithadditionalfields§ Providecontextorsupportingdetails
§ Translatefieldvaluestomoredescriptivedata§ Example:addtextdescriptionsforerrorcodes,IDs§ Example:addcontactdetailstousernamesorIDs§ Example:adddescriptionstoHTTPstatuscodes
§ File-basedorscriptedlookups
Lookups
43
![Page 44: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/44.jpg)
44
1.Upload/createtable
2.Assigntabletolookupobject
3.Maplookuptodataset
Convert a Code into a DescriptionConfigure a Static Lookup
![Page 45: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/45.jpg)
§ GetthelookupfromtheSplunkWiki(saveto.csv file)http://wiki.splunk.com/Http_status.csv
§ Lookuptablefiles>Addnew§ Name:http_status.csv (musthave.csv fileextension)§ Upload:<pathto.csv>
§ Verifylookupwascreatedsuccessfully
1.CreateHTTPStatusTable
45
| inputlookup http_status.csv
1
2
3
![Page 46: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/46.jpg)
§ Lookupdefinitions>Addnew§ Name:http_status§ Type:File-based§ Lookupfile:http_status.csv
§ Invokethelookupmanually
2.AddLookupDefinition
46
1
2
sourcetype=access_combined | lookup http_status status OUTPUT status_description
![Page 47: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/47.jpg)
§ Automaticlookups>Addnew§ Name:http_status (cannothavespaces)§ Lookuptable:http_status§ Applyto:sourcetype=access_combined§ Lookupinputfield:status§ Lookupoutputfield:status_description
§ Verifylookupisinvokedautomatically
3.ConfigureAutomaticLookup
47
1
2
sourcetype=access_combined
![Page 48: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/48.jpg)
§ Temporallookupsfortime-basedlookups§ Example:IdentifyusersonyournetworkbasedontheirIPaddress
andthetimestampinDHCPlogs
§ Usesearchresultstopopulatealookuptable§ … | outputlookup <tablename|filename>
§ Callanexternalcommandorscript§ Pythonscriptsonly§ Example:DNSlookupforIPßà Host
§ Createalookuptableusingarelationaldatabase§ ReviewmatchesagainstadatabasecolumnorSQLquery
FancyLookups
48
![Page 49: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/49.jpg)
§ CreatingandManagingAlerts(JobInspector)
§ Macros
§ WorkflowActions
MoreDataEnrichment
49
![Page 50: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/50.jpg)
LevelUponSearch&ReportingCommands
![Page 51: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/51.jpg)
Agenda
§ Doingmorewithbasicsearchcommands
§ Advancedsearchcommands
§ Doingmorewithbasicreportingcommands
51
![Page 52: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/52.jpg)
SearchSyntaxComponents
52
![Page 53: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/53.jpg)
AnatomyofaSearch
53
Disk
![Page 54: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/54.jpg)
§ top– limit§ rare– sameoptionsastop§ timechart– parameters§ stats– functions(sum,avg,list,values,sparkline)§ sort– inlineascendingordescending§ addcoltotals§ addtotals
DoingMorewithBasicSearchCommands
54
![Page 55: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/55.jpg)
§ Commandshaveparametersorqualifiers
§ topandrarehavesimilarsyntax
§ Eachsearchcommandhasitsownsyntax– showinlinehelp
FindMostandLeastActiveCustomersUsingthetop+rareCommands
... | top limit=20 clientip
... | rare limit=20 clientip
IPswiththemostvisits
IPswiththeleastvisits
![Page 56: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/56.jpg)
§ Sortinlinedescendingorascending
56
... | stats count by clientip | sort - count
... | stats count by clientip | sort + count
Numberofrequestsbycustomer- descending
Numberofrequestsbycustomer- ascending
SorttheNumberofCustomerRequestsUsingthesortCommand
![Page 57: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/57.jpg)
§ ShowSearchCommandReferenceDocs§ Functionsforeval+where§ Functionsforstats+chartandtimechart
§ Invokeafunction
§ Renameinline
57
... | stats sum(bytes) by clientip | sort - sum(bytes)
... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes
Totalpayloadbycustomer- descending
Totalpayloadbycustomer- ascending
DetermineTotalCustomerPayloadUsingfunctions+renamecommand
![Page 58: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/58.jpg)
§ Listallvaluesofafield
§ Listonlydistinctvaluesofafield
58
... | stats values(action) by clientip
... | stats list(action) by clientip
Activitybycustomer
Distinctactionsbycustomer
ObserveCustomerActivityUsingthelist+valuesFunctions
![Page 59: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/59.jpg)
§ Showdistinctactionsandcardinalityofeachaction
59
sourcetype=access_combined| stats count(action) as value by clientip, action| eval pair=action + " (" + value + ")"| stats list(pair) as values by clientip
AnalyzeCustomerActivityCombinelist+valuesFunctions
![Page 60: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/60.jpg)
§ Addcolumns
§ Sumspecificcolumns
60
... | stats count by clientip, action
2cols:clientip +action
... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes, count as totalevents by clientip | addcoltotals totalbytes, totalevents
Sumtotalbytesandtotaleventscolums
BuildingaTableofCustomerActivityAddColumnsandSumColumns
![Page 61: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/61.jpg)
61
... | stats sum(bytes) as totalbytes, sum(other) as totalother by clientip | addtotals fieldname=totalstuff
Foreachrow,addtotalbytes+totalother
Abetterexample:physicalmemory+virtualmemory=
totalmemory
BuildingaTableofCustomerActivitySumAcrossRows
![Page 62: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/62.jpg)
62
... | stats sparkline(count) as trendline by clientip
Incontextoflargereventset
... | stats sparkline(count) as trendline sum(bytes) by clientip
Inlineintables
TrendIndividualCustomerActivitySparklinesinAction
![Page 63: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/63.jpg)
AdvancedSearchCommandsCommand ShortDescription Hints
transaction Groupeventsbyacommonfieldvalue. Convenient,but resourceintensive.cluster Clustersimilareventstogether. Canbeusedon_raworfield.associate Identifiescorrelationsbetweenfields. Calculatesentropybtn fieldvalues.correlate Calculatesthecorrelationbetween
differentfields.Evaluatesrelationshipof allfieldsinaresultset.
contingency Buildsacontingencytablefortwofields. Computesco-occurrence,or%twofieldsexistinsameevents.
anomalies Computesanunexpectednessscoreforanevent.
Computessimilarityofevent(X)toasetofpreviousevents(P).
anomalousvalue Findsandsummarizesirregular,oruncommon,searchresults.
Considers frequencyofoccurrenceornumberofstdev fromthemean
![Page 64: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/64.jpg)
§ Seweventstogether+createsduration+eventcount
§ Sparklinesinlineintables
64
... | transaction JSESSIONID | table JSESSIONID, action, product_id
GroupbyJSESSIONID
ViewCustomerActivitybySessionUsingthetransactionCommand
![Page 65: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/65.jpg)
§ Intelligentgroup(createscluster_countandcluster_label)
65
... | cluster showcount=1 | table _raw, cluster_count, cluster_label
AutomaticallyGroupCustomerActivityUsingtheclusterCommand
![Page 66: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/66.jpg)
§ Predictovertime
§ ChartOverlaywithandwithoutstreamstats
§ Mapswithiplocation+geostats
§ Singlevalue
§ Meteredvisualswithgauge
DoMorewithBasicReportingCommands
66
![Page 67: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/67.jpg)
§ Predictfuturevaluesusinglower/upperbounds– singleandmultipleseries
67
... | timechart count as traffic | predict traffic
PredictWebsiteTrafficUsingthepredictCommand
![Page 68: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/68.jpg)
68
sourcetype=access_combined (action=view OR action=purchase)| timechart span=10m count(eval(action="view")) as Viewed,
count(eval(action="purchase")) as Purchased
CompareBrowsingvs.BuyingActivitySimpleChartOverlay
![Page 69: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/69.jpg)
69
... | iplocation clientip | geostats count by clientip
CombineIPlookupwithgeomapping
MapCustomerActivity GeographicallyGeolocation inAction
![Page 70: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/70.jpg)
70
... | stats count
DisplayaSimpleCountofEventsSingleValueinAction
![Page 71: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/71.jpg)
DisplayCountsUsingGaugesSingleValue,RadialandFillerGaugesinAction
71
... | stats count | gauge count 10000 20000 30000 40000 50000
![Page 72: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/72.jpg)
DataModelandPivot
![Page 73: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/73.jpg)
Agenda
§ Whatisadatamodel?
§ Buildadatamodel
§ PivotInterface
§ Accelerateadatamodel
73
![Page 74: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/74.jpg)
PowerfulAnalyticsAnyoneCanUse
Enablesnon-technicaluserstobuildcomplexreportswithoutthesearchlanguage
Providesmoremeaningfulrepresentationofunderlyingrawmachinedata
Accelerationtechnologydeliversupto1000xfasteranalyticsoverSplunk5
74
Pivot
DataModel
AnalyticsStore
![Page 75: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/75.jpg)
DefineRelationshipsinMachineDataDataModel• Describeshowunderlyingmachinedataisrepresentedandaccessed
• Definesmeaningfulrelationshipsinthedata
• Enablessingleauthoritativeviewofunderlyingrawdata
Hierarchicalobjectviewofunderlyingdata
Addconstraintstofilteroutevents
![Page 76: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/76.jpg)
TransparentAcceleration
• Automaticallycollected– Handlestimingissues,
backfill…• Automaticallymaintained– Usesaccelerationwindow
• Storedontheindexers– Peertothebuckets
• Faulttolerantcollection
Timewindowofdatathatisaccelerated
Checktoenableaccelerationofdatamodel
HighPerformanceAnalyticsStore
![Page 77: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/77.jpg)
Easy-to-UseAnalytics
• Drag-and-dropinterfaceenablesanyusertoanalyzedata
• Createcomplexqueriesandreportswithoutlearningsearchlanguage
• Clicktovisualizeanycharttype;reportsdynamicallyupdatewhenfieldschange
Selectfieldsfromdatamodel
Timewindow
Allcharttypesavailableinthecharttoolbox
Savereporttoshare
Pivot
![Page 78: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/78.jpg)
§ Definesleastcommondenominatorforadatadomain
§ Standardmethodtoparse,categorize,normalizedata
§ Setoffieldnamesandtagsbydomain§ PackagedasaDataModelsinaSplunkApp
§ Domains:security,web,inventory,JVM,performance,networksessions,andmore
§ MinimalsetuptousePivotinterface
CommonInformationModel(CIM)App
78
![Page 79: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/79.jpg)
§ Apps>FindMoreApps>
§ Search:“CommonInformationModel”
§ Installfree
§ Showfieldsforweb+WebDataModel
DownloadCIMApp
79
1
2
3
4
![Page 80: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/80.jpg)
DataModel&PivotTutorial
http://docs.splunk.com/Documentation/Splunk/latest/PivotTuto
rial/WelcometothePivotTutorial
80
![Page 81: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/81.jpg)
CustomVisualizationsandtheWebFrameworkToolkit
![Page 82: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/82.jpg)
Agenda
§ DeveloperPlatform
§ WebFrameworkToolkit(WFT)
§ RESTAPIandSDKs
§ GetaFlyingStart
82
![Page 83: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/83.jpg)
OptimizingtheAnalyticsProcess
83
Focusonthedata– intuitivetoolstoenabletheanalyst
Nosinglevisualizationexiststohandlealldatasets.
Neverlosesightoftherawdata
SplunkAnalytics
Explore
Context
Visualize
Algorithms
![Page 84: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/84.jpg)
6.0+6.1:Simple,Interactive,andExtensible
84
VISUALIZATIONEXPLORATION
CUSTOMIZABLEFRAMEWORK
POWERFULANALYTICS
PivotDataModels
InteractiveFormsContextualDrilldown
DashboardEditorWebFramework
![Page 85: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/85.jpg)
TheSplunkEnterprisePlatform
Collection
Indexing
SearchProcessingLanguage
CoreFunctions
Inputs,Apps,OtherContent
SDKContent
CoreEngine
UserandDeveloperInterfaces
WebFramework
RESTAPI
![Page 86: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/86.jpg)
What’sPossiblewiththeSplunkEnterprisePlatform?
PowerMobileApps
LogDirectly
ExtractData
CustomerDashboards
IntegrateBITools
IntegratePlatformServices
Developer Platform
![Page 87: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/87.jpg)
PowerfulPlatformforEnterpriseDevelopersDevelopersCanCustomizeandExtend
RESTAPI
BuildSplunkApps ExtendandIntegrateSplunk
SimpleXML
JavaScript
HTML5
WebFramework
JavaJavaScriptPython
RubyC#PHP
DataModels
SearchExtensibility
ModularInputs
SDKs
![Page 88: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/88.jpg)
SplunkSoftwareforDevelopers
GainApplicationIntelligence
BuildSplunkApps
IntegrateandExtendSplunk
![Page 89: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/89.jpg)
AWealthofSplunk AppsOver1,100appsavailableontheSplunkappssite
APISDKs UI
Server, Storage, Network
Server Virtualization
Operating Systems
Custom Applications
Business Applications
Cloud Services
App Performance MonitoringTicketing/ and
Other
WebIntelligence
Mobile Applications
Stream
![Page 90: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/90.jpg)
§ Interactive,cut/pasteexamplesfrompopularsourcerepositories:D3,GitHub,jQuery
§ Splunk6.xDashboardExamplesApphttps://apps.splunk.com/app/1603
§ CustomSimpleXML ExtensionsApphttps://apps.splunk.com/app/1772
§ SplunkWebFrameworkToolkitApphttps://apps.splunk.com/app/1613
ExampleAdvancedVisualizations
90
![Page 91: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/91.jpg)
91
http://www.d3js.org
![Page 92: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/92.jpg)
AddaD3BubbleChart
92
1. GotoFindMoreAppsandInstalltheSplunk6.xDashboardExamplesApp
2. EntertheApp3. GotoExamples>CustomVisualizations>
D3BubbleChart4. Copyautodiscover.js (file)+components/bubblechart (dir)
from:$SH/etc/apps/simple_xml_examples/appserver/staticto:$SH/apps/search/appserver/static
5. CopyandpastesimpleXMLtonewdashboard
![Page 93: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/93.jpg)
Resources
![Page 94: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/94.jpg)
SplunkDocumentation
94
• http://docs.splunk.com• OfficialProductDocs• Wikiandcommunitytopics• Updateddaily• Canbeprintedto.PDF
![Page 95: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/95.jpg)
SplunkAnswers
95
• http://answers.splunk.com• Communitydriven• Splunksupported• Knowledgeexchange• Q&A
![Page 96: Machine Data 101](https://reader034.vdocument.in/reader034/viewer/2022052514/586fb2bf1a28abe57d8b699f/html5/thumbnails/96.jpg)
SplunkEducation
96
• RecommendedforUsers– UsingSplunk– Searching&Reporting
• RecommendedforUI/DashboardDevelopers– DevelopingApps
• Instructor-LedCourses– Web– Onsite