![Page 1: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/1.jpg)
Copyright © 2013 Splunk Inc.
Nimish Doshi Principal Systems Engineer, Splunk #splunkconf
How and When to Use Dynamic Lookups
![Page 2: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/2.jpg)
Legal NoIces During the course of this presentaIon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauIon you that such statements reflect our current expectaIons and esImates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in this presentaIon are being made as of the Ime and date of its live presentaIon. If reviewed aSer its live presentaIon, this presentaIon may not contain current or accurate informaIon. We do not assume any obligaIon to update any forward-‐looking statements we may make. In addiIon, any informaIon about our roadmap outlines our general product direcIon and is subject to change at any Ime without noIce. It is for informaIonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaIon either to develop the features or funcIonality described or to include any such feature or funcIonality in a future release.
Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respecCve
owners.
©2013 Splunk Inc. All rights reserved.
2
![Page 3: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/3.jpg)
About Me: Nimish Doshi
! Principal Systems Engineer at Splunk – Cover the USA East Coast
! Have been at Splunk since 2008 ! Splunk Blogger ! AcIve App Template and Add-‐on developer at apps.splunk.com
3
![Page 4: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/4.jpg)
Agenda
! Lookups in General ! StaIc Lookups ! Dynamic Lookups – Retrieve fields from a web site – Retrieve fields from a database – Retrieve fields from a persistent cache
4
![Page 5: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/5.jpg)
Lookups in General
![Page 6: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/6.jpg)
Enrichment
6
Enrich your events with fields from external sources.
![Page 7: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/7.jpg)
Windows § Registry § Event logs § File system § sysinternals
Logfiles Configs Messages Traps Alerts
Metrics Scripts Tickets Changes
Capturing All This Data Occurs at INDEX Time
7
VirtualizaIon § Hypervisor § Guest OS § Guest Apps
Linux/Unix § ConfiguraIons § syslog § File system § ps, iostat, top
ApplicaIons § Web logs § Log4J, JMS,JMX § .NET events § Code, scripts
Databases § ConfiguraIons § Audit/query logs § Tables § Schemas
Networking § ConfiguraIons § syslog § SNMP § neglow
![Page 8: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/8.jpg)
Splunk Architecture, BIG DATA Plagorm
8
Splunk Web Interface Splunk CLI Interface Other Interfaces, SDKs
Search
Index
Monitor Files Listen to Network Ports Run Scripts
Detect File Changes
Data RouIng, Cloning and Load Balancing
Distributed Search
REST API
Users & Access Controls
Scheduling/AlerIng ReporIng Knowledge
Splunk > Engine
Distributed Search
Deployment Server
(WMI, Registry, OPSEC LEA, DBI, JMS, VMWare API, other APIs)
Lookups
![Page 9: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/9.jpg)
UlImate Knowledge Base
9
Lookups enrich your ability to act upon your data
hnp://sangrea.net/free-‐cartoons/comp_real-‐life-‐search-‐engine.jpg, Royalty Free Cartoons
*
![Page 10: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/10.jpg)
Before Lookups
10
![Page 11: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/11.jpg)
ASer Lookups
11
Top Status DescripIon
![Page 12: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/12.jpg)
Integrate External Data
12
Extend search with lookups to external data sources LDAP, AD Watch
Lists
CRM/ERP CMDB
Correlate IP addresses with locaIons, accounts with regions
![Page 13: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/13.jpg)
ü User’s Mailing Address (AD)
ü Error Code DescripIons
ü Product Names
ü Stock Symbol (from CUSIP)
InteresIng Things to Lookup
13
ü External Host Address
ü Database Query
ü Web Service Call for Status
ü Geo LocaIon
![Page 14: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/14.jpg)
Other Reasons for Lookup
14
! Bypass staIc developer or vendor that does not enrich logs ! ImaginaIve correlaIons
– i.e. Web site URL with like or dislike count stored in external source
! Make your data more interesIng – Bener to see textual descripIons than arcane codes
![Page 15: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/15.jpg)
StaIc Lookups
![Page 16: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/16.jpg)
StaIc vs. Dynamic Lookup
16
StaIc
Dynamic
External Data comes from a CSV file
External Data comes from output of an external script. Output
resembles a CSV file
![Page 17: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/17.jpg)
StaIc Lookups Review
17
! Pick what input fields will be used to get output fields ! Create or locate a CSV file that has all fields in proper order ! Tell Splunk via the Manager about your CSV file and your lookup ̶ You can also define lookups manually via props.conf and transforms.conf ̶ If you use automaIc lookups, they will run every Ime the source,
sourcetype, or associated host stanza is used in a search ̶ Non-‐automaIc lookups run only when the lookup command is invoked in
the search
![Page 18: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/18.jpg)
Example StaIc Lookup Conf files
18
props.conf!![access_combined] lookup_http = http_status status OUTPUT status_description, status_type
transforms.conf
[http_status] filename = http_status.csv
![Page 19: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/19.jpg)
Example AutomaIc StaIc Lookup
19
![Page 20: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/20.jpg)
Permissions
20
local.meta![lookups/http_status.csv] access = read : [ * ], write : [ * ] export = system [transforms/http_status] access = read : [ * ], write : [ * ] export = system
![Page 21: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/21.jpg)
Lookup Topics Not Covered in This Session
21
! Field extracIons are performed before lookups ! Lookups run on the indexer
– To ensure lookups do not run on remote peers, use local=true in lookup command
! You can also use outputlookup to populate a CSV file ! You can also use Ime based lookups to find fields that match your event’s Imestamp with an interval in the lookup CSV
![Page 22: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/22.jpg)
Dynamic Lookups
![Page 23: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/23.jpg)
Dynamic Lookups
23
! Write the script to simulate access to external source ! Test script with one set of inputs ! Create the Splunk Version of the lookup script ! Register the Script with Splunk via Manager or Conf files ! Test the script explicitly before using automaIc lookups
![Page 24: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/24.jpg)
Lookups vs. Custom Command
24
! Use dynamic lookups when returning fields given input fields – Standard for users who already know how to use lookup
! Use a custom command when doing more than just lookup – Not all use cases involve just returning fields
ê Decrypt event data ê Translate event data from one format to another with new fields (e.g. FIX Orders)
![Page 25: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/25.jpg)
Write/Test External Field Gathering Script
25
External Data in Cloud Your Python Script
Send Input Fields
Return Output Fields
Scripts
![Page 26: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/26.jpg)
Example Script to Test External Lookup
26
# Given a host, find the ip !def mylookup(host):! try:!! ipaddrlist = socket.gethostbyname_ex(host)!
return ipaddrlist! except:! return []
![Page 27: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/27.jpg)
Write/Test External Field Gathering Script
27
External Data in Cloud Your Python Script
Send Input Fields
Return Output Fields
Scripts
![Page 28: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/28.jpg)
Test External Field Gathering Script with Splunk
28
Output Fields External Data
in Cloud Your Python Script
Scripts
![Page 29: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/29.jpg)
Script for Splunk Simulates Reading Input CSV
29
hostname, ip!
a.b.c.com!
Zorrosty.com!
seemanny.com!
![Page 30: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/30.jpg)
Output of Script Returns Logically Complete CSV
30
hostname, ip!
a.b.c.com, 1.2.3.4!
Zorrosty.com, 192.168.1.10!
seemanny.com, 10.10.2.10!
![Page 31: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/31.jpg)
transforms.conf for Dynamic Lookup
31
[NameofLookup]!external_cmd = <name>.py field1…fieldN!external_type = python!fields_list = field1, …, fieldN!
![Page 32: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/32.jpg)
Example Dynamic Lookup Conf files
32
!transforms.conf!
!
# Note this is an explicit lookup![whoisLookup]!external_cmd = whois_lookup.py ip whois!external_type = python!fields_list = ip, whois!
![Page 33: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/33.jpg)
Dynamic Look Up Python Flow
33
def lookup(input):!
Perform external lookup based on input. Return result main()!
Check standard input for CSV headers.!
Write headers to standard output.!
For each line in standard input (input fields):!
!Gather input fields into a dictionary (key-value structure)!
!ret = lookup(input fields)!
!If ret:!
! !Send to standard output input values and return values from lookup!
![Page 34: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/34.jpg)
Whois Lookup
34
def main():!
if len(sys.argv) != 3:!
print "Usage: python whois_lookup.py [ip field] [whois field]"!
sys.exit(0)!
ipf = sys.argv[1]!
whoisf = sys.argv[2]!
r = csv.reader(sys.stdin)!
w = None!
header = []!
first = True…!
![Page 35: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/35.jpg)
Whois Lookup (cont.) to Read CSV Header
35
# First read the “CSV header” and output the fields names. Continue!
for line in r:!
if first:!
header = line!
if whoisf not in header or ipf not in header:!
print "IP and whois fields must exist in CSV data"!
sys.exit(0)!
csv.writer(sys.stdout).writerow(header)!
w = csv.DictWriter(sys.stdout, header)!
first = False!
continue…!
![Page 36: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/36.jpg)
Whois Lookup (cont.) to Populate Input Fields
36
# Read the result and populate the values for the input fields (ip address in our case) !
result = {}!
i = 0!
while i < len(header):!
if i < len(line):!
result[header[i]] = line[i]!
else:!
result[header[i]] = ''!
i += 1!
![Page 37: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/37.jpg)
Whois Lookup (cont.) to Populate Output Fields
37
# Perform the whois lookup if necessary !
if len(result[ipf]) and len(result[whoisf]):!
w.writerow(result)!
# Else call external website to get whois field from the ip address as the key!
elif len(result[ipf]):!
result[whoisf] = lookup(result[ipf])!
if len(result[whoisf]):!
w.writerow(result)!
![Page 38: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/38.jpg)
Whois Lookup FuncIon
38
LOCATION_URL=http://some.url.com?query=!
# Given an ip, return the whois response !
def lookup(ip):!
try:!
whois_ret = urllib.urlopen(LOCATION_URL + ip)!
lines = whois_ret.readlines()!
return lines!
except:!
return ''!
![Page 39: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/39.jpg)
Database Lookups in General
39
! Use DB Connect from apps.splunk.com if possible – Splunk supported – No code to be wrinen to do lookups to for
popular RDBMS
! Use your own DB lookup when – Your Database is not supported by DB Connect – You want to perform the lookup with custom
code to meet a requirement
![Page 40: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/40.jpg)
Database Lookup vs. Database Sent to Index
40
! Depends… ! Use a lookup when
– Using needle in the haystack searches with a few users – Using form searches returning few results
! Index the database table or view when ê Having lots of users and ad hoc reporIng is needed ê It is ok to have “stale” data (N minutes) old for a dynamic database
![Page 41: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/41.jpg)
Database Lookup
41
! Acquire proper modules to connect to the database ! Connect and authenIcate to database
– Use a connecIon pool, if possible ! Have lookup funcIon query the database
– Return a list ( [ ] ) of results
![Page 42: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/42.jpg)
Example Database Lookup Using MySQL
42
# See http://splunk-base.splunk.com/apps/36664/splunk-mysql-connector!
# for a general example using MySQL!
# First connect to DB outside of the for loop!conn = MySQLdb.connect(host = "localhost",! user = “name of user",! passwd = ”password",! db = ”Name of DB")!cursor = conn.cursor()!
![Page 43: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/43.jpg)
Example Database Lookup (cont.) Ssing MySQL
43
import MySQLdb….!
# Given a city, find its country !
def lookup(city, cur):!
try:!
selString = "SELECT country FROM city_country where city=”!
! cur.execute (selString + "\"" + city + "\"")!
row = cur.fetchone()!
return row[0]!
!except:!
return []!
![Page 44: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/44.jpg)
Example Mongdb Lookup
44
import pymongo!
from pymongo import Connection!
# Given a star name, find its magnitude !
def lookup(collection, key):!
!try:!
!star = collection.find_one({'name’: key})!
! !return star['magnitude'] !
!except:!
return None!
...!
connection = Connection()!
db = connection.test_database!
star_collection = db.stars ...!
![Page 45: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/45.jpg)
Web Services Lookup
45
! Acquire proper modules to connect to web service ! Connect and authenIcate to web service, if necessary ! Have lookup funcIon call your web service method
– Return a String of results or an empty String, if no match occurs
![Page 46: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/46.jpg)
Example Web Services Lookup Using Suds
46
From suds.client import Client…!
# Given a name, height, and weight, return percent body fat !
def lookup(client, name, height, weight):!
try:!
result=client.service.getPercentBodyFat(name, height, weight)!
if result!=None and result!=‘’:!
!return result!
! else:!
! !return “”!
!except:!
return “”!
… client=Client(<URL to some Web Service WSDL>)!
![Page 47: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/47.jpg)
Lookup Using Key Value Persistent Cache
47
! Download and install Redis ! Download and install Redis Python module ! Import Redis module in Python and populate key value DB ! Import Redis module in lookup funcIon given to Splunk to look up a value given a key
![Page 48: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/48.jpg)
Redis Lookup
48
##### CHANGE PATH TO your distribution FIRST ############
sys.path.append("/Library/Python/2.6/site-packages/redis-2.4.5-py2.6.egg")
import redis
…
def main():
….
# Connect to redis CHANGE for your DISTRIBTUION
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
redp = redis.Redis(connection_pool=pool)
![Page 49: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/49.jpg)
Redis Lookup (cont.)
49
# Note that this returns key value pairs. Redis can also return keys mapped to mulIple values
def lookup(redp, mykey):
try:
return redp.get(mykey)
except:
return “”
![Page 50: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/50.jpg)
Combine Persistent Cache with External Lookup
50
! For data that is “relaIvely staIc” – First see if the data is in the persistent cache – If not, look it up in the external source such as a database or web service – If results come back, add results to the persistent cache and return results
! For data that changes oSen, you will need to create your own cache retenIon policies
![Page 51: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/51.jpg)
Combining Redis with Whois Lookup
51
def lookup(redp, ip):!
try:!
ret = redp.get(ip)!
if ret!=None and ret!='':!
return ret!
else:!
whois_ret = urllib.urlopen(LOCATION_URL + ip)!
lines = whois_ret.readlines()!
if lines!='':!
redp.set(ip, lines)!
return lines…!
except:!
! return “”
![Page 52: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/52.jpg)
Add-‐On Download LocaIon Release
Whois hnp://splunk-‐base.splunk.com/apps/22381/whois-‐add-‐on 4.x
DBLookup hnp://splunk-‐base.splunk.com/apps/22394/example-‐lookup-‐using-‐a-‐database
4.x
Redis Lookup hnp://splunk-‐base.splunk.com/apps/27106/redis-‐lookup 4.x
Geo IP Lookup (not in these
slides)
hnp://splunk-‐base.splunk.com/apps/22282/geo-‐locaIon-‐lookup-‐script-‐powered-‐by-‐maxmind 4.x
Where to Get Add-‐ons Discussed Here
52
Splunkbase.
![Page 53: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/53.jpg)
53
Conclusion: Lookups are a powerful way to enhance your search experience
beyond indexing
So, What?
Enrich BIG DATA with external sources
![Page 54: How*and*When*to*Use* Dynamic*Lookups*...Stac*Lookups*Review * 17! Pick*whatinputfields*will*be*used*to*getoutputfields* Create*or*locate*aCSV*file*thathas*all*fields*in*proper*order*](https://reader033.vdocument.in/reader033/viewer/2022053012/5f0f2e607e708231d442e3ac/html5/thumbnails/54.jpg)
THANK YOU