splunk conf 2014 - getting the message
DESCRIPTION
Presentation from Splunk Conf 14TRANSCRIPT
![Page 1: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Damien DallimoreDev Evangelist , CSO Office @ Splunk
Getting the Message
Nimish DoshiPrincipal Systems Engineer @ Splunk
![Page 2: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/2.jpg)
2
DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events or the
expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important
factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
![Page 3: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/3.jpg)
3
Agenda
Damien’s Section
What is messagingJMS + DemoAMQP + DemoKafka + DemoCustom message handlingArchitecting for scale
Nimish’s Section
Using ZeroMQUsing JMS for underutilized computersQuestion time
![Page 4: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/4.jpg)
Damien’s Section
![Page 5: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/5.jpg)
5
From Middle Earth
Make Splunk Apps & Add-ons
Messaging background
![Page 6: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/6.jpg)
6
![Page 7: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/7.jpg)
7
apps.splunk.com
github.com/damiendallimore
![Page 8: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/8.jpg)
8
What is messaging ?Messaging infrastructures facilitate the sending/receiving of messages between distributed systems
Message can be encoded in one of many available protocols
A common paradigm involves producers and consumers exchanging via topics or queues
Topics (publish subscribe)
Queues (point to point)
QUEUE
TOPIC
![Page 9: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/9.jpg)
9
Why are messaging architectures used ?
Integrating Legacy Systems
Integrating Heterogeneous Systems
Distributed Applications
Cluster Communication
High Performance Streaming
![Page 10: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/10.jpg)
10
There’s a lot of information in the pipes
![Page 11: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/11.jpg)
11
The data opportunity
Easily tap into a massive source of valuable inflight data flowing around the veins
Don’t need to access the application directly ,pull data off the messaging bus
I can not think of a single industry vertical that does not use messaging
![Page 12: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/12.jpg)
12
Getting this data into Splunk
Many different messaging platforms and protocols
JMS (Java Message Service)
AMQP (Advanced Message Queueing Protocol)
Kafka
Nimish will cover some more uses cases also
![Page 13: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/13.jpg)
13
JMS
DEMO
Not a messaging protocol , but a programming interface to many different underlying message providers
WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…
Very prevalent in the enterprise software landscape
![Page 14: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/14.jpg)
14
AMQP
DEMO
RabbitMQ
Supports AMQP 0.9.1, 0.9, 0.8
Common in financial services and environments that need high performance and low latency
![Page 15: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/15.jpg)
15
Kafka
DEMO
Cluster centric design = strong durability and fault tolerance
Scales elastically
Producers and Consumers communicate via topics in a Kafka node cluster
Very popular with open source big data / streaming analytics solutions
![Page 16: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/16.jpg)
16
Custom message handling
These Modular Inputs can be used in a multitude of scenarios
Message bodies can be anything : JSON, XML, CSV, Unstructured text, Binary
Need to give the end user the ability to customize message processing
So you can plugin your own custom handlers
Need to write code , but it is really easy , and there are examples on GitHub
I’m a big data pre processing fan
![Page 17: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/17.jpg)
17
Cut the code
![Page 18: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/18.jpg)
18
Compile, bundle into jar file, copy to Splunk
![Page 19: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/19.jpg)
19
Declaratively apply it
Let’s see if it works
![Page 20: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/20.jpg)
20
Achieving desired scale
AMQP Queue
AMQP Mod Input
Single Splunk Instance
With 1 Modular Input instance , only so much performance / throughput can be achieved
You’ll hit limits with JVM heap , CPU , OS STDIN/STDOUT Buffer , Splunk indexing pipeline
![Page 21: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/21.jpg)
21
So go Horizontal
AMQP Queue
Universal Forwarders
Splunk Indexer Cluster
AMQP Broker
AMQP Mod Input AMQP Mod Input
![Page 22: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/22.jpg)
Nimish’s Section
![Page 23: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/23.jpg)
23
About Me
• Principal Systems Engineer at Splunk in the NorthEast• Session Speaker at all past Splunk .conf user conferences• Catch me on the Splunk Blogs
![Page 24: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/24.jpg)
24
Problem with Getting Business Data from JMS
The goal is to index the business message contents into SplunkMessage Uncertainty Principal:If you de-queue the message to look at it, you have affected the TXNIf you use various browse APIs for content, you may miss it– Message may have already been consumed by TXN
Suggestion: Use a parallel queue to log the message– Suggestion: Try ZeroMQ
![Page 25: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/25.jpg)
25
Why use ZeroMQ
Light WeightMultiple Client language support (Python, C++, Java, etc)Multiple design patterns (Pub/Sub, Pipeline, Request/Reply, etc)Open Source with community support
![Page 26: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/26.jpg)
26
Application Queue and ZeroMQ Example
Auto Load Balance
1
2
![Page 27: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/27.jpg)
27
Example Python Sender
context = zmq.Context()socket = context.socket(zmq.PUSH)socket.connect('tcp://127.0.0.1:5000')sleeptime=0.5
while True: num=random.randint(50,100) now = str(datetime.datetime.now()) sleep(sleeptime) payload = now + " Temperature=" + str(num) socket.send(payload)
![Page 28: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/28.jpg)
28
Python Receiver (Scripted Input)
context = zmq.Context()socket = context.socket(zmq.PULL)# Change address and port to match your environmentsocket.bind("tcp://127.0.0.1:5000")
while True: msg = socket.recv() print "%s" % msgexcept: print "exception"
![Page 29: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/29.jpg)
29
Python Subscriber (Scripted Input)
context = zmq.Context()socket = context.socket(zmq.SUB)
socket.connect ("tcp://localhost:5556")
# Subscribe to directionfilter = "east"socket.setsockopt(zmq.SUBSCRIBE, filter)
while True: string = socket.recv() print string
![Page 30: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/30.jpg)
30
Parallel Pipeline Example
![Page 31: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/31.jpg)
31
Getting Events out of SplunkSplunk SDK
Use Cases:– In Depth processing of Splunk events in a queued manner– Use as pivot point to drop off events into a Complex Event Processor– Batch Processing of Splunk events outside of Splunk
Divide and Conquer Approach as seen in last slide
![Page 32: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/32.jpg)
32
Java Example using SDK to load ZeroMQString query=search;Job job = service.getJobs().create(query, queryArgs);while (!job.isDone()) {
Thread.sleep(100);job.refresh();
}// Get Query Results and store in String str… (Code Omitted)// Assuming single line events StringTokenizer st = new StringTokenizer(str, "\n");while(st.hasMoreTokens()) {
String temp= st.nextToken();sock.send(temp.getBytes(), 0);byte response[] = sock.recv(0);
}
![Page 33: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/33.jpg)
33
Idle Computers at a Corporation
…
![Page 34: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/34.jpg)
34
Idea: Use Ideas from SETI @ Home
![Page 35: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/35.jpg)
35
Idle Computers Put to Work Using JMS
…
![Page 36: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/36.jpg)
36
Applications for Distributing Work
Application Server would free up computing resourcesWork could be pushed to underutilized computersExamples:– Massive Mortgage Calculation Scenarios– Linear Optimization Problems– Matrix Multiplication– Compute all possible paths for combinatorics
![Page 37: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/37.jpg)
37
Architecture
Optional
![Page 38: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/38.jpg)
38
Algorithm
Application servers push requests to queues, which may include data in the request object called a Unit of WorkJMS client implements doWork() interface to work with dataMessage Driven Bean receives finished work and implements doStore() interfaceWhat does this have to do with Splunk?– Time Series results can be stored in Splunk for further or historical analytics
![Page 39: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/39.jpg)
39
Matrix Example High Level Architecture
![Page 40: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/40.jpg)
40
Search Language Against Matrix ResultList Column Values of Each Stored Multiplied Matrix using Multikv
Screenshot here
![Page 41: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/41.jpg)
41
Search Language Against Matrix ResultVisualize the Average for Columns 2 to 5
Screenshot here
![Page 42: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/42.jpg)
42
Search Language Against Matrix ResultPerform arbitrary math on aggregate columns
Screenshot here
![Page 43: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/43.jpg)
43
Reference
ZeroMQ– http://apps.splunk.com/app/1000/– Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input/
Using JMS for Underutilized Computers– Github Reference: https://github.com/nimishdoshi/JMSClientApp/– Blog: http://blogs.splunk.com/2014/04/11/splunk-as-a-recipient-on-the-jms-grid/– Article:http:
//www.oracle.com/technetwork/articles/entarch/jms-distributed-work-082249.html
![Page 44: Splunk Conf 2014 - Getting the message](https://reader036.vdocument.in/reader036/viewer/2022062308/559323e61a28aba2038b4594/html5/thumbnails/44.jpg)
Questions ?