splunk conf 2014 - splunking the java virtual machine

Copyright © 2014 Splunk Inc.

Damien DallimoreDev Evangelist , CSO Office @ Splunk

Splunking the JVM

2

DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events or the

expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important

factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other

commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

3

From Middle Earth

Make Splunk Apps & Add-ons

JVM background

4

5

apps.splunk.com

github.com/damiendallimore

What is this JVM thing ?

7

JVM = Java Virtual Machine

First appeared in the early 90’s

Now the dominant runtime for enterprise applications

Application Servers Enterprise Service Buses Databases

NoSQL Distributed Big Data Web Servers

Directory Servers Search Engines Build Systems

Gaming Platforms Trading Systems Reservation Systems

Core Banking Messaging Infrastructure Proprietary Systems

8

9

Many JVM Variants

Oracle Hotspot

OpenJDK

Oracle JRockit

IBM J9

Azul Zing

More or less the same. OpenJDK is reference impl for Java 7/8

Getting merged into Hotspot

Commercial alternatives

10

JVMs are here to stay

It’s not just about the “J”

Big Data frameworks

You should care about getting insights into your JVM data

Getting at the data

12

Data Sources

LogsDevelopers

JMX (Java Management Extensions)Instrumentation Agents

Operating SystemDistributed Communications

Log Data

14

Standard Log Files

Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 1153 msOct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternalINFO: Starting service Catalina

Application logs that are part of the product

Developer logs for any code that was deployed

Written to local disk or network storage

15

Garbage Collector Logs

54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]

Generated by way of arguments passed to the JVM at startup

-verbose:gc-Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

16

SplunkJavaLogging

Sometimes you can’t write to file or deploy a UF

Appenders for Java Util Logging , Log4J , Logback

Simply add a logging appender to your logging configuration file

17

LogBack Appender Example

18

Code

19

Best practice semantic format

2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane”

20

Better Exception Logging

21

Easier to work with in Splunk

22

Splunk SDK for Java

Use the SDK from any JVM Language , Java / Groovy / Scala etc….

Send log events via REST , UDP or TCP directly to Splunk from your code

Spring Integration Adaptors available on Github

SDK available from dev.splunk.com

23

Using Java SDK

24

Using Spring

JMX

26

JMX = Java Management Extensions

Monitor JVM via MBean attributes , operations and notifications

JVM MBeans

Vendor MBeans

Custom Coded MBeans

27

Browse MBeans with JConsole

28

Getting this data into Splunk

Runs on all supported Splunk platforms

Works with all main JVM variants

100% Free and Open Source

29

Simple to Configure

30

Many Connectivity Options

Let’s look at this Splunk App in action

31

This is great for monitoring a single JVM……

32

But what if we have dozens , hundreds , even thousands ?

33

Multiple jmxserver elements per configuration file

34

Multiple configuration files per Splunk Instance

35

But stacking vertically will only get you so far

36

Go horizontal to achieve real scale

Indexer Cluster

UFs with JMX Mod Input

Monitored JVMs

37

In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases.

But what if you want to go deeper ?

Instrumentation Agents

39

SplunkJavaAgent

Dynamic Byte Code Injection Agent for JVMs

No need to change any code to get APM level insights

This is all you pass to the JVM at startup :

-javaagent:splunkagent.jar

40

What raw metrics can you get ?Bytecode Injection

• class loading• method execution• method timings • method call stack • caught & uncaught exceptions

JMX • running in local in-memory mode

Binary HPROF Memory dumps decoded in text• no more static hprof dumps and then loading into another Mem Analysis tool

41

Simple Configuration File

42

43

Raw events streamed into Splunk

44

Search with Splunk

45

Reveal APM insights

Operating System

47

The JVM executes as a process

Let’s see what the host Operating System can tell us : top

External programs that provide additional JVM insights : jstat

Index this data in Splunk and correlate

48

top

49

50

jstat

51

Distributed Communications

53

Valuable data flows in & out of JVMs

Messaging

Capture packets off the wire

Build a Splunk App !

55

Opportunity

Reuse some of the data collection approaches in this presentation and build out a specific JVM app

Publish it on apps.splunk.com

Make some moneyGenerate publicity for yourself / your companyGenerate sales leadsSell more Splunk licenses if you are also a partnerDo it because you love community collaborationBecome my beer buddy for life

56

Some JVM App Ideas

Splunk for JBossSplunk for Tomcat

Splunk for SolrSplunk for Cassandra

Splunk for “Apache Project”

I am here to help you

THANK YOU@damiendallimoreddallimore@splunk.com

Questions ?