Java Performance Tuning Toolbox

Java Performance Tuning Toolbox

Java Monitoring Tools

About MeMy name is Jerry KurianI am a solution architect and have spent 18 years in IT industry. I have worked a lot with Java and currently work a lot with Scala, Angular JS and of course JavaI am co-founder and CTO of RainCloud (www.raincloud.in) and MingleStats (www.minglestats.com)I enjoy coding and solving business problems. You can reach me @ jerrykurian@raincloud.in.Connect with me on Linked In https://in.linkedin.com/in/jerryk You can download sample code from http://bit.ly/1oDKIRU

Tools with JVMJDK comes with a number of tools that provide insight into the JVMjcmdPrints basic class, thread and VM information for a java process. This is suitable for use in shell scripts. jcmd process_id command optional_paramsJconsoleProvides a graphical view of the JVM activitiesJhatReads and helps analyze memory heap dumps

Tools with JVMJmapProvides heap dumps and other memory information about the JVM memory usageJinfoProvides visibility into the system properties and allows some system properties to be set dynamicallyJstackDumps the stacks of a java processJstatProvides information about GC and ClassLoading activities.Jvisualvm A GUI tool to profile a running application

Basic VM InformationJVM provides the basic information using the toolsUptimeJcmd process_id VM.uptimeSystem propertiesA set of system properties can be displayed usingJcmd process_id VM.system_propertiesOrJinfo sysprops process_idJVM VersionJcmd process_id VM.version provides the version of the VM that is running

Basic VM InformationJVM Command LineThe command line can be displayed using the VM summary tab of jconsole orJcmd process_id VM.command_lineJVM Tuning FlagsThe tuning flags in effect for the application can be obtained usingJcmd process_id VM.flags [-all]

Working with tuning flagsThere are hundreds of tuning flags in a JVM and most of them are pretty obscureYou need not ever change most of the flagsFiguring out which flags are in effect is a frequent task when diagnosing performance issuesA useful way of figuring out the flags for a particular platform isJava other_options XX:+PrintFlagsFinal version

PrintFlags

Flag data from this command is printed out in one of two waysThe colon in the first line indicates the value is not the default one. This can happen for following reasonsThe flags value was specified on command lineSome other option changed the valueThe JVM calculated default value ergonomicallyThe second line indicates that default value is set. {pd product} indicates default value is platform dependent

Manageable FlagsOther possible value for the last column in PrintFlag output is manageable

A manageable flag is one whose value can be changed dynamically during runtime

jinfoTo retrieve the values of all the flags in the processJinfo flags process_idTo inspect the value of an individual flagJinfo flag PrintGCDetails process_idTo dynamically modify the flag that is manageableJinfo flag PrintGCDetails process_id (Turns off PrintGCDetails)Jinfo can change value of the flag but that doesnt mean that the JVM will respond to that changeMost flags that affect GC algorithm are used at startup time and any change to it may not have any impact

Thread Information

Reading Thread InformationJconsole and jvisualvm display information in real time about the number of threads in applicationThe stack can be obtained viaJstack process_idStack information can also be obtained fromJcmd process_id Thread.print

Thread VisibilityJconsole is a tool that can give visibility into the state of threads in a VM

Blocked Threads VisibilityBest way to know when threads are blocked is to use tools that can look into the JVM and know at low level when the threads are blockedJava flight recorder is the tool that can give such visibilityWe can drill into events that JFR captures and look for those that cause a thread to blockJFR is a commercial tool, only available with the commercial version of the Oracle JDK

GC Analysis

Tuning Heap UsageGC tuning is important but better gain can sometimes be obtained by optimizing the codeThere are two conflicting goalsUsing less memory is the best way to improve the efficiency of garbage collectorFrequently re-creating some objects can lead to worse performance overall

Heap AnalysisTo get a clear understanding of heap usage, we need to get visibility into the heapReducing memory is an important goal. But as with most performance analysis, it helps to target effortsAnalysis must be performed to know which objects are consuming most memoryThe easiest way to do this is via a Heap Histogram

Heap HistogramHeap Histogram can be obtained by using the jcmd commandJcmd process_id GC.class_histogram

A sample output is seen above. Klass related objects are often near the top. They are the metadata objects from loading the classesSimilar output is available via following commandsJmap histo process_idJmap histo:live process_id

Heap DumpsHistograms are great at analysing issues caused by allocating too many resources of one or two classesBut deeper analysis can be better done by creating a heap dumpHeap dump can be generated from the command line usingJcmd process_id GC.heap_dump /path/to/dump.hprofJmap dump:live , file=/path/to/dump.hprof proc_idIncluding live in jmap will cause a full GC before the dump. This is default in case of jcmdIf you want to include other (dead) objects in jmap, then add all at the end of jcmd command

Analyzing Heap DumpsThere are various tools that help in analyzing the heap dumpsSome of the tools areJhatThis is the original analyzer tool that reads through the dump and shows analysis through a series of web pagesJvisualvmThe monitor tab of jvisual vm can view the dump of a live process or open an existing dump file to showcase the detailsMat (http://www.eclipse.org/downloads/download.php?file=/mat/1.5/rcp/MemoryAnalyzer-1.5.0.20150527-win32.win32.x86_64.zip)EclipseLink memory analyzer tool is an open source tool that can be used to browse through heap and run some SQL like commands

Analyzing Heap Dumps Retained MemoryThe first pass analysis of heap dumps generally involves retained memoryThe retained memory of an object is the amount of memory that will be freed if the object itself were garbage collected

Analyzing Heap DumpsIn the diagram, the retained memory of object Trio includes the objects Sally and DavidObject Michael is not included since it has other referencesObjects that retain a large amount of memory are called dominators

Analyzing Heap DumpsIf heap analysis shows that there are few objects that dominate the bulk of heap, then effort has to be made to create lesser number of these objects

Analyzing Heap Dumps - HistogramsThe problem with the retained memory is that if the program is sharing lots of objects then they will not show upHistograms aggregates types of objects and show the memory occupied by each typeThe objects taking up the largest space are obvious targetsThe mat can help trace those objects to find what is holding on to them

Analyzing Heap Dumps - Histograms

Profiling Tools

ProfilingProfiling tools are the most important tools in a performance analysts toolkitProfiling tools work by attaching themselves to the target VMThe target application and the profiling tool exchange information about the behaviour of the applicationProfiling happens in one of two modesSampling ModeInstrumented Mode

Sampling ProfilersSampling is the basic mode of profiling and carries the least amount of overheadReducing overheads is important since by introducing the profiling, the performance characteristics get alteredLimiting impact of profilers will lead to results that more closely model how application behaves in the real time

Sampling Profilers - IssuesSampling profilers are unfortunately prone to errorsSampling profiles work when a timer fires periodicallyThe profiler looks at which thread is executing which method at the time of samplingThat method is thenassumed to be executing since the last time timer firedThis can lead to sampling error where alternatively firing methods may not be accurately sampledThe way to overcome this is to sample over a longer period

Instrumented ProfilersInstrumented Profilers are much more intrusive than the sampling onesThey can also provide more beneficial information about whats happening inside a program

Sampling Vs InstrumentedRunning profiler in two different modes on same application could provide different viewsAs can be seen in the picture above, the method deemed to be taking the most time is different in both the cases

Which one to use?Instrumented profiler is more accurate and additional information is quite usefulBut, instrumented profilers work by altering the bytecode sequence of the classes and can thus affect performanceSampling profilers can only collect data when threads are at a safepoint that is they are allocating memoryIt might be better to start with the sampling profilers and if the information is not helpful enough then go for the instrumented profiling

Analyzing profile DataWhile going through the profiling results, you can certainly look at the methods that take up most of the timeBut Generally, the hot spots take up 1 to 2 % of the time and just optimizing them may not yield much benefits. Its best to use the profile data to identify areas that may be impacting. For example, if class loading is taking the most time, then the solution could be to ensure less classes are loaded in the beginning or loaded in parallel

Visual VM

JVisualvmJvisual vm is a free profiler tool that comes along with the JVMVisual vm allows a peek into various type of dataMemoryThreadsCore DumpHeap Dump

ConnectTo make use of visual vm, you will first need to connect to a running application

Application OverviewDouble clicking on the connected application will show the details of the application

Application MonitorMonitoring application will provide information on Heap, Permgen, classes and threads

Monitoring ThreadsThe thread view will provide details of the live threads in the application. You can see general and detailed viewsThere is a support for dumping a thread that is printed on console

Profiling ApplicationsThe Profiler tab of an application enables you to start and stop the profiling session of a local application.Profiling results are displayed in the Profiler tab. You can use the toolbar to refresh the profiling results, invoke garbage collection and save the profiling dataYou can choose the following profiling optionsMemoryCPU

CPU ProfilingThis profile command returns detailed data on method-level CPU performance (execution time), showing the total execution time and number of invocations for each method.