it’s good to have (jvm) options...choosing an alternate garbage collector parallelgc: optimized...

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

It’s Good to Have (JVM) Options

Chris Hansen / Sr Engineering Manager / @cxhansen

Tori Wieldt / Technical Evangelist / @ToriWieldt

JavaOne 2017

http://bit.ly/2g74cNh

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Safe Harbor

2

This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.

Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import.

Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.

New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided.

©2008–17 New Relic, Inc. All rights reserved.

Too Many Options….x10

Source: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

3

Source: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html

©2008–17 New Relic, Inc. All rights reserved. 4

©2008–17 New Relic, Inc. All rights reserved.

Focus

Source: https://issuu.com/malcolmdousha/docs/boeing_b-52_stratofortress

5

©2008–17 New Relic, Inc. All rights reserved.

Visualization software courtesy wordle.net

6

©2008–17 New Relic, Inc. All rights reserved.

Whence Data?● New Relic monitors 500,000 JVMs running apps and services● We report JVM options to aid in troubleshooting and discovery● The JVM option and other data here is fully anonymized● Filtered for Oracle Hotspot JVM only

7

©2008–17 New Relic, Inc. All rights reserved.

What do we count as the use of an option?The key...

○ -Xmx512M○ -Xloggc:/var/log/jvmgc.log○ -XX:MaxPermSize=64M○ -XX:+PrintGCDetails

8

Value, sort of

©2008–17 New Relic, Inc. All rights reserved.

Most Common JVM Options

9

©2008–17 New Relic, Inc. All rights reserved.

Categories of Options

Memory sizing GC tuningDiagnostics

Other tuning

10

©2008–17 New Relic, Inc. All rights reserved.

Basics of Java Heap and GC

Source: http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

Source: Jörge Prante http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html

11

©2008–17 New Relic, Inc. All rights reserved.

Object Tenure

Eden S0 S1 Tenured (i.e. Old)

req

sesh sesh sesh sesh

12

©2008–17 New Relic, Inc. All rights reserved.

Source: https://publicdomainvectors.org/en/free-clipart/Snellen-eye-test-chart-image/33940.html

13

©2008–17 New Relic, Inc. All rights reserved.

Resource Limits (Object Heap)● -Xmx* : Heap max (87%)● -Xms* : Heap initial (64%)● -Xss* : Per-thread stack size (11%)● -XX:MaxHeapFreeRatio=* : When to shrink heap (6%)● -XX:MinHeapFreeRatio=* : When to expand heap (5%)

● -Xmn* : Size of young generation (6%)● -XX:SurvivorRatio=* : Eden : Survivor (6%)

14

©2008–17 New Relic, Inc. All rights reserved.

Resource Limits (PermGen/Metaspace)● -XMaxPermSize=* : PermGen max (29%)● -XPermSize=* : PermGen threshold for GC (12%)● -XX:MaxMetaspaceSize=* : Class metadata max [unlimited] (13%)● -XX:MetaspaceSize=* : Class metadata threshold for GC (7%)

15

©2008–17 New Relic, Inc. All rights reserved.

Garbage Collector Diagnostics● -Xloggc:* : Log GC events to a file, which sets... (23%)● -XX:+PrintGC, -verbose:gc : Log at a basic level (18%)● -XX:+PrintGCDetails : Log at a detailed level (19%)● -XX:+PrintGCTimeStamps : Include relative timestamp (18%)● -XX:+PrintGCDateStamps : Include absolute timestamp (16%)● -XX:+PrintHeapAtGC : Include more heap detail (10%)● -XX:+PrintTenuringDistribution : Include post-GC tenure summary (8%)● -XX:+PrintGCApplicationStoppedTime : Log safepoint time (6%)

16

©2008–17 New Relic, Inc. All rights reserved.

-XX:+PrintGC (-Xloggc:*, -verbose:gc)

17

Java HotSpot(TM) 64-Bit Server VM (25.144-b01) for bsd-amd64 JRE (1.8.0_144-b01), built on Jul 21 2017 22:07:42 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)Memory: 4k page, physical 16777216k(780800k free)

/proc/meminfo:

CommandLine flags: -XX:MaxHeapSize=1610612736 -XX:MaxNewSize=268435456 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:SurvivorRatio=1 -XX:TargetSurvivorRatio=2 -XX:+UseParallelGC 1.087: [GC (Allocation Failure) 58880K->8704K(466432K), 0.0064928 secs]1.515: [GC (Allocation Failure) 67584K->12147K(466432K), 0.0090999 secs]2.197: [GC (Allocation Failure) 71027K->19407K(466432K), 0.0110946 secs]2.644: [GC (Allocation Failure) 78287K->31108K(525312K), 0.0199316 secs]4.221: [GC (Metadata GC Threshold) 136969K->18311K(525312K), 0.0111167 secs]4.232: [Full GC (Metadata GC Threshold) 18311K->17905K(525312K), 0.0390435 secs]

©2008–17 New Relic, Inc. All rights reserved.

-XX:+PrintTenuringDistribution

18

1.112: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 58880K->8719K(466432K), 0.0059594 secs]1.627: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 67599K->12121K(466432K), 0.0112993 secs]2.368: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 71001K->19519K(466432K), 0.0162481 secs]2.808: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 78399K->31713K(525312K), 0.0232932 secs]4.377: [GC (Metadata GC Threshold) Desired survivor size 32505856 bytes, new threshold 6 (max 15) 139103K->18620K(525312K), 0.0111938 secs]4.388: [Full GC (Metadata GC Threshold) 18620K->18060K(525312K), 0.0386663 secs]

©2008–17 New Relic, Inc. All rights reserved.

-XX:+PrintGCDetails

19

1.076: [GC (Allocation Failure) [ PSYoungGen: 58880K->8689K(116736K)] 58880K->8689K(466432K), 0.0067968 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 1.601: [GC (Allocation Failure) [PSYoungGen: 67569K->12219K(116736K)] 67569K->12219K(466432K), 0.0109878 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 2.248: [GC (Allocation Failure) [PSYoungGen: 71099K->19431K(116736K)] 71099K->19431K(466432K), 0.0104851 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 2.667: [GC (Allocation Failure) [PSYoungGen: 78311K->31397K(175616K)] 78311K->31397K(525312K), 0.0212616 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 4.247: [GC (Metadata GC Threshold) [PSYoungGen: 140097K->18470K(175616K)] 140097K->18470K(525312K), 0.0114535 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 4.259: [Full GC (Metadata GC Threshold) [PSYoungGen: 18470K->0K(175616K)] [ParOldGen: 0K->17922K(349696K)] 18470K->17922K(525312K), [Metaspace: 20851K->20847K(1069056K)], 0.0394262 secs] [Times: user=0.11 sys=0.01, real=0.04 secs]

©2008–17 New Relic, Inc. All rights reserved.

-XX:+PrintHeapAtGC

20

{Heap before GC invocations=2 (full 0): PSYoungGen total 116736K, used 67610K [0x00000007b0000000, 0x00000007baa80000, 0x00000007c0000000) eden space 58880K, 100% used [0x00000007b0000000,0x00000007b3980000,0x00000007b3980000) from space 57856K, 15% used [0x00000007b3980000,0x00000007b42069b0,0x00000007b7200000) to space 57856K, 0% used [0x00000007b7200000,0x00000007b7200000,0x00000007baa80000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K1.553: [GC (Allocation Failure) 67610K->11771K(466432K), 0.0087631 secs]Heap after GC invocations=2 (full 0): PSYoungGen total 116736K, used 11771K [0x00000007b0000000, 0x00000007be400000, 0x00000007c0000000) eden space 58880K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007b3980000) from space 57856K, 20% used [0x00000007b7200000,0x00000007b7d7ed80,0x00000007baa80000) to space 57856K, 0% used [0x00000007b3980000,0x00000007b3980000,0x00000007b7200000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K}

©2008–17 New Relic, Inc. All rights reserved.

Garbage Collector Diagnostics (Files)● -XX:+UseGCLogFileRotation (8%)● -XX:NumberOfGCLogFiles=* (9%)● -XX:GCLogFileSize=* (9%)

21

©2008–17 New Relic, Inc. All rights reserved.

Heap Dump● -XX:+HeapDumpOnOutOfMemoryError : hprof heap dump (20%)● -XX:HeapDumpPath=* : Path and file name for .hprof file (20%)● Analyze hprof file with VisualVM, jhat, or your favorite profiler

22

©2008–17 New Relic, Inc. All rights reserved.

Choosing an Alternate Garbage Collector

● ParallelGC: Optimized for throughput (60% — default)● CMS (Concurrent Mark Sweep): Optimized for low pause (28%)● G1: Low pause and high throughput (12%)● Serial GC: Good for batch processes

23

©2008–17 New Relic, Inc. All rights reserved.

Other Optimizations● -XX:+UseCompressedOops : Compress 64-bit ordinary object pointers

(12%)● -XX:+DisableExplicitGC : Ignore System.gc() (11%)● -XX:+AggressiveOpts : Use experimental runtime optimizations (5%)● -XX:-OmitStackTraceInFastThrow [5] : Java 5 JIT optimization (7%)● -XX:-TieredCompilation [7-] : Use simple JIT compilation (5%)

24

©2008–17 New Relic, Inc. All rights reserved.

What about the least used options?● -ea, -enableassertions (<1%)● Watch for typos!

○ Quoting/spacing○ Use proper key-value delimiter: “:”, “=”, “”, etc.○ Case sensitive○ JVM won’t always warn that these are ignored

● Watch for multiple with the same key! (Last one wins)

25

©2008–17 New Relic, Inc. All rights reserved.

Test and Monitor the Effects!

26

-XX:SurvivorRatio=1 [v 8] -XX:TargetSurvivorRatio=2 [v 50] -XX:MaxNewSize=256M [v ergo]

©2008–17 New Relic, Inc. All rights reserved.

Conclusion● People want diagnostics, especially for object allocation and collection● Know your garbage collector.

○ Go to CON4479 this afternoon.○ Consider this MOOC.

● Defaults are good!● Test and monitor your option changes.

27

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Bonus Content!Crunching the Docker Numbers

28Full details https://blog.newrelic.com/2016/06/16/docker-container-infographic/

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Glamour Stats

29

● New Relic has been using and monitoring containers since 2014● Monitored 418 million containers in the last two years● Monitor 1.5 million containers average a day● Average customer has more than 800 containers running at any

given moment● Our top ten customers (by Docker container usage) spin up more

than 92,000 containers a day

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Java and Docker

Docker is used on all kinds of applications:

• PHP (33%)• Java (31%)• Ruby (29%)• Node (28%)• Python (11%)• .NET (8%)

30

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Docker Growth

31

2015 2016

Average Number of Containers 96 280

Average # of containers per company for the year

28,000

Greatest number of containers in a single day for a single company

1,596 135,000

Greatest number of containers in a year for a single company

4,800,000

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Pets to Cattle to Bacteria* Average life of containers:

13 hours (2015) to 9.25 hours (2016)

* Average life of containers running less than an hour:10 minutes (2015) to 8 minutes (2016).

* 20% of all containers run for less than one minute.

32

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

Conclusions• Container usage continues to grow• From VM replacement to ephemeral• Paradigms are shifting

33

©2008–17 New Relic, Inc. All rights reserved.

finThis page is intentionally left blank

34