it’s good to have (jvm) options...choosing an alternate garbage collector parallelgc: optimized...

New Relic Template 2017

©2008–17 New Relic, Inc. All rights reserved.

It’s Good to Have (JVM) Options

Chris Hansen / Sr Engineering Manager / @cxhansen

Tori Wieldt / Technical Evangelist / @ToriWieldt

JavaOne 2017

http://bit.ly/2g74cNh

http://bit.ly/2g74cNh



Safe Harbor

2

This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.

Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import.

Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.

New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided.


Too Many Options….x10

Source: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

3

Source: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html


Focus

Source: https://issuu.com/malcolmdousha/docs/boeing_b-52_stratofortress

5


Visualization software courtesy wordle.net

6


Whence Data?● New Relic monitors 500,000 JVMs running apps and services● We report JVM options to aid in troubleshooting and discovery● The JVM option and other data here is fully anonymized● Filtered for Oracle Hotspot JVM only

7


What do we count as the use of an option?The key...

○ -Xmx512M○ -Xloggc:/var/log/jvmgc.log○ -XX:MaxPermSize=64M○ -XX:+PrintGCDetails

8

Value, sort of


Most Common JVM Options

9


Categories of Options

Memory sizing GC tuningDiagnostics

Other tuning

10


Basics of Java Heap and GC

Source: http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

Source: Jörge Prante http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html

11


Object Tenure

Eden S0 S1 Tenured (i.e. Old)

req

sesh sesh sesh sesh

12


Source: https://publicdomainvectors.org/en/free-clipart/Snellen-eye-test-chart-image/33940.html

13


Resource Limits (Object Heap)● -Xmx* : Heap max (87%)● -Xms* : Heap initial (64%)● -Xss* : Per-thread stack size (11%)● -XX:MaxHeapFreeRatio=* : When to shrink heap (6%)● -XX:MinHeapFreeRatio=* : When to expand heap (5%)

● -Xmn* : Size of young generation (6%)● -XX:SurvivorRatio=* : Eden : Survivor (6%)

14


Resource Limits (PermGen/Metaspace)● -XMaxPermSize=* : PermGen max (29%)● -XPermSize=* : PermGen threshold for GC (12%)● -XX:MaxMetaspaceSize=* : Class metadata max [unlimited] (13%)● -XX:MetaspaceSize=* : Class metadata threshold for GC (7%)

15


Garbage Collector Diagnostics● -Xloggc:* : Log GC events to a file, which sets... (23%)● -XX:+PrintGC, -verbose:gc : Log at a basic level (18%)● -XX:+PrintGCDetails : Log at a detailed level (19%)● -XX:+PrintGCTimeStamps : Include relative timestamp (18%)● -XX:+PrintGCDateStamps : Include absolute timestamp (16%)● -XX:+PrintHeapAtGC : Include more heap detail (10%)● -XX:+PrintTenuringDistribution : Include post-GC tenure summary (8%)● -XX:+PrintGCApplicationStoppedTime : Log safepoint time (6%)

16


-XX:+PrintGC (-Xloggc:*, -verbose:gc)

17

Java HotSpot(TM) 64-Bit Server VM (25.144-b01) for bsd-amd64 JRE (1.8.0_144-b01), built on Jul 21 2017 22:07:42 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)Memory: 4k page, physical 16777216k(780800k free)

/proc/meminfo:

CommandLine flags: -XX:MaxHeapSize=1610612736 -XX:MaxNewSize=268435456 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:SurvivorRatio=1 -XX:TargetSurvivorRatio=2 -XX:+UseParallelGC 1.087: [GC (Allocation Failure) 58880K->8704K(466432K), 0.0064928 secs]1.515: [GC (Allocation Failure) 67584K->12147K(466432K), 0.0090999 secs]2.197: [GC (Allocation Failure) 71027K->19407K(466432K), 0.0110946 secs]2.644: [GC (Allocation Failure) 78287K->31108K(525312K), 0.0199316 secs]4.221: [GC (Metadata GC Threshold) 136969K->18311K(525312K), 0.0111167 secs]4.232: [Full GC (Metadata GC Threshold) 18311K->17905K(525312K), 0.0390435 secs]


-XX:+PrintTenuringDistribution

18

1.112: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 58880K->8719K(466432K), 0.0059594 secs]1.627: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 67599K->12121K(466432K), 0.0112993 secs]2.368: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 71001K->19519K(466432K), 0.0162481 secs]2.808: [GC (Allocation Failure) Desired survivor size 59244544 bytes, new threshold 7 (max 15) 78399K->31713K(525312K), 0.0232932 secs]4.377: [GC (Metadata GC Threshold) Desired survivor size 32505856 bytes, new threshold 6 (max 15) 139103K->18620K(525312K), 0.0111938 secs]4.388: [Full GC (Metadata GC Threshold) 18620K->18060K(525312K), 0.0386663 secs]


-XX:+PrintGCDetails

19

1.076: [GC (Allocation Failure) [ PSYoungGen: 58880K->8689K(116736K)] 58880K->8689K(466432K), 0.0067968 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 1.601: [GC (Allocation Failure) [PSYoungGen: 67569K->12219K(116736K)] 67569K->12219K(466432K), 0.0109878 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 2.248: [GC (Allocation Failure) [PSYoungGen: 71099K->19431K(116736K)] 71099K->19431K(466432K), 0.0104851 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 2.667: [GC (Allocation Failure) [PSYoungGen: 78311K->31397K(175616K)] 78311K->31397K(525312K), 0.0212616 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 4.247: [GC (Metadata GC Threshold) [PSYoungGen: 140097K->18470K(175616K)] 140097K->18470K(525312K), 0.0114535 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 4.259: [Full GC (Metadata GC Threshold) [PSYoungGen: 18470K->0K(175616K)] [ParOldGen: 0K->17922K(349696K)] 18470K->17922K(525312K), [Metaspace: 20851K->20847K(1069056K)], 0.0394262 secs] [Times: user=0.11 sys=0.01, real=0.04 secs]


-XX:+PrintHeapAtGC

20

{Heap before GC invocations=2 (full 0): PSYoungGen total 116736K, used 67610K [0x00000007b0000000, 0x00000007baa80000, 0x00000007c0000000) eden space 58880K, 100% used [0x00000007b0000000,0x00000007b3980000,0x00000007b3980000) from space 57856K, 15% used [0x00000007b3980000,0x00000007b42069b0,0x00000007b7200000) to space 57856K, 0% used [0x00000007b7200000,0x00000007b7200000,0x00000007baa80000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K1.553: [GC (Allocation Failure) 67610K->11771K(466432K), 0.0087631 secs]Heap after GC invocations=2 (full 0): PSYoungGen total 116736K, used 11771K [0x00000007b0000000, 0x00000007be400000, 0x00000007c0000000) eden space 58880K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007b3980000) from space 57856K, 20% used [0x00000007b7200000,0x00000007b7d7ed80,0x00000007baa80000) to space 57856K, 0% used [0x00000007b3980000,0x00000007b3980000,0x00000007b7200000) ParOldGen total 349696K, used 0K [0x0000000760000000, 0x0000000775580000, 0x00000007b0000000) object space 349696K, 0% used [0x0000000760000000,0x0000000760000000,0x0000000775580000) Metaspace used 16469K, capacity 16654K, committed 16768K, reserved 1064960K class space used 2078K, capacity 2139K, committed 2176K, reserved 1048576K}


Garbage Collector Diagnostics (Files)● -XX:+UseGCLogFileRotation (8%)● -XX:NumberOfGCLogFiles=* (9%)● -XX:GCLogFileSize=* (9%)

21


Heap Dump● -XX:+HeapDumpOnOutOfMemoryError : hprof heap dump (20%)● -XX:HeapDumpPath=* : Path and file name for .hprof file (20%)● Analyze hprof file with VisualVM, jhat, or your favorite profiler

22


Choosing an Alternate Garbage Collector

● ParallelGC: Optimized for throughput (60% — default)● CMS (Concurrent Mark Sweep): Optimized for low pause (28%)● G1: Low pause and high throughput (12%)● Serial GC: Good for batch processes

23


Other Optimizations● -XX:+UseCompressedOops : Compress 64-bit ordinary object pointers

(12%)● -XX:+DisableExplicitGC : Ignore System.gc() (11%)● -XX:+AggressiveOpts : Use experimental runtime optimizations (5%)● -XX:-OmitStackTraceInFastThrow [5] : Java 5 JIT optimization (7%)● -XX:-TieredCompilation [7-] : Use simple JIT compilation (5%)

24


What about the least used options?● -ea, -enableassertions (<1%)● Watch for typos!

○ Quoting/spacing○ Use proper key-value delimiter: “:”, “=”, “”, etc.○ Case sensitive○ JVM won’t always warn that these are ignored

● Watch for multiple with the same key! (Last one wins)

25


Test and Monitor the Effects!

26

-XX:SurvivorRatio=1 [v 8] -XX:TargetSurvivorRatio=2 [v 50] -XX:MaxNewSize=256M [v ergo]


Conclusion● People want diagnostics, especially for object allocation and collection● Know your garbage collector.

○ Go to CON4479 this afternoon.○ Consider this MOOC.

● Defaults are good!● Test and monitor your option changes.

27

https://blogs.oracle.com/java-platform-group/free-java-virtual-machine-troubleshooting-mooc-starts-september-20th



Bonus Content!Crunching the Docker Numbers

28Full details https://blog.newrelic.com/2016/06/16/docker-container-infographic/



Glamour Stats

29

● New Relic has been using and monitoring containers since 2014● Monitored 418 million containers in the last two years● Monitor 1.5 million containers average a day● Average customer has more than 800 containers running at any

given moment● Our top ten customers (by Docker container usage) spin up more

than 92,000 containers a day



Java and Docker

Docker is used on all kinds of applications:

• PHP (33%)• Java (31%)• Ruby (29%)• Node (28%)• Python (11%)• .NET (8%)

30



Docker Growth

31

2015 2016

Average Number of Containers 96 280

Average # of containers per company for the year

28,000

Greatest number of containers in a single day for a single company

1,596 135,000

Greatest number of containers in a year for a single company

4,800,000



Pets to Cattle to Bacteria* Average life of containers:

13 hours (2015) to 9.25 hours (2016)

* Average life of containers running less than an hour:10 minutes (2015) to 8 minutes (2016).

* 20% of all containers run for less than one minute.

32



Conclusions• Container usage continues to grow• From VM replacement to ephemeral• Paradigms are shifting

33


finThis page is intentionally left blank

34

it’s good to have (jvm) options...choosing an alternate garbage collector parallelgc: optimized...

Documents