leak, lock and a long pause
TRANSCRIPT
![Page 1: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/1.jpg)
Leak, Lock and a Long Pause
Troubleshooting JVM issues
![Page 2: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/2.jpg)
You will learn that
• There are memory leaks in Java• Garbage collector can be a bottleneck• Thread locks are more common than you
think• You can automatically detect these problems
![Page 3: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/3.jpg)
How can JVM leak memory?
• Static fields are not gc’d• Thread-local + thread pools• Application scope• Complex bi-directional refs• OS resource handles: files, sockets, db
![Page 4: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/4.jpg)
Static fields
class Forever { // Unless you explicitly unload this class // GC will not touch it static final ArrayList list = new ArrayList(1000);}
![Page 5: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/5.jpg)
Thread-local
public void doFilter(ServletRequest req, ServletResponse res){ myThreadLocal.set(new BigLeak()); // thread goes back to the thread pool // with the big leak planted chain.doFilter(req, res);}
![Page 6: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/6.jpg)
Application scope
getServletContext().setAttribute(”leak", bigObj);// unless you call removeAttribute somewhere// this will live as long as Tomcat is up
![Page 7: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/7.jpg)
Bi-directional relationships
org.w3c.dom.Document doc = readXmlDocument();org.w3c.dom.Node child = doc.getDocumentElement().getFirstChild();doc.removeNode(child);doc = null;// child still has reference to doc. No GC for the doc.
![Page 8: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/8.jpg)
OS resources
try { Connection conn = ConnectionFactory.getConnection(); ... conn.close();} catch (Exception e) { // oops, connection is still open}
![Page 9: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/9.jpg)
GC Pause
source: oracle.com
![Page 10: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/10.jpg)
GC Pause
• Full garbage collection stops the world!• Heavily utilized heap = frequent pauses• Complex dependency graph = long pause• Lots of long living objects = long pauses
![Page 11: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/11.jpg)
Heap size vs pause
source: performanceterracotta.blospot.com
![Page 12: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/12.jpg)
Is it really our problem?
source: plumbr.eu
![Page 13: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/13.jpg)
Is it really our problem?
QUESTION:Does it impact our business if the system regularly suspends itself for several seconds?BETTER QUESTION:What is the level of unresponsiveness we allow in our system?As low as possible = bullshit answer
![Page 14: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/14.jpg)
No measurement = no answer
pauses for about half a second 2-3 times a minute
![Page 15: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/15.jpg)
GC friendly programming
• Avoid long living objects – minor gc is faster• Avoid mutable objects – questionable…• Design your classes small – faster compaction• Avoid complex dependency graphs• NEVER call System.gc() – disturbs optimal plan
![Page 16: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/16.jpg)
Thread locking
“Plain Threads are the GOTO of Today’s Computing”--Dr Hartmut Kaiser
![Page 17: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/17.jpg)
Thread locking
“Locking is not evil, lock contention is”Vladimir Šor
![Page 18: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/18.jpg)
Contention
![Page 19: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/19.jpg)
Thread contention
source: blogs.mulesoft.org
![Page 20: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/20.jpg)
Thread contention - causes
• Too broad sync scope• Overusing same locking object• Shared mutable state abuse
![Page 21: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/21.jpg)
What’s wrong here?
public boolean isAdmin(String name) { synchronized (privileges) { String privs= privileges.get("usr.”+name+".prv"); if (privs== null) return false; else return (privs.indexOf(”administrator") >= 0); }}
![Page 22: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/22.jpg)
Too broad sync scope// “Protect data, not code” – Michael Suesspublic boolean isAdmin(String name) { synchronized (privileges) { String privs= privileges.get("usr.”+name+".prv"); if (privs== null) return false; else return (privs.indexOf(”administrator") >= 0); }}
![Page 23: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/23.jpg)
What’s wrong here?public class Attributes { private HashMap usersStore; private HashMap servicesStore;
public synchronized void setUserAttributes(String usr, UserAttrs attrs) { usersStore.put(usr, attrs); }
public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) { servicesStore.put(svc, attrs); }
![Page 24: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/24.jpg)
Overusing a lockpublic class Attributes { private HashMap usersStore; private HashMap servicesStore;
public synchronized void setUserAttributes(String usr, UserAttrs attrs) { usersStore.put(usr, attrs); }
public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) { servicesStore.put(svc, attrs); }
// both methods sync on same Attributes instance
![Page 25: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/25.jpg)
Shared mutable state
“Every piece of mutable state you have is a liability”Martin Odersky
![Page 26: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/26.jpg)
Is there really no other way?
balance.deposit(amount)
![Page 27: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/27.jpg)
Is there really no other way?
newBalance = oldBalance.deposit(amount)// no sync needed
![Page 28: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/28.jpg)
Share immutable objects
public class Address { public final String streetName; public final int aptNo; public final String zipCode; public Address (String name, int apt, String zip){ // initialize all the fields }}
![Page 29: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/29.jpg)
Jhiccup
JVM AppJVM App
jhiccup thread
hiccup.hloghiccup.hlogmeasure thread sleepdelta(expected – actual)
src: azulsystems.com
![Page 30: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/30.jpg)
What is a JVM Hiccup?
• When your application is stalled• GC stop-the-world• Virtualization• CPU contention• OS Scheduler
![Page 31: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/31.jpg)
Plumbr
System running on JVMSystem running on JVM
plumbr agentplumbr agent
plumbr serverplumbr server
usage data
![Page 32: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/32.jpg)
DeMOtime for
![Page 33: Leak, lock and a long pause](https://reader035.vdocument.in/reader035/viewer/2022062420/55c54be3bb61eb667c8b4660/html5/thumbnails/33.jpg)
Sources
• plumbr.eu/blog/how-do-leak-detectors-work-plumbr-case-study
• java.dzone.com/articles/how-tame-java-gc-pausesthinkingparallel.com/2007/07/31/10-ways-to-reduce-lock-contention-in-threaded-programs/
• blogs.mulesoft.org/chasing-the-bottleneck-true-story-about-fighting-thread-contention-in-your-code
• ibm.com/developerworks/library/j-threads2/• http://www.azulsystems.com/giltene/how-java-got-
the-hiccups