oracle weblogic troubleshooting performance
TRANSCRIPT
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
1/37
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
2/37
Oracle Weblogic ServerBest Practices for Troubleshooting PerformanceIssues - CON8307
Laurent GoldsztejnFusion Middleware Proactive Support
September 2014@weblogicsupport
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
3/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor StatementThe following is intended to outline our general product direction. It is inteinformation purposes only, and may not be incorporated into any contract.commitment to deliver any material, code, or functionality, and should notin making purchasing decisions. The development, release, and timing of afunctionality described for Oracle’s products remains at the sole discretion
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
4/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Performance issues
Best practices with server logs
Best practices with thread dumps
Deconstructing Java threads
Examples & Demo
1
2
3
4
5
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
5/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Performance Issues
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
6/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Performance IssuesServer hanging
The server does not respond to new requests
Requests time out
Requests take longer and longer to process
The server is no longer reported as running
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
7/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Performance IssuesIssues that could lead to server hang or other severe performance issues
Application code or WLS internal issue
Too high number of file descriptors or open s
JVM spending too much time in GC
Long running JDBC requests
Lock contention between NM and Managed Server
Excessive or unneeded Remote JNDI look
WLS Session Replication problem
Memory leak
Deadlock
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
8/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Performance IssuesData needed to analyze a performance issue
Server logs, -verbose:gc to print information at every collection
Stdout redirected to server log:
Servers->Logging->General->Advanced-> Redirect Stdout Logging Enabled
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true
Java Flight Recordings (JFR)
Collections of general data about the application (GCs, Heap, Optimizations, Objects stat
RDA collections or configuration files
Thread dumps / Heap dumps
WLDF to collect metrics, setup watch and notifications, and to define instrumentatio
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
9/37Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with server logs
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
10/37Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with server logsErrors in Server Log file
Check for StuckThreads
Check for Too ManyOpen Files
Check foOutOfMem
message
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
11/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with server logsExample of Stuck Threads
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
12/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with server logs
• Socket connections use file descriptors
• A socket can remain in TIME_WAIT after they are closed to guarantee that delayed packets in the the corresponding socket
tcp_time_wait_interval on Linux (default 4 minutes)
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters on Windows (default 1 minute)
• If many connections are being opened and closed quickly then socket's in TIME_WAIT may begia system
– View sockets in TIME_WAIT using netstat
Too Many Open Files
Support Pattern: How To Troubleshoot Too Many Open Files Problems [ID 8674
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
13/37
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
14/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumps
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
15/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumps
•
A java thread dump is a snapshot that shows what every threaprocess is doing at a particular point in time
Definitions
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
16/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumpsWhat does a Thread Dump contain?
• State
• Name,Queue
• tid, nid, prio
• Classes
• Packages
• Method Calls
• Locks
• Vendor• Version
• Build
• Architecture
• Mode
• Date andtime thedump wascreated
Time Stamp JVM Info
ThreadsStack
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
17/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumpsDifferent ways to take thread dumps
Remote DiagnosticAgent (RDA)
Unix Command-linekill -3
WLST threadDump()
jcmprin
Hot
WLS Admin Console Jstack Java VisualVM
T
(Javac
Different ways to take thread dumps in WebLogic Server [ID 1098691.1]
How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?
https://blogs.oracle.com/fmwinstallproactive/entry/how_to_use_rda_to
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
18/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumps
• Different JVM Vendors display the data in different formats• Markers for start/end of thread dumps
• Reporting of locks
• Thread states and method signatures
•The underlying data exposed remains the same across vendors
• Thread dumps can contain lots of data
• Trying to read them in a text editor can be very tricky
Viewing a Thread Dump
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
19/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Best practices with thread dumpsFree parsing tools
Tool Description
Samuraihttp://yusuke.homeip.net/samurai/en/index.ht
ml
Light weight open source tool that
thread dumps and GC stats from lo
and presents them in different tab
with color code for easy visual ana
Thread Dump Analyzer (TDA)http://java.net/projects/tda/pages/Home
Swing GUI for analyzing Thread DuHeap Information
Thread Logichttp://java.net/projects/threadlogic/
Built upon TDA by adding logic for
patterns found in application serve
http://yusuke.homeip.net/samurai/en/index.htmlhttp://yusuke.homeip.net/samurai/en/index.htmlhttp://java.net/projects/tda/pages/Homehttp://java.net/projects/threadlogic/http://java.net/projects/threadlogic/http://java.net/projects/tda/pages/Homehttp://yusuke.homeip.net/samurai/en/index.htmlhttp://yusuke.homeip.net/samurai/en/index.html
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
20/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
TroubleshootingThread Dump Analysis – Five High level steps
Look at default and custom execute threads
Identity the cause of stuck threads by looking at the last method and java class
Ignore the idle threads
Check thread dumps from different managed servers to find potential relationshithe stuck threads
In most cases, WLS internal threads, such as muxer and listen threads, can be
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
21/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
22/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
23/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threadsThread.State
State Description
NEW Available thread ready to process work request for any work manager
BLOCKED
Thread blocked by another and waiting for the other thread to complete its o
release its locks. If more than one threads attempt to acquire the monitor of
object then only one thread (selected by the JVM scheduler ) is granted the m
other threads are put into BLOCKED state
RUNNING Thread actively working processing a task and therefore using some CPU
TIMED_WAITING
Thread in sleep, wait, join or park method and in a state of monitoring (waiti
condition) - waiting for another thread to perform an action for up to a specif
time is in this state
WAITINGThe thread is waiting by using a wait, join or park method without any maxim
indefinitely for another thread to perform a particular action is in this state
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
24/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
•
Mechanism used by Java to support thread synchronization – Mutual exclusion
• Supported via object locks
• Enables multiple threads to independently work on shared data without interfering with eac
– Cooperation
• Supported via the wait and notify methods of class Object
• Enables threads to work together towards a common goal
• Package java.util.concurrent
– Utility classes commonly useful in concurrent programming
Java monitors
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
25/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threadsJava Monitors - Example
"Thread-0" … in Object.wait()[…] at java.lang.Object.wait(Native Method)
- waiting on (a a.b.c.thread.MyClass)
at java.lang.Object.wait(Unknown Source)
at a.b.c.thread.Drop.take(ClassB.java:44)
- locked (a a.b.c.thread.ClassB)
at a.b.c.thread.exec(Test.java:15)
at java.lang.Thread.exec(Unknown Source)
"Thread-1" … waiting on condition[…]
at java.lang.Thread.sleep(Native Method)
at x.y.z.thread.run(ClassA.java:33)
at java.lang.Thread.run(Unknown Source)
"Thread-0" is waiting for anotification after it called
Object.wait()
"Thread-1" is sleeping on a
condition after it called
Thread.sleep
Conditions provide a means for one thread to suspend execution and wait until
notified by another thread that some state condition may now be true
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
26/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
• Locks are used to prevent that more than one thread accesses and workobject
• Only one thread can own a Lock object at a time
• Lock objects support a wait/notify mechanism
• A lock is called Thin with little or no contention
• A lock is promoted to Fat when contention increases
Java locks
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
27/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
•
If the lock is not held by another thread then the thread becomes the nelock until it relinquishes it
• If the lock is currently held then the thread requesting it joins the blockedcontenders for this lock
• Threads holding locks should relinquish them as soon as they stop needinavoid impairing on other threads
Thread Requesting Locks
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
28/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deconstructing java threads
•
Deadlocks describe a situation where two or more threads are blocked wother (circular dependency ). Neither block will ever end because each thfor the other
• Deadlocks can occur when multiple threads need the same locks, at the sobtain them in different order
• Restart of the server is the only option
Deadlocks
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
29/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples & Demo
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
30/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples and demo
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
31/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples and demo
[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x128 tid=15140 prio=1 alive, in native
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
…
at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
^-- Holding lock: java/io/BufferedInputStream@0x291bf2b8[thin lock]
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:75)
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1440)
…
at javax/mail/Service.connect(Service.java:105)
at javax/mail/Transport.send0(Transport.java:168)
…
Thread hanging on SocketRead from SMTP call
The thread i
receive data
mail server.
used as instr
communicatio
needs to
A lock i
BufferInpu
safegua
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
32/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples and demo
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x000000004c8a2000 nid=0
[0x000000004296c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a netscape.ldap.LDAPSearchListener)
at java.lang.Object.wait(Object.java:485)
at netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)
- locked (a netscape.ldap.LDAPSearchListener)
at netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)
…
at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3625)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:220)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
Thread hanging on LDAP response
The thread is waiting t
synchronized block and
Java monitor
Its wait will be over
receives the message
the LDAP serve
The health of the LDA
needs to be checkunderstand why it’s pe
slowly
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
33/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples and demo
"[ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)'" id=313028 idx=0x1d0 tid=4686
waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]
at java/lang/Object.wait(Object.java:474)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Idle Thread waiting for new task
The thread
notified of a ne
Several availa
are waitin
ExecuteThre
results
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
34/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Examples and demo
"[ACTIVE] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'" id=372 idx=0xfa t id=4257
prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock: weblogic/utils/KeyTable@0x16939098[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
...
- Holding lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
at weblogic/rjvm/ResponseImpl.getTxContext()Ljava/lang/Object;(ResponseImpl.java:100)[inlined]
..."[ACTIVE] ExecuteThread: '134' for queue: 'weblogic.kernel.Default (self-tuning)'" id=473785 idx=0x398 tid=32638 prio=5 alivedaemon
-- Blocked trying to get lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
...
^-- Holding lock: weblogic/utils/KeyTable@0x16939098[thin lock]
...
Deadlock
Thread #134
that thread
the same tim
lock that thre
Both thread
blocked wait
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
35/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Drinks. Food. Fun.My Oracle Support M
Tonight!
Monday, September 296:00 to 8:00 p.m.ThirstyBear Brewing Company(only ½ block from Moscone Center)
Oracle Confidential – Inte
Join us for a relaxing Happy Hour after a busy day at Oracle Open
•Take a break and unwind with your peers
•Get to know the Oracle support engineers you depend on
•Meet My Oracle Support executives and developers
•Enjoy drinks and hors d’oevres
•Admission is free with your Oracle OpenWorld badgeEvent details atwww.oracle.com
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
36/37
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
-
8/9/2019 Oracle WebLogic Troubleshooting Performance
37/37