troubleshooting with weblogic thread dumps 1

一、weblogic thread dump一般包含如下几部分
1.The Beginning of The Thread Dump
2.Stack Trace for Main Application Thread
3.JVM Internal Threads
4.Other Java Application Threads
5.Lock Chains

"Main Thread" id=1 idx=0x2 tid=48652 prio=5 alive, in native, waiting
-- Waiting for notification on: util/repro/Thread1@0x01226528[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Thread.join(Thread.java:1095)
^-- Lock released while waiting: util/repro/Thread1@0x01226528[fat lock]
at java/lang/Thread.join(Thread.java:1148)
at util/repro/DeadLockExample.main(DeadLockExample.java:23)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
线程名称：Main Thread
id=1: jvm内部唯一标识
idx=0x2：idx is the thread index in the threads array
tid=48652：对应系统线程ID，top命令（top -H -p pid）查看的线程pid对应
优先级: prio=5
线程状态： alive, in native, waiting
第二行到最后一行为当前的执行堆栈，一般从后往前看


二、线程的状态包含如下几种类型 Thread Status in Thread Dumps
1.Life States
2.Run States
3.Special States

Life States

State	Description
alive	This is a normal, running thread. Virtually all threads in the thread dump will be alive.
not started	The thread has been requested to start running by java.lang.Thread.start(), but the actual OS process has not yet started, or executed far enough to pass control to the JRockit JVM. It is extremely unlikely to see this value. A java.lang.Thread object that is created, but not has had start() executed, will not show up in the thread dump.
terminated	This thread has finished its run() method and has also notified any threads joining on it, but it is still kept in the JVM internal thread structure for running threads. It is extremely unlikely to see this value. A thread that has been terminated for a time longer than a few milliseconds will not show up in the thread dump.

Run States

State	Description
blocked	This thread has tried to enter a synchronized block, but the lock was taken by another thread. This thread is blocked until the lock gets released.
blocked (on thin lock)	This is the same state as blocked, but with the additional information that the lock in question is a thin lock.
waiting	This thread has called Object.wait() on an object. The thread will remain there until some other thread sends a notification on that object.
sleeping	This thread has called java.lang.Thread.sleep().
parked	This thread has called java.util.concurrent.locks.LockSupport.park().
suspended	The thread’s execution has been suspended by java.lang.Thread.suspend() or a JVMTI/JVMPI agent call

Special States

State	Description
interrupted	The user called java.lang.Thread.interrupt() on this thread.
daemon	This is either JVM internal thread or a thread that was marked as a daemon thread by java.lang.Thread.setDaemon().
in native	This thread is executing native code: either user-supplied JNI code or JVM internal code.
in suspend critical mode	This thread is executing JVM internal code and has marked itself as suspend critical. Garbage collection is stopped for a specified time period.
native_blocked	This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread.
native_waiting	This thread is executing JVM internal code and is waiting for notification from another thread about a JVM internal lock.

三、Troubleshooting with Thread Dumps

weblogic控制台如下两个表可关注
第一个表显示当前线程池的整体情况
第二个表显示每个当前每个线程的状态



第一个表的每一列说明如下：

Name	Description
Active Execute Threads	The number of active execute threads in the pool.
Execute Thread Total Count	The total number of threads in the pool.
Execute Thread Idle Count	The number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives
Queue Length	The number of pending requests in the priority queue. This is the total of internal system requests and user requests.
Pending User Request Count	The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
Completed Request Count	The number of completed requests in the priority queue.
Hogging Thread Count	returns the threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
Standby Thread Count	Returns the number of threads in the standby pool. Threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.

第二个表的每一列说明如下：

Name	Description
Name	The name of the execute thread.
Total Requests	The number of requests which have been processed by this queue.
Current Request	The request this thread is currently processing.
Transaction	The transaction on behalf of which the execute thread is currently working.
User	The user on behalf of which the execute thread is currently working.
Idle	True if the execute thread has no work assigned to it.
Stuck	True if the execute thread is stuck working on a request for more than the configured stuck thread maximum time.
Hogger	True if the execute thread is being hogged by a request for much more than the normal execution time, as automatically observed by the scheduler.
Standby	True if the execute thread is not a part of the active thread pool.
Work Manager	The name of the current work manager.
Application	The name only, without the version number, of the application using the context class loader of the thread.
Module	The module using the context class loader of the thread.

关于hogging线程和stuck线程的说明
What is Hogging Thread?
 A hogging thread is a thread which is taking more than usual time to complete the request and can be declared as Stuck .

How Weblogic determine a thread to declare as hogging?
A thread declared as Stuck if it runs over 600 secs (default configuration which you can increase or decrease from admin console).
There is an internal WebLogic polar which runs every 2 secs (by default 2 secs and can be alter)
It checks for the number of requests completed in last two minutes
Then it check how much times each took to complete
Then it takes the average time of all completed request (completed in last 2 sec)
Then multiply average time with 7, and the value came consider as “usual time to complete the request”
Now weblogic check each current executed thread in last 2 secs and compare with above average time, if for any of the thread it’s above this value then that thread will declare as Hogged thread.
For example –
At a particular moment, total number of completed requests in last two seconds – 4
Total time took by all 4 requests – 16 secs
Req1 took – 5 secs, Req2 took – 3 secs, Req3 took – 7 secs, Req4 took – 1 sec
Average time = 16/4 = 4 secs
7*4 = 28 secs
Now weblogic check all executed threads to see which taking more than 28 secs, if any then that thread(s) declared as Hogged Thread.
Only the thing you can change with respect to hogging threads configuration is Polar time (Stuck Thread Timer Interval parameter)which is 2 secs by default. You can change this polar value to some different value like 4 secs if you want polar to run in every 4 secs instead of 2 secs.

What is Stuck Thread?
A Stuck Thread is a thread which is processing a request for more than maximum time that you
configured in admin console.

How to deal with Stuck Thread?
Take multiple thread dumps immediately.
Review thread dumps or from console (managed server > monitoring > threads).
See how many threads got stuck?
If the stuck thread count is increasing or constant?
If constant then if got stuck on same area (application code etc ) or at different places ?
If getting increase then there would be some serious problem and you have to do a quick health check of you application server, database and other integrated technologies wherever your application reaching like ldap server for authentication, some other API’s or web services etc, and in parallel review thread dumps for stuck threads and share same with your developers to analyze quickly.
If you have one, two or few constant stuck  threads and it’s not increasing then you can monitor it for some more time to check if they get clear or not, if not then to clear them you have only option to restart your managed server(s), and its better to restart and clear them before they make further any impact.


未完待续。。。

参考资料：
https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/using_threaddumps.html#wp1094512
https://dzone.com/articles/how-analyze-java-thread-dumps
http://docs.oracle.com/cd/E21764_01/apirefs.1111/e13952/pagehelp/Corecoreserverservermonitorthreadstitle.html
http://rahulranyal.blogspot.jp/2015/06/weblogic-server-stuck-thread-hogging.html