9. 일반적인 자바 서버 애플리케이션의 구조
User AP
WAS
Client
Web Server
Legacy
JVM
DBMS
Network
10. 기본지식
• 쓰레드 풀링
Thread Pool
request
Thread Pool
Thread Pool
request
response
Idle Thread
(Waiting)
Working Thread
(Handling request)
Working Thread
11. 기본 지식
• 자바 쓰레드 상태 전이
new
Sleep/done
sleep
star
t
Runnable
dea
d
sto
p
Blocked
Wait/notify
Suspend/Resu
me
Block on IO/IO
complete
<< Thread State Diagram >>
- Runnable : running
- Suspended,Locked : waiting on monitor entry / waiting on condition /MW
- Wait : Object.Wait()
12. 일반적인 자바 서버의 구조
• 대부분 비슷
WebServer
OR Browser
OR Client
APP2 Thread
Queue
Dispatcher
Request
Queue
APP1 Thread
Queue
Connection
Pool
JMS Thread
Queue
Other
Resources
Connector
Thread
Pool
Idle
Thread
Thread
Queue
Working
Thread
대부분 Queue + Thread Pool 구조
(Tomcat,Jboss,WebLogic 등)
13. 쓰레드 덤프
• Thread dump
– Snapshot of java ap (X-ray)
– We can recognize thread running tread
state by “continuous thread dump”
14. Slow down in WAS & User AP
• Getting Thread Dump
– Unix : kill –3 pid , Windows : Ctrl + Break
– Get thread dump about 3~5 times between 3~5secs
Threads
Thread dump
Working thread
Time
15. Slow down in WAS & User AP
• Thread dump
– It displays all of thread state by “THREAD STACK”
"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable
[0x9607e000..0x9607fc68]
at
at
at
at
at
at
at
at
at
at
at
at
at
at
at
at
at
java.net.SocketInputStream.read(SocketInputStream.java:85)
oracle.net.ns.Packet.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730)
oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702)
oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373)
oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427)
oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911)
oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137)
oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344)
weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51)
weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56)
Sun JVM
Thread name
Thread State
Thread id (signature)
Program stack of this thread
16. Slow down in WAS & User AP
• How to analysis thread dump
MYTHREAD_RUN(){
call methodA();
}
methodA(){
//doSomething();
call methodB();
}
methodB(){
//call methodC();
}
methodC(){
// doSomething
1
Source code
“MYTHREAD” runnable 2
at …methodA()
at …MYTHREAD_RUN()
:
2
“MYTHREAD” runnable 3
at …methodB()
at …methodA()
at …MYTHREAD_RUN()
:
3
4
for(;;){// infinite loop }
}
“MYTHREAD” runnable 1
at …MYTHREAD_RUN()
:
5
“MYTHREAD” runnable 4
at …methodC()
at …methodB()
at …methodA()
at …MYTHREAD_RUN()
:
“MYTHREAD” runnable
at …methodC()
at …methodB()
at …methodA()
at …MYTHREAD_RUN()
:
계속 이모양이
반복됨
Thread dumps
17. Slow down in WAS & User AP
• CASE 1. Lock contention
–
–
–
–
In the java application java thread wait for lock in synchronized method
Occurred in multi threaded program
Reduce frequency of synchronized method
Synchronized block must be implemented to be end as soon as
possible (※ IO? )
public void synchronized methodA(Object param)
{
//do something
while(1){}
}
< sample code >
18. Slow down in WAS & User AP
• CASE 1. Lock contention
Thread2
Thread1
Thread3
Thread4
Sychronized
MethodA
Threads
Time
Thread 2.3.4 – waiting for lock
Thread1 – That has a lock
19. Slow down in WAS & User AP
• CASE 1. Lock contention : Thread dump example
"ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055ae20
nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530]
:
at java.lang.ClassLoader.loadClass(ClassLoader.java:255)
:
at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:252)
- locked <0x329fcf50> (a java.lang.Class)
"ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055bde0 nid=24
lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530]
at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:247)
- waiting to lock <0x329fcf50> (a java.lang.Class)
:
"ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0061dc20 nid=26
lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530]
at org.apache.axis.utils.XMLUtils.releaseSAXParser(XMLUtils.java:283)
- waiting to lock <0x329fcf50> (a java.lang.Class)
at
20. Slow down in WAS & User AP
• CASE 2. Dead lock
–
–
–
–
CWC “Circular waiting condition”
Commonly it makes WAS hang up
Remove CWC by changing lock direction
Some kind of JVM detects dead lock automatically
sychronized methodA(){
call methodB();
}
sychronized methodB(){
call methodC();
}
sychronized methodC(){
call methodA();
}
< sample code >
21. Slow down in WAS & User AP
• CASE 2. Dead lock
Thread1
Thread2
Threads
Thread 1
Sychronized
MethodA
Sychronized
MethodB
Thread 2
Thread 3
Time
Thread3
Circular waiting condition
Sychronized
MethodC
22. Slow down in WAS & User AP
• CASE 2. Dead lock
FOUND A JAVA LEVEL DEADLOCK:
---------------------------"ExecuteThread: '12' for queue: 'default'":
waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub),
which is locked by "ExecuteThread: '5' for queue: 'default'"
"ExecuteThread: '5' for queue: 'default'":
waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext),
which is locked by "ExecuteThread: '12' for queue: 'default'" JAVA STACK INFORMATION FOR THREADS
LISTED ABOVE:
-----------------------------------------------Java Stack for "ExecuteThread: '12' for queue: 'default'":
==========
at weblogic.servlet.internal.ServletStubImpl.destroyServlet(ServletStubImpl.java:434)
- waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub)
at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2377)
at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207)
:
Java Stack for "ExecuteThread: '5' for queue: 'default'":
==========
at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2370)
at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207)
at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:154)
- locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub)
at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:368)
:
Deadlock auto detect in Sun
Found 1 deadlock.========================================================
JVM
23. Slow down in WAS & User AP
• CASE 2. Dead lock
"ExecuteThread-6" (TID:0x30098180, sys_thread_t:0x39658e50, state:MW, native ID:0xf10) prio=5
at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java(Compiled Code))
at weblogic.jdbc.common.internal.ConnectionEnv.cleanup(ConnectionEnv.java(Compiled
:
"ExecuteThread-8" (TID:0x30098090, sys_thread_t:0x396eb890, state:MW, native ID:0x1112)
prio=5
at oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java(Compiled Code))
at weblogic.jdbcbase.pool.Connection.commit(Connection.java(Compiled Code))
:
sys_mon_t:0x39d75b38 infl_mon_t: 0x39d6e288:
oracle.jdbc.driver.OracleConnection@310BC380/310BC388: owner "ExecuteThread-8"
(0x396eb890) 1 entry 1)
Waiting to enter:
"ExecuteThread-10" (0x3977e2d0)
"ExecuteThread-6" (0x39658e50)
sys_mon_t:0x39d75bb8 infl_mon_t: 0x39d6e2a8:
oracle.jdbc.driver.OracleStatement@33AA1BD0/33AA1BD8: owner "ExecuteThread-6"
(0x39658e50) 1 entry 2)
Waiting to enter:
"ExecuteThread-8" (0x396eb890
Deadlock trace in AIX JVM
24. Slow down in WAS & User AP
• CASE 3. Wait for IO Response
– Situation in waiting for IO response (DB,Network)
– Commonly occurred caused by DB locking or slow response
Threads
Time
Wait for IO response
25. Slow down in WAS & User AP
• CASE 3. Wait for IO Response
"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427)
at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911)
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344)
at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51)
at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56)
at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:42)
at com.XXXX 생략
at ……..
26. Slow down in WAS & User AP
• CASE 4. High CPU usage
– When monitoring CPU usage by top,glance. WAS process consumes a
lot of CPU - almost 90~100%
– Find it using tools below
•
•
•
•
Sun : prstat,pstack,thread dump
AIX : ps –mp,dbx,thread dump
HP : glance,thread dump
See a document “How to find bottleneck in J2EE application “ in
http://www.j2eestudy.co.kr
27. Slow down in WAS & User AP
• CASE 4. High CPU usage
– Example in HP UX
1.
2.
3.
4.
5.
6.
HP glance
Start “glance”
Press “G” and enter the
WAS PID
Find the TID that
consumes a lot of CPU
resource
Get a thread dump
Find the thread in Thread
dump that has a matched
LWID with this TID
And find the reason and
fix it
28. Slow down in WAS & User AP
• Summarize
–
–
–
–
Thread dump is snapshot of Java Application.
If u meet a slowdown situation, Get a thread dump.
Analysis thread dump.
And fix it..
32. JVM Memory Layout
Eden
SS1 SS2
Old
New/Young
Old
Used in Application
Total Heap Size
•
•
•
Perm
New/Young – Recently created object
Old – Long lived object
Perm – JVM classes and methods
App
binary
33. Garbage Collection
•
Garbage Collection
– Collecting unused java object
– Cleaning memory
– Minor GC
• Collection memory in New/Young generation
– Major GC (Full GC)
• Collection memory in Old generation
37. Minor GC
3rd Minor GC
OLD
Objects moved old space when they become tenured
New Object
Garbage
Lived Object
38. Major GC
•
Major Collection
– Old Generation
– Mark and compact
– Slow
• 1st – goes through the entire heap, marking unreachable objects
• 2nd – unreachable objects are compacted
39. Major GC
Eden
SS1 SS2
Mark the objects to be removed
Eden
SS1 SS2
Compact the objects to be removed
Eden
SS1 SS2
40. Server option versus Client option
•
-X:NewRatio=2 (1.3) , -Xmn128m(1.4), -XX:NewSize=<size> XX:MaxNewSize=<size>
42. Support for –XX Option
•
•
Options that begin with -X are nonstandard (not guaranteed to be
supported on all VM implementations), and are subject to change
without notice in subsequent releases of the Java 2 SDK.
Because the -XX options have specific system requirements for
correct operation and may require privileged access to system
configuration parameters, they are not recommended for casual use.
These options are also subject to change without notice.
43. Garbage Collection Model
•
New type of GC
–
–
–
–
Default Collector
Parallel GC for young generation - JDK 1.4
Concurrent GC for old generation - JDK 1.4
Incremental Low Pause Collector (Train GC)
44. Parallel GC
•
Parallel GC
–
–
–
Improve performance of GC
For young generation (Minor GC)
More than 4CPU and 256MB Physical memory
required
threads
gc
threads
time
Young
Generation
Default GC
Parallel GC
45. Parallel GC
•
Two Parallel Collectors
– Low-pause : -XX:+UseParNewGC
• Near real-time or pause dependent application
• Works with
– Mark and compact collector
– Concurrent old area collector
– Throughput : -XX:+UseParallelGC
• Enterprise or throughput oriented application
• Works only with the mark and compact collector
46. Parallel GC
•
Throughput Collector
– –XX:+UseParallelGC
– -XX:ParallelGCThreads=<desired number>
– -XX:+UseAdaptiveSizePolicy
• Adaptive resizing of the young generation
47. Parallel GC
•
Throughput Collector
– AggressiveHeap
• Enabled By-XX:+AggresiveHeap
• Inspect machine resources and attempts to set various parameters to be optimal for
long-running,memory-intensive jobs
– Useful in more than 4 CPU machine, more than 256M
– Useful in Server Application
– Do not use with –ms and –mx
• Example) HP Itanium 1.4.2 java -XX:+ServerApp -XX:+AggresiveHeap -Xmn3400m spec.jbb.JBBmain -propfile Test1
48. Concurrent GC
•
Concurrent GC
–
–
–
Reduce pause time to collect Old
Generation
For old generation (Full GC)
threads
threads
Enabled by XX:+UseConcMarkSweepGC
gc
time
Old
Generation
Default GC
Concurrent GC
49. Incremental GC
•
Incremental GC
–
–
–
–
Enabled by –XIncgc (from JDK 1.3)
Collect Old generation whenever collect young generation
Reduce pause time for collect old generation
Disadvantage
• More frequently young generation GC has occurred.
• More resource is needed
• Do not use with –XX:+UseParallelGC and –XX:+UseParNewGC
50. Incremental GC
•
Incremental GC
Minor GC
Minor GC
Minor GC
After many time of Minor GC
Full GC
Young
Generation
Old
Generation
Default GC
Old Generation is
collected in Minor
GC
Incremental GC
51. Incremental GC
•
Incremental GC
-client –XX:+PrintGCDetails -Xincgc –ms32m –mx32m
[GC [DefNew: 540K->35K(576K), 0.0053557 secs][Train: 3495K->3493K(32128K), 0.0043531 secs] 4036K>3529K(32704K), 0.0099856 secs]
[GC [DefNew: 547K->64K(576K), 0.0048216 secs][Train: 3529K->3540K(32128K), 0.0058683 secs] 4041K>3604K(32704K), 0.0109779 secs]
[GC [DefNew: 575K->64K(576K), 0.0164904 secs] 4116K->3670K(32704K), 0.0169019 secs]
[GC [DefNew: 576K->64K(576K), 0.0057541 secs][Train: 3671K->3651K(32128K), 0.0051286 secs] 4182K>3715K(32704K), 0.0113042 secs]
[GC [DefNew: 575K->56K(576K), 0.0114559 secs] 4227K->3745K(32704K), 0.0191390 secs]
[Full GC [Train MSC: 3689K->3280K(32128K), 0.0909523 secs] 4038K->3378K(32704K), 0.0910213 secs]
[GC [DefNew: 502K->64K(576K), 0.0173220 secs][Train: 3329K->3329K(32128K), 0.0066279 secs] 3782K>3393K(32704K), 0.0325125 secs
Minor GC
Young Generation GC
Old Generation GC in Minor GC Time
Sun JVM 1.4.1 in Windows OS
Full GC
58. Garbage Collection Tuning
•
GC Tuning
– Find Most Important factor
• Low pause? Or High performance?
• Select appropriate GC model (New Model has risk!!)
– Select “server” or “client”
– Find appropriate Heap size by reviewing GC log
– Find ratio of young and old generation
59. Garbage Collection Tuning
•
GC Tuning
– Full GC Most important factor in GC tuning
•
•
•
•
How frequently ? How long ?
Short and Frequently decrease old space
Long and Sometimes increase old space
Short and Sometimes decrease throughput by Load balancing
– Fix Heap size
• Set “ms” and “mx” as same
• Remove shrinking and growing overhead
– Don’t
• Don’t make heap size bigger than physical memory (SWAP)
• Don’t make new generation bigger than half the heap
64. Example
Before Tuned
Old Area
PEAK TIME
52000~56000 sec
9시~ 1시간 가량
금요일 업무시간
2004-01-10
오후 6시 전후
2004-01-08
2004-01-09
오후 7:14 오전 8시 전후
2004-01-10
오전 10시 전후
2004-01-09
오후 7시 전후