Here are a few things that could be happening: 1. A memory leak is causing the heap usage to gradually increase over time as more and more objects are allocated but not garbage collected. 2. The application is performing some large operation that is temporarily using a lot more memory, such as loading a large data set into memory. 3. The garbage collector is not able to clean up memory fast enough to keep up with the allocation rate, possibly due to fragmentation or other GC tuning issues. More details would be needed to determine the exact cause, but in general a steadily increasing heap usage after GCs could indicate a memory leak or allocation rate that exceeds the GC's ability to free memory.

1. Do Your GC Logs
Speak To You
Using Measurements to Tune Java
Memory Management
Copyright 2011 Kodewerk Ltd. All rights reserved

2. Kodewerk
tm
Java Performance Services
About Me
• Independent consultant (www.kodewerk.com)
• performance tuning, benchmarking,
training course
• Co-author www.javaperformancetuning.com
• Speak frequently about performance
• Member of Java Champion program
• Other stuff... (google is you care to)
Copyright 2011 Kodewerk Ltd. All rights reserved

3. Kodewerk
tm
Java Performance Services
About Me
• Independent consultant (www.kodewerk.com)
• performance tuning, benchmarking,
training course
• Co-author www.javaperformancetuning.com
• Speak frequently about performance
• Member of Java Champion program
• Other stuff... (google is you care to)
Copyright 2011 Kodewerk Ltd. All rights reserved

4. Kodewerk
tm
Java Performance Services
Disclaimer
The resemblance of any
opinion, recommendation or
comment made during this
presentation to performance
tuning advice is merely
coincidental.
Copyright 2011 Kodewerk Ltd. All rights reserved

5. Why should I monitor GC?
Copyright 2011 Kodewerk Ltd. All rights reserved

6. hy
G
C d
at
Kodewerk be o
fo tha we
g
tm
in
is
re t
Java Performance Services
oc ha
no nn
g
th cu ve
in
e
tting a heap
ru
I am ge en rs th
th
ju is
an is
or no apparent d
OOM f of st lo
ha 0% PU
d
Hi!
can
W hat .
reasonc d
zen. C
g
the
o uld ay
in
stra
size
be t h he
why
GC’s
e
en
w
f re
d u r in e rea
h th
feT
gc
som
son f
gc.l
it h
g s ta
pp
10
nge
r tu p or co
que
ntinu
heap
CM
t f ro
exe
m w
ous f
r a
. Wi t
og.
eon
ull
, I
Wh
ntly
S G
cut
y
afte
oble
fr do
It s
e h
mca
t im e
ance
wan
ee es
C e
es?
se ze ou
and
elp
co up
eem
r a
and
a pr
t to
es to
nd
xec
fo
or m
pp
e of
s? r 1 lic
to
why
0s at
s v
Hello
ute
io
rows
kno
of
I’ve
perf
inut
n
see
laps
er y
s so
Copyright 2011 Kodewerk Ltd. All rights reserved fu l
w

7. Kodewerk
tm
Java Performance Services
? ? ?
?? ??
? ?
? ?
?? ? ? ?
? ??
? ? ?
? ? ?
? ? ?
? ? ??
? Copyright 2011 Kodewerk Ltd. All rights reserved

8. GC logs contain key
performance indicators
Copyright 2011 Kodewerk Ltd. All rights reserved

9. Kodewerk
tm
Java Memory Pools
Java Performance Services
Eden S0 S1 Tenured Perm
Young Old Gen
Gen
Normal Flow
- objects created in Eden
- copied into the active survivor space
- copied between survivor spaces
- copied to Old Gen when ages reaches
tenuring threshold
Copyright 2011 Kodewerk Ltd. All rights reserved

10. Kodewerk
tm
Java Performance Services
Collector Choices
uint i = 0;
if (UseSerialGC) i++;
if (UseConcMarkSweepGC || UseParNewGC) i++;
if (UseParallelGC || UseParallelOldGC) i++;
if (UseG1GC) i++;
if (i > 1) {
jio_fprintf(defaultStream::error_stream(),
"Conflicting collector combinations in option list; "
"please refer to the release notes for the combinations "
"allowedn");
status = false;
}
Copyright 2011 Kodewerk Ltd. All rights reserved

11. Kodewerk
tm
Java Memory Pools
Java Performance Services
Eden S0 S1 Tenured Perm
Young Old Gen
Gen
Abnormal Situations
- memory leaks (loitering objects)
- long pause times (heap too big, poor
parallelization)
- too frequent full GC (zombies, poor throughput)
- survivor space flooding (premature promotion)
- fragmented tenured space (compaction)
- high rates of object creation
- etc.....
Copyright 2011 Kodewerk Ltd. All rights reserved

12. Kodewerk
tm
Garbage Collection Logs
Java Performance Services
-verbose:gc
-Xloggc:gc.log
7.0 has new flags for rollover
Some many useful flags for monitoring GC
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+PrintHeapAtGC
-XX:+PrintApplicationStoppedTime
Copyright 2011 Kodewerk Ltd. All rights reserved

13. Kodewerk
tm
GC Log Fragment
Java Performance Services
21.026: [Full GC 21.026: [Tenured: 87725K->89582K(146212K), 0.3151328 secs] 117886K-
>89582K(243748K), [Perm : 30974K->30958K(30976K)], 0.3152271 secs] [Times: user=0.31 sys=0.02,
real=0.32 secs]
163.028: [GC 163.028: [DefNew
Desired survivor size 115671040 bytes, new threshold 15 (max 15)
- age 1: 736 bytes, 736 total
- age 2: 112 bytes, 848 total
- age 4: 40 bytes, 888 total
- age 6: 232 bytes, 1120 total
- age 7: 32 bytes, 1152 total
- age 11: 96 bytes, 1248 total
: 226048K->1K(451968K), 0.0018054 secs] 669064K->443016K(1129856K), 0.0018580 secs] [Times:
user=0.00 sys=0.00, real=0.00 secs]
Heap
def new generation total 451968K, used 193472K [0x10020000, 0x39620000, 0x39620000)
eden space 226048K, 85% used [0x10020000, 0x1bd0fe48, 0x1dce0000)
from space 225920K, 0% used [0x1dce0000, 0x1dce0560, 0x2b980000)
to space 225920K, 0% used [0x2b980000, 0x2b980000, 0x39620000)
tenured generation total 677888K, used 443015K [0x39620000, 0x62c20000, 0x62c20000)
the space 677888K, 65% used [0x39620000, 0x546c1d78, 0x546c1e00, 0x62c20000)
compacting perm gen total 36608K, used 36480K [0x62c20000, 0x64fe0000, 0x66c20000)
the space 36608K, 99% used [0x62c20000, 0x64fc0290, 0x64fc0400, 0x64fe0000)
No shared spaces configured.
Copyright 2011 Kodewerk Ltd. All rights reserved

14. Kodewerk
tm
Java Performance Services Full GC
21.026: [Full GC 21.026: [Tenured: 87725K->89582K(146212K), 0.3151328 secs] 117886K-
>89582K(243748K), [Perm : 30974K->30958K(30976K)], 0.3152271 secs] [Times: user=0.31 sys=0.02,
real=0.32 secs]
tenured space statistics
perm space statistics
- other stats are for the entire heap
Copyright 2011 Kodewerk Ltd. All rights reserved

15. Kodewerk
tm
Java Performance Services Full GC
21.026: [Full GC 21.026: [Tenured: 87725K->89582K(146212K), 0.3151328 secs] 117886K-
>89582K(243748K), [Perm : 30974K->30958K(30976K)], 0.3152271 secs] [Times: user=0.31 sys=0.02,
real=0.32 secs]
- occupancy before->after collection (memory pool configured size)
- reporting on tenured, perm memory pools plus total for all Java
heap spaces
Copyright 2011 Kodewerk Ltd. All rights reserved

16. Kodewerk
tm
Java Performance Services Full GC
21.026: [Full GC 21.026: [Tenured: 87725K->89582K(146212K), 0.3151328 secs] 117886K-
>89582K(243748K), [Perm : 30974K->30958K(30976K)], 0.3152271 secs] [Times: user=0.31 sys=0.02,
real=0.32 secs]
- pause time statistics for tenured and all Java heap
Copyright 2011 Kodewerk Ltd. All rights reserved

17. Kodewerk
tm
GC Log Fragment
Java Performance Services
163.028: [GC 163.028: [DefNew
Desired survivor size 115671040 bytes, new threshold 15 (max 15)
- age 1: 736 bytes, 736 total
- age 2: 112 bytes, 848 total
- age 4: 40 bytes, 888 total
- age 6: 232 bytes, 1120 total
- age 7: 32 bytes, 1152 total
- age 11: 96 bytes, 1248 total
: 226048K->1K(451968K), 0.0018054 secs] 669064K->443016K(1129856K), 0.0018580 secs] [Times:
user=0.00 sys=0.00, real=0.00 secs]
young gen statistics
statistics for all Java heap
Copyright 2011 Kodewerk Ltd. All rights reserved

18. Kodewerk
tm
GC Log Fragment
Java Performance Services
163.028: [GC 163.028: [DefNew
Desired survivor size 115671040 bytes, new threshold 15 (max 15)
- age 1: 736 bytes, 736 total
- age 2: 112 bytes, 848 total
- age 4: 40 bytes, 888 total
- age 6: 232 bytes, 1120 total
- age 7: 32 bytes, 1152 total
- age 11: 96 bytes, 1248 total
: 226048K->1K(451968K), 0.0018054 secs] 669064K->443016K(1129856K), 0.0018580 secs] [Times:
user=0.00 sys=0.00, real=0.00 secs]
surivor space sizing and tenuring thresholds
breakdown of bytes @ age
Copyright 2011 Kodewerk Ltd. All rights reserved

19. Kodewerk
tm
GC Log Fragment
Java Performance Services
Heap
def new generation total 451968K, used 193472K [0x10020000, 0x39620000, 0x39620000)
eden space 226048K, 85% used [0x10020000, 0x1bd0fe48, 0x1dce0000)
from space 225920K, 0% used [0x1dce0000, 0x1dce0560, 0x2b980000)
to space 225920K, 0% used [0x2b980000, 0x2b980000, 0x39620000)
tenured generation total 677888K, used 443015K [0x39620000, 0x62c20000, 0x62c20000)
the space 677888K, 65% used [0x39620000, 0x546c1d78, 0x546c1e00, 0x62c20000)
compacting perm gen total 36608K, used 36480K [0x62c20000, 0x64fe0000, 0x66c20000)
the space 36608K, 99% used [0x62c20000, 0x64fc0290, 0x64fc0400, 0x64fe0000)
No shared spaces configured.
- breakdown of occupancy and configured size of each memory pool
- pointer values describing each space (for debugging)
Copyright 2011 Kodewerk Ltd. All rights reserved

20. Kodewerk tm
Java Performance Services
Puzzler #1
600
Heap Usage after GC (MB)
500
400
300
200
100
1000 2000 3000 4000
Time (seconds)
Copyright 2011 Kodewerk Ltd. All rights reserved

21. Kodewerk tm
Java Performance Services
Puzzler #1
600
Heap Usage after GC (MB)
500
400
300
what happened here?
200
100
1000 2000 3000 4000
Time (seconds)
Copyright 2011 Kodewerk Ltd. All rights reserved

22. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ... ????
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
Copyright 2011 Kodewerk Ltd. All rights reserved

23. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ...
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
Copyright 2011 Kodewerk Ltd. All rights reserved

24. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ...
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
Copyright 2011 Kodewerk Ltd. All rights reserved

25. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ...
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
147.823 + 0.8074580 = 148.630 ~= 148.632
Copyright 2011 Kodewerk Ltd. All rights reserved

26. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ...
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
Copyright 2011 Kodewerk Ltd. All rights reserved

27. Kodewerk
tm
Java Performance Services
Puzzler #2
147.823: [Full GC [PSYoungGen: 7662K->0K(72896K)] [PSOldGen: 169093K->62518K(174784K)] 176755K-
>62518K(247680K) [PSPermGen: 27342K->27342K(55232K)], 0.8074580 secs]
148.632: [Full GC [PSYoungGen: 0K->0K(57856K)] [PSOldGen: 62518K->52158K(174784K)] 62518K-
>52158K(232640K) [PSPermGen: 27342K->26866K(61056K)], 0.3614330 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid18706.hprof ... ????
Heap dump file created [83874513 bytes in 3.330 secs]
160.552: [Full GC [PSYoungGen: 2624K->0K(44800K)] [PSOldGen: 86916K->76124K(174784K)] 89540K-
>76124K(219584K) [PSPermGen: 26902K->26902K(58880K)], 0.2801510 secs] [Times: user=0.28 sys=0.00,
real=0.28 secs]
Copyright 2011 Kodewerk Ltd. All rights reserved

28. Kodewerk
tm
Java Performance Services
Case Study
one web app
four different memory
configurations
Copyright 2011 Kodewerk Ltd. All rights reserved

29. Kodewerk
tm
Java Performance Services
Case Study
One Web app
Four different memory
configurations
heap size
new ratio
survivor ratio
Copyright 2011 Kodewerk Ltd. All rights reserved

30. Kodewerk
tm
Java Performance Services
1G, NR=8, SR=8
Ave: 149
Medien: 84
90%: 345
Max: 1728
Error: 0%
Copyright 2011 Kodewerk Ltd. All rights reserved

31. Kodewerk
tm
Java Performance Services
2G, NR=8, SR=8
Ave: 129
Medien: 80
90%: 287
Max: 4920
Error: 0%
Copyright 2011 Kodewerk Ltd. All rights reserved

32. Kodewerk
tm
Java Performance Services
1G, NR=1, SR=1
Ave: 82
Medien: 60
90%: 171
Max: 650
Error: 0%
Copyright 2011 Kodewerk Ltd. All rights reserved

33. Kodewerk
tm
Java Performance Services
1G, NR=8, SR=8
Copyright 2011 Kodewerk Ltd. All rights reserved

34. Kodewerk
tm
Java Performance Services
1G, NR=1, SR=1
Copyright 2011 Kodewerk Ltd. All rights reserved

35. Kodewerk
tm
Java Performance Services
2G, NR=1, SR=1
Ave: 326
Medien: 156
90%: 831
Max: 4976
Error: 0%
Copyright 2011 Kodewerk Ltd. All rights reserved

36. Questions
Copyright 2011 Kodewerk Ltd. All rights reserved