3. Performance tuning for Java
applications
Andrew Prentice
Atlassian
2

4. Topics
• How to make your apps run faster
• Tuning ideas
3

5. A note
• X vs Y will always depend on context
• Test with your application!
4

6. You are here
• How I did the testing
• Hardware & software
• Must have improvements
• Jave, Hardware, Virtualization and Garbage Collection
• Worth doing
• Gigabit vs 100 mbit, Heap size tuning, use an SSD 5

7. The contenders
• Dell PE 860 - “WALL-E”
• Xeon X3220 2.4 Ghz Quad core
• Quad Core, 2 x 4Mb L3
• 8 Gb DDR2 RAM
• 2 x 7.2K SAS Disks
• 1 x Intel X25-E 32 GB SSD
6

8. The contenders
• Dell PE 2950 - “Johnny 5”
• 2 x Xeon E5 405 2.0 Ghz
• Quad Core, 2 x 6M L3
• 32 Gb RAM
• 2 x 72 Gb 15 K Drives
• ~$4000
7

9. The contenders
• Dell PE R610 - “EVE”
• 2 x Xeon E5520 2.2Ghz
• Quad Core, 8M “SmartCache”
• 32 Gb RAM
• 2 x 146G 15K drives
• ~$4000
8

10. The Workload
• JIRA 4, MySQL 5, RHEL 5
• Java 1.6, 3G heap, ParNew, ParOld GC
• ~70,000 Issues, ~80 projects
• ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
• Traffic modeled on http://jira.atlassian.com with higher
writes
9

11. About the graphs
• Most show Average Response Time in milliseconds
• Includes time to generate AND deliver the response
• Does not include browser time
• Arrows show if lower or higher is better
10

12. You are here
• How I did the testing
• Hardware & software
• Must have improvements
• Jave, Hardware, Virtualization and Garbage Collection
• Worth doing
• Gigabit vs 100 mbit, Heap size tuning, use an SSD 11

13. Must. Have.
12

14. Upgrade Java 1.5 to 1.6
• Advances in complier/VM technology
• Produces better runtime code
• Able to use modern instructions
• Able to use modern hardware
13

15. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5 Java 1.6.0
1500
1125
750
375
6.3X
0
Average Response Time (ms)
14

16. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5.0 Java 1.6.0
380
285
190
95
26%
0
Average CPU (%)
15

17. Upgrade Java to 1.6
• HotSpot VM updates in 1.6
• Advancing technology
• Compressed OOPS on 64bit
• Escape Analysis
• NUMA
• http://java.sun.com/performance/reference/
whitepapers/6_performance.html 16

18. Java 1.6 vs 1.6 (Eve)
Java 1.6.0_7 Java 1.6.0_16
148
111
74
37
7%
0
Average Response Time (ms)
17

19. Hardware Upgrades
• Hyper-Threading
• QuickPath Interconnect
• Nehalem (Eve) have on die DDR3 Memory Controllers
• Memory throuput improvements - good for Java!
Technology Speed
DDR2-800 (dual channel) 6.4 GB/sec
DDR3-1333 (dual channel) 21.3 GB/sec
18

20. Hyper Threading
Thread 1
CPU
Thread 2
Thread 1 runs until it hits a cache miss.
Thread 2 runs on the same CPU while T1 is awaiting data
19

21. Hardware Upgrades
2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)
2000
1500
1000
500
8-12X
0
Average Response Time (ms)
20

22. Improved GC Throughput
2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)
30000
22500
15000
2-3X
7500
0
GC Throughput (MB/sec)
21

23. Garbage Collection
• Java’s Virtual Machine manages memory
• New objects are continually created during runtime
• GC is the process of collecting dead objects to reclaim
memory
22

24. Tune Garbage Collection
• Getting your GC tuning wrong is bad
• How bad?
• Are the defaults sane?
23

25. Tune Garbage Collection
CMS ParNew ParOld
4000
3000
2000
1000 2X
0
Wall-E (ms) Johnny 5 (ms) Eve (ms)
Average Response Time
24

26. Tune Garbage Collection
CMS ParNew ParOld
500
375
250
125
2X
0
Johnny 5 (ms) Eve (ms)
Average Response Time
25

27. Tune Garbage Collection
• Defaults are sane
• Parallel New + Parallel Old is fastest
• CMS prioritizes low pauses over CPU usage
• CMS does not compact - can lead to heap
fragmentation
26

28. Virtualisation
• Does being in a VM hurt?
• What if the risks are managed?
27

29. Virtualisation
Real VMWare ESX 3.5 VMWare ESX 4i
170.0
127.5
85.0
42.5 43%
0
Average Response Time (ms)
28

30. Virtualisation
• On average ~30% slower
• Under extreme resource starvation, up to 100% slower
• Avoid virtualization for high traffic instances
• See the Atlassian docs for recommendations
• http://confluence.atlassian.com/display/DOC/Running
+Confluence+in+a+Virtualised+Environment
29
• http://confluence.atlassian.com/display/JIRA/Running+JIRA+in

31. You are here
• How I did the testing
• Hardware & software
• Must have improvements
• Java, Hardware, Virtualization and Garbage Collection
• Worth doing
• Gigabit vs 100 mbit, Heap size tuning, use an SSD 30

32. Maybe.
31

33. Gigabit Network vs 100Mbit
• Gigabit is faster, sure
• But 100Mb is OK if you’re not doing big transfers,
RIGHT?!
32

34. Use Gigabit
100mb/sec 1000mb/sec
1900
1425
950
475 10%
0
Average Response Time (ms)
33

35. Use Gigabit
• Lower RTT on Gigabit
• If you have enough CPU power, keep the DB on
localhost
• Check your development environment
34

36. Use Gigabit
100Mbit 1000Mbit
200
150
100
50
26%
0
ReIndex Time (seconds)
35

37. Heap size tuning
• Memory starvation is bad, but can we “drown” Java
with too much memory?
36

38. Heap size - Wall-E
1.5G 3G 6G
2000
1500
1000
500
23%
0
Average Response Time
37

39. Heap size - Johnny 5
1.5G 3G 6G 15G
290.0
217.5
145.0
72.5 49%
0
Average Response Time
38

40. Heap size - Eve
1.5G 3G 6G 15G
170.0
127.5
85.0
42.5 28%
0
Average Response Time
39

41. Heap size tuning
• Heap too small - bad
• Heap too big - not bad!
• Bigger heaps give the JVM more room to work
• Diminishing returns from really big heaps
40

42. Buy an SSD
• SSDs are FAST!
• SSDs are EXPENSIVE!
• SSDs are all the rage!
• DBAs are finally happy (almost)
41

43. Buy an SSD
SATA SSD
800
600
400
200 33%
0
Browse Issue Create Issue
42

44. Buy an SSD
• Fast writes
• 0.1ms Avg Service Time
• Most improvement comes from SSD on the DB
• Writing to the Lucene index means re-opening
IndexReaders
• Much faster on SSD
43
• Lucene Search Term Cache fits in OS buffers

45. Remember!
• Upgrade!
• Java to 1.6
• Your server
• Use Parallel Garbage Collection with a big heap
• Remove virtualization if you can
• Use 1000mbit network
44

46. ...Part Two:
Environment and
Joanna Thurmann
JIRA Systems Administrator
2

47. Don’t let this happen to you!
3

48. “Step1 - We admitted we were powerless -
our JIRA had become unmanageable.”
JIRAcholics Anonymous
4

49. Contents
• Performance Considerations
• Hardware & Software Upgrade
• JIRA and Plugin Upgrades
• Structure for Scale
5

50. Our JIRA
6

51. Our JIRA
• 210,000 issues
6

52. Our JIRA
• 210,000 issues
• 142 projects
6

53. Our JIRA
• 210,000 issues
• 142 projects
• 66 workflows
6

54. Our JIRA
• 210,000 issues
• 142 projects
• 66 workflows
• 320 custom fields
6

55. Our JIRA
• 210,000 issues
• 142 projects
• 66 workflows
• 320 custom fields
• 6,000 user directory (~400 concurrent)
6

56. Our JIRA
• 210,000 issues
• 142 projects
• 66 workflows
• 320 custom fields
• 6,000 user directory (~400 concurrent)
• 24x7 usage 6

57. Considerations for
Performance and Usability
• Total Issue Count
• User Concurrency
• # and Complexity of Configurations
• Integrations
• Plugins
7

58. In General…
• JIRA uses the disk much more than db (due to Lucene
indexing)
• JIRA needs considerable memory
8

59. Our Upgrade Summary
Before After
 HP ProLiant DL580 G4  HP ProLiant DL380 G7
 4 x Dual-Core processors - Intel Xeon  2 x 6-CORE processors - Intel Xeon X5680
7140M 3.4GHz 3.33GHz
 RAID5 array  5 x HP 300GB 10K HDD  RAID10 array  11 x HP 300GB 10K HDD
 512 MB cache (25%R-75%W)  1GB cache (50%R-50%W)
 3.5 GB for JIRA (8 GB total)  6 GB for JIRA (32 GB total)
 Windows Server 2003 Std Edition 64-bit  Windows Server 2008 R2 Std Edition 64-bit
 SQL Server 2005 Std Edition  SQL Server 2008 R2 Std Edition
9

60. Our Upgrade – Disk and CPU
• Chose Raid10 over Raid5 for better performance
• Upgraded Controller from 512 MB cache to 1GB cache
• Chose 12 CPUs for concurrency
10

61. Our Upgrade – Memory
11

62. Our Upgrade – Memory
• 3.5GB  6GB
11

63. Our Upgrade – Memory
• 3.5GB  6GB
• Full garbage collections times lower and less frequent
11

64. Our Upgrade – Memory
• 3.5GB  6GB
• Full garbage collections times lower and less frequent
• Free up more memory
11

65. Our Upgrade – Memory
• 3.5GB  6GB
• Full garbage collections times lower and less frequent
• Free up more memory
• Eliminate out-of-memory errors
11

66. Our Upgrade - Memory
Before (3.5 GB JIRA, 8 GB Total) After (6 GB JIRA, 32 GB Total)
Full GC: 11-50s, 1x/hour Full GC: 3-7s, <5x/day
2.5 – 3.25 GB avg. usage 3-5 GB avg. usage
20-30% free 50-75% free
Out-of-memory: 1x/month Out-of-memory: 0
12

67. Our Upgrade - Memory
Before (3.5 GB JIRA, 8 GB Total) After (6 GB JIRA, 32 GB Total)
Full GC: 11-50s, 1x/hour Full GC: 3-7s, <5x/day
2.5 – 3.25 GB avg. usage 3-5 GB avg. usage
20-30% free 50-75% free
Out-of-memory: 1x/month Out-of-memory: 0
12

68. Our Upgrade - Software
• SQL Server 2008 R2
• OK to run JIRA and DB on the same server
• Allocate sufficient memory to the db
• Use the latest JDBC driver & JDK
13

69. What’s the bottom line?
14

70. What’s the bottom line?
14

71. Results – backend
Then Now Percentage
faster
Total Reindex 60 mins 25 mins 59%
XML Backup 30 mins 4 ½ mins 85%
SQL Server Backup 7 mins 2 mins 71%
Metrics service 20 mins 6 mins 70%
Index opNmizaNon 7+ mins 25 secs 94%
Index excepNons 3‐4 per day 0
15

72. Results – front end
Then Now Percentage
faster
Search all issues 3 secs 2 secs 34%
Create / Edit issue 4 secs 2 secs 50%
16

73. Plugin challenges we faced
• Memory leak
• End of support of *key* workflow plugin
• Odd behavior in JIRA 4.1.1 upgrade
• Performance issue with plugin
17

74. Plugin Install & Upgrades
18

75. Plugin Install & Upgrades
• Scrutinize Plugins
18

76. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
18

77. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
18

78. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Will it be supported?
18

79. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Will it be supported?
• How widely used is it?
18

80. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Will it be supported?
• How widely used is it?
• Does it perform well?
18

81. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Will it be supported?
• How widely used is it?
• Does it perform well?
• Do you really need it?
18

82. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Will it be supported?
• How widely used is it?
• Does it perform well?
• Do you really need it?
• Does it introduce other bugs?
18

83. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Get the latest release • Will it be supported?
• How widely used is it?
• Does it perform well?
• Do you really need it?
• Does it introduce other bugs?
18

84. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Get the latest release • Will it be supported?
• How widely used is it?
• Test in QA environment • Does it perform well?
• Do you really need it?
• Does it introduce other bugs?
18

85. Plugin Install & Upgrades
• Scrutinize Plugins Ask yourself…
• Is the plugin reputable?
• Get the latest release • Will it be supported?
• How widely used is it?
• Test in QA environment • Does it perform well?
• Do you really need it?
• Test! Test! Test! • Does it introduce other bugs?
18

86. Why Upgrade JIRA Regularly?
19

87. Why Upgrade JIRA Regularly?
• Performance improvements & key fixes in each release for
BOTH end-users AND admins
19

88. Why Upgrade JIRA Regularly?
• Performance improvements & key fixes in each release for
BOTH end-users AND admins
• Example: in-place DB upgrades in 4.3
19

89. My Weekend Upgrade
20

90. My Weekend Upgrade
Before
20

91. My Weekend Upgrade
Before
20

92. My Weekend Upgrade
Before After
20

93. My Weekend Upgrade
Before After
20

94. Why else?
21

95. Why else?
• Fewer internal support calls
21

96. Why else?
• Fewer internal support calls
• Less time to resolve them
21

97. Why else?
• Fewer internal support calls
• Less time to resolve them
• Easier & faster for Atlassian to support you
21

98. Why else?
• Fewer internal support calls
• Less time to resolve them
• Easier & faster for Atlassian to support you
• Early bird gets the worm
21

99. Our last JIRA upgrade rocked!
22

100. Our last JIRA upgrade rocked!
• Resolved 4 bugs found internally
22

101. Our last JIRA upgrade rocked!
• Resolved 4 bugs found internally
• Delivered 5 must-have features
22

102. Our last JIRA upgrade rocked!
• Resolved 4 bugs found internally
• Delivered 5 must-have features
• Resulted in ~25% fewer support calls
22

103. Our last JIRA upgrade rocked!
• Resolved 4 bugs found internally
• Delivered 5 must-have features
• Resulted in ~25% fewer support calls
• Saved ~3hrs/week with JQL instead of
SQL
22

104. Structure – For Startup JIRAs
23

105. Structure – For Startup JIRAs
• Configurations  Less is more since JIRA caches them
23

106. Structure – For Startup JIRAs
• Configurations  Less is more since JIRA caches them
• Use fewer issue types, resolutions, statuses, custom fields
23

107. Structure – For Startup JIRAs
• Configurations  Less is more since JIRA caches them
• Use fewer issue types, resolutions, statuses, custom fields
• Use custom field contexts vs. new fields
23

108. Structure – For Startup JIRAs
• Configurations  Less is more since JIRA caches them
• Use fewer issue types, resolutions, statuses, custom fields
• Use custom field contexts vs. new fields
• Use a simple workflow and apply to many projects
23

109. Structure – Project Roles
24

110. Structure – Project Roles
• Use Project Roles vs individual names & groups in
permission schemes
24

111. Structure – Project Roles
• Use Project Roles vs individual names & groups in
permission schemes
Before
Permission Scheme for Project 1
Administer Projects Group (“”Project 1 Admins”)
Edit Issues Group(“Project 1 Team”)
24

112. Structure – Project Roles
• Use Project Roles vs individual names & groups in
permission schemes
Before After
Global Permission Scheme
Permission Scheme for Project 1
Administer Projects Project Role (Administrators)
Administer Projects Group (“”Project 1 Admins”)
Edit Issues Project Role (Developers)
Edit Issues Group(“Project 1 Team”)
24

113. Structure – Project Roles
• Use Project Roles vs individual names & groups in
permission schemes
Before After
Global Permission Scheme
Permission Scheme for Project 1
Administer Projects Project Role (Administrators)
Administer Projects Group (“”Project 1 Admins”)
Edit Issues Project Role (Developers)
Edit Issues Group(“Project 1 Team”)
Project Roles for Project 1
Administrators “Project 1 Admins”
Developers “Project 1 Team”
24

114. Structure – Project Roles
• Use Project Roles vs individual names & groups in
permission schemes
Before After
Global Permission Scheme
Permission Scheme for Project 1
Administer Projects Project Role (Administrators)
Administer Projects Group (“”Project 1 Admins”)
Edit Issues Project Role (Developers)
Edit Issues Group(“Project 1 Team”)
Project Roles for Project 1
Administrators “Project 1 Admins”
Developers “Project 1 Team”
24

115. Structure – For Epic JIRAs
25

116. Structure – For Epic JIRAs
• Consolidate any and all schemes
25

117. Structure – For Epic JIRAs
• Consolidate any and all schemes
• Use Scheme Tools: e.g. Scheme Comparison Tool,
Group to Project Role Mapping Tool
25

118. Structure – For Epic JIRAs
• Consolidate any and all schemes
• Use Scheme Tools: e.g. Scheme Comparison Tool,
Group to Project Role Mapping Tool
• ‘Archive’ projects - hide them from view of users, so that
they don’t clutter the menus. Create ‘Archived’ config
schemes
25

119. We Declutter
26

120. We Declutter
• Created project naming conventions
26

121. We Declutter
• Created project naming conventions
• Eliminated & consolidated custom fields (25 out of 345)
26

122. We Declutter
• Created project naming conventions
• Eliminated & consolidated custom fields (25 out of 345)
• Took custom fields out of global context
26

123. We Declutter
• Created project naming conventions
• Eliminated & consolidated custom fields (25 out of 345)
• Took custom fields out of global context
• Eliminated issue types, status, resolution choices (15-30)
26

124. We Tidy
27

125. We Tidy
• Super User (SU) plugin
27

126. We Tidy
• Super User (SU) plugin
• Easier to troubleshoot user issues
27

127. We Tidy
• Super User (SU) plugin
• Easier to troubleshoot user issues
• Helps us “terminate” JIRA accounts
•Delete their dashboards
•Delete their personal filters
27

128. We Tidy
• Super User (SU) plugin
• Easier to troubleshoot user issues
• Helps us “terminate” JIRA accounts
•Delete their dashboards
•Delete their personal filters
•Delete subscriptions (the “silent killer”)
27

129. We Train
28

130. We Train
• Limit General Permissions & train the users on how/when to use
28

131. We Train
• Limit General Permissions & train the users on how/when to use
• Bulk Change
28

132. We Train
• Limit General Permissions & train the users on how/when to use
• Bulk Change
• Create Shared Objects (filters + dashboards)
28

133. We Train
• Limit General Permissions & train the users on how/when to use
• Bulk Change
• Create Shared Objects (filters + dashboards)
• Manage Group Filter Subscriptions
28

134. We Train
• Limit General Permissions & train the users on how/when to use
• Bulk Change
• Create Shared Objects (filters + dashboards)
• Manage Group Filter Subscriptions
• Train project administrators / super-users
28

135. We Do More
29

136. We Do More
• Use db-specific vs. XML-backup
29

137. We Do More
• Use db-specific vs. XML-backup
• Schedule regular maintenance
29

138. We Do More
• Use db-specific vs. XML-backup
• Schedule regular maintenance
• Regularly check logs for errors
29

139. We Do More
• Use db-specific vs. XML-backup
• Schedule regular maintenance
• Regularly check logs for errors
• Read the JIRA performance doc
29

140. 30

141. We
Keep
Going
30

142. “Step12 - we promised to help other
JIRAcholics, and to always practice what we
preach
JIRAcholics Anonymous
31

143. A word from my sponsors….

144. A word from my sponsors….

145. A word from my sponsors….
Polycom, Inc. (Nasdaq: PLCM) is a global leader in unified communications
solutions with industry-leading telepresence, video, voice, and infrastructure
solutions built on open standards. Polycom powers smarter conversations,
transforming lives and businesses worldwide. Please visit
www.polycom.com for more information.

146. blogs.atlassian.com/developer
Flickr Attributions:
Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/
Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/

147. blogs.atlassian.com/developer
Flickr Attributions:
Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/
Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/