Troubleshooting Live Java Web Applications

1. Troubleshooting Live Java Web Applications By Ashley Puls Sunday, November 10, 13

2. Sunday, November 10, 13

3. Troubleshooting Locate and eliminate sources of trouble Determine and settle problems Solve problems quickly using a systematic approach A logical process of elimination to identify the true source of the problem Sunday, November 10, 13

4. How to Become a Good Troubleshooter Practice makes perfect Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg Sunday, November 10, 13

5. How to Become a Good Troubleshooter Practice makes perfect Have a good set of tools Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg, http://blogs.telerik.com/images/default-source/jeff-fritz/toolbox.jpg?sfvrsn=0 Sunday, November 10, 13

6. Outline Examine a web application with multiple performance problems Troubleshoot the problems using several tools Discuss the pros and cons of each tool Sunday, November 10, 13

7. What is the Application? Sunday, November 10, 13

8. What is the Application? Dear Developer, I had some engineers develop “Running Central” to search for runners, races, and training teams. Can you please keep the site running? Sincerely, Max Siegel CEO of USA Track and Field Image From: http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx Sunday, November 10, 13

9. What is the Application? Tomcat Application Server Client racerunners.war jsp Client Controller Manager Entities Servlet hibernate Client MySQL Database Sunday, November 10, 13

10. Should we Jump into Troubleshooting? Image From: http://toosoxy.files.wordpress.com/2011/04/cliff.jpg Sunday, November 10, 13

11. What is Trouble? Image From: http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg Sunday, November 10, 13

12. What is Trouble? Site Inaccessible Sunday, November 10, 13

13. What is Trouble? Site Inaccessible Exceptions In Logs Sunday, November 10, 13

14. What is Trouble? Site Inaccessible Loading Slowly Exceptions In Logs Sunday, November 10, 13

15. What is Normal? Images From: http://espn.go.com/mlb/statistics, http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera Sunday, November 10, 13

16. What Are The Stats? Application Response Time Memory Consumed time per request heap and PermGen Sunday, November 10, 13 Thread State active vs blocked Thread Pool State active vs max thread count Database Response Time time per request

17. How To Gather The Stats Logs core JDK tools JMX Metrics Application Performance Monitoring Tools (APM) Sunday, November 10, 13 Proﬁlers Framework/App Server Monitoring Tools

18. Basic Information Tools Overview Installation Cost Logs New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, November 10, 13 Basic information about each tool Where to Find It For Production

19. Data In Tool Tools Logs New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, November 10, 13 App Response Time Database Response Time Thread State Thread Pool State Memory

20. Lets Talk to Some of Our Users Sunday, November 10, 13

21. Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg Sunday, November 10, 13

22. Usain Bolt : Six Time Olympic Champion Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg Sunday, November 10, 13

23. I was looking for a 5K race on your site, but it was loading like I was on a 56k Image From: http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg Sunday, November 10, 13

24. Lets Investigate! Sunday, November 10, 13

25. New Relic Application Performance Monitoring (APM) tool that monitors web apps, mobile apps, servers, and databases Sign up and download agent from the New Relic Website Sunday, November 10, 13

26. Basic Information Tools Overview Installation Cost Where to Find It APM solution restart app with agent basic free, advanced cost www.newrelic.com Logs New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, November 10, 13 For Production

28. Application Response Time Sunday, November 10, 13

32. YourKit Proﬁler that provides in depth CPU, thread, and memory proﬁling Get license key and download from YourKit Website Sunday, November 10, 13

33. Basic Information Tools Overview Installation Cost Where to Find It New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler Logs JVisualVm JConsole JMap/JHat JStack Sunday, November 10, 13 restarts or attaches all features at cost www.yourkit.com For Production

37. CPU Proﬁler Sunday, November 10, 13

46. Heap Memory Sunday, November 10, 13

47. Heap Memory Non-Heap Memory Sunday, November 10, 13

48. Heap Memory Non-Heap Memory Sunday, November 10, 13

49. JMap Prints the memory map of a Java process Commands jmap -heap JAVA_PID > /tmp/jmap.out jmap -dump:ﬁle=/tmp/heap_dump.map JAVA_PID Sunday, November 10, 13

50. Basic Information Tools Overview Installation Cost Where to Find It APM solution restart app with agent basic free, advanced cost www.newrelic.com prints memory map of Java process attaches to running process free included with JDK Logs New Relic YourKit JVisualVm JConsole JMap/JHat JStack Sunday, November 10, 13 For Production

51. In a Terminal Sunday, November 10, 13

53. The File Sunday, November 10, 13

54. The File Heap Memory Sunday, November 10, 13

55. The File Heap Memory Non-Heap Memory Sunday, November 10, 13

56. The File Heap Memory Non-Heap Memory Sunday, November 10, 13

72. YourKit Proﬁler that provides in depth CPU, thread, and memory proﬁling Get license key and download from YourKit Website Sunday, November 10, 13

75. Heap Memory Sunday, November 10, 13 Non-Heap Memory

76. Heap Memory Sunday, November 10, 13 Non-Heap Memory

81. /** * Service that records stats on the cities and states queried by users when looking * for races. */ public class RaceStatsServiceImpl implements RaceStatsService { /** Holds the private final /** Holds the private final cities queried by users. */ Map<String, Integer> citiesQueried; states queried by users. */ Map<String, Integer> stateQueried; public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); } Sunday, November 10, 13

82. /** * Service that records stats on the cities and states queried by users when looking * for races. */ public class RaceStatsServiceImpl implements RaceStatsService { /** Holds the private final /** Holds the private final cities queried by users. */ Map<String, Integer> citiesQueried; states queried by users. */ Map<String, Integer> stateQueried; Two ConcurrentHashMaps public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); } Sunday, November 10, 13

83. /** * Service that records stats on the cities and states queried by users when looking * for races. */ public class RaceStatsServiceImpl implements RaceStatsService { public void addCity(String city); public void addState(String state); public void String getMostQueriedForCity(); public void String getMostQueriedForState(); Sunday, November 10, 13

84. /** * Service that records stats on the cities and states queried by users when looking * for races. */ public class RaceStatsServiceImpl implements RaceStatsService { public void addCity(String city); public void addState(String state); public void String getMostQueriedForCity(); public void String getMostQueriedForState(); Sunday, November 10, 13 None of these methods remove from the concurrent hash maps

85. Key Takeaway Clean up memory in long running objects Sunday, November 10, 13

90. Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg Sunday, November 10, 13

91. Paula Radcliffe Female Marathon World Record Holder Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg Sunday, November 10, 13

92. It was taking too long to ﬁnd my fellow runners on your site Image From: http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg Sunday, November 10, 13

94. JVisualVM Visual tool which provides heap dumps, memory leak information, garbage collection monitoring, memory proﬁling, and CPU proﬁling Attaches to local and remote running processes Sunday, November 10, 13

95. Basic Information Tools Overview Installation Cost Where to Find It New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler restart or attach feature dependent all features at cost www.yourkit.com JVisualVm proﬁler attaches to running process free included with JDK prints memory map of Java process attaches to running process free included with JDK Logs JConsole JMap/JHat JStack Sunday, November 10, 13 For Production

99. Memory Information Sunday, November 10, 13

107. Kill Dash 3 Provides the current stack trace for each thread in a local JVM (thread dump) Available on linux and macs (ctrl+break on windows) Command: kill -3 PID Sunday, November 10, 13

108. JStack Provides the current stack trace for each thread in a local or remote JVM (thread dump) Command: jstack -l JAVA_PID > /tmp/jstack.out Sunday, November 10, 13

109. Basic Information Tools Overview Installation Cost Where to Find It New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler restart or attach feature dependent all features at cost www.yourkit.com JVisualVm proﬁler attaches to running process free included with JDK JMap/JHat prints memory map of Java process attaches to running process free included with JDK JStack prints thread dump of Java process attaches to running process free included with JDK Logs JConsole Sunday, November 10, 13 For Production

123. /** Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100); /** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } /** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . } Sunday, November 10, 13

124. /** Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100); /** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } Where the threads are sleeping /** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . } Sunday, November 10, 13

125. /** Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100); /** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } queue limited to 100 strings Where the threads are sleeping /** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . } Sunday, November 10, 13

126. /** Stores stats on the last names of runners which have been queried. */ public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100); /** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } } queue limited to 100 strings Where the threads are sleeping /** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . } Sunday, November 10, 13

127. Key Takeaway Do not block threads Sunday, November 10, 13

129. The File waiting for more requests to come in Sunday, November 10, 13

132. Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg Sunday, November 10, 13

133. Patrick Makau Musyoki Previous Marathon World Record Holder Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg Sunday, November 10, 13

134. I tried to look for marathons on your site but could not access it Image From: http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg Sunday, November 10, 13

136. JConsole Graphical User Interface (GUI) which displays Java Management Extension (JMX) metrics for a local or remote JVMs JMX metrics should be enabled in your application server Sunday, November 10, 13

137. Basic Information Tools Overview Installation Cost Where to Find It New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler restart or attach feature dependent all features at cost www.yourkit.com JVisualVm proﬁler attaches to running process free included with JDK JConsole JMX metrics attaches to running process free included with JDK JMap/JHat prints memory map of Java process attaches to running process free included with JDK JStack prints thread dump of Java process attaches to running process free included with JDK Logs Sunday, November 10, 13 For Production

143. JVM speciﬁc mbeans Sunday, November 10, 13

144. Tomcat Application Server mbeans JVM speciﬁc mbeans Sunday, November 10, 13

150. Can you do this in JVisualVM? Sunday, November 10, 13

158. Tomcat Application Server mbeans JVM speciﬁc mbeans Sunday, November 10, 13

168. The File Sunday, November 10, 13 vi tomcat/conﬁg/server.xml

171. What Should Be The Size Of The Pool? • If the pool size is too small, then requests back up Image From: http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg Sunday, November 10, 13

172. What Should Be The Size Of The Pool? • If the pool size is too large, then system resources might become exhausted Sunday, November 10, 13

173. What Should Be The Size Of The Pool? • If the pool size is too large, then system resources might become exhausted Images From: http://static.ddmcdn.com/gif/airport-security-detector1.jpg, http://graphics8.nytimes.com/images/2012/10/07/travel/07GETAWAY_SPAN/ 07GETAWAY_SPAN-articleLarge.jpg Sunday, November 10, 13

174. What Should Be The Size Of The Pool? • Pool size limiting factors – The amount of memory • In Java 6, the default thread stack size on Sparc is 512k for the 32-bit VM and 1024k for the 64-bit VM Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html Sunday, November 10, 13

175. What Should Be The Size Of The Pool? • Pool size limiting factors – The amount of memory • In Java 6, the default thread stack size on Sparc is 512k for the 32-bit VM and 1024k for the 64-bit VM –The number of CPUs • Context switching could become an issue Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html Sunday, November 10, 13

176. Key Takeaway Thread pools need to be tuned Sunday, November 10, 13

180. Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg Sunday, November 10, 13

181. Carmelita Jeter Fastest Woman Alive in the 100 meter sprint Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg Sunday, November 10, 13

182. My team’s 4x100 time is faster than your website. Get that ﬁxed! Image From: http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg Sunday, November 10, 13

184. Logs Application, server, and database logs provide information developers consider important for evaluating the status of the application The quality of the logs varies signiﬁcantly between products Tools like Splunk and Loggly can help analyze logs Sunday, November 10, 13

185. Basic Information Tools Overview Installation Cost Where to Find It Logs application, server, framework, database depends free depends New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler restart or attach feature dependent all features at cost www.yourkit.com JVisualVm proﬁler attaches to running process free included with JDK JConsole JMX metrics attaches to running process free included with JDK JMap/JHat prints memory map of Java process attaches to running process free included with JDK JStack prints thread dump of Java process attaches to running process free included with JDK Sunday, November 10, 13 For Production

186. The File Sunday, November 10, 13 The Application Log

189. The File Sunday, November 10, 13 The App Server Log

190. The File Sunday, November 10, 13 host_access_log

191. The File Sunday, November 10, 13 host_access_log

192. The File Sunday, November 10, 13 catalina.out

195. The File Sunday, November 10, 13 applicationContext.xml

196. The File Sunday, November 10, 13 applicationContext.xml

197. The File Sunday, November 10, 13 MySql Logs

198. The File Sunday, November 10, 13 MySql Logs - error log

199. The File Sunday, November 10, 13 MySql Logs - slow sql conﬁg

200. The File Sunday, November 10, 13 MySql Logs - slow sql log

201. MySql Logs - slow sql log The File Query Time Sunday, November 10, 13 Rows Examined

202. MySql Logs - slow sql log The File Query Time Sunday, November 10, 13 Rows Examined

222. Key Takeaway Use indexes appropriately Sunday, November 10, 13

227. Basic Information Tools Overview Installation Cost Where to Find It Logs application, server, framework, database depends free depends New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com YourKit proﬁler restart or attach feature dependent all features at cost www.yourkit.com JVisualVm proﬁler attaches to running process free included with JDK JConsole JMX metrics attaches to running process free included with JDK JMap/JHat prints memory map of Java process attaches to running process free included with JDK JStack prints thread dump of Java process attaches to running process free included with JDK Sunday, November 10, 13 For Production

228. Basic Information Tools Application Response Time JConsole JVisualVm - No Profiler New Relic JStack Logs Your Kit - No Profiler JMap Your Kit - With Profiler JVisualVM - With Profiler Sunday, November 10, 13 What is the impact of each tool?

229. Basic Information Tools Application Response Time JConsole None JVisualVm - No Profiler None New Relic Low JStack Low Logs Low - High Your Kit - No Profiler High JMap High Your Kit - With Profiler Very High JVisualVM - With Profiler Very High Sunday, November 10, 13

230. Basic Information Tools Application Response Time JConsole None JVisualVm - No Profiler None New Relic Use In Production Low JStack Low Logs Low - High Your Kit - No Profiler High JMap High Your Kit - With Profiler Very High JVisualVM - With Profiler Very High Sunday, November 10, 13

231. Basic Information Tools Application Response Time JConsole None JVisualVm - No Profiler None Use In Production New Relic Low JStack Low Logs Low - High Your Kit - No Profiler High JMap High Your Kit - With Profiler Use In Development JVisualVM - With Profiler Sunday, November 10, 13 Very High Very High

232. Basic Information Tools Overview Installation Cost Where to Find It For Production Logs application, server, framework, database depends free depends yes New Relic APM solution restart app with agent basic free, advanced cost www.newrelic.com yes YourKit profiler restart or attach feature dependent all features at cost www.yourkit.com no JVisualVm profiler attaches to running process free included with JDK basic: yes, profiler: no JConsole JMX metrics attaches to running process free included with JDK yes JMap/JHat prints memory map of Java process attaches to running process free included with JDK no JStack prints thread dump of Java process attaches to running process free included with JDK yes Sunday, November 10, 13

233. I encourage you to add a new tool to your tool box Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg Sunday, November 10, 13

234. I encourage you to add a new tool to your tool box Add the tool sooner rather than later Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg Sunday, November 10, 13

235. You should learn how to use the tools before you need them Image From: http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg Sunday, November 10, 13

236. WE WANT HAPPY USERS Images From: http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg, http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg, http:// www.runnersworld.com/sites/default/files/rw/Jeter-Carmelita-Happy-300x200.jpg, http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif Sunday, November 10, 13

237. Questions? Ashley Puls ashley@newrelic.com Sunday, November 10, 13

Troubleshooting Live Java Web Applications

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

Troubleshooting Live Java Web Applications

Plus De Contenu Connexe

Les utilisateurs ont également aimé

Similaire à Troubleshooting Live Java Web Applications

À la une

Livres associés

Troubleshooting Live Java Web Applications