With Atlassian Support as your guide learn how to diagnose your JIRA and Confluence installation like a pro, where to look for solutions, and tips for getting a quick response from us when all else fails.

2. Troubleshooting JIRA and
Confluence!
While keeping your sanity!
Jeff Curry!
!
Atlassian Support!

4. Agenda!
• Troubleshooting process!
• Reviewing logfiles!
• Reading thread dumps!
• Taking it further!
• Getting support ( quicker )!
!

5. Case Review!

6. Case Review!
• Entitled Error Message !
• Daily crashes, poor
performance!
• Larger instance!

7. Case Review – Reviewing Logfiles!
logs/atlassian-jira.log!
( NOT catalina.out )!

8. QuartzWorker-0 ERROR ServiceRunner xxx
[service.services.mail.MailFetcherService] xxx[10030]: Error connecting to host xxx'
as user xxx' via protocol 'pop3s': javax.mail.MessagingException: Connect failed;
nested exception is:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX
path validation failed: java.security.cert.CertPathValidatorException: signature check
failed
javax.mail.MessagingException: Connect failed;
nested exception is:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX
path validation failed: java.security.cert.CertPathValidatorException: signature check
failed!

9. org.apache.jk.core.MsgContext action!
WARNING: Error sending end packet!
java.net.SocketException: Software caused connection abort: socket write error!
at java.net.SocketOutputStream.socketWrite0(Native Method)!
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)!
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)!
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)!
at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)!
at org.apache.jk.core.MsgContext.action(MsgContext.java:293)!
at org.apache.coyote.Response.action(Response.java:182)!
at org.apache.coyote.Response.finish(Response.java:304)!
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)!
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)!
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)!
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)!
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)!
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)!
at java.lang.Thread.run(Thread.java:534)!

10. DefaultIndexManager - Lock attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Lock attempt timed out - waited 30000 milliseconds
!at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:463)
!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:286)
!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:252)
!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:260)
!at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
!at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
!at java.lang.reflect.Method.invoke(Method.java:618)
!at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)
!at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
!at $Proxy4.reIndexIssueObjects(Unknown Source)
!at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:138)
!at com.atlassian.jira.event.listeners.search.IssueIndexListener.issueCommented(IssueIndexListener.java:78)
!at com.atlassian.jira.event.issue.AbstractIssueEventListener.workflowEvent(AbstractIssueEventListener.java:118)
!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:125)
!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:83)
!at com.atlassian.jira.web.action.issue.AddComment.alertSystemOfComment(AddComment.java:101)
!at com.atlassian.jira.web.action.issue.AddComment.doExecute(AddComment.java:85)
!at webwork.action.ActionSupport.execute(ActionSupport.java:153)
!at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)
!at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)
!at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:178)
!at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
!

11. Case Review – Reviewing Logfiles!
Garbage collection ( GC ) logs!
-verbose:gc -Xloggc:<home>/logs/gc.log - XX:
+PrintGCTimeStamps -XX:+PrintGCDetails -XX:
+PrintGCDateStamps!
!
[ http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html ]!

12. Case Review – Reviewing Logfiles!
!
• Plain!
user=0.06 sys=0.01, real=0.02 secs]
48.630: [GC [PSYoungGen: 112384K->5460K(131072K)] • Filtered!
112384K->5460K(1029888K), 0.0158160 secs] [Times: user=0.03
sys=0.01, real=0.03 secs] ! grep 'Full GC' gc.log | awk '{print $1, $16}'!
93.865: [GC [PSYoungGen: 117844K->16468K(131072K)]
117844K->16468K(1029888K), 0.0329740 secs] [Times: 9285.005: real=7.80!
user=0.06 sys=0.03, real=0.03 secs] !
9474.161: real=10.99!
105.728: [GC [PSYoungGen: 128852K->18302K(131072K)]
128852K->18302K(1029888K), 0.0287090 secs] [Times: 10106.270: real=5.57!
user=0.07 sys=0.01, real=0.02 secs] !
122.928: [GC [PSYoungGen: 130686K->18679K(131072K)] 10355.367: real=12.84!
130686K->21123K(1029888K), 0.0221030 secs] [Times:
10758.017: real=15.17!

13. Case Review - Workaround!
• GC tuning for stability!
• Concurrent !
• -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:SurvivorRatio=8!
• 2 GB allocation!
• Added vCPU!
!

14. Case Review – Thread dumps!
"QuartzWorker-0" prio=10 tid=0x0a2ac400 nid=0xe3e runnable [0x72f04000]!
java.lang.Thread.State: RUNNABLE!
at java.net.SocketInputStream.socketRead0(Native Method)!
at java.net.SocketInputStream.read(SocketInputStream.java:129)!
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)!
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)!
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)!
- locked <0x744f7a38> (a java.lang.Object) !

15. Case Review – Thread dumps!
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)!
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)!
- locked <0x74501140> (a java.io.BufferedInputStream)!
at java.io.DataInputStream.readLine(DataInputStream.java:496)!
at com.sun.mail.pop3.Protocol.simpleCommand(Protocol.java:360)!
at com.sun.mail.pop3.Protocol.<init>(Protocol.java:104)!
at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:214)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:157)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!

16. Case Review – Thread dumps!
at javax.mail.Service.connect(Service.java:288)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:146)!
at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)!
at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)!
at org.quartz.core.JobRunShell.run(JobRunShell.java:195)!
at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)!
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)!

17. Case Review – Thread dumps!
"IssueIndexer:thread-3" prio=10 tid=0x08ec1000 nid=0x1b64 runnable [0x71f44000]!
java.lang.Thread.State: RUNNABLE!
...!
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java)!
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)!
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)!
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)!
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)!

18. Case Review – Thread dumps!
at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:605)!
at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)!
at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:790)!
at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:775)!
at org.ofbiz.core.entity.GenericDelegator.getRelated(GenericDelegator.java:1211)!
at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(GenericDelegator.java:1169)!
at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(GenericValue.java:260)!
at
com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(DefaultIssueManager.j
ava:318)!

19. Case Review – Thread dumps!
at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(DefaultCommentManager.java:135)!
at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:40)!
at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:29)!
at com.atlassian.jira.issue.index.DefaultIssueIndexer$CommentDocumentBuilder.get(DefaultIssueIndexer.java:460)!
at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:315)!
at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:
371)!
!
!

20. Case Review - Resolution!
JIRA
POP Mail Issue Database
service
inbox
Key

21. Case Review - Resolution!

22. Taking it further
!

23. Taking it further - VisualVM!
• http://visualvm.java.net!
• Monitor performance!
• Profile memory usage!
• Review thread dumps!
• Browse heap dumps!

24. Taking it further – OS Monitoring!
Linux 2.6.18-164.11.1.el5PAE (test.jira.com) !05/17/2012! 01:20:01 AM all 3.48 0.00 1.14 0.06 0.00 95.33!
! 01:30:01 AM all 3.80 0.00 0.95 0.06 0.00 95.19!
12:00:01 AM CPU %user %nice %system %iowait %steal %idle! 01:40:01 AM all 3.43 0.00 0.90 0.07 0.00 95.60!
12:10:01 AM all 5.22 0.00 1.50 6.50 0.00 86.79! 01:50:01 AM all 3.41 0.00 0.92 0.07 0.00 95.61!
12:20:01 AM all 3.42 0.00 0.90 0.08 0.00 95.60! 02:00:01 AM all 3.42 0.00 0.91 0.19 0.00 95.49!
12:30:01 AM all 4.77 0.00 1.01 0.29 0.00 93.92! 02:10:01 AM all 3.71 0.00 0.94 3.59 0.00 91.76!
12:40:01 AM all 3.47 0.00 0.90 0.07 0.00 95.56! 02:20:01 AM all 3.38 0.00 0.90 0.08 0.00 95.65!
12:50:01 AM all 3.43 0.00 1.13 0.07 0.00 95.38! 02:30:01 AM all 3.77 0.00 0.94 0.07 0.00 95.23!
01:00:01 AM all 3.48 0.00 1.12 0.06 0.00 95.33! 02:40:01 AM all 3.38 0.00 0.89 0.07 0.00 95.66!
01:10:01 AM all 4.27 0.00 0.95 1.74 0.00 93.04! 02:50:01 AM all 3.37 0.00 0.89 0.06 0.00 95.69!
03:00:01 AM all 3.37 0.00 0.89 0.07 0.00 95.67!

25. Taking it further - SAR!
http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html!

26. Taking it further – HTTP inspection!
• Fiddler!
• http://www.fiddler2.com/fiddler2/!
• Live HTTP Headers ( Firefox addon )!
• https://addons.mozilla.org/en-US/firefox/addon/live-http-headers/!

27. Taking it further – HTTP inspection!

28. Taking it further – HTTP inspection!

29. Getting support ( quicker )
!

30. Getting support – self-help!

31. Getting support – self-help!
• Knowledge base articles!
• https://confluence.atlassian.com/display/KB/Knowledge+Base+Home!
• Answers.atlassian.com!
• #atlassian on irc.freenode.com!

32. Getting support – self-help!

33. !

34. Getting support – Atlassian view!
• Initial response time SLA!
• 1 – 1 hour!
• 2 – 4 hours!
• 3 – 8 hours!
• 4 – 24 hours!
!

35. Getting support – Atlassian view!
Open
Waiting for Atlassian
Waiting for
Customer
Investigating
Customer
IRT

36. Getting support – file a ticket!
• When I… I expect… however…!
• Application log files !
• Screenshots!
• Steps to reproduce!

37. Getting support – file a ticket!
• When I use JIRA and try to login!
• I expect JIRA to display a login page!
• however I get this 404 page!
• log files attached!

38. Getting support – Atlassian view!
Open
Atlassian
Waiting for
Investigating
Customer
IRT

39. Summary!
• Basic!
• Application and GC log files!
• Thread dumps!
• Advanced!
• VisualVM!
• HTTP inspection!

40. I am my own Atlassian support engineer AMAA
#atlassiansummit

41. Thank you!