"We’re going on a Bug Hunt" Gerald Benischke, Software Engineer, Equal Experts.

1. Making Software. Better.
Simple solutions to big business problems.
Equal Experts is a network of talented, experienced, software
consultants, specialising in agile delivery.

2. The network
I’m part of the Equal Experts network,
a community of highly-experienced
software professionals.
At Equal Experts we’re given the
freedom to get on with what’s
important - delivering better software
for our clients’ customers.

3. We’re going on a Bug Hunt
Gerald Benischke
Software Engineer

4. Bug Hunter Prerequisites
! Goldilocks Logs
! Distributed Tracing
! Event Stores
! Source Code (sometimes decompiled classes will do)
! Metrics
! Ability to run locally
! HTTP Proxy
! Debugger
! Patience

5. Database Connectivity?
java.sql.SQLTransientConnectionException:
myPoolName - Connection is not available,
request timed out after 20000ms.

6. Talking to Oracle
WAR
Spring
JDBC
Hikari
CP
ELB
DB
Proxy
VPN

7. Talking to Oracle

8. Talking to Oracle Checklist
! Speak with the DBAs to check that the database is up
! Check the DB proxy that traffic is passing through
! Check whether any firewall / VPN changes were made
! Check Logs
! Check Stacktraces
! Check Metrics - and hope we’ve got decent ones!
! Check Logs Again

9. Is it a Database Problem?

10. Get sidetracked - What does JMS have to do with a DB Bug?
WAR
Spring
Jms
Template
Active
MQ
Client
ELB
Camel
MQ
Bridge
Active
MQ
VPN

11. Why would that be connected?
jms.brokerUrl: "failover:(tcp://activemq-1:61616,tcp://activemq-2:61616,tcp://activemq-3:61616)?
maxReconnectAttempts=1&startupMaxReconnectAttempts=1"
October 26th 2017, 10:40:39.681 Waiting 30000 ms before attempting connection
October 26th 2017, 10:40:31.470 Attempting 1088th connect to: tcp://ip-10-10-10-217:61616
October 26th 2017, 10:40:09.623 Attempting 1084th connect to: tcp://ip-10-10-10-217:61616
October 26th 2017, 10:40:01.469 Waiting 30000 ms before attempting connection
October 26th 2017, 10:39:39.592 Attempting 1084th connect to: tcp://ip-10-10-10-25:61616
October 26th 2017, 10:39:31.434 Attempting 1087th connect to: tcp://ip-10-10-10-25:61616
October 26th 2017, 10:39:09.592 Waiting 30000 ms before attempting connection
October 26th 2017, 10:39:01.407 Attempting 1087th connect to: tcp://ip-10-10-10-217:61616

12. NO timeout?
! ActiveMQ was restarted - but bound to different DHCP addresses
! App lost connection - and waits forever in sending

13. Symptoms and Causes
can be miles apart

14. Work on Sunday

15. Work on Sunday
! It works locally - against a DB running in docker
! It is broken on the server
! The settings have not been changed
! The stored procedure is the same
! The server versions are the same
! The driver versions are the same
! What else could it be?
! The logs are definitely showing the production server is
returning wrong results!

16. TO_CHAR
NLS_TERRITORY=AMERICA
! Friday=6
! Saturday=7
NLS_TERRITORY=UK
! Saturday=6
! Sunday=7

17. Oracle JDBC driver
Sets NLS_TERRITORY based on client Locale
Developer workstation was British
Server used Default Locale (US)
Fix the Application Servers settings
-Duser.language: en
-Duser.country: GB

18. Challenge Assumptions

19. Duplicate Submissions

20. Retries out of nowhere
! Backend team reports duplicate submissions
! Looking at the code, we don’t retry submissions
! What could cause it?
WAR
Http
Client
ELB
App
Proxy
VPN

21. Surely it’s not an App Issue?

22. Retries out of nowhere
! Combination of ELB and Http Client and timeouts

23. Retries out of nowhere solved
! Http Client timeout (not set)
! ELB timeout (60 seconds)
! ELB in TCP mode - connection just gets dropped on
timeout
! Results in a NoHttpResponseException
! Http Client automatically retries (even POSTs)!

24. Understand the Whole System
Be More Ops

25. Memory Leak

26. Memory Leak
! Instances crashed with an OutOfMemoryError (but it took
time)
! A single request triggered the above

27. Define a Memory Leak

28. Memory Leak Solved
! Due to corrupt data meant duplicate identifiers in a list
! A loop tried to find the “next” item in the list by identifier
and always found the first one - thereby entering an
infinite loop
! As part of the loop a Java object was added to another list
! The infinite loop kept going and adding to the list
! Eventually the JVM would only GC
! Healthcheck would kill the JVM eventually and restart

29. Know how to use dumps and profilers

30. SSL Performance

31. Entropy isn’t what it used to be
! When we moved to AWS - we experienced a dramatic
slow down in performance for SSL
! It turned out that in the old cloud the images had installed
havged package - which was missing from the new
! /dev/random blocked….

32. Summary

33. To Go on a Bug Hunt
! Bring the right tools
! Bring patience
! Experience helps
! Challenge assumptions
! Look at the whole system

34. QUESTIONS?

35. Thank You
United Kingdom
+44 203 603 7830
helloUK@equalexperts.com
Equal Experts UK Ltd
30 Brock Street
London NW1 3FG
India
+91 20 6607 7763
helloIndia@equalexperts.com
Equal Experts India Private Ltd
Office No. 4-C
Cerebrum IT Park No. B3
Kumar City, Kalyani Nagar
Pune, 411006
Canada
+1 403 775 4861
helloCanada@equalexperts.com
Equal Experts Devices Inc
205 - 279 Midpark way S.E.
T2X 1M2
Calgary, Alberta
Portugal
+351 211 378 414
helloPortugal@equalexperts.com
Equal Experts Portugal
Avenida Dom João II, Nº35
Edificio Infante 11ºA
1990-083 Parque das Nações
Lisboa – Portugal
Thank You
USA
+1 866-943-9737
helloUSA@equalexperts.com
Equal Experts Inc
1460 Broadway
New York
NY 10036
LinkedIn
linkedin.com/company/equal-experts
Twitter
@EqualExperts
Web
www.equalexperts.com