A high level overview of typical Java Vitual Machine issues and how to measure them.

1. Leak, Lock and a Long Pause
Troubleshooting JVM issues

2. You will learn that
• There are memory leaks in Java
• Garbage collector can be a bottleneck
• Thread locks are more common than you
think
• You can automatically detect these problems

3. How can JVM leak memory?
• Static fields are not gc’d
• Thread-local + thread pools
• Application scope
• Complex bi-directional refs
• OS resource handles: files, sockets, db

4. Static fields
class Forever {
// Unless you explicitly unload this class
// GC will not touch it
static final ArrayList list = new ArrayList(1000);
}

5. Thread-local
public void doFilter(ServletRequest req, ServletResponse res){
myThreadLocal.set(new BigLeak());
// thread goes back to the thread pool
// with the big leak planted
chain.doFilter(req, res);
}

6. Application scope
getServletContext().setAttribute(”leak", bigObj);
// unless you call removeAttribute somewhere
// this will live as long as Tomcat is up

7. Bi-directional relationships
org.w3c.dom.Document doc = readXmlDocument();
org.w3c.dom.Node child =
doc.getDocumentElement().getFirstChild();
doc.removeNode(child);
doc = null;
// child still has reference to doc. No GC for the doc.

8. OS resources
try {
Connection conn = ConnectionFactory.getConnection();
...
conn.close();
} catch (Exception e) {
// oops, connection is still open
}

9. GC Pause
source: oracle.com

10. GC Pause
• Full garbage collection stops the world!
• Heavily utilized heap = frequent pauses
• Complex dependency graph = long pause
• Lots of long living objects = long pauses

11. Heap size vs pause
source: performanceterracotta.blospot.com

12. Is it really our problem?
source: plumbr.eu

13. Is it really our problem?
QUESTION:
Does it impact our business if the system
regularly suspends itself for several seconds?
BETTER QUESTION:
What is the level of unresponsiveness we allow
in our system?
As low as possible = bullshit answer

14. No measurement = no answer
pauses for about half a second 2-3 times a minute

15. GC friendly programming
• Avoid long living objects – minor gc is faster
• Avoid mutable objects – questionable…
• Design your classes small – faster compaction
• Avoid complex dependency graphs
• NEVER call System.gc() – disturbs optimal plan

16. Thread locking
“Plain Threads are the GOTO of Today’s
Computing”
--Dr Hartmut Kaiser

17. Thread locking
“Locking is not evil, lock contention is”
Vladimir Šor

18. Contention

19. Thread contention
source: blogs.mulesoft.org

20. Thread contention - causes
• Too broad sync scope
• Overusing same locking object
• Shared mutable state abuse

21. What’s wrong here?
public boolean isAdmin(String name) {
synchronized (privileges) {
String privs= privileges.get("usr.”+name+".prv");
if (privs== null)
return false;
else
return (privs.indexOf(”administrator") >= 0);
}
}

22. Too broad sync scope
// “Protect data, not code” – Michael Suess
public boolean isAdmin(String name) {
synchronized (privileges) {
String privs= privileges.get("usr.”+name+".prv");
if (privs== null)
return false;
else
return (privs.indexOf(”administrator") >= 0);
}
}

23. What’s wrong here?
public class Attributes {
private HashMap usersStore;
private HashMap servicesStore;
public synchronized void setUserAttributes(String usr, UserAttrs attrs) {
usersStore.put(usr, attrs);
}
public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) {
servicesStore.put(svc, attrs);
}

24. Overusing a lock
public class Attributes {
private HashMap usersStore;
private HashMap servicesStore;
public synchronized void setUserAttributes(String usr, UserAttrs attrs) {
usersStore.put(usr, attrs);
}
public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) {
servicesStore.put(svc, attrs);
}
// both methods sync on same Attributes instance

25. Shared mutable state
“Every piece of mutable state you have is a
liability”
Martin Odersky

26. Is there really no other way?
balance.deposit(amount)

27. Is there really no other way?
newBalance = oldBalance.deposit(amount)
// no sync needed

28. Share immutable objects
public class Address {
public final String streetName;
public final int aptNo;
public final String zipCode;
public Address (String name, int apt, String zip){
// initialize all the fields
}
}

29. Jhiccup
JVM AppJVM App
jhiccupthread
hiccup.hloghiccup.hlog
measure thread sleep
delta(expected – actual)
src: azulsystems.com

30. What is a JVM Hiccup?
• When your application is stalled
• GC stop-the-world
• Virtualization
• CPU contention
• OS Scheduler

31. Plumbr
System running on JVMSystem running on JVM
plumbr agentplumbr agent
plumbr serverplumbr server
usage data

32. DeMO
time for

33. Sources
• plumbr.eu/blog/how-do-leak-detectors-work-plumbr-case-s
• java.dzone.com/articles/how-tame-java-gc-pauses
thinkingparallel.com/2007/07/31/10-ways-to-reduce-
lock-contention-in-threaded-programs/
• blogs.mulesoft.org/chasing-the-bottleneck-true-story-
about-fighting-thread-contention-in-your-code
• ibm.com/developerworks/library/j-threads2/
• http://www.azulsystems.com/giltene/how-java-got-
the-hiccups