1. Jconsole JMX data capture
of a Camel server
This document contains screenshots taken on a production
server. Its solely purpose is to share the collected data with
people on the Camel list and help diagnose some performance
issues.

3. Memory

11. Threads

13. Camel threads
• Some 1000 of them, most of them waiting for some lock, or parked.
• 424 <route> tags in a 4200 lines config file, there are some 626 routes available in the
Mbeans :-? (several routes are FTP from one URI to many URI)
Name: Camel (myCamel) thread #10 - ftp://retalixcon@SRV0043/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16c83b9
Total blocked: 147 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Name: Camel (myCamel) thread #192 - ftp://retalixcon@SRV0126/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e91163
Total blocked: 28 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)

14. Camel threads (continued)
Name: Camel (myCamel) thread #463 - ftp://retalixcon@SRV0153/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a1156f
Total blocked: 426 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Name: Camel (myCamel) thread #942 - ftp://retalixcon@SRV1004/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@195f6a7
Total blocked: 130 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)

15. Quartz threads
• 10 of them, mostly seem to be waiting (which should be normal).
Name: DefaultQuartzScheduler_Worker-1
State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@1efb21d
Total blocked: 0 Total waited: 123.407
Stack trace:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
Name: DefaultQuartzScheduler_Worker-5
State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@188145f
Total blocked: 0 Total waited: 123.428
Stack trace:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

16. Classes

18. Summary

20. JMX Mbeans
• Many thread pools (some 100) of
FileConsumer and FtpConsumer type alike.
• Many ProducerCache