With the increasing adoption of cloud native technologies and containerization; the gap between Java development and system administration is decreasing. Whether you are using Docker Swarm, Kubernetes or Mesos/Marathon as a container orchestrator; fundamental challenges for running docker in production are common. In this talk, I would like to share some of the basic linux concepts about CPU scheduling every Java Developer should know to be able to perform effective configuration and troubleshooting for docker containers. Yes, Docker provides isolation, but only if you know how best to configure it.

1. Docker, JVM and CPU
Aparna Chaudhary (@aparnachaudhary)

2. System is slow!
Service foo is
slow everyday at
23:00
All services on
this specific node
are slow
Noisy Neighbours
@aparnachaudhary

3. CPU Shares
● Default CPU isolation
● Provides a priority weighting across all cpu cycles across all cores.
● Default weight for any container is 1024
Containers
per node
increases
CPU per
container
reduces
@aparnachaudhary
docker run --rm -d mytinyservice:1.0.0

4. Core Core
Core Core
23:00 @aparnachaudhary
CPU Shares and noisy neighbours
16:00
FooFoo
Bar Bar
OnDietOnDiet
Foo becomes slow
because of Bar

5. CPU Set
● Limits container’s processes to specific CPU cores
● A comma-separated list of cores
● A hyphen separated range of cores
Used typically in
case of a database
@aparnachaudhary
docker run --rm -d cpuset=0-1 mydb:1.0.0
docker run --rm -d cpuset=1,3 mydb:1.0.0

6. CPU Limits
Allows container to use CPU time for duration of cpu-quota every cpu-period
--cpu-period = 100 microseconds (default)
--cpu-quota = 20 microseconds
Then application is allowed to use 20 microseconds of CPU time every 100
microseconds.
@aparnachaudhary

7. Scenario: Application uses all CPU quota at the start
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80 µs20 µs 80 µs20 µs 80 µs20 µs
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 80 µs Run
Throttle
@aparnachaudhary

8. Scenario: STW GC Run
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 95 µs
Run
Throttle
GC
5 15 805 15 805 15
@aparnachaudhary

9. Scenario: Concurrent GC Run (STW + non-STW phases)
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 95 µs
Run
Throttle
GC
10 8010 8010
CPU time stolen by GC -
end user experiences
STW.
10 10 10
Larger the number of GC
threads, higher the latency
experienced by end user.
@aparnachaudhary

10. Running JVM in docker
needs thorough
understanding of how
JVM GC interacts with
cgroup CPU scheduling
@aparnachaudhary

11. CPU, Cgroups, JVM - What do I do?
CPU Shares:
● Unpredictable performance because of
noisy neighbours
● Simple to configure
● Allows use of idle CPU resources
● Difficult capacity planning
CPU Limits:
● Predictable performance if tuned properly
for GC behavior
● Difficult to configure properly
● Idle CPU resources are not utilized
● Better capacity planning
@aparnachaudhary

12. Docker and Diagnostics

13. Docker REST API
@aparnachaudhary

14. cgroups Pseudo files
● Fastest way to gather stats
● No root access needed
@aparnachaudhary

15. Remember...

16. Decide on QoS (Best
Effort, Burstable,
Guaranteed)
Use cgroup pseudo files to
understand resource
utilization
CPU share will cause
unpredictable
performance because of
noisy neighbours
CPU limits may cause
throttling of application
during GC.
Make periodic
thread dumps
@aparnachaudhary
Use JVM Flags
-XX:+PrintFlagsFinal
-XX:ParallelGCThreads
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-Xloggc:<file-path>

17. References
● GC analysis - http://gceasy.io/
● Thread dump analysis - http://fastthread.io/
● CFS CPU Scheduler -
https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt
@aparnachaudhary