Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Why you’re going to fail running java on docker!

2 542 vues

Publié le

Why you’re going to fail running java on docker! - Rafael Benavides

Publié dans : Logiciels
  • Soyez le premier à commenter

Why you’re going to fail running java on docker!

  1. 1. 1 Why you’re going to FAIL running Java on docker! bit.ly/javadockerfail @burrsutter @rafabene
  2. 2. @burrsutter Change History 0.1 - JBCNConf.com 2017 - Barcelona 0.2 - JavaOne 2017 - San Francisco 0.3 - JavaDay UA 2017 - Kiev
  3. 3. Big Wins for Developers ● Highly Portable Packaging solution - for microservices, web apps ● Lightweight, Encapsulated OS abstraction - carry your OS with you ● Getting Started (docker run -it centos/wildfly) Instantly ● Dev Environments that more closely match Prod Environments ● Dev Environments that match OTHER Dev Environments (no more...but it works on my machine) ● No more waiting 3+ weeks for a VM to be provisioned by Ops just so you can run a series of tests Linux Container
  4. 4. @burrsutter
  5. 5. @burrsutter
  6. 6. Virtualization vs Containers Server Machine Host Operating System Hypervisor Guest OS Guest OS Guest OS Bins Libs Bins Libs Bins Libs App 1 App 2 App 3 Server Machine Host Operating System Bins Libs Bins Libs App 1 App 2 App 3 Docker ● Shared bins/libs and Host Operating System make containers lighter ● You can run multiple versions of app dependencies (Java 6 and 7) ● “Golden Image” deployment model ● Process isolation
  7. 7. Pros vs Cons Pros: ● Docker containers start very fast, take less memory ● Super easy to try out a new technology (e.g docker pull centos) ● Supports the “immutable image” deployment model ● It caches layers to make builds faster Cons: ● It is not a VM - it is a process running within the Host OS, do not assume you can run any flavor of container OS on any host OS ● It is not perfectly portable - you will eventually wish to get out of boot2docker-vm into a real Linux VM - one that more closely matches your production environment ● Lineage of docker images at Docker Hub is unknown
  8. 8. History of Containers 2000 2010 2005 2015 2000: JAILS ADDED TO FREEBSD 2006: GENERIC PROCESS CONTAINERS 2008: KERNEL AND USER NAMESPACES 2014: GOOGLE KUBERNETES 2008: LINUX CONTAINER PROJECT (LXC) 2015: STANDARDS VIA OCI AND CNCF 2013: RED HAT ENTERPRISE LINUX 2013: DOTCLOUD BECOMES DOCKER 2007: GPC RENAMED CONTROL GROUPS 2003: SELINUX ADDED TO LINUX MAINLINE 2015: RHT CONTAINER PLATFORM 2001: LINUX -VSERVER PROJECT 2013: DOTCLOUD PYCON LIGHTNING TALK 2005: FULL RELEASE OF SOLARIS ZONES
  9. 9. History of Java Java 1.0.2 - 30 May 1996 J2EE 1.2 - 12 Dec 1999 https://en.wikipedia.org/wiki/Java_EE_version_history#J2EE_1.2_.28December_12.2C_1999.29 https://en.wikipedia.org/wiki/Java_version_history
  10. 10. A Challenge Server Hardware Operating System Application Server .war or .ear Java Virtual Machine Custom Configuration Linux Kernel Version & Distribution Java 1.6.6_45 or Java 1.7.0_67 Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y JDBC driver, datasource, JMS queue, users Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code? Containerize Your App
  11. 11. Email MyApp.war has been tested with the following On my Windows 7 desktop JDK 1.8.43 Wildfly 9 Configuration: Datasource: MySQLDS Tested with: mysql-connector-java-5.1.31-bin.jar Production Environment Red Hat Enterprise Linux 6.2 JRE 1.7.3 WebSphere 8.5.5 Oracle 9
  12. 12. Dockerfile Container Guest OS Java App Server custom configuration FROM centos/wildfly COPY standalone.xml /opt/wildfly/standalone/configuration/ COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/ COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/ COPY myapp/target/your.war /opt/wildfly/standalone/deployments/ your.war dependencies Note: There are better ways to handle Java apps, this is for illustration purposes
  13. 13. @burrsutter Behind the Scenes
  14. 14. Cgroups and Namespaces Cgroups - cpu (cpu shares) - cpuacct - cpuset (limit processes to a CPU) - memory (swap, dirty pages) - blkio (throttle reads/writes) - devices - net_prio (packet class and priority) - freezer Namespaces - pid (processes) - net (network interfaces, routing) - ipc (system V ipc) - mnt (mount points, filesystems) - uts (hostname) - user (UIDs)
  15. 15. /sys/fs/cgroup/memory cgroup.clone_children memory.kmem.tcp.limit_in_bytes memory.move_charge_at_immigrate cgroup.event_control memory.kmem.tcp.max_usage_in_bytes memory.oom_control cgroup.procs memory.kmem.tcp.usage_in_bytes memory.pressure_level memory.failcnt memory.kmem.usage_in_bytes memory.soft_limit_in_bytes memory.force_empty memory.limit_in_bytes memory.stat memory.kmem.failcnt memory.max_usage_in_bytes memory.swappiness memory.kmem.limit_in_bytes memory.memsw.failcnt memory.usage_in_bytes memory.kmem.max_usage_in_bytes memory.memsw.limit_in_bytes memory.use_hierarchy memory.kmem.slabinfo memory.memsw.max_usage_in_bytes notify_on_release memory.kmem.tcp.failcnt memory.memsw.usage_in_bytes tasks
  16. 16. @burrsutter
  17. 17. DevOps Challenges for Multiple Containers ▪ How to scale? ▪ How to avoid port conflicts? ▪ How to manage them on multiple hosts? ▪ What happens if a host has trouble? ▪ How to keep them running? ▪ How to update them? ▪ Where are my containers? Node Node Node Node Node Logger Node
  18. 18. @burrsutter
  19. 19. ▪ Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator) ▪ Container orchestrator ▪ Supports multiple cloud and bare-metal environments ▪ Inspired by Google’s experience with containers ▪ Open source, written in Go ▪ Manage applications, not machines Meet Kubernetes
  20. 20. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  21. 21. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  22. 22. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  23. 23. @burrsutter Java
  24. 24. Why Fail? 8GB 4 cores 64GB 16 cores Developer Workstation Production Server
  25. 25. @burrsutter System.out.println("Memory: " + Runtime.getRuntime().maxMemory()/1024/1024); System.out.println("CPUs: " + Runtime.getRuntime().availableProcessors()); Why does Java Fail? It sees ALL the HOST resources, it is blind to cgroups by default Note: free also fails https://fabiokung.com/2014/03/13/memory-inside-linux-containers/
  26. 26. @burrsutter
  27. 27. @burrsutter Tomcat https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/core/ContainerBase.java#L304 https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/util/net/NioEndpoint.java#L138 https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/websocket/AsyncChannelGroupUtil.java#L85 https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/manager/StatusTransformer.java#L224 Wildfly https://github.com/wildfly/wildfly-common/blob/master/src/main/java/org/wildfly/common/cpu/ProcessorInfo.java#L61 Vert.x https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/VertxOptions.java#L41 Who uses those APIs? Tomcat, Wildfly, Vert.x
  28. 28. @burrsutter Memory (max heap defaults to ¼ of available memory) JIT Optimizations Mode: Client or Server Thread Management Garbage Collector Do NOT just “java -jar myVertxApp.jar” nor “java -jar mySpringBootApp.jar” Do try “java -XX:+PrintFlagsFinal -XX:+PrintGCDetails $JAVA_OPTIONS -jar myApp.jar” JVM Ergonomics https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.html
  29. 29. @burrsutter Memory
  30. 30. @burrsutter Heap (-Xms -Xmx) ------------------------------ Native JRE Perm (-XX:MaxPermSize), Java8 Meta (-XX:SurvivorRatio, -XX:MaxNewSize, -XX:NewRatio) JIT Bytecode JNI NIO Threads (defaults to 512K to 1024K depending on platform) Tip: Either -Xmx at 50% of container constrained memory or -XX:MaxRam=500m - your heap will be approximately 250mb * JVM Memory: Heap is about 50%
  31. 31. @burrsutter
  32. 32. @burrsutter https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/ https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits max_memory() { # High number which is the max limit until which memory is supposed to be # unbounded. local max_mem_unbounded="$(cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes)" local mem_file="/sys/fs/cgroup/memory/memory.limit_in_bytes" if [ -r "${mem_file}" ]; then local max_mem="$(cat ${mem_file})" if [ ${max_mem} -lt ${max_mem_unbounded} ]; then echo "${max_mem}" fi fi } Workaround docker pull fabric8/java-jboss-openjdk8-jdk
  33. 33. @burrsutter Heap Patched: Java 8u131, included in Java 9 https://bugs.openjdk.java.net/browse/JDK-8170888
  34. 34. @burrsutter Try it yourself Just need Docker for Mac or Windows docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version vs docker run -m 100MB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version UseCGroupMemoryLimitForHeap tells the JVM to look to the cgroup’s value for memory in /sys/fs/cgroup/memory/memory.limit_in_bytes
  35. 35. @burrsutter Cores
  36. 36. @burrsutter docker run --cpus="1.5" -m 400M openjdk:8u131 *docker run --cpu-quota=150000 --cpu-period=100000 -m 400M openjdk:8u131 docker run --cpu-shares=1536 -m 400M openjdk:8u131 **docker run --cpuset-cpus=0,2 -m 400M openjdk:8u131 Docker CPU Settings https://docs.docker.com/engine/admin/resource_constra ints/#configure-the-default-cfs-scheduler * Kubernetes uses this model ** JVM properly calculates availableProcessors()
  37. 37. @burrsutter docker run -it --cpus="2" -m 200M java java -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 -XX:MaxRam=200m // Half a core cpu-quota / cpu-period docker run -it --cpu-quota=50000 --cpu-period=100000 -m 200M openjdk:8u131 java -XX:+UseSerialGC -Xmx=100m Note: Set Max heap (-Xmx) to about 50% of total container memory Note: UseSerialGC when core count is below 2 Docker CPU Mapped to Java Some “rules of thumb”
  38. 38. @burrsutter CPU Set Patched https://bugs.openjdk.java.net/browse/JDK-6515172
  39. 39. @burrsutter Demo Boom https://github.com/burrsutter/javadockerfail Demo Script https://docs.google.com/document/d/1Td_nnjk12Miq6SpRs3RysDZbpQUjpxXk_p0J526VhcE/edit?usp=sharing
  40. 40. @burrsutter Base Docker Image FROM fabric8/java-jboss-openjdk8-jdk:1.2.7 Maven Plugin: - Docker builds leveraging base image - Deploys to Kubernetes and OpenShift easily https://fabric8.io/gitbook/mavenPlugin.html mvn io.fabric8:fabric8-maven-plugin:3.3.5:setup mvn fabric8:deploy Hot Tips https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/
  41. 41. @burrsutter More and More Resources https://developers.redhat.com/products/cdk/download/ https://github.com/burrsutter/javadockerfail https://developers.redhat.com/blog/2017/03/14/java-inside-docker/ https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/ https://dzone.com/articles/running-a-jvm-in-a-container-without-getting-kille https://github.com/redhat-developer-demos/java-container https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how- pods-with-resource-limits-are-run https://github.com/fabric8io-images/java/tree/master/images/jboss/openjdk8/jdk https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015014.html bit.ly/javadockerfail
  42. 42. 52 Why you’re going to FAIL running Java on docker! bit.ly/javadockerfail @burrsutter @rafabene
  43. 43. @burrsutter Previously Used Slides
  44. 44. Creative Commons/Flikr, SF History Center, SF Public Library, GG Bridge 1935-1936 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  45. 45. Creative Commons/Flikr, SF History Center, SF Public Library, GG & Bay Bridges 1935 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  46. 46. Creative Commons Found SF Historical Archive, GG & Bay Bridges, at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  47. 47. Creative Commons Photo ‘For Bankrupt Hanjin, Costs and Containers Pile Up’ at https://www.workboat.com/news/bluewater/hanjin-costs-containers-pile-up-us-ports/
  48. 48. Wikimedia Commons ‘Container Port & Toxic Fire San Francisco’ at https://commons.wikimedia.org/wiki/File:Containter_port_and_toxic_fire,_San_Francisco.jpg

×