SlideShare une entreprise Scribd logo
1  sur  44
Télécharger pour lire hors ligne
ssThe State of Linux Containers
2
Gaikai
PS Now announcement at CES 2014
3
Gaikai
- caching
+ controller feedback
1. “Linux Container” / “Docker Ecosystem” in a Nutshell
2. Confusion about Ecosystem / Vision to tackle it
3. Docker -> SWARM -> SLURM 

-> BigData
4. Discussion of Opportunities and Problems
4
Agenda
The Bits and Pieces…
Userland	(OS)Userland	(OS) Userland	(OS)
Userland	(OS)
Ubuntu:14.04 Ubuntu:15.10 RHEL7.2
Tiny	Core	Linux	
Linux Containers
6
SERVER
HOST	KERNEL
HYPERVISOR
KERNEL
SERVICE
Userland	(OS)
KERNEL KERNEL
Userland	(OS)Userland	(OS) Userland	(OS)
SERVICE SERVICE
SERVER
HOST	KERNEL
SERVICE SERVICE SERVICE
Traditional Virtualisation Containerisation
Containers do not spin up a distinct kernel
all containers & the host share the same
user-lands are independent
they are separated by Kernel Namespaces
Containers are ‘grouped processes’
isolated by Kernel Namespaces
resource restrictions applicable through CGroups (disk/netIO)
HOST
container1
7
Kernel Namespaces
bash
ls -l
container2
apache
container3
mysqld
consul consul
PIDNamespaces: Network Mount IPC UTS
container4
slurmd
ssh
consul
Container Runtime Daemon
creates/…/removes containers, exposes REST API
handles Namespaces, CGroups, bind-mounts, etc.
IP connectivity by default via ‘host-only’ network bridge
Docker Engine
8
SERVER
eth0
docker0
container1
container2
Docker-Engine
Docker Compose
9
Describes stack of container configurations
instead of writing a small bash script…
… it holds the runtime configuration as YAML file.
Docker Networking spans networks across engines
KV-store to synchronise (Zookeeper, etcd, Consul)
VXLAN to pass messages along
SERVER0 SERVER1 SERVER<n>
Docker Networking
10
Consul
Docker-Engine
Consul Consul
Docker-Engine Docker-Engine
Consul DC
global
container0 container1 containerN
Docker Swarm proxies docker-engines
serves an API endpoint in front of multiple docker-engines
does placement decisions.
SERVER0 SERVER1 SERVER<n>
Docker Swarm
11
Docker-Engine Docker-Engine Docker-Engine
swarm-client swarm-client swarm-client
swarm-master
:2376 :2376 :2376
:2375
container1
-e constraint:node==SERVER0
Docker Swarm [cont]
12
query docker-enginequery docker-swarm
Introduce new Technologies
Introducing new Tech
14
Self-perception when introducing new tech…
credit: TF2 - Meet the Pyro
Introducing new Tech
15
… not always the same as the perception of others.
credit: TF2 - Meet the Pyro
Docker Buzzword Chaos!
Distributions
Solutions
Auto-Scaling
On-Premise & OverSpill
Orchestration
self-healing
16
production-ready
enterprise-grade
1. No special distributions
useful for certain use-cases, such as elasticity and green-field
deployment
not so much for an on-premise datacenter w/ legacy in it.
2. Leverage existing processes/resources
install workflow, syslog, monitoring
security (ssh infrastructure), user auth.
3. keep up with docker ecosystem
incorporate new features of engine, swarm, compose
networking, volumes, user-namespaces
17
Vision
Reduce to the max!
Hardware (courtesy of )
8x Sun Fire x2250, 2x 4core XEON, 32GB, Mellanox ConnectX-2)
Software
Base installation
CentOS 7.2 base installation (updated from 7-alpha)
Ansible
consul, sensu
docker v1.10, docker-compose
docker SWARM
19
Testbed
node1
node2
node8
20
Docker Networking
Synchronised by Consul
Consul
Consul

DC
Consul
Consul
Docker-Engine
Docker-Engine
Docker-Engine
node1
node2
node8
21
Docker SWARM
Docker SWARM
Synchronised by Consul KV-
store
Consul
Consul

DC
Consul
Consul
Docker-Engine
Docker-Engine
Docker-Engine
swarm
swarm
SWARM
swarm master
node8
node2
node1
22
SLURM Cluster
Consul
Consul

DC
Consul
Consul
SLURM within SWARM
slurmctld slurmd
slurmd
slurmd
Docker-Engine
Docker-Engine
Docker-Engine
swarm
swarm
SWARM
swarm master
SLURM
23
SLURM Cluster [cont]
node8
node2
node1
24
SLURM Cluster [cont]
Consul
Consul

DC
Consul
Consul
SLURM within SWARM
slurmd within app-container
pre-stage containers slurmctld slurmd
slurmd
slurmd
Docker-Engine
Docker-Engine
Docker-Engine
swarm
swarm
hpcg
hpcg
SWARM
hpcg
swarm master
SLURM
25
MPI Benchmark
http://qnib.org/mpi
http://qnib.org/mpi-paper
node8
node2
node1
26
SLURM Cluster [cont]
Consul
Consul

DC
Consul
Consul
SLURM within SWARM
slurmd within app-container
pre-stage containers slurmctld slurmd
slurmd
slurmd
Docker-Engine
Docker-Engine
Docker-Engine
swarm
swarm
hpcg
hpcg
SWARM
hpcg
OpenFOAM
OpenFOAM
OpenFOAM
swarm master
SLURM
27
OpenFOAM Benchmark
http://qnib.org/immutable
http://qnib.org/immutable-paper
node1
node2
node8
28
Samza Cluster
Consul
Consul

DC
Consul
Consul
Distributed Samza
Zookeeper and Kafka cluster
Samza instances to run jobs Docker-Engine
Docker-Engine
Docker-Engine
swarm
swarm
SWARM
swarm masterzookeeper
zookeeper
zookeeper
kafka
kafka
kafka
samza
samza
samza
$ cat test.log |awk ‘{print $1}’ |sed -e ’s/HPC/BigData/g’ |tee out.log
To Be Explored
1. Where to base images on?
Ubuntu/Fedora: ~200MB
Debian: ~100MB
Alpine Linux: 5MB (musl-libc)
2. Trimm the Images down at all cost?
How about debugging tools? Possibility to run tools on the host
and ‘inspect’ namespaced processes inside of a container.
If PID-sharing arrives, carving out (e.g.) monitoring could be a
thing.
30
Small vs. Big
1. In an ideal world…
a container only runs one process, e.g. the HPC solver.
2. In reality…
MPI want’s to connect to a sshd within the job-peers
monitoring, syslog, service discovery should be present as well.
3. How fast / aggressive to break traditional
approaches?
31
One vs. Many Processes
Plugin System
VXLAN
MACVLAN
How about IPoIB?
32
Docker Network
Running OpenFOAM on small scale is cumbersome
manually install OpenFOAM on a workstation
be confident that the installation works correctly
A containerised OpenFOAM installation tackles both
33
Reproducibility / Downscaling
http://qnib.org/immutablehttp://qnib.org/immutable-paper
1. Since the environments are rather dynamic…
how does the containers discover services?
external registry as part of the framework?
discovery service as part of the container stacks?
34
Service Discovery
With Docker Swarm it is rather easy
to spin up a Kubernetes or Mesos cluster within Swarm.
35
Orchestration Frameworks
SERVER0 SERVER1 SERVER<n>
Docker-Engine Docker-Engine Docker-Engine
swarm-client swarm-client swarm-client
swarm-master
etcd
kubelet
scheduler apiserver
etcd
kubelet
etcd
kubelet
1. Containers should be controlled via ENV or flags
External access/change of a running container is discouraged
2. Configuration management
Downgraded to bootstrap a host?
36
Immutable vs. Config Mgmt
If containers are immutable within pipeline
testing/deployment should be automated
developers should have a production replica
37
Continuous Dev./Integration
38
Docker Momentum
Software Dev
DatacenterOps
IT Tinkering (Hello World)
Continuous Dev/Int/Dep
Microservices, hyper scale
Big Data
High Performance Computing
HPC
Disclaimer: subjective exaggeration
Spinning up production-like environment is great
MongoDB, PostreSQL, memcached as separate containers
python2.7, python3.4
39
Docker in Software Development
Like python’s virtualenv on steroids,
iteration speedup through reproducibility
Spinning up production-like environment is…
…not that easy
focus more on engineer/scientist, not the software-developer
1. For development it might work
close to non-HPC software dev
2. But is that the iteration-focus?
rather job settings / input data?
40
Docker in HPC development
Split input iteration / development from operation
non-distributed stays vanilla
transition to HPC cluster using tech to foster operation
41
Separation of Concerns?
http://gmkurtzer.github.io/singularity
Input/Dev
Docker-Engine 1.11 will not be the parent of containers
runC usage under the hood
42
containerd Integration
1. Separat Dev and Ops
don’t block the momentum fostering 

iteration speed in Development 
2. Using vanilla docker-tech
keep up with the ecosystem and prevent vendor/ecosystem
lock-in
3. 80/20 rule
have caveats on the radar but don’t bother too much
everything is so fast moving - it’s hard to predict
43
Recap aka. IMHO
Q&A
https://github.com/qnib/hpcac-cluster2016
http://qnib.org
eGalea Workshop (Pisa)

<plz ping me if you are interested>
23.06.2016

Contenu connexe

Tendances

Introduction to Red Hat
Introduction to Red HatIntroduction to Red Hat
Introduction to Red HatAlbert Wong
 
Introduction to the Helium release of OpenDaylight
Introduction to the Helium release of OpenDaylightIntroduction to the Helium release of OpenDaylight
Introduction to the Helium release of OpenDaylightSDN Hub
 
VMware NSX + Cumulus Networks: Software Defined Networking
VMware NSX + Cumulus Networks: Software Defined NetworkingVMware NSX + Cumulus Networks: Software Defined Networking
VMware NSX + Cumulus Networks: Software Defined NetworkingCumulus Networks
 
Building Resilient Applications with Cloudflare DNS
Building Resilient Applications with Cloudflare DNSBuilding Resilient Applications with Cloudflare DNS
Building Resilient Applications with Cloudflare DNSDevOps.com
 
VMworld 2015: VMware NSX Deep Dive
VMworld 2015: VMware NSX Deep DiveVMworld 2015: VMware NSX Deep Dive
VMworld 2015: VMware NSX Deep DiveVMworld
 
OpenSolaris Introduction
OpenSolaris IntroductionOpenSolaris Introduction
OpenSolaris Introductionsatyajit_t
 
SDN Service Provider use cases Network Function Virtualization (NFV)
SDN Service Provider use cases Network Function Virtualization (NFV)SDN Service Provider use cases Network Function Virtualization (NFV)
SDN Service Provider use cases Network Function Virtualization (NFV)Brent Salisbury
 
Virtualization technolegys for amdocs
Virtualization technolegys for amdocsVirtualization technolegys for amdocs
Virtualization technolegys for amdocsSamuel Dratwa
 
VMware NSX - Lessons Learned from real project
VMware NSX - Lessons Learned from real projectVMware NSX - Lessons Learned from real project
VMware NSX - Lessons Learned from real projectDavid Pasek
 
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)SDN Scale-out Testing at OpenStack Innovation Center (OSIC)
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)PLUMgrid
 
NSX Reference Design version 3.0
NSX Reference Design version 3.0NSX Reference Design version 3.0
NSX Reference Design version 3.0Doddi Priyambodo
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSSteve Wong
 
Presentation cloud orchestration
Presentation   cloud orchestrationPresentation   cloud orchestration
Presentation cloud orchestrationxKinAnx
 
Cisco UCS for OpenStack Cloud
Cisco UCS for OpenStack CloudCisco UCS for OpenStack Cloud
Cisco UCS for OpenStack CloudLora O'Haver
 
DPDK IPSec performance benchmark ~ Georgii Tkachuk
DPDK IPSec performance benchmark ~ Georgii TkachukDPDK IPSec performance benchmark ~ Georgii Tkachuk
DPDK IPSec performance benchmark ~ Georgii TkachukIntel
 
OpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get StartedOpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get StartedAll Things Open
 

Tendances (20)

Cloud Computing using OpenStack
Cloud Computing using OpenStackCloud Computing using OpenStack
Cloud Computing using OpenStack
 
Introduction to Red Hat
Introduction to Red HatIntroduction to Red Hat
Introduction to Red Hat
 
Introduction to the Helium release of OpenDaylight
Introduction to the Helium release of OpenDaylightIntroduction to the Helium release of OpenDaylight
Introduction to the Helium release of OpenDaylight
 
VMware NSX + Cumulus Networks: Software Defined Networking
VMware NSX + Cumulus Networks: Software Defined NetworkingVMware NSX + Cumulus Networks: Software Defined Networking
VMware NSX + Cumulus Networks: Software Defined Networking
 
Building Resilient Applications with Cloudflare DNS
Building Resilient Applications with Cloudflare DNSBuilding Resilient Applications with Cloudflare DNS
Building Resilient Applications with Cloudflare DNS
 
BRKSDN-2115
BRKSDN-2115 BRKSDN-2115
BRKSDN-2115
 
VMworld 2015: VMware NSX Deep Dive
VMworld 2015: VMware NSX Deep DiveVMworld 2015: VMware NSX Deep Dive
VMworld 2015: VMware NSX Deep Dive
 
OpenSolaris Introduction
OpenSolaris IntroductionOpenSolaris Introduction
OpenSolaris Introduction
 
SDN Service Provider use cases Network Function Virtualization (NFV)
SDN Service Provider use cases Network Function Virtualization (NFV)SDN Service Provider use cases Network Function Virtualization (NFV)
SDN Service Provider use cases Network Function Virtualization (NFV)
 
Virtualization technolegys for amdocs
Virtualization technolegys for amdocsVirtualization technolegys for amdocs
Virtualization technolegys for amdocs
 
VMware NSX - Lessons Learned from real project
VMware NSX - Lessons Learned from real projectVMware NSX - Lessons Learned from real project
VMware NSX - Lessons Learned from real project
 
Cont0519
Cont0519Cont0519
Cont0519
 
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)SDN Scale-out Testing at OpenStack Innovation Center (OSIC)
SDN Scale-out Testing at OpenStack Innovation Center (OSIC)
 
NSX Reference Design version 3.0
NSX Reference Design version 3.0NSX Reference Design version 3.0
NSX Reference Design version 3.0
 
Opening Up Your Network with SDN
Opening Up Your Network with SDNOpening Up Your Network with SDN
Opening Up Your Network with SDN
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS
 
Presentation cloud orchestration
Presentation   cloud orchestrationPresentation   cloud orchestration
Presentation cloud orchestration
 
Cisco UCS for OpenStack Cloud
Cisco UCS for OpenStack CloudCisco UCS for OpenStack Cloud
Cisco UCS for OpenStack Cloud
 
DPDK IPSec performance benchmark ~ Georgii Tkachuk
DPDK IPSec performance benchmark ~ Georgii TkachukDPDK IPSec performance benchmark ~ Georgii Tkachuk
DPDK IPSec performance benchmark ~ Georgii Tkachuk
 
OpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get StartedOpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get Started
 

En vedette

Presentacin webinar move_up_to_power8_with_scale_out_servers_final
Presentacin webinar move_up_to_power8_with_scale_out_servers_finalPresentacin webinar move_up_to_power8_with_scale_out_servers_final
Presentacin webinar move_up_to_power8_with_scale_out_servers_finalDiego Alberto Tamayo
 
Oracle Solaris Cloud Management and Deployment with OpenStack
Oracle Solaris Cloud Management and Deployment with OpenStackOracle Solaris Cloud Management and Deployment with OpenStack
Oracle Solaris Cloud Management and Deployment with OpenStackOTN Systems Hub
 
Understanding the IBM Power Systems Advantage
Understanding the IBM Power Systems AdvantageUnderstanding the IBM Power Systems Advantage
Understanding the IBM Power Systems AdvantageIBM Power Systems
 
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3OTN Systems Hub
 
Co-Design Architecture for Exascale
Co-Design Architecture for ExascaleCo-Design Architecture for Exascale
Co-Design Architecture for Exascaleinside-BigData.com
 
Solaris 11.2 What's New
Solaris 11.2 What's NewSolaris 11.2 What's New
Solaris 11.2 What's NewOrgad Kimchi
 
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Accelerating Business Intelligence Solutions with Microsoft Azure   passAccelerating Business Intelligence Solutions with Microsoft Azure   pass
Accelerating Business Intelligence Solutions with Microsoft Azure passJason Strate
 
Georgia Azure Event - Scalable cloud games using Microsoft Azure
Georgia Azure Event - Scalable cloud games using Microsoft AzureGeorgia Azure Event - Scalable cloud games using Microsoft Azure
Georgia Azure Event - Scalable cloud games using Microsoft AzureMicrosoft
 
OpenPOWER Roadmap Toward CORAL
OpenPOWER Roadmap Toward CORALOpenPOWER Roadmap Toward CORAL
OpenPOWER Roadmap Toward CORALinside-BigData.com
 
Oracle Solaris Software Integration
Oracle Solaris Software IntegrationOracle Solaris Software Integration
Oracle Solaris Software IntegrationOTN Systems Hub
 
Open Innovation with Power Systems
Open Innovation with Power Systems Open Innovation with Power Systems
Open Innovation with Power Systems IBM Power Systems
 
Expert summit SQL Server 2016
Expert summit   SQL Server 2016Expert summit   SQL Server 2016
Expert summit SQL Server 2016Łukasz Grala
 
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...Mark Rittman
 
Oracle Solaris Build and Run Applications Better on 11.3
Oracle Solaris  Build and Run Applications Better on 11.3Oracle Solaris  Build and Run Applications Better on 11.3
Oracle Solaris Build and Run Applications Better on 11.3OTN Systems Hub
 

En vedette (20)

OpenPOWER Update
OpenPOWER UpdateOpenPOWER Update
OpenPOWER Update
 
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
Presentacin webinar move_up_to_power8_with_scale_out_servers_finalPresentacin webinar move_up_to_power8_with_scale_out_servers_final
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
 
Oracle Solaris Cloud Management and Deployment with OpenStack
Oracle Solaris Cloud Management and Deployment with OpenStackOracle Solaris Cloud Management and Deployment with OpenStack
Oracle Solaris Cloud Management and Deployment with OpenStack
 
Understanding the IBM Power Systems Advantage
Understanding the IBM Power Systems AdvantageUnderstanding the IBM Power Systems Advantage
Understanding the IBM Power Systems Advantage
 
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3
Oracle Solaris Simple, Flexible, Fast: Virtualization in 11.3
 
Co-Design Architecture for Exascale
Co-Design Architecture for ExascaleCo-Design Architecture for Exascale
Co-Design Architecture for Exascale
 
Solaris 11.2 What's New
Solaris 11.2 What's NewSolaris 11.2 What's New
Solaris 11.2 What's New
 
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Accelerating Business Intelligence Solutions with Microsoft Azure   passAccelerating Business Intelligence Solutions with Microsoft Azure   pass
Accelerating Business Intelligence Solutions with Microsoft Azure pass
 
Georgia Azure Event - Scalable cloud games using Microsoft Azure
Georgia Azure Event - Scalable cloud games using Microsoft AzureGeorgia Azure Event - Scalable cloud games using Microsoft Azure
Georgia Azure Event - Scalable cloud games using Microsoft Azure
 
OpenPOWER Roadmap Toward CORAL
OpenPOWER Roadmap Toward CORALOpenPOWER Roadmap Toward CORAL
OpenPOWER Roadmap Toward CORAL
 
IBM POWER8 as an HPC platform
IBM POWER8 as an HPC platformIBM POWER8 as an HPC platform
IBM POWER8 as an HPC platform
 
Bitcoin explained
Bitcoin explainedBitcoin explained
Bitcoin explained
 
Blockchain
BlockchainBlockchain
Blockchain
 
Oracle Solaris Software Integration
Oracle Solaris Software IntegrationOracle Solaris Software Integration
Oracle Solaris Software Integration
 
Open Innovation with Power Systems
Open Innovation with Power Systems Open Innovation with Power Systems
Open Innovation with Power Systems
 
IBM Power8 announce
IBM Power8 announceIBM Power8 announce
IBM Power8 announce
 
Expert summit SQL Server 2016
Expert summit   SQL Server 2016Expert summit   SQL Server 2016
Expert summit SQL Server 2016
 
Puppet + Windows Nano Server
Puppet + Windows Nano ServerPuppet + Windows Nano Server
Puppet + Windows Nano Server
 
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...
IlOUG Tech Days 2016 - Big Data for Oracle Developers - Towards Spark, Real-T...
 
Oracle Solaris Build and Run Applications Better on 11.3
Oracle Solaris  Build and Run Applications Better on 11.3Oracle Solaris  Build and Run Applications Better on 11.3
Oracle Solaris Build and Run Applications Better on 11.3
 

Similaire à The State of Linux Containers

State of Linux Containers for HPC
State of Linux Containers for HPCState of Linux Containers for HPC
State of Linux Containers for HPCinside-BigData.com
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Henryk Konsek
 
Evolution of Linux Containerization
Evolution of Linux Containerization Evolution of Linux Containerization
Evolution of Linux Containerization WSO2
 
Evoluation of Linux Container Virtualization
Evoluation of Linux Container VirtualizationEvoluation of Linux Container Virtualization
Evoluation of Linux Container VirtualizationImesh Gunaratne
 
Containers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container ServicesContainers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container ServicesNEXTtour
 
Docker Devoxx UK - Never mind the bollocks here's the Linux Containers
Docker Devoxx UK - Never mind the bollocks here's the Linux ContainersDocker Devoxx UK - Never mind the bollocks here's the Linux Containers
Docker Devoxx UK - Never mind the bollocks here's the Linux ContainersPatrick Chanezon
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryImesh Gunaratne
 
Revolutionizing the cloud with container virtualization
Revolutionizing the cloud with container virtualizationRevolutionizing the cloud with container virtualization
Revolutionizing the cloud with container virtualizationWSO2
 
Accelerate your development with Docker
Accelerate your development with DockerAccelerate your development with Docker
Accelerate your development with DockerAndrey Hristov
 
Accelerate your software development with Docker
Accelerate your software development with DockerAccelerate your software development with Docker
Accelerate your software development with DockerAndrey Hristov
 
Academy PRO: Docker. Lecture 1
Academy PRO: Docker. Lecture 1Academy PRO: Docker. Lecture 1
Academy PRO: Docker. Lecture 1Binary Studio
 
Linux container & docker
Linux container & dockerLinux container & docker
Linux container & dockerejlp12
 
Weave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapWeave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapPatrick Chanezon
 
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael CrosbyDocker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael CrosbyDocker, Inc.
 
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby Michelle Antebi
 
Docker at the gate
Docker at the gateDocker at the gate
Docker at the gateTony Deng
 
Evolution of containers to kubernetes
Evolution of containers to kubernetesEvolution of containers to kubernetes
Evolution of containers to kubernetesKrishna-Kumar
 

Similaire à The State of Linux Containers (20)

State of Linux Containers for HPC
State of Linux Containers for HPCState of Linux Containers for HPC
State of Linux Containers for HPC
 
Dockers zero to hero
Dockers zero to heroDockers zero to hero
Dockers zero to hero
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.
 
Kubernetes Basics
Kubernetes BasicsKubernetes Basics
Kubernetes Basics
 
Evolution of Linux Containerization
Evolution of Linux Containerization Evolution of Linux Containerization
Evolution of Linux Containerization
 
Evoluation of Linux Container Virtualization
Evoluation of Linux Container VirtualizationEvoluation of Linux Container Virtualization
Evoluation of Linux Container Virtualization
 
Containers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container ServicesContainers and Nutanix - Acropolis Container Services
Containers and Nutanix - Acropolis Container Services
 
Docker Devoxx UK - Never mind the bollocks here's the Linux Containers
Docker Devoxx UK - Never mind the bollocks here's the Linux ContainersDocker Devoxx UK - Never mind the bollocks here's the Linux Containers
Docker Devoxx UK - Never mind the bollocks here's the Linux Containers
 
Docker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshitDocker meetup-20-apr-17-openshit
Docker meetup-20-apr-17-openshit
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
 
Revolutionizing the cloud with container virtualization
Revolutionizing the cloud with container virtualizationRevolutionizing the cloud with container virtualization
Revolutionizing the cloud with container virtualization
 
Accelerate your development with Docker
Accelerate your development with DockerAccelerate your development with Docker
Accelerate your development with Docker
 
Accelerate your software development with Docker
Accelerate your software development with DockerAccelerate your software development with Docker
Accelerate your software development with Docker
 
Academy PRO: Docker. Lecture 1
Academy PRO: Docker. Lecture 1Academy PRO: Docker. Lecture 1
Academy PRO: Docker. Lecture 1
 
Linux container & docker
Linux container & dockerLinux container & docker
Linux container & docker
 
Weave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapWeave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 Recap
 
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael CrosbyDocker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
 
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby
 
Docker at the gate
Docker at the gateDocker at the gate
Docker at the gate
 
Evolution of containers to kubernetes
Evolution of containers to kubernetesEvolution of containers to kubernetes
Evolution of containers to kubernetes
 

Plus de inside-BigData.com

Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...inside-BigData.com
 
Transforming Private 5G Networks
Transforming Private 5G NetworksTransforming Private 5G Networks
Transforming Private 5G Networksinside-BigData.com
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...inside-BigData.com
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...inside-BigData.com
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...inside-BigData.com
 
HPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural NetworksHPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural Networksinside-BigData.com
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean MonitoringBiohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoringinside-BigData.com
 
Machine Learning for Weather Forecasts
Machine Learning for Weather ForecastsMachine Learning for Weather Forecasts
Machine Learning for Weather Forecastsinside-BigData.com
 
HPC AI Advisory Council Update
HPC AI Advisory Council UpdateHPC AI Advisory Council Update
HPC AI Advisory Council Updateinside-BigData.com
 
Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19inside-BigData.com
 
Energy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic TuningEnergy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic Tuninginside-BigData.com
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODHPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODinside-BigData.com
 
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud AccelerationVersal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Accelerationinside-BigData.com
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance EfficientlyZettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficientlyinside-BigData.com
 
Scaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's EraScaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's Erainside-BigData.com
 
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computingCUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computinginside-BigData.com
 
Introducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi ClusterIntroducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi Clusterinside-BigData.com
 

Plus de inside-BigData.com (20)

Major Market Shifts in IT
Major Market Shifts in ITMajor Market Shifts in IT
Major Market Shifts in IT
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...
 
Transforming Private 5G Networks
Transforming Private 5G NetworksTransforming Private 5G Networks
Transforming Private 5G Networks
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
 
HPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural NetworksHPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural Networks
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean MonitoringBiohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
 
Machine Learning for Weather Forecasts
Machine Learning for Weather ForecastsMachine Learning for Weather Forecasts
Machine Learning for Weather Forecasts
 
HPC AI Advisory Council Update
HPC AI Advisory Council UpdateHPC AI Advisory Council Update
HPC AI Advisory Council Update
 
Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19
 
Energy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic TuningEnergy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic Tuning
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODHPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
 
State of ARM-based HPC
State of ARM-based HPCState of ARM-based HPC
State of ARM-based HPC
 
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud AccelerationVersal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Acceleration
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance EfficientlyZettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
 
Scaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's EraScaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's Era
 
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computingCUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computing
 
Introducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi ClusterIntroducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi Cluster
 
Overview of HPC Interconnects
Overview of HPC InterconnectsOverview of HPC Interconnects
Overview of HPC Interconnects
 

Dernier

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Dernier (20)

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 

The State of Linux Containers

  • 1. ssThe State of Linux Containers
  • 4. 1. “Linux Container” / “Docker Ecosystem” in a Nutshell 2. Confusion about Ecosystem / Vision to tackle it 3. Docker -> SWARM -> SLURM 
 -> BigData 4. Discussion of Opportunities and Problems 4 Agenda
  • 5. The Bits and Pieces…
  • 6. Userland (OS)Userland (OS) Userland (OS) Userland (OS) Ubuntu:14.04 Ubuntu:15.10 RHEL7.2 Tiny Core Linux Linux Containers 6 SERVER HOST KERNEL HYPERVISOR KERNEL SERVICE Userland (OS) KERNEL KERNEL Userland (OS)Userland (OS) Userland (OS) SERVICE SERVICE SERVER HOST KERNEL SERVICE SERVICE SERVICE Traditional Virtualisation Containerisation Containers do not spin up a distinct kernel all containers & the host share the same user-lands are independent they are separated by Kernel Namespaces
  • 7. Containers are ‘grouped processes’ isolated by Kernel Namespaces resource restrictions applicable through CGroups (disk/netIO) HOST container1 7 Kernel Namespaces bash ls -l container2 apache container3 mysqld consul consul PIDNamespaces: Network Mount IPC UTS container4 slurmd ssh consul
  • 8. Container Runtime Daemon creates/…/removes containers, exposes REST API handles Namespaces, CGroups, bind-mounts, etc. IP connectivity by default via ‘host-only’ network bridge Docker Engine 8 SERVER eth0 docker0 container1 container2 Docker-Engine
  • 9. Docker Compose 9 Describes stack of container configurations instead of writing a small bash script… … it holds the runtime configuration as YAML file.
  • 10. Docker Networking spans networks across engines KV-store to synchronise (Zookeeper, etcd, Consul) VXLAN to pass messages along SERVER0 SERVER1 SERVER<n> Docker Networking 10 Consul Docker-Engine Consul Consul Docker-Engine Docker-Engine Consul DC global container0 container1 containerN
  • 11. Docker Swarm proxies docker-engines serves an API endpoint in front of multiple docker-engines does placement decisions. SERVER0 SERVER1 SERVER<n> Docker Swarm 11 Docker-Engine Docker-Engine Docker-Engine swarm-client swarm-client swarm-client swarm-master :2376 :2376 :2376 :2375 container1 -e constraint:node==SERVER0
  • 12. Docker Swarm [cont] 12 query docker-enginequery docker-swarm
  • 14. Introducing new Tech 14 Self-perception when introducing new tech… credit: TF2 - Meet the Pyro
  • 15. Introducing new Tech 15 … not always the same as the perception of others. credit: TF2 - Meet the Pyro
  • 16. Docker Buzzword Chaos! Distributions Solutions Auto-Scaling On-Premise & OverSpill Orchestration self-healing 16 production-ready enterprise-grade
  • 17. 1. No special distributions useful for certain use-cases, such as elasticity and green-field deployment not so much for an on-premise datacenter w/ legacy in it. 2. Leverage existing processes/resources install workflow, syslog, monitoring security (ssh infrastructure), user auth. 3. keep up with docker ecosystem incorporate new features of engine, swarm, compose networking, volumes, user-namespaces 17 Vision
  • 19. Hardware (courtesy of ) 8x Sun Fire x2250, 2x 4core XEON, 32GB, Mellanox ConnectX-2) Software Base installation CentOS 7.2 base installation (updated from 7-alpha) Ansible consul, sensu docker v1.10, docker-compose docker SWARM 19 Testbed
  • 20. node1 node2 node8 20 Docker Networking Synchronised by Consul Consul Consul
 DC Consul Consul Docker-Engine Docker-Engine Docker-Engine
  • 21. node1 node2 node8 21 Docker SWARM Docker SWARM Synchronised by Consul KV- store Consul Consul
 DC Consul Consul Docker-Engine Docker-Engine Docker-Engine swarm swarm SWARM swarm master
  • 22. node8 node2 node1 22 SLURM Cluster Consul Consul
 DC Consul Consul SLURM within SWARM slurmctld slurmd slurmd slurmd Docker-Engine Docker-Engine Docker-Engine swarm swarm SWARM swarm master SLURM
  • 24. node8 node2 node1 24 SLURM Cluster [cont] Consul Consul
 DC Consul Consul SLURM within SWARM slurmd within app-container pre-stage containers slurmctld slurmd slurmd slurmd Docker-Engine Docker-Engine Docker-Engine swarm swarm hpcg hpcg SWARM hpcg swarm master SLURM
  • 26. node8 node2 node1 26 SLURM Cluster [cont] Consul Consul
 DC Consul Consul SLURM within SWARM slurmd within app-container pre-stage containers slurmctld slurmd slurmd slurmd Docker-Engine Docker-Engine Docker-Engine swarm swarm hpcg hpcg SWARM hpcg OpenFOAM OpenFOAM OpenFOAM swarm master SLURM
  • 28. node1 node2 node8 28 Samza Cluster Consul Consul
 DC Consul Consul Distributed Samza Zookeeper and Kafka cluster Samza instances to run jobs Docker-Engine Docker-Engine Docker-Engine swarm swarm SWARM swarm masterzookeeper zookeeper zookeeper kafka kafka kafka samza samza samza $ cat test.log |awk ‘{print $1}’ |sed -e ’s/HPC/BigData/g’ |tee out.log
  • 30. 1. Where to base images on? Ubuntu/Fedora: ~200MB Debian: ~100MB Alpine Linux: 5MB (musl-libc) 2. Trimm the Images down at all cost? How about debugging tools? Possibility to run tools on the host and ‘inspect’ namespaced processes inside of a container. If PID-sharing arrives, carving out (e.g.) monitoring could be a thing. 30 Small vs. Big
  • 31. 1. In an ideal world… a container only runs one process, e.g. the HPC solver. 2. In reality… MPI want’s to connect to a sshd within the job-peers monitoring, syslog, service discovery should be present as well. 3. How fast / aggressive to break traditional approaches? 31 One vs. Many Processes
  • 32. Plugin System VXLAN MACVLAN How about IPoIB? 32 Docker Network
  • 33. Running OpenFOAM on small scale is cumbersome manually install OpenFOAM on a workstation be confident that the installation works correctly A containerised OpenFOAM installation tackles both 33 Reproducibility / Downscaling http://qnib.org/immutablehttp://qnib.org/immutable-paper
  • 34. 1. Since the environments are rather dynamic… how does the containers discover services? external registry as part of the framework? discovery service as part of the container stacks? 34 Service Discovery
  • 35. With Docker Swarm it is rather easy to spin up a Kubernetes or Mesos cluster within Swarm. 35 Orchestration Frameworks SERVER0 SERVER1 SERVER<n> Docker-Engine Docker-Engine Docker-Engine swarm-client swarm-client swarm-client swarm-master etcd kubelet scheduler apiserver etcd kubelet etcd kubelet
  • 36. 1. Containers should be controlled via ENV or flags External access/change of a running container is discouraged 2. Configuration management Downgraded to bootstrap a host? 36 Immutable vs. Config Mgmt
  • 37. If containers are immutable within pipeline testing/deployment should be automated developers should have a production replica 37 Continuous Dev./Integration
  • 38. 38 Docker Momentum Software Dev DatacenterOps IT Tinkering (Hello World) Continuous Dev/Int/Dep Microservices, hyper scale Big Data High Performance Computing HPC Disclaimer: subjective exaggeration
  • 39. Spinning up production-like environment is great MongoDB, PostreSQL, memcached as separate containers python2.7, python3.4 39 Docker in Software Development Like python’s virtualenv on steroids, iteration speedup through reproducibility
  • 40. Spinning up production-like environment is… …not that easy focus more on engineer/scientist, not the software-developer 1. For development it might work close to non-HPC software dev 2. But is that the iteration-focus? rather job settings / input data? 40 Docker in HPC development
  • 41. Split input iteration / development from operation non-distributed stays vanilla transition to HPC cluster using tech to foster operation 41 Separation of Concerns? http://gmkurtzer.github.io/singularity Input/Dev
  • 42. Docker-Engine 1.11 will not be the parent of containers runC usage under the hood 42 containerd Integration
  • 43. 1. Separat Dev and Ops don’t block the momentum fostering 
 iteration speed in Development  2. Using vanilla docker-tech keep up with the ecosystem and prevent vendor/ecosystem lock-in 3. 80/20 rule have caveats on the radar but don’t bother too much everything is so fast moving - it’s hard to predict 43 Recap aka. IMHO