Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2 (NET410) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workshop: Deep Dive on Container Networking
at Scale on Amazon EKS, Amazon ECS, &
Amazon EC2
N E T 4 1 0
Liwen Wu
Software Development Engineer
Amazon Web Services
Ikenna Izugbokwe
Sr. Solutions Architect
Amazon Web Services
Pratik Mankad
Partner Solutions Architect
Amazon Web Services

Agenda
• Workshop environment
• ECS networking
• Kubernetes:
• Key concepts and architecture
• Networking requirements
• Networking implementation
• Amazon EKS:
• EC2 and VPC considerations
• CNI details
• Kubernetes on EC2: Kops
• Workshop activity: Amazon EKS & Kubernetes on EC2 (Kops)

Workshop Environment Setup…
• Launch CloudFormation template:
• GitHub: https://github.com/liwenwu -amazon/reinvent2018-NET410
• CloudFormation template creates:
• Two Amazon EC2 instances (workshop instances)
• Sets up environment
• Two kubernetes clusters:
1. Kops Cluster
2. EKS Cluster
• Two EC2 instances for workshop activities:
• net410-workshop-kops-mgmt
• net410-workshop-eks-mgmt

Deployment Options
Amazon EC2
Kubernetes
+

ECS networkMode: none
EC2
Task 1
lo
eth0
10.1.1.11
127.0.0.1
• Disables all networking
• IP for the container not
configured
• No access to the external
network as well as for other
containers.
• Has the loopback address.
docker0: 172.17.0.1/16

ECS networkMode: host
EC2
Task 1
lo
eth0
10.1.1.11
127.0.0.1
• Shares the host’s
network stack
• All interfaces from the
host will be available
to the container
• Host name will match
the host name on the
host system
• No network isolation
eth0 docker0
10.1.1.11 172.17.0.1
docker0: 172.17.0.1/16

ECS networkMode: bridge
EC2
Task 1
eth0
docker0: 172.17.0.1/16eth0
10.1.1.11
172.17.0.2:1234
• Uses software bridge:
docker0
• Containers are attached to
docker bridge
• Own IP address and all TCP
ports are available
• No network isolation
between containers
vethxx
Task 2
eth0
172.17.0.3:4567
vethxx

ECS networkMode: bridge
• Container on across EC2 instances…
EC2-1 EC2-2
docker0: 172.17.0.1/16
eth0
10.1.1.11
vethxx
eth0
172.17.0.2:1234
docker0: 172.17.0.1/16
eth0
10.1.2.11
vethxx
eth0
172.17.0.2:4567
Router
Task 1 Task 1

ECS networkMode: awsvpc
EC2
• Each task has it’s own
elastic network
interface
• Achieves network isolation
• Security group on the
primary and task ENI
• Container gets its own
addressable IP
ecs-bridge: 169.254.172.1/30
eth0
10.1.1.11
vethxx
ecs-eth0
169.254.172.2/30
Task 1
eth1
10.1.1.12

ECS networkMode: awsvpc
EC2-1
10.1.1.12:1234
eth0
eth1
10.1.1.11
EC2-2
10.1.2.12:1234
eth0
eth1
10.1.2.11

Key Kubernetes concepts:
• Kubernetes Control Plane
• Provided by Master Node Objects/processes
• Kubernetes Data Plane
• Provided by Worker Nodes Objects/processes
• Kubernetes Master Node
• Kube-apiserver
• Kube-controller-manager
• Kube-scheduler
• Kubernetes Worker Nodes
• Kubelet
• Kube-proxy
• Pods

Kubernetes Master Node

Kubernetes Worker Node

Kubernetes objects and examples
• Represent the state of the cluster
• They are records of “intent” or cluster desired state.
• Acted upon by calling k8s API directly from client libraries or via cli –
kubectl
• Identified by a name and a UID
• Labels
• Key/value pair attached to objects. Used by service to define sets of pods (key use case).
• Annotations
• Control load balancer configuration in EKS.

Kubernetes architecture:
Cloud

Four networking problems:
• Container-to-container communications
• Pod-to-pod communications
• Pod-to-service communications
• External-to-internal communications

What is a pod ...
• Smallest and simplest computing unit
• Group of one or more containers
• Co-located and co-scheduled
• Share a network stack and storage
• Containers within a pod share an IP address

From Kubernetes perspective …
• Pods can communicate with other pods
• Every pod gets its own IP address
• Mapping container ports to node (host) port not required

Kubernetes CNI’s (plugins)
• Kubenet (default for kops)
• Utilizes VPC route table for pod -to-pod traffic between nodes
• Calico
• Uses BGP to distribute routes
• Weave
• All nodes in cluster share same subnet—single L2 broadcast domain
• Flannel
• Leverages VXLAN overlay network
• Amazon-vpc-cni-k8s
• Amazon routed ENI VPC CNI

EC2 and VPC considerations—EKS
• AWS EC2 instance type:
• Determine the maximum number of Elastic Network Interfaces ( ENI)
• Determine the number of IP addresses
• Larger instances support more IP addresses:
• c5d.18xlarge supports 15 ENIs and each ENI can support up -to 50 IPv4
addresses
• EKS requires subnets in at least two AZ’s
• Use separate VPC for each EKS cluster
• Warm pool of IP addresses

amazon-vpc-cni-k8s Container Networking
Interface (CNI) Tenets:
• Integrates Amazon Virtual Private Cloud networking into Kubernetes
• Should use Amazon VPC networking natively to forward pod-to-pod
traffic
• Use AWS routable IP addresses for Pods
• Pods is 1st class citizen in Amazon VPC networking
• There is NO on-ramp/off-ramp for
• Pod to AWS services (e.g. Amazon S3, Amazon DynamoDB)
communication
• Pod to on-premises communication (e.g. VPN/direct -connect)
• Should make sure pods have fast startup time:
• Pods/Containers MUST be able to send and receive traffic in the matter
of seconds (compare to minutes for VM)

CNI networking details—Data plane
VPC network
EC2
POD
L3 routing
POD
L3 routing
POD POD
L3 routing
POD
ENI ENI ENI

Life of a packet: pod1-to-pod2, inside node
EC2
Pod1
eth0
Pod2
eth0
ENI
root
veth-pod1 veth-pod2

Life of a packet: pod1-to-pod3, across nodes
ENI
EC2
node1
Pod1
eth0
Pod2
eth0
root
veth-pod1 veth-pod2
EC2
node2
Pod3
eth0
Pod4
eth0
root
veth-pod3 veth-pod4

Kubernetes Service
Service
PodPodPod

For example: Kubernetes Service IP (10.100.0.1)
kubectl describe svc
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.100.0.1
Port: https 443/TCP
TargetPort: 443/TCP
Endpoints: 192.168.119.102:443,192.168
.154.135:443
Session Affinity: ClientIP
Events: <none>

Kubernetes DNS Pod (kube-dns)
• Kubernetes schedules a DNS Pod (kube-dns) and Service on
the cluster
• Configures the Kubelets to tell individual containers to use
the DNS Service’s IP to resolve DNS names
• DNS Pod communicate with Kubernetes Service IP (e.g
10.100.0.1) and build map of Service Name and Service IP

Life of a packet: pod (kube-dns)-to-service
(Kubernetes service)
ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
veth-dns-pod veth-pod2
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135I
Customer VPC EKS VPC
10.100.0.1
APIServer
Availability Zone 1
APIServer
Availability Zone 2

ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135
APIServer
Availability Zone 1
APIServer
Availability Zone 2

ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135I
APIServer
Availability Zone 1
APIServer
Availability Zone 2

CNI networking details—Control plane
• Kubelet invokes CNI add or delete commands for pods
• CNI request secondary IPs from ipamD and setups
networking stack for pod
• For fast pods startup time, IP address manager database
(ipamD) creates a secondary IP warm pool with one more
ENI and its IP address

CNI Networking details—Control plane
EC2

Networking explained …
eth0 eth0
vethxxxx vethxxxx
eth0
crb0
EC2

Pod-to-Pod Communications (kops):
EC2
Pod 1
eth0 eth0
vethxxxxvethxxxx
crb0: 100.65.129.1eth0
10.0.1.11
• Pods on the same instance…
Pod 2
100.65.129.11 100.65.129.12

Pod-to-Pod Communications (kops):
• Pods across EC2 instances…
EC2-1
eth0
10.1.1.11
crb0: 100.65.130.1
eth0Pod 1
vethxxxx
100.65.130.11
EC2-2
eth0
10.1.2.11
crb0: 100.65.129.1
eth0Pod 2
vethxxxx
100.65.129.11
Router (GW)
100.65.129.0/24
via eni: 10.0.1.11
100.65.130.0/24
via eni: 10.0.1.12

Pod-to-Service Communications (kops):
EC2-1
eth0
10.0.1.11
crb0: 100.65.130.1
eth0Pod 1
vethxxxx
100.65.130.11
Router (GW)
100.65.130.0/24
via eni: 10.1.1.11
100.65.129.0/24
via eni: 10.1.2.11
kube-proxy
Connect To:
100.65.130.11:8080
iptables:
service xx
endpoints yy
EC2-2
eth0
10.0.1.11
crb0: 100.65.130.1
eth0Pod 2
vethxxxx
100.65.130.11
kube-proxy
SERVICE:
NAME: service-net410
TYPE: ClusterIP
CLUSTER-IP: 100.64.31.116
EXTERNAL-IP: <none>
PORTS: 80/TCP

External-to-internal communications (kops):
VPC CIDR: 10.0.0.0/16
eth0
vethxxxx vethxxxx
crb0: 100.64.129.1
eth0
Router: 10.0.1.1
eth0: 10.0.1.11
100.64.129.0/24
via eni: 10.0.1.11
100.64.130.0/24
via eni: 10.0.1.12
100,64.129.11
eth0
100,64.129.12
kube-proxy
Iptables
service xx
endpoints yy
EC2-2
eth0
vethxxxx vethxxxx
crb0: 100.64.129.1
eth0
eth0: 10.0.1.12
100,64.130.11
eth0
100,64.130.12
SERVICE:
NAME: service-net410
TYPE: LoadBalancer
CLUSTER-IP: 100.64.31.116
EXTERNAL-IP: LB DNS
PORTS: 80:31787/TCP kube-proxy
Iptables
service xx
endpoints yy
EC2-1
Internet
EC2-1
EC2-2
ELB DNS: a266xxyy.us-west-2.elb.amazonaws.com

NET410 workshop activity
• To begin workshop activity:
• Access https://github.com/liwenwu -amazon/reinvent2018-NET410

References:
• https://kubernetes.io/
• https://kubernetes.io/docs/concepts/cluster-
administration/networking/
• https://kubernetes.io/docs/concepts/services-networking/service/
• https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-
kops/
• https://github.com/aws/amazon-vpc-cni-k8s

Thank you!
Ikenna Izugbokwe
Sr. Solutions Architect
Amazon Web Services
Liwen Wu
Software Development Engineer
Amazon Web Services
Pratik Mankad
Partner Solutions Architect
Amazon Web Services

Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2 (NET410) - AWS re:Invent 2018

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2 (NET410) - AWS re:Invent 2018

Similaire à Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2 (NET410) - AWS re:Invent 2018 (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2 (NET410) - AWS re:Invent 2018