DevOps Days Boston 2017 Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices? In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.

1. Kubernetes in about 45 minutes
Everything you need to know to be dangerous
Philip Lombardi
Platform Engineer

2. datawire.io
Who is this Phil Lombardi guy?
Twitter: @TheBigLombowski
2

3. datawire.io
Why are we all here?
● You are curious about Kubernetes (and friends) and want a primer!
● You are invested in Kubernetes but are looking to learn about some techniques
and tools to make your developers lives better.
● This is the last preso and you feel guilty about leaving early :)
3

4. datawire.io
Agenda...
● Part 1: Containers and Docker and Kubernetes, Oh my!
● Part 2: Kubernetes Core Concepts
● Part 3: Development Workflow
● Part 4: Logging, Debugging and Resiliency
● Q & A
4

5. datawire.io
Lesson 1: Containers and Docker and
Kubernetes, Oh my!
5

6. datawire.io
What is a container?
● Lightweight Linux environment. It is a form of virtualization… but very different
from a full virtual machine.
● Immutable, deployable artifact.
● Runnable.
● Popularized by Docker but there are many implementations (e.g. LXC, Rkt).
6

7. datawire.io
What is Docker?
● A tool, ecosystem and platform for building, pushing and
running containers.
● The most popular container runtime currently.
● Default container runtime in Kubernetes.
7

8. datawire.io
Why Containers?
● Easy and fast to produce.
● Great way to isolate different components in a complex system.
● Ensures a reproducible runtime for your app along the dev -> build -> test -> prod
pipeline.
● Easy to share in a team or with external partners.
8

9. datawire.io
What is Kubernetes?
● Runs massive numbers of containers based on
lessons learned by Google.
● Schedules and runs all kinds of containers
○ long-lived (e.g. services)
○ short-lived (e.g. pre-launch hooks, cronjobs etc)
● Kubernetes can be thought of as a Distributed OS or process manager
9

10. datawire.io
The Office Tower Analogy
Your product is the building
as a whole.
Your business logic is
the offices and workers
10

11. datawire.io
The Office Tower Analogy
Kubernetes provides the
infrastructure to build your
app around.
11
It is the foundational app
platform for your team to
build your businesses apps
around.

12. datawire.io
Why Kubernetes?
It is not the only kid in the neighborhood…
● Amazon ECS
● Docker Swarm
● Hashicorp Nomad
● Apache Mesos
12

13. datawire.io
Why Kubernetes?
Three big reasons to use Kubernetes over the other solutions:
1. Biggest ecosystem of the bunch and there is a hugely massive community
2. Runnable just about anywhere: cloud, bare-metal, and engineers laptops.
3. Unprecedented cloud portability.
13

14. datawire.io
Kubernetes Architecture
Types of nodes: Masters and Workers
14
Docker Kubelet
Kubeproxy
Kubernetes Node
Docker Kubelet
Kubeproxy
Kubernetes Node
Docker Kubelet
Kubeproxy
Kubernetes Node
Etcd API Server
Controller Manager
Kubernetes Master
Scheduler

15. datawire.io
Lesson 2: Core Kubernetes Concepts
15

16. datawire.io
The “Big Five” of Kubernetes Concepts
● Pods
● Deployments
● Services
● ConfigMaps
● Secrets
16

17. datawire.io
A Pod you say?
● One or more strongly-related containers…
17
Story
Server
name: blog
Redis
Comment
Server
Frontend
host: kube-worker-0

18. datawire.io
A Pod you say?
● Containers in a pod share the same host, pod IP and port space.
18
IP: 100.124.71.175
Story
Server
name: blog
Redis
Comment
Server
Frontend
host: kube-worker-0

19. datawire.io
A Pod you say?
● Unit of scaling is a Pod
19
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster

20. datawire.io
Pods Summary
● Like a host. All containers inside of a pod are run on the same underlying worker
machine.
○ Can therefore reference localhost
○ … or share the filesystem
○ … or use unix domain sockets
● All containers in a Pod share the same IP and port space.
● Pods are not durable.
● Pods are a very low-level primitive construct. Necessary to know, but not
commonly used directly.
20

21. datawire.io
Deployments
● Simple mechanism to configure, scale and update applications.
● Kubernetes does the rest of the hard work of scheduling the Pods across the
cluster to meet desired capacity numbers.
● Works like a thermostat… Ensures the current state is always consistent with the
desired state.
21

22. datawire.io
Services
● Services are stable “names” in Kubernetes that enable you to route traffic to Pods
across the entire cluster.
● Every service gets its own IP address.
● Services route traffic to pods by matching labels.
22

23. datawire.io
Services Illustrated
How to talk to both apps despite different IP addresses?
23
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster

24. datawire.io
Services Illustrated
Add a Service which becomes a DNS A record pointing the pod IP addresses
24
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster
blog DNS (short) => blog
DNS (long) => blog.default.cluster.local

25. datawire.io
Services Illustrated
You can have multiple services target pods using labels and selectors.
25
app=blog
env=prod
blog-0
kube-worker-0
app=blog
env=prod
blog-1
kube-worker-1
Kubernetes cluster
blog blog-staging
app=blog
blog-0
kube-worker-1
app=blog
env=prod
app=blog

26. datawire.io
Service Flavors
● Many different flavors of “Service” in Kubernetes
○ ClusterIP
○ NodePort
○ LoadBalancer
○ ExternalName - often forgotten, but very useful!
26

27. datawire.io
Services Summary
● Creates DNS A records pointing at Pod IP addresses
● Powerful label matching capabilities that enable you to route traffic to particular
pods (e.g. for blue-green or canary releases).
● Supports DNS SRV records so you can avoid hard coding port numbers in your
app code as well.
27

28. datawire.io
ConfigMap
● Containers are immutable… so how do you provide runtime configuration to
them?
● Age old question for immutable infrastructure lots of good (and bad) solutions
have been built over the years.
● Kubernetes solution is built-in as the ConfigMap. Inject configuration information
as…
○ Environment variables
○ Volumes
28

29. datawire.io
Secret
● Cousin of the ConfigMap
● Operates almost exactly the same as a ConfigMap but designed for storing
sensitive information.
● Secret information only sent from master to worker nodes when needed by a
pod. The data lives in memory so it is not on the disk.
● One important aspect of secrets… the master currently stores them in plaintext.
Work in Progress to eliminate this in the future, but worth being aware of.
29

30. datawire.io
Kubernetes to AWS
30
Kubernetes AWS
Pod EC2 Instance
Deployment AutoScaling Group + Launch Configuration
Service ELB
ConfigMap N/A
Secret N/A

31. datawire.io
Lesson 3: Developer Workflow
31

32. datawire.io
Developers...
● Part of our role involves aiding Developers and making them faster and more
productive.
● Kubernetes is awesome and it comes with a lot of power.
● Great power comes with lots of potential for learning pain.
● How do we make developers productive?
32

33. datawire.io
Manifests
● A declarative YAML/JSON config format that describe at a high level how
Kubernetes should operate.
● Kubernetes operates like a thermostat. Transforms current state -> desired state
based on config in the Manifest.
● Manifests often need to be parameterized (e.g. to change the container image).
General approach is to use some kind of templating.
33

34. datawire.io
Structuring an Application
● Often asked, all this stuff is cool, but how should we structure our apps to be
consistent and compatible with tooling?
● Strongly recommend a k8s/ directory in the top of your project
○ Manifests can be concrete and ready to use by just running `kubectl apply -f k8s/ `
○ … or you can take an alternate approach and put templates in that directory and do some kind of
config generation with say Python + Jinja2 (for example).
34

35. datawire.io
Writing Manifests
● Avoid hard coding environment configuration into manifests.
○ Templating
○ Kubectl switches (e.g. for namespaces)
● Stick to YAML even if Kubernetes supports JSON… not uncommon to want
comments in the manifests.
● Kubernetes manifests can be spread across multiple files or kept in a single file.
Strongly recommend using a single file until it becomes bothersome to maintain.
35

36. datawire.io
Development Workflows
● No single workflow that works for all developers or teams
● Need tools that can adapt to changing requirements and process
● Personally, great success with:
○ Trunk-based development model.
○ Using parameterized templates in k8s/ directory.
○ monorepo or “pseudo-monorepo”.
○ Dev-tooling the focuses on speed and maintaining fast iteration cycles
36

37. datawire.io
Forge (https://forge.sh)
● Build and deploy Kubernetes-based microservices quickly.
● Can deploy 1 or 100 services from source to Kubernetes in seconds.
● Changes are applied incrementally. Computes the diff of a change and then
pushes the update to Kubernetes.
37

38. datawire.io
Lesson 4: Logging, Debugging and
Resiliency
38

39. datawire.io
Logging...
● Kubernetes has a built-in log aggregation but it is limited (STDOUT, STDERR only).
● `kubectl logs` is good enough for devs but invest in a real logging solution
for prod.
● You will want something like fluentd and elasticsearch because Kubernetes does
not track historical logs for crashed or terminated pods.
○ Also search capabilities are limited to how much of a grep wizard you are.
39

40. datawire.io
Logging...
● There is more to logging than just application logs.
● Consider introducing a service mesh to your cluster that allows you to do
per-request logging and tracing.
40

41. datawire.io
Services Mesh
● This was covered in a presentation yesterday so here is the recap:
A dedicated infrastructure layer for making service-to-service communication
safe and reliable.
● Kubernetes and CNCF are really pushing Lyft Envoy as the mechanism to build a
service mesh.
41

42. datawire.io
Simple Log Query Tools
● Because the kubectl logs command is so limited many tools have been
written to make it easier...
○ ktail https://github.com/atombender/ktail
○ kubetail https://github.com/johanhaleby/kubetail
○ stern https://github.com/wercker/stern
42

43. datawire.io
Debugging...
Kubernetes IS complex. There are a lot of failure scenarios in all kinds of places. The
Kubernetes docs are pretty helpful for doing some troubleshooting
Application Troubleshooting:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/
Cluster Troubleshooting:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/
43

44. datawire.io
Debugging The Cool Way
● Sometimes you need something more powerful than logs…
● Or your developers use a shared development-staging cluster and something is
broken...
● Classic problem with building web services… How do I attach a local process to
the running cloud environment? What about a debugger?
● Super easy in Kubernetes with a tool called Telepresence!
http://www.telepresence.io/
44

45. datawire.io
● Telepresence proxies network
requests, environment variables,
and volumes to local
Telepresence client
● Code locally on your laptop using
your favorite editor and your local
filesystem :)
Telepresence
45

46. datawire.io
Wrapping Up
● Kubernetes is awesome!
● There’s a lot of power and flexibility
● We need to empower developers by providing them excellent tools that make
their lives easier!
● As infrastructure and ops engineers we need to build a stable platform that
developers can use without feeling restricted. The service mesh makes this easier.
46

47. datawire.io
Preso Over! Thank you!
● If you’re building cloud applications on top of Kubernetes, check out our open
source tools:
● Contact us @datawireio or hello@datawire.io
47