Let’s see what it takes to build an automated testing/QA setup with Kubernetes and GitLab from scratch, using off-the-shelf components. The goal is to automatically deploy any feature-x or hotfix-x branch to individual environments for automated tests and to make it easier to share intermediate results within the team. This talk will touch on abstract topics like continuous delivery, the value of developer tooling and well-defined workflows, before diving deep into implementation details, learnings, gotchas and stuff-which-went-wrong.

1. vsupalov.com
Automated Testing
Environments With
Kubernetes & GitLab
Vladislav Supalov, 15th November 2017

2. vsupalov.comvsupalov.com
Hi, I’m Vladislav!
2
● Computer science background: ML, CV & data
● Casual web development
● Fascination with DevOps
○ Efficient, reliable, (huge), infrastructure setups
○ Monitoring, automation, processes
● Automation engineer
● Data engineering consulting business
○ Data pipelines! Small data :)
● Co-founding a startup - Pivii Technologies
○ Visual artificial intelligence for social media content marketing
● Building a consulting business around Kubernetes automation
2
vsupalov

3. vsupalov.comvsupalov.com
One Fine Weekend:
3
3
● How hard can it be from scratch?
● Entirely self-hosted solutions
● Using Kubernetes and GitLab
● Free Google Cloud Platform credits :)
● Project: any changes to a Git feature-x or hotfix-x branch →
● Automatically deploy to individual environments
● Purpose: share intermediate results, review progress, run automated tests
● How far can I get?

4. vsupalov.com
● What? Why would you?
● Words: CI & CD
● Deployment pipelines
● GitLab & Kubernetes ❤
● Lab: automating QA envs
Envs are non-critical & disposable
Using lazy, off-the-shelf components
4

5. vsupalov.com
What? Why?
5

6. vsupalov.com
We’re not doing this for fun.
6

7. vsupalov.com
I am doing this for fun.
Most of you as well, I assume.
Giving developers superpowers.
Awesome, reliable, neat, tidy setups.
One command → everything works.
7
We’re not doing this for fun.

8. vsupalov.com
Ideally, good companies and
great teams are investing
resources (time & money) into
adopting and using practices &
tech for very good reasons.
Examples: processes, Scrum, workflows, automation,
fancy/immutable infrastructure, Git, Docker, Kubernetes
8

9. vsupalov.comvsupalov.com
● Company: Save money or make more money
● Reduce risk
● Fix things which are broken
● Less mistakes, faster iteration times, better for company
● Improve team communication
● Save expensive developer time
● Get more team efficiency
● Reduce friction for devs
● Find issues as early and with as little effort as possible
● Create feedback loop between ops and devs
● Happy developers = less employee churn
Good Reasons
9
9

10. vsupalov.com
Words & Practices:
Continuous Integration
Continuous Delivery
Continuous Deployment
Testing, Staging, Production
10

11. vsupalov.comvsupalov.com
Continuous Integration
11
11
“Continuous integration is the practice of
routinely integrating code changes into the
main branch of a repository, and testing the
changes, as early and often as possible.
Ideally, developers will integrate their code
daily, if not multiple times a day.”
Source: atlassian.com

12. vsupalov.comvsupalov.com
Continuous Delivery & Deployment
12
12 Source: atlassian.com

13. vsupalov.comvsupalov.com
Testing, Staging, Production Envs
13
13 Source: agiledata.org

14. vsupalov.com
Deployment Pipelines
14

15. vsupalov.comvsupalov.com
● Continuous Delivery (2010)
● By Jez Humble and David Farley
● ~440 pages
● Don’t be discouraged by the age
● Really good foundation and !juicy! details
● Literally the best book on the topic
Reading Recommendation
15
15 Image source: amazon.de

16. vsupalov.comvsupalov.com
Deployment Pipeline
16
16 Image source: Continuous Delivery, 2010, Jez Humble and David Farley, Chapter 5

17. vsupalov.comvsupalov.com
Basic Deployment Pipeline
17
17 Image source: Continuous Delivery, 2010, Jez Humble and David Farley, Chapter 5

18. vsupalov.com
GitLab & GitLab CI/CD &
Kubernetes
18
I just want to deploy my app.
Repeatedly. When I want to.
Without busy waiting or thinking too much about it.

19. vsupalov.com
Personal opinion: Kubernetes is
the best choice for dealing with
containerized apps for anything
beyond toy projects.
19

20. vsupalov.com
GitLab: betting on Kubernetes!
My interpretation of the founder’s tweets...
20

21. vsupalov.com
GitLab: open source, reasonably
simple to self-host
GitLab CI/CD: very cool product
Used both in a client projects -
really liked ‘em
21

22. vsupalov.comvsupalov.com
GitLab CI/CD: Overview
22
22

23. vsupalov.comvsupalov.com
GitLab CI/CD: Pipeline
23
23 #YOLO

24. vsupalov.comvsupalov.com
GitLab CI/CD: Job
24
24

25. vsupalov.com
And now: my weekend project -
steps taken, results & learnings
25

26. vsupalov.comvsupalov.com
Installing & Configuring - From Scratch
26
26
● New Google Kubernetes Engine cluster
○ Helm, (Prometheus), Nginx ingress, Stackdriver, Deis
● Well-sized Ubuntu Compute Engine VM for GitLab
○ Gitlab omnibus edition (EE)
● Setup, configuration
○ Docker
■ net.ipv4.ip_forward=1
○ GitLab task runner
○ Fixing Gitlab configs
○ GCE free quotas, non-ephemeral (aka static) IP
○ Let’s Encrypt & Nginx reverse proxy config for the fun of it
○ Own Docker registry
○ Toy projects
● Kubernetes integration configuration
○ Token, base64 cert...

27. vsupalov.comvsupalov.com
gitlab-ci.yml
27
27 Link: docs

28. vsupalov.comvsupalov.com
Starting Out: A Skeleton Pipeline
28
28

29. vsupalov.comvsupalov.com
Simplest Project → Deis Workflow → Docker Push
29
29

30. vsupalov.comvsupalov.com
GitLab’s Auto DevOps (Beta)
30L
k30
“Auto DevOps automatically detects, builds, tests, deploys,
and monitors your applications.”
Links: intro, quickstart

31. vsupalov.comvsupalov.com
Push A Flask Project
31
31

32. vsupalov.comvsupalov.com
Building...
32
32

33. vsupalov.comvsupalov.com
But How? Enable & Configure.
33
33

34. vsupalov.comvsupalov.com
Building: Python + Flask @ First Try
34
34

35. vsupalov.comvsupalov.com
Oh.
35
35

36. vsupalov.comvsupalov.com
Ah.
36
36

37. vsupalov.comvsupalov.com
Alright, A Rails App Then...
37
37

38. vsupalov.comvsupalov.com
Whoo!
38
38

39. vsupalov.comvsupalov.com
Shipped.
39
39

40. vsupalov.comvsupalov.com
Environment Overview
40
40

41. vsupalov.comvsupalov.com
41
41

42. vsupalov.com
In Conclusion
42

43. vsupalov.com
Works. Not quite what I aimed
for.
It’s a work intensive task.
A month of work if you’re starting
from scratch to get it right?
43

44. vsupalov.com
Deployment pipeline, team and
product are closely linked. They
should grow together.
No one size fits all - only up to a
certain point.
44

45. vsupalov.comvsupalov.com
● Auto DevOps might be neat-o eventually
● Neglected here:
○ Integrations
○ Lockdown: Security + VPN
○ Wildcard certs with Let’s Encrypt ~2018 + Kube Lego
○ Handle data
○ Monitoring, usable Logs, dashboards
○ Deploy your own Helm chart
○ Maintenance issues
● Beyond magic:
○ Start with a Skeleton
○ Build what’s needed
○ Deployment pipeline for a specific project & team
Closing Thoughts
45
45

46. vsupalov.com
Thanks! Want to get regular
tips on building deployment
pipelines and automating
Kubernetes with GitLab & co?
Mail: yt@vsupalov.com