Der Deutschen liebstes Kind ist nicht etwa der Hund oder das Auto, sondern ganz eindeutig der Tatort am Sonntagabend. Wenn man die App waipu.tv entwickelt, mit der die Kund*innen fernsehen und Aufnahmen machen können, ist diese Erkenntnis von entscheidender Bedeutung. Denn wie stellt man es an, dass sowohl das Backend als auch die Entwickler*innen während dieser Lastspitzen nicht ins Schwitzen kommen? Der Vortrag zeigt einige Best Practices aus dem realen Leben und könnte dabei Spuren von Microservices, DevOps, AWS, Monitoring, GitLab CI, Terraform etc. enthalten. Speaker: Michael Bruns, inovex Event: x-celerate, 09.10.2018 Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Mehr Tech-Artikel: https://www.inovex.de/blog/

1. TDD –
Tatort-Driven Development
Oder: Wie wir bei waipu.tv auch an einem Sonntagabend
die Nerven behalten
Michael Bruns, inovex GmbH

2. › Q: How many people watched Tatort on 03 April 2017?
› A: 14.56m
› Q: What’s the market share of this?
› A: 39.6%
2TDD –Tatort-Driven Development
Let’s start with a quiz
(Source: http://bit.ly/2uHEt7M)

3. 3TDD –Tatort-Driven Development
Please allow me to introduce myself:
› Software developer, architect, therapist, ...
› AWS fanboy
› Built quite a bit of waipu.tv
› http://sayat.me/mbruns
Michael Bruns
inovex GmbH

4. 4TDD –Tatort-Driven Development
What did we build?
AuthRegistration PlayoutEPG Recording
Users Streaming
......
Third
Party
...

5. 5TDD –Tatort-Driven Development
How do people use it?
20:15
› Anybody who likes working at 20:15
every day?
› Including and especially on weekends?

6. › Don’t build a platform, use one - in our case: AWS
› Use what’s already there: EC2, RDS, S3, Elasticsearch,
Kinesis, Route53, Lambda ...
› Add shared stuff (e.g. JVM, nginx) to base image (AMI)
› Leave the rest to the services
6TDD –Tatort-Driven Development
How did we build it?

7. › Foster DevOps, i.e. tear down all political and
technological barriers
› Make infrastructure reproducible
› Choose the right tool for the job:
Terraform, GitLab CI, Prometheus, …
7TDD –Tatort-Driven Development
How did we build it?

8. 8TDD –Tatort-Driven Development
Use the right tool for the job...
...and use it wisely!

9. › 50+ EC2 instances, 25+ Auto-Scaling Groups,
25+ ELBs, 75+ Security Groups
› ~30m objects / 3+ TB in S3 Buckets
› ~60m documents stored in Elasticsearch
› 50+ Git Repositories with GitLab CI
9TDD –Tatort-Driven Development
What did we use?

10. › ~15 Lambdas (growing a lot)
› 8 PostgreSQL databases, 10 Redis Caches
› 75+ DNS entries
› Java, JS, Clojure, Kotlin (and some Ruby, Lua, Go, …)
› ~15 people working in 5+ locations
10TDD –Tatort-Driven Development
What did we use?

11. › Infrastructure as Code
› Independent of provider (AWS, Azure, OpenStack, …)
› Multi-cloud strategy is possible
› https://www.terraform.io/
11TDD –Tatort-Driven Development
Terraform - Basics

12. 12TDD –Tatort-Driven Development
Terraform - Examples

13. 13TDD –Tatort-Driven Development
Terraform - Examples

14. › Version control
› Reproducible
› Manages dependencies (both implicit and explicit)
› State can be “planned” (i.e. validated)
› Remote state for collaboration
14TDD –Tatort-Driven Development
Terraform - Advantages

15. › Continuous Integration & Deployment
› Pipelines
› Stages, Environments, Variables, Tags, …
› Easy integration of runners
15TDD –Tatort-Driven Development
GitLab CI

16. 16TDD –Tatort-Driven Development
The whole pipeline
Local Git
Repo
GitLab
Repo
GitLab CI Build
(Gradle,
Leiningen, …)
GitLab CI
Deployment
(Terraform)
Dev
GitLab CI
Deployment
(Terraform)
GitLab CI
Deployment
(Terraform)
Preview Prod
auto auto
manually manually
GitLab CI
Tests
auto
GitLab CI
Tests
auto

17. › Launch Configuration for services
› Automatic scaling of services based on
› Network/CPU usage
› Number of requests
› Schedules
17TDD –Tatort-Driven Development
AWS - Auto Scaling Groups

18. 18TDD –Tatort-Driven Development
AWS - Auto Scaling Groups - Example

19. › “Serverless” functions
› Stateless
› Java (or better: JVM), Node.js, Python, C#, Go
› React to events (Kinesis, S3, Alexa, CloudWatch, …)
› “Infinite” scaling
19TDD –Tatort-Driven Development
AWS - Lambda

20. › I’ve been at x-celerate a year ago ranting about
Docker & Kubernetes
› Definitely useful technologies, but we still don’t use them
› If you’re application is trash without a container,
it will be trash inside a container as well
20TDD –Tatort-Driven Development
My Two Cents on Containers

21. › These are all just tools
› A fool with a tool is still a fool
› Tools won’t fix your people issues
21TDD –Tatort-Driven Development
The Bad News

22. Don’t put your unmaintainable monolith
into the cloud and expect things to get better!
22TDD –Tatort-Driven Development
Most important advice

23. › You will get automatic scaling
› You will get continuous delivery
› You will get a better distribution
23TDD –Tatort-Driven Development
Unmaintainable monolith in the cloud
(of errors)
(of a big ball of mud)
(of the stuff you already
didn’t know how to debug before and which you now
can’t even find because you aren’t writing the logs to a
central store yet and once the machine is gone all logs
are gone and now you sit in your office yelling and…)

24. › Improve the interiors (package/module structure, DB
access, number/distribution of requests, …)
› Automate as much as possible
› Start small, go micro only if necessary
› But start! Today is the best day to improve your
architecture.
24TDD –Tatort-Driven Development
Where to Start?

25. › Talk to each other
› Take responsibility
› Don’t point your finger at someone
› Don’t fall into this trap:
“Weeks of coding can save you hours of planning!”
25TDD –Tatort-Driven Development
The Social Aspects

26. › Firefighter for bugs or urgent issues
› Take turns
› Create an easy way to notify the
person in charge (e.g. @firefighter)
› Don’t be afraid to ask for help
26TDD –Tatort-Driven Development
Share the operational load

27. › Be open for constant change
› However, don’t change everything just because you can
› Sometimes legacy is not necessarily a bad thing
› Some changes might have consequences you don’t expect
27TDD –Tatort-Driven Development
Embrace Change

28. › Create a central board to request improvements
› Let the team make decisions which improvements are
most important
› Accept if the others think that your suggestion isn’t useful
28TDD –Tatort-Driven Development
Track Change

29. › Set the same goal(s) for everybody, but don’t set
too many rules
› Let teams find their best modus operandi
› Be careful with stuff like coding guidelines, static code
analysis, Git flow, …
29TDD –Tatort-Driven Development
The Everyday Work

30. › Repeat these sentences ten times every day:
› We are not Netflix
› We are not Google
› We are not Facebook
› We are not Spotify
30TDD –Tatort-Driven Development
The Mantra

31. › Don’t build a single point of failure
› Monitor your stuff
› Couple your services not as loosely as possible,
but as loosely as feasible
› Do you really need Kafka, Event Sourcing, CQRS, …?
31TDD –Tatort-Driven Development
Coupling Services

32. › Loosely coupled, small services
› Isolated from each other
› Supposed to fit into your head
› But what if they don’t just because of the sheer number
of services?!
32TDD –Tatort-Driven Development
“Micro”services

33. “One common anti-pattern when introducing DevOps to
an organization is to assign someone the role of 'DevOps'
or to call a team a 'DevOps team'. Doing so perpetuates
the kinds of silos that DevOps aims to break down and
prevents DevOps culture and practices from spreading
and being adopted by the wider organization.”
(Rouan Wilsenach,
https://martinfowler.com/bliki/DevOpsCulture.html)
33TDD –Tatort-Driven Development
DevOps - How not to do it

34. 34TDD –Tatort-Driven Development
The Fear of DevOps
› Do I have to become an nginx ninja?
› Am I doing deployments 4 hours/day?
› Will my weekends look like this?

35. › There’s this one person constantly calling me at the most
awkward times because something isn’t working:
35TDD –Tatort-Driven Development
The Fear of DevOps
› But my mum’s phone not
working the way she wants
it to work has nothing to do
with DevOps!
Of course this isn’t really my mum!
And her calls are perfectly fine.

36. 36TDD –Tatort-Driven Development
One final note...
➔ ➔

37. › Wear a helmet when riding your bike, i.e.:
› Expect failure
› The person to trust least is you
› Accept help
38TDD –Tatort-Driven Development
One final note...

38. Vielen Dank
Michael Bruns
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
michael.bruns@inovex.de
Twitter: @der_miggel Feedback:
http://sayat.me/mbruns