Every software development cycle is rife with inefficiency. Seasoned devs know the pain of getting access to essential remote systems, waiting for tests to run (and then fail), or debugging with only log files. This talk teaches you how to best leverage Kubernetes, remote infrastructure and related tooling to create a dev cycle that maximizes velocity and minimizes developer friction and frustration. Using tools such as Kubernetes, Docker and Telepresence, I will walk attendees through several advanced techniques that can be used to produce an effective developer experience and optimized dev loop. The goal of this is to eliminate many sources of frustrating inefficiency and reduce cycle time between releases. I will demonstrate how to incrementally adopt some of these techniques and how to approach introducing new and unfamiliar technology and techniques to skeptical dev teams.

1. Effective Kubernetes
Development
Turbocharge Your Dev Loop
Philip Lombardi
@TheBigLombowski

2. datawire.io
This Talk
● Share the Cloud Native workflow we have
evolved over three years
● A journey about how we got where we are
along with some broad stroke technical and
communication insights
2@TheBigLombowski

3. datawire.io
$(whoami)
● Internal dev tooling and infrastructure
● Occasionally hack on product
● Very impatient… on a mission to make Cloud
Native development less cumbersome
3@TheBigLombowski

4. datawire.io 4
Dev-Test
Deploy Build
Code
Dev-Loop
@TheBigLombowski

5. datawire.io 5
Dev-Test
curl ...
e2e
“Stuff”
Deploy
kubectl <...>
Data migrations
“Stuff”
Build
Binary
Docker Image
Assets
Code
func main() { ... }
if __name__ == “main”
fun main(args: Array<String>)
Dev-Loop
@TheBigLombowski

6. datawire.io
Late 2015
● Not on Kubernetes yet (kube 1.0 brand new)
● Bootstrapping our backend… need to ship fast
and ship often
● Dev loop is very primitive…
6@TheBigLombowski

7. datawire.io 7
Dev-Test
Wild West
Deploy
Mix of: Shell, Terraform and Ansible
Build
Build + Package Binary
Build VM image
Code
fun main(args: Array<String>)
Dev-Loop
Late 2015 - Early
2016
@TheBigLombowski

8. datawire.io
Problems 1.0
1. VM images take forever to create (10 min)
2. Build fails… start over again (10 mins x Failures)
3. Developers hate dealing with Packer as part of
their dev loop.
4. Shell + Terraform + Ansible stuff is pretty janky
too and causes problems.
8@TheBigLombowski

9. datawire.io
Solutions 1.0
1. Ignore all the speed complaints!
2. Slap a fresh coat of paint on packer by hiding
it in Makefiles and Gradle scripts.
9@TheBigLombowski

10. datawire.io 10@TheBigLombowski

11. datawire.io
Reality...
11
Me CTO + Coworkers
@TheBigLombowski

12. datawire.io
Lessons 1.0
1. I shouldn’t ignore my developers!
2. I need to listen and understand their problems
better...
12@TheBigLombowski

13. datawire.io
Problems 1.1 and 1.2
Problem: Our devs do not always know
what their problem is…
Problem: I listened to our developers
and we fixed the wrong problem
13@TheBigLombowski

14. datawire.io 14
Dev-Test
Wild West
Build
Build JAR (Java/Kotlin)
Build VM Image
Deploy
Code
fun main(args: Array<String>)
Fix
Deploy!
Early 2016
@TheBigLombowski

15. datawire.io 15
Dev-Test
Wild West
Build
Build JAR (Java/Kotlin)
Build VM Image
Code
fun main(args: Array<String>)
@TheBigLombowski
- Automate
Workflow!
- Fancy
Custom
Tools!
Fix
Deploy!
Early 2016

16. datawire.io
swap.py
● Does all the fancy orchestration to accomplish
an automated blue-green deployment.
● Solved some pain… but it also created lots of
new pain and still didn’t address the real pain.
● We’re smart people though so we can fix this!
16@TheBigLombowski

17. datawire.io
Solution 2.0: Doubledown!
17@TheBigLombowski

18. datawire.io 18
Expected Outcome... :D
@TheBigLombowski

19. datawire.io 19
… Actual Reaction D:
@TheBigLombowski

20. datawire.io
Lessons 2.0
● Thankful my bosses are patient people ;)
● We need to do a better job identifying the real
pain.
● Don’t get too technology crazy.
20@TheBigLombowski

21. datawire.io
Solution 3.0
● Scrap it all(!)
● Re-evaluate where things were slow
● Finally we all realized that the inability to build
quickly was killing us as well.
21@TheBigLombowski

22. datawire.io 22
Dev-Test
Wild West
Build
Switch to Docker!
Code
func main() ...
@TheBigLombowski
Reboot!
Early 2017
Deploy
Swarm? Kubernetes?
Mesos? Options!

23. datawire.io 23
Dev-Test
Wild West
Build
Switch to Docker!
Code
func main() ...
@TheBigLombowski
Reboot!
Early 2017
Deploy
Kubernetes!

24. datawire.io
Good News!
● Scrapping + Rethink a success!
● Devs Happy! Velocity increased!
24@TheBigLombowski

25. datawire.io 25
Dev-Test
Build
WEH! This is still too
slow!11!!
Deploy
Code
Dev-Loop
Mid-2017
@TheBigLombowski

26. datawire.io
They’re kind of Right...
● Compile code (30s+)
● Assembly binary / package (30s+)
● Build Container Image (15-30s+)
● Push Container Image (15-30s+)
26@TheBigLombowski

27. 120 seconds*
@TheBigLombowski
* Not scientific… possibly higher, but most likely not lower

28. datawire.io
● (Java) Compile in around 30 seconds? Nope.
Maven is still downloading the world 30
seconds in.
● (Python) Native dependencies require all kinds
of build toolchain stuff to be installed.
● Slow networks…
28@TheBigLombowski
120 seconds is Optimistic

29. datawire.io
And it’s really 120s x # of Rebuilds
Adds up real quick! Easy to throw away an hour or
more just waiting.
29@TheBigLombowski

30. datawire.io 30
Dev-Test
Optimize
Dockerfile
Deploy
Code
Build Speed
Attempt 1
@TheBigLombowski

31. datawire.io 31
Dev-Test
Multi-Stage BuildsDeploy
Code
Build Speed
Attempt 2
@TheBigLombowski

32. datawire.io 32
Dev-Test
Small Base
Images
Deploy
Code
Build Speed
Attempt 3
@TheBigLombowski

33. datawire.io
Not Enough!
● Shaves some seconds… but not enough.
● Even the fastest builds gets stuck in time sink of
registry push, redeploy, registry pull.
33@TheBigLombowski

34. datawire.io
Faster Builds… Hail Mary Pass
1. Wrote a custom tool that did some crazy stuff
around managing Docker layers and
continuously rebuilding…
2. Faster builds but generated monster sized
images that took ages to push and pull… so
net gain of nothing.
34@TheBigLombowski

35. datawire.io 35@TheBigLombowski

36. datawire.io
What if...
Your dev machine ran your code and it just
worked the same as if you ran it on Kubernetes?
36@TheBigLombowski

37. datawire.io
What if...
37@TheBigLombowski
CNCF Sandbox!
Open Source!

38. datawire.io 38
Dev-Test
Build + “Deploy”
Restart local process
Code
Dev-Loop
Mid-late 2017
@TheBigLombowski

39. datawire.io
Telepresence Topology
39@TheBigLombowski
Proxy Connection
Telepresence Pod
replaces
pod/Foo-App
Foo App
Bar App
pod/Bar-App
LB/NodePort
svc/Foo-App svc/Bar-App

40. datawire.io
Kill Two Birds w/ One Stone
● “Deploy” is merged into Build because “Deploy” just means relaunching local
process.
● Drastically simplifies the problem.
40@TheBigLombowski

41. datawire.io
Secondary Benefit!
● Code is on YOUR machine… so...
● You can run a debugger from your local
machine easily…
● No more printf debugging and grepping
logs!
○ Connect a debugger to your local process and voila breakpoints! Stack information!
○ Debugging is an often a painful part of Dev-Test so the value of this cannot be overstated.
41@TheBigLombowski

42. datawire.io 42
Expected Outcome... :D
@TheBigLombowski

43. datawire.io 43
Actual Outcome... :D
@TheBigLombowski

44. datawire.io 44@TheBigLombowski

45. datawire.io
Not Quite...
● Devs are never satisfied!
● Build speed is great but Getting Started and
Bootstrapping are real problems.
45@TheBigLombowski

46. datawire.io 46
Deploy: Bootstrapping
It is a big problem!
Dev-Loop
Late 2017
@TheBigLombowski

47. datawire.io
Attempt #1
● Use Minikube on local machine
● Apply manifests of stuff
● It should all just work out right?
47@TheBigLombowski

48. datawire.io
Outcome...
● Minikube is fragile and tends to break in mysterious ways…
● Our deployment logic for stuff is not as clean as we would hope. Tends to also
make assumptions about running from a well-configured CI env and not a dev
machine.
● Developers banging their heads against their desks because of bootstrapping
pain.
48@TheBigLombowski

49. datawire.io
Minikube is for Unicorns
49@TheBigLombowski

50. datawire.io
For the rest of us… remote shared environments
● If you have any cloud services you likely already have a deploy pipeline for
production.
● Leverage the deploy tools to spin remote dev-test environments for individuals or
teams.
50@TheBigLombowski

51. datawire.io 51
Attempt #2: Kubernaut
● Remote clusters that take <5 seconds to access.
● Runs a pool of clusters in the cloud.
● Developers claim (start exclusive lock) or discard
(end exclusive lock) a cluster.
● Pluggable to support a range of cluster configs
and backends
● Going Open Source in 2019
@TheBigLombowski

52. datawire.io 52
Dev-Test
Dev-Loop
Late 2017-2018
@TheBigLombowski
How we (almost) do it now kubernaut claim

53. datawire.io 53
Dev-Test Code
Dev-Loop
Late 2017-2018
@TheBigLombowski
How we (almost) do it now kubernaut claim

54. datawire.io 54
Dev-Test
Dev-Loop
Late 2017-2018
@TheBigLombowski
How we (almost) do it now kubernaut claim
Start telepresence
Code

55. datawire.io 55
Dev-Test
Dev-Loop
Late 2017-2018
@TheBigLombowski
How we (almost) do it now kubernaut claim
Start telepresence
Code
More Code

56. datawire.io
The Ugly: Dev Testing
● Dev-Testing is painful without shared remote
infrastructure
● Recommended way of doing it
○ Nested API Gateways and some combination of HTTP
Host/Header/Path based routing.
● Your API Gateway needs to be self-service for
developers and easy to safely reconfigure.
56@TheBigLombowski

57. datawire.io 57
Dev-Test
Build + “Deploy”
1. Claim Cluster
2. Start Telepresence
3. Code
Code
Dev-Loop
Late 2018
Dev Routing
@TheBigLombowski

58. datawire.io 58
0.0.0.0/0
Load Balancer
Ambassador
foo-serv bar-serv
Ambassador
foo-serv bar-serv
“Prod”
Kube
“Dev” Kube
@TheBigLombowski
Our Topology

59. datawire.io 59
0.0.0.0/0
Load Balancer
Ambassador
foo-serv bar-serv
Ambassador
foo-serv bar-serv
Telepresence
Proxy
Prod Kube Phils Kube
@TheBigLombowski
Our Topology
HTTP Header “Dev = Phil”
Proxy Connector
bar-serv
Phils Laptop

60. datawire.io
That’s where we are
● Still plenty of work to do
● Devs are way more productive than when we
started: minimal waiting for things to happen
● Keep evolving this as we scale out our team.
60@TheBigLombowski

61. datawire.io
Takeaways
● Rome was not built in a day
● Identify and fix the right problems!
● Identify your loops slow points… tackle low
hanging fruit first. Repeat…!
● You’re most likely going to need remote
development clusters for your projects.
61@TheBigLombowski

62. datawire.io
I will be at the Datawire.io booth
Come and chat :)
Check out Ambassador and Telepresence!
Tweet me! @TheBigLombowski
62@TheBigLombowski
Thanks!

63. datawire.io
I will be at the Datawire.io booth
Come and chat :)
Check out Ambassador and Telepresence!
Tweet me! @TheBigLombowski
63@TheBigLombowski
Thanks!

64. datawire.io 64

65. datawire.io 65