You're building your startup and you know it will be big. You don't want to spend a lot of time on infrastructure, but you also don't want to be putting out fires after you get mentioned on Hacker News. In this session, we will give you real practical tips that you can take home with you on building an infrastructure that will scale quickly with minimal up front work on your part, using time tested techniques in infrastructure as code, SaaS, and Serverless, among other things.

1. DevOps for Startups
Email: {anything}@jedberg.net
Twitter: @jedberg
Web: www.jedberg.net
Facebook: facebook.com/jedberg
Linkedin: www.linkedin.com/in/jedberg
Jeremy Edberg
Founder and CEO
MinOps
https://minops.com
https://sql.bot

2. If it won’t scale, it'll fail.

3. The key to scaling is finding the
bottlenecks before your users
do

4. Why am I here?

5. Why should we
learn from other
people’s mistakes?

7. Takeaways
• Infrastructure as Code
• Microservices/Serverless
• Queueing Theory
• Chaos Engineering
• Logs
• Incident reviews

8. Infrastructure as Code
• Changes are routine, small,
easy, and repeatable
• Resources are easily
managed by users and
disposable
• Enables continuous
deployment and
improvement
• Solutions can be easily
tested, measured, and then
rolled back

9. • Losing track of servers
and resources
• Configuration drift
• Snowflakes
• Fear of a fully
automated system
(lack of trust in oneself)
Infrastructure as Code
Challenges

10. Automate all the things!
http://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html

11. • Application startup
• Configuration
• Code deployment
• System
deployment
Automate all the things!

17. Choosing your infrastructure

18. λ

19. Test and prod are different
Prod is in need of constant updates
Slow iteration and deployment
Polyglot unfriendly
Deploy in weeks, live for years
Physical Servers

20. Prod is immutable
Rapid iteration and deployment
Multi-tenancy
Polyglot friendly
Deploy in minutes, live for weeks
Virtual Machines

21. Test and prod are the same
Prod is immutable
Rapid(er) iteration and deployment
High multi-tenancy
Polyglot friendly
Deploy in seconds, live for hours
Containers

22. Smallest unit of compute
Super scalable
Rapid iteration
Extreme multi-tenancy
Very polyglot friendly
Easier to collaborate
Deploy independently, live for seconds
Serverless
λ

23. A whole lot of choices
Amazon’s EcosystemHodgepodge of services

24. A whole lot of choices

25. Amazon’s Serverless Ecosystem
Lambda
SNS
DynamoDB
SQS
S3
Kenisis

26. What is serverless anyway?
• There are still servers, you just don’t
manage them anymore
• It also means you don’t access them
anymore
• So you don’t need to (or get to)
optimize them.

27. Serverless computing is all about
speeding up development by allowing
rapid iteration and removing
management overhead

28. Choosing your unit of compute
• VMs
Machine as the unit of scale
Abstracts the hardware
• Containers
Application as the unit of scale
Abstracts the OS
• Serverless
Functions as the unit of scale
Abstracts the language runtime
EC2
ECS
Lambda

29. How do I choose?
• VMs
“I want to configure machines,
storage, networking, and my
OS”
• Containers
“I want to run servers,
configure applications, and
control scaling”
• Serverless
“Run my code when it’s
needed”
EC2
ECS
Lambda
I didn’t write
the software
myself

30. Monorepo or Services?

31. Advantages to a Monorepo
• No worrying about
dependencies
• Don’t have to account for
data movement
• Deployments are simple
• Coordination is easy

32. Multiple services
Internal Microservices Platform
Monolithic
Success follows a standard pattern

33. ReleaseTestBuild
Developer Deployment Pain: High
DevOps Deployment Pain: Medium

34. ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
Developer Deployment Pain: Medium
DevOps Deployment Pain: High

35. ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
ReleaseTestBuild
λ
λ
λ }
Developer Deployment Pain: Low
DevOps Deployment Pain: 🔥

37. Advantages to a Service Oriented
Architecture
• Easier auto-scaling
• Easier capacity planning
• Identify problematic code-paths more easily
• Narrow in the effects of a change
• More efficient local caching

38. Highly aligned, loosely coupled
• Services are built by different
teams who work together to
figure out what each service
will provide.
• The service owner publishes
an API that anyone can use
and returns proper response
codes

39. Distributed Computing and a
Distributed Workforce
• The two go hand in hand
when you have a good
distributed systems culture
• Microservices and Micro
Teams

40. Proper
Microservices
Architecture
Service
and
Resource
Discovery
Network
and Traffic
Config
Automated
Testing
Continuous
Deployment
Security
Monitoring
and Alerting

41. Mature companies spend 25% of their
engineering resources on their internal platform

42. And when you’re done it is only “good
enough”
Building an internal
microservices platform is hard

43. So what about Serverless?

44. What do all the parts
of microservices have
in common?

45. Servers
Capacity planning
Right-sizing
Autoscaling
Load and performance
Patches
Tuning
Configuration
Utilization
Access control
Packages and AMIs

46. Serverless
Right-sizing
Autoscaling
Load and performance
Patches
Tuning
Configuration
Utilization
Access control
Packages and AMIs
Fully managed
Continuous Scaling
Function is the deployment unit
Capacity planning

47. Proper
Microservices
Architecture
Service
and
Resource
Discovery
Network
and Traffic
Config
Automated
Testing
Continuous
Deployment
Security
Monitoring
and Alerting

48. Proper
Microservices
Architecture
Automated
Testing
Continuous
Deployment
Security

49. Security
• Shorter TTL == less
chance for an attack
to take hold

50. Continuous scalingNo servers to
manage
Never pay for idle
– No cold servers
(only happy
accountants)
Benefits of AWS Lambda

51. What does Lambda do for you?
• Scales server capacity automatically
• API to trigger execution
• Ensures function is executed in parallel
and at scale
• Logging, monitoring, etc
• Easy pricing

52. Monitoring
• Everything is in Cloudwatch or Cloudwatch
logs

53. Pricing
• Choose your RAM
from 128MB to
1500MB
• CPU and Network
scaled based on RAM

54. Cost Comparison

55. Cost Comparison
There’s about 2.5M seconds in a month, so 3M requests is about 1.2 per second
The T2.Small is $18.98 a month, more than Lambda already

59. Lambda lets you manage
your code and infrastructure
in the same place

60. Lambda lets your developers manage
their code and your infrastructure
in the same place

61. All the problems you have with
microservices are multiplied 10X
with serverless

62. Problems with
Serverless
• efficient dependency usage
• local dev environments
• making sure everyone has the same
dependencies
• knowing when someone else is
deploying the same function

66. Testing
• You can’t test the network, but
a good application test should
obviate the need to do so.
• Not really a solved problem.
Can do local testing.
• Can also send json to the
function and compare the
results.

67. Tips and Tricks
• Limit your function size
(JVM startup time
especially)
• Remember execution is
async
• Don’t assume function
container reuse but
take advantage of it

68. Tips and Tricks
• Remember the 500MB in /tmp
• Use function aliases
• Use the included logger

69. Tips and Tricks
• Set up alarms on all
Lambda Cloudwatch
metrics
• Avoid throttling by using
SNS between any service,
such as S3
• Beware of infinite loops by
functions calling each other.

70. Avoiding Infinte Loops
• With a distributed team, this
is an easy mistake to make
• To avoid it, pass a call stack
and check for self in the
stack
8

71. So where does that leave
us?
Serverless or containers?
Services or monorepo?

72. Proper
Microservices
Architecture
Service
and
Resource
Discovery
Network
and Traffic
Config
Automated
Testing
Continuous
Deployment
Security
Monitoring
and Alerting

73. Actionable MetricsMonitoring
and Alerting

74. Actionable MetricsMonitoring
and Alerting

75. Monitoring
and Alerting Actionable Metrics

76. Choosing a metric
Monitoring
and Alerting

77. Self Serve is the Key
• Let developers choose what
metrics to submit
• What graphs they put on
their dashboards
• What to alert on
• They are closest to the app,
so they know best
Monitoring
and Alerting

78. Alert on increase of
failure, not lack of success
Increase in 500s Decrease in 200s
Monitoring
and Alerting
👍 👎

79. P50, P90, P99
Monitoring
and Alerting

80. P50, P90, P99
0
15
30
45
60
1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 11m 12m 13m 14m 15m
P50 P90 P99
Monitoring
and Alerting

81. Immutable Data
• If you can, write your software
such that everything in the
cache is immutable.

82. Moving data is
the single
biggest cost your
distributed
system will incur

83. But you need to move data for reliability,
so it’s a tradeoff
Use queues as often
as possible

84. Which is greater?
Queues or Sliced Bread?

85. Queuing
• Queue anything you are
writing to a data store
• Monitor your queue
lengths for great insight
and scaling!
0
2
4
6
8
10
12
14
16
18
1 3 5 7 9 11 13 15 17 19

86. 0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Items
Seconds
Queue Depth

87. 0
10
20
30
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Cumulative Flow Diagram
Items
Seconds
Arrivals
Departures

88. Capacity utilization increases
queues exponentially
• Every time you reduce the excess capacity
by 1/2, you double the average queue size.
• This has a direct effect on the ratio of wait
time to work time for a single work unit
• Use this to balance cost vs. latency
0
2
4
6
8
10
10 20 30 40 50 60 70 80 90 100

89. • Variability increases
queue sizes linearly
• Operating at high
utilization increases
variability
The price of
variability

90. The price of
variability

91. The price of
variability
Fast Medium Slow

92. Chaos Engineering
• Simulate things
that go wrong
• Find things that
are different

93. Two most important
things to test
Instance Loss
Increased Latency

94. Logs

95. • What went wrong?
• How could we have detected it
sooner?
• How could we have prevented it?
• How can we prevent this class of
problem in the future?
• How can we improve our behavior
for next time?
Ask the key questions:
Incident Reviews

96. Takeaways
• Infrastructure as Code
• Microservices/Serverless/
Monolith
• Queuing Theory
• Chaos Engineering
• Logs Suck
• Incident reviews

97. Questions?
Email: {anything}@jedberg.net
Twitter: @jedberg
Web: www.jedberg.net
Facebook: facebook.com/jedberg
Linkedin: www.linkedin.com/in/jedberg
Company: minops.com