Looking in from the outside, serverless seems so simple! And yet, many companies are struggling on their journey to serverless. In this talk, I highlight a number of mistakes companies are making when they adopt serverless.

1. MIND
THE
POTHOLES
MIND
THE
POTHOLES

2. Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

3. http://bit.ly/yubl-serverless

4. Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

6. Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant

7. What do you mean
by ‘serverless’?

8. “Serverless”

10. Gojko Adzic
It is serverless the same way
WiFi is wireless.
http://bit.ly/2yQgwwb

11. Serverless means…
don’t pay for it if no-one uses it
don’t need to worry about scaling
don’t need to provision and manage servers

12. in other words, it’s a lot like taking a cab

13. Ownership
Fuel
Navigate
To get there!
Focus on
getting there!

14. HW Ownership
OS
Runtime & Scale
Code
Focus on
getting there!
Physical
Servers
Virtual
Machines
Containers Serverless

15. Nano Services Self Managed Cost Paradigm
ChangeAsync
Dynamic agile env

16. “why are we failing at this?”

17. hidden dangers

18. monolith microservices serverless

19. monolith microservices serverless
observability
distributed
systems
bounded
context

20. monolith microservices serverless
observability
distributed
systems
bounded
context

21. monolith microservices serverless
observability
distributed
systems
bounded
context
event driven

22. monolith serverless
missing learnings
from microservices

23. monolith serverless
missing learnings
from microservices
poor decisions

24. #1
not letting go of legacy
thinking

25. “we’re doing serverless,
but why aren’t thing
going faster?”

26. Socio Technical

27. there are no silver bullets

28. centralised team
Team A Team B Team C Team D …

29. “but the developers don’t understand AWS and how
our infrastructure is set up”

30. “but the developers don’t understand AWS and how
our infrastructure is set up”
let’s solve this
problem instead!

31. what got you here won’t get you there

32. if (path == “/user” && method == “GET”) {
return getUser(…);
} else if (path == “/user” && method == “DELETE”) {
return deleteUser(…);
} else if (path == “/user” && method == “POST”) {
return createUser(…);
} else if ….
Monolithic Functions

33. GET /user
POST /user
DELETE /user
Single-Purposed Functions

34. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user

35. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
find related
functions by prefix

36. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
discoverability
(without having to dig into the code)

37. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
what does it do?

38. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem

39. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem
no least privilege…

40. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

41. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

43. more dependecies equals
slower cold start

44. author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)
worse cold start
performance

45. keep functions simple, and single-purposed

46. #2
one account that rules
them all

47. mind the shared limits

48. no. of DynamoDB tables
no. of API Gateway regional APIs
no. of API Gateway edge-optimized APIs
no. of Kinesis shards
no. of IAM roles
no. of S3 buckets
no. of CloudFormation stacks
no. of SNS subscription filters
no. of SSM parameters
…
Resource Limits

49. DynamoDB read & write
API Gateway requests/second
Lambda concurrent executions
SSM parameter ops/second
…
Throughput Limits

50. One account per Team per Environment

53. compartmentalise security breaches

54. #3
do first, research later

55. https://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/

59. the platforms need to do better at educating users on
how to choose between different services

60. SNS vs SQS vs Kinesis vs MKS?
the platforms need to do better at educating users on
how to choose between different services

61. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many

62. https://medium.com/theburningmonk-com/all-my-posts-on-serverless-aws-lambda-43c17a147f91

63. https://www.jeremydaly.com/newsletter/

64. #4
not using a deployment
toolkit

66. https://lumigo.io/blog/comparison-of-lambda-deployment-frameworks/

67. don’t write your own deployment framework

68. #5
console-driven
development

71. #6
one repo per function

72. github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

73. github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

74. monorepo?

75. github
repo

76. https://lumigo.io/blog/mono-repo-vs-one-per-service/

77. one repo per service?

85. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

86. CI/CD pipeline per service

87. functions are deployed together, as a stack

88. unencrypted secrets
in env vars
#7

89. secrets should NEVER be in plain text in env variables

90. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …

91. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …
yay!

94. SSM Parameter Store
Secret 1
Secret 2
IAM
fetch at cold start,
cache,
invalidate every x mins

95. https://github.com/middyjs/middy

97. SSM Parameter Store
Secret 1
Secret 2
IAM
switch to Higher
Throughput if you need
more than 40 ops/s

98. not following least
privilege principle
#8

102. missing DLQs
#9

103. async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2v7Kc3b
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS

104. async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2vs2lIg
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS
Lambda handles retries
(twice, then DLQ)

105. configure DLQ for async functions so you don’t lose failed events

106. too much/too little
concurrency
#10

107. “Lambda generates too much load for the downstream system”

108. one invocation
per message
SNS
Lambda

109. Downstream
System
SNS
Lambda

110. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many

111. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis

112. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out

113. if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out
SQS DynamoDB
Streams

114. ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many

115. cold starts
#11

116. “cold starts only happen to the first request”

117. function invocationconcurrent execution
i.e. a container

118. function invocationconcurrent execution
i.e. a container
class instance method call

119. Lambda scales the number of concurrent executions
based on traffic

120. existing “containers” are reused where possible

121. time
invocation

122. time
invocation
invocation

123. time
invocation
invocation

124. time
invocation
invocation
invocation
invocation

125. time
invocation
invocation
invocation
invocation
invocation
invocation

126. time
invocation
invocation
invocation
invocation
invocation
invocation

127. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation

128. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

129. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

130. time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

135. time
invocation
invocation
ping
invocation
invocation
invocation
ping ping

136. Lambda warmers don’t work when you have > 1
concurrent executions

137. FREQUENCY DURATION

138. FREQUENCY DURATION
dictated by user traffic,
out of your control

139. cold starts is generally not an issue if you have a
steady traffic pattern

140. time
req/s

141. time
req/s
El Classico

142. time
req/s
lunch dinner

143. FREQUENCY DURATION
optimize this!

144. minimise the duration of cold starts so they
fall within acceptable latency range

145. RDS connection
handling
#12

146. default RDS configs are bad for Lambda

147. default RDS configs are bad for Lambda
idle connections are
not closed
too many connections
per “container”
max open connection
is too low

148. https://www.jeremydaly.com/manage-rds-connections-aws-lambda/

149. set “wait_timeout” and “interactive_timeout” to 10 mins
(default is 8 hours!)

150. increase “max_connections” setting

151. set client socket pool size to 1

153. (lack of) observability
#13

154. happened system repaireduser impact
reduce MTTR

155. Identify & Resolve
Issues
Understanding
costs
Visibility

156. Identify & Resolve
Issues
Understanding
costs
Visibility

157. happened system repaireduser impact
MTTDiscovery

159. “What alerts should I have?”

160. It depends on what you’re building…

161. But, this is a good starting point

162. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

163. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

164. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

165. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

166. SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

167. “Can’t you codify these?”

170. https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

171. @theburningmonk
theburningmonk.com
github.com/theburningmonk