Serverless functions (like AWS Lambda, Google Cloud Functions, and Azure Functions) have the ability to scale almost infinitely to handle massive workload spikes. While this is a great solution for compute, it can be a MAJOR PROBLEM for other downstream resources like RDBMS, third-party APIs, legacy systems, and even most managed services hosted by your cloud provider. Whether you’re maxing out database connections, exceeding API quotas, or simply flooding a system with too many requests at once, serverless functions can DDoS your components and potentially take down your application. In this talk, we’ll discuss strategies and architectural patterns to create highly resilient serverless applications that can mitigate and alleviate pressure on “non-serverless” downstream systems during peak load times.

1. Building Resilient Serverless Systems
with Non-Serverless Components
Jeremy Daly
@jeremy_daly

2. Jeremy Daly
• Independent consultant working with companies
building in the cloud
• 20+ year veteran of technology startups
• Started working with AWS in 2009 and started using
Lambda in 2015
• Blogger (jeremydaly.com), OSS contributor, speaker
• Publish the Off-by-none serverless newsletter
• Host of the Serverless Chats podcast and the
Serverless Spotlight web series
@jeremy_daly

3. Agenda
• What is resiliency and what is serverless?
• Working with “less-than-scalable” RDBMS
• Using unreliable APIs
• Managing API quotas
• Decoupling our services
• Other non-serverless components
@jeremy_daly

4. What is resiliency?
@jeremy_daly
“The ability of a software solution to absorb the impact
of a problem in one or more parts of a system, while
continuing to provide an acceptable service level to the
business.” ~ IBM
IT’S NOT ABOUT PREVENTING FAILURE
IT’S UNDERSTANDING HOWTO GRACEFULLY DEALWITH IT

5. What does it mean to be Serverless?
• No server management
• Flexible scaling
• Pay for value
• Automated high availability
• LOTS of configuration and knowledge of cloud services
• Highly event-driven
@jeremy_daly
Flexible scaling 👈

6. What does it mean to be Serverless?
@jeremy_daly
ElastiCache
RDS
EMR Amazon ES
Redshift
Fargate
Anything “on EC2”Lambda Cognito Kinesis
S3 DynamoDB SQS
SNS API Gateway
AppSync IoT Comprehend
Serverless Managed Not Serverless
DocumentDB
(MongoDB)
Managed Streaming
for Kafka
Definitely
EventBridge

7. Everything has limits!
• Reserved Concurrency 🚦
• FunctionTimeouts ⏳
• Memory Limits 🧠
• NetworkThroughput 🚰
Some components are better than others
@jeremy_daly
Know
Your
Limits

8. Simple Serverless Web Service
Client
API Gateway Lambda DynamoDB
@jeremy_daly
Highly Scalable Highly Scalable Highly Scalable

9. “I want my, I want my, I want my SQL”
~ Dire Straits

10. Simple Serverless Web Service
Client
API Gateway Lambda
@jeremy_daly
Highly Scalable Highly Scalable NotThat Scalable 😳
RDS
^
not so
RDBMS and FaaS don’t play nicely together:
• Concurrency model doesn’t allow connection pooling
• Limited number of DB connections available
• Recycled containers create zombies

11. Ways to Manage DB Connections
• Increase max_connections setting
• Limit concurrent executions
• Lower your connection timeouts
• Limit connections per username
• Close connection before function ends
@jeremy_daly
🤞
😡
⚠
🎲
😱
👎

12. Better Ways to Manage DB Connections
• Implement a good caching strategy 💾
• Buffer events for throttling and durability 🏋
• Utilize a proxy service 🛰
• Manage connections ourselves 🤔
@jeremy_daly
👎

13. misshit
Implement a good caching strategy
Client API Gateway RDSLambda
Elasticache
Key Points:
• Create new RDS connections ONLY on misses
• Make sureTTLs are set appropriately
• Include the ability to invalidate cache
@jeremy_daly
YOU STILL NEEDTO
SIZEYOUR DATABASE
CLUSTERS APPROPRIATELY

14. Do you really need immediate feedback?
Synchronous Communication ⏳
Services can be invoked by other services and must wait for a reply.
This is considered a blocking request, because the invoking service
cannot finish executing until a response is received.
Asynchronous Communication 🚀
This is a non-blocking request. A service can invoke (or trigger)
another service directly or it can use another type of communication
channel to queue information.The service typically only needs to wait
for confirmation (ack) that the request was received.
@jeremy_daly

15. RDS
Buffer events for throttling and durability
Client API Gateway
SQS
Queue
SQS
(DLQ)
Lambda Lambda
(throttled)
ack
“Asynchronous”
Request
Synchronous
Request
@jeremy_daly
Key Points:
• SQS adds durability
• Throttled Lambdas reduce downstream pressure
• Failed events are stored for further inspection/replay
Limit the
concurrency to match
RDS throughput
x
Utilize Service
Integrations

16. Utilize a Proxy Service
• PgBouncer 🏀
• SQL Relay 🏃
@jeremy_daly
Client
API
Gateway
Lambda RDSEC2x
Fargate
🙀
• Amazon RDS Proxy
In a “serverless” application?
FOR SHAME! 😿

17. Amazon RDS Proxy
• Pool and share database connections
• Reroute requests on failover (66% reduction in
failover time)
• Use IAM credentials for database access control
• “Connection pool maximum connections”
• Only works on Amazon RDS (you still need to
do capacity planning)
• It is an added cost!
@jeremy_daly
Silver Bullet?

18. Manage connections ourselves
1. Count open connections
2. Close connection if connection ratio threshold exceeded
3. Close sleeping connections with high time values
4. Retry connections with exponential back off
@jeremy_daly

19. Serverless MySQL
https://github.com/jeremydaly/serverless-mysql
@jeremy_daly

20. Count open connections
@jeremy_daly
Query the
processlist to get
the total number
of active
connections

21. Close connection if over ratio threshold
@jeremy_daly
If we exceed the
connection ratio
Calculate our timeout
Try to kill zombies
If no zombies,
terminate connection
Else, just try to kill
zombies

22. Close sleeping connections with high time values
@jeremy_daly
Query processlist for zombies
Kill zombies

23. Retry connections with exponential back off
@jeremy_daly
If error trying to connect
Retry with Jitter

24. Does this really work?
@jeremy_daly
• Aurora Serverless (2 ACUs)
• 90 connections available
• 1,024 MB of memory
• 500 users/sec for one minute
• Avg. response time was 41 ms
• ZERO ERRORS

25. We shouldn’t have to do this!
@jeremy_daly
Amazon
Aurora Serverless
Aurora Serverless
DATA API
Doesn’t solve the
max_connections issue
Slower throughput, not quite
ready for synchronous workloads
Amazon
RDS Proxy
Added cost, still doesn’t
address auto scaling
🥰

26. Third-Party APIs

27. Manage calls to third-party APIs
• Implement a good caching strategy 💾
• Buffer events for throttling and durability 🏋
• Implement circuit breakers 🚦
@jeremy_daly

28. DynamoDB
Stripe API
The Circuit Breaker
Client API Gateway Lambda
Key Points:
• Cache your cache with warm functions
• Use a reasonable failure count
• Understand idempotency
Status
Check CLOSED
OPEN
Increment Failure Count
HALF OPEN
“Everything fails all the time.”
~WernerVogels
@jeremy_daly
🔥
🔥
🔥
🔥
🔥
Elasticache
or

29. What about quotas?
• Concurrency has no effect on frequency ⏰
• Stateless functions are not coordinated 😿
• Step Functions StandardWorkflows would be very expensive 💰
• Adding state wouldn’t prevent needless invocations 🗑
@jeremy_daly

30. Can we build a better system?
• 100% serverless
• Cost effective
• Scalable
• Resilient
• Efficient
• Coordinated
@jeremy_daly

31. Lambda Orchestrator
(concurrency 1)
The Lambda Orchestrator
DynamoDB
LambdaWorker
LambdaWorker
LambdaWorker
Concurrent Executions
of the SAME function
SQS (DLQ)
@jeremy_daly
CloudWatch Rule
(trigger every minute)
SQS QueueSQS (DLQ)
Status?
Gmail API
250 Quota Units
per minute

32. Decoupling Our Services

33. @jeremy_daly
Multicasting with EventBridge
Key Points:
• Allows multiple subscribers with RULES, PATTERNS and FILTERS
• Forward events to other accounts
• 24 hours of automated retries
Asynchronous
“PutEvents” Request
ack
w/ event id
Amazon
EventBridge
Lambda
SQS
Client
Step Function
Event Bus
+16 others

34. Key Points:
• Filter events to selectively trigger services
• Manage throttling/quotas per service
• Use Lambda Destinations with asynchronous events
Stripe API
@jeremy_daly
Distribute & Throttle
ack
SQS
Queue Lambda
(concurrency 25)
Client API
Gateway
Lambda
Order Service
"total": [{ "numeric": [ ”>", 0 ]}]
RDS
SQS
Queue Lambda
(concurrency 10)
SMS Alerting Service
Twilio API
SQS
Queue Lambda
(concurrency 5)
Billing Service
"detail-type": [ "ORDER COMPLETE" ]
EventBridge

35. Other non-serverless components
• Managed Services
• Other cloud services (MongoDB Atlas, ElasticSearch, etc.)
• Legacy Systems
• Our own serverless APIs 🤔
@jeremy_daly

36. Non-serverless components are inevitable
• Know the limits of your components
• Use a good caching strategy
• Embrace asynchronous processes
• Buffer and throttle events to distributed systems
• Utilize eventual consistency
@jeremy_daly
👈

37. Thank You!
Blog: JeremyDaly.com
Podcast: ServerlessChats.com
Newsletter: Offbynone.io
DDBToolbox: DynamoDBToolbox.com
Lambda API: LambdaAPI.com
GitHub: github.com/jeremydaly
Twitter: @jeremy_daly
@jeremy_daly