You shall not Fail! (in the face of turbulent conditions)
•
1 j'aime•643 vues
Lambda gives you a lot of scalability and multi-AZ out-of-the-box, but still, things can go wrong in production. There are region-wide outages, and performance degradation in services your function depends on can cause it to time out or error. And what if you're dealing with downstream systems that just aren't as scalable and can't handle the load you put on them? The bottomline is many things can go wrong and they often do at the worst times. The goal of building resilient systems is not to prevent failures, but to build systems that can withstand these failures. In this talk, we will look at a number of practices and architectural patterns that can help you build more resilient serverless applications. Such as multi-region, active-active, employing DLQs and surge queues and using chaos experiments to identify failure modes before they manifest in production. The recording is available here: https://www.youtube.com/watch?v=elVeOYYtLM0
Recommandé
Recommandé
Contenu connexe
Plus de Yan Cui
Plus de Yan Cui (20)
Dernier
Dernier (20)
You shall not Fail! (in the face of turbulent conditions)
- 1. @theburningmonk @sarutule 1 "I COME BACK TO YOU NOW AT THE TURN OF THE TIDE" 1
- 3. @theburningmonk @sarutule Failures in distributed systems 3
- 4. @theburningmonk @sarutule Failures on load: exhaustion of resources 4
- 5. @theburningmonk @sarutule Failures on load: exhaustion of resources 5
- 6. You Shall Not Fail! in the face of turbulent conditions TM what is RESILIENCE chaos ENGINEERING multi-region STRATEGIES retries & TIMEOUTS lambda SCALING decoupled INVOCATION
- 7. PRODUCERS Yan Cui, @theburningmonk Sara Gerion, @sarutule SPEAKERS Yan Cui, @theburningmonk Sara Gerion, @sarutule SPEAKING AT AWS Community Summit Online SPECIAL THANKS Phil Horn Joe Park
- 8. @theburningmonk @sarutule 8 Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
- 9. @theburningmonk @sarutule 9 Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
- 11. @theburningmonk @sarutule 11 Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery
- 12. SARA GERION Italian living in Amsterdam, The Netherlands Passionate about cloud, scalability, resilience Twitter: @Sarutule Backend engineer at DAZN @dazneng Director of Tech at SheSharp @SheSharpNL
- 14. @theburningmonk @sarutule Serverless - multiple AZ’s out of the box 14 Total resources created: 1 API Gateway 1 Lambda
- 17. @theburningmonk @sarutule REST API - Lambda autoscaling 17 Concurrency limits: 3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Later bursts: 500 new containers / each minute
- 18. @theburningmonk @sarutule REST API - Lambda autoscaling 18 X number of execution environments pre-initialized (ready to respond to invocations) Note: standard burst concurrency limits when over the provisioned capacity Concurrency limits: 3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Later bursts: 500 new containers / each minute
- 19. @theburningmonk @sarutule REST API - Lambda autoscaling 19 Adjustable provisioned capacity based on CloudWatch metrics X number of execution environments pre-initialized (ready to respond to invocations) Note: standard burst concurrency limits when over the provisioned capacity Concurrency limits: 3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Later bursts: 500 new containers / each minute
- 20. @theburningmonk @sarutule REST API - Lambda limitations & throttling 20
- 21. @theburningmonk @sarutule HOW TO SOLVE IT? 21
- 22. @theburningmonk @sarutule HOW TO SOLVE IT? IT DEPENDS 22
- 23. @theburningmonk @sarutule The importance of retry policies 23
- 24. @theburningmonk @sarutule Scenario: client only needs an acknowledgement 24
- 25. @theburningmonk @sarutule If fast acknowledgement not possible…
- 27. @theburningmonk @sarutule Scenario: predictable spikes 27 Holidays, weekends, celebrations (Black Friday) Planned launch of resources (new series available) Sport events
- 28. @theburningmonk @sarutule Scenario: unpredictable spikes 28 Traffic generated by user actions Jennifer Aniston’s first post
- 29. @theburningmonk @sarutule Possible mitigations for REST API’s 29 Use 1 Lambda for each endpoint 29
- 30. @theburningmonk @sarutule One Lambda function for each endpoint 3030
- 31. @theburningmonk @sarutule Possible mitigations for REST API’s 31 Use 1 Lambda for each endpoint Raise limits with an AWS support ticket 31
- 32. @theburningmonk @sarutule Possible mitigations for REST API’s 32 Use 1 Lambda for each endpoint Optimise performance Raise limits with an AWS support ticket 32
- 33. @theburningmonk @sarutule Possible mitigations for REST API’s 33 Use 1 Lambda for each endpoint Optimise performance Offload computing operations to an async flow (SQS, SNS, …) Raise limits with an AWS support ticket 33
- 34. @theburningmonk @sarutule Offload computing operations to queues 34
- 35. @theburningmonk @sarutule Offload computing operations to queues 35
- 36. @theburningmonk @sarutule Possible mitigations for REST API’s 36 Use 1 Lambda for each endpoint Optimise performance Offload computing operations to an async flow (SQS, SNS, …) Use provisioned capacity (plus autoscaling) Raise limits with an AWS support ticket 36
- 37. @theburningmonk @sarutule Reminder: beware of long timeouts 37 API Gateway Integration timeout Default: 29s Lambda Timeout Max: 15 minutes SQS Visibility timeout Default: 30s Min: 0s Max: 12 hours
- 41. @theburningmonk @sarutule Active-active & data replication 41
- 42. @theburningmonk @sarutule Multi-region architecture - benefits & tradeoffs 42 Protection against regional failures
- 43. @theburningmonk @sarutule Multi-region architecture - benefits & tradeoffs 43 Protection against regional failures Higher complexity
- 44. @theburningmonk @sarutule Multi-region architecture - benefits & tradeoffs 44 Protection against regional failures Higher complexity Very hard to test
- 47. @theburningmonk @sarutule 47 “the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production” principlesofchaos.org
- 48. @theburningmonk @sarutule 48 “You don't choose the moment, the moment chooses you! You only choose how prepared you are when it does.” Fire Chief Mike Burtch
- 49. @theburningmonk @sarutule 49 identify weaknesses before they manifest in system-wide, aberrant behaviors GOAL
- 50. @theburningmonk @sarutule 50 learn about the system’s behavior by observing it during a controlled experiments HOW
- 51. @theburningmonk @sarutule 51 learn about the system’s behavior by observing it during a controlled experiments HOW game days failure injection
- 52. @theburningmonk @sarutule 52 MUST KILL SERVERS! RAWR!! RAWR!! ahhhhhhh!!!! HELP!!! OMG!!! F***!!!
- 54. @theburningmonk @sarutule 54 STEP 1. define steady state i.e. “what does normal look like”
- 55. @theburningmonk @sarutule 55 STEP 2. hypothesis that steady state continues in control and experimental group e.g. “the system stays up if a server dies”
- 56. @theburningmonk @sarutule 56 STEP 3. inject realistic failures e.g. “slow response from 3rd-party service”
- 57. @theburningmonk @sarutule 57 STEP 4. try to disprove hypothesis i.e. “look for difference between control and experimental group”
- 59. @theburningmonk @sarutule 59 identify weaknesses before they manifest in system-wide, aberrant behaviors GOAL
- 60. @theburningmonk @sarutule 60 “Corporation X lost millions due to a chaos experiment went wrong and destroyed key infrastructure, resulting in hours of downtime and unrecoverable data loss.”
- 61. @theburningmonk @sarutule 61 Chaos Engineering doesn't cause problems. It reveals them. Nora Jones
- 63. @theburningmonk @sarutule 63 CONTAINMENT run experiments during office hours
- 64. @theburningmonk @sarutule 64 CONTAINMENT run experiments during office hours let others know what you’re doing, no surprises
- 65. @theburningmonk @sarutule 65 CONTAINMENT run experiments during office hours let others know what you’re doing, no surprises avoid important dates
- 66. @theburningmonk @sarutule 66 CONTAINMENT run experiments during office hours let others know what you’re doing, no surprises avoid important dates make the smallest change possible
- 67. @theburningmonk @sarutule 67 CONTAINMENT run experiments during office hours let others know what you’re doing, no surprises avoid important dates make the smallest change possible have a rollback plan before you start
- 69. @theburningmonk @sarutule 69 by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
- 70. @theburningmonk @sarutule 70 chaos monkey kills an EC2 instance latency monkey induces artificial delay in APIs chaos gorilla kills an AWS Availability Zone chaos kong kills an entire AWS region
- 72. @theburningmonk @sarutule 72 there are no servers to kill! SERVERLESS
- 73. @theburningmonk @sarutule 73 by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
- 74. @theburningmonk @sarutule 74 by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
- 79. @theburningmonk @sarutule 79 “what if DynamoDB has an elevated error rate?”
- 80. @theburningmonk @sarutule 80 hypothesis: the AWS SDK retries would handle it
- 82. @theburningmonk @sarutule 82 TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms
- 83. @theburningmonk @sarutule 83 TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) this is Marc Brooker’s fav formula!
- 85. @theburningmonk @sarutule 85 result: function times out after 6s (hypothesis is disproved)
- 86. @theburningmonk @sarutule 86 action: set max retry count + fallback
- 87. @theburningmonk @sarutule 87 outcome: a more resilient system
- 88. @theburningmonk @sarutule 88 “what if service X has elevated latency?”
- 89. @theburningmonk @sarutule 89 hypothesis: our try-catch would handle it
- 91. @theburningmonk @sarutule 91 result: function times out after 6s (hypothesis is disproved)
- 92. @theburningmonk @sarutule 92 TIL: most HTTP client libraries have default timeout of 60s. API Gateway has an integration timeout of 29s. Most Lambda functions default to timeout of 3-6s.
- 98. @theburningmonk @sarutule 98 outcome: a more resilient system
- 100. @theburningmonk @sarutule Failures in distributed systems 100
- 101. @theburningmonk @sarutule Serverless - multiple AZ’s out of the box 101 Total resources created: 1 API Gateway 1 Lambda
- 102. @theburningmonk @sarutule Beware of timeouts 102 API Gateway Integration timeout Default: 29s Lambda Timeout Max: 15 minutes SQS Visibility timeout Default: 30s Min: 0s Max: 12 hours
- 103. @theburningmonk @sarutule Offload computing operations to queues 103
- 105. @theburningmonk @sarutule 105 “You don't choose the moment, the moment chooses you! You only choose how prepared you are when it does.” Fire Chief Mike Burtch
- 106. @theburningmonk @sarutule 106 by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
- 107. @theburningmonk @sarutule 107 by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361