Serverless Architectures on AWS Lambda - The OpsGenie Experience

1. SERHAT CAN • SOFTWARE ENGINEER • @SRHTCN•
Serverless Architectures on
AWS Lambda
The OpsGenie Experience

2. Forward IT alerts to
OpsGenie
View alerts using the
mobile app
Get notified via email,
SMS, mobile push
and phone calls
Define On-call schedules
and escalations policies
Respond to alerts
directly from
the OpsGenie app

3. • Does not mean running code without servers
• Cloud provider manages servers and provisioning
• No more long-running virtual machines
• Event driven - starts & stops execution between each
request
• Pay only when your code executed
What is Serverless (FaaS)

4. What is Serverless (FaaS)

6. • Amazon S3
• Amazon DynamoDB
• Amazon Kinesis Streams
• Amazon Simple Notification Service
• Amazon Simple Email Service
• Amazon Cognito
• AWS CloudFormation
• Amazon CloudWatch Logs
Events
• Amazon CloudWatch Events
• AWS CodeCommit
• Scheduled Events (powered by Amazon CloudWatch Events)
• AWS Config
• Amazon Echo
• Amazon Lex
• Amazon API Gateway
• Client SDK

7. Limitations
Language limitations (Java, Nodejs, Python, C#)
Max memory is 1,5GB
Maximum execution duration per request 300 seconds
Concurrent executions 1000 (Immediate Increase to 3000)

8. Some of the Important Features
• VPC Support
• Environment Variables
• Dead Letter Queue
• Function Versioning and Aliases
• Tagging

9. Real Life
Use Cases
from OpsGenie

10. • Java8 (mostly)
• Node.js & Python for custom solutions
• Custom Deployment Solution written in Java
(in DevOps Team)
• Serverless.js
Our Serverless Stack

11. • Operating on multiple AZ’s is good but not enough
• Even services with 99.99999999999% durability, 99.99%
availability can’t be trusted
• Needed to be ready for regional failures
• Goal is full region switch within a few minutes if not seconds
• Next step: operate on multiple regions simultaneously
Reliability Needs of OpsGenie

12. Use Cases

13. DynamoDB Cross Region Replication
Motivation:
• Needed to replicate our data in DynamoDB to another region
for high availability

14. DynamoDB
Cross Region
Replication

15. Reporting
Motivation:
• Robust architecture with minimal effort.
• Extremely scalable reporting service with SQL like aggregation
features.

16. Reporting

17. DynamoDB AutoScale
Motivation:
• Needed an automated way to scale DynamoDB to avoid
read & write throttles
• Capacity needs to be adjusted during migration

18. DynamoDB AutoScale

19. Elasticsearch Indexing
Motivation:
• Needed to index our data to ES
• Our ES data should be available in every region we operate
• Cross region replication should not be included in the
application logic as much as possible

20. Elasticsearch Indexing - Very Soon

21. • AWS Lambda is a life saver for custom solutions, because;
• Customers do not need to manage servers
• Easy to deploy (give a .zip file)
• Real pay what you use pricing
(not hourly but for each 100ms execution time)
Custom Solutions & Integrations

22. Custom Solutions & Integrations

23. Custom Solutions & Integrations

24. Service and Incident Management
Motivation:
• A fully featured pilot project implement
in Serverless Architecture

25. Service and Incident Management

26. Challenges
Java Cold Start Time
Proper Monitoring
Deployment

27. • Lambda functions, written in Java, experience a latency
during new container initialization mainly caused by Java
class loading
• To avoid this we are experimenting an approach called
“Warmup Lambda”
• Warmup Lambda is a lambda function that aims to keep a
given number of concurrent lambda functions up, to avoid the
startup latency
Java Cold Start Time

28. • No way to attach NewRelic agent to AWS Lambda
• We wanted to use NewRelic’s capabilities like NRQL
• Logs are processed using Cloudwatch trigger and sent to
NewRelic, Elasticsearch etc.
• Just started experimenting new
AWS X-Ray (Distributed Tracing System)
Proper Monitoring

29. AWS X-Ray
• X-Ray provides an end-to-end view of requests as they travel
through your application, and shows a map of your
application’s underlying components

30. AWS X-Ray

31. AWS X-Ray

32. • Blue green deployment using Alias support
• Allows our applications to call different Lambda function
versions during deployment using Lambda SDK
Deployment

33. For more
engineering.opsgenie.com

35. Thank you!
Stay connected!
@srhtcn