Performance monitoring is a crucial part when running serverless applications. Serverless applications are highly distributed with business workflows and data spread across multiple managed services and APIs. As a result, in order to make sure that everything is running correctly in production you’d need full observability into the application that will help you find the root cause, and troubleshoot quickly. In this workshop, you will learn the critical observability considerations when operating a serverless application, what is distributed tracing and why it's important, and what tools can be used to gain full observability to your applications.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
W e b i n a r
Serverless Observability
Heitor Lessa
Specialist Solutions Architect
AWS
Ran Ribenzaft
CTO
Epsagon
@heitor_lessa @ranrib

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS operational responsibility models
On-Premises Cloud
Less More
Compute Virtual Machine
Amazon EC2 AWS Elastic Beanstalk AWS LambdaFargate
Databases MySQL
MySQL on EC2 Amazon RDS MySQL Amazon Aurora Amazon Aurora Serverless Amazon DynamoDB
Storage Storage
Amazon S3
Messaging ESBs
Amazon MQ Amazon Kinesis Amazon SNS / SQS
Analytics
Hadoop Hadoop on EC2 Amazon EMR Amazon Elasticsearch Service Amazon Athena

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Three Pillars of Observability
Event Logs Metrics Tracing
Distributed Systems Observability by Cindy Sridharan

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability
Event Logs Metrics Tracing
Log aggregation
& analytics
VisualizationsAlerting

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability
CloudWatch
Logs
Metrics Tracing
CloudWatch Logs
Insights
VisualizationsAlerting

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – CloudWatch Logs, Insights

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability
CloudWatch
Logs
CloudWatch
Metrics
Tracing
CloudWatch Logs
Insights
Visualizations
CloudWatch
Alarms

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – CloudWatch Custom Metrics
const metricData = await cloudWatch.putMetricData({
MetricData: [
{
MetricName: 'My Business Metric',
Dimensions: [
{
Name: 'Location',
Value: 'Paris'
}
],
Timestamp: new Date(),
Value: 123.4
}
],
Namespace: METRIC_NAMESPACE
}).promise();
• Metric name
• Dimensions
• Timestamp
• Value
• Namespace

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – CloudWatch Metrics/Dashboard
[DEV 302] Monitor All Your Things: Amazon CloudWatch in Action with BBC

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability
CloudWatch
Logs
CloudWatch
Metrics
Tracing
CloudWatch Logs
Insights
Visualizations
CloudWatch
Alarms
CloudWatch
Metric Filter

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – CloudWatch Logs Metric Filter
[DEV 302] Monitor All Your Things: Amazon CloudWatch in Action with BBC

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – CloudWatch Logs Subscription
Amazon
CloudWatch
Amazon Elasticsearch Service
API
AWS Lambda
Anywhere

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability
CloudWatch
Logs
CloudWatch
Metrics
AWS X-Ray
Traces
CloudWatch Logs
Insights
AWS X-Ray Service
Graphs
CloudWatch
Alarms
CloudWatch
Metric Filter

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – AWS X-Ray Service Graph

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Observability – AWS X-Ray Traces
The story of the request

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. W e b i n a r © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Serverless Shop
Simple e-commerce example
Browse and purchase products
Built 100% on Serverless resources
https://store.epsagon.io

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Serverless Shop: Architecture
Client
Amazon S3
Amazon
API Gateway
API
/get_items
/buy_item
Website
Amazon
CloudFront
AWS Lambda
Catalog Images
API
Google
Images
Amazon DynamoDB
Catalog DB
SYNCSYNC
Payment
AWS Lambda Amazon SNS
Amazon SESAWS LambdaAmazon SNS
Order Fulfillment
API
SYNC

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Serverless Shop: Catalog feature
AWS Lambda
Catalog Images
API
Google
Images
Amazon DynamoDB
Catalog DB

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Serverless Shop: Payment feature
ASYNC
Amazon SESAWS LambdaAmazon SNS
Order FulfillmentPayment
AWS Lambda

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business flows and KPIs
Catalog
Time to Catalog
How long it takes for customers to
see our products
Payment
Time to purchase
How long it takes from checkout
to confirmation mail

23. W e b i n a r © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability in action: Scenarios
Growth
Catalog grows by N
Image searching becomes a
bottleneck
No stock
Runs out of products
And yet purchase is successful

25. W e b i n a r © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges of Serverless applications
• No access to the underlying infrastructure that runs the code
• Troubleshooting event driven applications is hard
• Cost calculation requires understanding of business transactions
• Ephemeral and limited compute duration

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges of Serverless distributed applications
Managing lots of resources and APIs
End to end performance
Correlation between logs
Distributed errors
Amazon
API Gateway
AWS Lambda Amazon Simple
Notification Service
Amazon DynamoDB
AWS Lambda AWS Lambda
AWS Lambda
ASYNCSYNC
ASYNC
ASYNC
API
Amazon Simple Queue
Service

28. W e b i n a r © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Q&A

29. W e b i n a r © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Appendix

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some ideas you can use to improve the shop
• Lazy loading images (front-end)
• Check stock before allowing purchase
• Structured logging
• Custom metrics (CloudWatch API) for KPIs
• Order service
• Metrics from front-end
• DLQs

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #1: store grows
• More and more items are being added to the store
• Searching for images becomes a bottleneck as it’s done on the back-end
• Result -> timeout -> user can’t see any item in the store

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #2: stock run out
• 2 customers try buying the last piece of an item
• Both get the confirmation screen, only one get’s a success mail
• Result -> Un-happy customer
• Mis-handled request leads to business impact

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example Epsagon tracing - Payment