Even the best continuous delivery and DevOps practices cannot guarantee that there will be no issues in production. The rise of Site Reliability Engineering (SRE) has promoted new ways to automate resilience into your system and applications to circumvent potential problems, but it’s time to “shift-left” this effort into engineering. In this session, learn to leverage AWS Lambda functions as “remediation as code.” We show how to make it part of your continuous delivery process and orchestrate the invocation of Self-Healing Lambda functions in case of unexpected situations impacting the reliability of your system. Gone are the days of traditional operation teams—it’s the rise of “shift-lefters”! This session is brought to you by AWS partner, Dynatrace.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shift-Left SRE: Self-Healing with
AWS Lambda
Andreas Grabner
Global Technology Lead & DevOps Activist
Dynatrace
D E V 3 1 3 - S

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. Remediation use cases
2. PREVENT in CI/CD vs. Repair in PROD with
AWS Lambda
3. “Auto-Remediation as Code” with Lambda
4. The “Unbreakable Delivery Pipeline”

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Crash -> Restart

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Full or slow disk -> Clean up
$ find ./my_dir -mtime +10 -type f -
delete

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bad configuration changes -> Revert

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bad configuration changes -> Revert

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Low on resources -> Scale up

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Overprovisioned after drop in traffic -> Scale down

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blue vs. Green -> Redirect traffic
BLUE
GREEN

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
End user impact -> Reverse Blue / Green
Deploy Blue Back to Green

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
List of remediation action we discussed
• Process restarts
• Resource (for example, disk) cleanup
• Revert bad configuration changes
• Scale up
• Scale down
• Blue vs. Green switching

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add key metrics from incidents to quality gates
1 2 3Staging Production
CI CD CI CD
Code / Config change 4 End users
5 Issue impacting SLAs6 Add metric to quality gate

16. Use cases and metrics
we can “Shift-Left”!

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in log behavior
• Use cases
• Are we logging too much? Did we turn on verbose logging by accident?
• Metrics
• Total log size
• Number of total and critical log messages
• How to query?
• For example: Using Amazon CloudWatch log filters
aws logs put-metric-filter
--log-group-name MyApp/access.log
--filter-name EventCount
--filter-pattern ""
--metric-transformations
metricName=MyAppEventCount,metricNamespace=MyNamespace,metricValue=1,defaultValue=0

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in resource consumption
• Use cases
• Bad coding leads to higher costs?
• Metrics
• Memory usage
• Bytes sent/received
• Overall CPU
• CPU per transaction type
• How to query?
• Some through CloudWatch API
• Dynatrace Timeseries API

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in dependencies
• Use cases
• Do we have new dependencies? On purpose?
• Are we connecting to the services we are supposed to connect?
• How many container instances are required?
• Metrics
• Number of incoming / outgoing dependencies
• Number of instances running on
• How to query?
• Maybe CloudWatch API
• Dynatrace SmartScape API

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Use cases
• Did we introduce new “hidden” exceptions?
• Metrics
• Total exceptions
• Exceptions by class & service
• How to query?
• Dynatrace Timeseries API
Detect change in application exception handling

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in performance behavior
• Use case
• Are we jeopardizing our SLAs?
• Does load balancing work?
• Difference between canaries?
• Metrics
• Response time (percentiles)
• Throughput & perf per instance / canary
• How to query
• Dynatrace Timeseries API
• Dynatrace SmartScape API

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in error behavior
• Use cases
• New unexpected error conditions?
• Metrics
• HTTP Failure Rate
• JavaScript Error Rate
• Query through
• Real user monitoring (RUM) solution
• Dynatrace Timeseries API

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detect change in end user scenarios
• Use cases
• Average number of page requests per user increased?
• How does this impact resource and capacity requirements?
• Metrics
• Number of user interactions / session
• Page sizes, number of resources
• Query through
• RUM solution
• Dynatrace Timeseries API

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
List of metrics we just discussed
• Logging
• Total log size
• Number of total and critical log messages
• Resources
• Memory usage
• Bytes sent / received
• Overall CPU
• CPU per transaction type
• Dependencies
• Number of incoming / outgoing
dependencies
• Number of instances running on
• Exceptions
• Total exceptions
• Exceptions by class & service
• Performance
• Response time (percentiles)
• Throughput & perf per instance
• Errors
• HTTP failure rate
• JavaScript error rate
• End user scenarios
• Number of user interactions / session
• Page sizes, number of resources

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to add this to our pipeline?
1 2 3Staging Production
CI CD CI CD
Code / Config change 4 End users

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inspiration from Curtis Bray (re:Invent 2017)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inspiration from Thomas Steinmaurer @ Dynatrace
“Performance Signature”
for Build Nov 16
“Performance Signature”
for Build Nov 17
“Performance Signature”
for every build
“Multiple Metrics”
compared to prev
timeframe
Simple Regression Detection
per metric

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Build validation / Monitoring as code”
monspec.json
{
...
"perfsignature" : [
{
"timeseries" : "com.dynatrace.builtin:service.responsetime",
"aggregate" : “p90", // min, max, avg, sum, median, count, percentile
"validate" : "upper", // upper or lower
// "upperlimit" : 100, // Optional: Can be used to define a FIXED THRESHOLD
// "lowerlimit" : 50, // Optional: Can be used to define a FIXED THRESHOLD
},
{
"timeseries" : "com.dynatrace.builtin:service.failurerate",
"aggregate" : "avg"
},
{
"timeseries" : "com.dynatrace.builtin:service.requestspermin",
"aggregate" : "count",
"validate" : "lower"
},
{
"smartscape" : "toRelationships:calls",
"aggregate" : "count",
"upperlimit" : 1 // Validate that we only call to the one backend service and nowhere else!
}
],
Metrics: Which metrics, aggregation, upper/lower boundaries?
Dependencies: How many involved services?

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automate validation into AWS CodePipeline with
Lambda
StagingToProduction,5,ApproveStaging
Invoke
RegisterStagingValidation
AWS Lambda
registerDynatraceBuildValidation
Monspec

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automate validation into AWS CodePipeline with
Lambda
Staging: Register Build Validation!
registerDynatraceBuildValidation
Adds build validation request
Adds item
Build validation request item
- Pipeline Information
- Monspec
- Timestamp + Timeframe
- Comparison Definition Name
- Action Name to Approve / Reject
validateBuildDynatraceWork CloudWatch Events
(e.g:, 1min)
Triggers
Approves/Rejects IF “In Progress” & if RegisterBuildValidation
was called with that Action Name
Monspec from Amazon S3
Dynatrace entities & Timeseries REST API
Resolves tags and gets list of entities
Queries metrics for these entities
Updates build validation request
- Updated Monspec
- Updated status

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automate validation into AWS CodePipeline with
Lambda

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build over build results pulled from Amazon DynamoDB
GoodBuild
GoodBuild
GoodBuild
BadBuild
BadBuild
BadBuild
BadBuild
GoodBuild

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inspiration from Beachbody
ChatOps
Erik Landsness, Beachbody
Problem evolution

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Self-healing: Path to autonomous Ops
Auto-mitigate!
1 CPU exhausted? Add a new service instance to distribute load!
3 Caused by Canary Release? Redirect traffic to main canary!
How to escalate?
2 Exhausted connection pool? Increase pool size!
Escalate? Still ongoing?
1
2
Update teams
…
Impact mitigated??
Inform #WebTeam about JavaScript issue on IE
Push status update to inform our customers
Inform Support about potential incoming user complaints!
?

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Auto-Remediation as Code” triggered by Dynatrace
#1: Push deployment information,
e.g: CodeDeploy DeploymentId
#2: Calling Lambda via API Gateway
handleDynatraceProblemNotification
#4: Redeploy previous
revision
#3
Uses Dynatrace Events API
to pull CUSTOM_DEPLOYMENT events
#5: Push comment to Dynatrace

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: Unbreakable cloud-native pipelines
1 2 4 53
Production
Staging Approve staging Production Approve production
CI CD CI CD CI CD CI CD
Pushes deployment into
Dynatrace entities
Compares builds and
approves / rejects pipeline
Pushes deployment info into
Dynatrace entities
Validates production and
approves / rejects pipeline
Executes auto-remediating
actions e.g., roll-back
Build #17 Build #18

41. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Andreas Grabner
twitter: @grabnerandi
email: andreas.grabner@dynatrace.com

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample code slide
var pd = require('pretty-data').pd;
var xml_pp = pd.xml(data);
var xml_min = pd.xmlmin(data [,true]);
var json_pp = pd.json(data);
var json_min = pd.jsonmin(data);
var css_pp = pd.css(data);
var css_min = pd.cssmin(data [, true]);
var sql_pp = pd.sql(data);

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample code slide
var pd = require('pretty-data').pd;
var xml_pp = pd.xml(data);
var xml_min = pd.xmlmin(data [,true]);
var json_pp = pd.json(data);
var json_min = pd.jsonmin(data);
var css_pp = pd.css(data);
var css_min = pd.cssmin(data [, true]);
var sql_pp = pd.sql(data);
var pd = require('pretty-data').pd;
var xml_pp = pd.xml(data);
var xml_min = pd.xmlmin(data [,true]);
var json_pp = pd.json(data);
var json_min = pd.jsonmin(data);
var css_pp = pd.css(data);
var css_min = pd.cssmin(data [, true]);
var sql_pp = pd.sql(data);