The document discusses how artificial intelligence can be applied to performance engineering to make it self-healing and self-service. It describes how monitoring needs have evolved from just looking at dashboards and logs to dealing with dynamic cloud environments. It outlines how AI can be used for full-stack monitoring with one agent, automated end-to-end tracing, automated log analytics and change detection. It then discusses how AI can enable shifting work left to break the pipeline earlier, improve mean time to resolution with auto-mitigation, and shift work right with tags, deployments and events to create actionable feedback loops across development, operations and business teams.

1. Confidential, Dynatrace, LLC
Applying Artificial Intelligence to Performance
Engineering – Self-Healing, Self-Service
Andreas Grabner - December 2017
@grabnerandi

2. Monitoring used to
be about looking at
dashboards …

3. Process Memory (GB)
CPU Graphs (%)

4. .. and about
analyzing logs &
exceptions …

5. confidential
Top Exceptions
Top Logs

6. But the apps and
services we build
have transformed to
something more
dynamic…

7. Develop
Ship
Deploy
Run
Scale
Compute
nodejs mongo db netty cassandra redis
ansible jenkins puppet chef
docker cloudfoundry rh openshift rh atomic rocket
core os rancher kvm busybox
mesos marathon kubernetes swarm
Amazon azure openstack mesosphere calico weave
eureka/hystrix
A whole new technology stack & polyglot development
Amazon
DynamoDB AWS Lambda
AWS
CodeDeploy
Amazon EC2
Container Services
Amazon EC2
AWS Elastic
Beanstalk
Amazon
API Gateway

8. Cloud
OS, Disks
Containers, Processes,
Logs
Application- & Webserver
Mobile*
Services
Network
Browser
3rd parties
FullStackDiscovery,Modeling&Analytics

9. confidential
One Agent to monitor them all

10. confidential
DynatraceFullStackMonitoring

11. confidential
Automated End-to-End Tracing

12. All Timeseries Data you can wish for 
Network Container
Servers Hosts
Cloud

13. confidential
Everything automatically baselined!

14. confidential
Automated Log Analytics and Change Detection

15. Your Apps/ServicesYour Users Dynatrace OneAgent AI Supported Performance Engineering

16. Dev Perf/Test Ops Biz
Shift-Left: Break Pipeline Earlier
Improve MTTR: Auto-Mitigation
Shift-Right: Tags, Deploys, Events
Actionable Feedback Loops

17. Shift-Right: Tags, Deployments & Events
docker run –e DT_TAGS=BLUE
dtcli tag srv CartServicev2 GREEN
dtcli evt push host .*demo
version=123 source={git_commit}
dtcli evt push pg tomcat1
desc=JVMMemIncr hint=+100MB
Dynatrace Smartscape
Release
Automation
Dynatrace Automation
API, CLI, Auto-Detection

18. Improve MTTR: Automate Mitigate with AI Data
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
?Escalate at 2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5 Still ongoing? Initiate Rollback!
Escalate
? Still ongoing?5
1
2
3
4
Mark Bad Commits
Update Dev Tickets
…
…
Impact Mitigated??
?

19. Shift-Left: Break Pipeline Earlier
c0123bd
nov17
myservice:nov17 myservice:nov17
space:UAT
space:PERF
myservice:nov17
Selenium Perf Data
space:PERF
myservice:nov17myservice:nov16 space:PROD
myservice:BLUE
myservice:GREEN
myservice:nov17
space:PROD
myservice:BLUE
myservice:GREEN
space:PROD
myservice:nov16

20. Shift-Left: Performance as Self-Service
myservice:tmp57 myservice:tmp57
space:PERF
c0123bd

21. Actionable Feedback Loops: Business
Success Criteria
Labels become Key User Action
Live Data Queries
New Requirement Definition

22. Filter by tags, versions, …
Access to all key metrics
Access to every service
Actionable Feedback Loops: Architects

23. C:dynatrace-cli> py dtcli.py dqlr srv tags/?key=.*prod.* service.requestspermin[count%180:0],service.failurerate[avg%180:0]
Actionable Feedback Loops: SRE’s
Live data access
through REST API/CLI

24. Actionable Feedback Loops: Load Testing
Dynatrace Data

25. Actionable Feedback Loops: Load Testing
Extracted from HTTP Header Tag(s)
PurePath for Load
Test Requests

26. Actionable Feedback Loops: Operations
Filter by Infrastructure,
Service, Application

27. Dev Perf/Test Ops Biz
Shift-Left: Break Pipeline Earlier
Improve MTTR: Auto-Mitigation
Shift-Right: Tags, Deploys, Events
Actionable Feedback Loops

28. confidential
Demo Time or Done 

29. Confidential, Dynatrace, LLC
Applying Artificial Intelligence to Performance
Engineering
Andreas Grabner - November 2017
@grabnerandi