Cerner devised a way to gather AppDynamics End User Monitoring (EUM) data without modifying application code by leveraging existing F5 load balancers. This allowed Cerner to deploy the EUM technology rapidly, without interrupting development pipelines or requiring vendor assistance for integration with third-party software. Hear how EUM helps Cerner build data-driven acceleration and caching policies in F5's application acceleration manager to maximize the value of its infrastructure investments and strategically place new accelerators where they provide the greatest improvements to users. Learn about the value Cerner gained by deriving F5 application acceleration policies from EUM data. Key takeaways: o How to simplify EUM deployment using both native and custom F5 functionality (with technical configuration steps) o How to gather actionable insights from data provided by EUM o Understand how to translate the value Cerner has gained into your own success For more information go to: www.appdynamics.com

1. Improving the User Experience with
AppDynamics EUEM and F5
Tim Lukasiewicz
Technology Architect, CTS CorpOps CSM
@securi_tron | Tim.Lukasiewicz@cerner.com

2. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Agenda
Background
Our Challenge – Regionalize the Business Service
#1 - Understand the End User’s Experience
#2 – Find ways to make it faster
EUEM Technical Implementation Steps
Cerner’s Remedy Use Case Review
Driving F5 Application Acceleration with AppDynamics
Results
Q&A

3. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Background
Global Healthcare Information Technology Company
200+ Offices located around the world
27,680 Total associates
Primary datacenter presence in Kansas City

4. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Corporate Systems Management
About our CTS CSM team…
Corporate Systems Management
47 team members (31 US, 16 BLR, India)
• Service Delivery Managers
• Production Owner Architects
• Technology Architects
• System Engineers
• Interns – Cerner Scholars, MIC Interns, KCIT Apprentices
Responsible for all Enterprise Application Operations
If you use it to do your job at Cerner, we probably manage it!

5. The Challenge: Speed

6. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Challenge: Regionalize the Business Services
33
Countries
Cerner associates
are hereBusiness Service
lives here

7. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Cerner’s Challenges
Applications fail to perform outside of Kansas City
>1 minute page loads (or timeouts) reported by users
“System” looks fine to engineers
Rapid growth of Cerner Associates outside US
New acquisitions
Expanding Cerner’s global presence and new client-bases
Reduced Operational Efficiency
Associates hampered by technological and physical limitations
Investigation struggles without solid diagnostic data

8. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Why does it matter?
Every 100ms delay
74% are willing to wait
5 seconds or less
for a single web page to load
before leaving the site.
Costs Amazon 1%
in
sales.
Original Slide Courtesy of F5,
Source Citation: http://www.webperformancetoday.com/2013/05/06/psychology-waiting-faster-online-checkout/

9. Reactive Monitoring

10. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
…doesn’t always give us the big picture

11. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Real User Monitoring
Gather the metrics directly from real users performing their
work with no impact
Identify performance trends from real-world scenarios

12. Automated HTML Rewriting with
F5
Infrastructure As A Developer

13. Injecting the page-modifications with the F5 means
we aren’t reliant on developers or vendors to
implement our monitoring

14. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Setting up the F5 for Injection – First time
Name the rule ‘PrependEUMAgent’ if its
general-use
Append the application to the end if
specific, eg ‘PrependEUMAgent_Remedy’
Profiles -> Content -> HTML -> Click 'Rules‘
Click the down arrow next to "Create New" and
select "Prepend HTML"

15. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Setting up the F5 for Injection – First time
Copy HTML from AppDynamics UI
Paste into the Prepend window under
“Action Settings”
Adjust “src=adrum.js” to match your
application
Click Match Settings
Enter ‘/head’ in the tag name field
Leave others blank

16. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Setting up the F5 for Injection – First Time
Profiles -> Content -> Click “HTML”
Create a New Profile and name it “html_insertEUM”
If specialized- append the name of the application
Example: “html_insertEUM_Remedy”
Select '/Common/html' as the parent profile.
HTML Rules tab and add the relevant rule we created before to this
profile.

17. Setting up Injection – Every time

18. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Setting up Injection – Every time
Open the "Local Traffic" Heading and navigate through the
hover-menus
Virtual Servers -> Virtual Server List
Select the VS for your application, then change the "HTML
Profile" option from 'None' to the html_insertEUM profile.
Click "Update" at the bottom of the page to save your
changes.

19. Hosting the adrum.js file
More F5 magic

20. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Adrum.js Hosting – Hosted Model
Adrum.js file is served by
application web tier
Requires placing the file in
the document-root of the
website
Ask the developers if you
aren’t sure where that lives

21. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Adrum.js Hosting – Redirected Model
Adrum.js file is served by
arbitrary web server,
hidden by the F5
Needs ACLs from
applications F5s to web
server
Requires consistency in
naming

22. Cerner’s Use Case: Remedy

23. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Remedy at Cerner
2,100 daily concurrent users
10,000+ active users
5,000 Incidents records created per day
135 seconds to create an incident record
666,360 seconds per day creating incident records

24. “Remedy is Slow”
(Not any better than our basic up/down monitors)

25. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Global Performance Analysis

26. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Performance Testing – Load Testing (Bangalore)
0
10
20
30
40
50
60
70
80
Landing Login Console Load Logout
RemedyCert from India (AAM Off)
AVERAGE Time Across All Console Load Scenarios
50% 75% 100%
Percentages of Average Prod Workload

27. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
BMC & CDNetworks Whitepaper
https://www.cdnetworks.com/wp-content/uploads/2016/01/CDNetworks-BMC-Remedy-OnDemand-WP-EN-US.pdf

28. HTTP Performance Overview

29. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Life of an HTTP Request
DNS -> Connect -> Request -> Receive
(Repeat)

30. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
The HTTP dilemma
"Waterfall" of associated
resources required to
compose
the page.
• Modern browsers only
support 6 concurrent
connections per hostname
• JS files are a blocking
download
• CSS files are a blocking
download for most IE
browser versions (although
not intended to be)
http://www.igvita.com/slides/2012/webperf-crash-course.pdf
Slide courtesy of Ilya Grigorik @ Google:

31. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
PLT: latency vs. bandwidth
http://www.igvita.com/slides/2012/webperf-crash-course.pdf
Slide courtesy of Ilya Grigorik @ Google:

32. F5 Application Acceleration

33. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Life of an HTTP Request
http://www.igvita.com/slides/2012/webperf-crash-course.pdf
Slide courtesy of Ilya Grigorik @ Google:

34. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Acceleration in the Data Center
Load balance
• Distribute application
load across multiple
servers to increase
availability
Offload
• Increase server capacity
• Accelerate SSL
processing
• Manage TCP connections
more efficiently
SPDY gateway
• Leverage SPDY or
HTTP/2.0 and other
protocols without
recoding applications
Fast cache
• Offload repetitive traffic
from web and application
servers to increase
server capacity

35. Cerner’s Results

36. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Performance Testing – Load Testing (Bangalore)
0
10
20
30
40
50
60
70
80
Landing Login Console Load Logout
RemedyCert from India (AAM Off)
AVERAGE Time Across All Console Load Scenarios
50% 75% 100%
Percentages of Average Prod Workload

37. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Performance Testing – Load Testing (Bangalore)
0
1
2
3
4
5
6
7
8
9
10
Landing Login Console Load Logout
RemedyCert from India (AAM Off)
AVERAGE Time Across All Console Load Scenarios
50% 75% 100%
Percentages of Average Prod Workload

38. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Performance Testing – Load Testing (Bangalore)
0
10
20
30
40
50
60
70
80
Landing Login Console Load Logout
RemedyCert from India (AAM On)
AVERAGE Time Across All Console Load Scenarios
50% 75% 100% 150% 200%
Percentages of Average Prod Workload

39. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Performance Testing – Load Testing (Bangalore)
0
1
2
3
4
5
6
7
8
9
10
Landing Login Console Load Logout
RemedyCert from India (AAM On)
AVERAGE Time Across All Console Load Scenarios
50% 75% 100% 150% 200%
Percentages of Average Prod Workload

40. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
The Way Forward – Cerner Global CDN

41. Q&A

42. Please give us your feedback—Session T5906
• Complete the online survey you'll receive via
email later today or via text at:
Text this number: 878787
Text this word: APPSPHERE
• Every time you submit a session survey, your
name will be entered in a random drawing.
We're giving away Amazon Echos
to 5 lucky winners!
• Thank you for your input
APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 41
Win!

43. Thank you for attending AppSphere 2016!
Contact me with any other questions:
@securi_tron or Tim.Lukasiewicz@cerner.com