1.
LET’S LEARN
FROM THE TOP
PERF MISTAKES
@grabnerandi
http://apmblog.compuware.com

2.
What to do with the fastest car …

3.
… if it fails to reach the finish line

4.
What to do with
millions of $$ for
building a web
site …

5.
Performance,
Scalability & Architecture

6.
#1: Architectural
Decisions

7.
#1: “We want more Web 2.0”

8.
#1: Load Test Prior to Change

9.
#1: Load Test After Change

10.
Metrics: # Visitors
# Requests / User
Business: Do we need
all these bells and
whistles?

11.
#2: Disconnected
Teams

12.
#2: “Teamwork” between Dev and Ops
SEV1 Problem in Production
Need access to log files
Where are they? Can’t get them
Need to increase log level
Can’t do! Can’t change config
files in prod!

13.
#2: Solution: Implement a Custom “On Demand”
Remote Logger

14.
#2: Implementation and Rollout
Implemented
Custom Logger
Worked well in
Load Testing

15.
#2: What happened?
~ 1Mio Lock Exceptions in 30
mins

16.
#2: Root Cause: A special WebSphere Setting!
Log Service provides a synchronized
log file across ALL JVMs
Log Service provides a
synchronized log file across
ALL JVMs

17.
Metrics: # Log Messages,
# Exceptions
Share: Same Server
Settings

18.
#3:
Implementation
Flaws

19.
#3: Business Impact requires Action!

20.
#3: Solution: Cache to the RESCUE!!

21.
#3: Implementation and Rollout
Implemented
InMemory Cache
Worked well in
Load Testing

22.
#3: Result: Out of Memory Crashes!!
Still crashes
Problem fixed!Fixed Version Deployed

23.
Metrics: Heap Size,
# Objects Allocated,
# Objects in Cache
Cache Hit Ratio
Test: With realistic Data

24.
#4: Push
without a Plan

25.
#4: Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~
20MB

26.
#4: m.store.com redirects to www.store.com
ALL CSS and JS files are
redirected to the www domain
This is a lot of time “wasted”
especially on high latency mobile
connections

27.
#4: Critical Pages not Optimized!
Browse, Search
and Product Info
performs well
… because they don’t follow
best practices: 87 Requests, 28
Redirects, …
Critical Pages such as
Shopping Cart are very
slow …

28.
Metrics: Load Time,
# Resources (Images, …),
# HTTP 3xx, 4xx, 5xx
Dev: Build for Mobile
Test: Test on Mobile

29.
#5: “Blindly”
(Re)use Existing
Components

30.
#5: Requirement: We need a report

31.
#5: Using Hibernate results in 4k+ SQL Statements
to display 3 items!
Hibernate
Executes 4k+
Statements
Individual
Execution VERY
FAST
But Total SUM
takes 6s

32.
#5: Requirement: We need a fancy UI

33.
#5: Using Telerik Controls Results in 9s for Data-
Binding of UI Controls
#1: Slow Stored Procedure
Depending on Request
execution time of this SP
varies between 1 and 7.5s
#2: 240! Similar SQL Statements
Most of these 240! Statements
are not prepared and just differ in
things like Column Names

34.
Metrics: # Total SQLs
# SQLs / Web Request
# Same SQLs / Request
Transferred Rows
Test: With realistic Data
Dev: “Learn” Frameworks

35.
#6: No “Agile”
Deployment

36.
Adonair
#6: Load Spike resulted in Unavailability

37.
#6: Alternative: “GoDaddy goes DevOps”
1h before
SuperBowl KickOff
1h after
Game ended

38.
#6: Behind the Scenes

39.
Metrics: Availability
Page Size, # Objects
# Hosts, # Connections
DevOps: “Feature”
Switches

40.
What have we
learned today?

41.
# of Requests / User
# of Log Messages
# of Exceptions
# Objects Allocated
# Objects In Cache
Cache Hit Ratio
# of Images
# of SQLs
# SQLs per RequestAvailability
# HTTP 3xx, 4xx
Page Size

42.
A final thought …

43.
How about this idea?
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Let’s look behind the
scenes
Exceptions probably reason
for failed tests
Problem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regression
Now we have the functional and
architectural confidence

44.
How? Performance Focus in Test Automation
Cross Impact of KPIs
Analyzing All Unit /
Performance Tests
Analyze Perf
Metrics
Identify
Regressions

45.
More Info
• My Blog: http://apmblog.compuware.com
• Tweet about it: @grabnerandi
• dynaTrace Enterprise
– Full End-to-End Visibility in your Java, .NET, PHP Apps
– Sign up for a 15 Days Free Trial on http://compuwareapm.com
• dynaTrace AJAX Edition
– Browser Diagnostics for IE + FF
– Download @ http://ajax.dynatrace.com

46.
THANK YOU
@grabnerandi