Contenu connexe Similaire à Testing Mobile App Performance Similaire à Testing Mobile App Performance (20) Testing Mobile App Performance2. !
!
!!!!! !
!!!!
Brad Stoner is a senior performance engineer at Neotys. Previously, he managed the load
and performance team at H&R Block where he spent seven years leading his five-person team
in the pursuit of improved application performance and quality. Brad and his team managed
the performance testing process for more than fifty projects annually. He founded Sandbreak
Digital Solutions, a consulting company specializing in web application performance testing,
web page optimization, front end optimization, capacity testing, infrastructure validation, and
cloud testing. In his twelve years of IT experience, Brad has held roles from systems
engineering to operations management.
Brad Stoner
Neotys
3. www.neotys.com© 2015 Neotys. All Rights Reserved.
Ensuring the Performance of Mobile Apps
On Every Device and Network…
StarCANADA 2015 Brad Stoner, Neotys USA
4. www.neotys.com© 2015 Neotys. All Rights Reserved.
BIOGRAPHY
NEOTYS USA
Senior Performance Engineer
H&R Block
IT Manager – Load and Performance Team
Sandbreak Digital Solutions LLC
Former Founder/Principal Consultant
Webinars on neotys.com
5. www.neotys.com© 2015 Neotys. All Rights Reserved.
OBJECTIVES
Review of the Problem
Why is performance testing necessary?
How does it apply mobile?
Types of Mobile Testing
How they relate to performance
Mobile Testing Challenges
Overview of Performance Testing
Mobile specifics
Suggested Approaches
6. www.neotys.com© 2015 Neotys. All Rights Reserved.
PERFORMANCE MATTERS! From a Nice-to-Have to a MUST-Have
1 Second Could Cost
Amazon $1.6 Billion In Sales
US Online Businesses =
$1.7 trillion in 2013
100K$ TV ADS WEBSITE CRASHES!
The Obamacare Website
Failed For The Same Reason
The Soviet Union Did
Failures
Obamacare Apps
Bank of America
Facebook IPO
Poor Performance =
Lost Revenues
Meanwhile…
Entertainment, video &
mobile usage are exploding!
Website = Brand
Secure Marketing Investments
7. www.neotys.com© 2015 Neotys. All Rights Reserved.
MARKET DRIVERS: BUSINESS
Web and Mobile Applications are Business Critical
Organizations compete via the internet
Software and Brand are synonymous
Technology usage growing
Failure and poor performance are catastrophic and costly
Time To Market pressure increasing
End user expectations are rising
Investments in Software Quality are Up
Load testing is a must-have
Time pressure on dev & tests teams
18%
23%
28%
CY 2012 CY 2013 CY 2014
% IT BUDGET FOR APP QUALITY
Source: World Quality Report 2013-2014
0%
40%
20%
8. www.neotys.com© 2015 Neotys. All Rights Reserved.
MOBILE IS GROWING!
Unprecedented User
Expectations
Exploding Expansion
of Devices
• 1.2 billion new smartphones
over the next 5 years
• Smartphones over
40% of all handset
shipments
• 85% of users
expect mobile app
to load at least as
quickly as desktop
sites
• 25% of users abandoned a
mobile app after 3 second
delay
9. www.neotys.com© 2015 Neotys. All Rights Reserved.
PERFORMANCE PERSPECTIVE
How can we tell where the problem is?
Combinations of functional and performance testing but…
“End User Experience”
• Time between a client initiating a transaction and the
time the result is rendered back in the device
• Stand-alone application - EUE is determined completely
on the device
• “Other” application: - EUE can be measured on device
but is sum of:
- Round trip time from device to server and back
- Device-side rendering
Does it matter where the problem is?
(device, server, network, etc.)
• To the business…maybe not.
• To you…maybe (probably) yes!!
10. www.neotys.com© 2015 Neotys. All Rights Reserved.
CHALLENGES/CONSIDERATIONS – DEVICE TYPE
DEVICE CHALLENGES
Operating Systems
App Technologies
Screen Sizes
iOS Android Windows
Design of what content is
above/below the fold
When that content
gets delivered
Different content for
different devices
Web SocketHTML 5 Streaming
Video/Audio
Google SPDY
11. www.neotys.com© 2015 Neotys. All Rights Reserved.
MOBILE APPLICATION TYPES
WEB
Browser-Based
NATIVE
Client Downloaded From
App Store or Google Play
• Usually connected
to the internet
- If not, “Standalone”
• Generally faster (client-side)
than browser app
• Often a different application
site than if accessed via
desktop browser
HYBRID
Combo of Web
and Native
• Usually
wrappers for
web clients
12. www.neotys.com© 2015 Neotys. All Rights Reserved.
TYPES OF MOBILE TESTING
PLATFORM TESTING
LOAD TESTING
FUNCTIONAL
TESTING
FEATURES
DEVICE
PERFORMANCE
• Does the app do what
it’s expected to do?
How does the app work on
this device?
• Supported?
• CPU, Battery consumption
How fast is it displayed?
• Speed of use
• Can 1000 users use this app?
• What is the user experience
under load?
• What about 500 users on 3G
and 500 on 4G LTE?
13. www.neotys.com© 2015 Neotys. All Rights Reserved.
TESTING STANDALONE APPLICATIONS
Functional / Platform Testing Will Suffice
• All logic on the device itself
• No back-end server
Metrics to Correlate
• Custom Timers
• Device “Vitals” (CPU, Battery, Throughput)
Automated or Manual?
• Hard to do it manually for all different configurations!!
• How to get Vitals manually
Solution Requirements
• Consider an automation tool / vendor that allows:
- ON DEVICE testing
- Multi-device testing
- Vitals
14. www.neotys.com© 2015 Neotys. All Rights Reserved.
TESTING NATIVE, WEB, OR HYBRID APPLICATIONS
Remember: EUE = device-side rendering + server-side performance
How do we test each?
DEVICE-SIDE
Performance impacted by concurrent user load
Similar to browser-based apps but with nuances
SERVER-SIDE
Essentially a stand-alone application
Measurements made similar to prior slide
However performance impacted by server
Have To Separate Device-Side From Server-Side
How do we correlate the two?
15. www.neotys.com© 2015 Neotys. All Rights Reserved.
LOAD AND PERFORMANCE TESTING – OVERVIEW
What?
• The process of putting demand on a system or
device and measuring its response
• Non-functional
• Bridges, roller coasters, elevators…
• Software applications: Generating network traffic
• (Impossible to do manually!!)
Why?
• To avoid:
o Loss of revenue
o Customer “migration”
o Decreased productivity
• To understand an application’s
o Scalability
o Stability
16. www.neotys.com© 2015 Neotys. All Rights Reserved.
LOAD / PERFORMANCE TESTING SOLUTIONS
1
2
3
4
HOW THEY WORK
Record Usage Scenarios
Measure Response Times
Generate the Network Load
Analyze Results
- Network traffic between client and server
- “Models” of real-world activity
- Round-trip times from client to server and back
17. www.neotys.com© 2015 Neotys. All Rights Reserved.
LOAD / PERFORMANCE TESTING SOLUTIONS
Cloud Load Generators
Controller
On-Premise Load Generators
Firewall
Application
Server
Database
Server
LOAD TESTING SOLUTION APPLICATION INFRASTRUCTURE
LOAD
18. www.neotys.com© 2015 Neotys. All Rights Reserved.
LOAD / PERFORMANCE TESTING SOLUTIONS
Cloud Load Generators
Controller
On-Premise Load Generators
Firewall
Application
Server
Database
Server
LOAD TESTING SOLUTION APPLICATION INFRASTRUCTURE
LOAD
Monitoring
Engine
Monitoring
19. www.neotys.com© 2015 Neotys. All Rights Reserved.
DEVICE CHALLENGES FOR PERFORMANCE TESTING
Recording the Application
• Type of app
- Native apps
- Web apps through mobile browsers
- Hybrid apps
• Technologies used in the app
- HTML 5
- Web Socket
- Streaming Video/audio
- Google SPDY
Replay scenarios for specific devices and browsers
• Different content is delivered depending on the specific device or browser
• Different browsers and devices have different numbers of parallel connections that
affect performance
20. www.neotys.com© 2015 Neotys. All Rights Reserved.
CHALLENGES/CONSIDERATIONS – NETWORK CONDITIONS
Bandwidth
• How are users with different download and
upload capacities experiencing performance on
the app?
Latency
• How are network delays affecting the
performance of the app?
Packet Loss
• How are users experiencing the app when
switching cell towers, traveling, losing signal, etc.
SHOULD WE EMULATE THESE CONDITIONS?
YES!!!
21. www.neotys.com© 2015 Neotys. All Rights Reserved.
LATENCY: THE REAL BOTTLENECK
Bandwidth Isn’t Everything
• Benefits of increasing bandwidth over HTTP
caps out around 5 Mbps
• Increasing bandwidth from 5 to 10 Mbps
decreases page load time by only about 5%
Latency is Worse for Mobile
• In 2012, the average round trip time to
Google was ~100ms worldwide and ~50-
60ms within the US
• On mobile, RTT is around 100-1000ms
• If a device has been idle and requires the
radio to turn on first, add 1000-2000ms to
that time
Benefit of Adding an Extra Mbps
Bandwidth
PercentImprovement
20
0
10
30
40
22. www.neotys.com© 2015 Neotys. All Rights Reserved.
PACKET LOSS INFLUENCE
Packet Loss
Consequences
Packet retransmission
End user experience degraded
because packet loss increases jitter
Errors could occur if the packet loss
is in pure data transmission
Packet Loss
Inevitable
Mobility of the user
Quickly switching
to the next cell tower
Temporary lose the signal due to
congestion of the current cell
23. www.neotys.com© 2015 Neotys. All Rights Reserved.
CHALLENGES/CONSIDERATIONS - HIGH, DISTRIBUTED LOAD
High Number of Users
• Mobile applications usually have large user bases
• High user growth creates need for highly scalable load generation infrastructure
Geographical Distributed Load
• Users are accessing apps from remote
locations
• Distance and location greatly affect
performance
Mix of Web & Mobile Users
• Many mobile apps have a “partnered” web app with a
shared backend
• High load on the web app can affect performance on
mobile
TIP: USE CLOUD-BASED LOAD GENERATION
24. www.neotys.com© 2015 Neotys. All Rights Reserved.
MOBILE PERFORMANCE TESTING – SOLUTION REQUIREMENTS
Record from Any Device
• From real devices and emulators
• Native or browser-based applications
Simulate Any Device or Browser
• Ensures the right content is delivered for
that device or browser
• Simulates parallel connections for each
specific device and browser
Emulate Realistic Network Conditions
• Bandwidth
• Latency
• Packet Loss
Generate Load from the Cloud
• Geographic realism
• Test full delivery chain
Load
Command
3GH+ 4G WiFi
APPMonitor
Network
Emulation
& Device
Simulation
Cloud Load Generators On-premise Load Generators
Database ServerApp Server Web Server
25. www.neotys.com© 2015 Neotys. All Rights Reserved.
WHAT HAPPENED TO EUE?
The Problem:
• Performance solutions don’t provide on-device measures
• Functional/platform solutions don’t separate round-trip or server-side measures
HOLY GRAIL
SOLUTION
COMBINE THE TWO!
(Manual stop watch approach not feasible)
Automated
Functional/Platform
Automated
Performance
On-device measures
Device Vitals
“Round-Trip” measures
INTEGRATED FOR RESULTS CORRELATION
26. www.neotys.com© 2015 Neotys. All Rights Reserved.
LOAD + FUNCTIONAL TESTING
• Automatically trigger
functional test with the
start of a load test
• See real device rendering
times and vitals while the
backend of the
application is under load
• Results correlated for
“end-to-end”
Load
3GH+
4G WiFi
APPMonitor
Cloud LGsOn-premise LGs
Database ServerApp Server Web Server
Controller
Real
Devices &
Networks
Functional
Simulated
Devices
w/
Network
Emulation
Real Device Metrics
4G
3GH+
Real Device Metrics Under Load
27. www.neotys.com© 2015 Neotys. All Rights Reserved.
COMPLETE MOBILE EUE
Real Device Users
Virtual Users Load
Real Device
Testing Tool
Load
Testing Tool Application
Server
Load End User
Experience
Round-Trip
Time
00:00:00 00:00:30 00:01:00 00:01:30
28. www.neotys.com© 2015 Neotys. All Rights Reserved.
SUMMARY
Mobile devices are everywhere. Application Performance is critical.
Performance usually means “End User Experience”
Test with a combo of
AUTOMATED & INTEGRATED
solutions
MEASURE & CORRELATE