With an ever increasing amount of traditional, web-based content being accessed from mobile devices performance testing of mobile apps has become critical. According to the 2012-2013 World Quality Report, 64% of firms identified performance as a focus for mobile testing over both functional testing and security. It’s no wonder why firms are placing this degree of focus on mobile performance. Consumers expect their mobile content to be delivered as fast as it would be through a desktop / high bandwidth connection and they won’t hesitate to find other options when it’s not. On the surface, performance testing of mobile apps may appear to be no different from web apps. However, applying the same testing techniques used for traditional web-based systems may produce inaccurate results, or fail to expose performance issues. Mobile apps present unique differences and challenges that must be addressed to ensure performance tests accurately represent production user behavior and load profiles. This session will discuss how to address unique mobile app challenges such as device / OS diversity, mobile user behavior and network variability. Attendees will walk away with an understanding of best practices for ensuring the performance of their business critical mobile applications.

1. © Utopia Solutions
Best Practices for Performance Testing Mobile Apps
Lee Barnes, CTO
Utopia Solutions

2. © Utopia Solutions
Apology
I don’t believe in “best practices”…
I do believe in guidelines and better
approaches for a given situation

3. © Utopia Solutions3
Key Questions
How is mobile different?
How can I adapt?
Why is performance of mobile
important?

4. © Utopia Solutions4
Why Should I Care?
M-Commerce
66%
OF SHOPPERS
ABANDON
TRANSACTIONS
5 OUT OF 10
BECAUSE OF POOR
PERFORMANCE
1DELAY
SECOND
7%DROP IN
CONVERSIONS
equals a
2013 Mobile Commerce Insights
study conducted by Jumio

5. © Utopia Solutions5
Still Not Convinced?
2012-2013 World Quality
Report

6. © Utopia Solutions6
Evolution of Performance Testing
1990 2000 2010
2-Tier
Client Server Web Mobile
Challenges
State of the
Practice
Testing Large
Loads
Complex /
Integrated
Systems
Proprietary
Protocols
Skilled
Resources
In-Lab
Testing
Tool
Maturation
Open Source
Tools
Specialized
Resources
Cloud-based
Testing

7. © Utopia Solutions
Mobile Challenges & Adaptations

8. © Utopia Solutions8
Mobile Challenges – User Access
Native App Mobile Site
Full / Responsive
Site
Your users have options…

9. © Utopia Solutions9
Assess the User and Load Profile
1. Determine the user profile
• Business processes
• Native app, mobile site, full site
• Preferred browser
• Network conditions
• Geographic location
2. Determine the load profile
• Volume of users by business
process and location
• Volume of users by access type
• Frequency / throughput
Incorporate constant feedback from application logs
and other monitoring solutions

10. © Utopia Solutions10
Mobile Challenges - Network
Mobile User Connected User
3G 4G InATunnelG
Network bandwidth and quality affect both end user
experience and system load

11. © Utopia Solutions11
Assess Impact of Network Variability
1. Assess baseline performance
(single user)
Observe UX with real devices
behind network virtualization
solutions
2. Assess performance under
load
Put system under virtual user load
Observe UX with real devices behind
network virtualization solutions
NV
Virtual Users
NV
Real
Devices

12. © Utopia Solutions12
Network Virtualization Tools
Single User (Device and/or
Emulator)
• Apple Network Link Conditioner
• Android Emulator
• Charles Proxy
• Shunra NV
Multiple Virtual Users
• Shunra NV
• Load test tools (built-in)
• Charles Proxy

13. © Utopia Solutions13
Mobile Challenges – User Location and Volume
A high volume of global users is difficult to reproduce in a lab…

14. © Utopia Solutions14
Utilize Cloud-based Testing Solutions
• Achieve production level loads
• Test entire infrastructure (vs.
behind the firewall)
• Distribute load geographically
• Assess user impact on real
devices

15. © Utopia Solutions15
Mobile Challenges – Device Performance
Measuring System / Network Metrics is Not Enough…
Memory
CPU
Battery Drain
Graphics
I / O

16. © Utopia Solutions16
Measuring Device Performance
Critical device metrics:
• CPU
• Memory
• Battery
• Storage
Mobile development platforms
• Android DDMS
• iOS Instruments
Mobile test automation tools
• TouchTest (SOASTA)
• Trust (Mobile Labs)
• MonkeyTalk (Cloud Monkey)
Mobile lab / cloud solutions
• Device Connect (Mobile Labs)
• Keynote DeviceAnywhere
• Perfecto Mobile
Capture metrics while exercising app
Include appropriate devices /
configurations
Incorporate automation for consistency
Device Performance
Measurement Solutions

17. © Utopia Solutions17
Mobile Challenges – Agile Development
Analysis
Design
Develop &
Unit Test
Acceptance
Test
Final
Acceptance
Test
Deploy
Source: Agile and Test Estimation
Sharon Robson
How does performance
testing fit?

18. © Utopia Solutions18
Performance Testing in an Agile Environment
Plan Design Build Test Deploy
Waterfall
Performance Testing
Too late!

19. © Utopia Solutions19
Performance Testing in an Agile Environment
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance Testing
Same Process…
Same Risk!

20. © Utopia Solutions20
Performance Testing in an Agile Environment
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance
Testing
Performance
Testing
Performance
Testing
Performance
Testing
Waterfall Approach
• Rigid
• 4-6 week test cycle
Agile Approach
• Flexible
• 2 week sprints
+ =

21. © Utopia Solutions21
Early Cycle /
Ongoing
Performance
Analysis
Performance Testing in an Agile Environment
Ultimate Goal…
Late Cycle
Performance
Testing
Shift Left

22. © Utopia Solutions22
Performance Testing in an Agile Environment
How?
Map business and end user
requirements onto system components
Benchmark in development / QA
environment
Look for trends in measurements vs.
absolutes
Use service virtualization to stub out 3rd
party or yet to be developed services
Execute unit performance tests as part
of continuous integration
Unit / Component
Performance Testing

23. © Utopia Solutions23
Performance Testing in an Agile Environment
How?
Measure resource / performance
profile of native apps
Use mobile site analysis services / tools
to analyze services and pages
User APM solutions to profile code
Use device profiling solutions to
understand / compare base device
performance
Application / Code
Profiling

24. © Utopia Solutions24
Waterfall Agile
Service Model Perform turnkey
performance test
Timing End of development cycle
Duration Weeks
Tools Traditional load test tools
Environment Dedicated / production like
Role of
development
Throw system over wall
Wait for results
Waterfall Agile
Service Model Perform turnkey
performance test
Provide mentorship to agile
team
Timing End of development cycle Ongoing
Duration Weeks Hours / days
Tools Traditional load test tools Traditional load test tools
APM tools
Service virtualization
Continuous integration
Environment Dedicated / production like Development / desktop
Role of
development
Throw system over wall
Wait for results
Constant focus on performance
Performance Testing in an Agile Environment
What about my Performance
Testing Center of Excellence?

25. © Utopia Solutions25
Summary
Key takeaways…
• Mobile users are not the same as
connected users
• Performance is more than backend /
network
• Push performance analysis to the left
in the development cycle

26. © Utopia Solutions26
Questions…
…. and Answers!
Direct future questions to:
Lee Barnes
Founder and CTO
Utopia Solutions, Inc.
Email: lee.barnes@utopiasolutions.com
Twitter: twitter.com/USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: www.RaiseYourMQ.com
?