Eran Kinsbruner's presentation from Enterprise Apps World 2015, London Includes Mobile Test Coverage strategy, Best practices around mobile test automation and a cook book for a Continuous quality for mobile apps delivery

1. Mobile
Test
Automa/on
&
The
Need
for
Con/nuous
Tes/ng
to
Ensure
High,
Mul/pla=orm
Performance
June,
2015
Eran
Kinsbruner
Presented
By:
Gidi
Pridor

2. Introducing
Perfecto
Mobile
GeHng
No/ced
Good
Friends
At
a
glance
enabling
manual
and
automated
mobile
applica0on
tes0ng
as
well
as
applica0on
monitoring
for
increasingly
agile
delivery
processes.
“Perfecto Mobile marks shift in mobile
apps QA with Continuous Quality Lab”
•
Founded
in
late
2006,
Business
HQ
in
Boston
and
R&D
in
Israel
•
Proven
management
team
with
over
200
employees
globally
•
Funded
by
leading
VCs,
Total
-­‐
$57.8M

3. Agenda
3
Mobile
market
START
Mobile
TesQng
Coverage
Mobile
Projects
Challenges
Mobile
Test
automaQon
CI
&
Best
PracQces

4. Easily
Applied
for
Browsers
…
Mobile
Create
Challenges
…
UI
IntegraQon
Unit
TesQng

5. UI
IntegraQon
Unit
TesQng
Performance
Security
Mobile
Requires
ShiQing
the
Plan
…
Target
Devices
Target
Env.
OS
OS
versions
Form
factors
CPU
&
Memory
Manufacturers
Wearable
Networks
OrientaQon
Events
Device
events
LocaQons
NoQficaQons

6. 6
• Pain:
• Market
pressure
compressing
delivery
schedules
• Growing
App
complexity
driving
test
case
growth
• Rapidly
evolving
test
matrix
• Goal:
• Deliver
feedback
to
developers
faster
• Complete
test
coverage
within
each
sprint
cycle
• Strategy
• Increase
automaQon
• Adopt
conQnuous
integraQon
The
Challenges
–
Velocity
vs.
Quality
And
Or

7. What
does
it
take
to
cover
the
REAL
end
user
experience?
v
Device
&
Pla=orm
Model
OS
Version
Screen
Size
Form
Factors
v
Environment
LocaQon
Network
v
Condi/ons
Signal
Strength
Load
Others

8. Q1
2015
–
EU5
Index
50%
Preview
Supports:
• Sufficient
Device
Mix
• Legacy,
Reference,
New
• Allows
sufficient
OS
coverage
• Support
mix
of
Screen
Sizes
• Support
mix
of
hardware’s

9. UK
–
Q2
Sneak
Preview
Index
Smartphones:
1. iPhone
5S
2. iPhone
5C
3. Samsung
Galaxy
S4
4. iPhone
6
5. iPhone
6
Plus
6. Sony
Xperia
Z3
7. HTC
One
M8
8. Nexus
6
9. Samsung
Galaxy
S5
10. Samsung
Galaxy
Note
4
11. Samsung
Galaxy
A5
12. Samsung
Galaxy
S6
13. Samsung
Galaxy
S6
Edge
14. HTC
One
M9
15. LG
G4
16. LG
G3
Tablets:
1. Apple
iPad
Mini
3
2. Apple
iPad
Air
2
3. Apple
iPad
Air
4. Samsung
Galaxy
Tab
4
–
7.0’’
5. Samsung
Galaxy
Tab
S
10.5''
6. Alcatel
One
Touch
Pop
7
S
7. Sony
Xperia
Z3
Tablet
8. Google
Nexus
9
(HTC)

10. Application name position
Labels
Menu panel
Button images
Text colors
and images
Cross
Plakorm
Differences

11. Func/onal
Automa/on
–
Mobile
Objects
100%
accurate
naQve
object
rec.
Much
faster
than
OCR
Support
all
languages
Does
not
onen
detect
GUI
defects
ImperaQve
for
end
user’s
experience
ImperaQve
for
GUI
glitches
Implement
the
WYSIWYG
Slower
then
object
level
analysis
Dependent
on
3rd
party
Does
not
support
all
languages
Object
level
analysis
Visual
analysis
(OCR)
May
require
InstrumentaQon

12. Mobile
Test
Automa/on
That
Works
• Develop
for
testability
• Object
RecogniQon
– Mix
‘n
match
methods
• Image
recogniQon
• DOM
• Object
id
• XPath
• CSS
• Robust
– Error
handling
– CondiQonal
flow
– Reusable
funcQons
– Data
driven
Reliable
Repeatable
Maintainable
Scalable

13. Error
handling
Remove
CI/Automa/on
Blockers
System
level
control
ApplicaQon
control
Device
control

14. Mobile
Con/nuous
Integra/on
Recommended
Work
Flow
Dev Team
Source
Control
Code
Commit
CI
Server
ReporQng
Unit
Tests
Launch Perfecto Job
(Release Package)
Support today:
PM Native & Selenium
Automated
App
Deployment
On
Real
Devices
Functional Testing
Integration Testing
Performance Testing
Con/nuous
Feedback
Test
Planning
&
Test
Execu/on
Bug
ReporQng

15. Manage
the
Time
Budget
…
Build
Frequency
Test
Dura/on
Test
Scope
Device
Coverage
Unit
System
Smoke
Regression
Performance
Hourly
20-­‐40
Min
±4
3-­‐4
/mes
a
day
30-­‐60
Min
±6
Nightly
2-­‐7
Hours
±10
Weekend
10-­‐48
Hours
±20

16. Velocity!
Quality!
Strategies
for
achieving
5
stars

17. 1.
Assure
device
coverage

18. 2.
Lab
Sizing
Me
AnalyQcs
Teams
Projects
SDLC
Cycle
Requirements
v
Market
My
Space
Popular
&
emerging
The
Index
1.
Coverage
Industry
trends
&
insights
1.
Assure
device/OS
coverage

19. April
OCTOBER
JUNE
JULY
AUGUST
SEPTEMBER
Sony
Xperia
Z4
Tablet
New
iPhones
&
iPads
Samsung
Note
5
Motorola
Moto
X
(New)
iOS
9
Beta
Windows
10
for
Phones
DEVICES
PLATFORM
Mobile
Market
Calendar
2015
Android
M
?
MAY
Apple
Watch
Release
Samsung
Galaxy
S6
Samsung
Galaxy
S6
Edge
LG
G4
Nokia
Lumia
640
XL
ZTE
Blade
S6
Plus
Huawei
P8
Lenovo
A6000
Plus
New
trending
devices,
First
Samsung’s
to
run
Android
L
OTS
You
can
already
start
Beta
tesQng
in
June!
Seek
for
Emulators
of
iPhone
6S
and
more
close
to
the
release
date!
Seek
for
early
Beta
through
Android
Studio
New
nexus
devices
Perfecto
Mobile’s
Q2
Index
Release!!

20. 2.
Built
an
adap/ve
test
plan
Ø Update
your
test
plan
according
to
customer
feedback
Ø Automate
to
run
regularly

21. Mobile
Ready
AutomaQon
Fast
Impact
uQlizing
exisQng
knowledge
Test
on
real
devices
Sustainable
–
doesn’t
break
CI
Ready
Device
AgnosQc,
any
app
type
Unaxended
3.
Mobile-­‐Ready
Automa/on
is
Key
Become
a
Jedi
Master
you
will

22. Does
your
Lab
come
with
“Environment
Controls”?
Are
you
tesQng
“in
the
Wild”?
4.
Test
in
Real-­‐World
Condi/ons

23. Build
Your
Mobile
App
Test
Plan
Properly
Functional
• Validation of Functionality
• Smoke / Regressions Testing
• Offline access testing
• Negative Testing
Non Functional
• Network Strength / Outage /
Recovery
• Different NW Types
• Peripheral Testing
• Services
Interoperability (IOP)
• Voice / SMS interrupts
• Notifications
• Battery /Cable Removal
Memory Leak
• Memory Usage
• Memory Leaks
• Garbage Collection
Installation Testing
• New App Install
• Uninstall and Reinstall
• Upgrade testing
Language Testing
• Validation for Locales (French &
English)
• Images and Text
• Currencies, time zones etc.
• Context
Performance Testing
• CPU Usage testing
• Network Usage
• Page Render time or activity
Render time
Security Testing
• OWASP Vulnerabilities
• Sensitive data (App/Device)
• Dynamic Testing
• Static Code Analysis
• Data Encryption
Usability Testing
• User Experience
• Competitive Analysis
• Expert Review

24. Eran
Kinsbruner
Dir,
Technical
Evangelist
Perfecto
Mobile
Erank@perfectomobile.com
Gidi
Pridor
VP,
Strategic
MarkeQng
Perfecto
Mobile
Erank@perfectomobile.com