Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Mobile Testing challenges
Automation
Mobile Automation Meetup , February 2015
Yuval Golan , Erez Nachum , Tali Gilon , Von...
www.vonage.com 2
www.vonage.com 3
Mobile is “Taking over the world”
It’s a multi platform age with mass adoption.
Where is your Mobile Test...
www.vonage.com 4
Why is it
so simple
to QA
mobile apps?
Complexity Scope Execute Loop back CI Automation
www.vonage.com 5
Complexity
• More than 15K Android devices (last year 4k..)
• Different drivers
• Hundreds of carriers
• ...
www.vonage.com 6
Scope
• Functional
• Usability
• Load
• security
• Localization
• Bugs we wish to resolve
• Network Impai...
www.vonage.com 7
Execute
• Module
• Level
• Code coverage
• Recent bugs found / fix
• Recent commits / changes
Complexity ...
www.vonage.com 8
How to execute
• Manual
• Automate
• Crowd
• Perfecto / device any..
Complexity Scope Execute Loop back C...
www.vonage.com 9
Loopbacks: CI and field feedback
CI Continuous Integration
Pre Commit
QA in most organizations
Field Feed...
www.vonage.com 10
Metrics - Crashes
47% of the
apps crashes
1% of their use
32% of the
apps crashes
2% of their use
Comple...
www.vonage.com 11
Metrics , Crashes November 2013 – November 2014
Complexity Scope Execute
Loop
back CI Automation
www.vonage.com 12
Be alert
listen to the customers and market
• OS (7.1 no audio)
• IPV6 / IPV4 (crash)
Complexity Scope E...
www.vonage.com 13
Metrics – Star Ranks / user reviews
Overall Store Rank by Version
Complexity Scope Execute
Loop
back CI ...
www.vonage.com 14
Mobile CI for Commit
Developer commit
some code to the
repository
Jenkins (CI) get
notified and starts
c...
www.vonage.com 15
E2E (Seetest) Per commit (KIF +
Espresso)
Automation dev
investment
Write test once for both
platforms
D...
www.vonage.com 16
Vonage Automation Framework Features• Reporting system (using Jenkins abilities + Junit reports )
• Smar...
www.vonage.com 17
Nightly run
2 tests fail because of an Amazon issue
on uploading large video files.
Good morning
☺
Compl...
www.vonage.com 18
Web client
No need in secondary devices for manual/automated tests,
web client behaves like a real clien...
www.vonage.com 19
UI – system with Keywords
anyone can
write tests using
it!
Complexity Scope Execute Loop back CI Automat...
www.vonage.com 20
Module tests
Country Manager module tests we inject input
to the module's functions and compare the outp...
www.vonage.com 21
DDT – Data Driven Test
Send message setup- ~15 lines
Json file with scenarios-
Data Driven
testing
One f...
www.vonage.com 22
What’s next?
•Coverage , coverage , coverage
•Appium
•Appurify
•Qtest
What’s Next ?
Complexity Scope Exe...
www.vonage.com 23
Thank You
Yuval.golan.il@gmail.com
Prochain SlideShare
Chargement dans…5
×

Mobile QA Metrics Risks and Automation Presentation By Yuval Golan

977 vues

Publié le

Mobile QA automation Presentation
February 2015
At Google meetup center
By Yuval Golan

Publié dans : Mobile

Mobile QA Metrics Risks and Automation Presentation By Yuval Golan

  1. 1. Mobile Testing challenges Automation Mobile Automation Meetup , February 2015 Yuval Golan , Erez Nachum , Tali Gilon , Vonage Mobile team
  2. 2. www.vonage.com 2
  3. 3. www.vonage.com 3 Mobile is “Taking over the world” It’s a multi platform age with mass adoption. Where is your Mobile Testing,mobile CI and Mobile automation? Complexity Scope Execute Loop back CI Automation
  4. 4. www.vonage.com 4 Why is it so simple to QA mobile apps? Complexity Scope Execute Loop back CI Automation
  5. 5. www.vonage.com 5 Complexity • More than 15K Android devices (last year 4k..) • Different drivers • Hundreds of carriers • Bandwidth & Connectivity • Tens of OS • Resolution (LDPI , MDPI…) • Access rights • It is actually a phone .. it vibrate , Peripherals (BT ,earpiece), airplane mode , notifications… Environment Device Code Complexity Scope Execute Loop back CI Automation
  6. 6. www.vonage.com 6 Scope • Functional • Usability • Load • security • Localization • Bugs we wish to resolve • Network Impairments • Battery • Store / purchase • Disconnections / interrupts (alarm clock ; calls) Complexity Scope Execute Loop back CI Automation
  7. 7. www.vonage.com 7 Execute • Module • Level • Code coverage • Recent bugs found / fix • Recent commits / changes Complexity Scope Execute Loop back CI Automation
  8. 8. www.vonage.com 8 How to execute • Manual • Automate • Crowd • Perfecto / device any.. Complexity Scope Execute Loop back CI Automation
  9. 9. www.vonage.com 9 Loopbacks: CI and field feedback CI Continuous Integration Pre Commit QA in most organizations Field Feedback Complexity Scope Execute Loop back CI Automation
  10. 10. www.vonage.com 10 Metrics - Crashes 47% of the apps crashes 1% of their use 32% of the apps crashes 2% of their use Complexity Scope Execute Loop back CI Automation
  11. 11. www.vonage.com 11 Metrics , Crashes November 2013 – November 2014 Complexity Scope Execute Loop back CI Automation
  12. 12. www.vonage.com 12 Be alert listen to the customers and market • OS (7.1 no audio) • IPV6 / IPV4 (crash) Complexity Scope Execute Loop back CI Automation
  13. 13. www.vonage.com 13 Metrics – Star Ranks / user reviews Overall Store Rank by Version Complexity Scope Execute Loop back CI Automation
  14. 14. www.vonage.com 14 Mobile CI for Commit Developer commit some code to the repository Jenkins (CI) get notified and starts compilation On success “Static analysis” and post commit Code review running Deploy an environment Then automated “Commit tests” When commit is approved it is included in next QA build , the latest stable build will run on nightly regression New code version is distributed to QA/BETA Complexity Scope Execute Loop back CI Automation
  15. 15. www.vonage.com 15 E2E (Seetest) Per commit (KIF + Espresso) Automation dev investment Write test once for both platforms Different code per platform, duplicate tests Native actions V (Some IOS issues) X Scalability License required Open source (dev use it) Automation project location Automation repository Part of Dev repository Execution time 1m-2m (per E2E test, 2 devices) 0.2m-0.3m (1 device only) Reports Built in Requires development Number of mobile clients in use 1+ (we use 2) 1 Support V X Automation 3rd party Tools that we use Complexity Scope Execute Loop back CI Automation
  16. 16. www.vonage.com 16 Vonage Automation Framework Features• Reporting system (using Jenkins abilities + Junit reports ) • Smart investigation tools (search for specific exceptions and can connects them to existing Jira tickets, saves investigation time) • Web Client (Vonage Mobile code compiled on Mac/PC, simulates 2nd device ) • Automation Test Apps (in order to simulate native changes in native address book ,native settings…) • Retry mechanism (differentiates between one timers to 100% reproducible bugs) • Network traffic investigation tools (Http request- response, SIP flows in calls , uses app logs) • Private BE environment (Http request- response, SIP flows in calls, uses app logs) Our Framework Complexity Scope Execute Loop back CI Automation
  17. 17. www.vonage.com 17 Nightly run 2 tests fail because of an Amazon issue on uploading large video files. Good morning ☺ Complexity Scope Execute Loop back CI Automation
  18. 18. www.vonage.com 18 Web client No need in secondary devices for manual/automated tests, web client behaves like a real client and can even initiate events to your App. Complexity Scope Execute Loop back CI Automation
  19. 19. www.vonage.com 19 UI – system with Keywords anyone can write tests using it! Complexity Scope Execute Loop back CI Automation
  20. 20. www.vonage.com 20 Module tests Country Manager module tests we inject input to the module's functions and compare the output to the expected results. Complexity Scope Execute Loop back CI Automation
  21. 21. www.vonage.com 21 DDT – Data Driven Test Send message setup- ~15 lines Json file with scenarios- Data Driven testing One function + one “Jason” = ~1K tests Complexity Scope Execute Loop back CI Automation
  22. 22. www.vonage.com 22 What’s next? •Coverage , coverage , coverage •Appium •Appurify •Qtest What’s Next ? Complexity Scope Execute Loop back CI Automation
  23. 23. www.vonage.com 23
  24. 24. Thank You Yuval.golan.il@gmail.com

×