Поговорим о проблемах автоматизации для iOS и Web, о том как синхронизировать работу manual и automation QA, о проблемах которые возникают когда люди практически без опыта программирования начинают писать автоматические тесты и о том как мы эти проблемы решили. Затронем тему распараллеливания выполнения тестов. Поговорим о том, как выглядит процесс тестирования продукта, в котором 70-80% функционала автоматизированно. Цель доклада показать как можно построить эффективный процесс автоматизации и как manual QA могут помочь в этом.Хоть мы будем говорить о WebDriver и Appium, в целом доклад на них не фокусируется (так же не требуется знание какого-то конкретного языка программирования).

1. Mobile automatic testing: done by Readdle
Sergey Hizhnyak

2. About Readdle
Readdle creates popular productivity apps such Spark
email, Documents, Scanner Pro and PDF Expert.
The team is 110 people and we managed to build a
sustainable business, never taking VC money or external
funding.
We are the pioneers of iOS App Store and one of the first
companies to create file management and scanning
apps. Available on the App Store from the day one, our
apps have been downloaded more than 40 million
times.

3. About myself
• Started to work in QA more than 10 years ago
• Worked as automation QA, performance QA,
white box tester
• Currently QA manager for Fluix department in
Readdle

4. Setup
• WebDriver
• Appium
• Java
• Cucumber JVM
• Jenkins

5. Why?
• Why not Allure?
• Why Cucumber and BDD?
• Why managers read our reports?

6. Because
• Single development language
• Shared API and utilities
• Easy to add new platforms (Android, Windows)
• Easy to adopt by non-programmers

7. Testing process
• Tests in TestRail with unique ID
• Link Cucumber scenario
• Populate TestRail with execution status
• Visible coverage and testing progress
• We do not have a concept of passed/failed for the
whole automation

8. Parallel execution: issue
• Execution time exceeded 4 hours
• Cucumber is not designed to run in
parallel 

9. Parallel execution: solution
• Why we do not use Grid?
• Why we do not use some open-source
solution?
• Web service that builds queue from Cucumber
scenarios

10. Solution bonus
• Online execution tracking
• Current queue visualization
• Running scenario status
• Execution statuses and statistics

11. Tests maintenance: issue
• Tests fail a lot
• 1000+ tests report analysis
• Wasted time!

12. Tests maintenance: solution
• Sorting of failed tests
• Development
• Stable: passed 5 times in a row
• Verification
• Note: this is purely for our convenience

13. • 70% of test cases are automated
• Manual QA are “developing” Cucumber
scenarios
• Manual testing of all non-automated
functionality
• Manual retest for all failed/unstable automation
• 6-8 months of stable releases
• And we failed 
Our “perfect” testing process

14. Fail #1: Simulators
• Memory management and data protection
• Why do we run on simulator?
• Conclusion: test manually

15. Fail #2: TestRail<->Cucumber mess
• Test was moved/removed
• Link with wrong ID
• Copy-Paste

16. Fixing the mess
• Compare Cucumber with TestRail
• Multiple IDs
• Incorrect names
• Non-existing TestRail tests
• Etc.

17. Is it available somewhere?
• All code is open source (read-only)
• It is not very clean, but you could get some
ideas
• https://github.com/k-g-aleksandrov/cucumber-
queue-runner

18. Conclusion
• Develop your own solution
• Automate surroundings
• Check manually
• Involve everyone in developing your automation
• Have fun!

19. Q&A