In this SauceCon 2019 presentation, Mario describes the practices that ABN AMRO adopted in mobile teams when it comes to testing native applications on real devices. Since using Espresso and XCUITest is still relatively uncommon for large apps and there are various unique challenges due to being in an EU-regulated industry with various security restrictions, Mario will share the ABN AMRO team’s experiences, including: - A brief architectural overview of the Mobile Banking app: why it is all native (Objective-C/Swift and Java/Kotlin), how it communicates with other apps and websites - Why ABN AMRO choose to adopt Espresso and XCUITest: the advantages and the limitations of this choice - How ABN AMRO runs their test pipelines to spread them across time and devices and prevent teams from being blocked

1. CLOSE TO THE METAL
WHY AND HOW WE USE XCTEST & ESPRESSO
Mario Negro Ponzi – CoE Software Development and Tooling

3. THE APPS
Mobile Banking Grip Tikkie &Meer

4. PlanPrinciples
SUMMARY
Execution

5. PlanPrinciples
SUMMARY
Execution

6. GITFLOW PROBLEMSTHE PRINCIPLES
1. Anytime releases
2. DevOps teams
3. Infrastructure

7. Release
as often
as possible
ANYTIME RELEASES

8. Teams
must be
fully
autonomous
DEVOPS TEAMS

9. GITFLOW PROBLEMSINFRASTRUCTURE
Building
infrastructure
is not
our core
competence

10. GITFLOW PROBLEMSINFRASTRUCTURE

11. PlanPrinciples
SUMMARY
Execution

12. GITFLOW PROBLEMSTHE PLAN
1. What we tried
2. Tooling
3. CI/CD

13. • We did build a dedicated
test automation team
• Get the business people
write the tests
• Aim at cross-platform
compatibility
WHAT WE TRIED

14. WHAT WE TRIED – GHERKIN
Feature: Google Searching
As a web surfer, I want to search Google, so that I can
learn new things.
Scenario: Simple Google search
Given a web browser is on the Google page
When the search phrase "ABN AMRO" is entered
Then results for "ABN AMRO" are shown

15. 15
TOOLING

16. 16
TOOLING
What do we
lose?

17. 17
CI/CD
Automation also means
unessential custom
steps (tools) must be
avoided to reduce the
complexity of the
pipeline

18. PlanPrinciples
SUMMARY
Execution

19. 1. A third party must manage the devices
2. User management per team and individual
developers and testers allowing for
DevOps
3. Support for XCtest and Espresso
WE USE SAUCE LABS BECAUSE…

20. Web testing…
😀
BONUS POINTS

21. • UI Tests
• E2E tests
KIND OF TESTS

22. THE PIPELINES
Main
runs unit and smoke tests
(on every commit)
UI
runs UI tests (hourly)
E2E
runs E2E tests (nightly)

23. It does everything you would expect from a ‘normal’ pipeline
• It builds the apps
• It runs unit tests
• It performs a number of quality and security checks
• It runs smoke tests*
• It notifies via Slack
MAIN PIPELINE
* Some parts are still not completed

25. It checks that the UI looks right on multiple devices
• It runs hourly
• It runs UI tests on multiple random devices (4 + reference device)
• It notifies via Slack
• It does not require a server
• It will not require a Mac to run (even for iOS)
UI TESTS PIPELINE *
* Currently implemented only on Android

26. UI TESTS EXAMPLE

27. The real deal
• It runs nightly
• It runs E2E tests against a mock server
• The mock server runs on AWS Lambda and emulates Wiremock
• It notifies via Slack
• It will not require a Mac to run (even for iOS)
• If successful it will re-run the tests against our test environment
E2E TESTS PIPELINE *
* Not fully working, yet

28. E2E TESTS EXAMPLE

29. java -jar runner.jar android
--test <test.apk>
--app <app.apk>
--apikey <my project API key>
--datacenter EU
--device Samsung_Galaxy_Note8_ab
A SIMPLE COMMAND

30. Photos by various
authors on Unsplash