We Android developers use the "flakiness" word more than the "stability" word while talking about Espresso and Instrumentation tests. We cannot truly trust flaky tests. Instrumentation testing will continue to be part of our lives even with Jetpack Compose therefore, we need stability.
Running all UI tests before merging any code is possible, and it saves us a lot with 400+ integration tests. Unfortunately, stability is not something that comes out of the box. A certain level of architecture, proper thread usage, and test setup with helper rules are required. There is an uncountable number of things that affect UI tests.
Good thing, there are explanations and solutions for most of the flakiness. In this talk, you will learn more about best practices to increase test stability, how to utilise Test-Driven Development with UI testing, and how this will help you find bugs earlier.
Sinan Kozak
800 tests run on every commit under 20m
PR wont merge without succesful AndroidTest
18 Different features has their Android test in their module
We dont depend on UI test. Only most important features are tests end to end
Almost all main usacases covered
User interface (UI) testing lets you ensure that your app meets its functional requirements and achieves a high standard of quality such that it is more likely to be successfully adopted by users.
One approach to UI testing is to simply have a human tester perform a set of user operations on the target app and verify that it is behaving correctly
Time-consuming, tedious, and error-prone
The automated approach allows you to run your tests quickly and reliably in a repeatable manner.
androidTest folder
Creates apk to run on device
UI interaction
End to End flow of feature
Device resources
Pure java kotlin classes can be tested on jvm
They are faster
We should aim to have business logics in class we can test
In reality we wont be able to test everything on jvm
User interface
Device resource usage
End to end verification from user persfective
Who stop using android test because off flakiness
There is no out of box solution
Device control. We want to focus on WHAT to test
Delegate device control as much as possible
Threading issues
Framework issues
APIs to write AndroidTests.
Dont uses your classes
Use app from user persfective
Espresso focuses on how to find and how check views
Espresso is aware of multiple things
Espresso is low APIs focus on HOW to test
As developer we want to focus on WHAT to test
Espresso is API around actions and espresso is has HOW to find views or HOW to check conditions
10 minutes
How we can reduce learning curve
Kakao is a library around Espresso APIs.
The goal of the Kakao is to provide better readability and discoverability for Espresso testing.
Kakao doesn’t aim to change the behaviour of Espresso.
By default, Kakao enforces the same mentality of robot pattern. Kakao expects a Screen class. And in Screen class, you can define common UI elements or actions using Kotlin DSL.
Kakao has solutions for common UI element
Easy to discover
Type safety about actions and check
Easy to read tests
But this is not about stability
Lets talk about most common usecases
1st animation
Because Espresso isn't aware of any other asynchronous operations, including those running on a background thread, Espresso can't provide its synchronization guarantees in those situations. In order to make Espresso aware of your app's long-running operations, you must register each one as an idling resource.
For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
It would be great if we can have one idling resource for threading solution we have.
And actually we have some
20 minutes
For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
There is not a clean solution
We can use similar provider solution as ExecutorProvider
It is not possible to prevent libraries to use Default dispatcher
SetDefault vs setMain
With rx all of background operations covered
It feels like we are not goiing forward but keep craeting new problems
If you remember the usecases for espresso maybe we can find a different solution.
Robost solution
If you remember the usecases for espresso maybe we can find a different solution.
Robost solution
If we cannot control concurency framework, we can track background operations directly
For all other background operations, test authors should register the custom resource via IdlingRegistry and Espresso will wait for the resource to become idle prior to executing a view operation.
Create an interceptor and register idling resources with the information you have. In interceptor you can track current network operations.
After the network operation, you set the data to live data.
Everything will be good, right?
No
We set a countiing executor for arch component.
DataBinding will use value of livedata and there wont be a flakiness, right?
No
There is no easy way to keep track of when data binding applies changes to screen.
There is only hasPendingIntent information
Simple solution is not using databindiing
But if yyou have databindiing, yyou need to regiister and idliing resource and check all the views.
All theeee views
Network, livedata and databinding wont be a problem.
We can use ListAdapter and submit a new list and everything will allright?
You probably already understand the theme of this flow.
No
We started a data flow network layer until data reaches to screen, we needed 4 solution.
We will definitely need moree solutions
That is why it is nice to have a single place to share solutions
30 minutes
There are different options to run UI tests.
You can create your own test farm in the company with real devices.
You can use Github actions and run an emulator on Github action.
You can use other build frameworks like Jenkins and set up your own emulator on CI.
All of those options will need maintenance and they will be bound to your investment of money and time.
Run on google infrastructure
All of the devices on Firebase Test Lab have special adjustments
We prefer to use virtual devices as there is generally no queue for them
They are also active in Firebase slack.
You can ask questions or report issues. And from my own experience, they help individual cases.
One cool feature of Firebase test lab is sharding
Split tests into buckets and run them on different devices to parallelize execution
Cost will be same
Duration will depend on how small the test buckets are
Creating test matrices that should finish under desired time is a difficult problem.
Luckily we don’t do that setup by hand.
We need gradle integration for Flank
And Fladle is gradle wrapper
We have pipeline
Cover more usecases at scout
Good improvements coming to android testing
Espresso is great library.
I always feel that espresso is just starting point like any other coffees.
They provided base element so we can add milk or water and use it