Title: How to get the best ROI on Your Test Automation Effort?

1. How Much Test Automation is
Enough and Why?
- By ShriKant Vashishtha

2. Importance of Test Automation

3. The Paradox of Test Automation

4. • You may be writing a lot of test
code. Sometimes it's twice or
thrice of the size of base code.
• You don't know how much
coverage is enough?
• Even after doing good amount of
test automation, many teams are
still not sure if they really trust
automation suite
Let’s Investigate

5. • Because of the siloed way of working (developers vs testers),
many teams go for functional test automation in which specialist
automation developers develop these tests
• In a team of developers and testers, testers (who may not have
coding skills) may not be able to validate the developers tests
and still do their own testing.
• Writing automated tests (twice or thrice of base code size)
takes time
• People move towards the concept of 100% code coverage as
they may not know how much is enough.
Translates into…

6. • Why writing tests takes so much time?
• Is there a way to know if we have enough coverage other than
so called 100% code coverage?
Lead Us to Questions…

7. Why writing tests takes so
much time?

8. A Siloed Development Team

9. • Need to test
class/method in
isolation without its
dependencies.
A ‘Mock’ Understanding of the
Unit Test

10. • Dependencies are
mocked
• Results into mock
code which may be
bigger than the actual
code under test
A ‘Mock’ Understanding of the
Unit Test

11. Problems with Such Tests

12. Difficult to Know Sometimes Why
a Test Broke

13. Refactoring is Tied to
Implementation

14. • We should be able to refactor our code, keep it basically fit and
clean.
• We wanted to change our code and our tests were obstacle to
that change.
The Promise

15. • Changing the implementation details without breaking tests.
Let’s go Back and Understand The
Promise of Refactoring

16. Let’s Go Back to The Source

18. Avoid Implementation details, test
behaviours
-- Kent Beck

19. The trigger in TDD to create a new
test is that you have a requirement you
want to implement.

20. The SUT (System Under Test) is not a Class

21. • Do not write tests for implementation details, they change.
• Write tests only against the stable contract of the module
interface

23. Test the behaviour not the class

24. “What behaviour will we need to produce the revised report?
Put another way, what set of tests, when passed, will
demonstrate the presence of code we are confident
will compute the report correctly?
• We need to be able to add amounts in two different currencies
and convert the result given a set of exchange rates.
• We need to be able to multiply an amount (price per share) by
a number (number of shares) and receive an amount.”
Excerpt From: Kent Beck. “Test-Driven Development By Example”.
Apple Books.

25. Unit of isolation is test and not
class under test.

26. “When we write a test, we imagine the perfect interface for our
operation. We are telling ourselves a story about how
the operation will look from the outside. Our story
won't always come true, but it's better to start from the
best-possible application program interface (API) and
work backward than to make things complicated, ugly, and
"realistic" from the get-go.”
Excerpt From: Kent Beck. “Test-Driven Development By Example”.
Apple Books.

27. • Strive to decouple your tests from your implementation details
• Tests should not depend on details because then changing
implementation details break tests.
• Tests should depend on contracts or public interfaces
• Test Behaviours and Not Implementations
Summary on Mocking

28. Weinberg’s Decomposition Law

29. • Design classes and APIs using good domain design techniques that
generalize to the future. Code up their interfaces to express the
design. Implement only those data structures and
methods necessary to support the known,
forthcoming use cases that will be delivered. If the
system use cases change, you can continue to evolve the individual
methods and private data without disturbing the overall
architecture.
• The payoff in investment is higher in system level testing than in unit
testing: put your effort there. System testing typically finds twice as
many bugs as unit testing, even though agile developers spend most
of their time doing testing at the unit level, including TDD.
Instead This is What We Could Do

30. • A smarter approach would reduce the test code mass through formal test design: that is, to do formal boundary-condition
checking, more white-box testing, and so forth.
• Move Assertions directly into the code
• Use them to feed your fault-tolerance architecture on high-availability systems. This solves the problem of
maintaining a lot of extra software modules that assess execution and check for correct behavior; that’s one half of a unit
test. The other half is the driver that executes the code: count on your stress tests, integration tests, and system tests to do
that.
Turn Unit Tests to Assertions

31. Questions?