Ever tried doing Test First Test Driven Development? Ever failed? TDD is not easy to get right. Here's some practical advice on doing BDD and TDD correctly. This presentation attempts to explain to you why, what, and how you should test, tell you about the FIRST principles of tests, the connections of unit testing and the SOLID principles, writing testable code, test doubles, the AAA of unit testing, and some practical ideas about structuring tests.
3. Contents at a Glance
Why Test?
What to Test?
FIRST Tests, the 5+3 Requirements of Tests
Unit Testing and Solid
Writing Testable Code
Test Doubles
The AAAs of Unit Testing
How to Test
Structuring Tests
5. “
”
It is unit tests that keep our code flexible,
maintainable, and reusable. The reason is simple. If you
have tests, you do not fear making changes to the code!
Without tests every change is a possible bug. No matter
how flexible your architecture is, no matter how nicely
partitioned your design, without tests you will be
reluctant to make changes because of the fear that you
will introduce undetected bugs.
[Clean Code – Robert C. Martin]
Tests are there for the developer to enable refactoring.
Tests are there for the developer and the PO to provide a living specification.
Tests are there for the developer to provide a living documentation.
Tests are there for the business to provide a fast feedback loop for catching bugs
early.
6. The Cost of Fixing Bugs in Different
Phases of the SDLC
Design Coding Testing Post-Shipping
1 unit 6 units 15 units 60-100 units
Factors:
The length of the feedback loop
Time it takes for the information to propagate back to the developer
The context switch cost for the developer
The design modification consequences
Having to modify the design after it is already implemented is costly
Modifying existing code might just be exchanging known bugs for unknown bugs
Not in a well-tested application, though
7. Implications
Design well
Have as short a feedback loop between design and development as possible
(do you even agile, bro?)
Move as much of your testing to development-time as possible
Automate tests as much as possible
9. Test I/O, Not Internals, part 1
Suppose your task is to implement sorting on a list of integers
You remember bubble sort, because, well, that’s easy. You implement it,
together with 100% test coverage, in line with your DoD.
Your test case looks like this:
Given the input of 8, 2, 7, 3, 5
When I run the sorting
Then after the first step, my list will be 2, 8, 7, 3, 5
And after the second step, my list will be 2, 7, 8, 3, 5
And after the third step, my list will be 2, 7, 3, 8, 5
And after the fourth step, my list will be 2, 7, 3, 5, 8
And after the fifth step, my list will be 2, 3, 7, 5, 8
And after the sixth step, my list will be 2, 3, 5, 7, 8
10. Test I/O, Not Internals, part 2
The next day, your PO tells you, that your algorithm is not good enough, when
used on the 300 million records of the production database, results take ages
to compute.
You dust your Algorithms book off, look up QuickSerach, and implement it.
However, you have to write a whole new set of tests, as after one step of
QuickSort, the list will be 5, 2, 7, 3, 8, not 2, 8, 7, 3, 5.
Should you have only tested that
Given the input of 8, 2, 7, 3, 5
When I run the sorting
Then the sorted list is 2, 3, 5, 7, 8
then the test would have kept working, even after changing the underlying
algorithm.
11. Anatomy of Tests
A test should consist of some input, some action(s) to take, and some output.
Internal steps and interim states should not be validated.
12. Coverage
Code coverage tools measure how thoroughly tests exercise
“programs”/“code”.
Statement coverage: record if the line of code was executed
Multiple condition coverage: record if each logical condition in the code was
evaluated for both the true and the false branch
Is there a thing such as “high enough” coverage?
13. “
”
How much of the code should be tested with these automated unit
tests? Do I really need to answer that question? All of it! All. Of. It.
Am I suggesting 100% test coverage? No, I’m not suggesting it. I’m
demanding it. Every single line of code that you write should be
tested. Period.
Isn’t that unrealistic? Of course not. You only write code because you
expect it to get executed. If you expect it to get executed, you
ought to know that it works. The only way to know this is to test it.
[Clean Coder – Robert C. Martin]
Anyways, if you work in test first style TDD, your coverage will automatically be
very close to 100%.
15. The Test Pyramid, Part 2
System/E2E tests: load testing, usability
testing, performance testing,
compatibility testing, stress testing,
security testing, regression testing,
installation testing, etc. Partially
automated, partially manual, each type
covers a specific aspect of the system.
Automated UI tests
Integration tests: difficult to write, might
take long to run. Covers the “plumbing”.
Component/Acceptance tests: easy to
write and run, easy to obtain high
coverage, should cover any part of the
code that provides business value.
Unit tests: easy and fast to write, run,
should cover most of the code. (All of the
code, if using test first TDD.)
Manual
Exploratory
Tests
Automated UI
Tests
System/E2E Tests
Integration Tests
Component/Acceptance Tests
Unit Tests
17. Visibility and Unit Testing
Unit tests should test public methods (or, being more strict, public
interfaces might be an even better choice).
Don’t make a method public just in order to make is testable.
What about internal classes/methods?
Can work, use the InternalsVisibleTo attribute!
Should we have internal classes/methods anyways?
Think of APIs or adding your dll assembly as a reference to another Visual Studio
project, and seeing what’s visible to external developers.
(So yes, we should. Most likely that’s what you should be using instead of public.)
18. Difference between Unit Tests and
Acceptance Tests
Consider your tests as living specification of your application.
Acceptance tests are the business-level specification. Human-readable, with
test cases that are meaningful for the business. Used for feature validation.
Unit tests are the developer-level specification, that check how
functions/methods in the code work. The business does not really need to
understand the technical details, but other developers do. Used for
functionality verification.
23. Fast
You…
… create your test class
… reference your subject, new it up
(mock dependencies)
… create your test method
… make your test method fail
(Assert.False)
… call the method you want to test
(and generate it with the body
throwing a
NotImplementedException)
… run all your unit tests
… create the dumbest
implementation possible that makes
your unit test pass (return the value
that your test expects)
… run all your unit tests
… create another test method with
different input
… run all your unit tests
… implement your business logic
… run all your unit tests
… create test methods for edge cases
… run all your unit tests
… etc.
24. Fast
You probably end up having thousands of tests.
You run all of your tests about 3-5 times for each method you write.
How long are you willing to wait for them to run?
Developers should not hesitate to run the tests because they are slow.
Meaning that tests should run (including setup, the actual run, and teardown)
in a matter of milliseconds each, so the sum running time of all tests should
not be any longer than a couple of seconds.
25. Independent
Test running order should not interfere with the test result.
Some test runners enforce this by running your tests in a random order.
Running the tests one-by-one or in a bundle should not interfere with the
result.
26. Repeatable
Tests should be deterministic: every run of the same test code on the same
subject code should come up with the same result.
Do not depend on external data or environment the test is running in. Setup /
mock whatever is use in the test method.
Do not depend on date/time either!
Each test should arrange or setup its own data.
If some tests need common data, use some sort of data context for them. (Lots of
test frameworks provide those.)
Do not use random input. Use sensible input, but with such values that it is
visible from the first glance that the input is not production data.
You need a name? Use “TestName”.
You need an 8-digit number? Use “12345678”.
27. Repeatable
Do preprocessing in the Setup
Prepare environment if needed (setup mocks, flags, settings, sample data, etc.)
Do post-processing in the TearDown
Cleanup to the original state after the test has run
Remove created files, database entries, etc.
28. Self-Validating
The test should provide a binary output: Passed/Failed, Green/Red.
(Or ignored)
No manual inspection (checking log files, validating anything by hand) should
be required to validate the test results.
29. Timely
Tests should be written together with the code.
Covering existing code with unit tests later is difficult. (Anyone knows PEX?)
And it is rare that you have the time for doing that, anyways.
Testability influences code design. You don’t just write tests to have your code
tested, you write tests to aid you in designing SOLID code.
Unit testing provides a genuine need for implementing and following the SOLID principles.
To put it another way: The harder it is to unit test a piece of code, the greater the
chances that SOLID principles are being violated.
We’ll get back to this in just a moment…
30. +1T: Thorough and Reliable
Don’t just aim for a high coverage (100% should be the baseline anyways in test
first), aim for testing scenarios.
It is possible that there are cases you did not think about when writing your unit test.
Because of this, your might be 100%, but as soon as you create your new scenario for the
test case you did not think about previously, you realize the need for additional
production code, meaning that there was a deficiency in the code under test, in spite of
the high coverage.
If there is a bug in the system under test, at least one unit test should fail.
Creating and running failing unit tests in the TDD cycle serves the purpose of validating
that the test can actually fail. The fact that the test can fail should be validated,
otherwise the test can be a false positive, that passes no matter the bugs in the code.
Tests should be a living documentation of your code.
Whatever your code is doing should be understandable from your test suite.
To achieve this, you also need high coverage.
31. +1E: Easy to Read
Unit tests are living documentation.
This means that:
the tests should
be well-named
have a clear intent
the scenario should
be easy to understand
tell a story of a use-case or a functional aspect
in case of a failure
the cause of the failure should be obvious
the steps to take to fix the failure should be obvious
32. +1E: Easy to Write
As the coverage should be around 100%, there will be lots of tests. Writing
(modifying, maintaining) tests should not be difficult.
SOLID code helps a lot here.
Some libraries also tend to help a lot. Use a mocking library, use an assertion
helper.
Some libraries also have extensions / subparts to help in testing them
(mock/dummy implementations, etc.). Use those as well.
33. Unit Testing and SOLID
How Does Unit Testing Help You Write SOLID Code?
34. Single Responsibility
If your class violates the SRP, you will have massive test classes.
Class Size smell for the test class. You will generally end up 2-8 times the
number of test methods than the methods being tested.
The happy path and at least one unhappy path, can be way more based on the
parameters Say, double:
0.0, 15.0, -15.0,
double.MaxValue, double.MinValue,
double.NaN, double.NegativeInfinity, double.PositiveInfinity.
Some of your tests use some dependencies, other tests don’t use them.
This is abusing dependencies, a violation of the ISP as well.
35. Open/Closed
Injecting code that allows testing opens production code for extensibility.
Showing a MessageBox is difficult to test. However, injecting an Action<string> (or
an INotificationProvider) instead of calling MessageBox.Show(message) opens the
code to extension: notification by email, writing to a database (or using a
messagebox) becomes possible – the same way, a mock implementation that
facilitates testing also becomes possible.
And the class loses a responsibility as well.
Unit tests indirectly close the class for modifications: modifications will break
some of the tests.
36. Liskov Substitution
This is the reason our mocks work: the code under test cannot tell the
difference.
Code under test should not know it is running in “test mode”.
Unit tests should work on derived classes without any modifications.
Otherwise inheritance might not be the best option.
37. Interface Segregation
Mocking is generally done through an interface. If you find yourself needing to
mock a large interface or a complex interaction of methods you should think
about the design of those dependencies and about splitting those interfaces
or interactions into smaller chunks, so that you can mock only the chunk that
concerns your code under test right now.
A badly designed interface will make mocking difficult or will need several
different mocks depending on how it is being used.
38. Dependency Inversion
Some dependencies (database, web service, logger, etc.) may make your unit
test behave like an integration test. This is a sure sign that these
dependencies need to be mocked – thus, injected.
As the mock will generally not be the class itself, but a mock class that
implements the same interface, this also enforces to depend on abstractions
instead of implementations.
40. Not All Production Code Are Born Equal
Unit testing works best on… units.
Units can be separated effectively from their environment
Remember Dependency Inversion and Interface Segregation, aim for Loose Coupling
Dependencies should be made explicit and the new() operator should be avoided
Mocks should be injectable instead of these dependencies
Non-deterministic factors should be mockable (no hidden DateTime.Now or
Environment.MachineName in the middle of the method) with a deterministic mock
Side-effects should be avoided (frequently by passing method delegates in as
dependencies)
Static application state (static properties and static fields) should be avoided
Singletons should be avoided (singletons are hidden application states)
The Command and Query Segregation principle should be applied
41. Isolating Tests
Dependencies of the code under test should
be deterministic
not break the tests
Issue: production code (or the user, for that matter) is rarely deterministic
Solution: test should not depend on the environment
And this environment includes the dependencies of the class under test
Test Doubles allow you to mimic the behavior of classes and interfaces,
letting the code under test interact with them as if they were real. This
isolates the code you’re testing, ensuring that it works on its own and that
no other code will make the tests fail.
43. Test Doubles
Dummy:
Simple placeholder. Can be as simple as a null, a constant, or a method
implementation that throws NotImplementedException.
Fake:
Working implementation that is a simplification of the production version. E.g.
faking a data repository with a non-persistent in-memory collection.
Stub:
An implementation that holds pre-defined data and returns that during the test.
The input is configured from the test set-up or the test arrange.
Mock:
An implementation that records calls it receives. Can be used to validate that calls
to external dependencies happened, without caring about what the call returned.
44. Dummy
Placeholder that is not used during test execution. Can be as simple as a null, a
constant, or a method implementation that throws NotImplementedException.
public class Warehouse : IWarehouse {
public Warehouse(string location) { // … }
public void PlaceOrder(string productId, int amount) {
// …
}
}
public void OrderIsFulfilled_IfThereAreEnoughItemsInStock() {
IWarehouse w = new Warehouse(null);
w.SetStock(…); // …
}
45. Fake
Working implementation that is a simplification of the production version.
public class FakeAccountRepository : IAccountRepository {
IDictionary<,> accounts = new Dictionary<IUser, IAccount> {
{ new User(“john@mail.com”), new UserAccount() },
{ new User(“bob@xmail.com”), new UserAccount() }
}
public bool IsPasswordValid(User user, string pw) {
return pw == accounts[user].GetPasswordHash();
}
}
46. Stub
An implementation that holds pre-defined data.
public class FakeDie : IDie {
public int Roll() {
// chosen by fair dice roll.
// guaranteed to be random.
return 4;
}
}
47. Mock
An implementation that records calls it receives. Used for testing behavior
instead of state.
warehouseMock
.Setup(w => w.PlaceOrder(It.IsAny<string>(), It.IsAny<int>()))
.Verifiable();
warehouseMock.PlaceOrder();
// Placing an order should check the stock
wareHouseMock.Verify(warehouseMock => warehouseMock.CheckStock(),
Times.Once);
49. The AAA
Arrange: Setup the data, mocks, environment. Data setup is part of the test.
Act: Invoke the method that is being tested.
Assert: Check the results.
Fun fact: Mathematically this is a Finite State Machine:
Arrange is the Starting State
Act is a State Transition
Assert is the expected Finishing State
51. AAA Best Practices
Arrange
Don’t have an extensive Arrange block. Having too many things to set up is most
likely a sign of violating the SRP.
Do not Assert in your Arrange phase.
Invalid input data should probably throw an exception.
Consider using Code Contracts.
Assert
Single Concept per Test
Prefer one single assert, or a tightly knit group of asserts, per test method.
Very similar to the single point of return principle.
53. Coding Standards for Unit Tests
Apply the same quality standards (coding standards, reviews, pull requests, CI
builds) that you apply for your production code.
Tests must change as production code changes. The more dirty the tests are,
the more difficult it will be to maintain them.
Old tests will start to fail, and will be difficult to fix.
More time will be spent on fixing failing tests than writing production code.
Tests will be viewed as a liability, and developers will be tempted not to run the
tests.
Test code is just as important as production code.
54. Taking Unit Tests Seriously
All tests should be run on the developer machine before checking in.
All tests should be run after building, both on the developer machine and on
the CI server.
In a CI environment this means that tests should be run on all check-ins.
On all the branches, optimally.
Failing tests should break the build.
Failing a test should be considered an equal offense to committing non-
complying code.
When using gated check-ins or pull requests, failing tests should prevent the
check-in.
55. Ignoring Tests
Avoid ignoring a test that you broke. Refactor, cleanup, fix, don’t ignore.
Don’t ignore if the code needs to be fixed, not the test. You are hiding a bug with
the ignore attribute.
Avoid ignoring if the test needs to be fixed. You most likely will never have the
chance to return to fix the test for the same time concerns you might chose to
ignore it when you broke it.
Should you need to ignore, avoid Ignore attributes without explanation.
57. Specification by Example
Specification is difficult to code
Examples are easy to code
Lets see an… example.
Consider the following user story:
As a Math idiot
I want to be told the sum of two integer numbers
In order to not make silly mistakes
58. Formal Specification vs. Specification by
Example
Acceptance criteria:
I can enter two numbers
The result is the sum of the two
numbers
The result is the same irrespective
of the order I enter the numbers to
sum
Adding zero to a number does not
change its value
Acceptance criteria:
Examples:
|value1|value2|
| 0| 1|
| 1| 2|
| 2| 1|
| -25| 5|
When I add <value1> to <value2>
And I add <value2> to <value1>
Then <result1> equals <result2>
When I add <value1> to 0
Then my result is <value1>
59. Property-Based Testing
Choosing examples wisely is key
Edge cases are often a good choice
Apart from edge cases, the happy path values can be difficult to determine
And the developer is biased to create test cases that pass
However, if you can describe invariant properties of your method, you can use
these properties to create a good set of happy-path tests
60. Property-Based Testing, part 2
Some examples of properties:
Adding an item to the shopping cart increases the count of items in the shopping
cart
Adding an integer value to zero results in the input integer value
Multiplying an integer value by zero results in a zero value
Sorting a list of integers results in the same amount of integers in a list
Test for these properties. Either use a set of values that are easy to verify or
use (lots of) random values*
*: Didn’t we explicitly say that we should not test with random values? Well, yes, property-based testing might be an
exception to this rule – but it is not necessarily one. You can generate your input-values manually as well.
61. Fluent Assertions
Which one is correct?
Assert.AreEqual(actual, expected)
Assert.AreEqual(expected, actual)
Using multiple asserts for asserting one concept looks contrary to the single
assert per test principle
Enter Fluent Assertions
Compare the following assertions pairs and decide which ones are easier to
read
65. Organizing Unit Tests, Part 1
Where to place unit test projects?
Generally speaking, they should be rather close to the project being tested.
Certainly in the same solution.
But test assemblies should not be deployed with the production code!
Generally you will want to use the “ProjectName.UnitTests” folder and
namespace convention.
There are religions that promote using the .UnitTests for the folder, but not for the
namespace. Experiment, whether you like it or not. (Downside is that Intellisense
will display both when writing code in the test class.)
66. Organizing Unit Tests, Part 2
Mirror the folder/file structure (and of course namespace structure) of the
production code as closely as possible.
Place test-specific classes (helpers, custom assertions, etc.) into a separate
folder (TestUtilities).
Consider moving these to a shared dll that all your test projects can reference to
avoid code duplication.
67. Organizing Unit Tests, Part 3
For each class under test create a test fixture class, name it
“[ProductionClassName]Tests”. (e.g. PersonTests)
You will most likely want to make this class partial for size-related reasons.
This will contain your “subject” or “unitUnderTest”, that is, the instance of the
class you are testing.
For each method under test create a nested test fixture class that derives
from the containing class ([ProductionClassName]Tests). Name these after
the method that they test. (e.g. IsAllowedToDrink)
68. Organizing Unit Tests, Part 4
Create test methods inside the [MethodName] classes.
Naming conventions: IsAdult.[…]
StateUnderTest_ExpectedBehavior:
IfAgeIsLessThan18_IsFalse
ExpectedBehavior_StateUnderTest:
IsFalse_ForAgeLessThan18
Should_ExpectedBehavior_When_StateUnderTest:
Should_BeFalse_When_AgeIsLessThan18
When_StateUnderTest_Expect_ExpectedBehavior:
When_AgeIsLessThan18_Expect_ToBeFalse
Use a “Cannot”/”DoesNot” syntax when expecting exceptions
CannotAccept_NullArguments / DoesNotAccept_NullArguments
69. Summary
Tests are not just validation and verification
They are specification, documentation and design aids.
They save you a lot of engineering effort and engineering hours (thus money)
SOLID code is easy to test, easy to test code is most likely clean
Loose coupling, CQS, DI are keys in testability
Tests should be FIRST+TR+E2R+E2W
Tests are input, action, and output validation
AAA is a Finite State Machine
Specification by Example conveys acceptance criteria clearly
Tests are First Class Citizens in respect of standards and maintenance
Libraries can make testing easy (SpecFlow, Fluent Assertions, FsCheck)