I usually attend to several talks about testing. Everyone tells us how good, how important and valuable could be having a good test suite. But almost anybody tells us how hard is achieving this goal. With this talk, I wanted to provide an overview for people who is starting with software testing. Even for those who never tried it.
But this is not only an introduction. Though I start talking about the basics and principles and concepts, I also try to talk about my experience. What worked, where I failed and how I tried to solve the problems I found.
16. Feature Small Medium Large
Network access No localhost only Yes
Database No Yes Yes
File system access No Yes Yes
Use external systems No Discouraged Yes
Multiple threads No Yes Yes
Sleep statements No Yes Yes
System properties No Yes Yes
Time limit (seconds) 60 300 900+
21. DEPENDENCY INJECTION
➤ Separate business logic from creation logic
➤Avoid use of new for service objects.
➤Value objects can be created any where.
➤ Service objects in charge to implement business
logic.
➤ IOC Container or factories in charge of creation
logic.
22. DEPENDENCY INJECTION
public UserService(UserValidator userValidator, UserDao userDao) {
this.userValidator = userValidator;
this.userDao = userDao;
}
public User createUser(User user) throws ValidationException {
this.userValidator.validate(user);
user = this.userDao.create(user);
return user;
}
public User createUser(User user) throws ValidationException {
UserValidator userValidator = new UserValidator(...);
userValidator.validate(user);
UserDao userDao = new UserDao(...);
user = userDao.create(user);
return user;
}
VS
23. DEPENDENCY INJECTION
public UserService(UserValidator userValidator, UserDao userDao) {
this.userValidator = userValidator;
this.userDao = userDao;
}
public User createUser(User user) throws ValidationException {
this.userValidator.validate(user);
user = this.userDao.create(user);
return user;
}
public User createUser(User user) throws ValidationException {
UserValidator userValidator = new UserValidator(...);
userValidator.validate(user);
UserDao userDao = new UserDao(...);
user = userDao.create(user);
return user;
}
VS
this sucks
25. TEST DOUBLES (FAKE)
Fake implementation in order to make test pass.
public UserDaoFake implements UserDao {
public User create(User user) {
return ...;
}
}
26. TEST DOUBLES (STUB)
Stubs provide canned answers to calls made during the test,
usually not responding at all to anything outside what’s
programmed in for the test.
UserValidator validatorMock = mock(UserValidator.class);
stub(validatorMock.validate(any(User.class)))
.toThrow(new ValidationException());
var validateCall = Sinon.stub();
validatorStub.withArgs(user)
.onFirstCall().returns(validationError);
var userValidator = {
validate: validatorStub;
}
OR WITH JS
27. TEST DOUBLES (SPY)
Spies are objects that also record some information based on how
they were called
var validatorSpy = Sinon.spy();
var userValidator = {
validate: validatorSpy;
}
userValidator.validate(user);
sinon.assert.calledOnce(validatorSpy);
sinon.assert.calledWith(validatorSpy, user);
OR WITH JS
UserValidator validatorSpy = spy(new UserValidator());
doThrow(new ValidationException())
.when(validatorSpy).validate();
28. TEST DOUBLES (SPY)
Spies are objects that also record some information based on how
they were called
var validatorSpy = Sinon.spy();
var userValidator = {
validate: validatorSpy;
}
userValidator.validate(user);
sinon.assert.calledOnce(validatorSpy);
sinon.assert.calledWith(validatorSpy, user);
OR WITH JS
UserValidator validatorSpy = spy(new UserValidator());
doThrow(new ValidationException())
.when(validatorSpy).validate();
29. TEST DOUBLES (MOCKS)
Informal: think in a Stub which is also a Spy.
It also responds with default values to non-explicitly declared
methods
UserValidator validatorMock = mock(UserValidator.class);
when(validatorMock.validate(any(User.class)))
.thenTrhow(new ValidationException());
verify(validatorMock).validate(any(User.class))
var validatorAPI = {validate: function()};
var validatorMock = Sinon.mock(validatorAPI);
validatorMock.expects('validate').once()
.withArgs(user).throws(validationError)
validatorAPI.validate(user)
validatorMock.verify()
OR WITH JS
31. FIRST(IT)
➤ Fast
➤Hundreds or thousands per second
➤ Isolates
➤Failure reasons become obvious
➤ Repeatable
➤In any order, any time
➤ Self-validating
➤No manual execution required
➤ Timely
➤Written before code
➤ Immutable*
➤SUT is in the same state after execute the tests
➤ Trusted*
➤When the test fails, the system fail and when the test works, the system works
32. INTEGRATION TEST WHICH WORKS WITH EXTERNAL SYSTEM
Database
Fast
Isolates
Repeatable
Self-Validating
Timely
Inmutable*
Trusted*
33. INTEGRATION TEST WHICH WORKS WITH EXTERNAL SYSTEM
Database
Fast
Isolates
Repeatable
Self-Validating
Timely
Inmutable*
Trusted*
34. INTEGRATION TEST WHICH USES THE UI
Database
Fast
Isolates
Repeatable
Self-Validating
Timely
Inmutable*
Trusted*
35. INTEGRATION TEST WHICH USES THE UI
Database
Fast
Isolates
Repeatable
Self-Validating
Timely
Inmutable*
Trusted*
38. WHO, WHEN AND WHERE RUN THE TESTS?
➤ Unit
➤ Owner: developer
➤ When: after every change
➤ Where: every computer
➤ Integration
➤ Owner: developer || QA team
➤ When: as part or after commit stage
➤ Where: devel and pre-pro environments
➤ System
➤ Owner: QA team
➤ When: after commit stage
➤ Where: devel and pre-pro environments
42. WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another
4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
43. WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another
4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
a
b c
d
…x
44. WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another
4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
a
b c
d
…x
edges – nodes + 2 = predicate nodes +1 = number of regions = 4
45. BLACK BOX (PARTITIONING)
1. Identify equivalence classes
2. Select dataset:
1. Assign a unique value for every class
2. Select tests cases which cover the most
valid classes
3. Select tests cases which cover only one
invalid class at the same time
52. XUNIT
@Before
public void setUp() {
this.userValidator = mock(UserValidator.class);
this.userDao = mock(UserDao.class);
this.userService = new UserService(userValidator, userDao);
}
@Test
public void createValidUserShouldNotFail() {
//Exercise
User expectedCreatedUser = new User("irrelevantUser");
when(userValidator.validate(any(User.class)));
when(userValidator.validate(any(User.class))).thenReturn(createdUser);
User createdUser = userService.create(new User());
//Assertions
assertThat(createdUser, equalTo(expectedCreatedUser));
}
@Test(expected=ValidationException)
public void createInvalidUserShouldFail() {
when(userValidator.validate(any(User.class)))
.thenReturn(new ValidationException());
userService.create(new User("irrelevantUser"));
}
@After
public void tearDown() {
//clean the state here
}
53. RSPEC (SUITE PER CLASS)
describe('UserService test suite:', function(){
beforeEach(function(){
// setup the SUT
})
it('when create a valid user should not fail', function(){
// exercise + assertions
})
it('when create an invalid user should fail', function(){
// exercise + assertions
})
afterEach(function(){
// clean the state
})
})
• UserService test suite:
• When create a valid user should not fail √
• When create an invalid user should fail √
The report will look like:
54. RSPEC (SUITE PER SETUP)
describe('UserService test suite:', function(){
describe("when create a valid user ", function() {
beforeEach(function(){
// setup and exercise
})
it('should return valid user', function(){
// partial assertions
})
it('should call validator', function(){
// partial assertions
})
it('should call dao', function(){
// partial assertions
})
afterEach(function(){
// clean the state
})
})
})
56. BDD(IMPLEMENTATION)
@given("the user has introduced (w)+ and (w)+ into the
registration form")
public void populateForm(String username, String password) {
...
}
@given("has accepted terms and agreements")
public void acceptTerms() {
...
}
@when("send the registration from")
public void sendRegistrationForm() {
...
}
@then("the user with (w)+ should be created")
public void verifyUserIsCreated(String username) {
...
}
@then("the system should notify <error>")
public void verifyErrors(String error) {
...
}
58. NON-TESTABLE DESIGN SMELLS (BY MISKO HEVERY*)
➤Constructor does Real Work
➤Mixing business and creation logic
➤Class does too much work
➤Digging into collaborations
➤Brittle Global State & Singletons
*See hAp://misko.hevery.com/aAachments/Guide-WriIng Testable Code.pdf
59. CONSTRUCTOR DOES REAL WORK
➤new keyword in a constructor or at field declaration
➤static method calls in a constructor or at field declaration
➤Anything more than field assignment in constructors
➤Object not fully initialized after the constructor finishes
(watch out for initialize methods)
➤Control flow (conditional or looping logic) in a constructor
SOLID
60. MIXING BUSINESS AND CREATION LOGIC
➤Use of new inside class methods for a non value object
➤Use of getInstance
SOLID
61. CLASS DOES TOO MUCH WORK
➤Summing up what the class does includes the word “and”.
➤Class would be challenging for new team members to read an
quickly “get it”.
➤Class has fields that are only used in some methods.
➤Class has static methods that only operate on parameters.
SOLID
62. DIGGING INTO COLLABORATORS
➤Objects are passed in but never used directly only used to get
access to other objects)
➤Law of Demeter violation: method call chain walks an object
graph with more than one dot (.)
➤Suspicious names: context, environment, principal, container
or manager
SOLID
63. BRITTLE GLOBAL STATE & SINGLETONS
➤Adding or using singletons.
➤Adding or using static fields or static methods.
➤Adding or using static initialisation blocks.
➤Adding or using registries.
➤Adding or using service locators.
SOLID
65. QUESTIONS & STUPID QUESTIONS
➤Where I place my tests?
➤Who tests the classes which test our classes?
➤Could you be able to rewrite the code only reading the tests
definitions?
➤I spend more time writing code to setup my SUT than writing the test,
how do you solve it?
➤What is the minimum coverage should I expect for my code?
➤I’ve never write a test, where can I start?
➤My code is not testable at all, what can I do?
➤Should I verify every single interaction with my mocks?
➤How do I run my tests?
66. WHERE I PLACE MY TESTS?
➤Unit tests:
➤Test Class per Class
➤Test Class per SetUp (useful in Xunit frameworks)
➤Important naming convention (<ClassName>Test,
<TestSuite>Integra*onTest, …)
➤System tests:
➤Different project
67. WHERE I PLACE MY TESTS?
Java Project (Test Class per Class)
MyProject/
src/
main/
java/
com.groupId.arIfactId.MyClass.java
resources/
test/
java/
com.groupId.arIfactId.MyClassTest.java
com.groupId.arIfactId.it.suite.MyTestCaseIntegraIonTest.java
resources/
NodeJs Project
MyProject/
lib/
myClass.js
main.js
test/
ut/
/suite
it/
lib/
myClassTest.js
Java Project (Class per SetUp)
MyProject/
src/
main/
…
test/
java/
com.groupId.arIfactId.myclass.<SetUp1>Test.java
com.groupId.arIfactId.myclass.<SetUp2>Test.java
…
68. WHERE I PLACE MY TESTS?
IOS Project
MyIOSProject/
MyIOSProject/
... app code ...
MyIOSProjectTests/
... test code ...
Android Project
MyProject/
AndroidManifest.xml
res/
... (resources for main applicaIon)
src/
main/
… (source code for main applicaIon) …
test/
… (unit tests) …
androidTest/
… (android specific tests / UI, etc) …
tests/
AndroidManifest.xml
res/
… (resources for tests)
src/
... (source code for tests)
70. ¿COULDYOUBEABLETOREWRITETHECODEONLYREADINGTHETESTSDEFINITIONS?
➤Tests (specially Black Box tests) should tell us an story.
➤Use well defined, and complete scenarios for system tests:
➤Use business vocabulary for acceptance tests:
public void testValidaterUser1 { ... }
VS
public void validateUserWithNoPasswordShouldThrowsError { ... }
com.mycompany.artifactId.it.TestSteps ...
VS
com.mycompany.artifactId.it.usermanagement.UserCreationSteps ...
72. IDUPLICATETOOMUCHCODEONOBJECTSCREATION,MOCKSDEFINITIONANDASSERTION…
➤Writing a lot of code to initialize value objects?
➤Create DataBuilders / MotherObjects
➤Writing a lot of code to initialize mock/stub objects?
➤Create Mock“Mother”
➤Writing a lot of asserts (TDD purist says only one asser<on)?
➤Create CustomAsserts
User user = new User("irrelevantUsername", "v4l1dP4ss",
irrelevant@myCompany.com", ...);
VS
User user = userMother.createValidUser();
assertNotNull(user.getUsername());
assertNotNull(user.getPassword());
assertNotNull(user.getEmail());
...
VS
assertContainsAllMandatoryData(user);
75. MYCODEISNOTTESTABLEATALL,WHATCANIDO?
➤First of all, go to http://refactoring.com/
➤I suggest:
1. Add integration regression test.
2. Remove new from methods and ad it to constructors (this will prepare
your class for dependency injection).
3. Creates a constructor which receive every dependency your class will need.
4. Remove static classes and methods (adding the new non-static as a class
dependency).
5. Add as much tests as you want to ;)
➤Important!!! Do it step by step
76. REMOVES NEW FOR DEPENDENCIES
public UserService() {
this(new UserValidator(), new UserDao());
}
public UserService(UserValidator userValidator, UserDao userDao) {
this.userValidator = userValidator;
this.userDao = userDao;
}
public User createUser(User user) throws ValidationException {
this.userValidator.validate(user);
user = this.userDao.create(user);
return user;
}
public User createUser(User user) throws ValidationException {
UserValidator userValidator = new UserValidator(...);
userValidator.validate(user);
UserDao userDao = new UserDao(...);
user = userDao.create(user);
return user;
}
TO
77. REMOVE STATIC CLASSES AND METHODS
class MyJsonSerializer {
public static String serialize(Object myObject) {
//put your magic here
}
}
class UserClient {
public void createUser(User user) {
Http.post(url, JsonSerizalizer.serialize(user));
}
}
78. REMOVE STATIC CLASSES AND METHODS
class MysonSerializer {
public String serialize(Object myObject) {
//put your magic here
}
}
class HttpClientWrapper {
public void post(String url, String body) {
Http.post(url, body);
}
}
class UserClient {
private JsonSerializer serializer;
private HttpClientWrapper httpClient
public UserClient() {
serializer = new JsonSerializer();
httpClient = new HttpClientWrapper();
}
public void createUser(User user) {
httpClient.post(url, serializer.serialize(user));
}
}
79. SHOULD I VERIFY EVERY SINGLE INTERACTION WITH MY MOCKS?
It depends
80. HOW DO I RUN MY TESTS?
➤ Java / Android
➤$ mvn test
➤$ gradle test
➤ Ruby
➤$ rake test
➤PHP
➤$ phpunit <testFolder/File>
➤ NodeJs
➤$ npm test
➤$ grunt test
➤ Javascript (client)
➤$ karma star
81. RECOMMENDED READING
➤Growing Object Oriented Software Guided Tests
➤Xunit Patterns
➤Continuous delivery
➤Hexagonal Architecture
➤Inversion of Control Container and the Dependency Injection
pattern
➤Mocks aren’t Stubs
➤Misko Hevery blog
➤http://refactoring.com/
➤…