IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
Testing an Erlang Backend
1. Introduction
Why should I test?
Types of tests
Good Practices
Testing an Erlang Backend
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Enrique Paz
Making it
happen Senior Backend Developer @ Team Services
TDD & BDD
Automating
everything
Q&A EUG-NL / 12-07-2012
1/40
2. Introduction
Why should I test?
Types of tests
Introduction
Good Practices
Testing
Techniques
Unit tests
Testing Techniques
Integration tests
Acceptance Tests
Performance Tests
Making it Making it happen
happen
TDD & BDD
Automating
everything
Q&A
Q&A
2/40
3. Reasons for testing
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests • It increases your confidence in the code you write
Integration tests
Acceptance Tests
Performance Tests
• Tests themselves are useful documentation
Making it
happen
• It makes refactoring really an option
TDD & BDD
Automating
• It helps finding bugs earlier, so the impact is much lower
everything
Q&A
3/40
4. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
4/40
5. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
4/40
6. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
4/40
7. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
4/40
8. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen
TDD & BDD
Automating
everything
Q&A
4/40
9. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen Have you consider refactoring?
TDD & BDD
Automating
everything
Q&A
4/40
10. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen Have you consider refactoring?
TDD & BDD
Automating • “Setting up the scenario for testing my code is too complex”
everything
Q&A
4/40
11. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen Have you consider refactoring?
TDD & BDD
Automating • “Setting up the scenario for testing my code is too complex”
everything
Q&A
So. . . you do it manually everytime?
4/40
12. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen Have you consider refactoring?
TDD & BDD
Automating • “Setting up the scenario for testing my code is too complex”
everything
Q&A
So. . . you do it manually everytime?
Automating that scenario for testing will be expensive. . . but only once!
4/40
13. Yeah, but...
Introduction
Why should I test?
Types of tests
Good Practices
• “My code is so clear/simple it doesn’t need tests”
Testing simplicity and clarity are not universal words
Techniques
Unit tests
The simpler your code is, the easier for you to test it
Integration tests
Acceptance Tests
• “It takes me much longer testing the code than writing it”
Performance Tests
Was your original code working or you spent the time fixing it?
Making it
happen Have you consider refactoring?
TDD & BDD
Automating • “Setting up the scenario for testing my code is too complex”
everything
Q&A
So. . . you do it manually everytime?
Automating that scenario for testing will be expensive. . . but only once!
• In short, stop arguing and start testing!
4/40
14. Types of tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• Unit tests What you have written is exactly what you were trying to write
Integration tests
Acceptance Tests • Integration tests The API of your component can interact with other APIs
Performance Tests
Making it
• Acceptance tests The API you have implemented fits the bussiness
happen
TDD & BDD
requirements
Automating
everything • Performance tests Your solution is suitable for the expected scenario
Q&A
5/40
15. General Good Practices
Introduction
Why should I test?
Types of tests
Good Practices
Testing • Tests should be repeatable (setup and cleanup)
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
6/40
16. General Good Practices
Introduction
Why should I test?
Types of tests
Good Practices
Testing • Tests should be repeatable (setup and cleanup)
Techniques
Unit tests • Tests should be independent from each other (use sequences when
Integration tests
Acceptance Tests necesary)
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
6/40
17. General Good Practices
Introduction
Why should I test?
Types of tests
Good Practices
Testing • Tests should be repeatable (setup and cleanup)
Techniques
Unit tests • Tests should be independent from each other (use sequences when
Integration tests
Acceptance Tests necesary)
Performance Tests
Making it • Tests should be isolated from each other (leave the environment as you
happen
TDD & BDD
found it)
Automating
everything
Q&A
6/40
18. General Good Practices
Introduction
Why should I test?
Types of tests
Good Practices
Testing • Tests should be repeatable (setup and cleanup)
Techniques
Unit tests • Tests should be independent from each other (use sequences when
Integration tests
Acceptance Tests necesary)
Performance Tests
Making it • Tests should be isolated from each other (leave the environment as you
happen
TDD & BDD
found it)
Automating
everything • Tests should test specs and not depend on the implementation details
Q&A
6/40
19. General Good Practices
Introduction
Why should I test?
Types of tests
Good Practices
Testing • Tests should be repeatable (setup and cleanup)
Techniques
Unit tests • Tests should be independent from each other (use sequences when
Integration tests
Acceptance Tests necesary)
Performance Tests
Making it • Tests should be isolated from each other (leave the environment as you
happen
TDD & BDD
found it)
Automating
everything • Tests should test specs and not depend on the implementation details
Q&A
• Separate the tests from the code. If not possible, refactor!
6/40
20. Introduction
Why should I test?
Types of tests
Introduction
Good Practices
Testing
Techniques
Unit tests
Testing Techniques
Integration tests
Acceptance Tests
Performance Tests
Making it Making it happen
happen
TDD & BDD
Automating
everything
Q&A
Q&A
7/40
21. Main tools
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
8/40
22. Main tools
Introduction
Why should I test?
Types of tests
Good Practices
• Eunit
Testing
Techniques The classic *unit approach
Unit tests
Integration tests
Starting to use it is very easy
Acceptance Tests
Performance Tests
Out of the box integration with rebar
Making it
But. . . Might be annoying to debug it
happen
TDD & BDD
Automating
everything
Q&A
8/40
23. Main tools
Introduction
Why should I test?
Types of tests
Good Practices
• Eunit
Testing
Techniques The classic *unit approach
Unit tests
Integration tests
Starting to use it is very easy
Acceptance Tests
Performance Tests
Out of the box integration with rebar
Making it
But. . . Might be annoying to debug it
happen
TDD & BDD
• Common Test
Automating
everything Much more powerful
Q&A Setting it up can be painful
It shines in complex scenarios, so usually is better for integration tests
8/40
24. A different approach: property testing
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests Finding good testcases might be hard:
Integration tests
Acceptance Tests
Performance Tests
• There might be lots of them
Making it • They might be just hard to predict
happen
TDD & BDD
Automating
• They might be expensive / hard to generate
everything
Q&A
9/40
25. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
10/40
26. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
• You fed them with properties your code should satisfy
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
10/40
27. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
• You fed them with properties your code should satisfy
Integration tests
Acceptance Tests • You specify how the input should look like
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
10/40
28. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
• You fed them with properties your code should satisfy
Integration tests
Acceptance Tests • You specify how the input should look like
Performance Tests
Making it • They smartly generate input and try to verify whether the properties are
happen
TDD & BDD
satisfied
Automating
everything
Q&A
10/40
29. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
• You fed them with properties your code should satisfy
Integration tests
Acceptance Tests • You specify how the input should look like
Performance Tests
Making it • They smartly generate input and try to verify whether the properties are
happen
TDD & BDD
satisfied
Automating
everything • If they find a counterexample, they simplify it for you!
Q&A
10/40
30. Property testing tools
Introduction
Why should I test?
Types of tests
Good Practices
PropEr / Quickcheck Let’s use some math!
Testing
Techniques
Unit tests
• You fed them with properties your code should satisfy
Integration tests
Acceptance Tests • You specify how the input should look like
Performance Tests
Making it • They smartly generate input and try to verify whether the properties are
happen
TDD & BDD
satisfied
Automating
everything • If they find a counterexample, they simplify it for you!
Q&A
• They can also verify the transitions in your gen_servers or gen_fsms
10/40
31. Unit Tests Examples
To be tested
Introduction
Why should I test?
−spec re mo ve _du pl ic at es ( [ i n t e g e r ( ) ] ) −> [ i n t e g e r ( ) ] .
Types of tests
Good Practices
r e mo ve _d upl ic at es ( [ ] ) −>
Testing [];
Techniques r e mo ve _d upl ic at es ( L ) when i s _ l i s t ( L ) −>
Unit tests
l i s t s : usort (L) .
Integration tests
Acceptance Tests r e m o v e _ d u p l i c a t e s _ f r o m _ l i s t s _ f i l e ( F i l e ) −>
Performance Tests case l o a d _ l i s t s _ f r o m _ f i l e ( F i l e ) o f
{ e r r o r , Reason } −>
Making it { e r r o r , Reason } ;
happen { ok , L i s t s } −>
TDD & BDD { ok , [ rem ov e_ du pli ca te s ( L ) | | L <− L i s t s ] }
Automating end .
everything
Q&A % Internal functions
%
l o a d _ l i s t s _ f r o m _ f i l e ( F i l e ) −>
case f i l e : c o n s u l t ( F i l e ) o f
{ e r r o r , Reason } −>
{ e r r o r , Reason } ;
{ ok , Terms } −>
{ ok , [ L | | L <− Terms , i s _ l i s t ( L ) ] }
end .
11/40
32. Unit Tests Examples
Eunit
Introduction
Why should I test?
s i m p l e _ c a s e s _ t e s t _ ( ) −>
Types of tests
[ ? _ a s s e r t E q u a l ( [ ] , t s _ l i s t s : re mov e_ du pl ica te s ( [ ] ) ) ,
Good Practices
% Bad t e s t c a s e ( dependent / f o r c i n g on t h e i m p l e m e n t a t i o n ! ! )
Testing ? _ a s s e r t E q u a l ( [ 1 , 2 , 3 , 4 , 5 ] , t s _ l i s t s : re mo ve_ du pl ic at es ( [ 1 , 2 , 2 , 3 , 4 , 4 , 5 ] ) ) ] .
Techniques l o a d _ l i s t s _ f r o m _ f i l e _ t e s t _ ( ) −>
Unit tests
{ setup ,
Integration tests % SETUP c r e a t e s a v a l i d f i l e w i t h 2 l i s t s and ensures I n v a l i d F i l e does n o t e x i s t
Acceptance Tests f u n ( ) −> s e t u p _ f i l e s ( ) end ,
Performance Tests f u n ( { V a l i d F i l e , _ I n v a l i d F i l e } ) −> ok= f i l e : d e l e t e ( V a l i d F i l e ) end ,
f u n ( { V a l i d F i l e , I n v a l i d F i l e } ) −>
Making it [ ? _ a s s e r t E q u a l ( { ok , [ [ 1 , 2 , −2, −2, −1] , [ −2, 2 , 2 , 1 ] ] } , t s _ l i s t s : l o a d _ l i s t s _ f r o m _ f i l e (
happen ValidFile ) ) ,
TDD & BDD ? _ a s s e r t E q u a l ( { e r r o r , enoent } , t s _ l i s t s : l o a d _ l i s t s _ f r o m _ f i l e ( I n v a l i d F i l e ) ) ]
Automating end
everything
}.
Q&A r e m o v e _ d u p l i c a t e s _ f r o m _ l i s t s _ f i l e _ t e s t _ ( ) −>
{ setup ,
f u n ( ) −> s e t u p _ f i l e s ( ) end ,
f u n ( { V a l i d F i l e , _ I n v a l i d F i l e } ) −> ok= f i l e : d e l e t e ( V a l i d F i l e ) end ,
f u n ( { V a l i d F i l e , I n v a l i d F i l e } ) −>
[ ? _ a s s e r t E q u a l ( { e r r o r , enoent } , t s _ l i s t s : r e m o v e _ d u p l i c a t e s _ f r o m _ l i s t s _ f i l e ( I n v a l i d F i l e ) ) ,
? _assertMatch ( { ok , [ L1 , L2 ] } when i s _ l i s t ( L1 ) and i s _ l i s t ( L2 ) , t s _ l i s t s :
remove_duplicates_from_lists_file ( ValidFile ) ) ]
end } .
12/40
33. Unit Tests Examples
Proper
Introduction
Why should I test?
p r o p _ r e m o v e _ d u p l i c a t e s _ k e e p s _ a l l _ e l e m e n t s ( ) −>
Types of tests
numtests (1000 , ?FORALL( L , n e _ l i s t ( i n t e g e r ( ) ) ,
Good Practices
begin
Testing Processed= t s _ l i s t s : re mo ve _d upl ic at es ( L ) ,
Techniques ?WHENFAIL(
Unit tests
i o : f o r m a t ( " L=~p , Processed=~p~n " , [ L , Processed ] ) ,
Integration tests l i s t s : a l l ( f u n (N) −> l i s t s : member (N, Processed ) end , L )
Acceptance Tests )
Performance Tests end
)).
Making it p r o p _ r e m o v e _ d u p l i c a t e s _ c o n t a i n s _ n o _ d u p l i c a t e s ( ) −>
happen numtests (1000 , ?FORALL( O r i g i n a l , n e _ l i s t ( i n t e g e r ( ) ) ,
TDD & BDD begin
Automating L= t s _ l i s t s : r em ov e_ du pli ca te s ( O r i g i n a l ) ,
everything
Seq= l i s t s : seq ( 1 , l e n g t h ( L ) ) ,
Q&A ?WHENFAIL( i o : f o r m a t ( " O r i g i n a l =~p , Processed=~p~n " , [ O r i g i n a l , L ] ) ,
l i s t s : a l l ( f u n (N) −>
Elem= l i s t s : n t h (N, L ) ,
{ _Before , [ Elem | A f t e r ] } = l i s t s : s p l i t w i t h ( f u n (X) −> X/ = Elem end , L ) ,
n o t l i s t s : member ( Elem , A f t e r )
end , Seq )
)
end
)).
13/40
34. Integration Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Common Test A flexible tool for complex scenarios
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
14/40
35. Integration Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Common Test A flexible tool for complex scenarios
Techniques
Unit tests • per_suite and per_testcase setup and teardown
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
14/40
36. Integration Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Common Test A flexible tool for complex scenarios
Techniques
Unit tests • per_suite and per_testcase setup and teardown
Integration tests
Acceptance Tests
Performance Tests
• Easy to automatically avoid certain testcases under certain conditions
Making it
({skip, Reason})
happen
TDD & BDD
Automating
everything
Q&A
14/40
37. Integration Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Common Test A flexible tool for complex scenarios
Techniques
Unit tests • per_suite and per_testcase setup and teardown
Integration tests
Acceptance Tests
Performance Tests
• Easy to automatically avoid certain testcases under certain conditions
Making it
({skip, Reason})
happen
TDD & BDD • Also possible to run the test suites against a configurable set of running
Automating
everything nodes
Q&A
14/40
38. Integration Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Common Test A flexible tool for complex scenarios
Techniques
Unit tests • per_suite and per_testcase setup and teardown
Integration tests
Acceptance Tests
Performance Tests
• Easy to automatically avoid certain testcases under certain conditions
Making it
({skip, Reason})
happen
TDD & BDD • Also possible to run the test suites against a configurable set of running
Automating
everything nodes
Q&A
• ...
14/40
39. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
15/40
40. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
1. We use the SUITE setup to prepare the pair of projects we want to
Testing
Techniques integrate
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
15/40
41. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
1. We use the SUITE setup to prepare the pair of projects we want to
Testing
Techniques integrate
Unit tests
Integration tests
Acceptance Tests
2. Still in the SUITE setup, we use ct_slave to start the VMs for those projects
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
15/40
42. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
1. We use the SUITE setup to prepare the pair of projects we want to
Testing
Techniques integrate
Unit tests
Integration tests
Acceptance Tests
2. Still in the SUITE setup, we use ct_slave to start the VMs for those projects
Performance Tests
3. We use per testcase setup for creating specific conditions in each node
Making it
happen via RPC
TDD & BDD
Automating
everything
Q&A
15/40
43. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
1. We use the SUITE setup to prepare the pair of projects we want to
Testing
Techniques integrate
Unit tests
Integration tests
Acceptance Tests
2. Still in the SUITE setup, we use ct_slave to start the VMs for those projects
Performance Tests
3. We use per testcase setup for creating specific conditions in each node
Making it
happen via RPC
TDD & BDD
Automating
everything 4. Being A and B the nodes we want to integrate, testcases execute
Q&A something in A that requires it to interact with B
15/40
44. Common Test
The way we use it
Introduction
Why should I test?
Types of tests
Good Practices
1. We use the SUITE setup to prepare the pair of projects we want to
Testing
Techniques integrate
Unit tests
Integration tests
Acceptance Tests
2. Still in the SUITE setup, we use ct_slave to start the VMs for those projects
Performance Tests
3. We use per testcase setup for creating specific conditions in each node
Making it
happen via RPC
TDD & BDD
Automating
everything 4. Being A and B the nodes we want to integrate, testcases execute
Q&A something in A that requires it to interact with B
5. Testcases assert the obtained response matches the expected one given
the testcase preparation
15/40
45. General Considerations
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
16/40
46. General Considerations
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• These tests should check the whole stack, top to bottom
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
16/40
47. General Considerations
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• These tests should check the whole stack, top to bottom
Integration tests
Acceptance Tests • People without technical knowledge about the project should be able to
Performance Tests
write them
Making it
happen
TDD & BDD
Automating
everything
Q&A
16/40
48. General Considerations
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• These tests should check the whole stack, top to bottom
Integration tests
Acceptance Tests • People without technical knowledge about the project should be able to
Performance Tests
write them
Making it
happen
TDD & BDD
• They can be written in any language
Automating
everything
Q&A
16/40
49. General Considerations
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• These tests should check the whole stack, top to bottom
Integration tests
Acceptance Tests • People without technical knowledge about the project should be able to
Performance Tests
write them
Making it
happen
TDD & BDD
• They can be written in any language
Automating
everything • They don’t look for bugs, they validate the requirements
Q&A
16/40
50. What we use at Team Services
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests • Since we are developing an HTTP stack, we use JMeter for validation
Integration tests
Acceptance Tests • We could use any of the already described erlang tools, but. . .
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
17/40
51. What we use at Team Services
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests • Since we are developing an HTTP stack, we use JMeter for validation
Integration tests
Acceptance Tests • We could use any of the already described erlang tools, but. . .
Performance Tests
Making it
We want QA people from other teams to be able to black-box test our code
happen
TDD & BDD
Automating
everything
Q&A
17/40
52. What we use at Team Services
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests • Since we are developing an HTTP stack, we use JMeter for validation
Integration tests
Acceptance Tests • We could use any of the already described erlang tools, but. . .
Performance Tests
Making it
We want QA people from other teams to be able to black-box test our code
happen We want to avoid corrupting the acceptance tests with our knowledge of the
TDD & BDD
Automating system internals
everything
Q&A
17/40
53. What we use at Team Services
JMeter
Introduction
Why should I test?
Types of tests • Simple assertion-based mechanism
Good Practices
Testing
• Nice management interface
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
18/40
54. Some other tools used inside Spil Games
Cucumber
Introduction
Why should I test?
Types of tests
Good Practices
• Scenario descriptions in Gherkin (close to natural language)
Testing
Techniques • Multi-language support (Ruby, Php, Python. . . )
Unit tests
Integration tests # language : en
Acceptance Tests Feature : Payment Method Count
Performance Tests As a user , I want t o see a c e r t a i n number o f payment methods
f o r my c o u n t r y .
Making it
happen
Scenario O u t l i n e : Payment Method Choice
TDD & BDD
Given I am a user i n " <Country > "
Automating
everything And I open t h e PSS
Then I should see " <PaymentMethodCount> " payment methods
Q&A Examples :
| Country | PaymentMethodCount |
| Poland | 6 |
| France | 5 |
| Brazil | 6 |
19/40
55. Some other tools used inside Spil Games
Cucumber
Introduction
Why should I test?
Types of tests • Scenario implementation
Good Practices
/∗∗ @Given / ^ I am a user i n " ( [ ^ " ] ∗) " $ / ∗/
Testing p u b l i c f u n c t i o n iAmAUserIn ( $ c o u n t r y )
Techniques {
Unit tests $country = s t r t o l o w e r ( $country ) ;
Integration tests $ t h i s−>s e s s i o n−>setRequestHeader ( ’ SPILTESTIPADDRESS ’ , S p i l T e s t I P : : addressFor ( $ c o u n t r y ) ) ;
Acceptance Tests }
Performance Tests /∗∗ @Given / ^ I open t h e PSS$ / ∗/
p u b l i c f u n c t i o n iOpenThePss ( )
Making it
{
happen
r e t u r n $ t h i s−>iOpenThePaymentSelectionScreen ( ) ;
TDD & BDD
}
Automating
everything /∗∗ @Then / ^ I should see " ( [ ^ " ] ∗) " payment methods$ / ∗/
p u b l i c f u n c t i o n iShouldSeePaymentMethods ( $nummethods )
Q&A {
$page = $ t h i s−>s e s s i o n−>getPage ( ) ;
$ e l = $ t h i s−>getDomNodeByXpath ( $page , ’ / / ∗ [ @id=" p s s _ m e t h o d _ c o l l e c t i o n " ] ’ ) ;
$foundmethods = count ( $ e l−> f i n d A l l ( ’ css ’ , ’ d i v . pss_method ’ ) ) ;
i f ( $nummethods ! = $foundmethods ) {
throw new Ex c e p t i o n ( " Found payment methods does n o t equal $nummethods " ) ;
}
}
20/40
56. Basic Benchmarking
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques • Testing a function doesn’t have to be painful
Unit tests
Integration tests
Acceptance Tests
• The sooner you realize is bad, the less impact
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
21/40
57. Basic Benchmarking
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques • Testing a function doesn’t have to be painful
Unit tests
Integration tests
Acceptance Tests
• The sooner you realize is bad, the less impact
Performance Tests
−spec t e s t _ f u n ( any ( ) , f u n ( ( ) −> ok ) , i n t e g e r ( ) , i n t e g e r ( ) ) −> { f l o a t ( ) , f l o a t ( ) } .
Making it
happen t e s t _ f u n (Name, TestFun , ParCount , SeqCount ) −>
TDD & BDD
RS = t e s t _ f u n _ x _ t i m e s (Name, TestFun , SeqCount ) ,
Automating
RP = t e s t _ f u n _ x _ t i m e s _ p a r a l l e l (Name, TestFun , ParCount , SeqCount ) ,
everything {RS, RP} .
Q&A
21/40
58. Basic Benchmarking
Sync Execution
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests t e s t _ f u n _ x _ t i m e s (Name, Fun , SeqCount ) −>
Integration tests T i c k = now ( ) ,
Acceptance Tests l i s t s : f o r e a c h ( f u n ( _ ) −>
Performance Tests Fun ( )
end , l i s t s : seq ( 1 , SeqCount ) ) ,
Making it Tock = now ( ) ,
happen AvgTime = t i m e r : n o w _ d i f f ( Tock , T i c k ) / 1 0 0 0 ,
TDD & BDD Speed = SeqCount ∗1000/AvgTime ,
Automating i o : f o r m a t ( " ~40p req / sec ~.1 f ~n " , [ Name, Speed ] ) ,
everything
Speed .
Q&A
22/40
59. Basic Benchmarking
Async Execution
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques t e s t _ f u n _ x _ t i m e s _ p a r a l l e l (Name, Fun , ParCount , SeqCount ) −>
Unit tests T i c k = now ( ) ,
Integration tests pmap ( f u n ( _ ) −>
Acceptance Tests l i s t s : f o r e a c h ( f u n ( _ ) −>
Performance Tests Fun ( )
end , l i s t s : seq ( 1 , SeqCount ) )
Making it end , l i s t s : seq ( 1 , ParCount ) , 250000) , %t i m e o u t
happen Tock = now ( ) ,
TDD & BDD
AvgTime = t i m e r : n o w _ d i f f ( Tock , T i c k ) / 1 0 0 0 ,
Automating
everything
Speed = ParCount∗SeqCount ∗1000/AvgTime ,
i o : f o r m a t ( " ~40p req / sec ~10.1 f ~n " , [ Name, Speed ] ) ,
Q&A Speed .
23/40
60. Tsung
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests • It can be used to generate waves of load according to different parameters
Acceptance Tests
Performance Tests
• It generates graphics showing how the target system reacted to the load
Making it
happen • Uses XML especifications
TDD & BDD
Automating
everything
Q&A
24/40
61. Tsung
Introduction
Why should I test?
• getToken waves of 100 and 500 req/s during 600 secs
Types of tests <?xml v e r s i o n = " 1 . 0 " ?>
Good Practices < !DOCTYPE tsung SYSTEM " / u s r / share / tsung / tsung −1.0. d t d " [ ] >
<tsung l o g l e v e l = " debug " d u m p t r a f f i c = " p r o t o c o l " >
Testing
<clients>
Techniques
< c l i e n t h o s t = " l o c a l h o s t " u s e _ c o n t r o l l e r _ v m = " t r u e " >< / c l i e n t >
Unit tests
</ clients>
Integration tests
<servers>
Acceptance Tests
< s e r v e r h o s t = " a p i . spilgames . com" p o r t = " 443 " t y p e = " s s l " >< / s e r v e r >
Performance Tests
< / servers>
Making it <load>
happen < a r r i v a l p h a s e phase= " 1 " d u r a t i o n = " 600 " u n i t = " second " >
TDD & BDD <users a r r i v a l r a t e = " 100 " u n i t = " second " >< / users >
Automating </ arrivalphase>
everything < a r r i v a l p h a s e phase= " 2 " d u r a t i o n = " 600 " u n i t = " second " >
<users a r r i v a l r a t e = " 500 " u n i t = " second " >< / users >
Q&A < / load>
<sessions>
< s e s s i o n name= " a p i " p r o b a b i l i t y = " 100 " t y p e = " t s _ h t t p " >
<request>
< d y n _ v a r i a b l e name= " g u es t t ok e n " j s o n p a t h = " auth . token " / >
< h t t p u r l = ’ / account / getToken / ’ v e r s i o n = ’ 1 . 1 ’ c o n t e n t s = ’ {" ; t y p e" ; : " ;
sample" ; } ’ c o n t e n t _ t y p e = ’ t e x t / j s o n ’ method= ’POST ’ >
</ http>
< / request>
< / session>
< / sessions>
< / tsung >
25/40
62. Introduction
Why should I test?
Types of tests
Introduction
Good Practices
Testing
Techniques
Unit tests
Testing Techniques
Integration tests
Acceptance Tests
Performance Tests
Making it Making it happen
happen
TDD & BDD
Automating
everything
Q&A
Q&A
26/40
63. The purist approach
Introduction
Why should I test?
Types of tests
Good Practices
1. A developer receives some requirements from his manager, who ideally
Testing
Techniques has writting some acceptance tests
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
27/40
64. The purist approach
Introduction
Why should I test?
Types of tests
Good Practices
1. A developer receives some requirements from his manager, who ideally
Testing
Techniques has writting some acceptance tests
Unit tests
Integration tests 2. He designs the simplest (flexible) solution fitting those requirements
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
27/40
65. The purist approach
Introduction
Why should I test?
Types of tests
Good Practices
1. A developer receives some requirements from his manager, who ideally
Testing
Techniques has writting some acceptance tests
Unit tests
Integration tests 2. He designs the simplest (flexible) solution fitting those requirements
Acceptance Tests
Performance Tests
3. He documents and specs the API, and implements only the exported
Making it
happen functions headers all returning unimplemented
TDD & BDD
Automating
everything
Q&A
27/40
66. The purist approach
Introduction
Why should I test?
Types of tests
Good Practices
1. A developer receives some requirements from his manager, who ideally
Testing
Techniques has writting some acceptance tests
Unit tests
Integration tests 2. He designs the simplest (flexible) solution fitting those requirements
Acceptance Tests
Performance Tests
3. He documents and specs the API, and implements only the exported
Making it
happen functions headers all returning unimplemented
TDD & BDD
Automating
everything
4. He writes as many unit and integration tests as necesary for that API
Q&A
27/40
67. The purist approach
Introduction
Why should I test?
Types of tests
Good Practices
1. A developer receives some requirements from his manager, who ideally
Testing
Techniques has writting some acceptance tests
Unit tests
Integration tests 2. He designs the simplest (flexible) solution fitting those requirements
Acceptance Tests
Performance Tests
3. He documents and specs the API, and implements only the exported
Making it
happen functions headers all returning unimplemented
TDD & BDD
Automating
everything
4. He writes as many unit and integration tests as necesary for that API
Q&A 5. He hands over his work to another developer, who after agreeing with the
design fills the implementation until all the tests pass
27/40
68. A more practical approach
Introduction
Why should I test?
Types of tests When writting new code
Good Practices
Testing • Write the tests ASAP, if possible before writing the code itself
Techniques
Unit tests • While implementing, keep in mind everything you write you have to test it
Integration tests
Acceptance Tests
Performance Tests
• Start executing your tests ASAP, specially to validate those small simple
Making it functions you use all around (yes, those that can never fail)
happen
TDD & BDD
Automating
everything
Q&A
28/40
69. A more practical approach
Introduction
Why should I test?
Types of tests When writting new code
Good Practices
Testing • Write the tests ASAP, if possible before writing the code itself
Techniques
Unit tests • While implementing, keep in mind everything you write you have to test it
Integration tests
Acceptance Tests
Performance Tests
• Start executing your tests ASAP, specially to validate those small simple
Making it functions you use all around (yes, those that can never fail)
happen
TDD & BDD When changing existing code
Automating
everything
• Even if the existing coverage is poor, write tests for the changes you
Q&A
introduce
• Make sure all the tests are running before touching a single line!
28/40
70. TDD refactor
Original Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
• One of the following tests crashed on June 23
Performance Tests
g e t _ a g e _ t e s t _ ( ) −>
Making it [
happen ? _ a s s e r t E q u a l ( 4 7 , su_user : get_age ( # date { year =1965 , month =2 , day=1 } ) ) ,
TDD & BDD
? _ a s s e r t E q u a l ( 2 7 , su_user : get_age ( # date { year =1984 , month =6 , day=27 } ) )
Automating
].
everything
Q&A
29/40
71. TDD refactor
Original Code
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
• The implementation had an obvious flaw
Integration tests
Acceptance Tests −spec get_age ( # date { } ) −> non_neg_integer ( ) .
Performance Tests
get_age ( # date { year =0 , month =0 , day=0 } ) −>
Making it 0;
happen get_age ( # date { year=Y , month=M, day=D} ) −>
TDD & BDD B i r t h G r e g o r i a n D a y s = c a l e n d a r : d a t e _ t o _ g r e g o r i a n _ d a y s ( { Y , M, D} ) ,
Automating { NowDate , _ } = c a l e n d a r : now_to_datetime ( os : timestamp ( ) ) ,
everything
CurrentGregorianDays = c a l e n d a r : d a t e _ t o _ g r e g o r i a n _ d a y s ( NowDate ) ,
Q&A ( CurrentGregorianDays−B i r t h G r e g o r i a n D a y s ) d i v 365.
30/40
72. TDD refactor
Updated Tests
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques • We update the tests with the requirements for a human-like approach
Unit tests
Integration tests g e t _ a g e _ t e s t _ ( ) −>
Acceptance Tests [
Performance Tests ? _ a s s e r t E q u a l ( 4 7 , su_user : get_age ( # date { year =1965 , month =2 , day=1 } ) )
].
Making it i s _ b i r t h d a y _ g o n e _ t e s t _ ( ) −>
happen Dob=# date { year =1972 , month =4 , day=21 } ,
TDD & BDD [
Automating ? _ a s s e r t E q u a l ( f a l s e , su_user : i s _ b i r t h d a y _ g o n e ( Dob , 4 , 20) )
everything
? _ a s s e r t E q u a l ( t r u e , su_user : i s _ b i r t h d a y _ g o n e ( Dob , 4 , 22) ) ,
Q&A ? _ a s s e r t E q u a l ( t r u e , su_user : i s _ b i r t h d a y _ g o n e ( Dob , 4 , 21) ) ,
].
31/40
73. TDD refactor
Updated Code
Introduction
Why should I test?
Types of tests
Good Practices
Testing • The resultant code is still very clean. And now it works!
Techniques
Unit tests −spec get_age ( # date { } ) −> non_neg_integer ( ) .
Integration tests get_age ( # date { year =0 , month=_ , day=_ } ) −>
Acceptance Tests 0;
Performance Tests get_age ( # date { year=Y , month=_ , day=_ } =Dob ) −>
{ {NY, NM, ND} , _ } = c a l e n d a r : now_to_datetime ( os : timestamp ( ) ) ,
Making it
case i s _ b i r t h d a y _ g o n e ( Dob , NM, ND) o f
happen
t r u e −> NY − Y ;
TDD & BDD
f a l s e −> NY − Y −1
Automating
everything end .
Q&A −spec i s _ b i r t h d a y _ g o n e ( # date { } , p o s _ i n t e g e r ( ) , p o s _ i n t e g e r ( ) ) −> boolean ( ) .
i s _ b i r t h d a y _ g o n e ( # date { month=M, day=D} , CurrentMonth , CurrentDay ) −>
(M∗100+D) =< ( CurrentMonth ∗100+CurrentDay ) .
32/40
74. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
33/40
75. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
• Builds and tests a release your projects after every commit (Unit,
Testing
Techniques Integration and Acceptance)
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
33/40
76. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
• Builds and tests a release your projects after every commit (Unit,
Testing
Techniques Integration and Acceptance)
Unit tests
Integration tests • This automatic V&V process should never take longer than 5 minutes
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
33/40
77. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
• Builds and tests a release your projects after every commit (Unit,
Testing
Techniques Integration and Acceptance)
Unit tests
Integration tests • This automatic V&V process should never take longer than 5 minutes
Acceptance Tests
Performance Tests
• If the process fails, immediatly blames the originator of the change
Making it
happen
TDD & BDD
Automating
everything
Q&A
33/40
78. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
• Builds and tests a release your projects after every commit (Unit,
Testing
Techniques Integration and Acceptance)
Unit tests
Integration tests • This automatic V&V process should never take longer than 5 minutes
Acceptance Tests
Performance Tests
• If the process fails, immediatly blames the originator of the change
Making it
happen • Long duration test suites can be put together in a regression task
TDD & BDD
Automating
everything
performed during the night
Q&A
33/40
79. CI Environment
Introduction
Why should I test?
Types of tests
Good Practices
• Builds and tests a release your projects after every commit (Unit,
Testing
Techniques Integration and Acceptance)
Unit tests
Integration tests • This automatic V&V process should never take longer than 5 minutes
Acceptance Tests
Performance Tests
• If the process fails, immediatly blames the originator of the change
Making it
happen • Long duration test suites can be put together in a regression task
TDD & BDD
Automating
everything
performed during the night
Q&A
CI + AutomaticDeployment = ContinuousDelivery
33/40
80. CI Environment
Jenkins
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
34/40
81. Keeping an eye on quality
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests • Most CI systems have desktop or browser plugins
Acceptance Tests
Performance Tests
• Screens on the wall can be an option as well
Making it
happen • The quality of the final product starts with quality of your code
TDD & BDD
Automating
everything
Q&A
35/40
82. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
36/40
83. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
36/40
84. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
• Dialyzer performs static type analysis and clear specs help a lot when
Performance Tests testing
Making it
happen
TDD & BDD
Automating
everything
Q&A
36/40
85. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
• Dialyzer performs static type analysis and clear specs help a lot when
Performance Tests testing
Making it
happen • Sonar analyses your code looking for risks
TDD & BDD
Automating
everything
Q&A
36/40
86. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
• Dialyzer performs static type analysis and clear specs help a lot when
Performance Tests testing
Making it
happen • Sonar analyses your code looking for risks
TDD & BDD
Automating
Acts as an interface for cover and dialyzer results
everything
Q&A
36/40
87. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
• Dialyzer performs static type analysis and clear specs help a lot when
Performance Tests testing
Making it
happen • Sonar analyses your code looking for risks
TDD & BDD
Automating
Acts as an interface for cover and dialyzer results
everything
Points you to the weakest parts of your source
Q&A
36/40
88. Measuring code quality
Introduction
Why should I test?
Types of tests
Good Practices
• Cover can be integrated with eunit and Common Test to show you the test
Testing
Techniques coverage
Unit tests
Integration tests
Acceptance Tests
• Dialyzer performs static type analysis and clear specs help a lot when
Performance Tests testing
Making it
happen • Sonar analyses your code looking for risks
TDD & BDD
Automating
Acts as an interface for cover and dialyzer results
everything
Points you to the weakest parts of your source
Q&A
Shows nice graphics about test coverage, unreachable code, nested
code. . . over the time
36/40
89. Quality control with Sonar
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
37/40
90. Quality control with Sonar
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests
Acceptance Tests
Performance Tests
Making it
happen
TDD & BDD
Automating
everything
Q&A
38/40
91. Introduction
Why should I test?
Types of tests
Introduction
Good Practices
Testing
Techniques
Unit tests
Testing Techniques
Integration tests
Acceptance Tests
Performance Tests
Making it Making it happen
happen
TDD & BDD
Automating
everything
Q&A
Q&A
39/40
92. Thanks Everyone!
Introduction
Why should I test?
Types of tests
Good Practices
Testing
Techniques
Unit tests
Integration tests • Questions?
Acceptance Tests
Performance Tests
• Comments?
Making it
happen • Suggestions?
TDD & BDD
Automating
everything
Q&A
40/40