SlideShare une entreprise Scribd logo
1  sur  118
cbell@CamilleBellConsulting.com 1
Testing for Agility:
Bringing Testing into Everything
Camille Bell
Agile Coach & Trainer
cbell@CamilleBellConsulting.com
Twitter @agilecamille
TDD
User Story: Vetting Sighting
In order to show
confirmation of a sighting,
As a US-CERT analyst,
I want to mark a sighting a
vetted
• XYZ sighting is viewed by
Charley US-CERT analyst
• Charley marks sighting as vetted
• Tracy from Treasury searches for
sightings vetted today
• Tracy sees XYZ as a confirmed
sighting
cbell@CamilleBellConsulting.com 2
About Me
•  Agile Coach, Trainer and Developer
•  Over 4 decades of IT. All sorts of companies,
domains, technologies & IT roles.
•  Email: cbell@CamilleBellConsulting.com
•  Twitter: @agilecamille
•  Slideshare: camille_bell
cbell@CamilleBellConsulting.com 3
• Waiting to test
• Transforming Waterfall into Agility using Feedback
• Agile testing and requirements
• Agile testing and development
• Agile testing and integration and deployment
• Agile testing and acceptance
• Agile testing ratios
• Agile testing, bug fixes and maintenance work
• Agile testing and management
• Code Coverage
Agenda
cbell@CamilleBellConsulting.com 4
The Trouble
with
Waiting to Test
cbell@CamilleBellConsulting.com 5
Agility puts more focus on
testing to improve quality
and time to market
Which is great assuming 21st
Century Agile testing and
complementary practices
cbell@CamilleBellConsulting.com 6
But many shops bolt on Waterfall
phased testing to Agile development
While still better than no Agility,
it is costly in time and money
cbell@CamilleBellConsulting.com 7
Release Testing vs. Continuous Testing
Ref: ”Lean from the Trenches” by Henrick Kniberg
cbell@CamilleBellConsulting.com 8
End Testing vs. Continuous Testing
Ref: ”Lean from the Trenches” by Henrick Kniberg
cbell@CamilleBellConsulting.com 9
• It doesn’t guarantee customer acceptance !
• It’s extremely labor intensive !
• It’s error prone !
• It’s often ambiguous !
• It doesn’t stop fixed bugs from reappearing in the code !
• Regression testing is a nightmare !
• It doesn’t prevent “but it works on my machine” thinking !
• It’s sooner, but still after the fact ! (code exists)
• It kills your test team !
Some other Waterfall phased testing
(there are many more)
cbell@CamilleBellConsulting.com 10
Agile Development
practices turn Waterfall
completely around
with Feedback
cbell@CamilleBellConsulting.com 11
Design
High Level
Requirements
Requirements
Analysis
Code
Business
Vision
Deploy
Traditional Phased Waterfall
No feedback – Assumes perfection!
Plan
cbell@CamilleBellConsulting.com 12
Code
The Waterfall Ideal Writes All the Code
and then Tests it
Ref: “BDD in 5 minutes” video by Corey Haines
"How test-driven development works" blog by J. B. Rainsberger
cbell@CamilleBellConsulting.com 13
Code
bugs
But that never works creating
the code, test & fix cycle
cbell@CamilleBellConsulting.com 14
Code
Writing the Test First Prevents Coding Bugs
cbell@CamilleBellConsulting.com 15
Code
Test First Feedback Loop
Short cycles make it work
cbell@CamilleBellConsulting.com 16
Code
Test-First, while better isn’t enough
Design
untestable
design
cbell@CamilleBellConsulting.com 17
Code
The Test-Code Cycle can provide feedback to design
Design
cbell@CamilleBellConsulting.com 18
Code
Heavy refactoring of design is part of
Test Driven Development (TDD)
Design
Again short cycles are key
cbell@CamilleBellConsulting.com 19
Code
Design
Requirements
Analysis
Now Test, Code & Design all provide
continuing feedback for improvement, but ….
Not what
the
customer
really wants
cbell@CamilleBellConsulting.com 20
The Test Driven Development Cycle
provides feedback to analysis
Requirements
Analysis
Code
Design
cbell@CamilleBellConsulting.com 21
Code
Adding in Feature Analysis to TDD creates
Behavior Driven Development (BDD)
Design
Short cycles again
Requirements
Analysis
cbell@CamilleBellConsulting.com 22
Code
Design
Requirements
Analysis
High Level
Requirements
Not exactly what
the Market
Needed
(even though
customer rep
thought it was)
Now Behavior Driven Development provides
continuing feedback for improvement, but ….
cbell@CamilleBellConsulting.com 23
Code
Design
Requirements
Analysis
The Behavior Driven Development Cycle, when
deployed to the field provides feedback to marketing
Deployed
High Level
Requirements
cbell@CamilleBellConsulting.com 24
Continuous Deployment further
improves feedback
Code
Design
Requirements
Analysis
Deployed
High Level
Requirements
cbell@CamilleBellConsulting.com 25
But what if our initial vision was flawed?
Code
Design
Requirements
Analysis
Deployed
High Level
Requirements
Business
Vision
Product Concept
wrong !!!
Entire effort fails at
great expense !!!
cbell@CamilleBellConsulting.com 26
Lean Startup’s Build-Measure-Learn feedback loop
tests the ideas behind the business vision
cbell@CamilleBellConsulting.com 27
Agility Testing
starts with
Testable
Agile Requirements
cbell@CamilleBellConsulting.com 28
Idea
Test Cases
Software
Conversation
Agile Requirements Evolve from Feedback
Ref: Rachel Davies & Liz Sedley – “Agile Coaching”
cbell@CamilleBellConsulting.com 29
Customer Collaborates with Dev Team on
Confirmation of Stories
User Story: Vetting Sighting
In order to show confirmation of a sighting,
As a US-CERT analyst,
I want to mark a sighting a vetted
• XYZ sighting is viewed by Charley US-CERT analyst
• Charley marks sighting as vetted
• Tracy from Treasury searches for sightings vetted today
• Tracy sees XYZ as a confirmed sighting
Ref: Ron Jeffries 3 Cs of User Stories
Card
Confirmation
Collaboration
cbell@CamilleBellConsulting.com 30
User Stories Confirmed Through
Automated Tests During Iteration
• XYZ sighting is viewed by Charley US-CERT analyst
• Charley marks sighting as vetted
• Tracy from Treasury searches for sighting vetted today
• Tracy sees XYZ as a confirmed sighting
User Story: Vetting Sighting
In order to show confirmation of a sighting,
As a US-CERT analyst,
I want to mark a sighting a vetted
describe ”Imports TEWI sightings" do
context "a simple line item should know its name, quanity and price" do
let(:purchase) { LineItem.new("1 book at 12.49") }
it "should have a name of 'book'" do
purchase.name.should == 'book'
end
it "should have a quanity of 1" do
purchase.quantity.should == 1
end
it "should have a price of 12.49" do
purchase.price.should == 12.49
end
end
describe ”CERT analyst finds a recent TEWI sighting" do
context "a simple line item should know its name, quanity and price" do
let(:purchase) { LineItem.new("1 book at 12.49") }
it "should have a name of 'book'" do
purchase.name.should == 'book'
end
it "should have a quanity of 1" do
purchase.quantity.should == 1
end
it "should have a price of 12.49" do
purchase.price.should == 12.49
end
end
describe ”Vets a sighting" do
context "a simple line item should know its name, quanity and price" do
let(:purchase) { LineItem.new("1 book at 12.49") }
it "should have a name of 'book'" do
purchase.name.should == 'book'
end
it "should have a quanity of 1" do
purchase.quantity.should == 1
end
it "should have a price of 12.49" do
purchase.price.should == 12.49
end
end
describe ” Trusted partner finds un-vetted sighting" do
context "a simple line item should know its name, quanity and price" do
let(:purchase) { LineItem.new("1 book at 12.49") }
it "should have a name of 'book'" do
purchase.name.should == 'book'
end
it "should have a quanity of 1" do
purchase.quantity.should == 1
end
it "should have a price of 12.49" do
purchase.price.should == 12.49
end
end
describe "Trusted partner finds vetted sighting" do
context ”a vetted sighting should be clearly vetted" do
before(:each) do
new_sighting.build_from_TEWI()
cert_user.new(‘default_cert’)
new_sighting.vet(cert_user)
new_partner.new(‘default_partner’)
end
it "should have a tag of ’vetted'" do
new_sighting.tag.should_include == ’vetted'
end
it "should be viewable by partner" do
new_sighting.accesable_by(new_partner).should == true
end
end
cbell@CamilleBellConsulting.com 31
Incrementally turn
High Level Requirements
(features, epics)
into Tests
cbell@CamilleBellConsulting.com 32
In order to conduct banking
when the bank is closed
As a bank customer
I want to use an ATM
Why?
Who?
What?
Why does your
user want this?
Your idea probably needs to be broken down into smaller stories.
Who is this user?
How is he/she
different from
other users.
What exactly does
your user want?
Start with an idea for an App (like ATM banking)
These ideas are called User Stories.
cbell@CamilleBellConsulting.com 33
Break Down Big Stories into
Smaller User Stories
In order to get money
when the bank is closed
As a bank customer
I want to withdraw cash at the ATM
In order to deposit my checks
when the bank is closed
As a bank customer
I want to deposit checks at the ATM
In order to earn interest even
when the bank is closed
As a bank customer
I want to transfer money from
checking to saving at the ATM
In order to not overdraw my
account when the bank is closed
As a bank customer
I want to transfer money from
savings to checking at the ATM
cbell@CamilleBellConsulting.com 34
Choose a Story
(to become an automated test)
In order to get money
when the bank is closed
As a bank customer
I want to withdraw cash at the ATM
Feature: Cash Withdrawal
In order to get money when the
bank is closed
As a bank customer
I want to withdraw cash at the ATM
cbell@CamilleBellConsulting.com 35
Define Some High Level Scenarios
for that Story
Feature: Cash Withdrawal
In order to get money when the
bank is closed
As a bank customer
I want to withdraw cash at the ATM
• Successful Withdrawal
• Withdrawal Failed Because
Cash Machine Doesn’t Have
Dollar Bills
• Withdrawal Failed Due to
Insufficient Funds
• Withdrawal Failed Because
Account Closed
cbell@CamilleBellConsulting.com 36
• Successful Withdrawal
Given my account has been
credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account
should be $80
Choose a Scenario and
Define the Steps in Testable Detail
cbell@CamilleBellConsulting.com 37
• Successful Withdrawal
Given my account has been
credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account
should be $80
Use Given, When, Then format.
OK to add And and
But for readability
Test steps must be unambiguous.
Quantities are especially testable.
cbell@CamilleBellConsulting.com 38
With Minor Changes it can become an
Automated Test (Cucumber example)
• Successful Withdrawal
Given my account has been
credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account
should be $80
Scenario: Successful Withdrawal
Given my account has been
credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account
should be $80
cbell@CamilleBellConsulting.com 39
Functional & Unit Tests
Behavior Driven Development
&
Test Driven Development
cbell@CamilleBellConsulting.com 40
Behavior Driven Development
(aka Specification by Example)
•  Builds upon and formalizes the best practices of Test
Driven Development
•  Works outside-in
•  Starts with a failing customer acceptance test
•  Test describes needed behavior from a customer
viewpoint
•  Tests must be readable by anyone on team
cbell@CamilleBellConsulting.com 41
So To Understand BDD,
First Understand TDD
cbell@CamilleBellConsulting.com 42
3 Rules of Test Driven Development
1.  You are not allowed to write any production code
unless it is to make a failing unit test pass.
2.  You are not allowed to write any more of a unit test
than is sufficient to fail; and compilation failures are
failures.
3.  You are not allowed to write any more production
code than is sufficient to pass the one failing unit
test.
With tests you are in control; without tests you aren’t!
Bob Martin - The Three Rules of TDD
cbell@CamilleBellConsulting.com 43
The Test Driven Development Cycle
Write a failing test
for new functionality
Write just enough
code to pass test
Refactored code
must also pass tests,
no new functionality
CamillesCareer@gmail.com 43
cbell@CamilleBellConsulting.com 44
JUnit 4 Tests for Movie List Story
Starting Test An empty list should have a size of zero.
import static org.junit.Assert.*;
import org.junit.Test;
public class MovieListTest {
@Test
public void testEmptyListSize() {
MovieList emptyMovieList = new MovieList();
assertEquals("Size of movie list should be 0.", 0,
emptyMovieList.size());
}
}
Ref: Java code examples from
“Test-Driven Development: A Practical Guide” by David Astels
cbell@CamilleBellConsulting.com 45
cbell@CamilleBellConsulting.com 46
Class Code for Movie List Story
Test An empty list should have a size of zero.
public class MovieList {
public int size() {
return 0;
}
}
Couldn't be simpler, but that is the point!
cbell@CamilleBellConsulting.com 47
cbell@CamilleBellConsulting.com 48
Old Test An empty list should have a size of zero.
New Test Adding a movie to an empty list should result in a list with the size of one.
public class MovieListTest {
@Test
public void testEmptyListSize() {
MovieList emptyMovieList = new MovieList();
assertEquals("Size of movie list should be 0.", 0,
emptyMovieList.size());
}
@Test
public void testSizeAfterAddingOne () {
Movie starWars = new Movie();
MovieList oneItemList = new MovieList();
oneItemList.add(starWars);
assertEquals("Size of movie list should be 1.", 1,
oneItemList.size());
}
}
JUnit 4 Tests for Movie List Story
cbell@CamilleBellConsulting.com 49
cbell@CamilleBellConsulting.com 50
Class Code for Movie List Story
public class Movie {
}
public class MovieList {
private int numberOfMovies = 0;
public int size() {
return numberOfMovies;
}
public void add(Movie movieToAdd) {
numberOfMovies = 1;
}
}
Baby Steps: Change as little as possible to get to GREEN
New Test Adding a movie to an empty list should result in a list with the size of one.
cbell@CamilleBellConsulting.com 51
cbell@CamilleBellConsulting.com 52
•  Programmers restructure system by improving the design of existing
code without changing its behavior
–  to remove duplication, improve communication, simplify, or add flexibility
•  Refactoring is not random change, instead it’s driven by learning
from our tests and from identifying and fixing code smells
•  DRY (Don’t Repeat Yourself)
•  Refactoring can occur just prior or just after writing new code
•  Test, code, refactor, re-test
–  Kent Beck suggests short cycle (10 minutes)
Refactoring
When fixing bugs, write a Red test, BEFORE fixing the bug. Test will run Green when bug fixed.
When working with legacy code make sure to write automated tests BEFORE refactoring code,
If tests didn’t exist.
cbell@CamilleBellConsulting.com 53
Refactored Class Code after 4th Test
import java.util.ArrayList;
public class Movie {
}
public class MovieList {
private int numberOfMovies = 0;
private ArrayList<Movie> movies = new ArrayList<Movie>();
public int size() {
return movies.size();
}
public void add(Movie movieToAdd) {
numberOfMovies++;
movies.add(movieToAdd);
}
public boolean contains(Movie movieToCheckFor) {
return movies.contains(movieToCheckFor);
}
}
New Test If we add a movie to a list, we should be able to ask if it’s there and receive
a positive response.
Delete
Delete
Change
cbell@CamilleBellConsulting.com 54
cbell@CamilleBellConsulting.com 55
BDD & Cucumber
Stories to Tests
cbell@CamilleBellConsulting.com 56
Parts of a Cucumber Feature File
Feature: Cash Withdrawal
As a bank customer
I want to withdraw cash at the ATM
So that I can get money when the bank is closed
Scenario: Successful Withdrawal
Given my account has been credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account should be $80
Scenario Title
Scenario
Feature Title
Feature
Steps
cash_withdrawal.feature
Ref: Cucumber code examples from
“The Cucumber Book” by Matt Wynne and Aslak Hellesøy
cbell@CamilleBellConsulting.com 57
The Three Parts of
Cucumber
cbell@CamilleBellConsulting.com 58
The Structure of
Cucumber Features
Feature: [ feature title ]
As a user [ role ]
I want [ feature ]
So that [ business value ]
Scenario: [ scenario title ]
Given [ a pre-condition ]
And [another pre-condition ]
When [ event ]
Then [ post-condition outcome ]
And [ another post-condition outcome ]
Plain
TextKey
Words
From
User
Story
Ref: Cucumber code examples from
“The Cucumber Book” by Matt Wynne and Aslak Hellesøy
cbell@CamilleBellConsulting.com 59
Example
Cucumber Feature
cbell@CamilleBellConsulting.com 60
Step Definitions
cbell@CamilleBellConsulting.com 61
Cucumber Command Line
cbell@CamilleBellConsulting.com 62
Plus the Evolving Code Under Test
cbell@CamilleBellConsulting.com 63
The Cucumber
BDD Cycle
cbell@CamilleBellConsulting.com 64
Create Cucumber Feature File
from BDD Test Definition
cbell@CamilleBellConsulting.com 65
Run Feature from the Command Line
cbell@CamilleBellConsulting.com 66
Copy Snippet from
the Command Line Output
cbell@CamilleBellConsulting.com 67
Create Step Definition File
and Pasted Snippet
cbell@CamilleBellConsulting.com 68
Run Cucumber Again
(Notice the Pending
and Skipped Steps)
cbell@CamilleBellConsulting.com 69
Replace the Pending in the First Step
with Test Code You Need to Implement
cbell@CamilleBellConsulting.com 70
Run Cucumber Again
(Notice the Failed
and Skipped Steps)
cbell@CamilleBellConsulting.com 71
Implement Just Enough Code
to Make the Test Step Pass
cbell@CamilleBellConsulting.com 72
Run Cucumber Again
(Notice the Passed,
Pending and Skipped Steps)
cbell@CamilleBellConsulting.com 73
Repeat Cycle Until All Test Steps Pass
Write
Scenario
Create Step
Definitions
from Snippet
Turn Pending
Step into Test
Watch it Fail
Make it Pass
by Evolving
Code
Pending
Steps?
YesNoDone
with
Scenario
Start
with
User
Story
cbell@CamilleBellConsulting.com 74
Example
Scenario Outline
cbell@CamilleBellConsulting.com 75
Fit, Selenium examples
cbell@CamilleBellConsulting.com 76
cbell@CamilleBellConsulting.com 77
Browser Testing with Selenium
Example Selenium Testing with
jUnit and Java in Eclipse
Selenium IDE (Firefox only)
Record/Playback
Recorded tests don’t scale,
Only run test individually or
single test suite – good for first
cut tests
Selenium Remote Control
cross browser tests
with Java, Ruby, Python,
Perl, PHP or .Net
cbell@CamilleBellConsulting.com 78
Agility Testing
Ratios
cbell@CamilleBellConsulting.com 79
Dysfunctional Test Ratio
cbell@CamilleBellConsulting.com 80
Healthy Test Ratio
cbell@CamilleBellConsulting.com 81
Healthy Test Ratio
• Very, very, very Slow
• Usability testing
• Exploratory testing
• Labor intensive
• Automate exploration
• Very slow
• Validate UI functionality
• Brittle, if using capture playback
• Very, very fast
• Uses mocks
• Validate class and
function/method code
• Some fast, some slow
• Data and other integrity
• Validate at higher level
than unit tests, but not
directly user facing
cbell@CamilleBellConsulting.com 82
Agile Testing
and
Integration & Deployment
cbell@CamilleBellConsulting.com 83
3. Update
Build
Workspace
Continuous Integration Cycle
Developer
Deploy
Server
Version Control
Repository
1. Check-in
CI Server2. Monitor changes
4. Build
5. Run Tests
6. Send Build
& Test Report
7. Deploy App
at Release
cbell@CamilleBellConsulting.com 84
3. Update
Build
Workspace
Continuous Integration Cycle
with Performance Testing
Developer
Deploy
Server
Version Control
Repository
1. Check-in
CI Server2. Monitor changes
4. Build
5. Run Tests
6. Send Build
& Test Report
7b. Deploy App
at Release
Performance
Test Server
Frequently
7a. Performance
Test App
cbell@CamilleBellConsulting.com 85
Only Check-in Working Tested Code
Version Control
Repository
cbell@CamilleBellConsulting.com 86
Continuous Deployment Cycle
Builds on Continuous Integration
Developer
Deploy
Server
Version Control
Repository
1. Check-in
2. Monitor changes
3. Update
Build
Workspace
4. Build
5. Run Tests
CI Server
6. Send Build & Test Report
7. Clean Build
& Tests Pass
Deploy App
Immediately
cbell@CamilleBellConsulting.com 87
Tools for the Deployment Pipeline
Configuration
Management
Continuous
Integration
Testing
Strategy
Deployment
Capistrano
cbell@CamilleBellConsulting.com 88
cbell@CamilleBellConsulting.com 89
Build-Measure-Lean with Lean Startup
Split Tests
Customer Interviews
Customer Development
5 Whys Root Cause Analysis
Customer Advisory Board
Falsifiable Hypotheses
Product Owner Accountability
Customer Archetypes
Cross-functional Teams
Semi-autonomous Teams
Smoke Tests
Unit Tests
Usability Tests
Continuous Integration
Incremental Deployment
Free & Open-Source Components
Cloud Computing
Cluster Immune System
Just-in-time Scalability
Refactoring
Developer Sandbox
Split Tests
Clear Product Owner
Continuous Deployment
Usability Tests
Real-time Monitoring
Customer Liaison
Funnel Analysis
Cohort Analysis
Net Promoter Score
Real-Time Altering
Predictive Monitoring
Measure Faster
Learn Faster Code Faster
cbell@CamilleBellConsulting.com 90
Examples of Testing of Ideas with Lean Startup
• Split A/B Testing
– controlled experiments
– two versions of product released simultaneously
– test metric comparison ($ earned, # of signups, etc.)
– Netflix huge user of A/B testing
• Crowd Testing: Validating the existence of a customer base
– bypassing investors with Kickstarter, etc. (e.g. Torment)
– creating dummy site to sign up and buy (e.g. Zappos)
cbell@CamilleBellConsulting.com 91
cbell@CamilleBellConsulting.com 92
Bug Growth on
Non-Agile Projects
T i m e
B
u
g
s
code fix code codecode codecodecode fixfixfixfix fix
release 1
release 2
release 3
release 4
release 5
release 6
release 7
cbell@CamilleBellConsulting.com 93
TDD and BDD prevents a Great Many Bugs, but …
a few will probably still occur
Write a failing test
for new functionality
Write just enough
code to pass test
Refactored code
must also pass tests,
no new functionality
cbell@CamilleBellConsulting.com 93
cbell@CamilleBellConsulting.com 94
Create a RED test BEFORE fixing the bug,
… so you are sure you really fixed it,
and the bug never sneaks through the tests again.
cbell@CamilleBellConsulting.com 94
Write a failing test
that exposes the bug
Write just enough
code to fix the bug
Refactor if needed,
must also pass tests,
no new functionality
cbell@CamilleBellConsulting.com 95
cbell@CamilleBellConsulting.com 96
• By acting as a servant leader and removing obstacles
• By providing whole team training in agile processes
• By providing technical training in automated test tools to
current employees
• By requiring automated test experience for new hires
• By making testability a must have for languages & tools
(many of the best are open source)
• By providing the automated test tools themselves
• By providing continuous integration and deployment servers
• By tracking progress differently
Management Needs to Support Agility
cbell@CamilleBellConsulting.com 97
•  Agile processes like Scrum, Lean, Kanban and XP:
•  often make short near term predictions
•  track actual progress
•  determine if the predictions, if made, were correct
•  use actual progress metrics to project future outcomes
•  adapt and adjust based on real outcomes
• Managers must ensure that agile practices:
•  help the team improve projections
•  never become a stick
Agility Organizational Processes
Have Built-in Feedback Loops and Validation
cbell@CamilleBellConsulting.com 98
Running Tested Features Definition
1.  The	
  desired	
  so,ware	
  is	
  broken	
  down	
  into	
  named	
  features	
  
(requirements,	
  stories),	
  which	
  are	
  part	
  of	
  what	
  it	
  means	
  to	
  
deliver	
  the	
  desired	
  system.	
  	
  
2.  For	
  each	
  named	
  feature,	
  there	
  are	
  one	
  or	
  more	
  automated	
  
acceptance	
  tests	
  which,	
  when	
  they	
  work,	
  will	
  show	
  that	
  the	
  
feature	
  in	
  quesAon	
  is	
  implemented.	
  
3.  The	
  RTF	
  metric	
  shows,	
  at	
  every	
  moment	
  in	
  the	
  project,	
  how	
  
many	
  features	
  are	
  passing	
  all	
  their	
  acceptance	
  tests.	
  	
  
Ron Jeffries, A Metric Leading to Agility
cbell@CamilleBellConsulting.com 99
Calculating RTF is Simple
•  No weighting - No partials credit: A Feature is
either all running & passing all tests or it isn’t.
•  So a feature either counts as 0 or 1.
•  Add up the 1s.
•  Run all tests for each measurement.
•  Track over time.
cbell@CamilleBellConsulting.com 100
RTF Works on any Project, even Waterfall
RTF on Agile software
development projects
should steadily increase
over time
RTF on Waterfall
development only
increases at project end
cbell@CamilleBellConsulting.com 101
Example 1: Buggy New Feature Breaks Old
Monday 6 running tested
features. Tuesday no
new features added, but
existing features still
pass tests.
On Wednesday, new
feature added which fails
test and causes side
effect that causes pre-
existing feature to also
fail tests.
On Thursday, new tests
added to old feature and
new feature fixed. All
tests pass.
cbell@CamilleBellConsulting.com 102
Example 2: New Feature Breaks Everything
Monday 6 running tested
features. Tuesday no
new features added, but
existing features still
pass tests.
On Wednesday, new
feature added which
breaks the build, crashes
system, corrupts DB or
something similarly
catastrophic.
On Thursday, new tests
added and new feature
fixed. All tests pass.
cbell@CamilleBellConsulting.com 103
CUMULATIVE BURNUP
StoryPointsorFeaturesComplete(RTF)
Time
Burnup of Story Points or Running Tested Features metrics can be more informative than burndowns.
New Stories when added
caused existing stories to
fail automated tests
Stories fixed,
tests added, all
tests now pass
Stories Removed to
meet Release DateStories
Added
New Baseline
New Baseline
Baseline
ReleaseDate
Complex
Code
needed
refactoring
Ref: Alistair Cockburn http://alistair.cockburn.us/Earned-value+and+burn+charts
Ron Jeffries http://xprogramming.com/articles/jatrtsmetric/
cbell@CamilleBellConsulting.com 104
Camille Bell
Agile Coaching & Consulting
Retrospectives
Agile Boot Camps
Agile Training
Updated Slides
or just to chat about things agile
CamillesCareer@gmail.com
cbell@CamilleBellConsulting.com 105
cbell@CamilleBellConsulting.com 106
Code Coverage
C0, C1 and C2 Coverage
C0 – Line coverage analysis measures which lines of code have been
executed. C0 coverage is typically used it to find the areas of your
program that have not been sufficiently tested, i.e. those that were not
run by any your test cases.
C1 – Branch coverage analysis measures which of the different possible
branches of conditional statements have been tested. It is easy to have
100% C0 line coverage and only partial C1 branch coverage, because
if, then and if, then, else statements may be contained all
on one line in many languages.
C2 - Path coverage measures which of the different possible execution
paths through your code were tested. Paths are a combination of linear
code execution with alternate branches. Since each new conditional that
is encountered gives rise to new path choices, the permutations of
possible unique paths gets huge.
cbell@CamilleBellConsulting.com 107
Code Coverage
C0, C1 and C2 Tools
C0- Every automated code coverage tool measures at least Line
Coverage. Open source tools rcov for Ruby and Cobertura for Java
provide C0 coverage.
C1 - Many automated code coverage tools (both open source and
commercial) measure Branch coverage. For instance open source tool
Emma for Java measures both Line and Branch coverage as does
Clover, a commercial Java test tool.
C2 – Complete path coverage throughout an entire large application is
almost unheard of because of its complexity and cost. I know of only two
large software applications (both in Ada) that had 100% path coverage.
Path coverage through multiple limited sections of code is much more
feasible. All path coverage tools also include line and branch coverage.
All are commercial and very expensive.
cbell@CamilleBellConsulting.com 108
public class CoverageExample {
// This code is logically equivalent to the next example, but because of the
// code structure, untested branches are detected by line coverage.
public int echo (int x, boolean state1, boolean state2, boolean state3) {
if (state1) {
x++;
}
if (state2) {
x--;
}
if (state3) {
x = x;
}
return x;
}
}
// This test provides 100% Line coverage, but it doesn’t provide any branch coverage.
public class CoverageTestLineOnly {
CoverageExample lineBranchPath;
@Before
public void runBeforeEveryTest() {
lineBranchPath = new CoverageExample();
}
@Test
public void testReturnInput0FalseFalseFalse() {
assertEquals(0, lineBranchPath.echo(0, false, false, false);
}
}
Code Coverage
C0 – Line Coverage
cbell@CamilleBellConsulting.com 109
Code Coverage
C0 – Line Coverage Limitations
public class CoverageExample
// This code is logically equivalent to the prior example, but because of the
// code structure, untested branches are not detected by line coverage.
public int echo (int x, boolean state1, boolean state2, boolean state3) {
if (state1) { x++; }
if (state2) { x--; }
if (state3) { x = x; }
return x;
}
}
// This test provides 100% Line coverage, but it doesn’t provide any branch coverage.
public class CoverageTestLineOnly {
CoverageExample lineBranchPath;
@Before
public void runBeforeEveryTest() {
lineBranchPath = new CoverageExample();
}
@Test
public void allFalse() {
assertEquals(0, lineBranchPath.echo(0, false, false, false);
}
}
cbell@CamilleBellConsulting.com 110
Code Coverage
C1 – Branch Coverage
public class CoverageExample
// This code is identical to the prior example.
public int echo (int x, boolean state1, boolean state2, boolean state3) {
if (state1) { x++; }
if (state2) { x--; }
if (state3) { x = x; }
return x;
}
}
// Adding the new test provide 100% branch coverage, but doesn’t detect the bug.
public class CoverageTestLineAndBranch {
CoverageExample lineBranchPath;
@Before
public void runBeforeEveryTest() {
lineBranchPath = new CoverageExample();
}
@Test
public void allFalse() {
assertEquals(0, lineBranchPath.echo(0, false, false, false);
}
@Test
public void allTrue() {
assertEquals(0, lineBranchPath.echo(0, true, true, true);
}
}
cbell@CamilleBellConsulting.com 111
Code Coverage
C2 – Path Coverage Explosion
false false false
false false true
false true false
false true true
true false false
true false true
true true false
true true true
To test all possible paths of N
Boolean conditionals is 2 ^ N.
So for our example of 3 Booleans
that’s 2x2x2 or 8. As shown in the
example on the left, it’s not too
bad.
However, the set of all possible
paths grows exponentially, so a
combination of 10 Booleans,
becomes 2x2x2x2x2x2x2x2x2x2
or 1024 possible paths, an
unmanageable number of tests.
cbell@CamilleBellConsulting.com 112
Code Coverage
C2 – Basis Path Simplification
false false false
true false false
false true false
false false true
Basis path testing is a simplification of
path testing, that significantly lowers the
number of needed tests
A complete set of basis path sets are the
number of boolean decisions +1. It grows
linearly instead of exponentially. that’s
3+1 or 4 instead of 8 for our example
and 10+1 or 11 instead of 1024.
Path 1: Any path will do for the baseline,
so pick all trues or all falses for
simplification. We are picking false. This
is the first path in our basis set.
Path 2: To find the next basis path, we
flip the first decision (only) in our
baseline.
Path 3: Next we flip the second decision
(only) in our baseline path, the first
baseline decision remains fixed with the
false outcome.
Path 4: Finally, you flip the third decision
in your baseline path. Again, the first
baseline decision remains fixed with the
false outcome.
cbell@CamilleBellConsulting.com 113
Code Coverage
Line, Branch & Basis Path
// JUnit 4 tests with 100% line, 100% branch coverage, and 100% basis path.
public class CoverageTestPath {
CoverageExample lineBranchPath;
@Before
public void runBeforeEveryTest() {
lineBranchPath = new CoverageExample();
}
@After
public void runBeforeEveryTest() {
lineBranchPath = null ;
}
@Test
// First basis path.
public void allFalse() {
assertEquals ("All false inputs ", 0, lineBranchPath.echo(0, false, false, false);
}
@Test
// Second basis path.
public void trueFalseFalse() {
assertEquals ("True-False-False inputs ", 0, lineBranchPath.echo(0, true, false, false);
}
@Test
// Third basis path.
public void falseTrueFalse() {
assertEquals ("False-True-False inputs ", 0, lineBranchPath.echo(0, false, true, false);
}
@Test
// Fourth basis path.
public void falseFalseTrue() {
assertEquals ("False-False-True inputs ", 0, lineBranchPath.echo, false, false, true);
}
}
cbell@CamilleBellConsulting.com 114
Cucumber Code
Coverage with
RCov
(today Simplecov would better tool)
cbell@CamilleBellConsulting.com 115
RCov and Clover
•  RCov, an open source Ruby Gem provides code
coverage measurement for Cucumber
•  RCov provides C0 – line code coverage
•  Clover, a commercial tool provides code coverage
measurement for JUnit
•  Clover provides C1 – branch code coverage
cbell@CamilleBellConsulting.com 116
RCov Cucumber/Ruby Example
cbell@CamilleBellConsulting.com 117
RCov Cucumber/Ruby Example
cbell@CamilleBellConsulting.com 118
RCov Cucumber/Ruby Example

Contenu connexe

Tendances

Flexing your Agile Muscle - Agile Technical Concepts Explained
Flexing your Agile Muscle - Agile Technical Concepts ExplainedFlexing your Agile Muscle - Agile Technical Concepts Explained
Flexing your Agile Muscle - Agile Technical Concepts ExplainedSandy Mamoli
 
Improving the Quality of Existing Software
Improving the Quality of Existing SoftwareImproving the Quality of Existing Software
Improving the Quality of Existing SoftwareSteven Smith
 
Promises in JavaScript with jQuery
Promises in JavaScript with jQueryPromises in JavaScript with jQuery
Promises in JavaScript with jQueryRyan Blunden
 
Production code without tests
Production code without testsProduction code without tests
Production code without testsAkim Khalilov
 
Automated Acceptance Tests in .NET
Automated Acceptance Tests in .NETAutomated Acceptance Tests in .NET
Automated Acceptance Tests in .NETWyn B. Van Devanter
 
Cucumber From the Ground Up - Joseph Beale
Cucumber From the Ground Up - Joseph BealeCucumber From the Ground Up - Joseph Beale
Cucumber From the Ground Up - Joseph BealeQA or the Highway
 
Selenium + Specflow
Selenium + SpecflowSelenium + Specflow
Selenium + Specflowcromwellryan
 
Behavior Driven Testing - A paradigm shift
Behavior Driven Testing - A paradigm shiftBehavior Driven Testing - A paradigm shift
Behavior Driven Testing - A paradigm shiftAspire Systems
 
Web based automation testing on Node.js environment
Web based automation testing on Node.js environmentWeb based automation testing on Node.js environment
Web based automation testing on Node.js environmentYu-Lin Huang
 
Improving the Quality of Existing Software
Improving the Quality of Existing SoftwareImproving the Quality of Existing Software
Improving the Quality of Existing SoftwareSteven Smith
 
Agile testing for mere mortals
Agile testing for mere mortalsAgile testing for mere mortals
Agile testing for mere mortalsDave Haeffner
 
Producing Testable Requirements
Producing Testable RequirementsProducing Testable Requirements
Producing Testable RequirementsIntergen
 
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016Steven Smith
 
ATDD in Practice
ATDD in PracticeATDD in Practice
ATDD in PracticeSteven Mak
 
Selenium Users Anonymous
Selenium Users AnonymousSelenium Users Anonymous
Selenium Users AnonymousDave Haeffner
 
Test Driven Development with Laravel
Test Driven Development with LaravelTest Driven Development with Laravel
Test Driven Development with LaravelTyler Johnston
 

Tendances (20)

Flexing your Agile Muscle - Agile Technical Concepts Explained
Flexing your Agile Muscle - Agile Technical Concepts ExplainedFlexing your Agile Muscle - Agile Technical Concepts Explained
Flexing your Agile Muscle - Agile Technical Concepts Explained
 
Improving the Quality of Existing Software
Improving the Quality of Existing SoftwareImproving the Quality of Existing Software
Improving the Quality of Existing Software
 
Promises in JavaScript with jQuery
Promises in JavaScript with jQueryPromises in JavaScript with jQuery
Promises in JavaScript with jQuery
 
Production code without tests
Production code without testsProduction code without tests
Production code without tests
 
Automated Acceptance Tests in .NET
Automated Acceptance Tests in .NETAutomated Acceptance Tests in .NET
Automated Acceptance Tests in .NET
 
eXtreme Programming
eXtreme ProgrammingeXtreme Programming
eXtreme Programming
 
Developer Testing
Developer TestingDeveloper Testing
Developer Testing
 
Cucumber From the Ground Up - Joseph Beale
Cucumber From the Ground Up - Joseph BealeCucumber From the Ground Up - Joseph Beale
Cucumber From the Ground Up - Joseph Beale
 
Selenium + Specflow
Selenium + SpecflowSelenium + Specflow
Selenium + Specflow
 
Behavior Driven Testing - A paradigm shift
Behavior Driven Testing - A paradigm shiftBehavior Driven Testing - A paradigm shift
Behavior Driven Testing - A paradigm shift
 
Web based automation testing on Node.js environment
Web based automation testing on Node.js environmentWeb based automation testing on Node.js environment
Web based automation testing on Node.js environment
 
Improving the Quality of Existing Software
Improving the Quality of Existing SoftwareImproving the Quality of Existing Software
Improving the Quality of Existing Software
 
Agile testing for mere mortals
Agile testing for mere mortalsAgile testing for mere mortals
Agile testing for mere mortals
 
Producing Testable Requirements
Producing Testable RequirementsProducing Testable Requirements
Producing Testable Requirements
 
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016
 
ATDD in Practice
ATDD in PracticeATDD in Practice
ATDD in Practice
 
Selenium Users Anonymous
Selenium Users AnonymousSelenium Users Anonymous
Selenium Users Anonymous
 
Test Driven Development with Laravel
Test Driven Development with LaravelTest Driven Development with Laravel
Test Driven Development with Laravel
 
Great! another bug
Great! another bugGreat! another bug
Great! another bug
 
Agile Testing
Agile TestingAgile Testing
Agile Testing
 

Similaire à Testing for Agility: Bringing Testing into Everything

What CS Class Didn't Teach About Testing
What CS Class Didn't Teach About TestingWhat CS Class Didn't Teach About Testing
What CS Class Didn't Teach About TestingCamille Bell
 
Run your project like a project manager by patrice embry for eeconf 2018
Run your project like a project manager by patrice embry for eeconf 2018Run your project like a project manager by patrice embry for eeconf 2018
Run your project like a project manager by patrice embry for eeconf 2018Patrice Colancecco Embry
 
Code Reviews @ Quatico
Code Reviews @ QuaticoCode Reviews @ Quatico
Code Reviews @ QuaticoJan Wloka
 
Testable Requirements
Testable Requirements Testable Requirements
Testable Requirements Bharti Rupani
 
Expanding skill sets - Broaden your perspective on design
Expanding skill sets - Broaden your perspective on designExpanding skill sets - Broaden your perspective on design
Expanding skill sets - Broaden your perspective on designroskakori
 
Intro to TDD & BDD
Intro to TDD & BDDIntro to TDD & BDD
Intro to TDD & BDDdevObjective
 
ITB2015 - Behavior Driven Development, Automation and Continuous Integration
ITB2015 - Behavior Driven Development, Automation and Continuous IntegrationITB2015 - Behavior Driven Development, Automation and Continuous Integration
ITB2015 - Behavior Driven Development, Automation and Continuous IntegrationOrtus Solutions, Corp
 
Pivots for real for LASTConf sept 2015
Pivots for real for LASTConf sept 2015Pivots for real for LASTConf sept 2015
Pivots for real for LASTConf sept 2015Mark Smith
 
Radical Requirements: Tips For Delivering What They Want
Radical Requirements: Tips For Delivering What They WantRadical Requirements: Tips For Delivering What They Want
Radical Requirements: Tips For Delivering What They WantRob Bogue
 
Bdd with Cucumber and Mocha
Bdd with Cucumber and MochaBdd with Cucumber and Mocha
Bdd with Cucumber and MochaAtish Narlawar
 
Rev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringRev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringMarketo
 
Building software products
Building software productsBuilding software products
Building software productsChris Bohnert
 
Beyond Fast Good and Cheap
Beyond Fast Good and CheapBeyond Fast Good and Cheap
Beyond Fast Good and CheapAlan Stevens
 
The business case for contributing code
The business case for contributing codeThe business case for contributing code
The business case for contributing codeZivtech, LLC
 
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...HostedbyConfluent
 
Modeling Requirements Using Examples
Modeling Requirements Using ExamplesModeling Requirements Using Examples
Modeling Requirements Using ExamplesExcella
 
6 Things to Think About Before Building Your Website
6 Things to Think About Before Building Your Website6 Things to Think About Before Building Your Website
6 Things to Think About Before Building Your WebsiteFloown
 
Startup Secrets - Building a Compelling Value Proposition
Startup Secrets - Building a Compelling Value PropositionStartup Secrets - Building a Compelling Value Proposition
Startup Secrets - Building a Compelling Value PropositionMichael Skok
 

Similaire à Testing for Agility: Bringing Testing into Everything (20)

What CS Class Didn't Teach About Testing
What CS Class Didn't Teach About TestingWhat CS Class Didn't Teach About Testing
What CS Class Didn't Teach About Testing
 
Run your project like a project manager by patrice embry for eeconf 2018
Run your project like a project manager by patrice embry for eeconf 2018Run your project like a project manager by patrice embry for eeconf 2018
Run your project like a project manager by patrice embry for eeconf 2018
 
Code Reviews @ Quatico
Code Reviews @ QuaticoCode Reviews @ Quatico
Code Reviews @ Quatico
 
Testable Requirements
Testable Requirements Testable Requirements
Testable Requirements
 
Expanding skill sets - Broaden your perspective on design
Expanding skill sets - Broaden your perspective on designExpanding skill sets - Broaden your perspective on design
Expanding skill sets - Broaden your perspective on design
 
Testable requirements
Testable requirementsTestable requirements
Testable requirements
 
Intro to TDD & BDD
Intro to TDD & BDDIntro to TDD & BDD
Intro to TDD & BDD
 
ITB2015 - Behavior Driven Development, Automation and Continuous Integration
ITB2015 - Behavior Driven Development, Automation and Continuous IntegrationITB2015 - Behavior Driven Development, Automation and Continuous Integration
ITB2015 - Behavior Driven Development, Automation and Continuous Integration
 
2015 in tothebox-introtddbdd
2015 in tothebox-introtddbdd2015 in tothebox-introtddbdd
2015 in tothebox-introtddbdd
 
Pivots for real for LASTConf sept 2015
Pivots for real for LASTConf sept 2015Pivots for real for LASTConf sept 2015
Pivots for real for LASTConf sept 2015
 
Radical Requirements: Tips For Delivering What They Want
Radical Requirements: Tips For Delivering What They WantRadical Requirements: Tips For Delivering What They Want
Radical Requirements: Tips For Delivering What They Want
 
Bdd with Cucumber and Mocha
Bdd with Cucumber and MochaBdd with Cucumber and Mocha
Bdd with Cucumber and Mocha
 
Rev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringRev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive Scoring
 
Building software products
Building software productsBuilding software products
Building software products
 
Beyond Fast Good and Cheap
Beyond Fast Good and CheapBeyond Fast Good and Cheap
Beyond Fast Good and Cheap
 
The business case for contributing code
The business case for contributing codeThe business case for contributing code
The business case for contributing code
 
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...
You Put *What* in Your Stream?! Patterns and Practices for Event Design with ...
 
Modeling Requirements Using Examples
Modeling Requirements Using ExamplesModeling Requirements Using Examples
Modeling Requirements Using Examples
 
6 Things to Think About Before Building Your Website
6 Things to Think About Before Building Your Website6 Things to Think About Before Building Your Website
6 Things to Think About Before Building Your Website
 
Startup Secrets - Building a Compelling Value Proposition
Startup Secrets - Building a Compelling Value PropositionStartup Secrets - Building a Compelling Value Proposition
Startup Secrets - Building a Compelling Value Proposition
 

Dernier

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 

Dernier (20)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 

Testing for Agility: Bringing Testing into Everything

  • 1. cbell@CamilleBellConsulting.com 1 Testing for Agility: Bringing Testing into Everything Camille Bell Agile Coach & Trainer cbell@CamilleBellConsulting.com Twitter @agilecamille TDD User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sightings vetted today • Tracy sees XYZ as a confirmed sighting
  • 2. cbell@CamilleBellConsulting.com 2 About Me •  Agile Coach, Trainer and Developer •  Over 4 decades of IT. All sorts of companies, domains, technologies & IT roles. •  Email: cbell@CamilleBellConsulting.com •  Twitter: @agilecamille •  Slideshare: camille_bell
  • 3. cbell@CamilleBellConsulting.com 3 • Waiting to test • Transforming Waterfall into Agility using Feedback • Agile testing and requirements • Agile testing and development • Agile testing and integration and deployment • Agile testing and acceptance • Agile testing ratios • Agile testing, bug fixes and maintenance work • Agile testing and management • Code Coverage Agenda
  • 5. cbell@CamilleBellConsulting.com 5 Agility puts more focus on testing to improve quality and time to market Which is great assuming 21st Century Agile testing and complementary practices
  • 6. cbell@CamilleBellConsulting.com 6 But many shops bolt on Waterfall phased testing to Agile development While still better than no Agility, it is costly in time and money
  • 7. cbell@CamilleBellConsulting.com 7 Release Testing vs. Continuous Testing Ref: ”Lean from the Trenches” by Henrick Kniberg
  • 8. cbell@CamilleBellConsulting.com 8 End Testing vs. Continuous Testing Ref: ”Lean from the Trenches” by Henrick Kniberg
  • 9. cbell@CamilleBellConsulting.com 9 • It doesn’t guarantee customer acceptance ! • It’s extremely labor intensive ! • It’s error prone ! • It’s often ambiguous ! • It doesn’t stop fixed bugs from reappearing in the code ! • Regression testing is a nightmare ! • It doesn’t prevent “but it works on my machine” thinking ! • It’s sooner, but still after the fact ! (code exists) • It kills your test team ! Some other Waterfall phased testing (there are many more)
  • 10. cbell@CamilleBellConsulting.com 10 Agile Development practices turn Waterfall completely around with Feedback
  • 12. cbell@CamilleBellConsulting.com 12 Code The Waterfall Ideal Writes All the Code and then Tests it Ref: “BDD in 5 minutes” video by Corey Haines "How test-driven development works" blog by J. B. Rainsberger
  • 13. cbell@CamilleBellConsulting.com 13 Code bugs But that never works creating the code, test & fix cycle
  • 14. cbell@CamilleBellConsulting.com 14 Code Writing the Test First Prevents Coding Bugs
  • 15. cbell@CamilleBellConsulting.com 15 Code Test First Feedback Loop Short cycles make it work
  • 16. cbell@CamilleBellConsulting.com 16 Code Test-First, while better isn’t enough Design untestable design
  • 17. cbell@CamilleBellConsulting.com 17 Code The Test-Code Cycle can provide feedback to design Design
  • 18. cbell@CamilleBellConsulting.com 18 Code Heavy refactoring of design is part of Test Driven Development (TDD) Design Again short cycles are key
  • 19. cbell@CamilleBellConsulting.com 19 Code Design Requirements Analysis Now Test, Code & Design all provide continuing feedback for improvement, but …. Not what the customer really wants
  • 20. cbell@CamilleBellConsulting.com 20 The Test Driven Development Cycle provides feedback to analysis Requirements Analysis Code Design
  • 21. cbell@CamilleBellConsulting.com 21 Code Adding in Feature Analysis to TDD creates Behavior Driven Development (BDD) Design Short cycles again Requirements Analysis
  • 22. cbell@CamilleBellConsulting.com 22 Code Design Requirements Analysis High Level Requirements Not exactly what the Market Needed (even though customer rep thought it was) Now Behavior Driven Development provides continuing feedback for improvement, but ….
  • 23. cbell@CamilleBellConsulting.com 23 Code Design Requirements Analysis The Behavior Driven Development Cycle, when deployed to the field provides feedback to marketing Deployed High Level Requirements
  • 24. cbell@CamilleBellConsulting.com 24 Continuous Deployment further improves feedback Code Design Requirements Analysis Deployed High Level Requirements
  • 25. cbell@CamilleBellConsulting.com 25 But what if our initial vision was flawed? Code Design Requirements Analysis Deployed High Level Requirements Business Vision Product Concept wrong !!! Entire effort fails at great expense !!!
  • 26. cbell@CamilleBellConsulting.com 26 Lean Startup’s Build-Measure-Learn feedback loop tests the ideas behind the business vision
  • 28. cbell@CamilleBellConsulting.com 28 Idea Test Cases Software Conversation Agile Requirements Evolve from Feedback Ref: Rachel Davies & Liz Sedley – “Agile Coaching”
  • 29. cbell@CamilleBellConsulting.com 29 Customer Collaborates with Dev Team on Confirmation of Stories User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sightings vetted today • Tracy sees XYZ as a confirmed sighting Ref: Ron Jeffries 3 Cs of User Stories Card Confirmation Collaboration
  • 30. cbell@CamilleBellConsulting.com 30 User Stories Confirmed Through Automated Tests During Iteration • XYZ sighting is viewed by Charley US-CERT analyst • Charley marks sighting as vetted • Tracy from Treasury searches for sighting vetted today • Tracy sees XYZ as a confirmed sighting User Story: Vetting Sighting In order to show confirmation of a sighting, As a US-CERT analyst, I want to mark a sighting a vetted describe ”Imports TEWI sightings" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ”CERT analyst finds a recent TEWI sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ”Vets a sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe ” Trusted partner finds un-vetted sighting" do context "a simple line item should know its name, quanity and price" do let(:purchase) { LineItem.new("1 book at 12.49") } it "should have a name of 'book'" do purchase.name.should == 'book' end it "should have a quanity of 1" do purchase.quantity.should == 1 end it "should have a price of 12.49" do purchase.price.should == 12.49 end end describe "Trusted partner finds vetted sighting" do context ”a vetted sighting should be clearly vetted" do before(:each) do new_sighting.build_from_TEWI() cert_user.new(‘default_cert’) new_sighting.vet(cert_user) new_partner.new(‘default_partner’) end it "should have a tag of ’vetted'" do new_sighting.tag.should_include == ’vetted' end it "should be viewable by partner" do new_sighting.accesable_by(new_partner).should == true end end
  • 31. cbell@CamilleBellConsulting.com 31 Incrementally turn High Level Requirements (features, epics) into Tests
  • 32. cbell@CamilleBellConsulting.com 32 In order to conduct banking when the bank is closed As a bank customer I want to use an ATM Why? Who? What? Why does your user want this? Your idea probably needs to be broken down into smaller stories. Who is this user? How is he/she different from other users. What exactly does your user want? Start with an idea for an App (like ATM banking) These ideas are called User Stories.
  • 33. cbell@CamilleBellConsulting.com 33 Break Down Big Stories into Smaller User Stories In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM In order to deposit my checks when the bank is closed As a bank customer I want to deposit checks at the ATM In order to earn interest even when the bank is closed As a bank customer I want to transfer money from checking to saving at the ATM In order to not overdraw my account when the bank is closed As a bank customer I want to transfer money from savings to checking at the ATM
  • 34. cbell@CamilleBellConsulting.com 34 Choose a Story (to become an automated test) In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM Feature: Cash Withdrawal In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM
  • 35. cbell@CamilleBellConsulting.com 35 Define Some High Level Scenarios for that Story Feature: Cash Withdrawal In order to get money when the bank is closed As a bank customer I want to withdraw cash at the ATM • Successful Withdrawal • Withdrawal Failed Because Cash Machine Doesn’t Have Dollar Bills • Withdrawal Failed Due to Insufficient Funds • Withdrawal Failed Because Account Closed
  • 36. cbell@CamilleBellConsulting.com 36 • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Choose a Scenario and Define the Steps in Testable Detail
  • 37. cbell@CamilleBellConsulting.com 37 • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Use Given, When, Then format. OK to add And and But for readability Test steps must be unambiguous. Quantities are especially testable.
  • 38. cbell@CamilleBellConsulting.com 38 With Minor Changes it can become an Automated Test (Cucumber example) • Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Scenario: Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80
  • 39. cbell@CamilleBellConsulting.com 39 Functional & Unit Tests Behavior Driven Development & Test Driven Development
  • 40. cbell@CamilleBellConsulting.com 40 Behavior Driven Development (aka Specification by Example) •  Builds upon and formalizes the best practices of Test Driven Development •  Works outside-in •  Starts with a failing customer acceptance test •  Test describes needed behavior from a customer viewpoint •  Tests must be readable by anyone on team
  • 41. cbell@CamilleBellConsulting.com 41 So To Understand BDD, First Understand TDD
  • 42. cbell@CamilleBellConsulting.com 42 3 Rules of Test Driven Development 1.  You are not allowed to write any production code unless it is to make a failing unit test pass. 2.  You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures. 3.  You are not allowed to write any more production code than is sufficient to pass the one failing unit test. With tests you are in control; without tests you aren’t! Bob Martin - The Three Rules of TDD
  • 43. cbell@CamilleBellConsulting.com 43 The Test Driven Development Cycle Write a failing test for new functionality Write just enough code to pass test Refactored code must also pass tests, no new functionality CamillesCareer@gmail.com 43
  • 44. cbell@CamilleBellConsulting.com 44 JUnit 4 Tests for Movie List Story Starting Test An empty list should have a size of zero. import static org.junit.Assert.*; import org.junit.Test; public class MovieListTest { @Test public void testEmptyListSize() { MovieList emptyMovieList = new MovieList(); assertEquals("Size of movie list should be 0.", 0, emptyMovieList.size()); } } Ref: Java code examples from “Test-Driven Development: A Practical Guide” by David Astels
  • 46. cbell@CamilleBellConsulting.com 46 Class Code for Movie List Story Test An empty list should have a size of zero. public class MovieList { public int size() { return 0; } } Couldn't be simpler, but that is the point!
  • 48. cbell@CamilleBellConsulting.com 48 Old Test An empty list should have a size of zero. New Test Adding a movie to an empty list should result in a list with the size of one. public class MovieListTest { @Test public void testEmptyListSize() { MovieList emptyMovieList = new MovieList(); assertEquals("Size of movie list should be 0.", 0, emptyMovieList.size()); } @Test public void testSizeAfterAddingOne () { Movie starWars = new Movie(); MovieList oneItemList = new MovieList(); oneItemList.add(starWars); assertEquals("Size of movie list should be 1.", 1, oneItemList.size()); } } JUnit 4 Tests for Movie List Story
  • 50. cbell@CamilleBellConsulting.com 50 Class Code for Movie List Story public class Movie { } public class MovieList { private int numberOfMovies = 0; public int size() { return numberOfMovies; } public void add(Movie movieToAdd) { numberOfMovies = 1; } } Baby Steps: Change as little as possible to get to GREEN New Test Adding a movie to an empty list should result in a list with the size of one.
  • 52. cbell@CamilleBellConsulting.com 52 •  Programmers restructure system by improving the design of existing code without changing its behavior –  to remove duplication, improve communication, simplify, or add flexibility •  Refactoring is not random change, instead it’s driven by learning from our tests and from identifying and fixing code smells •  DRY (Don’t Repeat Yourself) •  Refactoring can occur just prior or just after writing new code •  Test, code, refactor, re-test –  Kent Beck suggests short cycle (10 minutes) Refactoring When fixing bugs, write a Red test, BEFORE fixing the bug. Test will run Green when bug fixed. When working with legacy code make sure to write automated tests BEFORE refactoring code, If tests didn’t exist.
  • 53. cbell@CamilleBellConsulting.com 53 Refactored Class Code after 4th Test import java.util.ArrayList; public class Movie { } public class MovieList { private int numberOfMovies = 0; private ArrayList<Movie> movies = new ArrayList<Movie>(); public int size() { return movies.size(); } public void add(Movie movieToAdd) { numberOfMovies++; movies.add(movieToAdd); } public boolean contains(Movie movieToCheckFor) { return movies.contains(movieToCheckFor); } } New Test If we add a movie to a list, we should be able to ask if it’s there and receive a positive response. Delete Delete Change
  • 55. cbell@CamilleBellConsulting.com 55 BDD & Cucumber Stories to Tests
  • 56. cbell@CamilleBellConsulting.com 56 Parts of a Cucumber Feature File Feature: Cash Withdrawal As a bank customer I want to withdraw cash at the ATM So that I can get money when the bank is closed Scenario: Successful Withdrawal Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80 Scenario Title Scenario Feature Title Feature Steps cash_withdrawal.feature Ref: Cucumber code examples from “The Cucumber Book” by Matt Wynne and Aslak Hellesøy
  • 58. cbell@CamilleBellConsulting.com 58 The Structure of Cucumber Features Feature: [ feature title ] As a user [ role ] I want [ feature ] So that [ business value ] Scenario: [ scenario title ] Given [ a pre-condition ] And [another pre-condition ] When [ event ] Then [ post-condition outcome ] And [ another post-condition outcome ] Plain TextKey Words From User Story Ref: Cucumber code examples from “The Cucumber Book” by Matt Wynne and Aslak Hellesøy
  • 64. cbell@CamilleBellConsulting.com 64 Create Cucumber Feature File from BDD Test Definition
  • 66. cbell@CamilleBellConsulting.com 66 Copy Snippet from the Command Line Output
  • 67. cbell@CamilleBellConsulting.com 67 Create Step Definition File and Pasted Snippet
  • 68. cbell@CamilleBellConsulting.com 68 Run Cucumber Again (Notice the Pending and Skipped Steps)
  • 69. cbell@CamilleBellConsulting.com 69 Replace the Pending in the First Step with Test Code You Need to Implement
  • 70. cbell@CamilleBellConsulting.com 70 Run Cucumber Again (Notice the Failed and Skipped Steps)
  • 71. cbell@CamilleBellConsulting.com 71 Implement Just Enough Code to Make the Test Step Pass
  • 72. cbell@CamilleBellConsulting.com 72 Run Cucumber Again (Notice the Passed, Pending and Skipped Steps)
  • 73. cbell@CamilleBellConsulting.com 73 Repeat Cycle Until All Test Steps Pass Write Scenario Create Step Definitions from Snippet Turn Pending Step into Test Watch it Fail Make it Pass by Evolving Code Pending Steps? YesNoDone with Scenario Start with User Story
  • 77. cbell@CamilleBellConsulting.com 77 Browser Testing with Selenium Example Selenium Testing with jUnit and Java in Eclipse Selenium IDE (Firefox only) Record/Playback Recorded tests don’t scale, Only run test individually or single test suite – good for first cut tests Selenium Remote Control cross browser tests with Java, Ruby, Python, Perl, PHP or .Net
  • 81. cbell@CamilleBellConsulting.com 81 Healthy Test Ratio • Very, very, very Slow • Usability testing • Exploratory testing • Labor intensive • Automate exploration • Very slow • Validate UI functionality • Brittle, if using capture playback • Very, very fast • Uses mocks • Validate class and function/method code • Some fast, some slow • Data and other integrity • Validate at higher level than unit tests, but not directly user facing
  • 83. cbell@CamilleBellConsulting.com 83 3. Update Build Workspace Continuous Integration Cycle Developer Deploy Server Version Control Repository 1. Check-in CI Server2. Monitor changes 4. Build 5. Run Tests 6. Send Build & Test Report 7. Deploy App at Release
  • 84. cbell@CamilleBellConsulting.com 84 3. Update Build Workspace Continuous Integration Cycle with Performance Testing Developer Deploy Server Version Control Repository 1. Check-in CI Server2. Monitor changes 4. Build 5. Run Tests 6. Send Build & Test Report 7b. Deploy App at Release Performance Test Server Frequently 7a. Performance Test App
  • 85. cbell@CamilleBellConsulting.com 85 Only Check-in Working Tested Code Version Control Repository
  • 86. cbell@CamilleBellConsulting.com 86 Continuous Deployment Cycle Builds on Continuous Integration Developer Deploy Server Version Control Repository 1. Check-in 2. Monitor changes 3. Update Build Workspace 4. Build 5. Run Tests CI Server 6. Send Build & Test Report 7. Clean Build & Tests Pass Deploy App Immediately
  • 87. cbell@CamilleBellConsulting.com 87 Tools for the Deployment Pipeline Configuration Management Continuous Integration Testing Strategy Deployment Capistrano
  • 89. cbell@CamilleBellConsulting.com 89 Build-Measure-Lean with Lean Startup Split Tests Customer Interviews Customer Development 5 Whys Root Cause Analysis Customer Advisory Board Falsifiable Hypotheses Product Owner Accountability Customer Archetypes Cross-functional Teams Semi-autonomous Teams Smoke Tests Unit Tests Usability Tests Continuous Integration Incremental Deployment Free & Open-Source Components Cloud Computing Cluster Immune System Just-in-time Scalability Refactoring Developer Sandbox Split Tests Clear Product Owner Continuous Deployment Usability Tests Real-time Monitoring Customer Liaison Funnel Analysis Cohort Analysis Net Promoter Score Real-Time Altering Predictive Monitoring Measure Faster Learn Faster Code Faster
  • 90. cbell@CamilleBellConsulting.com 90 Examples of Testing of Ideas with Lean Startup • Split A/B Testing – controlled experiments – two versions of product released simultaneously – test metric comparison ($ earned, # of signups, etc.) – Netflix huge user of A/B testing • Crowd Testing: Validating the existence of a customer base – bypassing investors with Kickstarter, etc. (e.g. Torment) – creating dummy site to sign up and buy (e.g. Zappos)
  • 92. cbell@CamilleBellConsulting.com 92 Bug Growth on Non-Agile Projects T i m e B u g s code fix code codecode codecodecode fixfixfixfix fix release 1 release 2 release 3 release 4 release 5 release 6 release 7
  • 93. cbell@CamilleBellConsulting.com 93 TDD and BDD prevents a Great Many Bugs, but … a few will probably still occur Write a failing test for new functionality Write just enough code to pass test Refactored code must also pass tests, no new functionality cbell@CamilleBellConsulting.com 93
  • 94. cbell@CamilleBellConsulting.com 94 Create a RED test BEFORE fixing the bug, … so you are sure you really fixed it, and the bug never sneaks through the tests again. cbell@CamilleBellConsulting.com 94 Write a failing test that exposes the bug Write just enough code to fix the bug Refactor if needed, must also pass tests, no new functionality
  • 96. cbell@CamilleBellConsulting.com 96 • By acting as a servant leader and removing obstacles • By providing whole team training in agile processes • By providing technical training in automated test tools to current employees • By requiring automated test experience for new hires • By making testability a must have for languages & tools (many of the best are open source) • By providing the automated test tools themselves • By providing continuous integration and deployment servers • By tracking progress differently Management Needs to Support Agility
  • 97. cbell@CamilleBellConsulting.com 97 •  Agile processes like Scrum, Lean, Kanban and XP: •  often make short near term predictions •  track actual progress •  determine if the predictions, if made, were correct •  use actual progress metrics to project future outcomes •  adapt and adjust based on real outcomes • Managers must ensure that agile practices: •  help the team improve projections •  never become a stick Agility Organizational Processes Have Built-in Feedback Loops and Validation
  • 98. cbell@CamilleBellConsulting.com 98 Running Tested Features Definition 1.  The  desired  so,ware  is  broken  down  into  named  features   (requirements,  stories),  which  are  part  of  what  it  means  to   deliver  the  desired  system.     2.  For  each  named  feature,  there  are  one  or  more  automated   acceptance  tests  which,  when  they  work,  will  show  that  the   feature  in  quesAon  is  implemented.   3.  The  RTF  metric  shows,  at  every  moment  in  the  project,  how   many  features  are  passing  all  their  acceptance  tests.     Ron Jeffries, A Metric Leading to Agility
  • 99. cbell@CamilleBellConsulting.com 99 Calculating RTF is Simple •  No weighting - No partials credit: A Feature is either all running & passing all tests or it isn’t. •  So a feature either counts as 0 or 1. •  Add up the 1s. •  Run all tests for each measurement. •  Track over time.
  • 100. cbell@CamilleBellConsulting.com 100 RTF Works on any Project, even Waterfall RTF on Agile software development projects should steadily increase over time RTF on Waterfall development only increases at project end
  • 101. cbell@CamilleBellConsulting.com 101 Example 1: Buggy New Feature Breaks Old Monday 6 running tested features. Tuesday no new features added, but existing features still pass tests. On Wednesday, new feature added which fails test and causes side effect that causes pre- existing feature to also fail tests. On Thursday, new tests added to old feature and new feature fixed. All tests pass.
  • 102. cbell@CamilleBellConsulting.com 102 Example 2: New Feature Breaks Everything Monday 6 running tested features. Tuesday no new features added, but existing features still pass tests. On Wednesday, new feature added which breaks the build, crashes system, corrupts DB or something similarly catastrophic. On Thursday, new tests added and new feature fixed. All tests pass.
  • 103. cbell@CamilleBellConsulting.com 103 CUMULATIVE BURNUP StoryPointsorFeaturesComplete(RTF) Time Burnup of Story Points or Running Tested Features metrics can be more informative than burndowns. New Stories when added caused existing stories to fail automated tests Stories fixed, tests added, all tests now pass Stories Removed to meet Release DateStories Added New Baseline New Baseline Baseline ReleaseDate Complex Code needed refactoring Ref: Alistair Cockburn http://alistair.cockburn.us/Earned-value+and+burn+charts Ron Jeffries http://xprogramming.com/articles/jatrtsmetric/
  • 104. cbell@CamilleBellConsulting.com 104 Camille Bell Agile Coaching & Consulting Retrospectives Agile Boot Camps Agile Training Updated Slides or just to chat about things agile CamillesCareer@gmail.com
  • 106. cbell@CamilleBellConsulting.com 106 Code Coverage C0, C1 and C2 Coverage C0 – Line coverage analysis measures which lines of code have been executed. C0 coverage is typically used it to find the areas of your program that have not been sufficiently tested, i.e. those that were not run by any your test cases. C1 – Branch coverage analysis measures which of the different possible branches of conditional statements have been tested. It is easy to have 100% C0 line coverage and only partial C1 branch coverage, because if, then and if, then, else statements may be contained all on one line in many languages. C2 - Path coverage measures which of the different possible execution paths through your code were tested. Paths are a combination of linear code execution with alternate branches. Since each new conditional that is encountered gives rise to new path choices, the permutations of possible unique paths gets huge.
  • 107. cbell@CamilleBellConsulting.com 107 Code Coverage C0, C1 and C2 Tools C0- Every automated code coverage tool measures at least Line Coverage. Open source tools rcov for Ruby and Cobertura for Java provide C0 coverage. C1 - Many automated code coverage tools (both open source and commercial) measure Branch coverage. For instance open source tool Emma for Java measures both Line and Branch coverage as does Clover, a commercial Java test tool. C2 – Complete path coverage throughout an entire large application is almost unheard of because of its complexity and cost. I know of only two large software applications (both in Ada) that had 100% path coverage. Path coverage through multiple limited sections of code is much more feasible. All path coverage tools also include line and branch coverage. All are commercial and very expensive.
  • 108. cbell@CamilleBellConsulting.com 108 public class CoverageExample { // This code is logically equivalent to the next example, but because of the // code structure, untested branches are detected by line coverage. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // This test provides 100% Line coverage, but it doesn’t provide any branch coverage. public class CoverageTestLineOnly { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void testReturnInput0FalseFalseFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } } Code Coverage C0 – Line Coverage
  • 109. cbell@CamilleBellConsulting.com 109 Code Coverage C0 – Line Coverage Limitations public class CoverageExample // This code is logically equivalent to the prior example, but because of the // code structure, untested branches are not detected by line coverage. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // This test provides 100% Line coverage, but it doesn’t provide any branch coverage. public class CoverageTestLineOnly { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void allFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } }
  • 110. cbell@CamilleBellConsulting.com 110 Code Coverage C1 – Branch Coverage public class CoverageExample // This code is identical to the prior example. public int echo (int x, boolean state1, boolean state2, boolean state3) { if (state1) { x++; } if (state2) { x--; } if (state3) { x = x; } return x; } } // Adding the new test provide 100% branch coverage, but doesn’t detect the bug. public class CoverageTestLineAndBranch { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @Test public void allFalse() { assertEquals(0, lineBranchPath.echo(0, false, false, false); } @Test public void allTrue() { assertEquals(0, lineBranchPath.echo(0, true, true, true); } }
  • 111. cbell@CamilleBellConsulting.com 111 Code Coverage C2 – Path Coverage Explosion false false false false false true false true false false true true true false false true false true true true false true true true To test all possible paths of N Boolean conditionals is 2 ^ N. So for our example of 3 Booleans that’s 2x2x2 or 8. As shown in the example on the left, it’s not too bad. However, the set of all possible paths grows exponentially, so a combination of 10 Booleans, becomes 2x2x2x2x2x2x2x2x2x2 or 1024 possible paths, an unmanageable number of tests.
  • 112. cbell@CamilleBellConsulting.com 112 Code Coverage C2 – Basis Path Simplification false false false true false false false true false false false true Basis path testing is a simplification of path testing, that significantly lowers the number of needed tests A complete set of basis path sets are the number of boolean decisions +1. It grows linearly instead of exponentially. that’s 3+1 or 4 instead of 8 for our example and 10+1 or 11 instead of 1024. Path 1: Any path will do for the baseline, so pick all trues or all falses for simplification. We are picking false. This is the first path in our basis set. Path 2: To find the next basis path, we flip the first decision (only) in our baseline. Path 3: Next we flip the second decision (only) in our baseline path, the first baseline decision remains fixed with the false outcome. Path 4: Finally, you flip the third decision in your baseline path. Again, the first baseline decision remains fixed with the false outcome.
  • 113. cbell@CamilleBellConsulting.com 113 Code Coverage Line, Branch & Basis Path // JUnit 4 tests with 100% line, 100% branch coverage, and 100% basis path. public class CoverageTestPath { CoverageExample lineBranchPath; @Before public void runBeforeEveryTest() { lineBranchPath = new CoverageExample(); } @After public void runBeforeEveryTest() { lineBranchPath = null ; } @Test // First basis path. public void allFalse() { assertEquals ("All false inputs ", 0, lineBranchPath.echo(0, false, false, false); } @Test // Second basis path. public void trueFalseFalse() { assertEquals ("True-False-False inputs ", 0, lineBranchPath.echo(0, true, false, false); } @Test // Third basis path. public void falseTrueFalse() { assertEquals ("False-True-False inputs ", 0, lineBranchPath.echo(0, false, true, false); } @Test // Fourth basis path. public void falseFalseTrue() { assertEquals ("False-False-True inputs ", 0, lineBranchPath.echo, false, false, true); } }
  • 114. cbell@CamilleBellConsulting.com 114 Cucumber Code Coverage with RCov (today Simplecov would better tool)
  • 115. cbell@CamilleBellConsulting.com 115 RCov and Clover •  RCov, an open source Ruby Gem provides code coverage measurement for Cucumber •  RCov provides C0 – line code coverage •  Clover, a commercial tool provides code coverage measurement for JUnit •  Clover provides C1 – branch code coverage