With a fragile test suite, the Continuous Testing that's vital to agile just isn't feasible. If you truly want to automate the execution of a broad test suite—embracing unit, component, integration, functional, performance, and security testing—during continuous integration, discover the tips to ensure your test suite is up to the task. Logically-componentized: Tests need to be logically-componentized so you can assess the impact at change time. When tests fail and they're logically correlated to components, it is much easier to establish priority and associate tasks to the correct resource. Incremental: Tests can be built on each other, without impacting the integrity of the original or new test case. Repeatable: Tests can be executed over and over again with each incremental build, integration, or release process. Presented at Better Software Conference East 2014 (Agile Development Conference East 2014).

1. Paraso&
Proprietary
and
Confiden1al
1
11/13/14
Are
Your
Con*nuous
Tests
Too
Fragile
for
Agile
Arthur
Hicken
Paraso&
Evangelist

2. Paraso&
Proprietary
and
Confiden1al
2
Paraso&
Proprietary
and
Confiden1al
2
Agenda
§ Challenges
of
Con1nuous
Agile
§ Tes1ng
defined
§ Tes1ng
is
hard
(no
one
wants
to
do
it)
§ Best
prac1ces
§ A
modest
proposal:
Spring
cleaning

3. Paraso&
Proprietary
and
Confiden1al
3
Paraso&
Proprietary
and
Confiden1al
3
The
role
of
QA
in
Agile
§ QA
is
part
of
development,
not
an
a&erthought
§ Review
developer
tes1ng
efforts
§ Addi1onal
tes1ng
of
requirements
§ Itera1ve
during
development
§ Fast
discovery
and
repor1ng
of
defects
§ Inves1gate
root
cause
of
defects
§ Influence
development
policy

4. Paraso&
Proprietary
and
Confiden1al
4
Paraso&
Proprietary
and
Confiden1al
4
Challenges
of
Agile
§ Need
to
be
fast
§ Need
to
be
flexible
§ Tes1ng
can
be
overlooked
if
not
baked
in
§ Architectural
compromises
from
schedule

5. Paraso&
Proprietary
and
Confiden1al
5
Paraso&
Proprietary
and
Confiden1al
5
Challenges
of
Con1nuous
§ Tests
must
produce
binary
decision
go/no-­‐go
§ Reuse
unit
test
and
func1onal
tests
from
dev
to
QA
§ High
level
of
automa1on
§ Requires
disciplined
mature
process
§ Tes1ng
must
automa&cally
answer
§ Is
it
stable
§ Will
it
do
what
it’s
supposed
to
do

6. Paraso&
Proprietary
and
Confiden1al
6
Paraso&
Proprietary
and
Confiden1al
6
Importance
of
Tes1ng
§ How
do
you
know
when
you're
done?
§ How
do
you
know
if
a
fix
for
a
minor
bug
broke
a
major
func1on
of
the
system?
§ How
can
the
system
evolve
into
something
more
than
is
currently
envisioned?
§ Tes1ng,
both
unit
and
func1onal,
needs
to
be
an
integrated
part
of
the
development
process.

7. Paraso&
Proprietary
and
Confiden1al
7
Paraso&
Proprietary
and
Confiden1al
7
Tes1ng
delivers
A
maintained
suite
of
unit
tests:
§ Represents
the
most
prac1cal
design
possible
§ Provides
the
best
form
of
documenta1on
for
classes
§ Determines
when
a
class
is
"done"
§ Gives
a
developer
confidence
in
the
code
§ Is
a
basis
for
refactoring
quickly

8. Paraso&
Proprietary
and
Confiden1al
8
Paraso&
Proprietary
and
Confiden1al
8
Why
Tes1ng
is
O&en
Ignored
§ Perceived
as
“burdensome”
§ Too
much
effort
§ Too
much
1me
§ We’re
on
a
schedule
–
no
1me
to
test.
§ Why
is
there
always
1me
to
do
it
over,
but
never
to
do
it
right?
§ Developer
Ego
§ “I
don’t
need
to
test
my
code
–
I
don’t
make
mistakes.”
§ Tes1ng
means
I
have
to
admit
that
I
might
have
made
a
mistake.
§ I
don’t
like
to
test
–
it’s
beneath
me.
§ Tes1ng
isn’t
fun
–
I
just
want
to
code!
§ That’s
for
the
QA
dept.

9. Paraso&
Proprietary
and
Confiden1al
9
Paraso&
Proprietary
and
Confiden1al
9
Defini1ons
§ Unit
§ Func1onal
§ Regression

10. Paraso&
Proprietary
and
Confiden1al
10
Paraso&
Proprietary
and
Confiden1al
10
Unit
Tes1ng
Unit
tests
that
validates
your
method
behaves
a
certain
way
given
a
specific
parameters
§ Regressions
tes1ng
§ Validate
behavior
of
exis1ng
code
hasn’t
changed
§ Scheduled
and
run
automa1cally
§ Test
Driven
Development
§ Validate
behavior
of
new
code
to
be
wrijen
§ Created
before
developing,
turned
into
regression
tests
a&er

11. Paraso&
Proprietary
and
Confiden1al
11
Paraso&
Proprietary
and
Confiden1al
11
Func1onal
Tes1ng
§ Incarna1ons
of
func1onal
tests
§ Unit
tests
(Junit,
CppUnit,
Nunit,
etc)
§ Integra1on
tests
§ Scripted
tests
§ Manual
tests
§ Ways
to
create
func1onal
tests
§ Manually
wrijen
§ Machine
generated
from
applica1on
tracing
§ Machine
generated
from
capturing
traffic
§ Machine
generated
from
recording
clicks

12. Paraso&
Proprietary
and
Confiden1al
12
Paraso&
Proprietary
and
Confiden1al
12
Regression
Tes1ng
§ Contributors
to
the
regression
test
suite
§ Func1onal
unit
tests
that
pass
§ Machine
generated
unit
tests
for
current
behavior
§ Unit
test
wrijen
by
developers
and
testers
§ Tests
wrijen
to
validate
bug
fixes
§ Regression
test
failures
§ Unwanted
change
in
behavior
–
fix
the
code
§ Inten1onal
change
in
specifica1on
–
update
test
controls

13. Paraso&
Proprietary
and
Confiden1al
13
Paraso&
Proprietary
and
Confiden1al
13
What’s
the
Difference?
Unit
Tests
• The
code
is
doing
things
right
• Wrijen
from
programmers
perspec1ve
• Confirm
code
gives
proper
output
based
on
specific
input
Func1onal
Tests
• The
code
is
doing
the
right
thing
• Wrijen
from
users
perspec1ve
• Confirm
the
system
does
what
the
user
expects
it
to
do
Regression
Tests
• It
s1ll
works

14. Paraso&
Proprietary
and
Confiden1al
14
Paraso&
Proprietary
and
Confiden1al
14
Why
do
Unit
Tes1ng
So&ware
bugs
are
inserted
during
coding
phase,
but
o&en
aren’t
found
un1l
func1onal
tes1ng,
or
worse
they
are
found
by
customers.
The
longer
a
bug
stays
in
the
system,
the
more
expensive
it
is
to
fix
it.
Unit
tes1ng
is
a
strategy
that
allows
you
to
find
out
if
your
system
is
behaving
the
way
you
expect
early
in
the
SDLC.
Unit
tes1ng
allows
for
easy
regression
tes1ng,
so
you
can
find
out
quickly
whether
you
have
introduced
some
unexpected
behavior
into
the
system.

15. Paraso&
Proprietary
and
Confiden1al
15
The
Cost
of
Defects

16. Paraso&
Proprietary
and
Confiden1al
16
Paraso&
Proprietary
and
Confiden1al
16
Best
Prac1ces

17. Paraso&
Proprietary
and
Confiden1al
17
Paraso&
Proprietary
and
Confiden1al
17
What’s
Important
§ Unit
tests
require
a
LOT
of
care
§ Unit
tes1ng
is
a
con1nuous
opera1on
§ Test
suite
maintenance
is
a
commitment
§ Control
data
(asser1ons)
in
the
code

18. Paraso&
Proprietary
and
Confiden1al
18
Paraso&
Proprietary
and
Confiden1al
18
Test
incorpora1on
§ Take
exis1ng
unit
tests
and
schedule
them
into
build
process
§ Run
and
fix
tests
as
part
of
build
process
§ Ideal
goal:
0
failures
and
100%
code
coverage
§ Realis1c
goal:
10%
test
failures
and
80%
code
coverage
§ If
the
unit
test
suite
is
noisy,
don’t
qualify
the
build
based
on
asser1on
failures

19. Paraso&
Proprietary
and
Confiden1al
19
Paraso&
Proprietary
and
Confiden1al
19
Unit
Test
Crea1on
§ Maintainability
§ Proper
asser1ons
§ Nightly
runs
§ Tests
should
be:
§ Automa1c
(no
human
input
or
review)
§ Repeatable
(no
strange
dependencies)

20. Paraso&
Proprietary
and
Confiden1al
20
Paraso&
Proprietary
and
Confiden1al
20
Legacy
Test
Suites
§ Leverage
your
exis1ng
suite
§ Clean
up
incrementally
§ Set
policy
based
on
current
results
§ Tighten
policy
when
in
compliance

21. Paraso&
Proprietary
and
Confiden1al
21
Paraso&
Proprietary
and
Confiden1al
21
Maintaining
and
Crea1ng
UT
§ Maintaining
is
more
of
a
problem
than
crea1ng.
§ Don’t
create
a
test
if
you
can’t
or
don’t
plan
to
maintain
it.
§ Crea1ng
as
you
go
creates
a
bejer
test
suite
§ Tests
not
run
regularly
become
irrelevant

22. Paraso&
Proprietary
and
Confiden1al
22
Paraso&
Proprietary
and
Confiden1al
22
Write
tests
with
the
code
§ Create
the
test
when
you’re
wri1ng
the
code
§ Have
a
test
for
every
task
you
complete
§ Increases
understanding
§ Improve
asser1ons

23. Paraso&
Proprietary
and
Confiden1al
23
Paraso&
Proprietary
and
Confiden1al
23
Connect
Unit
Tests
§ Associate
tests
with
Code
and
Requirements
§ Naming
Conven1ons
should
have
meaning
§ Use
code
comment
tags
@task
@pr
@fr
§ Use
check-­‐in
comments
@task
@pr
@fr
§ Improve
change-­‐based
tes1ng

24. Paraso&
Proprietary
and
Confiden1al
24
Paraso&
Proprietary
and
Confiden1al
24
Specific
Genera1on
§ Don’t
be
lured
by
100%
coverage
§ Auto
asser1ons
are
not
meaningful
unless
validated
§ Review
the
tests
as
you
work
on
the
code

25. Paraso&
Proprietary
and
Confiden1al
25
Paraso&
Proprietary
and
Confiden1al
25
Code
Review
Test
§ Code
for
unit
tests
should
be
reviewed
§ Consider
doing
test-­‐design
review
§ Ensures
you’re
tes1ng
what
you
think
you
are
§ Helps
reduce
noise
§ Helps
improve
understanding
§ Review
the
code
and
the
test
code
together

26. Paraso&
Proprietary
and
Confiden1al
26
Paraso&
Proprietary
and
Confiden1al
26
Tes1ng
Excep1on
Handling
§ Frequently
Skipped
§ Prevents
crashes
§ Prevents
death
spiral
§ Rule
of
thirds
§ 1/3
basic
func1onality
§ 1/3
advanced
features
/
infrequent
serngs
§ 1/3
error
handling

27. Paraso&
Proprietary
and
Confiden1al
27
Paraso&
Proprietary
and
Confiden1al
27
All
About
Asser1ons
§ Verifies
that
the
code
behaves
as
intended
§ True/false
§ Null
or
not
null
§ Specific
values
§ Complex
condi1ons

28. Paraso&
Proprietary
and
Confiden1al
28
Paraso&
Proprietary
and
Confiden1al
28
Why
Assert?
§ Excuses:
§ My
code
already
has
console
output
§ I
can
check
it
visually/manually
§ Binary
output
(pass/fail)
§ Part
of
overall
test
suite
§ No
human
interven1on

29. Paraso&
Proprietary
and
Confiden1al
29
Paraso&
Proprietary
and
Confiden1al
29
Asser1on
failures
Introduced
coding
error
• Fix
the
new
code
Func1onality
changed
• Update
the
asser1on
Improper
asser1on
• Comment
it
out

30. Paraso&
Proprietary
and
Confiden1al
30
Paraso&
Proprietary
and
Confiden1al
30
Asser1on
Policy
§ Assign
failures
as
tasks
§ Author
of
the
test
or
of
the
changed
code
§ Try
to
reach
developer
that
has
the
highest
chance
to
understand
the
change
§ Developer
assesses
failure:
§ Bug
(fix)
§ Changed
func1onality
(update
asser1on)
§ Bad
asser1on
(improve
or
remove)
-­‐
don’t
delete;
instead,
comment
out

31. Paraso&
Proprietary
and
Confiden1al
31
Paraso&
Proprietary
and
Confiden1al
31
What
to
assert
§ Ease
of
crea1ng
asser1ons
leads
to
poor
choices
§ Anything
CAN
be
checked
–
but
should
it
be?
§ Asser1ons
should
be
connected
to
what
you’re
trying
to
test
§ Asser1ons
should
be
logically
invariant
§ Check
values
by
range,
not
equivalence
§ “What
do
I
expect
here
logically?”

32. Paraso&
Proprietary
and
Confiden1al
32
Paraso&
Proprietary
and
Confiden1al
32
Asser1on
Example:
Java
public
Integer
getNumber()
{
Scanner
input
=
new
Scanner(
System.in
);
System.out.print(
"Enter
a
number
between
0
and
10:
"
);
int
number
=
input.nextInt();
//
assert
that
the
absolute
value
is
>=
0
assert
(
number
>=
0
&&
number
<=
10
)
:
"bad
number:
"
+
number;
return
number
}

33. Paraso&
Proprietary
and
Confiden1al
33
Paraso&
Proprietary
and
Confiden1al
33
How
to
recognize
bad
asser1ons
§ How
o&en
do
you
WANT
to
manually
verify
an
asser1on?
§ Daily
or
near-­‐daily
checking
indicates
poor
asser1on.
§ Troublesome
possibili1es:
§ Date
§ User
or
machine
name
§ OS
§ Filesystem
names
and
paths

34. Paraso&
Proprietary
and
Confiden1al
34
Paraso&
Proprietary
and
Confiden1al
34
Missing
Asser1ons
§ Error
was
found
a&er
code
changed
§ Test
suite
never
caught
the
error
§ Update
unit-­‐test
with
new
asser1on
§ Write
new
unit
tests
as
necessary

35. Paraso&
Proprietary
and
Confiden1al
35
Paraso&
Proprietary
and
Confiden1al
35
Cleaning
It
Up

36. Paraso&
Proprietary
and
Confiden1al
36
Paraso&
Proprietary
and
Confiden1al
36
The
Challenge:
Management’s
Perspec1ve
§ Need
tests
to
reduce
risks
§ Tes1ng
takes
1me
from
features/fixes
§ Not
enough
1me
§ Not
enough
people
§ Tests
and
reports
lack
context

37. Paraso&
Proprietary
and
Confiden1al
37
Paraso&
Proprietary
and
Confiden1al
37
The
Challenge:
Development’s
Perspec1ve
§ Hundreds
of
failing
or
broken
tests
§ Test
requirements
unclear
§ Exis1ng
tests
are
dubious
§ Deadline
pressure

38. Paraso&
Proprietary
and
Confiden1al
38
Paraso&
Proprietary
and
Confiden1al
38
Spring
Cleaning
Solu1on
§ Throw
out
the
useless,
old
junk
§ Iden1fy
and
eliminate
“noisy”
tests
§ Organize
the
remaining
chores
§ Priori1ze
unit
test
tasks
§ Keep
it
clean
§ Establish
and
follow
a
rou1ne
test
maintenance
plan

39. Paraso&
Proprietary
and
Confiden1al
39
Paraso&
Proprietary
and
Confiden1al
39
Throw
out
the
Junk:
Iden1fying
Noisy
Tests
§ Tests
that
are
fragile
§ Pass/Fail/Incomplete
results
frequently
change
§ Frequent
changes
are
required
to
keep
up-­‐to-­‐date
§ Tests
that
are
outdated
§ More
than
“n”
releases
have
occurred
despite
this
test
failure
§ The
test
applies
to
legacy
code
that
is
frozen
§ The
test
has
not
been
executed
for
“x”
period
of
1me
§ Tests
that
are
unclear
§ It
takes
more
than
5
minutes
to
understand
why
the
test
exists
§ The
failure
message
is
incorrect/unhelpful/irrelevant
§ There
is
no
associated
requirement
or
defect
§ Be
merciless!

40. Paraso&
Proprietary
and
Confiden1al
40
Paraso&
Proprietary
and
Confiden1al
40
Organize
Chores:
Priori1ze
Tasks
with
Policy
§ All
tasks
are
created
equal
§ Some
tasks
are
more
equal
than
others
§ Policies
establish
priori1za1on
and
assignment

41. Paraso&
Proprietary
and
Confiden1al
41
Paraso&
Proprietary
and
Confiden1al
41
Keep
it
Clean:
Establish
a
Rou1ne
Process
§ Management
and
Development
need
to
co-­‐define
a
Policy
§ Clear
tes1ng
goals
that
1e
tests
to
business
needs
§ Clear
delinea1on
of
responsibility
§ Test
crea1on
and
dele1on
§ Test
failure
resolu1on
§ Naming
conven1ons
§ Peer
review
§ Clear
and
frequent
communica1on
§ Understand
the
technical
and
business
impact
§ Enhance
traceability
to
project
requirements
§ Ask
ques1ons
§ Focus
on
safety/quality,
not
coverage
§ Keep
the
number
of
tests
minimal
§ Run
all
tests
on
a
regularly
scheduled,
frequent
basis
§ Clean
as
you
go

42. Paraso&
Proprietary
and
Confiden1al
42
Paraso&
Proprietary
and
Confiden1al
42
Spring
Cleaning
Recap
§ Find
and
delete
“noisy”
tests
§ Use
policies
to
priori1ze,
assign,
and
plan
§ Co-­‐design
and
follow
a
process
to
keep
it
clean

43. Paraso&
Proprietary
and
Confiden1al
43
Paraso&
Proprietary
and
Confiden1al
43
§ Web
§ hjp://www.paraso&.com/jsp/resources
§ Blog
§ hjp://alm.paraso&.com
§ Social
§ Facebook:
hjps://www.facebook.com/paraso&corpora1on
§ Twijer:
@Paraso&
@MustRead4Dev
@CodeCurmudgeon
§ LinkedIn:
hjp://www.linkedin.com/company/paraso&
§ Google+:
+Paraso&
+ArthurHickenCodeCurmudgeon
§ Google+
Community:
Development
Tes1ng
Because
its
Bejer