Beware The Brittle Dragon - A Test Automation Epic Tale

🐉 Beware the Brittle Dragon - Luca Giovenzana & Angelo Caovilla

Moral of the Story

What exactly is a
brittle test?

What exactly is a brittle test?
Fragile, flaky, easy to break
Not deterministic result
Run 1
Input
Run 2
Run 3
Run 4
Not due to only the test code

Raise your hand if
you ever had to deal
with brittle tests!

What are the
consequences
of brittle tests?

High Maintenance Cost
Keyword
test 1
test 2
test 3
test 4
75% Fail
Rollercoaster result trend

High Maintenance Cost
Debug and manual test
Fix broken tests

Real Example
Test 1...50 Precondition Step Assert
Nightly regression tests (1000+)
Most of the time working
This time many tests failed
RELEASE DROP
What would you do?

Hidden Fail
Test 1 Precondition Step Assert
Cognitive bias: test fails only because of the precondition
What if a real fail has been hidden?
Test 50 Precondition Step Assert BUG!

Useless feedback

Low Trust - Low Morale
Worthless feedback ignored
Maintenance avoided
Tests get even more broken
Automation is not trusted
V
i
c
i
o
u
s
L
o
o
p
Effort with no
results
No result ==
No commitment

What makes our
tests brittle?

How can we defeat
the Dragon?

Ice Cream
Testing Pyramid Real world: Ice Cream
Practical Test Pyramid Ham Vocke - Martin Fowler’s blog

Ice Cream
Move
Down to the Pyramid
Tests
Part of tests
Setup & Teardown
Diamo una chance alle codebase legacy Nicola Mincuzzi

Multipurpose / Holistic
Test 1 Setup Register Assert
Registration test:
Login Assert ... Assert

“There should never be more than
one reason for a class to change”
Robert C. Martin (Uncle Bob)
“There should never be more than
one reason for a test to fail”
Single Responsibility Principle

Test 1 Registration Check User
Single assert
Single Responsibility
Setup 1 Checkout
Setup 2 Streaming
Test 1 Registration
Test 2 Login
Test one thing

Poor Design
Sleep
Best effort automation
Quick and Dirty
Longer learning curve

Poor Design
Invest in automation
Agree on development guidelines
(Pekka’s how to write good test cases)
How to write good test cases using Robot Framework Pekka Clark

Poor Design
Reduce technical debt
Pair development
Boy Scout Rule:
“Always leave the code you are editing better
than you found it” Robert C. Martin (Uncle Bob)

Good and Bad Technical Debt
“Anything about your code that slows you down over the long term.
Hard-to-read code, duplication, tangled dependencies, etc.”
Good And Bad Technical Debt Henrik Kniberg

Unreliable and slow environment
Undersized
Environment
Front End Developer
Dev - Ops experiment
Business test
Backend Mad Developer
(internal server error hero)
Super complicated auto implosion mega query
Third part never-working services

Dedicated test environment
Pay for limitless resources

Mock third party systems
Strive for performances

Hard coded data
Limited products
Test product: 3E24B278-D1EC-358C-ADE5-F51703BBB342

Hard coded Data
Data Preparation
Data Targeting
Teardown Matters

No continuous Testing
Test suite needs to be executed
People is lazy forget to run
Not comparable results

Continuous testing to collect feedback

Remove/reduce barriers to test execution

Daily Standup
Include test feedback
in release process
Discuss reports together

Happy Ending

Automate everything

Quality assistance
Whole team in charge for quality
More TDD and pair programming
Quality Assistance over
Quality Assurance
BDD and TDD Outside in

Robot Framework BDD
*** Settings ***
Library SeleniumLibrary
Resource settings.resource
*** Test Cases ***
App Game Available
Given The Browser
When Navigate To The App Game
Then Title Should Be Tic Tac Toe
*** Keywords ***
Given The Browser
Open Browser ${EMPTY} ${BROWSER}
When Navigate To The App Game
Go To ${APP_URL}

Quality Assistance Team
Angelo Caovilla
Roberta Massimino
Fabrizio Premoli
Emanuele Bonanno
Art Director
Matteo Villa
All CHILI ex-colleagues
Special Thanks to

Contacts
Angelo Caovilla
acaovilla
Robot Framework Milano
lucagiove
Luca Giovenzana
@luboxit

Beware The Brittle Dragon - A Test Automation Epic Tale

Recommandé

Recommandé

Contenu connexe

Similaire à Beware The Brittle Dragon - A Test Automation Epic Tale

Similaire à Beware The Brittle Dragon - A Test Automation Epic Tale (20)

Dernier

Dernier (20)

Beware The Brittle Dragon - A Test Automation Epic Tale