WELCOME AT
AGILE NXT
FUTURE FRIDAY
#agilenxt
#futurefriday

MEASURE THE IMPACT
OF COACHING –
USING METHODS FROM
DATA SCIENCE
Pieter Rijken

Sports
Laid off coaches in soccer 2010-2014
Short term results
Long term results
0
20
40
60
80
Source: https://www.tussendelinies.nl/het-schokeffect-van-de-trainerswissel

Apparently the coach was wrong!
.....or not??
.....how can one tell??

Douglas W. Hubbard
Hard questions
1. How do you know if it [coaching] works?
2. Can someone [in the org] tell if it doesn’t work?
3. What is the impact if it doesn’t work?

What is the impact if it doesn’t
work?
By Power.corrupts - Own work, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=4769885
Known as Iatrogenesis
“brought forth by the healer”

Any parallel with Agile coaching?
...or interventions in general?
...or ‘improvement’ actions?

We need to compare against a
baseline
...for example no coaching
...or no interventions

Compared to the baseline,
the effect of coaching can be
1. Worse
2. Similar
3. Better
Perhaps coaching leads to worse results and no one notices!
Or.....coaching works and the organization concludes it doesn’t!

Virginia
Satir
Change
Model

Demand for Agile coaches since
2006
Source: https://www.itjobswatch.co.uk/jobs/uk/agile%20coach.do

We need to know when...
...coaching / interventions work
...when they do not
work
...who can tell us?

State of Agile Report 2017
(Version ONE)
• 65% using Agile practices for 3+ years
• 84% at or below ‘still maturing level’
• Agile practices are enabling greater adaptability to market conditions 4%
• 98% report success in Agile projects
• 74% indicate more than half of Agile projects are success
• No mention of coaching effectiveness

Organizational Behavior
Management
behavior
actions
business
results
Coaching
Measure!

What’s important to your customer?
• Crisp Scrum check list
• Kanban Litmus test
Source: Essential Kanban Condensed
Source:
https://www.crisp.se/wp-content/uploads/2012/05/Scrum-checklist.pdf

Services
Black Box
[Way of Working]
• Number of customer requests (stories, features, products,
releases, ...) fulfilled
• Number of projects, marketing campaigns, a/b tests, ...
Delivery Rate

Delivery rate
• important to the customer
• as seen by the customer
• very simple to measure
• not too many data points needed to get started
• look at the team / department / organization as a block box

Incoming emails
6
3
1
0
8
10
5
6
5
7
10
4
3
8
9
8
5
2
1
6
5
4 4
5
7
6
4
2
4
5
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Count
Week number
RECEIVED EMAILS JAN-JUL 2019
• Jan – Jul 2019
• Pattern or totally random?
Average = 5.0 emails per week

Pattern or totally random?
• It seems reasonable to assume:
• Independent (no conversations)
• Constant average rate
• Known as Poisson distribution

Examples that are Poisson
distributed
• Number of emails received per day (or per week)
• Number of cars passing a certain point
• Number of trains passing a station
• Number of goals in a soccer match
• Number of phone calls per day
• Number of customer requests completed per week
• Very relaxed conditions!

Let’s try!!
Hmmmm....

Uncertainty expressed with bars

Summary
• 153 emails in the period January – July 2019
• Grouped per week results in 30 data points
• In a histogram we are left with 10 data points
• Average rate between 4.8 and 5.2
• Despite variation rate is determined reasonably precise (within 10%)

What to watch out for
• Assumptions of the model
• independent
• constant delivery rate
• Bias (unconsciously) built into the model
We will use
this later on!

Team Mars
SCRUM
• ...uses Scrum framework with 2-week sprints
• ...delivers user stories
• ...part of a department of 6 (scrum) teams
• ...consists of specialists with slight collaboration
• ...does retrospectives every sprint
Delivery Rate

Application to team Mars
Frequency diagram

Observations
• Data consistent with a single Poisson distribution
• Stable for almost 1 year
• Great for forecasting based on delivery rate of user stories!!
• Despite the team doing retrospectives, the data shows no statistically
significant improvements as seen from the outside

Emails revisited: August
6
3
1
0
8
10
5
6
5
7
10
4
3
8
9
8
5
2
1
6
5
4 4
5
7
6
4
2
4
5
6
3
1
0
8
10
5
6
5
7
10
4
3
8
9
8
5
2
1
6
5
4 4
5
7
6
4
2
4
5
1 1
10 10
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
Count
Week number
RECEIVED EMAILS JAN-JUL 2019

Let’s try again!!
Mathematicians have an indicator
(statistic) for ‘how good’ the fit is.
Chi-squared per degree of freedom.
Ideally equals ~ 1
0.35 for January – July data
2.8 for January – August data
Too good
to be true
Not so
good

What to look for in data
6
3
1
0
8
10
5
6
5
7
10
4
3
8
9
8
5
2
1
6
5
4 4
5
7
6
4
2
4
5
6
3
1
0
8
10
5
6
5
7
10
4
3
8
9
8
5
2
1
6
5
4 4
5
7
6
4
2
4
5
1 1
10 10
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
Count
Week number
RECEIVED EMAILS JAN-JUL 2019
(a) change in delivery rate or (b) violation of independence

Method
• Use the delivery rate data
• Assume à priori a model(eg Poisson distribution) to correctly describe the
data
• Discover at what points in time this assumption is incorrect

Team Jupiter
Towards
SCRUM
• ...uses Scrum framework with 2-week sprints
• ...delivers tasks
• ...part of a larger tribe
• ...consists of specialists
• ...does retrospectives every sprint
Delivery Rate
Tasks Epics

Histogram
Data does not look like Poisson!
Indeed: chi-squared > 750K
Binning

Histogram
Periods of stability:
• January – May [rate of 3.5]
• October – December [rate of 1.2]
• July – September [rate of 3.3]
Transition period in December-January
MayJul
Start of new
organization
structure
Impactful
changes after
‘chaotic’ period

Split in periods
Jul - May
Jul - Sep Oct - Dec Jan - May

Observations
• Periods of change are recognized
• Helps to choose sequential subsets of data without trending
• In hindsight, transitions between periods correspond to major events of
change for the team
• Periods of stability (no significant change) are long; order of months

How long for change to be
noticeable?

Takeaways
• Supports coaches (and scrum masters) in learning which interventions
are effective and which are not,
• Compatibility of delivery rate as an indicator of significant changes
• Simple to use....you probably already have the data!
• Be patient!
• Beware of time delays in interventions
• Tool used for this presentation is online available as Jupyter Notebook
https://hub.docker.com/r/pietertje/chi2fit

Recipe to get started tomorrow
• Find a success criteria that matters to your customer
• Decide how to measure it
• Get data & model it
• Make a hypothesis (what results do you expect and when do you expect
results)
• Use statistical tools/tests to determine if change is significant!
• Test the hypothesis and learn from it!

THANK YOU!
WHAT IS YOUR
NEXT STEP IN
AGILITY?
powered by