Data driven continuous improvement

1. Unlocking Excellence with
Agile Metrics
Data-Driven Continuous Improvement
May 2016 – Lean Agile Glasgow
@MartinBurnsSCO
www.linkedin.com/in/MartinBurnsUK

2. Avoiding EEEEEVIL with
Agile Metrics
May 2016 – Lean Agile Glasgow
@MartinBurnsSCO
www.linkedin.com/in/MartinBurnsUK

3. 3 © 2015 CA. ALL RIGHTS RESERVED.
Martin Burns
Lives in Edinburgh but
spends too much time on
planes
Improvement obsessive
and caffeine addict
Been doing incremental
and iterative since the 90s.
The usual certifications
and many logos meant to
impress you
Lapsed Partick Thistle
Fan.

4. 4 © 2015 CA. ALL RIGHTS RESERVED.
In God we trust
All others
bring data
W. Edwards Deming

5. 5 © 2015 CA. ALL RIGHTS RESERVED.
Without data our decision-making process is corrupted
Source: http://www.vectorfreebie.com/wp-content/uploads/2016/01/Heart-Sketch.png
https://s-media-cache-ak0.pinimg.com/736x/40/05/a7/4005a76193de7788886ea0d0339284f8.jpg
Canberra
Sydney
Perth
Melbourne
Brisbane
London
Adelaide
Glasgow
Hobart
Auckland
546mm
557mm
569mm
597mm
603mm
725mm
1115mm
1158mm
1223mm1124mm

6. 6 © 2015 CA. ALL RIGHTS RESERVED.
The first step is admitting you have a problem…

7. 7 © 2015 CA. ALL RIGHTS RESERVED.
• We are a data-driven company
• This is data
• Therefore
• We should be driven by it

8. 8 © 2015 CA. ALL RIGHTS RESERVED.
Oh my!

9. 9 © 2015 CA. ALL RIGHTS RESERVED.

10. 10 © 2015 CA. ALL RIGHTS RESERVED.
Why are we collecting Data?
To make decisions

11. 11 © 2015 CA. ALL RIGHTS RESERVED.
Question:
What Metrics are
Coveted/Abused/Over-used
in Your Organisation?

12. 12 © 2015 CA. ALL RIGHTS RESERVED.
Common Metrics Traps
What will impede our outcomes?

13. 13 © 2015 CA. ALL RIGHTS RESERVED.
Common Metrics Traps
Manipulation
Unbalanced metrics
Availability bias
Mechanical decision-making
False positives/negatives
Lagging vs Leading Indicators
Mean, Median and Mode
- 13 -

14. 14 © 2015 CA. ALL RIGHTS RESERVED.
Summary Stats
§ The average x value is 9 for each
dataset
§ The average y value is 7.50 for each
dataset
§ The variance for x is 11 and the
variance for y is 4.12
§ The correlation between x and y is
0.816 for each dataset
§ A linear regression (line of best fit)
for each dataset follows the
equation y = 0.5x + 3
http://data.heapanalytics.com/anscombes-quartet-and-why-summary-statistics-dont-tell-the-whole-story

15. 15 © 2015 CA. ALL RIGHTS RESERVED.
Summary Stats

16. 16 © 2015 CA. ALL RIGHTS RESERVED.
Poor Data Integrity
Image coursesy of Flickr

17. 17 © 2015 CA. ALL RIGHTS RESERVED.
Beware the One-Metric to Solve Them All

18. 18 © 2015 CA. ALL RIGHTS RESERVED.
Photo courtesy of Flickr : Creative Commons

19. 19 © 2015 CA. ALL RIGHTS RESERVED.
if management sets
quantitative targets and
makes people’s job
depend on meeting them,
they will likely meet the
targets – even if they have
to destroy the enterprise
to do it.
W. Edwards Deming

20. 20 © 2015 CA. ALL RIGHTS RESERVED.
Photo courtesy of Flickr : Creative Commons

21. 21 © 2015 CA. ALL RIGHTS RESERVED.
How Do I Unravel This All
What metrics help us achieve our desired outcomes?

22. 22 © 2015 CA. ALL RIGHTS RESERVED.
First: You Need To Organize Based on
Outcomes
••How Deep/Wide is our Agile?
Adoption Outcomes
••Are We Delivering Value Faster?
Operations Outcomes
••Are We Meeting Expectations?
Business Outcomes

23. 23 © 2015 CA. ALL RIGHTS RESERVED.
Second– You Need Context (2 dimensions)
Q1: How Mature are the Agile Practices
of Your Group
(Team/Program/Department/Business
Unit)?
Just
Getting
Started
We’re
Doing It
and …..
We’re
Doing
Our Best
and We
Can
Improve

24. 24 © 2015 CA. ALL RIGHTS RESERVED.
Q2: Which Layer of the
Business System are we
focused on?
Second– You Need Context (2 dimensions)

25. Activity
Q. What Metrics are you Using
to Track Agile Adoption?

26. 26 © 2015 CA. ALL RIGHTS RESERVED.
Now Let’s Zero in on Operational
Metrics
Steering not Aiming

27. 27 © 2015 CA. ALL RIGHTS RESERVED.
numerical goals set
for other people,
without a road map
to reach the goal,
have effects opposite
to the effects sought.
W. Edwards Deming
Out of the Crisis, p69

28. 28 © 2015 CA. ALL RIGHTS RESERVED.
not aiming …
… but steering
Photo courtesy of Flickr : Creative Commons

29. 29 © 2015 CA. ALL RIGHTS RESERVED.
Focus at the right level
0
500
1000
1500
2000
2500
3000
3500
07/04/2015
14/04/2015
21/04/2015
28/04/2015
05/05/2015
12/05/2015
19/05/2015
26/05/2015
02/06/2015
09/06/2015
16/06/2015
23/06/2015
30/06/2015
All Trains - Story Count CFD – Release 3
Done Ready To Validate In Progress To Do Blocked
0
50
100
150
200
250
300
350
400
14/04/2015
21/04/2015
28/04/2015
05/05/2015
12/05/2015
19/05/2015
26/05/2015
02/06/2015
09/06/2015
16/06/2015
23/06/2015
30/06/2015
07/07/2015
All Trains - Features CFD – Release 3
Done In Progress To Do

30. 30 © 2015 CA. ALL RIGHTS RESERVED.

31. 31 © 2015 CA. ALL RIGHTS RESERVED.
Some of My Favorites: Do It Fast
Responsiveness: Feature Cycle Time
Velocity (Planned and Actual) / Work Capacity
e.g., In Planning we average 6 stories planned per human
per sprint. We finish about 2 stories per human per sprint
Productivity: Focus Factor *
http://www.scruminc.com/wp-content/uploads/2014/05/Hyper-Productive-Metircs.pdf

32. 32 © 2015 CA. ALL RIGHTS RESERVED.
Some of My Favorites:
Do It On Time: Sprint Predictability Ranges
Keep Doing It: Employee NPS Score

33. 33 © 2015 CA. ALL RIGHTS RESERVED.
Some of My Favorites:
Trade Me talk at Agile ANZ and nod to Dan Pink (include you Tube video)
Keep Doing It: Autonomy, Mastery and Purpose

34. 34 © 2015 CA. ALL RIGHTS RESERVED.
Some of My Favorites:
Keep Doing It: Simple Mood

35. 35 © 2015 CA. ALL RIGHTS RESERVED.
Metrics Mantra #1: Teams Execute
Will we complete our work in a sprint
(stories, tasks, defects, etc.)?

36. 36 © 2015 CA. ALL RIGHTS RESERVED.
Teams Execute
- 36 -

37. 37 © 2015 CA. ALL RIGHTS RESERVED.
Teams: Consistency over Throughput

38. 38 © 2015 CA. ALL RIGHTS RESERVED.
Metrics Mantra #2: Programs
Deliver Value
Will we meet our plan objectives
(features, milestones, quality, etc.)?

39. 39 © 2015 CA. ALL RIGHTS RESERVED.
Programs Deliver Value – Dashboard

40. 40 © 2015 CA. ALL RIGHTS RESERVED.
Programs Deliver Value – Feature Progress

41. 41 © 2015 CA. ALL RIGHTS RESERVED.
Programs Deliver Value – Timeline View

42. 42 © 2015 CA. ALL RIGHTS RESERVED.
Metrics Mantra #3: Portfolios Steer

43. 43 © 2015 CA. ALL RIGHTS RESERVED.
Portfolios Steer – Capacity Modeling

44. 44 © 2015 CA. ALL RIGHTS RESERVED.
Metrics Mantra #4: Occasionally
Glance Backward

45. 45 © 2015 CA. ALL RIGHTS RESERVED.
Variance not Average

46. 46 © 2015 CA. ALL RIGHTS RESERVED.
One final thought
Photo courtesy of Flickr : Creative Commons
http://cultivatingthought.com/author/aziz-ansari/

47. 47 © 2015 CA. ALL RIGHTS RESERVED.
ROTI – return on time invested (this session)
Source: http://blogs.adobe.com/agile/files/2013/12/hands-counting.jpg
https://s-media-cache-ak0.pinimg.com/736x/4e/5c/f7/4e5cf7d4ccb9c59b6620a9c71944d51e.jpg
https://dq9ucdqjq6nrk.cloudfront.net/wp-content/uploads/2014/02/01_sad_emoticon-300x269.jpg
https://s-media-cache-ak0.pinimg.com/736x/e9/9b/54/e99b541789969921ebd0110309a35f63.jpg

48. Senior Principal Transformation Consultant
Martin.burns@ca.com
Martin Burns
MartinBurnsSCO
Suzanne Nottage