Contenu connexe Similaire à Measuring Agility: Top 5 Metrics And Myths (16) Measuring Agility: Top 5 Metrics And Myths1. Measuring Agility
Top 5 Metrics and Myths
Pete Behrens
Agile Organization & Process Coach
© 2009 Trail Ridge Consulting, LLC pete@trailridgeconsulting.com 303.819.1809
2. Pete Behrens
Agile Organization & Process Coach
Certified Scrum Trainer
Certified Scrum Coach
Guide enterprise organizations in transitioning to an agile
organization implementing agile methods
Services for agile assessment, alignment, training and coaching
Previous Experience
Led development of the requirements management solution
RequisitePro – a core product in the IBM Rational product line – using
the Rational Unified Process (RUP)
Consulted with EDS leading development of large data warehouse
solutions using Rapid Application Development (RAD)
© 2009 Trail Ridge Consulting, LLC 2
4. How are projects measured today?
Failed
23%
Median
Overrun
Succeeded
28%
Cost 50%
Challenged
49%
Schedule 100%
• On Time
• On Budget The average project
• With all initially planned features Costs 50% more and takes
twice as long as planned
Source: Chaos Report, Standish Group, 2001
© 2009 Trail Ridge Consulting, LLC 4
5. Traditional project visibility is
often too late
Surprise !
n
la
tP
s
Features
Te
al
t&
ctu
en
t A
pm s
Te
lo
&
ve
t
en
De
m
e lop
D ev
Analysis & Requirements
Time
© 2009 Trail Ridge Consulting, LLC 5
6. Agile seeks transparency from
the outset of the project
Surprise !
n
la
Features
stP
Te
i ty
ibil
t&
is al
l eV en ctu
i
m
A
Ag st
p
lo
Te
ve
&
De
e nt
m
e lop
v
Analysis & Requirements De
Sprint 1 Sprint 2 Sprint 3 Sprint 4
Time
© 2009 Trail Ridge Consulting, LLC 6
7. Predictability Metric - Velocity
25
20 Average = 19
15
10
Teams will tend toward a
5 consistent velocity after a
few sprints if the team and
domain stay consistent
0
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7 Sprint 8
© 2009 Trail Ridge Consulting, LLC 7
8. Sprints drive predictability
Traditional Project
Predictable Uncertain Unpredictable
Agile Project with Timeboxed Iterations
Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint
Project Timeline
Definition:
Sprint = Iteration = Timebox
© 2009 Trail Ridge Consulting, LLC 8
9. Velocity - Advanced Burn Down
Measures team
velocity (work
complete per sprint)
Measures scope
change over time
Guides release-level
decision making
© 2009 Trail Ridge Consulting, LLC 9
10. Predictability Metric -
On Time Delivery
Last non-agile release
Since March 2007 every
Salesforce.com agile
release has been
deployed on-time
(down to the exact minute)
Source: Scrum Gathering 2008 - Salesforce.com Keynote Address
© 2009 Trail Ridge Consulting, LLC 10
11. Agile is value-driven
Predictive Process Adaptive Process
(Waterfall) (Agile)
Constraints Requirements Cost Schedule
Plan Value/Vision
Driven Driven
Estimates Cost Schedule Features
The plan creates The vision creates
cost/schedule estimates feature estimates
© 2009 Trail Ridge Consulting, LLC 11
12. What is valued?
Actual use of requested features in predictive projects
Always
7% Never
Often 45%
13% Results:
• 64% Rarely or never used
Sometimes
16%
• 20% Frequently used
Rarely
19%
Source: Standish Group study
presented at XP2002 by Jim Johnson
© 2009 Trail Ridge Consulting, LLC 12
13. The Value of Time
Traditional Single Release
Delivered
1-5
Agile Incremental Release
Delivered Delivered Delivered Delivered Delivered
1 2 3 4 5
Time
Value
Gap
Value
© 2009 Trail Ridge Consulting, LLC 13
14. Value Metric - Feature Delivery
94% feature
request increase
from 2006 - 2007
38% increase in
feature request
delivered per
developer
Source: Scrum Gathering 2008 - Salesforce.com Keynote Address
© 2009 Trail Ridge Consulting, LLC 14
15. Value Metric - Customer Survey
Ask your customers!
Set a baseline and
measure quarterly
Qualitative & Quantitative
Questions cover
Responsiveness
Quality of features
Support provided
Delivery timeliness
Feature value
© 2009 Trail Ridge Consulting, LLC ... 15
16. Quality Metric -
Running Tested Features (RTF)
Measures the number of automated unit and
functional tests for a team/product over time
Measures quality as a leading indicator
Measures productivity with respect to complexity better
than other measures
© 2009 Trail Ridge Consulting, LLC Source: http://www.xprogramming.com/xpmag/jatRtsMetric.htm 16
17. RTF Example
Israeli Air Force in 2005
Increased confidence in
team and management
Enabled accurate and
effective decision making
Motivated writing tests
Motivated writing smaller
tests - more adaptable
# of tests generally reflected
http://www.cs.huji.ac.il/~davidt/papers/Agile_Metrics_AgileUnited05.pdf
complexity better than other
methods (e.g. SLOC,
Function Points, etc.)
© 2009 Trail Ridge Consulting, LLC 17
18. Quality Metric -
Issue / Defect Costs
Measure the # of product issues and defects
multiplied by the cost of addressing them
Measures quality as a lagging indicator
Measures support cost impact of quality
© 2009 Trail Ridge Consulting, LLC 18
19. Issue/Defect Cost @ IBM
1,056
1,056
792 2008
Expected Cost per
Actual Defect = $16,000
528 Ticket = $500
Savings = $2.6M
264
168 67 Economics of Agile Development
32 Sue McKinney, IBM - 2008
Agile2008 Conference Case Study
0
Defects Tickets
© 2009 Trail Ridge Consulting, LLC 19
20. Myth - Metrics drive team
performance
Metrics are not inherently good or bad
It is the use of the metric that drives team dysfunction
© 2009 Trail Ridge Consulting, LLC 20
21. Myth - Velocity measures
productivity (or value)
Story points are relative
Cannot compare velocity across teams
All teams, products, environments,
constraints, and dependencies are different
Some stories are more valued than others
© 2009 Trail Ridge Consulting, LLC 21
22. Myth - 100% Committed vs.
Actual drives estimation accuracy
40
78% 64% 91% 100% 120%
30 32
Predictability
28
Story Points
25 24
20 22
20 20 20 20 Committed
18
Actual
10
Productivity
0
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5
© 2009 Trail Ridge Consulting, LLC 22
23. Commitment vs. Actual
Completed
Remaining
Share commitment vs. actual as
a fact to drive discussions:
1. Why didn’t we get it done?
2. What are we doing about it?
3. What are the impact to the
release goals?
© 2009 Trail Ridge Consulting, LLC 23
24. Commitment vs. Actual
150.0
Completed
Remaining
112.5
Share commitment vs. actual as
a fact to drive discussions:
Points
75.0
1. Why didn’t we get it done?
37.5
2. What are we doing about it?
3. What are the impact to the
0
Jan Feb Mar Apr May Jun Jul Aug Sep
release goals?
© 2009 Trail Ridge Consulting, LLC 23
25. Myth - Higher velocity is
always a good thing
Technical debt is bad
Technical debt is any “not-
quite-right” code not fixed
(e.g. Bugs, refactors,
workarounds, etc.)
Pushing too hard on new
product value and velocity
tends to increase technical
debt
Measure and limit technical
debt accumulation
© 2009 Trail Ridge Consulting, LLC 24
26. Myth - Sprints “Fail”
Maximum information is generated when the
probability of failure is 50% - not when
hypothesis are always correct.
It is necessary to have a reasonable failure rate
in order to generate a reasonable amount of new
information.
- Reinertsen, Managing the Design Factory
© 2009 Trail Ridge Consulting, LLC 25
27. Treat Sprints as Practice
Sprints allow teams to practice the skill of
delivering high-quality software on time
Preventing failure in sprints limits team
learning, growth, discipline, empowerment
and productivity
© 2009 Trail Ridge Consulting, LLC 26
28. Top 5 (or 6) Agile Metrics
Predictability Value
2. On time delivery
3. Customer
1. Velocity Surveys
4. # Features or
Value Delivered
6. Issue/Defect Cost
5. Running Tested Features
Quality Productivity
© 2009 Trail Ridge Consulting, LLC 27
29. Top 5 (or 6) Agile Metric Myths
1. Metrics drive team performance
2. Velocity measures productivity
3. Achieving 100% commitment to actual
increases estimation accuracy
4. Increasing velocity is always a good thing
5. Sprints “Fail”
6. An Agile tool will make you agile
© 2009 Trail Ridge Consulting, LLC 28
30. V1 provides metrics and an agile
framework - You guide agility
© 2009 Trail Ridge Consulting, LLC 29
31. Extending the Metrics
7. Velocity --> Investment, $/sprint, $/story pt
8. Features Delivered --> Earned Value
9. Customer Surveys --> Employee Surveys
10.Quality/Productivity --> Technical Debt
© 2009 Trail Ridge Consulting, LLC 30