This presentation covers details on how we can measure that Agile Transformation is providing the intended outcome or not. I presents a research & survey which tries to understand how different people measure value of Agile Transformation
How to measure the outcome of agile transformation
1. How to measure the outcome
of Agile transformation?
Rahul Sudame
CSP, SAFe SPC, PMI-ACP
2. Agenda
Lets play some game!
We need to have clear goals
Transformation Roadmap
Agile Readiness Assessment Model
Current State Assessment Model
‘Agile’ Metrics
Agile maturity Model
Survey on ROI for Agile
Case Study
4. Agile Innovation Games for Chapter Strategies
Is Agile delivering value to you?
Innovation Game: Teams
to discuss if Agile is
delivering value to you?
Are you measuring the
outcome of Agile
Transformation in your
organization?
Is Agile implementation
providing the desired
impact?
6. Moving to Agile is not easy!
Everyone wants to go Agile!
But Agile does not come for
free, it needs:
◦ Breaking the bones
(waterfall structure) of the
organization
◦ Changing mindset
◦ Investment &
Commitment
So, how to ensure returns of
this huge investment?
23
35
41
46
52
0 20 40 60
No Mgmt support
Non-willingness
Alignment
Difficult to Transition
No Clear Metrics
Response % for challenges in
achieving its goals with Scrum
Ref: The 2015 State of Scrum Report - Scrum Alliance
7. We need to have clear goals
Clear identification of
Customer & Business benefits
Envisioning the goal, based
on type of the organization
Identification of investment
needs
Clear identification of ROI
parameters
Business Case of Agile Transformation
Business
Case
Customer
Benefits
Business
Benefits
Project
Benefits
Investment
Needs
ROI
Parameters
8. Building Transformation
Roadmap
Agile Transformation roadmap
in alignment with
organizational environment
Pilot to try Agile methodology
and for the groundwork
Inspect & Adapt at all the
milestones
Build the right environment
iteratively
Agile Transformation Roadmap
10. Agile Readiness Assessment
Model
Are we ready for Agile?
Identification of Agile
readiness using objective
assessment
Team’s interest / enthusiasm
to be factored in
Best suited teams to initiate
pilot
Gap analysis can highlight
organization’s preparedness
Agile Readiness Questionnaire
Question Guidelines Points
Team size • 5 – Team size of 7 to 9
3 – Team size less than 7
2 – Team with 10 to 15
members
• 1 – More than 15
members
2
Team
Composition
• 5 - Completely cross
functional
3 - Cross functional with
some dependency on
external teams
0 - Not Cross functional
2
Business
Value Stream
Coverage
• 5 - Team covers the entire
business value stream
1- Less than 50% of the
value stream
0 - Less than 10% of the
business value stream
1
11. Focus on values, not just
practices
Coach the team to
implement Agile ‘principles’
and not just the ‘practices’
Build an environment of
continuous improvement
Don’t get trapped in ‘Trough
of disillusionment’
Deliver value periodically
Sprint
Cycle
Define
Roles
Backlog
Grooming
Agile
Estimation
Release
Planning
Sprint
Rituals
Activities in Sprint Cycles
12. Agile Current State Assessment
Conduct periodic assessment of
team’s Agile practices with
involvement of the entire team
Assess the effectiveness of
different aspects such as
Release Planning, Sprint
Planning, Product Ownership
Identify the Gaps and define a
plan of bridging those gaps
Current State Assessment Model
13. ‘Agile’ Metrics
Identify the metrics that
provide ‘real’ status of
progress
Define a mechanism to
aggregate required data
periodically
Do NOT enforce waterfall
metrics on Agile teams
Don’t get trapped in ‘Perfect’
burn down chart
• Earned Business Value / ROI
• Release Cycle
• Cost of Delay (COD)
• Lead / Cycle Time
Business Metrics
• Velocity / Burn down
• Backlog Size
• User Stories Done per Iteration
• User Stories carried forward to next iteration
Process Metrics
• Defects count per story / Defect density
• Acceptance tests per story
• Code Quality & code coverage metrics
• Test Automation metrics
Quality Metrics
14. Adaptability meets Predictability
Consistent
implementation of Agile
and usage of tools can
provide release
predictability
Dynamic data
extrapolation can help in
analyzing impact of
scope changes
Real-time visibility to all
the stakeholders
Release Predictability Chart
15. Agile Maturity Model
Maturity level of a
team can be
assessed based on
different parameters
Organizational
benchmarks can be
provided as
guidelines
Agile Maturity
Model itself should
be lean & aligned
with Agile
philosophy
Level
1
Level
2
Level
3
Level
4
Level
5
Process Group Rating Applicability Score
Maturity
Index
Release Mgmt
Effectiveness
6 7 86%
1.79
Sprint Mgmt Effectiveness 7.5 9 83%
Product Backlog Mgmt 7 7 100%
Frequent Product Delivery 7 8 88%
0.88Quality Assurance
Effectiveness
5.5 6 92%
Retro & Improvements 1 3 33% 0.33
Customer focus 5 5 100% 1.00
Maturity Index 4.12
Agile Maturity Model
16. Reduce Lead Time
Optimizing Lead time & Cycle
Time would help in reducing
‘waste’ in the system
‘Cost of Delay’ identification
would help in delivery of higher
value items sooner
Focus on business value would
help in reducing delays at all
levels
Deployment
1 month delay for deployment
QA
3 weeks delay for signoff
Development
2 weeks delay for integration
Design
8 days delay for signoff
Requirements
2 weeks delay for clarifications
Delays at various phases
17. Best Agile Team Award
Healthy competition amongst
the teams can be
encouraged
Effectiveness of teams on
various parameters related
with Agile implementations
can be considered
Team’s approach, Scrum
Master / PO’s effectiveness
can also be factored in
Team Retro
Effectiveness
Scrum Master
Effectiveness
Points
Team 1 12 24 73
Team 2 18 27 65
Team 3 14 11 62
Team 4 8 14 62
Team 5 10 13 61
Team 6 9 11 46
Assessment for Best Team Award
18. Are we making progress?
Agile Project Progress can
be reviewed periodically
Quantitative as well as
Qualitative measures to be
presented
Organization-wide changes
across different value-
streams to be reviewed
Project Progress Report
Quantitative
Benefits
• Delivery cycle reduction
• Revenue realization
($$$)
Qualitative
Benefits
• Value delivered to
customer
• Team efficiency
• Customer/Team
satisfaction
Benefits Not
Realized
• Value delivered to
customer
Project Progress Status Repot
20. Survey on Outcome of Agile
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile
In Progress:
◦ Details on survey participants
◦ Details on organization, industries, demographics represented
◦ Survey methodology
21. Survey on Outcome of Agile
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile
22. Survey Findings
Delivery cycle time - 22%
Delivering business value - 19%
Working Software / Customer Satisfaction – 14%
Not measuring ROI – 11%
Reduction in defects, waste & risks – 8%
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile
23. Survey Findings: Parameters
Parameters for measuring outcome of Agile:
◦ Delivery Cycle Time / WIP Reduction
◦ Achievement of business goals/value
◦ Working Software / Customer Satisfaction
◦ Reduction in defects, waste & risks
◦ Improvement in Business/Engineering processes
◦ On time delivery
◦ Delivery within the budget
◦ Break-even point - bang for your buck
◦ Team satisfaction
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile
24. Survey Findings: Feedback
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Start from what you
expect... what value
did you expect the
transformation to
bring?
If I could choose
just one measure I'd
choose Cycle Time
- time (in days)
Lean up front, big picture
planning, determine
business goals, and,
especially, how those goals
interrelate.
Its not worth to measure
the ROI of Agile.
Reducing WiP can have
up to a 4x impact on
Quality, half your Time-
to-Market, and improve
your Predictability
26. Case Study
Product development organization applied Agile enterprise-wide for
product development cycles
Outcome:
◦ Release Cycle reduced from 1.5 years to 3 months
◦ First release was not appreciated by customers due to low user
experience – Agile helped in Failing Faster!
◦ Manual regression testing efforts (for sanity suite): 5 days
◦ Automated regression testing efforts (for sanity suite): 4 hours
90% QA effort reduction
◦ Conversion of a short-term customer contract into a long-term
engagement
◦ Time taken for matured environment - 1 year
27. Summary
Agile Transformation is a journey and involves continuous improvement
For effectively presenting ROI, build a baseline and set of
parameters/metrics for before and after Agile state
◦ E.g. Pre & Post Agile Delivery Cycles / Durations
◦ Achievement of business / customer goals
Identify and present both quantitative and qualitative benefits to the
stakeholders periodically
◦ E.g. Customer Satisfaction
◦ Reduction in production hot fixes, change requests
Retrospect Agile Maturity at all levels of product development and focus
on building effectiveness iteratively
E.g. Target: Agile maturity Level 3 or Project Maturity Index > 3.5