Metrics are powerful tools when used to effect positive change in a project or organization. However, the value and benefits of metrics are often dependent on the context. While certain metrics provide information and insight to drive decision making for a traditional development approach, they may not be useful in an agile landscape—and vice versa. QA and agile experts Shaun Bradshaw and Bob Galen delve into the value, pitfalls, pros, and cons of various metrics in agile and waterfall development environments. Hear their experiences as they discuss and explore a variety of project-level, software development, and software testing metrics through the lens of both traditional and agile development contexts. Although Bob and Shaun respect each other’s knowledge and skill, they don’t often agree on metrics. And in this showdown, you’ll see why! Be prepared to learn, be entertained, and be ready to get in on the action as these two metrics titans go head-to-head.
Yin and Yang: Metrics within Agile and Traditional Lifecycles
1. W6
Special Topics
5/1/2013 11:30:00 AM
Yin and Yang: Metrics within Agile and
Traditional Lifecycles
Presented by:
Shaun Bradshaw, Zenergy Technologies, Inc.
Bob Galen, Rgalen Consulting
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Shaun Bradshaw
Cofounder of Zenergy Technologies, Inc., a QA and agile solutions consulting firm
(zenergytechnologies.com), Shaun Bradshaw has spent the past fifteen years advising, teaching, and
mentoring clients to improve their QA and test processes with effective testing and test management
techniques. He is the coauthor, editor, and trainer of a suite of methodologies covering testing, test
management, and test metrics. A popular speaker at many of the major QA industry conferences, Shaun
is known for his presentations on test metrics, the S-Curve, and the Zero Bug Bounce.
Bob Galen
Bob Galen is an agile coach at RGalen Consulting and director of agile solutions at Zenergy
Technologies, a North Carolina-based firm specializing in agile testing and leading agile adoption
initiatives. Bob regularly speaks at international conferences and professional groups on topics related to
software development, project management, software testing, and team leadership. He is a Certified
Scrum Master Practicing (CSC), Certified Scrum Product Owner (CSPO), and an active member of the
Agile Alliance and Scrum Alliance. Bob published Scrum Product Ownership–Balancing Value from the
Inside Out, which addresses the gap in guidance toward effective agile product management. Contact Bob
at bob@rgalen.com or bob.galen@zenergytechnologies.com.
3. 4/16/2013
The Yin & Yang
Title
of Software Metrics
STAREAST 2013
Presented By:
Bob Galen and Shaun Bradshaw
Slide 1
Software Peace of Mind in Action
???
Introductions
Bob Galen & Shaun Bradshaw
Director, Agile Solutions – Zenergy
Technologies
Determined agile adoption coach, trainer,
and consultant
10+ years of agile experience across
Software, QA/Test, and Project Management
Specialty in Agile at-Scale challenges
bob.galen@zenergytechnologies.com
Slide 2
VP of Consulting Solutions – Zenergy
Technologies
Experienced test manager, consultant,
trainer
IT Testing strategist
15 years of multi-domain experience
Agile experience; Scrum Master
shaun.bradshaw@zenergytechnologies.com
Software Peace of Mind in Action
1
4. 4/16/2013
???
Format For This Session
Focus for this session: QA & Testing Metrics
Panel format
15 minutes for Shaun
15 minutes for Bob
Write down your questions, ideas, metrics, etc. on 3x5
cards
30 minutes of Q&A interaction and debate
Slide 3
Software Peace of Mind in Action
What is Measurement?
• “Measurement is the process by which numbers
or symbols are assigned to attributes of entities
in the real world in such a way as to
characterize them according to clearly defined
rules.”
• “Measurement is the empirical, objective
assignment of numbers, according to a rule
derived from a model or theory, to attributes of
objects or events with the intent of describing
them.”
2
5. 4/16/2013
???
Software Metrics Traps
Karl Wiegers 1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
http://www.processimpact.com/articles/mtraps.html
Lack of Management Commitment
Measuring Too Much, Too Soon
Measuring Too Little, Too Late
Measuring the Wrong Things
Imprecise Metrics Definitions
Using Metrics Data to Evaluate Individuals
Using Metrics to Motivate, Rather than to Understand
Collecting Data That is Not Used
Lack of Communication & Training
Misinterpreting Metrics Data
Slide 5
Software Peace of Mind in Action
???
Case Study
Large, traditional financial / insurance organization
100 year company, traditionally staffed IT organization
Regulatory support
Large-scale project; joint with other consulting firms
Slide 6
Software Peace of Mind in Action
3
6. 4/16/2013
???
Waterfall is ‘Real’
Not every project can be broken down into 2-week
‘chunks’
Not every company wants to be agile
Scale of the project impacts everything
We’ll know we’re done when we’re done is NOT an
acceptable answer in many contexts
Waterfall and Agile
Slide 7
Software Peace of Mind in Action
???
Monitoring Phases & Transitions
The overriding goal was to plan a series of maturation
phases
Testing would serve as the guide between phases
Qualifying progress
Verifying functionality
Measuring variation / defects
Guiding next steps
Slide 8
Software Peace of Mind in Action
4
7. 4/16/2013
???
Risk-Based Testing
All testing was focused towards what
Was delivered
Worked
Was planned
Was considered Higher Priority features
Was considered Higher Risk items
Slide 9
Software Peace of Mind in Action
???
Key Metrics – “Exit Criteria”
Test Execution by priority
Have we tested enough? Did we cover our plans?
Overall pass rates
Is the software working? Where are we ‘Blocked’?
Are there any ‘Patterns’ to the failures
Percent of defects corrected by severity
Are we catching up on repairs?
Is the software maturing at a sufficient rate to support the plan?
Slide 10
Software Peace of Mind in Action
5
8. 4/16/2013
???
Customer Override…
Basically, the customer has the final authority to adjust
Metrics
Analysis
(Re)Actions
Goals
So, ’agility’ from a leadership & governance perspective
Slide 11
Slide 12
Software Peace of Mind in Action
Software Peace of Mind in Action
6
9. 4/16/2013
Transition…
Slide 13
Software Peace of Mind in Action
???
Agile Metrics
Agile methods are generally not metrics centered,
They’re much more team-centered and progress or
outcome centered
Rarely do you want to examine the individual; leave that to the
team
Exceptions – experienced coaches
Avoid “Metrics Dysfunction” metrics
Lines of code produced
Bugs found
Estimate vs. Actual
Slide 14
Software Peace of Mind in Action
7
10. 4/16/2013
???
Big 4 Agile Metrics
1.
2.
3.
4.
Value Delivered
Velocity & Throughput
Quality
Team Morale & Satisfaction
Slide 15
Software Peace of Mind in Action
???
Value-Delivered
Prioritization
Always deliver the highest priority work first
Earned value – it’s “Done”
Customer Acceptance
User Story sign-off
Active customer demonstration; embrace changes
ROI – Usage
Sales & usage targets; confirm
Did we make the right decisions on value? Adjustments?
Slide 16
Software Peace of Mind in Action
8
11. 4/16/2013
???
Velocity & Throughput
Productivity matters in agile teams
First, the software is only accepted if it meets clear Done-Ness
criteria
Velocity – the number of Story Points produced per Sprint
Throughput – the number of ‘Days’ that a story was InProgress, relative to it’s size; also a risk play
Continuous improvement
Velocity – UP; Throughput - DOWN
Slide 17
Software Peace of Mind in Action
???
Quality
Test Automation
Continuous Integration & Continuous Deployment
Every check-in…run automation; Stop-the-Line
Defect ‘Escapes’
Escaping your Done-Ness
Escaping your Sprint
Escaping into the wild…Customer Escapes
Pairing, inspections, RCA
Continuous Improvement – downward trending
Slide 18
Software Peace of Mind in Action
9
12. 4/16/2013
???
Team Morale & Satisfaction
Self-directed; Team autonomy; Listened to
Attrition is analyzed; New hire onboarding
Sustainable pace
Periodic surveys
Team overall health & satisfaction
Feedback on adjustments
Agile values
Make the survey and the actions Transparent
Make it clear—people are central to your success
Slide 19
Software Peace of Mind in Action
???
That’s it…
Aligned with Goals
Sprint Goals & Release Goals
Organizational Goals & Company Goals
Attributes:
They’re all important…perhaps “People First”
There are no “fixed targets”
Keep them simple; everyone is aware of the goal
Measure as a team
Stop the individual metrics dysfunction
Slide 20
Software Peace of Mind in Action
10
14. 4/16/2013
???
Questions for your Consideration
What ‘healthy’ metrics have you personally experienced?
What about stories of dysfunctional metrics?
Challenges you’ve seen in moving from Waterfall to Agile
or vice versa?
If you only had 1-3 metrics to measure software in your
own company, what would they be?
Slide 23
Software Peace of Mind in Action
???
Questions for your Consideration
Example of what we could discuss
Estimation vs. Actuals
Planning Poker & ‘Relative’ estimation
Anyone use GQM?
Slide 24
Software Peace of Mind in Action
12