Benchmarking As a Tool for Optimising Software Development Performance

Measure. Optimize. Deliver.
Phone +1.610.644.2856
Measuring Success
Benchmarking as a Tool for Optimising
Software Development Performance
Bram Meyerson
QuantiMetrics, a DCG Partner

©2013 David Consulting Group and QuantiMetrics
QuantiMetrics
•  A two decade track record as an independent trusted advisor
•  Established in 1992 and remained independent
•  Acquired Research Services Benchmarking (ex: Butler Cox) from CSC
in 2000
•  Owns the largest validated and calibrated benchmark database
covering project and application data that spans thousands of projects
from hundreds of organisations
•  Retained by world-class organisations and consultancies to help them
evaluate suppliers and to improve their in-house efficiencies
•  Provides client-centric service … underpinned by trust, confidentiality
and teamwork
•  Offices in London, Wiesbaden, Johannesburg
2

Proven Capabilities and a Focus
on Ongoing Innovation
•  Fact-based IT executive advisory and benchmark provider
•  Optimisation of software development and support processes
•  Due diligence and professional review of portfolio, programs and
project budgets and plans
•  Software application sizing (in-house development, outsourced
applications and vendor-supplied packages)
•  Evaluation of 3rd party software development services
•  Commercial software delivery and support dispute advisory
3

Agenda
•  Why benchmark?
•  Our unique approach to benchmarking
•  Examples of how benchmarking has underpinned performance
improvement
•  Critical success factors for benchmarking
4

Why benchmark?
•  Provides an objective 3rd party scorecard
•  Provides the cross-industry breadth perspective. Most IT executives have
a depth of MIS about their project environment
•  Acts as a catalyst for change by setting realistic improvement targets
•  Provides an accurate mechanism to calibrate project estimation models.
•  Focuses attention on measurement and improvement
5

6
Our Unique Approach to Benchmarking:
Process
Applications
Development
Project portfolio
Proposed or
disputed Project
Standard
project
data
(current)
QPeP
Benchmark
Report
Calibrated
client
baseline
QuantiMetrics
Validated Project
Database
QEsT
Risk
Report
High Moderate Acceptable
<-------------- Estimation Scenarios --------------à
Efficiency
Staffing IndexCost per FTE
Unit cost
Elapsed months
All testing errors
Operational
errors
Schedule conformance
Budget conformance
Client Minimum Maximum
Median Top quartile Top decile
IT Development – Summary Performance
Efficiency
Staffing IndexCost per FTE
Unit cost
Elapsed months
All testing errors
Operational
errors
Schedule conformance
Budget conformance
Client Minimum Maximum
Median Top quartile Top decile
IT Development – Summary Performance
Identifies good practices
and successes – but
also highlights
inefficiencies and
problems in
methodology, staffing,
planning, productivity,
cost or capability across
sizes, types &
technologies – and
proposes improvement
actions
Identifies probability of
successful delivery against
time, budget & quality
targets – and proposes
alternative scenarios with
higher likelihood
Project
characteristics

•  By project, programme, group of projects and organisation
•  The impact of size on project performance (finding the “sweet spot”)
•  Project type
•  Development technologies
•  The impact of time pressure on project performance (finding the ideal
time-box)
•  Industry sector
7
Key Focus on Normalization

©2013 David Consulting Group and QuantiMetrics 8
Base Data
•  Classification data
•  Size data
•  Resource data
•  Testing errors
•  Operational errors
•  Qualitative data
Outcomes
•  Productivity
•  Duration
•  Unit Cost
•  Test Errors
•  Reliability
•  Time Slip
•  Effort Overrun
Drivers
•  Efficiency
•  Staffing Style
•  Technical Quality
•  Planning Style
Context
•  Technology
•  Cost
•  Project Size
Raw Metrics Derived Metrics
Influence
Assist in
interpreting
Appropriate Level of Metrics

Good performance for Reliability & Effort Overrun; poorest performance in
Productivity, Unit Cost, & Test Errors.
The more the blue-
lined (Customer)
footprint
approaches the
outer edge of the
polygon, the better
the performance;
the closer to the
origin (centre), the
worse the
performance
Note: Project
sensitive cost
benchmarks are UK-
based and blended
where necessary to
take account of use
of external and off-
shore resources
Productivity
Duration
Unit cost
Test errorsReliability
Time slip
Effort overrun
Minimum Maximum Median
Top quartile PEP member Bottom quartile
OUTCOMES
Graphical Reporting

Excellent budget conformance, weak underlying efficiency and typical staffing
levels.
Efficiency
Staffing
QualityPlan conformance
Budget
conformance
DRIVERS
The more the blue-
lined (Customer)
footprint approaches
the outer edge of the
polygon the better
the performance, the
closer to the origin
(centre) the worse
the performance
Graphical Reporting

Technology gearing is typical, but projects are much smaller than typical, and staff
costs close to the top 25%.
Gearing examines the
power of the toolset /
languages (i.e. function
points per 1,000 lines of
code)
Gearing
FTE costSize
CONTEXT
These costs are fully
loaded. Benchmark rates
quoted here are global,
dominantly based on
internal staff plus
increasing contribution of
off-shore resourcing
The more the blue-
lined (Customer)
footprint
approaches the
outer edge of the
polygon, the better
the performance;
the closer to the
origin (centre), the
worse the
performance
Graphical Reporting

Cost-driven approach – high use of low-cost, off-shore suppliers whose productivity
is low; time-driven characteristic largely accounts for this, but also contributes to
high efficiency … productivity and efficiency are both better than previous
Staffing Style
Efficiency
Current projects
Previous projects
Current Agile
Previous Agile
Current not-Agile
Lean Time
driven
UnproductiveStretched
H
H
L
LStaff cost
Productivity
Current projects
Previous projects
Current Agile
Previous Agile
Current not-Agile
Economic Skill driven
Cost driven Uneconomic
H
H
L
L
Trade-offs

Agenda
•  Why benchmark?
•  Our unique approach to benchmarking
•  Examples of how benchmarking has underpinned performance
improvement
•  Critical success factors for benchmarking
13

Case Study 1: Large Telco
•  Leading communications company, serving the needs of customers in
almost 200 countries
•  Mandated to become more cost effective and embarked on an
aggressive performance transformation initiative. The company
participated in benchmarking to underpin their transformation journey
•  Summary of achievements:
–  Benchmarking reports have been used as a catalyst to speed up,
sharpen costs and improve reliability of software delivery
–  Benchmarking deliverables identified the impact of the adoption of
various Agile practices and assisted in identifying which were/were
not making a difference
–  Dramatically shortened cycle times, with small effort tradeoffs
–  Quality of software products also improved
14

Case Study 1: Large Telco
Relative to equivalent benchmarks, cycle time reduced dramatically, offset
by small productivity loss
15
Achieved by:
•  Increasing adoption
of time boxing (and
other elements of
Agile practice)
•  Increasing proportion
of outsourcing to a
small set of suppliers
•  The decision to go for speed was more of a concern than productivity, as simultaneous use of off-
shore cheaper resources would more than compensate.
•  The client could not continue to do things the same old way; to remain competitive in an era of
deregulation required being smart
Relativetobenchmark
Time

•  Leading Life, Pensions and Investment company
•  Non measurable achievements: benchmarking reports have been used
as a catalyst to speed up, sharpen costs and improve reliability of
software delivery.
•  Summary of achievements:
–  Improved quality of software products
–  54% reduction in project costs ($/fp)
–  More disciplined project management
–  More accurate schedule and cost estimation
–  Better technical quality (fewer errors)
Case Study 2: Large Insurance Company

Case Study 2: Large Insurance Company
Significant productivity and quality improvements
17
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
Proportionofmembers
Function points perstaff-month
IQR
Year 3Year 1
Inter-quartile
range
All testing errors
Earlyoperationerrors
Current proje
Previous proj
Current Agile
Previous Agil
Current not-A
Ineffective testing Poor quality
approach
Good quality
approach
Effective testing
H
H
L
L

Technical Quality
Efficiency
Previous projects
Current projects
Current Agile
Previous Agile
Current not-Agile
Quality deferred Quality led
Quality laggedQuality ignorant
H
H
L
L
Focus on Process Leads to Getting it Right the
First Time
18

Critical Success Factors
•  Sponsorship
•  Maturity
•  Comparability
•  Confidentiality
•  Taking a systemic view
19

High
Productivity
(fp/sm)
Low resource
cost ($/FTE)
Low Unit cost
(R/fp)
Effective Testing
(reducing
Fault rework)
High technology
gearing (Kloc/fp)
Process
Efficiency
Quality
Process (approach)
High
requirements
churn
Right Staffing
(approach to
Staffing levels)
Planning
Efficiency
Effective
requirements
management
Higher skill
levels
Right project
size (fp)
Excessive business
(time) demands
Late business scope creep
High speed
Lean
Processes
Reuse
“Efficeincy
over
Utelisation”
A Systems-thinking View on
Project Performance

Contact Us
Alan Cameron, DCG-SMS, Europe
Email: a.cameron@dcg-sms.com
Phone: +44 843 2895174
Mobile: +44 7825 271344
http://www.dcg-sms.com
21
@DavidConsultGrp
/DavidConsultGrp
/company/David-Consulting-Group
Mike Harris, DCG, United States
Email: m.harris@davidconsultinggroup.com
Phone: +1 (610) 644-2856
Mobile: +1 (484) 753-3935
http://www.davidconsultinggroup.com
Measure. Optimize. Deliver.
Phone +1.610.644.2856

Benchmarking As a Tool for Optimising Software Development Performance

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (7)

Similaire à Benchmarking As a Tool for Optimising Software Development Performance

Similaire à Benchmarking As a Tool for Optimising Software Development Performance (20)

Dernier

Dernier (20)

Benchmarking As a Tool for Optimising Software Development Performance