Project Performance Metrics - Creating project and portfolio metrics and dashboards for business and IT projects/portfolios

1. Project Performance Metrics
Jason Uyder, MSIS, PMP, ITIL
Fireman’s Fund Insurance Company
February 16, 2012

2. Definitions

Metric:
 A system of related measures that facilitates the
quantification of some particular characteristic

Metic:
 A standard of measurement
 Mathematics A geometric function that describes the
distances between pairs of points in a space.

3. Definitions

Quality
 The standard of something as measured against other
things of a similar kind; the degree of excellence of
something: "quality of life".
 General excellence of standard or level: "quality beers".

Quality
 The delta between expectations and experience – (Unknown)

Quality Metric
 The quantifiable measurement of the delta between
expectations and reality

4. Importance

As we all know, having good solid metrics and
measures is ‘key’ for every Project:
 How else would we report status?
 Know when the project is complete?
 Know if the project was successful?

5. Challenges (1)

Projects often defined or characterized by IT
activities:


When really the project is to improve a business
activity/function:


Implement new system, install software
Improve Customer Service, gain market share, increase
efficiency
We often assume good project pre-work was
done and if that we complete project activities
we will realize the business benefit.

6. Challenges (2)





People sponsoring the project don’t understand
project management
Don’t understand the ‘core’ business and/or the
‘core’ processes well enough
As Frank said in December – ‘Today people seem
to value decision making more than analysis’
You may be challenging assumptions and beliefs,
forcing folks to confront reality
Often forces a culture change.

7. Challenges (3)
 Often Metrics Programs are viewed as compliance
activities
 Make sure you understand the driving force behind
the desire for Metrics
 ‘Figures don’t lie but liars can figure’
 Metrics are the ‘shoe horn’ into the project, the real
value is ‘action’ and ‘take away items’
 Have a Charter for the program, treat the program
as a project, have your own CSF, Communication
Plan, Remediation Plan

8. General Quality Metrics

Product Metrics describe characteristics of a
product such as it’s size, complexity, features,
and performance.


Common product metrics are mean time to failure,
defect density, customer problem, and customer
satisfaction metrics.
Process Metrics are strictly for evaluating &
improving the effectiveness of development and
maintenance processes.

9. Product Metrics

Critical Success Factors

Why we are doing the project

Can be challenging to measure


Lack of data, common understanding of what and
how to measure
Can be financial, but should be measurable in the short
term

Not usually ROI, should be linked to CBA

Mostly operations based measurements

Should provide clear evidence of success

In Service Based Economy - Transactions are often the
focus

10. Critical Success Factors
Baseline
Target
Process (Speed &
Accuracy)
Frequency
50%
End to end cycle time –
Reduction in
Sample @ 90% Average = 34 Days
Reduce the cycle (Point at which a candidate
average
Confidence Level
time
starts an application to the
Monthly
Cycle time
time rejected or licensed)
Process
Measurement
Reduce the
Number of Process Steps
number of steps
(End to End)
in the process
Financial
Criteria
Reduce staffing
costs in the
Agent Licensing
Department by
$500,000 and
improve service
Departmental budget
1.5M RR
Results
TBD
Confirm process
steps at
deployment and
when in steady
state
57 Current Steps
50%
Reduction
TBD
Annual
1.5M RR
1M RR after
project costs
TBD
A limited number of CSF is crucial, too many usually means
you are trying to do too much
don’t really know what you’re doing
© Copyright Allianz
10

11. Critical Success Factors
Need
Drivers
CTQs
How long does it take to
Fill out Paperwork
Minimal Redundancy
User Friendly Forms
Easy Access To Current
Forms
Fast/
Reliable/Consistent
& Cost Efficient
Agent
Licensing Process
Quick Response
Minimal Data Entry
(# of Fields)
Complete/Legible forms
Technology to speed
investigation
Minimal Handling
Consistent Decision
Process
IT Systems won’t address these CTQs
Localize Decision
Making
Minimize Special Case
Decisions
VOC’, Root Causes &
CTQs
Collected the ‘VOC’ which
determined that due to the
industry compensation
model that getting an
Agent Licensed &
Appointed quickly greatly
increased the likelihood of
individual Agent success.
Therefore, Speed &
accuracy became this
projects CSF:
How fast ‘cycle time’
and consistently ‘Variance’
can we process Agents
through the Licensing
process correctly.
What are the CTQs for a
fast, reliable ….start with
the Drivers and drill down
to specific measurable
CTQs

12. Critical Success Factors
Average Cycle Time Went
From 34 Days to 16 Days

13. Critical Success Factors
Criteria
Measurement
Frequency
Reduce the cycle
time
End to end cycle time – (Point at
which a candidate starts an
application to the time rejected or
Licensed)
Sample @ 90%
Confidence Level
Monthly
Average = 34 Days
Reduce the number
of steps in the
process
Number of Process Steps
(End to End)
Confirm process
steps at deployment
and when in steady
state
57 Current Steps
Reduce staffing costs
in the Agent
Licensing
Departmental budget 1.5M RR
Department by
$500,000 and
improve service
Annual
Baseline
1.5M RR
Target
Results
50% Reduction
in average
Pass
Cycle time
New Average 16 Days
50% Reduction
Pass
New Process has 8
steps
TBD –
1M RR after
6FTE reduction
project costs ($450K Salary Reduction
+)
PROCESS STEPS
57
8
“HANDOFFS”
34
6
AVERAGE TIME SAVED PER APPLICANT: 2 1/4 HOURS
H.O. STAFF REDUCTION OF 6 DUE TO PROCESS IMPROVEMENT
THE ABILITY TO ACCESS INFORMATION AND WORK ANYWHERE IN THE
WORLD WITH AN INTERNET CONNECTION
© Copyright Allianz
13

14. Process Metrics

Project Quality and Process Compliance Metrics
 How is the project performing as a process, often
called projects status or included in project status
reports – Fairly Standard Project Metrics:

Schedule Performance Index

Cost Performance Index

Deliverable Timeliness (Monthly)

Deliverable Timeliness (Cumulative)

Change Request Impact

Defects per 1K Hrs Development

Testing Effectiveness

% Pre-Production Defects

16. Initial Project Quality Scorecard

17. Project Quality and Process Compliance Metrics
Project/
Release
Findings
Proj A

CPI is over target at
1.13, however it has
come down .03 from
last month. The root
causes of the high CPI
were overestimates of
requirements and
design activities and
resources not applying
time to correctly to
activities.
Proj B

Change Request
Impact is currently at
13%. The majority of
this score is the result
of PCR#3 (Scope
Realignment) 1100
Vendor hrs + 300
internal hours.
Action Items

None required at this time.
The release is creating a
mitigation plan and looking at
the potential
impacts/dependencies to the
overall E4 Release.

Outcomes
Since refresher trainings and reminders
on the importance of proper timekeeping
activities, CPI has trended back down to
close (.03) to targets.
TBD
Ultimately the real value is in the ‘action items’ and ‘take away items’ !
How can you influence and improve the project and the project
management practices in your organization ?
Mentor & Coach

18. Update
Not yet available
Not applicable or not available for release
Neutral
Trending
* PCR to Budget Ratio data is go-forward from 2/1/11 only

19. Quality & Performance Program Expansion
and Continuous Improvement
Feb 2010
February 2009 - first quality scorecard produced for XXXX
with only 11 scores and several metrics that were not
included or TBD.
May 2011
All Division programs are now represented with a more
comprehensive suite of KPIs. Enterprise Systems have completed a
calibration period and will be included going forward.
• Project & Program leadership meet monthly to review findings, discuss remediation plans, and most importantly gain
a better understanding of how to manage and improve project performance with quantitative data.
• Scores that are out of range are discussed, root causes identified, and where warranted, process improvement
initiatives are implemented.
• Remediation plans are in place to ensure process compliance, and overall performance and related scores. Lunch
and Learn sessions are provided, as well as various training sessions in an effort to minimize the learning curve for the
metrics program.

20. Quality & Performance Program Expansion
and Continuous Improvement

21. Summary
Yes – This is really hard, but should be really simple !
You may be challenging assumptions and beliefs, forcing folks to
confront reality
Often forces a culture change, or highlights the needs for one
Can be a double edge sword – make sure leaders really want to know
As companies ‘tighten their belt’ this type of analysis becomes more
valuable or is perceived as more valuable