Cmmi and quality practices to support military operational readiness
Assuring assessments leaders team member’s quality and qualification
1. Assuring Our
Assessments, Leaders
and Team Member’s
Quality and
Qualification Process
and Quality
Kobi Vider
K.V.P Consulting
Kobi.vider@hotmail.com
kobivp@aol.com
+972522946676
2. Agenda and Topics
• Opening
• The Internal and External Assessors Community
• The Challenge Statement
• The ‘Project’ Goals / Success Factors
• Tool DEMO And Short Walkthrough
• Results and Benefits
• Conclusions and Recommendations
• Future Plans
• Discussion Points
3. Opening
• As for today most of major industries mainly defense and
aerospace, as well as armed forces
• Which runs and mange large and complex programs
• Need to comply with more than just one quality standards
in many disciplines (e.g. HW, optics, software)
• They use large groups of internal and external assessors
that perform implementation checks, progress checks,
readiness reviews and formal appraisals.
• These communities are typically composed from groups of
very experienced and professional individuals
• These individuals have the best knowledge in their
professional domain but not necessarily on how to conduct
an efficient and effective appraisal which provide
meaningful results.
4. The Internal and External
Assessors Community
• ~450 auditors (internal military staff and externals reserves)
• ~80 externals (contractors, that perform audits for and on the XYZ
ORG)
• Few tens that are the contractors internal audits and assessments teams
(quality wise)
• ~200 quality engineers (full time)
• ~150 quality practitioners / trustees (part time)
• ~X00 project / program and acquisitions officers
5. The Challenge Statement
• The combination of the effort and expected
resources increase the risks on qualification
of auditors, domain knowledge, and
calibration of results and findings
effectiveness.
• When I have initial try to map the risks and
challenges, I have discovered that the main
ones are:
6. The Challenge Statement
• Large diversity in tailoring the methods for ‘of the
same’ appraisal (such as scope, coverage, rating
understanding)
• Inconsistency in ratings
• Inconsistency of interpretations
• Reports calibration for the same unit / group and
on organizational level is very difficult if not
impassible
• Performing quality assurance of the performed
process is very challenging
7. The ‘Project’ Goals / Success
Factors
• Role based profile and criteria
• Role development path
• Calibration mechanism and criteria
• Evaluation mechanism and criteria
• Leveling the different quality engineers and
‘auditors’ in the XYZ ORG by specific
performances and graduations
8. The ‘Project’ Goals / Success
Factors
• Flowing specific trainings (on different levels) as
personal development and qualification criteria
• Listing specific performances as indicators for
leveling justifications
• Structuring the different audits and reporting
guidelines in a single mandatory to follow (based
on MDD and SLA-BOK) process, to increase
calibration and quality of professional conduct and
results (for process and meaningfully
improvements)
12. ISO 9000:2008 Correlation Snapshot
20 are covered
Management
Responsibility
Measurement
Analysis &
Improvement
Product
Realization
Resource
Management
Quality
Management
System
REQM SSD OPP DAR RSKM
CAM OT QPM M&A IRP
SAM OPD SST PPQA SCON
PMC OPF IPM OID SD
PP CM STSM CAR
13. PMBOK Correlation Snapshot
15 PAs are covered
Integration Procurement Risk Communications Human
Resource
Scope
Time
Cost
Quality
REQM SSD OPP DAR RSKM
CAM OT QPM M&A IRP
SAM OPD SST PPQA SCON
PMC OPF IPM OID SD
PP CM STSM CAR
14. ITIL – CMMI Correlation Snapshot
Service Delivery
Financial Management
for
IT Services
Capacity
Management
IT Service
Continuity Management
Availability
Management
Service Level
Management
REQM SSD OPP DAR RSKM
CAM OT QPM M&A IRP
SAM OPD SST PPQA SCON
PMC OPF IPM OID SD
PP CM STSM CAR
15. ITIL – CMMI Correlation Snapshot
Service Support
The Service Desk
Incident Management
Problem Management
Configuration
Management
Change Management
Release Management
REQM SSD OPP DAR RSKM
CAM OT QPM M&A IRP
SAM OPD SST PPQA SCON
PMC OPF IPM OID SD
PP CM STSM CAR
16. ISO 20000 – CMMI
Correlation Snapshot
Control processes Planning and Resolution processes Release process
implementing
service management
new or changed
services
Service delivery
process
Requirements for a
management system
Relationship processes
REQM SSD OPP DAR RSKM
CAM OT QPM M&A IRP
SAM OPD SST PPQA SCON
PMC OPF IPM OID SD
PP CM STSM CAR
24. Statistical Methods
High Level Maturity Chi-Square ..................................................................................................................................................... 4
Chi-Square Test for Goodness of Fit ......................................................................................................... 4
Appraisal Team Chi-Square Test for Independence ........................................................................................................... 5
Assumptions and Accuracy of the Chi-Square Test .................................................................................. 7
Member Common Cause of Variation ......................................................................................................................... 8
Continuous Probability Distribution ........................................................................................................... 11
References Control Chart ............................................................................................................................................... 14
Control Charts for Variables .................................................................................................................... 15
Design of Experiments (DOE) ...................................................................................................................... 19
Description .............................................................................................................................................. 19
When to Use DOE.................................................................................................................................... 19
DOE Procedure ........................................................................................................................................ 20
Discrete Probability Distributions ............................................................................................................... 22
One-sided Test ............................................................................................................................................ 22
Example ................................................................................................................................................... 23
Example ................................................................................................................................................... 24
One Sample t-test ....................................................................................................................................... 25
Two Sample t-test ....................................................................................................................................... 25
P-Value ........................................................................................................................................................ 26
Discrete Data ............................................................................................................................................... 26
Exponential Distribution ............................................................................................................................. 26
Probability density function ................................................................................................................ 26
Hypothesis Testing ...................................................................................................................................... 28
Null Hypothesis ....................................................................................................................................... 29
Alternative Hypothesis ............................................................................................................................ 29
Simple Hypothesis ................................................................................................................................... 29
Composite Hypothesis ............................................................................................................................ 30
25. Acknowledgments............................................................................................................................ 1
High Level Maturity
Table of Content .............................................................................................................................. 1
Abstract ............................................................................................................................................ 1
Appraisal Team 1.1.
1.2.
Purpose ............................................................................................................................ 2
Objectives......................................................................................................................... 2
1.3. CMMI Terminology .......................................................................................................... 2
Member 1.4. Industry Terminology ....................................................................................................... 2
1.5. Terms and definitions ...................................................................................................... 2
References 1.6. Process Performance Models References in the CMMI Suite ......................................... 2
1.7. Process Performance Models relationships with Measurements in the CMMI Suite ..... 2
1.8. Relationships from PSP and TSP to support CMMI Measurements and high maturity ... 2
1.1 high level listen for in process performance models ....................................................... 4
1.1.1 Overview .................................................................................................................. 4
1.1.2 In Small Setting Organizations ................................................................................. 4
1.1.3 In Agile like development and teams ....................................................................... 4
1.1.4 In different type Organizations ................................................................................ 4
1.1.5 At Level 2 & 3 Organizations .................................................................................... 4
1.1.6 At Level 4 & 5 Organizations .................................................................................... 4
1.2 Business Objectives; quantitative objectives, Needs and Between ................................ 4
1.2.1 Business Objectives .................................................................................................. 4
1.2.2 Quantitative objectives ............................................................................................ 4
1.3 Analyzing the organization lifecycle and process ............................................................ 4
1.3.1 Key players in the process ........................................................................................ 4
1.3.2 Key users of performance models ........................................................................... 4
1.3.3 Key elements (building blocks) of the process......................................................... 4
1.3.4 Domain Specific Perspective .................................................................................... 4
1.3.5 Process perspective.................................................................................................. 4
1.3.6 Organizational Role Perspective .............................................................................. 4
3.1 Project Management Process Areas ................................................................................ 5
3.1.1 Project Planning (PP) 5
26. PROCESS PERFORMANCE MODELS ........................................................................................................... 1
APPLICATION AND APPROPRIATE USAGE (METHODOLOGY) .................................................................... 1
PPM-BOK ACKNOWLEDGMENTS ....................................................................................................................................... 1
TABLE OF CONTENT .......................................................................................................................................... 1
ABSTRACT ...................................................................................................................................................... 6
INTRODUCTION ........................................................................................................................................ 7
1.1. Purpose ...................................................................................................................................... 7
1.2. Objectives .................................................................................................................................. 7
1.3. CMMI Terminology .................................................................................................................... 7
1.4. Industry Terminology ................................................................................................................. 7
1.5. Process Performance Models References in the CMMI Suite .................................................... 7
1.6. Process Performance Models relationships with Measurements in the CMMI Suite ................ 7
1.7. Relationships from PSP and TSP to support CMMI Measurements and high maturity ............. 7
INTRODUCTION TO PROCESS PERFORMANCE MODELS ............................................................................ 8
1.1 Why to Measure the process performance ............................................................................... 8
1.2 What is all about? ..................................................................................................................... 8
1.3 Benefits from using process performance models ..................................................................... 9
1.3.1 Overview .............................................................................................................................................. 9
1.3.2 Application in Small Setting Organizations ........................................................................................... 9
1.3.3 Application in Agile like development and teams ................................................................................ 9
1.3.4 Application in different type Organizations ......................................................................................... 9
1.3.4.1 Software ..................................................................................................................................... 9
1.3.4.2 System ........................................................................................................................................ 9
1.3.4.3 Services ....................................................................................................................................... 9
1.3.4.4 Acquisition .................................................................................................................................. 9
1.3.4.5 Maintenance ............................................................................................................................... 9
1.3.4.6 Support ....................................................................................................................................... 9
1.3.5 Application and support low maturity organization ............................................................................. 9
1.3.6 At Level 2 & 3 Organizations ................................................................................................................ 9
1.3.7 At Level 4 & 5 Organizations ................................................................................................................ 9
1.4 The Measurements and Analysis Team ................................................................................... 10
1.4.1 As Organizational Level Functional Team ........................................................................................... 10
1.4.1.1 Position in the Organizational Structure ................................................................................... 10
1.4.1.2 Team Suggested Structure ........................................................................................................ 10
1.4.1.3 Team Roles, Responsibilities and authority .............................................................................. 10
1.4.1.3.1 Roles ..................................................................................................................................... 10
1.4.1.3.2 Responsibilities ..................................................................................................................... 10
1.4.1.3.3 Authority .............................................................................................................................. 10
1.4.1.4 Principles of Professional Conduct............................................................................................ 10
1.4.1.5 Basic Activities .......................................................................................................................... 10
1.4.1.6 Additional Activities .................................................................................................................. 10
1.4.1.7 Knowledge and Skills ................................................................................................................ 10
1.4.1.8 Personal Development Path ..................................................................................................... 10
1.4.2 As Group of Individuals ...................................................................................................................... 11
1.4.2.1 Position in the Business Organization Structure ....................................................................... 11
1.4.2.2 Position in the Quality Organization Structure ......................................................................... 11
27. Acknowledgments............................................................................................................................ 1
Table of Content .............................................................................................................................. 1
Abstract ............................................................................................................................................ 1
1.1. Purpose ............................................................................................................................ 2
1.2. Objectives......................................................................................................................... 2
1.3. CMMI Terminology .......................................................................................................... 2
1.4. Industry Terminology ....................................................................................................... 2
1.5. Terms and definitions ...................................................................................................... 2
1.6. Process Performance Models References in the CMMI Suite ......................................... 2
1.7. Process Performance Models relationships with Measurements in the CMMI Suite ..... 2
High Level Maturity 1.8.
1.1
Relationships from PSP and TSP to support CMMI Measurements and high maturity ... 2
High level SCAMPI scripts for process performance models ........................................... 4
1.1.1 Overview .................................................................................................................. 4
Lead Appraisal 1.1.2
1.1.3
In Small Setting Organizations ................................................................................. 4
In Agile like development and teams ....................................................................... 4
Guideline 1.1.4
1.1.5
In different type Organizations ................................................................................ 4
At Level 2 & 3 Organizations .................................................................................... 4
1.1.6 At Level 4 & 5 Organizations .................................................................................... 4
1.2 SCAMPI scripts for Business Objectives; quantitative objectives, Needs and Between .. 4
1.2.1 Business Objectives .................................................................................................. 4
1.2.2 Quantitative objectives ............................................................................................ 4
1.3 SCAMPI scripts for analyzing the organization lifecycle and process .............................. 4
1.3.1 Key players in the process ........................................................................................ 4
1.3.2 Key users of performance models ........................................................................... 4
1.3.3 Key elements (building blocks) of the process......................................................... 4
1.3.4 Domain Specific Perspective .................................................................................... 4
1.3.5 Process perspective.................................................................................................. 4
1.3.6 Organizational Role Perspective .............................................................................. 4
3.1 Project Management Process Areas ................................................................................ 6
3.1.1 Project Planning (PP) 6
28. Results and Benefits
Methodology conclusions.
• Model formulation is complex, but evaluation is straightforward.
• Standard tools, easy to integrate on domain cases.
• Method is better at capturing SMEs insights and project dynamics.
• Opportunity to further include additional domains
SME related conclusions (preliminary).
• Small units have better acceptance for audits
• Engineering related organizations have a good case for higher maturity levels
• Critical dependency on allowed investment horizon.
• Appraisal costs impacts on overall investment not dominant.
• Benefits for organizations operating at volatile (uncertain) segments.
29. Conclusions and Recommendations
• Method can be implemented in order to develop a decision
support system in the management of new product
development domain
• This model addresses various aspects of new product
development over multiple stages, and easily can be
adjusted to other domains
• The model can deal with quantitative and qualitative input
and missing data
• Method can be implemented to address our managerial
problems and to support our managers with strong ‘gate
keepers’
• The implementation of a graphical user interface hiding the
complexities of the method and model is significant
30. Future Plans
• Continue refinement of model.
• Parameter ranges.
• Roadmap involving multiple SCAMPI activities.
• Complete Validation and Verification.
• Integrate work and receive feed from other
initiatives.
• Continue sharing with other domains
31. Discussion Points
• Performance data
• Cost of poor planning
• Quantifying the operational impact
• Effecting and effected stakeholders
mapping
• Quantifying the impact of support planning
on the development teams
• Appling this model on other domains
