Dreaming Music Video Treatment _ Project & Portfolio III
2019 August - Jad Abou Ibrahim
1.
2. Jad Abou Ibrahim
Maximizing Profitability through Project
Quality Assurance:
Designing and Implementing Projects Audit
KPI Tool
3. About Khatib & Alami
Early adoption of advanced technologies is part of K&A’s DNA
Khatib & Alami (K&A) is among the top 50 international design firms and the top 10
in the Middle East, according to respected industry journal Engineering News-
Record (ENR).
Today, K&A is one of the leading GIS services and utility solutions providers in the
Middle East, Gulf and North Africa.
K&A was the first Esri Platinum Partner in the MENA region, has a similar
relationship with Schneider Electric, and a close relationship with the likes of
Microsoft.
“Hi!”
Senior Business Planning and Performance Analyst
Business Operations Unit in the Geospatial Systems Integration (GSI) Department
MBA graduate with 10+ years of international experience in
Business Planning, Strategic Management, Process
Management & Improvement, and Marketing & Advertising.
Jad.abouibrahim@khatibalami.com
+9613189025
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
4. Quality Assurance Process – Definition
The Quality Assurance Process covers process and product audits at the project and
organizational level.
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
Audits of compliance to the Khatib and Alami GIS Division processes (G-QMS) are a critical
component of software/project quality.
The objectives of the Internal Process audit are to:
‣ Establish Compliance to the defined processes
‣ Determine ways to improve effectiveness of the deployed processes
The feedback given by the auditing process provides a self-correcting mechanism to improve
the G-QMS.
5. Quality Assurance Process – Workflow
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
START
Head QAG
prepares the QA
Plan for the EPG
Plan
Head QAG
prepares the Audit
Schedule
Head QAG
prepares the Audit
Team
Auditor prepares
the relevant audit
checklists
EPG Plan
Audit
Schedule
Audit
Checklists
Head QAG
conducts the
opening meeting
Auditor conducts
the audit
NCR
Audit
Checklists
Auditor submits
NCR to auditee
Auditee corrects
the Non-
Conformities
Auditor conducts
the Follow-up
Audit
Head QAG
updates the NCR
Log
NCR Log
Head QAG
calculates the
Audit KPI results
Audit KPI
Tool
Head QAG
conducts the
closing meeting
END
KEY
EPG Engineering Process Group
KPI Key Performance Indicator
NCR Non-Compliance Report
QA Quality Assurance
QAG Quality Assurance Group
6. Audit KPI Tool – Inception
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
Purpose
Objectives
Impact on Quality – Reduce the number of all types of defects
Impact on Productivity – Link a team member’s metrics with their project’s metrics
Impact on the Bottom Line – Expand a project’s profit margin
Impact on the Clientele – Increase client satisfaction
Impact on Performance – Improve a project team’s teamwork
- Conceived and implemented in Q2-2016 (upgraded in Q4-2017)
- To reduce the high average of NCs per process audit
- Encourage proactive measures to improve and maintain our QMS
7. Audit KPI Tool – Walkthrough
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
Evaluation Criteria Evaluation
Audit
Ref.
Project Audit Scope
Project
Manager
Maj.
NCs
Min.
NCs
Resolved
NCs
% Resolved
Previous
Audit
Pending
NCs
Critical
PMGT
NCs
closed?
Current
Audit
Closed
on
Time?
Previous
Audit
Closed
on
Time?
# of PIs
# of
Positive
Obs.
Ratio
KPI
Result
2019-
Q1
Project A G-QMS PM 1 2 2 1 25% 2 No No 6 0 1.31 Fail
2019-
Q1
Project B Start-up PM 2 2 4 6 100% Yes Yes 0 0 3.00 Pass
2019-
Q1
Project C Requirements PM 3 8 8 12 75% 4 No No No 1 1 1.04 Fail
2019-
Q1
Project D Application PM 4 5 13 9 50% 0 Yes No Yes 2 1 1.22 Pass
2019-
Q1
Project E Project Closure PM 5 10 4 6 43% 0 No No Yes 4 0 1.36 Fail
2019-
Q1
Project F Configuration PM 6 4 7 8 73% 15 Yes No No 2 1 0.71 Fail
Evaluation
Ratio = [Organizational Standard
For G-QMS Audits: Organizational Standard = 5
For Start-up Audits: Organizational Standard = 10
For all other Audits: Organizational Standard = 25
+ 2 * Number of PIs+ 2 * Number of Positive Observations]+52
KEY
G-QMS GSI – Quality Management System
KPI Key Performance Indicator
NC Non-Compliance
PI Process Improvement
PM Project Manager
PMGT Project Management
[Total Major NCs + Total Minor NCs
2
2 * Open NCs from current audit+ 2 * Open NCs from previous audit]+
Passing Criteria
At least 50% of current NCs have been resolved
No unresolved NCs related to critical PMGT items
Evaluation Ratio ≥ 1.00
+
8. The Impact – Situational Analysis
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
- Total of 61 individual projects underwent process audits from 2015-Q2 to 2018-Q4
- 14 projects were selected for the Situational Analysis and split into the following:
Pre-Audit KPI Tool
- Six projects satisfied the following criteria:
1. Life-cycle must be Full-Development
2. Must have done at least 50% of the audits be-tween 2015-Q2 and 2016-Q1
3. Must have done through at least 25% of the audits be-tween 2016-Q2 and 2018-Q4
4. Must have implemented the Waterfall Methodology
Post-Audit KPI Tool
- Eight projects satisfied the following criteria:
1. Life-cycle must be Full-Development
2. Must have done at least 50% of the audits between 2016-Q2 and 2018-Q4
3. Must have implemented the Waterfall Methodology
9. ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
19
52
23 24
17 17
12
15
12 12
16 15 16
13 11
0
10
20
30
40
50
60
2015-Q2 2015-Q3 2015-Q4 2016-Q1 2016-Q2 2016-Q3 2016-Q4 2017-Q1 2017-Q2 2017-Q3 2017-Q4 2018-Q1 2018-Q2 2018-Q3 2018-Q4
Average Number of NCs per Project per Quarter
Post-Audit KPI ToolPre-Audit KPI Tool
Impacted KPIs Pre-Audit KPI Tool Post-Audit KPI Tool
Avg. number of NCs per project per quarter 25 15
% of audits that resolved their NCs on time 6% 28%
The Impact – Non-Compliances and Profitability
10. ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
4
40
12
15
32
32
10
12
18
11
28
34
6
7
7
29
10
10
14
21
0 10 20 30 40 50 60 70 80
Project N
Project M
Project L
Project K
Project J
Project I
Project H
Project G
Project F
Project E
Project D
Project C
Project B
Project A
Timeline of the Average Number of Non-Compliances per Project
Pre-Audit KPI Tool
Post-Audit KPI Tool
* = Ongoing Project
*
- 64% of the projects made a profit with an average of 11 NCs per project per quarter
- 36% of the projects made a loss with an average of 27 NCs per project per quarter
- Projects A and E incurred a loss while projects B, C, D, and F made a profit
- Projects G, H, K, M, and N made a profit while projects I, J, and L made a loss
Impacted KPIs Pre-Audit KPI Tool Post-Audit KPI Tool
% of Projects that made a Profit 50% 66%
The Impact – Non-Compliances and Profitability
12. ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
0% 0%
13%
18% 20%
60%
50% 50%
75%
67%
56%
33%
38%
75%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
0
2
4
6
8
10
12
14
16
2015-Q2 2015-Q3 2015-Q4 2016-Q1 2016-Q2 2016-Q3 2016-Q4 2017-Q1 2017-Q2 2017-Q3 2017-Q4 2018-Q1 2018-Q2 2018-Q3 2018-Q4
Pass Fail % of Success
Impacted KPIs Pre-Audit KPI Tool Post-Audit KPI Tool
KPI Status: Avg. % of audits that passed 22% 55%
Avg. number of bugs per project 68 26
The Impact – KPI Status and Bugs Reporting:
13. ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
Impacted KPIs Pre-Audit KPI Tool Post-Audit KPI Tool
Average number of NCs per project per quarter 25 15
Percentage of audits with on-time fully resolved NCs 6% 28%
Percentage of Projects that made a Profit 50% 66%
Average number of valid PIs per quarter 0 1
KPI Status: Average percentage of audits that passed 22% 55%
Average number of bugs per project 68 26
Conclusion – Bringing it all together
14. Conclusion – Lessons Learned and Beyond…
ABOUT
PROCESS
AUDITS
AUDITKPI
TOOL
THEIMPACTCONCLUSION
What went wrong:
- Resistance to Change and new staff need time to adjust to our QMS
- Lack of Automation
- Data integrity
What went right:
- Results from the Audit KPI Tool are factored in an employee’s annual evaluation
- The Audit KPI Tool became a gateway to push for CMMI Level 4 and Level 5
- Sense of ownership within project teams have increased
- Encouraging many project managers to become PMP Certified
Future Developments:
- Automate the Auditing Process and the Audit KPI Tool
- Develop a Generic KPI Dashboard to include Audit Results:
- Submitting deliverables on time e.g. actual v/s planned time
- Reducing rework in order to reduce cost
- Increase rate of first-time approval from clients
With our holding company based in Singapore, and our Executive Office in Dubai, we operate in more than 30 locations worldwide and employ around 5,000 people.
We were an early adopter of GIS expertise, and this year, we are celebrating 30 years since our first project, a land information system design study in Oman.
We have vast experience in providing GIS and Location-Based Services for Enterprise solutions, Application Design and Development, Cloud and Hybrid based services, and much more. Our 200+ GIS experts operate in 24 countries to support both public and private sector clients.
We have become an independent software vendor using Microsoft’s Azure platform.
Tell a story on why I decided to develop the Audit KPI Tool:
The goal of this KPI is increase adherence to G-QMS processes by becoming a factor in the yearly staff evaluation. The PMs along with the team are responsible for implementing the process and thus, we need to identify their strengths and weaknesses; and ultimately resolve those weaknesses in a proactive manner.
Implementation of the processes were inconsistent across many projects
Financial performances of major projects were not convincing
Process audits on projects tend to be unpopular among project team members as not only do they consider it as “extra work” for them to be part of but they are unable to regard audits as a (proactive) means for a qualitative output
This is usually the case if the results of the audits are utilized to order the project teams to rectify any non-compliance rather than demonstrate to them the added value.
It is to their advantage in knowing how an audit can contribute to the seamlessness of a project’s operations, encourage enhancements and improve practices within the project, and, ultimately, increase a project’s bottom-line.
Objectives:
Impact on the client:….by delivering on time with immediate approval
Impact on Quality…..by adhering to the G-QMS processes
Impact on Performance…..through cross-functional collaboration
Impact on Bottom Line…..by reducing cost of rework
First we just performed the audits using the checklists and filling out the NCRs…then I asked myself how can we make this process measurable?
Started addressing Key Success Factors that allows for an audit can be objectively evaluated e.g. Number of Major & Minor NCs, and resolving a large number of NCs and on time
Next we added factors that can promote continuous improvement e.g. Number of PIs and Positive Observations
These would then result to a score which would define if the project passed its audit
After a few quarterly audits we realized that we can encourage PMGT teams to be more proactive by adding an additional constraint: Making sure critical PMGT items are valid and updated regularly
During the pre-audit KPI Tool phase, a total of 18 process audits took place and only 6% had non-compliances fully resolved within the allotted time.
Whereas after the implementation, 67 process audits in total, 28% of the audits fully resolved their non-compliances
Process improvements are suggestions from employees to:
Reduce defects
Time & cost overruns
Improve risk management
Improve decision making
Reduce non-compliances to processes
The suggestions can result from lessons learned during the project, an employee’s own creativity or even from the audits.
What is interesting, after the implementation of the Audit KPI Tool, is that the projects that made a loss had a 6% passing rate whereas those that made a profit had a 68% passing rate.
Based on our analysis from the data over the past 15 quarterly audits, we are confident to state that a project that has an average of more than 15 NCs per quarterly audit is very likely to incur a loss by the end of its life-cycle while a project that has 15 NCs or less is very likely to make a profit
