This presentation was given at GRC Conference in Boston (October 2010) and explains the importance of measuring performance for real value. It goes into the world of metrics and balanced scorecards
1. Strategic Governance
Performance Management Systems
Ramsés Gallego
CISM, CGEIT, CISSP, SCPM, CCSK, ITIL, COBIT(f), Six Sigma Black Belt
General Manager
Entel Security & Risk Management
rgallego@entel.es
2. Program
Cause-effect analysis IT investment
Strategy KPIs
Outcomes
Processes Maturity models
Reliability
Alignment DMAIC Efficiency
Data Business goals Normalization
Real-time
Portfolio Proactive
Dashboards
Govern Lean Management
Mapping Indicators Balanced
Scorecard
KGIs Improvement
Metrics
Effectiveness
2
3. The need for IT to reinvent itself
Despite the projections of renewed economic health, the business
will continue to expect IT leadership to show strong financial
competencies, that IT projects realize tangible business value, and
that the IT organization demonstrates competitive effectiveness.
“..IT organizations that rise to the challenge will be rewarded with
substantial opportunities to develop a new type of service
organization. Those that don’t will face a grimmer future”
Gartner – CIO Update - 2009
4
4. We will be talking today about...
Some quotes and definitions
The myths on metrics
The power of Performance Management Systems
Metrics: characteristics & classification
What are CSFs, KGIs and KPIs?
Examples of governance indicators and KPIs
Process and architectures for Performance Systems
The SMART side of metrics & indicators
5
5. Let’s think about this
• ‘Measure what is measurable and make measurable what is not
so’ - Galileo Galilei (1564-1642)
• ‘If you cannot measure it, you cannot improve it’ - William
Thomson (Lord Kelvin), (1824-1907)
• ‘You cannot control what you cannot measure’ - DeMarco, 1982
• ‘Even when it is not clear how we might measure an attribute, the
act of proposing such measures will open a debate that leads to a
greater understanding’ - Fenton and Pfleeger, 1997
5
6. Definitions
Governance: “The set of responsibilities and
practices exercised by the board and executive
management with the goal of providing strategic
direction, ensuring that objectives are achieved,
ascertaining that risks are managed appropriately
and veryfing that the enterprise’s resources are
used responsibly”
6
7. Definitions: what is a performance
management system?
• Tools designed to facilitate decision-making and improve
performance and accountability through collection, analysis and
reporting of relevant performance-related data
• Metrics are simply a standard or system of measurement. In this
case, it is a standard for measuring the value driven by IT, an
organization’s value delivery posture. Although there are some
published standards for measuring the IT dimension, ideally
measurements should be adjusted and tuned to fit a specific
organization or situation
7
8. Goals of this effort
Develop a strategic governance performance
framework that allows management and other
stakeholders to assess their business
improvements (time-relevant), guide their
governance thinking and aid in the assessment
for their environments
8
9. Some myths
• #1 - a little data goes a long way
– Fact: you can only improve what you can measure
• #2 - measurement is for punishing the guilty
– Fact: metrics are for problem solving and identifying opportunity areas
• #3 - we can’t measure what we cannot control
– Fact: measure what you can influence
• #4 - metrics are for measuring people
– Fact: measure the team contribution. They are an organizational tool
• #5 - we must measure everything
– Fact: keep it simple so that everybody understands it
9
10. The power of metrics
• It’s not in the details but in their clarity
• Metrics allow executive management to:
• Measure achievement
• Drive performance
• Improve and realign (towards goals)
• Metrics should provide a holistic and balanced view of the
business
• Need to talk about RoI
10
11. Metrics: what is needed?
• The 7 attributes of Information criteria (also known as the “IC
Profile”)
Key conditions before defining a framework:
• Having a pre-defined business process
• Having clear goals/performance requirements
• Having quantitative/qualitative measures for
the business process
11
12. Characteristics & classification
• Process
Secure coding standards in use
• Objective/Subjective Avg. time to correct critical vulnerabilities
• Quantitative/Qualitative • Vulnerability metrics
By vulnerability type
• Static/Dynamic
By ocurrence within a software development life
• Absolute/Relative cycle phase
• Management
• Direct/Indirect
% of applications that are currently accepted by
business partners
Trending: critical unresolved, accepted risks
12
14. CSFs, KGIs, KPIs: what are they?
• CSFs: Critical Success
Factors or “vital elements”
• KGIs: Key Goals Indicators or
“what” has to be accomplished
• KPIs: Key Performance
Indicators or “how well” the
process is performing
14
16. Example of metrics and KPIs
• % reduction in repeat security incidents
• Increased number of secure assets from risk analysis audits
• % reduction of blank passwords on critical systems
• % improvement on time-to-access applications
• Improved bandwith use due to only-professional web surfing
• % reduction in the unavailabilty of services and components (linked with
corporate infrastructure management)
• % efficiency improvement based on number of RFCs processed
regarding vulnerabilities
• % reduction in installed software not taken from DML
16
17. Where do we show metrics?
Dashboards and BSCs
• Single point of information for governance
• Help to make decisions and provide real-time answers to managers
• Talk about the business, not about figures!
• Need the involvement of the business and operations to be
developed/designed in order to provide value
• Web and role-based so as to get the right data (becoming the tool that
consolidates siloed information)
17
20. Monitor vs. Manage
R A GE
ITO Refine, observe, MA
N
MON analize and
classify data
provided by
Value (and cost)
Act with business
systems knowledge, in a
Centralize single place
according to
access to data Apply business business needs
content and relevance to the
applications information to
determine
business
priorities
DATA INFORMATION KNOWLEDGE ACTION
Level 1
Level 1 Level 2
Level 2 Level 3 Level 4
21
21. The road to manage IT information
Management
Alarm Escalation, Invoke Management
ACTION Response
Console, Response Model
Management/Alert
● email ● Pager ● Cell ●
Presentation
Event Manage/Report Event Display, Trend Analysis, Security
Reports, Performance Reports, Security
KNOWLEDGE Pattern Discovery System Health, Assigning Ownership
Prioritization
Event Correlation
Event Prioritization, Event Associations,
Security Modeling
Event Aggregation
Data Normalization and Log Data Reduction, Event Matching,
Monitoring
Reduction
De-Duplicating Events
INFORMATION
Data Filtering
Data Repository Event Monitoring, Third-Party Integration,
Data Collection/Capture Protocol Support
DATA ● Syslog ● SNMP ● API ●
21
33. What can be achieved
• KPIs that are a measure of how well a process is performing
• The capability of predicting the probability of success or failure in
the future
• KPIs that are business-focused, process-oriented but IT-driven
• KPIs that are expressed in precisely measurable terms
• KPIs that, when acted upon, will help to improve the process
• FOCUS on what is really important and has impact
33
34. The SMART side of metrics
• First business needs, then processes,
then metrics, then tools
• Keep them simple
• Use “as is/to be” & “is/is not” lists
• Metrics should be S-M-A-R-T
34
35. THANK YOU
Strategic Governance
Performance Management Systems
Ramsés Gallego
CISM, CGEIT, CISSP, SCPM, CCSK, ITIL, COBIT(f), Six Sigma Black Belt
General Manager
Entel Security & Risk Management
rgallego@entel.es