An introduction and overview presentation I gave in 2008 to the Northern UK Security Group in Leeds.

1. 14/07/2008
Security Metrics
Phil Huggins

2. 14/07/2008 Security MetricsPage 2
Core Text
Security Metrics : Replacing Fear, Uncertainty and Doubt
Andy Jaquith, 2007
0-321-34998-9

3. 14/07/2008 Security MetricsPage 3
Recommended Texts

4. 14/07/2008 Security MetricsPage 4
Growing field
► Areas of interest
► Software security
► Modelling
► Benchmarking
► Return on investment
► Breach data
► Standards
► ISO / IEC 27004
► NIST SP800-55
► Communities
► Securitymetrics.org
► Metricsexchange.org
► Cybersecurity KTN

5. 14/07/2008 Security MetricsPage 5
Securitymetrics.org
► Open mailing list and wiki
► Active community
► Established by Andy Jaquith
► Runs the US based Metricon and MiniMetricon events
each year

6. 14/07/2008 Security MetricsPage 6
Metricsexchange.org
► New open community group
► Established by Elizabeth Nichols
► Sharing metrics definitions, learning and data
► Early days, big ideas

7. 14/07/2008 Security MetricsPage 7
Cybersecurity KTN – Metrics SIG
► UK Knowledge Trading Networks established by DTI
► Promoting collaboration between industry, academia and
government
► Metrics Special Interest Group has focused on the
delivery of the Internet Threat Exposure (ITE) Index
► Threat and Countermeasure focused metric of exposure
► Appears to be aimed at less sophisticated security
practitioners
► Risk assessment-lite ?
► Currently being developed in an open group

8. 14/07/2008 Security MetricsPage 8
Standards
► NIST SP800 – 55
► Exhaustive list of possible security metrics to measure
► 99 pages
► No real sense of what is a useful metric
► Defines useful characteristics to describe a metric
► Performance Goal, Performance Objective, Metric, Purpose,
Implementation Evidence, Frequency, Formula, Data Source,
Indicator
► ISO/IEC 27004
► Currently in draft / closed group
► Metrics covering the performance of an ISMS as defined in 27001
and 27002

9. 14/07/2008 Security MetricsPage 9
Types of Security Metrics
► Risk Metrics
► Compliance Metrics
► Operational Metrics
► Quality Metrics
► Management Metrics
► Business Metrics
► Confusion among practitioners

10. 14/07/2008 Security MetricsPage 10
Focus problems
► Technical Focus
► “What do we count?”
► Business Focus
► “What do we need to do and why?”
► Counting is the mechanical foundation
► The business wants the story the numbers tell
► Metrics are not the answer to funding problems

11. 14/07/2008 Security MetricsPage 11
Other common problems
► Managing to the metric
► No longer focused on the result
► Measuring emerging threats
► Measuring last years breaches

12. 14/07/2008 Security MetricsPage 12
Questions from the board
► Am I safe?
► Can I take responsibility for the actions of my company?
► Who handles my data?
► Who am I doing business with?
► Are they accountable?

13. 14/07/2008 Security MetricsPage 13
Metricon practitioners top 10 metrics
► Data volumes transmitted to competition
► Coverage metrics
► Availability of business systems
► End user perception of security
► Legal fees paid out
► Total cost of information security
► Information asset value
► Count of events on systems
► Security control success rate
► Cost of security monitoring and reporting

14. 14/07/2008 Security MetricsPage 14
Balanced Security Scorecards
► Complete:
► People, Process, Technology, Budgeting, Innovation,
Organisational Planning, Operations
► Traditionally include four primary perspectives:
► Financial
► Customer
► Internal Processes
► Learning and Growth
► Jaquith has a comprehensive chapter on balanced
security scorecards in his book

15. 14/07/2008 Security MetricsPage 15
Geer’s Scorecard
► Finance
► Cost of data security per transaction
► Downtimes lost to attack by attack class
► Data flow per transaction and source
► Budget correlation with risk measures
► Process
► % of critical systems under a DR plan
► % of critical systems obeying the security policy
► MTBF & MTTR for security incidents
► Frequency of security team internal consultations
► Latency to obey security change orders by department

16. 14/07/2008 Security MetricsPage 16
Geer’s Scorecard
► Learning and growth
► % of job reviews involving security
► % of security workers with training
► Ratio of B.U. security staff to central security staff
► Timely new system security consultations
► % of programs with budgeted security
► Customer
► % of SLAs with security standards
► % of tested external facing applications
► Number of non-employees with access
► % of data secure by default
► % of customer data outside the data centre

17. 14/07/2008 Security MetricsPage 17
GE Global experience
► Metrics to drive behaviour
► Scorecard approach
► Business unit drill down and comparison views
► Communication plan was key
► Built a custom system piecemeal over several years
► Started with manual data, automated over time
► Now moving to a common platform
► Monolithic vs Composite data sources
► Centralised vs Business unit data sources

18. 14/07/2008 Security MetricsPage 18
Dept of Veterans Affairs’ experience
► Didn‟t have common definitions of:
► What IT Security was
► What better IT security looked like
► The value of security
► Identified the security events that drove perception of
security
► Focused on the frequency and impact of those events
► Did not ignore uncertainty!
► Results-focused

19. 14/07/2008 Security MetricsPage 19
Intel’s experience
► Developed predictive model for future security incidents
► Used to provide ROI on „reduce the occurrence‟ controls
NOT „reduce the effect‟ controls
► Needed to gather current state data first in order to
identify „Annual Rate of Occurrence‟
► 2 years of data from 20+ global locations
► Needed to estimate „Single Loss Expectancy‟ value for
target environment
► Identified limited target groups to pilot controls in first to
measure results
► Needed a LOT of data
► 87% accurate predictions over a 12 month period

20. 14/07/2008 Security MetricsPage 20
Verizon 2008 Data Breach Investigations
Report
► 500 Investigations over 4 Years
► 18% of breaches were the result of an unpatched system
► 90% of unpatched breaches had had patches publicly
available for 6 months or more
► No more would have been prevented by a patch cycle
shorter than a month
► There is a lot of useful data in this report

21. 14/07/2008 Security MetricsPage 21
Dan Geer’s counterpoint
► We are losing
► The bad guys are in it for the money
► Attackers costs are continually falling
► Need to start measuring „attack metrics‟
► Focus on increasing their cost of attack
► More cost effective to redirect than to resist

22. 14/07/2008 Security MetricsPage 22
Marcus Ranum’s counterpoint
► Statistics only work where:
► Population is large
► Problems are common and widely shared
► Aggressors act consistently
► The only scores that matter are 0% and 100%
► Security is not „risk management‟ it is „complexity
management‟

23. 14/07/2008
Thank you
phuggins@uk.ey.com