From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Geer - Hutton - Shannon - A Pilot Project On The Use Of Prediction Markets In Information Security
1. A Pilot Project on the
Use of Prediction Markets
in Information Security
Dan Geer, In-Q-Tel
Alex Hutton, Verizon Business
Greg Shannon, Carnegie Mellon
April 20th, 2011
alpha-pilot at securitypredictions dot com
2. Overview
Motivation (dg)
Prediction Market Examples (gs)
What is the pilot; what information will it generate? (gs)
Why is this valuable to the infosec industry? (ah)
How is this helpful to security teams and professionals? (ah)
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 2
3. Motivations
Our Goal: Accelerated aggregation and dissemination of
actionable security information from diverse sources
Purpose of this talk: Explain the Pilot Project
Purpose of the pilot: Validate that we can use a market to
collect informed opinions from participants that when
aggregated and shared is of interest to individuals,
organizations and the information security industry.
Excellent overview and references in:
"Using Prediction Markets to Enhance US Intelligence Capabilities," CIA Center for
the Study of Intelligence, 2006, v50 n6, PDF 17pp. http://tinyurl.com/6kdqpl
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 3
4. The Art in Prediction
In prediction markets, the art is selecting the questions,
i.e., prediction markets are invulnerable to idiots but not
to idiotic questions.
Science and practice alike have shown that prediction
markets have greater accuracy than surveys and, unlike
surveys, can be run continuously.
As the rewards available to market participants rise, the
precision of the market's predictions improves.
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 4
6. A Simple Market Example
http://en.wikipedia.org/wiki/Prediction_market
Will candidate X win election Y? Yes or no?
Three elements: Participants, Contracts, Incentives
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 6
7. Primer
What are Prediction Markets?
Large groups of people are smarter than an elite few,
no matter how brilliant — better at solving problems,
fostering innovation, coming to wise decisions,
even predicting the future.
— James Surowiecki, author of The Wisdom of Crowds
def. Speculative markets used to make predictions of specific
events. Contracts representing the event, or outcome, are
bought and sold resulting in contract price fluctuations. The
current price represents the current group estimate of the
likelihood of the event.
April 2011 7
8. How They Work:
Reflecting Confidence in Outcomes
Individual answers are anonymous, market aggregates consensus
Participants are incented to express the strength of their confidence
Participants are rewarded based on the accuracy of their contributions
Social collaboration and comments by question, surface root causes
April 2011 8
9. How They Work:
Revealing Early Warning Indicators
Participants invest in stocks (buy/sell) and thus drive the price up or down.
The price reflects the crowd’s confidence in the stated outcome.
Decision-makers receive an analytical, real-time consensus view into the true
state of key issues.
Project Aries will achieve customer acceptance by 30-Sept-2011.
Information
contained in
dropping
confidence
April 2011 9
10. Social Analytic Reports &
Decision Dashboards
Tracking changing trends in
consensus opinions
Identifying divergent opinions
among participants subgroups –
Monitor
par*cipa*on
where does the information
to
ensure
diversity
reside?
April 2011 10
11. Pilot Overview
60-day alpha pilot
Use Consensus Point as the market platform
20-30 hand-picked participants
Internal (market) recognition as the incentive
Binary contracts varying in topic and duration
Written by Geer, Hutton, Shannon
Pilot objectives:
At least 10 contracts open at all times
20 contracts with at least 10 participants,100 trades
Positive survey results from participants at the end
At least 3 unclosed contracts estimating future events
Have a contract payout on an unexpected security event
Gain enough confidence to start a half-year beta
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 11
12. What Do We Want To Know?
What is the collective, anonymous, incented opinion
about actionable information security events and states of
the world?
How accurate and stable is this opinion/knowledge?
Can this knowledge benefit participants, 3rd parties and
the industry to improve information security?
Can a prediction market mitigate the unavailability of
detailed operational infosec data?
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 12
13. Criteria For Contracts
A binary question
Good: The market-cap leader in consumer operating systems issues a press-release on a
security-critical patch this quarter.
Poor: The number of software vulnerabilities discovered in the most popular consumer
operating system increased this quarter over the previous quarter.
A definitive authority on the result
Good: government agency, public company, nationally-recognized institution
Poor: news, an individual, on-line poll, micro-blog traffic
A history of indisputable previous outcomes
Good: Alerts issued, scores published, reports published
Poor: News articles, court documents, non-public sources
Market information is likely actionable
Good: A disruptive OS patch is in the pipeline
Poor: Companies will lose more data this year than last
Morally benign
Difficult for single entities to influence the outcome of the underlying event
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 13
15. Other Candidate Sources & Contracts
US-CERT alerts
Botnet species announced
Statistics from data breach reports
Trends in security surveys and indexes
Statistics from software security or controls reports
MITRE CVE reports
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 15
16. Criteria for Alpha Participants
Demonstrated knowledge of information security
At least 5 years of professional experience in such
Diverse across
Sectors: Government, Industry, Academic
Verticals: Civilian Gov’t, Health, Financial, DoD, Telecom, etc.
Layers: hosts, networks, applications, infrastructure, content
Life cycle: creation, installation, operation, incidents, remediation
Specialties: privacy, risk, availability, integrity, etc.
Demographics
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 16
17. Incentive Criteria
Is legal
Is sufficient to entice participants to divulge their
knowledge through market activity
Benefits are tangible to all participants
Not just the top performers
Does not encourage market manipulation or spectuation
Scales to 50 active contacts and 1,000 participants
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 17
18. Value to the InfoSec Industry
Opportunity for big-time benefit to the industry.
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 18
19. Value to the InfoSec Industry
A prediction market is a specifically framed piece of
knowledge (belief as a probability)
What do you want knowledge about?
Understand trends as they happen (or don’t happen)
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 19
20. Value to the InfoSec Industry
Suggested context:
Capability to manage
(skills, resources,
asset
landscape decision quality…)
impact
landscape
risk
threat
landscape
controls
landscape
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 20
21. Value to the InfoSec Industry
Example: Mobile Malware
% Mobile devices as targeted asset in 2011 DBIR
% Mobile devices as targeted asset in 2012 DBIR
% Mobile devices as targeted asset in 2013 DBIR
The effect of new vulnerability research on the above contracts...
The effect of new security technologies on the above contracts...
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 21
22. Value to the InfoSec Industry
Suggested context:
Capability to manage
(skills, resources,
asset
landscape decision quality…)
impact
landscape
risk
threat
landscape
controls
landscape
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 22
23. Value to InfoSec Teams and Professionals
An internally facing prediction market can be used for
decision support
Success/Failure of big dollar security projects
What current projects (both security and non-security) mean
to the frequency or impact of security events
Impact of current security events
This breach will cost how much?
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 23
24. Value to InfoSec Teams and Professionals
Calibration
Ability to better qualify the subjective evidence around us
Ability to “mine” changes in “price” for causes
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 24
25. Recap
Our Goal: Accelerated aggregation and dissemination of
actionable security information from diverse sources
To follow or join the pilot send e-mail to:
alpha-pilot at security predictions dot com
Geer Hutton Shannon Pilot Project for an InfoSec Prediction Market April 2011 25
26. On The Use of Prediction Markets in
Information Security (from src-bos program)
A tool created to help establish beliefs as probabilities, prediction markets are
speculative markets created for the purpose of understand the probability of future
events. Not widely used in Information Security, Prediction Markets may have
benefits to our industry. Dan Geer, Alex Hutton and Greg Shannon will give a
background around what prediction markets are, how they can be used by the
information security industry as a whole, and how security departments and
professionals can use them as a tool to help defend their environments.
Dan Geer is a computer security analyst and risk management specialist and
currently the chief information security officer for In-Q-Tel.
Alex Hutton is a principal for Research & Intelligence with the Verizon Business
RISK Team.
Dr. Greg Shannon is the chief scientist for the CERT® Program at Carnegie Mellon
University’s Software Engineering Institute.
http://www.sourceconference.com/boston/speakers_2011.asp#dgeer
Geer Hutton Shannon Pilot Project for an InfoSec Prediction April 2011 26
Market