TRACK 1 A - The Big Data Imperative for Smarter Defence - Tim Paydos.pdf

1. Defense Intelligence & The Information Challenge
Tim Paydos
Director, IBM Threat Prediction & Prevention
9 October 2012
1 © 2009 IBM Corporation

2. “In our ever-changing world, America's first line of defense is timely, accurate
intelligence that is shared, integrated, analyzed and acted upon quickly and effectively.”
- President Barack Obama, 1/7/10
This isn’t an Information Sharing Problem. Its an Information
Management Problem. Information Sharing Is only one piece of
the larger problem
General Michael Hayden, Former Director CIA, Former Director NSA
It is not just about sharing information. Its about making sense of it and exploiting it
once we have it. There is too much information to identify that which is important. We
need tools that tell us what’s important, and what is relevant.
Juan Zarate, Deputy National Security Adviser for Combating Terrorism
2
© 2009 IBM Corporation
© 2009 IBM Corporation

3. Today’s Discussion
If you only remember four things
Intensifying threat and policy pressures have pushed agencies to a
1 tipping point – they know they need to transform
Agency leaders embrace this, are defining the new requirements,
2 and are demonstrating success
Achieving transformation requires a broad set of capabilities, and a
3 combination of technology & expertise
The experience and assets exist to help you define a strategy and
4 a roadmap to guide your transformation
3 © 2009 IBM Corporation
FOR INTERNAL IBM USE ONLY

4. Agency Leaders Telling Us What They Need
Trusted Information -- On Multiple Levels
Insightful
Comprehensive analytics to drive
new meaning from your data
Can I extract new Insight to make it all worthwhile?
Governed & Secure
Rules are in place and tools are deployed to limit visibility,
secure sensitive information, and protect privacy
Can I Trust My Partner and My Own Agency?
Accurate & Complete
Complex and disparate data transformed, cleansed,
reconcile and delivered
Can I Trust The Information?
© 2009 IBM Corporation

5. Military & Intelligence Agency Challenges
Leaders Often Stuck Between A Rock And A Hard Place
Today’s intensifying challenges
mandate a fresh approach to Traditional Approaches
managing threat information have become obsolete
Multiplication of threat types, and frequency Information is compartmentalized – lack of full
with which they occur integration obscures visibility
Threats are increasingly asymmetrical Query State limits ability to address complexity of
threats
Explosion in complexity of threat identification
Inability to manage and search across an expanding
Multi cultural nature of citizenry array of unstructured data sources
Frequency of transaction/interactions Inability to link unstructured content with structured
data and manage together
Social Media and “Big Data” playing an
increasingly important role Untimely – Sense & Respond vs. Predict & Act
Transparency is clouding Out of context – lack of visual analysis,
collaboration, and support/guidance once threat is
Citizen, Policy Maker and Regulatory identified
expectations and pressures are increasing
5 © 2009 IBM Corporation

6. The Big Data Challenge Is Only Making It Harder
Big Data Is An Explosion In Volume, Variety & Velocity
In 2008, we created 40 Exabytes – more than in all recorded history . Now we we create
2.5 Quintillion bytes, or 2.5 Exabytes of data each day — so much that 90% of the data in
the world today has been created in the last two years alone.
• For Every 5 Minutes In Realtime, 60 Hours Of Video Is Up-Loaded to You
Tube
• 12 terabytes of Tweets are created each day, providing insight into public
sentiment
Volume of Digital Data
• 80% of all the data created daily is unstructured – videos, images, emails,
and social media
• Structured data now includes a massive range of sensors, click streams,
log files, call records, transactions
Variety of Information • 5 billion financial transactions occur every single day
• There are 30,000 commercial air flights, accounting for 1,500,000 air
passengers – every single day
Velocity of Decision Making © 2009 IBM Corporation

7. Smarter Agencies Leading The Way
Optimize decision making with actionable insights
Aware
Obstructed view
Aligned
Variety
Fragmented
Inability to predict of insight
Lack
Volume
Inefficient access Volume
VelocityVelocity Rigid
Agile
Variety
Inability to predict
Volume
Velocity
Inefficient access Skilled analytics experts
Everyone
Lack of insight
Variety
Velocity Distracted
Focused
Reactive
Proactive
© 2009 IBM Corporation
7

8. Leading the Way
Government Leaders Moving to Address These Challenges
Streamlined Information Sharing Western National
8 Law Enforcement Registry of Identities, Objects and Events
Across Fed and Local Agencies
Automated content extraction, entity
Western National resolution and analysis from seized
Child Predator Investigation Law Enforcement
assets
US Department of Connect the dots, predict and prevent
Protecting the homeland Homeland Security threats
US Department of Perpetual credentialing and vetting
Insider Threat Defense across branches and bases
Centralized Proactive enterprise data activity
Screening Database monitoring & extrusion prevention
Insider Threat
© 2009 IBM Corporation

9. Leading the Way
Government Leaders Moving to Address These Challenges
State of The Art covert surveillance system
National Border & Security based on Streams platform
Realtime Information Sharing, Discovery &
Crime Prediction & Prevention analysis
National Borders & Security: Real time network intrusion detection, sub-
Cyber Security millisecond analysis and response
National Borders & Security: Real time threat detection & prevention through
Big Data & Watson
Threat Prediction & Prevention
© 2009 IBM Corporation

10. Leading the Way
Maritime Threat Intelligence
10 © 2009 IBM Corporation

11. Leading the Way
Who is Talking to Whom
Stream A
Denoising & Social
Conversation Pairing Speaker Detection Network Analysis
A B Olivier Mihalis
talks to talks to
C D Ching-Yung Upendra
Stream B talks to talks to
Stream C
E Deepak
After denoising
- Just-in-time - Just-in-time - Social network
Stream D
- Features: Volumetrics - Features: GSM domain - Fusion technique
- Very high accuracy - High accuracy - Iterative method
- Very low complexity - Moderate complexity
- Robust to noise - Robust to noise
© 2009 IBM Corporation

12. Leading the Way
Department of Immigration and Citizenship (Australia)
Challenges
DIAC, Australia has embarked on a major transformation program to fix key identity issues :
Siloed databases, little connectivity between sources
Gaps and vulnerabilities in their existing implementation
Inconsistent user, citizen, and immigrant identity verification techniques
Receive 4 million Visa applications annually
Solution
Launched a four year program to transform existing systems using a Service Oriented Architecture (SOA) and implementing a
client-centric organization by providing:
Roles-based portals to case management personnel
Single view of a person’ across all client data sources (over 150 million ‘person’ records)
SOA transformation and integration of legacy environment
Business Benefits
United previously fragmented person’ data and improved processing time 20%
Improved ‘person’ identity resolution resulted in significant reduction in false negatives and inappropriate actions being
taken
Dramatic cost savings from streamlining associated business processes
12 © 2009 IBM Corporation

13. What We Have Learned
Big Data Requires A Different Approach – It Breaks The Traditional Analytics Model
Traditional Approach Big Data Approach
Business Users IT
Determine what question Delivers a platform to enable
to ask creative discovery
IT
Structures the data to Business
answer that question Explores what questions could
be asked
Structured & Repeatable Analytics Iterative & Exploratory Analytics
•Query Based -- Questions Drive Data •Autonomic -- Insight Drives Answers
•Citizen Surveys – Push VS. •Citizen Sentiment – Pull
•Monthly, Weekly, Daily •Persistent & Ad Hoc
•Data At Rest •Data In Motion
© 2009 IBM Corporation

14. What We Have Learned
A New Approach Is Required – Smarter Analytics Supporting Big Data
Advanced Analytic Applications
– Integrate and manage the full breadth of data
types, and at full volume scale
– Gain Insight from Data In Motion, Versus Data At
Rest
– Apply advanced analytics to information in its
native form
– Visualize all available data for ad-hoc analysis
– Development environment for building new
Big Data Platform
analytic applications
Process and analyze any type of data – Workload optimization and scheduling
– Security and Governance
Accelerators
© 2009 IBM Corporation

15. What We Have Learned
A Complete Set of Capabilities Is Required To Address The Challenge
© 2009 IBM Corporation

16. What’s Required To Bridge The Gap
Establish, Govern, Manage & Deliver Information That You Can Trust
IBM InfoSphere Information Server
Unified Deployment
Understand Cleanse Transform Deliver
Discover, model, and Standardize, merge, Combine and Synchronize, virtualize
govern information and correct information restructure information and move information
structure and content for new uses
Unified Metadata Management
Parallel Processing
Rich Connectivity to Applications, Data, and Content
© 2009 IBM Corporation
16

17. What’s Required To Bridge The Gap
And Monitor Continuously & In Real Time
Traditional Approach: Native Logging
• Lives within the silo – limits visibility
• Ex Post Facto, insecure, costly, inaccurate
• Can’t identify who and when data is accessed
• Does not block unauthorized data-extrusion
• Severe negative impact on database performance
4th Generation Enterprise Activity Monitoring
Discovers and classifies sensitive data
Lock down on who has access to what
Identifies the specific person (IP address) who touched
the data, how and when – tamper proof audit
Provides separation of duties – access to DB functions
does not grant the individual the right to see all
sensitive information
© 2009 IBM Corporation

18. What’s Required To Bridge The Gap
Persistent Relationship Awareness & Rules of Visibility
Queries & Data Flow
Through The Same
DATA “Smart” Channel Enterprise
DATA
Limited
DATA Alerts sent to
analyst
proactively
Restricted
18 © 2009 IBM Corporation

19. What’s Required To Bridge The Gap
Persistent Relationship Awareness – IBM Identity Insight
A highly-specialized Identity Analytics repository providing real-time detection of obvious and non-obvious
relationships between people, organizations, events, and other identity types
Addresses The Thorny Issues Of:
•Realtime Discovery
•Enterprise Amnesia - Continuous
•Enterprise Brain Hemorrhage - Persistent
•Complex and Expanding attribution
•Degrees of Separation
DATA •Extensibility
•Complex event processing Linked to Identities
•Information pedigree and audit trail of
DATA merges/splits
•Anonymization
DATA
19 © 2009 IBM Corporation

20. What’s Required To Bridge The Gap
Rules of Visibility – IBM Initiate
Provides accurate, real-time access to entity, object and event data across
disparate sources, systems and networks
Relationship & Hierarchy Management Views
Collaborative, Visual Data Stewardship Capabilities
Data Security & Privacy
Enterprise
Search Capabilities
Highly configurable/custom composite views
Open integration options
Dynamic implementation models
Limited
Data security & privacy
Reporting and analytics
Restricted
20 © 2009 IBM Corporation

21. Taking it to the Next Level
with Advanced Business Analytics
What is
happening What could
happen?
Simulation
How many,
how often,
where?
What if
How can we achieve
these trends Language &
the best outcome?
continue? Sentiment
What Optimisation
Forecasting
exactly is
the
problem?
What will How can we achieve
Structured Data What Are
Made What
happen next the best outcome and Extracting
People
& Unstructured consumable and actions are
if? address variability? concepts and
Predictive Stochastic Talking About
Content accessible to relationships
needed? & Feeling
everyone Modelling Optimisation
Descriptive Predictive Prescriptive Content Web
Analytics Analytics Analytics Analytics Analytics
21 © 2009 IBM Corporation

22. Today’s Discussion
If you only remember four things
Intensifying threat and policy pressures have pushed agencies to a
1 tipping point – they know they need to transform
Agency leaders embrace this, are defining the new requirements,
2 and are demonstrating success
Achieving transformation requires a broad set of capabilities, and a
3 combination of technology & expertise
The experience and assets exist to help you define a strategy and
4 a roadmap to guide your transformation
22 © 2009 IBM Corporation
FOR INTERNAL IBM USE ONLY