Security and Audit for Big Data

IBM Security Systems

Security and Audit for Big Data

Tina Chen
Guardium Enablement
chenti@us.ibm.com

© 2013 IBM Corporation
1



Please note

IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract. The development,
release, and timing of any future features or functionality described for our products remains
at our sole discretion.

2



Innovative technology changes everything

1 trillion
connected
objects

1 billion mobile
workers

Social
business

Bring your
own IT
Cloud and
virtualization

3



Compromises take weeks and months to discover & remediate
Time span of events by percent of breaches

4

http://www.verizonbusiness.com/resources/reports/rp_data-breach-investigations-report-2012_en_xg.pdf?CMP=DMC-SMB_Z_ZZ_ZZ_Z_TV_N_Z038


Four key drivers for data security
“To mitigate business risk, you must proactively protect what matters — customer
data, financial data and intellectual property — from both outside attackers and
privileged insiders.”
GLOBAL
COMPLIANCE
Aggressive New
Regulations

APTs DATA BREACHES
98% Stolen Records
From Large Orgs2

CLOUD ADOPTION

Compliance
– PCI-DSS, HIPAA/HITECH, SOX, GLBA, Basel III and others
– National Data Encryption Laws: UK Data Protection Act, EU Data
Protection Directive, South Korean PIPA and others
– Require encryption, separation of duties, privileged user controls

Data Breaches
– Valuable data is being targeted by sophisticated attackers - data
breaches increasingly result
– IP Protection, US Federal and State Data Protection laws,
Data across Borders
– Encryption + access controls limit risk, meet safe harbor requirements

Enterprise Security
#1 Inhibitor1

Cloud Adoption – Security the #1 concern

BIG DATA

Big Data = Big Risks

Big Data is a
Big Target

5

– Cloud efficiency and flexibility highly desired
– Encryption + access control limits exposure to cloud admins
and other new security risks

– Large data sets inevitably include sensitive data
– All data stores and report locations require protection

From Vormetric
1. Global State of Information Security® Survey by PwC, CIO magazine, and CSO magazine – October 2012
2. Verizon Data Breach Investigation Report – March 2012



The importance of monitoring
Can you answer these questions?
Big Data Platform
Structured

- Who is running map reduce
jobs and what are those jobs
accessing?
- Is there a new job in the
system that hasn’t been vetted?

Unstructured

- Is someone possibly trying to
hack into the file system?

Streaming

Clients

6

Hadoop Cluster



7



Complement existing security with secure databases

8



InfoSphere Guardium protects sensitive data in Hadoop environments and
helps ensure compliance
Introducing Hadoop
Activity Monitoring
o Protect your sensitive data with real time
activity monitoring
o Gain insights into data activity
throughout the stack: Hive, MapReduce,
HBase and HDFS
o Detect unauthorized applications or users
o Real time alerts reduce time to discovery
for possible breach or infraction of
compliance
o Automate compliance and management
tasks
o Infrastructure in place to provide
additional real-time controls over time

9

Monitor and Audit
Hive
Application
MapReduce
MapReduce
Oozie

Storage

HBase
HDFS



How it’s done
InfoSphere Guardium monitors key Hadoop events:
•Session and user information
•HDFS operations – Commands (cat, tail, chmod, chown, expunge, etc), files, permissions
•MapReduce jobs - Job, operations, permissions
•Exceptions, such as authorization and access control failures
•Hive/HBase queries , - Alter, count, create, drop, get, put, list,..

InfoSphere
Guardium
Collector
Appliance

InfoSphere Guardium Reports

S-TAPs

Clients

10

Hadoop Cluster

Heavy lifting occurs on Guardium collector! Very low
overhead on monitored nodes. Architecture supports
separation of duties

Sensitive data
alert!



How it is done

11



Capture and Parsing Overview
S-TAP

Guardium
Collector
Analysis
engine

Hadoop fs –mkdir ….
Parse
commands
then log

Namenode

Hadoop fs –mkdir /user/data/sundari

Sessions

Hadoop fs –mkdir …

Read Only
Hardened Repository
(no direct access)
Objects
/user/data/
sundari

Joe

Commands

Hadoop commands

Joe

12

Hadoop
Client

Hadoop fs –
mkdir/user/data/sundari

mkdirs



A recommended approach

1. Identify users and classes of users – “privileged” users,
data scientists…Who is allowed to access sensitive data
Validate with activity monitoring
2. Identify the applications, jobs, ad-hoc analysis
Validate with activity monitoring
3. When possible identify and mask sensitive data before it
enters the cluster and identify specific directory location in
cluster for that data. Put tighter monitoring controls around
that data.
4. Look at exceptions – permission exceptions, other
operational errors

13



Use cases
Let’s do the following:
•
•
•

Log and/or alert access to sensitive files by “unauthorized or unknown” user
Reporting on new jobs entering the system (identify new MapReduce jobs in the
system)
Exception reporting for permission errors on sensitive data

And for each scenario, how to:
Plan
Monitor
Automate

14



Planning for sensitive data access and monitoring
Plan

o Do you have PCI or other sensitive data?
Is sensitive data already identified in
“source” systems?
o How do you carry that sensitive
identification over to Hadoop?
o What are the internal and external
compliance requirements for monitoring
sensitive data access?
o What is the plan for handling violations?
Who need to be alerted and when?

15



Where is sensitive data?
Keep sensitive data localized, encrypted, and
under monitoring control

Plan

Non-Sensitive
data or known

Sensitive data or
unknown

16

Configuration
files


Monitoring sensitive data
Monitor

Real-time security policies
Default Hadoop Policy

Your policy rules go here, such as sensitive data alerting…

Flexible, granular rules….

17



Determine who is accessing sensitive data
When?

From where?

Monitor

Who?
What?

18



Alerts reduce time to discovery

Automate

Unauthorized access
to sensitive data!

Incorporate Data Events
into QRadar unified view
and real time analytics
19



Planning for application access
Plan

What MapReduce jobs are being used? What
kind of ad hoc analysis is allowed on the
system?
Have they been vetted for access to sensitive
data?
What is a normal pattern of activity?
What process should the team use to
communicate new deployments?

Task

Map

Reduce

(break task into
small parts)

(many results to a
single result set)

No human communication process is infallible…

20



What applications are using the data?
Monitor

MapReduce reports ….

Now, reduce the noise by filtering out authorized jobs….
21



Monitor

Focus your resources on the unknown –
unauthorized MapReduce jobs

22



Audit process workflow and administrative automation

Automate

Should this job be approved?

23




Automate

Business Owner approves or rejects
new applications/jobs

24




Automate


Information Security confirms
Business Owner recommendation

25




Automate

Guardium Admin adds authorized
jobs to “authorized job list”

Information Security confirms
Business Owner recommendation

26



Populate new vetted applications automatically

27

Automate




28

Automate




29

Automate



Alerting off the Exception reports

Monitor

Automate

Table already exists

IO exception

30



Continuous database activity monitoring

• PCI, SOX and HIPAA accelerators included with DAM (guidance, reports,
and more)

31



PCI – Data Access Report

32



Streamline and simplify compliance processes for Hadoop

InfoSphere Guardium Top Benefits
InfoSphere Guardium Top Benefits

Proven track record in data security
Proven track record in data security
Complete separation of duties
Complete separation of duties
Sensitive data monitoring to pass compliance audits
Sensitive data monitoring to pass compliance audits
Privileged user monitoring
Privileged user monitoring
Real-time alerting for abnormal/suspicious activity
Real-time alerting for abnormal/suspicious activity
Full forensics: Any activity – views, changes, updates…
Full forensics: Any activity – views, changes, updates…
Heterogeneous support – IBM, HortonWorks, Cloudera,
Heterogeneous support – IBM, HortonWorks, Cloudera,
Greenplum…
Greenplum…
Same platform for all databases in your enterprise
Same platform for all databases in your enterprise

33



34



IBM Securing all types of data…
Data at Rest

Data privacy for non-production environments:
Optim Data Privacy (DP)

Stored
(Databases, File Servers, Big Data, Data
Warehouses, Application Servers,
Cloud/Virtual ..)

Optim Test Data Management (TDM)

Securing static data on repository :
Guardium Data Encryption

Data in Motion

Data privacy for unstructured data (documents)
– Guardium Data Redaction

Data privacy for production environments:
Over Network
(SQL, HTTP, SSH, FTP, email,. …)

– Guardium Data Activity Monitoring

Configuration

Ensuring database is configured and patched
properly
Repository Vulnerability
(Database Configuration, Patch Level,
OS Security, …)

35

– Guardium Vulnerability Assessment
– QRadar/QVM


Data Masking for Data Privacy
Mask confidential data to avoid data breach & meet privacy compliance

Requirements
Mask files
Mask files

Hadoop

Database
IMS
VSAM
More…

Extract, mask & load
Extract, mask & load

Mask in-Hadoop
Mask in-Hadoop
using MapReduce
using MapReduce

Mask in-database
Mask in-database

• Protect confidential data
while preserving analytics
• Mask data anytime,
anywhere
• Mask data in Hadoop
using MapReduce
• Implement proven built-in
masking algorithms
• Support compliance with
privacy regulations

Benefits
InfoSphere Optim

Mask
JASON MICHAELS
JASON MICHAELS

Before Masking
36

ROBERT SMITH
ROBERT SMITH

• Protect sensitive
information from misuse
and fraud
• Prevent data breaches and
associated fines
• Achieve better information
governance

After Masking


Optim Data Masking implementation in Hadoop
Optim Masking can also be executed in
Hadoop for delimited files.
Java application/interface for masking.
•
•
•
•

MapReduce base classes and helpers.
Distributed cache.
Shared libraries.
Use of masking in Reducers.

Declarative specification of:
• Metadata of data files
• Masking rules

OPTIM
JASON MICHAELS
JASON MICHAELS

Before Masking

37

ROBERT SMITH
ROBERT SMITH

After Masking



Intelligence: A comprehensive portfolio of products and services

38

New in 2012

Products

Services



All domains feed Security Intelligence

Correlate new threats based on
X-Force IP reputation feeds

Hundreds of 3rd party
information sources

Guardium

Identity and Access Management

Database assets, rule logic and
database activity information

Identity context for all security
domains w/ QRadar as the dashboard

Tivoli Endpoint Manager
Endpoint Management
vulnerabilities enrich QRadar’s
vulnerability database
39

IBM Security Network
Intrusion Prevention System
Flow data into QRadar turns NIPS
devices into activity sensors

AppScan Enterprise
AppScan vulnerability results feed
QRadar SIEM for improved
asset risk assessment


Key Business Drivers for InfoSphere Guardium
Continuously Monitor All Access too..

Prevent data breaches

Assure data governance

Reduce cost of
compliance

40



Extend real-time Data Activity Monitoring to also protect sensitive data in
data warehouses, Big Data Environments and file shares
DATA

Big Data
Environments
InfoSphere
BigInsights

GreenPlum
CouchDB
Integration with
LDAP, IAM,
SIEM, TSM,
Remedy, …

41



Information and community

InfoSphere Guardium YouTube Channel – includes overviews
and technical demos
developerWorks forum (very active)
Guardium DAM User Group on Linked-In (very active)
Community on developerWorks (includes content and links to
a myriad of sources, articles, etc)
New! InfoSphere Guardium Virtual User Group.
Open, technical discussions with other users.
Send a note to bamealm@us.ibm.com if
interested.

42


Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response
to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed or misappropriated
or can result in damage to or misuse of your systems, including to attack others. No IT system or product should be considered completely secure
and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to
be part of a comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems,
products or services to be most effective. IBM DOES NOT WARRANT THAT SYSTEMS AND PRODUCTS ARE IMMUNE FROM THE
MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.

ibm.com/guardium
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes
only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use
of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any
warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement
governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in
all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole
discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any
way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United
43 States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.


Disclaimer
Please Note:
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion.

44


Security and Audit for Big Data

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Security and Audit for Big Data

Similaire à Security and Audit for Big Data (20)

Plus de Nicolas Morales

Plus de Nicolas Morales (14)

Dernier

Dernier (20)

Security and Audit for Big Data