Big Data, Big Picture: Can You See It?

Big Data Big Picture:
Can You See It?
Mike Harer
Senior Principal Product Manager, Big Data
Troy Coleman,
Senior Principal Product Manager, Chorus and DB2 for z/OS

2 © 2015 CA. ALL RIGHTS RESERVED.
Today’s Discussion
THE FUTURE
EMERGING NEEDS
BIG DATA MANAGEMENT INNOVATIONS FROM CA
THE PATH FORWARD
1
2
3
4
Q & A5

THIS IS THE AGE OF
THE APPLICATION
ECONOMY
AND IT’S ALL ABOUT
THE DATA…
DATA IS THE FUEL
THAT CAN MAKE OR
BREAK YOUR
BUSINESS

DEMANDING
GROWTH
SPINNING UP
NEW PROJECTS
USING ALL DATA
AVAILABLE
Businesses Moving Fast Into Big Data
Source: 2015 CA Sponsored Research: Vanson Bourne Global Big Data User Survey
60% want better
customer experience
54% want new
customers
84% will deploy an
initiative in the next 1
year
94% plan to use all data
available to them

How The BI Ecosystem Is Evolving
Example: Retail Store Analytics
NAS
POST-2014
HADOOP DISTRIBUTED FILE SYSTEM
(UNSTRUCTURED/STRUCTURED)
NFS
FTP
ETL
M/R
Hive
Pig
R
Mahout
RDBMS
HBASE
Storm
Kafka
Cassandra
Drill
Solr
Elastic
Search
SAN
(RDBMS/FILES)
Prod. Catalog
Transaction Log
(Credit)
Activity Log
Log Analysis
Customer Insight
Recommendation Engine
Web Session Analysis
Transaction Analysis
Personalized Offers
Transaction Capture
Web Sessions
Shopping Carts
FLASH
INGESTION BATCH PROCESSING/
MACHINE LEARNING
OLTP STREAM
PROCESSING
INTERACTIVE
ANALYTICS
SEARCH
PRE-2014
Extract, Transform & Load
all data into
Datawarehouse
Capture All Transactions in
ERP/Relational DB
Publish Promotions every
week
Create Datamarts and Run
Reports every day/week
ERP OPERATIONS
OLTP OPERATIONS
ETL OPERATIONS BUSINESS
INTELLIGENCE
BATCH OUTPUT
MANAGEMENT
SAP, Oracle
Peoplesoft
Informatica
FTP/SFTP
Oracle DB, DB2DB
SQL Server
CRON, Email

Result Is Complex Ecosystems
MULTIPLE → DOMAINS – IMPLEMENTATIONS – ENVIRONMENTS
Big Data Analytics
Platforms
Traditional BI –
Data Warehouse
Mainframe
and Distributed
Amazon EMR Console
Unstructured Data Structured Data

Key Insight: Complexity Is The New Normal
GROWTH IN # OF BIG DATA PROJECTS OVER TIME
AS BIG DATA DISRUPTS COMPUTING PARADIGMS – GET AHEAD OF
THE MANAGEMENT OF INFRASTRUCTURE NOW OR FACE THE
CHALLENGES OF DEALING WITH COMPLEXITY

What We Hear From Users
92%
admit to major
challenges managing
their organization’s
infrastructure to
support current and
future Big Data
initiatives
31%
36%
40%
43%
48%
Lack of visibility into information and
processes
Difficulty reintegrating analysis
Cost
Complexity of managing so many different
solutions
Complexity of managing such a large
implementation
COMPLEXITY IS 2 OF THE 5 LARGEST CHALLENGES
IN MANAGING BIG DATA ENVIRONMENTS

Key Project Success Factors
40%
45%
47%
49%
57%
Cloud/hosted infrastructure services
Management tools for the infrastructure in place
Hire new resources with required skills
New infrastructure (storage, etc.)
Train existing resources on Big Data technologies
TOP 5 MAJOR INVESTMENTS NEEDED FOR PROJECT SUCCESS

The “Big” Big Data Management Pains
The Need to Overcome Many Challenges
 Managing complex multi-vendor Big Data environments
 Finding Hadoop/Big Data experts
 Understanding capacity requirements for rapidly changing
business needs
 As complexity increases, manual processes are often required
 System problems are hard to isolate, downtime increases
 Unique tools and shortcomings
 Driving forces… acquisitions, department consolidations
demand greater operational efficiency

Big Data Management Innovations from CA
UNLOCK DATA
Getting broader:
Unlocking insights from
your mainframe data
VSTORM CONNECT DATA
STREAMING FOR BIG DATA
MANAGE
INFRASTRUCTURE
Getting to growth: Big
data projects are critical
for business value –
protect this revenue by
managing diversityCA BIG DATA
INFRASTRUCTURE
MANAGEMENT
BDIM Pre-Release Registration and to
Learn More:
www.ca.com/bigdata

DESIGNING FOR ROLE OF BIG DATA ADMINISTRATOR
The Role Of The Big Data Administrator
 Perform day-to-day operations and support
of Hadoop infrastructure
 Monitor/maintain existing clusters and
provision new ones
 Integrate enterprise monitoring tools
 Analyze current workloads and perform
capacity planning
 Hadoop Multi-Vendor Management
 Hadoop Resource Management /Reporting
 Hadoop Process Management /
Automation
 Hadoop Job Management & Monitoring
 Hadoop System Health Monitoring & Alerts
ROLE / RESPONSIBILITIES: KEY MANAGEMENT CAPABILITIES:

360 Degree Approach
SINGLE, CONSISTENT
MANAGEMENT UI EXPERIENCE
Linux / x86
SINGLE ACCESS POINT
INTO
HETEROGENEOUS
ENVIRONMENT
OPERATIONALIZE , MANAGE MULTI-VENDOR
HADOOP MANAGEMENT DOMAINS
Big Data Infrastructure
Management Server
CA BIG DATA INFRASTRUCTURE MANAGEMENT BIG DATA INFRASTRUCTURE
Current
Future
Mainframe
HADOOPECOSYSTEM

CA Big Data Infrastructure Management
Single Unified View
Job monitoring
Heterogeneous system
management
Intelligent alert
management
Resource
reporting
Cluster/Job/Node
management
BDIM
Pre-Release Registration
and to Learn More:
www.ca.com/bigdata

vStorm Connect Data Streaming for Big Data
 It’s extremely difficult for Data Scientists, CMOs and other stakeholders to get access to their raw System z data in
tandem with machine logs and other types of transactional information.
 Need to make it easier for Mainframe customers to participate in Big Data projects.
 Simplifies and increases the ability to move Mainframe data into the Hadoop Big Data environment.
LAS VEGAS, November 10, 2014 — CA WORLD ’14 — CA Technologies (NASDAQ:CA) today announced a new global distribution
agreement with Veristorm, a software company focused on Big Data management. The agreement strengthens CA’s ability to
help customers leverage key business data on the mainframe for Big Data and analytics projects.
Logs
IMS
VSAM
Datacom
IDMS
DB2
WHY?
 A new software solution that enables efficient data
integration into the Big Data ecosystem.
 Provides secure, near real-time access to mainframe
and distributed data for processing and self-service
analytics.
 New Extract-Hadoop-Transform technology, much
faster than traditional ETL and staging processes,
allowing enterprise data to be accessed by the analytics
platform of choice.
WHAT?

How it Works
 vStorm Connect extracts data from DB2, VSAM, IMS, IDMS
and Datacom and streams into Hadoop on the customer
platform of choice (System z, Power Series or x86) thru a
point-and-click, self-service portal.
 Decreases the use of Mainframe general processors (reduction in
MIPS/MSU utilization and associated expenses).
 Lowers Mainframe storage costs, which can often reach up to
$100k per TB, by offloading System z data into Hadoop off-
platform.
 Enables customer to gain new insight into key factors driving
business performance.
 Superior Technology - Unique because it does not require staging
like traditional ETL solutions, lowering TCO & providing the data
scientist real-time access to their most important transactional
data.
HOW IT WORKS
KEY BENEFITS

Obstacles to Include Mainframe Data
1 2
Data Governance as
the data moves off z/OS
operational systems
Data Ingestion from z/OS into Hadoop (on or off
platform) is a bottleneck (MIPS & ETL cost, Security
around data access and transfer, Process Agility)
 Existing security policies must be applied to data access and transfer.
 There needs to be high speed / optimized connectors between traditional z/OS
LPARs and the Hadoop clusters
 Ability to serve data transparently into Hadoop clusters on mainframe AND on
distributed platform
LEAD TO KEY REQUIREMENTS:

Data Ingestion Challenges
Extract from
proprietary
formats.
Aggregate or
summarize.
Staging
Transform
Load
Hadoop, MongoDB,
Cassandra, Cloud, Big Data
Ecosystem, Java, Python, C++,
Interface skills
JCL, DB2, HFS, VSAM, IMS, OMVS,
COBOL Copybooks, EBCDIC, Packed
Decimal, Byte ordering, IPCS, z/VM,
Linux on z
z/OS
Logs
IMS
VSAM
Datacom
IDMS
DB2

vStorm Connect – Mainframe data to Mainstream
Broad Platform Coverage
 ELT alternative:
Extract-Hadoop-
Transform saves
MIPS, staging
storage, complexity
 Dest: Cloudera,
Hortonworks, etc
 Metadata preserved:
Use SQL on Hive
 Governance,
sandbox: zDoop
Datacom
IDMS
Logs
IMS
VSAM
DB2
System Z
z/OS
vStorm Connect
Data Streaming
for Big Data
Data+
MetadataNo Staging!
vStorm Enterprise
Power Systems
vStorm Enterprise
x86
vStorm Enterprise
Linux on z
Linux

CA’s Big Data Management Approach
Deliver analytics solutions that remove
the need for special skills and knowledge
of the tools and infrastructure
Enable use of MF data (structured and
unstructured) for analytics on any
platform
1
Manage the complex analytics
environment and simplify the complexity
2
Secure and enable compliant access
control to data in analytics databases
3
4

CA DB2 Database Management for DB2 for z/OS

CA Chorus for DB2 Database Management
Modernizing Management of DB2 for z/OS
Released 4.0 in April 2015
 Custom Dashboard
 Performance Warehouse
Charting and Comparing
 Policy Health Alerts and
Investigation

CA Database Management Solutions for DB2 for z/OS
Agile Incremental Release 19.0
 CA Database Administration
Suite for DB2 for z/OS
 CA SQL Performance Suite for
DB2 for z/OS
 CA Sysview Performance
Management Option for DB2
for z/OS
 CA Database Backup and
Recovery Suite for DB2 for z/OS
Sprint Reviews
with Users
Prototyping
User Research
User Feedback
Usability
Testing
Product
Backlog
4 week
sprints

BDIM
Pre-Release Registration
and to Learn More:
www.ca.com/bigdata

For Informational Purposes Only
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. This presentation is
intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer
references relate to customer's specific use and experience of CA products and solutions so actual results may vary.
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect
the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating
to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This
presentation is based on current information and resource allocations as of March 01, 2015 and is subject to change or
withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in
this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release
referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major
product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance
and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any
contract.
CA does not provide legal advice. Neither this document nor any CA software product referenced herein shall serve as a
substitute for your compliance with any laws (including but not limited to any act, statute, regulation, rule, directive, policy,
standard, guideline, measure, requirement, administrative order, executive order, etc. (collectively, “Laws”)) referenced in this
document. You should consult with competent legal counsel regarding any Laws referenced herein.
Terms Of This Presentation

Big Data, Big Picture: Can You See It?

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Big Data, Big Picture: Can You See It?

Similaire à Big Data, Big Picture: Can You See It? (20)

Plus de CA Technologies

Plus de CA Technologies (20)

Dernier

Dernier (20)

Big Data, Big Picture: Can You See It?