Conflict in the Cloud – Issues & Solutions for Big Data

Conflict in the Cloud
Big data and cloud computing
Keith Peterson, CEO
Halo BI
©2014 Halo Business Intelligence | All Rights Reserved

Starting points
• Information management and analytics issues hurting business objectives
• Taking days and weeks to get to the data
• Multiple copies of data around the organization
• No shared view of the truth
• ETL and data warehouse unable to handle loads
• BI and reporting eating up capacity
• Data volumes growing but budgets static
• Desire to leverage new machine data sources

The “Big Data” challenge (Executive View)
…
© 2014 Halo Business Intelligence | All Rights Reserved

The “Big Data” challenge (Business View)
…
Big Data Is…
• Ever escalating volumes
• Expanding sources, such as Internet Of Things
• Increasingly high velocities
• With a widening variety of unstructured formats and semantic contexts
Big Data Is Not Really Useful
Unless insight can be gleaned through analytics...with a reasonable effort!

Three Strategies to Deal with Big Data
1 212 313
Ignore It Archive It Analyze It
Don’t jump on the
bandwagon
You have better things to
focus on
Just collect and store it
You can always analyze
it when resources free
up
With a clear business
problem and ROI
Invest in infrastructure to
derive the insights
needed

Big Data
Google 1 Trillion Web Pages per Year
Facebook 1 Million GB of Disk Storage
Yelp! 100 GB of log data per day
Youtube.com 20 Petabytes new video per year
.
.
Regional medical center – patient sensors 25 TB
Mid-market retailer – POS 10 TB
Mid-market manufacturer – machine sensors 6 TB
http://www.google.com/trends/explore#q=%2Fm%2F04y7lrx%2C%20Amazon%20Aws%2C%20Rackspace&cmpt=q

Five Big Data Questions
111 212 313 414 515
Left Behind? Cloud? Data? Tools? Usefulness?
Everyone is doing
it…you need to.
Really?
Or will cost
exceed benefit?
Big Data requires Big
Compute
Outsourcing risks:
• Loss of Control
• Platform Reliability
• Privacy
• Security
Which data and
sources?
Too much to handle
Some or all?
All vendors have
a Big Data suite
Which one?
How to query the data
Skills needed
Machine Learning

Big Data and Cloud Computing
Commodity computing to execute distributed queries across
multiple data sets
Rent commodity server instances to execute computation
remotely
Cloud hosting for $10/TB/Mo

Traditional BI Architecture
On Premise Or Cloud
Operational Data
• Data Volumes = 100 GB – 5 TB
• Manageable on-premise or in the cloud
Data Warehouse

Add Big Data
Data volumes = 6 TB +
image
Big Data Logs
Cluster

Big Data in the Cloud
The “Traditional” Approach
Data Platform
Commodity Storage Traditional RDMS
Client
Familiar BI Tools
MPP
SQL
SSAS
Sharepoint
BI
Stream
Machine
Learning
Browser
SQOOP
HIVE ODBC
• Use Amazon
Redshift,
Azure
HDInsight or
similar
• Use Blob
storage to
persist big
data
• Spin up
Compute
clusters as
needed
• Keep Data in
Cloud
perpetually

Big Data Storage Costs
Cost per TB
per year
Sources:
http://calculator.s3.amazonaws.com/index.html
http://azure.microsoft.com/en-us/pricing/calculator/
As of Nov 2014
Provider Type
Cost per PB
per year
Amazon EBS SSD storage $ 1,229 $ 1,258,291
Amazon EBS Magnetic Storage $ 614 $ 629,145
Amazon S3 Storage $ 411 $ 420,372
Azure Tables & Queues $ 792 $ 811,302
Azure Blob Storage $ 288 $ 294,912

Hosting Considerations
• What if you host big data on-premise?
• Cost of managing hundreds of servers, expensive processing power
• Costs can be hidden in data center budget until too late
• What is your Big Data output?
• Beyond about 25 TB of data, cloud hosting costs become significant.
• Data Transfer costs must be considered as well
• Inbound is usually free
• Outbound can be $1,000’s per month
• Direct connect or physically ship
• For audit purposes, data may need to be kept for up to 7 years
• Factor this into your storage costs
• Location
• Will regulations impact ability to store or process on machines in different countries

Big Data Considerations
Databases
• High speed analysis of transactional data
• Multi-step computations
• Interactive querying
• Lots of updates (adds/deletes/mods)
MapReduce HDFS
• Low cost storage and compute
• High performance queries on large data
• Complex data simple query
• Simple scaling
Note: Ideas in this slide are borrowed and adapted from “Running, Managing, and Adapting Hadoop at Sears,” by Andy McNalis, Senior Manager,
Hadoop Infrastructure, Sears Holdings.

Cloud Considerations
• Big Data needs Big Compute
• Which cloud services will you choose?
• Time, effort and skills will vary considerably
• Microsoft Azure
• Amazon EC2
• Google Cloud Platform
• Verizon Cloud
• Rackspace
http://online.wsj.com/articles/little-space-remains-for-rackspace-ahead-of-
the-tape-1415557510

The “Traditional” Approach
Data Platform
Traditional RDMS
Commodity Storage Client
Familiar BI Tools
MPP
SQL
SSAS
Sharepoint
BI
Stream
Machine
Learning
Browser
SQOOP
HIVE ODBC

Premise-Cloud Hybrid Approach
Data Platform
Traditional RDMS
Commodity Storage Client
Familiar BI Tools
MPP
SQL
SSAS
Sharepoint
BI
Stream
Machine
Learning
Browser
SQOOP
HIVE ODBC
ETL and Pre-aggregate on-premise
Analyze Visualize in Cloud

Enterprise Data Hub
On-premise Hadoop
Clusters
Data Warehouse
Accelerator
Cloud Hosting
Cloud BI Reporting and
Analytics

ROI Strategies
Finding critical applications
Cost of Labor
 Use lower skill-lower cost resources
 Avoid extra headcount
 Share experiences among plants
 Move experienced talent to higher
value activity
Cost of Capital
 Use under-resourced equipment /
assets more efficiently
 Make equipment last longer, run more
efficiently
 Avoid more equipment purchases
Cost of Materials
 User fewer raw materials
 Improve quality of raw materials
sourced
 Improve delivery and inventory
Cost of Overheads
 Reduce transportation costs
 Reduce or optimize energy and
resource costs
 Reduce management layers
Cost of Lost Opportunities
 Reduce time to market
 Improve product end-of-life
 Reduce downtime
 Reduce order to cash
Cost of Reputation
 Reduce product defects
 Anticipate customer reactions
 Tailor service and response profiles
More available: info@halobi.com

Warehouse Operations
Machine sensor data for inventory and labor optimization
$300K
Cases per man hour
Picking accuracy

Drought Management for Growers
Smarter water use
$475K potential
Water per output

Retail promotions
Demand forecasting, sentiment analysis, and pricing
$6.2M
Sales per Square Foot
Returns Rate

Summary
• The value of investing in Big Data in the Cloud
depends on your use case
• Cost is an issue – 25 TB
• Skills are an issue – steep learning curves
• Process is an issue – requires change in the way
people think and operate
• Partners are an issue – do you want a large or niche
provider
• Database design is important

Conflict in the Cloud – Issues & Solutions for Big Data

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Conflict in the Cloud – Issues & Solutions for Big Data

Similaire à Conflict in the Cloud – Issues & Solutions for Big Data (20)

Dernier

Dernier (20)

Conflict in the Cloud – Issues & Solutions for Big Data

Notes de l'éditeur