Welcome to the Age of Data

Welcome to the age of data!
BIGDATA.BE

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

who am i

» Steven Noels
» Founder & VP Product
» Makers of Lily: Interactive Big Data
platform
» Open Source / Apache Software
Foundation
» co-founder bigdata.be

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 2

Houston,
we have
a problem.

We’re
drowning.


Drowning
in a
Sea
of
Data.

Mountains of
Metadata.

The firehose
of UGC.

Still, we
can’t make
much sense
of it.


... and we
throw a lot of
it away.

We regard
DATA as cost.


But data is an
opportunity.


Think about it.


advertisements

recommendations

fraud detection

eyeballs

churn

The future is
for
datanerds.

This is what Big
Data is about:
new insights,
new business.

3 issues for
BIG DATA

volume
need:
more
capacity
data

moore
1
time


solution:
distributed
systems 1

1

distributed
systems are
1
hard.

2
database


2
database data warehouse


2
database data warehouse analytics


data shuffling, data duplication
2
database data warehouse analytics


“Top-performing
organizations are twice
as likely to apply
analytics to activities.”
3
(MIT Sloan Management
Review, Winter 2011)


enter

HBase


what is hadoop ?

1 server RAM
CPU
Disk


RAM HBASE

CPU MAP/REDUCE

DISK HDFS

many servers

map/reduce


map/reduce

» Batch-oriented
» Data locality (code is shipped around)
» Heavy parallellization
» Process management
» Append-only ﬁles


Hadoop ecosystem
» Hadoop Common » Hive: A data warehouse infrastructure

» Subprojects that provides data summarization and
ad hoc querying.
» Flume/SQOOP: Data collection systems
» MapReduce: A software framework for
for large distributed systems.
distributed processing of large data
» HBase: A scalable, distributed database sets on compute clusters.
that supports structured data storage
» Pig: A high-level data-ﬂow language
for large/wide tables.
and execution framework for parallel
» HDFS: A distributed ﬁle system that computation.
provides high throughput access to
» ZooKeeper: A high-performance
application data.
coordination service for distributed
applications.
» Mahout: machine learning libraries


High-level data model / easy API indexes

UI Framework SDK
(HUE) (HUE SDK)

Search
Dev2Dev
Workflow Scheduling Metadata tutoring,
(OOZIE) (oozie) (HIVE) integrated
deployment
and
Languages / enterprise
Data Compilers Fast usage metrics, support
Integration (PIG, HIVE) Read/Write analytics &
(FLUME, Access recommen-
SQOOP) (HBASE) dations
(PIG, HIVE)

Coordination
(ZOOKEEPER)

CDH

real-time big data architecture

1. compensate for high latency of updates to serving layer
speed layer 2. fast, incremental algorithms
3. batch layer eventually overrides speed layer
storm

1. random access to batch views
serving layer 2. updated by batch layer

1. store master dataset (append-only)
batch layer 2. compute arbitrary views


Hadoop, interactive.

Analytics Interactics (RDBMS)
batch interactive
static ﬁles data management

1018 1015 109-12


news & media

smart data management
insights indexing
search
commerce

ﬁnance
interactive
audience proﬁle
metrics harvesting

telecom

My baby: Lily.


The start of Lily.


Thank you !
for your attention
for your questions

» steven.noels@outerthought.com

» @stevenn


Welcome to the Age of Data

Recommandé

Recommandé

Contenu connexe

Similaire à Welcome to the Age of Data

Similaire à Welcome to the Age of Data (20)

Plus de NGDATA

Plus de NGDATA (6)

Dernier

Dernier (20)

Welcome to the Age of Data