The document discusses challenges related to large volumes of data, or "Big Data". Traditional technologies try to divide and separate data across different systems, but this becomes difficult to manage at scale. The presenter introduces Hadoop as an alternative approach that can handle large volumes of data in a single system and democratize access to data. Hadoop provides a framework for storage, management and processing of large datasets in a distributed manner across commodity hardware.

1. Adam
Muise
–
Solu/on
Architect,
Hortonworks
ELEPHANT
AT
THE
DOOR:
HADOOP
AND
NEXT
GENERATION
DATA

2. Who
am
I?

3. Who
is
?

4. 100%
Open
Source
–
Democra/zed
Access
to
Data
The
leaders
of
Hadoop’s
development
We
do
Hadoop
Drive
Innova/on
in
the
plaForm
–
We
lead
the
roadmap
Community
driven,
Enterprise
Focused

5. We
do
Hadoop
successfully.
Support
Training
Professional
Services

6. We
do
Hadoop
successfully
everywhere.

7. We
do
Hadoop
successfully,
everywhere,
with
partners.

8. What
is
Hadoop?
What
is
everyone
talking
about?

9. Data

10. “Big
Data”
is
the
marke/ng
term
of
the
decade
in
IT

11. What
lurks
behind
the
hype
is
the
democra/za/on
of
Data.

12. You
need
data.

13. But
what
do
you
do
with
your
data
now?

14. We
are
obsessive
compulsive
about
collec/ng
and
structuring
our
data.

15. Put
it
away,
delete
it,
tweet
it,
compress
it,
shred
it,
wikileak-­‐it,
put
it
in
a
database,
put
it
in
SAN/NAS,
put
it
in
the
cloud,
hide
it
in
tape…

16. You
need
data.
Your
customers
expect
you
to
know
what
they
want
before
they
do.

17. Let’s
talk
challenges…

18. Volume
Volume
Volume
Volume

19. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

20. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

21. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

22. Storage,
Management,
Processing
all
become
challenges
with
Data
at
Volume

23. Tradi/onal
technologies
adopt
a
divide,
drop,
and
conquer
approach

24. Another
EDW
Analy/cal
DB
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
The
solu/on?
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
OLTP
Data
Data
Data
Data
Data
Data
Data
Data
Data
Yet
Another
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data

25. Another
EDW
Analy/cal
DB
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
OLTP
Ummm…you
dropped
something
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Yet
Another
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

26. Analyzing
the
data
usually
raises
more
interes/ng
ques/ons…

27. …which
leads
to
more
data

28. Wait,
you’ve
seen
this
before.
…
Data
Data
Data
Analy/cs
Sausage
Factory
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
…
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

29. Data
begets
Data.

30. What
keeps
us
from
our
Data?

31. “Prices,
Stupid
passwords,
and
Boring
Sta/s/cs.”
-­‐
Hans
Rosling
h)p://www.youtube.com/watch?v=hVimVzgtD6w

32. Your
data
silos
are
lonely
places.
EDW
Accounts
Customers
Web
Proper/es
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

33. …
Data
likes
to
be
together.
EDW
Accounts
Customers
Data
Data
Web
Proper/es
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

34. CDR
Data
Data
Data
Machine
Data
Facebook
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Weather
Data
Twi^er
Data
Data
likes
to
socialize
too.
Data
Data
EDW
Data
Data
Data
Data
Data
Data
Accounts
Data
Web
Proper/es
Data
Data
Data
Customers
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

35. New
types
of
data
don’t
quite
fit
into
your
pris/ne
view
of
the
world.
Logs
Data
Data
Data
Data
Data
Data
Data
Machine
Data
Data
Data
Data
Data
Data
Data
Data
My
Li^le
Data
Empire
Data
?
Data
?
Data
Data
Data
Data
Data
?
?
Data
Data

36. To
resolve
this,
some
people
take
hints
from
Lord
Of
The
Rings...

37. …and
create
One-­‐Schema-­‐To-­‐
Rule-­‐Them-­‐All…
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data

38. ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data
…but
that
has
its
problems
too.
ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data

39. ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data
Fragile
workflows
make
suppor/ng
the
analy/cal
models
you
want
expensive
and
/me-­‐consuming.
ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data

40. What
do
you
want
to
do
with
data?

41. Marke/ng
Analy/cs
needs
data.
Work
with
the
popula/on,
not
just
a
sample.

42. Town/City
Middle
Income
Band
Your
segmenta/on
today.
Female
Age:
25-­‐30
Male
Product
Category
Preferences

43. GPS
coordinates
Looking
to
start
a
business
Walking
into
Starbucks
right
now…
Spent
25
minutes
looking
at
tea
cozies
Unhappy
with
his
cell
phone
plan
$65-­‐68k
per
year
Your
segmenta/on
with
Pregnant
be^er
data.
Tea
Party
Hippie
A
depressed
Toronto
Maple
Leaf’s
Fan
Gene
Expression
for
Risk
Taker
Male
Female
Age:
27
but
feels
old
Product
recommenda/ons
Thinking
about
a
new
house
Products
lek
in
basket
indicate
drunk
amazon
shopper

44. Pick
up
all
of
that
data
that
was
prohibi/vely
expensive
to
store
and
use.

45. Why
do
viewer
surveys…

46. …when
raw
data
can
tell
you
what
bu^on
on
the
remote
was
pressed
during
what
commercial
for
the
en/re
viewer
popula/on?

47. To
approach
these
use
cases
you
need
an
affordable
plaForm
that
stores,
processes,
and
analyzes
the
data.

48. So
what
is
the
answer?

49. Enter
the
Hadoop.
………
h^p://www.fabulouslybroke.com/2011/05/ninja-­‐elephants-­‐and-­‐other-­‐awesome-­‐stories/

50. Hadoop
was
created
because
tradi/onal
technologies
never
cut
it
for
the
Internet
proper/es
like
Google,
Yahoo,
Facebook,
Twi^er,
and
LinkedIn

51. Tradi/onal
architecture
didn’t
scale
enough…
App
App
App
App
App
App
App
App
DB
DB
DB
SAN
App
App
App
App
DB
DB
DB
SAN
DB
DB
DB
SAN

52. Databases
can
become
bloated
and
useless

53. $upercompu/ng
Tradi/onal
architectures
cost
too
much
at
that
volume…
$/TB
$pecial
Hardware

54. So
what
is
the
answer?

55. If
you
could
design
a
system
that
would
handle
this,
what
would
it
look
like?

56. It
would
probably
need
a
highly
resilient,
self-­‐healing,
cost-­‐efficient,
distributed
file
system…
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage

57. It
would
probably
need
a
completely
parallel
processing
framework
that
took
tasks
to
the
data…
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

58. It
would
probably
run
on
commodity
hardware,
virtualized
machines,
and
common
OS
plaForms
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

59. It
would
probably
be
open
source
so
innova/on
could
happen
as
quickly
as
possible

60. It
would
need
a
cri/cal
mass
of
users

61. Hadoop
2
just
hit
the
ground:
Introducing
YARN

62. YARN
lets
you
run
more
data
apps
than
ever
before
MapReduce
V2
MapReduce
V?
STORM
Giraph
Tez
YARN
HDFS2
MPI
HBase
…
and
more

63. YARN
turns
Hadoop
into
a
smart
phone:
An
App
Ecosystem
hortonworks.com/yarn/

64. YARN:
Yeah,
we
did
that
too.
hortonworks.com/yarn/

65. Storm
HDFS
YARN
Pig
MapReduce
Apache
Hadoop
HCatalog
Hive
HBase
Ambari
Sqoop
Falcon
Flume

66. Storm
Pig
HDFS
YARN
MapReduce
Hortonworks
Data
PlaForm
HCatalog
Hive
HBase
Ambari
Sqoop
Falcon
Flume

67. What
else
are
we
working
on?
hortonworks.com/labs/

68. Hadoop
is
the
new
Data
Opera/ng
System
for
the
Enterprise

69. There is NO second place
Hortonworks
…the
Bull
Elephant
of
Hadoop
InnovaDon
© Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Page
69