Another version of the What is Hadoop deck with a couple of more slides on YARN.

1. Adam
Muise
–
Hortonworks
WELCOME
TO
HADOOP

2. Who
am
I?

3. Why
are
we
here?

4. Data

5. “Big
Data”
is
the
marke=ng
term
of
the
decade

6. What
lurks
behind
the
hype
is
the
democra=za=on
of
Data.

7. You
need
to
deal
with
Data.

8. You’re
probably
not
as
good
at
that
as
you
think.

9. Put
it
away,
delete
it,
tweet
it,
compress
it,
shred
it,
wikileak-­‐it,
put
it
in
a
database,
put
it
in
SAN/NAS,
put
it
in
the
cloud,
hide
it
in
tape…

10. You
are
obsessive
compulsive
about
collec=ng
and
structuring
your
data.

11. Seek
help.

12. Let’s
talk
challenges…

13. Volume
Volume
Volume
Volume

14. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

15. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

16. Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume
Volume

17. Storage,
Management,
Processing
all
become
challenges
with
Data
at
Volume

18. Tradi=onal
technologies
adopt
a
divide,
drop,
and
conquer
approach

19. Another
EDW
Analy=cal
DB
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
The
solu=on?
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
OLTP
Data
Data
Data
Data
Data
Data
Data
Data
Data
Yet
Another
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data

20. Another
EDW
Analy=cal
DB
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
OLTP
Ummm…you
dropped
something
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Yet
Another
EDW
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

21. Analyzing
the
data
usually
raises
more
interes=ng
ques=ons…

22. …which
leads
to
more
data

23. Wait,
you’ve
seen
this
before.
Data
Data
Data
…
Sausage
Factory
Data
Data
Data
Data
Data
Data
Data
Data
Data
…
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

24. Data
begets
Data.

25. What
keeps
us
from
Data?

26. “Prices,
Stupid
passwords,
and
Boring
Sta=s=cs.”
-­‐
Hans
Rosling
h"p://www.youtube.com/watch?v=hVimVzgtD6w

27. Your
data
silos
are
lonely
places.
EDW
Accounts
Customers
Web
Proper=es
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

28. …
Data
likes
to
be
together.
EDW
Accounts
Customers
Data
Data
Web
Proper=es
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

29. CDR
Data
Data
Data
Machine
Data
Facebook
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Weather
Data
TwiYer
Data
Data
likes
to
socialize
too.
Data
Data
EDW
Data
Data
Data
Data
Data
Data
Accounts
Data
Web
Proper=es
Data
Data
Data
Customers
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data

30. New
types
of
data
don’t
quite
fit
into
your
pris=ne
view
of
the
world.
Logs
Data
Data
Data
Data
Data
Data
Data
Machine
Data
Data
Data
Data
Data
Data
Data
Data
My
LiYle
Data
Empire
Data
?
Data
?
Data
Data
Data
Data
Data
?
?
Data
Data

31. To
resolve
this,
some
people
take
hints
from
Lord
Of
The
Rings...

32. …and
create
One-­‐Schema-­‐To-­‐
Rule-­‐Them-­‐All…
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data

33. ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data
…but
that
has
its
problems
too.
ETL
Data
Data
Data
ETL
ETL
ETL
EDW
Data
Data
Data
Data
Data
Schema
Data
Data
Data
Data

34. So
what
is
the
answer?

35. Enter
the
Hadoop.
………
hYp://www.fabulouslybroke.com/2011/05/ninja-­‐elephants-­‐and-­‐other-­‐awesome-­‐stories/

36. Hadoop
was
created
because
Big
IT
never
cut
it
for
the
Internet
Proper=es
like
Google,
Yahoo,
Facebook,
TwiYer,
and
LinkedIn

37. Tradi=onal
architecture
didn’t
scale
enough…
App
App
App
App
App
App
App
App
DB
DB
DB
SAN
App
App
App
App
DB
DB
DB
SAN
DB
DB
DB
SAN

38. Databases
become
bloated
and
useless

39. $upercompu=ng
Tradi=onal
architectures
cost
too
much
at
that
volume…
$/TB
$pecial
Hardware

40. So
what
is
the
answer?

41. If
you
could
design
a
system
that
would
handle
this,
what
would
it
look
like?

42. It
would
probably
need
a
highly
resilient,
self-­‐healing,
cost-­‐efficient,
distributed
file
system…
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage

43. It
would
probably
need
a
completely
parallel
processing
framework
that
took
tasks
to
the
data…
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

44. It
would
probably
run
on
commodity
hardware,
virtualized
machines,
and
common
OS
pladorms
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

45. It
would
probably
be
open
source
so
innova=on
could
happen
as
quickly
as
possible

46. It
would
need
a
cri=cal
mass
of
users

47. {Processing
+
Storage}
=
{MapReduce/YARN+
HDFS}

48. HDFS
stores
data
in
blocks
and
replicates
those
blocks
block1
Processing
Processing
Processing
Storage
Storage
Storage
block2
block2
Processing
Processing
Processing
block1
Storage
Storage
Storage
block3
block2
Processing
Storage
block3
Processing
Processing
block1
Storage
Storage
block3

49. If
a
block
fails
then
HDFS
always
has
the
other
copies
and
heals
itself
block1
Processing
Processing
Processing
block3
Storage
Storage
Storage
block2
block2
Processing
Processing
Processing
block1
Storage
Storage
Storage
block3
block2
Processing
Storage
block3
Processing
Processing
block1
Storage
Storage
X

50. MapReduce
is
a
programming
paradigm
that
completely
parallel
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Mapper
Mapper
Mapper
Mapper
Mapper
Reducer
Data
Data
Data
Reducer
Data
Data
Data
Reducer
Data
Data
Data

51. MapReduce
has
three
phases:
Map,
Sort/Shuffle,
Reduce
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Mapper
Mapper
Key,
Value
Key,
Value
Key,
Value
Reducer
Key,
Value
Key,
Value
Key,
Value
Mapper
Reducer
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Mapper
Reducer
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Key,
Value
Mapper
Key,
Value
Key,
Value
Key,
Value

52. MapReduce
applies
to
a
lot
of
data
processing
problems
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Mapper
Mapper
Mapper
Mapper
Mapper
Reducer
Data
Data
Data
Reducer
Data
Data
Data
Reducer
Data
Data
Data

53. Introducing
YARN

54. YARN
=
Yet
Another
Resource
Nego=ator

55. YARN
abstracts
resource
management
so
you
can
run
more
than
just
MapReduce
MapReduce
V2
MapReduce
V?
STORM
Giraph
Tez
YARN
HDFS2
MPI
HBase
…
and
more

56. Node
Manager
Resource
Manager
Container
Scheduler
Pig
AppMaster
Container
Resource
Manager
+
Node
Managers
=
YARN
Node
Manager
Container
Container
Storm
Node
Manager
Node
Manager
MapReduce
AppMaster
Container
Container
Container
Container
Container
AppMaster

57. YARN
turns
Hadoop
into
a
smart
phone:
An
App
Ecosystem
hortonworks.com/yarn/

58. Check
out
the
book
too…
Preview
at:
hortonworks.com/yarn/

59. YARN
is
an
essen=al
part
of
a
balanced
breakfast
in
Hadoop
2.2.0

60. Hadoop
has
other
open
source
projects…

61. Tez
=
{
Generic
Tasks
+
Pipelining
}
Super
Fast
MapReduce

62. Hive
=
{SQL
-­‐>
Tez
||
MapReduce}
SQL-­‐IN-­‐HADOOP

63. Pig
=
{PigLa=n
-­‐>
Tez
||
MapReduce}

64. HCatalog
=
{metadata*
for
MapReduce,
Hive,
Pig,
HBase}
*metadata
=
tables,
columns,
par==ons,
types

65. Oozie
=
Job::{Task,
Task,
if
Task,
then
Task,
final
Task}

66. Falcon
Feed
Feed
Feed
Feed
Hadoop
DR
Feed
Replica=on
Feed
Feed
Hadoop
Feed

67. Flume
Files
Flume
JMS
Weblogs
Events
Flume
Flume
Flume
Flume
Flume
Hadoop

68. Sqoop
DB
DB
Sqoop
Hadoop
Sqoop

69. Ambari
=
{install,
manage,
monitor}

70. HBase
=
{real-­‐=me,
distributed-­‐
map,
big-­‐tables}

71. Storm
=
{Complex
Event
Processing,
Near-­‐Real-­‐Time,
Provisioned
by
YARN
}

72. Storm
HDFS
YARN
Pig
MapReduce
Apache
Hadoop
HCatalog
Hive
HBase
Ambari
Sqoop
Falcon
Flume

73. Storm
Pig
HDFS
YARN
MapReduce
Hortonworks
Data
Pladorm
HCatalog
Hive
HBase
Ambari
Sqoop
Falcon
Flume

74. What
else
are
we
working
on?
hortonworks.com/labs/

75. Hadoop
is
the
new
Data
Opera=ng
System
for
the
Enterprise

76. There is NO second place
Hortonworks
…the
Bull
Elephant
of
Hadoop
Innova@on
© Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Page
76