This document discusses big data and provides motivation for why companies need to learn how to manage large amounts of data. It defines big data as data that exceeds the processing capacity of traditional databases due to its large volume, velocity, or variety. The document outlines the characteristics of big data using the "5 Vs" - volume, velocity, variety, viability and value. It then provides an overview of how big data works by presenting the big data supply chain. The key points are that big data enables new insights but also poses challenges for storage, analysis and use due to its scale and complexity.

1. Big
Data
Ins+tuto
Superior
Técnico
Technical
University
of
Lisbon
Bruno
Lopes
Catarina
Moreira
João
Pinho

2. Mo#va#on
2
220
2
PetaBytes
Of
data
that
people
create
every
day!

3. Mo#va#on
3
90
%
of
Data
UNSTRUCTURED
Which
is
difficult
to
manage
and
interpret!

4. Mo#va#on
4
2
hours
per
day
Spent
searching
for
the
right
informa#on

5. Mo#va#on
5
$5
million
per
year
Money
lost
due
to
data-­‐related
problems

6. Mo#va#on
6
1
in
3
Business
leaders
make
decisions
based
on
informa#on
they
don’t
trust
or
don’t
have
1
in
2
Business
leaders
say
they
don’t
have
access
to
the
informa#on
they
need
to
do
their
jobs
60%
CEO’s
need
to
do
a
beQer
job
capturing
and
understanding
informa+on
rapidly
in
order
to
make
swiS
business
decisions
Of
world’s
data
is
unstructured
80
%
Organiza+ons
Need
Deeper
Insights
…
Informa+on
is
at
the
Center
of
a
New
Wave
of
Opportunity
Source:
Big
Data
by
Ami
Redwan
Haq,
Founder
at
Sen:nel
Solu:ons
Ltd
44
%
as
much
Data
and
Content
Over
the
Coming
Decade
2009
800
000
PetaBytes
2020
35
ZeQaBytes

7. What
is
Big
Data?
7
Big
Data
(video)

8. What
is
Big
Data?
Data
that
exceeds
the
processing
capacity
of
a
conven+onal
database.
The
data
is
too
big,
moves
too
fast
or
doesn’t
fit
the
structures
of
your
database
architectures.
8
Source:
J.
Hurwitz,
A.
Nugent,
F.
Halper
&
M.
Kaufman,
Big
Data
for
Dummies,
Willey
&
Sons,
2013

9. Why
Big
Data?
Big
data
enables
organiza+ons
to
store,
manage
and
manipulate
vast
amounts
of
data
at
the
right
speed
and
at
the
right
#me
to
gain
the
right
insights.
9
Source:
J.
Hurwitz,
A.
Nugent,
F.
Halper
&
M.
Kaufman,
Big
Data
for
Dummies,
Willey
&
Sons,
2013

10. Why
Big
Data?
• The
value
of
Big
Data
falls
into
2
categories:
1. Analy#cal
Use.
• Reveal
new
insights
that
were
hidden
in
too
costly
to
process
massive
amounts
of
data;
• Example:
Peer
influence
among
customers;
2. Enabling
New
Products.
• Combining
large
number
of
signals
from
user’s
ac+ons
and
friends
• Example:
Facebook
10

11. The
V’s
Big
Data
is
defined
as
any
kind
of
data
source
that
has
at
least
three
shared
characteris+cs:
11
Velocity
Variety
Volume
Viability
Value

12. Velocity
Increasing
rate
at
which
data
flows
into
organiza#ons
12

13. Velocity
The
Large
Hadron
Collider
at
CERN
generates
13
Scien+sts
are
forced
to
25
PetaBytes
per
Year!
Discard
Massive
Data
In
order
to
keep
requirements
prac+cal
storage

14. Velocity
The
way
we
deliver
and
consume
products
and
services
is
genera+ng
a
massive
data
flow
to
the
provider
• It’s
possible
to
stream
fast-­‐moving
data
into
bulk
storage
for
later
batch
processing.
• The
importance
lies
in
taking
data
from
input
through
decision.
14

15. Volume
Ability
to
store
massive
amounts
of
(un)structured
data
15

16. Volume
16
No
longer
a
problem!

17. Volume
17
Don’t
forget
the
cloud!

18. Volume
-­‐
Challenges
• How
to
determine
relevance
within
large
data
volumes
• How
to
use
analy+cs
to
create
value
from
relevant
data.
18

19. Variety
Managing,
merging
and
governing
different
varie+es
of
data.
19

20. Variety
• Source
data
is
diverse
and
doesn’t
fall
into
neat
rela+onal
structures
• Data
comes
in
the
form
of
emails,
photos,
videos,
monitoring
devices,
PDFs,
audio,
etc.
• This
variety
of
(un)structured
data
creates
problems
for
storage,
mining
and
analyzing
data
20

21. Viability
Quickly
and
cost-­‐effec#vely
test
and
confirm
a
par+cular
variable’s
relevance
before
inves+ng
in
the
crea+on
of
a
fully
featured
model.
21

22. Value
Gain
new
insights
about
data
by
analyzing
variables
that
were
hidden
22

23. Big
Data
23
So,
how
does
it
work?

24. The
Big
Data
Supply
Chain
24
Source:
Edd
Dumbill,
Planning
for
Big
Data,
O’Reilly,
2012

25. Summary
• Mo+va+on
of
why
companies
need
to
learn
how
to
deal
with
Big
Data!
• Presented
a
defini+on
of
Big
Data
and
analyzed
the
V’s.
• Presented
an
analysis
of
the
Big
Data
Supply
chain
for
enterprises.
25

26. 26
Technologies
for
Big
Data

27. The
Reach
of
Big
Data
• So
far,
the
analy+cal
use
of
Big
Data
has
been
in
reach
only
to
leading
coopera+ons:
27
But
at
a
fantas+c
cost!

28. The
Reach
of
Big
Data
But
now,
there
are
several
open
source
applica+ons
than
can
tackle
Big
Data
related
challenges!
28

29. Apache
Hadoop
• Developed
and
released
by
Yahoo!
• Open
source
framework
for
storage
and
large
scale
parallel
processing
of
datasets
on
clusters.
• Ability
to
cheaply
process
large
amounts
of
data,
regardless
of
its
structure.
29

30. Apache
Hadoop
30

31. Apache
Hadoop
31

32. (Video
-­‐
MapReduce)
• IBM
–
MapReduce
• Health
Care
-­‐
Real
Time
Alerts
32

33. HDFS
–
Hadoop
Distributed
File
System
33

34. 34
• Detec+on
of
faults
and
quick,
automa+c
recovery
• Emphasis
on
high
throughput
of
data
access
• Tuned
to
support
large
datasets
• Write-­‐once-­‐read-­‐many
access
model:
once
a
file
is
created,
wriQen
and
closed,
needs
not
to
be
changed
HDFS
–
Hadoop
Distributed
File
System

35. 35
• Moving
computa+on
is
cheaper
than
moving
data
– minimizes
network
conges+on
– Increases
the
overall
throughput
of
the
system
• Portability
Across
Heterogeneous
Hardware
and
SoSware
Plaqorms
HDFS
–
Hadoop
Distributed
File
System

36. HDFS
–
Hadoop
Distributed
File
System
36

37. The
Core
of
Hadoop
-­‐
MapReduce
37
• Created
by
Google
for
web
search
indexes
• Ability
to
take
a
query
over
a
dataset,
divide
it
and
run
it
in
parallel
over
mul+ple
nodes
• Consists
in
three
stages:
– Map
– Shuffle
– Reduce

38. The
Core
of
Hadoop
-­‐
MapReduce
38
Input
Output
Reduce
Shuffle
Map

39. The
Core
of
Hadoop
-­‐
MapReduce
39
Input
Output
Reduce
Reduce
Shuffle
Map

40. The
Core
of
Hadoop
-­‐
MapReduce
40
• Imagine
that
you
want
to
apply
MapReduce
to
word
coun+ng
in
books

41. The
Core
of
Hadoop
-­‐
MapReduce
41
Input
Output
Reduce
Reduce
Shuffle
Map
E
…
com
a
mulher
e
o
filho…

42. The
Core
of
Hadoop
-­‐
MapReduce
42
Input
Output
Reduce
Reduce
Shuffle
Map
E
…
com
a
mulher
e
o
filho…
(
1,
E)
(2,
com)
(3,
a
)
(4,
mulher)
(5,
e
)
(6,
o)
(7,
filho)

43. The
Core
of
Hadoop
-­‐
MapReduce
43
Input
Output
Reduce
Reduce
Shuffle
Map
e
…
com
a
mulher
e
o
filho…
(
1,
e)
(2,
com)
(3,
a
)
(4,
mulher)
(5,
e
)
(6,
o)
(7,
filho)
(1,
e)
-­‐>
2
(2,
com)
-­‐>
1
(3,
a
)
-­‐>
1
(4,
mulher)
-­‐>
1
(5,
o)
-­‐>
1
(6,
filho)
-­‐>
1

44. The
Core
of
Hadoop
-­‐
MapReduce
44
Input
Output
Reduce
Reduce
Shuffle
Map
e
…
com
a
mulher
e
o
filho…
(
1,
e)
(3,
com)
(2,
a
)
(6,
mulher)
(5,
e
)
(7,
o)
(4,
filho)
(1,
e)
-­‐>
2
(3,
com)
-­‐>
1
(2,
a
)
-­‐>
1
(6,
mulher)
-­‐>
1
(7,
o)
-­‐>
1
(4,
filho)
-­‐>
1
e
[4,2,6]
a
[3,1,2]
com
[5,1,2]
filho
[2,1,1]
mulher
[3,1,10]
O
[7,1,9]

45. The
Core
of
Hadoop
-­‐
MapReduce
45
Input
Output
Reduce
Reduce
Shuffle
Map
e
…
com
a
mulher
e
o
filho…
(
1,
e)
(3,
com)
(2,
a
)
(6,
mulher)
(5,
e
)
(7,
o)
(4,
filho)
(1,
e)
-­‐>
2
(3,
com)
-­‐>
1
(2,
a
)
-­‐>
1
(6,
mulher)
-­‐>
1
(7,
o)
-­‐>
1
(4,
filho)
-­‐>
1
e
[4,2,6]
a
[3,1,2]
com
[5,1,2]
filho
[2,1,1]
mulher
[3,1,10]
O
[7,1,9]
e
-­‐>
12
a
-­‐>
6
com
-­‐>
8
filho
-­‐>
4
mulher
-­‐>
14
o
-­‐>
17

46. Who
Uses
Hadoop?
46

47. The
Social
Genome
product:
• Reach
customers,
or
friends
of
customers,
who
have
men+oned
product
and
include
a
discount
• Combines
public
data
from
the
web,
social
data
and
proprietary
data
47
Walmart
(Real
Time)

48. The
Social
Genome
product:
• Helps
Walmart
to
understand
the
context
of
what
their
customers
are
saying
online
• When
a
person
tweets
I
love
Salt,
Walmart
can
understand
that
she
is
talking
about
the
movie
Salt
and
not
the
condiment.
48
Walmart
(Real
Time)

49. Apache
Hive
49
• Data
warehouse
soSware
that
facilitates
querying
and
managing
large
datasets
residing
in
a
distributed
storage
• Uses
HiveSQL
to
analyze
data
• Allows
programmers
to
use
their
own
map
reduce
func+ons

50. Apache
Cassandra
50
• Database
with
high
scalability
and
high
availability
without
compromising
performance.
• Offers
– Denormaliza+on
– Materialized
views
– Powerful
built
in
cache

51. Google
Big
Query
51
Built
on
GoogleFS

52. Summary
52
• Apache
Hadoop
techhnology
– HDFS
file
system
– MapReduce
paradigm
• Apache
Hive
• Apache
Cassandra
• Google
BigQuery

54. Project
HEIDI
54
How
to
index
high
dimensional
data?

55. Project
HEIDI
55
• Mul+media
datasets
are
growing!
– Ex:
Audio,
video,
medical
images,
photos,
etc.
• We
need
techniques
to
efficiently
manage
and
access
informa+on
in
such
large
datasets!
• Mul+media
datasets
cannot
be
searched
like
tradi+onal
databases
(e.g.
Text
search).
They
are
searched
in
metric
spaces.

56. Project
HEIDI
Given
a
mul#media
object
as
input,
return
the
most
similar
objects
from
a
mul+media
database
56

57. Project
HEIDI
57
• In
content-­‐based
image
retrieval,
images
are
described
by
their
proper+es:
• Colors
• Texture
• Shape
• Shadows
• etc

58. Project
HEIDI
58
• The
process
of
conver+ng
a
mul+media
object
into
a
vector
with
its
contents/features
is
called
feature
extrac,on.

59. 59

60. Project
HEIDI
60
The
problem
in
high
dimensional
indexing
is
the

61. Project
HEIDI
61
The
problem
in
high
dimensional
indexing
is
the
OF
DIMENSIONALITY!!!

62. The
Curse
of
Dimensionality
62
• A
phenomena
that
arises
when
analyzing
and
organizing
data
in
high
dimensional
spaces.
• It
does
not
occur
in
lower
dimensions!

63. The
Curse
of
Dimensionality
63
• When
dimensionality
increases,
the
volume
of
the
space
increases
so
fast
that
the
data
become
sparse.
• Tradi+onal
tree
indexing
structures
do
not
provide
any
advantages
in
the
index
process
(outperformed
by
linear
scan)…

64. The
Curse
of
Dimensionality
64
•
Consequences:
– (hyper)
cubes
– (hyper)
spheres
• Volume
increases
exponen#ally
with
growing
dimension
(while
the
edge
length
remains
constant).

65. The
Curse
of
Dimensionality
65
How
can
this
curse
affect
datasets
in
metric
spaces?

66. The
Curse
of
Dimensionality
66
•
In
metric
spaces,
there
is
the
Nearest
Neighbor
paradigm.
• The
NN-­‐sphere
contains
one
point.
•
For
higher
dimensions,
the
radius
of
the
NN-­‐sphere
will
be
larger
than
the
size
of
the
database.

67. The
Curse
of
Dimensionality
67
Literature
addresses
this
issue
in
two
ways:
1. Approximate
Nearest
Search
2. Reduce
the
dimensionality
of
the
mul+media
objects

68. Project
HEIDI
68
The
algorithm
that
is
proposed
in
this
project
does
not
suffer
from
the
curse
of
dimensionality!

69. Project
HEIDI
69
The
Linear
SubSpace
Indexing
Algorithm
• Based
on
the
idea
of
subspaces!
– Feature
extrac+on
func+on
that
maps
the
high
dimensional
objects
into
low
dimensional
equivalent
objects.

70. Project
HEIDI
70
The
Quick
and
Dirty
Paradigm
• Discard
data
objects
in
lower
dimensional
spaces
• Massive
comparison
opera+ons
are
less
expensive
in
lower
dimensions

71. Project
HEIDI
71
However,
this
approach
has

72. Project
HEIDI
72
The
Quick
and
Dirty
Paradigm
• Can
produce
lots
of
false
hits!
• The
mapping
to
lower
dimensions
can
lead
to
closer
data
points

73. Project
HEIDI
73
Are
false
hits
really
a

74. Project
HEIDI
74

75. Project
HEIDI
75
• Just
perform
a
final
comparison
in
the
original
space!
• All
false
hits
will
be
discarded!
• The
computa+on
is
quick,
because
the
massive
amount
of
objects
were
discarded
in
the
lower
dimensions.

76. Project
HEIDI
76

77. Project
HEIDI
77
But…
How
to
map
high
dimensional
objects
into
lower
dimensional
spaces?

78. Project
HEIDI
78
The
Quick
and
Dirty
Paradigm
• Lower
bonding
lemma:
– Distances
between
objects
in
the
feature
space
are
always
smaller
or
equal
that
in
the
original
space

79. Project
HEIDI
79
The
Quick
and
Dirty
Paradigm
• The
Principal
Component
Analysis
algorithm
maps
objects
into
lower
dimensions
• Sa+sfies
the
lower
bounding
lemma

80. Project
HEIDI
80
The
Principal
Component
Analysis

81. Project
HEIDI
81
The
Principal
Component
Analysis
1. Compute
Co-­‐Variance
Matrix

82. Project
HEIDI
82
The
Principal
Component
Analysis
2. Compute
eigen
vectors
and
eigen
values
from
co-­‐variance
matrix
3. Get
the
most
significant
eigen
vectors
and
create
a
projec+on
matrix
4. Project
data

83. Project
HEIDI
83
The
Hierarchical
Linear
SubSpace
Indexing
Method
1. Index
Phase
(off-­‐line)
– Par++on
large
dataset
into
chunks
of
data
– Itera+vely
apply
the
Principal
Component
Analysis
to
project
the
dataset
into
lower
dimensions
– This
ac+on
can
be
parallelized
in
many
servers

84. Project
HEIDI
84
The
Hierarchical
Linear
SubSpace
Indexing
Method

85. Project
HEIDI
85
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space

86. Project
HEIDI
86
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA

87. Project
HEIDI
87
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA

88. Project
HEIDI
88
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA

89. Project
HEIDI
89
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA

90. Project
HEIDI
90
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA

91. Project
HEIDI
91
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA

92. Project
HEIDI
92
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA
PCA

93. Project
HEIDI
93
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA
PCA

94. Project
HEIDI
94
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA
PCA
PCA

95. Project
HEIDI
95
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA
PCA
PCA

96. Project
HEIDI
96
The
Hierarchical
Linear
SubSpace
Indexing
Method
Original
Space
Lower
Space
PCA
PCA
PCA
PCA
PCA
PCA

97. Project
HEIDI
97
The
Hierarchical
Linear
SubSpace
Indexing
Method

98. Project
HEIDI
98
The
Hierarchical
Linear
SubSpace
Indexing
Method

99. Project
HEIDI
99
The
Hierarchical
Linear
SubSpace
Indexing
Method

100. Project
HEIDI
100
The
Lower
Bounding
Lemma
is
sa#sfied
Euclidean
Distance
Most
Similar
Images
Retrieved

101. Project
HEIDI
101
The
Hierarchical
Linear
SubSpace
Indexing
Method

102. Project
HEIDI
102
The
Hierarchical
Linear
SubSpace
Indexing
Method
2. Query
Phase
(on-­‐line)
– Project
queries
using
projec+on
matrices
from
step
1
– Star+ng
in
the
lowest
dimensional
space,
itera+vely
discard
all
objects
that
are
different
from
the
query
– Keep
doing
this
un+l
the
original
space
is
reached
and
the
false
hits
are
discarded.

103. Project
HEIDI
103
The
Hierarchical
Linear
SubSpace
Indexing
Method

104. Summary
• Important
concepts
in
Big
Data
for
High
Dimensional
Datasets
• The
curse
of
dimensionality
• Projec+on
techniques
• The
Lower
Bounding
Lemma
• The
Quick
and
Dirty
paradigm
(some
rela+on
to
the
MapReduce
paradigm)
104

105. Summary
Mathema#cs
is
more
important
than
engineering!
105

106. Now…
Time
to
See
This
Working
for
REAL!!!
106