Ameet Talwalker, Postdoctoral Fellow at UC Berkeley: Divide-and-Conquer Matrix Factorization

1. Divide-­‐and-­‐Conquer
Matrix
Factoriza5on
Ameet
Talwalkar
UC
Berkeley
November
15th,
2013
Collaborators:
Lester
Mackey2,
Michael
I.
Jordan1,
Yadong
Mu3,
Shih-­‐Fu
Chang3
1UC
Berkeley
2Stanford
University
3Columbia
University

2. Three
Converging
Trends

3. Three
Converging
Trends
Big
Data

4. Three
Converging
Trends
Big
Data
Distributed
CompuOng

5. Three
Converging
Trends
Machine
Learning
Big
Data
Distributed
CompuOng

6. Goal:
Extend
ML
to
the
Big
Data
SeAng
Challenge:
ML
not
developed
with
scalability
in
mind
✦
Does
not
naturally
scale
/
leverage
distributed
compuOng
Machine
Learning
Big
Data
Distributed
CompuOng

7. Goal:
Extend
ML
to
the
Big
Data
SeAng
Challenge:
ML
not
developed
with
scalability
in
mind
✦
Does
not
naturally
scale
/
leverage
distributed
compuOng
Our
approach:
Divide-­‐and-­‐conquer
✦
Apply
exisOng
base
algorithms
to
subsets
of
data
and
combine
Machine
Learning
Big
Data
Distributed
CompuOng

8. Goal:
Extend
ML
to
the
Big
Data
SeAng
Challenge:
ML
not
developed
with
scalability
in
mind
✦
Does
not
naturally
scale
/
leverage
distributed
compuOng
Our
approach:
Divide-­‐and-­‐conquer
✦
Apply
exisOng
base
algorithms
to
subsets
of
data
and
combine
✓
✓
✓
Build
upon
exisOng
suites
of
ML
algorithms
Preserve
favorable
algorithm
properOes
Naturally
leverage
distributed
compuOng
Machine
Learning
Big
Data
Distributed
CompuOng

9. Goal:
Extend
ML
to
the
Big
Data
SeAng
Challenge:
ML
not
developed
with
scalability
in
mind
✦
Does
not
naturally
scale
/
leverage
distributed
compuOng
Our
approach:
Divide-­‐and-­‐conquer
✦
Apply
exisOng
base
algorithms
to
subsets
of
data
and
combine
✓
✓
✓
✦
Build
upon
exisOng
suites
of
ML
algorithms
Preserve
favorable
algorithm
properOes
Naturally
leverage
distributed
compuOng
E.g.,
✦
✦
✦
Machine
Learning
Big
Data
Matrix
factorizaOon
(DFC) [MTJ, NIPS11; TMMFJ, ICCV13]
[KTSJ, ICML12; KTSJ,
Assessing
esOmator
quality
(BLB) JRSS13; KTASJ, KDD13]
Genomic
Variant
Calling [BTTJPYS13, submitted, CTZFJP13, submitted]
Distributed
CompuOng

10. Goal:
Extend
ML
to
the
Big
Data
SeAng
Challenge:
ML
not
developed
with
scalability
in
mind
✦
Does
not
naturally
scale
/
leverage
distributed
compuOng
Our
approach:
Divide-­‐and-­‐conquer
✦
Apply
exisOng
base
algorithms
to
subsets
of
data
and
combine
✓
✓
✓
✦
Build
upon
exisOng
suites
of
ML
algorithms
Preserve
favorable
algorithm
properOes
Naturally
leverage
distributed
compuOng
E.g.,
✦
✦
✦
Machine
Learning
Big
Data
Matrix
factorizaOon
(DFC) [MTJ, NIPS11; TMMFJ, ICCV13]
[KTSJ, ICML12; KTSJ,
Assessing
esOmator
quality
(BLB) JRSS13; KTASJ, KDD13]
Genomic
Variant
Calling [BTTJPYS13, submitted, CTZFJP13, submitted]
Distributed
CompuOng

11. Matrix
CompleOon

12. Matrix
CompleOon

13. Matrix
CompleOon
Goal: Recover a matrix from a
subset of its entries

14. Matrix
CompleOon
Goal: Recover a matrix from a
subset of its entries

15. Matrix
CompleOon
Goal: Recover a matrix from a
subset of its entries

16. Matrix
CompleOon
Goal: Recover a matrix from a
subset of its entries
Can we do this at scale?
✦
✦
✦
✦
✦
Netflix: 30M users, 100K+ videos
Facebook: 1B users
Pandora: 70M active users, 1M songs
Amazon: Millions of users and products
...

17. Reducing
Degrees
of
Freedom

18. Reducing
Degrees
of
Freedom
✦
Problem: Impossible without
additional information
✦
mn degrees of freedom
n
m

19. Reducing
Degrees
of
Freedom
✦
Problem: Impossible without
additional information
✦
✦
mn degrees of freedom
Solution: Assume small # of
factors determine preference
n
m
r
=m
n
r
‘Low-rank’

20. Reducing
Degrees
of
Freedom
✦
Problem: Impossible without
additional information
✦
✦
mn degrees of freedom
Solution: Assume small # of
factors determine preference
✦
O(m + n) degrees of freedom
✦
Linear storage costs
n
m
r
=m
n
r
‘Low-rank’

21. Bad
Sampling
✦
Problem:
We
have
no
raOng
informaOon
about

22. Bad
Sampling
✦
Problem:
We
have
no
raOng
informaOon
about
✦
SoluOon:
Assume
˜
+ m))
⌦(r(n
observed
entries
drawn
uniformly
at
random

23. Bad
InformaOon
Spread

24. Bad
InformaOon
Spread
✦
Problem:
Other
raOngs
don’t
inform
us
about
missing
raOng
bad
spread
of
informaOon

25. Bad
InformaOon
Spread
✦
Problem:
Other
raOngs
don’t
inform
us
about
missing
raOng
✦
SoluOon:
Assume
incoherence
with
standard
basis [Candes and Recht, 2009]
bad
spread
of
informaOon

26. Matrix
CompleOon
=
In
+
‘noise’
Low-rank
Goal:
Recover
a
matrix
from
a
subset
of
its
entries,
assuming
✦
low-­‐rank,
incoherent
✦
uniform
sampling

27. Matrix
CompleOon
=
In
+
Low-rank
✦
Nuclear-­‐norm
heurisOc
+
strong
theoreOcal
guarantees
+
good
empirical
results
‘noise’

28. Matrix
CompleOon
=
In
+
Low-rank
✦
Nuclear-­‐norm
heurisOc
+
strong
theoreOcal
guarantees
+
good
empirical
results
⎯
very
slow
computa5on
‘noise’

29. Matrix
CompleOon
=
In
+
‘noise’
Low-rank
✦
Nuclear-­‐norm
heurisOc
+
strong
theoreOcal
guarantees
+
good
empirical
results
⎯
very
slow
computa5on
Goal:
Scale
MC
algorithms
and
preserve
guarantees

30. Divide-­‐Factor-­‐Combine
(DFC)
[MTJ, NIPS11]

31. Divide-­‐Factor-­‐Combine
(DFC)
[MTJ, NIPS11]
✦
D
step:
Divide
input
matrix
into
submatrices

32. Divide-­‐Factor-­‐Combine
(DFC)
[MTJ, NIPS11]
✦
D
step:
Divide
input
matrix
into
submatrices
✦
F
step:
Factor
in
parallel
using
a
base
MC
algorithm

33. Divide-­‐Factor-­‐Combine
(DFC)
[MTJ, NIPS11]
✦
D
step:
Divide
input
matrix
into
submatrices
✦
F
step:
Factor
in
parallel
using
a
base
MC
algorithm
✦
C
step:
Combine
submatrix
esOmates

34. Divide-­‐Factor-­‐Combine
(DFC)
[MTJ, NIPS11]
✦
D
step:
Divide
input
matrix
into
submatrices
✦
F
step:
Factor
in
parallel
using
a
base
MC
algorithm
✦
C
step:
Combine
submatrix
esOmates
Advantages:
✦
Submatrix
factorizaOon
is
much
cheaper
and
easily
parallelized
✦
Minimal
communicaOon
between
parallel
jobs
✦
Retains
comparable
recovery
guarantees
(with
proper
choice
of
division
/
combinaOon
strategies)

35. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:

36. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC

37. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
✦
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space

38. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons

39. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
✦
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons

40. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
✦
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons
=

41. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
✦
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons
=

42. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
✦
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons
=

43. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
✦
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons
=
=

44. DFC-­‐Proj
✦
D
step:
Randomly
parOOon
observed
entries
into
t
submatrices:
✦
F
step:
Complete
the
submatrices
in
parallel
✦
Reduced
cost:
Expect
t-­‐fold
speedup
per
iteraOon
✦
Parallel
computaOon:
Pay
cost
of
one
cheaper
MC
C
step:
Project
onto
single
low-­‐dimensional
column
space
✦
✦
✦
✦
Roughly,
share
informaOon
across
sub-­‐soluOons
Minimal
cost:
linear
in
n,
quadraOc
in
rank
of
sub-­‐soluOons
Ensemble: Project onto column space of each sub-solution and
average

45. Does
It
Work?
Yes,
with
high
probability.
Theorem:
Assume:
✦ L
0
is
low-­‐rank
and
incoherent,
✦
˜
entries
sampled
uniformly
at
random,
⌦(r(n + m))
✦
Nuclear
norm
heurisOc
is
base
algorithm.

46. Does
It
Work?
Yes,
with
high
probability.
Theorem:
Assume:
✦ L
0
is
low-­‐rank
and
incoherent,
✦
˜
entries
sampled
uniformly
at
random,
⌦(r(n + m))
✦
Nuclear
norm
heurisOc
is
base
algorithm.
ˆ
Then
L
=
L0
with
(slightly
less)
high
probability.

47. Does
It
Work?
Yes,
with
high
probability.
Theorem:
Assume:
✦ L
0
is
low-­‐rank
and
incoherent,
✦
˜
entries
sampled
uniformly
at
random,
⌦(r(n + m))
✦
Nuclear
norm
heurisOc
is
base
algorithm.
ˆ
Then
L
=
L0
with
(slightly
less)
high
probability.
✦
Noisy
seang:
(2
✏)
approximaOon
of
original
bound
+
✦
Can
divide
into
an
increasing
number
of
subproblems
˜
(
t
!
1
)
when
number
of
observed
entries
in ! (r2 (n + m))

48. DFC
Noisy
Recovery
MC
0.25
Proj−10%
Proj−Ens−10%
Base−MC
RMSE
0.2
0.15
0.1
0.05
0
0
2
4
6
8
10
% revealed entries
✦
Noisy recovery relative to base algorithm ( n = 10K, r = 10 )

49. DFC Speedup
MC
3500
Proj−10%
Proj−Ens−10%
Base−MC
3000
time (s)
2500
2000
1500
1000
500
0
1
2
3
m
✦
4
5
4
x 10
Speedup over APG for random matrices with 4% of entries
revealed and r = 0.001n

50. Matrix
CompleOon
NeIlix
Prize:
✦ 100
million
raOngs
in
{1,
...
,
5}
✦ 18K
movies,
480K
user
✦ Issues:
Full-­‐rank;
Noisy,
non-­‐uniform
observaOons

51. Matrix
CompleOon
NeIlix
Prize:
✦ 100
million
raOngs
in
{1,
...
,
5}
✦ 18K
movies,
480K
user
✦ Issues:
Full-­‐rank;
Noisy,
non-­‐uniform
observaOons
NeIlix
Method
Error
Time
Nuclear
Norm
DFC,
t=4
DFC,
t=10
DFC-­‐Ens,
t=4
DFC-­‐Ens,
t=10
0.8433
2653.1s

52. Matrix
CompleOon
NeIlix
Prize:
✦ 100
million
raOngs
in
{1,
...
,
5}
✦ 18K
movies,
480K
user
✦ Issues:
Full-­‐rank;
Noisy,
non-­‐uniform
observaOons
NeIlix
Method
Error
Time
Nuclear
Norm
DFC,
t=4
DFC,
t=10
DFC-­‐Ens,
t=4
DFC-­‐Ens,
t=10
0.8433
0.8436
0.8484
0.8411
0.8433
2653.1s
689.5s
289.7s
689.5s
289.7

53. Matrix
CompleOon
NeIlix
Prize:
✦ 100
million
raOngs
in
{1,
...
,
5}
✦ 18K
movies,
480K
user
✦ Issues:
Full-­‐rank;
Noisy,
non-­‐uniform
observaOons
NeIlix
Method
Error
Time
Nuclear
Norm
DFC,
t=4
DFC,
t=10
DFC-­‐Ens,
t=4
DFC-­‐Ens,
t=10
0.8433
0.8436
0.8484
0.8411
0.8433
2653.1s
689.5s
289.7s
689.5s
289.7

54. Robust
Matrix
FactorizaOon
[Chandrasekaran, Sanghavi, Parrilo, and Willsky, 2009; Candes, Li, Ma, and Wright, 2011; Zhou, Li, Wright, Candes, and Ma, 2010]
Matrix
Comple5on
=
In
+
Low-rank
‘noise’

55. Robust
Matrix
FactorizaOon
[Chandrasekaran, Sanghavi, Parrilo, and Willsky, 2009; Candes, Li, Ma, and Wright, 2011; Zhou, Li, Wright, Candes, and Ma, 2010]
Matrix
Comple5on
=
In
Principal
Component
Analysis
+
+
‘noise’
Low-rank
=
In
‘noise’
Low-rank

56. Robust
Matrix
FactorizaOon
[Chandrasekaran, Sanghavi, Parrilo, and Willsky, 2009; Candes, Li, Ma, and Wright, 2011; Zhou, Li, Wright, Candes, and Ma, 2010]
Matrix
Comple5on
=
In
Principal
Component
Analysis
+
+
In
Low-rank
=
In
‘noise’
Low-rank
=
Robust
Matrix
Factoriza5on
‘noise’
+
Low-rank
+
Sparse
Outliers
‘noise’

57. Video
Surveillance
✦
Goal:
separate
foreground
from
background
✦
✦
✦
Store
video
as
matrix
Low-rank
=
background
Outliers
=
movement

58. Video
Surveillance
✦
Goal:
separate
foreground
from
background
✦
✦
✦
Store
video
as
matrix
Low-rank
=
background
Outliers
=
movement
Original
Frame

59. Video
Surveillance
✦
Goal:
separate
foreground
from
background
✦
✦
✦
Store
video
as
matrix
Low-rank
=
background
Outliers
=
movement
Original
Frame
Nuclear
Norm
(342.5s)

60. Video
Surveillance
✦
Goal:
separate
foreground
from
background
✦
✦
✦
Store
video
as
matrix
Low-rank
=
background
Outliers
=
movement
Original
Frame
Nuclear
Norm
(342.5s)
DFC-­‐5%
(24.2s)
DFC-­‐0.5%
(5.2s)

61. Subspace
SegmentaOon
[Liu, Lin, and Yu, 2010]
Matrix
Comple5on
=
In
+
Low-rank
‘noise’

62. Subspace
SegmentaOon
[Liu, Lin, and Yu, 2010]
Matrix
Comple5on
=
In
Principal
Component
Analysis
+
+
‘noise’
Low-rank
=
In
‘noise’
Low-rank

63. Subspace
SegmentaOon
[Liu, Lin, and Yu, 2010]
Matrix
Comple5on
=
In
Principal
Component
Analysis
+
+
‘noise’
Low-rank
=
In
Subspace
Segmenta5on
‘noise’
Low-rank
=
In
+
Low-rank
‘noise’

64. MoOvaOon:
Face
images

65. MoOvaOon:
Face
images
...
Principal
Component
Analysis
...
In

66. MoOvaOon:
Face
images
...
Principal
Component
Analysis
...
In
=
+
‘noise’
Low-rank
✦
Model
images
of
one
person
via
one
low-­‐dimensional
subspace

67. MoOvaOon:
Face
images

68. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

69. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

70. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

71. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

72. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

73. MoOvaOon:
Face
images
Subspace
Segmenta5on
In

74. MoOvaOon:
Face
images
Subspace
Segmenta5on
=
In
+
‘noise’
Low-rank
✦
Model
images
of
five
people
via
five
low-­‐dimensional
subspaces

75. MoOvaOon:
Face
images
Subspace
Segmenta5on
=
In
+
‘noise’
Low-rank
✦
Model
images
of
five
people
via
five
low-­‐dimensional
subspaces
✦
Recover
subspaces
cluster
images

76. MoOvaOon:
Face
images
Subspace
Segmenta5on
=
In
✦
+
‘noise’
Low-rank
Nuclear
norm
heurisOc
to
provably
recovers
subspaces
✦ Guarantees
are
preserved
with
DFC [TMMFJ, ICCV13]

77. MoOvaOon:
Face
images
Subspace
Segmenta5on
=
In
+
‘noise’
Low-rank
✦
Toy
Experiment:
IdenOfy
images
corresponding
to
same
person
(10
people,
640
images)
✦
DFC
Results:
Linear
speedup,
State-­‐of-­‐the-­‐art
accuracy

78. Video
Event
DetecOon

79. Video
Event
DetecOon
✦
✦
Input:
videos,
some
of
which
are
associated
with
events
Goal:
predict
events
for
unlabeled
videos

80. Video
Event
DetecOon
✦
✦
✦
Input:
videos,
some
of
which
are
associated
with
events
Goal:
predict
events
for
unlabeled
videos
Idea:
✦
Featurize
each
video

81. Video
Event
DetecOon
✦
✦
✦
Input:
videos,
some
of
which
are
associated
with
events
Goal:
predict
events
for
unlabeled
videos
Idea:
✦
✦
Featurize
each
video
Learn
video
clusters
via
nuclear
norm
heurisOc

82. Video
Event
DetecOon
✦
✦
✦
Input:
videos,
some
of
which
are
associated
with
events
Goal:
predict
events
for
unlabeled
videos
Idea:
✦
✦
✦
Featurize
each
video
Learn
video
clusters
via
nuclear
norm
heurisOc
Given
labeled
nodes
and
cluster
structure,
make
predicOons

83. Video
Event
DetecOon
✦
✦
✦
Input:
videos,
some
of
which
are
associated
with
events
Goal:
predict
events
for
unlabeled
videos
Idea:
✦
✦
✦
Featurize
each
video
Learn
video
clusters
via
nuclear
norm
heurisOc
Given
labeled
nodes
and
cluster
structure,
make
predicOons
Can
do
this
at
scale
with
DFC!

84. DFC
Summary
✦
DFC:
distributed
framework
for
matrix
factorizaOon
✦ Similar
recovery
guarantees
✦ Significant
speedups
✦
DFC
applied
to
3
classes
of
problems:
✦ Matrix
compleOon
✦ Robust
matrix
factorizaOon
✦ Subspace
recovery
✦
Extend
DFC
to
other
MF
methods,
e.g.,
ALS,
SGD?

85. Big
Data
and
Distributed
CompuOng
are
valuable
resources,
but
...

86. Big
Data
and
Distributed
CompuOng
are
valuable
resources,
but
...
✦
Challenge
1:
ML
not
developed
with
scalability
in
mind

87. Big
Data
and
Distributed
CompuOng
are
valuable
resources,
but
...
✦
Challenge
1:
ML
not
developed
with
scalability
in
mind
Divide-­‐and-­‐Conquer
(e.g.,
DFC)

88. Big
Data
and
Distributed
CompuOng
are
valuable
resources,
but
...
✦
Challenge
1:
ML
not
developed
with
scalability
in
mind
Divide-­‐and-­‐Conquer
(e.g.,
DFC)
✦
Challenge
2:
ML
not
developed
with
ease-­‐of-­‐use
in
mind

89. Big
Data
and
Distributed
CompuOng
are
valuable
resources,
but
...
✦
Challenge
1:
ML
not
developed
with
scalability
in
mind
ML base
ML base
Divide-­‐and-­‐Conquer
(e.g.,
DFC)
ML base
ML base
✦
Challenge
2:
ML
not
developed
with
ease-­‐of-­‐use
in
mind
ML base
ML base
www.mlbase.org
ML base