C. Guyon, T. Bouwmans. E. Zahzah, “Foreground Detection via Robust Low Rank Matrix Factorization including Spatial Constraint with Iterative Reweighted Regression”, International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 2012.
1. Foreground Detection via Robust Low Rank Matrix
Factorization including Spatial Constraint with Iterative
Reweighted Regression
C. Guyon, T. Bouwmans and E. Zahzah
charles.guyon@univ-lr.fr
MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France
Presenter: Muriel Visani (L3i lab - University of la Rochelle)
—
ICPR2012, Tsukuba, Japan
November 14, 2012
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
1 Spatia
2. Summary
1 Introduction and motivation on IRLS
2 Temporal constraint with an adapted norm
3 Diagram flow and spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
2 Spatia
3. Introduction and motivation
Purpose
Foreground detection : Segmentation of moving objects in video sequence
acquired by a fixed camera.
Background modeling : Modelization of all that is not moving object.
Involved applications
Surveillance camera
Motion capture
On the importance
Crucial Task : Often the first step of a full video surveillance system.
Strategy used
Eigenbackground decomposition.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
3 Spatia
4. Eigenbackgrounds
Find an « ideal » subspace of the video sequence, which describes the best
as possible the (dynamic) background.
Fig.1 The common process of background subtraction via PCA (Principal Component
Analysis). At the final step, an adaptative threshold is used to get a binary image.
Without a robust framework, the moving object may be absorbed in the model !
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
4 Spatia
5. Data Structure Transformation
First, we consider a video sequence as a matrix A ∈ Rn×m
n is the amount of pixels in a frame (∼ 106 )
m is the number of frames considered (∼ 200)
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
5 Spatia
6. IRLS : Vector version (1)
The usual IRLS (Iteratively Reweighted Least Squares) scheme for solve
argmin ||Ax − b||α is given by :
x
D (i) = diag((ε + |b − Ax (i) |)α−2 )
(1)
x (i+1) = (At D (i) A)−1 At D (i) b
This IRLS method is convergent for 1 ≤ α < 3. An more suitable
formulation is :
r (i) = b − Ax (i)
D = diag((ε + |r (i) |)α−2 )
(2)
y (i) = (A DA)−1 A Dr (i)
x (i+1) = x (i) + (1 + λopt )y (i)
for λopt computed in a second inner loop. It is convergent for 1 ≤ α < +∞
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
6 Spatia
7. IRLS : Vector version (2)
For spatio/temporal RPCA, it needs to solve the following general problem :
argmin ||Ax − b||α + λ||Cx − d||β (3)
x
By derivation, the associated IRLS scheme is,
r1 = b − Ax (i) , r2 = d − Cx (i) , e1 = ε + |r1 |, e2 = ε + |r2 |
α 1 β 1 −1 β−2
D1 = ( e1 ) α −1 diag(e1 ), D2 = λ( e2 ) β diag(e2 )
α−2
(i) −1
(4)
y = (A D1 A + C D2 C ) (A D1 r1 + C D2 r2 )
x (i+1) = x (i) + (1 + λopt )y (i)
Good news : Just few lines in Matlab !
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
7 Spatia
8. IRLS : Matrix Version
More generally, we consider the following matrix regression problem with
two parameters norm (α, β) and a weighted matrix W ,
n m
α 1
min ||AX − B||α,β with ||Mij ||α,β = ( ( Wij |Mij |β ) β ) α (5)
X W W
i=1 j=1
The problem is solved in the same manner on matrices with a reweighted
regression strategy,
Until X is stable, repeat on each k-column
R ← B − AX
S ← ε + |R| (6)
α −1
β−2 β
Dk ← diag(Sik ◦ ( j (Sij ◦ Wij )) β ◦ Wik )k
Xik ← Xik +(1+Λ(max(α, β)))(At Dk A)−1 At Dk Rik
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
8 Spatia
9. Various RPCA formulation (only for α = 1)
PCA with a fixed rank is : min ||S||F
L,S
s.t. Rank(L) = k (7)
A=L+S
R(obust)PCA is (Non convex and NP-hard ) :
min ||σ(L)||0 + λ||S||0
L,S (8)
s.t. A=L+S
Convex relaxed problem of (8) is RPCA-PCP proposed by Candès et al. [1] :
min ||σ(L)||1 + λ||S||1
L,S (9)
s.t. A=L+S
where σ(L) means singular values of L.
A mix is Stable PCP of Zhou et al. [2] (both entry-wise and sparse noise) :
min ||σ(L)||1 + λ||S||1
L,S (10)
s.t. ||A − L − S||F < δ
All of them could be solved by Augmented Lagrangian Multipliers (ALM).
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images2012
November 14, & Applications),
9 Spatia
10. Video examples
Some examples, temporal RPCA and ideal RPCA with ground truth fitting.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 10 Spatia
11. Summary
1 Introduction and motivation on IRLS
2 Temporal constraint with an adapted norm
3 Diagram flow and spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 11 Spatia
12. Sparse solution
In RPCA, residual error is sparse.
Using the RPCA decomposition on a synthetic low-rank random matrix
plus noise, the error looks like :
Same principle with video. Sparse noise (or outliers) are the moving objects.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 12 Spatia
13. Let’s play with norms
Varying the α, β norm → Different kind of recovering pattern error.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 13 Spatia
14. Let’s play with norms...(2)
Some issues
What is the best specific norm for temporal constrain ?
Initial assumption is ||.||2,1 . Confirmed experimentally ?
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 14 Spatia
15. Validation
If ideal eigenbakgrounds are that, best norm should be ...
Let us denote Lopt , the ideal low-rank subspace which outliers do not contribute to PCA
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 15 Spatia
16. Experimental validation
Let us denote Lα,β , the low-rank
recovered matrix with a ||.||α,β -PCA.
The plot shows the error between
||Lopt − Lα,β ||F for parameters α
and β chosen freely. The darkest
value means that the error is the
smallest here.
||S||2,1 is not optimal, but for convenience we use it.
The benefit of the ad hoc block-sparse hypothesis is confirmed by
testing its efficiency directly on video dataset.
Experimentation done on dynamic category of dataset change detection
workshop 2012 : http://www.changedetection.net/
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 16 Spatia
17. Summary
1 Introduction and motivation on IRLS
2 Temporal constraint with an adapted norm
3 Diagram flow and spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 17 Spatia
18. Overview & addition of a spatial constraint via TV
Figure: Overview of the learning and evaluation process. Learning process needs
GT (Ground Truth) for better fits the eigenbackground components.
Spatial Constraint via TV
Suppose A = L + S where L and S are computed via some kind of
RPCA techniques with the addition of Total Variation penalty on S.
This penalty increases connected (or connexe) shapes.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 18 Spatia
19. Exemple with a synthetic 1-D signal
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 19 Spatia
20. Summary
1 Introduction and motivation on IRLS
2 Temporal constraint with an adapted norm
3 Diagram flow and spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 20 Spatia
21. Experimental Protocol
RPCA-IRLS is compared for the following four recent robust methods :
Low-Rank Block sparse Decomposition (LBD, 2011) [3]
Low-Rank Representation (LRR, 2011) [4]
Symmetric Alternating Direction Augmented Lagrangian (SADAL, 2011) [5]
Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA, 2012) [6]
References
[1] E. Candes, X. Li, Y. Ma, and J. Wright, Robust principal component analysis, International Journal of ACM,
vol. 58, no. 3, May 2011.
[2] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma, Stable principal component pursuit,IEEE ISIT
Proceedings, pp. 1518-1522, Jun. 2010.
[3] G. Tang and A. Nehorai, Robust principal component analysis based on low-rank and block-sparse matrix
decomposition, CISS 2011, 2011.
[4] Z. Lin, R. Liu, and Z. Su. Linearized alternating direction method with adaptive penalty for low-rank
representation. NIPS 2011, Dec. 2011.
[5] S. Ma. Algorithms for sparse and low-rank optimization : Convergence, complexity and applications. Thesis,
2011.
[6] J. He, L. Balzano, and A. Szlam. Incremental gradient on the grassmannian for online foreground and
background separation in subsampled video. Conference on Computer Vision and Pattern Recognition
(CVPR), June 2012.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 21 Spatia
22. Experimental Protocol & Quantitative Results
Optimal threshold is chosen for maximizing F-measure criterion
which is based 2 × 2 histogram of True/false/positive/negative :
TP TP 2 DR Prec
DR = , Prec = , F =
TP + FN TP + FP DR + Prec
Good performance is then obtained when the F-measure is closed to 1
Time consumption is not take into account in the evaluation process.
Figure: F-Measure on the Wallflower and I2R dataset.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 22 Spatia
23. Quantitative Results
Here, we show other experimental results on the real dataset of BMC 2012,
Video Recall Precision F-measure PSNR Visual Results
1 0.9139 0.7170 0.8036 38.2425
2 0.8785 0.8656 0.8720 26.7721
3 0.9658 0.8120 0.8822 37.7053
4 0.9550 0.7187 0.8202 39.3699
5 0.9102 0.5589 0.6925 30.5876
6 0.9002 0.7727 0.8316 29.9994
7 0.9116 0.8401 0.8744 26.8350
8 0.8651 0.6710 0.7558 30.5040
9 0.9309 0.8239 0.8741 55.1163
Table: Quantitative results with common criterions. Last column : sample of the
original video, GT and our results of the first four real video sequences.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 23 Spatia
24. Summary
1 Introduction and motivation on IRLS
2 Temporal constraint with an adapted norm
3 Diagram flow and spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 24 Spatia
25. Conclusion
Advantages
Experiments on video surveillance datasets show that this approach is more
robust than other recent RPCA formulation in presence of dynamic
backgrounds (DC) and illumination changes (IC).
Well suited for video with spatially spread and temporarily sparse outliers.
Disadvantages
Small local motions, like « waving trees » are not (yet) well modelized by
this kind of global PCA. For example, IC needs few eigenBackground and
DC needs more with the risk to integrate moving objects into the model.
Future Works
Lack in computation time : Further research consists in developping an
incremental version to update the model at every frame and to achieve the
real-time requirements.
C. Guyon, T. Bouwmans and E. Zahzah charles.guyon@univ-lr.fr via Robust Low Rank Matrix Factorization including / 25 U
Foreground Detection (MIA Laboratory (Mathematics Images & Applications),
November 14, 2012 25 Spatia