PhD final dissertation, Computer Engineering, University of Cagliari, Italy, March 2018

1. Pattern Recognition
and Applications Lab
University
of Cagliari, Italy
Department of
Electrical and
Electronic Engineering
Fast Person Re-Identification for
Intelligent Video Surveillance Systems
PhD final dissertation
PhD Program in Electronic and Computer Engineering
March 2018
Bahram Lavi
lavi.bahram@diee.unica.it

2. http://pralab.diee.unica.it
Intelligent Video Surveillance
2
Application tools:
●
Person re-identification
●
On-line tracking of people and objects
●
Off-line image retrieval for an object of
interest within different camera views
●
Recognizing of suspicious
actions/behaviors
…
Intelligent
video surveillance
systems
Machine
learning
Pattern
recognition
Computer
vision

3. http://pralab.diee.unica.it
What is person re-identification?
• Recognizing a person in non-overlapping camera views
• Application tools:
– on-line (e.g. tracking people among camera network)
– off-line (e.g. support to human operators, video-surveillance
operators, and forensic investigators)
3

4. http://pralab.diee.unica.it
Challenges and difficulties
4
Low
quality
ofthe
im
agesPose
variation
O
cclusions
Illum
ination
change
face-recognition
cannotbe
applied

5. http://pralab.diee.unica.it
Person re-identification systems
5
Focus of this work: off-line support to human operators
probe
template gallery: n images

6. http://pralab.diee.unica.it
Image descriptors
6
One possible cue to generate an image signature:
clothing appearance based on

Color histograms

Textures

Convolutional neural networks
...
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
probe

7. http://pralab.diee.unica.it
Image descriptors
7
a) pedestrian image
b) symmetry-based silhouette partition
c) weighted color histograms (WCH)
d) maximally stable color regions (MSCR)
e) recurrent highly structured patches
(RHSP)
Example: SDALF (Symmetry-Driven Accumulation of Local Features)
Farenzena et al., CVPR 2010

8. http://pralab.diee.unica.it
Matching score computation
8
Can be computed by:
●
manually defined a distance/similarity metric (e.g Euclidean dist.)
●
learnt from data
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
probe
Matching scores
computation

9. http://pralab.diee.unica.it
Matching score computation
9
Example of matching score computation
(probe image vs. each image in gallery set)
probe
IA
template
IB
dWCH
dMSCR
dRHSP
Final matching score:
Descriptor: SDALF

10. http://pralab.diee.unica.it
Matching score computation
10
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
probe
Matching scores
computation
Matching scores

11. http://pralab.diee.unica.it
Ranked list generation
11
A ranked list can be created by sorting the computed matching scores
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
Matching scores
computation
Ranked list
Ranked list
probe

12. http://pralab.diee.unica.it
Performance of re-identification
systems
Recognition accuracy:
Cumulative Matching Characteristic (CMC) curve
12
Example: CMC curve
for a template gallery of size 55

13. http://pralab.diee.unica.it
Processing time
Two kind of processing times should be taken into account:
● To generate descriptor (tD
)
● To compute matching scores (tM
)
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
probe
Matching scores
computation
Ranked list
creation
Off-line step
On-line step
In real-world applications:
processing time of matching phase
can be very high due to:
● Large size of template gallery
● Or using a high cost
similarity measurement
13

14. http://pralab.diee.unica.it
Has ever processing time addressed
on person Re-Id task?
NOT explicitly...
ONLY:

Durta et al. 2013: a two-stage system proposed
➔
Possibly losing the correct identity within the 2nd
stage

Satta et al. 2012: dissimilarity-based approach
14

15. http://pralab.diee.unica.it
First solution
A multi-stage ranking approach
FOR
 Proc. time tM using a given descriptor D and number of stage N
By

Cascading simplified versions of descriptor D
D1
,D2
,…,DN
,where DN
=D
With

an application constraint on processing time:
t ≤ tmax, with tM > tmax
15

16. http://pralab.diee.unica.it
Goal of this work
16
D1
D1 D2
D2 DN = DDN = D
probe
template gallery
final ranking
...
...
...
Trade-off between processing time and recognition accuracy for
a given descriptor D

17. http://pralab.diee.unica.it
Example: two-stage system
17
probe
template gallery: n images

18. http://pralab.diee.unica.it
Example: two-stage system
18
D1
D1
probe
1st
stage: t1
template gallery: n1 images
A fast first-stage descriptor is used to rank all the n1 images in the
template gallery, where n1
=n
n1=10

19. http://pralab.diee.unica.it
Example: two-stage system
19
D1
D1
probe
rank 1 2 3 4 5 6 7 8 9 10
1st
stage: t1
A fast first-stage descriptor is used to rank all the n1 images in the
template gallery

20. http://pralab.diee.unica.it
Example: two-stage system
20
D1
D1
probe
rank 1 2 3 4 5 6 7 8 9 10
1st
stage: t1
D2
D2
2nd
stage: t2 > t1
A slower but more accurate second-stage descriptor is used to
re-rank the top-ranked templates
n2=5

21. http://pralab.diee.unica.it
Example: two-stage system
21
D1
D1
probe
rank 1 2 3 4 5 6 7 8 9 10
1st
stage: t1
A slower but more accurate second-stage descriptor is used to
re-rank the top-ranked templates
D2
D2
2nd
stage: t2 > t1
n2=5

22. http://pralab.diee.unica.it
Example: two-stage system
22
D1
D1
probe
1 2 3 4 5 6 7 8 9 10
1st
stage: t1
Whole procedure of a two-stage system
final ranked listfinal ranked list
D2
D2
2nd
stage: t2 > t1
n2=5
template gallery

23. http://pralab.diee.unica.it
Example: two-stage system
23
D1
D1
probe
1 2 3 4 5 6 7 8 9 10
1st
stage: t1
Whole procedure of a two-stage system
final ranked listfinal ranked list
D2
D2
2nd
stage: t2 > t1
processing time:
n2=5
template gallery

24. http://pralab.diee.unica.it
Example: two-stage system
24
D1
D2
D1
D1 D2
D2

25. http://pralab.diee.unica.it
Example: two-stage system
25
D1
D2
D1
D1 D2
D2
n2=?
Rank

26. http://pralab.diee.unica.it
Simplified versions of a descriptor
26
●
Generating the simplified versions by
(1) Ad hoc (e.g. histogram-based descriptor; by reducing number of bins)
(2) OR dimensionality feature reduction methods (e.g. PCA)
●
Descriptors that
– generates fixed-size feature vector
●
LOMO (Liao et al., CVPR-2015), ~26000 elements
●
gBiCov (Ma et. al. IVC-2014), ~6000 elements

27. http://pralab.diee.unica.it
Simplified versions of a descriptor
27
●
Generating the simplified versions by
(1) Ad hoc
Example:
MCM → reducing the number of patches
●
Descriptors that
– does NOT generate fixed-size feature vector
●
SDALF (Farenzena et al., CVPR-2010),
●
MCM (Satta et. al. ICIAP-2011),

28. http://pralab.diee.unica.it
Multi-stage system design
28
DD
tM > tmax
D1
t1
D1
t1
D2
t2
D2
t2
DN = D
tN
=tM
DN = D
tN
=tM
...
n2 n3 nNn1=n
● Given a descriptor D with matching processing time tM
● Deploying into a multi-stage system
● Where
t1
<t2
<...<tN-1
<tN
AND n=n1
>n2
>...>nN
>1

29. http://pralab.diee.unica.it
✔ trial-and-error procedure to choose
●
Number of stages N
● N-1 Simplified versions of D = {D1, ..., DN-1}
✔ Number of templates n2
, n3
,…, nN
to be ranked by
D = {D1,..., DN-1, DN
} are chosen by

Two-stage system (N=2):

N-stage system (N>2):
✔ Accordingly, processing time for a multi-stage system:
Multi-stage system design
29
n2=⌊
n1(tmax−t1)
t2
⌋
ni=⌊ ´αni−1⌋, ´α=max α:t≤tmax

30. http://pralab.diee.unica.it
Experimental set-up
• Benchmark datasets:
– VIPeR (1264 images, 632 pedestrians)
– i-LIDS (476 images, 119 pedestrians)
– ETHZ sequence 1 (4,857 images, 83 pedestrians)
30
• State-of-the-art descriptors:
– SDALF (Farenzena et al., 2010)
– gBiCov (Ma et al., 2014)
– LOMO (Liao et al., 2015)
– MCM (Satta et al., 2011)

31. http://pralab.diee.unica.it
Experimental set-up
31
● As in Farenzena et al. (2010):
● for each dataset and each individual: one image as probe, one image as
template (is selected randomly)
● 10 runs of experiments
● average CMC curves over 10 runs
● Two- and three-stage system
● Application constraint, for a given descriptor D
➔ t ≤ 0.3tM
➔ t ≤ 0.4tM
➔ t ≤ 0.5tM

32. http://pralab.diee.unica.it
Results: two-stage systems
32
D1
D2
t1 = 2.08 sec
t2 = 9.44 sec
D1
D1 D2
D2

33. http://pralab.diee.unica.it
Results: two-stage systems
33
D1
D2
t1 = 2.08 sec
t2 = 9.44 sec
t <= 0.3t2
D1
D1 D2
D2
7
n2
n2=7

34. http://pralab.diee.unica.it
Results: two-stage systems
34
D1
D2
t1 = 2.08 sec
t2 = 9.44 sec
t <= 0.4t2
D1
D1 D2
D2
7 15
n2
n2=15

35. http://pralab.diee.unica.it
Results: two-stage systems
35
D1
D2
t1 = 2.08 sec
t2 = 9.44 sec
t <= 0.5t2
D1
D1 D2
D2
7 15 23
n2
n2=23

36. http://pralab.diee.unica.it
Results: three-stage systems
36
D1
D3
D2
D1
D1 D3
D3D2
D2
t1 = 0.0015 sec
t2 = 0.0057 sec
t3 = 0.0400 sec

37. http://pralab.diee.unica.it
Results: three-stage systems
37
D1
D3
D2
D1
D1 D3
D3D2
D2
16 37
n2n3
n2=37 n3=16
t1 = 0.0015 sec
t2 = 0.0057 sec
t3 = 0.0400 sec
t <= 0.3t3

38. http://pralab.diee.unica.it
Results: three-stage systems
38
D1
D3
D2
D1
D1 D3
D3D2
D2
16 23 37 44
n2n3
n2=44 n3=23
t1 = 0.0015 sec
t2 = 0.0057 sec
t3 = 0.0400 sec
t <= 0.4t3

39. http://pralab.diee.unica.it
Results: three-stage systems
39
D1
D3
D2
D1
D1 D3
D3D2
D2
16 23 30 37 44 50
n2n3
n2=50 n3=30
t1 = 0.0015 sec
t2 = 0.0057 sec
t3 = 0.0400 sec
t <= 0.5t3

40. http://pralab.diee.unica.it
Second solution
A feature reduction method
FOR
●
Descriptor with fixed-size feature vector (e.g. gBiCov)
40
Descriptor
generation
Descriptor
generation
Descriptor
generation
Descriptor
generation
template gallery
probe
Matching scores
computation
Ranked list
Off-line step
On-line step
Feature reduction
method

41. http://pralab.diee.unica.it
Experimental set-up
• Benchmark datasets:
– VIPeR (1264 images, 632 pedestrians)
– i-LIDS (476 images, 119 pedestrians)
• State-of-the-art descriptors:
– gBiCov (Ma et al., 2014): ~6000 elements
– LOMO (Liao et al., 2015): ~26000 elements
41
• Feature reduction methods:
– Principal Component Analysis (PCA)
– K-PCA
– ISOMAP
• Reduction in features size for:
r=[2,5 ,20 ,50,80,100,150,200,500,1000,1200,2000,2500,3500]

42. http://pralab.diee.unica.it
Results: gBiCov on VIPeR using PCA
42
Feature size reduced for:
and average processing
time in second
r=
[
2
5
20
100
200
500
1000
]

43. http://pralab.diee.unica.it
Results: Reconstruction error
43
Frobenius norm:
E=
‖X−X‖F
2
‖X‖
r=2
r=5
r=20
r=100
r=200
r=500

44. http://pralab.diee.unica.it
Publications
1) Bahram Lavi, G. Fumera, and F. Roli:
'Multi-stage ranking approach for fast person re-identification’,
Journal of IET Computer Vision, 2018
2) Bahram Lavi, M. Fatan, and D. Valls:
'Comparative Study of the Behavior of Feature Reduction Methods in
Person Re-identification Task',
7th International Conference on Pattern Recognition Applications and Methods,
ICPRAM, 2018
3) Bahram Lavi, G. Fumera, and F. Roli:
'A Multi-Stage Approach for Fast Person Re- identification',
Joint IAPR International Workshops on Statistical Techniques in Pattern
Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR),
Springer, 2016.
44

45. http://pralab.diee.unica.it
Conclusions
• Goal: to tune the trade-off between processing time and
accuracy in person re-identification systems
– for any given descriptor
– under constraint on processing time
• Proposed solutions:
– multi-stage ranking approach
– feature reduction methods for fixed-sized descriptors
45

46. http://pralab.diee.unica.it
Open issues
• Multi-stage ranking approach: design a criteria to optimize
the multi-stage system by suitably choosing:
– number of stages
– simplified versions of the chosen descriptor
➢ For instance, using the Pareto optimization technique to jointly
optimized all the design parameters
• Feature reduction method: design a criteria to choose a
suitable reduced feature size for a fixed-size descriptor
46