This document summarizes a research paper on real-time 3D modeling of vehicles from monocular camera images. The researchers developed a system that can count and classify vehicles in real-time using a single low-cost camera. It extracts 2D blobs from camera images, then tracks vehicles in 3D using a Bayesian framework and particle filter. The system was able to classify over 13,000 vehicles with over 90% accuracy in challenging scenarios like dusk, rain and traffic jams, demonstrating its flexibility and real-time capabilities.
10. System overview
Calibration and rectification
Radial distortion Metric information
We can obtain the approximate
intrinsic and extrinsic parameters
of the calibration from the
homography
Perspective distortion
Algarve (Portugal) - VISAPP 2011/03/05-07 10 10
11. System overview
Segmentation
Pixel-level classification
Colour (IHLS) and gradient
Algarve (Portugal) - VISAPP 2011/03/05-07 11 11
13. System overview
3D tracking
Infer 3D boxes that projects onto detected 2D blobs
Propagate through time
Algarve (Portugal) - VISAPP 2011/03/05-07 13 13
14. Bayesian framework
State (t-1) State (t)
(Motion) Prior
Models Likelihood
Algarve (Portugal) - VISAPP 2011/03/05-07 14 14
15. Bayesian framework
MCMC-based particle filter
Recursive Bayes’ rule
Prediction
Posterior Likelihood
Approximate with a set of samples (particles)
Sample-based
Obtain samples with Metropolis-Hastings approach
Algarve (Portugal) - VISAPP 2011/03/05-07 15 15
16. Bayesian framework
Likelihood model
Plane perpendicular to the ray
passing through
3D hypotheses
Projection of into the ray
3D best box
2D blob
Algarve (Portugal) - VISAPP 2011/03/05-07 16 16
18. Real-time operation
Real-time is a hard requirement
High speed roads + short stretchs = few frames
MCMC particle filter
Propagate a chain of samples using Metropolis Hastings
It might be slow (depend on the processor)
Simplifications are welcome
One single sample
From 3D to 1D search
Check discrete posterior values and compute argmax
Algarve (Portugal) - VISAPP 2011/03/05-07 18 18
19. Real-time operation
Metropolis-Hastings
Full computation
Previous set of
particles
Single-sample posterior representation
Single previous
hypothesis
Algarve (Portugal) - VISAPP 2011/03/05-07 19 19
20. Real-time operation
From 3D to 1D search
Need to sample 3D
space WHL
Measures the distance of
the hypothesis
to the ray
s.t.
Need to sample 1D ray
Let us consider only
points on the ray
Algarve (Portugal) - VISAPP 2011/03/05-07 20 20
21. Real-time operation
Finite (small) number of posterior evaluations
Approximate MAP as product of most likely model and prior
Project each vehicle model to the ray given by the 2D observation
We reduce the search to the
evaluation of M points of the ray
These points are the projection of
the set of prior models on the ray
Algarve (Portugal) - VISAPP 2011/03/05-07 21 21
22. Results
Remarkable capabilities
Flexibility to adapt to large vehicles
Algarve (Portugal) - VISAPP 2011/03/05-07 22 22
24. Results
Capabilities
Adapts to visible part of vehicles
Better fits when several frames have led the filter to converge
Algarve (Portugal) - VISAPP 2011/03/05-07 25 25
25. Results
Known problems / difficulties
Shadows Dark vehicles Visual artifacts
Merging vehicles Night time Occlusions / perspective
Algarve (Portugal) - VISAPP 2011/03/05-07 26 26
28. Results
Light vehicles Heavy vehicles
1 1
0,95 0,95
Dusk Dusk
Precision
Precision
Rain and shadows Rain and shadows
0,9 0,9
Traffic jam Traffic jam
Dusk and rain Dusk and rain
0,85 0,85
Perspective Perspective
Color noise Color noise
0,8 0,8
0,8 0,85 0,9 0,95 1 0,8 0,85 0,9 0,95 1
Recall Recall
Dusk Rain and shadows Traffic jam Dusk and rain Perspective Colour noise
Algarve (Portugal) - VISAPP 2011/03/05-07 29 29
29. Conclusions
Vehicle detection and tracking
3D volume estimation
Real-time operation
Using some approximations within the particle filter framework
Flexible
Automatically adapts to illumination changes
Requires calibration to work under different perspectives
Average counting and classification rates above 90% in challenging
scenarios
Future work: interaction between objects, autocalibration,
automatic selection of parameters…
Algarve (Portugal) - VISAPP 2011/03/05-07 30 30