1. Robot Vision for the Visually Impaired
Vivek Pradeep, Gerard Medioni, James Weiland
presented by
Phongsathorn Eakamongul
Department of Computer Science
Asian Institute of Technology
2010, December 7
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 1 / 18

2. Outline
1 Abstracts
2 System Description
3 Result
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 2 / 18

3. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

4. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

5. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

6. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

7. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

8. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

9. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

10. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

11. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

12. Abstracts
head-mounted : wide-field information compare to shoulder or waist-mounted
design in literature which require body rotations
stereo-vision
navigational assistance device
visual odometry : dense 3D with 2D elevation grids
metric-topological SLAM
build vicinity map
3D traversability analysis to steer subjects away from obstacles in the path
use microvibration motors provides cues for taking evasive action : they use tactile
cues instead of audio since the latter impose greater cognitive load on the subject,
and blind users rely on hearing to perform a wide variety of other tasks
experiment running at 10 Hz
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 3 / 18

13. Introduction
visual impairment : need long cane or guide dog
In US, 109,000 people : use long canes, 7,000 use dog guides
only 1,500 graduate from dog-guid user program
Electronic travel aids (ETAs), leveraging ultrasonic, laser, or vision sensors
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 4 / 18

14. Introduction
visual impairment : need long cane or guide dog
In US, 109,000 people : use long canes, 7,000 use dog guides
only 1,500 graduate from dog-guid user program
Electronic travel aids (ETAs), leveraging ultrasonic, laser, or vision sensors
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 4 / 18

15. Introduction
visual impairment : need long cane or guide dog
In US, 109,000 people : use long canes, 7,000 use dog guides
only 1,500 graduate from dog-guid user program
Electronic travel aids (ETAs), leveraging ultrasonic, laser, or vision sensors
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 4 / 18

16. Introduction
visual impairment : need long cane or guide dog
In US, 109,000 people : use long canes, 7,000 use dog guides
only 1,500 graduate from dog-guid user program
Electronic travel aids (ETAs), leveraging ultrasonic, laser, or vision sensors
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 4 / 18

17. Introduction
visual impairment : need long cane or guide dog
In US, 109,000 people : use long canes, 7,000 use dog guides
only 1,500 graduate from dog-guid user program
Electronic travel aids (ETAs), leveraging ultrasonic, laser, or vision sensors
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 4 / 18

18. wearable array of microvibration motors provides a tactile cues
and guide user along safe path
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 5 / 18

19. Outline
1 Abstracts
2 System Description
3 Result
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 6 / 18

20. Online SLAM + obstacle detection
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 7 / 18

21. Stereo Vision Odometry
t−1 t−1 t t−1 t−1 t
matched correspondences across (PL , PR , PL ) or (PL , PR , PR ) can be
computed using three-point algorithm in RANSAC setting
for robustness, features matching and reprojection errors are measured across
four views
Sparse Bundle Adjustment
feature covariances can be propagated to get motion uncertainty for use in the
SLAM filter
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 8 / 18

22. Stereo Vision Odometry
t−1 t−1 t t−1 t−1 t
matched correspondences across (PL , PR , PL ) or (PL , PR , PR ) can be
computed using three-point algorithm in RANSAC setting
for robustness, features matching and reprojection errors are measured across
four views
Sparse Bundle Adjustment
feature covariances can be propagated to get motion uncertainty for use in the
SLAM filter
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 8 / 18

23. Stereo Vision Odometry
t−1 t−1 t t−1 t−1 t
matched correspondences across (PL , PR , PL ) or (PL , PR , PR ) can be
computed using three-point algorithm in RANSAC setting
for robustness, features matching and reprojection errors are measured across
four views
Sparse Bundle Adjustment
feature covariances can be propagated to get motion uncertainty for use in the
SLAM filter
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 8 / 18

24. Stereo Vision Odometry
t−1 t−1 t t−1 t−1 t
matched correspondences across (PL , PR , PL ) or (PL , PR , PR ) can be
computed using three-point algorithm in RANSAC setting
for robustness, features matching and reprojection errors are measured across
four views
Sparse Bundle Adjustment
feature covariances can be propagated to get motion uncertainty for use in the
SLAM filter
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 8 / 18

25. Stereo Vision Odometry
t−1 t−1 t t−1 t−1 t
matched correspondences across (PL , PR , PL ) or (PL , PR , PR ) can be
computed using three-point algorithm in RANSAC setting
for robustness, features matching and reprojection errors are measured across
four views
Sparse Bundle Adjustment
feature covariances can be propagated to get motion uncertainty for use in the
SLAM filter
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 8 / 18

26. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

27. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

28. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

29. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

30. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

31. SLAM
Rao-Blackwellised particle filter (RBPF) in FastSLAM framework
which use KLT and SIFT tracking
construct 2 maps
SLAM map : collection of sparse landmarks that propagated every frame to yield
consistent camera pose estimates, for SLAM purpose only
traversability map : dense 3D cloud from triangulation
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 9 / 18

32. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

33. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

34. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

35. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

36. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

37. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

38. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

39. Metric-Topological SLAM
serveral thousands of landmarks environment
two levels of environment representation
local, metric (submap) : estimates state information
six dimensional camera trajectory st
sparse map mt
feature observations (KLT/SIFT) z t
camera motion estimates u t
RBPF
p(st , mt |z t , u t ) ≈ p(st |z t , u t ) i p(mt (i)|st , z t , u t )
mt (i) : ith landmark in the map represented by N(µi , σ i )
each time feature is observed, the corresponding lankmark is updated using EKF
RBPF enables us to only update the observed landmark instead of the whole map
global topological
map is represents as a collection of submap
annotated graph
G = ({i M}i∈Ωt , {b Λ}a,b∈Ωt )
a
i
M : annotated submaps
Ωt : set of computed submaps
b
a Λ : coordinate transformations between adjacent maps
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 10 / 18

40. Traversability Map
5 radius sphere
multi-surface elevation map : point cloud is quantized into 2D grid
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 11 / 18

41. Traversability Map
5 radius sphere
multi-surface elevation map : point cloud is quantized into 2D grid
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 11 / 18

42. Traversability Map
5 radius sphere
multi-surface elevation map : point cloud is quantized into 2D grid
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 11 / 18

43. Prediction Motion and Cue Generation
if magnitude of translation respect to previous position exceeds certain threshold,
the direction of motion and reference position are updated
little translation -> no update
cue generation : most continuous traversable path ( Green color in picture )
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 12 / 18

44. Outline
1 Abstracts
2 System Description
3 Result
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 13 / 18

45. Result
Green : travesible
Red : not
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 14 / 18

46. error of camera frame-to-frame heading (yaw), when compared with
readings from a commercially Inertial Measurement Unit (IMU)
camera motion : slow (< 5 degree/s), medium (5-20 degree/s), fast (20-30 degree/s)
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 15 / 18

47. SLAM result
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 16 / 18

48. Traversability Map
one frame exp
patch that has thickness > 30 cm is labeled as vertical
5 horizontal patches is labeled as traversable
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 17 / 18

49. Experiment
Manually generate cues : wireless remote control
Autonomous generate cues, like group 4
Phongsathorn (AIT) Robot Vision for the Visually Impaired Short Occasion 18 / 18