We define a global matching framework based on energy pyramid, the Global Matching via Energy Pyramid (GM-EP) algorithm, which estimates the disparity map from a single stereo-pair by solving an energy minimization problem. We efficiently address this minimization by globally optimizing a coarse to fine sequence of sparse Conditional Random Fields (CRF) directly defined on the energy. This global discrete optimization approach guarantees that at each scale we obtain a near optimal solution, and we demonstrate its superiority over state of the art image pyramid approaches through application to real stereo-pairs. We conclude that multiscale approaches should be build on energy pyramids rather than on image pyramids.
Traditional Agroforestry System in India- Shifting Cultivation, Taungya, Home...
Fast Global Stereo Matching Via Energy Pyramid Minimization
1.
ISPRS – PCV 2014
Fast global matching via energy
pyramid (disparity esAmaAon)
Zurich, 9/5/2014
Bruno Conejo, Phd student (bconejo@caltech.edu)
with S. Leprince, F. Ayoub & JP. Avouac (GPS, Caltech)
2. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
2
Disparity
is
inversely
propor?onal
to
depth!
Epipolar
geometry
stereo-‐imaging
setup
Introduc?on:
Reference
Image
Target
Image
Disparity
map
3. Given
a
stereo-‐pair
of
images
(Ir
,It)
how
to
retrieve
the
most
probable
disparity
map
d*?
Regulariza?on:
priors
on
disparity
Matching:
encourages
similarity
In
term
of
probability,
we
need
to
es?mate
the
Maximum
A
Posteriori
(MAP)
of:
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
3
Modeling:
a
bayesian
approach
Gibbs
measure
relates
probability
density
func?on
to
energy:
Energy
of
configura?on
x
Normalizing
constant
Reference
Image:
Ir
Target
Image:
It
4. From
the
Gibbs
measure
we
relates
probabili?es
to
the
energies
(EM
,
ER
,
E):
Matching:
Similarity
criteria
(L1,
L2,
ZNCC,
...)
Regulariza?on:
Piecewise
constant
prior:
Modulated
by
radiometric
discon?nuity:
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
4
Modeling:
con?nuous
Condi?onal
Random
Field
(CRF)
First
order
Condi?onal
Random
Field
(CRF):
p
q
Associated
graph
Reference
Image
Set
of
nodes
Set
of
edges
5. We
need
to
globally
op?mize
a
con?nuous
CRF
over
all
possible
disparity
maps
(D):
However,
this
is
a
non-‐convex
problem:
varia?onal
approaches
can
not
work!
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
5
Modeling:
non-‐convexity
Many
local
minima!
6. Solu%on:
Restrict
d
to
take
value
in
a
finite
discrete
set,
i.e.,
the
“search
space”
encoded
by
a
label
space.
This
leads
to
globally
op?mize
a
first
order
discrete
CRF
(s?ll
NP-‐Hard)
:
-‐ Message
passing
(quadra?c
w.r.t
search
space):
Loopy
BP,
TRW-‐S,
DD-‐MRF,
…
-‐ Making
move
(linear
w.r.t
search
space)
:
α-‐exp,
β-‐swap,
Fast-‐PD,
…
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
Discrete
op?miza?on
6
Pairwise
term:
Encodes
prior
(regulariza?on)
Unary
term:
Encodes
similarity
(matching)
Label
space:
Encodes
for
each
node
all
poten?al
disparity
to
evaluate
7. Mul?-‐scale
approaches
We
work
with
large
images
(30,000
by
30,000)
and
we
have
large
disparity
range
(-‐300,300).
A
direct
approach
is
inefficient
(even
impossible)
and
unnecessary!
Locally
the
disparity
range
is
“small”.
We
can
use
a
mul?-‐scale
approach:
-‐ Coarsest
scales:
“large”
dispari?es
with
low
spa?al
frequencies
(natural
topography).
-‐ Finest
scales:
“small”
dispari?es
with
high
spa?al
frequencies
(man
made
objects).
Two
mul?-‐scale
schemes
are
possible:
-‐ Image
pyramid
(classic,
GM-‐IP
algorithm).
-‐ Energy
pyramid
(ours,
GM-‐EP
algorithm).
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
7
Reference
Image
Associated
disparity
8. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
8
Mul?-‐scale:
GM-‐IP
algorithm
(Image
Pyramid)
Build
&
Opt.
CRF
Build
&
Opt.
CRF
Ir
It
Algorithm
1) Build
pyramid
of
image
for
each
image
by
itera?ve
downsampling
2) Compute
and
op?mize
CRF
at
coarsest
scale
3) Define
new
search
space
around
current
solu?on
4) Repeat
(2-‐3)
un?l
finest
scale
10. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
10
Mul?-‐scale:
CRF
sparsifica?on
Label
before
op?m.
Label
amer
op?m.
Label
space
to
explore
Removed
label
range
11. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
11
Mul?-‐scale:
GM-‐IP(Image
Pyramid)
vs
GM-‐EP
(Energy
Pyramid)
The
image
pyramid
yields
a
smoothed
representa?on
of
the
energy
and
destroys
local
minimums,
especially
at
coarse
scale:
Different
minima!
Energy
pyramid
Image
pyramid
12. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
12
Stereo-‐imaging
in
urban
context:
(Reference
Image)
13. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
13
Stereo-‐imaging
in
urban
context:
(Target
Image)
14. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
14
Stereo-‐imaging
in
urban
context:
(Reference
Image)
15. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
15
Stereo-‐imaging
in
urban
context:
Area
of
interest
in
reference
image
Baseline
=
no
pyramid
(direct
op@miza@on)
Quan?ta?ve
comparison
between
algorithms:
Unary
terms:
ZNCC
with
5x5
windows
4
scales,
1
itera?on
per
scale.
16. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
16
Coarse
scale:
GM-‐EP
(Energy
Pyramid)
vs
GM-‐IP(Image
Pyramid)
17. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
17
Coarse
scale:
GM-‐EP
(Energy
Pyramid)
vs
GM-‐IP(Image
Pyramid)
18. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
18
Finest
scale:
GM-‐EP
(Energy
Pyramid)
vs
GM-‐IP(Image
Pyramid)
19. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
19
Finest
scale:
GM-‐EP
(Energy
Pyramid)
vs
GM-‐IP(Image
Pyramid)
20. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
20
GM-‐EP
(Energy
Pyramid)
vs
MicMac
(Image
Pyramid)
21. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
21
GM-‐EP
(Energy
Pyramid)
vs
MicMac
(Image
Pyramid)
22.
Key
points:
• A
versa?le
matching
model
efficiently
op?mized
with
state
of
the
art
discrete
op?miza?on
technique.
• Energy
pyramid
yields
a
beper
representa?on
of
the
energy.
Future
work:
• Modeling:
• Impact
of
images’
noise,
• Symmetry
w.r.t.
the
images,
• Occlusions,
• CRF
parameters
(unary
terms,
weights
of
CRF,
distance
func?on).
• Op?miza?on
• Auto
defini?on
of
the
search-‐space,
• Mul?grid
instead
of
mul?scale,
• Paralleliza?on
for
shared
and
distributed
memory
architectures.
B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
22
Conclusion
&
Future
work:
24. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
24
Stereo-‐imaging
in
urban
context:
(Reference
Image)
25. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
25
Stereo-‐imaging
in
urban
context:
(Target
Image)
26. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
26
Finest
scale:
GM-‐EP
(Energy
Pyramid)
vs
MicMac
(Image
Pyramid)
Micmac
GM-‐EP
(Energy
Pyramid)
27. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
27
Finest
scale:
GM-‐EP
(Energy
Pyramid)
vs
MicMac
(Image
Pyramid)
28. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
28
Mul?-‐scale:
GM-‐IP
algorithm
(Image
Pyramid)
29. B.Conejo
-‐
Fast
global
Matching
via
Energy
Pyramid
29
Mul?-‐scale:
GM-‐EP
algorithm
(Energy
Pyramid)