Lecture 2 of the Visual Perception module of the Erasmus Mundus MSc in Vision and Robotics (VIBOT)

1. 1
Lecture 2: Camera Calibration
Lecture 2
Camera Calibration
Joaquim Salvi
Universitat de Girona
Visual Perception

2. 2
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

3. 3
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

4. 4
Lecture 2: Camera Calibration
– Dense reconstruction – Visual inspection
– Object localization – Camera localization
2.1 Calibration Introduction
• Some applications of this capability include

5. 5
Lecture 2: Camera Calibration
Image courtesy of C. Taylor
“The Scholar of Athens,” Raphael, 1518
2.1 Calibration Introduction – Perspective Imaging

6. 6
Lecture 2: Camera Calibration
W
Z
W
Y
W
X
W
O
w
P
Image Plane
{ }W
u
P
I
Y
I
X
I
O
{ }I
Focal Point
W
Z
W
Y
W
X
W
O
w
P
Image Plane
{ }W
u
P
I
Y
I
X
I
O
{ }I
Focal Point
1
I
u
I I
u u
X
P Y
 
 
  
 
 
1
W
w
W
W w
w W
w
X
Y
P
Z
 
 
 
 
  
 
In pixels
In metrics?
w
l
w
l
0
W
w
W
W w
w W
w
X
Y
l
Z
 
 
 
 
  
 
2.1 Calibration Introduction

7. 7
Lecture 2: Camera Calibration
Modelling
G(X) X ?
Calibration
X !!!
Modelling:
• Determine the equation that approximates the camera behaviour.
• Define the set of unknowns in the equation (camera parameters).
• The camera model is an approximation of the physics & optics of the camera.
Calibration:
• Get the numeric value of every camera parameter.
G(X)
2.1 Calibration Introduction

8. 8
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

9. 9
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

10. 10
Lecture 2: Camera Calibration
2.2 Pinhole Model
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
dP
IY
IXIO
{ }I
RY
RX
{ }R
RO
Image
coordinate
system

11. 11
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 1: World to Camera)
Camera
coordinate
system
World
coordinate
system
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
C
WK
Step 1

12. 12
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 2: Projection)
Camera
coordinate
system
World
coordinate
system
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
f
Step 2
wX
wY
wZ
uX
uY
RY
RX
{ }R
RO

13. 13
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 3: Lens Distortion)
Camera
coordinate
system
World
coordinate
system
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
RY
RX
{ }R
RO
Step 3
dP

14. 14
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 3: Lens Distortion)
dP
uP dr
CY
CX
Observed position
Ideal
projection
dr: radial distortion
a
b
Radial distortion effect (a: negative, b: positive)
Radial Distortion

15. 15
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 3: Lens Distortion)
Axis with
maximum
radial
distortion
Axis with
minimum
tangential
distortion
CY
CX
Ideal
projection
Observed
position
dr: radial distortion
dt: tangential distortion
dPuP
dr
CY
CX
dt
Radial and Tangential Distortion
Image with distortionImage without distortion

16. 16
Lecture 2: Camera Calibration
2.2 Pinhole Model (Step 4: Camera to Image)
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
IY
IXIO
{ }I
RY
RX
{ }R
RO
Image
coordinate
system
Step 4
dP

17. 17
Lecture 2: Camera Calibration
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
dP
IY
IXIO
{ }I
RY
RX { }R
RO
C
WK
Image
coordinate
system
Step 1
Step 2Step 3
Step 4
2.2 Pinhole Model

18. 18
Lecture 2: Camera Calibration
2.2 Calibration Methods (I)
• Method of Hall
– Lineal method
– Transformation matrix
• Method of Faugeras-Toscani
– Lineal method
– Obtaining camera parameters
• Method of Faugeras-Toscani with distortion
– Iterative method
– Radial distortion
• Method of Tsai
– Iterative method
– Radial distortion
– Focal distance estimation
• Method of Weng
– Iterative method
– Radial and tangential distortion
• … and many more

19. 19
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

20. 20
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

21. 21
Lecture 2: Camera Calibration
2.3 The Method of Hall
• Method of Hall
– Lineal method
– Transformation matrix
• Method of Faugeras-Toscani
– Lineal method
– Obtaining camera parameters
• Method of Faugeras-Toscani with distortion
– Iterative method
– Radial distortion
• Method of Tsai
– Iterative method
– Radial distortion
– Focal distance estimation
• Method of Weng
– Iterative method
– Radial and tangential distortion
• … and many more

22. 22
Lecture 2: Camera Calibration
World
coordinate
system
WZ
WY WX
WO
wP
Image plane
{ }W
uP
IY
IXIO
{ }IImage
coordinate
system
2.3 The Method of Hall - Modelling

23. 23
Lecture 2: Camera Calibration
11 12 13 14
21 22 23 24
31 32 33 34
1
W
I w
u W
I w
u W
w
X
s X A A A A
Y
s Y A A A A
Z
s A A A A
 
    
        
        
 
Assume light is captured on the image plane by a linear projection
The matrix is defined up to a scale factor  Multiple Solutions
A component is fixed to the unity  Unique Solution
11 12 13 14
21 22 23 24
31 32 33 1
1
W
I w
u W
I w
u W
w
X
s X A A A A
Y
s Y A A A A
Z
s A A A
 
    
        
        
 
2.3 The Method of Hall - Modelling

24. 24
Lecture 2: Camera Calibration
11 12 13 14
31 32 33
21 22 23 24
31 32 33
1
1
W W W
I w w w
u W W W
w w w
W W W
I w w w
u W W W
w w w
A X A Y A Z A
X
A X A Y A Z
A X A Y A Z A
Y
A X A Y A Z
  

  
  

  
11 12 13 14
21 22 23 24
31 32 33 1
1
W
I w
u W
I w
u W
w
X
s X A A A A
Y
s Y A A A A
Z
s A A A
 
    
        
        
 
11 31 12 32 13 33 14
21 31 22 32 23 33 24
W I W W I W W I W I
w u w w u w w u w u
W I W W I W W I W I
w u w w u w w u w u
A X A X X A Y A X Y A Z A X Z A X
A X A Y X A Y A Y Y A Z A Y Z A Y
      
      
2.3 The Method of Hall - Calibration

25. 25
Lecture 2: Camera Calibration
 
 
2 1
2
1 0 0 0 0
0 0 0 0 1
W W W I W I W I W
i w w w u w u w u wi i i i i i i i i
W W W I W I W I W
i w w w u w u w u wi i i i i i i i i
Q X Y Z X X X Y X Z
Q X Y Z Y X Y Y Y Z
    
   
 
 
2 1
2
I
i ui
I
i ui
B X
B Y
 

 
T
11 12 13 14 21 22 23 24 31 32 33A A A A A A A A A A A A
 
1
t t
A Q Q Q B


QA B
Pseudoinverse leads to a unique solution:
1
A Q B

Obtaining 11 unknowns and each 2D point gives two equations
So, at least 6 points are needed. More points leads to a more accurate solution.
11 31 12 32 13 33 14
21 31 22 32 23 33 24
W I W W I W W I W I
w u w w u w w u w u
W I W W I W W I W I
w u w w u w w u w u
A X A X X A Y A X Y A Z A X Z A X
A X A Y X A Y A Y Y A Z A Y Z A Y
      
      
2.3 The Method of Hall - Calibration

26. 26
Lecture 2: Camera Calibration
Camera
coordinate
system
CY
CX CZ
CO
{ }C
IY
IXIO
{ }I
RY
RX
RO
{ }R
World
coordinate
system
WZ
WY
WX
WO
{ }W
Reconstruction
Area
Image of the calibrating pattern
 
 
2 1
2
1 0 0 0 0
0 0 0 0 1
W W W I W I W I W
i w w w u w u w u wi i i i i i i i i
W W W I W I W I W
i w w w u w u w u wi i i i i i i i i
Q X Y Z X X X Y X Z
Q X Y Z Y X Y Y Y Z
    
   
 
 
2 1
2
I
i ui
I
i ui
B X
B Y
 
  
1
t t
A Q Q Q B


2.3 The Method of Hall - Calibration

27. 27
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

28. 28
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

29. 29
Lecture 2: Camera Calibration
2.4 The Method of Faugeras-Toscani
• Method of Hall
– Lineal method
– Transformation matrix
• Method of Faugeras-Toscani
– Lineal method
– Obtaining camera parameters
• Method of Faugeras-Toscani with distortion
– Iterative method
– Radial distortion
• Method of Tsai
– Iterative method
– Radial distortion
– Focal distance estimation
• Method of Weng
– Iterative method
– Radial and tangential distortion
• … and many more

30. 30
Lecture 2: Camera Calibration
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
IY
IXIO
{ }I RY
RX { }R
RO
C
WK
Image
coordinate
system
Step 1
Step 2Step 3
Step 4
2.4 The Method of Faugeras-Toscani

31. 31
Lecture 2: Camera Calibration
• Extrinsic parameters: Model the situation and orientation of the camera with
respect to a world co-ordinate system.
• Intrinsic parameters: Model the behaviour of the internal geometry and the optical
characteristics of the camera.
u
w
v
Yc
Xc
Zc
Oc
Oi
(u0, v0)
Pu
P
image
co-ordinate
system
(píxels)
retinal
co-ordinate
system
(mm.)
Image plane
Retinal plane
Yr
Xr
Zr
w
World
co-ordinate
system
W
Z
W
Y
W
X
W
O { }W
Camera
co-ordinate system
2.4 The Method of Faugeras-Toscani

32. 32
Lecture 2: Camera Calibration
Yc
Xc
Zc
Zw
Yw
Oc
Ow
Pw
Camera
co-ordinate
system World
co-ordinate system
Retinal Plane
K
Xw
X
C
W Y
Z
t
T t
t
 
 
  
 
 
     
11 12 13
21 22 23
31 32 33
, , ,C
W
C
W
R Rot X Rot Y Rot Z
r r r
R r r r
r r r
    
 
 
  
 
 
C W
w w
C C W C
w W w W
C W
w w
X X
Y R Y T
Z Z
   
   
    
   
   
1 1
C W
Cw w
W
P P
K
   
   
   
3 3 3 1
1 30 1
C C
C W Wx x
W
x
R T
K
 
  
 
2.4 Extrinsic Parameters

33. 33
Lecture 2: Camera Calibration
CPw
CPu
Yc
Xc
Zc
Oc C
f
PZc
Yu
PYc Xu
PXc
C
C w
u C
w
C
C w
u C
w
X
X f
Z
Y
Y f
Z


2.4 The Intrinsic Parameters: Ideal Projection

34. 34
Lecture 2: Camera Calibration
pixel
Retinal
plane
(0, 0)
Yr
Xr (0, 0)
(Xd, Yd)
Image
Plane
(Xp, Yp)
R C
d u u
R C
d v u
X k X
Y k Y


2.4 The Intrinsic Parameters: Pixel Conversion

35. 35
Lecture 2: Camera Calibration
Yr
Xr
V
U
(0, 0)
Principal point
(u0,v0)
Computer image
co-ordinate
system
Camera
co-ordinate
system
0
0
I R
d d
I R
d d
X X u
Y Y v
  
  
2.4 The Intrinsic Parameters: Principal Point

36. 36
Lecture 2: Camera Calibration
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
IY
IXIO
{ }I RY
RX { }R
RO
C
WK
Image
coordinate
system
Step 1
Step 2Step 3
Step 4
2.4 The Method of Faugeras-Toscani

37. 37
Lecture 2: Camera Calibration
Real projection on the image plane(Xi, Yi)
(Xw, Yw, Zw) 3D object point with respect to world co-ordinate system
Affine transformation.
Modelled parameters: R, T
(Xc, Yc, Zc) 3D object point with respect to camera co-ordinate system
Perspective transformation.
Modelled parameter: f
(Xu, Yu) Ideal projection on the retinal plane
Pixel adjustment
Modelled parameters: ku, kv
(Xp,Yp) Real projection on the image plane
Adaptation to the computer image buffer
Modelled parameters: u0, v0
2.4 The Method of Faugeras-Toscani

38. 38
Lecture 2: Camera Calibration
C
C w
u C
w
C
C w
u C
w
X
X f
Z
Y
Y f
Z


R C
d u u
R C
d v u
X k X
Y k Y


0
0
I R
d d
I R
d d
X X u
Y Y v
  
  
0
0
C
I w
u u C
w
C
I w
u v C
w
X
X k f u
Z
Y
Y k f v
Z
  
  
0
0
0 0
0 0
0 0 1 0
1
C
I w
u u C
I w
u v C
w
X
s X u
Y
s Y v
Z
s


 
    
        
        
 
vv
uu
fk
fk




2.4 The Method of Faugeras-Toscani - Modelling

39. 39
Lecture 2: Camera Calibration
11 12 13
0
21 22 23
0
31 32 33
0 0
0 0
0 0 1 0
0 0 0 1 1
W
xI w
u u W
yI w
u v W
z w
r r r t X
s X u
r r r t Y
s Y v
r r r t Z
s


  
      
            
          
  
1 0 3 0
2 0 3 0
3
u u x z
v v y z
z
r u r t u t
A r v r t v t
r t
 
 
  
 
   
 
 
Intrínsecs Extrínsecs
2.4 The Method of Faugeras-Toscani - Modelling

40. 40
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

41. 41
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

42. 42
Lecture 2: Camera Calibration
11 12 13
0
21 22 23
0
31 32 33
0 0
0 0
0 0 1 0
0 0 0 1 1
W
xI w
u u W
yI w
u v W
z w
r r r t X
s X u
r r r t Y
s Y v
r r r t Z
s


  
      
            
          
  
1 0 3 0
2 0 3 0
3
u u x z
v v y z
z
r u r t u t
A r v r t v t
r t
 
 
  
 
   
 
 
Intrínsecs Extrínsecs
2.5 The Method of Faugeras-Toscani – Modelling

43. 43
Lecture 2: Camera Calibration
 
 
1 14 3 34
2 24 3 34
0
0
W I W
w u w
W I W
w u w
A P A X A P A
A P A Y A P A
   
   
31 14
34 34 34
32 24
34 34 34
I W W I
u w w u
I W W I
u w w u
AA A
X P P X
A A A
AA A
Y P P Y
A A A
  
  
1 1 2
3 2 2
I W W I
u w w u
I W W I
u w w u
X T P C T P X
Y T P C T P Y
  
  
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
2.5 The Method of Faugeras-Toscani – Calibration
1 0 3 0
2 0 3 0
3
u u x z
v v y z
z
r u r t u t
A r v r t v t
r t
 
 
  
 
   
 
 











343
242
141
AA
AA
AA





























1
343
242
141
w
W
u
I
u
I
P
AA
AA
AA
s
Ys
Xs

44. 44
Lecture 2: Camera Calibration
1
2
3
1
2
T
T
X T
C
C
 
 
 
 
 
 
 
 
B QX
1 3
1 3
0 1 0
0 0 1
t tW I W
w u w xi i i
t tI W W
x u w wi i i
P X P
Q
Y P P
 
 
 
  
 
 
 
I
ui
I
ui
X
B
Y
 
 
 
 
  
 
 
1
t t
X Q Q Q B


2.5 The Method of Faugeras-Toscani – Calibration
1 1 2
3 2 2
I W W I
u w w u
I W W I
u w w u
X T P C T P X
Y T P C T P Y
  
  
11 unknowns
minimum 6 points

45. 45
Lecture 2: Camera Calibration
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
3 1r 
2
1
zt
T

2.5 The Method of Faugeras-Toscani – tz
𝑅 =
𝑟1
𝑟2
𝑟3

46. 46
Lecture 2: Camera Calibration
1 2 1 2
1 2 1 2
cos
sin
v v v v
v v v v



 
0
1
1
0
t
i j
t
i j
i j
i j
r r i j
r r i j
r r i j
r r i j
 
 
  
  
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
2.5 The Method of Faugeras-Toscani – Intrinsics
02
3313
0
3331
0
3
2121
·
· u
t
rr
t
r
t
r
u
t
r
t
r
t
r
t
r
u
t
r
TTTT
z
u
zzzzz
u
zz
t






 
2
1
zt
T

2
2
21
0
T
TT
u
t


47. 47
Lecture 2: Camera Calibration
1 2 1 2
1 2 1 2
cos
sin
v v v v
v v v v



 
0
1
1
0
t
i j
t
i j
i j
i j
r r i j
r r i j
r r i j
r r i j
 
 
  
  
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
2 31 2
0 02 2
2 2
1 2 2 3
2 2
2 2
tt
t t t t
u v
T TT T
u v
T T
T T T T
T T
 
 
 
 
2.5 The Method of Faugeras-Toscani – Intrinsics

48. 48
Lecture 2: Camera Calibration
1 2 1 2
1 2 1 2
cos
sin
v v v v
v v v v



 
0
1
1
0
t
i j
t
i j
i j
i j
r r i j
r r i j
r r i j
r r i j
 
 
  
  
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
2.5 The Method of Faugeras-Toscani – Extrinsics
tt
t
tt
t
u
z
z
u
zz
TT
T
T
TT
TTr
TTT
T
T
TT
TTr
t
u
t
r
Tr
t
r
u
t
r
T
21
2
2
2
21
211
221
2
2
2
2
21
2110
3
11
1
0
3
1
1



























49. 49
Lecture 2: Camera Calibration
1 2 1 2
1 2 1 2
cos
sin
v v v v
v v v v



 
0
1
1
0
t
i j
t
i j
i j
i j
r r i j
r r i j
r r i j
r r i j
 
 
  
  
v
z
y
v
zz
z
u
z
x
u
zz
t
t
vC
t
r
v
t
r
T
t
r
T
t
t
uC
t
r
u
t
r
T





02
2
0
3
3
3
2
01
1
0
3
1
2 1 2
1 1 22
1 2 2
2 2 3
2 3 22
2 3 2
2
3
2
t
t t
t
t t
T T T
r T T
T T T
T T T
r T T
T T T
T
r
T
 
  
   
 
  
   

2 1 2
1 2
1 2 2
2 2 3
2 2
2 3 2
2
1
t
x t t
t
y t t
z
T T T
t C
T T T
T T T
t C
T T T
t
T
 
  
   
 
  
   

2.5 The Method of Faugeras-Toscani – Extrinsics

50. 50
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

51. 51
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

52. 52
Lecture 2: Camera Calibration
2.6 The Method of Faugeras-Toscani with distortion
• Method of Hall
– Lineal method
– Transformation matrix
• Method of Faugeras-Toscani
– Lineal method
– Obtaining camera parameters
• Method of Faugeras-Toscani with distortion
– Iterative method
– Radial distortion
• Method of Tsai
– Iterative method
– Radial distortion
– Focal distance estimation
• Method of Weng
– Iterative method
– Radial and tangential distortion
• … and many more

53. 53
Lecture 2: Camera Calibration
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
dP
IY
IXIO
{ }I
RY
RX { }R
RO
C
WK
Image
coordinate
system
Step 1
Step 2Step 3
Step 5
Step 4
2.6 The Method of Faugeras-Toscani with distortion

54. 54
Lecture 2: Camera Calibration
Ideal
projection
Observed
position
dr dt
Xr
Yr
dr: radial distortion
dt: tangential distortion
Pu
Pd
2.6 Lens Distortion

55. 55
Lecture 2: Camera Calibration
a
b
Radial distorsion effect Tangential distorsion effect
Xr
Axe of a
maximum
tangential
distortion
Axe of a
minimum
tangential
distortion
Radial distorsion is the most important and usually the only considered in
calibration.
2.6 Lens Distortion

56. 56
Lecture 2: Camera Calibration
X X Du d x  Y Y Du d y 
D X k rx d 1
2
D Y k ry d 1
2
r X Yd d 2 2
 
 
2 4
1 2
2 4
1 2
2 2
C
x d
C
y d
C C
d d
D X k r k r
D Y k r k r
r X Y
  
  
 
k1 is the most important component
and usuallly sufficient in most
applications.
2.6 Lens Distortion
Model of Faugeras-Toscani with distortion:

57. 57
Lecture 2: Camera Calibration
u
w
v
Yc
Xc
Zc
Oc
Oi
(u0, v0)
Pu
P
Camera
co-ordinate system
image
co-ordinate
system
f Pd
retinal
co-ordinate
system
Image plane
Retinal plane
Yr
Xr
Zr
X
f
P
P
u Xc
Zc

Y
f
P
P
u Yc
Zc

X X Du d x  Y Y Du d y 
D X k rx d 1
2
D Y k ry d 1
2
r X Yd d 2 2
X k Xp u d Y k Yp v d
X X ui p   0 Y Y vi p   0
2.6 The Method of Faugeras-Toscani with distortion

58. 58
Lecture 2: Camera Calibration
Camera
coordinate
system
World
coordinate
system
 0 0,u v
f
CY
CX CZ
CO
WZ
WY WX
WO
wP
Image plane
{ }W
{ }C
uP
dP
IY
IXIO
{ }I
RY
RX { }R
RO
C
WK
Image
coordinate
system
Step 1
Step 2Step 3
Step 5
Step 4
2.6 The Method of Faugeras-Toscani with distortion

59. 59
Lecture 2: Camera Calibration
(Xw, Yw, Zw) 3D object point with respect to world co-ordinate system
Affine transformation.
Modelled parameters: R, T
(Xc, Yc, Zc) 3D object point with respect to camera co-ordinate system
Perspective transformation.
Modelled parameter: f
(Xu, Yu) Ideal projection on the retinal plane
Radial lens distortion.
Modelled parameter: k1
(Xd, Yd) Real projection on the retinal plane
Pixel adjustment
Modelled parameters: ku, kv
(Xp,Yp ) Real projection on the image plane
Adaptation to the computer image buffer
Modelled parameters: u0, v0
(Xi, Yi) Real projection on the image plane
2.6 The Method of Faugeras-Toscani with distortion

60. 60
Lecture 2: Camera Calibration
2
1
2
1
C
C Cw
d dC
w
C
C Cw
d dC
w
X
f X k r X
Z
Y
f Y k r Y
Z
 
 
 
 
0
0
I
dC
d
u
I
dC
d
v
X u
X
k
Y v
Y
k





 1 1
C W
w w
C W
Cw w
WC W
w w
X X
Y Y
K
Z Z
   
   
   
   
      
   
r X Yd d 2 2
The model is NON-LINEAR
Iterative minimisation:
• Newton-Raphson
• Levenberg-Marquardt
2.6 The Method of Faugeras-Toscani with distortion

61. 61
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

62. 62
Lecture 2: Camera Calibration
Contents
2. Camera Calibration
2.1 Calibration introduction
2.2 The pinhole model
2.3 The method of Hall
2.4 The method of Faugeras-Toscani – Modelling
2.5 The method of Faugeras-Toscani – Calibration
2.6 The method of Faugeras-Toscani with distortion
2.7 Experimental comparison of methods

63. 63
Lecture 2: Camera Calibration
Hall Faugeras Faugeras
distorted
Tsai Weng
Transformation
matrix
Step 3
Lens
Distortion
Step 2
Projection
Step 1
World2camera
Transformation
with , , , tx, ty and tz
Projection with f
Radial distortion with k1
Undistorted
Multiple
distortion
k1, g1, g2, g3, g4
Transformation with
u0, v0 , ku and kv
Transformation
with u0, v0 and sx Transformation
with u0, v0,
ku and kv
Step 4
Camera2image
C C W C
w W w WP P T R
,
C C
C Cw w
u uC C
w w
X Y
X f Y f
Z Z
 
C C
w uP P
C C
u dP P
C I
d dP P
C C
u dP P
0
0
I C
d u d
I C
d v d
X k X u
Y k Y v
  
  
 
 
2 2
1
2 2
1
C C C C C
u d u u u
C C C C C
u d u u u
X X k X X Y
Y Y k Y X Y
  
  
1'
0
1
0
I C
d x x d
I C
d y d
X s d X u
Y d Y v


  
  
=
W C
w wP P
I W
d wP P A
2.7 Experimental Comparison - Methods

64. 64
Lecture 2: Camera Calibration
wP
Optical Ray
3Dd
2.7 Experimental Comparison - Accuracy Evaluation
• 3D Measurement
– Distance with respect to the optical ray
– Normalized Stereo Calibration Error
• 2D Measurement
– Accuracy of distorted image coordinates
– Accuracy of undistorted image
coordinates    
 
1 22 2
2 2 2
1
ˆ ˆ
1
NSCE
ˆ 12
C C C C
n
w w w wi i i i
C
i w u vi
X X Y Y
n Z  
 

   
 
 
  

Camera
coordinate
system
World
coordinate
system
f
CY
CX
WZ
WX
WO
wP
Image plane
{ }W
uP
dP
IX{ }I
RX
{ }R
RO
Image
coordinate
system
ˆ
uP
ˆ
dP
 0 0,u v
WY
CZ
CO { }C
IY
IO
RY
uP
ˆ
uP
 0 0,u v
dd
Observed Point
Linear Projection
- distortion
+ distortion
ud
ˆ
dP
dP

65. 65
Lecture 2: Camera Calibration
2.7 Experimental Comparison: Synthetic Images (I)
2D distorted image (pix.) 2D undistorted image (pix.)
Mean Standard
desviation
Max Min Mean Standard
desviation
Max Min
1 Hall 0.2676 0.1979 1.2701 0.0213 0.2676 0.1979 1.2701 0.0213
2 Faugeras 0.2689 0.1997 1.2377 0.0075 0.2689 0.1997 1.2377 0.0075
3 Faugeras with distortion 0.0840 0.0458 0.2603 0.0081 0.0834 0.0454 0.2561 0.0080
4 Tsai 0.0838 0.0457 0.2426 0.0035 0.0832 0.0453 0.2386 0.0035
5 Weng 0.0845 0.0455 0.2608 0.0019 0.0843 0.0443 0.2584 0.0129
2D distorted
0
0,05
0,1
0,15
0,2
0,25
0,3
1 2 3 4 5
pix.
Mean Standard deviation
2D undistorted
0
0,05
0,1
0,15
0,2
0,25
0,3
1 2 3 4 5
pix.
Mean Standard deviation

66. 66
Lecture 2: Camera Calibration
2.7 Experimental Comparison: Synthetic Images (II)
3D position (mm) NSCE
Mean Standard
desviation
Max Min
1 Hall 0.1615 0.1028 0.5634 0.0113 n/a
2 Faugeras 0.1811 0.1357 0.8707 0.0147 0.6555
3 Faugeras NR with distortion 0.0566 0.0307 0.1694 0.0055 0.2042
4 Tsai optimized 0.0565 0.0306 0.1578 0.0087 0.2037
5 Weng 0.0570 0.0305 0.1696 0.0088 0.2064
Normalized Stereo Calibration Error
Normalized Stereo Calibration Error
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
1 2 3 4 5
NSCE
3D position
0
0,05
0,1
0,15
0,2
1 2 3 4 5
mm.
Mean Standard deviation

67. 67
Lecture 2: Camera Calibration
Computing Time
160 punts 1800 punts
• Hall 1 ms 70 ms
• Faugeras 1 ms 70 ms
• Faugeras with distortion 10 ms 380 ms
• Tsai 10 ms 530 ms
• Weng 51 ms 4216 ms
Pentium III at 1 GHz.
2.7 Experimental Comparison: Synthetic Images (III)

68. 68
Lecture 2: Camera Calibration
2.7 Experimental Comparison: Real Images (I)
Camera
coordinate
system
CY
CX CZ
CO
{ }C
IY
IXIO
{ }I
RY
RX
RO
{ }R
World
coordinate
system
WZ
WY
WX
WO
{ }W
Reconstruction
Area
Image of the calibrating pattern
3D position (mm) NSCE
Mean Standard
desviation
Max Min
Hall 0.5219 0.2595 1.1370 0.0143 n/a
Faugeras 0.7782 0.4253 2.0210 0.0187 4.0649
Faugeras with distortion 0.4967 0.3367 1.5642 0.0094 2.5489
Tsai 0.4815 0.3023 1.4014 0.0093 2.4836
Weng 0.4740 0.2904 1.2669 0.0087 2.4556

69. 69
Lecture 2: Camera Calibration
2.7 Experimental Comparison: Real Images (II)
3D position (mm) NSCE
Mean Standard
desviation
Max Min
Hall 1.5698 0.9842 8.9249 0.0247 n/a
Faugeras 1.6187 0.9856 8.8812 0.0302 2.0175
Faugeras with distortion 0.9930 0.5660 3.2386 0.0154 0.9909
Tsai 0.9927 0.5655 3.2311 0.0153 0.9908
Weng 0.9896 0.5724 3.3526 0.0149 0.9869
Image of the calibration patternStereo camera over a mobile robot

70. 70
Lecture 2: Camera Calibration
2.7 Experimental Comparison - Conclusions
• Implementation of 5 of the most used camera calibration
methods
– Notation was unified
– The methods were compared in terms of model and
calibration
• The accuracy of non-linear methods is better than linear
methods
• Modelling of radial distortion is quite sufficient when high
accuracy is required
• Accuracy measuring methods obtain similar results if they are
relatively compared
Additional bibliography:
J. Salvi, X. Armangué and J. Batlle. A Comparative Review of Camera
Calibrating Methods with Accuracy Evaluation. Pattern Recognition,
PR, pp. 1617-1635, Vol. 35, Issue 7, July 2002.