Slides for a course on signal and image processing.

1. Signal and Image Processing
with Orthogonal Decompositions
Gabriel Peyré
www.numerical-tours.com

2. Signals, Images and More
2
Continuous signal: f 2 L ([0, 1]).

3. Signals, Images and More
2
Continuous signal: f 2 L ([0, 1]).
2
Continuous image: f 2 L ([0, 1]2 ).

4. Signals, Images and More
2
Continuous signal: f 2 L ([0, 1]).
2
Continuous image: f 2 L ([0, 1]2 ).
General setting: f : [0, 1] ! R
d s
Videos: d = 3, s = 1.

5. Signals, Images and More
2
Continuous signal: f 2 L ([0, 1]).
2
Continuous image: f 2 L ([0, 1]2 ).
General setting: f : [0, 1] ! R
d s
Videos: d = 3, s = 1.
Color image: d = 2, s = 3.

6. Signals, Images and More
2
Continuous signal: f 2 L ([0, 1]).
2
Continuous image: f 2 L ([0, 1]2 ).
General setting: f : [0, 1] ! R
d s
Videos: d = 3, s = 1.
Color image: d = 2, s = 3.
Multi-spectral: d = 2, s 3.

7. Overview
•Orthogonal Representations
•Linear and Non-linear Approximations
•Compression and Denoising

8. Orthogonal Decompositions
Continuous signal/image f L ([0, 1] ).
2 d
Orthogonal basis { m }m of L ([0, 1]d )
2

9. Orthogonal Decompositions
Continuous signal/image f L ([0, 1] ).
2 d
Orthogonal basis { m }m of L ([0, 1]d )
2
f= f, m m
m
||f || = |f (x)|2 dx = | f, m ⇥|2
m

10. Orthogonal Decompositions
Continuous signal/image f L ([0, 1] ).
2 d
Orthogonal basis { m }m of L ([0, 1]d )
2
f= f, m m
m
m
||f || = |f (x)|2 dx = | f, m ⇥|2
m

11. 1-D Wavelet Basis
Wavelets:
1 x 2j n
j,n (x) =
2j/2 2j
Position n, scale 2j , m = (n, j).

12. 1-D Wavelet Basis
Wavelets:
1 x 2j n
j,n (x) =
2j/2 2j
Position n, scale 2j , m = (n, j).

13. 2-D Fourier Basis
m2 m1 ,m2
Basis { m (x)}m of L ([0, 1])
2
tensor m1
product
Basis { m1 ,m2 (x1 , x2 )}m1 ,m2 of L ([0, 1]2 )
2
m1 ,m2 (x1 , x2 ) = m1 (x1 ) m2 (x2 )

14. 2-D Fourier Basis
m2 m1 ,m2
Basis { m (x)}m of L ([0, 1])
2
tensor m1
product
Basis { m1 ,m2 (x1 , x2 )}m1 ,m2 of L ([0, 1]2 )
2
m1 ,m2 (x1 , x2 ) = m1 (x1 ) m2 (x2 )
x
m
Fourier
transform
f (x) f, m1 ,m2

15. 2-D Wavelet Basis
3 elementary wavelets { H
, V
, D
}.
H
(x) V
(x) D
(x)
Orthogonal basis of L ([0, 1]2 ): 2
k=H,V,D
j,n (x) =2 (2
k j j
x n)
j<0,2j n [0,1]2

16. 2-D Wavelet Basis
3 elementary wavelets { H
, V
, D
}.
H
(x) V
(x) D
(x)
Orthogonal basis of L ([0, 1]2 ): 2
k=H,V,D
j,n (x) =2 (2
k j j
x n)
j<0,2j n [0,1]2

17. Example of Wavelet Decomposition
x
wavelet
transform
(j, n, k)
f (x) f, k
j,n

18. Discrete Computations
Discrete orthogonal basis { m } of CN .
f= f, m m
m

19. Discrete Computations
Discrete orthogonal basis { m } of CN .
f= f, m m
m
1 2i nm
m [n] = eN
N
Fast Fourier Transform (FFT), O(N log(N )) operations.

20. Discrete Computations
Discrete orthogonal basis { m } of CN .
f= f, m m
m
1 2i nm
m [n] = eN
N
Fast Fourier Transform (FFT), O(N log(N )) operations.
Discrete Wavelet basis: no closed-form expression.
Fast Wavelet Transform, O(N ) operations.

21. Overview
•Orthogonal Representations
•Linear and Non-linear Approximations
•Compression and Denoising

22. Linear Approximation

23. Linear Approximation

24. Linear Approximation

25. Non-Linear Approximation

26. Non-Linear Approximation
Coe cients

27. Non-Linear Approximation
Coe cients

28. Efficiency of Transforms
log(||f fM ||)
Fourier DCT
log(M )
Local DCT Wavelets
Best basis Fastest error decay ||f 2
fM ||

29. Efficient Approximation

30. Efficient Approximation

31. Efficient Approximation

32. Efficient Approximation

33. Efficient Approximation

34. Efficient Approximation

35. Efficient Approximation

36. Overview
•Orthogonal Representations
•Linear and Non-linear Approximations
•Compression and Denoising

37. JPEG-2000 vs. JPEG, 0.2bit/pixel

38. Compression by Transform-coding
forward
f
transform
a[m] = ⇥f, m⇤ R
Image f Zoom on f

39. Compression by Transform-coding
forward
f
transform
a[m] = ⇥f, m⇤ R q[m] Z
bin T
⇥ 2T
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Image f Zoom on f Quantized q[m]

40. Compression by Transform-coding
forward codi
f
transform
a[m] = ⇥f, m⇤ R q[m] Z ng
bin T
⇥ 2T
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Entropic coding: use statistical redundancy (many 0’s).
Image f Zoom on f Quantized q[m]

41. Compression by Transform-coding
forward codi
f
transform
a[m] = ⇥f, m⇤ R q[m] Z n g
bin T
ng
c odi
de
q[m] Z
⇥ 2T
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Entropic coding: use statistical redundancy (many 0’s).
Image f Zoom on f Quantized q[m]

42. Compression by Transform-coding
forward codi
f
transform
a[m] = ⇥f, m⇤ R q[m] Z n g
bin T
ng
c odi
dequantization de
a[m]
˜ q[m] Z
⇥ 2T
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Entropic coding: use statistical redundancy (many 0’s).
⇥
1
Dequantization: a[m] = sign(q[m]) |q[m] +
˜ T
2
Image f Zoom on f Quantized q[m]

43. Compression by Transform-coding
forward codi
f
transform
a[m] = ⇥f, m⇤ R q[m] Z n g
bin T
ng
c odi
backward dequantization de
fR = a[m]
˜ m a[m]
˜ q[m] Z
transform
⇥ 2T
m IT
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Entropic coding: use statistical redundancy (many 0’s).
⇥
1
Dequantization: a[m] = sign(q[m]) |q[m] +
˜ T
2
Image f Zoom on f Quantized q[m] f , R =0.2 bit/pixel

44. Compression by Transform-coding
forward codi
f
transform
a[m] = ⇥f, m⇤ R q[m] Z n g
bin T
ng
c odi
backward dequantization de
fR = a[m]
˜ m a[m]
˜ q[m] Z
transform
⇥ 2T
m IT
|a[m]| 2T T T a[m]
Quantization: q[m] = sign(a[m]) Z
T a[m]
˜
Entropic coding: use statistical redundancy (many 0’s).
⇥
1
Dequantization: a[m] = sign(q[m]) |q[m] +
˜ T
2
Theorem: ||f fM ||2 = O(M ) =⇥ ||f fR ||2 = O(log (R)R )
Image f Zoom on f Quantized q[m] f , R =0.2 bit/pixel

45. Noise in Images

46. Denoising

47. Denoising
N 1
thresh. ˜
f= f, m⇥ m f= f, m⇥ m
m=0 | f, m ⇥|>T

48. Denoising
N 1
thresh. ˜
f= f, m⇥ m f= f, m⇥ m
m=0 | f, m ⇥|>T
Theorem: if ||f0 f0,M ||2 = O(M ), In practice:
˜
E(||f f0 || ) = O(
2 2
+1 ) for T = 2 log(N ) T 3

49. Inverse Problems

50. Inverse Problems

51. Inverse Problems

52. Inverse Problems

53. Inverse Problems

54. Restoration with Sparsity

55. Restoration with Sparsity

56. Restoration with Sparsity

57. Conclusion

58. Conclusion

59. Conclusion