3. Blur?
Degrade image/video quality
Shot/record under dim light circumstance
Long exposure!
3 CG Lab., POSTECH
4. Blur
Various kinds of blur
Camera shake (Camera motion blur) Object movements (Object motion blur)
Out of focus (Defocus blur) Others (Wind, vibration, etc.)
4 CG Lab., POSTECH
5. Deblurring?
Remove blur and restore a sharp image
Input blurred image Deblurring result
5 CG Lab., POSTECH
6. Importance
Image/video in our daily lives
But, sometimes a retake is difficult
Strong demand for high quality
E.g. CCTV, car’s black box, medical imaging
6 CG Lab., POSTECH
7. Contents
Fast Motion Deblurring
Sunghyun Cho and Seungyong Lee (POSTECH)
ACM SIGGRAPH Asia 2009
Text Image Deblurring Using Text-Specific Properties
Hojin Cho (POSTECH), Jue Wang (Adobe), and Seungyong
Lee (POSTECH)
ECCV 2012 (to appear)
7 CG Lab., POSTECH
8. Sunghyun Cho Seungyong Lee
POSTECH
ACM SIGGRAPH Asia 2009
10. Image formation model
Convolution
Motion blur kernel
Trace of a sensor
Blurred image Latent sharp image Blur kernel
* : convolution operator
10 CG Lab., POSTECH
11. Deblurring
Non-blind deconvolution
Ill-posed (Due to the loss of information caused by motion blur)
Blurred image Latent image PSF
Blind deconvolution
Severely ill-posed
Blurred image Latent image PSF
11 CG Lab., POSTECH
12. Blind deconvolution
Possible solutions
Severely ill-posed problem
No unique solution
Blurred image
12 CG Lab., POSTECH
13. Related work
Parametric kernels
Ex) 1D linear motion blur
[Yitzhakey et al. 1998], [Rav-Acha and Peleg 2005], [Cho et
al. 2007], [Money and Kang 2008], …
Blurred image Latent sharp image PSF
13 CG Lab., POSTECH
14. Related work
More complex motion blur
[Fergus et al. 2006], [Jia 2007], [Shan et al. 2008]
Excessive amount of computation
Blurred image Latent sharp image PSF
14 CG Lab., POSTECH
15. Motivation
Computation time Important for practical
purpose
Previous methods are slow
[Fergus et al. 2006] took 1 hr 25 min.
[Shan et al. 2008] took 4 min 48 sec.
Our method took 5.766 sec. in CPU
and 0.734 sec. using GPU accel.
Image size: 640 x 480
kernel size: 25 x 25
15 CG Lab., POSTECH
16. Contributions
Fast motion deblurring
Only a few sec.
Fast latent image estimation
Fast blur kernel estimation
40x ~ 60x faster than [Shan et al. 2008]
GPU acceleration
600x ~ 800x faster than [Shan et al. 2008]
16 CG Lab., POSTECH
17. Motion deblurring: Common framework
Iteratively solve
1. Estimate a PSF
Blurred image Latent image PSF
2. Estimate a latent sharp image using a complex image
prior
Blurred image Latent image PSF
17 CG Lab., POSTECH
18. Motion deblurring: Common framework
Blur model
B L* K N
Blurred image B Latent image L Blur kernel K Noise N
Energy function
f ( L, K ) B L * K q( L) r ( K )
2
* : convolution operator
q(L), r(K) : regularization terms or priors for L, K
18 CG Lab., POSTECH
19. Motion deblurring: Common framework
Blurred image
Latent image estimation Kernel estimation
L ' arg min B L * K q( L)
2
K ' arg min B L * K r ( K )
2
L K
Deblurred result
19 CG Lab., POSTECH
20. Latent image estimation: Blurred image Latent image
estimation
Kernel
estimation
Deblurred result
Analysis of prev. methods
Two important properties
Restoration of strong edges
Inspecting around strong edges,
we can find a blur kernel
Noise suppression in
smooth regions
Avoids the effect of noise Blurry input Latent image
estimation of
on kernel estimation
[Shan et al. 2008]
Edge of a blurred image Edge of a sharp image
20 CG Lab., POSTECH
21. Latent image estimation: Blurred image Latent image
estimation
Kernel
estimation
Deblurred result
Analysis of prev. methods
Two important properties
Restoration of strong edges
Inspecting around strong edges,
we can find a blur kernel
Noise suppression in
smooth regions
Avoids the effect of noise Blurry input Latent image
estimation of
on kernel estimation
[Shan et al. 2008]
21 CG Lab., POSTECH
22. Latent image estimation: Blurred image Latent image
estimation
Kernel
estimation
Deblurred result
Analysis of prev. methods
Two important properties
Restoration of strong edges
Inspecting around strong edges,
we can find a blur kernel
Noise suppression in
smooth regions
Avoids the effect of noise Blurry input Latent image
estimation of
on kernel estimation
[Shan et al. 2008]
Computationally expensive priors for q(L)
L ' arg min B L * K q( L)
2
L
22 CG Lab., POSTECH
24. Latent image estimation: Blurred image Latent image
estimation
Kernel
estimation
Deblurred result
Basic idea for acceleration
Simple Deconvolution Prediction
Current kernel Updated kernel
24 CG Lab., POSTECH
25. Deblurring process
Blurred image
Prediction Kernel estimation Deconvolution Final
deconvolution
Deblurred result
* Deconvolution + prediction = latent image estimation
25 CG Lab., POSTECH
26. Results
Blurry input Our result Blur
kernel
Processing time Processing time
Image size Blur kernel size
(CPU) (GPU)
1024 x 768 49 x 47 18.656 sec. 2.125 sec.
26 26 CG Lab., POSTECH
27. Results
Blurry input Our result Blur kernel
Processing time Processing time
Image size Blur kernel size
(CPU) (GPU)
972 x 966 65 x 93 18.813 sec. 5.766 sec.
27 27 CG Lab., POSTECH
28. Results
Blurry input Our result Blur kernel
Processing time Processing time
Image size Blur kernel size
(CPU) (GPU)
858 x 558 61 x 43 8.969 sec. 0.703 sec.
28 28 CG Lab., POSTECH
29. Comparison (quality)
Blurry input
[Yuan et al. 2007] [Shan et al. 2008] our method
* This method uses two input images.
29 CG Lab., POSTECH
35. Difficulties
Why is handling text difficult?
(1) Different statistics against natural images
(2) Spatial distribution of text is highly regulated
Log-scale Gradient Log-scale Gradient
Natural image Text image
gradient histogram magnitude gradient histogram magnitude
Directly applying a natural image deblurring method to a text image
will result in an erroneous result!
35 CG Lab., POSTECH
36. Our Approach
Exploit desired “general” properties of text images
Introduce a new optimization framework to
incorporate the text properties in deblurring
Utilizing domain-specific properties
Real blurred image Our result
36 CG Lab., POSTECH
37. Desired Properties of Latent Text Images – 1/3
Text boundary
High contrasts along the texts’ boundary
37 CG Lab., POSTECH
38. Desired Properties of Latent Text Images – 2/3
Character color
Near-uniform color inside characters
POSTECH
1) Near-uniform color
2) Gradients are close to zero
38 CG Lab., POSTECH
39. Desired Properties of Latent Text Images – 3/3
Background gradients
Too restrictive to assume a single background color (e.g.,
advertisements or posters)
Similar to the natural image
39 CG Lab., POSTECH
40. Optimization Framework Blur model
b = k* l + n
Deblurring problem
arg min b k * l l (l ) k (k )
2
l ,k
l (l ) : A prior for the latent image
k (k ) : A prior for the motion blur kernel
arg min b k * l l (l ) k (k ) a (a ) l a
2 2
l ,k ,a
a (a) : prior for the auxiliary (reference) image a
a : reflects the text properties
40 CG Lab., POSTECH
41. Deblurring Algorithm
Kernel Estimation
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image b Result image l
• Estimating l
Restore sharp edges along the boundaries of texts
considering the text properties 1 and 2
• Estimating k
Compute the blur kernel (camera’s trajectory)
• Final deconvolution
Restores texts and background using k & b
41 CG Lab., POSTECH
42. Deblurring Algorithm
Kernel Estimation
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image b Result image l
Blur kernel k
Computing l
Blurred image b
Result image l
Computing a
42 CG Lab., POSTECH
43. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
Step 1: Computing l
a reflects the text properties
The text properties soak into l by solving:
arg min b k * l l (l ) l a
2 2
l
a plays a role as a guidance(reference) image
1
(k ) (b) (a)
l
(k ) (k ) (1) l () ()
Can be solved efficiently using FFTs and component-wise operators(multiplication/division)
43 CG Lab., POSTECH
44. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
Step 2: Computing a
Initialize a by refining l computed from Step 1
Reduce ringing artifacts and noise caused by simple deconvolution
through L0 gradient minimization
Bilateral filter Total variation (TV) L0 gradient minimization
The result of L0 gradient minimization can provide a good initial for a
[Property 1] The boundaries of texts have large gradient values
[Property 2] Gradients inside characters are close to zero
Noise and ringing artifacts generally have small magnitudes, thus being reduced
* The figures are provided in [Xu et al., SIGGRAPH Asia 2011] 44 CG Lab., POSTECH
45. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
Step 2: Computing a
With the initial a, we iteratively solve:
arg min a (a) l a
2
a
a (a) : a cost function which is minimized when a satisfies the text image properties
l a
2
: a data term to penalize non-similarity between a and l
0 if ai =aiP
For each pixel at i, a (ai )
d MAX otherwise
aP : an ideal image satisfying the text image properties 1 & 2 completely.
After minimizing the equation,
a will reflect the text image properties while preserving the similarity against l
45 CG Lab., POSTECH
46. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
Computing the ideal image aP:
Detect text regions from l using the stroke width transform (SWT)
Typical stroke Searching in the direction Detected stroke width
of the gradient at edges
For detected text regions (connected strokes), we force the pixels
to have the same color value
Through the SWT and re-coloring, the ideal image satisfies the following properties:
[Property 1] The large gradient around the boundaries of text characters
[Property 2] Colors are near uniform inside text characters
46 CG Lab., POSTECH
47. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
0 if ai =aiP
arg min a (a) l a
2
where a (ai )
a d MAX otherwise
a (a) : a cost function which is minimized when a satisfies the text image properties
l a
2
: a data term to penalize non-similarity between a and l
Solution
a P
if d MAX li aiP
2
For each pixel at i, ai i
*
li
otherwise
47 CG Lab., POSTECH
48. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image
Computing k
Adopted from [Shan et al. SIGGRAPH 2008] and [Cho and
Lee, SIGGRAPH Asia 2009]
( ) *b k * *a k ,
2
E (k )
2
*
*
{ x , y , xx , xy , yy }
The auxiliary image a is used instead of the latent image l
a contains less ringing artifacts and noise due to the L0 gradient
minimization
48 CG Lab., POSTECH
49. Kernel Estimation
Algorithm Details
Estimating l
Input Final Deconvolution
Computing l Computing a Estimating k
Blurred image Result image l
Final deconvolution
Given the estimate kernel and the input blurred image, we
minimize:
(1) (2) (3)
E (l ) b k * l 1 li 2 li ai 3 li
2 2 2 0.8
iT iT iT
T {i | a (ai ) 0} : Set of pixel indices for text regions
(1): [Property 1 & 2] Gradient values inside each character should be close to zero
(2): [Property 1 & 2] Each character has a near-uniform color
(3): [Property 3] Gradient values of background are sparse
49 CG Lab., POSTECH
51. Original image
Synthetic blurred image
Fast motion deblurring result
Our result
51
52. Results on Real Photographs (1/4)
Blurred image
Our result
52
53. Results on Real Photographs (2/4)
Blurred image Our result
53
54. Results on Real Photographs (3/4)
Blurred image Our result
54
55. Results on Real Photographs (4/4)
Blurred image
Our result (very large blur: 105x105) 55
56. Comparison – Real Images
Blurred image Fergus et al. Shan et al. Cho and Lee
SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009
Xu and Jia Levin et al. Chen et al. Our method
ECCV 2010 CVPR 2011 56 CVPR 2011 CG Lab., POSTECH
57. Comparison – Real Images
Blurred image Fergus et al. Shan et al. Cho and Lee
SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009
Xu and Jia Levin et al. Chen et al. Our method
ECCV 2010 CVPR 2011 CVPR 2011
57 CG Lab., POSTECH
58. Comparison – Real Images
Blurred image Fergus et al. Shan et al. Cho and Lee
SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009
Not available
Xu and Jia Levin et al. Chen et al. Our method
ECCV 2010 CVPR 2011 CVPR 2011
* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
59. Comparison – Real Images
Blurred image Fergus et al. Shan et al. Cho and Lee
SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009
Not available
Xu and Jia Levin et al. Chen et al. Our method
ECCV 2010 CVPR 2011 CVPR 2011
* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
60. Comparison – Real Images
Not available
Blurred image Fergus et al. Shan et al. Cho and Lee
SIGGRAPH 2006 SIGGRAPH 2008 SIGGRAPH Asia 2009
Not available Not available
Xu and Jia Levin et al. Chen et al. Our method
ECCV 2010 CVPR 2011 CVPR 2011
* The methods of Shan et al. and Levin et al. could not handle this image.
* Since Chen et al.’s approach requires training, re-implementing it is impossible without having the training data.
61. Conclusion
Previous natural image deblurring methods do not work well
for text images
Due to the lack of consideration of text-specific properties
We analyzed the desired properties for latent text images
We proposed a novel text image deblurring algorithm which
explicitly incorporates the text-specific properties into the
optimization framework
61 CG Lab., POSTECH