To get rid of (an) object(s) at a picture or to restore a picture from scratches or holes, Criminisi at el. suggested an algorithm which is combied "texture synthesis" and "inpainting". I made the slide to present at a class to introduce this algorithm. I refered a slide http://bit.ly/1Ng7DNt. I wish this slide may help you to understand the algorithm. Thank you.

1. Region Filling and Object Removal by
Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., and Toyama, K. (2004)
Lee, Woonghee
M.S. student at the Big Data Mining Lab.
Department of computer science and engineering at the Hanyang University
October 4th, 2015

2. Contents
 Prerequisite
 Key Observations
 Region Filling Algorithm
 Results and Comparisons

3. Prerequisite
 Texture synthesis
 Inpainting

4. Prerequisite
To fill large image repetitively 2-D texture synthesis
 inpainting
Exmaples from Ashikimin [1]

5. Prerequisite
Properties
• Cheap cost to generate image
• Effectively generating image
• Difficult to fill holes in photos
• A complex product of mutual
influences between different
boundaries
 texture synthesis
 inpainting

6. Prerequisite
To fill holes in images by
propagating linear structures
(called isophote)
 texture synthesis
 inpainting

7. Prerequisite
To fill holes in images by
propagating linear structures
(called isophote)
Depends on Gestalt Law of
Continuation
 texture synthesis
 inpainting

8. Prerequisite
Gestalt Law of Continuation texture synthesis
 inpainting

9. Prerequisite
Gestalt Law of Continuation texture synthesis
 inpainting

10. Prerequisite
Gestalt Law of Continuation
Human perceives a dotted line as
a full line by implicit continuation.
 texture synthesis
 inpainting

11. Prerequisite
Propagation direction texture synthesis
 inpainting
propagate along isophotes

12. Prerequisite
Properties
• Effective to fill speckles,
scratches, and overlaid text
• Causes noticeable blur to fill
large regions
• Extremely slow (83’-158’ on a
384 X 256 image)
 texture synthesis
 inpainting

13. Main Idea
To combine the advantages of
“texture synthesis” and “inpainting”

14. Key Observations
A. Exemplar-Based Synthesis Suffices
Algorithm Core: Isophote-driven image-
sampling process

15. Key Observations
A. Exemplar-Based Synthesis Suffices

16. Key Observations
B. Filling Order is Critical
artefacts

17. Key Observations
B. Filling Order is Critical
Onion peel(concentric-layer odering) causes
“over shooting” → To achieve balancing
between the structured regions and texture
regions.

18. Region Filling Algorithm
1) Computing Patch Priorities
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =
Σ 𝑞∈Ψ 𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ 𝑝|
𝐷 𝑝 =
|∇𝐼 𝑝
⊥∙𝑛 𝑝|
𝛼
Initialization:
𝐶 𝑝 = 0, ∀ 𝑝∈ Ω and 𝐶 𝑝 = 1, ∀ 𝑝∈ 𝐼 − Ω

19. Region Filling Algorithm
1) Computing Patch Priorities
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =
Σ 𝑞∈Ψ 𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ 𝑝|
𝐷 𝑝 =
|∇𝐼 𝑝
⊥∙𝑛 𝑝|
𝛼
Initialization:
𝐶 𝑝 = 0, ∀ 𝑝∈ Ω and 𝐶 𝑝 = 1, ∀ 𝑝∈ 𝐼 − Ω
higher priority
lower priority

20. Region Filling Algorithm
1) Computing Patch Priorities
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =
Σ 𝑞∈Ψ 𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ 𝑝|
𝐷 𝑝 =
|∇𝐼 𝑝
⊥∙𝑛 𝑝|
𝛼
Initialization:
𝐶 𝑝 = 0, ∀ 𝑝∈ Ω and 𝐶 𝑝 = 1, ∀ 𝑝∈ 𝐼 − Ω
similar priority

21. Region Filling Algorithm
2) Propagating Texture and Structure Information
After computing priorities, setting the highest
priority Ψ 𝑝
To avoid diffusion, propagating image texture
from the source region
Ψ 𝑞 = arg 𝑚𝑖𝑛Ψ 𝑞∈Φ 𝑑(Ψ 𝑝, Ψ𝑞)

22. Region Filling Algorithm
3) Updating Confidence Values
After filling the patch Ψ 𝑝, the confidence term
is updated
𝐶 𝑝 = 𝐶 𝑝 , ∀ 𝑝∈ Ψ 𝑝 ∩ Ω
It does not require additional parameter to
specify image.

23. Region Filling Algorithm
The 𝑡 indicates the current iteration.

24. Region Filling Algorithm
Properties of the region filling algorithm
Recall 𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
The priority equation achieves balance of
effects and an organic synthesis

25. Region Filling Algorithm
Properties of the region filling algorithm
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
• avoids an arbitrary fill order.
• eliminates the risk of “broken-structure”
artefacts.
• propagates strong edges.
• reduces blocky and misalignment artefacts
without additional step.

26. Region Filling Algorithm
Implementation Details
The target 𝛿Ω is manually selected.
The normal direction 𝑛 𝑝 is computed as
1) Contour’s “control” points are filtered via
2D Gaussian kernel
2) estimated as the orthogonal unit vector of
𝛿Ω

27. Region Filling Algorithm
Implementation Details
The gradient ∇𝐼 𝑝is computed as the MAX value
in Ψ𝑝 ∩ 𝐼
Pixels are classified as belonging to
• The target region Ω
• The source region
• The remainder

28. Results and Comparisons
Experimental environment was a 2.5-GHz
Pentium IV with 1GB of RAM.
To compare with the results of earlier work.

29. Results and Comparisons
Kanizsa Triangle and the Connectivity Principle

30. Results and Comparisons
Comparing Different Filling Orders
original image target region
raster-scan concentric
Harrison’s
2 m 45 s
Ours
5 s

31. Results and Comparisons
Comparing Different Filling Orders
original image target region

32. Results and Comparisons
Comparing Different Filling Orders
raster-scan concentric

33. Results and Comparisons
Comparing Different Filling Orders
Harrison’s
45 m
Ours
2 s

34. Results and Comparisons
Comparing Different Filling Orders
Using only data term leads
the “over shoot”

35. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
original image target region

36. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
onion peel ours

37. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
onion peel ours

38. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
onion peel ours

39. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
onion peel ours

40. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
priority function for before image
Priority function is 0 for inside and 1 for outside Final priorities made the continuation of the pole

41. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
original image target region

42. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
Isophotes hits the thin boundary

43. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
ours traditional image inpainting (blurry)

44. Results and Comparisons
Comparisons With Diffusion-Based Inpainting
target region
texture and structure inpainting (blurry) ours

45. Comparison
With Drori et al.
Results and Comparisons
Drori et al. (blurry)

46. Examples on Photographs
Results and Comparisons
Drori et al. (blurry)

47. Examples on Photographs
Results and Comparisons
Drori et al. (blurry)
onion peel ours

48. Examples on Photographs
Results and Comparisons
ours
“bow-tie” effect

49. Thank you for listening