We propose a method for automatically guiding patch-based image completion using mid-level structural cues. Our method first estimates planar projection parameters, softly segments the known region into planes, and discovers translational regularity within these planes. This information is then converted into soft constraints for the low-level completion algorithm by defining prior probabilities for patch offsets and transformations. Our method handles multiple planes, and in the absence of any detected planes falls back to a baseline fronto-parallel image completion algorithm. We validate our technique through extensive comparisons with state-of-the-art algorithms on a variety of scenes. Project page: https://sites.google.com/site/jbhuang0604/publications/struct_completion

1. Image Completion using
Planar Structure Guidance
Jia-Bin Huang Johannes KopfNarendra AhujaSing Bing Kang

2. The image completion problem
Credit: ©Flickr user
remonrijper

3. The image completion problem
Credit: ©Flickr user
remonrijper

4. The image completion problem
Credit: ©Flickr user
remonrijper

5. Large Hole
Small Hole
Non-stationaryStationary

6. Large Hole
Small Hole
Non-stationary
Texture Synthesis
Stationary

7. Texture Synthesis
Large Hole
Small Hole
Non-stationary
Image Inpainting
Stationary

8. Texture Synthesis
Large Hole
Small Hole
Non-stationary
Image Completion
Stationary

9. Input Image Credit: ©Flickr user
remonrijper

10. Region to fill

11. Photoshop Content Aware Fill
[Wexler et al. 2007] [Barnes et al. 2009]

12. Statistics of Patch Offsets
[He and Sun ECCV 2012]
Translational patches
are not sufficient!

13. Searching patch transformation
space is HARD!
Image Melding
[Darabi et al. 2012]

14. Our Result We use mid-level
information

15. • Based on non-parametric framework of [Wexler et al. 2007]

16. • Based on non-parametric framework of [Wexler et al. 2007]
• The basic form

17. • Based on non-parametric framework of [Wexler et al. 2007]
• The basic form
• Patch matching cost
• PatchMatch [Barnes et al. 2009]
for efficient nearest neighbor
field search.

18. • Low-level processing is not sufficient

19. • Low-level processing is not sufficient
• Idea: guide the completion using mid-level information!

20. • Low-level processing is not sufficient
• Idea: guide the completion using mid-level information!
• Transformation of source patches

21. • Low-level processing is not sufficient
• Idea: guide the completion using mid-level information!
• Transformation of source patches
• Translational
(Photoshop CAF)

22. • Low-level processing is not sufficient
• Idea: guide the completion using mid-level information!
• Transformation of source patches
• Translational
(Photoshop CAF)
• Rotation and scale
(Image Melding)

23. • Low-level processing is not sufficient
• Idea: guide the completion using mid-level information!
• Transformation of source patches
• Translational
(Photoshop CAF)
• Rotation and scale
(Image Melding)
• Projective transform
(Ours)

24. Mid-Level cues for guidance
•Planes:
Cues for how to deform patches
•Regularity:
Cues for where to sample patches

25. Mid-level cues – planes
Credit: ©Flickr user
micromegas

26. Plane detection – vanishing point detection

27. Plane detection – vanishing point detection

28. Plane detection – localization
Support lines for VPs

29. Plane detection – localization
Support lines for VPs Support line density

30. Plane detection – localization
⊙
Support lines for VPs Support line density Plane location density

31. Plane detection – localization
⊙
⊙
Support lines for VPs Support line density Plane location density

32. Plane detection – localization
⊙
⊙
⊙
Support lines for VPs Support line density Plane location density

33. Plane detection – posterior probability

34. Our Result

35. Photoshop Content Aware Fill Statistics of Patch Offsets
Image Melding Our Result

36. Photoshop Content Aware Fill Statistics of Patch Offsets
Image Melding Our Result

37. Input and Hole Credit: ©Flickr user
theenmoy

38. Our Result (with planar guidance only)
Need regularity
guidance!

39. Extracting mid-level cues - regularity detection

40. Regularity detection – feature detection and matching

41. Regularity detection – mode detection
• For fronto-parallel scenes, similar to [He and Sun ECCV 2012]
• Can handle perspectively distorted planes and multiple surfaces

42. Regularity detection without rectification

43. Plane and regularityWith planar guidance only

44. Guided sampling and propagation
• Slight modification of PatchMatch algorithm
• Plane guided sampling and propagation
• Draw a plane index based on posterior probability
• Draw random samples using rejection-based
sampling

45. Guided sampling and propagation
• Slight modification of PatchMatch algorithm
• Plane guided sampling and propagation
• Draw a plane index based on posterior probability
• Draw random samples using rejection-based
sampling
• Regularity guided sampling
• Randomly draw a offset vector in rectified space

46. Results

47. Input and Hole Credit: ©Flickr user
danielfoster

48. Photoshop Content Aware Fill Statistics of Patch Offsets
Image Melding Our Result

49. Input and hole
Credit: ©Flickr user
the-o

50. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result
[Wexler et al. 2007]
[Barnes et al. 2009]
[He and Sun 2012] [Darabi et al. 2012]

51. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result
[Wexler et al. 2007]
[Barnes et al. 2009]
[He and Sun 2012] [Darabi et al. 2012]

52. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result
[Wexler et al. 2007]
[Barnes et al. 2009]
[He and Sun 2012] [Darabi et al. 2012]

53. Input and Hole
Credit: ©Flickr user
chrisschoenbohm

54. Photoshop Content Aware Fill Statistics of Patch Offsets
Image Melding Our Result

55. Input and Hole
Credit: ©Flickr user
99667320@N06

56. Photoshop Content Aware Fill Statistics of Patch Offsets
Image Melding Our Result

57. Input and hole
Credit: ©Flickr user
sunshinepictures

58. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result
[Wexler et al. 2007]
[Barnes et al. 2009]
[He and Sun 2012] [Darabi et al. 2012]

59. What if there are no structures?
• Reduces to baseline image completion algorithms which
use translational patches (same as Photoshop content-
aware fill)

60. What if there are no structures?
• Reduces to baseline image completion algorithms which
use translational patches (same as Photoshop content-
aware fill)
• Works well with man-made scenes
• Backward-compatible with natural scenes
• Tested on 25 natural images from [Kopf et al. 2012]

61. Input and Mask Completion

62. Input and Mask Completion

63. Input and Mask Completion

64. Check out our project website!
• http://bit.ly/planarcompletion
• Full comparison of 110 images
• 85 urban scenes + 25 natural
• Code will be released
soon!

65. Failure modes
• Difficult to find correct demarcation lines (boundary between
regions)
• Need higher-level information
• Missed, false, or inaccurate detection of plane parameters or
regularity
• Computer vision algorithms are not perfect

66. Input
Credit: ©Flickr user
loungerie

67. Mask

68. Our result

69. Photoshop
Priority BPImage Melding
Statistics of Patch Offset

70. Input
Credit: ©Flickr user
addictive_picasso

71. Mask

72. Our result

73. Photoshop CAF
[Wexler et al. 2007]
[Barnes et al. 2009]
Image Melding
[Darabi et al. 2012]
Statistics of Patch Offset
[He and Sun 2012]

74. Conclusions
• Mid-level constraints for the win!

75. Conclusions
• Mid-level constraints for the win!
• Our work: planes and regularity

76. Conclusions
• Mid-level constraints for the win!
• Our work: planes and regularity
• Great for man-made scenes

77. Conclusions
• Mid-level constraints for the win!
• Our work: planes and regularity
• Great for man-made scenes
• Backward-compatible with natural scenes

78. Thank You!