Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Examplar-based inpainting

1 947 vues

Publié le

This presentation deals with examplar-based inpainting. It is based on the following papers:
(i) C. Guillemot and O. Le Meur, Image inpainting: overview and recent advances, IEEE Signal Processing Magazine, Vol. 1, pp. 127-144, 2014.
(ii) O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchical super-resolution-based inpainting, IEEE TIP, vol. 22(10), pp. 3779-3790, 2013.
(iii) O. Le Meur and C. Guillemot, Super-resolution-based inpainting, ECCV 2012.

Publié dans : Ingénierie
  • Soyez le premier à commenter

Examplar-based inpainting

  1. 1. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Examplar-based inpainting Olivier Le Meur olemeur@irisa.fr IRISA - University of Rennes 1 June 19, 2014 1 / 44
  2. 2. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Inpainting: context and issues (1/3) This talk is about inpainting. We will heavily rely upon these papers: C. Guillemot & O. Le Meur, Image inpainting: overview and recent advances, IEEE Signal Processing Magazine, Vol. 1, pp. 127-144, 2014. O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchical super-resolution-based inpainting, IEEE TIP, vol. 22(10), pp. 3779-3790, 2013. O. Le Meur & C. Guillemot, Super-resolution-based inpainting, ECCV 2012. 2 / 44
  3. 3. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Inpainting: context and issues (2/3) Inpainting Inpainting corresponds to filling holes (i.e. missing areas) in im- ages (Bertalmio et al., 2000). Let be an image I defined as I : Ω ⊂ Rn −→ Rm Let be a degradation operator M M : Ω −→ {0, 1} M(x) = 0, if x ∈ U 1, otherwise Let F the observed image: F = M ◦ I n = 2 for a 2D image m = 3 for (R,G,B) image Ω = S ∪ U, • S being the known part of I • U the unknown part of I ◦ is the Hadamard product 3 / 44
  4. 4. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Inpainting: context and issues (3/3) Different configurations according to the definition of M: Original image 80% of the pixels have been removed. damaged portions in black, scratches object removal Sparsity and low-rank methods Diffusion-based methods Examplar-based methods 4 / 44
  5. 5. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting 3 Variants of Criminisi’s method 4 Super-resolution-based inpainting method 5 Results and comparison with existing methods 6 Conclusion 5 / 44
  6. 6. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting Presentation Notation Criminisi et al.’s method 3 Variants of Criminisi’s method 4 Super-resolution-based inpainting method 5 Results and comparison with existing methods 6 Conclusion 6 / 44
  7. 7. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Examplar-based inpainting (1/4) Texture synthesis Examplar-based inpainting methods rely on the assumption that the known part of the image provides a good dictionary which could be used efficiently to restore the unknown part (Efros and Leung, 1999). The recovered texture is therefore learned from similar regions. ª This can be done simply by sampling, copying or combining patches from the known part of the image; Template Matching ª Patches are then stitched together to fill in the missing area. 7 / 44
  8. 8. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Examplar-based inpainting (2/4) Notations: ª a patch ψpx is a discretized N × N neighborhood centered on the pixel px. This patch can be vectorized in a raster-scan order as a mN2 -dimensional vector; ª ψuk px denotes the unknown pixels of the patch; ª ψk px denotes its known pixels; ª ψpx(i) denotes the ith nearest neighbour of ψpx ; ª δU is the front line; 8 / 44
  9. 9. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Examplar-based inpainting (3/4) Criminisi et al.’s algorithm Criminisi et al. (Criminisi et al., 2004) has brought a new momentum to inpainting applications and methods. They proposed a new method based on two sequential stages: 1 Filling order computation; 2 Texture synthesis. 1 Filling order computation: P(px) = C(px) × D(px) Confidence term C(px) = q∈ψk px C(q) |ψpx | where |ψpx | is the area of ψpx . Data term D(px) = | I⊥ (px) · npx | α where α is a normalization constant in order to ensure that D(px) is in the range 0 to 1. 9 / 44
  10. 10. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Examplar-based inpainting (4/4) 2 Texture synthesis: A template matching is performed within a local neighborhood: py = arg min q∈W d(ψk pq , ψk px∗ ) ª W ⊆ S is the window search; ª ψk px∗ are the known pixels of the patch ψpx∗ with the highest priority; ª ψk py are the known pixels of the nearest patch neighbor; ª d(a, b) is the sum of squared differences between patches a and b. The pixels of the patch ψuk py are then copied into the unknown pixels of the patch ψpx∗ . 10 / 44
  11. 11. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting 3 Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations 4 Super-resolution-based inpainting method 5 Results and comparison with existing methods 6 Conclusion 11 / 44
  12. 12. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Filling order computation (1/4) P(px) = C(px) × D(px) Two variants are here presented: ª Tensor-based data term (Le Meur et al., 2011); ª Sparsity-based data term (Xu and Sun, 2010). Many others: edge-based data term, transformation of the data term in a nonlinear fashion, entropy-based data term... 12 / 44
  13. 13. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Filling order computation (2/4) Tensor-based data term Instead of using the gradient, (Le Meur et al., 2011) used the structure tensor which is more robust: D(px) = α + (1 − α)exp − η (λ1 − λ2)2 where η is a positive value and α ∈ [0, 1]. The structure tensor is a symmetric, positive semi-definite matrix (Weickert, 1999): Jρ,σ [I] = Kρ ∗ m i=1 (Ii ∗ Kσ) (Ii ∗ Kσ)T where Ka is a Gaussian kernel with a standard deviation a. The parameters ρ and σ are called integration scale and noise scale, respectively. 13 / 44
  14. 14. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Filling order computation (3/4) D(px) = α + (1 − α)exp − η (λ1 − λ2)2 When λ1 λ2, the data term tends to α. It tends to 1 when λ1 >> λ2. 14 / 44
  15. 15. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Filling order computation (4/4) Sparsity-based data term Sparsity-based data term (Xu and Sun, 2010) is based on the sparse- ness of nonzero patch similarities: D(px) = |Ns(px)| |N(px)| × pj ∈Ws w2 px ,pj where Ns and N are the numbers of valid and candidate patches in the search window. Weight wpx ,pj is proportional to the similarity between the two patches centered on px and pj ( j wpx ,pj = 1). A large value of the structure sparsity term means sparse similarity with neighboring patches ⇒ a good confidence that the input patch is on some structure. 15 / 44
  16. 16. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Texture synthesis (1/4) Texture synthesis with more than one candidate From K patches ψpx(i) which are the most similar to the known part ψk px of the input patch, the unknown part of the patch to be filled ψuk px is then obtained by a linear combination of the sub-patches ψuk px(i) . ψuk px = K i=1 wiψuk px(i) How can we compute the weights wi of this linear combination? Note: K is locally adjusted by using an -ball including patches within a certain radius. 16 / 44
  17. 17. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Texture synthesis (2/4) ψuk px = K i=1 wiψuk px(i) Different solutions exist (Guillemot et al., 2013): ª Average template matching: wi = 1 K , ∀i; ª Non-local means approach (Buades et al., 2005): wi = exp − d(ψpk x , ψpk x(i) ) h2 ª Least-square method minimizing E(w) = ψk px − Aw 2 2,a w∗ = arg min w E(w) 17 / 44
  18. 18. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Texture synthesis (3/4) ψuk px = K i=1 wiψuk px(i) ª Constrained Least-square optimization with the sum-to-one constraint of the weight vector ⇒ LLE method (Saul and Roweis, 2003) E(w) = ψk px − Aw 2 2,a w∗ = arg min w E(w) s.t. wT 1K = 1 ª Constrained Least-square optimization with positive weights ⇒ NMF method (Lee and Seung, 2001) w∗ = arg min w E(w) s.t. wi ≥ 0 18 / 44
  19. 19. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Texture synthesis (4/4) Similarity metrics: ª Using a Gaussian weighted Euclidean distance dL2 (ψpx , ψpy ) = ψpx − ψpy 2 2,a where a controls the decay of the Gaussian function g(k) = e− |k| 2a2 , a > 0; ª A better distance introduced in (Bugeau et al., 2010, Le Meur and Guillemot, 2012): d(ψpx , ψpy ) = dL2 (ψpx , ψpy ) × (1 + dH (ψpx , ψpy )) where dH (ψpx , ψpy ) is the Hellinger distance dH (ψpx , ψpy ) = 1 − k p1(k)p2(k) where p1 and p2 represent the histograms of patches ψpx , ψpy , respectively. 19 / 44
  20. 20. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Some Examples (1/2) Inpainted pictures with (Criminisi et al., 2004)’s method (Courtesy of P. P´erez): 20 / 44
  21. 21. Results from (Le Meur et al., 2011).
  22. 22. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Limitations Very sensitive to the parameter settings such as the filling order and the patch size: Examplar-based methods are a one-pass greedy algorithms. 22 / 44
  23. 23. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting 3 Variants of Criminisi’s method 4 Super-resolution-based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution 5 Results and comparison with existing methods 6 Conclusion 23 / 44
  24. 24. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Proposed approach (1/1) Objectives of the proposed method We apply an examplar-based inpainting algorithm several times and fuse together the inpainted results. less sensitive to the inpainting setting; relax the greedy constraint. The inpainting method is applied on a coarse version of the input picture: less demanding of computational resources; less sensitive to noise; K candidates for the texture synthesis without introducing blur. Need to fuse the inpainted images and to retrieve the highest frequencies Loopy Belief Propagation and Super-Resolution algorithms. 24 / 44
  25. 25. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution More than one inpainting (1/1) The baseline algorithm is an examplar-based method: ª Filling order computation; ª Texture synthesis. ª Decimation factor n = 3 ª 13 sets of parameters Table: Thirteen inpainting configurations. Setting Parameters 1 Patch’s size 5 × 5 Decimation factor n = 3 Search window 80 × 80 Sparsity-based filling order 2 default + rotation by 180 degrees 3 default + patch’s size 7 × 7 4 default + rotation by 180 degrees + patch’s size 7 × 7 5 default + patch’s size 11 × 11 6 default + rotation by 180 degrees + patch’s size 11 × 11 7 default + patch’s size 9 × 9 8 default + rotation by 180 degrees + patch’s size 9 × 9 9 default + patch’s size 9 × 9 + Tensor-based filling order 10 default + patch’s size 7 × 7 + Tensor-based filling order 11 default + patch’s size 5 × 5 + Tensor-based filling order 12 default + patch’s size 11 × 11 + Tensor-based filling order 13 default + rotation by 180 degrees + patch’s size 9 × 9 + Tensor-based filling order 25 / 44
  26. 26. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Loopy Belief Propagation (1/5) . . . . . . Loopy Belief Propagation is used to fuse together the 13 inpainted images. Let be a finite set of labels L composed of M = 13 values. E(l) = p∈ν Vd(lp) + λ (n,m)∈N4 Vs(ln, lm) where, lp the label of pixel px, ν represents the pixel in U and N4 is a neighbourhood system. λ is a weighting factor. 26 / 44
  27. 27. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Loopy Belief Propagation (2/5) E(l) = p∈ν Vd(lp) + λ (n,m)∈N4 Vs(ln, lm) ª Vd(lp) represents the cost of assigning a label lp to a pixel px: Vd(lp) = n∈L u∈υ I(l) (x + u) − I(n) (x + u) 2 ª Vs(ln, lm) is the discontinuity cost: Vs(ln, lm) = (ln − lm) 2 The minimization is performed iteratively (less than 15 iterations) (Boykov and Kolmogorov, 2004, Boykov et al., 2001, Yedidia et al., 2005). 27 / 44
  28. 28. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Loopy Belief Propagation (3/5) LBP convergence: ª 13 inpainted image in input; ª 25 iterations; ª resolution=80 × 120. 28 / 44
  29. 29. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Loopy Belief Propagation (4/5) LBP convergence: ª 13 inpainted image in input; ª 25 iterations; ª resolution=120 × 80. 29 / 44
  30. 30. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Loopy Belief Propagation (5/5) LBP convergence: ª 13 inpainted image in input; ª 25 iterations; ª resolution=200 × 135. 30 / 44
  31. 31. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Super-resolution (1/2) For the LR patch corresponding to the HR patch having the highest priority: ª We look for its best neighbour; ª Only the best candidate is kept; ª The corresponding HR patch is simply deduced. ª Its pixel values are then copied into the unknown parts of the current HR patch. 31 / 44
  32. 32. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Super-resolution (2/2) To speed-up the process, we can perform the search: ª within a search window; ª within a dictionary (as illustrated on the right) composed of LR patches with their corresponding HR patches. 32 / 44
  33. 33. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting 3 Variants of Criminisi’s method 4 Super-resolution-based inpainting method 5 Results and comparison with existing methods Results Comparison with existing methods 6 Conclusion 33 / 44
  34. 34. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Results (1/4) 34 / 44
  35. 35. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Results (2/4) 35 / 44
  36. 36. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Results (3/4) 36 / 44
  37. 37. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Results (4/4) 37 / 44
  38. 38. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Comparison with existing methods (1/5) Three methods have been tested: ª [Komodakis] N. Komodakis, and G. Tziritas, Image Completion using Global Optimization. in CVPR 2007 (Komodakis and Tziritas, 2007); ª [Pritch] Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-Map Image Editing. in ICCV 2009 (Pritch et al., 2009); ª [He] K. He and J. Sun, Statistics of Patch Offsets for Image Completion. in ECCV 2012 (He and Sun, 2012). 38 / 44
  39. 39. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Comparison with existing methods (2/5) From left to right: Komodakis, Pritch, He, Ours. 39 / 44
  40. 40. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Comparison with existing methods (3/5) From left to right: Komodakis, Pritch, He, Ours. 40 / 44
  41. 41. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Comparison with existing methods (4/5) Much more results on the link: http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/ indexSoA.html 41 / 44
  42. 42. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Comparison with existing methods (5/5) Limitations and failure cases: From left to right: original, He’s method and proposed one. ª No semantic information are used... ª No objective quality metric. 42 / 44
  43. 43. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Outline of the presentation 1 Inpainting: context and issues 2 Examplar-based inpainting 3 Variants of Criminisi’s method 4 Super-resolution-based inpainting method 5 Results and comparison with existing methods 6 Conclusion 43 / 44
  44. 44. Examplar-based inpainting O. Le Meur Inpainting: context and issues Examplar-based inpainting Presentation Notation Criminisi et al.’s method Variants of Criminisi’s method Filling order computation Texture synthesis Some examples Limitations Super-resolution- based inpainting method Proposed approach More than one inpainting Loopy Belief Propagation Super-resolution Conclusion ª A new framework to perform inpainting of still color pictures: coarse inpainting + super-resolution. Binary file could be downloaded: http://people.irisa.fr/Olivier.Le_Meur/publi/2013_ TIP/index.html ª A natural extension is to deal with video inpainting. A paper dealing with video inpainting under revision in IEEE TIP. 44 / 44
  45. 45. Examplar-based inpainting O. Le Meur References References M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. In SIGGRPAH 2000, 2000. Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. On PAMI, 26(9):1124–1137, 2004. Y. Boykov, O. Veksler, and R.Zabih. Efficient approximate energy minimization via graph cuts. IEEE Trans. On PAMI, 20(12): 1222–1239, 2001. A. Buades, B. Coll, and J.M. Morel. A non local algorithm for image denoising. In IEEE Computer Vision and Pattern Recognition (CVPR), volume 2, pages 60–65, 2005. A. Bugeau, M. Bertalm´ıo, V. Caselles, and G. Sapiro. A comprehensive framework for image inpainting. IEEE Trans. on Image Processing, 19(10):2634–2644, 2010. A. Criminisi, P. P´erez, and K. Toyama. Region filling and object removal by examplar-based image inpainting. IEEE Trans. On Image Processing, 13:1200–1212, 2004. A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In IEEE Computer Vision and Pattern Recognition (CVPR), pages 1033–1038, 1999. C. Guillemot, M. Turkan, O. Le Meur, and M. Ebdelli. Object removal and loss concealment using neigbor embedding methods. Signal processing: image communication, 28:1405–1419, 2013. K. He and J. Sun. Statistics of patch offsets for image completion. In ECCV, 2012. N. Komodakis and G. Tziritas. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. On Image Processing, 16(11):2649 – 2661, 2007. O. Le Meur and C. Guillemot. Super-resolution-based inpainting. In ECCV, pages 554–567, 2012. O. Le Meur, J. Gautier, and C. Guillemot. Examplar-based inpainting based on local geometry. In ICIP, 2011. D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In In NIPS, pages 556–562. MIT Press, 2001. Y. Pritch, E. Kav-Venaki, and S. Peleg. Shift-map image editing. In ICCV’09, pages 151–158, Kyoto, Sept 2009. L.K. Saul and S.T. Roweis. Think globally, fit locally: Unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research, 4:119–155, 2003. J. Weickert. Coherence-enhancing diffusion filtering. International Journal of Computer Vision, 32:111–127, 1999. Z. Xu and J. Sun. Image inpainting by patch propagation using patch sparsity. IEEE Trans. on Image Processing, 19(5): 1153–1165, 2010. J.S. Yedidia, W.T. Freeman, and Y. Weiss. Constructing free energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51:2282–2312, 2005. 44 / 44

×