study

1. Seam Carving for Content-Aware Image Resizing Shai Aidan (Mitsubishi Electric Research Labs) Ariel Shamir (The Interdisciplinary Center & MERL) ACM SIGGRAPH 2007

2. Resize Seam carving & insertion

3. Abstract Seams are optimal 8-connected paths of pixels cross the image Carving out or inserting seams to achieve content-aware resizing

4. Outline Introduction Background Seam-carving operator Discrete image resizing Multi-size images Limitations Conclusions and future work

5. Introduction

6. Motivation HTML can support dynamic changes of page layout and text. Why can not an image deform to fit different layout automatically ? iGoogle How about aspect ratio of an image , such as fitting photo into PDA or phone cells ? Solution ? Resize – content independent Crop – remove pixels from the image periphery only

7. Basic Idea of Seam-Carving Use energy function to define the importanceof pixels Define seam-carving image operator Image reduction Carving out seams - the connected low energy pixels crossing the image Preserving the image structure Image enlarging Insert seams on low energy area The order of seam insertion ensures a balance between the original image content and the artificially inserted pixels

8. Application Discrete image resizing Aspect Ration Change, Image Retarget, Image Enlarging, Content Amplification, Seam Carving in gradient domain, Object Removal Multi-size images An image can continuously change their size in a content-aware manner Storing the order of seam removal and insertion

9. background

10. Image Retarget Seek to change the size of the image while maintaining the important features Face detector An automatic thumbnail creation [Suh03] ROI Fisheye-View warp [Liu and Gleicher 05, 06] Visual saliency [] [Suh 03] [Selur 04, decompose image to foreground obj and background origin

11. Saliency map [Itti IEEE99] Simulate neuroscience of human visual system Pyramid tech. to compute 3 feature maps, color, intensity and orientation [Suh 03], an automatic thumbnail creation, based on either a saliency map or the output of a face detector [Chen 03], adapting most important region of images to mobile devices.

12. [Liu 03], suggesting to trade time for space. Given a collection of regions of interest, they construct an optimal path through these regions and display them serially.

13. [Santella et al. 06] use eye tracking, in addition to composition rules to crop images intelligently.

14. ROI (Region-Of-Interest) Such a method was proposed by [Liu and Gleicher 05, 06] for image and video retargeting. For image retargeting they find ROI and construct a novel Fisheye-View warp that essentially applies a piecewise linear scaling function in each dimension to the image. This way the ROI is maintained while the rest of the image is warped. The retargeting can be done in interactive rates, once the ROI is found, so the user can control the desired size of the image by moving a slider. In their video retargeting work they use a combination of image and saliency maps to find the ROI. Then they use a combination of cropping, virtual pan and shot cuts to retarget the video frames.

15. Feature-aware warping The first solution to the general problem of warping an image into an arbitrary shape while preserving user-specified features was recently proposed by [Gal et al. 06]. The feature-aware warping is achieved by a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain. Since local constraints are propagated by the global optimization process, not all the constraints can always be satisfied at once

16. Seam Perfect seams to combine parts of a set of photo into a single composite picture [Agarwala et al. 04] Drag-and-Drop Pasting that extends the Poisson Image Editing to computer an optimal boundary (seam) between the source picture and target images [Jia et al. 06] AutoCollage, a program that automatically creates a collage image from a collection of images. [Rother et al. 06] Simultaneously solve matting and compositing. They allow the user to scale the size of the foreground object and paste it back on the original background. [Wang , Cohen 06] evaluated several cost functions for seamless image stitching and concluded that minimizing an L1error norm between the gradients of the stitched image and the gradients of the input images performed well in general [Zomet et al. 05]

17. Sear Optimal Seam Dijkstra’s shortest path algorithm [98] Dynamic programming [Efros 01] Graph cuts [Kwatra 03]

18. Seam-carving operator

19. Strategies of Image Reduction Original e1 energy Optimal global remove the lowest energy pixels Pixel remove the least energy in each row

20. Strategies of Image Reduction Original e1 energy Column removing columns with minimal energy Crop find a sub-win with the highest energy

21. Vertical Seam

22. Horizontal Seam

23. Optimal Seam Search

24. Optimal Seam Search S Dynamic Programming G

25. e1 energy

26. Image Energy Preservation The average energy of all pixels during resizing

27. Energy Functions L1 and L2-norm of the gradient, saliency measure [Itti 99]

29. Energy Functions Histogram of Gradient (HoG) [Dalal and Triggs 95] max(HoG(I(x,y)) makes sure the seams run parallel to the edge of objects and not cross them

30. Energy Functions Entropy Compute the entropy over a 9 x 9 window and add it to e1 eEntropy(x,y) = + e1 (x,y)

31. Energy Functions Segmentation and L1 Image segmentation [Christoudias 02] Apply e1 on the results

32. No single e function performs well across all images Similar range for resizing e1 or eHoG works well

33. Discrete image resizing Aspect Ratio Change, Retargeting with Optimal Seams-Order, Image Enlarging, Content Amplification,

34. Aspect Ratio Change Carving-out /insert seams Original Original Original 1D aspect ratio changing

35. 2D aspect ratio changing Optimal Seams-Order Search + DynamicProgramming = + min

36. Retargeting with Optimal Seams-Order h-first alternate v-first Transport map Original optimal

37. Image Enlarging Find first k seams for removal Duplicate them in order to arrive at I(-k) origin I(t): smaller image after t seam-carving I(-k): enlarged image after k seam insertion t I(-1) I(-k) I(t) I(-k) enlarged image insert seams in order of removal

38. Image Enlarging (＞50%) origin Break into several steps Each step does not enlarge the size of image more than a fraction

39. Content Amplification Amplified Original

40. Seam Carving in the Gradient Domain Seam + Poisson Reconstruction [Perez 03] Compute e function Work on the gradient domain Remove seams from the x and y derivatives of the original image Use Poisson Reconstruction original retarget retarget in Gradient Domain

41. Object Removal Mark the removing target Remove seams until all the marked pixels are gone * Employ seam insertion to maintain the original size

42. Object Removal Origin

43. Multi-size images Store the pre-computed representation that encodes, for each pixel in V/H map The index of the seam that removed it The negative index of the seam that inserted it Blue (first seam)  Red (last seam) origin V(i,j)=t : pixel (i,j) removed by t-th vertical seam H(i,j)=t : pixel (i,j) removed by t-th horizontal seam

44. Limitations Seam-Carving does not work well on all images Ex: face Origin Crop Scale Constraint the face Face the flower Bottom up feature detection

45. Limitations The amount of content Too density, no “less” important area The layout of the image content origin origin

46. Conclusions Present a content-aware resizing using the seam-carving image operator Seams are the optimal paths on a single image Carve-out seams Insert seams Application of seam-carving operator Aspect ratio change, image retargeting, content amplification, object removal Multi-size images that support continuous resizingin real-time

47. Future Work Video resizing Combination of scaling and seam-carving Define more robust multi-size image Better solution to combine horizontal and vertical seams in multi-size image

48. END