Improved seam carving for stereo image resizing
© Yue et al.; licensee Springer. 2013
Received: 9 March 2013
Accepted: 8 April 2013
Published: 26 April 2013
When stereo images are shown in three-dimensional (3D) display devices of different aspect ratios, the resizing algorithm for single image could lead to shape and depth distortion of the stereo image’s main content. This paper aims to propose a novel method for retargeting stereo image pairs without distorting important objects in the scene while still maintaining the consistency between the left and right images. We extended seam carving algorithm to stereo images. The novelty of our method is that important objects are determined by jointly considering the intensities of gradients and visual fusion area. The retargeted stereo pair has a feasible 3D interpretation that is similar to the original one. Our method protected the important content and reduced the visual distortion in each of the images as well as the depth distortion. Experimental results are presented to demonstrate that the proposed method effectively guaranteed the geometric consistency of resized stereo images.
With the development of three-dimensional (3D) display technology, there are many kinds of 3D display devices on the market, from cell phone to IMAX screen. The resolutions of these devices have various aspect ratios. When the same stereo image is displayed on screens with different aspect ratios, the stereo image needs to be resized without distorting the shape and depth of the main content.
Normal image scaling only resizes images uniformly and ignores the distortion of the main content. Cropping can protect the reserved content but loses many pixels of the image periphery. Image resizing methods attempt to adapt the image content to the screen without distorting the main objects in the scene. Seam carving is an effective method for image resizing. It is termed as content-aware resizing. The operator retargets the image to a new size considering the main content. When resizing the single image for aspect ratio change, seam carving gets much better results compared to scaling and cropping. However, when seam carving is applied, the only way to determine seams is to consider the change of intensities around the seam. If the intensities of the main content have fewer changes than those of the background, the carving or inserting seams will go through and distort the main content.
Existing resizing methods are effective for single image. However, these methods do not give enough protection to the important content, and using them for stereo images will distort the depth perception. For stereo images, the resizing operator considers not only the distortion of shape, but also the distortion of depth. Depth perception is derived from the small differences in the location of homologous, or corresponding, points in the image pair incident on the retina of the eyes . A general stereo image is composed of two planar images of the same scene from different viewpoints. The difference of corresponding points in the viewpoints generates disparity in the stereo image. When the left and right eyes respectively view different viewpoints, the observer perceives depth depending on stereo image disparity. Clearly, stereo image resizing should consider the relevancy of stereo image pairs, retargeting both images without distorting the depth perception of the main content.
Two retargeting algorithms have also been adjusted to work on stereo image pairs. In , stereo matching results are fused into a framework in the seam carving to preserve the consistency between the left and right images. Both image retargeting and depth adjustment algorithms were discussed. Similarly, a method for retargeting a pair of stereo images was proposed in . It takes into account the visibility relations between pixels in the image pair, and the geometric consistency was mostly preserved by generalizing seam carving to simultaneously carve a pair of seams in both images while reducing distortion in appearance and depth.
In this paper, we present a stereo image resizing method based on seam carving. The proposed method firstly calculates the disparity map of stereo image pairs and then segments the main content of the image within the scope of human vision which is determined with Panum's fusional area . At last, it resizes the stereo image pairs with seam carving algorithm and protects the main content.
This paper is organized as follows: In Section 2, Panum's fusional area of stereo image is calculated. Section 3 proposes our method for stereo image resizing based on seam carving. Experimental results and analysis are shown in Section 4. Finally, Section 5 concludes this paper.
2. Panum’s fusional area of stereo image
Panum presented that when the two eyes were aimed at one point F, only the objects within the area around F were seen as single fused images. The area was named Panum's fusional area .
Panum's fusional area of 3D display devices is expressed as [Pb, Pf]. Because the perceived depth depends on the disparity of the stereo image , [Pb, Pf] is converted into [Db, Df]. The minimum and the maximum disparity of Panum's fusional area of the stereo image are defined by Db and Df, respectively.
Many stereo images are obtained using parallel stereo cameras. If these stereo image pairs are not horizontally moved in changing the disparity, the perceived depth will only be in front of the screen . For this case, Agarwal and Blake have discussed Panum's fusional area in .
3. The proposed method
The resized stereo image for our method is a pair of rectified images IL and IR. The proposed method retargets IL and IR into a new size. The new stereo image is composed of paired images I′L and I′R.
In this paper, we described the method focusing on reducing the stereo image width. Carving and inserting are reciprocal, and horizontal resizing is similar as vertical resizing.
Calculate the disparity map (D) of the stereo image, then segment the main content base on D and Panum's fusional area of the stereo image, [D b, D f].
Calculate and select the seam of the left image (S L), then pick the seam of the right image (S R) with the disparity, D. Respectively carve S L and S R from I L and I R.
Repeat step 2 according to the image width which needs to be reduced.
3.1. Main content segmentation
According to whether the disparity of corresponding objects is in Panum's fusional area ([Db, Df]), the image was segmented into the main content and background.
3.2. Seam selection and carving
In our method, the seam is selected by both energy function and main content protection. Main content protection lets the seam bypass the main content without hurting the proportion.
Let SL denote the seam in IL and SR denote the seam in IR. SL and SR are correlative by the disparity map D. The disparity map shows the correspondence of each pixel in the left and right images. So SR can be obtained by SL and D.
SL is computed by the energy function. The energy function based on the gradient was used to select the energy which the pixels in the seams have, which is called the backward energy at first. The minimum energy tried to minimize the artifacts introduced in the generated image.
If it does not cross the main content, SL and its correlative seam SR will be removed for retargeting the stereo image.
Since the left and right images are captured from different views, some pixels around objects are occluded . Therefore, removing S′L and S′R takes a little effect on depth perception.
The energy function calculates the changing intensities of gradients. When the change of aspect ratio is large, many seams need to be removed from the stereo image. In this scenario, some seams may intensively appear around the objects with smooth gradients and lead to cracks in the resized stereo image. In order to prevent this condition, when n seams need to be removed, we calculate 2n − 1 seams at once and alternately remove n seams from the 2n − 1 seams.
For some stereo images with large main content, the excessive main content protection will cause overflow or seriously distort the global structure of stereo images. To address this problem, we set a ratio based on the image content. If the reduced width is more than the ratio of the image, the seams of the left image will be defined only by the energy function, without considering main content protection.
4. Experimental results and analysis
As can be observed from Figures 5 and 6, the main content is not impacted by resizing using our proposed method. The depth perception of important content is preserved well. It is also clearly seen that a significant depth distortion is caused when naive independent retargeting of each image is considered.
Since independently retargeting each image of the stereo image pair will distort the geometric structure of the main content, in this paper, we proposed a new seam carving method to work on stereo images without distorting the depth perception of the main content. We improved the seam carving to work on a pair of stereo images and proved that the proposed method is guaranteed to give a geometrically consistent result of main content. Our method takes advantage of both appearance and perceived depth and can deal with stereo images that are difficult to process with single image seam carving. The experimental results showed that our method had a satisfactory result on stereo images.
The authors are grateful to the anonymous referees for the constructive comments. This study was funded by The National High Technology Research and Development Program of China (2012AA03A301), The National Natural Science Foundation of China (60932007, 61201179), Ph.D. Programs Foundation of the Ministry of Education of China (20110032110029), and Key Projects in the Tianjin Science & Technology Pillar Program (11ZCKFGX02000).
- Avidan S, Shamir A: Seam carving for content-aware image resizing. ACM Transactions on Graphics (TOG) 2007, 26(3):10. 10.1145/1276377.1276390View ArticleGoogle Scholar
- Holliman N: 3D Display Systems. Bristol: IOP; 2004.Google Scholar
- Utsugi K, Shibahara T, Koike T, Takahashi K, Naemura T: Seam carving for stereo images, in 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). Volume 2010. IEEE: Piscataway; 2010:1-4.Google Scholar
- Basha T, Moses Y, Avidan S: Geometrically consistent stereo seam carving, in 2011 IEEE International Conference on Computer Vision (ICCV). IEEE: Piscataway; 2011:1816-1823.View ArticleGoogle Scholar
- Panum PL: Physiologische Untersuchungen über das Sehen mit zwei Augen. Nabu: Charleston; 2010.Google Scholar
- Krol JD, van der Grind WA: Rehabilitation of a classical notion of Panum’s fusional area. Perception 1982, 11(5):615-619. 10.1068/p110615View ArticleGoogle Scholar
- Son JY, Yeom S, Lee DS, Lee KH, Park MC: A stereoscopic camera model of focal plane detector array. Journal of Display Technology 2011, 7(5):281-288.View ArticleGoogle Scholar
- Agarwal A, Blake A: Dense stereo matching over the Panum band. IEEE Transactions on Pattern Analysis and Machine Intelligence 2010, 32(3):416-430.View ArticleGoogle Scholar
- Klaus A, Sormann M, Karner K: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure, in 18th International Conference on Pattern Recognition, 2006 (ICPR 2006). Volume 3. IEEE: Piscataway; 2006:15-18.Google Scholar
- Comaniciu D, Meer P: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002, 24(5):603-619. 10.1109/34.1000236View ArticleGoogle Scholar
- Geiger D, Ladendorf B, Yuille A: Occlusions and binocular stereo, in Computer Vision—ECCV'92. Berlin: Springer; 1992:425-433.Google Scholar
- Scharstein D, Szeliski R: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 2002, 47(1/2/3):7-42.View ArticleGoogle Scholar
- Scharstein D, Pal C: Learning conditional random fields for stereo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007). Minneapolis; 2007.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.