Improved seam carving for stereo image resizing
EURASIP Journal on Wireless Communications and Networking volume 2013, Article number: 116 (2013)
When stereo images are shown in three-dimensional (3D) display devices of different aspect ratios, the resizing algorithm for single image could lead to shape and depth distortion of the stereo image’s main content. This paper aims to propose a novel method for retargeting stereo image pairs without distorting important objects in the scene while still maintaining the consistency between the left and right images. We extended seam carving algorithm to stereo images. The novelty of our method is that important objects are determined by jointly considering the intensities of gradients and visual fusion area. The retargeted stereo pair has a feasible 3D interpretation that is similar to the original one. Our method protected the important content and reduced the visual distortion in each of the images as well as the depth distortion. Experimental results are presented to demonstrate that the proposed method effectively guaranteed the geometric consistency of resized stereo images.
With the development of three-dimensional (3D) display technology, there are many kinds of 3D display devices on the market, from cell phone to IMAX screen. The resolutions of these devices have various aspect ratios. When the same stereo image is displayed on screens with different aspect ratios, the stereo image needs to be resized without distorting the shape and depth of the main content.
Normal image scaling only resizes images uniformly and ignores the distortion of the main content. Cropping can protect the reserved content but loses many pixels of the image periphery. Image resizing methods attempt to adapt the image content to the screen without distorting the main objects in the scene. Seam carving is an effective method for image resizing. It is termed as content-aware resizing. The operator retargets the image to a new size considering the main content. When resizing the single image for aspect ratio change, seam carving gets much better results compared to scaling and cropping. However, when seam carving is applied, the only way to determine seams is to consider the change of intensities around the seam. If the intensities of the main content have fewer changes than those of the background, the carving or inserting seams will go through and distort the main content.
Existing resizing methods are effective for single image. However, these methods do not give enough protection to the important content, and using them for stereo images will distort the depth perception. For stereo images, the resizing operator considers not only the distortion of shape, but also the distortion of depth. Depth perception is derived from the small differences in the location of homologous, or corresponding, points in the image pair incident on the retina of the eyes . A general stereo image is composed of two planar images of the same scene from different viewpoints. The difference of corresponding points in the viewpoints generates disparity in the stereo image. When the left and right eyes respectively view different viewpoints, the observer perceives depth depending on stereo image disparity. Clearly, stereo image resizing should consider the relevancy of stereo image pairs, retargeting both images without distorting the depth perception of the main content.
Two retargeting algorithms have also been adjusted to work on stereo image pairs. In , stereo matching results are fused into a framework in the seam carving to preserve the consistency between the left and right images. Both image retargeting and depth adjustment algorithms were discussed. Similarly, a method for retargeting a pair of stereo images was proposed in . It takes into account the visibility relations between pixels in the image pair, and the geometric consistency was mostly preserved by generalizing seam carving to simultaneously carve a pair of seams in both images while reducing distortion in appearance and depth.
In this paper, we present a stereo image resizing method based on seam carving. The proposed method firstly calculates the disparity map of stereo image pairs and then segments the main content of the image within the scope of human vision which is determined with Panum's fusional area . At last, it resizes the stereo image pairs with seam carving algorithm and protects the main content.
This paper is organized as follows: In Section 2, Panum's fusional area of stereo image is calculated. Section 3 proposes our method for stereo image resizing based on seam carving. Experimental results and analysis are shown in Section 4. Finally, Section 5 concludes this paper.
2. Panum’s fusional area of stereo image
Panum presented that when the two eyes were aimed at one point F, only the objects within the area around F were seen as single fused images. The area was named Panum's fusional area .
For 3D display devices, Panum's fusional area is illustrated in Figure 1. The distances between the edges of Panum's fusional area and the eyes are denoted by zf and zb, respectively. zf and zb can be respectively calculated as
where v is the viewing distance, e is the eye separation, and deg is the angular disparity of Panum's fusional area which was discussed by Krol and van der Grind .
Panum's fusional area in front and behind the screen are denoted by Pf and Pb, respectively. Then we have
Panum's fusional area of 3D display devices is expressed as [Pb, Pf]. Because the perceived depth depends on the disparity of the stereo image , [Pb, Pf] is converted into [Db, Df]. The minimum and the maximum disparity of Panum's fusional area of the stereo image are defined by Db and Df, respectively.
Many stereo images are obtained using parallel stereo cameras. If these stereo image pairs are not horizontally moved in changing the disparity, the perceived depth will only be in front of the screen . For this case, Agarwal and Blake have discussed Panum's fusional area in .
3. The proposed method
The resized stereo image for our method is a pair of rectified images IL and IR. The proposed method retargets IL and IR into a new size. The new stereo image is composed of paired images I′L and I′R.
In this paper, we described the method focusing on reducing the stereo image width. Carving and inserting are reciprocal, and horizontal resizing is similar as vertical resizing.
The proposed method resized the stereo image as follows:
Calculate the disparity map (D) of the stereo image, then segment the main content base on D and Panum's fusional area of the stereo image, [D b, D f].
Calculate and select the seam of the left image (S L), then pick the seam of the right image (S R) with the disparity, D. Respectively carve S L and S R from I L and I R.
Repeat step 2 according to the image width which needs to be reduced.
3.1. Main content segmentation
According to whether the disparity of corresponding objects is in Panum's fusional area ([Db, Df]), the image was segmented into the main content and background.
The disparity map (D) of the stereo image is calculated by the belief propagation (BP) algorithm . We consider the disparity map with respect to IL, which is taken to be the reference image. The disparity map (D) is shown in Figure 2.
The BP algorithm we used for calculating the disparity map was based on belief propagation and mean shift segmentation . The disparity map and the reference image (IL) are segmented into some objects. The objects and the average disparity of these objects are denoted by and , respectively, i = 1, 2,…, m. If is in [Db, Df], is regarded as the main content, . If is not in [Db, Df], is regarded as the background, . That is,
Figure 3 depicts the result of the main content segmentation. The main objects are reserved, and the background is removed from the left image.
3.2. Seam selection and carving
A seam is an optimal eight-connected path of pixels on a single image from top to bottom (vertical) and consisted of one and only one pixel in each row, which guarantees that the image keeps a rectangle when the seams are removed. In , an energy function defines the cost of a seam. The optimal seam S* which minimizes this seam cost is selected:
where eHoG is the energy with Histogram of Gradients, which is defined as follows:
In our method, the seam is selected by both energy function and main content protection. Main content protection lets the seam bypass the main content without hurting the proportion.
Let SL denote the seam in IL and SR denote the seam in IR. SL and SR are correlative by the disparity map D. The disparity map shows the correspondence of each pixel in the left and right images. So SR can be obtained by SL and D.
SL is computed by the energy function. The energy function based on the gradient was used to select the energy which the pixels in the seams have, which is called the backward energy at first. The minimum energy tried to minimize the artifacts introduced in the generated image.
If it does not cross the main content, SL and its correlative seam SR will be removed for retargeting the stereo image.
If SL crosses the main content, its crossing part will be replaced. Let Scross be the parts that cross the main content and Snot cross be the parts that do not cross the object of the main content. Let SE denote the new part of the seam used to replace Scross. SE is selected beside the edge of the object; the vertical height of SE and Scross is the same, and the start points and end points of SE and Scross have the same ordinate value. That is, assuming that Scross starts from point a(i1, j1) to point b(i2, j2) to end, SE will start from point c(k1, j1) to point d(k2, j2). Let S′L denote the new seam that consists of SE and Snot cross. By S′L and D, the new seam of the right image (S′R) is obtained. Figure 4 shows the coupled seams S′L and S′R. S′L and S′R are not necessarily continuous:
Since the left and right images are captured from different views, some pixels around objects are occluded . Therefore, removing S′L and S′R takes a little effect on depth perception.
The energy function calculates the changing intensities of gradients. When the change of aspect ratio is large, many seams need to be removed from the stereo image. In this scenario, some seams may intensively appear around the objects with smooth gradients and lead to cracks in the resized stereo image. In order to prevent this condition, when n seams need to be removed, we calculate 2n − 1 seams at once and alternately remove n seams from the 2n − 1 seams.
For some stereo images with large main content, the excessive main content protection will cause overflow or seriously distort the global structure of stereo images. To address this problem, we set a ratio based on the image content. If the reduced width is more than the ratio of the image, the seams of the left image will be defined only by the energy function, without considering main content protection.
4. Experimental results and analysis
We tested our method on standard stereo images obtained from middlebury stereo datasets. We reduced by 20% the width of tsukuba and Reindeer by our method and single image seam carving, respectively. Figures 5 and 6 compare our results and single image seam carving results.
As can be observed from Figures 5 and 6, the main content is not impacted by resizing using our proposed method. The depth perception of important content is preserved well. It is also clearly seen that a significant depth distortion is caused when naive independent retargeting of each image is considered.
Since independently retargeting each image of the stereo image pair will distort the geometric structure of the main content, in this paper, we proposed a new seam carving method to work on stereo images without distorting the depth perception of the main content. We improved the seam carving to work on a pair of stereo images and proved that the proposed method is guaranteed to give a geometrically consistent result of main content. Our method takes advantage of both appearance and perceived depth and can deal with stereo images that are difficult to process with single image seam carving. The experimental results showed that our method had a satisfactory result on stereo images.
Avidan S, Shamir A: Seam carving for content-aware image resizing. ACM Transactions on Graphics (TOG) 2007, 26(3):10. 10.1145/1276377.1276390
Holliman N: 3D Display Systems. Bristol: IOP; 2004.
Utsugi K, Shibahara T, Koike T, Takahashi K, Naemura T: Seam carving for stereo images, in 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). Volume 2010. IEEE: Piscataway; 2010:1-4.
Basha T, Moses Y, Avidan S: Geometrically consistent stereo seam carving, in 2011 IEEE International Conference on Computer Vision (ICCV). IEEE: Piscataway; 2011:1816-1823.
Panum PL: Physiologische Untersuchungen über das Sehen mit zwei Augen. Nabu: Charleston; 2010.
Krol JD, van der Grind WA: Rehabilitation of a classical notion of Panum’s fusional area. Perception 1982, 11(5):615-619. 10.1068/p110615
Son JY, Yeom S, Lee DS, Lee KH, Park MC: A stereoscopic camera model of focal plane detector array. Journal of Display Technology 2011, 7(5):281-288.
Agarwal A, Blake A: Dense stereo matching over the Panum band. IEEE Transactions on Pattern Analysis and Machine Intelligence 2010, 32(3):416-430.
Klaus A, Sormann M, Karner K: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure, in 18th International Conference on Pattern Recognition, 2006 (ICPR 2006). Volume 3. IEEE: Piscataway; 2006:15-18.
Comaniciu D, Meer P: Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002, 24(5):603-619. 10.1109/34.1000236
Geiger D, Ladendorf B, Yuille A: Occlusions and binocular stereo, in Computer Vision—ECCV'92. Berlin: Springer; 1992:425-433.
Scharstein D, Szeliski R: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 2002, 47(1/2/3):7-42.
Scharstein D, Pal C: Learning conditional random fields for stereo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007). Minneapolis; 2007.
The authors are grateful to the anonymous referees for the constructive comments. This study was funded by The National High Technology Research and Development Program of China (2012AA03A301), The National Natural Science Foundation of China (60932007, 61201179), Ph.D. Programs Foundation of the Ministry of Education of China (20110032110029), and Key Projects in the Tianjin Science & Technology Pillar Program (11ZCKFGX02000).
The authors declare that they have no competing interests.
Authors’ original submitted files for images
About this article
Cite this article
Yue, B., Hou, Cp. & Zhou, Y. Improved seam carving for stereo image resizing. J Wireless Com Network 2013, 116 (2013). https://doi.org/10.1186/1687-1499-2013-116