Improved seam carving for stereo image resizing

Abstract

When stereo images are shown in three-dimensional (3D) display devices of different aspect ratios, the resizing algorithm for single image could lead to shape and depth distortion of the stereo image’s main content. This paper aims to propose a novel method for retargeting stereo image pairs without distorting important objects in the scene while still maintaining the consistency between the left and right images. We extended seam carving algorithm to stereo images. The novelty of our method is that important objects are determined by jointly considering the intensities of gradients and visual fusion area. The retargeted stereo pair has a feasible 3D interpretation that is similar to the original one. Our method protected the important content and reduced the visual distortion in each of the images as well as the depth distortion. Experimental results are presented to demonstrate that the proposed method effectively guaranteed the geometric consistency of resized stereo images.

1. Introduction

With the development of three-dimensional (3D) display technology, there are many kinds of 3D display devices on the market, from cell phone to IMAX screen. The resolutions of these devices have various aspect ratios. When the same stereo image is displayed on screens with different aspect ratios, the stereo image needs to be resized without distorting the shape and depth of the main content.

Normal image scaling only resizes images uniformly and ignores the distortion of the main content. Cropping can protect the reserved content but loses many pixels of the image periphery. Image resizing methods attempt to adapt the image content to the screen without distorting the main objects in the scene. Seam carving[1] is an effective method for image resizing. It is termed as content-aware resizing. The operator retargets the image to a new size considering the main content. When resizing the single image for aspect ratio change, seam carving gets much better results compared to scaling and cropping. However, when seam carving is applied, the only way to determine seams is to consider the change of intensities around the seam. If the intensities of the main content have fewer changes than those of the background, the carving or inserting seams will go through and distort the main content.

Existing resizing methods are effective for single image. However, these methods do not give enough protection to the important content, and using them for stereo images will distort the depth perception. For stereo images, the resizing operator considers not only the distortion of shape, but also the distortion of depth. Depth perception is derived from the small differences in the location of homologous, or corresponding, points in the image pair incident on the retina of the eyes [2]. A general stereo image is composed of two planar images of the same scene from different viewpoints. The difference of corresponding points in the viewpoints generates disparity in the stereo image. When the left and right eyes respectively view different viewpoints, the observer perceives depth depending on stereo image disparity. Clearly, stereo image resizing should consider the relevancy of stereo image pairs, retargeting both images without distorting the depth perception of the main content.

Two retargeting algorithms have also been adjusted to work on stereo image pairs. In [3], stereo matching results are fused into a framework in the seam carving to preserve the consistency between the left and right images. Both image retargeting and depth adjustment algorithms were discussed. Similarly, a method for retargeting a pair of stereo images was proposed in [4]. It takes into account the visibility relations between pixels in the image pair, and the geometric consistency was mostly preserved by generalizing seam carving to simultaneously carve a pair of seams in both images while reducing distortion in appearance and depth.

In this paper, we present a stereo image resizing method based on seam carving. The proposed method firstly calculates the disparity map of stereo image pairs and then segments the main content of the image within the scope of human vision which is determined with Panum's fusional area [5]. At last, it resizes the stereo image pairs with seam carving algorithm and protects the main content.

This paper is organized as follows: In Section 2, Panum's fusional area of stereo image is calculated. Section 3 proposes our method for stereo image resizing based on seam carving. Experimental results and analysis are shown in Section 4. Finally, Section 5 concludes this paper.

2. Panum’s fusional area of stereo image

Panum presented that when the two eyes were aimed at one point F, only the objects within the area around F were seen as single fused images. The area was named Panum's fusional area [5].

For 3D display devices, Panum's fusional area is illustrated in Figure 1. The distances between the edges of Panum's fusional area and the eyes are denoted by z_f and z_b, respectively. z_f and z_b can be respectively calculated as

$$z_{\mathrm{f}}=\frac{e}{2}\cot \left(\arctan \frac{e}{2v}+\frac{\left|\mathit{deg}\right|}{2}\right),$$

(1)

$$z_{\mathrm{b}}=\frac{e}{2}\cot \left(\arctan \frac{e}{2v}-\frac{\left|\mathit{deg}\right|}{2}\right),$$

(2)

where v is the viewing distance, e is the eye separation, and deg is the angular disparity of Panum's fusional area which was discussed by Krol and van der Grind [6].

Figure 1

Panum's fusional area of 3D screen.

Panum's fusional area in front and behind the screen are denoted by P_f and P_b, respectively. Then we have

$$P_{\mathrm{f}}=v-z_{\mathrm{f}},$$

(3)

$$P^b=v-z^b.$$

(4)

Panum's fusional area of 3D display devices is expressed as [P_b, P_f]. Because the perceived depth depends on the disparity of the stereo image [2], [P_b, P_f] is converted into [D_b, D_f]. The minimum and the maximum disparity of Panum's fusional area of the stereo image are defined by D_b and D_f, respectively.

Many stereo images are obtained using parallel stereo cameras. If these stereo image pairs are not horizontally moved in changing the disparity, the perceived depth will only be in front of the screen [7]. For this case, Agarwal and Blake have discussed Panum's fusional area in [8].

3. The proposed method

The resized stereo image for our method is a pair of rectified images I_L and I_R. The proposed method retargets I_L and I_R into a new size. The new stereo image is composed of paired images I′_L and I′_R.

In this paper, we described the method focusing on reducing the stereo image width. Carving and inserting are reciprocal, and horizontal resizing is similar as vertical resizing.

The proposed method resized the stereo image as follows:

• 1. Calculate the disparity map (D) of the stereo image, then segment the main content base on D and Panum's fusional area of the stereo image, [D _b, D _f].

• 2. Calculate and select the seam of the left image (S _L), then pick the seam of the right image (S _R) with the disparity, D. Respectively carve S _L and S _R from I _L and I _R.

• 3. Repeat step 2 according to the image width which needs to be reduced.

3.1. Main content segmentation

According to whether the disparity of corresponding objects is in Panum's fusional area ([D_b, D_f]), the image was segmented into the main content and background.

The disparity map (D) of the stereo image is calculated by the belief propagation (BP) algorithm [9]. We consider the disparity map with respect to I_L, which is taken to be the reference image. The disparity map (D) is shown in Figure 2.

Figure 2

The disparity map calculated by applying the BP algorithm.

The BP algorithm we used for calculating the disparity map was based on belief propagation and mean shift segmentation [10]. The disparity map and the reference image (I_L) are segmented into some objects. The objects and the average disparity of these objects are denoted by $o_{\mathrm{L}}^i$ and $d_{\mathrm{L}}^i$, respectively, i = 1, 2,…, m. If $d_{\mathrm{L}}^i$ is in [D_b, D_f], $o_{\mathrm{L}}^i$ is regarded as the main content, $o_{\mathrm{L}}^i\in O_{\mathrm{main}\mathrm{content}}$. If $d_{\mathrm{L}}^i$ is not in [D_b, D_f], $o_{\mathrm{L}}^i$ is regarded as the background, $o_{\mathrm{L}}^i\in O_{\mathrm{background}}$. That is,

$$o_{\mathrm{L}}^i\in \left\{\begin{array}{c}\hfill O_{\mathrm{main}\mathrm{content}}{d}_{\mathrm{L}}^i\in \left[D_{\mathrm{b}},D_{\mathrm{f}}\right]\hfill \\ \hfill O_{\mathrm{background}}{d}_{\mathrm{L}}^i\notin \left[D_{\mathrm{b}},D_{\mathrm{f}}\right]\hfill \end{array}\right..$$

(5)

Figure 3 depicts the result of the main content segmentation. The main objects are reserved, and the background is removed from the left image.

Figure 3

The main objects of the left image.

3.2. Seam selection and carving

A seam is an optimal eight-connected path of pixels on a single image from top to bottom (vertical) and consisted of one and only one pixel in each row, which guarantees that the image keeps a rectangle when the seams are removed. In [1], an energy function defines the cost of a seam. The optimal seam S^* which minimizes this seam cost is selected:

$$S^{*}=\underset{s}{\min }E\left(s\right)=\underset{s}{\min }{\displaystyle \sum }_{j=1}^n{e}_{\mathrm{HoG}}\left(I\left(s_i\right)\right),$$

(6)

where e_HoG is the energy with Histogram of Gradients, which is defined as follows:

$$e_{\mathrm{HoG}}\left(I\right)=\frac{\left|\frac{\partial }{\partial x}I\right|+\left|\frac{\partial }{\partial y}I\right|}{\max \left(\mathit{HoG}\left(I\left(x,y\right)\right)\right)}.$$

(7)

In our method, the seam is selected by both energy function and main content protection. Main content protection lets the seam bypass the main content without hurting the proportion.

Let S_L denote the seam in I_L and S_R denote the seam in I_R. S_L and S_R are correlative by the disparity map D. The disparity map shows the correspondence of each pixel in the left and right images. So S_R can be obtained by S_L and D.

S_L is computed by the energy function. The energy function based on the gradient was used to select the energy which the pixels in the seams have, which is called the backward energy at first. The minimum energy tried to minimize the artifacts introduced in the generated image.

If it does not cross the main content, S_L and its correlative seam S_R will be removed for retargeting the stereo image.

If S_L crosses the main content, its crossing part will be replaced. Let S_cross be the parts that cross the main content and S_not cross be the parts that do not cross the object of the main content. Let S_E denote the new part of the seam used to replace S_cross. S_E is selected beside the edge of the object; the vertical height of S_E and S_cross is the same, and the start points and end points of S_E and S_cross have the same ordinate value. That is, assuming that S_cross starts from point a(i₁, j₁) to point b(i₂, j₂) to end, S_E will start from point c(k₁, j₁) to point d(k₂, j₂). Let S′_L denote the new seam that consists of S_E and S_not cross. By S′_L and D, the new seam of the right image (S′_R) is obtained. Figure 4 shows the coupled seams S′_L and S′_R. S′_L and S′_R are not necessarily continuous:

$$S^{\text{'}}{}_{\mathrm{L}}=\left\{\begin{array}{ll}{S}_{\mathrm{L}}& \mathrm{if}{S}_{\mathrm{L}}\mathrm{does}\mathrm{not}\mathrm{cross}\mathrm{the}\mathrm{main}\mathrm{content}\\ S_{\mathrm{L}}-S_{\mathrm{cross}}+S_{\mathrm{E}}& \mathrm{if}{S}_{\mathrm{L}}\mathrm{crosses}\mathrm{the}\mathrm{main}\mathrm{content}\end{array}\right..$$

Figure 4

The coupled seam of the stereo image. (a) The seam of the left image. (b) The seam of the right image.

Since the left and right images are captured from different views, some pixels around objects are occluded [11]. Therefore, removing S′_L and S′_R takes a little effect on depth perception.

The energy function calculates the changing intensities of gradients. When the change of aspect ratio is large, many seams need to be removed from the stereo image. In this scenario, some seams may intensively appear around the objects with smooth gradients and lead to cracks in the resized stereo image. In order to prevent this condition, when n seams need to be removed, we calculate 2n − 1 seams at once and alternately remove n seams from the 2n − 1 seams.

For some stereo images with large main content, the excessive main content protection will cause overflow or seriously distort the global structure of stereo images. To address this problem, we set a ratio based on the image content. If the reduced width is more than the ratio of the image, the seams of the left image will be defined only by the energy function, without considering main content protection.

4. Experimental results and analysis

We tested our method on standard stereo images obtained from middlebury stereo datasets. We reduced by 20% the width of tsukuba[12] and Reindeer[13] by our method and single image seam carving, respectively. Figures 5 and 6 compare our results and single image seam carving results.

Figure 5

Comparison of our method and single image seam carving for tsukuba dataset. (a) The original pair of stereo image. (b) The resized pair of stereo image by our method. (c) The resized pair of stereo image by single image seam carving.

Figure 6

Comparison of our method and single image seam carving for Reindeer dataset. (a) The original pair of stereo image. (b) The resized pair of stereo image by our method. (c) The resized pair of stereo image by single image seam carving.

As can be observed from Figures 5 and 6, the main content is not impacted by resizing using our proposed method. The depth perception of important content is preserved well. It is also clearly seen that a significant depth distortion is caused when naive independent retargeting of each image is considered.

5. Conclusions

Since independently retargeting each image of the stereo image pair will distort the geometric structure of the main content, in this paper, we proposed a new seam carving method to work on stereo images without distorting the depth perception of the main content. We improved the seam carving to work on a pair of stereo images and proved that the proposed method is guaranteed to give a geometrically consistent result of main content. Our method takes advantage of both appearance and perceived depth and can deal with stereo images that are difficult to process with single image seam carving. The experimental results showed that our method had a satisfactory result on stereo images.