The resized stereo image for our method is a pair of rectified images IL and IR. The proposed method retargets IL and IR into a new size. The new stereo image is composed of paired images I′L and I′R.
In this paper, we described the method focusing on reducing the stereo image width. Carving and inserting are reciprocal, and horizontal resizing is similar as vertical resizing.
The proposed method resized the stereo image as follows:
-
1.
Calculate the disparity map (D) of the stereo image, then segment the main content base on D and Panum's fusional area of the stereo image, [D b, D f].
-
2.
Calculate and select the seam of the left image (S L), then pick the seam of the right image (S R) with the disparity, D. Respectively carve S L and S R from I L and I R.
-
3.
Repeat step 2 according to the image width which needs to be reduced.
3.1. Main content segmentation
According to whether the disparity of corresponding objects is in Panum's fusional area ([Db, Df]), the image was segmented into the main content and background.
The disparity map (D) of the stereo image is calculated by the belief propagation (BP) algorithm [9]. We consider the disparity map with respect to IL, which is taken to be the reference image. The disparity map (D) is shown in Figure 2.
The BP algorithm we used for calculating the disparity map was based on belief propagation and mean shift segmentation [10]. The disparity map and the reference image (IL) are segmented into some objects. The objects and the average disparity of these objects are denoted by and , respectively, i = 1, 2,…, m. If is in [Db, Df], is regarded as the main content, . If is not in [Db, Df], is regarded as the background, . That is,
(5)
Figure 3 depicts the result of the main content segmentation. The main objects are reserved, and the background is removed from the left image.
3.2. Seam selection and carving
A seam is an optimal eight-connected path of pixels on a single image from top to bottom (vertical) and consisted of one and only one pixel in each row, which guarantees that the image keeps a rectangle when the seams are removed. In [1], an energy function defines the cost of a seam. The optimal seam S* which minimizes this seam cost is selected:
(6)
where eHoG is the energy with Histogram of Gradients, which is defined as follows:
(7)
In our method, the seam is selected by both energy function and main content protection. Main content protection lets the seam bypass the main content without hurting the proportion.
Let SL denote the seam in IL and SR denote the seam in IR. SL and SR are correlative by the disparity map D. The disparity map shows the correspondence of each pixel in the left and right images. So SR can be obtained by SL and D.
SL is computed by the energy function. The energy function based on the gradient was used to select the energy which the pixels in the seams have, which is called the backward energy at first. The minimum energy tried to minimize the artifacts introduced in the generated image.
If it does not cross the main content, SL and its correlative seam SR will be removed for retargeting the stereo image.
If SL crosses the main content, its crossing part will be replaced. Let Scross be the parts that cross the main content and Snot cross be the parts that do not cross the object of the main content. Let SE denote the new part of the seam used to replace Scross. SE is selected beside the edge of the object; the vertical height of SE and Scross is the same, and the start points and end points of SE and Scross have the same ordinate value. That is, assuming that Scross starts from point a(i1, j1) to point b(i2, j2) to end, SE will start from point c(k1, j1) to point d(k2, j2). Let S′L denote the new seam that consists of SE and Snot cross. By S′L and D, the new seam of the right image (S′R) is obtained. Figure 4 shows the coupled seams S′L and S′R. S′L and S′R are not necessarily continuous:
Since the left and right images are captured from different views, some pixels around objects are occluded [11]. Therefore, removing S′L and S′R takes a little effect on depth perception.
The energy function calculates the changing intensities of gradients. When the change of aspect ratio is large, many seams need to be removed from the stereo image. In this scenario, some seams may intensively appear around the objects with smooth gradients and lead to cracks in the resized stereo image. In order to prevent this condition, when n seams need to be removed, we calculate 2n − 1 seams at once and alternately remove n seams from the 2n − 1 seams.
For some stereo images with large main content, the excessive main content protection will cause overflow or seriously distort the global structure of stereo images. To address this problem, we set a ratio based on the image content. If the reduced width is more than the ratio of the image, the seams of the left image will be defined only by the energy function, without considering main content protection.