Early disparity estimation skipping for multi-view video coding

Abstract

Multi-view video (MVV) and multi-view plus depth video became typical formats for 3D video systems which offer more realistic visual effects for consumers. The standard for encoding of MVV, known as multi-view video coding (MVC), was produced by ISO/IEC and ITU-T and has exhibited significant compression efficiency for MVV signals. However, the high computational complexity of MVC leads to excessive power consumption and limits its applications. In this study, new fast algorithms for MVC are proposed by skipping disparity estimation prediction. According to the characteristics of the motion and disparity vectors, unnecessary disparity estimation is skipped based on the amount of motion that can be measured through the use of motion activity and SKIP mode on the temporal axis. Experimental results showed that by skipping the disparity estimation, the proposed method reduced the encoding computational complexity by nearly 73% for inter-view prediction and 38% in the overall encoding process of B-views. In addition, no noticeable visual quality degradation was observed.

1. Introduction

Recently, 3D video has received increased attention because of the great success of 3D movies. Other mediums such as three-dimensional TV (3DTV) and 3D digital multimedia broadcasting (3D DMB) are also representative applications of 3D video [1]. 3DTV provides vivid and realistic scenes to users with its feeling of depth [2, 3]. The principles of 3DTV are based on a stereoscopic vision system. When left and right views of stereoscopic video are shown in a user's left and right eyes, respectively, users can perceive a feeling of depth due to binocular parallax. A 3DTV system contains not only stereoscopic video, but also a multi-directional 3D scene by multi-view. 3D DMB is a broadcasting system which offers 3D video to users in mobile circumstances [4]. Because the system can present more realistic video and information than conventional DMB, it is expected to be widely used in e-commerce and news media.

Among the video formats that support 3DTV and 3D DMB, multi-view video (MVV) and multi-view plus depth video (MVD) are typical formats. MVV is simultaneously acquired by two or more cameras placed at a certain distance from each other so that its viewpoint number is fixed as the number of cameras. MVV is very simple and intuitive, but it has difficulties in responding to various types of 3D displays because it provides limited viewpoints. On the other hand, MVD contains multi-view color and depth video which are used to synthesize virtual views in the decoder side. The MVD format can be applied to various 3D displays because of virtual view synthesis, which leads to a high degree of computational complexity for the decoder. However, MVV and MVD have one aspect in common: they receive an enormous amount of data from multiple viewpoints.

Multi-view video coding (MVC) is a video coding standard that reduces a large amount of MVV data. It was developed by the Joint Video Team (JVT), an organization jointly created by the Video Coding Expert Group and the Motion Picture Expert Group (MPEG) [5]. In general, the computational complexity of a video codec for MVV proportionally increases with the number of views. However, the complexity of MVC is far beyond the independent encoding of each view because of the exhaustive search for inter-view prediction, a key technique in MVC, and it leads to excessive power consumption and works as an obstacle for practical use. Therefore, it is necessary to decrease the computational complexity of MVC.

Some research has been performed to reduce the computational burden of MVC [6–10]. Kim et al. [6] proposed an adaptive search range control algorithm for motion and disparity estimation. The researchers reduced the computation time for motion and disparity estimation by controlling the search ranges. The reliability of the predicted vectors for the current block was used to control the range, and it was calculated based on the difference between two predicted vectors by two ways. Since the search ranges of motion and disparity estimation are reduced, the algorithm showed time saving performance of about 70%. However, the algorithm needs camera parameters for vector prediction, and it is difficult to apply to MVC because it was based on the different coding structure from the structure of MVC. Ding et al. [7] proposed a fast motion estimation algorithm that finds an initial motion vector based on the macroblock partitioning information of the reference view. The algorithm then refines the motion vector in a small search range near the initial motion vector. This approach simplified the mode selection process and reduced the search range of the motion estimation so that the overall processing time of MVC was reduced. However, there are many conventional fast mode decision and search range adaptation algorithms for motion estimation, Ding's algorithm overlaps with some of these conventional algorithms. Peng et al. [8] presented a hybrid fast macroblock mode selection algorithm for MVC. In base view, the algorithm partly stops the block mode selection process through multi-thresholding of the rate-distortion cost. In other views, the macroblock mode can be predicted from the frames of neighboring views. The fast algorithm decreased the number of cases in the mode selection process. It covered most of the pictures in the coding structure of MVC, but it is difficult to implement into hardware due to the complicated construction of the algorithm. Li et al. [9] proposed a fast disparity and motion estimation technique based on multi-view geometry. The algorithm reduced the search range for disparity estimation based on a correlation between neighboring cameras and selectively skipped the motion estimation process through the use of the relationship between the motion and disparity vectors. Li et al.'s algorithm showed a time saving performance of about 65%, but bit rate was increased by about 3%. Shen et al. [10] proposed reduced mode selection, search range adaptation, and view-adaptive disparity estimation (VDAE) for MVC. They were based on the mode complexity and motion homogeneity of the neighboring views. The algorithms do not overlap each other and show relatively high performance, but the reliability of the algorithms is not high because they use a global disparity vector to obtain the information of the encoded neighboring views.

In this article, new fast algorithms that skip the disparity estimation process for inter-view prediction without the information of neighboring views are proposed. While disparity vectors are rarely selected for the results of MVC, the disparity estimation process consumes much time. The proposed algorithms reduce the encoding time of MVC by skipping the unnecessary disparity estimation process. For skipping of disparity estimation, the motion activity and SKIP mode are introduced. Since these methods are not influenced by multi-view geometry, the proposed algorithms can provide consistent results regardless of the camera arrangement. It can also be applied with the conventional fast algorithms of other types such as search range adaptation and early mode determination, because the proposed algorithms do not theoretically overlap with the conventional fast algorithms.

This article is organized as follows. An overview of the MVC prediction structure is presented in Section 2. In Section 3, two new methods for skipping disparity estimation are proposed. Experimental results and analysis are given in Section 4 and conclusions are followed in Section 5.

2. Overview of MVC prediction structure

MVC reference software was chosen at the 75th MPEG meeting, and it was based on version 3.5 of the JSVM (Joint Scalable Video Model) software [11]. It used reordered MVV as a 2D sequence because conventional 2D video codec was used for MVC without syntax modification. After the MVC standardization activity was moved to JVT, a new version of MVC software considered the characteristics of MVV was developed. It included two special properties such as coding structure for inter-view prediction and hierarchical B-picture structure.

Figure 1 shows the acquisition and coding structure for MVV. The MVV is simultaneously acquired by two or more cameras as shown in Figure 1a, and then it is encoded by the MVC software with coding structure as shown in Figure 1b. The MVC standard adopted inter-view prediction to remove the redundancy among neighboring views. The full inter-view prediction is applied to every other view, i.e., S1 and S3 in Figure 1b, because excessive estimation process for the inter-view prediction causes high computational complexity. However, for all the pictures at T0 and T8, the inter-view prediction is performed regardless of view order. These pictures are called 'anchor picture,' and are used as stamps of synchronization and random access. According to the picture type of the anchor pictures, a view type is determined. For instance, the view type of S0 is I-view, and the view types of S2 and S1 are P-view (predictive coded view) and B-view (bi-directional predictive coded view), respectively.

Figure 1

Acquisition and coding structure of MVV. (a) MVV acquisition. (b) MVC structure.

A hierarchical B-picture structure was adopted for coding performance regardless of the characteristics of the MVV. In contrast to the conventional structure such as 'IPPP' or 'IBBP', the predicted picture has its own hierarchy level in the hierarchical B-picture structure and the pictures are encoded sequentially according to their level [12]. This concept provides the benefit of increased flexibility at the picture/sequence level through the availability of the multiple reference picture technique [13]. In general, the maximum hierarchy level is four because of the encoding complexity. For the multi-view test sequences, the hierarchical structure is determined based on the group of picture (GOP) length, as shown in Figure 2.

Figure 2

Hierarchical B-picture structures for multi-view sequences.

The inter-view prediction technique enhances the coding efficiency of MVC, but leads to a high degree of computational complexity. The technique yielded coding gains of up to 3.2 dB and an average coding gain of 1.5 dB [14]. However, its complexity was much higher when compared to that of view-independent coding due to the exhaustive search for inter-view prediction. The computational complexity for B-views was twice that of a single view coding. The proportion of motion and disparity estimation time of the B-views are shown in Table 1. Disparity estimation occupies almost one half of the total processing time. Nevertheless, disparity vectors are rarely selected. Merkle et al. [14] found that the proportion of selected inter-view prediction is about 13% in MVC for several sets of multi-view test data. Because temporal prediction was selected for the inter-coding, the inter-view prediction process is unnecessary for the other blocks. Thus, the computational complexity of MVC can be reduced by skipping the unnecessary process of inter-view prediction.

Table 1 Processing time ratio of motion and disparity estimation in B-view

3. Fast algorithm for MVC by skipping inter-view prediction

To determine the unnecessary disparity estimation process, we observed motion and disparity estimation processes. It was found that motion vectors are most likely to be selected in static or slow motion areas and disparity vectors are used only in fast motion areas. Table 2 shows the average magnitude values of motion vectors for two regions, encoded by temporal prediction and inter-view prediction. The average magnitude values of the motion vectors in blocks encoded by inter-view prediction are far larger than the values in the temporally predicted blocks as shown in Table 2. This is due to the characteristics of temporal and inter-view prediction. In general, the performance of temporal prediction is superior to inter-view prediction, because inter-view prediction has inherent disadvantages such as a perspective effect and a color imbalance problem between views. The same objects may be seen as different shapes and pixel values in each view. However, temporal prediction exhibits poor performance in fast motion areas, because the correlation between the previous picture and the current picture decreases and its rate-distortion cost increases due to large motion vectors. Thus, by skipping the disparity estimation process in motionless or slow motion areas, the computational complexity of MVC can be decreased with no degradation in video quality. To measure the amount of motion and to simplify inter-view prediction, two methods are proposed in the following sections.

Table 2 Average magnitude of the motion vectors in two regions

3.1. Motion activity method

The first proposed method to determine the motionless or slow motion area is based on a thresholding technique. Motion activity is defined to represent the amount of motion in a macroblock as follows:

$$\mathsf{\text{Motion}}\mathsf{\text{Activity}}=\left|\mathsf{\text{M}}{\mathsf{\text{V}}}_x\right|+\left|\mathsf{\text{M}}{\mathsf{\text{V}}}_y\right|$$

(1)

where MV _x and MV _y are the horizontal and vertical components of the motion vector, respectively. There are some vision techniques used to accurately measure the amount of motion, such as optical flow and feature tracking. However, these techniques have a high degree of computational complexity and are not suitable for a codec. The information from motion vectors is appropriate for measuring the amount of motion in a video codec, because the vectors roughly reflect the amount of motion in a block. The information of the motion vectors can be obtained after the motion estimation. The motion activity has a physical meaning similar to the magnitude of the motion vectors and exhibits more hardware-friendly characteristics than the magnitude. Thus, the motion activity was used as a variable for the thresholding.

To formalize the calculation process of the motion activity in various macroblock partition modes, an 8 × 8 block was selected as a basic unit for the motion activity. In the case of inter 16 × 16 block mode, the motion activity is calculated by multiplying the sum of the motion vector components by 4, which is the number of 8 × 8 blocks in the 16 × 16 block. Since intra mode is usually selected in very fast motion areas or new object areas, the motion activity is defined as infinite in intra mode. Equations for calculating the motion activity for all of the cases in this study are given in Table 3.

Table 3 Motion activity in block modes

It is possible to select various threshold values of the motion activity based on the video data and quantization parameters (QPs) for performance. However, in this study, only one threshold value was selected for every case due to practical use. To observe the motion activity values under various conditions, MVC was performed in four QPs and eight video sequences. The average motion activity values for temporal prediction and inter-view prediction are shown in Table 4. The average motion activity of the temporally predicted region was 17.3 for the test sequences. Based on this result, the proposed algorithm was performed with threshold values of the motion activity in the range of 10 to 30. The peak signal-to-noise ratio (PSNR) values were similar to all the threshold values we used. However, as the threshold value increased, the bit rate increased and the processing time decreased, as shown in Table 5. From the results, 22 was selected as the threshold, because it maintained a bit rate increment of about 0.5% and time was sufficiently saved.

Table 4 Average value of the motion activity in B-view

Table 5 Simulation results for various motion activity thresholds

The overall process of the disparity estimation skipping algorithm based on the motion activity is shown in Figure 3. At first, only motion estimation is performed for a target macroblock for the temporal prediction. The results of the motion estimation include motion information such as the block mode and the motion vectors. This motion activity is calculated based on the information. Then, the motion activity is compared with an empirically predefined threshold value to determine whether disparity estimation was performed or not. If the motion activity is less than the threshold, disparity estimation is skipped, because the macroblock is located in a motionless or slow motion area. Otherwise, disparity estimation will be performed. When disparity estimation is performed for the macroblock, the rate-distortion cost of the disparity estimation is compared with that of motion estimation in order to select the best block mode and vectors.

Figure 3

The overall process of the proposed algorithm based on the motion activity.

3.2. Method used by SKIP mode in temporal prediction

Another method to determine skipping of the disparity estimation process refers to the selection of SKIP mode in temporal prediction. SKIP mode is one of the inter prediction modes in H.264/AVC, and a 1-bit flag is transmitted for the mode. The blocks encoded as the mode are reconstructed by reference indexes and motion vectors derived by causal blocks. SKIP mode is usually selected in stationary regions or slow motion regions. Figure 4 shows the distribution of selected SKIP, inter and intra modes in a B-picture. The moving object (the dancer in the middle of the scene) is encoded as intra or inter mode, but the background and static objects are encoded as SKIP mode. Table 6 also shows the motion activity values in blocks encoded as SKIP mode and non-SKIP modes. In case of SKIP mode, the motion activity values are smaller than the values of blocks encoded as non-SKIP modes. It presents that blocks selected as SKIP mode are located in slow motion regions or stationary regions. Thus, SKIP mode can be used to determine whether disparity estimation is skipped or not.

Figure 4

Macroblock type status in the B-picture of the Breakdancers sequence.

Table 6 Average value of the motion activity in SKIP mode and non-SKIP mode

The overall process of the proposed algorithm based on SKIP mode is shown in Figure 5. At first, motion estimation is performed in the temporal axis. If SKIP mode is employed for the macroblock, the disparity estimation process is skipped. Otherwise, disparity estimation is processed for inter-view prediction. The result of the disparity estimation is then compared to the result of the motion estimation in order to obtain the best macroblock mode and vectors.

Figure 5

The overall process of the proposed algorithm based on SKIP mode.

4. Experimental results

Implementation of the proposed algorithm was based on JMVM 8.0 to verify its performance [15]. All of the experiments have been performed on a desktop computer with a 2.4-GHz CPU and 2 GB of memory. The test conditions for the simulation are shown in Table 7 and information on the four test sequences are presented in Table 8 [16]. Each view image of the multi-view test sequences are shown in Figure 6.

Table 7 Test conditions for the experiments

Table 8 Parameters of the test sequences

Figure 6

Test sequences (top to bottom: Ballroom, Breakdancers, Exit, and Uli).

Three metrics were defined for evaluating the performance of the proposed algorithm: ΔBDPSNR, ΔBDBR, and ΔT. The metric ΔBDPSNR is defined as the change between the average PSNR of the conventional MVC and the proposed method by the Bjonteggard delta measurement [17]. As the performance improves, this criterion approaches zero. The metric ΔBDBR is the bit rate difference, as a percentage, between the compared methods by a Bjonteggard delta measurement. When the proposed algorithm exhibits the same performance as the conventional method, ΔBDBR approaches zero. If ΔBDBR has a negative value, the proposed algorithm is better than the conventional method. The parameter ΔT is the processing time saving factor and is defined as:

$$\mathrm{\Delta }T=\frac{T_{\mathsf{\text{conventional}}}-T_{\mathsf{\text{proposed}}}}{T_{\mathsf{\text{conventional}}}}\times 100\left(\%\right)$$

(2)

As ΔT increases in a negative direction, the performance speed is increased. With the predefined threshold, the proposed algorithm was performed for B-views of the test sequences. The experimental results of the proposed algorithm are shown in Table 9. For the B-views, the encoding speed was improved by about 73% in the disparity estimation for inter-view prediction and about 38% in the total encoding process. When compared with VADE [10], the proposed algorithm shows better performance by 14% in disparity estimation and 8% in total encoding time. As shown in Figure 7, no noticeable degradation in quality was observed. The ΔBDPSNR decrease was less than 0.05 dB and the ΔBDBR increase was about 1% when compared to the original results. For the Breakdancers and Uli sequences, the time saving performance was a little worse than in the other sequences because the Breakdancers sequence includes a dancer with fast movements and the Uli sequence includes many moving objects in the scene. The motion activity method presents a smaller bit rate increment and lower PSNR degradation than those of the method used by SKIP mode. The average encoding time of the motion activity method was slightly faster. This is due to the characteristics of the method used by SKIP mode. The method indirectly measures the amount of motion. SKIP mode roughly reflects the tendency of motion in a scene, but it can be selected in a fast motion area and not selected in a slow motion area. Table 10 shows the ratio of false alarms for two algorithms. The false positive affects the coding efficiency of the fast algorithm, and the false negative affects the time saving performance. The motion activity method showed more skipping of the disparity estimation process and fewer false alarms than method used by SKIP mode. However, the method used by SKIP mode is simpler than the motion activity method in terms of implementation. It does not need threshold values and a calculation process to determine disparity estimation skipping. The motion activity method is suitable for software implementation because it can provide better performance through the use of an additional process for selection of the threshold values. On the other hand, the method used by SKIP mode is appropriate for a limited hardware development environment, such as mobile and embedded hardware, due to its hardware-friendly characteristics.

Table 9 Performance of the proposed algorithm in B-views

Figure 7

RD curve of the proposed algorithm. (a) Ballroom; (b) Breakdancers; (c) Exit; (d) Uli.

Table 10 The comparison of two proposed algorithms

The proposed algorithm and other conventional fast algorithms can simultaneously be applied without a loss in time saving performance, because the proposed algorithm does not theoretically overlap with the conventional fast algorithms such as search range reduction and early mode determination. Table 11 shows the performance of the proposed algorithm and the TZ-search algorithm [18], which is used in JMVM software as a fast search method. Results of the proposed algorithm with the TZ-search algorithm showed improvement of about 45% in encoding time of the B-views when compared to the results of the TZ-search algorithm alone. About 70% of the disparity estimation process was skipped by the proposed algorithm in B-view, although the conventional fast algorithm was simultaneously applied with the proposed algorithm. This amount of time saving is similar to the results obtained by the proposed algorithm shown in Table 9.

Table 11 Performance of the proposed algorithm simultaneously performed with TZ search algorithm

5. Conclusion

New fast algorithms for MVC using disparity estimation skipping for inter-view prediction were proposed. The proposed algorithms determine whether disparity estimation is performed based on the amount of motion or not. To measure the amount of motion in a macroblock level, two methods were employed. The motion activity method uses motion activity as a variable for thresholding. The method used by SKIP mode omits the disparity estimation process when SKIP mode is selected in temporal prediction. The proposed methods are simple and have hardware-friendly characteristics. The encoding complexity was decreased by about 73% in disparity estimation for inter-view prediction and about 38% in the total encoding process of B-views without noticeable degradation in quality. Because the proposed algorithm uses an approach that is different from the conventional fast algorithms, it exhibits consistent time saving performance when it is simultaneously applied with the conventional fast algorithms.