New rank detection methods for reducedrank MIMO systems
 Qian Wang^{1}Email author and
 Yindi Jing^{1}
https://doi.org/10.1186/s1363801504574
© Wang and Jing. 2015
Received: 1 July 2015
Accepted: 5 October 2015
Published: 19 October 2015
Abstract
In practical multiinput multioutput (MIMO) systems, the channel matrices often have reduced rank. Reliable detection of the channel rank is essential in achieving the significant gain provided by MIMO configuration. Existing work on MIMO channel rank detection assume a static channel model, so the proposed methods only consider the noise distributions while the distributions of the MIMO channels are not considered. In this paper, we employ a random channel model and propose three thresholdbased rank detection methods which take into account the distributions of both the channels and the noises. In our first algorithm, following existing singlethreshold rank detection scheme, we rigorously derive an analytical lower bound on the correct rank detection probability and propose a systematic threshold selection method by maximizing the lower bound. Then we propose two new rank detection methods which use multiple thresholds, where each threshold corresponds to one possible rank value. The thresholds are optimized based on the derived lower bounds on the rank detection probability for different channel rank values. The convergence and complexity of the proposed algorithms are analyzed. Simulation results on the correct rank detection probability of the proposed schemes are provided to show their advantage over existing schemes. The meansquarederror (MSE) and outage probability are also simulated to show the importance of reliable rank detection in MIMO communications.
Keywords
1 Introduction
In the past two decades, a configuration, called multiinput multioutput (MIMO) system, which utilizes multiple antennas at both the transmitter and the receiver, has been extensively studied [1, 2]. MIMO systems have outstanding performance in increasing the data throughput and link range over singleantenna systems without consuming additional bandwidth or transmit power. Early studies focused on MIMO systems with fullrank channels. But in many practical propagation environments such as the number of surrounding scatterers which is finite and limited, the MIMO channel matrix is likely to have reduced rank [3, 4], especially when the channel dimension is large [5]. In [3], the finite scatterer channel model was described and its capacity was analyzed. In [4], the model was applied to the largescale MIMO system, referred to as massive MIMO. When the channel matrix has reduced rank, the number of its entries is larger than its real dimension, and thus designs based on fullrank channels become inefficient. This motivates the research on reducedrank technologies for MIMO systems [6–23].
In [6–14], various reducedrank filtering technologies were proposed, where a reducedrank transformation is first used on the observed signal vector to obtain a lowerdimension vector, then a filter is designed to estimate the desired signal vector. The reducedrank transformation can lower the order of the filter and thus require less computation complexity and shorter training length. If the covariance matrix of the observed data vector is known, the optimal reducedrank design is singular value decomposition (SVD)based [13, 14]. But the covariance matrix estimation needs a very large number of training symbols. To overcome this drawback, a reducedrank multistage Wiener filter design was proposed in [6], which relaxes the requirement on explicit estimate of the signal subspace. It was later extended to its adaptive versions in [7, 8] and applied to MIMO equalization in [9]. Another related design is the auxiliaryvector filtering algorithm [10]. Compared with the multistage Wiener filter, it achieves better performance and does not involve matrix inversion operation. Joint transformation and filter designs can be found in [11, 12].
Another important issue for MIMO communications is the channel rank detection and channel matrix estimation, since most MIMO transceiver techniques require channel state information (CSI) at the transmitter side and/or the receiver side for smart signal processing. For example, in the MIMO multiplexing transmissions, the transmitter should align multiple data streams with the eigenspaces of the channel. To achieve this, precise estimates on the channel rank and channel matrix are needed. This paper is concerned with the channel rank detection, which is an important part of channel estimation. Thus, in what follows, we explain related literature on channel estimation and rank detection.
Some existing work on MIMO channel estimation focus on fullrank MIMO channel matrices with independent or correlated entries [24–27]. Their channel estimation schemes are entrybased, where the unknown channel matrix is parameterized by its entries. For reducedrank channel estimation, these schemes will cause performance degradation. A more natural and efficient approach is to use SVDbased channel estimation methods [15–23]. It was shown in [16, 17] that the maximumlikelihood (ML) estimation of the reducedrank MIMO channel with Gaussian noise is the truncated SVD method. In truncated SVD, if the channel rank is known to be r, the MIMO channel matrix is estimated from the SVD of the received signalplusnoise matrix, by keeping the largest r singular values and their corresponding singular vectors. While the truncated SVD method traces back to the 1930s [28], it was rediscovered in [16, 17] for MIMO channel estimation and further improved to reduce the mean squared error (MSE) of the estimation [18–22]. One improved scheme is shrinkageandthreshold SVD, where the truncated singular values are further shrunk to remove the noise effect.
In both truncated SVD and shrinkageandthreshold SVD, the channel estimation accuracy largely depends on the correct truncation of the singular values, which is the rank detection of the MIMO channel matrix. Thus correct rank detection can improve the channel estimation quality, which is crucial for advanced MIMO techniques such as beamforming and power allocation among data streams. Furthermore, for a MIMO channel matrix, the rank is the indicator of how many data streams can be spatially multiplexed on the channel, and the data streams are represented by the singular values and the corresponding singular vectors. Similarly, for a multiuser system with multiple antennas at the base station, the rank of the MIMO channel matrix from all users to the base station determines how many users can be served by the base station within the same timefrequency bandwidth. Thus, accurate channel rank detection is an important part of channel estimation and is essential for MIMO systems.
Various rank detection methods are available in the literature [17–23]. In [17], a minimum description length (MDL)based rank detection was used for MIMO channels. A threshold is calculated at each instance of the channel to minimize the MDL. The MDLbased detection aims at minimizing the MSE. It also requires a large number of samples to work. In [23], several rank detection methods were proposed for a realvalued channel matrix. In the proposed method, the singular values or functions of singular values are compared with a threshold for rank detection. Lower and upper bounds on the threshold selection were discussed. Josse and Sardy [22] considered both the MIMO channel rank detection and the shrinkage of the singular values for the channel estimation. The rank detection scheme is also thresholdbased, where two threshold selection methods are proposed. In the first method, the threshold parameter and the shrinkage parameter are jointly optimized for each channel realization to minimize a Stein unbiased risk estimate (SURE) of the MSE of the MIMO channel estimation. In the second method, the threshold is calculated from the distribution of the largest singular value of the noise matrix. In [18–21], for asymptotic MIMO channels where both dimensions of the channel matrix approach infinity with a fixed ratio, simple closedform thresholds were derived for thresholdbased rank detection.
1.1 Summary of our results and distinction to existing work
This paper is on the rank detection of MIMO channels. We propose a threshold optimization method for the traditional singlethresholdbased rank detection scheme and two new multiplethresholdbased rank detection schemes. For the threshold optimization of the traditional singlethreshold scheme, we first derive a closedform lower bound on the probability of correct rank detection based on the a priori channel rank distribution, then the optimal threshold is decided via the maximization of the lower bound. For the two new multiplethreshold schemes, different thresholds are used for different possible rank values, and each threshold is derived by maximizing the lower bound on the probability of correct rank detection when a specific rank value is assumed. Properties (e.g., wellposedness, convergence, complexity) of the two new schemes are discussed. Simulation results on the probability of correct rank detection of the proposed schemes are shown, and their advantages over existing schemes are discussed. The MSE and outage probability are also simulated to show that better rank detection improves the MSE and outage performance of MIMO systems.
Our model, problem formulation, and methods differ from existing ones in the following major aspects. First, we assume a random channel matrix where the channel entries follow Rayleigh flatfading and the distribution of the channel coefficients is taken into account in the threshold optimization. On the contrary, in all existing work, the channels are assumed to be static and the distribution of the channel matrix is not used in the rank detection designs [14, 17–23]. Also, in our model, a general training length and unitary training matrix are considered, while most existing work (e.g., [17–23]) apply to identity training matrix only, where the training length equals the number of transmit antennas. Finally, in this work, we use the probability of correct rank detection as the performance measure and optimization objective, while existing work (except [23]) targeted at minimizing the MSE of the MIMO channel estimation [17–22].
In what follows, we clarify the major difference of our channel rank detection and channel estimation problem to the reducedrank filter design problem in [6–14]. The goal of channel estimation is to estimate the channel matrix itself given a limited training time. In MIMO communications, channel estimation is usually required in the transceiver designs to optimize the communication performance such as outage probability and bit error rate. For the filter design, the goal is to obtain a precise estimate of the signals by filtering the observations, and usually the MSE is used as the design criterion. Regardless of the channel rank, a reducedrank filter can be used to lower the computational complexity and required training length. Naturally, precise channel rank detection and channel estimation can be helpful in reducedrank filter design, but it is not a necessary step in filter design. Also the optimal rank for the filter may not be the true rank of the channel matrix.
The rest of this paper is organized as follows. In Section 2, the MIMO channel model, the truncated SVDbased channel estimation, and the rank detection problem are presented. In Section 3, for the traditional singlethresholdbased rank detection, we derive a lower bound on the probability of correct rank detection and propose to optimize the threshold via the maximization of the lower bound. Two new rank detection algorithms based on multiple thresholds corresponding to different possible rank values are introduced in Section 4, as well as discussions on their properties. Simulation results on the probability of correct rank detection, MSE, and outage probability are shown in Section 5. Section 6 contains the conclusions.
In this paper, bold upper case letters and bold lower case letters are used to denote matrices and vectors, respectively. For a matrix A, its Hermitian, trace, rank, and determinant are denoted by A ^{∗}, tr(A), rank(A), and det(A), respectively. I _{ n } is the n×n identity matrix. \(\mathbb {E}(\cdot)\) denotes the average operator, and diag{a _{1},…,a _{ n }} denotes the diagonal matrix whose diagonal entries starting in the upper left corner are a _{1},⋯,a _{ n }.
2 Reducedrank MIMO channel and rank detection problem
2.1 Reducedrank MIMO channel model
Denote the rank of the channel matrix as r, i.e., rank(H)=r. If r=K, the channel has full rank. If r<K, the channel has reduced rank. For a reducedrank MIMO channel, the number of degrees of freedom in the channel matrix is less than its dimension. A rank analysis on typical propagation environments shows that MIMO channels often experience rank deficiency [3, 4, 29]. For example, in [29], the rank distribution of 8×8 MIMO channels under four scenarios, namely generalized typical urban, generalized bad urban, generalized hill terrain, and generalized rural area, is reported. For all four scenarios, the probability that the channel matrix has full rank is 0. Especially for the scenarios of generalized typical urban and generalized hill terrain, the rank of the 8×8 MIMO channel is always no higher than 4.
where A is an M×r fullrank matrix and B is a r×N fullrank rectangular unitary matrix. In this work, we focus on Rayleigh fading by assuming that entries of A are independent and identically distributed (i.i.d.) and follow circularly symmetric complex Gaussian (CSCG) distribution, with zero mean and unitvariance, i.e., \(a_{\textit {ij}}\sim \mathcal {C}\mathcal {N}(0,1)\), where a _{ ij } is the (i,j)th entry of A. It can thus be shown straightforwardly that each entry of H has CSCG distribution and \(h_{i,j}\sim \mathcal {C}\mathcal {N}(0,\\mathbf {b}_{j}\_{F}^{2})\), where b _{ j } is the jth column of B. It is noteworthy that the rank detection schemes proposed in this paper are not constrained to Rayleigh distribution and can be extended to more general channel fading models. The schemes are also not constrained to the unitary B case and can be extended to any general deterministic B matrix.
2.2 Training model, SVDbased channel estimation, and rank detection problem
where W is the T×N noise matrix. Entries of the noise matrix are assumed to be i.i.d. CSCG random variables with zero mean and unitvariance. The pilot and the noise are assumed to be independent to the channel matrix, which applies to most practical systems. It is noteworthy that the channel model and training model also apply to multiuser massive MIMO systems with M singleantenna users and N base station antennas, where channel training is conducted in the uplink [5].
Since S ^{∗} is an M×T unitary matrix, \(\tilde {\mathbf {W}}\) is an M×N random matrix. Entries of \(\tilde {\mathbf {W}}\) can be shown to be i.i.d. CSCG random variables with zero mean and their variances are M/(P T). From (6), \(\tilde {\mathbf {Y}}\) is a noisy observation of the channel matrix H with white Gaussian noises.
In (8), an estimation with rank r is obtained by keeping the subspaces with respect to the r strongest singular values of \(\tilde {\mathbf {Y}}\). Subspaces with respect to the K−r smallest singular values are seen as the noise effect and are ignored. This process guarantees that the estimator has the same rank with the real channel.
The entrybased estimation leads to a fullrank matrix. When r=K, i.e., the channel has full rank, the SVDbased estimation and entrybased estimation are equivalent. When r<K, i.e., the channel has reduced rank, the entrybased estimation will contain subspaces due to the noise effect only and thus have a worse performance.
Therefore, the rank detection is an essential problem for the SVDbased estimation. Wrong rank detection will lead to channel estimation error. In addition, it can degrade the performance of the MIMO communications. If the detected rank is smaller than r, some singular values and the corresponding singular spaces existing in the channel matrix may be detected as the noise effect only, and the subspaces will be lost in the estimated MIMO channel. On the other hand, if the detected rank is larger than r, some singular values and the corresponding singular spaces which do not exist in the channel matrix but appear in \(\tilde {\mathbf {Y}}\) because of the noise disturbance may be detected as part of the channel matrix. In MIMO communications, information and power will be allocated to such subspaces, which cause loss of information and wasting of power since the subspaces do not exist in the channel. Our problem of this paper is to detect the channel rank from the received signal Y, or the transformed received signal \(\tilde {\mathbf {Y}}\). It is noteworthy that the rank detection does not require extra training since the same observations can be used for both rank detection and channel estimation.
3 Threshold selection for singlethresholdbased rank detection
We can detect the rank of H from the singular values of \(\tilde {\mathbf {Y}}\). Thresholdbased algorithm appears to be a natural and common strategy [23], where the rank of H is detected as the number of singular values of \(\tilde {\mathbf {Y}}\) that are larger than the threshold. With this scheme, singular values of \(\tilde {\mathbf {Y}}\) that are smaller than the threshold are seen as the effect of the noise only; while singular values of \(\tilde {\mathbf {Y}}\) that are larger than the threshold are seen as the effect of nonzero component of the channel matrix with small noise disturbance.
If no singular value is larger than ε _{th}, i.e., σ _{1}<ε _{th}, the rank detection result is set to be 1 since the rank of a MIMO channel cannot be 0. The algorithm is given in Algorithm 1.
This singlethresholdbased rank detection idea is not new and was proposed and used in [17–23]. But the major challenge of using this scheme for rank detection lies in the selection of the threshold ε _{th}. Appropriate selection of the ε _{th} value is crucial to the detected result.
In this section, we first derive a lower bound on the probability of correct detection in Section 3.1, then propose a systematic method for the threshold selection based on maximizing the lower bound in Section 3.2, and finally discuss the difference of the proposed method with existing ones in Section 3.3.
3.1 Derivation of a lower bound on the conditional probability of correct rank detection
To find a systematic way of optimizing the threshold, we first derive a lower bound on the probability of correct detection conditioned on an arbitrary rank value of the MIMO channel matrix. The lower bound takes into consideration the system dimensions (e.g., T, M, and N), training power P, and the distributions of the channel coefficients and the noises. It will be used in the threshold optimization in later sections.
where γ(k,u) and Γ(k,u) are the lower and upper incomplete gamma functions [30], respectively.
The following proposition on the probability of correct rank detection under the condition that the rank of the MIMO channel matrix is r (for 1≤r≤K) is derived.
Proposition 1.
Proof 1.
Recall that σ _{ i }’s are the singular values of \(\tilde {\mathbf {Y}}\) in nonincreasing order, i.e., σ _{1}≥⋯≥σ _{ r }≥⋯≥σ _{ K }≥0. Let λ _{ i } and γ _{ i } be the singular values of H and \(\tilde {\mathbf {W}}\), respectively, both in nonincreasing order, i.e., λ _{1}≥⋯≥λ _{ r }≥0 and γ _{1}≥⋯≥γ _{ r }≥⋯≥γ _{ K }≥0. Since rank(H)=r, we have λ _{ r+1}=⋯=λ _{ K }=0.
By noticing that λ _{ r+1}=0, from (15) with i=r+1, we have σ _{ r+1}≤γ _{1}. Thus when λ _{ r }≥2ε _{th}≥2γ _{1}, we have σ _{ r+1}≤γ _{1}≤ε _{th}. Also from (15) with i=r, σ _{ r }≥λ _{ r }−γ _{1}≥2ε _{th}−γ _{1}≥ε _{th}. By noticing that σ _{ i }’s are in nonincreasing order, we can conclude that the rank detection result of Algorithm 1 is r, which is the correct detection.
where the last step is because that γ _{1}, the largest eigenvalue of \(\tilde {\mathbf {W}}\), is independent of both λ _{ r } and the rank of H.
By using (18) and (20) in (16), the lower bound in (13) is obtained.
3.2 Threshold optimization for Algorithm 1
Given Algorithm 1 and threshold ε _{th}, the overall probability of correct rank detection can be lower bounded by ϕ(ε _{th}). The derivations are as follows.
where in (22) and (23), the results in Proposition 1 have been used.
The optimization problem in (24) is onedimensional and can be optimally solved via exhaustive grid search. But there is a natural tradeoff between precision and computational complexity. For low computational complexity, in solving (24), we can find a zero point of d lnϕ(ε _{th})/d ε _{th} via bisection method and use it as the threshold. This lowcomplexity method can result in suboptimality when d lnϕ(ε _{th})/d ε _{th} has multiple zero points.
In this section, for the traditional singlethresholdbased rank detection, we rigorously derived an analytical lower bound on the correct rank detection probability, based on which a systematic threshold optimization scheme that maximizes this lower bound is proposed. The derived optimal threshold is adaptive to the number of transmit and receive antennas of the MIMO channel, the training length and power, and the distributions of the channel coefficients and the noise. However, it is independent of the instantaneous channel values or singular values of the channel matrix. Thus the threshold optimization can be conducted offline, which largely reduces the delay of channel rank detection in real applications.
It is noteworthy that the proposed method is not limited to Rayleigh fading channel model but can be extended to other distributions. Our lower bound calculation and threshold optimization need the distributions of the singular values of the channel matrix. For other channel fading models, even if no closedform expressions for the distributions of the singular values are available, numerical estimations of the singular value distributions can be obtained via simulation, and our method can still be used. Especially as our methods can be conducted offline, the computation complexity is not an issue in the realtime channel rank detection and channel estimation within a coherence interval of the channel.
Since γ _{ r }<γ _{1}, it can be shown straightforwardly that \(\tilde {\phi }_{r}(\epsilon _{\text {th}})>\phi _{r}(\epsilon _{\text {th}})\) and \(\tilde {\phi }(\epsilon _{\text {th}})>\phi (\epsilon _{\text {th}})\), which means that the new lower bounds are tighter, and we may obtain a better threshold by maximizing \(\tilde {\phi }(\epsilon _{\text {th}})\). However, the expression for \(\mathbb {P}(\gamma _{r} \leq \epsilon _{\text {th}})\) is much more complex than \(\mathbb {P}(\gamma _{1} \leq \epsilon _{\text {th}})\). Notice that it is the rth largest singular value of \(\tilde {\mathbf {W}}\), not the smallest singular value since \(\tilde {\mathbf {W}}\) has full rank with probability 1. Thus the calculation of the derivative of \(\tilde {\phi }(\epsilon _{\text {th}})\) is more involved, and the maximization of \(\tilde {\phi }(\epsilon _{\text {th}})\) has higher computational complexity. Our simulations show that the use of \(\tilde {\phi }(\epsilon _{\text {th}})\) improves the performance but the improvement is moderate. Thus, balancing the performance and computation complexity, we choose ϕ(ε _{th}) for our algorithms.
3.3 Difference to existing singlethresholdbased rank detection schemes
In this subsection, we explain the difference of our work with existing results on thresholdbased rank detection in [19, 20, 22, 23].
First, the research in [19, 20, 22, 23] are for realvalued channel matrix and realvalued noise matrix. Also, they consider the special case of T=M and S=I _{ m }. Our work applies for complexvalued channel matrix and complexvalued noise matrix, a general training length T where T≥M, and unitary T×M pilot S. In what follows, we explain existing threshold selections and calibrate the results to our model and notation.
In [23], lower and upper bounds on the threshold selection were provided. No specific threshold value or optimization method was given.
In all three aforementioned works, the proposed threshold selection methods depend on the distribution of the noise matrix only, where [19, 20] used the asymptotic behavior of the singular values and singular vectors of the noisy observation matrix when the dimensions of the channel matrix approach infinity; and Josse and Sardy [22] used the distribution of the largest singular value of the noise matrix. It was assumed in their work that the MIMO channel matrix is deterministic and their results cannot take advantage of the distribution of the channel matrix. On the contrary, we adopt a random channel model, and our threshold selection takes into account both the distribution of the channel coefficients and the distribution of the noises.
4 Improved multiplethreshold rank detection methods
To use the rank detection scheme in Algorithm 1 with the proposed threshold in the previous section, the a priori probabilities of the channel rank need to be known. However, for some wireless communication systems, the channel rank distribution may not be precisely known due to the mobility and complexity of the signal propagation environment. In this case, a rank detection algorithm that does not rely on the channel rank distribution is required. In addition, the lower bound on the probability of correct rank detection in (21) provides the average rank detection performance over all possible rank values. It may not be sharp enough for one channel realization with a specific rank value. Thus, in this section, we propose two improved rank detection algorithms which do not need the rank distribution.
4.1 Rank detection algorithm with multiple thresholds
where ϕ _{ i }(ε) is defined in (13).
To guarantee the existence of a rank detection result, we also need to consider the case that there is no element in \(\mathcal {I}\), i.e., \(\mathcal {I}=\emptyset \). The following claim is proved.
Claim 1.
 1.
there exists an integer D, 1≤D≤K, such that a _{ i }<b _{ i } for i<D, and a _{ i }≥b _{ i } for i≥D
 2.
a _{ i }<b _{ i } for all i.
Proof 2.
Assume that there exists no integer i (for 1≤i≤K−1) such that a _{ i }≥b _{ i } and a _{ i+1}<b _{ i+1}. We consider the two cases a _{ K }<b _{ K } and a _{ K }≥b _{ K } separately.
When a _{ K }<b _{ K }, if there exists an i such that a _{ i }≥b _{ i }, let i _{max} be the largest i satisfying a _{ i }≥b _{ i }. We have \(a_{i_{\max }}\ge b_{i_{\max }}\) and \(a_{i_{\max +1}}< b_{i_{\max +1}}\), which contradicts the assumption. Thus a _{ i }<b _{ i } for all i.
When a _{ K }≥b _{ K }, if there exists an i such that a _{ i }<b _{ i }, let i _{max} be the largest i satisfying a _{ i }<b _{ i }. Then, we will have a _{ i }<b _{ i } for all i≤i _{max}, based on the same reasoning as above. In this case, D=i _{max}+1. When there does not exist an i such that a _{ i }<b _{ i }, meaning a _{ i }≥b _{ i } for all i, D=1.
Based on Claim 1, when \(\mathcal {I}=\emptyset \), we have either \(\sigma _{i}< \epsilon ^{*}_{\text {th},i}\) for i<D and \(\sigma _{i}\ge \epsilon ^{*}_{\text {th},i}\) for i≥D (1≤D≤K), in which case the rank detection result should be K; or \(\sigma _{i}< \epsilon ^{*}_{\text {th},i}\) for all i=1,⋯,K, in which case the rank detection result should be 1, which is the lowest possible rank for the random Rayleigh fading channel matrix H.
Given these discussions, our second rank detection scheme with multiple thresholds is described in Algorithm 2.
4.2 Iterative rank detection algorithm with multiple thresholds
In our third rank detection algorithm, we iteratively use the K thresholds \(\epsilon _{\text {th},1}^{*},\cdots,\epsilon _{\text {th},K}^{*}\) defined in (25) to refine our rank detection threshold and detection result. In each iteration, single thresholdbased rank detection is performed and the threshold value is set using the rank detection result of the previous iteration. The iteration ends when the rank detection result of the current iteration is the same as the result of the previous one. More specifically, first, a rank detection result r _{1} is initialized, e.g., r _{1}=1, then \(\epsilon ^{*}_{\text {th},r_{1}}\) is used for the threshold of the next iteration. The new rank detection result is found using Algorithm 1, that is, the rank is detected as the maximum index i such that \(\sigma _{i}\ge \epsilon ^{*}_{\text {th},r_{1}}\) or 1 if all singular values of \(\tilde {\mathbf {Y}}\) are smaller than \(\epsilon ^{*}_{\text {th},r_{1}}\). This new rank detection result is denoted as r _{2}. If r _{2}≠r _{1}, \(\epsilon ^{*}_{\text {th},r_{2}}\) is used as the threshold for the next iteration, and a new rank detection result can be obtained. The scheme is described in Algorithm 3. Please note that lines 3–8 of Algorithm 3 are the same as those of Algorithm 1.
Claim 2.
If \(\epsilon ^{*}_{\text {th},1}\ge \epsilon ^{*}_{\text {th},2}\ge \cdots \ge \epsilon ^{*}_{\text {th},K}\), Algorithm 3 always converges.
Proof 3.
We prove the convergence by contradiction. Assume that the algorithm does not converge. From the algorithm, r _{2} is the new rank detection result when using threshold \(\epsilon ^{*}_{\text {th}, r_{1}}\). For any initial value for r _{1}, if r _{2}=r _{1}, Algorithm 3 converges and the rank detection result is r _{1}, which causes a contradiction. Next we consider the cases r _{2}>r _{1} and r _{2}<r _{1} separately.

Case A: r _{2}=K and \(\sigma _{K}\ge \epsilon ^{*}_{\text {th}, r_{1}}\); or

Case B: \(\sigma _{r_{2}}\ge \epsilon ^{*}_{\text {th}, r_{1}}\) and \(\sigma _{r_{2}+1}<\epsilon ^{*}_{\text {th}, r_{1}}\).
Since \(\epsilon ^{*}_{\text {th},i}\)’s are in nonincreasing order, for Case A, we have \(\sigma _{K}\ge \epsilon ^{*}_{\text {th}, r_{1}}\ge \epsilon ^{*}_{\text {th}, K}\). Thus the new rank detection result is K, i.e., r _{3}=K. Then r _{3}=r _{2} and Algorithm 3 terminates, which contradicts the assumption. For Case B, since r _{1}<r _{2}, we have \(\sigma _{r_{2}}\ge \epsilon ^{*}_{\text {th}, r_{1}}\ge \epsilon ^{*}_{\text {th}, r_{2}}\). Thus the next rank detection result cannot be smaller than r _{2}, i.e., r _{2}≤r _{3}. If r _{2}=r _{3}, Algorithm 3 terminates, which contradicts the assumption. Thus r _{2}<r _{3}. The same situation happens for the next iterations. So, if Algorithm 3 does not converge, we will find an infinite strictly increasing integer sequence r _{1}<r _{2}<r _{3}<⋯. This contradicts the fact that r _{ m }’s are in the range of [1,K].
Case 2: r _{1}>r _{2}. Similarly, we have either Case A (r _{2}=1 and \(\sigma _{1}<\epsilon ^{*}_{\text {th}, r_{1}}\)) or Case B listed above. For Case A, we have \(\sigma _{1}<\epsilon ^{*}_{\text {th}, r_{1}}\le \epsilon ^{*}_{\text {th}, 1}\). Thus the new rank detection result is 1, i.e., r _{3}=1 and Algorithm 3 terminates, which contradicts the assumption. For Case B, since r _{1}>r _{2}, we have \(\sigma _{r_{2}+1}<\epsilon ^{*}_{\text {th},r_{1}}\le \epsilon ^{*}_{\text {th}, r_{2}}\). Thus the next rank detection result cannot be larger than r _{2}, i.e., r _{3}≤r _{2}. If r _{2}=r _{3}, Algorithm 3 terminates, which contradicts the assumption. Thus r _{2}<r _{3}. The same situation happens for the next iterations. So, if Algorithm 3 does not converge, we will find an infinite strictly decreasing integer sequence r _{1}>r _{2}>r _{3}>⋯. This contradicts the fact that r _{ m }’s are in the range of [1,K].
Claim 2 shows that when the thresholds corresponding to the K rank values are in nonincreasing order, Algorithm 3 is guaranteed to converge. Also, from the proof of Claim 2, we can see that the convergence is guaranteed within K iterations. Our limited simulation results indicate that the algorithm converges very fast (within 2–3 iterations). Intuitively, as \(\epsilon ^{*}_{\text {th}, i}\) is the threshold when the channel rank is i and the singular values are in a nonincreasing order, it is natural to have \(\epsilon ^{*}_{\text {th},1}, \epsilon ^{*}_{\text {th},2}, \cdots, \epsilon ^{*}_{\text {th},K}\) in a nonincreasing order. However, we cannot prove this analytically. When violation of the ordering happens on one threshold, we can simply reset the threshold to be the average of the one before and the one after to fix the ordering problem and have guaranteed convergence.^{1}
Although the convergence of Algorithm 3 is guaranteed, we cannot not guarantee the uniqueness of the rank detection solution with respect to different initial values for r _{1}. In other words, for different initial rank values, the algorithm may converge to different solutions. An example is as follows. Assume that \(\sigma _{1}>\epsilon ^{*}_{\text {th},1}> \sigma _{2}>\epsilon ^{*}_{\text {th},2}>\cdots >\sigma _{K}>\epsilon ^{*}_{\text {th},K}\). Then the final rank detection result of Algorithm 3 will equal to r _{1} for any initial r _{1} value.
4.3 Discussion on complexity
The computational complexity of Algorithms 2 and 3 is composed of two parts: the calculations of \(\epsilon ^{*}_{\text {th},1},\cdots,\epsilon ^{*}_{\text {th},K}\) and the rank detection part.
The optimization of \(\epsilon ^{*}_{\text {th},i}\) only depends on the dimensions of the channel matrix M and N, the training time T, and the training power P. It is independent of the channel realization of each coherence interval. Thus the optimization can be conducted offline. Further, the following lemma is proved which can be used to reduce the computational complexity of the optimization.
Lemma 1.
The function ϕ _{ r }(ε _{th}) in (13) is a logconcave function of ε _{th}.
Proof 4.
For the Hermitian matrices F ^{(1)}(μ) and F ^{(r)}(μ), we can show that all their leading principal minors are positive when μ>0 from the definitions in (11), (12), and the CDFs in (17), (19). Thus the two matrices are positive definite and \(\det \left (\boldsymbol {F}^{(1)}\left (\frac {PT}{M}\epsilon _{\text {th}}^{2} \right) \right)\) and \(\det \left (\boldsymbol {F}^{(r)}\left (4\epsilon _{\text {th}}^{2} \right)\right)\) are logconcave functions since the determinant of a positive definite matrix is logconcave [31]. Based on [31], the product of logconcave functions is also logconcave. This ends the proof.
Next, we analyze the complexity of the rank detection part. For Algorithm 2, the total number of comparisons in the worst scenario is 2K, thus the complexity is \(\mathcal {O}(K)\). For Algorithm 3, in the worst case, the number of iterations is K; and for each iteration, at most K+1 comparisons are needed. The overall number of comparisons is K(K+1). Thus the complexity is \(\mathcal {O}(K^{2})\). Notice that K= min{M,N}. Even for massive MIMO systems with N base station antennas and M singleantenna users, where N is large (e.g., hundreds), the complexities of the two proposed rank detection algorithms are linear and quadratic in the number of users, respectively.
5 Simulation results
5.1 Simulation on the probability of correct rank detection
In this section, simulation results are shown for Algorithm 1 with our proposed threshold optimization in Section 3, and the two new rank detection algorithms with multiple thresholds, Algorithm 2 and Algorithm 3, proposed in Section 4. We simulate the probability of correct rank detection for different parameters, such as the average training power P, the training length T, and the numbers of transmitter and receiver antennas M and N. In our simulations, channel coefficients are generated as Rayleigh fading following the model in (3). S is generated as a random T×M unitary matrix following the isotropic distribution. For comparison, we also show the rank detection accuracy of Algorithm 1 with the threshold values proposed in [19, 20, 22]. While, the results in [19, 20, 22] are for T=M and S=I _{ M }, we extend them for a general T≥M and unitary S as explained in Section 3.3.
5.2 Simulation on MSE and outage probability
which is the MSE per channel coefficient. Traditionally, the average in (29) is over the noises. But here it is over both the noises and the channels, since the channel coefficients are also random variables. In the conducted MonteCarlo simulation, a distinct channel realization is used for each iteration.
where \({\sigma _{n}^{2}}\) is the noise variance. In the following simulations, we assume that the noises have unit variance, i.e., \({\sigma _{n}^{2}}=1\).
6 Conclusions
In this paper, we proposed novel thresholdbased rank detection algorithms for reducedrank MIMO systems. Different from previous work, we consider a MIMO system with a random channel matrix model, a general training length, and a unitary training matrix. Lower bounds on the probability of correct rank detection were derived using the distribution of the channel matrix and noise matrix, based on which the rank detection thresholds can be optimized. In addition to the traditional singlethreshold detection algorithm, we further proposed two lowcomplexity multiplethreshold algorithms. Compared with the existing schemes, our proposed schemes can achieve higher rank detection rate for various scenarios. Our simulation results also show that better rank detection can improve the channel estimation quality and the outage performance of the MIMO system.
7 Endnotes
^{1} In simulations, the thresholds we numerically obtain from (25) are in nonincreasing order, except when the MIMO channel dimension gets large. When the values of M and N are large, violation of the nonincreasing order occasionally happens due to the limited precision of computer calculation.
^{2} The cases where the channel rank is 7 and 8 are not considered, since based on the the experimental results on several scenarios in [33], the rank of the 8×8 channel matrix is always less than 7.
^{3} For very small rank values, e.g., 1,2,3,4, our schemes are inferior to existing ones.
^{4} For example, for a 8×8 MIMO system whose channel rank is uniformly distributed between 1 and 8, under this setup, a lower bound on the outage probability will be \(\sum _{i=0}^{7}i/8/8=43.75\,\%\). Another way to show the comparison is to draw the simulated outage probability values subtracted by the lower bound. But either way, the same comparison result can be obtained.
Declarations
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 GJ Foschini, Layered spacetime architecture for wireless communication in a fading environment when using multielement antennas. Bell. Labs Tech. J. 1(2), 41–59 (1996).View ArticleGoogle Scholar
 IE Telatar, Capacity of multiantenna gaussian channel. Eur. Trans. Telecom. 10(6), 585–595 (1999).View ArticleGoogle Scholar
 AG Burr, Capacity bounds and estimates for the finite scatterers mimo wireless channel. IEEE J. Sel. Areas. Commun. 21(5), 812–818 (2003).View ArticleGoogle Scholar
 HQ Ngo, EG Larsson, TL Thomas, The multicell multiuser mimo uplink with very large antenna arrays and a finitedimensional channel. IEEE Trans. Commun. 61(6), 2350–2361 (2013).View ArticleGoogle Scholar
 F Rusek, D Persson, BK Lau, EG Larsson, TL Marzetta, O Edfors, F Tufvesson, Scaling up mimo: Opportunities and challenges with very large arrays. IEEE Signal Process. Mag. 30(1), 40–60 (2013).View ArticleGoogle Scholar
 JS Goldstein, IS Reed, LL Scharf, A multistage representation of the wiener filter based on orthogonal projections. IEEE Trans. Inf. Theory. 44(7), 2943–2959 (1998).MATHMathSciNetView ArticleGoogle Scholar
 ML Honig, JS Goldstein, Adaptive reducedrank interference suppression based on the multistage wiener filter. IEEE Trans. Commun. 50(6), 986–994 (2002).View ArticleGoogle Scholar
 RC de Lamare, M Haardt, R SampaioNeto, Blind adaptive constrained reducedrank parameter estimation based on constant modulus design for CDMA interference suppression. IEEE Trans. Signal Process. 56(6), 2470–2482 (2008).MathSciNetView ArticleGoogle Scholar
 Y Sun, V Tripathi, ML Honig, Adaptive turbo reducedrank equalization for mimo channels. IEEE Trans. Wireless Commun. 4(6), 2789–2800 (2005).View ArticleGoogle Scholar
 H Qian, SN Batalama, Data recordbased criteria for the selection of an auxiliary vector estimator of the MMSE/MVDR filter. IEEE Trans. Commun. 51(10), 1700–1708 (2003).View ArticleGoogle Scholar
 RC de Lamare, R SampaioNeto, Adaptive reducedrank processing based on joint and iterative interpolation, decimation, and filtering. IEEE Trans. Signal Process. 57(7), 2503–2514 (2009).MathSciNetView ArticleGoogle Scholar
 RC de Lamare, R SampaioNeto, Adaptive reducedrank equalization algorithms based on alternating optimization design techniques for mimo systems. IEEE Trans. Veh. Tech. 60(6), 2482–2494 (2011).View ArticleGoogle Scholar
 Y Hua, M Nikpour, P Stoica, Optimal reducedrank estimation and filtering. IEEE Trans. Signal Process. 49(3), 457–469 (2001).View ArticleGoogle Scholar
 P Stoica, M Viberg, Maximum likelihood parameter and rank estimation in reducedrank multivariate linear regressions. IEEE Trans. Signal Process. 44(12), 3069–3078 (1996).View ArticleGoogle Scholar
 Y Jing, X Yu, Mlbased channel estimations for nonregenerative relay networks with multiple transmit and receive antennas. IEEE J. Sel. Areas Commun. 30(8), 1428–1439 (2012).View ArticleGoogle Scholar
 E Lindskog, C Tidestav, in Proc. IEEE 49th Veh. Tech. Conf.: Jul. 1999, 2. Reduced rank channel estimation (IEE,Houston, TX, 1999), pp. 1126–1130.Google Scholar
 M Nicoli, U Spagnolini, Reducedrank channel estimation for timeslotted mobile communication systems. IEEE Trans. Signal Process. 53(3), 926–944 (2005).MathSciNetView ArticleGoogle Scholar
 AA Shabalin, AB Nobel, Reconstruction of a lowrank matrix in the presence of gaussian noise. J. Multivariate Anal. 118:, 67–76 (2013).MATHMathSciNetView ArticleGoogle Scholar
 M Gavish, DL Donoho, Optimal shrinkage of singular values. (arXiv preprint, 2015). http://arxiv.org/abs/1405.7511. Accessed 14 Oct. 2015.
 DL Donoho, M Gavish, The optimal hard threshold for singular values is 4/sqrt(3). IEEE Trans. Inf. Theory. 60(8), 5040–5053 (2014).Google Scholar
 R Nadakuditi, Optshrink: An algorithm for improved lowrank signal matrix denoising by optimal, datadriven singular value shrinkage. IEEE Trans. Inf. Theory. 60(5), 3002–3018 (2014).MathSciNetView ArticleGoogle Scholar
 J Josse, S Sardy, Adaptive shrinkage of singular values. Statistics and Computing (2015). doi:10.1007/s1122201595549.
 K Konstantinides, K Yao, Statistical analysis of effective singular values in matrix rank determination. IEEE Trans. Acoust., Speech, Signal Process. 36(5), 757–763 (1988).MATHView ArticleGoogle Scholar
 B Hassibi, BM Hochwald, How much training is needed in multipleantenna wireless links?IEEE Trans. Inf. Theory. 49:, 951–961 (2003).MATHView ArticleGoogle Scholar
 N Shariati, J Wang, M Bengtsson, Robust training sequence design for correlated mimo channel estimation. IEEE Trans. Signal Process. 61(1), 107–120 (2014).View ArticleGoogle Scholar
 C Huang, T Chang, X Zhou, YWP Hong, Twoway training for discriminatory channel estimation in wireless mimo systems. IEEE Trans. Signal Process. 61(10), 2724–2738 (2013).MathSciNetView ArticleGoogle Scholar
 M Biguesh, AB Gershman, Trainingbased mimo channel estimation: A study of estimator tradeoffs and optimal training signals. IEEE Trans. Signal Process. 54(3), 884–893 (2006).View ArticleGoogle Scholar
 C Eckart, G Young, The approximation of one matrix by another of lower rank. Psychometrika. 1(3), 211–218 (1936).Google Scholar
 M Nicoli, MultiUser Reduced Rank Receivers for TD/CDMA Systems, Dissertation, PhD thesis, (Politecnico di Milano, Milan, Italy, 2001).Google Scholar
 A Ordonez, D Palomar, J Fonollosa, Ordered eigenvalues of a general class of Hermitian matrices with application to the performance analysis of mimo systems. IEEE Trans. Signal Process. 57(2), 672–689 (2009).MathSciNetView ArticleGoogle Scholar
 S Boyd, L Vandenberghe, Convex Optimization (Cambridge University Press, Cambridge, 2004).MATHView ArticleGoogle Scholar
 Q Li, G Li, W Lee, M Lee, D Mazzarese, B Clerckx, Z Li, Mimo techniques in wimax and LTE: a feature overview. IEEE Commun. Mag. 48(5), 86–92 (2010).View ArticleGoogle Scholar
 A Molisch, H Asplund, R Heddergott, M Steinbauer, T Zwick, The cost259 directional channel modelpart i: Overview and methodology. IEEE Trans. Wireless Commun. 5(12), 3421–3433 (2006).View ArticleGoogle Scholar