### 2.1. System Model and Problem Formulation

In this section, we focus on the downlink of the multiuser cooperative MIMO relay system as illustrated in Figure 1. Assuming half duplex relaying, the scenario under analysis consists of a base station , a relay station and mobile station transmitting through two orthogonal channels, for instance, two separate time slots as time division multiple access (TDMA). During the first slot, The BS deployed with transmit antennas communicates with the fixed RS that has antennas and the MSs,each of which has single antenna. A MIMO channel denoted by is thus created between the BS and the RS while a MIMO broadcast channel (MIMO ) denoted by is also established. The precoding strategy at the BS includes an encoding operation and a subsequent linear operation with a filter matrix . The BS encodes data streams that are targeted to the MSs and broadcasts it to the RS and the . The RS processes the received signal with a filter matrix , and then forwards the data streams to the MSs through a MIMO BC denoted by in the second slot. Finally, in the cooperative scenario, each of the MSs combines the signals from the direct path and the relay path that are received in the first slot and the second slot, respectively. Note that all the matrices in this paper are assumed full rank for simplicity.

During the first slot, the signal model for the direct path of the proposed system in downlink is

where and is a zero-mean complex Gaussian noise vector received at the MSs with covariance matrix . Also, denotes a zero-mean complex Gaussian vector whose covariance matrix is , which indicates that uncorrelated data streams are transmitted.

During the second slot, assuming is the received signal at MS and , the signal model for the relay path of the proposed downlink system is

where and are zero-mean complex Gaussian noise vector received at the RS and MSs with covariance matrices and , respectively. In addition, we assume the signal and noise are uncorrelated as well. The assumptions with the afore-mentioned signal and noise can be summarized as

Then, the signal and are normalized as

where the scalar and can be interpreted as *automatic gain control* that are necessary to give reasonable expressions for the MSE in any real MIMO system.

Finally, we combine the signals from both the paths to get

Therefore, the optimization problem based on MSE can be formulated as

where we assume that BS and RS use the whole available average transmit power, that is, and , respectively. Since the transmitted signal from BS is Fs and the transmitted signal from the RS is , by using the assumption (3) simultaneously, the power constraints can be obtained.

However, from the following explicit expression of the objective function, it can be seen that the problem (7) is too complex to be solved optimally:

Hence, we separate it to be several independent subproblems as the following sections produce.

### 2.2. Filter Optimization for Direct Path

Based on the signal model (1) and (4), we first propose the optimization problem for the direct path as

As the direct path is actually a conventional MIMO link, a closed form solution is found for the optimization in [17]

where we define

Thus the optimal result for problem (10) is obtained.

### 2.3. Filter Optimization for Relay Path

For the relay path, the MSE is given by

Then using (2) in (15), the optimization problem for the relay path is formulated as

Here, note that

#### 2.3.1. Local Optimal Joint (OJ) MMSE Scheme

Aiming at the optimal solution of the problem (16), we can find necessary conditions for the transmit filter , the relay filter **,** and the weight by constructing the Lagrange function

with the Lagrange multiplier , and setting its derivative to zero:

where we use and . By introducing , the structure of the resulting relay filter follows from (21):

with

where the power constraint at the relay is used.

Applying (23) into (21), we get

which follows that

where and the power constraint at the relay are used.

Hence, using (23) and (25) in (22), we obtain that . Therefore, the filter matrix can be expressed as the function of the transmit matrix for the optimization in (16):

where we define

Similarly, the expression of the transmit filter matrix in terms of the relay filter matrix can be derived as

where we define

From the above results, it is obviously seen that and are functions of each other. Therefore, the solutions and for the problem (16) can be obtained via the following iterative procedures.

(1)Initialize the transmit filter matrix , satisfying the transmit power constraint.

(2)Calculate the relay filter matrix with the given according to (26).

(3)Calculate the transmit filter matrix with the new according to (29).

(4)Go back to **2** until convergence to get and .

Although the MSE function in (15) is not jointly convex on both the transmit filter matrix and the relay filter matrix, it is convex over either of them. This guarantees that the proposed iterative algorithm could at least converge on a local minimum.

#### 2.3.2. Suboptimal Joint (SOJ) MMSE Scheme

In this subsection, we present a simplified closed form solution to the suboptimal structure of and , in that the optimal scheme proposed above involves a complex iterative algorithm which is not quite practical in real systems.

First, we ignore the scalar and the power constraint at the relay for simplicity, and the problem can be changed into

Let the singular value decomposition (SVD) of be . Here, for simplicity of the derivation, we assume . Thus, is a diagonal matrix of singular values while and are square and unitary matrices. Then, our main theories are described as follows.

Theorem 1.

The objective of problem (32) can achieve its minimum when the BS filter and the relay filter are constructed as follows:

where and are diagonal matrices.

Proof.

The standard Lagrange multiplier technique, which is similar to that in the last section, is used to solve the optimization problem formulated in (32). By setting the derivative of the cost function to zero, we get

where is the Lagrange multiplier.

Supposing , the afore-mentioned two equations can be arranged as

which implies is Hermitian.

Thus, combining (35) and (36) gives

which follows that

where is a unitary matrix. Using (38) in (35), we have

Premultiply the equation by and postmultiply by to get

Let , and substituting the SVD of and in (40),we have

Since is Hermitian from (40), . Applying it in the afore-mentioned equation we get

which implies must be diagonal. Hence,

can be obtained, since no other matrices satisfy the property. Note that is a permutation matrix.

Similarly, we can also yield that

Substitute the SVD of and in (37) to get

Using uniqueness of SVD, we have

Without loss of generality, set the permutation matrix as . Then, using (43), (44), and (46) in (15), the MSE expression becomes

Since the trace of matrix depends only on its singular values, can be chosen to be any unitary matrix (e.g., ) without affecting the MSE. Therefore, we have

which leads to the desired result (33).

Theorem 2.

The optimum MMSE power allocation policy can be expressed as

Proof.

Using (43) and (44) in (42), we have

which produces the desired water filling result (49). Besides, from (45), (50) can be easily obtained.

Therefore, the filter matrices , and the scalar can be obtained via the following steps. First, calculate and according to Theorems 1 and 2. Then, let the relay filter matrix , where is chosen to meet the relay power constraint. In addition, the scalar is set to be equal to . Thus, the solutions of , , and form a suboptimal scheme for the optimization problem (32), which is simpler than the local optimal scheme.

### 2.4. Filter Design Schemes for Cooperative Scenario

After the signal from the direct path and the relay path and are obtained, the optimization problem for combining them based on minimizing the MSE is formulated as

By applying the standard Lagrange multiplier technique, the optimal weighing parameters are written as

where we assume and .

As is known, we are unable to find the optimal solution for problem (7). Then based on the optimal results for the subproblems (10), (16), and (52), we propose two schemes to approach the optimal results.

Cooperative Scheme 1 (CS1)

In this scheme, we first present the transmit filter matrix from view of the direct path, that is, . Then, based on (26), the relay filter is fixed. Besides, the scalar and at the MSs can be easily obtained by using (13) and (27), respectively. Conditioned upon the results above, the covariance matrix and can be worked out to get the weight and . Therefore, the above solutions form *Cooperative scheme* *1* for the downlink of proposed multiuser cooperative MIMO-relay systems.

Cooperative Scheme 2 (CS2)

Alternatively, this scheme takes the relay path into account primarily. Namely, the transmit filter matrix and the relay filter matrix follow the result deduced for the relay path, which is written as and . Then the scalar and can be calculated accordingly to normalize the received signal at the MSs. Similar with that in *Cooperative scheme* *1*, the weight and are obtained. Thus the *Cooperative scheme* *2* is created.

In summary, all the schemes above are useful for different scenarios. As we know, there may be three kinds of users in relaying networks: direct users, pure relay users, and cooperative users. The direct users communicate with the BS directly and can use the filter design results in Section 2.2. For the pure relay users, they receive the data stream signal only from the relay path neglecting the direct link. These users can adopt the filter design results in Section 2.3. The cooperative users are those who combine the signal from the direct path in first time slot and the signal from the relay path in the second time slot. For these users, we propose two different filter design schemes in Section 2.4.