Joint Linear Filter Design in Multiuser Cooperative Nonregenerative MIMO Relay Systems

Abstract

This paper addresses the filter design issues for multiuser cooperative nonregenerative MIMO relay systems in both downlink and uplink scenarios. Based on the formulated signal model, the filter matrix optimization is first performed for direct path and relay path respectively, aiming to minimize the mean squared error (MSE). To be more specific, for the relay path, we derive the local optimal filter scheme at the base station and the relay station jointly in the downlink scenario along with a more practical suboptimal scheme, and then a closed-form joint local optimal solution in the uplink scenario is exploited. Furthermore, the optimal filter for the direct path is also presented by using the exiting results of conventional MIMO link. After that, several schemes are proposed for cooperative scenario to combine the signals from both paths. Numerical results show that the proposed schemes can reduce the bit error rate (BER) significantly.

1. Introduction

Wireless relays are essential to provide reliable transmission, high-throughput, and broad coverage for next-generation wireless networks [1]. Deploying a relay between a source and a destination cannot only overcome shadowing due to obstacles but also reduce the required transmitted power from the source and hence interference to neighboring nodes. Relays can be regenerative [2] or nonregenerative [3]. The former employs decode-and-forward scheme and regenerates the original information from the source. The latter employs amplify-and-forward scheme, which only performs linear processing for the received signal and then transmits to the destinations. As a result of the above difference, a nonregenerative relay generally causes smaller delay than a regenerative relay.

MIMO techniques are well studied to promise significant improvements in terms of spectral efficiency and link reliability. In [4, 5], the capacity of point-to-point MIMO channel is investigated and extensive work on multi-user MIMO has been done for a decade [6]. Therefore, combined with the above two technologies, a novel system called MIMO-relay emerges to accommodate users with high data rate requests and extend the network coverage. Recently, there is a vigorous body of work on MIMO-relay systems [7–15]. For example, [7, 8] derives upper bounds and lower bounds for the capacity of MIMO-relay channels. In [9], the optimal design of non-regenerative MIMO relays is investigated. Assuming relays and receivers with multiple antennas, the optimal relay matrix that maximizes the capacity between the source and destination is developed when a direct link is not considered or is negligible. The same problem is studied in [10], and [11] extends the work to partial channel state information (CSI) scenario.

Despite significant research efforts and advances on MIMO relay systems, most of the aforementioned research is based on a point-to-point scenario with a single user equipped with multiple antennas. In practical systems, however, each relay will need to support multiple users. This motivates us to study multiuser MIMO-relay systems, where the relay forwards data to multiple users. The most different feature between the researches on the single-user (with multiantenna) and multiuser (with single antenna) system is that the signals of the multiple users cannot be cooperatively pretransformed (e.g., uplink of a cellular system) or posttransformed (e.g., downlink of a cellular system). While single-user MIMO-relay systems have been a primary focus of prior research, a few researchers begin to pay attention to multiuser scenario as well. In [16], the optimal design of nonregenerative relays for multiuser MIMO-relay systems based on sum rate is investigated. Assuming zero-forcing dirty paper coding at the base station and linear operations at the relay station , it proposes upper and lower bounds on the achievable sum rate, neglecting the direct links from the BS to the users.

In this paper, we consider the problem of joint linear optimization for both downlink and uplink in multiuser cooperative nonregenerative MIMO-relay systems based on MSE criterion, which is different from the sum rate criterion in [16]. The MSE criterion is motivated by robustness to channel estimation errors and a lower implementation complexity. Then our main contributions are as follows.

(i)We derive the optimal joint design of the BS and RS filter matrices that achieves the minimum mean squared error (MMSE) for both downlink and uplink of the multiuser MIMO relay systems at the absence of direct path.

(ii)We propose several schemes for the design of the BS and RS filter matrices based on MSE criterion in the presence of direct path, which is called cooperative scenario in this paper.

(iii)We compare different schemes for direct-path-only scenario, relay-path-only scenario and cooperative scenario, and the numerical results are provided to show the effectiveness joint filter design and cooperative combine operation.

The rest of this paper is organized as follows. Sections 2 and 3 formulate the system model and propose the joint filter design schemes for downlink and uplink of multiuser MIMO relay systems, respectively. Numerical results are given in Section 4. Finally, Section 5 concludes this paper.

Notations

Boldface capital letters and boldface small letters denote matrices and vectors, respectively. Superscripts , and stand for the conjugate, transpose, and complex conjugated transpose operation, respectively., while and represent inversion and pseudoinversion of matrices. Also, and denote the expectation and trace operation, respectively, and, finally, is the identity matrix.

2. Downlink Systems

2.1. System Model and Problem Formulation

In this section, we focus on the downlink of the multiuser cooperative MIMO relay system as illustrated in Figure 1. Assuming half duplex relaying, the scenario under analysis consists of a base station , a relay station and mobile station transmitting through two orthogonal channels, for instance, two separate time slots as time division multiple access (TDMA). During the first slot, The BS deployed with transmit antennas communicates with the fixed RS that has antennas and the MSs,each of which has single antenna. A MIMO channel denoted by is thus created between the BS and the RS while a MIMO broadcast channel (MIMO ) denoted by is also established. The precoding strategy at the BS includes an encoding operation and a subsequent linear operation with a filter matrix . The BS encodes data streams that are targeted to the MSs and broadcasts it to the RS and the . The RS processes the received signal with a filter matrix , and then forwards the data streams to the MSs through a MIMO BC denoted by in the second slot. Finally, in the cooperative scenario, each of the MSs combines the signals from the direct path and the relay path that are received in the first slot and the second slot, respectively. Note that all the matrices in this paper are assumed full rank for simplicity.

Figure 1

Multiuser cooperative MIMO relay system model in downlink.

During the first slot, the signal model for the direct path of the proposed system in downlink is

(1)

where and is a zero-mean complex Gaussian noise vector received at the MSs with covariance matrix . Also, denotes a zero-mean complex Gaussian vector whose covariance matrix is , which indicates that uncorrelated data streams are transmitted.

During the second slot, assuming is the received signal at MS and , the signal model for the relay path of the proposed downlink system is

(2)

where and are zero-mean complex Gaussian noise vector received at the RS and MSs with covariance matrices and , respectively. In addition, we assume the signal and noise are uncorrelated as well. The assumptions with the afore-mentioned signal and noise can be summarized as

(3)

Then, the signal and are normalized as

(4)

(5)

where the scalar and can be interpreted as automatic gain control that are necessary to give reasonable expressions for the MSE in any real MIMO system.

Finally, we combine the signals from both the paths to get

(6)

Therefore, the optimization problem based on MSE can be formulated as

(7)

(8)

where we assume that BS and RS use the whole available average transmit power, that is, and , respectively. Since the transmitted signal from BS is Fs and the transmitted signal from the RS is , by using the assumption (3) simultaneously, the power constraints can be obtained.

However, from the following explicit expression of the objective function, it can be seen that the problem (7) is too complex to be solved optimally:

(9)

Hence, we separate it to be several independent subproblems as the following sections produce.

2.2. Filter Optimization for Direct Path

Based on the signal model (1) and (4), we first propose the optimization problem for the direct path as

(10)

(11)

As the direct path is actually a conventional MIMO link, a closed form solution is found for the optimization in [17]

(12)

(13)

where we define

(14)

Thus the optimal result for problem (10) is obtained.

2.3. Filter Optimization for Relay Path

For the relay path, the MSE is given by

(15)

Then using (2) in (15), the optimization problem for the relay path is formulated as

(16)

(17)

Here, note that

(18)

2.3.1. Local Optimal Joint (OJ) MMSE Scheme

Aiming at the optimal solution of the problem (16), we can find necessary conditions for the transmit filter , the relay filter , and the weight by constructing the Lagrange function

(19)

with the Lagrange multiplier , and setting its derivative to zero:

(20)

(21)

(22)

where we use and . By introducing , the structure of the resulting relay filter follows from (21):

(23)

with

(24)

where the power constraint at the relay is used.

Applying (23) into (21), we get

which follows that

(25)

where and the power constraint at the relay are used.

Hence, using (23) and (25) in (22), we obtain that . Therefore, the filter matrix can be expressed as the function of the transmit matrix for the optimization in (16):

(26)

(27)

where we define

(28)

Similarly, the expression of the transmit filter matrix in terms of the relay filter matrix can be derived as

(29)

(30)

where we define

(31)

From the above results, it is obviously seen that and are functions of each other. Therefore, the solutions and for the problem (16) can be obtained via the following iterative procedures.

(1)Initialize the transmit filter matrix , satisfying the transmit power constraint.

(2)Calculate the relay filter matrix with the given according to (26).

(3)Calculate the transmit filter matrix with the new according to (29).

(4)Go back to 2 until convergence to get and .

Although the MSE function in (15) is not jointly convex on both the transmit filter matrix and the relay filter matrix, it is convex over either of them. This guarantees that the proposed iterative algorithm could at least converge on a local minimum.

2.3.2. Suboptimal Joint (SOJ) MMSE Scheme

In this subsection, we present a simplified closed form solution to the suboptimal structure of and , in that the optimal scheme proposed above involves a complex iterative algorithm which is not quite practical in real systems.

First, we ignore the scalar and the power constraint at the relay for simplicity, and the problem can be changed into

(32)

Let the singular value decomposition (SVD) of be . Here, for simplicity of the derivation, we assume . Thus, is a diagonal matrix of singular values while and are square and unitary matrices. Then, our main theories are described as follows.

Theorem 1.

The objective of problem (32) can achieve its minimum when the BS filter and the relay filter are constructed as follows:

(33)

where and are diagonal matrices.

Proof.

The standard Lagrange multiplier technique, which is similar to that in the last section, is used to solve the optimization problem formulated in (32). By setting the derivative of the cost function to zero, we get

(34)

where is the Lagrange multiplier.

Supposing , the afore-mentioned two equations can be arranged as

(35)

(36)

which implies is Hermitian.

Thus, combining (35) and (36) gives

(37)

which follows that

(38)

where is a unitary matrix. Using (38) in (35), we have

(39)

Premultiply the equation by and postmultiply by to get

(40)

Let , and substituting the SVD of and in (40),we have

(41)

Since is Hermitian from (40), . Applying it in the afore-mentioned equation we get

(42)

which implies must be diagonal. Hence,

(43)

can be obtained, since no other matrices satisfy the property. Note that is a permutation matrix.

Similarly, we can also yield that

(44)

Substitute the SVD of and in (37) to get

(45)

Using uniqueness of SVD, we have

(46)

Without loss of generality, set the permutation matrix as . Then, using (43), (44), and (46) in (15), the MSE expression becomes

(47)

Since the trace of matrix depends only on its singular values, can be chosen to be any unitary matrix (e.g., ) without affecting the MSE. Therefore, we have

(48)

which leads to the desired result (33).

Theorem 2.

The optimum MMSE power allocation policy can be expressed as

(49)

(50)

Proof.

Using (43) and (44) in (42), we have

(51)

which produces the desired water filling result (49). Besides, from (45), (50) can be easily obtained.

Therefore, the filter matrices , and the scalar can be obtained via the following steps. First, calculate and according to Theorems 1 and 2. Then, let the relay filter matrix , where is chosen to meet the relay power constraint. In addition, the scalar is set to be equal to . Thus, the solutions of , , and form a suboptimal scheme for the optimization problem (32), which is simpler than the local optimal scheme.

2.4. Filter Design Schemes for Cooperative Scenario

After the signal from the direct path and the relay path and are obtained, the optimization problem for combining them based on minimizing the MSE is formulated as

(52)

By applying the standard Lagrange multiplier technique, the optimal weighing parameters are written as

(53)

where we assume and .

As is known, we are unable to find the optimal solution for problem (7). Then based on the optimal results for the subproblems (10), (16), and (52), we propose two schemes to approach the optimal results.

Cooperative Scheme  1 (CS1)

In this scheme, we first present the transmit filter matrix from view of the direct path, that is, . Then, based on (26), the relay filter is fixed. Besides, the scalar and at the MSs can be easily obtained by using (13) and (27), respectively. Conditioned upon the results above, the covariance matrix and can be worked out to get the weight and . Therefore, the above solutions form Cooperative scheme  1 for the downlink of proposed multiuser cooperative MIMO-relay systems.

Cooperative Scheme  2 (CS2)

Alternatively, this scheme takes the relay path into account primarily. Namely, the transmit filter matrix and the relay filter matrix follow the result deduced for the relay path, which is written as and . Then the scalar and can be calculated accordingly to normalize the received signal at the MSs. Similar with that in Cooperative scheme  1, the weight and are obtained. Thus the Cooperative scheme  2 is created.

In summary, all the schemes above are useful for different scenarios. As we know, there may be three kinds of users in relaying networks: direct users, pure relay users, and cooperative users. The direct users communicate with the BS directly and can use the filter design results in Section 2.2. For the pure relay users, they receive the data stream signal only from the relay path neglecting the direct link. These users can adopt the filter design results in Section 2.3. The cooperative users are those who combine the signal from the direct path in first time slot and the signal from the relay path in the second time slot. For these users, we propose two different filter design schemes in Section 2.4.

3. Uplink Systems

3.1. System Model and Problem Formulation

In uplink systems, we also assume nonregenerative MIMO-relay system with direct link as depicted in Figure 2. As shown in downlink systems, there are also one BS equipped with antennas, one RS with antennas and users each of which with single antenna in uplink systems. During the first slot, the users transmit data streams, respectively, to the BS and the RS through two independent MIMO access channels (MIMO AC) denoted by and . The RS processes the received signal by , and then transmits to the BS in the second slot. The channel between them is a traditional MIMO link . Multiplied by the filter matrix , the signal from two slots is decoded to be data streams at the .

Figure 2

Multiuser cooperative MIMO relay system model in uplink.

Similar with that in downlink systems, the signal model for the direct path of proposed systems in uplink is

(54)

where is a zero-mean complex Gaussian noise vector received at the BS with covariance matrix . Also, denotes a zero-mean complex Gaussian vector whose covariance matrix is , which indicates uncorrelated data streams with equal power are transmitted. Note that is the total transmit power for all the . Here, is the filter matrix at the BS for the direct path.

Then the signal model for the relay path of the proposed multiuser nonregenerative MIMO-relay system in uplink is given by

(55)

where and are zero-mean complex Gaussian noise vectors received at the BS and RS with covariance matrices and , respectively. Also, is the filter matrix at the BS for the relay path. The afore-mentioned assumptions can be expressed as

(56)

where is defined.

Finally, we combine the signals from both the paths to get

(57)

that is,

(58)

where is assumed.

Therefore the optimization problem based on MSE can be formulated as

(59)

where we assume that the RS uses the whole available average transmit power .

3.2. Filter Optimization for Direct Path

Based on the signal model (54), we first propose the optimization problem for the direct path as

(60)

whose optimal solution can be expressed as [18]

(61)

3.3. Filter Optimization for Relay Path

For the relay path, the MSE is given by

(62)

Then using (55) in (62), the optimization problem for the relay path is formulated as

(63)

(64)

where we assume that the RS uses the whole available average transmit power .

As discussed in downlink systems, the Lagrange function is constructed as

(65)

with the Lagrange multiplier and by setting its derivative, we have

(66)

(67)

Obviously, and are function of each other. Iterative algorithms can be applied to get the optimal solution. However, it is too complex to be practical. Thus, a close-form solution will be derived in the following. Before the derivation, we introduce a useful lemma first [19] as follows.

Lemma 1.

If and are both Hermitian, there exists a unitary such that and are both diagonal if an only if is Hermitian.

Next, let the SVD of and be . Here, we also assume for simplicity. Then two main theorems involving the optimal scheme in uplink with their proofs are presented as follows.

Theorem 3.

The objective of problem (63) can achieve its minimum when the relay filter and the filter are constructed as follows:

(68)

where and are diagonal matrices.

Proof.

the derivation begins with the equivalent form of (66) and (67) that are expressed as

(69)

Comparing the above equations, we get

(70)

which implies is Hermitian since the right-hand side is Hermitian. In addition, and are Hermitian where is assumed. Hence, by using Lemma 1, we have where is a diagonal matrix. Without loss of generality, let , that is

(71)

Using the SVD of , , , and the result (71), it holds that (70) becomes

(72)

Since the left-hand side of the afore-mentioned equation is diagonal, the other term must be diagonal. Thus, and must be a permutation matrix , in that no other matrices can satisfy the property. Let , we have

(73)

Using SVD and (71), (73) in (67), we get

(74)

which implies that

(75)

Hence, substituting (71), (73), and (75) into the SVD of and , we can have the desired result (68), which decomposes the MIMO relay channel into parallel channels.

Theorem 4.

The optimum MMSE power allocation policy of the problem (63) can be expressed as

(76)

where is defined.

Proof.

Using the results (73) in (72), we get

(77)

that is

(78)

Substituting (75) and (78) into (74), the desired results are obtained.

Therefore, the afore-mentioned theorems form the closed form local optimal solution for uplink of proposed systems, that is, the filter matrices and can be easily calculated according to Theorems 3 and 4.

3.4. Filter Design Schemes for Cooperative Scenario

Based on the results derived earlier, we propose two schemes to approach the optimal results.

Cooperative Scheme  1

In this scheme, the relay filer matrix is given by the expression of as shown in Section 3.3. By regarding and as equivalent channel matrix and noise vector of the conventional MIMO link, the receive filter matrix can be obtained via the Linear MMSE receiver in [18], that is,

(79)

Cooperative Scheme  2

In this scheme, the relay filer matrix is also given by the expression of as shown in Section 3.3. Besides, the BS detects the soft estimate of the data streams from the direct path and relay path using the filter matrix and respectively. Finally, the receiver performs MRC combination over the separate data stream and then decodes them.

4. Numerical Results

The bit error rates (BER) of the proposed schemes in the previous sections are evaluated by applying them to a -user MIMO-relay system with antennas at the BS and antennas at the RS. We obtain the BER plots of OJ-MMSE/RP (Section 2.3.1), SOJ-MMSE/RP (Section 2.3.2), MMSB/DP (Section 2.2), CS1-MMSE/RDP (Section 2.4) and CS2-MMSE/RDP (Section 2.4) in downlink systems, together with OJ-MMSE/RP (Section 3.3), MMSB/DP (Section 3.2), CS1-MMSE/RDP (Section 3.4), and CS2-MMSE/RDP (Section 3.4) in uplink systems. Note that RP and DP denote direct path and relay path, respectively, while RDP represents the cooperative scenario with both the paths. In addition, we also evaluate the following two schemes as a reference for downlink and uplink systems, respectively.

() Transmit Amplify-and-Forward MMSE for relay path of downlink systems (TAF-MMSE/RP). This scheme only requires the relay to normalize the received signal to meet the power constraint and then forward the signal. In this case, the filter matrix at the relay is

(80)

where is given to meet the power constraint at the relay, and hence the BS filter matrix and the scalar are obtained by substituting (80) into (29).

() Receive Amplify-and-Forward MMSE for relay path of uplink systems (RAF-MMSE/RP). In this scheme, the filter matrix at the relay is also , where is given to meet the power constraint at the relay and hence the uplink signal model becomes

(81)

which is similar with that in conventional MIMO systems by regarding and as equivalent channel matrix and noise vector. Then the received MMSE filter can be obtained via the Linear MMSE receiver in [18].

In the simulation, we assume a flat fading channel in which each component of and is an i.i.d. complex random variable with zero mean and unit variance. Considering that the distance between BS and the MSs is usually larger than that between RS and BS, a relevant path loss is introduced to let where each component of is another i.i.d. complex random variable with zero mean and unit variance. In addition, uncorrelated data streams and noise are assumed. To be more specific, 10000 QPSK symbols are simulated for each of the data streams per channel realization and all the results are mean of 2500 channel realizations.

(82)

4.1. BER versus SNR

Figures 3 and 4 show the comparisons of the BER versus SNR in downlink of multiuser MIMO-relay systems. SNR1 denotes the average signal-to-noise ratio of BS-RS link, that is, , while SNR2 denotes the average signal-to-noise ratio of the RS-MS link, that is, . Besides, we assume and in the simulation. The graphs show that the BER of the schemes except MMSB/DP and CS1-MMSE/RDP is saturated when SNR1 or SNR2 becomes large. This is because the relay path is dominant in these schemes, and thus if the SNR of either link is fixed, the BER will converge to a lower bound with the increase of SNR of the other link. On the other hand, we can see that the BER of CS1-MMSE/RDP scheme does not only outperform other schemes much but also is not saturated when increasing SNR1. This is due to the fact that it takes into account both the direct path and relay path and performs joint filter design over the paths. By comparing both the OJ-MMSE/RP and TAF-MMSE/RP scheme for relay path, it can be observed that the joint BS and RS filter design show BER gain than conventional precoding at the BS and AF at therelay, especially in high SNR region. However, when SNR1 is larger enough than SNR2, the MMSB/DP scheme for direct path is better than other schemes except CS1-MMSE/RDP scheme due to the performance loss of two hop transmission.

Figure 3

BER versus SNR of BS-RS link in downlink

Figure 4

BER versus SNR of RS-MS link in downlink

Figures 5 and 6 show the comparisons of the BER versus SNR in uplink of multi-user MIMO-relay systems. Here, SNR1 denotes the average signal-to-noise ratio of RS-BS link, that is, , while SNR2 denotes the average signal-to-noise ratio of the MS-RS link, that is, . Similarly, the graphs also show benefit from the proposed cooperative operation for both paths and joint filter design at BS and RS.

Figure 5

BER versus SNR of RS-BS link in uplink

Figure 6

BER versus SNR of MS-RS link in uplink

4.2. BER versus the Number of Antennas per Node

Figures 7 and 8 show the BER of the schemes with various number of antenna sat the BS and RS for downlink systems and uplink systems, respectively. However, is also assumed. We can see that with the increase of the number of antennas per node, the BER of most schemes rises gradually due to the interference among the multiple data streams, but it converges when becomes large. However, the CS1-MMSE/RDP scheme in uplink systems performs differently, which shows that this scheme can eliminate the interference effectively.

Figure 7

BER versus number of antennas in downlink

Figure 8

BER versus number of antennas in uplink

4.3. BER versus the Relevant Path Loss of Direct Path

Figure 9 shows the BER of the schemes with various relevant path loss for downlink systems. Here, when the relevant path loss of direct path is small enough, the CS2-MMSE/RDP scheme becomes the best scheme instead of the CS1-MMSE/RDP scheme. This is because the bad direct channel condition brings little performance gain that can not offset the performance loss for the relay path. On the other hand, the CS2-MMSE/RDP scheme, together with other three schemes where only relay path is focused, is not affected by the change of the direct channel. Furthermore, comparing the schemes only considering relay path and the MMSE scheme only for direct path, it can be seen that the latter performs better than the former if the direct channel is good enough, which offers a measure for routing the users in cellular MIMO-relay networks.

Figure 9

BER versus pathloss of direct path in downlink

Figure 10 shows the BER of the schemes with various relevant path loss for uplink systems. Apart from the CS2-MMSE/RDP scheme, other schemes perform similar with that in downlink systems. As the CS2-MMSE/RDP scheme for uplink systems also takes both the direct path and the relay path into account, its BER decreases with the improvement of direct path.

Figure 10

BER versus pathloss of direct path in uplink .

4.4. Complexity

Finally, Tables 1 and 2 show computational complexity of the proposed schemes in downlink and uplink systems, respectively. The complexity is measured as the number of required complex multiplications. For simplicity, we only take matrix multiplication, matrix inversion, and SVD parts into account. In addition, for the scheme involving iterative algorithm, we approximate the average iteration time to be 50. For downlink systems, it is observed that the reference scheme MMSB/DP and TAF-MMSE/RP is lower than others due to their simple operations. In addition, CS1-MMSE/RDP only requires a little more multiplications while providing much better performance than others as showed in the previous subsections. Similarly from , we can see that CS2-MMSE/RDP can achieve an excellent tradeoff of complexity and performance. However, CS1-MMSE/RDP scheme sacrifices not much complexity for much better performance than other schemes.

Table 1 Computational complexity of the proposed schemes in downlink systems.

Table 2 Computational complexity ofthe proposed schemes in uplink systems.

4.5. Convergence of Iterative Algorithm

Figure 11 gives the average MSE versus the iteration number for OJ-MMSE/RP scheme in Section 2.3.1 under three different system configurations, that is, . In the figure the dash lines are the steady state performance of the corresponding configurations. As is seen Figure 11, it is obvious that the total MSE is monotonously decreasing and lower bounded to 0. These two facts guarantee the convergence of the scheme. In addition, simulation results have demonstrated that the system performance is very close to the steady-state solution after only a few numbers of iterations.

Figure 11

MSE performance versus number of iterations.

5. Conclusion

In this paper, the local optimal MSE-based joint (BS and RS) filters have been proposed for a multiuser cooperative nonregenerative MIMO-relay system. Both uplink and downlink are considered. It is clear that the cooperative system can be divided into two paths, that is, the direct path and the relay path. As the optimal filter for the direct path can be obtained by using the exiting results of conventional MIMO link, we focus on the optimization for the relay path first. To be more specific, we propose the joint local optimal filter scheme, which involves an iterative algorithm in downlink scenario. Thus a simpler suboptimal scheme is derived for practical use. Then, in uplink scenario a closed-form optimal solution is exploited based on matrix analysis theory. The proposed optimal scheme firstly transform the MIMO relay channel into parallel sub-channels and then the optimal power allocation among the sub-channels has been found to follow a water-filling pattern. Furthermore, based on the results for direct path and relay path, two schemes are proposed for downlink systems and uplink systems with different combination methods, respectively. Numerical results and analysis show that joint filter design and cooperative operation can offer significant performance gain in terms of BER.