In this paper, the main objective is to investigate the best performance among various transmission schemes and RS criteria in multiple-antenna two-way relay systems. By analyzing existing transmission schemes and RS criteria, we find^{1} that the Alamouti code with RS schemes can only achieve a diversity gain of *d*= min(*M*,*N*) at a code rate of *r*=1. The diagonal code with RS schemes can achieve a full diversity gain of *d*=*K* min(*M*,*N*) at a code rate of \(r=\frac {1}{\min (M, N)}\). In other words, the Alamouti code with RS and the diagonal code with RS schemes suffer performance loss in either the diversity gain or the code rate. Among the transmission schemes, beamforming schemes can be easily combined with RS and achieve the best performance. The scheme can minimize the overall system SEP and achieve a full diversity gain at a code rate of *r*=1. Thus, we focus on the performance analysis for the BF-RS scheme in this section. Prior to studying the performance of the BF-RS scheme, we first derive an expression of the system SEP.

Let \(P_{\text {MAC}}^{k}\) denote the SEP in the MAC phase at *R*
_{
k
} and \(P_{\text {BC},m}^{k}\) denote the SEP in the BC phase for given **h**
_{1k
} and **h**
_{2k
} at *S*
_{
m
}. Then, we have

$$\begin{array}{*{20}l} P_{\text{MAC}}^{k} &= \Pr \left\{ {\mathbf{z'}} \ne f\left(\mathbf{x},\mathbf{y}\right)|\left(\textbf{x},\textbf{y}\right),{\textbf{h}_{1\text{k}}},{\mathbf{h}_{2\text{k}}}\right\};\cr {P_{\text{BC},m}^{k}} &= {\Pr \left\{ \hat{\mathbf{z}}_{m} \ne \mathbf{{z'}}|{\mathbf{z'}},{\mathbf{h}_{mk}}\right\} }. \end{array} $$

((7))

It is noteworthy that a correct detection occurs if both the phases have erroneous decisions in a two-way transmission with the BPSK modulation. Therefore, averaging the SEPs over the two phases, the instantaneous SEP at source node *S*
_{
m
}, denoted by \(P_{E\text {to}E,m}^{k} \), is given by

$$\begin{array}{*{20}l} P_{E\text{to}E,m}^{k} = P_{\text{MAC}}^{k}\left(1 - P_{\text{BC},m}^{k}\right) + P_{\text{BC},m}^{k}\left(1 - P_{\text{MAC}}^{k}\right). \end{array} $$

((8))

Thus, the instantaneous end-to-end SEP of both the source nodes *S*
_{1} and *S*
_{2}, defined by \(P_{E \text {to}E}^{k} = \frac {1}{2}\sum \limits _{m = 1}^{2} {P_{E\text {to}E,m}^{k}}\), is given by

$$\begin{array}{*{20}l} {{P_{{E \text{to} E}}^{k}}} &= P_{\text{MAC}}^{k} + \frac{1}{{{2}}}\sum\limits_{m = 1}^{{2}} {P_{\text{BC},m}^{k}} - P_{\text{MAC}}^{k}\sum\limits_{m = 1}^{{2}} {P_{\text{BC},m}^{k}}. \end{array} $$

((9))

Note that in the rest of the paper, we confine ourselves to BPSK for tractable analysis as we will focus on the performance analysis of the BF-RS scheme.

### The optimal beamforming at the two sources

We redefine the transmission matrices at the two sources as **X**=[**x**
_{1}
**x**
_{2} … **x**
_{
L
}] and **Y**=[**y**
_{1}
**y**
_{2} … **y**
_{
L
}], where **x**
_{
j
}=**w**
_{1}
*x*
_{
j
} and **y**
_{
j
}=**w**
_{2}
*y*
_{
j
}. Here, **w**
_{1}=[*w*
_{11}
*w*
_{12} … *w*
_{1M
}]^{T} and **w**
_{2}=[*w*
_{21}
*w*
_{22} … *w*
_{2N
}]^{T} are the beamforming vectors chosen by source *S*
_{1} and *S*
_{2}. For simplicity, the time index subscript *j* is omitted in the following derivation. We assume that the transmission power of the sources and the relays are identical, i.e., *P*
_{1}=*P*
_{2}=*P*
_{
r
}=*P*. We also denote the SNR by \({{\bar \gamma }} = \frac {{{P}}}{{\sigma ^{2}}}\), where *σ*
^{2} is the Gaussian noise variance. We focus on the case of **g**
_{
mk
}=**h**
_{
mk
}, *m*=1,2, by considering that the links between *S*
_{
m
} and *R*
_{
k
} are reciprocal. In addition, we assume that the h_{1i,k
}’s, *i*=1,2,…,*M* and h_{2i,k
}’s, *i*=1,2,…,*N*,*k*=1,2,…,*K*, are independent and \({h_{mi,k}}\sim \mathcal {CN}(0,1)\), i.e., Rayleigh fading channels. At relay *R*
_{
k
}, we can rewrite (3) as

$$\begin{array}{*{20}l} {} [x',y'] = \mathop {\arg \min }\limits_{x \in {\mathcal{C}},y \in {\mathcal{C}}} {\left\| {{\mathbf{s}_{k}} - \sqrt {\frac{P}{M}} \mathbf{h}_{1k}^{T}{\mathbf{w}_{1}}x - \sqrt {\frac{P}{N}} \mathbf{h}_{1k}^{T}{\mathbf{w}_{1}}y} \right\|^{2}},, \end{array} $$

((10))

Recall that the signals *x* and *y* are transmitted from *S*
_{1} and *S*
_{2}, respectively. The estimate of *x* and *y* is written as *x*
^{′} and *y*
^{′}, respectively. Based on pairwise error probability (PEP) analysis and assuming that all code **u** are equally likely, the average error rate is upper bounded by

$$\begin{array}{*{20}l} \mathbf{P}_{\text{MAC}}^{k}&\le Q\left({\sqrt {\frac{{\bar \gamma {{\left\| {\textbf{h}_{1k}^{T}{\textbf{w}_{1}}\left(x - x{\prime}\right)} \right\|}^{2}}}}{{2M}}}} \right)\\ &+ Q\left({\sqrt {\frac{{\bar \gamma {{\left\| {\mathbf{h}_{2k}^{T}{\mathbf{w}_{2}}(y - y')} \right\|}^{2}}}}{{2N}}}} \right) \cr & \buildrel \Delta \over = \sum\limits_{i = 1}^{2} {Q(\sqrt {\gamma_{ib}})}, \end{array} $$

((11))

where \({\gamma _{1b}} = {\frac {{2 \bar \gamma {{\left \| {\textbf {h}_{1k}^{T}{{\mathbf {w }}_{1}}} \right \|}^{2}}}}{{{M}}}}\) and \({\gamma _{2b}} = {\frac {{2 \bar \gamma {{\left \| {\textbf {h}_{2k}^{T}{{\mathbf {w}}_{2}}} \right \|}^{2}}}}{{{N}}}}\) are the equivalent system SNR of *S*
_{1} and *S*
_{2}, respectively, and *Q*(·) is the Gaussian Q-function ([15], 2-1-97).

With power constraints at the sources, we minimize \(\textbf {P}_{\text {MAC}}^{k} \) subject to that the norms of *w*
_{1} and *w*
_{2} are bounded by constant, i.e., \({\left \| {\sqrt {\frac {P}{{{M}}}} {\textbf {w}_{1}}} \right \|^{2}} \le P\) and \({\left \| {\sqrt {\frac {P}{{{N}}}} {\textbf {w}_{2}}} \right \|^{2}} \le P\), respectively, which leads to ∥**w**
_{1}∥^{2}≤*M*,∥**w**
_{2}∥^{2}≤*N*.

The optimal beamforming vectors can be obtained by maximizing the SNR of *S*1 and *S*
_{2}, respectively, as the objective is the sum of two monotonous Q-function expressions, and the constraints are independent of each other. Thus, the optimal beamforming vectors in the multiple-antenna two-way relay system can be easily obtained as

$$\begin{array}{*{20}l} {\textbf{w}_{1,\text{opt}}} = \sqrt {{M}} \frac{{\textbf{h}_{1k}^{*}}}{{\left\| {\textbf{h}_{1k}^{}} \right\|}},\:{\textbf{w}_{2,\text{opt}}} = \sqrt {{N}} \frac{{\textbf{h}_{2k}^{*}}}{{\left\| {\textbf{h}_{2k}^{}} \right\|}}. \end{array} $$

((12))

Note that the results are similar to the maximum ratio transmission (MRT) [16]. It is proved that the MRT obtained from one-way relay systems is still valid in the two-way relay system [17, 18].

Using the optimal beamforming vectors in (12), the lower bound on \(\textbf {P}_{\text {MAC}}^{k}\) can be obtained exactly in the same way as in [19]. Assuming a genie-aided source knows the message from the other before transmission, we have

$$\begin{array}{*{20}l} Q\left({\sqrt {2\bar \gamma \beta_{\min }^{k}}} \right) \le \textbf{P}_{\text{MAC}}^{k} \le \sum\limits_{m = 1}^{2} {Q\left({\sqrt {2\bar \gamma {\beta_{m}^{k}}}} \right)}, \end{array} $$

((13))

where \(\beta _{\min }^{k} = \min \left ({{{\left \| {\textbf {h}_{1k}^{}} \right \|}^{2}},{{\left \| {\textbf {h}_{2k}^{}} \right \|}^{2}}} \right)\) and \({\beta _{m}^{k}} = {\left \| {{\textbf {h}_{\textit {mk}}}} \right \|^{2}}\).

### RS criterion

The opportunistic two-way relay scheme with a low complexity in [7] achieves the better performance compared to a fully-distributed two-way relay scheme. Thus, we focus on the schemes that only one best relay is selected out of *K* relays to decode-and-forward the signals in the second phase transmission. For tractable analysis, we use the equivalent SNR as an effective metric. When relay *R*
_{
k
} is selected, we define the equivalent SNR for the MAC phase at relay *R*
_{
k
} and the BC phase at *S*
_{
m
}, *m*=1,2 as \(\mathrm {SNR_{\textit {MAC}}}^{k} \le \bar \gamma \beta _{\min }^{k} \) and \(\mathrm {SNR_{\textit {BC}},m}^{k} = \bar \gamma {\beta _{m}^{k}} \label {EQ:P_bc_bound} \), respectively. Using the optimal beamforming, we compare two RS methods: one is an optimal beamforming and RS (O-BF-RS) scheme and the other is a suboptimal beamforming and RS (S-BF-RS) scheme.

1) O-BF-RS: For O-BF-RS, the destination node is to select the best relay among all the relays, which is denoted by \(R_{\mathcal {R}}\). The selected relay \(R_{\mathcal {R}}\) can achieve the minimum overall system SEP in (9). That is,

$$\begin{array}{*{20}l} \mathcal{R} = \mathop {\arg \min }\limits_{k = 1,2,\ldots,K} \left(P_{\text{MAC}}^{k} + \frac{1}{2}\sum\limits_{m = 1}^{2} P_{\text{BC},m}^{k}\right). \end{array} $$

((14))

2) S-BF-RS: The overall system performance is limited by the worst link. Thus, as a suboptimal scheme, we select the relay which minimizes the maximum SEP of all links. This can be done by selecting the relay that maximizes the minimum equivalent SNR of all links. Then, the RS criterion is given as

$$\begin{array}{*{20}l} {} {\mathcal{R}} \,=\, \mathop {\arg \min \max }\limits_{k = 1,2,\ldots,K} \left(\!P_{\text{MAC}}^{k},P_{\text{BC},1}^{k},P_{\text{BC},2}^{k}\!\right) \,=\, \mathop {\arg \max }\limits_{k = 1,2,\ldots,K} \left(\!\bar \gamma \beta_{\min }^{k}\right). \end{array} $$

((15))

Actually, the performance difference between the optimal scheme and the suboptimal scheme is negligible as the overall performance is generally decided by the worst link. This result can be verified in Section 6. However, even the performance is almost the same, the computation complexity of the two schemes are quite different. The general SEP expression which is conditioned on the instantaneous received SNR can be expressed as \(\text {SEP}(\gamma)=Q(\sqrt {(c\gamma)})\) [20], where *γ* is the received SNR, *c* is a constant which depends on the modulation mapping and *Q*(·) is the Gaussian Q-function ([15], 2-1-97). As a result, computing the Q-function to obtain the SEP expression and then determining the best relay for the O-BF-RS is much more complicated than that of the S-BF-RS, in which only the received SNR is needed. Thus, the S-BF-RS is more practical. Note that, for convenience, we use the BF-RS to replace the S-BF-RS in the rest of this paper.

In the following section, we present the performance analysis for the BF-RS scheme in the multiple-antenna two-way relay system.