A basic two-way communication system model is considered as shown in Figure 1. The system operates in the time division duplex mode. Source 1 and source 2 are required to exchange data between themselves with the assistance of a relay node. All three nodes have *N* antennas. The relay is located at a normalized distance *d* from source 1. Source 1-to-relay and source 2-to-relay transmissions undergo Rayleigh fading with {\mathbf{H}}_{1}\in {\u2102}^{N\times N} and {\mathbf{H}}_{2}\in {\u2102}^{N\times N}, respectively. Entries of **H**_{1} are assumed to be approximately \frac{1}{{d}^{\alpha}}\phantom{\rule{1em}{0ex}}\mathcal{C}\mathcal{N}(0,1), and entries of **H**_{2} are assumed to be approximately \frac{1}{{(2-d)}^{\alpha}}\phantom{\rule{1em}{0ex}}\mathcal{C}\mathcal{N}(0,1), where *α* is the path loss exponent and *d* is the normalized distance. The distance between any node to the midpoint of two nodes is considered as the reference distance.

We consider {\mathbf{R}}_{i}\in {\u2102}^{N\times N} as the antenna correlation matrix at source *i*(=1,2), and {\mathbf{R}}_{r}\in {\u2102}^{N\times N} is the antenna correlation matrix at the relay node. We consider these matrices to be symmetric. We denote {\mathbf{H}}_{m1}={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{H}}_{1}{\mathbf{R}}_{1}^{\frac{1}{2}} and {\mathbf{H}}_{m2}={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{H}}_{2}{\mathbf{R}}_{2}^{\frac{1}{2}} to be the channel matrices from source 1-to-relay and source 2-to-relay. During the BC stage, channel matrices are denoted as {\mathbf{H}}_{b1}={\mathbf{R}}_{1}^{\frac{1}{2}}{\mathbf{H}}_{1}^{T}{\mathbf{R}}_{r}^{\frac{1}{2}} and {\mathbf{H}}_{b2}={\mathbf{R}}_{2}^{\frac{1}{2}}{\mathbf{H}}_{2}^{T}{\mathbf{R}}_{r}^{\frac{1}{2}}.

During the MA stage, the source nodes transmit their signals and the summation of two signals are received at the relay node. The relay node estimates the sum of the two signals, which is the general scheme of PNC mapping [8]. However, a problem arises due to channel fading. With fading, the estimation at the relay node becomes exceedingly complex. Therefore, we consider designing precoders at the source nodes and a decoder at the relay node to overcome this problem. Source nodes use precoders {\mathbf{F}}_{1}\in {\u2102}^{N\times N} and {\mathbf{F}}_{2}\in {\u2102}^{N\times N} before their transmissions. The relay node receives both signals at the same time and uses the decoder \mathbf{G}\in {\u2102}^{N\times N} prior to performing any PNC operation as in [8] to estimate the XOR or the summation of two transmitted symbol vectors.

Next, we consider the BC stage of two-way communication. The relay node retransmits the estimated summation during this stage. The source nodes estimate the summation at their nodes and find the respective symbol transmitted by the other node with the help of its own symbol. Here, we consider the precoder at the relay node and decoders at the source nodes. The relay uses the precoder {\mathbf{F}}_{r}\in {\u2102}^{N\times N}. Both sources 1 and 2 use decoders {\mathbf{G}}_{1}\in {\u2102}^{N\times N} and {\mathbf{G}}_{2}\in {\u2102}^{N\times N}, and each reconstructs the required symbol with the help of their own information. All nodes dynamically adjust their precoder and decoder matrices with the channel information.

Both stages require CSI knowledge to design the precoders and the decoder, and the following procedure is used to estimate channels prior to their data transmission.

### 2.1 Channel estimation

We assume channel reciprocity and a quasi-static channel environment. The relay transmits an orthogonal sequence to estimate channels at the end of the BC time slot. Both source nodes receive this signal, and they find optimum precoder and decoder matrices. The source nodes then send optimum **G** and **F**_{
r
} to the relay node via dedicated feedback link. We assume that the channel estimation and relay feedback time durations are very small compared to the channel coherence time. Once the two-way relaying system is established with optimum precoders and decoders, the source nodes transmit data that are required to exchange between them. Antenna correlations are also assumed to be known.

The relay broadcasts **X**_{
t
} training sequence, where **X**_{
t
} is an *N*×*N* matrix. The received signal at source 1 is given by the following:

{\mathbf{Y}}_{1}={\mathbf{H}}_{b1}{\mathbf{X}}_{t}+{\mathbf{N}}_{1},

(1)

where **N**_{1} is an *N*×*N* matrix with entries independent and identically distributed (i.i.d.) \mathcal{C}\mathcal{N}(0,{\sigma}_{{N}_{1}}^{2}). As commonly used in orthogonal sequence channel estimation [24], we use {\mathbf{X}}_{t}={\mathbf{R}}_{r}^{-\frac{1}{2}}\mathbf{X}, where **X** is a unitary *N*×*N* matrix with multiplication factor \sqrt{{P}_{t}/\text{Tr}\left({\mathbf{R}}_{r}^{-1}\right)} with the relay transmit power *P*_{
t
}. Source 1 pre-multiplies the received signal matrix by {\mathbf{R}}_{1}^{-\frac{1}{2}} and post-multiplies it by **X**^{−1}. Therefore, the received channel matrix {\stackrel{~}{\mathbf{H}}}_{1}^{T} is given by the following:

\begin{array}{ll}{\stackrel{~}{\mathbf{H}}}_{1}^{T}& ={\mathbf{R}}_{1}^{-\frac{1}{2}}{\mathbf{R}}_{1}^{\frac{1}{2}}{\mathbf{H}}_{1}^{T}{\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{R}}_{r}^{-\frac{1}{2}}\mathbf{X}{\mathbf{X}}^{-1}+{\mathbf{R}}_{1}^{-\frac{1}{2}}{\mathbf{N}}_{1}{\mathbf{X}}^{-1}\phantom{\rule{2em}{0ex}}\\ ={\mathbf{H}}_{1}^{T}+{\mathbf{R}}_{1}^{-\frac{1}{2}}{\stackrel{~}{\mathbf{N}}}_{1},\phantom{\rule{2em}{0ex}}\end{array}

(2)

where {\stackrel{~}{\mathbf{N}}}_{1} is an *N*×*N* matrix with entries i.i.d. \mathcal{C}\mathcal{N}(0,{\sigma}_{1}^{2}) and {\sigma}_{1}^{2}={\sigma}_{{N}_{1}}^{2}\text{Tr}\left({\mathbf{R}}_{r}^{-1}\right)/{P}_{t}. The minimum MSE (MMSE) criterion is used to obtain the channel estimate from {\stackrel{~}{\mathbf{H}}}_{1}^{T}. The MMSE estimate is presented as follows:

{\stackrel{\u0304}{\mathbf{H}}}_{1}^{T}\phantom{\rule{1em}{0ex}}=\mathcal{E}\left\{{\mathbf{H}}_{1}^{T}\right|{\stackrel{~}{\mathbf{H}}}_{1}^{T}\}={\left[{\mathbf{I}}_{N}+{\sigma}_{1}^{2}{\mathbf{R}}_{1}^{-1}\right]}^{-1}{\stackrel{~}{\mathbf{H}}}_{1}^{T}.

(3)

The estimation error can be obtained [27] as {\mathbf{R}}_{1}^{-\frac{1}{2}}{\left[{\mathbf{I}}_{N}+{\sigma}_{1}^{2}{\mathbf{R}}_{1}^{-1}\right]}^{-\frac{1}{2}}{\mathbf{E}}_{1}, where **E**_{1} is an *N*×*N* matrix with entries i.i.d. \mathcal{C}\mathcal{N}(0,{\sigma}_{1}^{2}). Therefore, the channel matrix now consists of MMSE estimation and the estimation error part as

{\mathbf{H}}_{1}^{T}={\stackrel{\u0304}{\mathbf{H}}}_{1}^{T}+{\mathbf{R}}_{1}^{-\frac{1}{2}}{\left[{\mathbf{I}}_{N}+{\sigma}_{1}^{2}{\mathbf{R}}_{1}^{-1}\right]}^{-\frac{1}{2}}{\mathbf{E}}_{1}.

(4)

A similar result is valid for the estimation of {\mathbf{H}}_{2}^{T}, and it is given by {\mathbf{H}}_{2}^{T}={\stackrel{\u0304}{\mathbf{H}}}_{2}^{T}+{\mathbf{R}}_{2}^{-\frac{1}{2}}{\left[{\mathbf{I}}_{N}+{\sigma}_{2}^{2}{\mathbf{R}}_{2}^{-1}\right]}^{-\frac{1}{2}}{\mathbf{E}}_{2}, where **E**_{2} is an *N*×*N* matrix with entries i.i.d \mathcal{N}(0,{\sigma}_{2}^{2}) and {\sigma}_{2}^{2}={\sigma}_{{N}_{2}}^{2}\text{Tr}\left({\mathbf{R}}_{r}^{-1}\right)/{P}_{t}.

Since we assume the channel reciprocity, using (3) and (4), we can write the source-to-relay channels **H**_{
m
i
} as follows:

\begin{array}{ll}{\mathbf{H}}_{\mathit{\text{mi}}}& ={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{H}}_{i}{\mathbf{R}}_{i}^{\frac{1}{2}}={\mathbf{R}}_{r}^{\frac{1}{2}}{\stackrel{\u0304}{\mathbf{H}}}_{i}{\mathbf{R}}_{i}^{\frac{1}{2}}+{\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{E}}_{i}^{T}\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}\times {\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}{\mathbf{R}}_{i}^{-\frac{T}{2}}{\mathbf{R}}_{i}^{\frac{1}{2}}\phantom{\rule{1em}{0ex}}\phantom{\rule{1em}{0ex}}i=1,2.\phantom{\rule{2em}{0ex}}\end{array}

(5)

Correlation matrices are symmetric, and for simplicity, we denote {\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}={\mathbf{R}}_{r}^{\frac{1}{2}}{\stackrel{\u0304}{\mathbf{H}}}_{i}{\mathbf{R}}_{i}^{\frac{1}{2}} and {\mathbf{E}}_{\mathit{\text{mi}}}={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{E}}_{i}^{T}{\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}, where these represent each channel matrix with a mean part and an estimation error part. We use these estimated channels to design precoders and decoder.

### 2.2 Physical layer network coding

During data transmission, modulated symbol vectors are fed into sources 1 and 2, with each given as **x**_{1}=(*x*_{11}, *x*_{12}, *x*_{13},…,*x*_{1N})^{T} and **x**_{2}=(*x*_{21}, *x*_{22}, *x*_{23},…,*x*_{2N})^{T}, where {x}_{\mathit{\text{ii}}}\in \u2102 and {\mathcal{E}}_{\mathbf{x}}\left\{{\mathbf{x}}_{i}{\mathbf{x}}_{i}^{H}\right\}={\mathbf{I}}_{N} (*i*=1,2). The relay node estimates the summation of modulated signals (**x**_{1}+**x**_{2}) and transmits it during the next time slot. This is more general and a valid PNC scheme for any modulation alphabet. For simple modulation schemes like QPSK, this summation of two signals can be mapped to XOR of two transmitted unmodulated information [8]. During the next time slot, the modulated symbol of XOR version will then be transmitted. In summary, we can carry out PNC for any case if we design the precoders and decoders to minimize the MSE between received signal and summation of modulated signals.

During the first time slot, the received signal vector \mathbf{y}\in {\u2102}^{N\times 1} at the relay is given by the following:

\mathbf{y}=\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}{\mathbf{x}}_{1}+\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}{\mathbf{x}}_{2}+\mathbf{G}\mathbf{n},

(6)

where \mathbf{n}\sim \mathcal{C}\mathcal{N}(0,{\sigma}^{2}{\mathbf{I}}_{N}). The relay node estimates the **x**_{1}+**x**_{2}, and this leads us to consider *N* number of separate spatial streams. Therefore, the received signal at *i* th stream *y*_{
i
} can be used to obtain an estimate corresponding to *x*_{1i}+*x*_{2i}. This scheme reduces the complexity of the PNC mapping at the relay [32]. We denote the estimation of *x*_{1i}+*x*_{2i} as *x*_{3i}, and *x*_{3i} broadcasts to other nodes during the next time slot.

During the second time slot, the received signal vector {\mathbf{y}}_{1}\in {\u2102}^{N\times 1} at the source 1 is given as follows:

{\mathbf{y}}_{1}={\mathbf{G}}_{1}{\mathbf{H}}_{b1}{\mathbf{F}}_{r}{\mathbf{x}}_{3}+{\mathbf{G}}_{1}{\mathbf{n}}_{1},

(7)

where **x**_{3}=(*x*_{31}, *x*_{32}, *x*_{33},…,*x*_{3N})^{T} and {\mathbf{n}}_{1}\sim \mathcal{C}\mathcal{N}(0,{\sigma}^{2}{\mathbf{I}}_{N}). Similarly, received signal vector {\mathbf{y}}_{2}\in {\u2102}^{N\times 1} at the source 2 is given as follows:

{\mathbf{y}}_{2}={\mathbf{G}}_{2}{\mathbf{H}}_{b1}{\mathbf{F}}_{r}{\mathbf{x}}_{3}+{\mathbf{G}}_{2}{\mathbf{n}}_{1},

(8)

where {\mathbf{n}}_{2}\sim \mathcal{C}\mathcal{N}(0,{\sigma}^{2}{\mathbf{I}}_{N}). A source node estimates **x**_{3} and filter out its transmitted symbol. This gives the desired symbol, which is transmitted by the other source node. Here, the PNC operation can be considered independently at each relay antenna. In the case of nodes with different number of antennas, the maximum number of independent flows is limited to the minimum number of antennas at all nodes. The PNC operation is then considered in a similar manner as for the case where there are equal number of antennas at the nodes.

Accuracy of the PNC mapping is dependent on the estimated summation of two symbols. Therefore, it is evident that the optimum joint design is required to have accurate estimation process. Problem formulation and the solving method for designing optimum precoders and decoders are described in the next sections of the paper.

### 2.3 Problem formulation

As two-way communications have two phases, we can consider the analysis separately for both phases. For each phase, the total power can be limited, which can occur in many practical scenarios. Therefore, we consider *P*_{
t
} as the maximum total transmitted power available in each time slot. A similar problem formulation and solving procedure is valid for individual power constraints of nodes.

#### 2.3.1 Multiple-access stage

In this stage, both source nodes transmit to relay, and the transmitted powers of source nodes should satisfy the following constraint:

\text{Tr}\left({\mathbf{F}}_{1}{\mathbf{F}}_{1}^{H}\right)+\text{Tr}\left({\mathbf{F}}_{2}{\mathbf{F}}_{2}^{H}\right)\le {P}_{T},

(9)

The received signal (6) during the MA stage is used to estimate **x**_{1}+**x**_{2}. The estimation error vector **e**_{
m
} can be defined as follows:

{\mathbf{e}}_{m}=\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}{\mathbf{x}}_{1}+\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}{\mathbf{x}}_{2}+\mathbf{G}\mathbf{n}-{\mathbf{x}}_{1}-{\mathbf{x}}_{2}

(10)

Data streams may need different quality of service (QoS). We facilitate this by introducing weights for different streams. A diagonal *N*×*N* positive definite weight matrix **W** is used for that purpose. We express WMSE at the MA stage as follows:

\begin{array}{ll}{\text{WMSE}}_{m}& ={\mathcal{E}}_{\mathbf{x},\mathbf{n}}\{\parallel {\mathbf{W}}^{1/2}{\mathbf{e}}_{m}{\parallel}^{2}\}\phantom{\rule{2em}{0ex}}\\ ={\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{\text{Tr}\right({\mathbf{W}}^{1/2}{\mathbf{e}}_{m}{\mathbf{e}}_{m}^{H}{\mathbf{W}}^{H/2}\left)\right\}\phantom{\rule{2em}{0ex}}\\ =\text{Tr}\left(\mathbf{W}{\mathcal{E}}_{\mathbf{x},\mathbf{n}}\right\{{\mathbf{e}}_{m}{\mathbf{e}}_{m}^{H}\left\}\right),\phantom{\rule{2em}{0ex}}\end{array}

(11)

where {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{e}}_{m}{\mathbf{e}}_{m}^{H}\right\} is given by the following:

\begin{array}{l}{\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{e}}_{m}{\mathbf{e}}_{m}^{H}\right\}=(\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}-{\mathbf{I}}_{N}){(\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}-{\mathbf{I}}_{N})}^{H}\phantom{\rule{2em}{0ex}}\\ +(\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}-{\mathbf{I}}_{N}){(\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}-{\mathbf{I}}_{N})}^{H}+{\sigma}^{2}\mathbf{G}{\mathbf{G}}^{H}.\phantom{\rule{2em}{0ex}}\end{array}

(12)

We use {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{x}}_{1}{\mathbf{x}}_{1}^{H}\right\}={\mathbf{I}}_{N}, {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{x}}_{2}{\mathbf{x}}_{2}^{H}\right\}={\mathbf{I}}_{N}, {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{x}}_{1}{\mathbf{x}}_{2}^{H}\right\}={\mathbf{0}}_{N\times N}, {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{x}}_{1}{\mathbf{n}}^{H}\right\}={\mathbf{0}}_{N\times N}, {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{{\mathbf{x}}_{2}{\mathbf{n}}^{H}\right\}={\mathbf{0}}_{N\times N} and {\mathcal{E}}_{\mathbf{x},\mathbf{n}}\left\{\mathbf{n}{\mathbf{n}}^{H}\right\}={\sigma}^{2}{\mathbf{I}}_{N} to obtain (12).

We need to minimize WMSE_{
m
} during the MA stage subject to the total power constraint to find optimum precoders and decoders. However, for given channel instances of **H**_{m 1} and **H**_{m 2}, the estimation error becomes a random variable. We have to consider this channel estimation error with WMSE_{
m
}. The error has a Gaussian distribution, and we focus on the expected value of WMSE_{
m
}, given as follows:

\begin{array}{ll}\phantom{\rule{-15.0pt}{0ex}}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{m}\right\}& ={\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}(\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}-{\mathbf{I}}_{N})\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}\times {(\mathbf{G}{\mathbf{H}}_{m1}{\mathbf{F}}_{1}-{\mathbf{I}}_{N})}^{H}+\mathbf{W}(\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}-{\mathbf{I}}_{N})\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}\times {(\mathbf{G}{\mathbf{H}}_{m2}{\mathbf{F}}_{2}-{\mathbf{I}}_{N})}^{H}+{\sigma}^{2}\mathbf{W}\mathbf{G}{\mathbf{G}}^{H}\left)\right\}.\phantom{\rule{2em}{0ex}}\end{array}

(13)

Channel estimates in (5) consist of the MMSE value and the error part as {\mathbf{H}}_{\mathit{\text{mi}}}={\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}+{\mathbf{E}}_{\mathit{\text{mi}}} for *i*=1,2. Therefore, expanding (13) into the following:

\begin{array}{ll}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{m}\right\}& =\sum _{i=1}^{2}{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ +{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\mathbf{E}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ +\sum _{i=1}^{2}{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\mathbf{E}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ +{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\mathbf{E}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\mathbf{E}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ -\sum _{i=1}^{2}{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}-\mathbf{W}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{2em}{0ex}}\phantom{\rule{1em}{0ex}}\phantom{\rule{2.77626pt}{0ex}}\phantom{\rule{2.77626pt}{0ex}}\phantom{\rule{2.77626pt}{0ex}}-\mathbf{W}\mathbf{G}{\mathbf{E}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}-\mathbf{W}{\mathbf{F}}_{i}^{H}{\mathbf{E}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ +\text{Tr}(2\mathbf{W}+{\sigma}^{2}\mathbf{W}\mathbf{G}{\mathbf{G}}^{H})\phantom{\rule{2em}{0ex}}\end{array}

(14)

since {\mathcal{E}}_{\mathbf{E}}\left\{{\mathbf{E}}_{\mathit{\text{mi}}}\right\}={\mathbf{0}}_{N\times N} (14) reduces to the following:

\begin{array}{ll}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{m}\right\}& =\sum _{i=1}^{2}\text{Tr}\left(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\right)\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\mathbf{E}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\mathbf{E}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}-\sum _{i=1}^{2}\text{Tr}(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}-\mathbf{W}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H})\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+\text{Tr}(2\mathbf{W}+{\sigma}^{2}\mathbf{W}\mathbf{G}{\mathbf{G}}^{H})\phantom{\rule{2em}{0ex}}\end{array}

(15)

Moreover, we can expand the following term with the expectation as follows:

\begin{array}{l}{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\mathbf{E}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\mathbf{E}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\left)\right\}\\ \phantom{\rule{1em}{0ex}}={\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right(\mathbf{W}\mathbf{G}{\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{E}}_{i}^{T}{\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}\\ \phantom{\rule{2em}{0ex}}\times {\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}{\mathbf{E}}_{i}^{\ast}{\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{G}}^{H}\left)\right\}\\ \phantom{\rule{1em}{0ex}}={\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right({\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{G}}^{H}\mathbf{W}\mathbf{G}{\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{E}}_{i}^{T}{\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}\\ \phantom{\rule{2em}{0ex}}\times {\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\left[{\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right]}^{-\frac{T}{2}}{\mathbf{E}}_{i}^{\ast}\left)\right\}\\ \phantom{\rule{1em}{0ex}}={\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right({\mathbf{P}}_{i}{\mathbf{E}}_{i}^{T}{\mathbf{Q}}_{i}{\mathbf{E}}_{i}^{\ast}\left)\right\},\end{array}

(16)

where we use {\mathbf{Q}}_{i}={\left({\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right)}^{-\frac{1}{2}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\left({\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right)}^{-\frac{1}{2}} and {\mathbf{P}}_{i}={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{G}}^{H}\mathbf{W}\mathbf{G}{\mathbf{R}}_{r}^{\frac{1}{2}}. Next, we use relationships between trace and vectors to simplify (16) into the following:

\begin{array}{ll}{\mathcal{E}}_{\mathbf{E}}\left\{\text{Tr}\right({\mathbf{P}}_{i}{\mathbf{E}}_{i}^{T}{\mathbf{Q}}_{i}{\mathbf{E}}_{i}^{\ast}\left)\right\}& ={\mathcal{E}}_{\mathbf{E}}\left\{\text{vec}\right({\mathbf{E}}_{i}^{\ast}\left)\text{vec}\right({\mathbf{P}}_{i}{\mathbf{E}}_{i}^{T}{\mathbf{Q}}_{i}\left)\right\}\phantom{\rule{2em}{0ex}}\\ ={\mathcal{E}}_{\mathbf{E}}\left\{\text{vec}\right({\mathbf{E}}_{i}^{\ast}\left)\right({\mathbf{Q}}_{i}\otimes {\mathbf{P}}_{i}\left)\text{vec}\right({\mathbf{E}}_{i}^{T}\left)\right\}\phantom{\rule{2em}{0ex}}\\ =\text{Tr}\left(\right({\mathbf{Q}}_{i}\otimes {\mathbf{P}}_{i}\left)\mathcal{E}\right\{\text{vec}\left({\mathbf{E}}_{i}^{T}\right)\text{vec}\left({\mathbf{E}}_{i}^{\ast}\right)\left\}\right).\phantom{\rule{2em}{0ex}}\end{array}

(17)

We know that \mathcal{E}\left\{\text{vec}\right({\mathbf{E}}_{i}^{T}\left)\text{vec}\right({\mathbf{E}}_{i}^{\ast}\left)\right\}={\sigma}_{i}^{2}\mathbf{I} and using the relation Tr((**Q**_{
i
}⊗**P**_{
i
}))=Tr(**Q**_{
i
})Tr(**P**_{
i
}), we can find the following expression for (15):

\begin{array}{ll}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{m}\right\}& =\sum _{i=1}^{2}\text{Tr}(\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}\phantom{\rule{0.3em}{0ex}}-\mathbf{W}\mathbf{G}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}{\mathbf{F}}_{i}\phantom{\rule{2em}{0ex}}\\ -\mathbf{W}{\mathbf{F}}_{i}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{H}{\mathbf{G}}^{H}+\mathbf{W})\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+\sum _{i=1}^{2}{\sigma}_{i}^{2}\text{Tr}\left({\mathbf{Q}}_{i}\right)\text{Tr}\left({\mathbf{P}}_{i}\right)+\text{Tr}\left({\sigma}^{2}\mathbf{W}\mathbf{G}{\mathbf{G}}^{H}\right)\phantom{\rule{2em}{0ex}}\end{array}

(18)

Next, we formulate the optimization problem to minimize expected value of WMSE_{
m
} under a limited available transmit power at source nodes as presented in Problem 1:

##### Problem 1

\begin{array}{l}\begin{array}{ll}\underset{{\mathbf{F}}_{1},{\mathbf{F}}_{2},\mathbf{G}}{\text{min}}& {\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{m}\right\}\\ \text{subject to}& \text{Tr}\left({\mathbf{F}}_{1}{\mathbf{F}}_{1}^{H}\right)+\text{Tr}\left({\mathbf{F}}_{2}{\mathbf{F}}_{2}^{H}\right)\le {P}_{T}\end{array}\end{array}

(19)

This is a non-convex optimization problem. In Section 3, we propose an algorithm to solve this optimally.

#### 2.3.2 Broadcasting stage

Estimated **x**_{1}+**x**_{2}, i.e., **x**_{3} broadcasts during this time slot. Relay uses **F**_{
r
} precoder and transmits **x**_{3} to both source nodes. Sources 1 and 2 now have **G**_{1} and **G**_{2} decoders, respectively. At source *i* (=1,2), it estimates **x**_{3} and uses that to find desired symbol.

Similar to the MA stage, the joint design is considered to minimize the WMSE of received signals. We considered all nodes to satisfy a total power constraint for their transmission. Therefore, during the BC stage, transmit power at the relay node should satisfy the following constraint:

\text{Tr}\left({\mathbf{F}}_{r}{\mathbf{F}}_{r}^{H}\right)\le {P}_{T}/2.

(20)

This constraint becomes *P*_{
T
}/2 because {\mathcal{E}}_{\mathbf{x}}\left\{{\mathbf{x}}_{3}{\mathbf{x}}_{3}^{H}\right\} is now equal to 2**I**_{
N
}; **e**_{
b
i
} is the estimation error vector at *i* th source. It is given as follows:

{\mathbf{e}}_{\mathit{\text{bi}}}={\mathbf{G}}_{i}{\mathbf{H}}_{\mathit{\text{bi}}}{\mathbf{F}}_{r}{\mathbf{x}}_{3}+{\mathbf{G}}_{i}{\mathbf{n}}_{i}-{\mathbf{x}}_{3}.

(21)

We use the similar weights for different streams as in multiple-access analysis. WMSE at source *i* is denoted as WMSE_{
i
} and is given by the following:

\begin{array}{l}{\text{WMSE}}_{i}=2\mathbf{W}({\mathbf{G}}_{i}{\mathbf{H}}_{\mathit{\text{bi}}}{\mathbf{F}}_{r}-{\mathbf{I}}_{N}){({\mathbf{G}}_{i}{\mathbf{H}}_{\mathit{\text{bi}}}{\mathbf{F}}_{r}-{\mathbf{I}}_{N})}^{H}\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{2em}{0ex}}\phantom{\rule{2em}{0ex}}\phantom{\rule{2.77626pt}{0ex}}+{\sigma}^{2}\mathbf{W}{\mathbf{G}}_{i}{\mathbf{G}}_{i}^{H}\phantom{\rule{2em}{0ex}}i=1,2.\phantom{\rule{2em}{0ex}}\end{array}

(22)

**H**_{
b
i
} has an error component. Therefore, the expected value of WMSE_{
i
} is considered in the optimum precoder-decoder design. A similar procedure as in the MA stage is valid to find the expected value of WMSE_{
i
}. We find WMSE_{
i
} as follows:

\begin{array}{ll}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{i}\right\}& =2\text{Tr}(\mathbf{W}{\mathbf{G}}_{i}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{T}{\mathbf{F}}_{r}{\mathbf{F}}_{r}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{\ast}{\mathbf{G}}_{i}^{H}\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{3em}{0ex}}-\mathbf{W}{\mathbf{G}}_{i}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{T}{\mathbf{F}}_{r}-\mathbf{W}{\mathbf{F}}_{r}^{H}{\stackrel{\u0304}{\mathbf{H}}}_{\mathit{\text{mi}}}^{\ast}{\mathbf{G}}_{i}^{H}+\mathbf{W})\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+2{\sigma}_{i}^{2}\text{Tr}\left({\mathbf{L}}_{i}\right)\text{Tr}\left(\mathbf{K}\right)\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+\text{Tr}\left({\sigma}^{2}\mathbf{W}{\mathbf{G}}_{i}{\mathbf{G}}_{i}^{H}\right)\phantom{\rule{2em}{0ex}}i=1,2\phantom{\rule{1em}{0ex}},\phantom{\rule{2em}{0ex}}\end{array}

(23)

where {\mathbf{L}}_{i}={\left({\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right)}^{-\frac{1}{2}}{\mathbf{G}}_{i}^{H}\mathbf{W}{\mathbf{G}}_{i}^{H}{\left({\mathbf{I}}_{N}+{\sigma}_{i}^{2}{\mathbf{R}}_{i}^{-1}\right)}^{-\frac{1}{2}} and \mathbf{K}={\mathbf{R}}_{r}^{\frac{1}{2}}{\mathbf{F}}_{r}{\mathbf{F}}_{r}^{H}{\mathbf{R}}_{r}^{\frac{1}{2}}.

Both source nodes are trying to minimize expected values of the WMSE_{1} and WMSE_{2} during the BC stage. We are not considering a greedy approach, i.e., every node is trying to minimize its own WMSE component. Equal proportions of WMSE are considered to find optimum precoder and decoders. Therefore, we consider the sum of two components, and the problem is formulated in Problem 2

##### Problem 2.

\begin{array}{l}\begin{array}{ll}\underset{{\mathbf{F}}_{r},{\mathbf{G}}_{1},{\mathbf{G}}_{2}}{\text{min}}& \frac{1}{2}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{1}\right\}+\frac{1}{2}{\mathcal{E}}_{\mathbf{E}}\left\{{\text{WMSE}}_{2}\right\}\\ \text{subject to}& \text{Tr}\left({\mathbf{F}}_{r}{\mathbf{F}}_{r}^{H}\right)\le {P}_{T}/2.\end{array}\end{array}

(24)

This problem is a non-convex problem, and we propose solutions in the next section.