### 3.1 Opportunistic relaying

Traditionally, there are two strategies of opportunistic relay selection: reactive relay selection and proactive relay selection (Fig. 2). Specifically, the reactive strategy first chooses the nodes to form a collection, which can successfully decode signals between the links of *S*-*R*. Then, the best relay is selected from the collection that maximizes the instantaneous channel gains of *R*-*D* link. Conversely, the proactive selection would choose the best relay prior to the source transmission, which depends on instantaneous channel gains of both *S*-*R* and *R*-*D* links. In this paper, we adopt proactive selection strategy in the hybrid FD/HD relay system.

In opportunistic proactive selection, the best relay *R*
^{*} is selected from a collection of *M* possible cooperative nodes prior to the source transmission. It requires that each relay should know its own instantaneous channel gains of *S*-*R* and *R*-*D* links. The best relay is chosen to maximize the minimum of the weighted channel gains between *S*-*R* and *R*-*D* links for all the *M* relays.

$$ \begin{array}{l}{R}^{\ast }= \arg \max {W}_k,\ k=1,\dots, M\\ {}{W}_k= \min \left\{\zeta {\gamma}_{sk},\left(1-\zeta \right){\gamma}_{kd}\right\},\end{array} $$

(6)

where *ζ* denotes power allocation at the source. In this case, the communication through the best relay would fail to outage while either *S* ‐ *R*
^{*} or *R*
^{*} ‐ *D* links occur outage.

Since the minimum of two independent exponential distributed variables also follows exponential distribution with the sum of the two parameters, which is shown as follows:

$$ {W}_k\sim E\left(\frac{1}{\zeta {\lambda}_{sk}}+\frac{1}{\left(1-\zeta \right){\lambda}_{kd}}\right). $$

(7)

From (7), we obtain the outage probability of opportunistic proactive relay system as follows:

$$ \begin{array}{l}{P}_{\mathrm{Opp}\hbox{-} \mathrm{pro}}^{\mathrm{outage}}= \Pr \left\{{W}_k<\frac{N_0\left({2}^{2R}-1\right)}{P_{\mathrm{tot}}}\right\}\\ {}\kern3em = \Pr \left\{ \max {W}_k<\frac{N_0\left({2}^{2R}-1\right)}{P_{\mathrm{tot}}}\right\}\\ {}\kern3em ={\displaystyle \prod_{k=1}^M \Pr \left\{{W}_k<\frac{N_0\left({2}^{2R}-1\right)}{P_{\mathrm{tot}}}\right\}}\\ {}\kern3em ={\displaystyle \prod_{k=1}^M\left[1- \exp \left\{-\frac{N_0\left({2}^{2R}-1\right)}{P_{\mathrm{tot}}}\left(\frac{1}{\zeta {\lambda}_{sk}}+\frac{1}{\left(1-\zeta \right){\lambda}_{kd}}\right)\right\}\right]}.\end{array} $$

(8)

In proactive selection strategy, selecting a single relay before the source transmission could potentially results in degraded performance. On the other hand, selecting a single relay for information forwarding before source transmission simplifies the receiver design and the overall network operation, since proactive selection is equivalent to routing. Although active selection is more precise because it depends on instantaneous CSI during the communication, it would cost additional spectrum during transmission to select best relay. Conversely, proactive selection can facilitate the receiver operation and system design while all the cooperative nodes stay idle except the opportunistic relay during the transmission. Thus, our opportunistic proactive relay strategy can be viewed as energy efficient.

### 3.2 Cooperative jamming with robust design

Unlike traditional opportunistic relay selection, except the chosen relay, the remaining nodes would stay idle. In our proposed scheme, the other cooperative nodes operate as jammers to suppress the eavesdropping process.

If the ECSI is known (for instance, the eavesdropper is an active user in the wireless network, it has to feed back its CSI), the source transmits designed CJ beamformer to enhance the security of system, which suppresses or eliminates the information leakage to eavesdropper. During the two time slots, cooperative jamming beamformer depends on *J*-*R* and *J*-*D* links, respectively; thus, jamming signals would have no influence on the chosen relay and legitimate *D*.

First, under the assumption that ECSI is available, we apply zero-forcing (ZF) algorithm on the jamming beamforming design, which requires that \( {\mathbf{h}}_{jr}^H{\mathbf{f}}_r=0 \). At time slot *t*, we assume that beamformer **f**
_{
r
} is a unit-normalized one-dimensional vector for the cooperative jamming. For the case of perfect CSI, the constraint of ZF algorithm is given by

$$ \begin{array}{l}\underset{\mathbf{f}}{ \max }\ \left|{\mathbf{h}}_{je}^H{\mathbf{f}}_r\right|\\ {}\mathrm{s}.\mathrm{t}.\kern0.5em {\mathbf{h}}_{jr}^H{\mathbf{f}}_r=0,\kern0.5em \left|{\mathbf{f}}_r^H{\mathbf{f}}_r\right|=1.\end{array} $$

(9)

The solution of (9) is referred as the null-steering beamformer, which is written as

$$ {\mathbf{f}}_r=\frac{\left({\mathbf{I}}_M-\mathbf{H}\right){\mathbf{h}}_{je}}{\left\Vert \left({\mathbf{I}}_M-\mathbf{H}\right){\mathbf{h}}_{je}\right\Vert }, $$

(10)

where \( \mathbf{H}={\mathbf{h}}_{jr}{\left({\mathbf{h}}_{jr}^H{\mathbf{h}}_{jr}\right)}^{-1}{\mathbf{h}}_{jr}^H \) is the orthogonal matrix onto the subspace spanned by **h**
_{
jr
}. On the other hand, the optimal CJ beamformer **f**
_{
d
} at time slot *t* + 1 can be obtained with the same way.

However, it is impractical for the relay to obtain perfect ECSI in some cases (channel estimation errors, feedback delay, and quantization errors), and we cannot optimize system performance without knowledge of the eavesdropper’s CSI. Thus, we follow the robust approach of [12] to find the optimal CJ beamformer under imperfect CSI. And the optimization problem becomes

$$ \begin{array}{l}\underset{{\mathbf{Q}}_z}{ \max}\underset{\mathbf{e}}{ \min }\ {\left({\tilde{\mathbf{h}}}_{je}+{\mathbf{e}}_{je}\right)}^H{\mathbf{Q}}_z\left({\tilde{\mathbf{h}}}_{je}+{\mathbf{e}}_{je}\right)\\ {}\mathrm{s}.\mathrm{t}.\kern1em {\mathbf{h}}_{jr}^H{\mathbf{Q}}_z{\mathbf{h}}_{jr}=0,\end{array} $$

(11)

where \( {\mathbf{Q}}_z={\mathbf{f}}_r{\mathbf{f}}_r^H \) denotes jamming matrix; problem (11) can be transformed to

$$ \begin{array}{l}\underset{{\mathbf{Q}}_z}{ \max }\ \mathrm{t}\mathrm{r}\left[\left({\mathbf{Q}}_z-\boldsymbol{\upxi} \right){\tilde{\mathbf{h}}}_{jr}{\tilde{\mathbf{h}}}_{jr}^H\right]-\kappa {\varepsilon}^2\\ {}\mathrm{s}.\mathrm{t}.\kern1em \left[\begin{array}{l}\boldsymbol{\upxi} \kern3em {\mathbf{Q}}_z\\ {}{\mathbf{Q}}_z\kern0.75em \kappa {\mathbf{I}}_M+{\mathbf{Q}}_z\end{array}\right] \succeq 0\\ {}\kern2.25em {\mathbf{Q}}_z \succeq 0,\kern0.75em \kappa \ge 0\\ {}\kern2.25em \boldsymbol{\upxi} \succeq {\mathbf{Q}}_z{\left(\kappa {\mathbf{I}}_M+{\mathbf{Q}}_z\right)}^H{\mathbf{Q}}_z\\ {}\kern2.25em {\tilde{\mathbf{h}}}_{jr}^H{\mathbf{Q}}_z{\tilde{\mathbf{h}}}_{jr}=0.\end{array} $$

(12)

*Proof* See Appendix.

Problem (12) is a SDP with a set of linear matrix inequality (LMI) constraints. Note that **e**
_{
je
} is not explicit in (12), the optimal robust matrix \( {\mathbf{Q}}_z^{\ast } \) is derived from the hidden **e**
_{
je
}, which can be obtained through the following problem:

$$ \begin{array}{l}\underset{\mathbf{e}}{ \min }\ {\left({\tilde{\mathbf{h}}}_{je}+{\mathbf{e}}_{je}\right)}^H{\mathbf{Q}}_z^{\ast}\left({\tilde{\mathbf{h}}}_{je}+{\mathbf{e}}_{je}\right)\\ {}\mathrm{s}.\mathrm{t}.\kern0.5em \left\Vert {\mathbf{e}}_{je}\right\Vert \le \varepsilon .\end{array} $$

(13)

Using the Lagrange theorem, the problem (13) can be written as

$$ L\left({\mathbf{e}}_{je},\lambda \right)={\mathbf{e}}_{je}^H\left({\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\right){\mathbf{e}}_{je}+2\mathrm{R}\mathrm{e}\left({\mathbf{e}}_{je}^H{\mathbf{Q}}_z^{\ast }{\tilde{\mathbf{h}}}_{je}\right)+{\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_z^{\ast }{\tilde{\mathbf{h}}}_{je}-\lambda {\varepsilon}^2, $$

(14)

where *λ* ≥ 0. When \( {\mathbf{e}}_{je}=-{\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_z^{\ast }{\left({\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\right)}^{-1} \), the minimum of *L*(**e**
_{
je
}, *λ*) is achieved. The problem (13) is written as

$$ \begin{array}{l}\underset{\lambda }{ \max }{\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_{\mathbf{z}}^{\ast }{\tilde{\mathbf{h}}}_{je}-\lambda {\varepsilon}^2-{\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_z^{\ast}\left({\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\right){\mathbf{Q}}_z^{\ast H}{\tilde{\mathbf{h}}}_{je}\\ {}\mathrm{s}.\mathrm{t}.\kern0.75em {\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\succeq 0,\kern0.5em {\mathbf{Q}}_z^{\ast H}{\tilde{\mathbf{h}}}_{je}\in R\left({\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\right),\end{array} $$

(15)

where *λ* is solution of the following problem:

$$ \begin{array}{l}\underset{\lambda \succeq 0,\eta }{ \max}\kern0.75em \eta \\ {}\mathrm{s}.\mathrm{t}.\kern0.5em \left[\begin{array}{l}{\mathbf{Q}}_z^{\ast }+\lambda {\mathbf{I}}_M\kern3.5em {\mathbf{Q}}_z^{\ast H}{\tilde{\mathbf{h}}}_{je}\\ {}{\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_z^{\ast}\kern2.25em {\tilde{\mathbf{h}}}_{je}^H{\mathbf{Q}}_z^{\ast }{\tilde{\mathbf{h}}}_{je}-\lambda {\varepsilon}^2-\eta \end{array}\right]\succeq 0.\end{array} $$

(16)

Problem (16) can be efficiently solved by well-studied interior-point algorithm-based package SeDuMi [18]. After \( {\mathbf{Q}}_z^{\ast } \) and **e**
_{
je
} are obtained, then, we can get the optimal **f**
_{
d
}. Meanwhile, the optimal **f**
_{
r
} can be obtained with the same approach.