Secure massive MIMO system with two-way relay cooperative transmission in 6G networks

Abstract

With the advent of Internet of Everything and the era of big data, massive multiple-input multiple-output (MIMO) is considered as an essential technology to meet the growing communication requirements for beyond 5G and the forthcoming 6G networks. This paper considers a secure massive MIMO system, where the legitimate user and the base station exchange messages via two-way relays with the presence of passive eavesdroppers. To achieve the trade-off between the physical-layer security and communication reliability, we design a cooperative transmission mode based on multiple-relay collaboration, where some relays broadcast the received signals and other relays act as friendly jammers to prevent the interception by eavesdroppers. A quantum chemical reaction optimization (QCRO) algorithm is proposed to find the most suitable scheme for multiple-relay collaboration. Simulation results highlight excellent performance of the proposed transmission mode under QCRO in different communication scenarios, which can be considered as a potential solution for the security issue in future wireless networks.

1 Introduction

In contemporary society, as the booming development of information technology and the popularity of intelligent equipment, wireless networks have become an essential part of our daily life [1]. Inheriting the benefits achieved in 5G, 6G network is expected to expand to a wider level to realize the full coverage of land and air [2,3,4,5]. As networks become denser, how to efficiently utilize the system resources and how to meet the higher transmission demands of ultra-high speed, high quality, and low latency have become key issues in 6G networks [3,4,5]. Massive multiple-input multiple-output (MIMO) is an effective solution for the increasing challenge of wireless data traffic since it can serve a large number of IoT devices at the same time [6,7,8,9]. By utilizing large antenna arrays, massive MIMO can offer a significant improvement in system capacity, and can serve a large variety of devices at the same time, which can greatly improve the quality of service (QoS) and spectral efficiency of communication systems [10].

The openness and sharing of nature of wireless propagation channels make it easy for any smart device to get information. Since more information will be transmitted through wireless environment in the era of 6G communications, it is crucial to build a secure communication network [11, 12]. Compared with single-antenna transmission architecture, massive MIMO has significant advantages in building secure communication networks. The characteristic of large number of antennas at the base station (BS) makes it easy to transmit several directional beams to the desired terminals. As the size of the antenna array grows infinite, the beam formed by the BS will become narrow and have high directional selectivity, thus effectively reduce the risk of information interception. However, in the actual systems, the number of antennas is finite due to huge hardware cost, and the eavesdroppers will make every effort to intercept or break the information transmission. Typically, cryptographic encryption method was used to maintain the system privacy [13,14,15], but the implementation complexity is too high due to the cumbersome process of service management. By exploiting wireless transmission properties, physical-layer security has attracted considerable attentions [16, 17]. In order to protect the transmission security, several efforts such as artificial noise [18,19,20], antenna/relay selection [21,22,23], and cooperative strategies [24,25,26,27] were employed to deal with the information leakage by eavesdroppers. The utilization of antenna correlation diversity was demonstrated to improve the secrecy rate of massive MIMO systems [19]. With the assistance of cooperative relays, the authors in [24] and [25] investigated the secure transmission problem in the presence of passive and active eavesdroppers. The secrecy performance under a practical scenario where the eavesdroppers hide their existence in the system was investigated in [26]. For secure information transmission between the BS and the destination, the authors in [21] proposed a destination-based cooperative jamming scheme to improve the system capacity and transmission confidentiality.

Among recent works on the security issue, the combinations of cooperative relays and secure techniques have shown significant improvements on secrecy capacity and system coverage in massive MIMO networks. Due to the low-complexity relaying process and high spectral efficiency outperforms one-way relay network, two-way relays have attracted extensive attentions [28,29,30,31]. In two-way relay networks, eavesdroppers will receive the overlapped information when the two legitimate devices broadcast their signals at the same time [31]. In this condition, both two devices act as source and jammer to prevent the interception by eavesdroppers. By exploiting the dual property, the authors in [32, 33] investigated distributed relay selection criterion to optimize the system security performance. In order to enhance the spatial diversity gains and further improve the system performance, massive MIMO with two-way relay collaboration has become a hotspot in recent researches [34,35,36,37]. Feng et al. [35] indicated that the system performance with full-duplex operation outperforms the half-duplex mode under a certain self-loop interference. Zhang et al. [36] investigated the overall performance with hardware impairments. Pan et al. [37] adopted a power allocation strategy based on geometric programming to reduce the information interception of untrusted users. These works, however, did not consider the information leakage happens at the two phases of data transmission.

1.1 Motivation and contributions

Although existing researches have made influential achievements in improving the security performance, there are still some limitations. For better understanding, the comparison of previous works with our work is presented in Table 1. Many anti-eavesdropping methods [16,17,18,19,20] generally consider the case of direct transmission between the BS and the users, but it’s hard to meet the QoS demands of the desired terminals where there are no direct transmission links due to the long-distance fading.

Table 1 Comparison with existing works

For secure transmission issues with relay collaboration, most of the existing works [21,22,23,24,25,26,27,28,29,30,31,32,33, 37] do not simultaneously address the power allocation, resource utilization, and multiple relay selection in a scenario where information leakage happens at both transmission phases in massive MIMO networks with limited time–frequency resources. In this paper, we present a cooperative transmission mode for the security issue of massive MIMO two-way relay networks. Unlike one-way relaying systems, both the two legitimate devices play the roles of source and jammer via two-way relaying operation. However, the duality has not been well exploited in [28, 29, 31] to prevent the interception by eavesdroppers. In order to break through the limitations of previous studies and obtain a higher security performance, we propose a multiple-relay collaboration strategy considering the interception of eavesdroppers during the two phases of information transmission. According to the multiple-relay selection (MRS) scheme, the relays function as receivers/transmitters or friendly jammers to reduce the information leakage between the source and the legitimate user. Considering the energy conservation and communication reliability requirements, we propose a quantum chemical reaction optimization (QCRO) algorithm to obtain the optimal MRS result. The major contributions of our work are summarized in the following:

• We propose a low-complexity cooperative relaying and cooperative jamming (CRCJ) transmission mode to achieve the trade-off between the communication security and reliability of a massive MIMO two-way relay system. The multiple-relay collaboration strategy is employed to enhance the secrecy performance while prevent the interception by eavesdroppers in two transmission phases.

• Exact expression for secrecy sum-rate is derived considering the information leakage of two transmission phases in massive MIMO relay networks. The expression indicates that secrecy performance is affected by the transmission power of relays and MRS result with QoS and interference constraints in practical scenarios.

• A novel algorithm named QCRO is proposed to tackle the complicated MRS problem for the cooperative transmission in massive MIMO networks. Simulation results illustrate the excellent performance of QCRO over conventional algorithms in various system parameters. Besides, the proposed QCRO algorithm can be considered as a potential solution for other complicated problems in communication domain.

1.2 Organization and notations

The other sections of this work are presented as follows. Section 2 presents the system architecture and the analysis of a secure massive MIMO network with multiple-relay collaboration. The QCRO algorithm for MRS is addressed in Sect. 3, and Sect. 4 presents the simulations. In the end, we conclude this work in the final section.

Notations: The uppercase boldface symbols and lowercase boldface symbols represent matrixes and the vectors, respectively. ${\mathbf{0}}$ and ${\varvec{I}}_{N}$ represent the zero vector and identity matrix, respectively. ${\mathcal{C}\mathcal{N}}(\mu ,\sigma^{2} )$ denotes the complex Gaussian distribution with mean of $\mu$ and variance of $\sigma^{2}$. $(.)^{{\text{T}}}$ and $(.)^{{\text{H}}}$ represent transpose operator and conjugate transpose operator, respectively. $|.|$, $||.||$, and ${\text{abs}}(.)$ represent modulus, Euclidean norm, and absolute value functions, respectively. $E\{ .\}$ denotes the statistical expectation, and $[x]^{ + }$ stands for $\max \left\{ {0,x} \right\}$.

2 System model

In this paper, we consider a secure massive MIMO two-way relay network where there is a base station (BS) with $M_{t}$ antennas, a user, $L$ relays, and $K$ eavesdroppers as shown in Fig. 1. In order to facilitate practical implementation, the user, relays, and eavesdroppers are deployed with a single antenna. Since the direct transmission link between the BS and the user is so weak due to the long-distance fading, the BS and the user exchange their information via two-way relays. With the help of two-way relays, the information transmission can be divided into two phases. In the first phase, both the BS and the user transmit their signals to the two-way relays. In the second phase, the relays amplify and forward the received signals to the legitimate devices. The self-interference cancellation (SIC) is performed at the BS/user side [32]. By removing their own self-interference, both the BS and the user can get their desired signals. However, eavesdroppers cannot separate the desired information from the superimposed signal without the knowledge of prior information about the BS and the user, which makes it difficult to decipher the intended information. When eavesdroppers try to intercept the signal of the user, the signal from the BS can be regarded as interference. The same goes for the BS. However, when the eavesdropper is closely located at the legitimate device, it requires a stronger anti-eavesdropping mechanism to protect the transmission security. In this case, we develop a cooperative relaying and cooperative jamming (CRCJ) transmission mode, i.e., a portion of them act as cooperative relays assist in the information transmission between the BS and the user, other relays act as jammers to prevent the information leakage by the passive eavesdroppers.

Fig. 1

Diagram of the secure massive MIMO two-way relay network

Regarding propagation model, the channel state information (CSI) is defined as ${\varvec{g}}_{X,Y} = {\varvec{h}}_{X,Y} d_{X,Y}^{ - \xi /2}$ [26], where ${\varvec{h}}_{X,Y} \sim {\mathcal{C}\mathcal{N}}({\mathbf{0}},{\varvec{I}}_{N} )$ denotes the small-scale fading factor, and $d_{X,Y}^{ - \xi /2}$ denotes the large-scale fading factor. In it, $\xi$ denotes the path loss exponent, and $d_{X,Y}$ denotes the distance between the X − Y link. The global CSI is available by uplink training in time-division duplexed (TDD) massive MIMO networks [11, 38]. We consider the instantaneous CSI remains unchanged in one time slot. The definitions of CSI between the terminals of secure massive MIMO two-way relay networks are as follows:

• CSI from the BS to the $i{\text{ - th}}$ relay: ${\varvec{g}}_{{BS,r_{i} }} \in {\mathbb{C}}^{{1 \times M_{t} }}$.

• CSI from the BS to the $k{\text{ - th}}$ eavesdropper: ${\varvec{g}}_{{BS,e_{k} }} \in {\mathbb{C}}^{{1 \times M_{t} }}$.

• CSI from the user to the $k{\text{ - th}}$ eavesdropper: $g_{{u,e_{k} }}$.

• CSI from the $i{\text{ - th}}$ relay to the user: $g_{{r_{i} ,u}}$.

• CSI from the $i{\text{ - th}}$ relay to the $k{\text{ - th}}$ eavesdropper: $g_{{r_{i} ,e_{k} }}$.

• CSI from the $i{\text{ - th}}$ relay to the $j{\text{ - th}}$ relay ($i \ne j$): $g_{{r_{i} ,r_{j} }}$.

In the following, we will introduce the policy of the CRCJ mode for the secure massive MIMO two-way relay network.

2.1 CRCJ policy

Under CRCJ policy, the relays are divided into two classes for different purposes, i.e., some relays are selected to receive the mixed signal from the BS and the user, while the remaining relays act as jammers to transmit jamming signals to the eavesdroppers. For simplicity, the MRS scheme is expressed by a binary vector ${\varvec{b}} = [b_{1} ,b_{2} ,...,b_{L} ]$, where $b_{i} \in \{ 0,1\}$, $i = 1,2,...,L$. If $b_{i} = 1$, the $i{\text{ - th}}$ relay is selected to assist information transmission between the legitimate devices.

In the first phase, the BS and the user simultaneously broadcast their signals with transmit power $p_{BS}$ and $p_{u}$. The signal received at the $i{\text{ - th}}$ relay is given by

$$y_{i} = \sqrt {p_{BS} } {\varvec{g}}_{{BS,r_{i} }} {\varvec{Vs}}_{BS} + \sqrt {p_{u} } g_{{r_{i} ,u}} s_{u} + \sum\limits_{j = 1}^{L} {(1 - b_{j} )\sqrt {p_{{r_{j} }} } g_{{r_{i} ,r_{j} }} s_{{r_{j} }} } + \eta_{{r_{i} }}$$

(1)

where $p_{{r_{j} }}$ is the transmit power of the j-th jammer; ${\varvec{V}}$ is the precoding matrix at the BS; ${\varvec{s}}_{BS}$, $s_{u}$, and $s_{{r_{j} }}$ are unit-power signals with $E\{ ||{\varvec{s}}_{BS} ||^{2} \} = 1$, $E\{ |s_{u} |^{2} \} = 1$, and $E\{ |s_{{r_{j} }} |^{2} \} = 1$, respectively; $\eta_{{r_{i} }}$ is the additive white Gaussian noise (AWGN). For the $k{\text{ - th}}$ eavesdropper, the intercepted signal in the first phase can be expressed as

$$y_{{e_{k} }}^{(1)} = \sqrt {p_{BS} } {\varvec{g}}_{{BS,e_{k} }} {\varvec{Vs}}_{BS} + \sqrt {p_{u} } g_{{u,e_{k} }} s_{u} + \sum\limits_{j = 1}^{L} {(1 - b_{j} )\sqrt {p_{{r_{j} }} } g_{{r_{j} ,e_{k} }} s_{{r_{j} }} } + \eta_{{e_{k} }}^{(1)}$$

(2)

where $\eta_{{e_{k} }}^{(1)}$ is the AWGN. The signal-to-interference plus noise ratio (SINR) received at the $k{\text{ - th}}$ eavesdropper from the two legitimate devices (BS and user) can be respectively given by

$$\gamma_{{BS \to e_{k} }}^{(1)} = \frac{{p_{BS} \left\| {{\varvec{g}}_{{BS,e_{k} }} {\varvec{V}}} \right\|^{2} }}{{p_{u} \left| {g_{{u,e_{k} }} } \right|^{2} + \sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{j} ,e_{k} }} } \right|^{2} } + \sigma_{0}^{2} }}$$

(3)

$$\gamma_{{u \to e_{k} }}^{(1)} = \frac{{p_{u} \left| {g_{{u,e_{k} }} } \right|^{2} }}{{p_{BS} \left\| {{\varvec{g}}_{{BS,e_{k} }} {\varvec{V}}} \right\|^{2} + \sum\nolimits_{j = 1}^{L} {(1 - b_{j} )p_{{r_{j} }} \left| {g_{{r_{j} ,e_{k} }} } \right|^{2} } + \sigma_{0}^{2} }}$$

(4)

where $\sigma_{0}^{2}$ denotes the power of AWGN.

In the second phase of data transmission, the selected relays broadcast their received signals to the legitimate devices. We denote $p_{i}$ as the transmit power of the $i{\text{ - th}}$ relay. For the $i{\text{ - th}}$ relay, the normalized transmission signal is expressed as

$$y_{i}^{{{\text{norm}}}} = \sqrt {G_{i} } y_{i} = \frac{{y_{i} }}{{\sqrt {p_{BS} \left\| {{\varvec{g}}_{{BS,r_{i} }} {\varvec{V}}} \right\|^{2} + p_{u} |g_{{r_{i} ,u}} |^{2} + \sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } }}$$

(5)

where $G_{i} = \left( {p_{BS} \left\| {{\varvec{g}}_{{BS,r_{i} }} {\varvec{V}}} \right\|^{2} + p_{u} |g_{{r_{i} ,u}} |^{2} + \sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } \right)^{ - 1}$ is the normalization factor. Since the BS and the user can distinguish their transmitted signals, by removing their own self-interference, the corresponding signals at the BS and the user can be respectively expressed by

$${\varvec{y}}_{BS} = \sum\limits_{i = 1}^{L} {b_{i} \sqrt {G_{i} p_{i} } {\varvec{Wg}}_{{BS,r_{i} }}^{{\text{T}}} \cdot \left( {\sqrt {p_{u} } g_{{r_{i} ,u}} s_{u} + \sum\limits_{j = 1}^{L} {\left( {1 - b_{j} } \right)\sqrt {p_{{r_{j} }} } g_{{r_{i} ,r_{j} }} s_{{r_{j} }} } + \eta_{{r_{i} }} } \right)} + {\varvec{\eta}}_{BS}$$

(6)

and

$$y_{u} = \sum\limits_{i = 1}^{L} {b_{i} \sqrt {G_{i} p_{i} } g_{{r_{i} ,u}} \cdot \left( {\sqrt {p_{BS} } {\varvec{g}}_{{BS,r_{i} }} {\varvec{Vs}}_{BS} + \sum\limits_{j = 1}^{L} {\left( {1 - b_{j} } \right)\sqrt {p_{{r_{j} }} } g_{{r_{i} ,r_{j} }} s_{{r_{j} }} } + \eta_{{r_{i} }} } \right)} + \eta_{u}$$

(7)

where ${\varvec{W}}$ is the receiving matrix at the BS; ${\varvec{\eta}}_{BS}$ and $\eta_{u}$ are AWGN. The SINRs of the BS and the user are respectively shown by

$$\gamma_{u \to BS} = \frac{{\sum\nolimits_{i = 1}^{L} {b_{i} G_{i} p_{i} p_{u} \left\| {{\varvec{Wg}}_{{BS,r_{i} }}^{{\text{T}}} } \right\|^{2} \cdot \left| {g_{{r_{i} ,u}} } \right|^{2} } }}{{\sum\nolimits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left\| {{\varvec{Wg}}_{{BS,r_{i} }}^{{\text{T}}} } \right\|^{2} \cdot \left( {\sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }}\left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } \right)} \right]} + \sigma_{0}^{2} }}$$

(8)

and

$$\gamma_{BS \to u} = \frac{{\sum\nolimits_{i = 1}^{L} {b_{i} G_{i} p_{i} p_{BS} \left| {g_{{r_{i} ,u}} } \right|^{2} \cdot \left\| {{\varvec{g}}_{{BS,r_{i} }} {\varvec{V}}} \right\|^{2} } }}{{\sum\nolimits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left| {g_{{r_{i} ,u}} } \right|^{2} \cdot \left( {\sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } \right)} \right]} + \sigma_{0}^{2} }}$$

(9)

Then, the instantaneous transmission rates of the BS and the user are respectively given by

$$C_{BS \to u} = \frac{1}{2}\log_{2} \left( {1 + \gamma_{BS \to u} } \right)$$

(10)

and

$$C_{u \to BS} = \frac{1}{2}\log_{2} \left( {1 + \gamma_{u \to BS} } \right)$$

(11)

Without the knowledge of the CSI between the legitimate devices and cooperative relays, the eavesdroppers cannot separate the superimposed signal [32]. For the $k{\text{ - th}}$ eavesdropper, the intercepted signal in the second phase can be given by

$$\begin{aligned} y_{{e_{k} }}^{(2)} & = \sum\limits_{i = 1}^{L} {b_{i} \sqrt {G_{i} p_{i} } g_{{r_{i} ,e_{k} }} y_{i} } + \eta_{{e_{k} }}^{(2)} \\ & = \sum\limits_{i = 1}^{L} {b_{i} \sqrt {G_{i} p_{i} } g_{{r_{i} ,e_{k} }} \cdot \left( {\sqrt {p_{BS} } {\varvec{g}}_{{BS,r_{i} }} {\varvec{Vs}}_{BS} + \sqrt {p_{u} } g_{{r_{i} ,u}} s_{u} + \sum\limits_{j = 1}^{L} {(1 - b_{j} )\sqrt {p_{{r_{j} }} } g_{{r_{i} ,r_{j} }} s_{{r_{j} }} } + \eta_{{r_{i} }} } \right)} + \eta_{{e_{k} }}^{(2)} \\ \end{aligned}$$

(12)

where $\eta_{{e_{k} }}^{(2)}$ is the AWGN. According to (12), the SINR received at the $k{\text{ - th}}$ eavesdropper from the BS and the user can be respectively shown by

$$\gamma_{{BS \to e_{k} }}^{(2)} = \frac{{\sum\nolimits_{i = 1}^{L} {b_{i} G_{i} p_{i} p_{BS} \left| {g_{{r_{i} ,e_{k} }} } \right|^{2} \cdot \left\| {{\varvec{g}}_{{BS,r_{i} }} {\varvec{V}}} \right\|^{2} } }}{{\sum\nolimits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left| {g_{{r_{i} ,e_{k} }} } \right|^{2} \cdot \left( {p_{u} \left| {g_{{r_{i} ,u}} } \right|^{2} + \sum\nolimits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } \right)} \right]} + \sigma_{0}^{2} }}$$

(13)

$$\gamma_{{u \to e_{k} }}^{(2)} = \frac{{\sum\nolimits_{i = 1}^{L} {b_{i} G_{i} p_{i} p_{u} \left| {g_{{r_{i} ,e_{k} }} } \right|^{2} \cdot \left| {g_{{r_{i} ,u}} } \right|^{2} } }}{{\sum\nolimits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left| {g_{{r_{i} ,e_{k} }} } \right|^{2} \cdot \left( {p_{BS} \left\| {{\varvec{g}}_{{BS,r_{i} }} {\varvec{V}}} \right\|^{2} + \sum\limits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } + \sigma_{0}^{2} } \right)} \right]} + \sigma_{0}^{2} }}$$

(14)

Consider the situation that the eavesdroppers are independent of each other, the total received SINR at the passive eavesdroppers from the BS and the user can be respectively expressed by [21].

$$\gamma_{BS}^{{\text{E}}} = \mathop {\max }\limits_{{k \in \{ 1,2, \ldots ,K\} }} \left\{ {\gamma_{{BS \to e_{k} }}^{(1)} ,\gamma_{{BS \to e_{k} }}^{(2)} } \right\}$$

(15)

$$\gamma_{u}^{{\text{E}}} = \mathop {\max }\limits_{{k \in \{ 1,2, \ldots ,K\} }} \left\{ {\gamma_{{u \to e_{k} }}^{(1)} ,\gamma_{{u \to e_{k} }}^{(2)} } \right\}$$

(16)

The information leakage from the BS and the user are respectively shown by

$$C_{BS}^{{\text{E}}} = \frac{1}{2}\log_{2} \left( {1 + \gamma_{BS}^{{\text{E}}} } \right)$$

(17)

$$C_{u}^{{\text{E}}} = \frac{1}{2}\log_{2} \left( {1 + \gamma_{u}^{{\text{E}}} } \right)$$

(18)

Hence, the secrecy transmission rates of the BS and the user can be given by

$$C_{BS \to u}^{{\text{S}}} = \left[ {C_{BS \to u} - C_{BS}^{{\text{E}}} } \right]^{ + }$$

(19)

$$C_{u \to BS}^{{\text{S}}} = \left[ {C_{u \to BS} - C_{u}^{{\text{E}}} } \right]^{ + }$$

(20)

Finally, the secrecy sum-rate of the secure massive MIMO two-way relay network can be expressed by

$$C_{{{\text{sum}}}} = \left[ {C_{BS \to u} - C_{BS}^{{\text{E}}} ]^{ + } + [C_{u \to BS} - C_{u}^{{\text{E}}} } \right]^{ + }$$

(21)

2.2 Problem formulation

Based on the CRCJ policy, the problem of secrecy sum-rate maximization based on MRS is formulated as

$${\text{max }}C_{{{\text{sum}}}} ({\varvec{b}}) = \max \left\{ {\left[ {C_{BS \to u} ({\varvec{b}}) - C_{BS}^{{\text{E}}} ({\varvec{b}})} \right]^{ + } + \left[ {C_{u \to BS} ({\varvec{b}}) - C_{u}^{{\text{E}}} ({\varvec{b}})} \right]^{ + } } \right\}$$

(22a)

subject to

$$b_{i} \in \left\{ {0,1} \right\}\quad \forall i \in \{ 1,2, \ldots ,L\}$$

(22b)

$$\sum\limits_{i = 1}^{L} {b_{i} } \ge 1$$

(22c)

$$0 \le p_{i} , \, p_{{r_{j} }} \le p_{\max } \quad \forall i, \, j \in \left\{ {1,2, \ldots ,L} \right\}$$

(22d)

$$\sum\limits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left| {g_{{r_{i} ,u}} } \right|^{2} \cdot \left( {\sum\limits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } } \right)} \right]} \le Interference$$

(22e)

$$\sum\limits_{i = 1}^{L} {\left[ {b_{i} G_{i} p_{i} \left\| {{\varvec{Wg}}_{{BS,r_{i} }}^{{\text{T}}} } \right\|^{2} \cdot \left( {\sum\limits_{j = 1}^{L} {\left( {1 - b_{j} } \right)p_{{r_{j} }} \left| {g_{{r_{i} ,r_{j} }} } \right|^{2} } } \right)} \right]} \le Interference$$

(22f)

where $p_{\max }$ denotes the maximum transmission power of relays. The relay selection constraints are shown in (22b) and (22c), where (22b) indicates that each relay acts as a jammer or a cooperator to participate in transmission, and (22c) indicates that at least one relay is selected to broadcast confidential signals to the legitimate devices. (22d) denotes the power constraints of cooperative relays and cooperative jammers. (22e) and (22f) are interference constraints, and $Interference$ denotes the maximum interference allowed to the legitimate device. Due to the cooperative relaying and cooperative jamming policy, the jamming signals transmitted to eavesdroppers can also cause interference to the BS and the legitimate user. Therefore, in order to guarantee the transmission communication quality of the BS and the legitimate user, we consider interference constraints as in (22e) and (22f). However, these constraints are too complicated to address, which make the problem more difficult to solve. Since the interference is caused by the jammers, it’s more convenient to restrict the interference of jammers to the cooperative relays. Hence, to guarantee the system communication quality, we make the interference from the cooperative jammers should not exceed the maximum interference threshold $I_{{{\text{th}}}}$ of cooperative relays. For the sake of simplifying (22e) and (22f), we can convert them by limiting the transmit power of the relays as follows

$$p_{{r_{j} }} = \min \left\{ {p_{\max } ,\frac{{I_{{{\text{th}}}} }}{{L^{\prime}\left| {\mathop {\max }\limits_{{i \in \{ 1,2, \ldots ,L\} }} \left\{ {\left( {1 - b_{j} } \right) \cdot g_{{r_{i} ,r_{j} }} } \right\}} \right|^{2} }}} \right\}$$

(23)

where $L^{\prime}$ denotes the number of cooperative jammers. In this way, the interference constraint can be satisfied for any multiple-relay selection results. Then, we can obtain a higher secrecy rate while ensuring the quality of information transmission, thus achieving the trade-off between communication security and reliability.

Obviously, the optimization problem of (22) is a multi-constraint nonlinear programming problem, and the computational complexity exponentially increases with the number of relays, which is NP-hard to solve. However, traditional algorithms are difficult to get the good performance due to the slow convergence speed and poor convergence accuracy. To efficiently tackle the complicated problem, we propose a quantum chemical reaction optimization (QCRO) algorithm to obtain the appropriate solution.

3 Methods

In this section, a novel intelligent algorithm named QCRO is proposed for multiple-relay selection in a secure massive MIMO two-way relay network. Inspired by the chemical reaction optimization (CRO) algorithm [39] and quantum evolutionary theory [9], QCRO employs a set of quantum molecules, which is varied by different quantum evolutionary rules. Here we introduce the principle of QCRO.

3.1 QCRO for optimization problem

In an $L$-dimensional space, (where $L$ is the maximal dimension of the problem), there exist $N$ quantum molecules. The $n$-th quantum molecule ($n = 1,2, \ldots ,N$) of the $t{\text{ - th}}$ iteration is given by

$${\varvec{x}}_{n}^{t} = \left[ {x_{n,1}^{t} ,x_{n,2}^{t} , \ldots ,x_{n,L}^{t} } \right]$$

(24)

where $0 \le x_{n,l}^{t} \le 1$; $n = 1,2,...,N$; $l = 1,2,...,L$; $x_{n,l}^{t}$ denotes the $l{\text{ - th}}$ quantum bit of the $n{\text{ - th}}$ quantum molecule. For each quantum molecule, the quantum bits should be measured to the solution domain. The measurement state of the n-th quantum molecule can be obtained by the following rule:

$$\overline{x}_{n,l}^{t} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {\alpha_{n,l}^{t} > (x_{n,l}^{t} )^{2} } \hfill \\ {0,} \hfill & {\alpha_{n,l}^{t} \le (x_{n,l}^{t} )^{2} } \hfill \\ \end{array} } \right.$$

(25)

where $\overline{x}_{n,l}^{t}$ denotes the $l{\text{ - th}}$ measurement state of the n-th quantum molecule, and $\alpha_{n,l}^{t}$ is a random number distributed in [0,1].

The fitness value of the $n{\text{ - th}}$ quantum molecule is calculated by the fitness function, which can be expressed as $f(\overline{\user2{x}}_{n}^{t} )$. For the maximum optimization problem, the global optimal solution $\overline{\user2{\rho }}_{{{\text{best}}}}^{t} = \left[ {\overline{\rho }_{{{\text{best}},1}}^{t} ,\overline{\rho }_{{{\text{best}},2}}^{t} , \ldots ,\overline{\rho }_{{{\text{best}},L}}^{t} } \right]$ is denoted as the measurement state of the quantum molecule with the maximum fitness value until the $t{\text{ - th}}$ iteration.

In QCRO, the quantum molecules are updated by collision, decomposition, and synthesis. These processes are related to the kinetic energy (KE) of quantum molecules, where the top $\mu_{1} N$ quantum molecules with the highest KE are updated by collision, $\mu_{2} N$ quantum molecules with the smallest KE are updated by synthesis, and the remaining $\mu_{3} N$ quantum molecules are updated by decomposition. $\mu_{1}$, $\mu_{2}$, and $\mu_{3}$ are constants which respectively represent the reaction ratio of collision, synthesis, and decomposition. To make it easy, we sort the quantum molecules in a descending order according to the level of KE, and the $n^{\prime}{\text{ - th}}$ quantum molecule is denoted by ${\varvec{x}}_{{n^{\prime}}}^{t} = \left[ {x_{{n^{\prime},1}}^{t} ,x_{{n^{\prime},2}}^{t} , \ldots ,x_{{n^{\prime},L}}^{t} } \right]$ with the KE of $e_{{n^{\prime}}}^{t}$. The generation of new quantum molecules is related to the quantum rotation angle and measurement states of previous quantum molecules. For collision, the $n^{\prime}{\text{ - th}}$ quantum molecule is updated to a quantum molecule $m$, the quantum rotation angle and KE are given by

$$\theta_{m,l}^{t + 1} = c_{1} \cdot \left( {\overline{x}_{g,l}^{t} - \overline{x}_{{n^{\prime},l}}^{t} } \right)$$

(26)

$$e_{m}^{t + 1} = e_{{n^{\prime}}}^{t} \cdot \left( {1 - \vartriangle e} \right)$$

(27)

where $m = n^{\prime}$; $n^{\prime} = 1,2, \ldots ,\mu_{1} N$; $l = 1,2, \ldots ,L$; $\overline{x}_{g,l}^{t}$ denotes the $l{\text{ - th}}$ measurement state of the $g{\text{ - th}}$ quantum molecule with higher fitness value in the $t{\text{ - th}}$ iteration,$g \in \left\{ {1,2, \ldots ,N} \right\}$, $g \ne n^{\prime}$. $c_{1}$ represents the weight coefficient, and $\vartriangle e$ represents the loss rate of KE.

For decomposition, the $n^{\prime}{\text{ - th}}$ quantum molecule is decomposed into quantum molecule $m$ and $m + 1$, and the quantum rotation angles are respectively shown as

$$\theta _{{m,l}}^{{t + 1}} = \left\{ {\begin{array}{*{20}l} {c_{2} \cdot \left( {\bar{\rho }_{{{\text{best}},l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} } \right) + c_{3} \cdot (\bar{x}_{{a,l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} ),} \hfill & {\varphi _{{m,l}}^{{t + 1}} < \kappa _{1} } \hfill \\ {c_{4} \cdot \left( {\bar{x}_{{g,l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} } \right),} \hfill & {{\text{else}}} \hfill \\ \end{array} } \right.$$

(28)

$$\theta _{{m + 1,l}}^{{t + 1}} = \left\{ {\begin{array}{*{20}l} {c_{2} \cdot \left( {\bar{\rho }_{{{\text{best}},l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} } \right) + c_{3} \cdot \left( {\bar{x}_{{a,l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} } \right),} \hfill & {\varphi _{{m + 1,l}}^{{t + 1}} < \kappa _{1} } \hfill \\ {c_{4} \cdot \left( {\bar{x}_{{g,l}}^{t} - \bar{x}_{{n^{\prime},l}}^{t} } \right),} \hfill & {{\text{else}}} \hfill \\ \end{array} } \right.$$

(29)

where $m = 2n^{\prime} - \mu_{1} N - 1$; $n^{\prime} = \mu_{1} N + 1,\mu_{1} N + 2,...,\mu_{1} N + \mu_{3} N$; $l = 1,2,...,L$; $g \in \{ 1,2,...,N\}$, $g \ne n^{\prime}$; $\overline{x}_{a,l}^{t}$ denotes the $l{\text{ - th}}$ measurement state of a random quantum molecule in the $t{\text{ - th}}$ iteration, $a \in \{ 1,2,...,N\}$, $a \ne n^{\prime}$; $c_{2}$, $c_{3}$ and $c_{4}$ are weight coefficients. $\kappa_{1}$ is the mutation probability which is a fixed parameter that determines the decomposition style, $\varphi_{m,l}^{t + 1}$ and $\varphi_{m + 1,l}^{t + 1}$ are random variables distributed from 0 to 1. The KE of quantum molecule $m$ and $m + 1$ can be expressed by

$$e_{m}^{t + 1} = e_{{n^{\prime}}}^{t} \cdot {{\left( {1 - \vartriangle e} \right)} \mathord{\left/ {\vphantom {{\left( {1 - \vartriangle e} \right)} 2}} \right. \kern-0pt} 2}$$

(30)

$$e_{m + 1}^{t + 1} = e_{{n^{\prime}}}^{t} \cdot {{\left( {1 - \vartriangle e} \right)} \mathord{\left/ {\vphantom {{\left( {1 - \vartriangle e} \right)} 2}} \right. \kern-0pt} 2}$$

(31)

For synthesis, the $n^{\prime}{\text{ - th}}$ quantum molecule and the $(n^{\prime} + 1){\text{ - th}}$ quantum molecule are synthesized into a new quantum molecule $m$, the quantum rotation angle and KE are given by

$$\theta_{m,l}^{t + 1} = c_{5} \cdot \omega_{m,l}^{t} \cdot \left( {\overline{x}_{{n^{\prime} + 1,l}}^{t} - \overline{x}_{{n^{\prime},l}}^{t} } \right)$$

(32)

$$e_{m}^{{t + 1}} = \left( {e_{{n^{\prime}}}^{t} + e_{{n^{\prime} + 1}}^{t} } \right) \cdot \left( {1 - \vartriangle e} \right)$$

(33)

where $m = (n^{\prime} + \mu_{1} N + 3\mu_{3} N + 1)/2$; $n^{\prime} = \mu_{1} N + \mu_{3} N + 1,\mu_{1} N + \mu_{3} N + 3,...,N - 1$; $l = 1,2,...,L$; $c_{5}$ represents the weight coefficient; and $\omega_{m,l}^{t}$ is a random variable distributed from 0 to 1.

The $m{\text{ - th}}$ and $(m + 1){\text{ - th}}$ updated quantum molecules can be respectively obtained by

$$x_{m,l}^{t + 1} = \left\{ {\begin{array}{*{20}l} {\sqrt {1 - (x_{{n^{\prime},l}}^{t} )^{2} } ,} \hfill & {\theta_{m,l}^{t + 1} = 0\;{\text{and}}\;\hat{\varphi }_{m,l}^{t + 1} \le \kappa_{2} } \hfill \\ {{\text{abs}}\left( {x_{{n^{\prime},l}}^{t} \cdot \cos \theta_{m,l}^{t + 1} - \sqrt {1 - (x_{{n^{\prime},l}}^{t} )^{2} } \cdot \sin \theta_{m,l}^{t + 1} } \right),} \hfill & {{\text{else}}} \hfill \\ \end{array} } \right.$$

(34)

$$x_{m + 1,l}^{t + 1} = \left\{ {\begin{array}{*{20}l} {\sqrt {1 - \left( {x_{{n^{\prime},l}}^{t} } \right)^{2} } ,} \hfill & {\theta_{m + 1,l}^{t + 1} = 0\;{\text{and}}\;\hat{\varphi }_{m + 1,l}^{t + 1} \le \kappa_{2} } \hfill \\ {{\text{abs}}\left( {x_{{n^{\prime},l}}^{t} \cdot \cos \theta_{m + 1,l}^{t + 1} - \sqrt {1 - (x_{{n^{\prime},l}}^{t} )^{2} } \cdot \sin \theta_{m + 1,l}^{t + 1} } \right),} \hfill & {{\text{else}}} \hfill \\ \end{array} } \right.$$

(35)

where $\hat{\varphi }_{m,l}^{t + 1}$ and $\hat{\varphi }_{m + 1,l}^{t + 1}$ are random variables distributed from 0 to 1, $\kappa_{2}$ denotes the conversion probability, and ${\text{abs}}(.)$ represents the absolute value function. For collision, $m = n^{\prime}$; $n^{\prime} = 1,2,...,\mu_{1} N$. For decomposition, $m = 2n^{\prime} - \mu_{1} N - 1$; $n^{\prime} = \mu_{1} N + 1,\mu_{1} N + 2,...,\mu_{1} N + \mu_{3} N$. For synthesis, $m = (n^{\prime} + \mu_{1} N + 3\mu_{3} N + 1)/2$; $n^{\prime} = \mu_{1} N + \mu_{3} N + 1,\mu_{1} N + \mu_{3} N + 3, \ldots ,N - 1$.

Then, we obtain the corresponding measurement states of the updated quantum molecules and calculate the fitness value. The measurement state of the quantum molecule with the maximum fitness value until the $(t + 1){\text{ - th}}$ iteration is updated as the global optimal solution $\overline{\user2{\rho }}_{{{\text{best}}}}^{t + 1}$. The iteration ends when the QCRO algorithm achieves the terminal condition.

3.2 Computational complexity analysis

For the iterations of quantum molecules in QCRO, it is required to rank the kinetic energy of quantum molecules. The computational complexity is $O(N)$. According to their kinetic energy, the quantum molecules are updated by collision, decomposition, or synthesis reactions separately. The quantum rotation angles and new quantum molecules are generated according to different reactions, with the computational complexity of $O(2N \times L)$. These reactions also change the kinetic energy of quantum molecules, and the computational complexity is $O(N)$. Based on (25), the measurement states of the updated quantum molecules can be obtained. The computational complexity is $O(N \times L)$. Then, calculate the fitness value of the updated quantum molecules and update the global optimal solution of QCRO. The computational complexity is $O(2N)$.

When the QCRO algorithm terminates after running $t$ iterations, the computational complexity is $O_{{{\text{iteration}}}} = O(t \times N \times (4 + 3L))$.

3.3 Process of multiple-relay selection based on QCRO

In order to tackle the MRS problem of (22) in secure massive MIMO two-way relay networks, the fitness function of QCRO algorithm is set as $f\left( {\overline{\user2{x}}_{n}^{t} } \right) = \left\{ {\begin{array}{*{20}l} {C_{{{\text{sum}}}} (\overline{\user2{x}}_{n}^{t} ),} \hfill & {{\text{satisfy}}\;{\text{constraint}}\;{\text{conditions}}} \hfill \\ {0,} \hfill & {{\text{else}}} \hfill \\ \end{array} } \right.$. For each quantum molecule, the measurement state is corresponding to a MRS result, the global optimal solution of QCRO algorithm corresponds to the optimal MRS result. Then, the problem of finding the best MRS vector with the maximized secrecy sum-rate can be transformed into finding the global optimal solution of QCRO. Based on the iteration process of QCRO, we can easily get the global optimal solution. In general, the process of MRS based on QCRO for secrecy sum-rate optimization can be shown in Algorithm 1.

4 Results and discussion

Here we present the secrecy performance in the secure massive MIMO two-way relay network. We consider a two-dimensional network topology where the BS and the legitimate user are located at the positions (0, 0) and (100, 0) (unit: meters), respectively, $L$ relays are randomly located at (50, 0) with the radius of 20, and $K$ eavesdroppers are randomly located in the system. We set $L = 20$, $I_{{{\text{th}}}} = - 20\,{\text{dBm}}$, $\xi = 3.8$, and $M_{t} = 128$ [9]. The system bandwidth $B = 1\,{\text{MHz}}$, and noise power spectral density $N_{0} = - 174\,{\text{dBm/Hz}}$ [7]. To reduce the implementation complexity, maximum ratio transmission (MRT) and maximum ratio combining (MRC) methods are adopted at the BS for precoding and receiving [6]. The comparisons of the proposed QCRO algorithm, existing intelligent algorithms and relay selection strategies are presented in the first part. For the second part, we illustrate the impact on the secrecy sum-rate of the cooperative transmission mode based on QCRO algorithm with various system parameters. All results are the average of 200 Monte-Carlo simulations.

4.1 Performance comparisons with QCRO

The comparisons of QCRO algorithm, particle swarm optimization (PSO) algorithm [40], chemical reaction optimization (CRO) algorithm [39], single-relay selection (SRS) strategy [32], and random multiple-relay selection (RMRS) strategy on the secrecy performance are presented in this section. To tackle the MRS problem of (22), the PSO, CRO, SRS, and RMRS adopt the same fitness function as QCRO. The specific operations of PSO, CRO, and SRS are depicted in [40], [39], and [32], respectively. For the SRS, only one relay is selected to forward the received signal, while other relays transmits jamming signals. For the RMRS, all relays are randomly predefined as helper or jammer. To facilitate comparison, we set the maximum number of iterations for QCRO, PSO, and CRO algorithms to the same value, and all these algorithms are set to the same population size. The other parameters of PSO and CRO algorithms are set to the optimal values cited in [40] and [39], respectively. For QCRO algorithm, all quantum bits are initialized to 0.5 and the initial kinetic energy each quantum molecule is set to 1000. The parameter settings of QCRO algorithm are shown in Table 2.

Table 2 Parameter settings of QCRO algorithm

The convergence performance of QCRO algorithm, PSO algorithm, CRO algorithm, SRS, and RMRS strategies are presented in Fig. 2 with $p_{BS} = 35\,{\text{dBm}}$, $p_{u} = 30\,{\text{dBm}}$, $p_{\max } = 30\,{\text{dBm}}$, and $K = 1$. For the MRS problem, both PSO and CRO fall into the local optimum. We observe that QCRO algorithm has a rather fast convergence speed (converges after 30 iterations) and a higher convergent accuracy throughout the iterations. The reason is that QCRO algorithm combines the merit of chemical reaction process and the thought of quantum intelligence computation theory. In QCRO, the quantum molecules are updated via different quantum evolution strategies of collision, decomposition, and synthesis. The designed quantum evolution strategies can make full use of the interactions of quantum molecules, which increases the diversity of solutions. In addition, the searching speed and searching accuracy can be greatly improved by designing new quantum evolutionary rules (28) and (29). Simulation result shows that QCRO has strong search ability and ideal convergence compared with other algorithms. The results also illustrate that the prominent advantage of QCRO over the SRS and RMRS strategies on secrecy sum-rate in a massive MIMO system.

Fig. 2

Convergence performance comparisons. Blue line: QCRO; yellow line: PSO; green line: CRO; pink line: RMRS; black line: SRS

The secrecy performance of QCRO, PSO, CRO, RMRS and SRS with the variation of $p_{BS}$, $p_{u}$, $p_{\max }$, and $K$ are shown in Figs. 3, 4, 5, and 6. In Fig. 3, for most strategies, the secrecy sum-rate of the massive MIMO system increases along with $p_{BS}$. For QCRO, the rising tendency begins to slow down when $p_{BS}$ is over 15 dBm. This phenomenon is caused by information leakage. Since the eavesdropper tries to intercept the desired signals during the information transmission process, the signal strength received at the eavesdropper will become stronger as $p_{BS}$ increases. According to (3) and (13), the eavesdropper may obtain more information from the BS in a higher $p_{BS}$. When the level of $p_{BS}$ exceeds a certain threshold, the increment of the eavesdropping rate will be greater than that of legitimate transmission rate, which will lead to the reduction of secrecy sum-rate. From the simulation result, we can conclude that the secrecy sum-rate may tends to decrease when $p_{BS}$ is over 25 dBm. Compared with other schemes, QCRO can achieve the highest secrecy sum-rate in any $p_{BS}$.

Fig. 3

Secrecy sum-rate in different $p_{BS}$ with $p_{u} = 30\,{\text{dBm}}$, $p_{\max } = 30\,{\text{dBm}}$, and $K = 1$. Blue line: QCRO; yellow line: PSO; green line: CRO; pink line: RMRS; black line: SRS

Fig. 4

Secrecy sum-rate in different $p_{u}$ with $p_{BS} = 35\,{\text{dBm}}$, $p_{\max } = 30\,{\text{dBm}}$, and $K = 1$. Blue line: QCRO; yellow line: PSO; green line: CRO; pink line: RMRS; black line: SRS

Fig. 5

Secrecy sum-rate in different $p_{\max }$ with $p_{BS} = 35\,{\text{dBm}}$, $p_{u} = 30\,{\text{dBm}}$, and $K = 1$. Blue line: QCRO; yellow line: PSO; green line: CRO; pink line: RMRS; black line: SRS

Fig. 6

Secrecy sum-rate in different $K$ with $p_{BS} = 35\,{\text{dBm}}$, $p_{u} = 30\,{\text{dBm}}$, and $p_{\max } = 30\,{\text{dBm}}$. Blue line: QCRO; yellow line: PSO; green line: CRO; pink line: RMRS; black line: SRS

Figures 4 and 5 illustrate the impact of different transmit power of the user and maximum transmit power of the cooperative relays on the security performance. For all strategies in Fig. 4, a higher secrecy sum-rate can be achieved by increasing $p_{u}$. The results in Fig. 5 also illustrate that increasing $p_{\max }$ can boost the security performance. The reason is that a larger $p_{\max }$ will permit relays to broadcast the signals at a higher transmission power under certain interference conditions. After self-interference elimination, the increment of SINR at the legitimate devices is higher than that of the eavesdroppers.

The impact of different number of eavesdroppers on the secrecy sum-rate is presented in Fig. 6. The result shows that the existence of eavesdroppers has an adverse effect on the secrecy performance, and the information leakage increases along with $K$. That is because by increasing $K$, the probability of emerging an eavesdropper with a higher channel gain increases accordingly. For the non-colluding eavesdroppers, the information leakage is determined by the maximum received SINR at the eavesdroppers during the two transmission phases. From Figs. 3, 4, 5, and 6, we conclude that QCRO has great advantages over other strategies in improving the communication security of secure massive MIMO two-way relay networks.

4.2 Impact of different system parameters

The performance of the proposed cooperative transmission mode based on QCRO algorithm in different $p_{u}$ and $p_{BS}$ is studied in Fig. 7, where $p_{u}$ increases from 0 to 40 dBm, $p_{BS}$ = 0 dBm, 3 dBm, 5 dBm, and 10 dBm, respectively. From the simulations, the system can obtain a higher secrecy sum-rate with a larger level of $p_{u}$ at first. But when $p_{u}$ is over 30 dBm, the increment of secrecy sum-rate begins to slow down. For the cases of $p_{BS}$ = 3 dBm, 5 dBm, and 10 dBm, the secrecy sum-rate decreases with $p_{u}$ when $p_{u}$ is over 35 dBm. The reason is that the SINR received at the eavesdropper increases with $p_{u}$, which may cause more information leakage. Hence, we can appropriately increase $p_{BS}$ to confuse the eavesdroppers.

Fig. 7

Secrecy sum-rate in different $p_{u}$ and $p_{BS}$ with $p_{\max } = 30\,{\text{dBm}}$ and $K = 1$. Blue line: $p_{BS} = 10\,{\text{dBm}}$; green line: $p_{BS} = 5\,{\text{dBm}}$; yellow line: $p_{BS} = 3\,{\text{dBm}}$; pink line: $p_{BS} = 0\,{\text{dBm}}$

In Fig. 8, we investigate the impact of different $p_{\max }$ and $p_{u}$ levels with $p_{\max }$ varying from 0 to 40 dBm, $p_{u}$ = 0 dBm, 5 dBm, 10 dBm, 20 dBm, and 30 dBm, respectively. From the simulations, the increasing trend of secrecy sum-rate with $p_{u}$ is in accordance with the results of Fig. 4. We also observe that both higher levels of $p_{\max }$ and $p_{u}$ can help boost the communication security. The reason is that a higher $p_{\max }$ can permit each relay to use more power for its own transmission. With the increment of $p_{\max }$, the desired terminals can obtain higher transmission rates than that of the eavesdroppers according to the SIC criterion.

Fig. 8

Secrecy sum-rate in different $p_{\max }$ and $p_{u}$ with $p_{BS} = 35\,{\text{dBm}}$ and $K = 1$. Blue line: $p_{u} = 30\,{\text{dBm}}$; green line: $p_{u} = 20\,{\text{dBm}}$; yellow line: $p_{u} = 10\,{\text{dBm}}$; pink line: $p_{u} = 5\,{\text{dBm}}$; black line: $p_{u} = 0\,{\text{dBm}}$

The influence of $p_{BS}$ and number of eavesdroppers are studied in Fig. 9. In simulations, $p_{BS}$ varying from – 10 to 20 dBm, and $K$ = 1, 3, 10, 20, respectively. The results indicate that a larger number of eavesdroppers can cause more information leakage, which jeopardizes the communication security of massive MIMO system. For a certain value of $K$, we can see that the secrecy sum-rate increases with $p_{BS}$ at first. However, the growth starts to slow down as $p_{BS}$ increases. Take $K = 1$ as an example, there is a dropping trend in the secrecy sum-rate when $p_{BS}$ is over 15 dBm. Since the eavesdropper may intercept more information from the BS at a higher $p_{BS}$, when the increment of the eavesdropping rate exceeds the increment of legitimate transmission rate, the secrecy sum-rate will no longer increase with $p_{BS}$. Therefore, appropriate $p_{BS}$ plays a significant role in improving the secrecy sum-rate.

Fig. 9

Secrecy sum-rate in different $p_{BS}$ and $K$ with $p_{u} = 30{\text{dBm}}$ and $p_{\max } = 30{\text{dBm}}$. Blue line: $K = 1$; green line: $K = 3$; yellow line: $K = 10$; pink line: $K = 20$

The influence of different $p_{\max }$ and $L$ in single and multiple eavesdropper cases are investigated in Fig. 10. We consider $K = 1$ and $K = 10$, $p_{\max }$ varies from − 10 to 20 dBm, and $L$ increases from 5 to 20. The simulation results indicate that both higher $p_{\max }$ and more relays can boost the transmission security. Obviously, the secrecy sum-rate in the case of single eavesdropper is higher than that of multiple eavesdroppers.

Fig. 10

Secrecy sum-rate in different $p_{\max }$ and $L$ with $p_{BS} = 35{\text{dBm}}$, $p_{u} = 30{\text{dBm}}$, and $K = \{ 1,10\}$. Blue line: $K = 1, \, L = 20$; green line: $K = 1, \, L = 10$; yellow line: $K = 1, \, L = 5$; pink line: $K = 10, \, L = 20$; black line: $K = 10, \, L = 10$; purple line: $K = 10, \, L = 5$

5 Conclusions

In this paper, we have designed a cooperative transmission mode to achieve the trade-off between the communication security and reliability in a massive MIMO system with two-way relay cooperation. Based on the multiple-relay collaboration strategy, some relays function as helper while other relays act as jammer to against the interception of eavesdroppers. Considering the system capacity, energy conservation, and QoS requirements, we have formulated the MRS problem in a secure massive MIMO two-way relay network. Then, we have introduced the implementation of multiple-relay collaboration strategy based on QCRO to optimize the secrecy sum-rate and simulation results have demonstrated its effectiveness in different communication scenarios. In future research, we will incorporate the efforts of this work with energy harvesting, network slicing, and ultra-dense heterogeneous networks to meet the increasing demands of future wireless communication systems.