Joint trajectory and CoMP clustering optimization in UAV-assisted cellular systems: a coalition formation game approach

Abstract

In this paper, the flexibility of unmanned aerial vehicles (UAVs), as well as the benefits of coordinated multi-point (CoMP) transmission, are utilized for mitigating the interference in cellular networks. Specifically, the joint problem of CoMP clusters and UAVs’ trajectories is addressed for downlink transmission in a UAV-assisted cellular system. The problem is presented as a non-convex optimization problem that aims to maximize the sum rate of the ground users by taking into account the clustering, UAV mobility and backhaul capacity constraints. Since the formulated problem is known to be NP-hard, we partition it into two sub-problems. Particularly, by using coalitional game theory, the CoMP clusters are obtained with a given UAVs’ trajectories. Then, UAVs’ trajectories are optimized with given CoMP clusters using successive convex approximation technique. Based on the block coordinate descent method, the two sub-problems are solved alternatively until convergence. Numerical results are conducted and demonstrated the effectiveness of the proposed algorithm.

1 Introduction

Owning to the mobility and flexibility of unmanned aerial vehicles (UAVs), UAV-assisted cellular systems have been attracting comprehensive interest from both industry and academia. Different from the traditional cellular networks, UAVs can offer line-of-sight (LoS) channels with terrestrial users [1]. However, LoS channels may cause severe co-channel interference toward terrestrial users. To address this challenge, coordinated multi-point (CoMP) transmission can be used to mitigate the co-channel interference and improve the rate performance of the system [2]. To limit the processing time delay and overhead, the scale of CoMP is limited by clustering the entire network. However, users at the edge of each cluster may suffer from inter-cluster interference [3]. Therefore, a proper deployment for UAVs needs to be considered simultaneously with the CoMP design to limit the inter-cluster interference problem. Consequently, the UAVs’ trajectories should be optimized jointly with the CoMP clusters, which is a challenging problem and has not been explored in earlier works.

There have been some interesting works focused on optimizing the UAV trajectory in UAV-assisted terrestrial wireless communication systems. In [4], the traveling salesman problem is considered in free space optic-based wireless communication systems to determine the order of the ground terminals that the UAVs go through while maximizing the service time. The joint optimization problem of communication mode, resource allocation, and trajectory is studied in [5] in a single-cell orthogonal frequency division multiple access UAV relay network to maximize the network throughput. In [6], the trajectories of the UAV relays are optimized jointly with the transmit power to maximize the system throughput in cooperative UAV-enabled relaying systems. Joint UAV trajectory and resource allocation optimization problem is considered in [7] to maximize the energy efficiency in non-orthogonal multiple access-based UAV wireless networks. In [8], the flight and collection trajectory are jointly optimized to minimize the mission completion time under energy constraints in UAV-enabled wireless sensor networks.

Trajectories of multi-UAVs are designed in [9] to minimize the mission time with constraints of maximum speed and acceleration of UAVs, the anti-collision, and communication interference between UAVs in multi-UAV internet of things (IoT) network. Joint vehicle communication scheduling, UAV trajectory, and UAV power allocation optimization problem are considered in [10] to maximize the system throughput under anti-collision and communication interference between UAVs constraints in multi-UAV-enabled mobile internet of vehicles model. In [11], three-dimensional (3D) UAV trajectory is optimized in a UAV-assisted IoT system to maximize the data collected from IoT nodes under power and flight time limitations. The system’s average outage probability minimization problem is studied in [12] to optimize the 3D trajectory of the UAV under the constraints of velocity and on-board energy. In [13], the UAV trajectory and resource allocation are jointly optimized to maximize the average throughput with constraints of co-channel interference and completion time in time-constrained UAV-enabled cognitive radio networks.

On the other hand, different CoMP cluster schemes are presented in the literature. Optimal clustering and beamforming matrices are obtained in [14] using the weighted minimum mean square error-based algorithm in a user-centric multiple-input-multiple-output network to maximize the system throughput. A user-centric clustering scheme is presented in [3] for the CoMP clusters problem in cloud radio access network, where Nash Bargaining Solution is presented to attain the fairness between users in terms of the achievable rates. However, when deploying UAV-assisted terrestrial wireless communication systems, the works [4,5,6, 8,9,10,11,12] focus on orthogonal transmission systems. In addition, previous studies did not consider the co-channel interference toward terrestrial users caused by the presence of ground base stations (GBSs) alongside the deployed UAVs. Furthermore, the works cited above that investigated the CoMP clusters problem [3, 14] employ a user-centric clustering scheme, which results in significant processing overhead in the system.

In this paper, downlink transmission in a UAV-assisted cellular system is considered, where multiple UAVs and GBSs are cooperatively serving multiple ground users using the joint transmission as the CoMP technique. In a joint transmission scheme, each user’s data symbol will be transmitted from all UAVs and GBSs in the CoMP cluster of this user. There is no need for all transmission nodes in the entire network to jointly serve each user. This is due to the channel conditions and the limited capacity backhaul links for the transmission nodes. Moreover, increasing the cooperation range will lead to long time delay and a high required processing overhead for the user’s data. Therefore, the disjoint clustering approach is considered in this paper, where the whole network is split into non-overlapped clusters and the UAVs and GBSs in each cluster will jointly serve the users located within the range of them. To design the CoMP disjoint clusters and address the inter-cluster interference problem, an optimization problem is formulated, where the UAVs’ trajectories are optimized jointly with the CoMP clusters to maximize the sum rate of the ground users while ensuring clustering, UAV mobility, and backhaul capacity constraints.

The formulated problem is a mixed-integer nonlinear programming (MINLP) problem, which is challenging to be solved directly. To deal with this non-convex problem, we partition it into two sub-problems: CoMP clusters and UAVs’ trajectories optimization. In the first sub-problem, while fixing the UAVs’ trajectories, the CoMP clusters problem is reformulated as a coalitional formation game and solved by the merge and split method. Meanwhile, in the second sub-problem, UAVs’ trajectories problem is optimized with given CoMP clusters using successive convex approximation (SCA) technique. Then, the two sub-problems are solved alternatively until convergence using the block coordinate descent (BCD) method. Finally, numerical simulations are conducted to show the performance of the proposed algorithm in comparison with the other existing algorithms under a different number of users, different values of backhaul link’s capacity, and different sizes of clusters. The following are the main contributions of this article:

• This work focuses on a non-orthogonal transmission system using the joint transmission CoMP technique in a UAV-assisted cellular system to mitigate the co-channel interference caused by the offered LoS channels between UAVs and terrestrial users.

• A novel framework that jointly optimizes the UAVs’ trajectories with the CoMP disjoint clusters is proposed to address the inter-cluster interference problem. To the best of our knowledge, joint consideration of UAVs’ trajectories and CoMP clusters optimization in UAV-assisted cellular systems has not been substantially investigated.

• An approach is proposed to deal with the formulated MINLP problem. First, we partition the problem into two sub-problems. Then, the first sub-problem that obtains the CoMP clusters is reformulated as a coalitional formation game and solved by the merge and split method. Meanwhile, the second sub-problem that obtains the UAVs’ trajectories is solved using the SCA technique. Finally, the two sub-problems are solved alternatively until convergence using the BCD method.

• Numerical simulations show that the proposed algorithm achieves a higher sum rate than other existing schemes, which use the orthogonal transmission system and the user-centric clustering scheme.

The remainder of the paper is structured as follows: The system model, constraints, and problem formulation are given in Sect. 2. The proposed approach for solving the CoMP clusters and UAVs’ trajectories optimization problem is presented in Sect. 3. Section 4 illustrates the numerical results, and Sect. 5 concludes the paper.

2 System model and problem formulation

2.1 System model

Consider a downlink transmission in a UAV-assisted cellular system that consists of $U$ UAVs and $G$ GBSs, in which all UAVs and GBSs are connected to a ground central processor through limited capacity backhaul links. The ground central processor is responsible for most of the intensive system computational operations, including baseband signal processing, beamforming vector calculation, resource management, and channel estimation, as well as users’ data sharing toward the coordinated transmission nodes (i.e., GBSs and UAVs). Therefore, it can facilitate the coordination between all the transmission nodes and the implementation of the joint transmission CoMP scheme. UAVs and GBSs cooperatively serve $K$ ground users. The set of all transmission nodes is denoted by $\mathcal{L}=\left\{1, 2, \dots , L\right\}$, where $L=U+G$. The horizontal position of each GBS $g\in \mathcal{G}=\left\{1, 2, \dots , G\right\}$ is ${{\varvec{p}}}_{g}={\left[{a}_{g}, {b}_{g}\right]}^{T}$. We assume that all UAVs have the same flight cycle $T$, where $T$ is divided into $N$ equal-duration slots. Suppose also that all UAVs have the same flight height $H$. Without loss of generality, the duration of each slot $n\in \mathcal{N}=\{\mathrm{1,2},\dots ,N\}$ is sufficiently small in which the location of each UAV $u\in \mathcal{U}=\left\{1, 2, \dots , U\right\}$ can be considered fixed during the slot duration. Then, the horizontal coordinate of each UAV $u$ in slot $n$ is denoted by ${{\varvec{p}}}_{u}[n]={\left[{a}_{u}[n], {b}_{u}[n]\right]}^{T}$ and the UAVs’ location matrix is given by ${\varvec{P}}\left[n\right]=\left\{{{\varvec{p}}}_{u}\left[n\right],\forall u\in \mathcal{U}\right\}$.

In this paper, a joint transmission scheme is considered as a downlink CoMP technique. In addition, the disjoint clustering approach is considered to limit the cooperation range of CoMP as well as reduce the time delay and the processing overhead. The overall system model is illustrated in Fig. 1. Suppose that the network is divided at each slot $n$ into $M[n]$ clusters such that the set of all transmission nodes in cluster $m$ $\in \mathcal{M}\left[n\right]=\{1, 2, \dots , M[n]\}$ at slot $n$ is denoted by ${\mathcal{L}}_{m}[n]=\left\{1, 2, \dots , {L}_{m}[n]\right\}$, where ${L}_{m}[n]$ is the number of transmission nodes in this cluster. Meanwhile, the set of users in cluster $m$ at slot $n$ is represented by ${\mathcal{K}}_{m}[n]=\left\{1, 2, \dots , {K}_{m}[n]\right\}$, where ${K}_{m}[n]$ is the number of users in this cluster.

Fig. 1

System model showing a downlink UAV-assisted cellular system with disjoint clusters: This figure shows the overall system model, in which joint transmission scheme is considered as a downlink CoMP technique. In addition, the disjoint clustering approach is considered to limit the cooperation range of CoMP

Let ${h}_{uk,m}\left[n\right]$ be the channel gain between UAV $u$ and user $k$ in cluster $m$ at slot $n$. As such, ${h}_{uk,m}\left[n\right]$ can be given by ${h}_{uk,m}\left[n\right]=\sqrt{{h}_{uk,m}^{\mathrm{L}}\left[n\right]}{h}_{uk,m}^{\mathrm{S}}\left[n\right]$, where ${h}_{uk,m}^{\mathrm{L}}\left[n\right]$ and ${h}_{uk,m}^{\mathrm{S}}\left[n\right]$ are the large-scale and the small-scale channel coefficients from UAV $u$ to user $k$ in cluster $m$ at slot $n$, respectively. Specifically, ${h}_{uk,m}^{\mathrm{L}}\left[n\right]$ can be expressed as [15]

$$h_{uk,m}^{{\text{L}}} \left[ n \right] = h_{0} \sqrt {\left( {\left\| {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} + H^{2} } \right)^{{ - \beta^{{{\text{UAV}}}} }} } ,$$

(1)

where ${h}_{0}$ is the channel power gain at a distance 1 m, ${{\varvec{p}}}_{k,m}={\left[{a}_{k,m}, {b}_{k,m}\right]}^{T}$ is the horizontal location of user $k\in \mathcal{K}=\left\{1, 2, \dots , K\right\}$ in cluster $m$ and ${\beta }^{\mathrm{UAV}}$ denotes the path loss exponent related to the air-to-ground link. Furthermore, ${h}_{uk,m}^{\mathrm{S}}\left[n\right]$ is modeled by the Rician fading model [15] as follow:

$$h_{uk,m}^{{\text{S}}} \left[ n \right] = \sqrt {\frac{\mu }{1 + \mu }} \overline{h}_{uk,m} \left[ n \right] + \sqrt {\frac{1}{1 + \mu }} \tilde{h}_{uk,m} \left[ n \right],$$

(2)

where $\beta$ is the Rician factor, ${\overline{h} }_{uk,m}\left[n\right]$ with $\left|{\overline{h} }_{uk,m}\left[n\right]\right|=1$ represents the deterministic channel component and ${\widetilde{h}}_{uk,m}\left[n\right]\sim \mathcal{C}\mathcal{N}(0, 1)$ denotes the scattered fading channel parameter.

The ground channel gain between GBS $g$ and user $k$ in cluster $m$ is denoted by ${f}_{gk,m}$, where ${f}_{gk,m}=\sqrt{{f}_{gk,m}^{\mathrm{L}}}{f}_{gk,m}^{\mathrm{S}}$. Particularly, ${f}_{gk,m}^{\mathrm{S}}\sim \mathcal{C}\mathcal{N}(0, 1)$ is the small-scale fading coefficient from GBS $g$ to user $k$ in cluster $m$ and ${f}_{gk,m}^{\mathrm{L}}$ represents the large-scale channel gain between GBS $g$ and user $k$ in cluster $m$, which can be expressed as [16]

$$f_{gk,m}^{{\text{L}}} = f_{0} \left\| {{\varvec{p}}_{g} - {\varvec{p}}_{k,m} } \right\|_{2}^{{ - \beta^{{{\text{GBS}}}} }} ,$$

(3)

where ${f}_{0}$ represents the channel power gain at a distance 1 m and ${\beta }^{\mathrm{GBS}}$ is the path loss exponent related to the ground link.

Let ${x}_{k,m}$ be the unity power information signal destined to user $k$ in cluster $m$. Thus, the received signal at user $k$ in cluster $m$ at slot $n$ can be written as

$$\begin{aligned} \hat{x}_{k,m} \left[ n \right] = & \mathop \sum \limits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} \sqrt {P_{lk,m} \left[ n \right]g_{lk,m}^{{\text{L}}} \left[ n \right]} g_{lk,m}^{{\text{S}}} \left[ n \right]x_{k,m} \\ & + \mathop \sum \limits_{{i \ne k,i \in {\mathcal{K}}_{m} \left[ n \right]}} \mathop \sum \limits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} \sqrt {P_{li,m} \left[ n \right]g_{lk,m}^{{\text{L}}} \left[ n \right]} g_{lk,m}^{{\text{S}}} \left[ n \right]x_{i,m} \\ & + \mathop \sum \limits_{{\begin{array}{*{20}c} {m^{\prime} \ne m,} \\ {m^{\prime} \in M\left[ n \right]} \\ \end{array} }} \mathop \sum \limits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} \mathop \sum \limits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} \sqrt {P_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{{\text{L}}} \left[ n \right]} g_{{l^{\prime}k,m}}^{{\text{S}}} \left[ n \right]x_{{i^{\prime}, m^{\prime}}} + e_{k,m} , \\ \end{aligned}$$

(4)

where ${P}_{lk,m}\left[n\right]$, ${g}_{lk,m}^{\mathrm{L}}\left[n\right]$ and ${g}_{lk,m}^{\mathrm{S}}\left[n\right]$ are the transmit power, the large-scale channel gain, and the small-scale channel coefficient from the transmission node $l$ to user $k$ in cluster $m$ at slot $n$, respectively. In (4), the first term is the desired signal of the user that received from its own cluster, the second term represents the intra-cluster interference comes from the signals transmitted to other users by the same cluster, the third term is the inter-cluster interference resulting from the simultaneous transmission by other clusters, and ${e}_{k,m}\sim \mathcal{C}\mathcal{N}(0, {\sigma }^{2})$ denotes the additive white Gaussian noise (AWGN) received at user $k$ in cluster $m$. Then, the downlink achievable data rate ${R}_{k,m}[n]$ of user $k$ in cluster $m$ at slot $n$ can be expressed as

$$R_{k,m} \left[ n \right] = W\,\log_{2} \left( {1 + \frac{{\mathop \sum \nolimits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} \gamma_{lk,m} \left[ n \right]g_{lk,m}^{L} \left[ n \right]}}{{\sigma^{2} + \mathop \sum \nolimits_{{\begin{array}{*{20}c} {i \ne k,} \\ {i \in {\mathcal{K}}_{m} \left[ n \right]} \\ \end{array} }} \mathop \sum \nolimits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} \gamma_{li,m} \left[ n \right]g_{lk,m}^{L} \left[ n \right] + \mathop \sum \nolimits_{{\begin{array}{*{20}c} {m^{\prime} \ne m,} \\ {m^{\prime} \in M\left[ n \right]} \\ \end{array} }} \mathop \sum \nolimits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} \mathop \sum \nolimits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} \gamma_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{L} \left[ n \right]}}} \right)$$

(5)

where ${\gamma }_{lk,m}[n]={P}_{lk,m}\left[n\right]{\left|{g}_{lk,m}^{\mathrm{S}}\left[n\right]\right|}^{2}$, ${\gamma }_{li,m}[n]={P}_{li,m}\left[n\right]{\left|{g}_{lk,m}^{\mathrm{S}}\left[n\right]\right|}^{2}$, ${\gamma }_{{l}{\prime}{i}{\prime},{m}{\prime}}[n]={P}_{{l}{\prime}{i}{\prime},{m}{\prime}}\left[n\right]{\left|{g}_{{l}{\prime}k,m}^{\mathrm{S}}\left[n\right]\right|}^{2}$ and $W$ denotes the wireless channel bandwidth.

2.2 System constraints

The presented system model is subject to some requirements, which will be provided in this sub-section.

2.2.1 Clustering constraints

The formed clusters are subject to some constraints, that is, the transmission nodes and users in each cluster do not overlap with the members of other clusters; the union of all clusters does not exceed the set of all transmission nodes and the set of users; and the maximum number of the transmission nodes in each cluster $m$ is limited by ${L}_{m}^{\mathrm{max}}$. To this end, the clustering constraints are given by

$${\mathcal{L}}_{m} \left[ n \right] \cap {\mathcal{L}}_{{m^{\prime}}} \left[ n \right] = \emptyset , \cup {\mathcal{L}}_{m} \left[ n \right] = {\mathcal{L}},\forall m,m^{\prime} \in {\mathcal{M}}\left[ n \right],m \ne m^{\prime},\forall n \in {\mathcal{N}},$$

(6)

$${\mathcal{K}}_{m} \left[ n \right] \cap {\mathcal{K}}_{{m^{\prime}}} \left[ n \right] = \emptyset , \cup {\mathcal{K}}_{m} \left[ n \right] = {\mathcal{K}},\forall m,m^{\prime} \in {\mathcal{M}}\left[ n \right],m \ne m^{\prime},\forall n \in {\mathcal{N}},$$

(7)

$$L_{m} \left[ n \right] \le L_{m}^{\max } ,\forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}}.$$

(8)

2.2.2 UAV mobility constraints

Suppose that the maximum distance that each UAV $u$ can move between any two successive slots is ${d}^{\mathrm{max}}$, which is affected by the maximum speed of UAVs. Assume also that the minimum distance between any two UAVs is ${d}^{\mathrm{min}}$ to guarantee that no collision exists. In addition, the initial and final locations of each UAV $u$ are overlapped at the same position ${{\varvec{p}}}_{u}^{\mathrm{in}}$. Therefore, the trajectory of each UAV $u$ is subject to the following constraints,

$$\left\| {{\varvec{p}}_{u} \left[ {n + 1} \right] - {\varvec{p}}_{u} \left[ n \right]} \right\|_{2} \le d^{{{\text{max}}}} ,\forall u \in {\mathcal{U}},\forall n \in {\mathcal{N}},$$

(9)

$$\left\| {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{{u^{\prime}}} \left[ n \right]} \right\|_{2} \ge d^{{{\text{min}}}} ,\forall u,u^{\prime} \in {\mathcal{U}},u \ne u^{\prime},\forall n \in {\mathcal{N}\backslash }\left\{ {1, N} \right\},$$

(10)

$${\varvec{p}}_{u} \left[ 1 \right] = {\varvec{p}}_{u}^{{{\text{in}}}} ,{\varvec{p}}_{u} \left[ N \right] = {\varvec{p}}_{u}^{{{\text{in}}}} ,\forall u \in {\mathcal{U}}.$$

(11)

2.2.3 Backhaul capacity constraints

The maximum capacity of the backhaul link varies based on the physical medium utilized (e.g., copper, optical fiber, or microwave connections). Based on [17, 18], the backhaul capacity consumption for the transmission node $l$ can be considered as the sum of the achievable rates of the users in cluster $m$ where the transmission node $l$ is located at each slot $n$. Then, backhaul capacity constraints are defined by limiting the backhaul capacity consumption to the maximum capacity of the backhaul link ${C}_{l}^{\mathrm{BH}}$. Therefore, backhaul capacity constraints can be given by

$$\sum\nolimits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} {R_{k,m} \left[ n \right] \le C_{l}^{{{\text{BH}}}} ,\forall l \in {\mathcal{L}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right], \forall n \in {\mathcal{N}}.}$$

(12)

2.3 Problem formulation

It is noted that the achievable data rate at each user depends on the selected CoMP cluster of this user and the channel gain parameters (or the UAVs’ locations). Therefore, the optimization problem to jointly optimize CoMP clusters and UAVs’ trajectories to maximize the sum rate of the users while guaranteeing clustering, UAV mobility, and backhaul capacity constraints is formulated as follows:

$$\begin{aligned} & \mathop {\max }\limits_{{\left( {\begin{array}{*{20}c} {{\mathcal{L}}_{m} \left[ n \right], {\mathcal{K}}_{m} \left[ n \right], {\varvec{p}}_{u} \left[ n \right]|} \\ {\forall m \in M\left[ n \right],u \in U, n \in N} \\ \end{array} } \right)}} \mathop \sum \limits_{{n \in {\mathcal{N}}}} \mathop \sum \limits_{{m \in {\mathcal{M}}\left[ n \right]}} \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} R_{k,m} \left[ n \right] \\ & {\text{s}}.{\text{t}}.{ }\quad \left( 6 \right) - \left( {12} \right). \\ \end{aligned}$$

(13)

The optimization problem in (13) is combinatorial due to CoMP cluster variables, and non-convex due to the non-concave objective function and the non-convex constraints (10) and (12). Therefore, problem (13) is a NP-hard MINLP problem, which cannot be solved directly. The main variables used in the problem formulation and their meaning are summarized in Table 1.

Table 1 Main variables and their meaning

3 Proposed approach

In this section, problem (13) is partitioned into two sub-problems, where the CoMP clusters and UAVs’ trajectories are optimized separately while fixing other variables. Particularly, for the first sub-problem, CoMP clusters are optimized with given UAVs’ trajectories by using coalitional game theory [19]. Meanwhile, for the second sub-problem, UAVs’ trajectories are optimized with given CoMP clusters by means of SCA technique [20]. Then, BCD method [21] is used to alternatively solve these two sub-problems until convergence. The following sub-sections show the details of the proposed solution.

3.1 CoMP clusters optimization

With fixed UAVs’ trajectories, problem (13) can be reformed into

$$\begin{aligned} & \mathop {\max }\limits_{{\left( {\begin{array}{*{20}c} {{\mathcal{L}}_{m} \left[ n \right], {\mathcal{K}}_{m} \left[ n \right]|m \in M\left[ n \right], n \in N} \\ \end{array} } \right)}} \mathop \sum \limits_{{n \in {\mathcal{N}}}} \mathop \sum \limits_{{m \in {\mathcal{M}}\left[ n \right]}} \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} R_{k,m} \left[ n \right] \\ & {\text{s}}.{\text{t}}.{ }\quad \left( 6 \right) - \left( 8 \right), \left( {12} \right). \\ \end{aligned}$$

(14)

Although UAVs’ trajectories are fixed, problem (14) is still difficult to solve in its current form. To deal with this issue, the CoMP clusters problem is reformulated as a coalitional formation game, which can be used to provide efficient disjoint clustering. Particularly, the transmission nodes set $\mathcal{L}$ is treated as a set of players, which will negotiate with each other to form clusters. To avoid forming a grand coalition, the utility function for each cluster $m$ can be defined as follows:

$$\begin{aligned} & U\left( {{\mathcal{L}}_{m} \left[ n \right]} \right) = \left\{ {\begin{array}{*{20}l} {R\left( {{\mathcal{L}}_{m} \left[ n \right]} \right), } \hfill & {{\text{if}} \left( {C_{l}^{BH} - R\left( {{\mathcal{L}}_{m} \left[ n \right]} \right) \ge 0} \right),\forall l \in {\mathcal{L}}_{m} \left[ n \right],} \hfill \\ {0,} \hfill & {{\text{otherwise}}.} \hfill \\ \end{array} } \right. \\ & \quad \forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}}, \\ \end{aligned}$$

(15)

where $R\left({\mathcal{L}}_{m}\left[n\right]\right)=\sum_{k\in {\mathcal{K}}_{m}\left[n\right]}{R}_{k,m}\left[n\right]$ is the benefit for cluster $m$. Due to cost existence in the utility function, which is represented by the backhaul capacity limitation, the formulated $(\mathcal{L},U)$ coalitional formation game is non-superadditive with an empty core [22]. Thus, the merge and split method is used to deal will this problem, where two main rules are used for forming and breaking clusters as follows:

Definition 1

(Merge Rule) Merge any set of coalitions $\{{\mathcal{L}}_{1}\left[n\right], {\mathcal{L}}_{2}\left[n\right],\dots , {\mathcal{L}}_{{m}{\prime}}\left[n\right]\}$ whenever the utility function satisfies $\sum_{j=1}^{{m}{\prime}}U\left({\mathcal{L}}_{j}\left[n\right]\right)<U(\bigcup_{j=1}^{{m}{\prime}}{\mathcal{L}}_{j}\left[n\right])$, thus $\{{\mathcal{L}}_{1}\left[n\right], {\mathcal{L}}_{2}\left[n\right],\dots , {\mathcal{L}}_{{m}{\prime}}\left[n\right]\}\to \bigcup_{j=1}^{{m}{\prime}}{\mathcal{L}}_{j}\left[n\right]$.

Definition 2

(Split Rule) Split any coalition $\bigcup_{j=1}^{{m}^{{\prime}{\prime}}}{\mathcal{L}}_{j}\left[n\right]$ whenever the utility function satisfies $U(\bigcup_{j=1}^{{m}^{{\prime}{\prime}}}{\mathcal{L}}_{j}\left[n\right])<\sum_{j=1}^{{m}^{{\prime}{\prime}}}U\left({\mathcal{L}}_{j}\left[n\right]\right)$, therefore $\bigcup_{j=1}^{{m}^{{\prime}{\prime}}}{\mathcal{L}}_{j}\left[n\right]\to \{{\mathcal{L}}_{1}\left[n\right], {\mathcal{L}}_{2}\left[n\right],\dots , {\mathcal{L}}_{{m}^{{\prime}{\prime}}}\left[n\right]\}$.

The proposed clustering algorithm is presented in Algorithm 1. The candidate clusters are chosen in merge operation such that the maximum cluster size ${L}_{m}^{\mathrm{max}}$ satisfies. The stability of the proposed algorithm is shown in the following proposition.

Proposition 1

The obtained result from Algorithm 1 converges to ${\mathbb{D}}_{hp}$-stable clusters, which means that no players in $\mathcal{L}$ are interested to form another cluster sets as proved in [22].

3.2 UAVs’ trajectories optimization

With fixed CoMP clusters, UAVs’ trajectories optimization problem can be given by

$$\begin{aligned} & \mathop {\max }\limits_{{\left( {\begin{array}{*{20}c} { {\varvec{p}}_{u} \left[ n \right]|u \in U, n \in N} \\ \end{array} } \right)}} \mathop \sum \limits_{{n \in {\mathcal{N}}}} \mathop \sum \limits_{{m \in {\mathcal{M}}\left[ n \right]}} \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} R_{k,m} \left[ n \right] \\ & {\text{s}}.{\text{t}}.{ }\quad \left( 9 \right) - \left( {12} \right). \\ \end{aligned}$$

(16)

However, problem (16) is still a non-convex optimization problem. To deal with such non convexity, we first rewrite ${R}_{k,m}[n]$ as follows,

$$R_{k,m} \left[ n \right] = \hat{R}_{k,m} \left[ n \right] - \tilde{R}_{k,m} \left[ n \right],$$

(17)

where

$$\begin{aligned} & \hat{R}_{k,m} \left[ n \right] = W\,\log_{2} \left( {\sigma^{2} + \mathop \sum \limits_{{m^{\prime} \in {\mathcal{M}}\left[ n \right]}} \,\mathop \sum \limits_{{\begin{array}{*{20}c} {i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]} \\ \end{array} }} \,\mathop \sum \limits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} \gamma_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{L} \left[ n \right]} \right), \\ & \tilde{R}_{k,m} \left[ n \right] = W\,\log_{2} \left( {\sigma^{2} + \mathop \sum \limits_{{\begin{array}{*{20}c} {i \ne k,} \\ {i \in {\mathcal{K}}_{m} \left[ n \right]} \\ \end{array} }} \,\mathop \sum \limits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} \gamma_{li,m} \left[ n \right]g_{lk,m}^{L} \left[ n \right] + \mathop \sum \limits_{{\begin{array}{*{20}c} {m^{\prime} \ne m,} \\ {m^{\prime} \in M\left[ n \right]} \\ \end{array} }} \,\mathop \sum \limits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} \,\mathop \sum \limits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} \gamma_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{L} \left[ n \right]} \right). \\ \end{aligned}$$

Then, we introduce the auxiliary variables $\boldsymbol{\varrho }\left[n\right]=\left\{{\varrho }_{k,m}\left[n\right]\ge 0,\forall k\in {\mathcal{K}}_{m}\left[n\right],\forall m\in \mathcal{M}\left[n\right],\forall n\in \mathcal{N}\right\}$ and ${\varvec{\zeta}}[n]=\left\{{\zeta }_{uk,m}\left[n\right]={\Vert {{\varvec{p}}}_{u}\left[n\right]-{{\varvec{p}}}_{k,m}\Vert }_{2}^{2},\forall u\in \mathcal{U},\forall k\in {\mathcal{K}}_{m}[n],\forall m\in \mathcal{M}\left[n\right],\forall n\in \mathcal{N}\right\}$. Therefore, problem (16) can be reformed into

$$\mathop {\max }\limits_{{\left( {\begin{array}{*{20}c} { {\varvec{p}}_{u} \left[ n \right], \varrho_{k,m} \left[ n \right], \zeta_{uk,m} \left[ n \right]|} \\ {u \in U,k \in {\mathcal{K}}_{m} \left[ n \right],} \\ {\forall m \in M\left[ n \right], n \in N} \\ \end{array} } \right)}} \mathop \sum \limits_{{n \in {\mathcal{N}}}} \mathop \sum \limits_{{m \in {\mathcal{M}}\left[ n \right]}} \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} \varrho_{k,m} \left[ n \right]$$

(18)

$${\text{s.t.}} \quad \hat{R}_{k,m} \left[ n \right] - \tilde{R}_{k,m} \left[ n \right]\left( {\zeta \left[ n \right]} \right) \ge \varrho_{k,m} \left[ n \right],\forall k \in {\mathcal{K}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}},$$

(18a)

$$\mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} \left( {\hat{R}_{k,m} \left[ n \right]\left( {{\varvec{\zeta}}\left[ n \right]} \right) - \tilde{R}_{k,m} \left[ n \right]} \right) \le C_{l}^{{{\text{BH}}}} ,\forall l \in {\mathcal{L}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right], \forall n \in {\mathcal{N}},$$

(18b)

$$\begin{aligned} & \left\| {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} \ge \zeta_{uk,m} \left[ n \right],\forall u \in {\mathcal{U}},\forall k \in {\mathcal{K}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}}, \\ & \quad \left( 9 \right) - \left( {11} \right). \\ \end{aligned}$$

(18c)

It can be observed that problem (18) is still non-convex due to the new non-convex constraints. However, the SCA technique can be utilized, where the functions are approximated to their bounds in each iteration around a local point.

We consider first the function ${\widehat{R}}_{k,m}\left[n\right]$, which is convex for ${\Vert {{\varvec{p}}}_{u}\left[n\right]-{{\varvec{p}}}_{k,m}\Vert }_{2}^{2}$. Around a local point ${{\varvec{P}}[n]}^{\left(t\right)}$, the lower-bound first-order Taylor expansion ${\widehat{R}}_{k,m}^{\mathrm{lower}}{\left[n\right]}^{\left(t\right)}$ can be given as

$$\hat{R}_{k,m} \left[ n \right] \ge \hat{R}_{k,m}^{{{\text{lower}}}} \left[ n \right]^{\left( t \right)} = \hat{R}_{k,m} \left[ n \right]\left( {{\varvec{P}}\left[ n \right]^{\left( t \right)} } \right) + \hat{\delta }_{k,m} \left[ n \right]\left( {{\varvec{P}}\left[ n \right]^{\left( t \right)} } \right)\left( {\left\| {{\varvec{p}}_{{u^{\prime}}} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{k,m} } \right\|_{2}^{2} - \left\| {{\varvec{p}}_{{u^{\prime}}} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} } \right),$$

(19)

where

$$\hat{\delta }_{k,m} \left[ n \right] = \frac{{Wh_{0} \left( {\frac{{\beta^{{{\text{UAV}}}} }}{2}} \right){\text{log}}_{2} \left( e \right)\sum\nolimits_{{m^{\prime} \in {\mathcal{M}}\left[ n \right]}} {\sum\nolimits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} {\sum\nolimits_{{u^{\prime} \in {\mathcal{U}}_{{m^{\prime}}} \left[ n \right]}} {P_{{u^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]\left| {h_{{u^{\prime}k,m}}^{{\text{S}}} \left[ n \right]} \right|^{2} \left( {\left\| {{\varvec{p}}_{{u^{\prime}}} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} + H^{2} } \right)^{{ - \frac{{\beta^{{{\text{UAV}}}} }}{2} - 1}} } } } }}{{\sigma^{2} + \sum\nolimits_{{m^{\prime} \in {\mathcal{M}}\left[ n \right]}} {\sum\nolimits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} {\sum\nolimits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} {\gamma_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{{\text{L}}} \left[ n \right]} } } }}$$

Second, consider the function ${\widetilde{R}}_{k,m}\left[n\right]$, around a local point ${{\varvec{P}}[n]}^{\left(t\right)}$, the upper-bound first-order Taylor expansion ${\widetilde{R}}_{k,m}^{\mathrm{upper}}{\left[n\right]}^{\left(t\right)}$ can be obtained as

$$\begin{aligned} \tilde{R}_{k,m} \left[ n \right] \le \tilde{R}_{k,m}^{{{\text{upper}}}} \left[ n \right]^{\left( t \right)} = & \tilde{R}_{k,m} \left[ n \right]\left( {{\varvec{P}}\left[ n \right]^{\left( t \right)} } \right) \\ & + \tilde{\delta }_{1k,m} \left[ n \right]\left( {{\varvec{P}}\left[ n \right]^{\left( t \right)} } \right)\left( {\left\| {{\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{k,m} } \right\|_{2}^{2} - \left\| {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} } \right) \\ & + \tilde{\delta }_{2k,m} \left[ n \right]\left( {{\varvec{P}}\left[ n \right]^{\left( t \right)} } \right)\left( {{\varvec{p}}\left\| {_{{u^{\prime}}} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{k,m} } \right\|_{2}^{2} - \left\| {{\varvec{p}}_{{u^{\prime}}} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} } \right), \\ \end{aligned}$$

(20)

where

$$\begin{aligned} & \tilde{\delta }_{1k,m} \left[ n \right] = \frac{{Wh_{0} \left( {\frac{{\beta^{{{\text{UAV}}}} }}{2}} \right){\text{log}}_{2} \left( e \right)\left( {\begin{array}{*{20}c} {\sum\nolimits_{{\begin{array}{*{20}c} {i \ne k,} \\ {i \in {\mathcal{K}}_{m} \left[ n \right]} \\ \end{array} }} {\sum\nolimits_{{u \in {\mathcal{U}}_{m} \left[ n \right]}} {P_{ui,m} \left[ n \right]\left| {h_{uk,m}^{{\text{S}}} \left[ n \right]} \right|^{2} \left( {\left\| {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} + H^{2} } \right)^{{ - \frac{{\beta^{{{\text{UAV}}}} }}{2} - 1}} } } } \\ \end{array} } \right)}}{{\sigma^{2} + \sum\nolimits_{{\begin{array}{*{20}c} {i \ne k,} \\ {i \in {\mathcal{K}}_{m} \left[ n \right]} \\ \end{array} }} {\sum\nolimits_{{l \in {\mathcal{L}}_{m} \left[ n \right]}} {\gamma_{li,m} \left[ n \right]g_{lk,m}^{{\text{L}}} \left[ n \right]} } + \sum\nolimits_{{\begin{array}{*{20}c} {m^{\prime} \ne m,} \\ {m^{\prime} \in M\left[ n \right]} \\ \end{array} }} {\sum\nolimits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} {\sum\nolimits_{{l^{\prime} \in {\mathcal{L}}_{{m^{\prime}}} \left[ n \right]}} {\gamma_{{l^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]g_{{l^{\prime}k,m}}^{{\text{L}}} \left[ n \right]} } } }} \\ & \tilde{\delta }_{2k,m} \left[ n \right] = \frac{{Wh_{0} \left( {\frac{{\beta^{{{\text{UAV}}}} }}{2}} \right){\text{log}}_{2} \left( e \right)\left( {\begin{array}{*{20}c} {\sum\nolimits_{{\begin{array}{*{20}c} {m^{\prime} \ne m,} \\ {m^{\prime} \in M\left[ n \right]} \\ \end{array} }} {\sum\nolimits_{{i^{\prime} \in {\mathcal{K}}_{{m^{\prime}}} \left[ n \right]}} {\sum\nolimits_{{u^{\prime} \in {\mathcal{U}}_{{m^{\prime}}} \left[ n \right]}} {P_{{u^{\prime}i^{\prime},m^{\prime}}} \left[ n \right]\left| {h_{{u^{\prime}k,m}}^{{\text{S}}} \left[ n \right]} \right|^{2} \left( {\left\| {{\varvec{p}}_{{u^{\prime}}} \left[ n \right] - {\varvec{p}}_{k,m} } \right\|_{2}^{2} + H^{2} } \right)^{{ - \frac{{\beta^{{{\text{UAV}}}} }}{2} - 1}} } } } } \\ \end{array} } \right)}}{{}} \\ \end{aligned}$$

Moreover, considering the constraints in (18c), around a local point ${{\varvec{p}}}_{u}{\left[n\right]}^{\left(t\right)}$, the lower-bound first-order Taylor expansion for the convex function on the left side is applied. Then, (18c) can be written as follows:

$$\begin{aligned} & \left\| {{\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{k,m} } \right\|_{2}^{2} + 2\left( {{\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{k,m} } \right)^{T} \left( {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} } \right) \ge \zeta_{uk,m} \left[ n \right],\forall u \in {\mathcal{U}},\forall k \\ & \quad \in {\mathcal{K}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}}. \\ \end{aligned}$$

(21)

Similarly, the constraints in (10) can be converted as follows:

$$\begin{aligned} & \left\| {{\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{{u^{\prime}}} \left[ n \right]^{\left( t \right)} } \right\|_{2}^{2} + 2\left( {{\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{{u^{\prime}}} \left[ n \right]^{\left( t \right)} } \right)^{T} \left( {{\varvec{p}}_{u} \left[ n \right] - {\varvec{p}}_{u} \left[ n \right]^{\left( t \right)} + {\varvec{p}}_{{u^{\prime}}} \left[ n \right]^{\left( t \right)} - {\varvec{p}}_{{u^{\prime}}} \left[ n \right]} \right) \\ & \quad \ge d^{\min 2} ,\forall u,u^{\prime} \in {\mathcal{U}},u \ne u^{\prime},\forall n \in {\mathcal{N}}\backslash \left\{ {1, N} \right\}. \\ \end{aligned}$$

(22)

By utilizing the above approximations, problem (18) can be represented at iteration $(t+1)$ by the following problem

$$\mathop {\max }\limits_{{\left( {\begin{array}{*{20}c} { {\varvec{p}}_{u} \left[ n \right], \varrho_{k,m} \left[ n \right], \zeta_{uk,m} \left[ n \right]|} \\ {u \in U,k \in {\mathcal{K}}_{m} \left[ n \right],} \\ {\forall m \in M\left[ n \right], n \in N} \\ \end{array} } \right)}} \mathop \sum \limits_{{n \in {\mathcal{N}}}} \mathop \sum \limits_{{m \in {\mathcal{M}}\left[ n \right]}} \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} \varrho_{k,m} \left[ n \right]$$

(23)

$${\text{s}}.{\text{t}}.\quad \hat{R}_{k,m}^{{{\text{lower}}}} \left[ n \right]^{\left( t \right)} - \tilde{R}_{k,m} \left[ n \right]\left( {{\varvec{\zeta}}\left[ n \right]} \right) \ge \varrho_{k,m} \left[ n \right],\forall k \in {\mathcal{K}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right],\forall n \in {\mathcal{N}},$$

(23a)

$$\begin{aligned} & \mathop \sum \limits_{{k \in {\mathcal{K}}_{m} \left[ n \right]}} \left( {\hat{R}_{k,m} \left[ n \right]\left( {{\varvec{\zeta}}\left[ n \right]} \right) - \tilde{R}_{k,m}^{{{\text{upper}}}} \left[ n \right]^{\left( t \right)} } \right) \le C_{l}^{{{\text{BH}}}} ,\forall l \in {\mathcal{L}}_{m} \left[ n \right],\forall m \in {\mathcal{M}}\left[ n \right], \forall n \in {\mathcal{N}}, \\ & \quad \left( 9 \right),\left( {11} \right), \left( {21} \right), \left( {22} \right). \\ \end{aligned}$$

(23b)

Problem (23) at iteration $(t+1)$ is a convex optimization problem, thus it can be solved efficiently by CVX [23]. Using the SCA technique, the solution for UAVs’ trajectories optimization problem can be obtained by solving the problem (23) iteratively, where the local point ${{\varvec{P}}[n]}^{\left(t\right)}$ is updated in each iteration. The procedure for solving the UAVs’ trajectories problem is described in Algorithm 2.

3.3 Overall algorithm

Based on BCD method, the overall algorithm for solving the problem (13) is presented in Algorithm 3, where the convergence is guaranteed following the same proof in [24].

4 Numerical results and discussion

4.1 Experimental method

This section provides numerical results to evaluate the performance of the proposed solution. We consider a cellular network with a square coverage of 1 km × 1 km, where $G=18$ GBSs and $K=70$ users are randomly placed. The maximum UAV velocity and the duration of each slot in the UAV flight cycle are 60 mps and 5 s [25], respectively. Therefore, ${d}^{\mathrm{max}}$ is set as 300 m. Circular filling scheme [26] is adopted to define the initial UAVs’ trajectories. The common parameters used in the simulation are listed in Table 2. The dimensions of the area where UAVs are deployed are implicitly considered due to the UAV mobility constraints in (9) and (11). Particularly, the constraints in (9) limit the moving distance for each UAV, which is based on the maximum speed that the UAV can fly and the duration of the UAV flight cycle. In addition, the constraints in (11) force the UAVs to return to the same location. Furthermore, UAVs are required to fly near ground users that are located in the specified coverage area to improve the sum rate of the users. Therefore, the UAVs are guaranteed not to move far away from the considered coverage area of the cellular network. However, considering the dimensions of the deployment area for UAVs explicitly is interesting and can be studied in future work. The altitude of UAVs is fixed at 100 m [16] and the height of GBSs is fixed at 25 m [27, 28], and therefore, the collision between UAVs and GBSs can be prevented.

Table 2 Simulation parameters

The proposed algorithm is compared with the static trajectory scheme provided in [29] and the user-centric clustering scheme offered in [14]. In addition, five scenarios for the proposed solution are compared in the simulation, namely, Scenario 1: CoMP clusters and UAVs’ trajectories are collaboratively optimized, i.e., the proposed scheme in this paper (denoted as C-T-OPT); Scenario 2: CoMP clusters are optimized with circular filling trajectories (denoted as C-OPT-With-CFT); Scenario 3: UAVs’ trajectories are optimized in the non-CoMP scheme (denoted as T-OPT-With-non-CoMP); Scenario 4: CoMP clusters are optimized with static trajectories (denoted as C-OPT-With-ST); Scenario 5: UAVs’ trajectories are optimized with fixed cluster size (denoted as T-OPT-With-FCS). The locations of users, GBSs, UAVs’ circular filling trajectories and UAVs’ static trajectories are shown in Fig. 2.

Fig. 2

Locations of users, GBSs, and UAVs’ trajectories in the horizontal dimension: This figure depicts the simulation setup, where the locations of users, GBSs, UAVs’ circular filling trajectories and UAVs’ static trajectories are clarified

4.2 Performance evaluation

The cumulative distributions of the sum rate for the five scenarios of the proposed solution and the two comparable schemes are shown in Fig. 3 when the number of users is 70. It is obvious that all proposed solution scenarios outperform the scheme provided in [29]. This is because the solution in [29] did not optimize the UAVs’ trajectories and did not apply CoMP transmission which can help to mitigate the co-channel interference and hence improve the system performance. Moreover, C-T-OPT and C-OPT-With-CFT outperform the user-centric clustering scheme in [14], which validate that the optimization of the UAVs’ trajectories can help to reduce the inter-cluster interference. Furthermore, the optimization of trajectories in a non-CoMP scheme i.e., T-OPT-With-non-CoMP can improve the sum rate compared with the scheme in [14]. In addition, C-T-OPT can improve the sum rate by about 4.3% compared with C-OPT-With-ST. On the other hand, applying a fixed cluster size degrades the performance which validates that the collaborative optimization scenario can adapt the cluster size to improve the sum rate. Specifically, the sum rate of C-T-OPT is reduced by 72.93% when fixed cluster size is considered.

Fig. 3

Cumulative distribution functions of system sum rate: This figure shows the cumulative distributions of the sum rate for the five scenarios of the proposed solution and the two comparable schemes

Figure 4 illustrates the sum rate versus a different number of users for all scenarios of the proposed solution and the two comparable schemes. The number of users is changed from 40 to 80. The results show that all scenarios of the proposed solution outperform the scheme offered in [29] when the number of users is greater than 50. Specifically, the sum rate enhancement is about 4.2 ×, 4.1 ×, 4 ×, 4 × and 0.5 × for C-T-OPT, C-OPT-With-CFT, T-OPT-With-non-CoMP, C-OPT-With-ST, and T-OPT-With-FCS, respectively, in comparison with the scheme in [29]. However, compared with the algorithm in [14], C-T-OPT, C-OPT-With-CFT, T-OPT-With-non-CoMP, and C-OPT-With-ST can improve the sum rate performance at a large number of users. This is because the algorithm in [14] did not consider the backhaul capacity constraints and the performance is degraded due to the limited capacity that the backhaul link can support.

Fig. 4

Sum rate comparison for a different number of users when ${C}_{l}^{\mathrm{BH}}=20$ $\mathrm{Mbps}$: This figure illustrates the sum rate versus a different number of users for all scenarios of the proposed solution and the two comparable schemes, where the number of users is changed from 40 to 80

The convergence speed in terms of the number of iterations required by the proposed solution is depicted in Fig. 5. As shown, the proposed algorithm is converged after a small number of iterations under a different number of users. Specifically, the objective value of problem (13) can converge using Algorithm 3 within 32 and 33 iterations on average when the number of users is 70 and 80, respectively.

Fig. 5

Number of iterations statistics for a different number of users when ${C}_{l}^{\mathrm{BH}}=20$ Mbps: This figure shows the convergence speed in terms of the number of iterations required by the proposed solution, where the number of users is changed from 40 to 80

Figure 6 shows the sum rate versus different values of the maximum capacity that the backhaul links can support when the number of users is 70. The maximum capacity of the backhaul link is changed from 10 to 50 Mbps. As expected, in all schemes, the sum rate is increased when the maximum capacity of backhaul links increases. Compared with the algorithm in [14], the sum rate enhancement of C-T-OPT, C-OPT-With-CFT, T-OPT-With-non-CoMP, and C-OPT-With-ST is about 9.82%, 7.17%, 6.17%, and 4.32%, respectively. In addition, the sum rate improvement of C-T-OPT, C-OPT-With-CFT, T-OPT-With-non-CoMP, and C-OPT-With-ST is more significant than the algorithms in [14] and [29] when the maximum capacity of backhaul links increased. This is due to considering the backhaul capacity constraints in the proposed solution.

Fig. 6

Sum rate comparison with different values of backhaul link’s capacity when $K=70$ users: This figure illustrates the sum rate versus different values of the maximum capacity that the backhaul links can support when the number of users is 70 where the maximum capacity of the backhaul link is changed from 10 to 50 Mbps

Figure 7 represents the sum rate versus different values of the maximum cluster size that changed from 3 to 7 for all CoMP scenarios when the number of users is 70. The results show that the C-T-OPT, C-OPT-With-CFT, and C-OPT-With-ST can select the cluster size that achieves the maximum sum rate. However, increasing the cluster size in T-OPT-With-FCS will reduce the sum rate and degrade the CoMP performance. In contrast, the performance of the user-centric clustering scheme in [14] is improved when the cluster size is increased. However, when the cooperation range is increased in CoMP transmission, it will require a long-time delay and high processing overhead for the user’s data. In conclusion, the proposed solution can improve the CoMP performance when disjoint clustering is considered because of optimizing the UAVs’ trajectories jointly with the CoMP clusters, which can address the inter-cluster interference problem.

Fig. 7

Sum rate comparison with different sizes of clusters when ${C}_{l}^{\mathrm{BH}}=20$ Mbps and $K=70$ users: This figure represents the sum rate versus different values of the maximum cluster size that changed from 3 to 7 for all CoMP scenarios when the number of users is 70

5 Conclusion

In this paper, the downlink sum rate maximization problem is studied in a UAV-assisted cellular system. In particular, the joint CoMP clusters and UAVs’ trajectories optimization problem is formulated under clustering, UAV mobility, and backhaul capacity constraints. To solve this non-convex MINLP problem, we partitioned it into two sub-problems: CoMP clusters and UAVs’ trajectories optimization. For the CoMP clusters sub-problem, it is reformulated as a coalitional formation game and solved by the merge and split method. Meanwhile, for UAVs’ trajectories sub-problem, it is solved by the SCA technique. Then, a BCD-based algorithm is proposed to solve the two sub-problems alternatively till reaching convergence. The results demonstrate the improvement in the system performance in terms of sum rate. Specifically, the sum rate improvement of the proposed solution is about 4.2 × compared to an existing static trajectory scheme.