LEO laser microwave hybrid inter-satellite routing strategy based on modified Q-routing algorithm

Abstract

Low earth orbit (LEO) satellite communication networks require huge load capacity and information processing speed to carry global communication traffic. Inter-satellite links and the on-board processing are the key technologies to achieve this goal, but the new network architecture leads to great challenges on satellite routing. This paper designs a hybrid inter-satellite link with the same-orbit laser and the different-orbit microwave to increase the link capacity and adopts a CPU centralized scheduling to improve the utilization of computing resources. Then, this paper establishes minimum delay function by considering the inter-satellite transmission delay and the on-board processing delay. The transmission delay model bases on the orbital period, and the processing delay adopts the multi-services model, the limited-capacity single-service model, and the unlimited-capacity single-service model in the queuing theory to model the on-board CPU centralized scheduling, photoelectric converters, and electro-optical converters, respectively. Based on this model, this paper proposes an inter-satellite routing strategy with modified Q-routing algorithm. The modified algorithm uses Dijkstra algorithm to accelerate the convergence of Q-routing algorithm and retains the strong real-time performance of Q-routing algorithm. Simulations show that the delay of the modified algorithm is 83.3$\%$ lower than that of the Dijkstra algorithm, and the larger the network and the traffic, the more obvious the advantage.

1 Introduction

In recent years, the Information and Communications Technology (ICT) industry has valued the LEO communication satellite constellations due to its global communications advantages and limited orbital resources. Companies such as Space X, Boeing, and Oneweb have been carrying out LEO satellite launch plans [1]. Many advantages of satellite communication are difficult to realize in terrestrial communication systems. First, satellite communication networks are high-stable and are not destroyed by earthquakes, fires, tsunamis, and other natural disasters. Second, satellite communication networks can achieve seamless global coverage, then can provide low-cost broadband services for deserts, oceans, forests, and other extremely remote areas. During the second quarter of 2021, the Internet performance test application developer of Ookla launched a global test on the Starlink network, the test showed that Starlink was able to provide bandwidth of 15 Mbps for uplink, 100 Mbps for downlink, and 30 to 50 ms delay [2].

At this stage, all LEO satellite constellations do not have complete inter-satellite links, the transmission between two adjacent satellites relies on ground station relay services. In the same time, some satellite systems use microwave links, such as the United States Tracking and Data Relay Satellite System and Chinese Tianlian-1 satellite system. The microwave links of these satellites can achieve a maximum data volume of G bps, which basically meets the QoS for communication, navigation, and remote sensing. But the high-throughput links still face some challenges, such as narrow bandwidth, low rate, large transmitting antenna volume, and high power consumption. With the development of optical tracking, optical capture [3], the two-way laser link achieves 5.625 Gbps communication between the TerraSAR-X satellite and the Near Field Infrared Experiment satellite, the two-way laser link achieves 1.8 Gbps communication in the European Data Relay Satellite System. Compared with microwave communication, laser communication has the advantages of [4] high transmission efficiency, low power consumption, lightweight, etc. However, the huge relative motion between different-orbit satellites makes laser alignment very difficult, the fast-developing Starlink only announced the success of the laser link test in the same orbit in September 2020 [5]. Therefore, the same-orbit laser link and different-orbit microwave link are the development trend of inter-satellite links.

A satellite covers a large area with a small load, which makes computing resources constraint, the emerging resource pooling technology is just the proposal. The main control computer of Shenzhou Spacecraft in paper [6] includes a preprocessor plus a main processor to realize dual processor task sharing. Paper [7] introduces several architectures of small satellites with multiple CPUs. Paper [8] proposes an on-board Field programmable gate array with three CPUs, in which two CPUs are on-board task processors and the other CPU is Coordinate Rotation Digital Computer. Researchers and engineers have done a lot of related works on the theory and hardware of on-board resource pool. At present, satellite network resource virtualization mainly focuses on the earth station resource pool, but the on-board resource pool. Paper [9, 10] study the cloud-based satellite ground network. The ground resource pool can support the optimal resource allocation scheme and minimize energy consumption. Paper [11] puts forward a virtual resource utilization plan for the increasing traffic in the Fengyun meteorological satellite ground system. This plan allocates corresponding resources according to the priority and attributes of the tasks. Paper [12] applies the concept of resource virtualization to LEO satellite constellation for the first time and illustrates the feasibility and superiority of on-board software-defined network (SDN). Paper [13] presents a satellite cloud architecture based on resource virtualization, this method allows satellites to share idle resources each other. Obviously, the resource virtualization technology of satellite ground station has been relatively mature, while the on-board resource virtualization technology is still remaining on paper.

Under the background of inter-satellite laser link and CPU resource pooling, the delay of satellite routing will be different from existing LEO constellations. The receiving speed and cache capacity of photoelectric converter in laser communication system lead to new packet loss rate and processing delay. The pooling of on-board CPU resources brings a new generation method for processing delay of packets. Therefore, it is necessary to re-evaluate the cost of routing delay of satellite network in the same-orbit laser link and in the different-orbit microwave link.

Due to the huge topology of modern LEO satellite constellation as shown in Fig. 1, the routing calculation is difficult to broadcast the real-time information, traditional static algorithms hardly research the optimal path within limited calculation and cache resources. For example, Oneweb initially plans to build a network of 720 satellites, and Starlink plans to build a network of 42,000 satellites. The time and space cost of traditional algorithms are too large to calculate the shortest path, but reinforcement learning can solve large-scale mathematical problems. According to the next hop queue length and path length continuously iterating in real time, the reinforcement learning algorithm Q-routing can obtain the shortest delay path in dynamic networks. The algorithm also has the advantages of decentralized computation, small space cost, and short single iteration time. For satellite routing scenarios with limited computing resources, limited cache space, and strong real-time computing, its performance is significantly better than traditional algorithms.

Fig. 1

LEO satellite communication system

This paper studies the modified Q-routing algorithm to find the shortest delay path in terms of the processing delay and the transmission delay. The rest of this paper is organized as follows. The next section gives an overview of the development of satellite routing. Section 2 describes some mathematical definitions. Section 3 proposes a routing strategy based on modified Q-routing algorithm in laser microwave hybrid inter-satellite networks. Section 4 shows a performance evaluation on the proposed strategy. The last section concludes the paper.

2 Related work

This section shows the related works of the satellite link routing, intelligent satellite routing, and so on.

2.1 Microwave and laser link routing

Microwave links have been widely used on the ground communications, and wireless optical links have also been used to some extent. Paper [14] proposes the improved Hex DD algorithm in Wireless Sensor Networks (WSN), which bypasses nodes with poor microwave link quality and reduces data transmission delay and network energy consumption. Paper [15] proposes a vehicular ad hoc network algorithm called Link State aware Geographic Opportunistic routing protocol, which ensures a highly dynamic network packet delivery rate, moreover, the algorithm integrates the geographical information and link status information of nodes to select a more stable and faster route. Paper [16] shows two mainstream applications of terrestrial optical wireless (OW) communication are last-kilometer broadband connection and indoor optical communication. The issue of WSN routing has received considerable critical attention, but too little work has been devoted to OW. Most studies use inter-satellite links on routing of LEO satellite constellation. Paper [17] uses phased array antennas to optimize routing snapshots. This solution increases the time of routing snapshots and reduces the path delay of packets, but the frequent link reconfiguration increases the computational pressure and the power consumption. Paper [18] studies the LEO routing problem based on wavelength division multiplexing inter-satellite link and wavelength technology and proposes a method to solve the wavelength assignment problem in laser transmission, the method is suitable for low-latency, large-capacity routing scenarios. Paper [19] applies optical time slice switching technology in the LEO satellite constellation and proposes a distance-varying routing and time slice allocation scheme to improve network throughput. In a word, the laser link provides larger bandwidth than the microwave link, but rare researchers notice the hybrid microwave-laser links.

2.2 Resource pooling in routing

At present, plenty of research focuses on the resource pooling of ground stations and the research on satellites resource pooling is inadequate, since routing strategies are mostly based on satellite ground stations. Paper [20] sets up SDN controllers for all satellites on ground stations. The strategy calculates the shortest transmission delay path, and then performs congestion avoidance on the satellites with the longest waiting delay. This strategy requires the SDN controllers to monitor the satellite status in real time and transmit the information to the routing calculation node, the routing overhead is exponential. Paper [21] proposes a LEO/GEO hierarchical routing strategy, one controller places on the ground station, and three on-board controllers are set on the GEO satellite. This strategy requires three GEO satellites with powerful computing capabilities, which are sufficient to complete the routing of all LEO satellites. The paper [22] proposes to build resource pools on MEO satellites. Comparing with the strategy in [21], MEO satellites are responsible for fewer satellites, which reduces the pressure on on-board computing and reduces the transmission delay for routing information exchange. In summary, the ground resource pool routing requires a large amount of routing overhead, and the calculation of on-board routing requires a large amount of on-board computing resources, a well-known satellite SDN solution has not appeared yet.

2.3 Satellite routing algorithm

Some research of intelligent algorithms exists in satellite routing. In paper [23], ant colony algorithm for route discovery and time-dependent pheromone copes with the changing routing topology. However, this algorithm is not tested in detail. In paper [24], the reinforcement learning Q-learning algorithm calculates the shortest path. Compared with Q-routing, each node randomly selects the next hop, and if the destination is finally reached, the Q-value of all nodes on the path updates according to Bellman equation. This algorithm can indeed reach the destination in theory, but its Q-value represents the probability of the node reaching the destination, which has nothing to do with the speed or the quality of the path, so the path delay is not stable. The paper [25] analyzes the Q-routing algorithm bases on dynamic topology, but the theoretical analysis is not demonstrated by simulation results. The research is insufficient for satellite routing on machine learning algorithms, typical ant colony, q-learning are applied, but all the algorithms are lacking of completeness.

3 Preliminaries

In the satellite network, huge inter-satellite distance makes the transmission delay significant; and the new on-board processing methods causes new delays in the satellite routing. This section introduces the satellite network topology, the calculation method of transmission delay, and the calculation method of processing delay.

3.1 Network topology

The partial topological diagram of the LEO satellite network is as follows, extend the partial one can get the overall diagram. Nodes in the figure represent satellites, and connections represent inter-satellite links. No different-orbital link exists above 70$^\circ$ north and south latitudes (Fig. 2).

Fig. 2

Satellite constellation topology

Define the diagram as a directed graph G(V, E, P), G represents the topology of the satellite network, V represents the collection of nodes in it, E represents the collection of links between nodes, and P represents the collection of paths between any pairs of satellites. p(s, d) represents any path from starting satellite s to destination satellite d, path length $N_p$ represents the nodes number of one path, $u_i$ and $u_{i+1}$ represent any pair of adjacent satellites on the path.

3.2 Transmission delay

Compared with ground communication, transmission delay between two satellites is significant. Paper [26] introduces a method of calculating inter-satellite distance by latitude and longitude, this section briefly describes the method. As shown below, $\lambda$ and $\varphi$ are the longitude and the latitude of a satellite on the earth, respectively (Fig. 3).

Fig. 3

Inter-satellite link diagram

The above figure shows the cross section of two satellites and the earth center O. A and S are two satellites at the same altitude, their angle $\angle AOS$ is marked as $\xi$.

$$\begin{aligned} \xi =\arccos \left[ \sin \varphi _{A} * \sin \varphi _{B}+\cos \varphi _{A} * \cos \varphi _{B} * \cos \left( \lambda _{A}-\lambda _{B}\right) \right] \end{aligned}$$

(1)

According to the cosines rule, the distance between satellite and is as follows,

$$\begin{aligned} d_{A S}=R_{A} * \sqrt{2 *(1-\cos \xi )} \end{aligned}$$

(2)

where $R_A$ represents the orbital altitude of satellite A and S. Then, the inter-satellite transmission delay is as follows, where c is the speed of light.

$$\begin{aligned} T_{l}=d_{A S} / c \end{aligned}$$

(3)

3.3 Processing delay

This paper uses three models of queuing theory, namely $M/M/1/\infty$/$\infty$, $M/M/1/N/\infty$, $M/M/c/N/\infty$.

Note that in queuing theory, the queuing delay is the waiting time of packets which have entered the system but not yet received service, the service time is the total time from entering the system to leaving it. In a broad sense, processing time refers to the service time of queuing theory, this paper follows the broad sense.

In the model $M/M/1/\infty$/$\infty$, the first parameter M means the input process follows Poisson distribution, the second parameter M means the quit process follows Poisson distribution, 1 means the number of service stations; the first $\infty$ means the system capacity is infinite, and the second $\infty$ means the number of packets is infinite. The total service delay $W_S$ is as follows,

$$\begin{aligned} W_{s}=\frac{1}{\mu -\lambda } \end{aligned}$$

(4)

In the $M/M/1/N/\infty$ model, indicates the system capacity. The total service delay $W_S$ is as follows,

$$\begin{aligned} W_{s}=\frac{L_{s}}{\mu \left( 1-P_{0}\right) } \end{aligned}$$

(5)

where $L_{s}=\frac{\rho }{1-\rho }-\frac{(N+1) \rho ^{N+1}}{1-\rho ^{N+1}}$, $P_{0}=\frac{1-\rho }{1-\rho ^{N+1}}$, $\rho =\frac{\lambda }{c \mu }$.

In the $M/M/c/\infty /\infty$ model, c represents the number of service stations (CPU computing units in this paper). Then, the average service delay $W_S$ of each packet is as follows,

$$\begin{aligned} W_{S}=\frac{L_{S}}{\lambda } \end{aligned}$$

(6)

where $P_{0}=\left[ \sum _{n=c+1}^{\infty } \frac{1}{k !}\left( \frac{\lambda }{\mu }\right) ^{k}+\frac{1}{c !} * \frac{1}{1-\rho } *\left( \frac{\lambda }{\mu }\right) ^{c}\right] ^{-1}$, $\rho =\frac{\lambda }{c \mu }$. $P_0$ represents the probability that no packet from starting moment to the current moment.

4 Methods

The same-orbit satellite laser link expands the link capacity with low technical difficulty, the multi-CPU scheduling method reduces the on-board processing time and relieves the on-board computing pressure. This section proposes the routing strategy of LEO same-orbit laser and different-orbit microwave link based on Q-routing, introduces the delay caused by on-board CPU in this architecture, and applies the Q-routing algorithm to solve the routing problem.

4.1 On-board CPU resource pool and hybrid inter-satellite link

Under LEO communication circumstance, one satellite needs to communicate with five network neighbors generally, including four adjacent satellites and one ground station. In the figure below, satellites 1–4 and ground stations are neighbors of satellite 0. Particularly, if satellite 0 is near the reverse slit, the different-orbit link next to the slit is shut down, and if satellite 0 is near the pole, neither different-orbit links are available due to the large Doppler shift (Fig. 4).

Fig. 4

Microwave laser hybrid communication

Each satellite has five transceivers, four of which communicate with neighboring satellites and the remaining one with ground station. Among the five signal channels, two are laser channels, which transmit and receive signals in the same orbits; three are microwave links, which transmit and receive signals in different orbits and on the ground.

Due to the large relative motion of two satellites in different orbits, satellite aligning is difficult from thousands of kilometers within centimeters error. The relative motion of the same orbit is small, which makes laser communication relatively easy. Under existing technical conditions, the same-orbit satellites can use laser link, the different-orbit satellites still remain in microwave link. In the same time, direct processing of optical signals can’t realize, the laser signals need to convert into electrical ones for processing. Therefore, same orbits communications use photoelectric converters to receive laser signals, use electro-optical converters to transmit laser signals; different orbits and ground-sides use RF antennas to receive and transmit microwave signals.

Packets from five receivers queue in a cache buffer after entering the satellite, then the CPU resource pool delivers them to certain transmitters after calculation. A CPU resource pool contains five CPU units. For an empty pool, one packet arrives in first CPU unit for path planning. If the second packet arrives before the first one has finished, the second one enters the second CPU unit; if the third packet arrives before those two have finished, the third one enters the third CPU unit, and so on. If all units are occupied, new packets enter the cache buffer. The electrical signal turns to a laser or microwave signal at transmitters, so the buffer capacity of the transmission converters can be infinite. The following figure shows the queuing model for a satellite (Fig. 5).

Fig. 5

The queuing model of a single satellite

The satellite networks transmit packets from a ground station to another ground station. Per second packets number obeys Poisson distribution, so that in each path, the packets number still obey Poisson distribution. Packets pass through a satellite in three steps: satellite reception, on-board processing, and satellite transmission.

For a receiver, the mean arrival number rate is $\lambda _{1 j}(j \in 1,2,3,4,5)$, the mean departure rate is $\mu _{1j}$. Laser receivers follow $M/M/1/N/\infty$, and microwave receivers follow $M/M/1/\infty$/$\infty$.

Since the sum of several independent Poisson processes is still a Poisson process, the strength of the combined Poisson process equals to the sum of all previous strengths. Thus, the packets strength before resource pool is $\lambda _{2}=\sum _{i=1}^{5} \lambda _{1 i}$. The average processing speed of each CPU unit is $\mu _2$. The processing of a resource pool follows $M/M/c/\infty /\infty$.

The arrival rate at each transmitter is $\lambda _{3j}(j \in 1,2,3,4,5)$, the departure rate is $\mu _{3j}$, the queuing model is $M/M/1/\infty /\infty$.

In summary, a satellite includes three queuing models and generates three processing delay: $W_{S1}$ when reception, $W_{S2}$ when on-board processing, and $W_{S3}$ when transmission. The total processing delay $W_S$ is as follows,

$$\begin{aligned} W_{S}=W_{S 1}+W_{S 2}+W_{S 3} \end{aligned}$$

(7)

4.2 Routing model in mathematics

Based on the conclusion of delay analysis, this section presents the mathematical model of LEO routing problem.

The routing algorithm in this paper aims to find a path with the shortest delay. The topology of a satellite constellation is constantly changing, that is to say, the link distance (especially different-orbit links) changes continually. The academic method to cope with this dynamic characteristic is called “topology slicing,” which slices the continuously changing topology into n discrete topologies with little change between neighboring slices. This method replaces the topology changes over a small period of time with a constant topology, thus, the larger the n, the smaller the topological difference between each adjacent slice; if the n is suitable, the delay difference introduced by this method can be negligible. Above all, the topology used for routing calculations is static.

For a packet sent from the ground, the satellite routing is independent of the transmission between satellites and ground. The packet enters the starting satellite from the ground, passes through starting receiver, through the CPU resource pool, through the transmitter, and then reaches the next satellite. The packet repeats these steps until reaches destination satellite, which then transmits it to the aiming ground station. In this process, both the starting satellite and the destination satellite are fixed, the intermediary satellite responsible for the transmission is variate. Consequently, three delays are the same for each path: the transmission delay from the satellite to the ground, the processing delay generated at the starting receiver, and the processing delay at the destination transmitter. These delays do not affect the routing and can be ignored. In summary, the problem becomes finding the shortest path with the shortest delay from the starting resource pool to the destination resource pool. Entire constraints required for a routing path are as follows.

First, among all the paths that can connect the start and destination satellites, exclude excessively long ones

$$\begin{aligned} \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p} - 1}}}\limits _{({u_i},{u_{i + 1}}) \in p(s,d)} {T_l}({u_i},{u_{i + 1}})\le {T_{{l_{\max }}}},\,\,\, p(s, d) \in P \end{aligned}$$

(8)

Second, the load of the path can’t be too heavy, the processing delay must not be too large.

$$\begin{aligned} \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p}}}}\limits _{{u_i} \in V} {W_S}({u_i}) \le {W_{{S_{\max }}}},\,\,\, p(s, d) \in P \end{aligned}$$

(9)

Finally, the routing algorithm performs congestion control by detouring, but still does not make the path too remote, the total delay should be within a certain range.

$$\begin{aligned} \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p} - 1}}}\limits _{({u_i},{u_{i + 1}}) \in p(s,d)} {T_l}({u_i},{u_{i + 1}}) + \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p}}}}\limits _{{u_i} \in V} {W_S}({u_i}) + {W_S}({u_{{N_p}}}) \le {T_{\max }}, \,\,\,p(s, d) \in P \end{aligned}$$

(10)

In summary, follows expresses the mathematical model for the shortest delay path problem.

$$\begin{aligned} \begin{array}{l} {p^*} = {\mathrm{arg}}\mathop {\mathrm{min}}\limits _{p(s,d)} \left\{ \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p} - 1} }}\limits _{({u_i},{u_{i + 1}}) \in p(s,d)} {T_l}({u_i},{u_{i + 1}}) + \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p}}}}\limits _{{u_i} \in V} {W_S}({u_i}) \right\} \\ {\mathrm{s.t.}}\left\{ \begin{array}{l} \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p} - 1}}}\limits _{({u_i},{u_{i + 1}}) \in p(s,d)} {T_l}({u_i},{u_{i + 1}}) \le {T_{{l_{\max }}}}\\ \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p}}}}\limits _{{u_i} \in V} {W_S}({u_i}) \le {W_{{S_{\max }}}}\\ \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p} - 1}}}\limits _{({u_i},{u_{i + 1}}) \in p(s,d)} {T_l}({u_i},{u_{i + 1}}) + \mathop {\mathop {\sum }\limits _{i = 1}^{{{N_p}}}}\limits _{{u_i} \in V} {W_S}({u_i}) + {W_S}({u_{{N_p}}}) \le {T_{\max }}\\ p(s,d) \in P\\ {T_l}({u_i},{u_{i + 1}}) \ge 0\\ {W_S}({u_i}) \ge 0 \end{array} \right. \end{array} \end{aligned}$$

(11)

$p^{*}$ represents the shortest delay path, represents any path from s to d, $T_{l_{\max }}$ and $W_{S_{\max }}$ are both predefined upper bound, and $T_{\max }$ is the total delay threshold.

4.3 Modified Q-routing routing algorithm

In this paper, we use a modified Q-routing algorithm [27] to compute routes based on satellite topology slicing. Although topology is fixed, the load is dynamic. Therefore, traditional static algorithms are no longer applicable, routing algorithms that consider dynamic load in real time can find better paths. The algorithm used in this paper dynamically takes the processing delay into account, this algorithm uses Dijkstra algorithms to accelerate the convergence.

The following is a representation of the Q-value in the Q-routing algorithm.

$$\begin{aligned} Q_{u_{i}}\left( u_{N_{P}}, u_{i+1}\right) \end{aligned}$$

(12)

This value represents the estimated cost for the satellite to send the packet from the neighbor satellite $u_{i+1}$ to $u_{N_{P}}$.

The satellite $u_{i}$ sends the packet to the neighbor with the lowest Q-value, after receiving the packet, neighbor satellite $u_{i+1}$ immediately reports its minimum sending cost $Q_{u_{i+1}}\left( u_{N_{\rho }}, u_{i+1}\right)$, path delay $T_{l}\left( u_{N_{P}}, u_{i+1}\right)$, receive delay $W_{S1}\left( u_{i+1}\right)$ to the satellite $u_{i}$, then $u_{i}$ iterates over these value. The iteration is performed as follows,

$$\begin{aligned} \begin{array}{l} {\text {New}} Q_{u_{i}}\left( u_{N_{p}}, u_{i+1}\right) \\ \quad =\left( 1-\alpha \right) Q_{u_{i}}\left( u_{N_{p}}, u_{i+1}\right) \\ \qquad +\alpha \left( T_{l}\left( u_{i}, u_{i+1}\right) +W_{S}\left( u_{i}\right) + \mathop {\min }\limits _{u_{i+2}\in {\text {neighbors}}\,{\text{ of }}\,u_{i+1}} Q_{u_{i+1}}\left( u_{N_{p}}, u_{i+2}\right) \right) \end{array} \end{aligned}$$

(13)

Since the transmission delay is constant for each topology, Dijkstra algorithm can provide an approximate direction to the destination. However, the Dijkstra algorithm only take transmission delay into consideration, the reinforcement learning algorithm can use the dynamic processing delay to find the approximate optimal solution. Combining two algorithms, this paper come to a fast converging dynamic algorithm.

The flow of the modified Q-routing algorithm is as follows, paths find by Dijkstra algorithm only use transmission delay, so they can be pre-calculated before the arrival of packets and do not take up the routing delay. The computed path table stores in the satellite, which requires some storage space but greatly reduces the route computation time (Fig. 6).

Fig. 6

The modified Q-routing algorithm flow

Dijkstra algorithm first calculates the routing graph, obtains the shortest path between two points in advance based on path delay. Since the time complexity of this algorithm is $O\left( N^{2} \right)$, Dijkstra takes some time and some computational resources to calculate the routing table. For each satellite node, Dijkstra algorithm gets the routing table from this satellite to all other satellites in one computation, so the average computation of all packets is relatively low. Meanwhile, this calculation happens only one time in each time slice, so it only occupies the storage resources, the space complexity of the algorithm at the starting satellite is $O\left( N \right)$.

The Q-routing algorithm has a small amount of computation each iteration. When delay converges, each satellite have a Q-table of size ${\mathrm{Num}}\left( d\right) *{\mathrm{Num}}\left( y\right)$. The total space complexity of its algorithm is $O\left( NAH\right)$, while the average space complexity to each satellite is $O\left( AH\right)$ ($N={\mathrm{Num}}\left( u_{i}\right)$, $A={\mathrm{Num}}\left( u_{i+1}\right)$, $H={\mathrm{Num}}\left( d\right)$, $u_{i+1}$ is a neighbor of $u_i$, d is the destination and K is the number of iterations). Therefore for different system configurations, different algorithms are available. The complexity comparison of two algorithms is as follows (Table 1).

Table 1 Complexity analysis

5 Experiments

The simulation tests the performance of three algorithms under Walker constellation configuration. The three algorithms are: Dijkstra algorithm, Q-routing algorithm, and modified Q-routing algorithm. The orbit parameters are as follows (Table 2).

Table 2 Orbit parameters

Fig. 7

Routing of 2 million packets using three algorithms

Figure 7 shows the results of routing 2 million packets. 20,000 packets per batch route from random start to random destinations in an 18 * 40 network. The delay in the vertical coordinate represents the average delay per batch. As seen from the figure, the path delay of Dijkstra algorithm is very large. The reason of the distinguishing is that Dijkstra algorithm only considers static path delay, it cannot consider dynamic processing delay, and distinctly, this drawback will be more obvious when the higher load. At the same time, the convergence speed of the Q-routing algorithm is slow, and using Dijkstra algorithm for guidance accelerates the convergence of the Q-routing.

According to Fig. 7, the shortest path convergence of the network is about 20 ms, which meets the expectation of Oneweb and Stralink. Meanwhile, the complexity of this result is as follows (Table 3).

Table 3 Complexity analysis of Fig. 7

Fig. 8

Different batch using modified Q-routing

Fig. 9

Different batch using Q-routing

Figure 8 calculates the path delay based on the modified Q-routing algorithm, the 18 * 40 satellite network. The figure shows that for each batch of 2000 to 10,000 packets, the path delay eventually converges to less than 20 ms; the delay increases to more than 20 ms for batches within more than 10,000 packets. The delay converges to the minimum for all batches when the number of network iteration rounds near 30. As the load increases, the convergence delay also increases. The rate of delay convergence is slow at 2000 per batch but fast at 3000, which shows that 2000 is a threshold value.

Figure 9 shows the path delay based on the q-routing algorithm. Comparing with Fig. 8, the convergence speed of the modified algorithm is greater than that of the q-routing algorithm under any topology size.

The following is a simulation for the modified algorithm with the same load under different topologies.

Fig. 10

Modified algorithm under different topologies

Figure 10 shows the simulation of modified algorithm under topologies of 5 * 5, 10 * 10, 15 * 15, 20 * 20, 25 * 25. 2000 packets per batch is a large load for a 5 * 5 network. As the network gets larger, the packet routing distance gets longer, but the transmission delay shows a decreasing trend. This is due to the fact that the increase in transmission delay does not fit the decrease in processing delay. However, the reduction trend is getting smaller, if the network size increases again, the delay will increase. Also, as the network increases, the convergence rate becomes slower and slower. The first two networks converge almost immediately, but the 20 * 20 and the 25 * 25 converge at the 10th and 20th iteration, respectively.

Fig. 11

Packets hop number with three algorithms

Figure 11 shows the packets hop number with different algorithms. In Fig. 11, Dijkstra algorithm has the least hop number, followed by Q-routing, modified algorithm. However, from the previous simulation, the delays are in the opposite order. So, the smart algorithm increases the hop number to get a lower delay, the packets bypass the path with high processing delay and achieve a total delay reduction.

Fig. 12

Modified algorithm with different learning rates

Figure 12 shows the convergence of the modified algorithm with different learning rates and a network of 18 * 40. As presented in the picture, with large learning rates, the network takes 20 to 40 batches to converge; the smaller the learning rate, the slower the convergence rate. The convergence speed at the same learning rate is proportional to the network size, the smaller the network, the faster the convergence. See the following figure for details.

Fig. 13

Different networks with different learning rates

Figure 13 shows the simulation on networks of 5 * 5, 10 * 10, and 20 * 20 with learning rates of 1, 0.8, 0.5. In this figure, the delay converges quickly at about 40 ms for the network size of 10 * 10, regardless of learning rate; the 20 * 20 network converges at a slower rate than the 10 * 10, and the 30 * 30 network is even slower. Note that in the same network, the smaller the learning rate $\alpha$, the smaller the convergence delay, indicating a large learning rate in a small network makes traffic slow.

6 Results and discussion

This paper applies a modified Q-routing algorithm to the same-orbit laser and different-orbit microwave satellite links with on-board CPU resource pools. Hybrid links increase the broadband, resource pools meet the requirement of tight computing resources in satellite routing. This satellite architecture generates new processing delay. The modified Q-routing algorithm takes transmission delay and processing delay into account to find a faster convergence. Experiments show that the delay of the modified algorithm is significantly smaller than that of the Dijkstra algorithm, and the convergence delay is stable.