Minimum-length scheduling with rate control in wireless networks: a shortest path approach

Abstract

In this paper, the minimum-length scheduling problem in wireless networks is studied, where each source of traffic has a finite amount of data to deliver to its corresponding destination. Our objective is to obtain a joint scheduling and rate control policy to minimize the total time required to deliver this finite amount of data from all sources. First, networks with time-invariant channels are considered. An optimal solution is provided by formulating the minimum-length scheduling problem as finding a shortest path on a single-source directed acyclic graph. However, finding the shortest paths is computationally hard since the number of vertices and edges of the graph increases exponentially in the number of network nodes, as well as in the initial traffic demand values. Toward this end, a simplified version of the problem is considered for which we explicitly characterize the optimal solution. Next, our results are generalized to time-varying channels. First, it is shown that in case of time-varying channels, the minimum-length scheduling problem can be formulated as a stochastic shortest path problem and then an optimal policy is provided that is based on stochastic control. Finally, our analytical results are illustrated with a set of numerical examples.

I. Introduction

The problem of minimum-length scheduling involves obtaining a sequence of activations of wireless nodes so that a finite amount of data residing at a subset of the nodes in the network reaches its intended destinations in minimum time. It is closely related to the problems of network throughput or stable throughput maximization, since minimizing the time to deliver a fixed amount of data can be seen as maximizing the effective rate at which data traverse the network. However, it also differs from them in that some network resources become available to heavily loaded nodes when the lightly loaded ones become relieved. Furthermore, it can be introduced as a useful alternative metric that characterizes the traffic-carrying capabilities of wireless networks with non-stationary and non-ergodic channel variations, where the commonly used performance criteria of stable throughput and network capacity are not well defined. Although in this paper we focus on networks with stationary and ergodic channel behavior, we expect our analysis to yield valuable insights regarding the more general case of non-ergodic and non-stationary wireless channels.

The topic of obtaining schedules of minimum length has attracted the attention of the research community, as in [1–10] to sample a few. The first formulations of the problem of scheduling for efficient access to a shared channel, which we are aware of, appeared in [1, 2]. A simple collision channel model had been considered but with the possibility of spatial reuse. That is, an "interference map" was assumed in terms of a graph that described all the independent sets of nodes in the graph, namely those sets of nodes that do not include "adjacent" nodes. The objective was to determine the shortest length of a frame of slots that would allow all nodes to transmit once in the frame without violating the "interference" constraint imposed by the interference rules on the graph. It was shown that the problem is NP-complete, and a distributed heuristic was developed that showed decent performance compared to the optimum that was computable in "small" instances of the problem.

This problem was revisited in more generality through a continuous approximation of the structure of the frame schedule in [3], where the authors obtain a centralized, polynomial time algorithm for static networks that finds a schedule of minimum length satisfying a set of link-traffic requirements. However, in [3], modeling of the physical layer is overly simplified as it is assumed that any two links can be successfully activated simultaneously as long as they do not share any common vertices. This simplification relates the minimum-length scheduling problem to the problem of obtaining a maximal matching in a non-bipartite graph [11]. However, due to the broadcast nature of the wireless medium all, concurrent transmissions can potentially contribute to the total amount of interference at each receiver and, thus, cause a reception to fail.

A commonly accepted model for capturing the effects of interference is the Signal to Interference plus Noise Ratio (SINR) criterion under which the outcome of a transmission depends on the ratio of the signal power at a receiver to the noise and the total interference. If this ratio exceeds a certain threshold, then the transmission is assumed to be successful. Although still an approximation, this model is reasonable and captures the overall interference generated by the simultaneous transmissions in the network. A variation of this problem formulation that incorporated some physical layer attributes was studied in [4, 5].

In [5], the authors consider the problem of obtaining a schedule of minimum length under the SINR interference model. They assume that the transmission rates are fixed and that each transmitting node selects optimally its transmission power. In [5], the minimum-length scheduling problem is formulated as a linear program [12] that can possibly have a prohibitively large number of variables and thus is hard to solve. In [6–9], the authors consider the minimum-length scheduling problem for different sets of optimization parameters. Specifically, they consider the cases where (1) both the transmission powers and rates are fixed, (2) the transmission powers can be optimized, but the transmission rates are fixed, and (3) the transmission powers are fixed and each transmitter is allowed to choose its rate from a predetermined, finite set of rates, common to all transmitters. In [6–9], the minimum-length scheduling problem is also formulated as a complex linear program, with a relatively small number of constraints and a large number of variables. To address the high complexity, the authors employ the technique of column generation [12], whose running time is faster, on average, than that of the original linear program.

Most of the prior work on the minimum-length scheduling problem assumes that the transmission rates are fixed. However, due to the broadcast nature of the wireless medium, the parameters of the physical layer, such as the transmission powers and rates, are coupled with the scheduling decisions that can be made at the medium access control. In the recent years, there is ample evidence of how the variables associated with a particular layer depend on variables associated with other layers. Exploiting the links and dependencies between such variables usually yields superior solutions (see e.g., [13–17]). This is similar, if not equivalent, to maximizing a function of multiple variables by considering its dependence on all the variables rather than on a subset of them alone. Therefore, due to this coupling between the physical layer and the medium access control in wireless systems, it is clear that a joint optimization of link activation and rate control will yield a better performance, which is the focus of this paper.

In [10], the authors consider a cross-layer view of the minimum-length scheduling problem for static, single-hop networks through rate control and formulate the problem as a shortest path on a directed acyclic graph (DAG). In the first part of this paper, we consider static networks where the channel effect is due to pure path loss. We first assume a slotted-time model and formulate the minimum-length scheduling problem as a shortest path between a given source-destination pair on a DAG. We obtain an optimal joint scheduling and rate control solution that provides a shortest path. Although finding a shortest path on a DAG has a polynomial complexity in the number of vertices and edges, this number grows exponentially as the size of the network and initial data traffic load increase. For this reason, we make the following simplifications. We first map the discrete-time problem to a continuous-time equivalent, where slots are eliminated. We then restrict the possible scheduling and rate control decisions to either communication "one at a time", in a Time Division Multiple Access (TDMA) fashion, or "all together" for all time. A similar approach for the problems of sum-rate maximization and proportional fairness was considered in [18]. We explicitly characterize the optimal solution of this reduced problem. Understanding the behavior of the optimal policy, even for the reduced problem, is significant since it provides valuable intuition about when scheduling and rate control actions for one or the other extreme are preferable. This intuition, for example, can improve the performance of the column generation technique in [6–9] by providing the algorithm with those scheduling and rate control actions that are expected to be employed by an optimal policy in the reduced problem and thus improve that heuristic.

However, the cited work (i.e., [1–10]) studies the minimum-length scheduling problem only for time-invariant wireless networks. Since the wireless channel in reality is time-varying, in the second part of this paper, we consider the time-varying case. Our goal then becomes to find an optimal policy that minimizes the expected time required to deliver all the traffic to its respective destinations. We solve the minimum-length scheduling problem by formulating it as a stochastic shortest path, which is a special case of a Markov Decision Process (MDP) [19]. We obtain an optimal scheduling and rate control policy through stochastic control methods. Our approach is similar to the works in [20] and [21] where the minimum-length scheduling problem is formulated as an MDP under exact and statistical knowledge of the underlying channel conditions, respectively. For time-invariant channel processes, this model reduces to finding a shortest path on a DAG and methods described in the first part of this paper are applicable to compute the optimal solution.

Our work differs from [1–3] since we model the interference more accurately through the SINR interference model. We follow a different approach from [4–9] since we formulate the minimum-length scheduling problem as finding a shortest path on a single-source DAG, and we give an optimal graph-theoretic algorithm. Furthermore, we provide an explicit characterization of an optimal policy for a simplified, continuous-time model that is obtained by reducing the set of feasible scheduling and rate control decisions to either transmission in the "one at a time" fashion, as in TDMA, or in the "all together" mode. Our results are different from [4, 5] since we consider joint scheduling and rate control decisions. Finally, and more importantly, we generalize existing work by considering the more realistic case of time-varying channels.

The rest of this paper is organized as follows. In Section II, we present the network model. In Section III, we consider static wireless networks. In particular, in Section III-A, we present a graph-theoretic formulation of the minimum-length scheduling problem as a shortest path problem on a single-source directed acyclic graph. In Section III-B, we first map the problem to a continuous-time model and we then restrict the set of feasible scheduling and rate control actions that can be employed. By doing so, we are able to explicitly characterize an optimal policy that finds a schedule of minimum length. Then, in Section IV, we consider time-varying wireless networks. In Section IV-A, we formulate the minimum-length scheduling problem as a stochastic shortest path, and in Section IV-B, we provide an optimal solution by employing the principles of stochastic control theory. Specifically, we use the value iteration method [19] to find a stochastic shortest path. In Section V, we complement our analytical results with some numerical experiments, and finally, in Section VI, we conclude the paper. In Appendix 1, a table of variables is provided to make the notation comprehensively clearer to those who prefer to see the full notational picture. An optimal algorithm that computes a shortest path on a DAG is given in Appendix 2. The proofs of our results appear in Appendices 3-4.

II. Model formulation

We consider a slotted-time, single-hop, wireless network consisting of K transmitter and receiver pairs. Without loss of generality, we assume that the slot duration is equal to 1 sec. Each transmitter has a finite amount of data units, e.g., a file of packets or bits to deliver to its corresponding receiver. The objective is to activate the transmitters so that the time to deliver all the traffic to the intended receivers is minimized. The single-hop network assumption, albeit simplifying, is interesting and highly non-trivial since it captures the fundamental problems that arise due to interference, when multiple nodes attempt simultaneous channel access. We denote by $K=\left\{1,\dots ,K\right\}$ the set of all transmitter and receiver pairs in the network. At every time slot, each transmitter $k\in K$ can either transmit at its maximum transmission power $P_k^{\text{max}}$ or remain silent. We denote the transmission power level of the k th transmitter at time slot t by P _k ( t ), where $P_k\left(t\right)\in \left\{0,P_k^{max}\right\}$.

It is assumed that each transmitter k has a fixed amount of d _k bits to deliver to its corresponding destination. We denote by d = (d₁, ..., d _K ) the vector of initial data traffic at each transmitter. We also denote by X _k (t) the queue size at transmitter k at time slot t and by X(t) = (X₁(t), ..., X _K (t)) the corresponding vector of queue sizes at all transmitters in the network. The queue size of each transmitter at time slot 0 is equal to its initial data traffic, i.e., X(0) = d. The state space of the process ${\left\{\text{X}\left(t\right)\right\}}_{t=0}^{\infty }$ is denoted by $X$.

We also consider a channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$ that takes values from a finite set $G$. For every time slot t, the channel state $\text{G}\left(t\right)=\left(G_{\left(k,j\right)}\left(t\right),\forall k,j\in K\right)$ gives the channel quality between every transmitter k and receiver j in the network. This model captures the effects of channel variations due to e.g., node mobility, fading, or fixed path loss. It is assumed that the channel follows a block fading model with block length equal to the duration of a time slot. Hence, the channel conditions change only at the beginning of each time slot and remain constant throughout the slot duration. The network model under consideration is depicted in Figure 1.

Figure 1

A network of K transmitter/receiver pairs.

We include the physical layer effects by adopting the Signal to Interference plus Noise Ratio (SINR) criterion. Specifically, a transmission from transmitter k to receiver k is successful if the ratio of the received signal power to the sum of the thermal noise and the total interference exceeds a certain threshold. The exact value of the SINR threshold depends on various factors, such as the transmission rate, the target probability of bit error, the coding and modulation techniques employed at the transmission, etc. In this paper, we focus on the dependence of this threshold on the transmission rate and assume that the rest of the parameters affecting it are fixed. We denote by γ_t,k(r _k (t)) the SINR threshold value at receiver k that must be met or exceeded in order to receive successfully from transmitter k at rate r _k (t) at time slot t. Consequently, we say that at slot t transmitter k transmits successfully to receiver k at rate r _k (t) if

$$\text{SIN}{\text{R}}_k\left(t\right)=\frac{P_k\left(t\right)G_{\left(k,k\right)}\left(t\right)}{N_k+{\sum }_{j=1,j\ne k}^K{P}_j\left(t\right)G_{\left(j,k\right)}\left(t\right)}\ge {\gamma }_{t,k}\left(r_k\left(t\right)\right),$$

(1)

where N _k is the thermal noise power at receiver k.

It is known that the maximum transmission rate is an increasing function of the SINR threshold (see e.g., [22]). This gives rise to the following trade-off: By increasing the transmission rate, the number of transmitters that can successfully satisfy the SINR criterion concurrently decreases. On the other hand, by decreasing the transmission rate, a higher number of transmitters can jointly satisfy the SINR criterion. Thus, it is not clear whether allowing more concurrent transmissions (less time sharing) at lower rates is preferable to allowing fewer concurrent transmissions (more time sharing) at higher rates. The answer is tightly dependent on the performance objective and on the network parameters, such as the transmission powers, channel conditions, etc.

A joint scheduling and rate control policy at any given time needs to decide (a) which transmitters to activate and (b) their respective transmission rates. This information can be captured by the K-dimensional rate vector r(t) = (r₁(t), ..., r _K (t)), where r _k (t) is the rate of transmitter k at slot t. If a transmitter is assigned a zero rate, then it is not activated by the policy. In other words, a transmission rate vector implicitly specifies the scheduling decisions. We define the set of all feasible rate vectors to contain those that are obtained by the following two-step procedure: We first identify all possible subsets of activated transmitters (by assigning to each transmitter k either power 0 or $P_k^{max}$), and then, we assign them the maximum rates that allow all activated transmitters to jointly satisfy the SINR criterion. Thus, there exist 2 ^K - 1 such K-dimensional transmission power vectors, each of which corresponds to an achievable rate vector. Clearly, the set of achievable rates depends on the current channel state $\text{g}\in G$. Hence, for every channel state g, we denote by $R(\text{g})$ the finite, discrete set of feasible K-dimensional rate vectors. Then, the cardinality of $R(\text{g})$, i.e., $R(\text{g})$, is equal to 2 ^K- 1 for every channel state $\text{g}\in G$.

In this paper, we are interested in obtaining optimal policies that take joint scheduling and rate control decisions under the objective of minimizing the (expected) time to deliver all data to the intended destinations. The policies we consider are aware of the network queue sizes at all times. Furthermore, unlike in [21], they are assumed to know the current channel conditions in order to make accurate scheduling decisions. For every slot t, the pair of the channel state G(t) and queue sizes X(t) constitutes the system state S(t). We denote by $S$ the state space of the system state process ${\left\{S\left(t\right)\right\}}_{t=0}^{\infty }$, which is given by

$$S=\left\{\left(\text{x},\text{g}\right):\text{x}\in X,\text{g}\in G\right\}.$$

(2)

We restrict our attention to stationary policies that take decisions merely based on the current system state information. Consider a state $\text{s}=\left(\text{x},\text{g}\right)\in S$, and let the system state at time slot t satisfies S(t) = s. Let us also define the set $A\left(\text{s}\right)$ to be a subset of the overall feasible scheduling and rate control decisions corresponding to state s = (x, g), i.e., $A\left(\text{s}\right)\subseteq R\left(\text{g}\right)$. Then, the policies we consider are given by the mapping

$$\text{r}\left(t\right)=\mathbf{\pi }\left(\text{s}\right),\mathbf{\pi }:X\times G\to A\left(\text{s}\right)\subseteq R\left(\text{g}\right).$$

(3)

That is, we allow for the possibility of restricting the space of actions by limiting the range of allowable policies. If $A\left(\text{s}\right)$ is a strict subset of the overall feasible scheduling and rate control decisions, then scheduling will be suboptimal in general at the benefit of decreased complexity. Furthermore, by selectively choosing the elements of the set $A\left(\text{s}\right)$, it is possible to obtain performance close to optimal while achieving considerable reduction in computational complexity.

We assume that every admissible policy uses the channel state information rationally so that a scheduled transmission must always be successful. Naturally, as reflected by the cardinality of the set $R(\text{g})$, the policies we consider are non-idling, i.e., they always activate at least one transmitter that has a non-empty queue until all the queues in the network are empty. We call the class of stationary, non-idling policies given by the mapping (3) as admissible, and denote them by Π.

The queue size process evolves according to

$$\text{X}\left(t+1\right)={\left[\text{X}\left(t\right)-\text{r}\left(t\right)\right]}^{+},t\ge 0,$$

(4)

where the rate control and scheduling vectors r(t) are given according to the mapping in (3) and where [z]⁺ = max{z, 0}.

Clearly, the queue size at each transmitter k takes its maximum value at time slot 0, when it is equal to the initial demand d _k and, due to the absence of external arrivals, it keeps decreasing over time until it reaches zero. Under the above model, we proceed to formulate the minimum-length scheduling problem for static and time-varying networks.

III. Static networks

In this section, we restrict our attention to static networks, where the channel qualities G_{(k, j)}(t) are equal for every time slot t, i.e., we ignore effects of fading or user mobility. Thus, the cardinality of the set $G$ is equal to one. To simplify notation, in this section, we denote the channel quality G_{(k, j)}(t) as G(k, j). We will drop this assumption in Section IV where we will consider time-varying channel processes. Furthermore, since there is a single channel state g, to simplify notation, we will write $R$ to denote $R(\text{g})$ for $\text{g}\in G$ and $A$ to denote $A\left(\text{s}\right)$ for s = (x, g), $\text{g}\in G$. At every time slot t, the scheduling and rate control policy identifies a rate vector $\text{r}\left(t\right)=\left(r_1\left(t\right),\dots ,r_K\left(t\right)\right)\in A\subseteq R$ that specifies which transmitters are activated and their respective rates.

We can formulate the minimum-length scheduling problem as follows:

$$\text{minimize}:T$$

(5)

$$\text{subject to}:\text{X}\left(T\right)=0,\text{X}\left(0\right)=\text{d},$$

(6)

$$T\in \mathbb{N}.$$

(7)

In the specific case of pure TDMA scheduling, combined with rate control, where only a single transmitter can be active at any given time, the solution of the above problem becomes trivial. Specifically, each transmitter must be active for as many time slots as needed to empty its queue. The required number of such time slots for each transmitter k is equal to the ratio of its initial demand d _k divided by its corresponding rate when it accesses the channel individually, rounded upwards to the closest integer value. Then, the minimum total time that is needed until all the queues are empty is equal to the sum of the time slots required by each transmitter. The order in which the transmitters must be activated is immaterial; they can be chosen in a round-robin or random fashion.

However, the solution of the optimization problem given by (5)-(7) is, in general, a non-trivial discrete optimization problem. In the following subsections, we provide an optimal graph-theoretic algorithm by mapping it to a shortest path problem on a DAG, and in the sequel, we give an explicit characterization of the optimal policy for a reduced version of this problem.

A. The equivalent DAG representation

We construct the weighted DAG $Ḡ=\left(V,E\right)$ as follows: We assume that every vertex u ∈V of the DAG represents a queue size vector that can be obtained, starting from a vector of queues X(t), by employing some scheduling and rate control action chosen from the set $A$. Furthermore, every directed edge (u, v) ∈ E represents one such action in $A$. We say that the edge (u, v) is incident from u and incident to v. Hence, from every vertex x _i , we can have $\left|A\right|$ edges that are incident from x _i , each corresponding to a different rate vector ${\text{r}}^j,i=1,\dots ,\left|R\right|$. Each such edge is incident to a graph node y _i = [x _i - rⁱ ]⁺. We disallow those edges that correspond to rate vectors, which activate transmitters with empty queues. Therefore, the actual number of edges that are incident from a vertex can be less than $\left|A\right|$. The weight of each edge is equal to one. From now on, we will refer to action rⁱ by means of the edge (x _i , y _i ). The unique source node x₀ of the DAG corresponds to the vector of initial demands, X(0).

In Figure 2, we give an example of such a graph for a network of two transmitter and two receiver pairs. We assume that the initial demands are d₁ = 4 bits and d₂ = 6 bits and that we have three possible scheduling and rate control actions, namely (1) only transmitter 1 accesses the channel at a rate of 3 bits/sec, (2) only transmitter 2 accesses the channel at a rate of 3 bits/sec, and (3) both transmitters concurrently transmit at a rate of 2 bits/sec each. Figure 2 depicts the DAG that is obtained by these three actions. Note that from each vertex all the three rate control actions are allowed, as long as each action schedules transmitters with non-empty queues. For example, in Figure 2, the only viable rate control action for the queue size vector [4,0] is to activate transmitter 1 individually.

Figure 2

A DAG construction corresponding to initial demands d ₁ = 4 bits and d ₂ = 6 bits and three rate control actions r¹ = [3,0], r² = [0,3], and r³ = [2,2].

As we observe from Figure 2 for any path of vertices < x₀, x₁, x₂, ..., x _m >, the queue size vector of each vertex in the path has to be component-wise larger or equal to the queue size of any other vertex that succeeds it in the path and the queue size vectors of any two vertices on the graph cannot be the same. As a result, the overall graph of the different queues is directed and acyclic. Finally, it is clear that every path starting at the source x₀ ends at the 0-vector. Moreover, the weight of any sub-path < x₀, x₁, x₂, ..., x _m > is equal to its length m, which is effectively the number of time slots required to go from vertex x₀ to vertex x _m along the specified path, since by construction of this DAG the weight of each edge is equal to one time slot. Thus, the initial problem given by (5)-(7) is transformed into a shortest path problem on a weighted single-source DAG. A shortest path on the DAG G = (V, E) can be obtained through the DAG-SHORTEST-PATHS$\left(Ḡ,{\text{x}}_0\right)$ algorithm. The exact algorithm, taken from [11], is given in Appendix 2.

For the example given in Figure 2, the shortest path algorithm selects the sequence of actions r³, r³, r², and the minimum schedule length is equal to 3 sec (slots). Note that the sequences of actions r², r³, r³ and r³, r², r³ are also optimal as the order in which the actions are taken is immaterial in terms of minimizing the time needed to empty the queues, under the assumption of static channels. Also, it is worth mentioning that the length of the optimal schedule obtained through rate control is, naturally, no longer than that of TDMA (employing only actions r¹ and r²), which, in this example, has length 4. Furthermore, it is reasonable to expect that the difference in the schedule lengths under the two schemes can become significant when the number of transmitter/receiver pairs in the network increases or when the values of initial demands are large. This will also be illustrated through a set of numerical results in Section V.

The optimality of DAG-SHORTEST-PATHS$\left(Ḡ,{\text{x}}_0\right)$ can easily be verified (see e.g., [11], Theorem 24.5). Also, it is easy to see that its overall running time is Θ(|V | + |E|). Hence, the number of operations needed to compute a shortest path on a single-source DAG is of polynomial complexity in the number of vertices and edges. However, in our DAG construction, this number grows exponentially (1) in the number of transmitters when $A=R$ since from every vertex there exist 2 ^K - 1 potential edges that are incident from it and (2) as the initial demands increase. Therefore, the overall complexity of the algorithm becomes exponential, and despite its theoretical merit, it is rendered impractical.

B. Continuous-time model

To decrease the complexity that stems from the discrete nature of the minimum-length scheduling problem, we can map the problem given by (5)-(7) to a continuous-time one. Therefore, instead of seeking the minimum number of time slots required to deliver all data traffic to its respective destinations, we will be interested to obtain the minimum duration or period of time that has to elapse until all network queues empty. In this way, the minimum-length scheduling problem becomes a linear program with a relatively small number of constraints and a large number of variables as in the formulations of [5], [6], and [7]. In order to solve this linear program, we follow a different approach than [5], [6], and [7]. In particular, we reduce the number of variables involved, i.e., the scheduling and rate control decisions that the policy employs, and then obtain an optimal solution for this reduced problem.

As is commonly done, to understand the essential features of a difficult problem, one needs to consider a simplified version that inherits and keeps the key features of the problem. Specifically, here, we restrict the set $A$ to contain only feasible rate vectors obtained by two simple schemes, namely scheduling a single transmitter at a time (in a TDMA fashion) or concurrently activating all the transmitters, as considered in [23, 24]. By doing so, we decrease the cardinality of $A$ to K + 1. Clearly, such a reduction is expected to yield suboptimal results. However, this simplification focuses attention on the trade-off between transmitting more frequently (at a reduced rate) and transmitting less frequently (at a higher rate). We expect that this approach will help us gain valuable insights regarding the nature of optimal scheduling and rate control for the general problem.

We define Action k for $k\in K$ to consist of individually activating transmitter k and Action 0 to be the corresponding action when all K transmitters are activated simultaneously. Let the rate of transmitter k under individual operation be $r_k^k$ and the corresponding rate under concurrent operation be $r_k^0$. Furthermore, let us denote by τ _i for i ∈ {0, ..., K} the period of time that Action i is utilized. The above are illustrated in Figure 3.

Figure 3

The K + 1 possible actions obtained by either scheduling K transmitters "one-at-a-time" or "all-at-once". The rate of source k under Action j is denoted by $r_k^j$.

Then, the continuous-time equivalent of the problem given in (5)-(7) under the reduced space of actions is:

$$\text{minimize}:\sum _{i=0}^K{\tau }_i$$

(8)

$$\text{subject to}:d_k\le {\tau }_k{r}_k^k+{\tau }_0{r}_k^0,\forall k\in K$$

(9)

$${\tau }_i\ge 0,i\in \left\{0,\dots ,K\right\}.$$

(10)

The following theorem characterizes an optimal scheduling and rate control policy that solves (8)-(10).

Theorem 1: A minimum-length scheduling and rate control policy solving (8)-(10) takes actions according to the following:

• 1) If it is true that

  $$\sum _{k=1}^K\frac{r_k^0}{r_k^k}\le 1,$$

  (11)

then Action k is chosen (k = 1, ..., K) for a duration of

$${\tau }_k=\frac{d_k}{r_k^k},$$

(12)

while Action 0 is never employed, i.e.,

$${\tau }_0=0.$$

(13)

• 2) If it is true that

  $$\sum _{k=1}^K\frac{r_k^0}{r_k^k}\ge 1,$$

  (14)

then a subset of transmitters $J$, where $J\subseteq K$, is chosen such that for every $k\in J$, Action k is chosen for a duration of

$${\tau }_k=\frac{d_k-{\tau }_0{r}_k^0}{r_k^k},$$

(15)

and Action 0 is selected for a period of

$${\tau }_0=\underset{i\in K\backslash J}{max}\frac{d_i}{r_i^0},$$

(16)

where for any two sets $K$, $J$ the operation\is the set difference operation defined as

$$\mathrm{K}\backslash \mathrm{J}=\{x:x\in \mathrm{K}\text{and x}\notin \mathrm{J}\text{}}\text{.}$$

(17)

The proof appears in Appendix 3. To completely characterize the policy, we need to specify the set $J$, which results from the following lemma.

Lemma 1: Consider an ordering of the transmitters in decreasing order of the values $d_k∕r_k^0$ for every $k\in K$. Let the corresponding indexing of the transmitters be ${\left\{{\ell }_k\right\}}_{k=1}^K$, that is $d_{{\ell }_1}∕r_{{\ell }_1}^0\ge \cdots \ge d_{{\ell }_K}∕r_{{\ell }_K}^0$. Then, the set $J$ contains those transmitters with the highest ratios $d_k∕r_k^0$ and the cardinality $\left|J\right|$ of the set $J$ is given by

$$\left|J\right|=\underset{k\in \left\{0,\dots ,K\right\}}{argmin}\left\{\left(\underset{i\in K\backslash \left\{0,...,k\right\}}{max}\frac{d_{{\ell }_i}}{r_{{\ell }_i}^0}\right)+\sum _{\left\{j:j\ge 1,j\le k\right\}}\frac{d_{{\ell }_j}}{r_{{\ell }_j}^{{\ell }_j}}-\left(\underset{i\in K\backslash \left\{0,...,k\right\}}{max}\frac{d_{{\ell }_i}}{r_{{\ell }_i}^0}\right)\sum _{\left\{j:j\ge 1j\le k\right\}}\frac{r_{{\ell }_j}^0}{r_{{\ell }_j}^{{\ell }_j}}\right\}.$$

(18)

The proof of the lemma appears in Appendix 4.

From the above, we conclude that the set $J$ contains the transmitters with the highest $\left|J\right|$ values of $d_k∕r_k^0$, where $\left|J\right|$ is given by Lemma 1. Hence, an optimal scheduling and rate control policy individually activates the transmitters that either have a very high initial demand or whose rates under concurrent operation are very low, e.g., due to excessive amounts of interference caused by other concurrent transmissions. Those transmitters must be further assisted toward emptying their queues by being granted individual access to the channel.

IV. Time-varying networks

In the previous section, we focused on time-invariant channels. However, in reality, the wireless channel is time-varying, due to the effects of fading, node mobility, etc. In this section, we extend our model by considering time-varying channels. We make the following assumption on the wireless channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$.

Assumption 1: The channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$ varies according to a stationary Markov Chain with transition probability from some channel state $\text{g}\in G$ to another channel state $g^{\prime }\in G$ given by

$$p_{\text{G}}\left(\text{g},g^{\prime }\right)=Prob\left[\text{G}\left(t+1\right)=g^{\prime }|\text{G}\left(t\right)=\text{g}\right],\forall \text{g},g^{\prime }\in G.$$

(19)

Due to the time variability of the channel process, the length of the schedule T is a random variable and thus "minimum-length" is meant "in the expected sense". This can be formula ted as follows:

$$\text{minimize}:E\left[T\right]$$

(20)

$$\text{subject to}:\text{X}\left(T\right)=0,\text{X}\left(0\right)=\text{d},$$

(21)

$$T\in \mathbb{N},$$

(22)

where the expectation is with respect to the stationary probability distribution of the channel process.

We proceed to present a solution to the problem of (20)-(22) through stochastic control methods by considering admissible policies in the class Π.

A. Stochastic shortest path formulation

Since the wireless channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$ is Markov and since the process of the queue sizes evolves according to (4), for every admissible policy, it is easy to show that the system process ${\left\{S\left(t\right)\right\}}_{t=0}^{\infty }$ is also a Markov Chain, with state space $S$ given by (2). The Markovianness of the system process is shown and verified in Appendix 5. We further define a subset $S_{term}$ of the state space $S$ to be the set of terminating states that correspond to empty queues, i.e.,

$$S_{term}=\left\{\left(\text{x},\text{g}\right):\text{x}=0,\text{g}\in G\right\}.$$

(23)

Evidently, from (4), this Markov Chain is absorbing and from every non-terminating state a terminating state is reached with probability one in finite time under all admissible policies. Furthermore, once the system reaches any state in $S_{term}$, it remains there forever. Hence, the objective becomes to reach a terminating state in minimum expected time by choosing the next state. This will yield a schedule of minimum expected length. This is a stochastic shortest path problem, which is a special case of an MDP. If there is no randomness in the channel state, i.e., the entire wireless channel realization is known at priori at the very first time slot, our results of Sect ion III follow from this model as a special case.

The set of feasible scheduling and rate control actions corresponding to each system state $\text{s}=\left(\text{x},\text{g}\right)\in S$ is the set $A\left(\text{s}\right)\subseteq R\left(\text{g}\right)$. The system is driven by the time-varying channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$. Taking an action leads to different states with different probabilities depending on the evolution of the channel process unless the system has already reached a terminating state.

Let p _r (s, s ') be the transition probability from system state s = (x, g) to state s ' = (x ', g ') by taking action $\text{r}=\pi \left(\text{x},\text{g}\right)\in A\left(\text{s}\right)$. Then we have

$$p_{\text{r}}\left(\text{s},s^{\prime }\right)=Prob\left[\text{X}\left(t+1\right)=x^{\prime },\text{G}\left(t+1\right)=g^{\prime }|\text{X}\left(t\right)=\text{x},\text{G}\left(t\right)=\text{g},\pi \left(\text{x},\text{g}\right)=\text{r}\right].$$

(24)

From (4) and Assumption 1, it is easy to see that p _r (s,s ') can be written as

$$p_{\text{r}}\left(\text{s},s^{\prime }\right)=\left\{\begin{array}{cc}\hfill p_{\text{G}}\left(\text{g},g^{\prime }\right),\hfill & \hfill if{\left(\text{x}-\text{r}\right)}^{+}=x^{\prime },\text{s},s^{\prime }\in S\hfill \\ \hfill 0,\hfill & \hfill otherwise.\hfill \end{array}\right.$$

(25)

Note that from the Markovianess of the channel process and the admissibility of the policy π, the transition probability p _r (s, s ') is time invariant and does not depend on the previous system states.

We define the cost of taking action r and going from state s to state s ' as ${\tilde{c}}_{\text{r}}\left(\text{s},s^{\prime }\right)$. For every system state s, action $\text{r}\in A\left(\text{s}\right)$ and system state s ' such that p _r (s, s ') > 0, we assume that ${\tilde{c}}_{\text{r}}\left(\text{s},s^{\prime }\right)=1$. This represents the fact that in order to go from state s to state s ' by taking this action one needs to spend one time slot. Let us further define the cost per stage c _r (s) to be the expected cost when, being at state $\text{s}\in S\backslash S_{term}$, control action $\text{r}\in A\left(\text{s}\right)$ is chosen. It is clear that $c_{\text{r}}\left(\text{s}\right)={\sum }_{s^{\prime }\in S}{p}_{\text{r}}\left(\text{s},s^{\prime }\right){\tilde{c}}_{\text{r}}\left(\text{s},s^{\prime }\right)=1,\forall \text{s}\in S\backslash S_{term}$. Once a terminal state $\text{s}\in S_{term}$ is reached, no more cost is incurred and the system remains there forever, i.e., c _r (s) = 0, $\forall \text{r}\in A\left(\text{s}\right)$, $\text{s}\in S_{term}$. Having said the above, the minimum-length scheduling problem can be formulated as a stochastic shortest path. Next, we provide an optimal policy that is obtained through Dynamic Programming [19].

B. An optimal policy

Let $T^{\pi }\left(\text{s}\right)$ be the expected time to empty the queues in the network starting from state s under a policy π ∈ Π. Clearly, $T^{\pi }\left(\text{s}\right)=0$ under any policy π ∈ Π for every terminating state $\text{s}\in S_{term}$. Then, the minimum expected schedule length $T^{\star }\left(\text{s}\right)$ is given by

$$T^{\star }\left(\text{s}\right)=\underset{\pi \in \Pi }{min}{T}^{\pi }\left(\text{s}\right),\forall \text{s}\in S\backslash S_{term}.$$

(26)

A policy π' is optimal if it achieves the minimum $T^{\star }\left(\text{s}\right)$ for every non-terminating state $\text{s}\in S\backslash S_{term}$, i.e.,

$${T^{\pi }}^{\star }\left(\text{s}\right)=T^{\star }\left(\text{s}\right),\forall \text{s}\in S\backslash S_{term}.$$

(27)

To solve the above shortest path problem, two commonly used methods are the policy iteration and the value iteration[19]. Due to the large state space of the problem, value iteration is easier to compute and, hence, will be used here. Consider the value iteration algorithm and the corresponding "expected " time $T_k\left(\text{s}\right)$ to empty the queues starting from state s at the k th iteration. Assume that $T_0\left(\text{s}\right)=\infty$ for all states $\text{s}\in S$. We borrow the following properties from [19].

Lemma 2: The value iteration method converges to the optimal cost function, i.e.,

$$T^{\star }\left(\text{s}\right)=\underset{k\to \infty }{lim}{T}_k\left(\text{s}\right),\forall \text{s}\in S\backslash S_{term},$$

(28)

where

$$T_{k+1}\left(\text{s}\right)=1+\underset{\text{r}\in A\left(\text{s}\right)}{min}\sum _{s^{\prime }\in S}{p}_{\text{r}}\left(\text{s},s^{\prime }\right)T_k\left(s^{\prime }\right),\text{s}\in S\backslash S_{term}.$$

(29)

Lemma 29, borrowed from [19], shows the optimality of the value iteration method to solve problems of stochastic control. Therefore, by employing the value iteration method, we can obtain a solution to the minimum-length scheduling problem in time-varying networks.

Lemma 3: The optimal solution to a stochastic shortest path problem must satisfy Bellman's equation, i.e., for every non-terminating state $\text{s}\in S\backslash S_{term}$, it must be true that

$$T^{\star }\left(\text{s}\right)=1+\underset{\text{r}\in A\left(\text{s}\right)}{min}\left\{\sum _{s^{\prime }\in S}{p}_{\text{r}}\left(\text{s},s^{\prime }\right)T^{\star }\left(s^{\prime }\right)\right\}.$$

(30)

Hence, the optimal scheduling and rate control policy π' for every state $\text{s}\in S\backslash S_{term}$ is given by

$${\pi }^{\star }\left(\text{s}\right)=\underset{\text{r}\in A\left(\text{s}\right)}{argmin}\left\{\sum _{s^{\prime }\in S}{p}_{\text{r}}\left(\text{s},s^{\prime }\right)T^{\star }\left(s^{\prime }\right)\right\},\forall \text{s}\in S\backslash S_{term}.$$

(31)

Although the value iteration method optimally solves the aforementioned stochastic shortest path problem, in general it may require an infinite number of iterations until it converges. However, if the Markov Chain of the system evolution is acyclic, then it was shown in [19] that the value iteration method for each state converges in a finite number of iterations (at most as many as the number of non-terminating states of the Markov Chain). It is easy to see that the Markov Chain driving our system is acyclic. This is because starting from one of the states whose queue size satisfies X(0) = d, the queue sizes in the network are non-increasing with time as given in (4) under any admissible policy.

V. Numerical results

In this section, we consider slotted-time, static and time-varying single-hop wireless networks and illustrate our analytical results with a few numerical calculations. We did not find it essential to provide numerical results on the continuous-time reduced problem since for that case we explicitly characterize the corresponding policy. Thus, we only provide numerical results for the value iteration method in the discrete-time case for static and time-varying networks. We consider a network of two transmitter/receiver pairs as in Figure 1 with K = 2. The channel process ${\left\{\text{G}\left(t\right)\right\}}_{t=0}^{\infty }$ is Markov and switches between two states, namely a good state, G, and a bad state, B. When the channel is in the good state, both transmitters have channels of good quality to their receivers; otherwise, both channels are bad. The transition probabilities of this Markov Chain are shown in Figure 4.

Figure 4

A two-state Markovian channel process.

Since we have 2 transmitter/receiver pairs, there exist 3 possible rate vectors corresponding to each channel state, denoted by r^k (g), k = 1, 2, when only the k th transmitter is activated and by r³(g), when both transmitters are activated, under channel state g ∈ {B, G}.

We first assume that the initial demands are d₁ = 4 bits and d₂ = 6 bits, which is the case discussed in Section III. We consider 3 scenarios associated with different achievable rates for each channel state.

• Scenario 1: When the k th transmitter is activated alone, its achievable rate is 3 bits/sec, and when both transmitters are activated simultaneously, the corresponding rate is 2 bits/sec for each. Under this scenario, the channel realization is immaterial and the minimum expected time to empty the queues is 3 sec (slots), i.e., equal to the result of the static network example of Section III.

• Scenario 2: Under the good channel state, G, the achievable rates are equal to the case of Scenario 1, i.e., when the k th transmitter is activated alone, its achievable rate is 3 bits/sec and when both transmitters are activated simultaneously, the corresponding rate is 2 bits/sec for each. However, under the bad channel, B, the achievable rates are strictly worse (2 bits/sec for individual transmission and 1 bit/sec for each transmitter under concurrent transmission). Naturally, we anticipate that the expected time required to empty the queues is more than 3 sec (slots).

• Scenario 3: We assume that under the bad channel state, B, the achievable rates are equal to the ones in Scenario 1, but the good channel is better and thus allows higher rates (4 bits/sec when a transmitter is activated individually and 3 bits/sec when they are both activated simultaneously). Naturally, the expected time to empty the queues will decrease to a value less than 3 sec (slots).

The above 3 scenarios and the corresponding minimum expected schedule lengths are shown in Table 1, where we have assumed that the channel starts from a good channel state. Similar results were observed for higher initial demands (d₁ = d₂ = 100 bits), which are also given in Table 1. Note that the above values of initial demands and transmission rates can be scaled accordingly to give meaningful values for real systems. Note also that although time is measured in terms of time slots, the average minimum expected length takes fractional values since it is computed by taking the expectation with respect to the channel state probability distribution.

Table 1 Expected time required to empty the queues for different values of initial demands, under Scenarios 1-3, assuming that t he channel starts from a good state.

In Figure 5, we illustrate the performance comparison between the optimal policy and a pure TDMA scheme that activates only a single transmitter at a time. Specifically, we consider the same single-hop network of two transmitter/receiver pairs discussed above under Scenario 2. We plot the expected schedule length for the above two schemes as a function of the values of the initial data traffic, where slots are in units of seconds and queue sizes are in bits. For simplicity, the initial queue sizes at each transmitter are assumed to be equal. As expected, we observe that the difference between the expected time to empty the queues under the optimal policy and under the TDMA scheme increases as the initial queue sizes increase. This result illustrates the fact that permitting concurrent transmissions through rate control can provide considerable gains.

Figure 5

Performance comparison of the optimal policy with respect to TDMA scheduling.

VI. Conclusions

In this paper, we focused on the problem of joint scheduling and rate control in single-hop wireless networks under the objective of minimizing the required time to deliver all data traffic to the intended destinations. First, we focused on networks with time-invariant links. We presented a graph-theoretic formulation for the minimum-length scheduling problem as a shortest path on a single-source directed acyclic graph. Motivated by the combinatorial nature of the minimum-length scheduling problem, and we then mapped it to a continuous-time formulation and restricted the set of feasible scheduling and rate control actions. By doing so, we were able to explicitly characterize an optimal policy for the reduced problem that finds a schedule of minimum length. Finally, we considered time-varying wireless networks. We formulated the minimum-length scheduling problem as a stochastic shortest path and presented an optimal policy by employing the principles of stochastic control theory.

Appendix 1: Table 2 table of variables

Here, we give a list with the most commonly appearing variables in the paper to facilitate the reader. The variables are listed in the order they appear in Table 2 below.

Table 2 Table of variables

Appendix 2: Finding a shortest path on a dag

Shortest path problems on single-source DAGs can be solved optimally in polynomial time [11]. Below, we provide an optimal algorithm, taken from [11], that finds a shortest path on a DAG.

In order to compute a shortest path, we first need to sort the DAG in topological order and then use a sequence of edge relaxations until we obtain a shortest path from the source x₀ to the vertex corresponding to the 0 vector. Topological order is a linear ordering of all the vertices of the DAG so that for every edge (x _i , x _j ), the vertex x _i appears before x _j in the ordering. The process of edge relaxation verifies whether the current best-known path from the source x₀ to a vertex y can be improved by passing through a different vertex x.

We proceed with a few definitions that will be useful in the rest of this appendix. We define the distance of a vertex x to be the minimum distance in terms of edges that must be traversed from the source to reach x. We also denote by δ[x] an upper bound on the distance of vertex x. For every edge (x, y), we say that x is the predecessor of y and we write x = Pred[y]. We denote by Adj[x] a list that contains all the vertices y that are adjacent to x, i.e., such that there exists an edge (x, y) ∈ E. The pseudo-code of the algorithm is provided below:

DAG-SHORTEST-PATHS$\left(Ḡ,{\text{x}}_0\right)$

• 1    topologically sort the vertices of $\overline{G}$

• 2    INITIALIZE-SINGLE-SOURCE$\left(Ḡ,{\text{x}}_0\right)$

• 3    for each vertex x taken in topologically sorted order

• 4       do for each vertex y ∈ Adj[x]

• 5          do RELAX(x,y)

The topological sorting of the first line of the algorithm can be completed in Θ(|V| + |E|) time, by running a Depth-First Search (DFS) [11]. The second line of the algorithm involves the initialization of various variables as shown next:

INITIALIZE-SINGLE-SOURCE$\left(Ḡ,{\text{x}}_0\right)$

• 1    for each vertex x ∈ V

• 2       do δ[x] ← ∞

• 3          Pred[x] ← NIL

• 4    δ[x ₀] ← 0

This process requires time order Θ(|V|). Finally, in lines 3-5 of the DAG-SHORTEST PATHS$\left(Ḡ,{\text{x}}_0\right)$ algorithm, at each time step, the next vertex in the topological order is selected and a sequence of relaxations over all edges that are incident from this vertex is performed. The procedure RELAX(x,y), given next, verifies whether the current shortest path from x₀ to y can be improved by passing through x.

RELAX(x,y)

• 1    if δ[y] > δ[x] + 1

• 2       then δ[y] ← δ[x]+1

• 3          Pred[y]←x

Appendix 3: Proof of Theorem 1

We can write the Lagrangian of the problem described by (8)-(10) as:

$$L\left(\tau ,\mu ,\lambda \right)=-\sum _{i=0}^K{\tau }_i+\sum _{k=1}^K{\mu }_k\left({\tau }_k{r}_k^k+{\tau }_0{r}_k^0-d_k\right)+\sum _{i=0}^K{\lambda }_i{\tau }_i,$$

where μ and λ represent the Lagrange multipliers. The Karush-Kuhn-Tucker (KKT) conditions yield:

• 1) For every Action $k\in K$ we have

  $$\frac{\partial L\left(\tau ,\mu ,\lambda \right)}{\partial {\tau }_k}=-1+{\mu }_k{r}_k^k+{\lambda }_k=0.$$

  (32)

• 2) For Action 0 we have

  $$\frac{\partial L\left(\tau ,\mu ,\lambda \right)}{\partial {\tau }_0}=-1+\sum _{k=1}^K{\mu }_k{r}_k^0+{\lambda }_0=0.$$

  (33)

• 3) For every Action $k\in K$ it must be true that

  $$\begin{array}{c}{\mu }_k\left({\tau }_k{r}_k^k+{\tau }_0{r}_k^0-d_k\right)=0\Rightarrow \\ {\mu }_k\ge 0,{\tau }_k{r}_k^k+{\tau }_0{r}_k^0\ge d_k.\end{array}$$

  (34)

• 4) For all actions i ∈ {0, ...,K} we have

  $${\lambda }_i{\tau }_i=0\Rightarrow {\lambda }_i\ge 0,{\tau }_i\ge 0.$$

  (35)

Consider the following cases:

Case 1: Assume that Action 0 is never employed, i.e., τ₀ = 0. Since the traffic demands of every transmitter must be met, we have that τ _k > 0 for every $k\in K$. Hence, from (35), it follows that λ₀ ≥ 0 and λ _k = 0 for every$k\in K$. From (32), we obtain,

$${\mu }_k=\frac{1}{r_k^k}.$$

(36)

Furthermore, since μ _k > 0 and τ₀ = 0, (34) yields

$$d_k={\tau }_k{r}_k^k,$$

i.e., for every $k\in K$ we obtain

$${\tau }_k=\frac{d_k}{r_k^k}.$$

(37)

Finally, from (33) and (36) it follows that

$$\sum _{k=1}^K\frac{r_k^0}{r_k^k}\le 1.$$

(38)

Case 2: Assume that Action 0 is employed for a non-negative amount of time and that also a subset $J$ of the transmitters is further selected to transmit individually. This implies that τ₀ > 0, τ _j > 0 for every $j\in J$and τ _i = 0 for every $i\in K\backslash J$. Hence, (35) yields λ₀ = 0, λ _j = 0 for every $j\in J$ and λ _i ≥ 0 for every $i\in K\backslash J$. Also, for every $j\in J$, (32) yields

$${\mu }_j=\frac{1}{r_j^j},$$

(39)

and for every $i\in K\backslash J$, it follows that

$${\mu }_i\le \frac{1}{r_i^i}.$$

(40)

Moreover, from (34) and (39), for every $j\in J$, we obtain

$${\tau }_j=\frac{d_j-{\tau }_0{r}_j^0}{r_j^j},$$

(41)

and from (34) and (40), for every $i\in K\backslash J$, we have

$$d_i\le {\tau }_0{r}_i^0,$$

(42)

or equivalently

$${\tau }_0\ge \underset{i\in K\backslash J}{max}\frac{d_i}{r_i^0}.$$

(43)

Finally, from (33), (39), and (40), it follows that

$$\sum _{k=1}^K\frac{r_k^0}{r_k^k}\ge 1.$$

(44)

■

Appendix 4: Proof of Lemma 1

From (41), (42), and the fact that τ _j > 0 for every $j\in J$, for $i\in K\backslash J$, it follows that

$$0<{\tau }_j\le \frac{d_j{r}_i^0-r_j^0{d}_i}{r_j^j{r}_i^0},$$

which yields that

$$\frac{d_i}{r_i^0}<\frac{d_j}{r_j^0}.$$

(45)

Consider now a new ordering ${\left\{{\ell }_k\right\}}_{k=1}^K$ of the transmitters in decreasing order of the values $d_k∕r_k^0$ for every $k\in K$, i.e.,$d_{{\ell }_1}∕r_{{\ell }_1}^0\ge \cdots \ge d_{{\ell }_K}∕r_{{\ell }_K}^0$. Hence, from (45), it follows that there exists a threshold, i.e., a transmitter index value in the new ordering, below which all the transmitters must belong to the set $J$ and above which (that is, the remaining ones), they must belong to the set $K\backslash J$.

Since the objective is to minimize ${\tau }_0+{\sum }_{j\in J}{\tau }_j$, from (41) and (43), it follows that

$$\begin{array}{lll}\hfill \left|J\right|& =\underset{k\in \left\{0,\dots ,K\right\}}{argmin}\left\{{\tau }_0+\sum _{\left\{j:j\ge 1,j\le k\right\}}{\tau }_{{\ell }_j}\right\}& \hfill \text{(1)}\\ =\underset{k\in \left\{0,\dots ,K\right\}}{argmin}\left\{{\tau }_0+\sum _{\left\{j:j\ge 1,\le k\right\}}\frac{d_{{\ell }_j}-{\tau }_0{r}_{{\ell }_j}^0}{r_{{\ell }_j}^{{\ell }_j}}\right\}& \hfill \text{(2)}\\ =\underset{k\in \left\{0,\dots ,K\right\}}{argmin}\left\{\left(\underset{i\in K\backslash \left\{0,\dots ,K\right\}}{max}\frac{d_{{\ell }_i}}{r_{{\ell }_i}^0}\right)+\sum _{\left\{j:j\ge 1,j\le k\right\}}\frac{d_{{\ell }_j}}{r_{{\ell }_j}^{{\ell }_j}}-\left(\underset{i\in K\backslash \left\{0,\dots ,K\right\}}{max}\frac{d_{{\ell }_i}}{r_{{\ell }_i}^0}\right)\sum _{\left\{j:j\ge 1,j\le k\right\}}\frac{r_{{\ell }_j}^0}{r_{{\ell }_j}^{{\ell }_j}}\right\}.& \hfill \text{(3)}\\ \hfill \text{(4) }\end{array}$$

■

Appendix 5: Markovian property of the system process

To show that the process ${\left\{\text{S}\left(t\right)\right\}}_{t=0}^{\infty }$ is Markov, we need to prove that for any values that the system state can take s₀ = (x₀, g₀), ..., s_t+1 = (x_t+1, g_t+1) it is true that

$$Prob\left[\text{S}\left(t+1\right)={\text{s}}_{t+1}|\text{S}\left(t\right)={\text{s}}_t,\text{S}\left(t-1\right)={\text{s}}_{t-1},\dots ,\text{S}\left(0\right)={\text{s}}_0\right]=Prob\left[\text{S}\left(t+1\right)={\text{s}}_{t+1}|\text{S}\left(t\right)={\text{s}}_t\right].$$

(46)

Then, we have the following:

$$\begin{array}{l}\text{Prob[}\text{X}(t+1)={\text{x}}_{t+1}\mathrm{,}\text{G}(t+1)={\text{g}}_{t+1}|\text{X}(t)={\text{x}}_t\mathrm{,}\text{G}(t)={\text{g}}_t,\\ \text{X}(t-1)={\text{x}}_{t-1}\mathrm{,}\text{G}(t-1)={\text{g}}_{t-1}\mathrm{,}\dots \mathrm{,}\text{X}(0)={\text{x}}_0\mathrm{,}\text{G}(0)={\text{g}}_0]=\\ \text{Prob[}\text{X}(t)-\pi (\text{X}(t),\text{G}(t))={\text{x}}_{t+1}\mathrm{,}\text{G}(t+1)={\text{g}}_{t+1}|\text{X}(t)={\text{x}}_t\mathrm{,}\text{G}(t)={\text{g}}_t,\\ \text{X}(t-1)={\text{x}}_{t-1}\mathrm{,}\text{G}(t-1)={\text{g}}_{t-1}\mathrm{,}\dots \mathrm{,}\text{X}(0)={\text{x}}_0\mathrm{,}\text{G}(0)={\text{g}}_0]=\end{array}$$

(47)

$$\begin{array}{l}\text{Prob}[{\text{x}}_t-\pi ({\text{x}}_t\mathrm{,}{\text{g}}_t)={\text{x}}_{t+1}\mathrm{,}\text{G}(t+1)={\text{g}}_{t+1}|\text{X}(t)={\text{x}}_t\mathrm{,}\text{G}(t)={\text{g}}_t,\\ \text{X}(t-1)={\text{x}}_{t-1}\mathrm{,}\text{G}(t-1)={\text{g}}_{t-1}\mathrm{,}\dots \mathrm{,}\text{X}(0)={\text{x}}_0\mathrm{,}\text{G}(0)={\text{g}}_0]=\\ \text{Prob}[{\text{x}}_t-\pi ({\text{x}}_t\mathrm{,}{\text{g}}_t)={\text{x}}_{t+1}\mathrm{,}\text{G}(t+1)={\text{g}}_{t+1}|\text{X}(t)={\text{x}}_t\mathrm{,}\text{G}(t)={\text{g}}_t]=\\ \text{Prob}[\text{X}(t+1)={\text{x}}_{t+1}\mathrm{,}\text{G}(t+1)={\text{g}}_{t+1}|\text{X}(t)={\text{x}}_t\mathrm{,}\text{G}(t)={\text{g}}_t]\end{array}$$

(48)

where (47) follows from (3) and (4) and where (48) follows from Assumption 1 on the markovian property of the channel process. This completes the proof.

■