Adaptive access and rate control of CSMA for energy, rate, and delay optimization

Abstract

In this article, we present a cross-layer adaptive algorithm that dynamically maximizes the average utility function. A per stage utility function is defined for each link of a carrier sense multiple access-based wireless network as a weighted concave function of energy consumption, smoothed rate, and smoothed queue size. Hence, by selecting weights we can control the trade-off among them. Using dynamic programming, the utility function is maximized by dynamically adapting channel access, modulation, and coding according to the queue size and quality of the time-varying channel. We show that the optimal transmission policy has a threshold structure versus the channel state where the optimal decision is to transmit when the wireless channel state is better than a threshold. We also provide a queue management scheme where arrival rate is controlled based on the link state. Numerical results show characteristics of the proposed adaptation scheme and highlight the trade-off among energy consumption, smoothed data rate, and link delay.

1. Introduction

In wireless networks, mobile devices are usually battery powered with a limited amount of energy. Therefore, minimization of energy consumption while maintaining the quality of service in the network is crucial. This must be accomplished by adapting the transmission parameters to the system dynamics and to the time-varying channel of the links. In this article, we present a cross-layer adaptive algorithm that dynamically maximizes the average utility function of a carrier sense multiple access (CSMA)-based wireless link.

Benefits of such adaptation schemes are shown in some prior works in terms of energy efficiency [1–8]. In such works various control algorithms have been proposed that trade-off among different goals such as energy consumption, average delay, packet dropping probability and bit error rate, and dynamically adapt the transmission parameters to the channel and system state. The aforementioned works assume point-to-point links with dedicated channels. However, in data transmission networks, where data are generated at random time instances, random access schemes are used to efficiently exploit channel resources. In such systems, there are more users than available channels, and at any given time only a subset of users can access the channels. Therefore, the optimality of channel access decision is crucial in random access networks. Random access is widely used in ad hoc networks as it can be implemented in a distributed manner. Wireless local area networks (WLAN) and practical personal or sensor networks usually use random access control in their ad hoc operation mode [9, 10]. On the other hand, it is shown recently that CSMA protocols can achieve maximum stable throughput [11] while keeping bounded queuing delay [12], and it can achieve a collision free WLAN [13].

Optimization of random access networks was first proposed in order to achieve single hop proportional fairness for slotted ALOHA networks [14]. Different types of fairness are also considered and random access control is modeled as a utility maximization problem in [15]. In addition, the cross-layer optimization problem of random access control and transmission control protocol is solved as a network utility maximization problem [16]. Newton-like algorithms are also provided for energy and throughput optimization with end-to-end delay constraint in multi hop random access network [17]. However, in the aforementioned articles static transmission probability was used and opportunity of time varying and adaptive control was ignored.

On the other hand, queue-based random access algorithms were studied in [18], where access probabilities are assumed to be adapted based on queue sizes. Stability of the proposed algorithms was verified and their delay performance was shown to surpass fixed optimization algorithms. Also a heuristic differential queue-based scheduling algorithm is proposed in [19] which shows superior performance compared to 802.11 through experimental results. However, such queue-based algorithms are inappropriate for fading channels and prioritize links with low channel quality, which results in low energy efficiency [20].

In this article, we propose cross-layer adaptive algorithms; derived from dynamic programming, for distributed optimization of the links in CSMA-based wireless networks operating in mobile environments. As a performance metric, we define the per stage utility of the link as a weighted concave function of energy consumption, smoothed data rate, and smoothed queue size in the link, where the weights are assigned based on the desired tradeoff among them. The algorithms maximize the average utility by dynamically adapting the channel access decision and transmit data rate (by selecting different modulation and coding schemes) according to the queue size of the link and the availability and quality of the time-varying channel (channel state is assumed to be known at the transmitter). Both, finite-time horizon (FTH) and infinite-time horizon (ITH) problems are considered. In the first case, the utility sum is maximized for a finite time period, whereas in the second case, the long-term average utility is maximized.

We consider a mobile environment with frequency-flat time-varying channel response. This requires suitable models of the wireless channel dynamics. Here, we use finite-state Markov chains (FSMC) to model channel dynamics, such that channel time-correlation at network links is partially exploited by the proposed algorithms. Although the physical wireless channel is inherently non-Markovian, it has been shown that stationary Markov chains can capture the essence of the channel dynamics [21]. Many transmission adaptation algorithms are based on first-order Markov channel models [1, 2]. Here, we consider first- and second-order Markov chains to model characteristics of network links.

The numerical simulations show the benefits of the proposed adaptation algorithms in terms of energy efficiency, and highlight the trade-off among energy consumption, smoothed data rate, and delay in links of a CSMA network. They also show that the use of suitable Markov model for the wireless channel improves performance of the adaptation algorithm, mainly for slow fading channels. Algorithms based on uncorrelated, first- and second-order Markov models are considered and their performance is compared through simulations.

The rest of the article is organized as follows. Section 2 presents the system model and in particular it describes the model of the network links as well as wireless channel models. In Section 3, per stage utility of the links is defined. Consequently, the utility sum maximization for a finite time period is formulated as an optimal finite-horizon control problem. Similarly, the long-term average utility maximization is formulated as an optimal infinite-horizon control problem. Section 4 uses dynamic programming to compute the optimal adaptation policies for the problems formulated in Section 3. We have investigated structural properties of the optimal solution in Section 5. Numerical results and comparisons are described in Section 6. Finally, Section 7 concludes the article.

2. System model

In this section, we describe the model of the random access links as well as wireless channel models.

2.1. Link model

We consider an ad hoc network where links use CSMA protocol similar to the one provided in [22] which prevents collision among links and also resolves hidden and exposed node problems which exist in wireless networks [23]. As shown in Figure 1, we assume a slotted transmission model where each timeslot, of duration T_s, contains both a data slot and a number of control mini slots. When the link has a packet to transmit, it should wait for a random value of W control mini-slots, and if no other link has reserved the channel earlier, it will send a short request to send packet to reserve the channel. Then, the potential receiver which also perceives that the channel is idle will response with a clear to send (CTS) packet that allows the transmitter to transmit and informs possible interfering nodes that the channel will be used. Once the transmitter receives the CTS, it sends its packet in the data slot.

Figure 1

System and timing diagrams.

Timeslot k is defined as the time interval [(k - 1)T_s, kT_s). We use I _k to denote the channel access, where I _k = 1 indicates that the link has decided to access the channel at the k th timeslot. The control policy adapts I _k in each slot based on the system and channel state. Also B _k = 1 indicates that the link should delay its transmission because the channel is already occupied by another link. We model B _k as a Bernoulli process where P _B Pr{B _k = 1} is the channel occupancy probability. The Bernoulli distribution is widely used to model the statistics of B _k in CSMA networks [24].

The link has a queue of maximum size L. Let q _k denote the number of packets in the queue at the k th timeslot, which is assumed to be known at the transmitter. Obviously, I _k = 0 when q _k = 0. r _k denotes the controlled number of packets that arrive the queue in slot k, which we will call arrival rate hereafter. The value of r _k should be chosen both to provide suitable rate for source data and to prevent delay due to backlog through adapting source rate to the link state [25]. To avoid buffer overflow the arrival rate is constrained by r _k ≤ (L - q _k ). The queue update equation is

$$q_{k+1}=q_k-\min \left\{q_k,C_k\right\}{I}_k\left(1-B_k\right)+r_k$$

(1)

Where C _k indicates the maximum number of packets that can be transmitted during the k th data slot. C _k depends on the channel state, and it is assumed to be known at the transmitter at the beginning of each timeslot. We call the data that the physical layer transmits in one time slot a frame and the link consumes a constant energy e for transmission of frame in the data slot. Thus, the energy consumed in the k th timeslot will be E _k = eI _k (1 - B _k ).

We also consider the exponentially weighted moving average (EWMA) of the queue occupancy ${\bar{q}}_k$ and of the arrival rate ${\bar{r}}_k$ as the link state variables which are defined as follows

$${\bar{q}}_{k+1}={\theta }_q{\bar{q}}_k+\left(1-{\theta }_q\right)q_{k+1};{\theta }_q\in \left(0,1\right)$$

(2)

$${\bar{r}}_{k+1}={\theta }_r{\bar{r}}_k+\left(1-{\theta }_r\right)r_k;{\theta }_r\in \left(0,1\right)$$

(3)

Note that ${\bar{q}}_k$ and ${\bar{r}}_k$ can be viewed as "smoothed" measures of the delay and data rate in the link. The parameters θ _q and θ _r determines the time scale over which the smoothing is performed. The smaller the value of θ _r or θ _q , the shorter the time period of moving average (smoothing). Values of θ _r and θ _q are determined based on the tolerance of the applications to the delay and data rate variations in the link. Random early detection protocol has used the EWMA of the delay (${\bar{q}}_k$) as a criterion for congestion control [26]. In addition, the EWMA of the rate (or smoothed rate), ${\bar{r}}_k$, has been used in [27, 28] as a measure of the quality of service. EWMA is also used as a metric in statistical quality control [29].

2.2. Channel model

We consider a frequency-flat block-fading channel, where the channel remains constant during each timeslot, and can change for consecutive timeslots. Therefore, we assume that the duration of each timeslot (T _s ) is less than the coherence time of the channel. Hence, channel responses at different timeslots can be correlated. The channel power gain at the k th timeslot is denoted by γ _k . Since we assume constant transmit power, the received signal-to-noise ratio (SNR) in the link for the k th timeslot will be proportional to γ _k . The fading range 0 ≤ γ is partitioned into M disjoint regions so that the j th region is defined as R _j = {γ : A _j ≤ γ < A_j+1}, where A₁ = 0 and A_M+1= ∞. The channel for the k th timeslot is in state j if γ _k ∈ R _j . Also the values of A _j are selected according to the adaptive modulation and coding as follows. Consider that transmitter has a set of modulation and coding schemes {Q₁, Q₂, ... , Q _M } to select from in each time slot. We select A _j ; j = 2, ... , M such that if channel is in state j, transmitter can use Q _j and ensures that the frames transmitted with this scheme have error probability less than FER_th which is a target threshold for frame error rate (FER).

Let $\mathcal{C}=\left\{{Ĉ}_j|j=1,...,M\right\}$ denote the set of number of transmit packets associated with the set of channel states, if γ _k ∈ R _j then $C_k={Ĉ}_j$ where C _k is the number of packets that can be transmitted in the k th timeslot. Note that packet error rate will be below the same threshold, i.e., PER_th = FER_th, since (a) adaptive algorithm applies different Q _i schemes so that transmitter ensures the same error threshold for all frames, and (b) if a frame transmission was unsuccessful all packets in the frame will be lost. Therefore, the ratio of the lost packets to the total number of packets equals the ratio of the erroneous frames to the total number of frames, regardless of the channel state.

$$\mathsf{\text{PE}}{\mathsf{\text{R}}}_{\mathsf{\text{th}}}=\sum _{C_j=1}^M\left(\mathsf{\text{Prob}}\left\{\mathsf{\text{packet}}\mathsf{\text{in a frame of size}}{Ĉ}_j\right\}\times \mathsf{\text{FER}}\left({Ĉ}_j\right)\right)=\mathsf{\text{FE}}{\mathsf{\text{R}}}_{\mathsf{\text{th}}}$$

Subsequently, we consider three models for the random process C _k , with diverse degrees of complexity.

1. Uncorrelated model

In this model, the channel response at different timeslots are assumed uncorrelated so, where P _j is the probability of the channel state R _j . This simple model may be accurate for fading channels that exhibit high time-variability. It is also the fitting model when there is no prior information about the channel time correlation.

2. First-order markov model

To model the time correlation of the channel we use an M- state FSMC [30] with time discretized to T_s and transition probabilities as

Accordingly, the random process C _k will be modeled with the same M- state FSMC so:

$$\mathrm{Pr}\left\{C_{k+1}={Ĉ}_j|C_k={Ĉ}_i\right\}=P_{i,j}$$

(4)

The transition probabilities depend on the normalized Doppler frequency f_dT_s which determines the rate of variation of the channel with respect to the timeslot duration, where f_d is the channel Doppler frequency. Although the physical wireless channel is inherently non-Markovian, it has been shown that an FSMC can capture the essence of the channel dynamics when the number of regions/states (M) is low and the channel fades slow enough (see for example [21] and references therein). Note that the uncorrelated model can be viewed as a particular case of FSMC where P _i,j = P _j , ∀i.

3. Second-order Markov model

In order to model dynamics of C _k more accurately, we also consider second-order FSMC channel models. They are more accurate than the first-order FSMC since C_k+1depends on both C _k and C_k-1.

$$P_{l,i,j}=\mathrm{Pr}\left\{C_{k+1}={Ĉ}_j|C_{k-1}={Ĉ}_l,C_k={Ĉ}_i\right\}$$

(5)

In this article, we use the so-called Cartesian product method [21] for the second-order models. We will investigate the effect of the FSMC order on the performance of the resulting algorithm through numerical results. Note that the formulation of the first-order Markov model can be considered as a special case of the second-order model with P _i,j = P _l,i,j for any l.

3. Problem formulation

We consider a wireless link in a CSMA network which desires to optimize its transmission rate, energy consumption, and delay. We distinguish two dynamic optimization problems: FTH and ITH problems. In the FTH problem, the performance of the link is optimized over a finite number of timeslots, whereas in the ITH problem the link performance is optimized considering an infinite number of timeslots. Next, they are formulated as dynamic programming problems.

3.1. Finite time horizon

We define a utility maximization problem over N timeslots or stages as follows:

$${\max }_{\left\{u_k\right\}}\underset{k=1,\mathrm{...}N}{E_{\left\{C_k\right\}}}[g_{N+1}(s_{N+1})+{\displaystyle {\sum }_{k=1}^{N}g(s_k,}{\mu }_k)],$$

(6)

where the expectation is taken over the random process C _k . The function g(s _k , μ _k ) is the utility per stage and is a measure of the quality of service of the link at each timeslot. It depends on the action vector μ _k = (I _k , r _k ) and on the system state vector. We consider a second-order Markov model for C _k and include component C_k-1in the state vector $s_k=\left(q_k,{\bar{q}}_k,{\bar{r}}_k,C_k,C_{k-1}\right)$. Note that the first-order model can be considered as a special case with $s_k=\left(q_k,{\bar{q}}_k,{\bar{r}}_k,C_k\right)$. Considering φ(·) as the state update function we can write: s_k+1= φ (s _k , μ _k , C_k+1, B _k ). In (6) g_N+1is the final stage utility which depends only on the final state of the system, s_N+1and it can include some limitations or penalties on the final state of the system.

Here we consider a special format for utility per stage function in order to clarify how it controls system performance:

$$\begin{array}{c}g\left(s_k,{\mu }_k\right)=E_{B_k}\left[\mathsf{\text{U}}\left({\bar{r}}_k\right)-\alpha \mathsf{\text{V}}\left({\bar{q}}_k\right)-\beta E_k\right]=\mathsf{\text{U}}\left({\bar{r}}_k\right)-\alpha \mathsf{\text{V}}\left({\bar{q}}_k\right)-\\ \beta e{I}_k\left(1-P_B\right),k=1,...,N\end{array}$$

(7)

where U(·), and -V(·) are suitable continuous, concave functions, and parameters α and β control the tradeoff between rate, energy, and delay in the utility function. A similar formulation for per stage utility is used in [27, 28] for multi-period utility maximization while queue management and thus queue sizes were not considered.

The number of packets remaining in the queue at the final stage can be penalized with a price of η as follows:

$$g_{N+1}\left(s_{N+1}\right)=U\left({\bar{r}}_{N+1}\right)-\alpha \mathsf{\text{V}}\left({\bar{q}}_{N+1}\right)-\eta q_{N+1}$$

(8)

3.2. Infinite time horizon

In this case we maximize the average utility per stage which is defined by

$${\max }_{\left\{{\mu }_k\right\}}{\lim }_{N\to \infty }{E}_{\left\{C_k\right\}}\left[\frac{1}{N}{\sum }_{k=1}^{N}g\left(s_k,{\mu }_k\right)\right]$$

(9)

where the action and state vectors as well as the per stage utility function are defined similar to the FTH problem. We consider both the first- and second-order models for the channel state by applying appropriate format of s _k .

4. Optimal adaptive control

To maximize FTH or ITH utility functions the controller should decide optimal actions ${\mu }_k^{*}\left(s_k\right)$ at the beginning of each timeslot as a function of the system state s _k . Note that the decision must be causal since future system states are unknown due to the randomness of the channel state (C _k ) and occupancy (B _k ). In this section, by using the DP algorithm [31], we derive algorithms that compute the optimal control functions for the FTH and ITH problems. It is important to remark that the resulting optimal control functions are computed and stored offline. Then, they will be used online to dynamically adapt the actions to the system state. As described earlier, the system state definition can support uncorrelated, first- and second-order channel models so we do not limit the solution to any specific channel model.

4.1. Per stage adaptation to maximize FTH utility

The optimal control policy is the sequence of control functions (one for each timeslot) ${\pi }^{*}=\left\{{\mu }_1^{*}\left(s_1\right),{\mu }_2^{*}\left(s_2\right),...,{\mu }_N^{*}\left(s_N\right)\right\}$that maximize (6). Note that the control functions provide the optimal action for each of the possible system states at different stages. Using the DP algorithm, the optimal policy π* is obtained from the following backward recursion for k = N, N - 1, ..., 1:

$$J_{\left(N+1\right)}\left(s_{N+1}\right)=g_{N+1}\left(s_{N+1}\right)$$

(10)

$$J_k\left(s_k\right)=\underset{{\mu }_k}{\max }\left\{g\left(s_k,{\mu }_k\right)+E_{C_{k+1},B_k}\left[J_{k+1}\left(s_{k+1}\right)\right]\right\}$$

(11)

$${\mu }_k^{*}\left(s_k\right)=\underset{{\mu }_k}{\arg \max }\left\{g\left(s_k,{\mu }_k\right)+E_{C_{k+1},B_k}\left[J_{k+1}\left(s_{k+1}\right)\right]\right\}$$

(12)

The function J _k (s _k ) is the maximum expected accumulative utility, achieved under optimal decision, when the system is in state s _k at the k th stage. Thus, J₁ (s₁) is the expected total utility for N stages when the initial state is s₁.

The application of the DP algorithm requires computation of function J _k (s _k ) for all possible system states (s _k ) at each stage and necessitates the system state space to be finite. Since the state components ${\bar{r}}_k$, and ${\bar{q}}_k$ can take values from continuous spaces, we discretize them using finite grids $\left\{{\bar{q}}_m^d|m=1,2,...M_q\right\}$, and $\left\{{\bar{r}}_m^d|m=1,2,...,M_r\right\}$. Then, we can express each non-grid value as a linear interpolation of the nearby grid values:

$$\bar{q}={\sum }_{m=1}^{M_q}{w}_q^m\left(\bar{q}\right){\bar{q}}_m^d$$

(13)

$$\bar{r}={\sum }_{m=1}^{M_r}{w}_r^m\left(\bar{r}\right){\bar{r}}_m^d$$

(14)

where $w_q^m$ and $w_r^m$ are non-negative weights and ${\sum }_{m=1}^{M_q}{w}_q^m\left(\bar{q}\right)={\sum }_{m=1}^{M_r}{w}_r^m\left(\bar{r}\right)=1$. It can be shown that if Lipschitz condition holds for the functions $w_q^m\left(\bar{q}\right)$, $w_r^m\left(\bar{r}\right)$, and g(s _k , μ _k ), and for the state update functions (1-3), the DP solution of the discretized problem converges to the optimal policy for the original continuous problem, as the density of the grid increases [32]. For the problem in hand the utility and the state update functions are continuous and thus satisfy the Lipschitz condition. We select $w_q^m$ and $w_r^m$ to be suitable continuous functions of the state variables which are chosen on the basis of geometric considerations as suggested in [31] and each state will be described by two nearby discrete states:

$$w_x^m\left(\bar{x}\right)=\left\{\begin{array}{c}\begin{array}{cc}\hfill \left(\bar{x}-{\bar{x}}_{m+1}^d\right)/\left({\bar{x}}_m^d-{\bar{x}}_{m-1}^d\right),\hfill & \hfill {\bar{x}}_{m-1}^d<\bar{x}<{\bar{x}}_m^d\hfill \\ \hfill \left({\bar{x}}_{m+1}^d-\bar{x}\right)/\left({\bar{x}}_{m+1}^d-{\bar{x}}_m^d\right),\hfill & \hfill {\bar{x}}_m^d<\bar{x}<{\bar{x}}_{m+1}^d\hfill \end{array}\\ 0,\mathsf{\text{otherwise}}\end{array}\right.$$

(15)

The DP algorithm for the discretized state space $s_k^d=\left(q_k,{\bar{q}}_k^d,{\bar{r}}_k^d,C_k,C_{k-1}\right)$ and k = N, N - 1, ... , 1 will be

$$J_{\left(N+1\right)}\left(s_{N+1}^d\right)=g_{N+1}\left(s_{N+1}^d\right)$$

(16)

$$J_k\left(s_k^d\right)=\underset{{\mu }_k}{\max }\left\{g\left(s_k,{\mu }_k\right)+E_{C_{k+1},B_k}\left[{\tilde{J}}_{k+1}\left(s_{k+1}\right)\right]\right\}$$

(17)

$${\tilde{\mu }}_k^{*}\left(s_k^d\right)=\arg {\max }_{{\mu }_k}\left\{g\left(s_k,{\mu }_k\right)+E_{C_{k+1},B_k}\left[{\tilde{J}}_{k+1}\left(s_{k+1}\right)\right]\right\}$$

(18)

where ${\tilde{J}}_{k+1}\left(s_{k+1}\right)$ is the estimation of $J_{k+1}\left(s_{k+1}\right)$ by its values at discretized states and is given by:

$${\tilde{J}}_{k+1}\left(s_{k+1}\right)={\sum }_{m=1}^{M_q}{\sum }_{l=1}^{M_r}{w}_q^m\left({\bar{q}}_{k+1}\right)w_r^l\left({\bar{r}}_{k+1}\right)J_{k+1}\left(s_{k+1}^d\right)$$

(19)

In the above equation, we have $s_{k+1}^d=\left(q_{k+1},{\bar{q}}_m^d,{\bar{r}}_l^d,C_{k+1},C_k\right)$ and $s_{k+1}=\left(q_{k+1},{\bar{q}}_{k+1},{\bar{r}}_{k+1},C_{k+1},C_k\right)$. Furthermore q_k+1, ${\bar{q}}_{k+1}$ and ${\bar{r}}_{k+1}$ are given by (1), (2), and (3), respectively. We consider the second-order Markov model by using both C_k-1and C _k in the state vector. The other channel models can be considered as its special case. The solution provided in Equations (17)-(19) is valid for any concave and continuous function of g.

Next we replace g(s _k , μ _k ) in (17) and (18) with its format provided in (7) and calculate the expectation, $E_{C_{k+1},B_k}\left[\cdot \right]$, using the channel transition probabilities, $\mathrm{Pr}\left\{C_{k+1}={Ĉ}_j|C_k,C_{k-1}\right\}$ and channel occupancy probability, Pr{B_k = 1}. The expected accumulative utility and the optimal control functions for k = N, N - 1, ... , 1 will be

$$\begin{array}{c}{J}_k\left(s_k^d\right)=\mathsf{\text{U}}\left({\bar{r}}_k^d\right)-\alpha \mathsf{\text{V}}\left({\bar{q}}_k^d\right)+\underset{{\mu }_k}{\max }\{-\beta e{I}_k\left(1-P_B\right)+{\sum }_{j=1}^M{\sum }_{i=0}^1\mathrm{Pr}\left\{{\mathsf{\text{B}}}_{\mathsf{\text{k}}}=i\right\}\mathrm{Pr}\{C_{k+1}=\\ {Ĉ}_j|C_k,C_{k-1}\}{\tilde{J}}_{k+1}\left(s_{k+1}\right)\}\end{array}$$

(20)

$$\begin{array}{c}{\bar{\mu }}_k^{*}\left(s_k^d\right)=\\ \arg {\max }_{{\mu }_k}\left\{-\beta e{I}_k\left(1-P_B\right)+{\sum }_{j=1}^M{\sum }_{i=0}^1\mathrm{Pr}\left\{{\mathsf{\text{B}}}_{\mathsf{\text{k}}}=\mathsf{\text{i}}\right\}\mathrm{Pr}\left\{C_{k+1}={Ĉ}_j|C_k,C_{k-1}\right\}{\tilde{J}}_{k+1}\left(s_{k+1}\right)\right\}\end{array}$$

(21)

Since ${\bar{r}}_k^d$ and ${\bar{q}}_k^d$ are independent of the decision in the k th timeslot, $\mathsf{\text{U}}\left({\bar{r}}_k^d\right)-\alpha \mathsf{\text{V}}\left({\bar{q}}_k^d\right)$ do not affect the maximization in (21). Also the summations in (20) and (21) are over all M channel states and two possible channel occupancy conditions.

The discrete DP algorithm can be executed offline and the resulting optimal policy can be stored in a look-up table available at the transmitter. Then, it will be used online to dynamically adapt the action to the system state.

4.2. State-based adaptive control to optimize average utility per stage

To solve the ITH problem of (9) we first define the average utility per stage when using policy π and starting from the initial state s as

$$J_{\pi }\left(s\right)={\lim }_{N\to \infty }E\left[\frac{1}{N}{\sum }_{k=1}^{N}g\left(s_k\right)|\left(s_1=s\right),\pi \right]$$

(22)

We denote the optimal policy as π* which produces the maximum average utility per stage J*. Both π* and J* are independent of the initial state since the influence of the utility of the early stages on the average utility reduces to 0 as N → ∞. Moreover, since the utility per stage, the transition probabilities (4), and the state update Equations (1)-(3) are all stationary, the optimal policy will be stationary (does not change from stage to stage). Therefore, it is a single function, μ*(s), that maps the system states to actions regardless of the stage.

J*, together with the so-called relative value function h*(s), should satisfy the following Bellman's fixed point equation [31] for every state:

$$J^{*}+h^{*}\left(s\right)=\underset{\mu }{\max }\left\{g\left(s,\mu \right)+E\left[h^{*}\left(s^{+}\right)\right]\right\}$$

(23)

where s⁺ indicates the successor state of the current state s. Considering φ(·) as the state update function s⁺ = φ (s, μ, C⁺, B). The expectation in Equation (23) is over the random processes {B _k } and {C _k }.

We use a modified relative value iteration algorithm to solve the ITH problem [31]. First, we define a variant of the Bellman operator over any function f as

$${\mathcal{B}}_{\tau }f\left(s\right)=\underset{\mu }{\max }\left\{g\left(s,\mu \right)+\tau E\left[f\left(s^{+}\right)\right]\right\}$$

(24)

where parameter τ ∈ (0, 1) is a scalar. Then, the following iterative algorithm is used in order to calculate h⁽ⁿ⁺¹⁾(s) for all states of the state space in the iteration (n + 1):

$$h^{\left(n+1\right)}\left(s\right)=\left(1-\tau \right)h^{\left(n\right)}\left(s\right)+{\mathcal{B}}_{\tau }{h}^{\left(n\right)}\left(s\right)-{\mathcal{B}}_{\tau }{h}^{\left(n\right)}\left(s^{\prime }\right),$$

(25)

where s' is some fixed state. We initialize this algorithm with h⁽⁰⁾(s) = g(s, I = 0, r = 0). Convergence of (25) is guaranteed since queue and channel states are recurrent [31]. The decision will also be updated and will finally converge to the optimal decision as n → ∞:

$${\mu }^{\left(n\right)}\left(s\right)=\arg \underset{\mu }{\max }\left\{g\left(s,\mu \right)+\tau E\left[h^{\left(n\right)}\left(s^{+}\right)\right]\right\},$$

(26)

The practical application of (24) requires the state space to be discrete, so we use the same discretization procedure as in Section 4.1. This results in the following modified Bellman operator:

$${\mathcal{B}}_{\tau }^{d}f\left(s^d\right)=\underset{\mu }{\max }\left\{g\left(s^d,\mu \right)+\tau E\left[{\sum }_{m=1}^{M_q}{\sum }_{l=1}^{M_r}{w}_q^m\left(q^{+}\right)w_r^l\left(r^{+}\right)f\left(s^{+d}\right)\right]\right\}$$

(27)

Therefore, we apply (27) and compute h⁽ⁿ⁾(s^d ) for all possible discrete states. For the uncorrelated and first-order channel models there are (L × M _q × M _r × M) discrete states and for the second-order channel model this number should be multiplied by M.

5. Structural properties of the optimal solution

In the previous section, we provided DP algorithms that can be applied to find optimal decisions through numerical calculations. In this section, we investigate some structural properties of the solution. We use the following practical assumptions throughout this section.

Assumption 1: Per stage utility function has a format of (7) and (8) with U(·), and V(·) as increasing functions.

Assumption 2: Consider $\left(C^{-}={Ĉ}_{a^{-}},C={Ĉ}_a\right)$ as the channel state in the previous and current slot, respectively, and $\left({Ĉ}_{b^{-}},{Ĉ}_b\right)$ as other possible channel states in these two slots, where ${Ĉ}_a\le {Ĉ}_b$ and ${Ĉ}_{a^{-}}\le {Ĉ}_{b^{-}}$. We assume that there exists a j such that the following inequality holds for channel transition probabilities:

$$\sum _{i=1}^j{P}_{b^{-},b,i}\le \sum _{i=1}^j{P}_{a^{-},a,i}$$

where $P_{a^{-},a,i}$ is the probability of going from channel states $\left({Ĉ}_{a^{-}},{Ĉ}_a\right)$ to the next state $C^{+}={Ĉ}_i$ as defined for second order Markov model.

Assumption 2 is valid in practice for Markov channels since $\left({Ĉ}_{a^{-}},{Ĉ}_a\right)$ is supposed to be lower than and $\left({Ĉ}_{b^{-}},{Ĉ}_b\right)$and each side of the inequality calculates the probability of going to the first j states with lowest rates. For example, if $P_{b^{-},b,1}\le P_{a^{-},a,1}$ then the inequality will be true for j = 1 for and assumption 2 holds. If the inequality turns out to be true for any value of j then the assumption is correct. Based on this assumption we provide the following lemma:

Lemma 1: If f(C) and g(C) are two increasing functions, f(C) ≤ g(C), C⁺ is the next channel state, and similar to Assumption 2 ${Ĉ}_a\le {Ĉ}_b$ and $\left({Ĉ}_{a^{-}}\le {Ĉ}_{b^{-}}\right)$ then we have

$$E\left\{f\left(C^{+}\right)|C^{-}={Ĉ}_{a^{-}},C={Ĉ}_a\right\}\le E\left\{g\left(C^{+}\right)|C^{-}={Ĉ}_{b^{-}},C={Ĉ}_b\right\}$$

(28)

Proof is provided in the Appendix.

5.1. Structural properties of FTH solution

The following theorem indicates monotonicity of J _k (s _k ) versus the state variables.

Theorem 1: J _k (s _k ) is a decreasing function of q _k and ${\bar{q}}_k$, and an increasing function of ${\bar{r}}_k$ and C _k for all values of k.

Proof: In order to prove the theorem we show through induction that for k = N + 1, ..., 1 we have J _k (s _k + Δ) ≤ J _k (s _k ) for any vector Δ that increase q _k and ${\bar{q}}_k$, and decrease ${\bar{r}}_k$ and C _k .

Based on Equation (11) for optimal decision in the k th stage, we define G _k (s _k , μ _k ) as:

$$G_k\left(s_k,{\mu }_k\right)=g\left(s_k,{\mu }_k\right)+E\left[J_{k+1}\left(s_{k+1}\right)\right]$$

(29)

Thus, $J_k\left(s_k\right)=G_k\left(s_k,{\mu }_k^{*}\right)$ where ${\mu }_k^{*}$ is optimal decision for state s _k . Also we define Δ = (δ₁, δ₂, - δ₃, -δ₄, -δ₅) for any value of δ _i ≥ 0, i = 1, ... , 5 such that $s_k+\mathrm{\Delta }=\left(q_k+{\delta }_1,{\bar{q}}_k+{\delta }_2,{\bar{r}}_k-{\delta }_3,C_k-{\delta }_4,C_{k-1}-{\delta }_5\right)$ is an element of the state space.

For k = N + 1 we have $J_{N+1}\left(s_{N+1}\right)=U\left({\bar{r}}_{N+1}\right)-\gamma \mathsf{\text{V(}}{\bar{q}}_{N+1}\mathsf{\text{)}}-\eta q_{N+1}$ and using assumption 1 it is clear that J_N+1(s_N+1+ Δ) ≤ J_N+1(s_N+1). Assuming J_k+1(s_k+1) is a monotonic function we show J _k (s _k ) is also monotonic for k = N, ... , 1 which completes the proof.

We define ${\mu }_{k,\mathrm{\Delta }}^{*}$ as optimal decision for state s _k + Δ in stage k, so we can write $J_k\left(s_k+\mathrm{\Delta }\right)=G_k\left(s_k+\mathrm{\Delta },{\mu }_{k,\mathrm{\Delta }}^{*}\right)$, however ${\mu }_{k,\mathrm{\Delta }}^{*}$ is not an optimal decision for state s _k so we have:

$$G_k\left(s_k,{\mu }_{k,\mathrm{\Delta }}^{*}\right)\le G_k\left(s_k,{\mu }_k^{*}\right)=J_k\left(s_k\right)$$

(30)

Using Assumption 1 it is clear that g is a monotonic function of the state variables

$$g\left(s_k+\mathrm{\Delta },{\mu }_{k,\mathrm{\Delta }}^{*}\right)\le g\left(s_k,{\mu }_{k,\mathrm{\Delta }}^{*}\right)$$

(31)

We consider φ (·) as the state update function and define two possible next states $s_{k+1,\mathrm{\Delta }}^{*}=\phi \left(s_k+\mathrm{\Delta },{\mu }_{k,\mathrm{\Delta }}^{*},C_{k+1},B_k\right)$ and $s_{k+1}^{\#}=\phi \left(s_k,{\mu }_{k,\mathrm{\Delta }}^{*},C_{k+1},B_k\right)$. For known values of C_k+1and B _k we can use (1)-(3) and easily show that $s_{k+1,}^{{*}_{\mathrm{\Delta }}}=s_{k+1}^{\#}+{\mathrm{\Delta }}^{\prime }$ in which ${\mathrm{\Delta }}^{\prime }=\left({\delta }_1^{\prime },{\delta }_2^{\prime },-{\delta }_3^{\prime },-{\delta }_4^{\prime },-{\delta }_5^{\prime }\right)$ for some ${\delta }_i^{\prime }\ge 0$. Thus $J_{k+1}\left(s_{k+1,\mathrm{\Delta }}^{*}\right)\le J_{k+1}\left(s_{k+1}^{\#}\right)$.

We define $f\left(C_{k+1}\right)\triangleq E_B\left[J_{k+1}\left(s_{k+1,\mathrm{\Delta }}^{*}\right)\right]$ and $g\left(C_{k+1}\right)\triangleq E_B\left[J_{k+1}\left(s_{k+1}^{\#}\right)\right]$ since B is independent of the system state thus we have f(C_k+1) ≤ g (C_k+1) and since J_k+1(·) is an increasing function of C, then f(·) and g(·) are increasing functions. Applying Lemma 1 with ${Ĉ}_{a^{-}}=\left(C_{k-1}-{\delta }_5\right)$, ${Ĉ}_a=\left(C_k-{\delta }_4\right)$, ${Ĉ}_{b^{-}}=C_{k-1}$, and ${Ĉ}_b=C_k$ we find that

$$E_{C^{+}}\left\{E_B\left[J_{K+1}\left(s_{k+1,\mathrm{\Delta }}^{*}\right)\right]\right\}\le E_{C^{+}}\left\{E_B\left[J_{k+1}\left(s_{k+1}^{\#}\right)\right]\right\}$$

(32)

Combining (31), (32) and considering definition of G _k in (29) we get

$$J_k\left(s_k+\mathrm{\Delta }\right)=G_k\left(s_k+\mathrm{\Delta },{\mu }_{k,\mathrm{\Delta }}^{*}\right)\le G_k\left(s_k,{\mu }_{k,\mathrm{\Delta }}^{*}\right)$$

(33)

Equation (30) together with (33) prove the theorem by showing: J _k (s _k + Δ) ≤ J _k (s _k ).

   ■

Assuming uncorrelated channel model the following theorem indicates the "threshold structure" of the optimal transmission policy versus the channel state.

Theorem 2: If the optimal access decision in state $s_k=\left(q_k,{\bar{q}}_k,{\bar{r}}_k,C_k\right)$ is $I_k^{*}\left(s_k\right)=1$, then for another possible state $s_k^{\prime }=\left(q_k,{\bar{q}}_k,{\bar{r}}_k,C_k^{\prime }\right)$ in the same slot with improved channel state $C_k^{\prime }\ge C_k$ we have $I_k^{*}\left(s_k^{\prime }\right)=1$.

Proof: Assume ${\mu }_k^{*}\left(s_k\right)=\left(I_k=1,r_k\right)$ but ${\mu }_k^{*}\left(s_k^{\prime }\right)=\left(I_k^{\prime }=0,r_k^{\prime }\right)$ as the optimal decision for s _k and $s_k^{\prime }$, respectively. According to the definition of G _k in the proof of Theorem 1, ${\mu }_k^{*}\left(s_k\right)$ maximizes G _k (s _k , μ _k ) and we have

$$G_k\left(s_k,{\mu }_k^{*}\left(s_k^{\prime }\right)\right)\le G_k\left(s_k,{\mu }_k^{*}\left(s_k\right)\right)$$

(34)

On the other hand since s _k and $s_k^{\prime }$ differs only in the channel state, we have $g\left(s_k,{\mu }_k^{*}\left(s_k^{\prime }\right)\right)=g\left(s_k^{\prime },{\mu }_k^{*}\left(s_k^{\prime }\right)\right)$ and by using $\left(I_k^{\prime }=0,r_k^{\prime }\right)$ for both states, queue size will modify similarly for s _k , and $s_k^{\prime }$ which results in the same next state, $s_{k+1}=s_{k+1}^{\prime }$. Also for uncorrelated channel model the averaging over next channel state does not depend on the current state, thus $E_{C^{+}}\left\{E_B\left[J_{k+1}\left(s_{k+1}\right)\right]\right\}=E_{C^{+}}\left\{E_B\left[J_{k+1}\left(s_{k+1}^{\prime }\right)\right]\right\}$ and

$$G_k\left(s_k,{\mu }_k^{*}\left(s_k^{\prime }\right)\right)=G_k\left(s_k^{\prime },{\mu }_k^{*}\left(s_k^{\prime }\right)\right)$$

(35)

By applying decision ${\mu }_k^{*}\left(s_k\right)=\left(I_k=1,r_k\right)$ and since transmission with better channel state will decrease q and $\bar{q}$ which increase J _k according to Theorem 1, we have

$$G_k\left(s_k,{\mu }_k^{*}\left(s_k\right)\right)\le G_k\left(s_k^{\prime },{\mu }_k^{*}\left(s_k\right)\right)$$

(36)

Combining (34), (35), and (36) results in

$$G_k\left(s_k^{\prime },{\mu }_k^{*}\left(s_k^{\prime }\right)\right)\le G_k\left(s_k^{\prime },{\mu }_k^{*}\left(s_k\right)\right)$$

which is in contrast to optimality of the ${\mu }_k^{*}\left(s_k^{\prime }\right)=\left(I_k^{\prime }=0,r_k^{\prime }\right)$. Thus, we should have ${\mu }_k^{*}\left(s_k^{\prime }\right)=\left(I_k^{\prime }=1,r_k^{\prime }\right)$.

   ■

Note that Theorem 2 may be incorrect when channel state is time correlated. For example, consider two possible channel states C _k and $C_k^{\prime }$, with $C_k<C_k^{\prime }$ and assume that optimal decision is to transmit for a state with C _k . Also assume that probability of going from $C_k^{\prime }$ to a better channel state and from C _k to a worse channel state is high. So, we can argue heuristically that in this condition it may be optimal to transmit data when channel is in state C _k but not to transmit when it is in state $C_k^{\prime }$.

5.2. Structural properties of ITH solution

We provide structural properties of ITH solution in this section through the following theorems. First we show that relative value function, h*(s), is a monotonic function in Theorem 3 and then prove the threshold structure of access decision versus channel state in Theorem 4.

Theorem 3: h*(s), is a decreasing function of q and $\bar{q}$, and an increasing function of $\bar{r}$ and C.

Proof: We define Δ = (δ₁, δ₂, -δ₃, -δ₄, -δ₅) with δ _i ≥ 0 and show that h*(s + Δ) ≤ h*(s). We also define G _f (s, μ(s)) on function f as

$$G_f\left(s,\mu \right)=g\left(s,\mu \right)+\tau E\left[f\left(s^{+}\right)\right]$$

(37)

Assuming μ* as the decision that maximizes G _f and according to the Bellman equation (24) we have

$${\mathcal{B}}_{\tau }f\left(s\right)=G_f\left(s,{\mu }^{*}\right)$$

Taking into account h*(s) = lim_k→∞h⁽ⁿ⁾(s), we prove through induction for every iteration n, h⁽ⁿ⁾(s + Δ) ≤ h⁽ⁿ⁾(s). For n = 0 we define h⁽⁰⁾(s) = g(s, I = 0, r = 0) which according to Assumption 1 it is clear that h⁽⁰⁾ (s + Δ) ≤ h⁽⁰⁾ (s). We assume that h⁽ⁿ⁾(s) is monotonic and show h⁽ⁿ⁺¹⁾(s) is also monotonic. First, we show that B _τ h⁽ⁿ⁾(s) is monotonic. Using (26) for states s and s + Δ and assuming μ* and ${\mu }_{\mathrm{\Delta }}^{*}$, respectively, as maximizing actions we have

$${G_h}^{\left(n\right)}\left(s,{\mu }_{\mathrm{\Delta }}^{*}\right)\le {\mathcal{B}}_{\tau }{h}^{\left(n\right)}\left(s\right)={G_h}^{\left(n\right)}\left(s,{\mu }^{*}\right)$$

(38)

From definition of g it is clear that $g\left(s+\mathrm{\Delta },{\mu }_{\mathrm{\Delta }}^{*}\right)\le g\left(s,{\mu }_{\mathrm{\Delta }}^{*}\right)$. We can use the similar approach as the proof of Theorem 1 and apply Lemma 1 to show that

$$E_C+\left\{E_B\left[h^{\left(n\right)}\left({\left(s+\mathrm{\Delta }\right)}^{+}\right)|{\mu }_{\mathrm{\Delta }}^{*}\right]\right\}\le E_C+\left\{E_B\left[h^{\left(n\right)}\left(s^{+}\right)|{\mu }_{\mathrm{\Delta }}^{*}\right]\right\}$$

(39)

which results in

$${\mathcal{B}}_{\tau }{h}^{\left(n\right)}\left(s+\mathrm{\Delta }\right)=G_h\left(n\right)\left(s+\mathrm{\Delta },{\mu }_{\mathrm{\Delta }}^{*}\right)\le G_h\left(n\right)\left(s,{\mu }_{\mathrm{\Delta }}^{*}\right)$$

(40)

Combining (38) and (40) we find that B _τ h⁽ⁿ⁾(s + Δ) ≤ B _τ h⁽ⁿ⁾(s). Using Equation (25) and taking into account that B _τ h⁽ⁿ⁾(s') is independent of the state vector, it can be easily shown that h^{(n + 1)}(s) is a monotonic function.

   ■

Assuming uncorrelated channel model the following theorem indicates existence of a threshold for channel state that the link should decide to transmit when channel state is better than or equal to that threshold.

Theorem 4: There exists a threshold, C_th, that for $s_{\mathsf{\text{th}}}=\left(q=C_{\mathsf{\text{th}}},\bar{q},\bar{r},C_{\mathsf{\text{th}}}\right)$ with any $\bar{q}$ and $\bar{r}$, we have I*(s _th ) = 1. Also for any s with C ≥ C_th and q ≥ C_th we have I*(s) = 1.

Proof: Assume in timeslot k we have C _k = C^max and q _k = C^max, transmission at this time has the energy cost of β e but it will reduce q by C^max which will reduce $\bar{q}$ by θ _q C^max and also will reduce the future costs related to the queue size. However, transmission of theses C^max packets at any later time slot requires the same amount of energy. Thus, it is better to transmit these packets at state s_th to reduce the queue size as early as possible and reduce the future costs related to the queue size. We conclude that: "if C = C^max and q = C^max then I*(s) = 1" which proves existence of C_th.

In order to prove the second part of the theorem we assume $s=\left(q\ge C_{\mathsf{\text{th}}},\bar{q},\bar{r},C\ge C_{\mathsf{\text{th}}}\right)$ and consider optimal decisions μ*(s_th) = (I*(s_th), r*(s_th)) and μ*(s) = (I*(s), r*(s)) for states s_th and , respectively. If I(s) = 0 we can show similar to the proof of Theorem 2 that it cannot be an optimal transmission policy.

   ■

6. Numerical results

For numerical analysis of the adaptive control algorithms provided in Section 4 we consider a lightweight sensor in a wireless network that may transmit its status using few bits. In each timeslot the sensor may send its own packet or forward packets of other sensors. We assume a Rayleigh flat fading channel, and use a set of simple Modulation and Coding schemes. Note that our adaptive algorithm only requires the FSMC model which can be found for many practical fading channels [21] and do not depend on Rayleigh fading assumption or Modulation schemes. However, in this section we consider the following types of modulations joint with Reed-Solomon (RS) coding:

• Q₁: No transmission since link is in deep fade.

• Q₂: BPSK with RS (63,47).

• Q₃: QPSK with RS (127,94).

• Q₄: 16-QAM with RS (255, 188).

Note that in each time slot one frame will be transmitted and the time duration of the frames is identical for different schemes. Figure 2 illustrates FER of the aforementioned schemes. Setting 0.01 as the FER threshold, we find SNR thresholds A _j for the fading regions which ensure the required FER limit for (Q₁, Q₂, Q₃, Q₄) as {0, 3.8, 7.77, 33.1, ∞}. For example, we will use Q₃ while SNR is between 7.77 (8.9 dB) and 33.1 (15.2 dB). Assuming a Rayleigh fading channel, we use the Markov model proposed in [21, 30] to obtain the transition probabilities for a given normalized Doppler frequency. Unless otherwise indicated we assume the average SNR at the receiver is $\overline{\mathsf{\text{SNR}}}=10\mathsf{\text{dB}}$ and normalized Doppler frequency is f_dT_s = 0.02. Each frame that will be transmitted in the data slot contains a coded block. Considering the packet length of 47 bits, 0, 1, 2, or 4 packets can be transmitted in a frame based on the channel state so $\mathcal{C}=\left\{0,1,2,4\right\}$.

Figure 2

FER versus SNR for different modulations and RS coding schemes.

Regarding the per stage utility function (7) we use $U\left({\bar{r}}_k\right)=\log \left(\epsilon +{\bar{r}}_k\right)$, and $\mathsf{\text{V}}\left({\bar{q}}_k\right)={\left({\bar{q}}_k\right)}^2$ to avoid very small rates and large queue sizes. Logarithmic utility is used to provide proportional fairness in the network [14] and prevent selfish rate maximization of the link. Also it provides fairness among multiple flows over a single link [27]. Recently, it has been shown, based on experimental results and Weber-Fechner psychophysical law, that user experience and satisfaction follows logarithm laws, and quality of experience (QoE) versus rate is formulated as QoE(r) = log (ar + b) [33]. In the utility function, energy and queue sizes are used with negative weights to minimize energy consumption and delay. Remember that the energy consumed at the k th timeslot is given by E _k = eI _k (1 - B _k ). Therefore

$$g\left(s_k,{\mu }_k\right)=\log \left(\epsilon +{\bar{r}}_k\right)-\alpha {\left({\bar{q}}_k\right)}^2-\beta e{I}_k\left(1-P_B\right)$$

(41)

We also use the following parameters for simulations unless otherwise indicated: θ _q = θ _r = 0.7, L = 12, α = 0.005, βe = 1, , P _B = 0.1, ε = 0.001. Using these parameters, Figure 3 shows the selected per stage utility which is an increasing function of ${\bar{r}}_k$ and decreasing function of ${\bar{q}}_k$. Our selected parameters result in a utility function which is negative; however, behavior of this function versus system state is such that by maximizing it we will maximize rate while minimizing energy and delay.

Figure 3

Sample of per stage utility.

As described in Section 4, continuous state variables, ${\bar{r}}_k$ and ${\bar{q}}_k$, should be discretized in order to achieve a finite state system and dynamic programming solution. We set M _r = 21, and M _q = 13 for discretization. It is shown in our simulations that enhancement achieved by selecting greater values for M _r , and M _q is insignificant. Also the maximum queue size is assumed to be L = 12 and the number of arrival packets at each stage r _k is limited by 4.

6.1. FTH results

As indicated earlier for FTH problem we use (8) as the final stage utility function with η = 5 as the price for packets remaining in the queue where $U\left({\bar{r}}_k\right)=\log \left(\epsilon +{\bar{r}}_k\right)$, and $\mathsf{\text{V}}\left({\bar{q}}_k\right)={\left({\bar{q}}_k\right)}^2$. We assume that the initial state of the link is ${\bar{r}}_1=0.1$, ${\bar{q}}_1=q_1=0$ and consider a flat fading Rayleigh channel. The recursive algorithm (16)-(18) is used to obtain the optimal control policy (over the discretized state space) for different values of N. The transition probabilities of the Markov models are computed as described earlier.

The optimal control policies are then used in Monte Carlo simulations, over the channel response γ _k and the channel occupation B _k processes, to maximize J₁(s₁), the sum of the utilities of the N stages. Figure 4 illustrates the J₁(s₁), as a function of the number of timeslots N, for different channel correlation models and two values of average SNR. This figure shows that the performance is enhanced by exploiting the channel correlation through the FSMC models, mainly for large values of N. It also shows that the use of second-order FSMC is not worthwhile in these cases. For N > 50, J₁ (s₁) varies almost as a linear function of N where the slope depends on the channel correlation model and the average SNR.

Figure 4

FTH utility versus Maximum slot number for different SNRs and channel models.

We have also investigated the dependence of performance on the initial state of the link. Figure 5 illustrates the average utility, J₁(s₁)/N, versus the initial EWMA rate ${\bar{r}}_1$ for different values of N. As expected the higher initial EWMA rate, the higher average utility. It also shows that sensitivity to the initial state decreases as the number of slots increases.

Figure 5

Average utility versus initial EWMA rate for N = 20, N = 50 and N = 100.

Size of the grid used for discretization, M _q and M _r , can affect performance of the system. In Table 1 we provide FTH performance of the algorithms that have used different values of M _q and M _r for N = 40. We can see that enhancement achieved by selecting values greater than M _r = 21, and M _q = 13 is negligible.

Table 1 Performance versus discretization grid size

6.2. ITH results

We use the modified relative value iteration algorithm (25), with τ = 0.9, in order to find the optimal control policies for the infinite time horizon problems. Unless otherwise indicated, in the following results we have considered the first-order FSMC channel model. In each iteration the algorithm computes new values of I⁽ⁿ⁾(s) and r⁽ⁿ⁾(s) for all possible states and finally it converges to the optimal control policy (for the discretized problem) μ*(s) = (I*(s), r*(s)). Figure 6 illustrates the convergence of the iterative algorithm, by showing the percentage of decisions, (I⁽ⁿ⁾(s), r⁽ⁿ⁾(s)). which are modified in each iteration in comparison with the previous iteration. Apart from the optimal control policy, the algorithm also provides the optimal relative value function, h*(s). Figure 7a,b illustrates r*(s), and h*(s) versus some elements of the state vector while fixing the others. They show interesting properties of the optimal policy and relative value function with respect to the system state. For example, Figure 7a shows that the arrival rate should be reduced as the channel goes to the fade state or as the EWMA of rate increases. Figure 7b indicates that the relative utility function decreases as the queue size increases or the channel goes to the fade state.

Figure 6

Convergence of modified relative value iteration algorithm.

Figure 7

DP optimal policy: (a) Optimal arrival rate versus channel state and EWMA of packet arrival. (b) Relative utility versus queue state and channel state.

Figure 8 demonstrates the optimal actions for a particular realization of the channel process in a period of 200 timeslots. Note that when the channel goes to a deep fade during timeslots 231 to 282 the link does not access the channel (I _k = 0) so there is not energy consumption in this period (E _k = 0). Also, new packet arrivals are reduced to prevent high queue backlog but kept at a minimum rate to prevent $\log \left(\epsilon +{\bar{r}}_k\right)$ from very negative values. After the deep fade finishes the link starts to transmit backlogged packets while keeping slow arrival rate until timeslot 288.

Figure 8

Time variations of control and state variables.

Based on the selected format of the per stage utility (7), we can reduce the energy consumption by increasing β. However, this is achieved at the cost of reducing the transmission rate and increasing the delay as shown in Figure 9. In other words, the figure shows the tradeoff between energy, rate, and delay as a function of β. Here, the average delay is calculated using the little's low: $\bar{D}={\sum }_k{q}_k/{\sum }_k{r}_k$ [34].

Figure 9

Energy-(rate, delay) tradeoff.

Figure 10 shows the performance of the optimal policies, obtained from different channel correlation models, as a function of the time variability of the channel. In particular, it shows the resulting average utility per stage, as a function of the normalized Doppler frequency f_dT_s, for the first- and second-order FSMC models and different values of the EWMA parameters for packet arrivals and queue occupancy. It shows that average utility is higher for fading channels with higher f_dT_s since channel remains for a short time in deep fades. For θ = θ _q = θ _r = 0.7, which corresponds to larger averaging time of rate and queue size, both channel models exhibit similar performance. However, for θ = θ _q = θ _r = 0.3 we see that the more accurate second-order FSMC model enhances the performance of the link compared to the first-order FSMC model.

Figure 10

Average utility per stage as a function of the normalized Doppler frequency for different channel correlation models and EWMA smoothing factors.

7. Conclusions

We addressed the problem of optimal channel access and rate adaptation in the links of CSMA wireless networks. We defined a utility function that trades off the energy consumption and the average packet transmission rate and delay. By using dynamic programming, we derive algorithms and optimal policies that maximize the average utility by adapting the arrival packet rate and channel access as functions of the queue occupancy, channel state, and smoothed rate. The optimal policies can be computed and stored offline. Then, they can be used online for dynamic access control and queue management of the link. The proposed algorithms exploit the time correlation of the channel by means of different FSMC models. Both FTH and ITH problems were addressed. In the first case, the average utility is optimized for a finite time period, whereas in the second case, the long-term average utility is maximized. Structural properties of the optimal solution are investigated and it is shown that optimal transmission policy has a threshold structure versus the channel state. For the ITH problem we proved the existence of a channel state that the link should always transmit when the channel is in that state or in a better one. Numerical results show that the overall performance of the link can be enhanced by increasing the order of the FSMC channel model. However, it increases the complexity of the algorithms and the memory required to store the optimal policies.

Appendix

Proof of Lemma 1

The difference between right- and left-hand of inequality (28) can be calculated using the channel transition probabilities:

$$\mathcal{D}\triangleq E\left\{g\left(C^{+}\right)|{Ĉ}_{b^{-}},{Ĉ}_b\right\}-E\left\{f\left(C^{+}\right)|{Ĉ}_{a^{-}},{Ĉ}_a\right\}=\sum _{i=1}^M{P}_{b^{-},b,i}g\left({Ĉ}_i\right)-P_{a^{-},a,i}f\left({Ĉ}_i\right)$$

We partition the summation and rewrite it as

$$\begin{array}{c}\mathcal{D}=\sum _{i=1}^j{P}_{b^{-},b,i}\left[g\left({Ĉ}_i\right)-f\left({Ĉ}_i\right)\right]-\left(P_{a^{-},a,i}-P_{b^{-},b,i}\right)f\left({Ĉ}_i\right)\\ +\sum _{i=j+1}^M\left(P_{b^{-},b,i}-P_{a^{-},a,i}\right)g\left({Ĉ}_i\right)+P_{a^{-},a,i}\left[g\left({Ĉ}_i\right)-f\left({Ĉ}_i\right)\right]\\ \ge \sum _{i=j+1}^M\left(P_{b^{-},b,i}-P_{a^{-},a,i}\right)g\left({Ĉ}_i\right)-\sum _{i=1}^j\left(P_{a^{-},a,i}-P_{b^{-},b,i}\right)f\left({Ĉ}_i\right)\\ \ge g\left({Ĉ}_j\right)\sum _{i=j+1}^M\left(P_{b^{-},b,i}-P_{a^{-},a,i}\right)-f\left({Ĉ}_j\right)\sum _{i=1}^j\left(P_{a^{-},a,i}-P_{b^{-},b,i}\right)\end{array}$$

the first inequality is a result of $g\left({Ĉ}_i\right)-f\left({Ĉ}_i\right)\ge 0$ and the second one considers $g\left({Ĉ}_j\right)\le g\left({Ĉ}_i\right)$, i = j + 1, ... , M, and $f\left({Ĉ}_j\right)\ge f\left({Ĉ}_i\right)i=1,...,j$. Since ${\sum }_{i=1}^M{P}_{b^{-},b,i}={\sum }_{i=1}^M{P}_{a^{-},a,i}=1$ we have ${\sum }_{i=1}^i{P}_{a^{-},a,i}-P_{b^{-},b,i}={\sum }_{i=j+1}^M{P}_{b^{-},b,i}-P_{a^{-},a,i}$ thus

$$\mathcal{D}\ge \left[g\left({Ĉ}_j\right)-f\left({Ĉ}_j\right)\right]\sum _{i=1}^j\left(P_{a^{-},a,i}-P_{b^{-},b,i}\right)\ge 0$$

where the second inequality is a result of Assumption 2.    ■