In the previous section, we provided DP algorithms that can be applied to find optimal decisions through numerical calculations. In this section, we investigate some structural properties of the solution. We use the following practical assumptions throughout this section.
Assumption 1: Per stage utility function has a format of (7) and (8) with U(·), and V(·) as increasing functions.
Assumption 2: Consider as the channel state in the previous and current slot, respectively, and as other possible channel states in these two slots, where and . We assume that there exists a j such that the following inequality holds for channel transition probabilities:
where is the probability of going from channel states to the next state as defined for second order Markov model.
Assumption 2 is valid in practice for Markov channels since is supposed to be lower than and and each side of the inequality calculates the probability of going to the first j states with lowest rates. For example, if then the inequality will be true for j = 1 for and assumption 2 holds. If the inequality turns out to be true for any value of j then the assumption is correct. Based on this assumption we provide the following lemma:
Lemma 1: If f(C) and g(C) are two increasing functions, f(C) ≤ g(C), C+ is the next channel state, and similar to Assumption 2 and then we have
(28)
Proof is provided in the Appendix.
5.1. Structural properties of FTH solution
The following theorem indicates monotonicity of J
k
(s
k
) versus the state variables.
Theorem 1: J
k
(s
k
) is a decreasing function of q
k
and , and an increasing function of and C
k
for all values of k.
Proof: In order to prove the theorem we show through induction that for k = N + 1, ..., 1 we have J
k
(s
k
+ Δ) ≤ J
k
(s
k
) for any vector Δ that increase q
k
and , and decrease and C
k
.
Based on Equation (11) for optimal decision in the k th stage, we define G
k
(s
k
, μ
k
) as:
(29)
Thus, where is optimal decision for state s
k
. Also we define Δ = (δ1, δ2, - δ3, -δ4, -δ5) for any value of δ
i
≥ 0, i = 1, ... , 5 such that is an element of the state space.
For k = N + 1 we have and using assumption 1 it is clear that JN+1(sN+1+ Δ) ≤ JN+1(sN+1). Assuming Jk+1(sk+1) is a monotonic function we show J
k
(s
k
) is also monotonic for k = N, ... , 1 which completes the proof.
We define as optimal decision for state s
k
+ Δ in stage k, so we can write , however is not an optimal decision for state s
k
so we have:
(30)
Using Assumption 1 it is clear that g is a monotonic function of the state variables
(31)
We consider φ (·) as the state update function and define two possible next states and . For known values of Ck+1and B
k
we can use (1)-(3) and easily show that in which for some . Thus .
We define and since B is independent of the system state thus we have f(Ck+1) ≤ g (Ck+1) and since Jk+1(·) is an increasing function of C, then f(·) and g(·) are increasing functions. Applying Lemma 1 with , , , and we find that
(32)
Combining (31), (32) and considering definition of G
k
in (29) we get
(33)
Equation (30) together with (33) prove the theorem by showing: J
k
(s
k
+ Δ) ≤ J
k
(s
k
).
■
Assuming uncorrelated channel model the following theorem indicates the "threshold structure" of the optimal transmission policy versus the channel state.
Theorem 2: If the optimal access decision in state is , then for another possible state in the same slot with improved channel state we have .
Proof: Assume but as the optimal decision for s
k
and , respectively. According to the definition of G
k
in the proof of Theorem 1, maximizes G
k
(s
k
, μ
k
) and we have
(34)
On the other hand since s
k
and differs only in the channel state, we have and by using for both states, queue size will modify similarly for s
k
, and which results in the same next state, . Also for uncorrelated channel model the averaging over next channel state does not depend on the current state, thus and
(35)
By applying decision and since transmission with better channel state will decrease q and which increase J
k
according to Theorem 1, we have
(36)
Combining (34), (35), and (36) results in
which is in contrast to optimality of the . Thus, we should have .
■
Note that Theorem 2 may be incorrect when channel state is time correlated. For example, consider two possible channel states C
k
and , with and assume that optimal decision is to transmit for a state with C
k
. Also assume that probability of going from to a better channel state and from C
k
to a worse channel state is high. So, we can argue heuristically that in this condition it may be optimal to transmit data when channel is in state C
k
but not to transmit when it is in state .
5.2. Structural properties of ITH solution
We provide structural properties of ITH solution in this section through the following theorems. First we show that relative value function, h*(s), is a monotonic function in Theorem 3 and then prove the threshold structure of access decision versus channel state in Theorem 4.
Theorem 3: h*(s), is a decreasing function of q and , and an increasing function of and C.
Proof: We define Δ = (δ1, δ2, -δ3, -δ4, -δ5) with δ
i
≥ 0 and show that h*(s + Δ) ≤ h*(s). We also define G
f
(s, μ(s)) on function f as
(37)
Assuming μ* as the decision that maximizes G
f
and according to the Bellman equation (24) we have
Taking into account h*(s) = limk→∞h(n)(s), we prove through induction for every iteration n, h(n)(s + Δ) ≤ h(n)(s). For n = 0 we define h(0)(s) = g(s, I = 0, r = 0) which according to Assumption 1 it is clear that h(0) (s + Δ) ≤ h(0) (s). We assume that h(n)(s) is monotonic and show h(n+1)(s) is also monotonic. First, we show that B
τ
h(n)(s) is monotonic. Using (26) for states s and s + Δ and assuming μ* and , respectively, as maximizing actions we have
(38)
From definition of g it is clear that . We can use the similar approach as the proof of Theorem 1 and apply Lemma 1 to show that
(39)
which results in
(40)
Combining (38) and (40) we find that B
τ
h(n)(s + Δ) ≤ B
τ
h(n)(s). Using Equation (25) and taking into account that B
τ
h(n)(s') is independent of the state vector, it can be easily shown that h(n + 1)(s) is a monotonic function.
■
Assuming uncorrelated channel model the following theorem indicates existence of a threshold for channel state that the link should decide to transmit when channel state is better than or equal to that threshold.
Theorem 4: There exists a threshold, Cth, that for with any and , we have I*(s
th
) = 1. Also for any s with C ≥ Cth and q ≥ Cth we have I*(s) = 1.
Proof: Assume in timeslot k we have C
k
= Cmax and q
k
= Cmax, transmission at this time has the energy cost of β e but it will reduce q by Cmax which will reduce by θ
q
Cmax and also will reduce the future costs related to the queue size. However, transmission of theses Cmax packets at any later time slot requires the same amount of energy. Thus, it is better to transmit these packets at state sth to reduce the queue size as early as possible and reduce the future costs related to the queue size. We conclude that: "if C = Cmax and q = Cmax then I*(s) = 1" which proves existence of Cth.
In order to prove the second part of the theorem we assume and consider optimal decisions μ*(sth) = (I*(sth), r*(sth)) and μ*(s) = (I*(s), r*(s)) for states sth and , respectively. If I(s) = 0 we can show similar to the proof of Theorem 2 that it cannot be an optimal transmission policy.
■