In the previous section, we provided DP algorithms that can be applied to find optimal decisions through numerical calculations. In this section, we investigate some structural properties of the solution. We use the following practical assumptions throughout this section.

**Assumption 1:** Per stage utility function has a format of (7) and (8) with U(·), and V(·) as increasing functions.

**Assumption 2:** Consider \left({C}^{-}={\u0108}_{{a}^{-}},C={\u0108}_{a}\right) as the channel state in the previous and current slot, respectively, and \left({\u0108}_{{b}^{-}},{\u0108}_{b}\right) as other possible channel states in these two slots, where {\u0108}_{a}\le {\u0108}_{b} and {\u0108}_{{a}^{-}}\le {\u0108}_{{b}^{-}}. We assume that there exists a *j* such that the following inequality holds for channel transition probabilities:

\sum _{i=1}^{j}{P}_{{b}^{-},b,i}\le \sum _{i=1}^{j}{P}_{{a}^{-},a,i}

where {P}_{{a}^{-},a,i} is the probability of going from channel states \left({\u0108}_{{a}^{-}},{\u0108}_{a}\right) to the next state {C}^{+}={\u0108}_{i} as defined for second order Markov model.

Assumption 2 is valid in practice for Markov channels since \left({\u0108}_{{a}^{-}},\phantom{\rule{0.3em}{0ex}}{\u0108}_{a}\right) is supposed to be lower than and \left({\u0108}_{{b}^{-}},{\u0108}_{b}\right)and each side of the inequality calculates the probability of going to the first *j* states with lowest rates. For example, if {P}_{{b}^{-},b,1}\le {P}_{{a}^{-},a,1} then the inequality will be true for *j* = 1 for and assumption 2 holds. If the inequality turns out to be true for any value of *j* then the assumption is correct. Based on this assumption we provide the following lemma:

**Lemma 1:** If *f*(*C*) and *g*(*C*) are two increasing functions, *f*(*C*) ≤ *g*(*C*), *C*^{+} is the next channel state, and similar to Assumption 2 {\u0108}_{a}\le {\u0108}_{b} and \left({\u0108}_{{a}^{-}}\le {\u0108}_{{b}^{-}}\right) then we have

E\left\{f\left({C}^{+}\right)|{C}^{-}={\u0108}_{{a}^{-}},C={\u0108}_{a}\right\}\le E\left\{g\left({C}^{+}\right)|{C}^{-}={\u0108}_{{b}^{-}},C={\u0108}_{b}\right\}

(28)

Proof is provided in the Appendix.

### 5.1. Structural properties of FTH solution

The following theorem indicates monotonicity of *J*_{
k
} (*s*_{
k
} ) versus the state variables.

**Theorem 1:** *J*_{
k
} (*s*_{
k
} ) is a decreasing function of *q*_{
k
} and {\stackrel{\u0304}{q}}_{k}, and an increasing function of {\stackrel{\u0304}{r}}_{k} and *C*_{
k
} for all values of *k*.

*Proof:* In order to prove the theorem we show through induction that for *k* = *N* + 1, ..., 1 we have *J*_{
k
} (*s*_{
k
} + Δ) ≤ *J*_{
k
} (*s*_{
k
} ) for any vector Δ that increase *q*_{
k
} and {\stackrel{\u0304}{q}}_{k}, and decrease {\stackrel{\u0304}{r}}_{k} and *C*_{
k
} .

Based on Equation (11) for optimal decision in the *k* th stage, we define *G*_{
k
} (*s*_{
k
} , *μ*_{
k
} ) as:

{G}_{k}\left({s}_{k},{\mu}_{k}\right)=g\left({s}_{k},{\mu}_{k}\right)+E\left[{J}_{k+1}\left({s}_{k+1}\right)\right]

(29)

Thus, {J}_{k}\left({s}_{k}\right)={G}_{k}\left({s}_{k},{\mu}_{k}^{*}\right) where {\mu}_{k}^{*} is optimal decision for state *s*_{
k
} . Also we define Δ = (*δ*_{1}, *δ*_{2}, - *δ*_{3}, -*δ*_{4}, -*δ*_{5}) for any value of *δ*_{
i
} ≥ 0, *i* = 1, ... , 5 such that {s}_{k}+\mathrm{\Delta}=\left({q}_{k}+{\delta}_{1},{\stackrel{\u0304}{q}}_{k}+{\delta}_{2},{\stackrel{\u0304}{r}}_{k}-{\delta}_{3},{C}_{k}-{\delta}_{4},{C}_{k-1}-{\delta}_{5}\right) is an element of the state space.

For *k* = *N* + 1 we have {J}_{N+1}\left({s}_{N+1}\right)=U\left({\stackrel{\u0304}{r}}_{N+1}\right)-\gamma \mathsf{\text{V(}}{\stackrel{\u0304}{q}}_{N+1}\mathsf{\text{)}}-\eta {q}_{N+1} and using assumption 1 it is clear that *J*_{N+1}(*s*_{N+1}+ Δ) ≤ *J*_{N+1}(*s*_{N+1}). Assuming *J*_{k+1}(*s*_{k+1}) is a monotonic function we show *J*_{
k
} (*s*_{
k
} ) is also monotonic for *k* = *N*, ... , 1 which completes the proof.

We define {\mu}_{k,\mathrm{\Delta}}^{*} as optimal decision for state *s*_{
k
} + Δ in stage *k*, so we can write {J}_{k}\left({s}_{k}+\mathrm{\Delta}\right)={G}_{k}\left({s}_{k}+\mathrm{\Delta},{\mu}_{k,\mathrm{\Delta}}^{*}\right), however {\mu}_{k,\mathrm{\Delta}}^{*} is not an optimal decision for state *s*_{
k
} so we have:

{G}_{k}\left({s}_{k},{\mu}_{k,\mathrm{\Delta}}^{*}\right)\le {G}_{k}\left({s}_{k},{\mu}_{k}^{*}\right)={J}_{k}\left({s}_{k}\right)

(30)

Using Assumption 1 it is clear that *g* is a monotonic function of the state variables

g\left({s}_{k}+\mathrm{\Delta},{\mu}_{k,\mathrm{\Delta}}^{*}\right)\le g\left({s}_{k},{\mu}_{k,\mathrm{\Delta}}^{*}\right)

(31)

We consider *φ* (·) as the state update function and define two possible next states {s}_{k+1,\mathrm{\Delta}}^{*}=\phi \left({s}_{k}+\mathrm{\Delta},{\mu}_{k,\mathrm{\Delta}}^{*},{C}_{k+1},{B}_{k}\right) and {s}_{k+1}^{\#}=\phi \left({s}_{k},{\mu}_{k,\mathrm{\Delta}}^{*},{C}_{k+1},{B}_{k}\right). For known values of *C*_{k+1}and *B*_{
k
} we can use (1)-(3) and easily show that {s}_{k+1,}^{{*}_{\mathrm{\Delta}}}={s}_{k+1}^{\#}+{\mathrm{\Delta}}^{\prime} in which {\mathrm{\Delta}}^{\prime}=\left({\delta}_{1}^{\prime},{\delta}_{2}^{\prime},-{\delta}_{3}^{\prime},-{\delta}_{4}^{\prime},-{\delta}_{5}^{\prime}\right) for some {\delta}_{i}^{\prime}\ge 0. Thus {J}_{k+1}\left({s}_{k+1,\mathrm{\Delta}}^{*}\right)\le {J}_{k+1}\left({s}_{k+1}^{\#}\right).

We define f\left({C}_{k+1}\right)\triangleq {E}_{B}\left[{J}_{k+1}\left({s}_{k+1,\mathrm{\Delta}}^{*}\right)\right] and g\left({C}_{k+1}\right)\triangleq {E}_{B}\left[{J}_{k+1}\left({s}_{k+1}^{\#}\right)\right] since *B* is independent of the system state thus we have *f*(*C*_{k+1}) ≤ *g* (*C*_{k+1}) and since *J*_{k+1}(·) is an increasing function of *C*, then *f*(·) and *g*(·) are increasing functions. Applying Lemma 1 with {\u0108}_{{a}^{-}}=\left({C}_{k-1}-{\delta}_{5}\right), {\u0108}_{a}=\left({C}_{k}-{\delta}_{4}\right), {\u0108}_{{b}^{-}}={C}_{k-1}, and {\u0108}_{b}={C}_{k} we find that

{E}_{{C}^{+}}\left\{{E}_{B}\left[{J}_{K+1}\left({s}_{k+1,\mathrm{\Delta}}^{*}\right)\right]\right\}\le {E}_{{C}^{+}}\left\{{E}_{B}\left[{J}_{k+1}\left({s}_{k+1}^{\#}\right)\right]\right\}

(32)

Combining (31), (32) and considering definition of *G*_{
k
} in (29) we get

{J}_{k}\left({s}_{k}+\mathrm{\Delta}\right)={G}_{k}\left({s}_{k}+\mathrm{\Delta},{\mu}_{k,\mathrm{\Delta}}^{*}\right)\le {G}_{k}\left({s}_{k},{\mu}_{k,\mathrm{\Delta}}^{*}\right)

(33)

Equation (30) together with (33) prove the theorem by showing: *J*_{
k
} (*s*_{
k
} + Δ) ≤ *J*_{
k
} (*s*_{
k
} ).

■

Assuming *uncorrelated channel model* the following theorem indicates the "*threshold* *structure*" of the optimal transmission policy versus the channel state.

**Theorem 2:** If the optimal access decision in state {s}_{k}=\left({q}_{k},{\stackrel{\u0304}{q}}_{k},{\stackrel{\u0304}{r}}_{k},{C}_{k}\right) is {I}_{k}^{*}\left({s}_{k}\right)=1, then for another possible state {s}_{k}^{\prime}=\left({q}_{k},{\stackrel{\u0304}{q}}_{k},{\stackrel{\u0304}{r}}_{k},{C}_{k}^{\prime}\right) in the same slot with improved channel state {C}_{k}^{\prime}\ge {C}_{k} we have {I}_{k}^{*}\left({s}_{k}^{\prime}\right)=1.

*Proof:* Assume {\mu}_{k}^{*}\left({s}_{k}\right)=\left({I}_{k}=1,{r}_{k}\right) but {\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)=\left({I}_{k}^{\prime}=0,{r}_{k}^{\prime}\right) as the optimal decision for *s*_{
k
} and {s}_{k}^{\prime}, respectively. According to the definition of *G*_{
k
} in the proof of Theorem 1, {\mu}_{k}^{*}\left({s}_{k}\right) maximizes *G*_{
k
} (*s*_{
k
} , *μ*_{
k
} ) and we have

{G}_{k}\left({s}_{k},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right)\le {G}_{k}\left({s}_{k},{\mu}_{k}^{*}\left({s}_{k}\right)\right)

(34)

On the other hand since *s*_{
k
} and {s}_{k}^{\prime} differs only in the channel state, we have g\left({s}_{k},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right)=g\left({s}_{k}^{\prime},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right) and by using \left({I}_{k}^{\prime}=0,{r}_{k}^{\prime}\right) for both states, queue size will modify similarly for *s*_{
k
} , and {s}_{k}^{\prime} which results in the same next state, {s}_{k+1}={s}_{k+1}^{\prime}. Also for uncorrelated channel model the averaging over next channel state does not depend on the current state, thus {E}_{{C}^{+}}\left\{{E}_{B}\left[{J}_{k+1}\left({s}_{k+1}\right)\right]\right\}={E}_{{C}^{+}}\left\{{E}_{B}\left[{J}_{k+1}\left({s}_{k+1}^{\prime}\right)\right]\right\} and

{G}_{k}\left({s}_{k},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right)={G}_{k}\left({s}_{k}^{\prime},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right)

(35)

By applying decision {\mu}_{k}^{*}\left({s}_{k}\right)=\left({I}_{k}=1,{r}_{k}\right) and since transmission with better channel state will decrease *q* and \stackrel{\u0304}{q} which increase *J*_{
k
} according to Theorem 1, we have

{G}_{k}\left({s}_{k},{\mu}_{k}^{*}\left({s}_{k}\right)\right)\le {G}_{k}\left({s}_{k}^{\prime},{\mu}_{k}^{*}\left({s}_{k}\right)\right)

(36)

Combining (34), (35), and (36) results in

{G}_{k}\left({s}_{k}^{\prime},{\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)\right)\le {G}_{k}\left({s}_{k}^{\prime},{\mu}_{k}^{*}\left({s}_{k}\right)\right)

which is in contrast to optimality of the {\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)=\left({I}_{k}^{\prime}=0,{r}_{k}^{\prime}\right). Thus, we should have {\mu}_{k}^{*}\left({s}_{k}^{\prime}\right)=\left({I}_{k}^{\prime}=1,{r}_{k}^{\prime}\right).

■

Note that Theorem 2 may be incorrect when channel state is time correlated. For example, consider two possible channel states *C*_{
k
} and {C}_{k}^{\prime}, with {C}_{k}<{C}_{k}^{\prime} and assume that optimal decision is to transmit for a state with *C*_{
k
} . Also assume that probability of going from {C}_{k}^{\prime} to a better channel state and from *C*_{
k
} to a worse channel state is high. So, we can argue heuristically that in this condition it may be optimal to transmit data when channel is in state *C*_{
k
} but not to transmit when it is in state {C}_{k}^{\prime}.

### 5.2. Structural properties of ITH solution

We provide structural properties of ITH solution in this section through the following theorems. First we show that relative value function, *h**(*s*), is a monotonic function in Theorem 3 and then prove the threshold structure of access decision versus channel state in Theorem 4.

**Theorem 3:** *h**(*s*), is a decreasing function of *q* and \stackrel{\u0304}{q}, and an increasing function of \stackrel{\u0304}{r} and *C*.

*Proof:* We define Δ = (*δ*_{1}, *δ*_{2}, -*δ*_{3}, -*δ*_{4}, -*δ*_{5}) with *δ*_{
i
} ≥ 0 and show that *h**(*s* + Δ) ≤ *h**(*s*). We also define *G*_{
f
} (*s*, *μ*(*s*)) on function *f* as

{G}_{f}\left(s,\mu \right)=g\left(s,\mu \right)+\tau E\left[f\left({s}^{+}\right)\right]

(37)

Assuming *μ** as the decision that maximizes *G*_{
f
} and according to the Bellman equation (24) we have

{\mathcal{B}}_{\tau}f\left(s\right)={G}_{f}\left(s,{\mu}^{*}\right)

Taking into account *h**(*s*) = lim_{k→∞}*h*^{(n)}(*s*), we prove through induction for every iteration *n*, *h*^{(n)}(*s* + Δ) ≤ *h*^{(n)}(*s*). For *n* = 0 we define *h*^{(0)}(*s*) = *g*(*s*, *I* = 0, *r* = 0) which according to Assumption 1 it is clear that *h*^{(0)} (*s* + Δ) ≤ *h*^{(0)} (*s*). We assume that *h*^{(n)}(*s*) is monotonic and show *h*^{(n+1)}(*s*) is also monotonic. First, we show that B _{
τ
}*h*^{(n)}(*s*) is monotonic. Using (26) for states *s* and *s* + Δ and assuming *μ** and {\mu}_{\mathrm{\Delta}}^{*}, respectively, as maximizing actions we have

{{G}_{h}}^{\left(n\right)}\left(s,{\mu}_{\mathrm{\Delta}}^{*}\right)\le {\mathcal{B}}_{\tau}{h}^{\left(n\right)}\left(s\right)={{G}_{h}}^{\left(n\right)}\left(s,{\mu}^{*}\right)

(38)

From definition of *g* it is clear that g\left(s+\mathrm{\Delta},{\mu}_{\mathrm{\Delta}}^{*}\right)\le g\left(s,{\mu}_{\mathrm{\Delta}}^{*}\right). We can use the similar approach as the proof of Theorem 1 and apply Lemma 1 to show that

{E}_{C}+\left\{{E}_{B}\left[{h}^{\left(n\right)}\left({\left(s+\mathrm{\Delta}\right)}^{+}\right)|{\mu}_{\mathrm{\Delta}}^{*}\right]\right\}\le {E}_{C}+\left\{{E}_{B}\left[{h}^{\left(n\right)}\left({s}^{+}\right)|{\mu}_{\mathrm{\Delta}}^{*}\right]\right\}

(39)

which results in

{\mathcal{B}}_{\tau}{h}^{\left(n\right)}\left(s+\mathrm{\Delta}\right)={G}_{h}\left(n\right)\left(s+\mathrm{\Delta},{\mu}_{\mathrm{\Delta}}^{*}\right)\le {G}_{h}\left(n\right)\left(s,{\mu}_{\mathrm{\Delta}}^{*}\right)

(40)

Combining (38) and (40) we find that B _{
τ
}*h*^{(n)}(*s* + Δ) ≤ B _{
τ
}*h*^{(n)}(*s*). Using Equation (25) and taking into account that B _{
τ
}*h*^{(n)}(*s*') is independent of the state vector, it can be easily shown that *h*^{(n + 1)}(*s*) is a monotonic function.

■

Assuming uncorrelated channel model the following theorem indicates existence of a threshold for channel state that the link should decide to transmit when channel state is better than or equal to that threshold.

**Theorem 4:** There exists a threshold, *C*_{th}, that for {s}_{\mathsf{\text{th}}}=\left(q={C}_{\mathsf{\text{th}}},\stackrel{\u0304}{q},\stackrel{\u0304}{r},{C}_{\mathsf{\text{th}}}\right) with any \stackrel{\u0304}{q} and \stackrel{\u0304}{r}, we have *I**(*s*_{
th
} ) = 1. Also for any *s* with *C* ≥ *C*_{th} and *q* ≥ *C*_{th} we have *I**(*s*) = 1.

*Proof:* Assume in timeslot *k* we have *C*_{
k
} = *C*^{max} and *q*_{
k
} = *C*^{max}, transmission at this time has the energy cost of *β e* but it will reduce *q* by *C*^{max} which will reduce \stackrel{\u0304}{q} by *θ*_{
q
}*C*^{max} and also will reduce the future costs related to the queue size. However, transmission of theses *C*^{max} packets at any later time slot requires the same amount of energy. Thus, it is better to transmit these packets at state *s*_{th} to reduce the queue size as early as possible and reduce the future costs related to the queue size. We conclude that: "if *C* = *C*^{max} and *q* = *C*^{max} then *I**(*s*) = 1" which proves existence of *C*_{th}.

In order to prove the second part of the theorem we assume s=\left(q\ge {C}_{\mathsf{\text{th}}},\stackrel{\u0304}{q},\stackrel{\u0304}{r},C\ge {C}_{\mathsf{\text{th}}}\right) and consider optimal decisions *μ**(*s*_{th}) = (*I**(*s*_{th}), *r**(*s*_{th})) and *μ**(*s*) = (*I**(*s*), *r**(*s*)) for states *s*_{th} and , respectively. If *I*(*s*) = 0 we can show similar to the proof of Theorem 2 that it cannot be an optimal transmission policy.

■