Dynamic power control for energy harvesting wireless multimedia sensor networks

Abstract

Optimization of energy usage in wireless sensor networks (WSN) has been an active research field for the last decades and various approaches have been explored. In fact, A well designed energy consumption model is the foundation for developing and evaluating a power management scheme in network of energy constrained devices such as: WSN. We are interested in developing optimal centralized power control policies for energy harvesting wireless multimedia sensor networks (WMSN) equipped with photovoltaic cells. We propose a new complete information Markov decision process model to characterize sensor's battery discharge/recharge process and inspect the structural properties of optimal transmit policies.

1 Introduction

The recent technological advances in the fields of micro-electronic, wireless communication along with reduction of production costs have motivated the development of a novel generation of wireless networks. wireless sensor networks (WSN) are articulated over a set of miniaturized battery powered devices (sensors) with communication capabilities and are expected to become highly integrated into our daily activities. This class of networks is perceived as an evolution of AdHoc networks with specific energy and computation limitations. Also, the increasing availability, at low cost, of multimedia devices (cameras, microphones,...) has triggered the emergence of multimedia wireless sensor networks (WMSN) [1, 2]. With the diversity of their application domains, ranging from healthcare and intelligent patient monitoring to disaster relief and industrial process supervision through intrusion detection and border protection, WMSN hold a promising future [3]. It is noteworthy that the volume and the nature of carried multimedia content, mainly composed of images and/or video streams, impose severe requirements on sensor's residual energy and available bandwidth.

The energy scarcity represents one of the major limitations of WMSN, indeed, post-deployment replacement of the sensors batteries is generally not practical or even impossible. Therefore, a proper management strategy of the residual energy happens to be a crucial prerequisite to any large scale WMSN deployment. In order to preserve the sensors energy, an optimal choice of transmit powers and an efficient scavenging mechanism of energy from the deployment environment along with an adequate topological placement of the sensors are necessary.

A variety of topologies have been proposed for WMSN deployment, notably: single-tier flat, single-tier clustered and multi-tier (see Figure 1). Introducing hierarchy in WMSN benefits at various levels: indeed, since sensors forward exclusively packets produced within their cluster, the communication overhead is reduced and consequently the network lifetime is prolonged. Also, resourced devices (multi-processing hubs) realize heavy computations and aggregation of data reported by their cluster sensors, reducing by the way the energy consumption resulting from relaying redundant data.

Figure 1

Hierarchical wireless multimedia network [1].

Energy harvesting [4–7] in the context of WSN received increasing attention from research community. Indeed, enabling sensors to replenish their energy reserves, extends the WMSN's deployment lifetime and enlarges their application domains. Energy harvesting sources are various and encompass solar, wind and vibratory sources. In this work we will focus our attention on sensors equipped with photovoltaic cells that realize solar energy transformation into electric energy needed to recharge their batteries.

Energy consumption optimization remains a crucial issue even for energy harvesting WMSN. In fact, the harvested energy should be exploited optimally to cope with energy sources periodicity (day/night cycles) and unpredictability (wind activity/inactivity periods). Since, most of energy consumption is incurred at transceiver level, a balance should be found between conflicting objectives: maximizing the achieved throughput while reducing energy consumption and consequently extending the sensor's battery lifetime. Optimal energy management policies for energy harvesting sensors are considered in [8]. The discounted throughput is maximized over an infinite horizon, where queuing for data is also considered. In [9], the authors consider a binary power control problem: at each slot the wireless device could either transmit at a constant power or remain silent. The authors consider only the single user case and the optimal transmission policy is shown to be of a threshold type for the soft and strict delay constraint cases. The authors of [10] presented a decentralized power control with stochastic channel variation scheme. The proposed scheme considers a cost function that accounts for the QoS of each user and the interference to other users. The single user optimal policy is generalized to the multiple users scenario for the ergodic regime where the spreading factor and the number of users grow infinitely but their ratio remains constant. In [11], the authors consider a single transmitter-receiver scenario where the transmitter has a finite buffer, and solved the problem of dynamically assigning rates/powers to packets in order to minimize the long-term average transmission energy subject to an upper bound on the buffer overflow probability. The problem is formulated as a constrained Markov decision process (CMDP) and an analytical solution is given and proved to be monotone in queue length. The authors of [12] use an evolutionary game theoretic formulation to characterize the equilibrium policy for power allocation under channel un-certainty and delayed imperfect payoffs. A heterogonous learning framework that accounts for user and technology specificities is proposed. The authors of [13] address the problem of network resource allocation for energy-harvesting sensor platforms with time-varying battery recharging rates. They propose a joint approach that combines QuickFix for getting the optimal sampling rate and SnapIt that adapts the sampling rate with the objective of maintaining the battery at a target level. The considered networks are characterized by a special directed acyclic network graph (DAG) structure and the choice of a given rate implies a specific transmit power. In [14], an energy harvesting body sensor network formed by sensors with a two-state energy harvester device is considered. The authors develop policies based on the energy-error probability tradeoff to maximize successful transmission probability while minimizing probability of running out of energy. The developed strategies exploit the knowledge of the current energy level and the process governing event generation and battery recharge to select the appropriate transmission mode. The problem of throughput optimal energy allocation is studied for energy harvesting systems in a time constrained slotted setting in [15]. The structural properties of the optimal power allocation policy are obtained through dynamic programming and convex optimization. The optimal use of the harvested energy for different energy profiles and storage capabilities is discussed in [16] with the outage probability considered as a performance metric. The authors developed a discrete time Markov model of the evolution of battery and transmission state and provide optimal transmission strategy that minimizes outage probability.

Our objective is to define an optimal energy management policy for the centralized dynamic power control problem for energy harvesting WMSN. Thus, at each time slot, the transmission powers of all the sensors are fixed by the base station to maximize the expected system's throughput under minimum energy consumption.

The added value of our work covers the following points:

• We formalize the centralized power control problem in the context of hierarchical energy harvesting WMSN as a complete information MDP.

• A stochastic model for the discharge/recharge battery process for energy harvesting WMSN is provided.

• We consider a novel utility function that balances the sojourn in a each battery state along with achieved throughput for a given transmit policy.

• The structural properties of centrally computed optimal transmit policies are inspected.

This article is organized as follows: In Section 2, we give a mathematical formulation for dynamic power control problem in the context of energy harvesting WMSN. Optimal transmission policies are treated in Sections 3 and 4 for the single and multiple sensors scenarios. Then we present numerical simulation results in Section 5. Finally, we conclude the article and announce our future works in Section 6.

2 System model

We consider a WMSN formed by a set of sensors $\mathcal{N}=\left\{1,\dots ,n\right\}$, under the authority of a single base station or gateway. Each sensor is equipped by its own battery and uses power to communicate with the base station either directly or through multimedia processing hubs. We discretize each sensor's battery capacity to several intervals that give a more coarse-grained description of the battery state, e.g., full, medium, discharged. The sensors are equipped with photovoltaic cells that make them capable of harvesting solar energy while undergoing the discharge process. The residual energy of the battery dictates available transmission powers and the achieved throughput is affected by the chosen transmit powers of other sensors. Time is discrete and at each time slot t, each sensor knows its own battery state, whereas the battery state of the other sensors and their actions remain unknown.

The formulated problem fits within the MDP framework with full information and infinite planning horizon, where the base station will compute and provide each sensor with its optimal transmission strategy, given the fact that it has access to all sensors' information i.e environment, battery and radio channel status. In order to be aware of each sensor battery level, we assume that time is slotted into virtual slots that encompasses several physical slots. the initial physical slots will be affected to the sensors to communicate their battery levels to the base station in a TDMA fashion. The remaining slots will serve for data exchange with interference possibility. We use three bits to code the energy level of each battery and thus, we could represent up to eight battery states.

2.1 Mathematical formulation

Denote by $w={\left\{w_t\right\}}_{t\ge t_0}$ the set of environment states at each time slot post t₀. Thus, the power allocation dynamic problem for a sensor j could be modeled by an MDP:

$$\Omega =\left\{{\mathcal{X}}^j,{\left(A^j\left(w,s_j\right)\right)}_{s_j\in {\mathcal{X}}^j,j\in \mathcal{N},w},q^j,\lambda \right\}$$

(1)

Where:

• ${\mathcal{X}}^j=\left\{0,1,2,\dots ,m-1\right\}$ is a finite set of states of sensor's j battery. A state of the battery represents some interval of percentage of the remaining energy. The energy level of the battery increases (respectively decreases) sequentially. Initially, the sensor's battery is in its highest state (m - 1), as sensors perform their affected tasks (event detection, packets forwarding,...) they consume energy and their batteries energy levels decrease sequentially. The harvested solar energy will be converted into electrical energy and will increase the sensors batteries residual energy level sequentially.

• $\forall s_{j,t}\in {\mathcal{X}}^j,A^j\left(w_t,s_{j,t}\right)=\left\{p_0,\dots ,p_{s_{j,t}}\right\}$ is a finite set of available transmission powers for sensor j. This set satisfies A^j(w _t , s _{j, t} - 1) ⊂ A^j(w _t , s _{j, t} ): more powers are available at higher states. A sensor makes a decision on its transmit power $p_t^j\in A^j\left(w_t,s_{j,t}\right)$ based on its remaining energy s _{j, t} at time t.

• q^jis the state transition probability of sensor's j battery. Given the state of environment w _t , the state s _{j, t} of the battery of j, the state c _{j, t} of the radio channel in the vicinity of j, and the actions of the others $p_t^{-j}=\left(p_t^1,p_t^2,\dots ,p_t^{j-1},p_t^{j+1},\dots ,p_t^n\right)$ the new states are (s_{j, t+1}, c_{j, t+1}) with the probability $q^j\left(\left(s_{j,t+1},c_{j,t+1}\right)|\left(s_{j,t},c_{j,t}\right),p_t^j,p_t^{-j}\right)$.

• The discount factor λ indicates for a user the decay in the gain value with the evolution of the time.

Let η _t = (w _t , s_{1, t}, c_{1, t},..., s _{n, t} , c _{n, t} ) be the state profile of all the system at time t. The action profile of the system is: $p_t=\left(p_t^1,\dots ,p_t^n\right)$, where $\forall j\in \mathcal{N},p_t^j\in A^j\left(w_t,s_{j,t}\right)$. When w _t = 0, the SINR of sensor j is null. For w _t ≠0, the SINR of sensor j is given by:

$$\text{SIN}{\text{R}}_j\left({\eta }_t,p_t\right)=\frac{p_t^j{h}_j\left(w_t,s_{j,t},c_{j,t}\right)}{N_0+{\sum }_{i\ne j}{p}_t^i{h}_i\left(w_t,s_{i,t},c_{i,t}\right)}$$

(2)

where $h_j\left(w_t,s_{j,t},c_{j,t}\right)p_t^j$ represents the power received at the base station or the multimedia processing hub given that states are s _{j, t} (respectively c _{j, t} ) for the battery (respectively the radio channel in the vicinity) of sensor $j.p_t^j$ is the power level chosen by j and h _j (w _t , s _{j, t} , c _{j, t} ) is a function of the channel state and others exogenous characteristics, N₀ is the variance of the noise. The throughput of sensor j at time t is an increasing function of the SINR _j (η _t , p _t ).

$$\left\{\begin{array}{c}\text{Th}{\text{p}}_j\left({\eta }_t,p_t\right)=f^j\left(\text{SIN}{\text{R}}_j\left({\eta }_t,p_t\right)\right)\hfill \\ f^j\left(0\right)=0\hfill \end{array}\right.$$

(3)

In the rest of the article, we consider f ^j to be Shannon capacity [17] and thus:

$$f^j\left(\text{SIN}{\text{R}}_j\left({\eta }_t,p_t\right)\right)=\text{log}\left(1+\text{SIN}{\text{R}}_j\left({\eta }_t,p_t\right)\right)$$

(4)

2.2 Stochastic battery model for sensors with energy harvesting capabilities

The sensors harvest energy through a photovoltaic cell and use the scavenged power to recharge their batteries. Thus, the sensor battery will move from state s _{j, t} to state s _{j, t} +1 with probability $q^j\left(s_{j,t}+1|s_{j,t},p_t^j\right)=p_{\text{harvest}}$. The transmit power choice of j determines the transitions to the next state. Thus, when the sensor j selects a transmit power $p_t^j\in A^j\left(w_t,s_{j,t}\right)$, the new state of the battery is s_{j, t+1}with probability $q^j\left(s_{j,t+1}|s_{j,t},p_t^j\right),q^j\left(s_{j,t+1}|s_{j,t},p_t^j\right)=0$ if s_{j, t+1 ∉ {}s _{j, t} + 1, s _{j, t} , s _{j, t} -1_}. If the energy harvesting process is frozen for a long period of time (due to cloudy weather for example), the state 0 is reached (the battery is completely empty) and the sensor is considered to be out of service. Figure 2 shows the state transition probabilities of sensor's j battery.

Figure 2

Stochastic model of the sensor's battery discharge/recharge process.

The probability to move to the lower adjacent state increases with the energy consumption i.e $\forall p_t^j,p_t^{\prime j}\in A^j\left(w_t,s_{j,t}\right)$:

$$p_t^j>p_t^{\prime j}\Rightarrow q^j\left(s_{j,t}-1|s_{j,t},p_t^j\right)>q^j\left(s_{j,t}-1|s_{j,t},p_t^{\prime j}\right)$$

(5)

At each time slot t, depending on the remaining energy off the battery, the sensor j makes a decision on its transmit power $p_t^j$. Denote by the policy $d_j^{\infty }=\left(d_0^j,d_1^j,\dots ,d_{m-1}^j\right)$ the collection of decision rules of that sensor under infinite planning horizon, where $\forall s_i\in {\mathcal{X}}^j,d_i^j\in \Delta \left(A^j\left(s_i\right)\right)$ and Δ(A^j (s _i )) stands for the space of probability distribution over sensor's j action space for the battery state s _i .

The sojourn time $T^j\left(l,d_j^{\infty }\right)$ in the state l under transmit policy $d_j^{\infty }$ can be expressed as:

$$\begin{array}{cc}\hfill T^j\left(l,d_j^{\infty },t_0\right)& =\underset{t}{\text{arg}\text{min}}\left\{t>t_0|s_{j,t}=l-1,s_{j,t_0}=l,d_j^{\infty }\right\}\hfill \\ =1+q^j\left(l|l,p_{t_0}^j\right)T^j\left(l,d_j^{\infty },t_0+1\right).\hfill \end{array}$$

When $p_t^j$ depends only on the state of the battery but not on the time (stationary policy), the sojourn becomes:

$$T^j\left(l,d_j^{\infty }\right)=\frac{1}{q^j\left(l-1|l,d_j^{\infty }\right)+q^j\left(l+1|l,d_j^{\infty }\right)}$$

(6)

3 Optimal policy for a single sensor

When considering a single sensor, the interferences are omitted from the SINR expression that becomes equivalent to the signal to noise ratio (SNR): $\text{SN}{\text{R}}_j\left({\eta }_t,p_t\right)=\frac{p_t^j{h}_j\left(w_t,s_{j,t},c_{j,t}\right)}{N_0}$ and the sensor receives an immediate reward $r_t\left(s_{j,t},p_t^j\right)$ for choosing transmit power $p_t^j$ at slot t. The immediate reward reflects a balance between maximizing the expected sojourn in a given battery state and the corresponding achieved throughput for the chosen transmit power:

$$r_t\left(s_{j,t},p_t^j\right)=\left(1-\frac{1}{T^j\left(s_{j,t},p_t^j\right)}\right)\text{log}\left(1+\frac{p_t^j{h}^j}{N_0}\right)$$

(7)

Let ind: ${\mathcal{X}}^j\to ℝ$ be a non-decreasing function on ${\mathcal{X}}^j$. We consider transition probabilities that adhere to the following expression:

$$q^j\left(s_{j,t}^{\prime }|s_{j,t},p_t^j\right)=\left\{\begin{array}{cc}1-\frac{p_t^j}{pw\left(s_{j,t}\right)}-\frac{P_{\text{harvest}}}{\text{ind}\left(s_{j,t}\right)}\hfill & s_{j,t}^{\prime }=s_{j,t}\hfill \\ \frac{p_t^j}{pw\left(s_{j,t}\right)}-\frac{\left(\text{ind}\left(s_{j,t}\right)-1\right)p_{\text{harvest}}}{\text{ind}\left(s_{j,t}\right)}\hfill & s_{{}_{j,t}}^{\prime }=s_{j,t}-1\hfill \\ {}_{P_{\text{harvest}}}\hfill & s_{j,t}^{\prime }=s_{j,t}+1\hfill \\ {}_0\hfill & \text{else}\text{.}\hfill \end{array}\right.$$

(8)

In the discounted total reward problem the gains on the first stages are more important than the future ones. In particular, a gain acquired at time n is assumed to have a present value λⁿ r(s _j , p _j ) where 0 < λ < 1 is a discount factor. Under an infinite planning horizon and due to its elegant theory, the ease in which it allows inclusion of constraints, and its facility for sensitivity analysis, linear programming formulation [18] is adequate to solve our MDP. We randomly choose a set of real constants ${\left\{\alpha \left(s_j\right)\right\}}_{s_j\in {\mathcal{X}}^j}$ to be a distribution probability over the states of sensor's j battery. Therefore, the set of α (s _j ) should respect the following constraint: ${\sum }_{s_j\in {\mathcal{X}}^j}\alpha \left(s_j\right)=1$. We also consider for every state s _j and available action P^j∈A^j (s _j ) the linear program variable $x\left(s_j,p^j\right)={\sum }_{n=0}^{\infty }{\lambda }^n\text{Prob}\left(s_j,p^j\right)$ that indicates the expected discounted time of the sensor's battery being in state s _j and making decision p^j. The linear program equivalent to the λ discounted MDP Ω is described by:

$$\left\{\begin{array}{c}\text{Maximize}\sum _{s_j\in {\mathcal{X}}^j}\sum _{p^j\in A\left(s_j\right)}r\left(s_j,p^j\right)x\left(s_j,p^j\right)\hfill \\ \text{Subject}\text{to}\hfill \\ {\sum }_{p^j\in A^j\left(s_j\right)}x\left(s_j^{\prime },p^j\right)-\sum _{s_j\in {\mathcal{X}}^j}\sum _{p^j\in A^j\left(s_j\right)}\lambda q^j\left(s_j^{\prime }|s_j,p^j\right)x\left(s_j,p^j\right)=\alpha \left(s_j^{\prime }\right)\hfill \\ \forall s_j\in {\mathcal{X}}^j,\forall p^j\in A^j\left(s_j\right),x\left(s_j,p^j\right)\ge 0\hfill \\ \sum _{{s^{\prime }}_j\in {\mathcal{X}}^j}\alpha \left(s_j^{\prime }\right)=1\hfill \\ 0<\lambda <1.\hfill \end{array}\right.$$

(9)

The solution of this linear program (LP) is obtained through application of the simplex method. After solving the LP above, we recover the optimal decision rules of the associated MDP by applying the rule [19]:

• A discounted MDP has always a deterministic optimal decision rule [20] that we select based on the following criterion:

  $$\forall p^l\in A^j\left(s_k\right),d_k^j=p^l\Rightarrow x\left(s_k,p^l\right)>0,s_k\in {\mathcal{X}}^j.$$

  (10)

We argue that there exists structured decision rules for the MDP Ω.

Proposition 1. There exists optimal monotone non-decreasing decision rules on ${\mathcal{X}}^j$for the MDP Ω.

The detailed proof of Proposition 1 is given below:

Proof. Let $Q\left(k,l,p_t^j\right)={\sum }_{i=k}^{i=m-1}{q}^j\left(s_{i,t}|s_{l,t},p_t^j\right)$, and $p_t^j,p_t^{\prime j}$ two transmit powers such as $p_t^j<p_t^{\prime j}$:

• (1) The immediate reward r^j for sensor j is non-decreasing on ${\mathcal{X}}^j$ for all $p_t^{\prime j}\in A^j$.

• (2) The immediate reward r^j for sensor j is superadditive.

  $$\begin{array}{c}r\left(s_{l,t}+1,p_t^{\prime j}\right)+r\left(s_{l,t},p_t^j\right)-r\left(s_{l,t}+1,p_t^j\right)-r\left(s_{l,t},p_t^{\prime j}\right)=\hfill \\ \left(\frac{1}{T^j\left(s_{l,t},p_t^{\prime j}\right)}-\frac{1}{T^j\left(s_{l,t}+1,p_t^{\prime j}\right)}\right)\text{log}\left(1+\frac{p_t^{\prime j}{h}^j}{N_0}\right)+\left(\frac{1}{T^j\left(s_{l,t}+1,p_t^j\right)}-\frac{1}{T^j\left(s_{l,t},p_t^j\right)}\right)\text{log}\left(1+\frac{p_t^j{h}^j}{N_0}\right)\hfill \\ =g\left(p_t^{\prime j}\right)-g\left(p_t^j\right)\hfill \end{array}$$

Where $g:p_t^j\mapsto \frac{\text{pw}\left(s_{l,t}+1\right)-\text{pw}\left(s_{l,t}\right)}{\text{pw}\left(s_{l,t}\right)\text{pw}\left(s_{l,t}+1\right)}{p}_t^j+\frac{\text{ind}\left(s_{l,t}+\text{l}\right)-\text{ind}\left(s_{l,t}\right)}{\text{ind}\left(s_{l,t}\right)\text{ind}\left(s_{l,t}+1\right)}{p}_{\text{harvest}}$. Since the functions pw and ind are monotone non-decreasing we conclude that r is a superadditive function.

• (3) $Q\left(k,l,p_t^j\right)$ is non-decreasing on ${\mathcal{X}}^j:\text{ Let }\left(s_k,s_l\right)\in {\mathcal{X}}^j\times {\mathcal{X}}^j$,

  $$\begin{array}{c}{s}_k>s_l+1:\Delta Q\left(k,l,p_t^j\right)=Q\left(k,l+1,p_t^j\right)-Q\left(k,l,p_t^j\right)=Q\left(k,l+1,p_t^j\right)=p_{\text{harvest}}\ge 0.\hfill \\ s_k=s_l+1:\hfill \end{array}$$
  $$\begin{array}{cc}\hfill \Delta Q\left(k,l,p_t^j\right)& =Q\left(k,l+1,p_t^j\right)-Q\left(k,l,p_t^j\right)\hfill \\ =\sum _{i=k}^{i=m-1}{q}^j\left(s_{i,t}|s_{l,t}+1,p_t^j\right)-q^j\left(s_{i,t}|s_{l,t},p_t^j\right)\hfill \\ =q^j\left(s_{l,t}+1|s_{l,t}+1\right)+p_{\text{harvest}}-p_{\text{harvest}}\ge 0.\hfill \end{array}$$

$s_k=s_l:q^j\left(s_{l,t}|s_{l,t},p_t^j\right)$ is a monotone non-decreasing function on X^j , thus:

$$\begin{array}{cc}\hfill \Delta Q\left(k,l,p_t^j\right)& =\sum _{i=k}^{i=m-1}{q}^j\left(s_{i,t}|s_{l,t}+1,p_t^j\right)-q^j\left(s_{i,t}|s_{l,t},p_t^j\right)\hfill \\ =q^j\left(s_{l,t}+1|s_{l,t}+1,p_t^j\right)+q^j\left(s_{l,t}|s_{l,t}+1,p_t^j\right)+p_{\text{harvest}}-q^j\left(s_{l,t}|s_{l,t},p_t^j\right)-p_{\text{harvest}}\hfill \\ \ge q^j\left(s_{l,t}|s_{l,t}+1,p_t^j\right)\ge 0.\hfill \end{array}$$

s _k ≤ s _l - 1:

$$\begin{array}{cc}\hfill \Delta Q\left(k,l,p_t^j\right)& =\sum _{i=k}^{i=m-1}{q}^j\left(s_{i,t}|s_{l,t}+1,p_t^j\right)-q^j\left(s_{i,t}|s_{l,t},p_t^j\right)\hfill \\ =1-1=0.\hfill \end{array}$$

• (4) $Q\left(k,l,p_t^j\right)$ is globally superadditive on ${\mathcal{X}}^j\times A^j$, denote by $\vartheta =Q\left(k,l+1,p_t^{\prime j}\right)+Q\left(k,l,p_t^j\right)-Q\left(k,l+1,p_t^j\right)-Q\left(k,l,p_t^{\prime j}\right)$:

s _k > s _l + 1: ϑ = P_harvest - P_harvest = 0.

s _k = s _l + 1:

$$\begin{array}{cc}\hfill \vartheta & =p_{\text{harvest}}+q^j\left(s_{l+1}|{s_l}_{+1},p_t^{\prime j}\right)+p_{\text{harvest}}-q^j\left(s_{l+1}|s_{l+1},p_t^j\right)-p_{\text{harvest}}-p_{\text{harvest}}\hfill \\ \le 0.\hfill \end{array}$$

$$\begin{array}{c}{s}_k=s_l:\vartheta =1+q^j\left(s_l|s_l,p_t^j\right)+p_{\text{harvest}}-1-q^j\left(s_l|s_l,p_t^{\prime j}\right)-p_{\text{harvest}}\ge 0\hfill \\ s_k\le s_l-1:\vartheta =1-1=0.\hfill \end{array}$$

We conclude by virtue of Theorem 6.11.6 of [20] that deterministic optimal monotone non-decreasing policies over the set X^j exists for the MDP (1).

4 Optimality for all the sensors batteries lifetime

Under the assumption that each sensor uses CDMA with mutual orthogonal codes to communicate with the base station or the multimedia processing hubs, sensors transmissions do not interfere. Therefore, the overall policy $\left(d_1^{\infty *},\dots ,d_n^{\infty *}\right)$ realized when every sensor adopts its optimal transmit power happens to be the optimal transmit policy for the overall system noted d^∞*.

For the general case: non-orthogonal codes are used and sensors transmissions do interfere. The WMSN is modeled by the MDP: ${\Omega }^{+}=\left\{\mathcal{X},{\left(A\left(w_t,S_t\right)\right)}_{S_t\in \mathcal{X},w},Q,\lambda \right\}$ with full information. We extend the previously formulated mathematical model to account for multiple sensors and denote the augmented states and actions spaces respectively $\mathcal{X}={\prod }_{k=1}^n{\mathcal{X}}^k$ and $\mathbf{A}={\prod }_{k=1}^n{A}^k$. The transition probability from state S _t to S_t+1for the power profile P _t is given by the formula:

$$Q\left(S_{t+1}|S_t,P_t\right)=\prod _{k=1}^{n}q\left(s_{k,t+1}|s_{k,t},p_t^k\right),\left(s_{k,t+1}|s_{k,t}\right)\in {\mathcal{X}}^k\times {\mathcal{X}}^k.$$

(11)

With each sensor utility expressed as follows:

$$r_t\left(s_{j,t},p_t^j\right)=\left(1-\frac{1}{T^j\left(s_{j,t},p_t^j\right)}\right)\times \text{log}\left(1+\frac{p_t^j{h}^j}{N_0+{\sum }_{k\ne j}{p}_t^k{h}^k}\right)$$

(12)

The immediate reward for the network becomes:

$$R_t\left(S_t,P_t\right)=\sum _{j=1}^n{r}_t\left(s_{j,t},p_t^j\right)$$

(13)

We reconsider the LP in (9) for the augmented MDP:

$$\left\{\begin{array}{c}\text{Maximize}\sum _{S\in \mathcal{X}}\sum _{a\in A\left(S\right)}R\left(S,a\right)x\left(S,a\right)\hfill \\ \text{Subject}\text{to}\hfill \\ \sum _{a\in A\left(S\right)}x\left(S^{\prime },a\right)-\sum _{S\in \mathcal{X}}\sum _{a\in A\left(S\right)}\lambda Q\left(S^{\prime }|S,a\right)x\left(S,a\right)=\alpha \left(S^{\prime }\right).\hfill \\ \forall S\in \mathcal{X},\forall a\in A\left(S\right),x\left(S,a\right)\ge 0\hfill \\ \sum _{S^{\prime }\in \mathcal{X}}\alpha \left(S^{\prime }\right)=1\hfill \\ 0<\lambda <1.\hfill \end{array}\right.$$

5 Numerical investigations

We discretize each battery residual energy capacity to five states: near full, high, medium, low and discharged. The states set is: {0, 1, 2, 3, 4} and the transmit power panel for each state is detailed in Table 1:

Table 1 Transmit power panel

Our objective is to characterize the optimal transmit policy for a single sensor with a discount factor λ = 0.6. We solve the associated LP through the simplex algorithm to obtain the optimal policy summarized below:

$${\sigma }_j^{*}=\left(0\to P_0,1\to P_0,2\to P_2,3\to P_3,4\to P_4\right).$$

Table 2 describes the optimal transmit policy for a network formed by three sensor with three states {0, 1, 2} for a λ = 0.6 discounted Ω⁺ under infinite horizon planning. We notice that sensors tend to use their lowest available transmission powers as using higher ones result in reduced throughput due to interference and rapid depletion of their batteries.

Table 2 Optimal transmit power

6 Concluding remarks

In this article we considered the problem of dynamic centralized power allocation for energy harvesting WMSN. We focus on solar powered sensors and provide a stochastic model for the associated battery discharge/recharge process. The dynamic power control problem was formulated as a MDP and the structural properties of optimal transmission policies established. We plan, in a near future, to generalize our approach for the decentralized case with partial channel information using stochastic game theory.