Energy-efficient proactive edge caching with sleep scheduling for green networks

Abstract

Proactive content caching in small cells (SCs) (i.e., proactive edge caching) can significantly reduce energy consumption of networks. However, the sleep mode of SCs can make the cached contents in SCs unavailable. Therefore, a joint optimization of edge caching strategy and sleep scheduling should be conducted to maximize the effectiveness of edge caching. In this paper, we propose an energy-efficient proactive edge caching with sleep scheduling (E3CS) where the controller jointly adjusts the caching strategy and the sleep scheduling of SCs by considering the content popularity dynamics. To optimize the performance of E3CS (i.e., minimize the overall energy consumption), we formulate a Markov decision process (MDP) and the joint optimal policy on the caching strategy and sleep scheduling is obtained by a value iteration algorithm. Evaluation results demonstrate that E3CS with the optimal policy outperforms the comparison schemes in terms of the overall energy consumption.

1 Introduction

Recent report [1] predicts that mobile data traffic will grow at a compound annual growth rate of 47% from 2016 to 2021. In addition, it shows that most Internet traffic is governed by content retrieval applications such as content sharing and video on demand (VOD) services. In these applications, users retrieve contents from a remote content server, which causes a significant amount of traffic and energy consumption. To alleviate this problem, the content caching where contents are stored in intermediate nodes of networks has received a high attention by lots of network operators. By amplifying the caching efficiency, contents can be proactively stored on the network edge (e.g., small cells (SCs)) [2–5].¹

Specifically, the energy consumption for content transmission over the backhaul link can be significantly reduced by means of proactive edge caching in SCs. Meanwhile, SCs usually use the sleep mode to reduce the maintenance energy needed to be active. However, it can make the cached contents in SCs (in the sleep mode) unavailable. In this case, the contents should be delivered through the backhaul link from the remote content server even though the contents are stored in SCs, which increases the energy consumption. Therefore, the caching strategy and sleep scheduling should be jointly adjusted to reduce the overall energy consumption. However, only few works consider simultaneously both the caching strategy and sleep scheduling [6, 7].

The popularity of contents changes dynamically with temporal patterns [8]. Specifically, contents on micro-blogging platform such as Facebook and Twitter are very volatile, and they become popular and fade away in a matter of minutes. Moreover, their popularity can exhibit a variety of patterns [9]. Obviously, when the popular contents are cached, the utility of the cached contents (e.g., hit ratio) increases. Meanwhile, the content popularity dynamics have strong correlation between short-term popularity and long-term popularity, and thus, it can be well-estimated by several methods [10–15]. However, most of existing cache systems are not designed by considering the content popularity dynamics, which is reasonable only when the cache churn time is small compared with the popularity dynamics.

2 Methods

In this paper, we propose an energy-efficient proactive edge caching with sleep scheduling (E3CS) where the controller jointly adjusts the caching strategy and the sleep scheduling of SCs by considering the content popularity dynamics. Specifically, the controller does not command the sleep mode to SCs having high utility of the cached contents (e.g., popular contents) and caches proactively contents estimated to become popular in SCs to reduce the energy consumption for the content delivery. Moreover, the controller commands the sleep mode to SCs with low utility of the cached contents (e.g., when there is no user in the coverage of SC). Then, the energy consumption to maintain the active mode of SCs can be diminished. To optimize the performance of E3CS (i.e., minimize the overall energy consumption), we formulate a Markov decision process (MDP) and the joint optimal policy on the caching strategy and sleep scheduling is obtained by a value iteration algorithm. Evaluation results demonstrate that E3CS with the optimal policy can reduce the overall energy consumption by up to 25%.

The contribution of this paper can be summarized as follows: (1) to the best of our knowledge, this is the first work to simultaneously optimize the caching strategy and sleep scheduling with the consideration of the content popularity dynamics and (2) we present and analyze evaluation results under various environments, which provide guidelines for designing the edge caching system with high energy efficiency.

The remainder of this paper is organized as follows. Related works are summarized in Section 3 and E3CS is described in Section 4. After that, the MDP model is developed in Section 5 and the evaluation results are given in Section 6, and followed by the concluding remarks in Section 7.

3 Related works

To improve the energy efficiency of networks by means of the content caching, there are several works in the literature [6, 7, 10–26].

To maximize the effectiveness of caching, sociality between users is exploited in [16, 17]. Nikolaou et al. [16] proposed two cache placement strategies that take advantage of known relationships between clients (e.g., social links) and the workload on the service. In a similar context, Wang et al. [17] analyzed the impact of social relationships on the performance of edge caching by means of a Markov chain. Meanwhile, Kakar et al. [18] introduced the concept of delivery time per bit and investigated cache-aided heterogeneous wireless networks. Since the performance of the caching system can be improved when the coded contents are cached, some works focused on the coded contents and introduced appropriate caching schemes for these contents [19–21]. Maddah-Ali and Niesen [19] proposed a coded caching scheme to improve substantially the efficiency of content transmission over un-coded caching. Fadlallah et al. [20] provided an overview of some caching-aided coded multicast techniques and then discussed the potential of caching-aided coded multicast for improving bandwidth efficiency. Poularakis et al. [21] developed caching algorithms for layered encoding (i.e., scalable video coding (SVC)) to reduce the average video delivery time. In [22, 23], it is assumed that the content popularity distribution is fixed, which follows a specific distribution (i.e., Zipf distribution). Specifically, Zhou et al. [22] designed a cache strategy called caching as a cluster where SCs can exchange contents with each other within the cluster of SCs, and a probabilistic solution was derived based on the content popularity distribution to reduce the average content delivery latency. Meanwhile, Krishnan et al. [23] studied the effect of retransmissions on the cache strategy for both static and mobile user scenarios with the fixed content popularity distribution and demonstrated that the cache strategy is very sensitive to the number of retransmissions. Since the content popularity is fluctuated, Muller et al. [24] suggested an algorithm which learns context-specific content popularity by regularly observing context information of users and updating the cached contents. However, these works did not take into account the effect of SCs’ sleep mode on the performance of the content caching.

Some works analyzed the energy efficiency of edge caching-enabled networks [25, 26]. Liu and Yang [25] derived a closed form of the energy efficiency and identified key impacting factors to the energy efficiency. Similarly, Perabathini et al. [26] analyzed the energy efficiency and area power consumption based on stochastic geometry and provided the optimal transmission power to maximize the energy efficiency.

Since the popular contents should be cached to improve the performance of the caching system, it is important to estimate the content popularity dynamics. Therefore, some works focused on the content popularity dynamics estimation [10–15]. Yang and Leskovec [10] studied on temporal patterns of contents and how the content popularity grows and fades over time. Similarly, Zhou et al. [11] analyzed how the content popularity changes with time, for different types of contents, and then apply the results to design caching strategies. Meanwhile, some works developed a prediction model based on empirical data [12–14]. Szabo and Huberman [12] presented a model that predicts the long-term content popularity based on early measurements of user access. Li et al. [13] proposed a model that can capture the popularity dynamics based on popularity characteristics (e.g., early popularity evolution pattern and future popularity burst possibility). Similarly, Traverso et al. [14] devised a simple model to capture the dynamics of content popularity. Since most existing popularity prediction models are based on the big data analytics, their accuracy is highly dependent on the size of data sets. To mitigate this problem, He et al. [15] predicted the viewing probability of contents from the perspective of individuals by means of a discrete-time Markov chain.

Few works considered simultaneously both the caching strategy and the sleep scheduling [6, 7]. Li et al. [6] constructed the framework of edge caching with considering the interaction between caching and sleeping. Specifically, the content caching problem was formulated to maximize the average hit rate with an energy consumption constraint. Meanwhile, Chiang et al. [7] investigated a multi-cell cooperation-based approach where a cooperative transmission and sleep mode operation are jointly performed to tackle inter-cell interference while reducing energy dissipation. However, since these works did not consider the content popularity dynamics, the optimal performance of edge caching system cannot be achieved.

4 Energy-efficient proactive edge caching with sleep scheduling (E3CS)

Figure 1 shows the system model in this paper. We consider a heterogenous networks where a macro cell (MC) overlays with several SCs which are connected to MC through an interface (e.g., X2 interface in LTE/LTE-A networks).

Fig. 1

System model

Two types of energy consumption are considered in our system model: (1) the energy consumption to deliver the contents and (2) the energy consumption to maintain the active mode of SCs. The energy consumption to deliver the contents can be reduced by the edge caching. When a cached content is requested, SC can provide it and thus the energy consumption owing to backhaul link transmissions can be reduced.² To improve this effect, the popular contents should be previously cached before when these are requested. That is, contents should be proactively cached with the consideration of the content popularity dynamics (i.e., contents should be cached before those become popular).

Meanwhile, we assume that SCs in the active mode consume E_A energy during the unit time τ. To reduce this energy consumption, SCs can use the sleep mode. Since MC generally provides control messages to users, we assume that MC in our system model does not use the sleep mode. In the sleep mode, since most functions of SCs are turned off, the energy consumption in this mode can be negligible. However, the sleep mode makes the cached contents unavailable. Therefore, the contents should be delivered from the remote content server, which increases the energy consumption to deliver the contents. Especially when SC users in the sleep mode request excessively, huge energy can be consumed. Therefore, SCs use the sleep mode only when the utility of the cached contents is low (e.g., there is no user in the coverage of SC).

In E3CS, the controller at MC side jointly decides the caching strategy and conducts the sleep scheduling of SCs to minimize the overall energy consumption. For this, the controller maintains some information such as the content popularity dynamics and the number of users (i.e., mobility profiles of users) in each SC. Note that the content request frequency is dependent on the number of users in SCs. By using these information, the controller makes the policy table on the caching strategy and sleep scheduling and distributes the table to SCs. Basically, we assume that the library of candidate contents to be cached is not changed in a short-term scenario,³ and therefore, we can obtain the optimal policy in an offline manner. Since newly generated contents are generally not popular at the first time, these contents do not need to be cached in SCs. Therefore, we believe that our caching scenario is reasonable. Meanwhile, when some parameters (e.g., content popularity) are changed or new contents become popular, a new policy table should be reconstructed by the controller to achieve better performance. Since the complexity of making a policy table is not negligible, too frequent reconstructions can be huge overhead to the controller. Therefore, an appropriate frequency should be determined to balance the tradeoff between the performance improvement and the overhead, which is one of our future works.⁴ After receiving the policy table, SCs decide which contents are cached in their storages and whether to sleep or not by following the distributed policy table. To obtain the optimal policy table, we formulate MDP which will be elaborated in Section 5. Since MC has higher computational power than SCs and the policy table is made by the controller at MC side, the MDP model can be applied to SCs without any high computation overhead in SCs.

5 Markov decision process (MDP)

To construct the optimal policy table on the caching strategy and sleep scheduling, we formulate an MDP model in this section. The MDP model represents a mathematical framework to model decision making in situations in which outcomes are partially random and partially under the control of the decision maker [27]. Therefore, the MDP model appears suitable for simultaneously adjusting the cache strategy and sleep scheduling under the content popularity dynamics. The MDP model is constructed with five elements: (1) decision epoch, (2) action, (3) state, (4) transition probability, and (5) cost functions [28, 29]. Subsequently, we introduce the optimality equation and a value iteration algorithm to solve the equation. Important notations for the MDP model are summarized in Table 1.

Table 1 Summary of notations

5.1 Decision epoch

Figure 2 shows the timing diagram for the MDP model. A sequence T={1,2,3,…} represents the time epochs when successive decisions are made. S_t and A_t denote the state and the action chosen at the decision epoch t∈T, respectively. τ represents the duration of each decision epoch.

Fig. 2

Timing diagram

5.2 State space

We define the state space ${\mathbb {S}}$ as

$$\begin{array}{@{}rcl@{}} {\mathbb{S}} = {\mathbb{P}} \times {\mathbb{M}} \times {\mathbb{N}} \times \prod \limits_{i}^{{N_{S}}} {{{\mathbb{C}_{i}}}} \end{array} $$

(1)

where ${\mathbb {P}}$ and ${\mathbb {M}}$ denote the state spaces for representing the popularity of contents and the mode of SCs, respectively. In addition, ${\mathbb {N}}$ means the state space for describing the number of users in SCs. Moreover, ${\mathbb {C}_{i}}$ describes the state space for representing cached contents of SC i. Meanwhile, N_S is the number of SCs.

${\mathbb {P}}$ can be represented as

$$\begin{array}{@{}rcl@{}} {\mathbb{P}} = \prod\limits_{j}^{{N_{C}}} {{P_{j}}} \end{array} $$

(2)

where N_C is the number of contents. In addition, P_j denotes the popularity of content j. That is, P_j={0,1,2,…,M_P} where M_P is the maximum level of content popularity. Note that the content popularity can be quantized by a certain criterion (e.g., the popularity ranking).

Meanwhile, ${\mathbb {M}}$ is described by

$$\begin{array}{@{}rcl@{}} {\mathbb{M}} = \prod\limits_{i}^{{N_{S}}} {{M_{i}}} \end{array} $$

(3)

where M_i denotes the mode of SC i. That is, if SC i is in the sleep mode, M_i=0. Otherwise, M_i=1.

${\mathbb {N}}$ can be denoted by

$$\begin{array}{@{}rcl@{}} {\mathbb{N}} = \prod\limits_{i}^{{N_{S}}} {{N_{i}}} \end{array} $$

(4)

where N_i denotes the number of users in SC i. That is, N_i={0,1,2,…,M_U} where M_U is the maximum number of users in one SC.

On one hand, ${\mathbb {C}_{i}}$ can be represented as

$$\begin{array}{@{}rcl@{}} {{\mathbb{C}_{i}}} = \prod\limits_{j}^{{N_{C}}} {c_{i}^{j}} \end{array} $$

(5)

where $c_{i}^{j}$ denotes whether the content j is cached in SC i or not. In other words, if the content j is cached in SC i, $c_{i}^{j} = 1$. Otherwise, $c_{i}^{j} = 0$. The element of ${\mathbb {C}}_{i}$ can be represented by the vector C_i. For example, when the total number of contents is 5 and the first and third contents are stored in SC 2, C₂=[1,0,1,0,0] as shown in Fig. 1.⁵

5.3 Action space

The action space ${\mathbb {A}}$ can be described as

$$\begin{array}{@{}rcl@{}} {\mathbb{A}} = \left\{ { {\mathbf{A_{1}}},{\mathbf{A_{2}}},\ldots,{\mathbf{A_{{N_{P,A}}}}}} \right\} \end{array} $$

(6)

where N_P,A means the total number of possible combinations of actions. A_k represents the kth possible combination of actions. At each decision epoch, the controller determines which SCs are in the sleep mode (i.e., conducts the sleep scheduling). In addition, the controller decides whether each content is cached at each SC (i.e., decides the cache strategy). Therefore, one of the vectors in the action vector set (i.e., A) can be constructed by concatenating two vectors: (1) the action vector for the sleep scheduling, A^M and (2) the action vector for the cache strategy, A^C. That is, A=A^M⊕A^C where ⊕ means the concatenation operation between two vectors. Moreover, A^C can be constructed by concatenating N_S vectors where each vector ${\mathbf {A_{i}^{C}}}$ decides whether each content is cached at SC i$\left (\text {i.e.,}\ {\mathbf {{A^{C}}}} = {\mathbf {A_{1}^{C}}} \oplus {\mathbf {A_{2}^{C}}} \oplus \ldots \oplus {\mathbf {A_{{N_{S}}}^{C}}}\right)$.

The controller commands each SC whether it goes in the sleep mode or not. Therefore, A^M can be represented by $\left [ {a_{1}^{M},a_{2}^{M},\ldots,a_{{N_{S}}}^{M}} \right ]$, where $a_{i}^{M}$ denotes the commanded mode (i.e., sleep or active mode) to SC i. That is, if $a_{i}^{M} = 0$, the controller commands the sleep mode to SC i. On the contrary, $a_{i}^{M} = 1$ means that the controller commands SC i to be active.

Meanwhile, ${\mathbf {A_{i}^{C}}}$ can be denoted as $\left [ {a_{i,1}^{C},a_{i,2}^{C},\ldots,a_{i,{N_{C}}}^{C}} \right ]$ where $a_{i,j}^{C}$ represents whether SC i tries to cache content j. In other words, $a_{i,j}^{C} = 1$ means that SC i tries to cache content j (i.e., SC i downloads content j from the content server), whereas $a_{i,j}^{C} = 0$ means that SC i does not try. Note that the number of contents can be cached is limited due to the constraint of the storage capacity of SCs. Therefore, the possible action vector ${\mathbf {A_{i}^{C}}}$ which can be chosen is restricted. That is, when κ_j denotes the size of content j, only ${\mathbf {A_{i}^{C}}}$ with $\sum \limits _{j} \kappa _{j} a_{i,j}^{C} \le \xi$ where ξ is the maximum number of contents which can be cached in SCs can be chosen.

5.4 Transition probability

We can think that two type of actions (i.e., action for the sleep scheduling A^M and action for the cache strategy A^C) are chosen at each decision epoch. Then, cached content state C and SC mode state M are dependent on the chosen action A^C and A^M, respectively. Meanwhile, all states change independently with each other. Therefore, the transition probability from the current state, S=[P,M,N,C], to the next state, S^′=[P^′,M^′,N^′,C^′], can be represented by

$$\begin{array}{@{}rcl@{}} P\left[ {{\mathbf{S}}'|{\mathbf{S}},{\mathbf{A}}} \right] &=& P\left[ {{\mathbf{P}}'|{\mathbf{P}}} \right] \times P\left[ {{\mathbf{M}}'|{\mathbf{M}},{\mathbf{A^{M}}}} \right] \\ &&\times P\left[ {{\mathbf{N}}'|{\mathbf{N}}} \right] \times P\left[ {{\mathbf{C}}'|{\mathbf{C}},{\mathbf{A^{C}}}} \right]. \end{array} $$

(7)

The transition probability of each state can be derived as follows. Since the popularity of contents are varied independently, $P\left [ {{\mathbf {P}}'|{\mathbf {P}}} \right ] = \prod \limits _{j}^{{N_{C}}} {P\left [ {{{P'_{j}}}|{{{P_{j}}}}} \right ]}$ where P_j and Pj′ denote the popularity of content j at the next states, respectively. Meanwhile, P[Pj′|P_j] can be obtained by various methods such as statistic manner and model-based approach [10–15]. For example, by using shot noise model (SNM) [14], ⁶ the request rate can be modeled as a function of the time. Since the request rate at a certain time can represent the popularity at that time, P[Pj′|P_j] can be defined.

Since the modes of SCs are decided independently according to the elements in the chosen action vector A^M, $P\left [ {{\mathbf {M}}'|{\mathbf {M}}} \right ] = \prod \limits _{i}^{{N_{S}}} {P\left [ {M_{i}'|{M_{i}},a_{i}^{M}} \right ]}$ where M_i and Mi′ represent the mode of SC i at the current and next states, respectively. Since Mi′ is decided by $a_{i}^{M}$ (i.e., SC i changes its mode by following the command of the controller), ${P\left [ {M_{i}'|{M_{i}},a_{i}^{M}=0} \right ]}$ and ${P\left [ {M_{i}'|{M_{i}},a_{i}^{M}=1} \right ]}$ can be represented by

$$\begin{array}{@{}rcl@{}} P\left[ {{M_{i}'}|{M_{i}},a_{i}^{M} = 0} \right] = \left\{ \begin{array}{l} 1,\ {\text{if}}\ M_{i}' = 0\\ 0,\ {\text{otherwise}} \end{array} \right. \end{array} $$

(8)

and

$$\begin{array}{@{}rcl@{}} P\left[ {{M_{i}'}|{M_{i}},a_{i}^{M} = 1} \right] = \left\{ \begin{array}{l} 1,\ {\text{if}}\ M_{i}' = 1\\ 0,\ {\mathrm{otherwise.}} \end{array} \right. \end{array} $$

(9)

Meanwhile, P[N^′|N] can be defined by means of a statistical manner or a mathematical assumption (e.g., uniform distribution and point Poisson process [30]). Note that P[N^′|N] means that the population distribution of SCs and the mobility of users can be reflected in this probability.

On the other hand, the contents stored in SC i are independently decided by the chosen action vector ${\mathbf {A_{i}^{C}}}$. Therefore, $P\left [ {{\mathbf {C}}'|{\mathbf {C}},{\mathbf {{A^{C}}}}} \right ] = \prod \limits _{i}^{{N_{S}}} {P\left [ {{\mathbf {C'{~}_{i}}}|{{\mathbf {C_{i}}}},{\mathbf {A_{i}^{C}}}} \right ]}$ where C_i and C^′ _i represent the cached contents in SC i at the current and next states, respectively. Meanwhile, whether content j is stored in SC i or not is decided according to $a_{i,j}^{C}$. That is, $P\left [ {{\mathbf {C'{~}_{i}}}|{{\mathbf {C_{i}}}},{\mathbf {A_{i}^{C}}}} \right ]$ can be calculated by $\prod \limits _{j}^{{N_{C}}} {P\left [ {c'{~}_{i}^{j}|c_{i}^{j},a_{i,j}^{C}} \right ]}$ where $c_{i}^{j}$ and $c'{~}_{i}^{j}$ denote the current and next states for the cached status of content j, respectively. ${P\left [ {c'{~}_{i}^{j}|c_{i}^{j},a_{i,j}^{C}} \right ]}$ is derived as follows. If SC i does not have content j and the controller decides that content j is stored at SC i$\left (\text {i.e.,}\ c_{i}^{j} = 0 \ \text {and}\ a_{i,j}^{C} = 1\right)$, SC i downloads content j by requesting it to the content server. We assume that the download completion time for content j follows an exponential distribution with mean 1/μ_j.⁷ Then, the probability that content j is stored in SC i at the next state can be calculated by μ_jτ [31]. Meanwhile, when SC i does not have content j and the controller decides that content j is not stored at SC i$\left (\text {i.e.,}\ c_{i}^{j} = 0\ \text {and}\ a_{i,j}^{C} = 0\right)$, any state transition does not occur. Therefore, the corresponding state transition probabilities can be derived as

$$\begin{array}{@{}rcl@{}} P\left[ {c'{~}_{i}^{j}|c_{i}^{j} = 0,a_{i,j}^{C} = 1} \right] = \left\{ \begin{array}{ll} {\mu_{j}}\tau,& {\text{if}}\ c'{~}_{i}^{j} = 1\\ 1 - {\mu_{j}}\tau,& {\text{if}}\ c'{~}_{i}^{j} = 0.\\ \end{array} \right. \end{array} $$

(10)

and

$$\begin{array}{@{}rcl@{}} P\left[ {c'{~}_{i}^{j}|c_{i}^{j} = 0,a_{i,j}^{C} = 0} \right] = \left\{ \begin{array}{l} 1,\ {\text{if}}\ c'{~}_{i}^{j} = 0\\ 0,\ {\mathrm{otherwise.}} \end{array} \right. \end{array} $$

(11)

When the controller decides that content j is stored at SC i$\left (\text {i.e.,}\ a_{i,j}^{C} = 1\right)$, if SC i has already content j, $\left (\text {i.e.,}\ c_{i}^{j} = 1\right)$, SC i needs not to download content j. Thus, any state transition does not occur. Meanwhile, SC i can delete content j immediately. Therefore, when the controller decides that content j is not stored at SC i$\left (\text {i.e.,}\ a_{i,j}^{C} = 0\right)$, the next state for content j is 0 $\left (\text {i.e.,}\ c'{~}_{i}^{j} = 0\right)$ even though the current state for content j is 1 $\left (\text {i.e.,}\ c_{i}^{j} = 1\right)$. Then, the corresponding transition probabilities can be described by

$$\begin{array}{@{}rcl@{}} P\left[ {c'{~}_{i}^{j}|c_{i}^{j} = 1,a_{i,j}^{C} = 1} \right] = \left\{ \begin{array}{l} 1,\ {\text{if}}\ c'{~}_{i}^{j} = 1\\ 0,\ {\text{otherwise}} \end{array} \right. \end{array} $$

(12)

and

$$\begin{array}{@{}rcl@{}} P\left[ {c'{~}_{i}^{j}|c_{i}^{j} = 1,a_{i,j}^{C} = 0} \right] = \left\{ \begin{array}{l} 1,\ {\text{if}}\ c'{~}_{i}^{j} = 0\\ 0,\ {\mathrm{otherwise.}} \end{array} \right. \end{array} $$

(13)

5.5 Cost function

To define the cost function, we consider the overall energy consumption. The overall energy consumption consists of (1) the energy consumption to deliver the contents, (2) the energy consumption to maintain the active mode of SCs, and (3) the energy consumption to change of the cached contents. Since one SC in the active mode consumes E_A energy during τ, the energy consumption to maintain the active mode of SCs can be calculated as ${E_{A}}\sum \limits _{i}^{{N_{S}}} {\delta \left [ {{M_{i}} = 1} \right ]}$, where δ[·] is a delta function to return 1 if the given condition (e.g., M_i=0) is true. If the condition is not true, a delta function returns 0. Meanwhile, the expected number of requests for content j in SC i at each decision epoch, $N_{R}^{i,j}$, is proportional to its popularity and the number of users in SC i. That is, $N_{R}^{i,j}$ can be calculated by αP_jn_i where α is the coefficient parameter reflecting the duration of decision epoch.⁸ When the content j is requested, if SC i is in the active mode and has content j in its storage $\left (\text {i.e.,}\ M_{i}=1\ \text {and}\ c_{i}^{j}=1\right)$, SC i can transport content j to users. In this case, a small amount of delivery energy consumption E_C,E is needed. On the contrary, when SC i is in the sleep mode or it does not have content j in its storage $\left (\text {i.e.,}\ M_{i}=0\ \text {or}\ c_{i}^{j}=0\right)$, the requested content should be delivered from the content server. Since the energy consumption to deliver the contents is generally proportional to the distance between the source and the destination, a large amount of energy E_C,C can be assumed when the content is delivered from the content server. Meanwhile, when SC i tries to change its stored content $\left (\text {i.e.,}\ a_{i,j}^{C} = 0\ \text {and}\ a_{i,j}^{C} = 1\ \text {or}\ a_{i,j}^{C} = 1\ \text {and}\ a_{i,j}^{C} = 0\right)$, the energy E_S is needed.⁹ To sum up, the cost function for the total energy consumption of networks r(S,A) can be defined as

$$ {}\begin{aligned} r\left({{\mathbf{S}},{\mathbf{A}}} \right) &= {E_{A}}\sum\limits_{i}^{{N_{S}}} {\delta [ {{M_{i}} = 1} ]} \\ &\quad+ \sum\limits_{i}^{{N_{S}}} \sum\limits_{j}^{{N_{C}}} \alpha {P_{j}}{n_{i}} \left[ {{E_{C,E}}\delta [ {{M_{i}} = 1} ]\delta [ {c_{i}^{j} = 1} ]} \right.\\ &\quad+ \left. {E_{C,C}}\left\{ {\delta [ {{M_{i}} = 0} ] + \delta [ {c_{i}^{j} = 0} ] - \delta [ {{M_{i}} = 0} ]\delta [ {c_{i}^{j} = 0} ]} \right\} \right] \\ &\quad+ \sum\limits_{i}^{{N_{S}}} \sum\limits_{j}^{{N_{C}}} E_{S} \left[ \delta [ {{a_{i,j}^{C}} = 0} ]\delta [ {c_{i}^{j} = 1} ] + \delta [ {{a_{i,j}^{C}} = 1} ]\delta [ {c_{i}^{j} = 0} ] \right]. \end{aligned} $$

(14)

5.6 Optimality equation

To minimize the expected total cost and obtain the optimal policy, we choose the expected total discount cost optimality criterion [32, 33]. Let v(S) be the minimum expected total cost when the initial state is S. Then, we can describe v(S) as

$$\begin{array}{@{}rcl@{}} v({\mathbf{S}}) = \mathop{\min }\limits_{\pi \in \Pi} v^{\pi} ({\mathbf{S}}) \end{array} $$

(15)

where v^π(S) is the expected total cost when the policy π with an initial state S is given.

The optimality equation is given by [28]

$$\begin{array}{@{}rcl@{}} v({\mathbf{S}}) = \mathop{\min }\limits_{{\mathbf{A}} \in \mathbb{A}} \left\{ {r({\mathbf{S}},{\mathbf{A}}) + \sum\limits_{{\mathbf{S}}' \in \mathbb{S}} \lambda P[{\mathbf{S}}'|{\mathbf{S}},{\mathbf{A}}]v({\mathbf{S}}')} \right\} \end{array} $$

(16)

where λ is a discount factor in the MDP model. A value of λ closer to 1 gives more weight to future costs. The solution of the optimality equation corresponds to the minimum expected total cost and the optimal policy. To solve the optimality equation and to obtain the optimal policy, δ^∗, we use a value iteration algorithm, as shown in Algorithm 1, which is one of the conventional algorithms to solve MDP problem, where |v|=min[v(S)] for ${\mathbf {S}}~\in ~\mathbb {S}$ and v^k(S) is the expected total cost in the step k. First, parameters are initialized (line 1 in Algorithm 1). Then, the expected total cost in the step k for each state is computed (line 2 in Algorithm 1). After that, the algorithm determines whether the iteration is sufficiently conducted or not based on the inequality |v^k+1(S)−v^k(S)|<ε(1−λ)/2λ (line 3 in Algorithm 1), where ε is a stopping criterion. If the iteration is not enough, the algorithm conducts an additional iteration. Otherwise, the algorithm computes the stationary optimal policy (line 4 in Algorithm 1).

Generally, each iteration in the value iteration algorithm is performed in $O(|\mathbb {A}||\mathbb {S}|^{2})$ [34].¹⁰ Note the overall state space and the action space are represented by (1) and (6), respectively. Therefore, the complexity of this algorithm is decided by the number of contents, the number of SCs, the number of users, and the cache size. If these parameters have large values (i.e., state and/or action space are large), the complexity cannot be neglected even though it is a polynomial function. Therefore, the controller makes a table to store the optimal policy, which can be pre-computed by the value iteration algorithm in an offline manner, and then distributes the table to SCs. This table includes the state and decision in each state. Then, SCs can decide whether to be in the sleep mode or not and which contents are stored according to the policy stored in the table.

6 Evaluation results

For performance evaluation, we compare the proposed algorithm, E3CS, with the following five schemes: (1) ADAP-SLEEP where SCs use the sleep mode only when the number of users in their area does not exceed δ_N while the contents are cached by the MDP, (2) NO-SLEEP where SCs do not use the sleep mode while the contents are cached by the MDP, (3) ALL-SLEEP where SCs are always in the sleep mode,¹¹ (4) HALF-SLEEP where half of SCs which are randomly selected are in the sleep mode are in the sleep mode whereas the others are in the active mode while the contents are cached by the MDP, and (5) POP-CACHE where the content is cached based on the current content popularity (not content popularity dynamics) while the mode of SCs are decided by the MDP. Note that, since this is the first work on the joint optimization of the caching strategy and sleep scheduling by considering the content popularity dynamics, we introduce simple comparison schemes to demonstrate the effectiveness of the propose scheme. The performance metric is the energy consumption ratio, ζ, based on the energy consumption of ALL-SLEEP. That is, the energy consumption of ALL-SLEEP is normalized as 1.

The default parameter settings are as follows. The number of SCs, N_S, and the total number of contents, N_C, are set to 25 and 100, respectively [35, 36]. The cache size of SCs and the average content size are assumed by 1 GB and 500 MB, respectively [36]. In particular, the content size κ_j is set to the value between 250 and 750 MB. Then, we can assume that the maximum number of contents which can be cached in SCs ξ is set to 2. Meanwhile, we assume that the link bandwidth between SC and the content server is 250 MB/s. Then, the average download completion time, 1/μ_j, is set to the value between 1/3 and 1. We assume that the content popularity is dynamically changed to show effectively the outperforming of the proposed scheme. The maximum number of users in one SC M_U is set to 5 [36]. The number of users in each SC is dynamically changed with the different probability. That is, the number of users in 40% of SCs is usually kept small, while the number of users in 20% of SCs is decided by a uniform distribution. On the one hand, many users usually exist in 40% of SCs. Meanwhile, the energy consumption E_A of the active mode during τ is 6.8 W [37]. In addition, delivery energy consumptions, E_C,E and E_C,C, are set to 0.13 and 4.2 W, respectively [37, 38].¹² The energy consumption to change contents, E_S, is set to 1 W [39].¹³ On the other hand, α and τ are set to 0.1 and 1, respectively. The discount factor λ is set to 0.95. Meanwhile, we have set the x-axis values based on the real energy consumption in cellular systems [37, 38].

6.1 Effect of E _A

Figure 3 shows the effect of the energy consumption, E_A, for the active mode during τ. From Fig. 3, it can be found that the energy consumption ratio of E3CS is the lowest among the comparison schemes. This is because, in E3CS, caching strategy and sleep scheduling are jointly adjusted. In doing so, the case where the cached contents become unavailable due to the sleep mode of SCs rarely occurs. Specifically, E3CS considers the content popularity dynamics, and therefore, contents estimated to become popular are cached proactively in SCs. Therefore, the ζ of E3CS is smaller than that of POP-CACHE. Note that, in POP-CACHE, the content is cached based on the current content popularity.

Fig. 3

Effect of the energy consumption E_A for the active mode during τ

As shown in Fig. 3, when E_A is smaller than 5, the energy consumption of E3CS is comparable to that of NO-SLEEP. This is because the energy consumption for the active mode is not a dominant factor to reduce the overall energy consumption. Therefore, E3CS commands all SCs to be active to reduce the energy consumption for the content delivery.

Meanwhile, as E_A increases, the energy consumption to exploit contents cached in SCs also increases. Note that SCs should be in the active mode to use contents cached in SCs. Therefore, ζ of all schemes except ALL-SLEEP increases as E_A increases. However, since E3CS commands the sleep mode adaptively depending on E_A (i.e., E3CS frequently commands the sleep mode to more SCs when E_A is large), the energy consumption ratio ζ of E3CS increases slowly when E_A is over 6.

6.2 Effect of E _C,C

The effect of the energy consumption E_C,C to retrieve contents from the remote content server on ζ is shown in Fig. 4. From Fig. 4, it can be seen that E3CS operates adaptively even when E_C,C is changed. Small E_C,C means that contents can be retrieved from the remote content server without high energy consumption. In such case, E3CS commands the sleep mode to SCs to reduce the energy consumption for the active mode in SCs, and thus, the ζ of E3CS is the same as that of ALL-SLEEP when E_C,C is 2. On the contrary, when E_C,C has larger value (i.e., when E_C,C is 5∼6), the energy consumption to retrieve contents from the remote content server is more influential than the energy consumption for the active mode. As a results, E3CS does not command the sleep mode to most SCs, and thus, contents can be delivered from the network edge (i.e., SCs). Therefore, the ζ of E3CS is comparable to that of NO-SLEEP.

Fig. 4

Effect of the energy consumption E_C,C to retrieve contents from the remote content server

6.3 Effect of E _C,E

The effect of the energy consumption E_C,E to deliver contents from the network edge on ζ is shown in Fig. 5. In this result, α is set to 0.25. When E_C,E is small (i.e., E_C,E<1), the ζ of E3CS is almost the same as that of NO-SLEEP. This is because, when E_C,E is small, the energy saving effect through the content delivery from the network edge is more influential than the energy consumption due to the active mode. In such a situation, SCs are always in the active mode to reduce the energy consumption for the content delivery. Meanwhile, as E_C,E increases, E3CS commands the sleep mode to some SCs. Especially when E_C,E is lager than 3, E3CS commands the sleep mode to all SCs, and therefore, the ζ of E3CS is the same as that of ALL-SLEEP. Meanwhile, since ADAP-SLEEP, NO-SLEEP, and HALF-SLEEP do not consider the energy consumption to deliver contents from the network edge, their ζ increases continuously regardless of E_C,E. Specifically, the increasing rate of NO-SLEEP is the highest, because the contents are always retrieved from the network edge in this scheme.

Fig. 5

Effect of the energy consumption E_C,E to deliver contents from the network edge

6.4 Effect of μ

Figure 6 shows the effect of the download rate μ between the content server and SCs. As μ increase, ζ of all schemes except ALL-SLEEP decreases. This can be explained as follows. Large μ represents that contents can be cached within short time. Then, the cached contents can be rapidly changed according to the popularity of contents, which means that the cached contents can be used more frequently. Consequently, the case where contents are delivered from the remote content server rarely occurs, and therefore, the overall energy consumption can be reduced. This result means that the bandwidth of backhaul link (i.e., the bandwidth between the content server and SCs) should be enhanced to reduce the energy consumption.

Fig. 6

Effect of the download rate μ between the content server and SCs

7 Conclusion

The sleep mode of small cells (SCs) can make the popular contents unavailable. To prevent this situation, in this paper, we propose an energy efficient proactive edge caching with sleep scheduling (E3CS) where the controller jointly adjusts the caching strategy and the sleep scheduling of SCs with the consideration of the content popularity dynamics. The joint optimal policy on the caching strategy and sleep scheduling minimizing the overall energy consumption is obtained by means of a Markov decision process (MDP). Evaluation results demonstrate that E3CS with the optimal policy reduces the overall energy consumption by up to 25% compared with other schemes. In addition, it can be also found that E3CS operates adaptively even when operating environments (e.g., energy consumptions for the active mode and the content delivery) are changed. In our future work, we will extend the MDP model to consider content sharing among users by means of device-to-device (D2D) communications. Moreover, to reduce the complexity of the MDP model, we will reformulate another MDP model that has an implementation-friendly threshold structure, and then evaluate the model in more practical scenarios.