 Research
 Open Access
Selfoptimized heterogeneous networks for energy efficiency
 Shaoshuai Fan^{1},
 Hui Tian^{1}Email author and
 Cigdem Sengul^{2}
https://doi.org/10.1186/s1363801502611
© Fan et al.; licensee Springer. 2015
 Received: 7 August 2014
 Accepted: 19 January 2015
 Published: 4 February 2015
Abstract
Explosive increase in mobile data traffic driven by the demand for higher data rates and everincreasing number of wireless users results in a significant increase in power consumption and operating cost of communication networks. Heterogeneous networks (HetNets) provide a variety of coverage and capacity options through the use of cells of different sizes. In these networks, an active/sleep scheduling strategy for base stations (BSs) becomes an effective way to match capacity to demand and also improve energy efficiency. At the same time, environmental awareness and selforganizing features are expected to play important roles in improving the network performance. In this paper, we propose a new active/sleep scheduling scheme based on the user activity sensing of small cell BSs. To this end, coverage probability, network capacity, and energy consumption of the proposed scheme in Ktier heterogeneous networks are analyzed using stochastic geometry, accounting for cell association uncertainties due to random positioning of users and BSs, channel conditions, and interference. Based on the analysis, we propose a sensing probability optimization (SPO) approach based on reinforcement learning to acquire the experience of optimizing the user activity sensing probability of each small cell tier. Simulation results show that SPO adapts well to user activity fluctuations and improves energy efficiency while maintaining network capacity and coverage probability guarantees.
Keywords
 Heterogeneous networks
 Selfoptimization
 Energy efficiency
 Reinforcement learning
1 Introduction
To satisfy the explosive increase in mobile data traffic demand, heterogeneity is expected to be a key feature of future wireless networks [14]. Heterogeneous networks (HetNets) consist of a conventional cellular network overlaid with a diverse set of lower power small cell base stations (BSs), such as microcells, picocells, and femtocells, to improve spatial frequency reuse and coverage. This allows the network to achieve higher data rates while retaining seamless connectivity and mobility. However, the overall energy consumption and operating cost of networks are also increasing considerably by the deployment of these additional small cell base stations [5,6]. As a result, green wireless communication has attracted the attention of both researchers and network operators, and energy efficiency has become one of the key network management parameters [2,7]. Additionally, the future heterogeneous networks are also expected to operate in selforganizing manner to reduce operational expenditures (OPEX) due to the deployment of large numbers of BSs [8].
An effective way to adapt to the traffic demand while improving energy efficiency is performing active/sleep scheduling by taking advantage of the fluctuations in traffic demand over time and space [9]. In [10], using a sleep mode is shown to be effective especially when the cell size is small and under light traffic conditions for a singletier network. For heterogeneous networks, Soh et al. [2] applied the tools from stochastic geometry to analyze the impact of loadaware sleeping strategy on coverage probability, finding its performance to be at least as good as without using a sleep mode. Active/sleep scheduling can be controlled via either the user equipment, the small cell, or the core network [11]. If it is networkcontrolled as proposed in [12], the information about the traffic load and user location are needed to identify hotspots to make the active/sleep decisions. Therefore, it is attractive to deploy distributed sleep mode strategies which do not involve the UE equipments, extra signaling overhead, and user location awareness. Wildemeersch et al. [5] investigated using small cells in a distributed way to offload the traffic from the macrocell network and exploiting their cognitive capabilities of user activity sensing to improve the energy efficiency by active/sleep scheduling. However, their analysis in a twotier network environment only considered the network performance of traffic offloading and the user detection. The quality of service (QoS) of users such as coverage probability and throughput which should be guaranteed as the baseline of energy saving was ignored. Moreover, the operation status of BSs were not considered by their proposed user detection model in the literature, and additional energy consumption would be caused by the active BSs due to unnecessary sensing. Also a user’s cell association with small cell tiers will affect the detection of the user because only macrocell users could be detected under their proposed model. This issue makes the scheme not applicable to the general multitier heterogeneous network scenario.
In this paper, we propose an active/sleep scheduling scheme for Ktier heterogeneous networks exploiting selforganizing capabilities. In our scheme, to guarantee coverage, macrocells are always active. However, when a small cell does not serve any active users, it goes into a sleep mode, during which it wakes up only to sense macrocell user activity. If the small cell detects an active user within its coverage during the sensing period, it becomes active to offload traffic from the macrocell. We analyze the coverage probability, network capacity, and energy consumption of the proposed scheme in a Ktier heterogeneous network using stochastic geometry, accounting for cell association uncertainties due to random positioning of users and BSs, channel conditions, and interference. To save as much energy as possible, user detection follows a sensing probability, which is selfoptimized by the network. The sensing probability optimization (SPO) approach based on reinforcement learning is proposed to acquire the experience of optimizing the user activity sensing probability of each small cell tier, considering the user activity fluctuations and user QoS such as coverage and throughput.
The rest of the paper is organized as follows: In Section 2, we describe the system model and propose the user activity sensingbased active/sleep scheduling scheme. In Section 2, we describe the energy efficiency optimization problem and present the details of the proposed fuzzy Qlearningbased SPO approach. In Section 2, we present the simulation results. Finally, we draw the conclusions.
2 User activity sensingbased active/sleep scheduling scheme
2.1 System model and assumptions
We consider a heterogeneous network that consists of K tiers of BSs, where the first tier of macrocell BSs is overlaid with K−1 tiers of denser and lower power small cell BSs. We consider that all tiers share the full spectrum and, hence, interference exists between tiers. All small cell BSs operate in openaccess mode, such that they are accessible to all users. In order to improve energy efficiency, we propose an active/sleep scheduling scheme which makes use of monitoring user activity and selforganizing capabilities.
where λ _{ u } p _{ k } is the user intensity associating with the kth tier.
To reduce user detection energy, the sensing period of BSs in the kth small cell tier follows a certain probability p _{ sk } (k=2,3,⋯,K) which is selfoptimized by the network using the sensing probability optimization approach described in Section 2.
2.2 Analysis of the active/sleep scheduling scheme
where Q(·) is the complementary distribution function of the standard Gaussian, η is the detection threshold used by energy detection, σ ^{2} is the variance of the additive white Gaussian noise, γ is the signaltonoiseplusinterference ratio (SINR), N=⌊τ _{s} f _{s}⌋ is the total sample size, f _{s} is the sample frequency. Note that, the detection probability and false alarm probability could be adjusted to certain target values, \(p_{\mathrm {d}}^{*}\) and \(p_{\mathrm {f}}^{*}\), by setting sensing threshold and sample frequency to appropriate values η ^{∗} and \(f_{\mathrm {s}}^{*}\), which is out of scope of this paper.
Theorem 1.
where P _{ k } is the transmit power of BSs in the kth tier, and α is the path loss exponent.
Proof.
See Appendix 1.
The coverage probability is defined as the probability that a user’s SINR from its associated BS is higher than the target SINR value τ.
Theorem 2.
where \(\rho \left ({\tau,\alpha }\right) = { {\tau }^{2/\alpha }}\int _{{{{\tau } }^{ 2/\alpha }}}^{\infty } {\frac {1}{{1 + {x^{\alpha /2}}}}dx} \).
Proof.
See Appendix 2.
In Equation 7, C _{0} is the ergodic rate of a user associated with the first tier during the sensing time t _{s} when there is no interference from the other tiers, and C _{ k } is the ergodic rate of a user associated with the kth tier during the time T−t _{s}.
Theorem 3.
Proof.
See Appendix 3.
3 Selfoptimization of user activity sensing based on fuzzy Qlearning
where ε _{p} and ε _{c} are, respectively, the threshold coverage probability and average capacity offered to a user. P _{c} and C are as defined in Equations 6 and 7, respectively.
To solve the problem P, we propose a SPO approach based on fuzzy Qlearning [2123], which optimizes the key sensing probabilities of the proposed active/sleep scheduling scheme by interacting with the uncertain environment and learning from the past experience. Our approach tunes the sensing probability for each K−1 tiers in a selfoptimized manner according to the active user density λ _{ u }. Assuming that the active user density does not fluctuate fast, we avoid realtime tuning and execute the tuning of the sensing probability periodically. Therefore, our approach accepts centralized operation, and the new values for sensing probabilities are computed by a centralized management entity and transmitted periodically to the BSs at each tier.
Here, a _{ ij } is the discrete sensing probability tuning action vector of the jth inference result responding to the ith rule. q _{ ij } represents the elementary quality, and the higher value of q _{ ij }, the higher the trust for the corresponding sensing probability configuration.
In addition, during the trialanderror process of action policy exploration, to avoid bad actions that result in negative performance, a must be checked according to the constraints of coverage probability and capacity (see Equations 14 and 15). Although the coverage probability and the ergodic capacity are not given in a closedform expression, the integrals are fairly easy to compute. If the coverage probability and capacity derived from the output sensing probabilities do not meet the constraints, the action for current state should be reselected according to Equation 16 excluding the faulty actions.
s _{ t } and a _{ t } denote the state and the action of the fuzzy inference rule at step t, and θ is the discount factor.
ξ is the learning rate for Qlearning.
4 Simulation results
Simulation parameters
Parameters  Value 

Tiers of networks K  3 
Network density λ _{1},λ _{2},λ _{3} (m ^{−2})  10^{−6},5×10^{−5},10^{−4} 
Transmit power P _{1},P _{2},P _{3} (dBm)  43, 30, 20 
Variance of noise σ ^{2} (dBm)  −104 (10 MHz bandwidth) 
Constant power \({E_{c}^{1}},{E_{c}^{2}},{E_{c}^{3}}\) (W)  75, 20, 4 
Sensing power \({E_{s}^{2}},{E_{s}^{3}}\) (W)  5, 4 
Processing power \({E_{p}^{1}},{E_{p}^{2}},{E_{p}^{3}}\) (W)  150, 50, 8 
Target SINR τ (dB)  1 
Sensing time t _{s}  20%T 
Target detection probability \(p_{\mathrm {d}}^{*}\)  0.9 
Target false alarm probability \(p_{\mathrm {f}}^{*}\)  0.1 
Path loss exponent α  4 
Threshold of user capacity ξ _{ c } (nat/s/Hz)  0.05 
Threshold of coverage probability ξ _{ p }  0.5 
4.1 Performance of user activity sensingbased active/sleep scheduling scheme
4.2 Performance of selfoptimization approach

Scheme 1: SPO. The sensing probabilities of Ktier heterogeneous networks are selfoptimized periodically adapting to the user activity fluctuations using reinforcement learning.

Scheme 2: always sensing. All BSs in all small cell tiers always sense user activity during the sensing time. Hence, the sensing probability of BSs in every small cell tier is 1.

Scheme 3: always active. All BSs are always active, and they do not perform user activity sensing.

Scheme 4: only macrocell. All users are served by macrocells (i.e., there are no active small cells in the network).

Scheme 5: random sensing. Each small cell senses user activity with a certain probability (e.g., 0.3 in our evaluation).

Scheme 6: random sleep. Each small cell goes into the active/sleep mode with a certain probability (e.g., 0.3 in our evaluation) and does not do the user activity sensing.
5 Conclusions
This paper proposed an active/sleep scheduling scheme for Ktier heterogeneous networks, which senses and adapts to user activity. Coverage probability, network capacity, as well as energy consumption of the proposed active/sleep scheduling were analyzed using stochastic geometry, accounting for cell association uncertainties due to random positioning of users and BSs, propagation channel, and network interference. A reinforcement learningbased SPO approach was proposed to optimize the user activity sensing probability of each small cell tier, considering user activity fluctuations and user QoS. Simulation results showed that SPO achieves low energy consumption with guaranteed network capacity and coverage probability. Possible future work includes the exploitation of more environmental awareness capabilities. And it would be of interest to extend the proposed scheme to the case, where small cells perform opportunistic usage of the frequency spectrums, for higher frequency spectrum usage and energy efficiency.
6 Appendices
6.1 Appendix 1
6.1.1 Proof of Theorem 1
6.2 Appendix 2
6.2.1 Proof of Theorem 2
6.3 Appendix 3
6.3.1 Proof of Theorem 3
Plugging (31), (33), (39), and (40) into (38), we obtain the ergodic throughput of a user associated with the first tier during the time t _{s} in (8).
Declarations
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 61471060), the National Major Science and Technology Special Project of China (No. 2013ZX03003016), and the Funds for Creative Research Groups of China (No. 61421061).
Authors’ Affiliations
References
 S Parkvall, A Furuskar, E Dahlman, Evolution of LTE toward IMTadvanced. IEEE Commun. Mag. 49(2), 84–91 (2011).View ArticleGoogle Scholar
 YS Soh, TQS Quek, M Kountouris, H Shin, Energy efficient heterogeneous cellular networks. IEEE J. Sel. Areas Commun. 31(5), 840–850 (2013).View ArticleGoogle Scholar
 A Damnjanovic, J Montojo, Y Wei, T Ji, T Luo, M Vajapeyam, T Yoo, O Song, D Malladi, A survey on 3GPP heterogeneous networks. IEEE Wireless Commun. 18(3), 10–21 (2011).View ArticleGoogle Scholar
 A Ghosh, N Mangalvedhe, R Ratasuk, B Mondal, M Cudak, E Visotsky, T Thomas, J Andrews, P Xia, H Jo, H Dhillon, T Novlan, Heterogeneous cellular networks: from theory to practice. IEEE Commun. Mag. 50(6), 54–64 (2012).View ArticleGoogle Scholar
 M Wildemeersch, TQS Quek, CH Slump, A Rabbachin, Cognitive small cell networks: energy efficiency and tradeoffs. IEEE Trans. on Commun. 61(9), 4016–4029 (2013).View ArticleGoogle Scholar
 Y Chen, S Zhang, S Xu, GY Li, Fundamental tradeoffs on green wireless networks. IEEE Commun. Mag. 49(6), 30–37 (2011).View ArticleGoogle Scholar
 Z Hasan, H Boostanimehr, VK Bhargava, Green cellular networks: a survey, some research issues and challenges. IEEE Commun. Surveys Tutorials. 13(4), 524–540 (2011).View ArticleGoogle Scholar
 OG Aliu, A Imran, MA Imran, B Evans, A survey of self organisation in future cellular networks. IEEE Commun. Surveys Tutorials. 15(1), 336–361 (2013).View ArticleGoogle Scholar
 G Auer, V Giannini, C Desset, I Godor, P Skillermark, M Olsson, M Imran, D Sabella, M Gonzalez, O Blume, A Fehske, How much energy is needed to run a wireless network?IEEE Wireless Commun. 18(5), 40–49 (2011).View ArticleGoogle Scholar
 W Guo, T O’Farrell, in Conference on Wireless Advanced. Green cellular network: deployment solutions, sensitivity and tradeoffs (IEEEPiscataway, New Jersey, USA, London, England, 20–22 June2011), pp. 42–47.Google Scholar
 I Ashraf, F Boccardi, L Ho, SLEEP mode techniques for small cell deployments. IEEE Commun. Mag. 49(8), 72–79 (2011).View ArticleGoogle Scholar
 L Saker, SE Elayoubi, R Combes, T Chahed, Optimal control of wake up mechanisms of femtocells in heterogeneous networks. IEEE J. Sel. Areas Commun. 30(3), 664–672 (2012).View ArticleGoogle Scholar
 T Yucek, H Arslan, A survey of spectrum sensing algorithms for cognitive radio applications. Commun. Surveys Tutorials IEEE. 11(1), 116–130 (2009).View ArticleGoogle Scholar
 D Bhargavi, CR Murthy, in 2010 IEEE Eleventh International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). Performance comparison of energy, matchedfilter and cyclostationaritybased spectrum sensing (IEEEPiscataway, New Jersey, USA, Marrakech, Morocco, 20–23 June2010), pp. 1–5.View ArticleGoogle Scholar
 HS Jo, YJ Sang, P Xia, JG Andrews, Heterogeneous cellular networks with flexible cell association: a comprehensive downlink SINR analysis. Wireless Commun. IEEE Trans. on. 11(10), 3484–3495 (2012).View ArticleGoogle Scholar
 JG Andrews, F Baccelli, RK Ganti, A tractable approach to coverage and rate in cellular networks. IEEE Trans. on Commun. 59(11), 3122–3134 (2011).View ArticleGoogle Scholar
 B Blaszczyszyn, MK Karray, HP Keeler, in 2013 Proceedings IEEE INFOCOM. Using Poisson processes to model lattice cellular networks (IEEEPiscataway, New Jersey, USA, Truin,Italy, 14–19 April2013), pp. 773–781.View ArticleGoogle Scholar
 JFC Kingman, Poisson Processes (Oxford University Press, Oxford, 1992).Google Scholar
 YC Liang, Y Zeng, EC Peh, AT Hoang, Sensingthroughput tradeoff for cognitive radio networks. IEEE Trans. on Wireless Commun. 7(4), 1326–1337 (2008).View ArticleGoogle Scholar
 I Ashraf, LT Ho, H Claussen. Improving energy efficiency of femtocell base stations via user activity detection (IEEEPiscataway, New Jersey, USA, Sydney, Australia, 18–21 April2010), pp. 1–5.Google Scholar
 WX Shi, SS Fan, N Wang, CJ Xia, Fuzzy neural network based access selection algorithm in heterogeneous wireless networks. J. China Inst. Commun. 31(9), 151–156 (2010).Google Scholar
 S Fan, H Tian, C Sengul, Selfoptimization of coverage and capacity based on a fuzzy neural network with cooperative reinforcement learning. EURASIP J. Wireless Commun. Netw. 2014(1), 1–14 (2014).View ArticleGoogle Scholar
 J Li, J Zeng, X Su, W Luo, J Wang, Selfoptimization of coverage and capacity in LTE networks based on central control and decentralized fuzzy Qlearning. Int. J. Distributed Sensor Networks. 2012(1), 1–10 (2012).MATHGoogle Scholar
 3GPP 36.814 V9.0.0, Further advancements for EUTRA physical layer aspects. http://www.3gpp.org. Accessed 30 March 2010.
Copyright
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.