Analysis of opportunistic spectrum access in cognitive radio networks using hidden Markov model with state prediction
 JyuWei Wang^{1}Email author and
 Ramzi Adriman^{2}
https://doi.org/10.1186/s1363801402385
© Wang and Adriman; licensee Springer. 2015
Received: 21 August 2014
Accepted: 30 December 2014
Published: 24 January 2015
Abstract
We propose a study using hidden Markov model (HMM) with state prediction for opportunistic spectrum access (OSA) in cognitive radio (CR) networks. The primary channels are assumed to be operating in a TDMA manner with synchronous time slots of equal length and alternating between idle and busy states. The secondary users (SUs) may use them when they are idle by channel sensing. In contrast to the traditional scheme relying only on channel sensing for exploring spectrum opportunities, the proposed prediction scheme takes advantage of state prediction, channel sensing, and acknowledgments (ACKs) from the receiver in an attempt to maximize the utility. In the prediction scheme, there are three distinct actions: direct skip, sensing then conditional access, and direct access. We impose some constraints on the system parameters and derive thresholds by which we can specify the optimal action. We then conduct simulations to compare the performance of the prediction scheme to that of the traditional scheme. Results show that the former is superior to the latter. We believe the proposed prediction scheme is suitable for the OSA system using spectrum sensing.
Keywords
1 Introduction
The spectrum resources are rare, and a large part of the spectrum licensed for various applications are not fully used in times or in spaces. These facts motivate the use of cognitive radio (CR) networks coexisting with the licensed primary networks [1,2]. In CR networks, the secondary users (SUs) are allowed to utilize the frequency channels assigned for the primary users (PUs) when they are currently not being used. This kind of spectrum sharing is referred to as opportunistic spectrum access (OSA) or spectrum overlay. To detect the spectrum opportunities, spectrum sensing may be used. In [36], the authors consider the OSA system in which an ad hoc CR network operating with a primary network for which N frequency channels are assigned. The channels are assumed to be operating in a TDMA manner with synchronous time slots of equal length and alternating between idle and busy states. The SU may use them when they are idle by channel sensing.
To effectively implement the concept of CR networking and alleviate the processing delays, spectrum prediction technique has been employed in the functions of spectrum sensing, spectrum decision, and spectrum mobility [710]. With spectrum prediction, SUs may skip the sensing duty on channels that are predicted to be busy or directly access when channels are predicted to be idle. In this way, sensing time and energy consumption may be reduced.
CR networks can operate in either centralized or distributed mode [11]. In the centralized mode, a central controller is implemented to perform spectrum decisions and resource allocation. In contrast to the centralized mode requiring a central controller, the distributed mode does not rely on any central controller. In this mode, SUs may cooperate by exchanging their own local information to enhance the sensing performance or operate independently to pursue their own benefits [12].
Many researchers have used partially observable Markov decision process (POMDP) to model distributed OSA networks [36]. The idea to apply the POMDP model arises from the considerations that states of the system cannot be totally observed. Optimal action is determined by using the partial information and the history of observations. Assuming that the channel statistics are the same within T time periods, the optimal value functions for each system state are solved for a Tfinite horizon optimal problem in a recursive manner. The problem is quite involved because there are a total of 2^{ N } system states [4,5]. In a POMDP model, the actions taken by the agents control the state transitions of the hidden process. Different actions taken in a time change the future behavior of the process, so we have to solve the problem based on finite or infinite horizon.
In this work, we consider the OSA system with distributed mode. We assume that channels are statistically independent and focus on the treatment for a single channel. With imperfect sensing and by considering the fact that actions taken by SUs do not affect the evolution of the channel state, we thus model the system as a hidden Markov model (HMM) [1320]. An HMM can be considered as consisting two processes: the variation of the hidden states is a Markov process and the observation under a specific hidden state is a random process. This model is suitable for modeling Markov process with imperfect sensing. Operating in this model, SUs try to track the state of the hidden process and use the channel when idle state is detected. As actions do not control the hidden process, the current action does not affect the value function in the future. Thus, in finding optimal policy, no recursive procedures are required. Using the HMM in conjunction with state prediction, we derive thresholds on the probability space for the action policy, which specifies the optimal action achieving maximum expected utility based on the predicted probability.
The rest of this paper is organized as follows. In Section 2, we give a description of the system. Section 2 describes actions and derives the decision rule. Simulation results are given in Section 2 and finally we draw conclusions in Section 2.
2 System description
The same OSA system as studied in [36] is considered in this work. In the OSA system we assume that a dedicated control channel is available for SUs in the system to exchange signaling messages, as suggested in [21]. We also assume each primary channel is characterized by a twostate Markov chain. As SUs cannot know the channel state exactly, we further model each channel as an HMM, which may be viewed as a discretetime bivariable random process {S(t), O(t)}, where t=1,2, ⋯, is the discrete time, S(t) is the hidden process, and O(t) is the observable process having states as the hiddenprocess.
For the HMM under considerations, we make some assumptions stating as follows [14]. The first one is the Markov property, which means the next state depends only on the current state and independent of previous states. The second one is the stationary property, which states that the transition probabilities are independent of the actual time the transitions take places. The third one is the observation independence assumption; that is, the current observation is statistically independent of previous observations.
The hidden process S(t) of a channel is characterized by two transition probabilities α and β, where α is the probability of transition from idle state (s _{0}) to busy state (s _{1}) and β is the probability of transition from s _{1} to s _{0}. The observable process also has idle and busy states. However, due to imperfect sensing, observations may be incorrect. When a busy state is observed, denoted by o _{1}, if the hidden process is actually idle, we say a false alarm occurs and denote this probability by p _{fa}; otherwise we have a correct detection, denoting this probability by p _{d}. Thus, (1−p _{d}) is the miss detection probability. Perfect sensing means p _{d}=1 and p _{fa}=0, but it is extremely hard to achieve in practice. A spectrum opportunity detector can be considered as performing a binary hypotheses test. As can be seen from a receiver operating characteristic curve, a smaller value of p _{fa} implies a larger value of (1−p _{d}). This results in more packet collisions. Contrarily, a larger p _{fa} implies a smaller (1−p _{d}) but a large part of the unused channel resources will be wasted. In general, we would like to make p _{d} as large as possible and p _{fa} as small as possible. However, these are usually conflicting objectives. To solve the problem, we may resort to NeymanPearson criterion [22]; that is, determining a value of p _{fa} which is acceptable and seeking a decision strategy on the channel energy that constrains p _{fa} to this value while simultaneously maximizing p _{d}.
3 Actions and decision rule
In the traditional scheme for the OSA system, at the beginning of a time slot, the SU first senses the channel. If the outcome is idle, the SU then transmits over the channel; otherwise it skips the rest of the slot. Thus, no attempts are made to track the state of the channel. In this work, we propose to use state prediction in conjunction with channel sensing. In contrast to the traditional scheme always performing sensing in each slot, the prediction scheme may skip the slot or directly transmit without sensing, so as to avoid sensing as much as possible. As the concept of Bayesian filters plays a vital role in the proposed scheme, before going into further details, we first describe the concept behind the Bayesian filter.
3.1 Bayesian filter
These equations show that an update of the Bayesian filter may be viewed as composed of two major steps. The first step is to predict the onestep ahead probability vector by the transition probabilities and the previous belief vector, as in (3). The second step is to update current belief by the predicted probability vector and the observation, as in (4). However, obtaining observations requires channel sensing and doing so consumes energy. If the hidden process is predicted to be idle with high probability, one may avoid channel sensing and directly transmit. Similarly, if the channel is predicted to be busy with high probability, channel sensing may be avoided as well. These considerations are the motivation of this work.
3.2 Actions and probability update
The update in (6) tends to lower the prediction probability in the next slot and in turn helps induce action a _{1} to be chosen because when \(\hat {b}(t)\) is moderate, in the denominator, the first term may be quite small compared to the second term. If the sensing outcome is o _{0}, the SU then transmits over the channel. This action is referred to as sensing then conditional access, denoted by a _{2}, which is also the only action in the traditional scheme.
Note that if the channel is assumed to be errorfree, the SU sets b(t)=0 when a NACK is received. In fact, both updates in (8) and (9) may be as small as zero because the error rate usually is quite small compared to the other quantities.
3.3 Formulation of decision rule
Before we can formulate the decision rule, we have to specify the utility for each action. In the OSA system, we assume the SU can earn positive rewards only from successful transmissions. Suppose the SU has to pay a cost for a transmission, as doing so consuming energy. If the transmission is successful, the SU receives a reward. When the transmission collides with a PU’s transmission, the SU has to pay a cost for the collision but no further cost is paid when collapsed by the channel noise. As the transmission cost itself is a constant quantity and the channel error rate plays its role as a weighting factor on the utilities, both of them may be incorporated into the reward for a successful transmission, R _{succ}, and the cost for a collision, C _{coll}. Channel sensing consumes energy, so each time the SU senses a channel is assumed to be charged for a cost C _{sens}. In addition, we assume that C _{coll}>R _{succ}>C _{sens}.
where p(o _{0}s _{0})=1−p _{fa} and p(o _{1}s _{1})=p _{d}.
Having determined all the utility values as a function of \(\hat {b},\) we now proceed with the formulation of the decision rule.
Clearly, \( U_{a_{1}} (0)\) is the maximum one. In order to have \(U_{a_{3}} (0) < U_{a_{2}} (0)\), we require C _{coll}(1−p _{d})+C _{sens}<C _{coll}. A setting of p _{d}>C _{sens}/C _{coll} satisfies the inequality. We find this constraint reasonable as a higher p _{d} is desirable. With the above treatments, at \(\hat {b} = 1\), we have \(U_{a_{3}} (\hat {b}) > U_{a_{2}} (\hat {b}) > U_{a_{1}} (\hat {b})\) and at \(\hat {b} = 0\), we have \(U_{a_{3}} (\hat {b}) < U_{a_{2}} (\hat {b}) < U_{a_{1}} (\hat {b})\). Therefore, \(U_{a_{3}} (\hat {b}),_{ }U_{a_{2}} (\hat {b})\), and \(U_{a_{1}} (\hat {b})\) are not parallel to each other and any two lines intersect at one point. For illustrating purposes, we in the rest of this paper focus on this scenario. However, the idea applies to other scenarios as well.
If channels are statistically independent, then extension to the case of multiple channels is straightforward. Suppose the SU is capable of treating a number of channels up to l. If l≥N, all the N channels can be treated simultaneously; otherwise the first l channels having the largest utility values are chosen with potential ties being broken randomly.
3.4 Effects of transition probabilities

Case α+β<1: For this case, if b(t)<β/(α+β), the last two terms on the righthand side of (24) sum to a negative value, which increases toward zero as n increases; that is, \(\hat {b}(t + n)\) approaches the steadystate probability β/(α+β) as n increases. As a consequence, if η _{0}>β/(α+β), action a _{1} will be taken forever. Thus, we desire η _{0}<β/(α+β). On the other hand, if b(t)>β/(α+β), the two mentioned terms sum to a positive value, which approaches zero as n increases. In addition, \(\hat {b}(t + 1)=\beta \) if b(t)=0 and \(\hat {b}(t + 1)=1 \alpha \) if b(t)=1. Clearly, for any given b(t), values of \(\hat {b}(t + n)\) are limited to [ β,1−α]. Hence, for action a _{1} (resp. action a _{3}) to be active, we require η _{0}>β (resp. η _{1}<1−α).

Case α+β=1: For this case, \(\hat {b}(t + n)=\beta /(\alpha +\beta)\), for n=1,2,⋯, irrespective of b(t). Thus, the prediction scheme degenerates, as only one action is available. If the action for this case happens to be action a _{2}, the prediction scheme degenerates into the traditional scheme.

Case α+β>1: For this case, if b(t) is below β/(α+β), \(\hat {b}(t + 1)\) achieves the maximum value, which is above β/(α+β), in the sequence of \(\hat {b}(t + n)\). Contrarily, if b(t) is above β/(α+β), \(\hat {b}(t + 1)\) reaches the minimum value, which is below β/(α+β), in the sequence of \(\hat {b}(t + n)\). As the rate of change of the state is fast, the actions in adjacent time slots tend to be different so as to follow the channel state. Performing the same action a _{1} (or a _{3}) in consecutive time slots is seldom.
may fit the preferences described above. Under this condition, the prediction scheme definitely outperforms the traditional scheme. However, values of α and β are given by PUs and the SU has no choices. But it is clear that the prediction scheme does at least as well as the traditional scheme as long as utilities are of concerns.
4 Simulation results
In this section, we compare the performance of the prediction scheme to that of the traditional scheme by simulations. Before conducting simulations, we have to set system parameters appropriately so that the imposed constraints are met. With this consideration in mind, we first set C _{coll}=4, C _{succ}=3, and C _{sens}=0.4. For the given values, the constraints imposed in (17) and (18) are p _{fa}<0.866 and p _{d}>0.1, respectively. These constraints are not stringent.
5 Conclusions
We proposed a study using HMM with state prediction for spectrum sensing in OSA networks. The SU may exercise one of the three distinct actions: direct skip, sense then conditional access, or direct access. We imposed some constraints on the system parameters and derived thresholds by which we can specify the optimal action which maximizes the utility. We compared the performance of the prediction scheme to that of the traditional scheme. Simulation results show that as trying to avoid sensing as much as possible, the prediction scheme may earn a gain roughly a magnitude of the sensing cost relative to the traditional scheme. Simulation results also show that when the false alarm probability is high, the performance gain can be significant. This suggests that using the prediction scheme instead of relying only on sensing results can alleviate the detriment arising from unreliable sensing, as prediction results become more reliable. With channel errors taken into account, we observe from the simulation results that the effect of channel errors on the superiority of the proposed scheme over the traditional scheme is not significant. This means that superiority exists equally in any kind of channels. Thus, we believe the proposed scheme is suitable for OSA systems using spectrum sensing.
Declarations
Acknowledgements
The research is supported by the Ministry of Science and Technology of the Republic of China under Grant No. MOST 1032221E468005.
Authors’ Affiliations
References
 S Haykin, Cognitive radio: brainempowered wireless communications. IEEE JSAC. 23(2), 201–220 (2005).Google Scholar
 I Akyildiz, W Lee, M Vuran, S Mohanty, Next generation/dynamic spectrum access/cognitive radio wireless networks: a survey. Comput. Networks. 50, 2127–2159 (2006).View ArticleMATHGoogle Scholar
 Q Zhao, BM Sadler, A survey of dynamic spectrum access. IEEE Signal Process. 24(3), 79–89 (2007).View ArticleGoogle Scholar
 Q Zhao, L Tong, A Swami, Y Chen, Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: a POMDP framework. IEEE JASAC. 25(3), 589–600 (2007).Google Scholar
 DV Djonin, Q Zhao, V Krishnamurthy, in Communications (ICC), 2007 IEEE International Conference on. Optimality and complexity of opportunistic spectrum access: a truncated, Markov decision process formulation, (24–28 June 2007), pp. 5787–5792.Google Scholar
 L Li, C Chen, F Lin, X Guan, in the 32nd Chinese Control Conference. Optimal dynamic sensing policy for cognitive radio network: a, POMDP based method, (26–28 July 2013), pp. 6392–6397.Google Scholar
 X Xing, T Jing, W Cheng, Y Huo, X Cheng, Spectrum prediction in cognitive radio networks. IEEE Wireless Comm. 20(2), 90–96 (2013).View ArticleGoogle Scholar
 X Xing, T Jing, Y Huo, H Li, X Cheng, in INFOCOM 2013 Proceedings IEEE. Channel quality prediction based on, Bayesian inference in cognitive radio networks, (14–19 April 2013), pp. 1465–1473.Google Scholar
 H Ahmadi, YH Chew, PK Tang, YA Nijsure, in Personal Indoor and Mobile Radio Communications (PIMRC), IEEE 22nd International Symposium on. Predictive opportunistic spectrum access using learning based hidden, Markov models, (11–14 Sept 2011), pp. 401–405.Google Scholar
 Z Chen, N Guo, Z Hu, RC Qiu, Experimental validation of channel state prediction considering delays in practical cognitive radio. IEEE Trans. Veh. Technol. 60(4), 1314–1325 (2011).View ArticleGoogle Scholar
 EZ Tragos, S Zeadally, AG Fragkiadakis, VA Siris, Spectrum assignment in cognitive radio networks: a comprehensive survey. IEEE Commun. Surveys Tutorials. 15(3), 1108–1135 (2013).View ArticleGoogle Scholar
 P Ren, Y Wang, Q Du, J Xu, A survey on dynamic spectrum access protocols for distributed cognitive wireless networks. EURASIP J Wireless Commun. Netw. 2012(60), 1–21 (2012).MATHGoogle Scholar
 Y Ephraim, N Merhav, Hidden Markov processes. IEEE Trans. Inf. Theory. 48(6), 1518–1569 (2002).View ArticleMATHMathSciNetGoogle Scholar
 C Oliver, Markov Processes for Stochastic Modeling (Elsevier Academic Press, USA, 2009).Google Scholar
 D Treeumnuk, DC Popescu, Using hidden Markov models to evaluate performance of cooperative spectrum sensing. IET Commun. 7(17), 1969–1973 (2013).View ArticleGoogle Scholar
 E Chatziantoniou, B Allen, V Velisavljevic, in Personal Indoor and Mobile Radio Communications (PIMRC), IEEE 24th International Symposium on. An HMMbased spectrum occupancy predictor for energy efficient cognitive radio, (8–11 Sept 2013), pp. 601–605.Google Scholar
 IA Akbar, WH Tranter, in SoutheastCon, 2007 Proceedings IEEE. Dynamic spectrum allocation in cognitive radio using hidden, Markov models: Poisson Distributed Case, (22–25 March 2007), pp. 196–201.Google Scholar
 T Nguyen, BL Mark, Y Ephraim, in Information Sciences and Systems (CISS) 2011 45th Annual Conference on. Hidden Markov process based dynamic spectrum access for cognitive, (8–11 Sept 2013), pp. 1–6.Google Scholar
 Z Sun, JN Laneman, in Communications (ICC) 2012 IEEE International Conference on. Secondary access policies with imperfect sensing in dynamic spectrum access networks, (10–15 June 2012), pp. 1752–1756.Google Scholar
 T Yucek, H Arslan, A survey of spectrum sensing algorithms for cognitive radio applications. IEEE Commun. Surveys Tutorials. 11(1), 116–130 (2009).View ArticleGoogle Scholar
 Q Zhao, J Jia, X Shen, in Cognitive Wireless Communication Networks, eds. by E Hossain, V Bhargava. Optimal spectrum sensing decision for hardwareconstrained cognitive networks (Springer,USA, 2007). Chap. 13.Google Scholar
 Q Yan, RS Blum, Distributed signal detection under the NeymanPearson criterion. IEEE Trans. Inf. Theory. 47(4), 1368–1377 (2001).View ArticleMATHMathSciNetGoogle Scholar
Copyright
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.