Decision making for cognitive radio equipment: analysis of the first 10 years of exploration

This article draws a general retrospective view on the first 10 years of cognitive radio (CR). More specifically, we explore in this article decision making and learning for CR from an equipment perspective. Thus, this article depicts the main decision making problems addressed by the community as general dynamic configuration adaptation (DCA) problems and discuss the suggested solution proposed in the literature to tackle them. Within this framework dynamic spectrum management is briefly introduced as a specific instantiation of DCA problems. We identified, in our analysis study, three dimensions of constrains: the environment's, the equipment's and the user's related constrains. Moreover, we define and use the notion of a priori knowledge, to show that the tackled challenges by the radio community during first 10 years of CR to solve decision making problems have often the same design space, however they differ by the a priori knowledge they assume available. Consequently, we suggest in this article, the "a priori knowledge" as a classification criteria to discriminate the main proposed techniques in the literature to solve configuration adaptation decision making problems. We finally discuss the impact of sensing errors on the decision making process as a prospective analysis.

The increase of computational capacity associated with (rather) cheap flexible hardware technologies (such as programmable logic devices, digital signal processors and central processing units) offer a glimpse into new ways to designing and managing future non military communication systems.^a As a matter of fact in 1991, Joseph Mitola III argued that in a few years, at least in theory, software design of communication systems should be possible. The term coined by Joseph Mitola to present such technologies is software defined radio (SDR) [1]. For illustration purposes, today's radio devices need a specific dedicated electronic chain for each standard, switching from one standard to another when needed (known as the Velcro approach [2]). With the growth of the number of these standards (GSM, EDGE, Wi-Fi, Bluetooth, LTE, etc.) in one equipment, the design and development of these radio devices has become a real challenge and the practical need for more flexibility became urgent. Recent hardware advances have offered the possibility to design, at least partially, software solutions to problems which were requiring in the past hardware signal processing devices: a step closer to SDR systems.

In specific, several possible definitions exist--and are still a matter of debate in the community--to define SDR systems. For consistency reasons, we briefly describe software related radio concepts as agreed on by the SDR Forum [3]. This matter is further discussed in [4]. The SDR Forum defines SDR as radio in which some or all of the physical layer functions are software defined where physical layer and software defined terms are respectively described as:

• Physical layer: The layer within the wireless protocol in which processing of radio frequency, intermediate frequency, or baseband signals including channel coding occurs. It is the lowest layer of the ISO seven-layer model as adapted for wireless transmission and reception.

• Software defined: Software defined refers to the use of software processing within the radio system or device to implement operating (but not control) functions.

Thus, SDR systems are defined only from the design and the implementation perspectives. Consequently it appears as a simple evolution from the usual hardwired radio systems. However, with the added software layer, it is technically possible with current technology to control a large set of parameters in order to adapt on the fly radio equipment to their communication environment (e.g., bandwidth, modulation, protocol, power level adaptation to name a few). Nevertheless the control and optimization of reconfigurable radio devices need the definition of optimization criteria related to the equipment hardware capabilities, the users' needs as well as the regulators' rules. Introducing autonomous optimization capabilities in radio terminals and networks is the basis of cognitive radio (CR), term also suggested and coined by Joseph Mitola III [5, 6].

Mitola [6] defined CR, in his Ph.D dissertation as follows: The term CR identifies the point at which wireless personal digital assistant (PDAs) and the related networks are sufficiently computationally intelligent about radio resources and related computer to computer communication to:

1. (1)

   Detect user communication needs as a function of use context, and

2. (2)

   Provide radio resources and wireless services most appropriate to these needs.

Thus, the purpose of this new concept is to autonomously meet the user's expectations, i.e., maximizing his profit (in terms of QoS, throughput or power efficiency to name a few) without compromising the efficiency of the network. Hence, the needed intelligence to operate efficiently must be distributed in both the network and the radio device.

In this article, we suggest to provide a brief discussion on the decision making problems seen from CR equipment's perspective and discussed in the literature as well as the main solutions suggested to tackle these problems. For that purpose, we revisit in Section 2 the rise of CR paradigm from which we discuss a basic definition. Then, in order to objectively compare the techniques introduces to address CR related decision making problem, we describe a conceptual object referred to as design space in Section 3. This conceptual object was introduced in the literature [7] to suggest that the CR design problem, from the decision making perspective, is better defined by a set of constrains rather than by a set of degrees of freedom. Thus, this section reminds us of the three considered dimensions of constrains viz., the environment's constraint, the equipment's limits and the user's needs. Moreover, in Section 4, we define and use the notion of a priori knowledge, to show that the tackled challenges by the radio community to solve configuration adaptation decision making problems have often the same design space, however they differ by the a priori knowledge they assume available on this design space. Consequently, in Section 4, we suggest the a priori knowledge as a classification criteria to discriminate the main proposed techniques in the literature to solve configuration adaptation decision making problems. Section 5, extends previous classification by adding the impact of observation accuracy and the benefit of learning techniques in such contexts. Section 6 concludes this analysis.

2.1. The rise of CR

To fulfill the requirements to enable smart and autonomous equipment, Mitola and Maguire introduced the notion of cognitive cycle as described in Figure 1, [5, 6], where the cognitive cycle presupposes the capacity to collect information from the surrounding environment (perception), to digest it (i.e., learning, decision making, and predicting tools) and to act in the best possible way by considering several constraints and the available information. The reconfiguration of radio equipment is not discussed in depth, however, it is generally accepted that SDR in an enabling to technology support CR [4].

Cognitive radio decision making context: the CR cycle as introduced by Joseph Mitola III [6].

Full size image

As illustrated in Figure 1, a full cognitive cycle^b demands at every iteration five steps: observe, orient, plan, decide, and act. The observe step deals with internal as well as external metrics. It aims at capturing the characteristics of the environment of the communication device (e.g., channel state, interference level or battery level to name a few.). This information is then processed by the three following steps: orient, plan, and decide steps, where priorities are set, schedules are planed according to the systems constraints, and decisions are made. Finally an appropriate action is taken during the act step (such as send a message, reconfigure, modify power level to name a few). In order to complete the cognitive cycle, a last and final step is needed to enhance the decision making engine of the communication device: the learn step. As a matter of fact, learning abilities enable communication equipment to evaluate the quality of their past actions. Thus, the decision making engine learns from its past successes and failures to tune its parameters and adapt its decision rules to its specific environment. Learning can consequently help the decision making engine to improve the quality of future decisions.

As far as we can track the emergence of a CR literature and to the best of authors' knowledge, the today's plethoric publications started with three major contributions: On the one hand, the federal communication commission (FCC) pointed out in 2002 the inefficiency of static frequency bands' allocation to specific wireless applications, and suggested CR as a possible paradigm to mitigate the resulting spectrum scarcity [8, 9]. Then, Haykin in article [10] in 2005, suggested a simplified cognitive cycle to represent CR decision making engines as illustrated in Figure 2. Haykin's model tackled the particular dynamic spectrum management problem and discussed different possible models to design future CR networks. Article [10] inspired many studies on CR application fields such as theory based cognitive networks. Eventually, this two subjects led to two very actives research fields as illustrated in this recent surveys [11–13]. On the other hand, while the two contributions [8, 10] focus on spectral efficiency, Rieser suggested, through various publications, synthesized in his Ph.D. dissertation, [14] in 2004, a biologically inspired CR engine that relies on genetic algorithms (GA). To the best of authors' knowledge, it was the first suggested and partially implemented CR engine presented to the community.

Cognitive radio decision making context: simplified cognitive cycle as introduced by Haykin [10].

Full size image

In this article although we cannot avoid mentioning CR applications from spectrum management perspective, we focus on the decision making and learning mechanisms designed to deal with broader frameworks, i.e., configuration adaptation problems. Thus, spectrum management problems are, from the equipment point of view, but a subset of configuration adaptation problems.

2.2. Basic cognitive cycle

Since the original definition suggested by Joseph Mitola III, several other definitions were proposed to define the edges of CR [4, 8–10, 15–17]. However, defining cognition is, in general, a harsh task. In the context of CR, basic cognitive abilities are considered:

• environment perception (or observation)

• and reasoning (or analysis/decision).

Based on these cognitive abilities, a CR needs to take appropriate actions to adapt itself to its surrounding environment.

Once again these notions know several possible definitions that we do not explicit in this article. However, the basic cognitive cycle considers three macro-steps as illustrated in Figure 3 and that we can define as follows:

Illustration of the basic cognitive cycle [35, 81]. As illustrated, an agent, usually referred to as CA faces an environment in a broad sense. The CA repeats the cognitive cycle where he observes the environment, analyzes the collected information and decides the next action to take. Notice that the arrow action could suggest always an action on the environment. This is possible in order to evaluate the reaction on the environment to given stimuli. However, the arrow also suggests an action on the CR in order to adapt to the environment.

Full size image

1. (1)

   Observation: Through its sensors the CR gathers information on its environment. Raw data and preprocessed information helps the agent to build a knowledge base. In this context, the term environment is used in a broad sense referring to any source of information that could improve the CR's behavior (internal state, interference level, regulators' rules and enforcement policies, to name a few).

2. (2)

   Analysis/decision: This macro-step, presented as a black box in this case, includes all needed operations before given specific orders to the actuators (i.e., before reconfiguration in CR contexts). Depending on the level of sophistication, this step can deal with metric analysis, performance optimization, scheduling, and learning.

3. (3)

   Action: Mainly parameter reconfiguration and waveform transmission. A reconfiguration management architecture needs to be implemented to ensure efficient and quick reconfigurations [18].

This definition is quite general. It can incorporate simple designs as well as complex ones. Most of the published articles deal however with a restricted problem: spectrum management. In such context, the term environment finds more specific definitions such as the followings to name a few: Environment:

• Geolocation [19–22].

• Spectrum occupation [23–27].

• Interference level (or interference temperature [10]).

• Noise level uncertainty [28–30].

• Regulatory rules (that define the open opportunities [11] for instance).

Thus, depending on the considered environment, specific sensors are to be designed [4, 31, 32]. The captured -and/or computed- metrics by the sensors are then processed by the decision making engine. The kind of process highly depends on the quality of the metrics (level of uncertainty on the captured numerical value for instance) as well as the global information held by the CR. Finally, the made decisions are translated into appropriate bandwidth occupation and power allocation actions.

Within the basic cognitive cycle, we focus in this section on the analysis step, and more specifically on learning and decision making. We mainly find, in the literature two approaches. On the one hand, some of the articles focus on implementing smart behavior into radio devices to enable more adequate configurations, adapted to their environment, than those imposed by radio standards. As a matter of fact, standard configurations are usually over dimensioned to meet the requirements of various critical communication scenarios. This approach mainly focuses on one equipment, ignoring the rest of the network. We refer to the problem related to the first approach as dynamic configuration adaptation (DCA) problem. On the other hand due to a more pressing matter, most of CR related articles focus on spectrum management. These latter articles aim at enabling a more efficient use of the frequency resources because of its scarcity. This second problem is usually referred as dynamic spectrum access problem (DSA).

3.1. Design space and DCA problem

In this section, we discuss some of the limits related to the idealized CR concept before introducing the so called DCA problem. Several questions arise when designing a CR engine. We summarize our conceptual approach, presented in article [7], to dimension the decision making and learning abilities of a cognitive engine. Thus, we introduce the notion of design space as a conceptual object that defines a set of CR decision making problems by their constraints rather than by their degrees of freedom. We identified, in our analysis study, three dimensions of constrains: the environment's, the equipment's, and the user's related constrains.

Ideally speaking, CR concept--supported by an SDR platform--opens the way to infinite possibilities. Autonomous and aware of its surrounding environment as well as of it own behavior (and thus of its own abilities), any part of the radio chain could be probed and tested to evaluate its impact on the device's performance. This however implies that the equipment is also able, in its reasoning process, to validate its own choices. Namely, it must self-reference its cognition components [33]. Unfortunately, this class of reasoning is well known in the theory of computing to be a potential black hole for computational resources. Specifically, any turing-capable (TC) computational entity that reasons about itself can enter a Göel-turing^cloop from which it cannot recover[33].

To mitigate this paradox, time limited reasoning has been suggested by Mitola. As a matter of fact, radio systems need to observe, decide, and act within a limited amount of time: The timer and related computationally indivisible control construct is equivalent to the computer-theoretic construct of a step-counting function over "finite minimalization." It has been proved that computations that are limited with reliable watchdog timers can avoid the Gödel-turing paradox to the reliability of the timer. This proof is a fundamental theorem for practical self-modifying systems[33].

Realistic CR frameworks need to take into account a large set of possible configurations, however, as mentioned hereabove through the Gödel-paradox, the decision making engine also needs to be constrained in order to avoid the system to crash. We argue in the rest of this paragraph that, in general, CR decision making problems are better defined by their constraints rather than by their degrees of freedom.

When designing such CR equipments the main challenge is to find an appropriate way to correctly dimension its cognitive abilities according to its environment as well as to its purpose (i.e., providing a certain service to the user). Several articles in the literature have already been concerned by this matter however their description of the problem usually remained fuzzy (e.g., [6, 14, 34–36]). We summarize their analysis by defining three "constraints" on which the design of a CR equipment depends: First, the constraints imposed by the surrounding environment, then the constraints related to the user's expectations and finally, the constraints inherent to the equipment. We argue that these constraints help dimensioning the CR decision making engine. Consequently, an a priori formulation of these elements helps the designer to implement the right tools in order to obtain a flexible and adequate CR.

• The environment constraints: since a CR is a wireless device that operates in a surrounding communicating environment, it shall respect its rules: those imposed by regulation for instance (e.g., allocated frequency bands, tolerated interference, etc.) as well as its physical reality (propagation, multi-path and fading to name a few) and network conditions (channel load or surrounding users' activities for instance). Thus the behavior of CR equipments is highly coordinated by the constraints imposed by the environment. As a matter of fact, if the environment allows no degree of freedom to the equipments, this latter has no choice but to obey and thus looses all cognitive behavior. On the other side, if no constraints are imposed by the environment, the CR will still be constrained by its own operational abilities and the expectations of the user.

• User's expectations: when using his wireless device for a particular application (voice communication, data, streaming and so on), the user is expecting a certain quality of service. Depending on the awaited quality of service, the CR can identify several criteria to optimize, such as, minimizing the bit error rate, minimizing energy consumption, maximizing spectral efficiency, etc. If the user is too greedy and imposes too many objectives, the designing problem to solve might become intractable because of the constraints imposed by the surrounding environment and the platform of the CR. However if the user is expecting nothing, then again there is no need for a flexible CR. Usually it is assumed that the user is reasonable in a sense that he accepts the best he could get with a minimum cost as long as the quality of service provided is above a certain level.^d

• Equipment's operational abilities: These limitations are perhaps the most obvious since one cannot ask the CR equipment to adapt itself more than what it can perform (sense and/or act). It is usually assumed in the CR literature that the equipment is an ideal software radio, and thus, that it has all the needed flexibility for the designed framework. On a real application the efficiency of CR equipments depends of course on the degrees of freedom (or equivalently the constraints) inherent to the wireless platform used to communicate. As examples of commonly analyzed degrees of freedom one can find: modulation, pulse shape, symbol rate, transmit power, equalization to name a few. In all cases, a CR is designed to target and support given scenarios. We do not consider that CR can be designed to answer all scenarios or concepts [18].

The interaction between all three constraints is further emphasized through the notion of design space. We denote by CR design space an abstract three dimensional space that characterizes the CR decision making engine as shown in Figure 4. It is indeed abstract since it does not have any rigorous mathematical meaning but it is only used to visually and conceptually illustrate the dependencies of the CR decision making engine to the "design dimensions": environment, parameters (usually referred to as knobs) and objectives (or criteria defined from the user's expectations).

Cognitive radio decision making design space.

Full size image

In Figure 4, we represent two sub-spaces referred to as actual design space and virtual design space. On the one hand, the virtual design space refers to the upper bound support of the design space where all three dimensions are considered independently from each others. Its volume can be interpreted as the largest space of decision problems one could define from the three dimensions. On the other hand, the actual design space is included in the virtual design space. It results from the reduction of the design space when taking into account the correlation between the different constraints imposed by every dimension of the design space. For instance, some constraints on the environment such as, "imposed fixed waveform" might limit some objectives such as "find a waveform that maximizes the spectral efficiency".

To define a specific decision making problem, one needs to introduce a last-possibly implicit- function. This latter represents a functional relationship between all three dimensions, more specifically the correlation between the different constraints as illustrated by the design space. Thus, it models the interdependence of all three constraints. A simple representation of this interdependence can be expressed through an explicit objective function which numerical value is computed as a function of the equipment parameters, the environment's conditions as well as the values of other objective functions. Unfortunately such functions are not always available and might remain implicit. In such scenarios, optimization might prove problematic without using appropriate learning tools.

Finally, based on the here above presented analysis, all configuration adaptation problems seem to have the same roots. However, to define a specific problem among the set of possibilities in the design space, prior knowledge is important. This latter notion is further detailed in Section 4, where a classification of decision making tools as a function of prior knowledge is suggested. Nevertheless, the general DCA problem can be described as the most general decision making design space that we can state as follows [7]:

Within this framework, we assume that the environment constrains the CR by allowing only K possible configurations to use. This condition characterizes the environment and the equipment. Moreover we assume that there exist M ≥ 1 objectives that evaluate how well the equipment performs to meet the users expectations.

To conclude, we usually observe in the literature that these constrained based characterizations are implicitly made. Thus, usually the assumptions introduced to define the decision making framework are, unfortunately, hardly explained. These assumptions concern what we refer to as the "a priori model knowledge". In Section 4, we introduce and explain the notion of a priori knowledge and we present a brief state of the art on decision making for CR configuration adaptation using the DCA design space. We show that although the design space is the same, depending on the a priori model knowledge, different approaches are suggested by the community to tackle the defined decision making problems.

The following section describes an important case of DCA know as DSA that we briefly describe for the sake of consistency.

3.2. Spectrum scarcity and dynamic spectrum access

Since the early 90s, the radio community captured the potential industrial and economic opportunities that could emerge from a better frequency resource usage as noticed in 2004 in article [37]: A trend that has the potential to change the current industrial structure is the emergence of alternative spectrum management regimes, such as the introduction of so called "unlicensed bands", where new technologies can be introduced if they fulfil some very simple and relaxed "spectrum etiquette" rules to avoid excessive interference on existing systems. The most notable initiative in this area is the one of the federal communications commission (FCC, the regulator in USA) in the early 90s driving the development of short range wireless communication systems and wireless local area networks (WLANs).

Exploiting portions of the spectrum to unlicensed usage was a first step to introducing alternative frequency management schemes. Rethinking the main regulatory frameworks imposed for decades is the next step. As a matter of fact, during the last century, most of the meaningful spectrum resources were licensed to emerging wireless applications, where the static frequency allocation policy combined with a growing number of spectrum demanding services led to a spectrum scarcity. However, several measurements conducted in the United-States first, and then in numerous other countries [8, 23–27], showed a chronic underutilization of the frequency band resources, revealing substantial communication opportunities.

With the advent of SDR technology, it became, at least theoretically, possible to design agile systems capable of switching from one frequency band to another depending on given communication constraints. Thus, during the years 2002 and 2003 several task forces and researches suggested new frequency management policies and regulatory frameworks to enable efficient use of the spectrum resource [8, 38–43]. The consequences of this new framework are that the spectrum management model of today is abolished for large parts of the spectrum. Instead, "free"^espectrum trading becomes the preferred mechanism and technical systems that allow for the dynamic use and re-use of spectrum becomes a necessity[37].

The DSA encompasses all suggested approaches that emerged from the early definitions of efficient and "free" spectrum access or trading. In 2007, article [44] suggested one possible and simple taxomony^f to classify the different suggested spectrum management approaches as illustrated in Figure 5. Three main approaches can be discriminated: dynamic exclusive use model, open sharing model (spectrum commons model), and hierarchical access model:

Classification of several dynamic spectrum access approaches as suggested in article [44]. Three main approaches can be discriminated: dynamic exclusive use model, open sharing model (spectrum commons model), and hierarchical access model.

Full size image

• Dynamic exclusive use model: the spectrum basically is allocated exclusively to specific services or operators. However, the spectrum property rights framework allows opening a secondary market where the licensed users can sell and trade portion of their spectrum, whereas the dynamic spectrum allocation framework aims at providing a better allocation of the spectrum, to exclusive services, by adapting the spectrum allocation to space and time network load information.

• Open sharing model (spectrum commons model): aims at generalizing the success encountered by WLAN technologies within the ISM band. In other words, it mainly suggests opening portions if the spectrum to unlicensed users.

• Hierarchical access model: this framework introduced a secondary network that aims at exploiting resources left vacant by the incumbent users [usually referred to as primary users (PU)]. Secondary users (SUs) are able to communicate as long as they do not cause harmful interference to PUs. In this article, we do not subdivide this framework. As a matter of fact, their are as many subsets as the possible communication opportunities to exploit: power control, ultra-wide band communication under PUs noise level, spectrum hole detection and exploitation, directional communications to name a few [11]. In general, it is refers to as opportunistic spectrum access (OSA).

Since the seminal article of Haykin [10] in 2005, OSA research community has been, to the best of authors' knowledge the most active in the field of DSA. With several network models based on game theory [13], Markov chains or multi-armed Bandit (MAB) (and machine learning in general) [44–50], to name a few, and relying on the concept of CR, the community tackled several challenges encountered when dealing with OSA such as (non exhaustive): dynamic power allocation, optimal band selection (with or without prior knowledge on the occupancy pattern of the spectrum bands by PUs), as well as cooperation among the different SUs [12] centralized or decentralized, with or without observation errors.

In Section 5.2 an OSA scenario based on a MAB model, described in article [48], is summarized and illustrates the impact of observation errors on decision making for CR. In the following section, however, we introduce prior knowledge as a classification criteria among the main learning and decision making tools suggested in CR articles.

The a priori knowledge is a set of assumptions made by the designer on the amount and representation of the available information to the decision making engine when it first deals with the environment. As a matter of fact, "knowledge" is defined by the Oxford english dictionary as: (i) expertise, and skills acquired by a person through experience or education; the theoretical or practical understanding of a subject, (ii) what is known in a particular field or in total; facts and information or (iii) awareness or familiarity gained by experience of a fact or situation. Consequently, within the CR framework, we can define the a priori knowledge as the set of theoretical or practical assumptions provided by the designer to the CR decision making engine. These assumptions, if they are accurate, provide the CR with valuable information on the problem to deal with. These remarks lead us to suggest that the decision making problems the CR has to deal with are defined by the set {design space, a priori knowledge}. In other words, depending on the a priori knowledge on the environment, some decision making approaches offer a better fit to the decision making framework than others. Moreover, we assert that a few, if not many, different cognitive engines could cohabit in a single CR equipment and will have to coordinate their actions [51]. Thus, recently (2011), a CR decision making engine based on prior knowledge has been suggested in [52]. In the following sections we briefly describe the different approaches provided by the community depending on the a priori knowledge assumed relevant to tackle the environment the CR might face during its life time. In Figure 6 we suggest to classify these techniques depending on the a priori knowledge provided to the cognitive decision making engine.

Suggested decision making techniques depending on the assumed a priori knowledge.

Full size image

4.1. Expert approach

The expert approach relies on the important amount of knowledge collected by telecommunication engineers and researchers. This knowledge is based on theoretical consideration and practical measures on the environment and radio communication parameters. It was first suggested by Mitola in his Ph.D. dissertation on CR [6]. Through intensive off-line simulations, expert systems are provided with a set of inference rules. These rules are then used on-line to adapt the equipment depending on the context faced by CR equipments. Thus, the more available knowledge, the better the equipment can adapt itself to its surrounding dynamic environment. However, this knowledge is usefully as long as if the CR can represent its knowledge in a way that enables to exploit it and to react to the environment by adequate adaptations of its operating configuration. For that purpose, Mitola suggested representing the knowledge of CR equipments using a new dedicated language radio communication: "radio knowledge representation language" (RKRL) [6, 33]. This representation of knowledge uses web semantic such as XML (eXtensible Markup Language), RDF (resource description framework), and OWL (web ontology language). The expert knowledge based approach had a large success especially due to the XG project (neXt Generation) supported by the DARPA (e.g., [53] and for spectrum sharing: [54]). As a matter of fact, if the knowledge is well represented and provided to the equipment as a set of rules, the decision making process becomes very simple. However this approach has a few drawbacks:

• The behavior of the designed system is not tuned to a particular user but to all users and to a set of probable environments. Moreover in order to acquaint the CR decision making engine with valuable and large knowledge, an important amount of effort is needed from the designer.

• Expert knowledge is mainly based on models. Thus the system might behave in a poor way when it is facing unexpected dynamics in the environment.

The techniques based on expert systems can, however be supported by several other tools (some are discussed later) to help them acquire new knowledge on the environment or help them avoid conflicts between different configuration adaptation rules. A similar approach, based on an ontology to model the knowledge of the decision making engine was recently suggested [55–58]. Where a common language to radio devices is suggested based on an ontology, expressed in OWL and implemented on the USRP card [59] using GNU radio [60].

4.2. Exploration based decision making

In some contexts, one can consider that there is a priori knowledge available on the complex relationships existing between, the metrics observed, the parameters to adapt and the criteria to satisfy as described in Figure 7. In this case the problem appears to be a multi-criteria optimization problem. Within this framework, the CR decision making engine aims at finding the best parameters to meet the users expectations by solving a set of equations as shown in Table Two of article [61] from which is extracted Figure 7). This problem is known to be complex for several reasons:

Multi-criteria optimization problem [61].

Full size image

• there exists no universal definition of optimality in this case. Thus the solution of this problem are satisfactory (or not) with respect to a certain function, usually named fitness that evaluates how well the criteria were satisfied.

• Thus usually a large space of possible "good" configurations can be available.

• The criteria are correlated and can be in conflict (e.g., Figure 7).

If we assume that the previously mentioned off-line expert rule extraction phase has not been (or partially) accomplished an exploration of the space of possible configurations is needed.

There exists various possible algorithm to explore a large set of potential candidates. The most obvious one is probably "exhaustive search", where all possible candidates are computed and evaluated in order to find the best solution. However, when the number of candidates grows large, such approaches can become computationally burdensome and miss the imposed decision making deadlines. Usually in such contexts, heuristics are preferred. In the context of CR, finding the best solution might not be necessary. Instead, the cognitive engine would rather find, within the imposed limited amount of time, a satisfactory solution.

Consequently, if the following criteria are met:

• Available a priori knowledge on the complex relationships existing between, the metrics observed, the parameters to adapt and the criteria to satisfy.

• Possible heavy parallel computing.

Then a large set of decision making tools are possible such as: simulated annealing, GAs, and swarm algorithms to name a few [62]. Notice that such approaches did not wait for CR to be used on radio technologies. In 1993, article [63] already suggested simulated annealing as a possible solution to deal with channel assignment for cellular networks.^g

Genetic algorithms [14, 34, 61], Swarm Algorithms [64, 65] and insect colony inspired algorithms [66]^h techniques are usually referred to as bio-inspired or evolutionary techniques.

This defined CR decision making framework was first analyzed by Rieser and Rondeau. They suggested the use of GAs to tackle this framework [14, 34, 61]. GAs were first designed to mimic Darwin's evolutionary theory and are well known for their capacity to adapt themselves to a changing environment. Without using our formalism, their study showed that under what we define as design space and with the described a priori knowledge, the GAs provide cognitive radios with an efficient and flexible decision making engine. But we cannot consider their model as a generality for all CR use cases, so that other solutions have to be considered additionally. Further details on the different versions suggested and implemented by Virginia Tech can be found in the following recent survey [67].ⁱ

Notice, that once again, prior knowledge can substantially enhance the behavior of these algorithms. An interesting illustration can be found in article [52] in the case of GAs based decision making engines.

4.3. Learning approaches: exploration and exploitation

As we argued in the previous sections and as several other authors [36, 68] noticed, "Many CR proposals, such as[61, 69, 70], rely on a priori characterization of these performance metrics which are often derived from analytical models. Unfortunately, [...], this approach is not always practical due to e.g., limiting modeling assumption, non-ideal behaviors in real-life scenarios, and poor scalability" [68]. To avoid these limitations and in order to tackle more realistic scenarios, many methods based on learning techniques were suggested: artificial neuronal networks (ANN), evolving connectionist systems (ECS) [71, 72], statistical learning [73], regression models and so on. All of these approaches have their cons and pros, however they all have in common that they mainly rely on trials conducted within a real environment to try and infer from it decision making rules for CR equipments. Since this learning tools aim at representing the functional relationship between the environment (through the sensed metrics), the systems parameters and the criteria to satisfy, they need a direct interaction with the environment in order to build a posteriori knowledge on their environment. In this study we sub-classify these methods depending on the way they learn and exploit their rules. On the one hand (i), we find a set of techniques that separates exploration and exploitation phases. On the other hand (ii), we find other techniques more flexible that combine both processes.

In the first mentioned case (i) we find several tools such as ANN or statistical learning already used and exploited in other domain requiring some cognitive abilities (robotics, video games, etc.). These methods have two phases: a phase of pure "exploration" where the CR decision making engine learns and infers to find (explicitly or implicitly) decision making rules, then uses in a second phase this a posteriori knowledge to make decision. Since these learning techniques rely on a first learning phase, a large amount of data and computational power is needed in order to extract reliable knowledge. This difficulty is already known concerning ANN for instance. It is still true for statistical learning. As noticed by Weingart in article [73], the provided techniques are still computationally prohibitive, and not ready yet to be used in a real equipment. However if the first phase is well achieved the second phase is usually very simple and does not require much time or energy [68]. In the second case (ii), we find promising techniques recently introduced to the community and still need to be further investigated [17, 36] in the case of configuration adaptation.^j These techniques try to provide the CR with a flexible and incremental learning decision making engine. In the case of ECS based decision making engine, Colson suggested the use of an evolving neural network [71, 72]. Unlike the usual ANN, the ECS-NN can change its structure without "forgetting" already learned knowledge. Thus new rules can be learned by adding new neurons to the neural structure. In order to be efficient the architecture proposed in [36] needs some expert advice (a priori knowledge) on the several available configurations. These added information ranks the different configurations based on some criteria (robustness, spectral efficiency, etc.) but without knowing a priori which one is more adequate when facing a certain environment.

More recently, article [17] however assumes that no a priori knowledge is provided and that the performance of the equipment can only be estimated when trying a specific configuration. The associated tools are based on the so-called MAB framework. One advantage here is to provide learning solutions while operating, even if the cognitive engine is facing a completely new environment. Of course, performance increase while the learning process progresses. Note that this approach is also proving its accuracy in the OSA context [47].

To conclude this section, we would like to emphasize the fact that the proposed classification in this article shows that a CR equipment cannot depend on only one core decision making tool but on a pool of techniques. Every time it faces an environment, the equipment needs to have an estimation of its a priori knowledge and on its reliability. To tackle a particular context, the general process can be summarized through three questions: What can't I do (design space)? What do I already know (a priori knowledge)? And what technique should I select to solve the decision making problem?

In the following section we extend the analysis to the specific and practical context of imperfect sensing. As a matter of fact the impact of sensing errors can be significant on decision making techniques. However, unfortunately, very few studies seem to tackle this specific problem within CR contexts. Hence, we further discuss this matter hereafter.

As illustrated through the notion of basic cognitive cycle, decision making, and learning rely on prior observations of the environment. Consequently, the performance of the implemented decision making tools highly depends on the quality of the observations. Unfortunately, we could not find substantial quantitative material evaluating the impact of sensing errors on decision making and learning tools. Thus, we suggest to qualitatively^k discuss, in this section, the impact of sensing errors on the previously discussed decision making tools for CR. For that purpose we rely on a specific problem borrowed from the OSA^l community to illustrate this discussion where the problem of decision making in the context of sensing errors is clearly formalized and the impact of such errors on the considered learning algorithm's performance is quantified.

5.1. An example of learning approach

Opportunistic spectrum access is a particularly interesting framework that illustrates the challenge faced when learning under uncertainty. When tackling the general DCA problem, described hereabove, while considering K channels to probe, the problem that consists in maximizing the cumulated throughput of the user over the number of transmission trials appears to be consistent with a MAB paradigm [74, 75]. In a nutshell, based on the analogy with the one-armed bandit (also known as slot machine), it models a gambler sequentially pulling one of the several levers (MAB) on the gambling machine. Every time a lever^m is pulled, it provides the gambler with a random income usually referred to as reward. Although we assume that the gambler has no a priori information on the rewards' stochastic distributions, he aims at maximizing his cumulated income through iterative pulls. In the OSA framework, the SU is modeled as the gambler while the frequency bands represent the levers. The gambler faces at each trial a trade-off between pulling the lever with the highest estimated payoff (known as exploitation phase) and pulling another lever to acquire information about its expected payoff (known as exploration phase). We usually refer to this trade-off as the exploration-exploitation dilemma. If the problem is assumed modeled as a MAB framework an interesting way to tackle the problem is to use the class of so-called upper confidence bound algorithmsⁿ (UCB) [17, 47, 48, 50, 76]. The main advantage of UCB methods for CR is to offer a balance between exploration and exploitation phases without interrupting the communication process, i.e., while providing a certain service to the user [17]. Namely, a CR based on UCB can jointly communicate and learn. Thus it avoids the instantiation of two steps: a learning step during which the user has to wait. And a communication step that depends on how well the first step performed. It is worth noticing that the suggested illustration, in the article, is based on the so-called UCB₁. This latter has been selected for its rather low computational complexity compared to other techniques in the literature.

For illustration purpose, we use the following decision model for OSA of a SU having the choice between ten frequency bands, each one used by PUs with a different probability, usually unknown to the CR decision making engine. A complete model is provided in [48]. Only one band can be sensed and tried at each iteration in order to keep the system's complexity reasonable. Consequently, the cognitive engine only has a partial information on the environment at each iteration and should derive the probability of availability of the bands based on its previous trials. It provides a confidence bound on every band and selects, for the next iteration, the band most likely to be free. Communication can be performed if the band is detected as free; otherwise the SU backs off. However, the SU can make errors due to the non perfect accuracy of its sensing detector. More specifically, the detector might detect the presence of a PU while the band is in fact free and vice-versa. The consequence is that the SU does not transmit during this iteration whereas he could, or transmits when he should not causing interference to the incumbent users. We usually speak of false alarm in the former case and miss-detection in the latter case.

We see in Figure 8 the impact of false alarms on the proportion of time a cognitive engine, relying on the UCB₁[77, 78] algorithm, choses the most available channel (considered as optimal in this case). This proportion increases as the number of trials grow large, thus as the SU learns more on the availability of the bands. We can see that with a probability of false alarm equal to zero, the decision making engine needs 1,000 trials to obtain a selection rate of 72% of the most available channel. This ratio falls to 50% after 1,000 trials for a probability of false alarm of 0.4, which is quite decent considering the scenario and the heavy deterioration of sensing accuracy. In fact, in this case, a little bit more than twice the number of iterations has been necessary compared to a perfect sensing scenario. But after 10,000 trials, the ratio grows to achieve 96 and 92%, respectively. Consequently, the cognitive engine is able to communicate and to converge towards the most available band in spite of the sensing errors it is suffering.

Percentage of time the UCB ₁ -based CA selects the optimal channel under various sensing errors frameworks (over ten available channels): impact of sensing errors on the learning performance of the algorithm UCB ₁ .

Full size image

5.2. The impact of observation error and uncertainty on decision making

Analyzing the impact of uncertainty and sensing errors on the performance of a CR decision making engine is very difficult. However due to the importance of this problem to the community, we suggest as a closing point of this article, an intuitive and brief insight view on this matter. Within this framework we consider that the sensing information we capture from the environment may contain errors. Then we describe the potential consequence of such errors on the performance of class of algorithms previously classified.

Due to their lack of flexibility, expert decision making techniques seem to be the most vulnerable to uncertainty. As a matter of fact, their decision making process, based on either rules or predefined policies, leads the CR to consider all observations as being correct. Hence a sensing errors provokes a behavioral error. GA based decision making engines rely on explicit relationships between parameters, observations and criteria. Consequently, sensing errors can highly impact the selection process as it introduces biases in the performance evaluation of the different candidates. Moreover, generation after generation, these errors would probably propagate leading to an inefficient selection process. Such decision making engines would probably need to interact with environment to test the candidates and confirm their performance. In such scenarios, the CA might be able to mitigate the impact of sensing errors at the cost however of a burdensome process. ANN are usually depicted, when they fulfill given requirements, as universal approximators. In other words, if the neural network is correctly designed to fit the decision making problem, it can efficiently learn the implicit relationship that exists between parameters, observations and criteria. Consequently even when sensing errors are present, the learning process can lead to capture average patterns and thus appropriately mitigate their impact. Thus, the more learning abilities and flexibility a decision making shows the more robust it become to uncertainty and sensing errors. This analysis is further depicted in Figure 6. Thus, we can summarize this intuitive insight view as follows: the more the decision making technique is at the right of Figure 6, the more robust to observation flaws it seems to be. Notice that the learning process enable the CA to acquire knowledge on its environment. Consequently a learning process fully achieved should lead to an expert decision. Figure 9 illustrates moreover a vertical axis that suggests, when possible, that collaboration helps CR users to acquire through diversity a better information on their environment. And thus, it enables them to improve the performance of their decision making engine considering a given uncertainty level.

Decision-making techniques classification based on a priori knowledge in the context of noisy sensing.

Full size image

Taking into account the uncertainty on the environment sensing, we may assert that learning-oriented techniques are more efficient. This is emphasized by the proposed classification based on the a priori knowledge criteria on the environment. Hence, we believe that such approaches should be particularly addressed by the CR community in the second decade of CR decision making era.

We tackled in this article decision making in the sense of a mono-equipment problem. In a multi-equipment context, a higher level of decision (rule, policy, etc.) should specify how equipments cooperate or not. This is out of the scope of this article. However, at the level of each equipment, decision goes back to what has been stated in this article.

In this article, we presented a brief yet original retrospective view on the first 10 years of CR. More specifically of the different challenges faced by the CR decision making community and the suggested solution to answer them. We state that most of these decision making models have the same design space however they differ by the a priori knowledge they assume available. Consequently, we suggested the "a priori knowledge" as a classification criteria to discriminate the main proposed techniques in the literature to solve configuration adaptation decision making problems. Moreover as a qualitative and prospective analysis, we depicted through an toy example the impact of observation errors and uncertainty on CR decision making engine.

We believe that this analysis made on the first 10 years of exploration of decision making for CR may help gaining perspective on the topic and thus help addressing this research domain for the next coming 10 years.

^aBoth US and European military have been working on such flexible and inter-operable defense systems since the late 1970s. ^bIt is called full CR to oppose it to other simplified versions suggested in the literature [4]. ^cA specific example of such paradox can be illustrated by the following sentence: 'This sentence is false!' [79] as suggested by Mitola during a recent seminar at Supélec, http://www.rennes.supelec.fr/ren/rd/scee/seminaire.html. ^dNotice that this assumption introduces the notion of satisfactory behavior. We oppose it to rational thinking where the decision making engine always aims at the most rewarding option. Thus when the decision making engine needs to learn in an uncertain environment, satisfaction based reasoning can be introduced to accelerate the convergence rate of learning algorithms for instance. ^e[...] "Trade, lease, and rent of licenses were possible without incurring excessive administrative procedures and overhead costs" [37]. ^fA different, more detailed and more exhaustive, DSA taxomony can be found in article [80]. ^gIt is indeed a very restrictive case of DCA and DSA where a centralized entity, seen as the cognitive agent (CA) assigns frequency channels to its users depending on the channel conditions. ^hTo the best of authors' knowledge Swarm algorithms have only been exploited in case of resource allocation. No complex configuration adaptation decision making engine was found in the literature based on such techniques. ⁱThis document is presented as a survey of the various suggested decision making architectures for CR. We notice however, that except the one designed by Mitola, during the DARPA xG Program, and those designed and implemented by Virginia Tech, the community around this topic seems thin and advances slowly toward efficient architectures. Other suggested architectures relying mostly on bio-inspired techniques tackle spectrum resource allocation related problems. ^jThese same techniques, based on a MAB model, prove to be efficient to tackle some DSA related problems as already discussed in Section 3.2. ^kTo the best of authors' knowledge such studies have only been conducted when dealing with OSA problems. Consequently, the presented results are exploratory and need further investigations to fully confirm them. We find however the overall discussion interesting to capture CR related decision making challenges. ^lAs mentioned earlier, we consider in this article that OSA problems are but specific instantiations of DCA problems. ^mFrom a DCA problems perspective, a lever is a specific configuration to be tested. Thus in OSA, it refers to a band to probe for instance. ⁿUCB algorithms are given here as an example of learning-oriented approach. But the philosophy and conclusions of this section would match other learning techniques.

Affiliations

1. SUPELEC, SCEE/IETR, Avenue de la Boulaie, CS 47601, 35576, Cesson Sévigné Cedex, France
   • Wassim Jouini
   • , Christophe Moy
   •  & Jacques Palicot

Corresponding author

Correspondence to Wassim Jouini.

Competing interests

The authors declare that they have no competing interests.

Reprints and Permissions