 Research
 Open Access
 Published:
Radio resource management for public femtocell networks
EURASIP Journal on Wireless Communications and Networking volume 2011, Article number: 181 (2011)
Abstract
With evolution and popularity of radio access technologies, the radio resource is becoming scarce. However, with fastgrowing service demands, the future advanced wireless communication systems are expected to provide ubiquitous mobile broadband coverage to support higher data rate. Therefore, it is becoming an important problem that how to meet the greater demand with limited resources? In this situation, the femtocell has recently gained considerable attention. It is an emerging wireless access point that can improve indoor coverage as well as reduce bandwidth load in the macrocell network, and seem to be more attracted since the indoor traffic is up to 75% of all in 4G network. According to the newest researches, deployment of femtocell base station in public places' applications (campus, enterprise, etc.) is of much broad prospect, which could provide high quality, high rate wireless services to multiple users as well as effectively improved resource utility. However, a key challenge of the public femtocell networks is the utilitybased resource management. In public femtocell networks, multiunits are necessary to jointly provide high rate, high quality services to indoor users, but there is often heavy resource competition as well as mutual interference between multiple femtocells. Therefore, it's very critical to optimize the radio resources allocation to meet femtocells' requirement as possible and reduce interference. What is more, using some ingenious resource allocation technique, multiple femtocells can cooperate and improve the system performance further. In this article, we proposed a systematic way to optimize the resource allocation for public femtocell networks, including three schemes of different stages: (1) longterm resource management, which is to allocate spectrum resource between macrocell and femtocell networks; (2) mediumterm resource management, which is to allocate radio resources to each femtocell; (3) fast resource management, which is to further enable multiple femtocells to cooperate to improve the network's coverage and capacity. Numerical results show that these radio resource management schemes can effectively improve radio resource utility and system performance of the whole network.
1. Introduction
With evolution and popularity of radio access technologies, the radio resource is becoming scarce. However, with fastgrowing service demands, the future advanced wireless communication systems are expected to provide ubiquitous mobile broadband coverage to support higher data rate. Therefore, it is becoming an important problem that how to meet the greater demand with limited resources? In this situation, the femtocell has recently gained considerable attention. It is an emerging lowpower, lowcost data access point that can improve indoor coverage as well as reduce bandwidth load in the macrocell network [1], and seem to be more attracted since the indoor traffic is up to 75% of all in 4G network [2]. Although femtocells were initially targeted at consumer offers, it was immediately clear that this technology presents a number of benefits for the enterprise case and for the coverage of open spaces.
According to the newest researches [3], deployment of femtocell base station (FBS) in public places' applications (campus, enterprise, etc.) are of much broad prospect, which could provide high quality, high rate wireless services to multiple users as well as effectively improved resource utility. Small office/home office business users can have immediate benefits by utilizing a consumer unit, typically with local access enabled to connect to local LAN servers. Medium and large enterprises need a different solution as multiple units need to cooperate to provide the necessary coverage and capacity. Compared to picocells the public femtocells present obvious advantages in that they do not need dedicated links.
However, a key challenge of the public femtocell networks is the utilitybased resource management. Considering the large number of FBSs and the requirement of multiple units' cooperation [4], it may be highcost and inefficient to manually allocate resource for each FBS. What is more, in public places the path loss between femtocells are weak, so the interferences between femtocells are relatively serious and the common macrocell resource management schemes may not solve this problem well. Owing to the predicted widely adoption of femtocells, researchers have begun to consider the problem of coverage optimization [5–8]. However, all of these studies focus on single femtocell coverage optimization for smallarea residential users and provide good indoor coverage, preventing signals from leaking outdoors [6] as well as to increase the flexibility in deployment locations [7, 8], rather than on multiple femtocells that achieve joint coverage in large enterprise environments. For multiple femtocells, the main optimization goal is to optimize the resource allocation between femtocells and reduce coverage overlaps and gaps, as well as to balance the workload among femtocells.
In this article, we proposed a systematic way to optimize the resource allocation for public femtocell networks, including three schemes of different stages: (1) longterm resource management, which is to allocate spectrum resource between macrocell and femtocell networks. We proposed an adapted soft frequency reuse (ASFR) approach to combat traditional intercell interference by inheriting the conventional soft frequency reuse (SFR) functionality and to mitigate intertier interference (ITI) of macro/femtocells by applying an orthogonal spectrum reuse between macro/femtocells. In addition, we make the femtocells dynamically access macrocell's spectrum through cognitive radio (CR) technology, without interference with macrocell UEs; (2) mediumterm resource management, which is to allocate radio resources to each femtocell. In this stage, we used a Qlearningbased selfconfiguration scheme to configure the FBS's power and work channel according to the environment in public femtocell networks. The numerical results show that the proposed scheme performed well in improving network performance as well as complexity comparing with some other common approaches; (3) fast resource management, which is to fast manage radio resources between femtocells. We proposed a coordinated multipoint transmission technique to enable femtocells to cooperate to improve the network's coverage and capacity.
The remainder of this article is organized as follows: the system model will be described in Section 2. The details and analysis of the proposed schemes will be presented in Section 3, some analytical results and performance evaluation is given in Section 4, and the last section concludes the article.
2. System model
Consider OFDMAbased macrocells whose frequency reuse factor ζ > 1, public femtocell networks (enterprise femtocells, airport femtocells, etc.) and residential femtocells are deployed in the macrocell's coverage, as shown in Figure 1. According to the traditional SFR scheme, the macrocells are partitioned into two parts: central part and outer part, femtocells may be in either central or outer part. To mitigate interference between intermacrocells, the outer part of a cell can only use fractional spectrum and spectrums of different macrocells' outer parts are orthogonal.
Figure 2 shows a plan of the enterprise femtocell network in Figure 1. The aim of enterprise femtocell network is to meet the increasing demands for higher speed and higherquality wireless data services within office buildings, factories, apartment buildings, and other indoor propagation environments, where the usual macrocell system can only provide degraded services or provide no coverage at all. In this article, we considered the proposed radio resource management schemes in a typical enterprise office scenario, in which there are two meeting rooms, five open offices, one demo room, and one sitting room. To provide highquality services, each room is equipped with a FBS and totally M femtocells and N user equipments (UEs) are distributed in this network. There is also a femto gateway which connects the FBSs with core network, collects and stores the information from all the FBSs and UEs, and allocates radio resources (power, channel etc.) between the FBSs as well as sends parameter adjustment prompt to the FBSs according to the predefined scheme.
The femtocells deployed in enterprise and campus environment usually use hybrid access mode, by which the subscribers can preferentially access the femtocells network and nonsubscribers can access only when there are excess resources. The femtocells cover open places, such as railway stations, airports, and shopping malls are very similar with the enterprise ones, but for open spaces only open access mode is ever used. The discussion of access mode is beyond the scope of this article, so we consider the UEs can access the femtocell network if only there are enough resources.
3. Radio resource management schemes
3.1. Longterm resource management: spectrum allocation between macrocell and public femtocells
Spectrum is one of the most important resources for wireless networks, public femtocell networks are no exception. Usually, there are typically two types of spectrum assignment schemes for coexistence of macrocells and femtocells [9]. One is shared spectrum allocation (cochannel), by which femtocell uses the same frequency band as the macrocell, this results in more effective use of resources and efficient handoff (due to easier cellsearch), but the interference from the macrocell BS may seriously degrade the performance. The other is split spectrum allocation, by which the femtocells use different frequency bands than those employed by the macrocell; while this avoids interference to/from the macrocell, additional spectrum resources are required.
In this article, we proposed an ASFR approach based on the traditional SFR, to eliminate interference between macrocell and femtocells, as well as maximize available spectrum resources for femtocell networks.
3.1.1. ASFR premier
Figure 3 gives us an overview of the ASFR spectrum allocation approach. Cells A, B, and C denote the three cells of a typical cluster. Like traditional SFR, the total available spectrum is divided into three orthogonal segments with equal size, respectively, utilized by celledge users of the three cells. Thus, the celledge bands of neighboring cells are orthogonal. As premier ASFR design, let FBS_{1,n}represent the n th femtocell access point within the coverage of macro base station (MBS) A. Assuming that the celledge UEs of MBS A are restricted to utilize spectrum segment j(j ∈ {1, 2, 3}), then the available spectrum segments for the FBS_{1,n}is (a) segment j (j ∈ {1, 2, 3}), if the FBS is at the cell center; (b) the other two segments, i.e., segment (j (mod 3) + 1) and segment ((j + 1) (mod 3) + 1), otherwise. In this way, FBSs reuse the spectrum of MBSs in an orthogonal approach and the intertier interference is mitigated. We should notice that the cellcenter borderline ρ, which is the distance from the local MBS and which divides the cellcenter and cellouter users, can be tuned for performance optimization. However, the ratio of available resource numbers for cell center and cell outer is fixed at 2:1. As a result, the available number ratio of cellcenter and cellouter FBSs is fixed at 1:2, which means that the cellcenter FBSs' capacity may be cut to only half of that in the cell edge, and which is unreasonable. We call this disadvantage as 1:2 issues. Therefore, to make the ASFR approach more applicable, designs to overcome this imbalance are essential.
3.1.2. ASFR evolution
The key ambition of this ASFR evolution is to provide designs immune to the 1:2 issue. Inspired by the EFFR approach introduced in literature [10], CR techniques is implemented in the ASFR design, and FBSs are offered additional secondary spectrumthe local macrocell radio channels, while their original available spectrum is made primary, as in Figure 3. Let FBS_{1,n}be in the cell edge of Cell A, by now it has one primary spectrum segment: j (j ∈ {1, 2, 3}), and two secondary spectrum segments: segment (j (mod 3) + 1) and segment ((j + 1) (mod 3) + 1). As we all know, secondary spectrum reuse is always accompanied by sorts of spectrum detection tools and criterions, and extra costs as well. To improve spectrum efficiency, while not to make the existing cellular system become too complex, detection of the secondary spectrum is triggered if and only if the primary spectrum is exhausted. In the following, an optimized detection criterion is generated, which further distinguish the two secondary segments of celledge FBSs: To improve the secondary spectrum detection efficiency, segment (j (mod 3) + 1) is designed as the primary one out of the two secondary spectrum of FBS_{1,n}, and ((j + 1) (mod 3) + 1) as the secondary one, respectively, named primarysecondary (PS) spectrum and secondarysecondary (SS) spectrum for FBS_{1,n}illustrated in Figure 3. The detection of SS is triggered if and only if PS is exhausted. This means that FBS_{1,n}starts secondary spectrum detection from PS (j mod 3 + 1) and only when the PS cannot satisfy the spectrum requirements, SS ((j + 1) (mod 3) + 1) is detected and allocated. Meanwhile, the MBS A is designed to start its spectrum allocation from a different point, i.e., SS other than PS of FBS_{1,n}. In this way, when loading factor of MBS A is lower than 1/3 and neglecting effects of other femtocells, the probability of successful secondary spectrum detection on PS of can be 100%, since the PS segment of FBS_{1,n}is not occupied by the local macrocell. Obviously, it would be too late to trigger secondary spectrum detection after the exhaustion of the primary one. Assuming that traffic load is estimated at each cell, a mechanism is designed to support suitable detection time: (1) two thresholds are defined: load thresh 1 and load thresh 2, with load thresh 1 < load thresh 2; (2) before the load of FBS_{1,n}reach load thresh 1, no channel measurements on PS is needed, and before the load of FBS_{1,n}reach load thresh 2, no measurements on the SS is needed. Thus, the overheads of detection are reduced and efficiency of detection is improved. Values of the two thresholds are closely correlated with the IFI condition of given environments.
3.2. Mediumterm resource management: channel and power allocation
Through the spectrum allocation between femtocell and macrocell, the interference between twotier networks can be mitigated, and by CR technology, the femtocells can access the macrocell's spectrum which the nearby macrocell UEs are not using, further improving the network's capacity and resource utility. However, among each femtocell of the femtocell networks, the small path losses may cause heavy interferences. Therefore, according to the bandwidth requirement of femtocells, the whole available spectrum including allocation and cognitive parts can be divided into several channels to be allocated to different femtocells, and the femtocells' power should be optimal configured as well.
Previous studies focused on the resource allocation between femtocells can be divided into two categories: (1) distributed selfconfiguring, each femtocell independently configures its work channel and power. This method has low complexity but does not consider impact on surrounding femtocells and probably causes interference. (2) Global resource allocation by calculating the optimal configuration for each femtocell. This method can achieve an optimal resource allocation but it needs a lot of information collecting and computing. In addition, this method solves the resource allocation problem by only one allocating process. In fact, the radio environment may change a lot over time as well as different femtocells switch on or off, so the realtime resource allocating is needed.
In this article, we proposed a Qlearningbased approach to deal with the resource allocation problem of femtocells for realtime. Simulation results show that the approach could configure the femtocell according to the environment, and optimize its performance without loss of other femtocells.
3.2.1. Reinforcementlearning
Qlearning is one kind of reinforcementlearning. The reinforcementlearning model consists of several factors [11, 12]: (1) S = {s_{1}, s_{2},..., s_{ n }} denotes the finite discrete possible environment states, (2) A = {a_{1}, a_{2},..., a_{ m }} denotes the possible using actions of the agent, and (3) r denotes the current reward value, (4) π:S → A is the agent's strategy. The relationships between these factors are shown in Figure 4:

1.
Agent perceives the environment and decides the state s;

2.
Agent choose an action a according to the current strategy π: S → A and have an effect on the environment.

3.
The environment receives the action a and then transforms from state s to s' by a certain probability p, after that it will generate a current reward r and feed back to agent.

4.
The agent updates its strategy π: S → A according to s' and r.
Through continuous implementation of the above process circle, the ultimate goal is to find the optimal strategies for the agent in each state s, making the cumulative return on a given optimization object maximum/minimum. One most common infinite horizon optimization objective is the mathematical expectation of the longterm cumulative return:
r is a constant time discount factor, which reflects the importance of the future return relative to the current return, the smaller r is, the less important the future return is. According to [13], (1) can be rewritten as
R(s, a) is the mathematical expectation of r(s_{ t }, a_{ t }), P_{ s, s' }(a) is the probability that state s transforms to s' after executing action a.
3.2.2. Qlearning
Compared with other reinforcementlearning algorithms, Qlearning has the advantages that it can directly find the optimal strategy through value iteration [14] to satisfy (2) [15, 16], without knowing R(s, a) and P_{ s, s' }(a). The specific method is each state and action pair (s, a) is associated with a Qvalue Q(s, a), the (2) deformation:
Its meaning is the expected cumulative return by executing action a in state s then following a serious of actions obeying the strategy π.
In order to obtain the optimal strategy π that makes
where ${V}_{*}\left(s\right)=\underset{a\in A}{max}{Q}_{*}\left(s,a\right)$,
We can make the Qlearning process as follows:
where α ∈ [0,1) is the learning rate, and ΔQ_{ t }(s, a) is Q value update error function, as follows:
It can be proved that if each Q(s, a) value can be updated through an infinite number of iterations, and in this process in some appropriate way α gradually reduced to 0, Q(s, a) will converge to the optimal value of the probability of a Q*(s, a). At this point, the optimal strategy*π can be obtained as
3.2.3. Qlearningbased channel and power allocation formulation
The proposed Qlearningbased selfconfiguration scheme is described as follows:

1.
The FGW maintains a Qvalue table for femtocell's selfconfiguration, as shown in Table 1. This table is twodimensional, one dimension is all possible states s, while the other denotes all possible actions a. Each unit Q(s, a) denotes the Q value, i.e., the value of the objective function, when the action a is chosen at state s.

2.
State s(P, C, T) is mainly describing the environment of the femtocell k which needs selfconfiguration, and we used three elements related to its neighbor femtocells to comprise s: (1) P = (p_{1},..., p_{ N }) denotes neighbor femtocells' (1  N) pilot power received by femtocell k, p_{ i }∈{0, 1, 2}, respectively, denote low (< 10 dBm), moderate (1015 dBm), and high (1520 dBm) power. To reduce the complexity without loss of performance, the most nearest two neighbor femtocells are chosen and N is supposed to be 2. (2) Neighbor femtocells' work channel vector C = (c_{1}~c_{ N }), c_{ i }∈{0, 1, 2,3}, respectively, denotes the four channels which the whole spectrum is divided into. (3) Neighbor femtocells' throughput vector T = (t_{1}~t_{ N }), t_{ i }∈{0, 1, 2}, respectively denotes low (< 2 bps/Hz), moderate (25 bps/Hz), and high (> 5 bps/Hz) throughput.

3.
Action a(p_{ a }, c_{ a }) is the possible combinations of power p_{ k }∈{10, 15, 20}dBm and work channel c_{ k }∈{0, 1, 2, 3} that femtocell k may be configured. a(p_{ k }, c_{ k })∈{0~12} denotes 1 of the 12 kinds of femtocell configurations.

4.
Selection criteria for action a(p_{ a }, c_{ a }): if we adopt the greedy algorithm, that is always at each iteration to select the action a(p_{ a }, c_{ a }) that makes Q(s, a) maximum in current state s(P, C, T), probably because the initial iteration algorithm improper selection (due to lack of accumulated experience) and ultimately "cover up" the optimal strategy. In this article, we choose a(p_{ a }, c_{ a }) more representative method: the Boltzmann distributionbased exploration algorithm. Specifically, in state s(P, C, T), Boltzmann distribution algorithm selected an action with following probability:
$$p\left(as\right)=\frac{{e}^{Q\left(s,a\right)\u2215{T}_{s}}}{\sum _{{a}^{\prime}\in A}{e}^{Q\left(s,{a}^{\prime}\right)\u2215{T}_{s}}}$$(8)
where T_{s} is the "temperature" parameter, and decreases with the Q value iterative process. Equation 8 expressed the basic idea that with the constant iteration of Qlearning algorithm update, the choice of state action will increasingly depend on the accumulated experience rather than random to explore.

5.
On reward R(s, a), we consider the whole benefits of the configured femtocell k and the entire network. After taking action a(p_{ a }, c_{ a }) in state s(P, C, T), femtocell k obtained throughput t_{ k }, and the throughputs of neighbor femtocell nb_{1} and nb_{2} change into ${{t}_{1}}^{\prime}$ and ${{t}_{2}}^{\prime}$ because of the interference of femtocell k. We define the reward R(s, a) as follows:
$$R\left(s,a\right)=\alpha \bullet {t}_{k}{\beta}_{1}\bullet \left({t}_{1}{t}_{1}^{\prime}\right){\beta}_{2}\bullet \left({t}_{2}{t}_{2}^{\prime}\right)$$(9)
where α, β_{1}, β_{2}∈(0, 1) are compute weights, α > β_{1} or β_{2}. Reward function means to improve the new configured femtocell k's performance as far as possible, under the premise of ensuring the gain in the overall network performance. With this reward function, the system's longterm cumulative return is the sum of network performance gains after all the resource allocations for each femtocell.

6.
Q value update: femtocell k configures its channel and power as a(p_{ a }, c_{ a }) in state s(P, C, T), and gets its current reward R(s, a). At the same time, neighbor femtocells nb_{1} and nb_{2} adjust their power to ${p}_{1}^{\prime}$ and ${p}_{2}^{\prime}$ according to the impact of femtocell k, as well as their throughput t_{1} and t_{2} change into ${{t}_{1}}^{\prime}$ and ${{t}_{2}}^{\prime}$. Therefore, the state s(P, C, T) transition to s'(P, C, T), and according to R(s, a) and $\underset{{a}^{\prime}\in A}{max}{Q}_{t}\left({s}^{\prime},{a}^{\prime}\right)$, the Q(s, a) is updated according to (5) and (6).
The Qlearningbased channel and power configuration process is shown in Figure 5, and the details are described as follows:
Initialization: Q value table is cleared. To ensure that all of the stateaction pairs (s, a) can be fully tried, each item of the Q value table is associated with a learning rate α(s, a) and initialized to 0. While each state s(P, C, T) is associated with a temperature T_{s} and initialized to T_{0}, initialize all the visiting number n(s, a) = 0 of each (s, a). Set the time discount factor in (6) to γ.
Selfconfiguring trigger: There are three cases that will trigger the femtocell k's selfconfiguring.

1.
FBS k switches up;

2.
I_{ k }> I_{0}, I_{0} denotes the set interference threshold, and I_{ k } is femtocell k caused interference to its neighbors, which is calculated as follows:
$${I}_{k}=\sum _{i\in NB\left(k\right)}{\beta}_{i,k}{P}_{i,k}^{\mathsf{\text{pilot}}},$$
where ${\beta}_{i,k}=\left\{\begin{array}{l}1,\mathsf{\text{if}}k\mathsf{\text{and}}i\mathsf{\text{areinthesamechannel}}\\ 0,\mathsf{\text{else}}\end{array}\right.$, NB(k) is the neighbor femtocell list of femtocell k, ${P}_{i,k}^{\mathsf{\text{pilot}}}$ is femtocell i's received power from femtocell k.

3.
Average SINR of femtocell k's UEs is below the threshold SINR_{0}:
$$\mathsf{\text{SIN}}{\mathsf{\text{R}}}_{k,\mathsf{\text{ave}}}=\sum _{j\in UE\left(k\right)}\mathsf{\text{SIN}}{\mathsf{\text{R}}}_{k,j}/N=\sum _{j\in UE\left(k\right)}\left({P}_{j,k}^{r}/\left(\sum _{l\ne k}{P}_{j,l}^{r}+{n}_{0}\right)\right)/N<\mathsf{\text{SIN}}{\mathsf{\text{R}}}_{0},$$
${P}_{j,l}^{r}$ is UE j's received power from femtocell l, N is the number of femtocell k.
Above parameters, such as received power, interference, etc., are reported to FGW by FBSs and UEs.
Determining the state s: according to the parameters reported by femtocell k and its neighbor nb_{1} and nb_{2}, FGW decides which states s(P, C, T) femtocell k is in, and finds all the Q(s, a) corresponding to s(P, C, T) in the Q value table.
Action selection: FGW chooses an action a(p_{ a }, c_{ a }) according to the probability calculated by (8), and configures femtocell k's channel and power, respectively, as c_{ a } and p_{ a }. While recording the visited time n(s, a) of (s, a).
Getting reward: After the implementation of action a(p_{ a }, c_{ a }), according to the throughputs reported by the femtocells, FGW calculates R(s, a) of this iteration using (9).
Q value update: According to the changed throughputs and powers reported by nb_{1} and nb_{2}, FGW decides the new state s'(P', C', T') which s(P, C, T) transferred to after implementation of action a(p_{ a }, c_{ a }), and updates Q(s, a) according to R(s, a) and $\underset{{a}^{\prime}\in A}{max}{Q}_{*}\left({s}^{\prime},{a}^{\prime}\right)$ using (4).
Parameters update: To ensure the convergence of strategies selection as well as Qvalue update, we make the learning rate α(s, a) negative exponential declining with increasing of visited number n(s, a), and temperature T_{s} negative exponential declining with increasing of visited number n(s), where ${n}_{s}=\sum _{a\in A}{n}_{s,a}$.
3.3. Fast resource management: coordinated multipoint transmission
The interference between multiple femtocells can be reduced through channel and power allocation. However, due to short distance and few obstacles among femtocells, the interference may be too heavy to be well eliminated by radio resource allocation. Therefore, we considered to jointly adjust femtocells' antenna for realtime to implement fast radio resource management. In addition, large density of femtocells deployment can also bring a gain of improved joint fast resource management. Here, we only consider the downlink transmission.
In the public femtocell network, due to the lack of precise planning, there will be inevitably some coverage holes and multifemtocell overlap area, resulting in some UEs' low received signal power or heavy interference. We plan to solve this problem through coordinated multipoint transmission technique of multiple femtocells, which in general is to make the highgain main lobes of the femtocells toward the coverage holes and lowgain side lobes toward the overlap areas. Through collaboration of femtocells the coverage holes and interference between each other can be reduced and thus improving the network performance.
3.3.1. Problem formulation
In the femtocell network shown in Figure 2, suppose UE i is served by femtocell k, then UE i's received signal SINR can be calculated as
where ${h}_{k,i}=1{0}^{\left(\frac{37+30{log}_{10}{d}_{ki}+18.3{{n}_{ki}}^{\left\{\frac{{n}_{ki}+2}{{n}_{ki}+1}0.46\right\}}}{10}\right)}$
${P}_{k,i}^{r}$ is the received power from FBS k at UE i, ${P}_{k}^{t}$ is the transmit power of FBS k, ${g}_{k,i}^{t}$ is the antenna transmit gain from FBS k to UE i, d_{ ki } and h_{ k, i }, respectively, denote the distance and channel gain between FBS k and UE i, ${\beta}_{l,i}^{n}$ is a binary indicator, if ${\beta}_{l,i}^{n}=1$, femtocell l does allocate some power in channel n that UE i is using, zero otherwise. ${\beta}_{l,i}^{n}=1$ if FBS k serves UE i. n_{ ki } denotes the number of walls in the path, n_{0} denotes the effect of an interference from a MBS and additive white Gaussian noise. The total capacity of all femtocell users can be expressed as
Using the above formulas as a basis, we formulated an optimization problem as
in order to maximize the network's capacity.
Previous studies about femtocell network optimizing have mainly focused on radio resource allocation schemes such as spectrum and power allocation to reduce interference and improve network capacity, i.e., selecting the optimal ${\beta}_{l,i}^{n}$ and ${P}_{l}^{t}$ (l = 1  M, i = 1  N). Here, we considered dynamic and realtime adjustment of femtocells' antenna gains ${g}_{l,i}^{t}$ (l = 1  M) to give a new way of optimizing femtocell networks.
Because the movement of UE in the indoor environment is slow, during the optimization process, which is supposed to be several seconds, the h_{ k, i }, ${P}_{k}^{t}$ are unchanged, and the objective function can be further noted as
where ${\epsilon}_{l,i}={\beta}_{l,i}^{n}\bullet {P}_{l}^{t}\bullet {h}_{l,i}$ is a constant. Assuming matrix E, E', and G, respectively, are
${\mathbf{E}}_{N,M}=\left(\begin{array}{ccccc}\hfill {e}_{1,1}\hfill & \hfill ..\hfill & \hfill ..\hfill & \hfill ..\hfill & \hfill {e}_{1,M}\hfill \\ \hfill .\hfill & \hfill .\hfill & \hfill .\hfill \\ \hfill .\hfill & \hfill .\hfill & \hfill .\hfill \\ \hfill .\hfill & \hfill .\hfill & \hfill .\hfill \\ \hfill {e}_{N,1}\hfill & \hfill ..\hfill & \hfill ..\hfill & \hfill ..\hfill & \hfill {e}_{N,M}\hfill \end{array}\right),$ where e_{ n, m } = ε_{ m, n }
the m th row of G_{ M, N }, ${\stackrel{\u0303}{g}}_{m}={g}_{m,y}$ (y = 1N) represented the m th FBS's transmit gains to each UE. Formula (13) can be noted as
where $F\left({\mathbf{A}}_{N\times N}\right)=\prod _{i=1}^{N}{a}_{i,i}$ and I_{ N } is N order unit matrix.
Because the E and E' are assumed to be constant matrixes, it is clear that in order to achieve the optimal objective f_{obj}, we are supposed to find an optimal G_{ M, N } for FBSs' antenna transmit gains in different directions, which can be further noted as
3.3.2. Coordinated antenna patterns selection
Due to the cost and size restrictions of the FBS, the recently proposed Eplane Horns Based Reconfigurable Antenna [17, 18] was used for femtocell, which is of low complexity and can form four optional patterns, one pattern can switch to arbitrary another by simple circuit switching as shown in Figure 6. Under different patterns, an FBS k has different beamforming gains in each direction, i.e., different ${\stackrel{\u0303}{g}}_{m}$ of G_{ M, N }. Based on that, we proposed a coordinated multipoint transmission scheme which is to select the optimal antenna patterns combination of all FBSs, in order to obtain the approximate optimal solution of (16) with low additional complexity, which can be called as coordinated antenna patterns selecting (COPS) and noted as
where AP_{ k } denotes FBS k's antenna pattern, according to the numerical results, the COPS scheme could well improve the network capacity with a low additional complexity.
Taking into account the tradeoff between performance and complexity, we search the optimal antenna patterns combination of all FBSs using the simulated annealing algorithm (SA) rather than other heuristicbased algorithms:
Cbn_{ t } = (AP_{1,t},...AP_{ k, t },...AP_{ N, t }) denotes one coordinated antenna patterns combination of all FBSs. Cbn* is assumed as the optimal FBSs' antenna patterns combination, ${f}_{Cb{n}_{t}}=\sum _{i=1}^{N}{log}_{2}\left(1+{S}_{k,i}\right)$ is the evaluation function of the simulated annealing algorithm, and it can be calculated according to S_{ k, i } reported by UEs after each antenna patterns selection for FBSs. T is the temperature parameter and is initiated as T_{0} = 3/(10 ln 0.5). According to our simulation when the iteration number is more than 1000, the algorithm's performance will not be improved obviously. Therefore, max_num = 1000 is the allowed maximized iteration number. The COPS scheme is shown in Table 2.
4. Simulation results
We evaluated the performance of the three proposed scheme in the twotier (macrocell and femtocell) network shown in Figure 1 and an Mcell topology of an enterprise femtocell networks, as shown in Figure 2. Each femtocell has an average of N randomly distributed UEs. In the simulation, continuous heterogeneous services with different weights are generated, including the full buffer, VoIP, video, HTTP, and FTP services. The system parameters are described in Table 3.
4.1. Longterm resource management
For the simulation of spectrum allocation between macrocell and public femtocell networks, all the users are uniformly distributed on a cell site, using the same service generation function, and the cellcenter band, which is 2/3 of the total available spectrum, serves approximately 2/3 of the total services. Figure 7 depicts throughput performance of proposed ASFR scheme compared to a cochannel macro/femtocell setting with proportional fair scheduling. With equal priority applied to all types of service, simulation results show that, compared to the reuse one cochannel macro/femtocells, the proposed mechanism provides comparable and more stable throughput performance on both macrocells and femtocells.
4.2. Mediumterm resource management
Based on the simulation results of the longterm resource management, we further evaluate the performance of the mediumterm resource management. We simulated channel and power allocation of femtocell k in a public femtocell network as shown in Figure 2, and set the user model by applying the statistical data of the UEs' distribution during the time from 8 a.m. to 8 p.m. that obtained from our institute. The indoor users moved at 1 m/s speed in accordance with preset probability and routes. During the simulation, we created different scenarios in which the channels and powers of femtocell k's neighbor femtocells' are set to different values. The femtocell k's configuration of channel and power is triggered by each of the three cases mentioned in part 3. The performance parameters of all the femtocells including femtocell k and its neighbors are recorded. Finally, we will evaluate the performance of proposed configuration scheme by the average values of the performance parameters.
In order to better measure the performance of the proposed configuration approach, we also simulate other two configuring algorithms of channel and power: a typical distributed configuring algorithm: enhanced modified iterative waterfilling [19] and a typical central configuring algorithm: joint channel allocation and power control [20]. Figures 8 and 9 depict coverage percentage and capacity of three algorithms, which are the average values of different channel allocation situations for neighbor femtocells of femtocell k. It can be seen from the pictures that the central configuring can achieve an excellent performance but with high complexity (which is O((M^{2}N+MN^{2})log_{2}N)), the distributed configuring achieved a relatively poor performance because the femtocell which implements selfconfiguration does not consider the impact on neighbors. The performance of the proposed approach is very close to the central configuring but with a much lower complexity (which is O(5 × K), K is the number of actions and in our algorithm K = 12).
4.3. Fast resource management
In addition to the simulation results of the mediumterm resource management, we further evaluate the performance of fast resource management. We implement coordinated antenna patterns selection after femtocell k's channel and power configuration in use of three different algorithms. Figures 10 and 11 show that when different mediumterm resource allocation schemes reached their optimization limits, the COPS scheme can further improve the network's coverage and capacity effectively with low complexity (which is O(MNlog_{2}N)).
5. Conclusion
This article focused on the resource management problem in public femtocell networks. With the growing of indoor communication requirement, femtocell is proposed as a solution of improving indoor coverage and network capacity. The public femtocell network is considered by operators as a promising way of meeting the increasing communication requirement of public places such as enterprise, campus, and airport. However, the deployment of public femtocell is very different from traditional macrocell or residential femtocell, as multifemtocells need to cooperate to provide coverage. Therefore, it's necessary to address the resource allocation between multiple femtocells, in order to eliminate the interference between femtocells and maximize system capacity.
In this article, we proposed a systematic approach to manage resources of femtocell network. For the longterm resource management, we adopt frequency soft reuse to solve spectrum allocation problem between macrocell and femtocell networks. In addition, we enable femtocells dynamically access spectrum allocated to macrocell through CR, which further improves femtocell network capacity and resource utility. For the mediumterm resource management, we proposed the Qlearningbased configuring approach to allocate channel and power to femtocells. At last, we proposed a coordinated antenna patterns selection scheme to implement fast resource management for femtocell networks, and make multiple femtocells to cooperate to improve the resource utility and system performance of femtocell networks.
Abbreviations
 ASFR:

adapted soft frequency reuse
 COPS:

coordinated antenna patterns selecting
 CR:

cognitive radio
 FBS:

femtocell base station
 ITI:

intertier interference
 MBS:

macro base station
 PS:

primarysecondary
 SA:

simulated annealing
 SFR:

soft frequency reuse
 SOHO:

small office/home office
 SS:

secondarysecondary
 UE:

user equipment.
References
 1.
Chandrasekhar V, Andrews JG: A Gatherer, Femtocell networks: a survey. IEEE Commun Mag 2008, 46(9):5967.
 2.
Presentation by ABI Research, Picochip, Airvana, IP access, Gartner, Telefonica Espana 2nd International Conference Home Access Points and Femtocells [http://www.avrenevents.com/dallasfemto2007/]
 3.
Saunders SR, Carlaw S, Giustina A, Bhat RR, Rao VS, Siegberg R: Femtocells: Opportunities and Challenges for Business and Technology. Wiley; 2009. ISBN: 9780470748169
 4.
Ashraf I, Claussen H, Ho LTW: Distributed radio coverage optimization in enterprise femtocell networks. IEEE ICC 2010 proceedings
 5.
Choi S, Lee T, Chung M, Choo H: Adaptive coverage adjustment for femtocell management in a residential scenario. Management Enabling the Future Internet for Changing Business and New Computing Services 2009, 221230.
 6.
Claussen H, Ho L, Samuel L: Selfoptimization of coverage for femtocell deployments. Proceedings of Wireless Telecommunication Symposium 278285.
 7.
Claussen H, Pivit F: Femtocell coverage optimization using switched multielement antennas. IEEE International Conference on Communication 2009, 16.
 8.
Duran A, Carrasco G: UMTS femtocell performance in massive deployments: capacity and GoS implications. Bell Labs Tech J 2009, 14(2):185202. 10.1002/bltj.20379
 9.
Chandrasekhar V, Andrews JG: Uplink capacity and interference avoidance for twotier cellular networks. In Proceedings of IEEE Global Telecommunication Conference (GLOBECOM). Washington, DC; 2007:33223326.
 10.
Zheng X, Walke B: Frequency reuse techniques for attaining both coverage and high spectral efficiency in ofdma cellular systems. WCNC 2010.
 11.
Sutton R, Barto A: Reinforcement Learning: An Introduction. MIT Press; 1998.
 12.
Samaan N, Karmouch A: An automated policybased management framework for differentiated communication systems. IEEE J Sel Areas Commun 2005, 23(12):22362247.
 13.
Watkins CJCH: Learning from delayed rewards. Ph.D. Dissertation, Cambridge University, Cambridge, UK; 1989.
 14.
Kaelbling LP, Littman ML, Wang X, et al.: Reinforcement learning: a survey. J Artif Intell Res 1996, 4: 237285.
 15.
Watkins CJCH, Dayan P: Qlearning. Mach Learn 1992, 8: 279292.
 16.
Nie J, Haykin S: A Qlearningbased dynamic channel assignment technique for mobile communication systems. IEEE Trans Veh Technol 1999, 48(5):16761687. 10.1109/25.790549
 17.
Tsai A, Huang J, Wang L, Hwang R: High capacity femtocells with directional antennas. IEEE WCNC 2010 proceedings
 18.
Claussen H, Pivit F: Femtocell coverage optimization using switched multielement antennas. ICC 2009.
 19.
Lee K, Jo O, Cho D: Cooperative resource allocation for guaranteeing intercell fairness in femtocell networks. IEEE Commun Lett 2011, 15(2):222235.
 20.
Guruacharya S, Niyato D, Hossain E, Kim D: Hierarchical competition in femtocellbased cellular networks. IEEE Globecom 2010 proceedings
Acknowledgements
This study was sponsored by the Key Project of National Natural Science Foundation of China (60632030, 60832009), the National Basic Research Program of China (973 Program) (2009CB320406), the National Key Technology R&D Program of China (2010ZX0300300101) and the Doctoral Fund of Ministry of Education of China (20070013017) and the contributions of the colleagues from Wireless Technology Innovation Institute of Beijing University of Posts and Telecommunications are hereby acknowledged.
Author information
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Received
Accepted
Published
DOI
Keywords
 radio resource management
 public femtocell networks
 resource utility
 system performance