The power density received by an antenna varies with the incidence angle. While receiving the signal, the antenna responds to an incoming wave from a given direction according to the pattern value in that direction. Each of *M* number of antenna elements holds different gain patterns in each direction. Therefore, the received power at each element from a particular angle varies due to the prospective antenna gain. Using the variation of received power for the *M* elements, the femtocell is trained to distinguish the outdoor and indoor users. The conversion of the power from user equipment to the antenna end also depends on the path loss, user equipment class, transmitting antenna gain, multi-path fading, shadowing, noise signals, and receiver port and feed cable losses. However, other terms vary very little within the elements as in practical; femtocells are small in size, with antenna placed very close to each other. The user antennas are considered isotropic that can transmit in all directions evenly. Even if the antenna has directional radiation pattern, it will obtain the *M* antenna elements location within a small angle from the user end.

Let us consider that the number of source is *K*. For *K*th source, the incident beam plane wave centered at frequency *w* impinging on the *M* elements of antenna in different angle *θ*
_{
K
}. The maximum received power on the effective area of the antenna relates proportionally to the transmitting power of the user equipment. Using complex power presentation, the received power at *i*th antenna element can be written as:

$$ {P}_i(t)={\displaystyle \sum_{m=1}^K}{Q}_m(t){e}^{-j{k}_{mi}}+{n}_i(t);\;i=1,2,\dots M $$

(1)

where,

$$ {k}_{mi}={k}_0\sqrt{x_i^2+{y}_i^2}\mathrm{C}\mathrm{o}\mathrm{s}\left({ \tan}^{-1}\frac{y_i}{x_i}\right) $$

(2)

The center point of the femtocell is (0, 0) and coordinates of each antenna element center point is (*x*
_{
i
},*y*
_{
i
}). Using vector notation and Friss transmission law, Equation 1 can be expressed as:

$$ P(t)=AQ(t)+N(t) $$

(3)

$$ P(t)={\left[{P}_1(t),{P}_2(t)\dots {P}_M(t)\right]}^T $$

(4)

$$ Q(t)={\left[{Q}_1(t),{Q}_2(t)\dots {Q}_K(t)\right]}^T $$

(5)

$$ N(t)={\left[{n}_1(t),{n}_2(t)\dots {n}_M(t)\right]}^T $$

(6)

$$ A=\left(\frac{\lambda_w{G}_R\left({\theta}_1\right)}{4\pi {R}_1},\frac{\lambda_w{G}_R\left({\theta}_2\right)}{4\pi {R}_2},\dots \frac{\lambda_w{G}_R\left({\theta}_K\right)}{4\pi {R}_K}\right) $$

(7)

The noise signal is independent of *Q*(*t*) for all antenna elements. The spiral co-relation matrix of the received power is as follows:

$$ \begin{array}{c}R=E\left\{P(t)P{(t)}^H\right\}\hfill \\ {}\hfill =AE\left[Q(t){Q}^H(t)\right]{A}^H+E\left[N(t){N}^H(t)\right]\hfill \end{array} $$

(8)

Here, *H* is the conjugate transpose. The femtocell is trained to perform mapping of *G*:*R*
^{K} → *C*
^{M} from the space of *Q*(*t*) to the *P*(*t*). The neural network is used to do the inverse mapping *F*:*C*
^{M} → *R*
^{K}.

In sample data, processing algorithm for signals applies the correlation matrix instead of actual output value. Here, the *Z* is formed from the first row of the correlation matrix *R* shown in N-MUST algorithm [14].

$$ R=\left[\begin{array}{c}\hfill {R}_{1,1}\dots \dots \dots {R}_{1,M}\hfill \\ {}\hfill \dots \dots \dots \dots \dots \dots \hfill \\ {}\hfill {R}_{M,1}\dots \dots \dots {R}_{M,M}\hfill \end{array}\right] $$

(9)

$$ Z=\frac{b}{\left|b\right|} $$

(10)

The size of *b* is *M* × 1, where *b* got both real and imaginary values. Since the neural network does not deal with the imaginary value, the size of *Z* is 2 *M* × 1. However, in [14], the authors calculated the angle of incidence only. The angle of incidence and the received power can be utilized (in theory) to train a neural network to identify the relative location of a user and grant access accordingly. This process appears rather redundant as the whole process can be carried out by a single neural network. Thus, in this paper, a neural network is trained using the vectors of comprising of *E* from all antennas over multiple instances of *n*.

$$ {E}_{i,n}={\displaystyle \underset{n\times T}{\overset{T\left(n+1\right)}{\int }}}{P}_i(t)dt $$

(11)

$$ \overline{E}=\left[\begin{array}{c}\hfill {E}_{1,n}\hfill \\ {}\hfill \cdot \hfill \\ {}\hfill \cdot \hfill \\ {}\hfill \begin{array}{c}\hfill \cdot \hfill \\ {}\hfill {E}_{i,n}\hfill \end{array}\hfill \end{array}\right] $$

(12)

where, *n* = 0,1,2,3…. and *T* is the sampling period.

The architecture of this paper involves two stages. The initial stage is the training process, that is to learn the traits of indoor and outdoor user using the value of *Ē*. In the next stage, the network will detect the user based on the previous learning. The network will be trained initially with some sample of indoor and outdoor user’s *Ē*. After the training, random users are generated. They are considered as isotropic sources. The values of *Ē* are calculated using Hata path loss model and considering unit shadow-fading and additive white Gaussian noise. When a source radiates towards the femtocell antenna elements, the varying gain of each antenna creates discrepancies in the received power. The neural network exploits these discrepancies to predict the category of the source. The neural network is trained to grant access if the location of the user is inside a certain region [17]. Based on the training experiences, the femtocell distinguishes the indoor and outdoor users. For random values of *Ē*, the neural network will determine the users’ category by giving an output of ‘1’ or ‘−1’ (sigmoid output neurons). A filtering stage is shown in MN-MUST algorithm that reduces the size of the correlation matrix for cylindrical array configuration for direction of arrival estimation [7]. In this case, the received power is a function of the antenna gain in the impinging direction. For a single user, the more power pattern in each antenna element is considered, the more perfection is achieved in the detection process with less number of training samples.