In-phase and quadrature filter shape index modulation

Abstract

This paper proposes a novel modulation scheme called “In Phase and Quadrature-Filter Shape Index Modulation” (IQ-FSIM). It aims to enhance the spectral/energy efficiencies (SE/EE) while generalizing several existing modulations and index modulation (IM) domains. In this system, the bitstream is divided into three sub-streams. The first is mapped using amplitude–phase modulation, and the other two are mapped separately to an index of a filter shape at the in-phase and quadrature components. IQ-FSIM in SISO mode enhances the SE by $2\log _2 N$ (double FSIM gain), thanks to the independent indexation of N different filter shapes on each component changing at the symbol rate. A low-complexity matched filter-based detector that reaches the optimal joint ML performance is proposed. The theoretical lower bounds for the probability of filter index error and the symbol/bit error rate (SER/BER) are derived and validated. The computational complexity of the proposed IQ-FSIM transceiver is estimated and compared to its predecessor FSIM, where it is shown that IQ-FSIM provides up to 93.7% complexity reduction compared to FSIM along with different advantages. In addition, the results reveal that both IQ-FSIM and its special case FSIM, even with non-optimal filter shapes, outperform the equivalent schemes with/without IM of the same SE in AWGN, flat, and frequency-selective fading channels.

1 Introduction

Index modulation (IM) can offer a significant improvement in spectral efficiency (SE) and/or energy efficiency (EE). Hence, it is a promising technology for low-power and ultra-high rate beyond 5G systems. The IM concept is widely explored in the time, frequency, and spatial domain [1]. The latter is based on systems with multiple antennas, where the index of the activated (targeted) antenna or set of antennas at the transmitter (receiver) conveys additional data bits implicitly (Virtual Bits (VBs)) [2,3,4,5]. In contrast, the frequency and time IM domains are investigated mainly in the single-input single-output (SISO) systems to convey the VBs by the index of activated frequency bands/sub-carriers [6,7,8,9], or time slots [10,11,12], respectively.

The frequency and time IM domains are very convenient for low-power and low data rates applications since they offer EE gain but at the expense of SE loss or limited enhancement due to sacrificing some available frequency and time resources. However, the modulation type IM domain is introduced later to overcome the SE limitation by using all available resources. In this IM domain, the primary and secondary activated slots/bands or sub-carriers are mapped using distinguishable amplitude–phase modulations (APMs) (no common constellation between both APMs), like in dual-mode single-carrier IM (DM-SC-IM) [13] and DM-orthogonal frequency division multiplexing-IM (DM-OFDM-IM) [14, 15]. Moreover, another strategy based on non-orthogonal frequency bands or pulses is proposed in [6] and [16], to increase the SE but at the cost of inherited interference that results in performance degradation and thus lower EE. Furthermore, IM non-orthogonal multiple access (IMMA) technique is proposed in the time IM domain [17], where different users can have concurrent transmissions at the same time slot, which leads to induced collision. Besides, these SISO-IM domains are extended to multiple-input multiple-output (MIMO) [18] or combined with spatial IM domain to overcome the limited SE enhancement in the frequency and time IM domains, or enhance the system performance in dispersive channels [19].

Recently, the filter IM domain has been proposed in [20], where any filter property at the transmitter side can be indexed whenever the receiver can detect this property. The filter IM domain generalizes most of the existing SISO-IM and several existing modulations schemes, as explained in Section II in [20]. Besides, the “Filter Shape Index Modulation” (FSIM) scheme is proposed where a filter shape is selected from a filter bank on each symbol period at the transmitter side to convey VBs through IM. This system allows reaching a higher SE and EE gain in SISO in contrast to most SISO-IM schemes.

In the context of the French national research project BRAVE [21], we target to achieve ultra-high wireless data rates in the terahertz (THz) and sub-THz bands, needed for different scenarios defined in [22]. The results reveal that increasing the SE of energy-efficient low-order modulation by spectral-efficient IM and MIMO techniques is advantageous for low-power wireless ultra-high data rate systems, especially with the current technological limitations in sub-THz bands. This conclusion motivated us to design the FSIM scheme [20] for better SE and EE enhancement in SISO while using low-order modulation before its exploitation in MIMO mode [23]. It is worth mentioning that MIMO-FSIM system is among the most spectral-efficient schemes, and it is a promising candidate for wireless terabits systems, as shown in the different perspectives comparison [23]. In this paper, in-phase and quadrature FSIM (IQ-FSIM) is proposed for further SE/EE enhancements as compared to the equivalent APM (e.g., QAM) and its predecessor FSIM that has proved its superiority over the existing state of art schemes with/without IM.

In the following, the major contributions of this paper are summarized:

• 1 A novel modulation scheme named “IQ-FSIM” is proposed in the filter IM domain. In contrast to our previously proposed FSIM scheme [20], the pulse shaping filters for the in-phase (I) and quadrature (Q) components are indexed separately to double the SE gain of FSIM. This new IQ-FSIM scheme achieves the highest SE gain among existing SISO-IM schemes due to considering different filter shapes at each symbol period, separating I and Q indexation, transmitting M-ary APM symbol (e.g., QAM, PSK, etc.), and most importantly exploiting all available time and frequency resources efficiently.

• 2 The proposed IQ-FSIM modulation is shown to be a more generalized scheme than its predecessor FSIM which is in turn generalizing most SISO schemes with/without IM. This paper highlights this fact and clarifies that the reconfiguration of IQ-FSIM system provides different SISO systems (e.g., QAM, PSK, on–off keying, pulse position modulation, FTN, etc.) and other IM techniques (e.g., time, frequency IM domains, etc.).

• 3 Two different detectors for IQ-FSIM are proposed. Firstly, the joint maximum likelihood (ML) detector, that jointly estimates the APM symbols and both filter shapes on I and Q, is described. Then, a matched filter (MF)-based detector is proposed to achieve optimal performance while providing lower complexity. The latter estimates firstly the filter shapes on I and Q in parallel before proceeding to APM symbol detection. Moreover, a generalized inter-symbol interference (ISI) estimation and cancellation block for FSIM and IQ-FSIM is presented.

• 4 The theoretical lower bounds for the filter index error probability, total symbol error rate (SER), as well as real/virtual and total bit error rate (BER) are derived to evaluate the proposed IQ-FSIM scheme. Then, the derived analytical expressions are validated by Monte Carlo simulations.

• 5 In the SISO context, different simulations are provided to show the significant EE and SE gains achieved by the proposed IQ-FSIM compared to several equivalent SISO schemes. More precisely, the IQ-FSIM, the previously proposed FSIM, and their equivalent QAM system are compared at the same SE using theoretical and numerical results in additive white Gaussian noise (AWGN) with the currently designed filters. Then, the comparison is extended to flat and frequency-selective multipath fading channels to provide a broader conclusion.

• 6 The transceiver computational complexity of FSIM and IQ-FSIM systems are estimated and compared. The transmitter complexity is shown to be the same for both, but IQ-FSIM receiver has a significant complexity reduction compared to FSIM at the same SE.

The paper is organized as follows. The proposed IQ-FSIM transceiver along with the two optimal detectors and the ISI estimation and cancellation (ISI-EC) technique are presented in Sect. 2. Section 3 shows the derivation of the theoretical lower bounds for IQ-FSIM in terms of probability of error of filter index detection, total SER, as well as real/virtual and total BER. Afterward, these theoretical results are validated in Sect. 4, and different simulations are performed to highlight the filter IM domain advantages. Finally, the conclusion is provided in Sect. 5.

Notations Bold lower case is used for vectors. $\Re \{x\}$ and $\Im \{x\}$ are used interchangeably with $x^I$ and $x^Q$ to denote the real and imaginary parts of the complex number x, respectively. The character $*$ stands for the convolution. $\langle {{\textbf {x}}},{{\textbf {y}}}\rangle$ denotes the vectors’ dot-product. $\Arrowvert .\Arrowvert$ denotes the Frobenius norm. ${{\mathcal {C}}}{{\mathcal {N}}}(\mu ,\,\sigma ^2)$ represents the distribution of a complex Gaussian random variable having $\mu$ mean and $\sigma ^2$ variance. $\lfloor x \rfloor$ ($\lceil x \rceil$) stands for the floor (ceil) function that corresponds to the largest (smallest) integer less (greater) than or equal to x. P(.), p(.), and $\mathop {{\mathbb {E}}}[.]$ denote, respectively, the probability of an event, the probability density function (PDF), and the expectation.

2 In-phase and quadrature filter shape index modulation

2.1 IQ-FSIM transmitter

Fig. 1

Block diagram of the proposed IQ-FSIM transceiver

The proposed IQ-FSIM transceiver in SISO system is depicted in Fig. 1, where the input bitstream b is divided into three streams $b_1$, $b^I_2$, and $b^Q_2$. The sub-bitstream $b_1$ is mapped by an M-ary APM, while the sub-bitstreams $b_2^I$ and $b_2^Q$ are mapped to the indices i and j of the selected filters at the I and Q components, respectively. Note that i and j are not necessarily the same in contrast to FSIM [20] that uses the same filter on I and Q. These filters ${{\textbf {f}}}_{i}[m]$ and ${{{\textbf {f}}}}_{j}[m]$ of length L are used as pulse shaping for the I and Q components, respectively, where the sample index m is an integer between 0 and $L-1$, and the filter spans $\eta$ APM symbols of $\uplambda$ samples each, i.e., $L=\eta .\uplambda +1$ samples.

The filter bank at each branch contains N different filter shapes as depicted in Fig. 1. Thus, the number of bits per IQ-FSIM symbol ${\mathcal {L}}_{\text {IQ-FSIM}}$ is given by:

$$\begin{aligned} {\mathcal {L}}_{\text {IQ-FSIM}}= 2\log _2 N + \log _2 M , \end{aligned}$$

(1)

where N is a power of 2. Thus, the SE enhancement is $2\log _2(N)$ compared to conventional M-ary linear APM systems (e.g., QAM, PSK, etc.), and it is the double of the previous SE gain achieved by FSIM ($\log _2(N)$), thanks to the separate and independent filter indexation on I and Q.

The filter bank’s outputs for the nth APM complex symbol $c_n$ on the I and Q branch are denoted, respectively, by the signals $s^I_n[m]$ and $s^Q_n[m]$ given by:

$$\begin{aligned} {s^I}_n[m]&=\left( {f_i}_n*\Re \{c'_n\}\right) [m]= c_n^I {f}_{i_n}[m], \end{aligned}$$

(2)

$$\begin{aligned} {s^Q}_n[m]&=\left( {f_j}_n*\Im \{c'_n\}\right) [m]= c_n^Q {f}_{j_n}[m], \end{aligned}$$

(3)

where ${c}'_n$ is the up-sampled version of the $n^{th}$ APM symbol $c_n=c_n^I+\textrm{j}\,c_n^Q$ by a factor L with $\textrm{j}=\sqrt{-1}$, $i_n \text { and } j_n \in \{0, 1, \ldots , N-1\}$ are the indices of the filter shapes being selected to filter $c_n$ components, at I and Q based on the sub-streams $b_2^I$ and $b_2^Q$, respectively. The convolution in (2–3) gives only L nonzero values from $2L-1$ and thus it is a merely simple multiplication of L real values by a complex APM symbol. Then, the complex signal $s_n[m]$ is deduced by combining both components $s_n[m]=s^I_n[m] + \textrm{j}\, s^Q_n[m]$.

Afterward, the overlap–add (OLA) block takes the signals ${s}_n[m]$ of length L each as depicted in Fig. 1 and then adds them after shifting by $\uplambda$ samples between consecutive signals. More details are provided in our previously proposed FSIM scheme [20]. The $\uplambda$ corresponding samples ${x}_n[p]$ for $c_n$ are given by:

$$\begin{aligned} {x}_n[p]=\sum _{n'=n-\lceil \eta /2 \rceil }^{n+\left\lfloor \eta /2\right\rfloor }{s}_{n'}[p-(n'-n)\uplambda ], \end{aligned}$$

(4)

where $p=p_{\text {center}} -\lceil \uplambda /2\rceil +1 ,\ldots , p_{\text {center}} +\lfloor \uplambda /2\rfloor$ and the index of the middle desired sample is $p_{\text {center}} = \frac{L-1}{2}$.

After overlapping and adding the signals of $N_s$ filtered data symbols, a frame ${{\textbf {x}}}_f$ is created by adding $N_{ZP}$ zero-prefix (ZP) symbols (other prefix can be used):

${{\textbf {x}}}_f=[{{\textbf {0}}}_{1},\ldots ,{{\textbf {0}}}_{N_{ZP}}, {{\textbf {x}}}_{-\lfloor \eta /2\rfloor },\ldots ,{{\textbf {x}}}_{\lceil N_s - 1 + \eta /2\rceil } ]$. This ZP is only needed to mitigate the interframe interference with frequency-selective multipath fading channel. In contrast to cyclic-prefix, the ZP does not consume additional power (silent transmission), but both leads to a SE reduction by a factor $N_s/(N_s+N_{ZP})$ to get : $\text {SE}_{IQ-FSIM}=\frac{N_s}{N_s+N_{ZP}}\mathcal {L_{IQ-FSIM}}$ bpcu.

Note that the previously proposed FSIM scheme [20] is a special case of IQ-FSIM (see Appendix A), and thus the traditional Nyquist and faster-than-Nyquist transceiver using any M-ary APM schemes remain a special case of the proposed IQ-FSIM system (proof similar to Appendix A in [20]). Hence, IQ-FSIM can be reconfigured to get FSIM by using the same indexed filter shapes on I and Q, while the traditional APM transceiver is obtained using the same and only Nyquist filter on I and Q for all symbols. Similarly, the FTN system can be achieved by decreasing the time shift in the OLA block between successive symbols according to the compression factor of Nyquist symbol period. In addition, it is clear that this reconfigurable IQ-FSIM maintains the ability of its special case FSIM to generalize many IM schemes as discussed in Section II in [20].

2.2 IQ-FSIM receiver

The received time-domain-based band signal y shown in Fig. 1 can be expressed as:

$$\begin{aligned} y(t)=(h*x_f)(t)+v(t), \end{aligned}$$

(5)

where v(t) is the AWGN following ${{\mathcal {C}}}{{\mathcal {N}}}(0,\,\sigma _v^2)$ distribution, and h(t) is the impulse response of a multipath frequency-selective fading channel with J paths and maximum delay spread of $(J-1)T_{sym}$, where $T_{sym}=\uplambda \,T_s$ is the APM symbol period and $T_s$ is the sampling period. In order to avoid the interframe interference, the guard time using ZP should be maintained larger than the channel maximum delay spread, i.e., $N_{ZP} \ge (J-1)$. The average power of transmitted symbols $c_n$ as well as all the filters f in the bank for I and Q components are normalized according to $\mathop {{\mathbb {E}}}[| c_n|^2]=1$ and $\sum _{m=0}^{L-1}{{f}^2[m]}=1,$ respectively.

Similar to FSIM, the Nyquist ISI-free criterion in AWGN channel is relaxed so that more distinguishable filter shapes can be designed and a higher SE can be reached. Hence, the proposed IQ-FSIM scheme can have some inherent predictable ISI due to the non-Nyquist filter shapes, in addition to the ISI resulting from the multipath channel effect.

The proposed IQ-FSIM receiver mitigates the fading channel effect on all received samples to generate the equalized signal $\hat{{{\textbf {x}}}}$ as shown in Fig. 1 and then proceeds to cancel the inherent ISI before the detection as it will be shown in Sect. 2.2.3. The former ISI can be mitigated by applying, just after the ZP removal from each frame, a linear frequency domain equalizer (FDE) like zero-forcing-FDE (ZF-FDE) or minimum mean square error-FDE (MMSE-FDE).

The following subsections describe the two proposed optimal detectors and the ISI-EC algorithm.

2.2.1 Joint ML detector

The ISI-EC block targets to eliminate the introduced ISI by the non-Nyquist filter shapes from the equalized signal $\hat{{{\textbf {x}}}}$ and generate the signal $\hat{{{\textbf {s}}}}$ of L samples for each symbol. Then, the detector takes these signals $\hat{{{\textbf {s}}}}$ to jointly estimate the filter indices and the APM symbol. The joint ML detector performs an exhaustive search over all possible combinations between the APM symbols, filters ${{\textbf {f}}}_i$ on I, and filters ${{\textbf {f}}}_j$ on Q, as highlighted in the following:

$$\begin{aligned} \displaystyle \{{\hat{\imath }},{\hat{\jmath }},{\hat{c}}\}=\mathop {\mathrm {arg\,min}}\limits _{ {{\textbf {f}}}_i\in \psi ^I,{{\textbf {f}}}_j\in \psi ^Q,c \in \chi }\Arrowvert \hat{{{\textbf {s}}}} - (c^I {{{\textbf {f}}}}_{i} +\textrm{j}\,c^Q {{{\textbf {f}}}}_{j}) \Arrowvert ^2 \end{aligned}$$

(6)

where $\psi ^I$/$\psi ^Q$ and $\chi$ denote the set of N filter’s shapes used for the I/Q components and the M-ary APM constellation, respectively. The detected ${\hat{\imath }}$/${\hat{\jmath }}$ and ${\hat{c}}$ are the recovered indices of ${{\textbf {f}}}_i$/${{\textbf {f}}}_j$ on I/Q, and the APM symbol, respectively. Note that the joint ML detector can replace the IQ-FSIM detector in Fig. 1.

2.2.2 Matched filter-based detector

Fig. 2

Matched filter-based detector for IQ-FSIM. This block estimates the index ${\hat{\imath }}$ (${\hat{\jmath }}$) of the filter shape being used at I (Q) component using the input ${\hat{s}}^I$ (${\hat{s}}^Q$)

After equalization and ISI-EC, the IQ-FSIM detector based on MFs performs the detection of filter indices used on I and Q components in parallel before proceeding to APM detection as depicted in Fig. 1. Note that the bank of correlators can replace the MFs and provides the same results [24]. Each filter shape detector contains N MFs $g_k$, where $g_k(t)=f_k(T_f-t)$ with $0\le t\le T_f$, $T_f=L.T_s$ as shown in Fig. 2. The filters’ outputs $r_k(t)$ on the I branch are given by:

$$\begin{aligned} r_k^I(t)&=\int _{0}^{t}{{\hat{s}}^I(\tau )g_k(t-\tau ) d\tau } , \quad k=1,2,\ldots ,N \nonumber \\&=\int _{0}^{t}{{\hat{s}}^I(\tau )f_k(T_f-t+\tau ) d\tau }. \end{aligned}$$

(7)

Then, these MFs’ outputs are sampled at the instant $T_f$:

$$\begin{aligned} r_k^I=\int _{0}^{T_f}{{\hat{s}}^I(\tau )f_k(\tau ) d\tau }, \quad k=1,2,\ldots ,N. \end{aligned}$$

(8)

Hence, the filter index used at the I branch is estimated by considering the maximum of $r_k^I$ squared:

$$\begin{aligned} {\hat{\imath }}= \mathop {\mathrm {arg\,max}}\limits _{ k} (r_k^I)^2. \end{aligned}$$

(9)

Similarly, the filter index on the Q branch is detected to deduce ${\hat{\jmath }}$ using the input signal ${\hat{s}}^Q$ and the corresponding MF bank. Once the filter indices $({\hat{\imath }},{\hat{\jmath }})$ are detected, $r=r^I_{{\hat{\imath }}}+\textrm{j}\,r^Q_{{\hat{\jmath }}}$ is used to perform the APM symbol detection like in the conventional systems without IM. Afterward, these detected pieces of information (${\hat{c}},{\hat{\imath }},{\hat{\jmath }}$) are demapped using the corresponding demapper, and the overall bitstream ${\hat{b}}$ is recovered, as shown in Fig. 1.

2.2.3 ISI estimation and cancellation (ISI-EC)

The designed filter bank in the IQ-FSIM scheme relaxes the Nyquist criterion for zero-ISI while keeping the introduced ISI as minimum as possible since the low filters’ cross-correlations cannot be guaranteed with only Nyquist filter shapes. This relaxation with FSIM and IQ-FSIM is added to improve the SE more than most SISO-IM schemes. Thus, these different filter shapes on I and Q will induce a predictable ISI that could be estimated and then removed or minimized before proceeding with the APM symbol detection.

In order to highlight the effect of the inherent ISI due to the non-Nyquist filter shapes, the baseband received signal without ZP can be expressed as follows in AWGN channel:

$$\begin{aligned} y(t)&=x(t)+v(t)=\sum _n{s_n(t-n\,T_{sym})} + v(t) \nonumber \\&=\sum _n{\Bigl ( c_n^I f_{i_n}(t-n\,T_{sym})+\textrm{j}\,c_n^Q f_{j_n}(t-n\,T_{sym})\Bigr )} +v(t). \end{aligned}$$

(10)

In the case of fading channel, the equalized signal ${\hat{x}}$ is passed to the ISI-EC block instead of y as depicted in Fig. 1. Note that the equalized signal ${\hat{x}}$ can be expressed similar to (10) when the channel effect is completely compensated (i.e., perfect equalization where only the noise component can be affected). Therefore, an AWGN channel can be considered in the sequel to explain this block without loss of generality.

After sampling y(t) in AWGN at the rate of $T_{s}$, the L samples corresponding to the $n^{th}$ APM symbol $c_n$ can be written as follows:

$$\begin{aligned} y_n[m]&= \Bigl ( c_n^I f_{i_n}[m]+\textrm{j}\,c_n^Q f_{j_n}[m]\Bigr ) \nonumber \\&+\underbrace{\sum _{n'>n}{(c^I_{n'}\,f_{i_{n'}} + \textrm{j}\,c^Q_{n'}\,f_{j_{n'}})[m- (n'-n)\uplambda ]}}_{anticausal\,ISI} \nonumber \\&+ \underbrace{\sum _{n'<n}{(c^I_{n'}\,f_{i_{n'}}+ \textrm{j}\,c^Q_{n'}\,f_{j_{n'}})[m -(n'-n)\uplambda ]} }_{causal\,ISI})+ v_n[m] \nonumber \\&= \Bigl ( c_n^I f_{i_n}[m]+\textrm{j}\,c_n^Q f_{j_n}[m]\Bigr )+ ISI_n[m] + v_n[m], \end{aligned}$$

(11)

where $f_{i_n}$ and $f_{j_n}$ represents the $i{th}$ and $j{th}$ shapes used to filter the real and imaginary components of the $n{th}$ APM symbol, respectively, and $v_n[m]$ is the m-th noise sample contaminating the nth APM symbol at $t=n\,T_{sym}+m\,T_s$. In contrast to a conventional transceiver that uses Nyquist filters, the received signal in the proposed scheme contains some ISI ($ISI\ne 0$ in (11)).

Since the filter detection needs all the L samples around each symbol, the ISI-EC requires eliminating the causal and anticausal ISI introduced on all related samples as with FSIM. Hence, the $\eta$ past and $\eta$ future APM symbols along with the filter indices on I and Q are required for ISI-EC. This block for IQ-FSIM is similar to that for the FSIM scheme that we proposed in [20]. However, the tentative detectors independently estimate the filter index on I/Q components, and the pulse shaping considers the separate filter shapes indexation for I/Q in the case of IQ-FSIM (the difference in ISI-EC block between FSIM and IQ-FSIM are highlighted in blue in Fig. 3).

Fig. 3

ISI estimation and cancellation (ISI-EC) for IQ-FSIM. The causal ISI is compensated in ISI-EC by using the feedback final decisions $\{{\hat{\imath }},{\hat{\jmath }},{\hat{c}}\}$ for past symbols, while the causal ISI is estimated by tentative detection of $\{{\tilde{\imath }},{\tilde{\jmath }},{\tilde{c}}\}$ for future symbols)

However, even if the input signal is perfectly equalized (no channel distortion), a residual ISI can remain in ${\hat{s}}_n[m]$ when an error occurs in past and/or future decisions, and thus ${\hat{s}}_n[m]$ is given by:

$$\begin{aligned} {\hat{s}}_n[m]= \Bigl ( c_n^I f_{i_n}[m]+\textrm{j}\,c_n^Q f_{j_n}[m]\Bigr )+ ISI_{resid}[m] +v_n[m], \end{aligned}$$

(12)

where $m=0,\ldots L-1$, and $ISI_{resid}[m]$ corresponds to the residual ISI remained due to error in feedback and/or tentative decisions. Note that the $ISI_{resid}[m]$ approaches zero if the filter bank is well designed. Then, the IQ-FSIM detector takes ${\hat{s}}_n[m]$ to estimate the final decisions $\{{\hat{\imath }}_{n},{\hat{\jmath }}_{n},{\hat{c}}_{n}\}$ as shown in Fig. 1.

Recalling, the filter bank design requirements and constraints for the IQ-FSIM system are similar to those for FSIM. These requirements can be summarized as follows: (1) minimum filters’ dot-product (sampled cross-correlation at $t=T_f$) to have distinguishable filter shapes, and (2) the inherent ISI from non-Nyquist filters should be kept as low as possible while having the same bandwidth and center frequency; (3) the out-of-band (OOB) radiations should remain within the acceptable level to respect the standards and regulations; (4) the filter shape length should be kept as low as possible. The minimization of the first two factors is crucial to achieve the best system performance, while the minimization of the OOB and the filter length is needed to limit the adjacent channel interference and maintain a low computational complexity, respectively.

3 Theoretical performance analysis

The theoretical performance of IQ-FSIM is assessed in this section by deriving the lower bounds for the probability of error of filter index detection, the total SER, as well as the real/virtual and total BER. Note that the low-complexity MF-based detector approaches the joint ML performance, as shown in the next section. Hence, the analytical expressions will consider only the former.

The errors in filter indices detection affect the subsequent APM symbol due to mismatched filtering on I and/or Q that leads to higher ISI most probably, which thus could degrade the IQ-FSIM overall performance.

The theoretical lower bound expressions will be derived by assuming that the inherent ISI from the IQ-FSIM filter shapes is completely mitigated at the receiver (i.e., $ISI_{resid}=0$). As shown in Fig. 2, the bank of MFs on I/Q took, on each symbol period, the signal ${\hat{s}}^I$/${\hat{s}}^Q$ of L samples that represents the real/imaginary part of ${\hat{s}}$ expressed in (12). Then, these outputs are sampled at $t=T_f$, giving ${r}^I_k$ (expressed in (8)) and ${r}^Q_k$. Firstly, the probability of filter index detection error on the I component is derived, then that on the Q component can be deduced similarly. Thus, the outputs on the I branch ${r}^I_k$ in AWGN channel can be rewritten as:

$$\begin{aligned} r^I_k&= \langle {{\textbf {f}}}_k,{{\textbf {f}}}_{i_n}\rangle \,c_n^I+ \langle {{\textbf {f}}}_k,{{\textbf {v}}}^I_n\rangle \nonumber \\&= {\left\{ \begin{array}{ll} c_n^I + \langle {{\textbf {f}}}_k,{{\textbf {v}}}^I_n\rangle &{} \text {for}\, k=i_n, \\ \langle {{\textbf {f}}}_k,{{\textbf {f}}}_{i_n}\rangle \,c_n^I+ \langle {{\textbf {f}}}_k,{{\textbf {v}}}^I_n\rangle &{} \text {for}\, k\ne i_n. \\ \end{array}\right. } \end{aligned}$$

(13)

The filter shapes in our proposed system share the same bandwidth around the same center frequency to enable a more significant SE improvement. Thus, they have a nonzero correlation or filters’ dot-product that should be kept as low as possible for better filter detection and overall system performance.

Let us suppose that the filter $f_1$ is selected at the transmitter side on the I component for the APM symbol $c_n$. Then, the vector ${{\textbf {r}}}^I_k$ composed of the elements $r^I_k$ for $k = 1, \ldots , N$ is expressed as follows:

$$\begin{aligned} {{{\textbf {r}}}^I_k}= [c^I_n+v^I_1, \langle {{\textbf {f}}}_2,{{\textbf {f}}}_1\rangle \,c^I_n + v^I_2, \ldots , \langle {{\textbf {f}}}_N,{{\textbf {f}}}_1\rangle \,c^I_n + v^I_N ], \end{aligned}$$

(14)

where $v^I_k= \langle {{\textbf {f}}}_k,{{\textbf {v}}}^I_n\rangle$ are real-valued zero-mean mutually statistically independent Gaussian random variables with equal variance $\sigma ^2=\sigma ^2_k=\frac{1}{2}N_0$, where $N_0$ is the noise spectral density [24]. The filter index on I is estimated by taking the maximum squared component of the vector ${{\textbf {r}}}^I_k$: $U^I_k= (r^I_k)^2$. Similarly, the detection is performed on the Q component using $U^Q_k$ that represents the square of the matched filter sampled output in the Q branch. Thus, $U^I_k$ and $U^Q_k$, in general, are described as a statistically independent non-central chi-square distribution for all k, each having 1 degree of freedom where the non-centrality parameter $\alpha ^2_{k,q}$ is given by:

$$\begin{aligned} \alpha ^2_{k,q}={\left\{ \begin{array}{ll} {\mathcal {E}}_q &{} \text {for}\, k=1 \\ \langle {{\textbf {f}}}_k,{{\textbf {f}}}_{1}\rangle ^2\, {\mathcal {E}}_q &{} \text {for}\, k\ne 1,\\ \end{array}\right. } \end{aligned}$$

(15)

where ${\mathcal {E}}_q$ is the square of the real part ${\mathcal {E}}^I_q$ (imaginary part ${\mathcal {E}}^Q_q$) of the transmitted APM symbol $c_n$. Note that the PDF of $U_k^I$ or $U_k^Q$ for IQ-FSIM follows a non-central chi-square PDF having 1 as degree of freedom (square of real numbers) in contrast to the PDF of $U_k$ for FSIM that has 2 as degree of freedom due to summing the squares of real and imaginary components in $U_k$ of [20] (energy of complex number). In the following, the superscripts I and Q for $U_k$ are omitted for the sake of simplicity. The probabilities of error for filter index on I and Q are considered similar with the following differences: the selected filter at the transmitter, and ${\mathcal {E}}_q$. Hence, the PDF of $U_k$ with a given ${\mathcal {E}}_{q}$ in general is defined as [24, p. 43 (2-1-115)]:

$$\begin{aligned} p(u_k)=\frac{1}{\sigma \sqrt{2\pi u_k}}e^{-\Bigl (\frac{u_k+\alpha ^2_{k,q}}{2\sigma ^2}\Bigr )} \cosh \Bigl (\frac{\alpha _{k,q}\sqrt{u_k}}{\sigma ^2}\Bigr ), \end{aligned}$$

(16)

where $u_k\ge 0$ and $\cosh (x)=\frac{e^{x}+e^{-x}}{2}$.

By first deriving the probability of correct filter detection, we can easily deduce the probability of filter index detection error. The former probability corresponds to the probability that $u_1$ is the largest $u_k$, and thus for a given ${\mathcal {E}}_q$ it can be written as follows:

$$\begin{aligned}&P_{c ,{\mathcal {E}}_{q}} = P(U_2<U_1,U_3<U_1,\ldots ,U_N<U_1) = \nonumber \\&\int _{0}^{\infty }{P(U_2<U_1,\ldots ,U_N<U_1 |u_1)\,p(u_1)du_1}, \end{aligned}$$

(17)

where $P(U_2<U_1,U_3<U_1,\ldots ,U_N<U_1 |U_1)$ denotes the joint probability that $u_2,u_3,\ldots u_N$ are all smaller than $u_1$ conditioned on a given $u_1$. Then the probability of correct filter detection is deduced by averaging this joint probability over all $u_1$. To derive the theoretical lower bound, an orthogonal filter bank is considered, which makes these $N-1$ variables $u_k$ statistically independent following a central chi-square distribution ($\alpha ^2_{k,q}=0$ for $k\ne 1$). In this case, the joint probability can be written as a product of $N-1$ marginal probabilities of the form:

$$\begin{aligned} P(u_k<u_1|u_1)=\int _{0}^{u_1}{p_{u_k}(x_k)dx_k},\quad k=2,3,\ldots ,N. \end{aligned}$$

(18)

Hence, the probability of a correct decision is expressed as:

$$\begin{aligned} P_{c ,{\mathcal {E}}_q}=\int _{0}^{\infty }{\left( \prod _{2}^{N}{ P(u_k<u_1|u_1)}\right) \,p(u_1)du_1} \end{aligned}$$

(19)

and the probability of filter detection error with a given ${\mathcal {E}}^I$ is deduced as follows:

$$\begin{aligned} P_{e ,{\mathcal {E}}_q}=1-P_{c ,{\mathcal {E}}_q}. \end{aligned}$$

(20)

Thus, the weighted average of the filter detection error over the set of possible energy levels ${\mathfrak {Q}}^I$ (${\mathfrak {Q}}^Q$) for real (imaginary) part of the APM symbols allows estimating the average probability of filter detection error $P_e^I$ on I ($P_e^Q$ on Q) :

$$\begin{aligned} P_e^I&=\sum _{{\mathcal {E}}^I_q \in {\mathfrak {Q}}^I}{P_{e ,{\mathcal {E}}^I_q}\,\,P({\mathcal {E}}^I_q)}, \end{aligned}$$

(21)

$$\begin{aligned} P_e^Q&=\sum _{{\mathcal {E}}^Q_q \in {\mathfrak {Q}}^Q}{P_{e ,{\mathcal {E}}^Q_q}\,\,P({\mathcal {E}}^Q_q)}, \end{aligned}$$

(22)

where $P({\mathcal {E}}_{q})$ is the probability of occurrence of the energy level ${\mathcal {E}}_{q}$ of each component. For example, when the used APM is a rectangular 8-QAM, ${\mathfrak {Q}}^I=\{1/6, 3/2\}$ (2 possible energy levels for the real part, i.e., 2 possible real squared values for all 8-QAM constellations), and their probability of occurrence is $P({\mathcal {E}}^I_{q})=[0.5,\,0.5]$ if all APM symbols are equiprobable. Also, there is one level for the imaginary part ${\mathfrak {Q}}^Q=\{1/6 \}$. Hence, the total probability of error $P_{e}$ for filter index on I and Q can be expressed as follows when the same filter bank size is used for both branches:

$$\begin{aligned} P_{e}=\frac{1}{2} (P_e^{I}+P_e^{Q}). \end{aligned}$$

(23)

In order to derive the theoretical lower bound for the overall IQ-FSIM system performance, the probability of APM symbol error will be considered. This probability $P_e^{APM}$ can be expressed according to the law of total probability as follows:

$$\begin{aligned}&P_e^{APM}=P_{(\text {APM error } \cap \text { correct filters at I and Q})} \nonumber \\&\quad \quad + P_{(\text {APM error } \cap \text { false filter(s) at I or Q})}\nonumber \\&= (1-P_e^I)(1-P_e^Q)P_{(\text {APM error/correct filters at I and Q})} \nonumber \\&+ P_e^I(1-P_e^Q)P_{(\text {APM error/false filter at I and correct at Q})}\nonumber \\&\quad\quad + P_e^Q(1-P_e^I)P_{(\text {APM error/false filter at Q and correct at I})}\nonumber \\&+\quad\quad P_e^I\,P_e^Q\, P_{(\text {APM error/false filters at I and Q})}, \end{aligned}$$

(24)

where $P_{(A/B)}$ is the probability of event A knowing B.

Note that the $P_{(\text {APM error/correct filters at I and Q})}$ with the assumption of $ISI_{resid}=0$ is the same as that of APM symbol error in a traditional APM system using Nyquist filter shapes. However, when a filter is wrongly estimated on I and/or Q components, the APM symbol is detected by using a mismatched filter output. In this case, the signal-to-noise ratio (SNR) seen at the input of the APM symbol detector is affected by the filters’ dot-product as shown in (13) when $k\ne i_n$. Let us define $\gamma$ as the post-MF signal-to-noise ratio seen at the input of APM symbol detector, and $P_e^{\text {Nyq APM}} (\gamma )$ as the probability of APM symbol error in general with traditional Nyquist transceiver ($P_e^{\text {Nyq APM}} (\gamma )$ for M-ary QAM is given by [24, p. 280 (5.2-78) and (5.2-79)]). According to (13), $\gamma$ in general, can be expressed as follows:

$$\begin{aligned} \gamma =\frac{\langle f_{k^I},f_i\rangle ^2{\mathcal {E}}^I_{q} +\langle f_{k^Q},f_j\rangle ^2 {\mathcal {E}}^Q_{q}}{N_0}, \end{aligned}$$

(25)

where $k^I$ ($k^Q$) denotes the k matched filter output for I (Q) component. It is worth mentioning that $\gamma$ can be simplified to $E_s/N_0$ only when the filters at I and Q are both correctly detected (i.e., correct MF: $k^I=i$ and $k^Q=j$), where $E_s$ is the APM symbol energy. Thus, $P_{APM \,error/correct\, filters}$ is calculated using $P_e^{\text {Nyq APM}}(\gamma )$ with $\gamma =E_s/N_0$.

However, the other conditional probabilities $P_{(\text {APM error/false filter at I and correct at Q})}$,

$P_{(\text {APM error/false filter at Q and correct at I})}$, and $P_{(\text {APM error/false filters at I and Q})}$ with at least one filter error are deduced by averaging $P_e^{\text {Nyq APM}}(\gamma )$ over all possible filters’ dot-product combination or post-MF SNRs $\gamma$s, i.e., averaging over the corresponding $\gamma$ set satisfying the condition ($\forall \, k^I\ne i$ and $k^Q=j$), ($\forall \, k^I=i$ and $k^Q\ne j$), or ($\forall \, k^I\ne i$ and $k^Q\ne j$), respectively.

Note that an IQ-FSIM symbol is considered to be estimated correctly when all the carried information (APM symbol and both filter indices) are correctly detected. Then the probability of a correct IQ-FSIM decision is $(1-P_e^I)(1-P_e^Q)(1-P_e^{APM})$ and the IQ-FSIM SER is given by:

$$\begin{aligned} SER=1-(1-P_e^I)(1-P_e^Q)(1-P_e^{APM}). \end{aligned}$$

(26)

The virtual BER can be easily deduced from the probability of error for filter indices, similar to the real BER calculated using the probability of error for APM symbol [24, p. 262 (5-2-24)], so these BERs are calculated as follows:

$$\begin{aligned} P_b^I&=\frac{N}{2(N-1)} P_e^I \end{aligned}$$

(27)

$$\begin{aligned} P_b^Q&=\frac{N}{2(N-1)} P_e^Q \end{aligned}$$

(28)

$$\begin{aligned} P_b^{APM}&=\frac{M}{2(M-1)} P_e^{APM}. \end{aligned}$$

(29)

Accordingly, the total BER is the weighted average of these probabilities in (27)-(29):

$$\begin{aligned} BER=\frac{\log _2 N}{{\mathcal {L}}_{\text {IQ-FSIM}}}P_b^I+\frac{\log _2 N}{{\mathcal {L}}_{\text {IQ-FSIM}}}P_b^Q+\frac{\log _2 M}{{\mathcal {L}}_{\text {IQ-FSIM}}}P_b^{APM}. \end{aligned}$$

(30)

Note that if different filter bank size N is used between I and Q, Eqs. (23) and (30) should adapt the weighted average, and (27) and (28) should consider the corresponding filter size. Finally, it is clear that the dot-product between all filters affects the filter and APM error probabilities as well as the SNR as shown in (15) and (25), respectively. Similarly, the inherent ISI from non-Nyquist filters can also degrade the detection if it was not correctly estimated and canceled because the Signal-to-Noise-plus-Interference Ratio (SNIR) will degrade by adding the residual ISI power to the denominator of (25). Therefore, the filter bank design should consider minimizing these two factors to approach the theoretical lower bound of the IQ-FSIM system. Note that the filter bank design is out of scope of this paper. In the next section, the theoretical lower bounds for the error probability of filter indices and BER are validated with different IQ-FSIM configurations.

4 Numerical results analysis and discussions

In this section, different theoretical and simulation results are presented and discussed. Firstly, Monte Carlo simulations are performed to validate the theoretical lower bound of probability of filter error (23), and the effects of the residual ISI and filter correlation are highlighted. Secondly, the BER performance of the proposed IQ-FSIM system is assessed with both proposed detectors and compared in AWGN channel to the equivalent conventional transceiver (QAM+Nyquist filters) of same SE. These comparisons are performed at different transmission rates (4 to 8 bits/symbol or bit per channel use (bpcu)) using the total BER theoretical lower bound (30) and the Monte Carlo results. This study validates the theoretical expressions, shows IQ-FSIM advantages and limitations, and highlights its best configuration. Afterward, these comparisons are extended to consider the previously proposed FSIM scheme in AWGN, flat, and frequency-selective Rayleigh fading channels. The main simulation parameters for all these subsections are summarized in Table 1.

Table 1 Simulation parameters

Fig. 4

Impulse response for the 2 designed IQ-FSIM filters and RRC pulse shaping filter having same $\eta =10$ and $\uplambda =8$

Fig. 5

Magnitude and phase responses of the filter $f_1$ depicted in Fig. 4

Fig. 6

Magnitude and phase responses of the filter $f_2$ depicted in Fig. 4

Fig. 7

Output of the correct matched filter at the receiver side for RRC and the 2 designed IQ-FSIM filters having same $\eta =10$ and $\uplambda =8$

Note that 2 and 4 non-optimal filters are designed according to the IQ-FSIM filter bank design requirements, and they are used in the following to show the SE and EE advantages of IQ-FSIM. Figure 4 presents the impulse response of the two filters, whereas Figs. 5 and 6 show their phase and magnitude responses. In addition, the correct matched filter outputs at the receiver (${{\textbf {f}}}_i*{{\textbf {g}}}_i$) are shown in Fig. 7. The inherent ISI clearly appears in $f_2$ because the result of ${{\textbf {f}}}_2*{{\textbf {g}}}_2$ has nonzero values for some $t = n.T_{sym}$. Note that these filters differ than those used in [20], more specifically the ‘Filter 2’ is an enhanced version of that in [20].

4.1 Performance analysis: theoretical lower bound versus Monte Carlo simulation

Fig. 8

Comparison of the total probabilities of filter error on I and Q using ISI-EC, perfect ISI cancellation, and the derived theoretical lower bound, $N=2$ (2-IQ-FSIM-MQAM). The SE is: a [4, 5] bpcu using $M=[4,8]$, b [6, 7] bpcu using $M=[16,32]$

The theoretical lower bound and simulated filter index error probabilities with perfect ISI cancellation and with ISI-EC are compared in Fig. 8. In the following, the notation $N-$IQ-FSIM$-M$APM is adopted to describe the proposed system.

It is clear from Fig. 8 that the simulation results with perfect ISI cancellation validate the analytical lower bound (23) derived under the assumption of perfect ISI cancellation and orthogonal filters. However, an SNR gap between these curves may appear due to non-orthogonal filters. Moreover, the difference between simulated results with perfect ISI cancellation and ISI-EC is due to the residual ISI when the ISI-EC proposed in Sect. 2.2.3 is adopted. It is worth mentioning that 2-IQ-FSIM-QPSK with ISI-EC is very tight to its lower bound because QPSK is more robust to ISI. However, the degradation due to residual ISI appears with higher M-ary QAM that are more sensitive to ISI, as shown in Fig. 8b, but it is minimal at low SNR values. Note that the used filter banks are non-optimal, but they satisfy all the filter bank design requirements summarized at the end of Sect. 3. Thus, better results can be achieved when optimal filter shapes are used. Therefore, this theoretical lower bound can be considered a helpful indicator for evaluating the proposed system’s performance.

4.2 Comparison to equivalent M-ary QAM in AWGN channel

Fig. 9

BER performance for 2-IQ-FSIM-MQAM, its theoretical lower bound, and its equivalent scheme 4MQAM of the same SE: a 4 bpcu using $M=4$, b 5 bpcu using $M=8$

Fig. 10

BER performance for 2-IQ-FSIM-MQAM, its theoretical lower bound, and its equivalent scheme 4MQAM of the same SE: a 6 bpcu using $M=16$, b 7 and 8 bpcu using $M=32$ and $M=64$, respectively

In the following, the proposed system IQ-FSIM is compared to its equivalent traditional system of the same SE that uses QAM and Nyquist RRC pulse shaping filter. The system performance is evaluated in AWGN channel using theoretical lower bound (30) and Monte Carlo simulations with ISI-EC. Figures 9 and 10 show the comparison between the BER performance of N-IQ-FSIM-M and its equivalent QAM with the same SE (i.e., QAM of order $N^2M$). It is clear that the low-complexity MF-based detector achieves the joint ML performance in all configurations. In addition, the proposed IQ-FSIM system has a significant performance gain in all SNR ranges compared to the equivalent QAM system of order less than 128 (7bpcu) while using these two non-optimal filters in 2-IQ-FSIM-64QAM. It is worth mentioning that the BER of 2-IQ-FSIM-QPSK with ISI-EC is very tight to its lower bound while achieving 3.55 dB gain at BER $=10^{-4}$ compared to its equivalent system 16 QAM as shown in Fig. 9a. Similarly, 2-IQ-FSIM-16QAM outperforms 64QAM by 2 dB, but a 3.55 dB gap appears with its lower bound mainly due to the residual ISI effect on higher M. Note that 2-IQ-FSIM-8QAM has 1 dB less gain compared to 32QAM due to the inherited performance degradation when using non-square QAM. Moreover, as M-ary QAM scheme is increased with IQ-FSIM system, the residual ISI effect becomes more important, as shown in Fig. 10b. For instance, the 2-IQ-FSIM-32QAM and 2-IQ-FSIM-64QAM systems can only achieve some gain at the low SNR region, thus increasing the filter bank size to reach such SE$=[7,8]$ bpcu is more favorable than using very large M with $N=2$ as it will be highlighted later. This degradation appears due to: (i) higher sensitivity of large M-ary QAM to ISI; (ii) error propagation that results from the residual ISI caused by the error in tentative and/or feedback decisions in the preceding block for ISI-EC. However, the IQ-FSIM BER lower bound depicted in Fig. 10b shows that the design of an optimal filter bank and a better ISI mitigation technique can allow achieving a better performance.

Fig. 11

Spectrum comparison between the conventional QAM system using RRC pulse shaping and the proposed IQ-FSIM system using 2 non-optimal filters

The spectrum of the proposed IQ-FSIM scheme is compared to that of the conventional transceiver in Fig. 11. The proposed scheme conserves most of the radiated power on the targeted in-band bandwidth, similar to the conventional M-ary QAM system, but a higher OOB radiation is observed due to energy redistribution between in-band and OOB mainly in filter 2, as shown in Fig. 6. Note that this OOB for IQ-FSIM can adhere to the spectrum regulations for different applications, and it will be considered in the future filter bank design to be kept as low as possible.

For further analysis of the proposed system, the 4 filter shapes designed for FSIM in [20] are used to evaluate IQ-FSIM performance, and their dot-product matrix $\Xi$ is given by:

$$\begin{aligned} \Xi =\begin{bmatrix} 1&{}0.7615&{}0.7284&{}0.334\\ 0.7615&{}1&{}0.6376&{}0.0887\\ 0.7284&{}0.6376&{}1&{}0.67\\ 0.334&{}0.0887&{}0.67&{}1\\ \end{bmatrix}. \end{aligned}$$

(31)

Fig. 12

BER performance for 4-IQ-FSIM-MQAM, its theoretical lower bound, and its equivalent scheme 16MQAM of the same SE: a 6 bpcu using $M=4$, b 8 bpcu using $M=16$

The performance of 4-IQ-FSIM-QPSK (6 bpcu) is depicted in Fig. 12a, where a 4.3 dB gain is achieved compared to 64 QAM at BER$=10^{-4}$. However, the gain with 4-IQ-FSIM-16QAM is more important in the low SNR region (5.2 dB at BER$=10^{-1}$), as shown in Fig. 12b, and it decreases to reach 0.5 dB at BER$=10^{-4}$ due to residual ISI effect on higher M-ary QAM and the used non-optimal filter bank. Although the residual ISI and the high dot-product between some filters lead to the observed degradation between the 4-IQ-FSIM performance systems with ISI-EC and their theoretical lower bounds, this IQ-FSIM scheme achieves higher SE gain than the existing schemes at lower SNR requirements. Hence, IQ-FSIM even with non-optimal filters also gives a significant EE gain in bits/joule, especially with low M-ary APM due to the important SNR gain that reduces the required transmit power to reach any target BER and thus the system energy consumption.

4.3 Comparison to equivalent systems with/without IM

FSIM scheme showed a significant gain compared to existing single carrier SISO-IM schemes in the different conditions [20]. Hence, it is enough to show that IQ-FSIM outperforms FSIM and conventional QAM to confirm the IQ-FSIM superiority over the existing SISO systems with/without IM. Nevertheless, the next section will confirm this fact by comparing the proposed scheme to QAM, FSIM, SC-IM [12], and DM-SC-IM [13].

4.3.1 Performance

Fig. 13

BER performance for IQ-FSIM and different equivalent systems of similar SE (4 and 6 bpcu) in: a AWGN channel, b flat Rayleigh fading channel ($J=1$)

Fig. 14

BER performance for IQ-FSIM and different equivalent systems of same SE in a frequency-selective Rayleigh fading $J=4$: a SE$=4$ bpcu, b SE$=6$ bpcu

Fig. 15

Computational complexity reduction by the proposed N-IQ-FSIM-QPSK ($M=4$) compared to FSIM scheme using either same modulation order or same number of filter shapes

In this subsection, the two proposed schemes in the filter IM domain, FSIM and IQ-FSIM, are compared to their equivalent QAM scheme and existing single carrier IM schemes of the same SE (SC-IM and DM-SC-IM) in AWGN and fading channels in Figs. 13 and 14. Note that the proposed low-complexity MF-based detector and the ISI-EC technique are used for both filter IM schemes, and the joint maximum likelihood is used for SC-IM and DM-SC-IM. The average BER in fading channels is evaluated using Monte Carlo simulations with 5000 channel realizations, and the frame setup in fading channels is: $N_s=1015, N_{ZP}=9$ in terms of symbol length for all schemes compared at the same SE (SE$\approx 4$bpcu and SE$\approx 6$ bpcu). In order to achieve SE$\approx 4$bpcu, SC-IM needs to transmit 64QAM in the $N_a=3$ activated time slots from each group of $N_g=5$ (number of groups $G=N_s/N_g=203$) because its SE is $\frac{G}{N_s+N_{ZP}} (N_a \log _2(M) + \lfloor \log _2 {N_g\atopwithdelims ()N_a}\rfloor$) bpcu. Similarly, DM-SC-IM is configured to achieve same SE$\approx 4$bpcu with $G=203, N_g=5$ according to: $\frac{G}{N_s+N_{ZP}} (N_a \log _2(M_a)+(N_g-N_a) \log _2(M_b) + \lfloor \log _2 {N_g\atopwithdelims ()N_a}\rfloor$), where $M_a=16$ and $M_b=4$ are the modulation order used at $N_a$ primary and $N_g-N_a$ secondary activated slots, respectively.

As depicted in Fig. 13a, N-IQ-FSIM-MQAM scheme having SE=4 bpcu (6 bpcu) outperforms in AWGN the equivalent QAM system by 3.72 dB (4.3 dB), and its predecessor FSIM by 2 dB (2.3 dB) when using either the same number of filters “N-FSIM-(NM)QAM” or the same modulation order “$N^2$-FSIM-(M)QAM”.

Recalling that FSIM and IQ-FSIM systems reach their theoretical lower bounds when the filter bank size is 2. However, when this number is larger, an SNR gap appears between the realistic performance of both systems and their lower bounds, as shown in previous BER figures. Although the theoretical curves are not added in Fig. 13a to avoid a more congested figure, it is worth mentioning that $N^2$-FSIM has better performance than N-IQ-FSIM only in terms of the theoretical lower bound in AWGN. However, the former needs the square of the number of filters used in the latter, and this larger number of filters adds more constraints in the filter design that results in higher filters’ dot-products as shown in (31), and thus more degradation compared to the lower bound.

It is worth mentioning that the performance gain of IQ-FSIM is more significant when compared to SC-IM (7.7 dB) and DM-SC-IM (5.5 dB) at SE$\approx 4$ in AWGN as shown in Fig. 13a. These results are expected because SC-IM requires much higher modulation order in activated slots to compensate the SE loss in deactivated time slots, and DM-SC-IM performance is highly affected by the ability to differentiate the two modulation sets used in primary and secondary activated slots. Hence, both filter IM schemes are much better than their equivalent $N^2M$QAM modulation and existing SISO-IM schemes of the same SE.

Similarly, IQ-FSIM achieves the best performance in flat Rayleigh fading channel as shown in Fig. 13b, where performance gain of 0.5 dB, 1.3 dB, 1.8 dB, and 5.2 dB is achieved at BER$=10^{-3}$ compared to $N^2$-FSIM-MQAM, N-FSIM-(NM)QAM, $(N^2M)$QAM, and (DM-)SC-IM, respectively, of the same SE$=4$ bpcu. While both filter IM schemes using the same number of filters $N=4$ are quite similar using these non-optimal filters at higher SE=6 bpcu, and they outperform QAM by 3.6 dB. Note that filter IM schemes have also a significant gain at SE$\approx =6$ bpcu compared to the other SISO-IM schemes ((DM)-SC-IM), but the curves are discarded to avoid more congested Fig. 13b.

The results in frequency-selective fading channels at different SEs prove that IQ-FSIM has better performance than equivalent conventional systems or quite similar to the best scheme ($<0.1$ dB difference), as shown in Fig. 14b.

This can be explained by the fact that IQ-FSIM allows using lower modulation order M (i.e., better robustness to ISI) or a lower number of filters (lower filters’ dot-product) to reach the same SE, making it the best scheme in the different channels. To summarize, N-IQ-FSIM-MQAM provides a significant gain compared to QAM and existing SISO-IM schemes in all configurations and channel conditions, and IQ-FSIM has always better performance than FSIM or similar in the worst case. Although FSIM can reach in some cases IQ-FSIM performance, it will be highlighted in the following subsection that the latter is still preferred due to its lower receiver complexity.

4.3.2 Computational complexity

Another critical factor to consider when comparing both schemes in the filter IM domain (i.e., FSIM and IQ-FSIM) is the computational complexity of the transceiver. This section considers the complexity in terms of real multiplications (RMs). The transmitter complexity of both schemes is mainly in the pulse shaping step (convolution) like any conventional system, but as shown in (2)–(3), the (IQ)-FSIM pulse shaping is merely a simple time-domain multiplication of L real samples by two real values that leads to 2L RMs per symbol in both systems. Hence, the transmitter complexity is the same for both schemes for any N and M system configuration because the mappers and OLA block do not include any RM.

However, the receiver complexity increases with N and M, as highlighted in the following. Without loss of generality, the equalizer complexity is omitted when comparing FSIM and IQ-FSIM complexities because it is the same and independent of the system configuration. The complexity of ISI-EC in RMs is the sum of (IQ)-FSIM detector for tentative decision and the pulse shaping complexities (2L RMs) where the latter is performed two times, as shown in Fig. 3. The sampled output of each matched filter in (IQ)-FSIM detector requires L multiplications of a complex number by real values (L multiplications of real numbers for I and L for Q), and thus it is 2L RMs in both systems. The energy calculation to detect each filter index is performed N times using 2 RMs (square of real and imaginary components with/without their addition in FSIM/IQ-FSIM). Similarly, the Euclidean distance in the APM ML is performed M times using 2 RMs each. The OLA block and ISI cancellation step in Fig. 3 require only additions/subtractions. Therefore, the computational complexities ${\mathcal {C}}$ per symbol in terms of RMs for FSIM or IQ-FSIM receiver main blocks are given by:

$$\begin{aligned} {\mathcal {C}}_\text {Pulse}\,\text {shaping}&=2L, \end{aligned}$$

(32)

$$\begin{aligned} {\mathcal {C}}_\text {(IQ)-FSIM detector}&= 2NL+2N+2M, \end{aligned}$$

(33)

$$\begin{aligned} {\mathcal {C}}_\text {ISI-EC}&= {\mathcal {C}}_\text {(IQ)-FSIM detector} + 2{\mathcal {C}}_\text {Pulse shaping}\nonumber \\&=2NL+2N+2M+4L, \end{aligned}$$

(34)

$$\begin{aligned} {\mathcal {C}}_\text {(IQ)-FSIM Rx}&={\mathcal {C}}_\text {(IQ)-FSIM detector}+{\mathcal {C}}_\text {ISI-EC}\nonumber \\&= 4NL+4N+4M+4L\nonumber \\&=4(NL+N+M+L). \end{aligned}$$

(35)

It is clear that a shorter filter length L is preferred to minimize the complexity, and both FSIM and IQ-FSIM transceivers have the same complexity only when using the same system configuration (N and M). However, their complexities should be compared at the same SE constraint for a fair comparison. In order to reach the SE in (1) of IQ-FSIM using N filter shapes on each I/Q component and M-ary APM (denoted by N-IQ-FSIM-MAPM), FSIM requires either to use: NM as QAM modulation order and the same number of filter shapes N (denoted by N-FSIM-(NM)APM), or $N^2$ filter shapes and the same modulation order M (denoted by $N^2$-FSIM-MAPM). Hence, the higher complexity of FSIM receiver compared to IQ-FSIM is evident as depicted in Fig. 15 where IQ-FSIM showed a significant complexity reduction up to $93.7\%$ compared to $N^2$-FSIM-MAPM using the same modulation order. It is worth mentioning that the latter FSIM configuration has the nearest performance to the IQ-FSIM scheme, but the latter reduces the detection complexity by $40\%$ at least.

Therefore, the IQ-FSIM scheme has the lowest complexity among the different FSIM configurations of the same SE. In addition, the former leads to the best performance or near it in AWGN and fading channels compared to different systems with/without IM (e.g., FSIM, QAM, and existing SISO-IM schemes).

5 Conclusion

This paper proposes a novel scheme called “In-phase and Quadrature Filter Shape Index Modulation” (IQ-FSIM) in the filter IM domain. The proposed scheme conveys information bits by an APM symbol (e.g., QAM, PSK, etc.) and filters shapes’ indices on the I and Q components. Thus, the IQ-FSIM system increases the SE by $2\log _2 N$ compared to M-ary APM, which is double the gain obtained with the FSIM scheme. This makes the IQ-FSIM one of the most spectral-efficient SISO-IM schemes. In addition, the proposed reconfigurable IQ-FSIM system generalizes the previously proposed FSIM scheme, and thus most of the existing Nyquist and faster-than-Nyquist SISO schemes with/without IM as they are already generalized by FSIM [20].

This novel scheme is proposed along with a joint ML detector that jointly estimates both filter indices and APM symbol using an exhaustive search. In addition, a low-complexity optimal MF-based detector is proposed to estimate the filter indices on I and Q in parallel, and then the transmitted APM symbol is deduced. Moreover, a linear ISI-EC technique is presented to mitigate the predictable ISI introduced by the filter shapes being used on I and Q.

In addition, the system performance of IQ-FSIM is characterized by deriving the theoretical lower bound of filter error probability, symbol error rate, and total BER. The Monte Carlo simulations validated these lower bounds and proved that the IQ-FSIM system with the appropriate configuration, and even with non-optimal filter shapes, offers significant SE and EE gains compared to the other existing SISO schemes of the same SE.

It is worth recalling that it is preferred to use a low-order APM for better ISI robustness and to increase the filter bank size for higher SE by exploiting more IM advantages (e.g., use 4-IQ-FSIM-QPSK instead of 2-IQ-FSIM-16QAM or 64QAM to transmit 6 bpcu). Hence, the IQ-FSIM system configuration should be carefully selected for the best SE and EE gains.

In addition, the results in the different channels proved that N-IQ-FSIM-MQAM is the most spectral-efficient and energy-efficient scheme compared to the currently existing SISO schemes with/without IM (FSIM, QAM, PSK, Time IM techniques, modulation type IM, etc.).

To summarize, IQ-FSIM provides several advantages such as: (1) higher SE while using low APM modulation order, which is beneficial for (2) better robustness to ISI and RF impairments in general (e.g., phase noise), and also for (3) lower peak-to-average power ratio (PAPR) that allows with (4) its smaller SNR requirement to achieve (5) higher energy efficiency and lower power consumption. However, these advantages come at the cost of linear computational complexity increase proportional to L at the receiver compared to QAM, but the filter length L can be minimized during the filter bank design. Nevertheless, IQ-FSIM provides up to 93.7% receiver complexity saving compared to FSIM and has similar transmitter complexity like QAM and FSIM.

Finally, these filter IM schemes insist that accepting a certain level of controllable and predictable ISI allows reaching a high system capacity even with a low-order M-ary APM and non-optimal filter shapes. It is worth mentioning that IQ-FSIM can achieve even higher EE and SE gains by designing an optimal filter bank and a larger filter bank (higher N), respectively. Hence, the optimal filter bank design for the filter IM domain is an open research topic.

6 Methods/experimental

The aim of performing simulation studies is to provide a theoretical and numerical evaluation of the performance and complexity of the proposed scheme and a comparison to conventional and existing techniques. For this purpose, Monte Carlo simulations were performed and the derived theoretical performance is validated. Although the methods and experimental setup are fully detailed in Sect. 4, some general guidelines are summarized in the following. The presented results for the proposed scheme can be categorized into three types: filter characteristics, performance, and computational complexity. The figures of filter characteristics are generated using the non-optimal filter shapes designed to satisfy the proposed system requirements, and these coefficients can be provided as mentioned in section availability of data. All performance analyses in AWGN channel (probabilities of filter error on I and Q, un-coded BER) are based on the mathematical equations validated by extensive Monte Carlo simulations using 10⁶ symbols. The Monte Carlo performance simulations in fading channels is averaged over 5000 randomly generated channel realizations, where each frame composed of 1015 data symbols and 9 zero-padding symbols is subjected to different channel. FSIM and IQ-FSIM system use constellation sets normalized to unity average power, and normalized filter shapes on I and Q. For instance, the average power of transmitted symbols $c_n$ as well as all the filters f in the bank for I and Q components is normalized according to $\mathop {{\mathbb {E}}}[| c_n|^2]=1$ and $\sum _{m=0}^{L-1}{{f}^2[m]}=1$, respectively. The SNR in the x-axis of the performance figures includes the oversampling effect for all schemes. The computational complexity analysis is based on the derived mathematical equations.

Appendix

1.1 Proof of the FSIM scheme generalization by IQ-FSIM

The overlap and add block input for IQ-FSIM can be expressed as follows according to (2) and (3):

$$\begin{aligned} s_n[m]= s^I_n[m] + \textrm{j}\, s^Q_n[m]=c_n^I {f}_{i_n}[m] +\textrm{j}\, c_n^Q {f}_{j_n}[m]. \end{aligned}$$

(36)

Using the same indexed filter shape on I and Q (${f}_{i_n}={f}_{j_n}$) leads to:

$$\begin{aligned} s_n[m]= (c_n^I +\textrm{j}\,c_n^Q){f}_{i_n}[m]= c_n{f}_{i_n}[m]. \end{aligned}$$

(37)

Thus, we obtain the FSIM scheme as a special case of IQ-FSIM ((37) which is the same as (2) in [20], and the overlap and add block is the same for both FSIM and IQ-FSIM schemes).