Improved demapping for channels with data-dependent noise

In this section, we describe the system components and present some examples of distortion. Subsequently, a reduced channel model is presented for optimal demapping.

2.1 Communication scenarios

We consider two communication scenarios depicted in Fig. 1. In both cases, the information bits are encoded with an error-correcting code (Encoder), mapped to symbols (Mapper), and converted to a signal for transmission (Tx Filter).

2.1.1 Scenario 1: nonlinear amplifier

In the first scenario, a predistorter is used to compensate for the nonlinear power amplifier (PA). This situation is common in satellite communications involving a traveling-wave tube amplifier (TWTA) [21] and wireless applications where power efficiency is important [2]. Imperfect compensation results in an overall system that is nonlinear. For this work, the TWTA characteristics are modeled with the memoryless model proposed by Ghorbani and Sheikhan [22]. The signal predistortion is implemented with a complex-valued polynomial [23], such that the output signal, y, is

$$\begin{array}{*{20}l} y(n) = \sum_{k=1}^{K}\sum_{l=0}^{L} c_{l,k} x(n-l)|x(n-l)|^{k-1}, \end{array} $$

(1)

where x(n) is the input signal at sample n and c _l,k are the complex coefficients for delay l and order k.

2.1.2 Scenario 2: phase noise

The second scenario assumes a linear PA (and no predistortion) but the addition of phase noise at the receiver. In this case, the output, y, is

$$\begin{array}{*{20}l} y(n) = x(n) e^{j\phi(n)} \end{array} $$

(2)

where the phase noise ϕ(n) is simulated using filtered Gaussian noise to match a given power spectral density mask [24]. Although phase noise is well studied and can be compensated in other ways (e.g., [25, 26]), we use this example to demonstrate the generality of our demapping approach.

2.2 Motivating examples

The above system imperfections affect the location of the received symbols. In addition to a shift in the centroid of each cluster of symbols, the spread and shape of the symbol clusters may also be modified, e.g., due to nonlinear intersymbol interference (ISI). As motivating examples, we simulate scenarios 1 and 2 with a 64-APSK and 64-QAM constellation, respectively. The additive noise had an E _s /N₀ of 26 dB and received symbols are plotted in Fig. 2a and b (more simulation details are given in Section 4). In scenario 1, the predistortion in (1), combined with the nonlinear amplifier, results in a small shift in centroid locations while the spread of the symbols varies substantially across constellation points. For example, symbols from the outer constellation ring exhibit larger variation compared to the symbols from the inner rings even though they experience the same level of thermal noise; further, correlation between the I and Q components leads to a non-isotropic distribution of symbol locations. Similarly for scenario 2, due to the nature of phase noise in (2), outer constellation points experience more degradation (Fig. 2b). These effects can be quantified by calculating the covariance matrix for each constellation point, as illustrated in Fig. 2c, d.

2.3 Reduced channel model

The distortions above can be modeled with a general bitwise channel model suitable for demapping, as shown in Fig. 3. Components between the mapper and demapper are represented at the symbol level as a discrete time channel with data-dependent additive noise Z. The input to the channel, X, is mapped from bits B₁,…,B _m by the modulator and takes values from a finite alphabet (or constellation), \(\mathcal S=\{x_{1},\ldots, x_{M}\}\) with M=2 ^m . For simplicity, we consider the Iand Qcomponents of a symbol as elements of a vector. Thus X, Y, and Z are in \(\mathbb R^{2}\). The output Y is defined by the signal model,

$$\begin{array}{*{20}l} Y = X + Z \end{array} $$

(3)

The noise is characterized by the conditional probability density function (PDF) of Z given X=x _k for k=1,…,M,

$$\begin{array}{*{20}l} p_{Z|X}(z | x_{k}) = \mathcal N (\delta_{k},\Sigma_{k}), \end{array} $$

(4)

where \(\mathcal N(\delta _{k},\Sigma _{k})\) denotes the bivariate Gaussian distribution with mean δ _k and covariance Σ _k ; the subscript k emphasizes the dependency of the noise distribution on the channel input x _k . The conditional PDF of the channel follows from the signal model,

$$\begin{array}{*{20}l} p_{Y|X}(y | x_{k}) = \mathcal N (\mu_{k},\Sigma_{k}), \end{array} $$

(5)

where μ _k =x _k +δ _k is the centroid of the received symbol cluster for the kth constellation point. The role of the demapper is to convert the received symbol Y into soft information about the bits, denoted L₁,…,L _m .

In this reduced channel model, the symbol covariance Σ _k captures the total uncertainty from two sources: (i) the AWGN and (ii) the nonlinear distortion or phase noise. We refer to the former as thermal noise, SNR, or E _s /N₀ and the latter as distortion. The reduced channel model is a generalization of the common additive Gaussian noise model used for demapping in QAM systems [7, 11], with a non-diagonal covariance that depends on the transmitted symbol. The extra degrees of freedom are useful to represent general nonlinearities such as those described in Section 2.1, while the Gaussian assumption keeps the computations tractable during demapping.

This work proposes an improved demapper that uses knowledge of the symbol clusters at the receiver. Demodulation based on the received symbol centroids, μ _k , has been proposed previously [13, 14]; however, this work extends the concept to exploit the additional information provided by the symbol covariances. Intuitively, for a hard-decision demapper, constellation points with a larger covariance should have a larger decision region to capture the uncertainty. This intuition follows for a soft-decision demapper based on log-likelihood ratios.

where \(\mathcal I_{i}(x)\) is the set of bit indices where symbol x has a bit value of 1 (excluding the current index). That is, \(\mathcal I_{i}(x)=\{ j\in \{1,\ldots,m\}\,|\, j\ne i \land b_{j}=1 \}\). For APP demapping, the L value in (6)–(8) is used. However, in this work, we also use the extrinsic L value, \(L_{i}^{e}\), directly for two purposes: first, to compute the maximum achievable rate in Section 4.1; secondly, to exchange soft information in iterative demapping and decoding in Section 4.4.

where |·| denotes the determinant of a matrix. The mean μ _k and covariance Σ _k for k=1…,M can be estimated from the received symbols, elaborated below. The key distinction between the proposed demapper and existing methods is the use of the symbol-dependent covariance matrix Σ _k .

3.1 Covariance estimation

In a practical system, a necessary step for the proposed demapping scheme is to estimate the covariance of each symbol cluster. A simple method to estimate the symbol distribution is to use known pilot symbols inserted in the transmitted frame. The received pilots, y _i , can be grouped according to the kth constellation point and the sample mean and covariance used as the unbiased estimates of the distribution parameters,

$$\begin{array}{*{20}l} \mu_{k} &\approx \frac{1}{N}\sum_{i=1}^{N} y_{i} \end{array} $$

(11)

$$\begin{array}{*{20}l} \Sigma_{k} &\approx \frac{1}{N-1}\sum_{i=1}^{N} (y_{i}-\mu_{k})(y_{i}-\mu_{k})^{T}. \end{array} $$

(12)

For a low number of pilots, improved results are possible using multiple frames or biased shrinkage estimates [27]. The estimates above are appropriate for the reduced channel model in Section 2.3. Improved estimates may be possible incorporating prior information from a more detailed nonlinear model; however, in this work, we keep the model general to widen the potential applications.

In the absence of pilot symbols, “blind estimation” of the centroids and covariances must be performed using the received symbols. A fixed set partitioning can be used to group symbols prior to calculation of the centroids [13, 28]. A two-step adaptive set partitioning was proposed in [14]. Alternatively, we conjecture that a probabilistic approach with soft boundaries would be optimal. We pose the problem as estimating the parameters of a Gaussian mixture model (GMM) from samples of the underlying distribution, where the mixture weights are equal and set to 1/M. That is, we consider the received symbols as the samples from the distribution,

$$\begin{array}{*{20}l} p(y | \theta) = \frac{1}{M}\sum_{k=1}^{M}\mathcal N(y;\mu_{k},\Sigma_{k}) \end{array} $$

(13)

where the goal is to estimate the unknown means and covariances θ=[μ₁,Σ₁,…μ _M ,Σ _M ] ^T from N received symbols y₁,…,y _N . These parameters can be estimated efficiently using the expectation-maximization (EM) algorithm [29, 30]. The advantage of this algorithm is that hard decisions about the transmitted symbols are not performed; instead, it computes “soft labels,” denoted γ _i,k , which are posterior probabilities that the kth constellation point was transmitted given the received symbol y _i . These are then used to weight the sample mean and covariance calculations. That is, after initialization, the following steps are repeated until convergence: E-step The posterior probabilities for the transmit symbols γ _i,k are computed using Bayes’ rule and the current parameters θ [30]. M-step The parameters θ are re-estimated using the current transmit probabilities γ _i,k as,

$$\begin{array}{*{20}l} \mu_{k} &\approx \frac{1}{\tilde N_{k}}\sum_{i=1}^{N} \gamma_{i,k} y_{i} \end{array} $$

(14)

$$\begin{array}{*{20}l} \Sigma_{k} &\approx \frac{1}{\tilde N_{k}}\sum_{i=1}^{N} \gamma_{i,k} (y_{i}-\mu_{k})(y_{i}-\mu_{k})^{T}, \end{array} $$

(15)

where \(\tilde N_{k} = \sum _{i} \gamma _{i,k}\) is the effective number of symbols assigned to cluster k.

We briefly compare the blind GMM algorithm and the pilot-based method through simulation. The symbol covariances were estimated from each algorithm after the nonlinear distortion in Scenario 1. The true covariance was calculated using all the received symbols and knowledge of the transmitted symbols. The normalized mean squared error (NMSE) between the estimated and true covariance was calculated for various levels of distortion at an SNR of 25 dB. Figure 4 illustrates that the GMM algorithm is more accurate for low levels of distortion since accurate assignment of symbols to constellation points can be performed. After assignment, the large number of symbols is used to reduce the error in the estimates. However, as the distortion increases, the cluster boundaries begin to overlap and the accuracy of the covariances decrease. Likewise, the performance of the GMM algorithm degrades for lower SNR. In practice, communication systems without pilots typically have low modulation orders and relatively high SNR—a situation where we expect the GMM algorithm to perform well. Estimation using 60 or 120 pilot symbols per constellation point results in a relatively constant error of 6 or 3%, respectively, for varying levels of distortion.

This demonstrates that the demapper is applicable to practical systems with or without pilots, using sample means and covariances in the former and blind GMM-based estimation in the latter case.

All simulations were performed in MATLAB (The Mathworks, Natick, MA). Information bits were generated with equal probability with a pseudorandom number generator before coding and modulation. We implemented the coding strategy defined by the Consultative Committee for Space Data Systems (CCSDS), consisting of serial concatenated convolutional codes (SCCC) at the transmitter and a turbo decoder at the receiver [31]. Various puncturing strategies are used to define different code rates. A symbol rate of 250 Msym/s was used for all simulations. After modulation, the signal was synthesized with four times oversampling and a root-raised-cosine transmit filter with a rolloff of 0.35. Complex Gaussian noise was added to simulate thermal noise before the signal was processed by the receive modules.

We compare two demappers: (i) a standard demapper, which assumes circularly symmetric noise with a constant covariance across constellation points and (ii) the proposed covariance-based demapper. Both demappers account for the mean of the symbol-dependent noise distribution using estimates of the cluster centroids. However, the proposed demapper also utilizes the covariance of each symbol cluster. Since the proposed demapper is a generalization of the standard APSK demapping algorithm, this comparison allows us to quantify the improvement obtained by using the covariance information. The means and covariances are estimated using 120 known pilot symbols per constellation point. Using CCSDS frames, this represents a 6% reduction in data rate for a 64 order modulation. Simulations below processed 50 frames (∼10⁷ bits) through the full system model, including distortions, using different demapping algorithms.

Details specific to the two distortion scenarios are elaborated below.

Scenario 1: nonlinear amplifier

A memoryless model was used to simulate a nonlinear power amplifier [22], where the model parameters were fit to data measured from an X-band TWTA. The level of distortion is controlled by the amplifier backoff, which is the average power of the input signal relative to the amplifier saturation point. A complex-valued polynomial with order 9 was used for predistortion, extracted using an indirect learning strategy with least squares [32]. We consider 64-APSK and 128-APSK constellations. Since the CCSDS standard is limited to modulation orders up to 64, we extended the CCSDS framework to include a 128-APSK constellation from the DVB-S2X standard [33]. The symbols are distributed in four or five constellation rings. There is no phase noise in this scenario.

Scenario 2: phase noise

A linear transmission channel is used, but phase noise is considered at the receiver. Phase noise was simulated at baseband using the phase noise mask from the receiver link in the DVB guidelines for the professional service scenario [34]. Different levels of phase noise were simulated by shifting the entire mask to achieve a given level (in dBc/Hz) at a frequency offset of 100 Hz from the carrier frequency [24]. Rectangular 64-QAM and 128-QAM schemes were simulated.

4.1 Maximum achievable rate

Independent of the demapping algorithm, the maximum achievable rate for the reduced channel model in Section 2.3 is given by the mutual information between the received and transmitted symbols. For a uniform input distribution, the mutual information is

$$\begin{array}{*{20}l} I(Y;X) = \frac{1}{M}\sum_{x\in\mathcal S}\int p(y|x) \log\frac{p(y|x)}{p(y)}dy. \end{array} $$

(16)

The mutual information could be computed efficiently using an approximation to the entropy of a Gaussian mixture, e.g., [35], although a loss in accuracy is expected for high modulation orders. Instead, we calculate the maximum rate by numerical integration of (16) over \(\mathbb R^{2}\) using a grid of 1000×1000 points. The integrand in (16) can be easily evaluated since the PDFs are known from (10). This maximum rate is independent of the demapper and serves as a benchmark for different demapping algorithms.

The achievable rate using a specific demapper can be computed from the mutual information between the extrinsic L values in (9) and the transmitted bits for the equivalent bit channels depicted in Fig. 3. In this case, the rate was calculated using the area property of the extrinsic information transfer (EXIT) chart [36]. Specifically, a binary erasure channel was simulated as the extrinsic channel to provide varying amounts of a priori information to the demapper. For a given SNR and distortion level, the reduced channel model was simulated with ∼10⁷ bits to compute the demapper’s EXIT function using the extrinsic L values in (9). The function was evaluated at 100 levels of a priori information between 0 and 1. The achievable rate for the demapper was computed with the mean (area) of the resulting EXIT function [36].

Figure 5 illustrates the maximum achievable rate for the proposed covariance-based demapper and the standard demapper. The nonlinear channel in scenario 1 was simulated with an amplifier backoff of 3 dB for two modulation schemes at various SNR levels. The achievable rate for the proposed demapper matches the theoretical maximum derived from the symbol-level channel model, demonstrating the optimality of the demapper. In contrast, the achievable rate of the standard demapper is reduced when the SNR is high. At high SNR, the symbol scattering is directional and mainly due the nonlinear distortion so knowledge of the covariance matrices is crucial to achieve maximum performance. It is also evident that there is a larger gain for the higher-order 128-APSK modulation. Note that the maximum rate for both modulation schemes is below the corresponding number of bits, even for infinite SNR, due to the fixed level of distortion.

To examine the effect of the distortion level on the demapper performance, we calculated the mutual information for SNR values between 5–30 dB and distortion levels defined by input powers between -10 and -2 dB. We define the gain as the difference in the achievable rate between the proposed demapper and the standard demapper. Figure 6 plots the achievable rate gain for a 128-APSK modulation scheme for different pairs of SNR values and distortion levels. We see that there is a gain in regions with high distortion and high SNR. This is expected since the symbol distributions are less isotropic, captured by the covariance of each cluster. We refer to this region as “distortion dominated.” Conversely, regions with low SNR or low distortion have relatively circular and constant symbol covariances, so the gain is minimal. We refer to this region as (thermal) “noise dominated.” The maximum gain is 0.16 bits per channel use, obtained when there is severe distortion and minimal noise.

Very similar plots for the case of phase noise in scenario 2 were generated (not shown for space limitations), where we consider the AWGN as thermal noise and phase noise as distortion. Analogous to the nonlinear channel, a large amount of phase noise creates very asymmetric covariances, which become the dominate source of error for sufficiently high SNR.

We remark that this analysis is ideal in the sense that the reduced channel model perfectly represents the received symbols. In a practical system, the bivariate Gaussian is an approximation of the effects of several communication components. We examine this approximation in more detail in the Appendix. In the next sections, we demonstrate that the superior performance of the proposed demapper is maintained for practical systems with non-Gaussian PDFs.

4.2 Bit error rates

We have shown a substantial gain in the maximum achievable rate when operating in a distortion dominated regime, where the symbol noise varies across the constellation. In this section, we examine how this gain translates to system performance by computing the bit error rate (BER) for different coding schemes.

As an initial experiment, we compute the BER using the proposed and standard demappers to make hard decisions about the transmitted bits. The BER in Fig. 7 is for an uncoded system under scenario 1 with a nonlinear amplifier at input power of -5 and -3 dB for two different constellations. The proposed demapper incorporating symbol covariances lowers the uncoded BER in distortion dominated regions, when the SNR is high. The BER increases for a denser constellation (e.g., 128-APSK) or a higher level of distortion (input power of -3 dB). It is also clear that an “error floor” exists for each scheme, due to the fixed level of distortion.

The uncoded simulations reenforce the common theme that demapping using the symbol covariance is advantageous for high SNR when the system is dominated by distortion. However, when error-correction is used, the threshold of the code also plays a significant role. Figure 8 illustrates the BER of a coded system for the same setup as Fig. 7. Notice that there is minimal improvement in the BER for 64-ASPK modulation with coding, despite a large reduction in BER for the uncoded case. To explain this, imagine decoding with a hard decision demapper. Although the proposed demapper reduces the errors at high SNR, the code can already correct for this number of errors (in the order of 10⁻²), so further reduction is unnecessary. In other words, the information gain from the proposed demapper occurs at SNR values above the threshold of the code. Conversely, higher order schemes, such as 128-APSK, inherently require a higher SNR to achieve the same level of error, so the proposed demapper becomes beneficial. Thus, the performance of the coded system will be substantially improved for 128-APSK modulation with covariance-based demapping, as shown in Fig. 8.

The performance of the entire system with the proposed demapper depends on the modulation scheme, level of distortion, and the SNR threshold of the code. Figure 9 presents the required SNR to achieve a BER of 10⁻⁶ when error-correction is used for the nonlinear amplifier in scenario 1. As the nonlinear distortion increases, the covariance-based demapper provides more accurate L values, which improves the decoder performance so it can operate at reduced SNR compared to the traditional demapper. The reduction in SNR is more pronounced as the spectral efficiency increases, e.g., by increasing the code rate or modulation order. This highlights the interplay between the quality of the demapper output and the errors after decoding. It is also worth noting that the required SNR increases smoothly with the distortion level, which acts to balance the increased uncertainty due to nonlinear distortion by reducing the uncertainty due to thermal noise.

Figure 10 plots the required SNR to achieve a BER of 10⁻⁶ for a coded system with phase noise, described in scenario 2. When we consider the phase noise as a distortion, separate from thermal noise, all the trends above are maintained. This demonstrates the utility of the proposed demapper for any system that can be modeled with data-dependent noise.

4.3 Effect of the number of pilot symbols

The robustness of the demapper to imperfect mean and covariance estimates is investigated by varying the number of pilot symbols. We calculated the SNR required to achieve a BER of 10⁻⁶ with a varying number of pilots for the coded system with the nonlinear amplifier in scenario 1. Figure 11 demonstrates that increasing the number of pilots leads to improved performance until the required SNR reaches a plateau, at which point further improvements in the mean and covariance estimates do not improve the BER performance. For 128-APSK modulation, more pilots are required to reach optimal performance, due to the increased number of parameters to be estimated. Apart for a very low number of pilots, the performance is relatively stable and a graceful performance degradation is observed for decreasing number of pilots, demonstrating a reasonable robustness to inaccurate covariance estimation.

4.4 Iterative demapping and decoding

In this section, we demonstrate that the proposed demapper can also benefit applications with iterative demapping and decoding [7]. Iterative demapping methods are strongly dependent on the chosen bit labeling, e.g., a Gray mapping exhibits little performance gain with joint demapping and decoding [37]. To provide a suitable example, we consider a coded system with phase noise described in scenario 2, with a 128-QAM constellation and a randomly generated bit labeling. With this bit labeling, the demapper will benefit from a priori information passed back from the decoder each iteration. We use the EXIT chart [36] to visualize iterations between the demapper and decoder in Fig. 12. The simulation was conducted close to the threshold of the turbo code, with an E _s /N₀ of 25 dB, and phase noise of −43 dBc/Hz at a frequency offset of 100 Hz. The standard demapper generates less accurate L values, and consequently, the mutual information captured in the EXIT function is lower. This results in the iterative receiver terminating when the demapper’s EXIT function intersects with that of the decoder (Fig. 12a). The proposed demapper, on the other hand, has a higher EXIT function, which allows for successful decoding using the a priori information generated from previous iterations (Fig. 12b). Successful decoding is achieved after two iterations. It is interesting to note that the turbo code is suboptimal in this example and below the maximum achievable rate, which could be obtained by designing a code/decoder to match the EXIT function of the demapper [38]. Nonetheless, this example demonstrates a clear advantage of the proposed demapper for iterative demapping and decoding. As with all joint demapping approaches, however, the performance gain will depend on the bit labeling and the particular coding scheme.

The proposed demapper works on the assumption that the conditional PDF for each constellation point is adequately represented by a bivariate Gaussian of the in-phase and quadrature components. In this section, we explore this assumption in more detail. We approach this investigation from the perspective of model selection [39], which addresses the question of whether a bivariate Gaussian is a “better” model than a circularly symmetric Gaussian. We use the Akaike information criterion (AIC) to compare the two models [39]. The AIC provides a trade-off between the goodness of fit of a model and the number of parameters in the model.

Figure 13 depicts the true PDF using a histogram of 10⁸ simulated symbols partitioned into 10⁴ bins. These symbols are used to fit a bivariate Gaussian model and a circularly symmetric Gaussian model, and the AIC for each model is displayed. The bivariate Gaussian model provides a better fit (lower AIC) than the circularly symmetric version even after penalizing the extra degrees of freedom. The bivariate Gaussian model had a lower AIC for all points in the constellation although the difference between the models is smaller for points in the inner rings since these points have less distortion. For example, the AIC for a constellation point in the outer ring is − 2.61×10⁷ for the bivariate Gaussian compared to − 2.26×10⁷ for the circularly symmetric Gaussian.

It is important to consider that our ultimate goal of modeling the PDF is to efficiently compute likelihoods to provide to the decoder. In addition to fitting the true PDF better, the results in this manuscript demonstrate that L values from the bivariate Gaussian model enable more accurate decoding than the standard circularly symmetric Gaussian.