 Research
 Open access
 Published:
Joint highdimensional soft bit estimation and quantization using deep learning
EURASIP Journal on Wireless Communications and Networking volumeÂ 2022, ArticleÂ number:Â 51 (2022)
Abstract
Forward error correction using soft probability estimates is a central component in modern digital communication receivers and impacts endtoend system performance. In this work, we introduce EQNet: a deep learning approach for joint soft bit estimation (E) and quantization (Q) in highdimensional multipleinput multipleoutput (MIMO) systems. We propose a twostage algorithm that uses soft bit quantization as pretraining for estimation and is motivated by a theoretical analysis of soft bit representation sizes in MIMO channels. Our experiments demonstrate that a single deep learning model achieves competitive results on both tasks when compared to previous methods, with gains in quantization efficiency as high as \(20\%\) and reduced estimation latency by at least \(21\%\) compared to other deep learning approaches that achieve the same endtoend performance. We also demonstrate that the quantization approach is feasible in singleuser MIMO scenarios of up to \(64 \times 64\) and can be used with different soft bit estimation algorithms than the ones during training. We investigate the robustness of the proposed approach and demonstrate that the model is robust to distributional shifts when used for soft bit quantization and is competitive with stateoftheart deep learning approaches when faced with channel estimation errors in soft bit estimation.
1 Introduction
Digital MIMO systems operate by transmitting a vector of discrete symbols across a singlechannel use and are a major component of current 5G and envisioned 6G communication systems [1]. Given the prevalent use of errorcorrecting codes, such as lowdensity parity check (LDPC) codes, two core tasks for modems in these systems consist in soft bit estimation (also commonly referred to as softoutput MIMO detection, or soft detection) and soft bit quantization. Using softâ€”instead of hardâ€”bit estimates for decoding errorcorrecting codes is known to provide an order of magnitude order in a reduction in endtoend error rates [2], and algorithms that operate on soft bits must ensure that they are close as possible to the optimal solution. In this work, our goal is to develop learningaided solutions for two different tasks, handled under the same framework: soft bit estimation and quantization. Starting with soft bit quantization using deep learning, we develop a framework that can address the two tasks simultaneously by reusing the information learned during quantization for better supervised training of a soft bit estimation algorithm.
Soft bit quantization is required when storing a large number of soft bits for a potentially long duration of time, such as in hybrid automatic repeat request (HARQ) schemes [3] or when the soft bits themselves need to be forwarded across wireless environments, such as in relay [4] or fronthaul [5] systems. For example, in distributed communications systems [6], lowresolution soft bit quantization is required whenever relaying or feedback is involved, because system capacity represents a bottleneck [7], and soft bits must be transmitted without errors, using a low communication overhead. Quantization is also required in systems that use the HARQ protocol, where it is beneficial for the receiver to store soft bit values from a failed transmission, and use a soft combining scheme [8, 9] to boost performance, making storage a potential system bottleneck.
Soft bit estimation in MIMO systems is a challenging practical problem due to the exponential complexity of the optimal solution [10] and stringent latency requirements in 5Gandbeyond communication systems [11]. For example, given a singleuser MIMO (SUMIMO) 5G system operating at sub6 GHz, the base station could be faced with estimating soft bits from as many as thousands of channel uses per data frame [12], each of these requiring an expensive algorithm. Nearoptimal estimation algorithms are a central part of endtoend performance in coded systems (e.g., that use lowdensity paritycheck (LDPC) codes) [13], with solutions that use machine learning to obtain highperformance and low latency being an active area of research. Our work builds upon this line of research and introduces a deep learningaided algorithm for nearoptimal soft bit estimation in moderately sized SUMIMO systems.
Deep learning methods have emerged as promising candidates for aiding or completely replacing signal processing blocks in MIMO communication systems [14, 15]. In this work, we focus on the case where deep learning is used on specific functional tasks in endtoend communication systems. These approaches are attractive, as they are modular and compatible with existing communication protocols. While previous solutions have been developed for both tasks, there is currently no solution that addresses soft bit quantization in large MIMO systems and that tackles soft bit estimation and quantization in the same framework. Furthermore, the robustness of deep learningbased methods is still an open problem in the broader machine learning field [16, 17] and has been recognized as an issue in digital communications as well [18, 19]. In this paper, we address this research gap and introduce EQNet, a datadriven architecture that aims to solve the challenges of lowlatency quantization and estimation, while still retaining the endtoend system performance of classical approaches under distributional shifts.
1.1 Related work
1.1.1 Soft bit quantization
Generally, there are two approaches developed in the prior work regarding soft bit quantization: scalar or vector quantization. In the scalar approach, each soft bit is separately quantized, independent of the others, and without considering structure at a channel use level. The work in [20] introduces an informationtheoretic optimal databased approach for quantizing soft bits from arbitrary distributions, such as corresponding to the same bit position in graycoded digital quadrature amplitude modulation (QAM). This approach also has the advantage that it does not make any assumptions about the underlying channel model and an algorithm is given for estimating optimal scalar quantization levels in arbitrary channels. The work in [21] proposes a solution for soft bit quantization in relay systems based on maximizing the mutual information between two transmitters and one receiver. While both of these approaches are competitive, they do not take advantage of redundancy between soft bit estimates derived from the same channel use.
A promising research avenue is to consider vector quantization techniques for soft bits in MIMO channels. The work in [22] extends the datadriven approach in [23] and proposes a vector extension to a maximum mutual information quantizer, but loses theoretical optimality guarantees. The method in [24] introduces a deep learning approach that leverages the redundancy between soft bit values corresponding to a single channel use and achieves excellent quantization results for SISO channels. However, there remains the issue of developing a vector quantization method for arbitrary MIMO scenarios, which is a major distinction between this paper and all other prior work in terms of quantization methods. Finally, prior work does not discuss or exploit the learned representations from the quantization task when maximum likelihood (ML) soft bits are used for training, which is also a component of EQNet.
1.1.2 Soft bit estimation
There are a broad number of classical (nonlearningbased) signal processing algorithms that have been developed for soft bit estimation, given that channel decoding using soft bit inputs is ubiquitous in practice [2]. A modern survey of MIMO soft bit estimation (also called softoutput detection) is presented in [13]. The VBLAST algorithm [25] uses the idea of sequential estimation, with subsequent work leading to zeroforcing with successive interference cancelation (ZFSIC) [26] as an algorithm for efficient detection, and the minimum mean square error with successive interference cancelation (MMSESIC) [27] to further improve performance. In these methods, the system is reduced to a (regularized) upper triangular form and data symbols are detected in a fixed order. Once a symbol is detected, it is assumed to be correct and subtracted from the remaining data streams. This leads to lowcomplexity, but also lowperformance methods, that may exhibit error floors. Sphere decoding [28, 29] formulates the detection problem as a tree search algorithm and performs a greedy search. In the softoutput version [29], multiple candidate solutions are used to estimate loglikelihoods. A drawback of this class of algorithms is the need for specialized hardware to accommodate latency budgets in 5Gandbeyond scenarios [30]. To address this, recent work has combined sphere decoding with deep learning for search radius and branch prediction [31].
More recently, there has been work in modelbased approaches for MIMO detection, where differentiable optimization steps are interleaved with deep learning models. The work in [32] proposes OAMPNet2 as a databased extension to the orthogonal approximate message passing (OAMP) detection algorithm [33], where step sizes are treated as learnable parameters. This method has the advantage of a very small number of learnable parameters, but has a relatively large endtoend latency because of the matrix operations involved during inference. The work in [34] takes a similar approach, but replaces the fixed computations of the OAMP algorithm with fully learnable transforms (i.e., layers of a deep neural network), resulting in competitive results for MIMO soft bit estimation and an architecture that can be scaled to highdimensional scenarios. Finally, the work in [35] trains a twolayer deep neural network using a supervised loss directly on the soft bits, in singleinput singleoutput (SISO) channels, leveraging that, in this case, a closedform linear approximation of the soft bits exists. This is not the case for MIMO detection, where singlestage supervised training may encounter convergence issues, as outlined in Sect.Â 3.2.
1.2 Contributions
Our contributions in this work are the following:

1
We prove lower and upper representation size bounds (TheoremsÂ 1 andÂ 2) for the ML and MMSESIC soft bit estimates in arbitrary MIMO channels. The upper bound for ML is proved constructively through a construction that holds for any channel, while the lower bound for ML comes from the class of diagonalized MIMO channels. In practice, we verify that the lower bound for ML can be achieved for arbitrary channels and learned using deep neural networks, which becomes a design criterion in the proposed approach. For the MMSESIC algorithm, we prove that the lower bound is achievable in arbitrary channels.

2
We introduce a methodology for the supervised training of a joint soft bit estimation (E) and quantization (Q) algorithm in MIMO scenarios, termed EQNet. The proposed approach involves a twostage training procedure: The first stage trains a deep autoencoder for quantization, while the second stage trains an estimation encoder, reusing the quantization decoder. We experimentally verify that the twostage training algorithm improves convergence and has better endtoend coded system performance than the singlestage supervised training baseline when using small depth (shallow) networks.

3
We experimentally evaluate endtoend block error rate performance, inference latency and the robustness of EQNet in soft bit quantization and estimation. We demonstrate competitive results in both tasks simultaneously, under realistic system simulations. We compare our method with classical signal processing algorithms, as well as deep learningbased approaches. We obtain gains of up to \(20\%\) in numerical quantization efficiency and estimation gains of up to 1 dB in endtoend system performance in coded MIMO orthogonal frequencydivision multiplexing (OFDM) scenarios and demonstrate latency improvements against other deep learning approaches. Finally, we demonstrate that EQNet is robust to inaccurate channel state information (CSI), different trainâ€“test distributions of the channel conditions and distributions of the input in largescale MIMO scenarios (up to \(64 \times 64\) SUMIMO), when evaluated on the quantization task, and is competitive with other deep learning methods on the robust estimation task in smallersized MIMO scenarios of up \(4 \times 4\).
2 Methods
2.1 System model
We assume a narrowband, instantaneous, digital, singlecell uplink MIMO communication model. This encompasses practical scenarios, such as singlecarrier MIMO communication, or an individual subcarrier of a MIMOOFDM transmission, and is flexible enough to model various distributions of the MIMO channel matrix. The communication model is given by [36, Eq.Â 7.55]:
where \({\mathbf {x}} \in {\mathbb {C}}^{N_\mathrm {t}}\) is a vector of transmitted symbols drawn from a finite, discrete constellation \({\mathcal {C}}\), and \({\mathbf {y}} \in {\mathbb {C}}^{N_\mathrm {r}}\) is a vector of received symbols. \(N_\mathrm {t}\) and \(N_\mathrm {r}\) represent the number of transmitted and received data streams, respectively, and \({\mathbf {n}} \in {\mathbb {C}}^{N_\mathrm {r}}\) is an i.i.d. complex Gaussian noise vector with covariance matrix \(\sigma _n^2 {\mathbf {I}}\). For the remainder of this work, we deal with spatial multiplexing scenarios, where \(N_\mathrm {r} \ge N_\mathrm {t}\) and the transmitted data streams are assumed to be independent, but an application of our method to transmit diversity is possible when considering the effective digital matrix as the product between the precoding matrix and the channel matrix. \({\mathbf {H}}\) is the digital channel matrix characterizing the baseband channel effects between the transmitter and the receiver and includes the effects of precoding and beamforming. In practice, \({\mathcal {C}}\) is the set of symbols in a QAM constellation with \(2^K\) elements, where K is the modulation order and represents the number of information bits transmitted in each of the \(N_\mathrm {t}\) data streams.
The kth loglikelihood ratio of the ith transmitted symbol is defined as [20]:
For the remainder of this work, we use the notion of soft bits, which are closely related to the loglikelihood ratios through an invertible transform [20] and are used in practical decoders for errorcorrecting codes due to operations being simpler to carry out in the hyperbolic tangent domain [37]:
In scenarios where the noise is nonzero, soft bits are constrained to the interval \((1, 1)\), with large magnitude values tending to nearcertain soft bits. This is beneficial for datadriven methods, because a limited dynamic range is also helpful in the stability of deep learning methods, where unbounded inputs can lead to exploding gradients [38]. By replacing the definition of the loglikelihood ratio from (2), we obtain that the soft bits are defined as
We organize the soft bits corresponding to a single MIMO channel use in the soft bit matrix \(\mathbf {\Lambda } \in {\mathbb {R}}^{N_\mathrm {t} \times K}\).
2.1.1 ML soft bits
The likelihoods \(P({\mathbf {y}}b_{i,k})\) in (4) can be expanded by the total probability law. For the soft bit corresponding to the ith transmitted stream and the kth bit position, let \({\bar{b}}_{i, k} = \{ b_{j, l}  (j, l) \ne (i, k) \}\) be the set of all remaining bit positions. The conditional probability of the (i,Â k)th bit position is given by
where the sum is taken over all possible values of the vector \({\bar{b}}_{i, k}\), i.e., all bits except the position of interest are marginalized. Because the terms in (4) involve conditioning the bit \(b_{i, k}\) on a specific outcome and together with the marginalization in (5), it follows that, under the i.i.d. Gaussian noise assumption, each probability can be expanded as (up to a normalization factor):
where we use \({\mathbf {x}}\) to denote each symbol corresponding to a specific combination of \({\bar{b}}_{i, k}, b_{i, k}\) obtained by using the given QAM constellation. Each exponential term in the sum in (6) corresponds to the conditional probability of the observed symbols given a specific transmitted vector and comes from the Gaussian noise assumption. Finally, the optimal maximum likelihood (ML) estimator for the soft bits in channels affected by i.i.d. Gaussian noise can be expressed as
2.1.2 MMSESIC soft bits
The exact evaluation of (7) is computationally prohibitive even in moderately sized systems. We consider the MMSESIC algorithm based on the QR decomposition in [39], with a random order, which sequentially estimates the data streams and their corresponding soft bits. The channel matrix \({\mathbf {H}}\) is first augmented by adding the extra rows:
where \({\mathbf {I}}_{N_\mathrm {t}}\) is a square identity matrix of size \(N_\mathrm {t} \times N_\mathrm {t}\). The QR decomposition of the augmented matrix is carried out as \(\underline{{\mathbf {H}}} = \underline{{\mathbf {Q}}} \underline{{\mathbf {R}}}\), where \(\underline{{\mathbf {Q}}}\) is an unitary matrix and \(\underline{{\mathbf {R}}}\) is upper triangular. The first \(N_\mathrm {r}\) rows of \(\underline{{\mathbf {Q}}}\) are denoted as
The matrix \({\mathbf {Q}}\) is used to equalize the received symbols \({\mathbf {y}}\) as
where we used that \(\underline{{\mathbf {Q}}}\) is unitary, and \(\tilde{{\mathbf {n}}}\) represents the filtered Gaussian noise, with zero mean and standard deviation of the ith entry given by the norm of the ith column of \({\mathbf {Q}}\).
As the equivalent MIMO system in (10) is now upper triangular, sequential detection of the symbols and soft bits starts from the \(N_\mathrm {t}\)th data stream, with the scalar system equation:
and the corresponding soft bits are obtained by marginalization only over the bits of the ith symbol, via (5). This populates the \(N_\mathrm {t}\)th row of the matrix \(\mathbf {\Lambda }^{(\mathrm {MMSESIC})}\), and detection continues sequentially, by subtracting all the previously estimated hard symbols from the estimated vector \({\mathbf {x}}^{(\mathrm {MMSESIC})}\). The generic scalar system equation for the ith stream is given by
where \({\hat{n}}_i\), in general, includes the remaining interference terms caused by incorrect detection of the previous streams. Assuming correct detection, we have that \({\hat{n}}_i = {\tilde{n}}_i\), and (5) can be used to estimate the soft bits of the current stream. Finally, the ZFSIC algorithm has a similar flow, with the difference that \(\underline{{\mathbf {H}}} = {\mathbf {H}}\), i.e., no matrix augmentation is performed before the QR decomposition.
2.1.3 LMMSE soft bits
In extremely large MIMO systems, the MMSESIC estimate may not be available even for quantization, because of the sequential nature of the algorithm in (12). The linear minimum mean squared error (LMMSE) algorithm uses the matrix \({\mathbf {G}} = \left( {\mathbf {H}}^\mathrm {H} {\mathbf {H}} + \sigma _n^2 {\mathbf {I}}_{N_\mathrm {t}} \right) ^{1} {\mathbf {H}}^\mathrm {H}\) to equalize the received symbol vector as
Assuming that the offdiagonal terms of \({\mathbf {G}} {\mathbf {H}}\) are negligible, the MIMO system can be decomposed in \(N_\mathrm {t}\) parallel SISO systems, where the ith system equation is
where \(g_i\) is the ith diagonal element of \({\mathbf {G}} {\mathbf {H}}\), and \({\tilde{n}}_i\) is Gaussian noise with zero mean and standard deviation given by the norm of the ith row of \({\mathbf {G}}\). Estimating the soft bit matrix \(\mathbf {\Lambda }^{(\mathrm {LMMSE})}\) is then done separately for each row, using the marginalization only over the bits of the ith symbol, via (5).
2.2 Soft bit quantization
The goal of soft bit quantization is to design the following pair of functions, given the full precision ML soft bit estimate \(\mathbf {\Lambda }^{(\text {ML})}\):

An encoder \(f_\text {Q}\) that compresses the matrix \(\mathbf {\Lambda }^{(\text {ML})}\) to a lower or equal dimension representation \({\mathbf {z}}\) and numerically quantizes \({\mathbf {z}}\) to a bit stream of finite length. In general, this function is lossy, and the bit stream encodes an entry in a quantization codebook that is fixed and is known to both the transmitter and receiver (e.g., similar to how beamforming codebooks are standardized in 5G systems [40]).

A decoder \(g_\text {Q}\) that recovers \(\mathbf {\Lambda }^{(\text {ML})}\) as accurately as possible from a given bit stream, with respect to a predefined error function.
2.3 Soft bit estimation
The goal of soft bit estimation is to design a function \(f_\text {E}\) that takes as input the received symbols \({\mathbf {y}}\), channel \({\mathbf {H}}\), the symbol constellation \({\mathcal {C}}\) and the noise standard deviation \(\sigma _n\) and outputs an estimate of the soft bit matrix \(\tilde{\mathbf {\Lambda }}\). While (7) is the optimal estimator for the soft bits when the noise is Gaussian and independentâ€”a common situation in practice [40]â€”the two distinct sums in (7) each involve a number of \(2^{N_\mathrm {t}K  1}\) terms. This leads to a prohibitive computational complexity even for moderate values of \(N_\mathrm {t}\) and K, and algorithms that approximate the solution \(\mathbf {\Lambda }^{(\text {ML})}\) by reducing the amount of performed computations are the main goal in efficient soft bit estimation.
2.4 Distortion metric for soft bits
While the tasks of soft bit quantization and estimation are different in terms of what inputs are available during deployment (inference), they share a common characteristic: For both tasks, the output has to approximate the desired soft bits as closely as possible. We use the following weighted mean squared error loss to quantify this deviation, which represents an extension of the loss in [24] to MIMO scenarios:
where \(\tilde{\mathbf {\Lambda }}\) is the output of the estimator. \(\mathbf {\Lambda }^{\text {(ML)}}\) is the desired output (in this case, the ML estimate), and the term \(\epsilon\) acts as a stabilization constant, since some bits can have an arbitrarily small absolute value, and would thus need to be reconstructed or estimated to an arbitrary precision. Since the impact of the soft bits with very low magnitude values is negligible in decoding algorithms [37], we use a value of \(\epsilon = 0.001\) to bound the loss in (15), chosen as a tradeoff between stability and accuracy.
2.5 Theoretical results
In this section, we prove lower and upper bounds on the representation size ratio of the soft bit matrices of the ML and MMSESIC algorithms. We define the representation size ratio of a surjective mapping \(f:{\mathbb {R}}^n \rightarrow {\mathbb {R}}^m\) as the ratio m/n, where any function with a ratio strictly higher than one can be used to represent the data (codomain of f) using a lowerdimensional feature space, thus achieving compression. Note that the theory in this section does not cover numerical quantization and requires infinite numerical precision for the domain and codomain of f. In Sects.Â 3.4 andÂ 3.6, numerical quantization is applied to the feature representation, and its impact is evaluated.
The two bounds derived in this section are helpful in choosing the feature dimension in a deep autoencoder architecture, where we would ideally like to always use as few features as possible and further quantize each of them using a fixed bit budget. For the ML soft bits, we prove that the lower bound is achievable for arbitrary channels by constructing a feature representation that achieves it. We prove the upper bound through a special class of diagonalized channels, and in practice we attempt, and are successful, in learning a deep autoencoder that achieves the upper bound for arbitrary channels in Sect.Â 3.1. For the MMSESIC soft bits, we prove that the upper bound can be achieved constructively, for arbitrary channels.
2.5.1 Lower representation size ratio bound for the ML estimate in arbitrary channels
We directly work with (1) and make no assumptions on \({\mathbf {x}}\), other than it being sampled from the constellation \({\mathcal {C}}\). We prove the following lower bound for the representation size ratio.
Theorem 1
Let \({\mathbf {y}} = {\mathbf {H}}{\mathbf {x}} + {\mathbf {n}}\) be the noisy received symbols in arbitrary channels. Then, there exists a surjective function \(f:{\mathbb {R}}^{N_\mathrm {t}(N_\mathrm {t}+2)} \rightarrow {\mathbb {R}}^{K\times N_\mathrm {t}}\) such that \(f({\mathbf {z}}) = \mathbf {\Lambda }^{\text {(ML)}}\) and a representation size ratio of \(R_{\mathrm {low, ML}} = K/(N_\mathrm {t}+2)\) is achieved.
The proof is provided in the Appendix and relies on the QR decomposition of \({\mathbf {H}}\). After equalizing the \({\mathbf {Q}}\) term, the statistics of the noise do not change, and the system is now represented by an upper triangular equation. The representation of the upper triangular channel itself takes \({\mathcal {O}}(N_\mathrm {t}^2)\) degrees of freedom, because we do not make any structural assumptions about the matrix \({\mathbf {H}}\), and storing the raw channel matrix and the received symbols is sufficient for exact soft bit recovery using (4).
2.5.2 Upper representation size ratio bound for the ML estimate in diagonalized channels
We consider a version of (1) where the transmitter has knowledge of the channel matrix \({\mathbf {H}}\). Furthermore, we assume that the transmitted symbols take the form \({\mathbf {s}} = {\mathbf {V}} {\mathbf {x}}\), where \({\mathbf {V}}\) represents the matrix of right singular vectors of \({\mathbf {H}}\) and \({\mathbf {x}}\) is drawn from \({\mathcal {C}}\). This type of linear precoding is shown to be optimal when using waterfilling [41] under transmit power constraints. Then, the following theorem holds.
Theorem 2
Let \({\mathbf {y}} = {\mathbf {H}}{\mathbf {s}} + {\mathbf {n}}\) be the noisy received symbols, where \({\mathbf {s}} = {\mathbf {V}} {\mathbf {x}}\) are the transmitted symbols. Then, there exists a surjective function \(f:{\mathbb {R}}^{3N_\mathrm {t}} \rightarrow {\mathbb {R}}^{K\times N_\mathrm {t}}\) such that \(f({\mathbf {z}}) = \mathbf {\Lambda }^{\text {(ML)}}\) and a representation size ratio of \(R_{\mathrm {up, ML}} = K/3\) is achieved.
The proof for the upper bound is provided in the Appendix. Intuitively, this is done by recognizing that the channel becomes diagonal after performing leftside multiplication with the conjugate transpose of the matrix of left singular vectors \({\mathbf {U}}\). As the MIMO channel can be separated in a set of \(N_\mathrm {t}\) virtual channels, the soft bits corresponding to each virtual channel can be exactly represented using a threedimensional vector, regardless of K [24].
2.5.3 Upper representation size ratio bound and achievability for the MMSESIC estimate in arbitrary channels
Theorem 3
Let \({\mathbf {y}} = {\mathbf {H}}{\mathbf {x}} + {\mathbf {n}}\) be the noisy received symbols in arbitrary channels. Then, there exists a surjective function \(f:{\mathbb {R}}^{3N_\mathrm {t}} \rightarrow {\mathbb {R}}^{K\times N_\mathrm {t}}\) such that \(f({\mathbf {z}}) = \mathbf {\Lambda }^{\text {(MMSESIC)}}\) and a representation size ratio of \(R_{\mathrm {up, MMSESIC}} = K/3\) is achieved.
The proof for this upper bound is provided in the Appendix. Intuitively, once the system is represented in its upper triangular form, we inductively prove that representing each row of the soft bit matrix can be done using a threedimensional vector, regardless of K.
2.5.4 Design implications of the theoretical results
In practice, the lower bound of \(K / (N_\mathrm {t} + 2)\) is too weak to use for compressing the ML soft bits, as it is possible that \(N_\mathrm {t} \ge K  2\). The upper bound, however, is invariant with respect to the number of transmitted data streams \(N_\mathrm {t}\) and always compresses for \(K \ge 4\) (i.e., a modulation of 16QAM or higher order). The two bounds coincide and a feature representation that achieves them can be derived in closed form for the particular case of \(N_\mathrm {t} = 1\) (SISO channels), which recovers the previous work in [24]. However, achieving the upper bound is nontrivial in arbitrary channels in the case of ML soft bits. A key component of EQNet is to assume that the upper bound for the ML soft bits can be achieved in arbitraryâ€”and not only in diagonalizedâ€”MIMO channels. This is verified and discussed in Sect.Â 3.1. While we do not derive such a representation explicitly for the ML case, this motivates us to attempt and learn it by using the representational power of deep neural networks. In particular, this makes a deep autoencoder a suitable choice, where we set the dimension of the bottleneck feature representation to \(3N_\mathrm {t}\), regardless of the algorithm used to estimate the soft bits.
2.6 EQNet: joint estimation and quantization
EQNet is a supervised method that uses a deep autoencoder and leverages compression as a supervised pretraining task for learning a soft bit estimator: When training the estimator, we do not learn a direct mapping between received symbols and the soft bit matrix, but rather split the learning in two separate stages:

1
In the first stage, we train a pair of quantization encoder and decoder functions by compressing to a feature representation of size \(3N_\mathrm {t}\) (the upper bound for both ML and MMSESIC in TheoremsÂ 2 andÂ 3, respectively).

2
In the second stage, we train an additional estimation encoder that takes the received symbols and CSI as input, and is trained to predict the features of the soft bits, obtained from the first stage. In this stage, the features are frozen, and only this new encoder is learned.
A highlevel functional diagram of EQNet is shown in Fig.Â 1. The ablation experiments in Sect.Â 3.2 show that the twostage procedureâ€”where we first train the pair \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\), and then reuse \(f_\mathrm {Q}\) to train \(f_\mathrm {E}\)â€”is an essential step when training a model with limited depth and width, and that singlestage endtoend training falls into unwanted local minima.
2.6.1 Stage 1: Compression and quantization
In the first stage, we train an autoencoder consisting of a quantization encoder \(f_\text {Q}\) and decoder \(g_\text {Q}\). The input to the quantization encoder is the ML estimate of the soft bits \(\mathbf {\Lambda }^\text {(ML)}\) or, when infeasible due to the large complexity, the MMSESIC or LMMSE estimates of the soft bits. The parameters of the quantization encoder and decoder are denoted by \(\theta _f\) and \(\theta _g\), respectively. The features output by the encoder are
where \(\mathbf {\Lambda }\) denotes either \(\mathbf {\Lambda }^{(\text {ML})}\), \(\mathbf {\Lambda }^{(\text {MMSESIC})}\), or \(\mathbf {\Lambda }^{(\text {LMMSE})}\). Let \(\tilde{\mathbf {\Lambda }}\) denote the matrix of reconstructed soft bits as
where \({\mathcal {Q}}\) is a quantization function that maps the interval \([1, 1]\) to a discrete and finite set of points \({\mathcal {C}}\) of size \(2^{N_b}\), where \(N_b\) is the length of the quantized bit word.
During training, we replace the \({\mathcal {Q}}\) operator with a differentiable approximation to obtain useful gradients by following previous work [24] and using the noise model \({\mathcal {Q}}(x) = x + u\), where u is drawn from \({\mathcal {N}}(0, \sigma _u)\) and \(\sigma _u = 0.001\) is chosen small enough to avoid an error floor in full precision and to control the effective numerical precision of the features during training. The choice of approximating quantization noise with i.i.d. Gaussian noise is motivated both by the ease of implementation (the original motivation in [24]), and by the recent work in [42], that connects the variance of the added noise to the quantization error. In particular, choosing \(\sigma _u^2 = 0.001\) leads to a feature resolution of approximately ten bits. While this is a finer quantization resolution than used in Sect.Â 3.4 (where we use as few as four bits per feature), it allows for flexibility in using a larger number of bits per feature, if desired, whereas injecting more noise during the training stage would remove this possibility.
The loss function used to learn the parameters \(\theta _f\) and \(\theta _g\) of the quantization autoencoder is given by
where B is the batch size and \({\mathcal {L}}\) is given by (15), applied elementwise and summed. Once the quantization autoencoder is trained, the feature representation \({\mathbf {z}}\) is obtained for all training samples by performing a forward pass with the encoder \(f_\text {Q}\).
2.6.2 Stage 2: Estimation
In the second stage, we train an estimation encoder \(f_\text {E}\) with parameters \(\theta _e\). This model uses the received data streams \({\mathbf {y}}\) and the coherent CSI to recover the feature representation learned by the quantization encoder. Given a pretrained pair of quantization functions, we can then further use the quantization decoder to recover the estimated soft bits. The supervised loss used to train the parameters \(\theta _E\) is given by
The choice of using an \(\ell _1\)loss for the feature loss in the estimation phase is empirical and investigated in Sect.Â 3.3, where it is compared to the more conventional mean squared error (MSE) loss.
Note that the ground truth soft bit matrix \(\mathbf {\Lambda }^{(\text {ML})}\) is never explicitly used as a target output in the second stage, and we instead rely on the pretrained feature extractor given by \(f_\mathrm {Q}\). Obtaining the estimated soft bits from the estimated feature representation is straightforward during inference and is done as
A simple singlestage baseline is given by training \(f_\mathrm {E}\) and \(g_\mathrm {Q}\) together (as a single deep neural network with a bottleneck size of \(3N_\mathrm {t}\)), without first constructing the features \({\mathbf {z}}\). This is done by using the loss in (18) between the reconstructed soft bits in (20) and the ground truth soft bits \(\mathbf {\Lambda }\). The twostage approach has the following advantages over this baseline:

Training using the twostage approach is more stable, converges faster and to a lower weighted MSE value when we evaluate using \(g_\text {Q}\). This claim is verified in Sect.Â 3.2.

Twostage training allows for the use of a compact and shallow estimation encoder, which benefits endtoend latency when compared to other deep learning approaches, as shown by the results in Sect.Â 3.3.
2.6.3 Implementation details
The models \(f_\mathrm {Q}\), \(f_\mathrm {E}\) and \(g_\mathrm {Q}\) are deep neural networks that use fully connected layers. A detailed block diagram of the models is shown in Fig.Â 2. All models use the rectified linear unit activation function in the hidden layers, given by \(\mathrm {relu} (x) = \max \{x, 0\}\), with the exceptions shown in Fig.Â 2.
The quantization encoder \(f_\mathrm {Q}\) uses a simple feedforward, fully connected deep neural network with six hidden layers, a width of \(4 N_\mathrm {t} K\) for the hidden layers, and \(3 N_\mathrm {t}\) for the output layer, which matches the upper bound in TheoremÂ 2. Importantly, the width of each layer scales with modulation order, while the depth remains fixed, which is a design choice taken to increase the representational power of the network without sacrificing latency in higherorder modulation scenarios. The quantization decoder \(g_\mathrm {Q}\) uses the branched architecture in Fig.Â 2b: The latent representation is separately processed using \(KN_\mathrm {t}\) parallel, feedforward, fully connected deep neural networks, each with six hidden layers and the same width as the encoder.
To numerically quantize the latent feature representation during testing, a factorized quantization codebook for the features \({\mathbf {z}}\) is learned by separately applying a scalar quantization function to each of its entries. We use the same databased approach as [24] and train a kmeans++ [43] scalar quantizer after \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\) have been trained. We learn a separate quantizer for each latent feature. The memory cost for storing all the scalar quantization codebooks is less than 2 kB in all the experiments and scales linearly with \(N_\mathrm {t}\).
The estimation encoder \(f_\mathrm {E}\) takes as input the triplet \(({\mathbf {y}}, {\mathbf {H}}, \sigma _n)\) and uses their flattened, realvalued (obtained by concatenating the real and imaginary parts) versions with a dense layer for each signal, followed by a concatenation operation. This is an early feature fusion strategy we have used due to the three input signals having different dimensions and scales. A single residual block of the estimation encoder contains an additional six hidden layers with residual connections between them, and is expanded in Fig.Â 2c.
We use the Adam optimizer [44] with a batch size of 32768 samples, learning rate of 0.001 and default TensorFlow [45] parameters for both stages. Training, validation and test data consist of pregenerated ML or MMSESIC (in highdimensional MIMO) soft bit estimates using (4) from 10000, LDPCcoded packets of size (324,Â 648), at six logarithmically spaced SNR values, dependent on the system size. The bits in a codeword are modulated, grouped in MIMO vectors, and transmission is simulated across different (randomly chosen) MIMO channels during training. The receiver uses either ML or MMSESIC algorithm to estimate the soft bits, which are used to train \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\). The data are split using a 80/10/10 train/validation/test ratio. We investigate the following SUMIMO scenarios:

1
\(2 \times 2\), 64QAM and \(4 \times 4\), 16QAM, both trained using ML soft bits. Our findings in Sects.Â 3.3 andÂ 3.4 show that EQNet is viable for both estimation and quantization, respectively.

2
\(8 \times 8\), 16QAM and \(16 \times 16\), 16QAM, both trained using the MMSESIC soft bits. We find that EQNet achieves stateoftheart performance for soft bit quantization in Sect.Â 3.5, and discuss the limitations of using estimation in Sect.Â 3.8.

3
\(32 \times 32\), 16QAM and \(64 \times 64\), 16QAM, where we investigate the performance of applying the models trained on \(16 \times 16\), MMSESIC scenarios to larger sizes, and the LMMSE estimate, in quantization mode. Results are presented in Sect.Â 3.5.
Publicly available implementations can be found in the online code repository^{Footnote 1}.
2.7 Operating modes
Training the components of EQNet is done offline, by first collecting (or simulating, if accurate environment models are available) a dataset of training ML soft bits \(\mathbf {\Lambda }\), the corresponding samples, and CSI information. Estimating the CSI information itself will require pilot symbols [46], but EQNet training is agnostic to their structure or number. Once this dataset is collected, the two phases described in Sect.Â 2.6 are applied in sequence to train the models in EQNet.
FigureÂ 3 shows the role of EQNet during receiver deployment with a HARQ protocol, and how the three blocks in Fig.Â 1 are used. In estimation mode, the algorithm uses \({\mathbf {y}}, {\mathbf {H}}\) and \(\sigma _n\) as inputs for the estimation encoder \(f_\mathrm {E}\), followed by the decoder \(g_\mathrm {Q}\), and outputs the estimated soft bit matrix \(\mathbf {\Lambda }\). As the results in Fig.Â 4 have showed, inaccurate CSI may, in general, degrade the performance of the EQNet soft bit estimation module. While not explored in this work, further training (adapting) the estimation module during deployment to account for inaccurate CSI using a robust optimization objective [47] is possible.
Any external soft bit estimation algorithm (e.g., ML, or MMSESIC) can be used, instead of the EQNet estimation encoder. This does not preclude EQNet being used for quantization, and the modes do not need to be used together, even though the model \(g_\mathrm {Q}\) is shared between them. If codeword decoding fails, the soft bits from the failed transmission must be stored, as shown in Fig.Â 3. If the soft bit estimation module outputs loglikelihood ratios instead of soft bits, then we apply the transform in (3) to convert them to soft bits. The failed, soft codeword is first split into matrices of soft bits corresponding to different MIMO channel uses, and each matrix \(\mathbf {\Lambda }\) is fed to the encoder \(f_\mathrm {Q}\) and quantized, yielding a bit string representing the quantized soft bit matrix. This representation is stored, and the soft bits are decoded (reconstructed) using \(g_\mathrm {Q}\) and converted to loglikelihood using the \({{\,\mathrm{arctanh}\,}}\) function when the second codeword transmission arrives and soft combining is performed.
3 Results and discussion
3.1 Verifying TheoremsÂ 1 andÂ 2
To verify the two bounds for ML soft bits in MIMO channels, and whether the upper bound can be achieved, we treat the latent feature dimension as a hyperparameter and perform an ablation experiment over its size. We train a series of quantization models (the pair of models \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\)), where the only parameter that changes is the size of this feature representation. For exemplification, we target a \(2 \times 2\), 64QAM, i.i.d. Rayleigh fading scenario, but this result also holds for the \(4 \times 4\), 16QAM, i.i.d. Rayleigh fading scenario used in Sect.Â 3.3. For the considered scenario, TheoremÂ 1 states that the latent feature dimension corresponding to the lower bound is eight, while TheoremÂ 2 gives a dimension of six matching the upper bound.
FigureÂ 5 shows the endtoend block error rate performance averaged over 10000 codewords when the feature dimension is the only parameter that changes in a series of autoencoder models. In all cases, the ML soft bit matrix is estimated using (4), and used to train and evaluate the autoencoder with a given bottleneck size. For reference, we also plot the performance of the ideal, uncompressed ML estimate. The following conclusions can be drawn from this plot:

1
The upper bound (in this case, corresponding to \(\text {dim}({\mathbf {z}}) = 6\)) is sufficient to achieve nearoptimal performance, with minimal (less than 0.1 dB) performance losses. We posit that this departure from the ideal curve in Fig.Â 5 is due to the residual training error, even when quantization is not applied to the feature space. Because optimizing the weights of a deep neural network with nonlinear activation is a nonconvex problem, the training loss in (18) is not exactly zero at the end of training, or for the test samples. This implies that soft bits will always be distorted by \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\), even without numerical quantization. In practice, this could be mitigated by training the model for significantly longer, and using significantly more training data, as well as using cyclical learning rate schedules that escape from local minima [48].

2
Any attempt to further compress the soft bit matrix beyond this limitâ€”that is, cases where \(\text {dim}({\mathbf {z}}) < 6\)â€”is met with an increase in error and departure from ML performance, providing evidence that the representation size ratio of TheoremÂ 2 is optimal for the ML soft bits.

3
To capture scenarios that require reliable communications, we simulate SNR values larger than 18.5 dB using one million codewords and still find no significant deviations from optimal performance for \(\mathrm {dim}({\mathbf {z}}) \ge 6\)â€”note that the error for the \(\mathrm {dim}({\mathbf {z}}) = 6\) curve is exactly zero at the 20.3 dB point, due to the finite number of simulated codewords. The matching bit error rate in high SNR (greater than 20 dB) is empirically less than \(10^{7}\), thus to further reduce this performance gap, an outer code such as the Reedâ€“Solomon code [49], with a high rate, could be used.
This result justifies the use of a latent feature space of dimension \(3N_\mathrm {t}\) in all the remaining experiments.
3.2 Importance of twostage training
We investigate the difference in performance of the proposed twostage learning approach against a singlestage learning baseline that combines (15) and (19). We use exactly the same architecture in both cases. FigureÂ 6 shows the validation loss during training (left), as well as the validation coded block error rate after 500 epochs of training (right), for both methods in a \(2 \times 2\), 64QAM, and i.i.d. Rayleigh fading scenario. This highlights the superior performance of the proposed twostage method: baseline singlestage training is unstable and converges much slower, whereas the twostage method is more stableâ€”as shown in the left side of Fig.Â 6â€”and converges to a solution with lower coded block error rateâ€”as shown in the right side of Fig.Â Â 6. While it is possible that the singlestage approach will eventually converge (either through significantly longer training time, or by using a significantly larger neural network), this can be achieved much faster with the proposed approach.
3.3 Estimation performance
We implement and evaluate two variants of EQNet: EQNet (L) (one residual block) and EQNet (P) (three residual blocks). We compare our method with two stateoftheart deep learning baselines: the scheme in [34]â€”that we further refer to as NNDet for brevityâ€”and the OAMPNet2 approach in [32]. We additionally compare with three nonlearningbased approaches: ZFSIC, MMSESIC (both implemented via the QR decomposition, as in [39]), and softoutput SD implemented using the default MATLAB algorithm based on [29]. For \(4 \times 4\), 16QAM, we train a single NNDet model with ten unfolded blocks, labeled as highperformance (P). For \(2 \times 2\), 64QAM, we train two variants of NNDet: NNDet (P) (four unfolded blocks), and NNDet (L) three unfolded blocks) which trades off some of the performance for lower endtoend latency. Implementations of the baselines are available along with our source code.
FiguresÂ 7 andÂ 8 plot the performance in \(4 \times 4\), 16QAM and \(2 \times 2\), 64QAM scenarios, respectively. For the 64QAM scenario, EQNet (L) achieves the same performance as the highperformance NNDet (P). This is better highlighted in Fig.Â 7, where the same conclusion is evident. We observe that using the \(\ell _1\)loss in (19) surpasses the MSE version. We attribute this to the fact that the feature representation is bounded to the \((1, 1)\) interval in the first training stage, causing the MSE to shrinkâ€”and slow down learningâ€”when squared errors are used in the second stage. In contrast, the \(\ell _1\)loss efficiently learns when features are concentrated around the origin. The performance of EQNet also surpasses the ZFSIC and MMSESIC baselines, but there is still a gap with respect to the SD algorithm.
We quantify endtoend latency for processing a batch of B samples (including transfer to/from GPU where applicable) by measuring the average (using 10,000 batches) execution times using the timeit Python module for all algorithms. For the CPU measurements, the processing is executed on a single thread. The CPU is an Intel i99900x running at 3.5 GHz, and the GPU is an NVIDIA RTX 2080Ti. From TableÂ 2 and TableÂ 1, it is observed that, while SD is efficient for a batch size of one, the degree of parallelism is much lower, explained by the heavy use of sorting and the tree search procedure [29]. In contrast, the deep learningbased approaches do not require sorting operations, leading to more efficient parallel implementations that favorably scale with the batch size. ZFSIC and MMSESIC also show low latency in both scenarios, but suffer from a performance drop. The number of parameters (weights) of EQNet is equal to 440k for EQNet (L), and three times more for EQNet (P), while NNDet (L) and NNDet (P) use a total of 195k and 260k, respectively. In contrast, OAMPNet2 does not use any deep neural networks and only has 16 trainable parameters (two for each of the eight unfolded blocks used).
The reduced latency compared to the other deep learning methods is attributed to the fact that the baselines rely on unfolded detection approaches and require additional linear algebra computations, such as matrix inversion and conjugate multiply after each iteration. These operations are sequential and increase the endtoend inference latency. In contrast, our approach does not involve any additional operations. The last lines in TablesÂ 1 andÂ 2 highlight the increased latency gap when the algorithm is used in a scenario with a large number of orthogonal (e.g., multiplexed in the frequency domain) SUMIMO users, where the base station performs computations in large batches. In this case, EQNet is competitive in terms of latency with the ZFSIC and MMSESIC baselines.
3.4 Quantization performance
For the quantization task, we investigate the performance of EQNet against the maximum mutual information (MI) quantizer in [20] and the deep learning baseline in [24], both designed for SISO scenarios. We extend these methods to MIMO scenarios by splitting the soft bit matrix into rows and quantizing each of them separately. To test the quantization methods, random payloads are generated, encoded using an LDPC code of size (324, 648), modulated using QAM, transmitted across a number of different MIMO channels (with fading and noise). At the receiver, soft bits are first estimated with either the ML, MMSESIC, or LMMSE algorithms, after which finally the soft bits are quantized and reconstructed to evaluate distortions. For EQNet, we train a separate kmeans++ quantizer with 64 levels for each scalar dimension of the latent space, thus requiring six bits of storage for each feature, per MIMO channel use. For a \(2 \times 2\), 64QAM scenario, this amounts to compressing the soft bit matrix (twelve soft bits per MIMO channel use) using a 36bit codeword, leading to an effective storage cost of three bits per soft bit.
The results in Fig.Â 9 show that EQNet is superior to both baselines and can efficiently quantize the soft bit matrix, with a minimal performance loss. Compared to the deep learning baseline, EQNet achieves a \(16\%\) compression gain at the same endtoend performance, whereas compared to the maximum MI quantizer, EQNet boosts the performance of the system by 0.65 dB while using the same storage budget. This increase in performance highlights the importance of jointly learning a feature space across the spatial dimensions of MIMO channels.
3.5 Quantization in largescale MIMO scenarios
When considering large, spatial multiplexing SUMIMO scenarios with a high modulation order, the ML estimate \(\mathbf {\Lambda }^{(\mathrm {ML})}\) is no longer practically attainable even during the offline training stage. Note that, according to TheoremÂ 2, a modulation order of at least \(K = 4\) (16QAM) is required to achieve feature compression, which justifies our choice of \(K = 4\) for the remainder of this section.
For \(8 \times 8\) and \(16 \times 16\) MIMO scenarios with 16QAM, Fig.Â 10a and b shows the performance of EQNet and the two quantization baselines, when three bits are used per soft bit. It can be seen that EQNet surpasses both baselines at the same bit budget. More interestingly, all quantization methods improve the performance compared to floating point soft bit estimation with MMSESIC. We attribute this effect to an effect similar to that which occurs in systems that use hybrid hard/soft bit estimation [50]â€”clipping loglikelihood ratios (in our method, implicitly, by quantizing the reconstructed soft bits \(\tilde{\mathbf {\Lambda }}\)) using welltuned clipping levels can compensate for hard errors introduced in the MMSESIC detection chain, especially for larger MIMO scenarios.
FigureÂ 10c and d plots the performance of EQNet and the baselines, when quantizing LMMSE soft bits in \(32 \times 32\) and \(64 \times 64\) scenarios with 16QAM, respectively. In the case of EQNet, we do not train new models \(f_\mathrm {Q}\) and \(g_\mathrm {Q}\), but instead reuse the models trained for \(16 \times 16\), 16QAM by separately quantizing submatrices of 16 rows from the soft bit matrix \(\mathbf {\Lambda }^{(\mathrm {LMMSE})}\). Empirically, we find that learning a model for \(32 \times 32\) from scratch involves very wide networks that are difficult and costly to train, as width scales with \(N_\mathrm {t}\), according to Fig.Â 2.
From Fig.Â 10c and d, we conclude that the quantization part of EQNet is robust to input shifts in the quantization mode and can scale to large MIMO scenariosâ€”even when trained on soft bits derived from the MMSESIC algorithm, the model can quantize LMMSE soft bits, surpassing the deep baseline, and remaining competitive with the MI baseline.
3.6 Robustness to distributional shifts
Practical communication scenarios involve testtime distributional shifts, or may be faced with imperfect CSI during deployment. For soft bit estimation and quantization, the most fragile part of the system is given by the matrix \({\mathbf {H}}\), and any distributional mismatch that occurs. To evaluate the robustness of our approach, we consider models trained on channels from an i.i.d. Rayleigh fading model and tested on realizations of the CDLA channel model adopted by the 5GNR specifications [51].
FigureÂ 11 shows the quantization performance under this shift in a 2by2 64QAM scenario, where EQNet (Rayleigh) is a model trained using Rayleigh channels and EQNet (5G) is an identical model trained using realizations of the CDLA channel model. While a performance gap of around 1 dB is apparent between EQNet (5G) and the ML solution, the EQNet (Rayleigh) model is robust and does not exhibit error floors or performance losses. This is a strong indicator that the quantization learned using Rayleigh channels can be used in a wide variety of conditions and is further discussed in the section.
FigureÂ 12 investigates the estimation performance under the same shift and reveals a higher degree of robustness compared to the NNDet approach, retaining a performance close to that of the ZFSIC and MMSESIC algorithms in the low and midSNR regimes, and being surpassed in the high SNR regime. We attribute this error floor to the illconditioned channels in the CDLA models: Because we are using spatial multiplexing in this scenario, all suboptimal algorithms suffer a performance drop, and the shallow EQNet is no longer sufficient to achieve nearML performance. A potential fix here is to increase the depth of the estimation encoder \(f_\mathrm {E}\), at the cost of increased latency.
We finally investigate the performance of our approach in the case of CSI estimation impairments at the receiver. For this, we use a corrupted version of \({\mathbf {H}}\) generated from the model \(\hat{{\mathbf {H}}} = {\mathbf {H}} + {\mathbf {N}}\), where \({\mathbf {N}}\) is Gaussian noise with zero mean and covariance matrix \(\sigma _\text {CSI}^2{\mathbf {I}}_{N_\mathrm {r} \times N_\mathrm {t}}\). This models impairments coming from the channel estimation module and is a widely used model to test the robustness of downstream estimation algorithms [52, 53]. FigureÂ 4 plots test performance with corrupted CSI in a \(2 \times 2\) MIMO scenario with 64QAM and shows that EQNet is as robust as the NNDet baseline to this type of impairment, while still benefiting from the lower latency in TableÂ 2.
3.7 Limitations: estimation in largescale MIMO scenarios
We have attempted to train the estimation part (second stage) of EQNet for large MIMO scenarios and found that this is faced in the following challenges:

Because the width of the estimation encoder is proportional to \(N_\mathrm {t}^2\), the total number of weights increases in the order of \({\mathcal {O}}(N_\mathrm {t}^4)\), becoming prohibitive even for \(N_\mathrm {t} = 8\) (more than three million weights).

Because we can only generate MMSESIC soft bits for training, learning to estimate them is difficult when using a feedforward architecture, due to the recursive nature of the algorithm. Using a recurrent architecture for \(f_\mathrm {E}\) could be a potential fix for this issue, but would involve major changes to the design in Fig.Â 2(c).

Finally, the latency of MMSESIC is still acceptable in \(8 \times 8\) and \(16 \times 16\) scenarios (e.g., below 5 ms for a batch size of one, on CPU)â€”thus, supervised learning of soft bit estimation with deep learning is faced with the challenge of learning from suboptimal estimates, and it is still an open problem if deep learning can significantly surpass the algorithm used to generate the training soft bits in performance.
Taken together, the above issues currently limit the estimation part of the EQNet framework to be advantageous against competing methods only in small MIMO scenarios, up to \(4 \times 4\), where the ML soft bits (which do not recurrent evaluation) are tractable during the offline training stage. A promising direction of future research is to incorporate algorithm specific changesâ€”such as the recurrent nature of MMSESICâ€”into the deep learning architecture of \(f_\mathrm {E}\), while still using the twostage approach of EQNet. A recent example of architectural design in this sense is given in [54], where interference cancelation steps are interleaved with a learnable model.
Finally, a large portion of the complexity of \(f_\mathrm {E}\) comes from taking as input the full CSI matrix \({\mathbf {H}}\). Without further assumptions on the distribution of \({\mathbf {H}}\) in an environment, this is required for accurate soft bit estimation. Using only incoherent information about \({\mathbf {H}}\), obtained offline, could be a way of significantly reducing inference complexity in environments with strong assumptions.
4 Conclusions
In this paper, we have proposed a deep learning framework that jointly solves the tasks of soft bit estimation and quantization in MIMO scenarios. We have derived theoretical lower and upper bounds on the size of distortionfree representations of ML and MMSESIC soft bits in MIMO channels and further showed evidence that the upper bound for ML soft bits is practically achievable in arbitrary channels. Our approach has been shown to be practical in terms of latency and is compatible with any MIMO system, such as the MIMOOFDM used in 5G scenarios, relaying scenarios, or distributed communication systems, which would benefit from both quantization and estimation gains. Throughout evaluation, our approach has shown superior performance, competitive distributional and impairment robustness to stateoftheart deep learningbased estimation methods.
One drawback that remains is the presence of an error floor when faced with severe distributional shifts at test time, as per Fig.Â 12. Even though our results show that this floor is much lower than that of the prior deep learningbased work in [34], there is still room for improvement, at least in overcoming the MMSESIC algorithm across the entire SNR range. For example, our method could be extended to account for perturbations during training or be trained on a dataset that pools together realizations of multiple channel models. Another promising direction for future work is removing the requirement of training a separate model for each MIMO configuration and leveraging flexible deep learning architectures to learn universal algorithms for soft bit processing.
Data availability statement
The software used to generate soft bit data for training and evaluation, as well as a complete implementation of the proposed approach and the baselines, is publicly available in the EQNet Github repository, located at https://github.com/utcsilab/eqnet.
Abbreviations
 CDL:

Clustered delay line
 CPU:

Central processing unit
 CSI:

Channel state information
 GPU:

Graphical processing unit
 HARQ:

Hybrid automatic repeat request
 LDPC:

Lowdensity parity check
 MIMO:

Multiple input multiple output
 ML:

Maximum likelihood
 MMSESIC:

Minimum mean squared error with successive interference cancelation
 MSE:

Mean squared error
 OAMP:

Orthogonal approximate message passing
 OFDM:

Orthogonal frequencydivision multiplexing
 QAM:

Quadrature amplitude modulation
 SD:

Sphere decoding
 SISO:

Single input single output
 SNR:

Signaltonoise ratio
 VBLAST:

VerticalBell Laboratories Layered SpaceTime
 ZFSIC:

Zeroforcing with successive interference cancelation
References
H. Viswanathan, P.E. Mogensen, Communications in the 6G era. IEEE Access 8, 57063â€“57074 (2020)
J. Hagenauer, Soft is better than hard. Communications and Cryptography, 155â€“171 (1994)
B. Zhao, M.C. Valenti, Practical relay networks: a generalization of hybridARQ. IEEE J. Sel. Areas Commun. 23(1), 7â€“18 (2005)
Y. Song, N. Devroye, Lattice codes for the gaussian relay channel: Decodeandforward and compressandforward. IEEE Trans. Inf. Theory 59(8), 4927â€“4948 (2013)
T.X. Vu, H.D. Nguyen, T.Q. Quek, Adaptive compression and joint detection for fronthaul uplinks in cloud radio access networks. IEEE Trans. Commun. 63(11), 4565â€“4575 (2015)
P. Kairouz, H.B. McMahan, B. Avent, A. Bellet, M. Bennis, A.N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al, Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019)
S. Feng, D. Niyato, P. Wang, D.I. Kim, Y.C. Liang, Joint service pricing and cooperative relay communication for federated learning. In: 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), pp. 815â€“820 (2019). IEEE
P. Frenger, S. Parkvall, E. Dahlman, Performance comparison of HARQ with Chase combining and incremental redundancy for HSDPA. In: IEEE 54th Vehicular Technology Conference. VTC Fall 2001. Proceedings (Cat. No. 01CH37211), vol. 3, pp. 1829â€“1833 (2001). IEEE
W. Ejaz, G. Hattab, N. Cherif, M. Ibnkahla, F. Abdelkefi, M. Siala, Cooperative spectrum sensing with heterogeneous devices: Hard combining versus soft combining. IEEE Syst. J. 12(1), 981â€“992 (2016)
E.G. Larsson, P. Stoica, J. Li, On maximumlikelihood detection and decoding for spacetime coding systems. IEEE Trans. Signal Process. 50(4), 937â€“944 (2002)
M. Bennis, M. Debbah, H.V. Poor, Ultrareliable and lowlatency wireless communication: Tail, risk, and scale. Proc. IEEE 106(10), 1834â€“1853 (2018)
3GPP: 38.214: Physical Layer Procedures for Data. (2021). 3GPP. Ver. 16.4
M.A. Albreem, M. Juntti, S. Shahabuddin, Massive MIMO detection techniques: a survey. IEEE Commun. Surv. Tutor. 21(4), 3109â€“3132 (2019)
C. Zhang, P. Patras, H. Haddadi, Deep learning in mobile and wireless networking: a survey. IEEE Commun. Surv. Tutor. 21(3), 2224â€“2287 (2019)
T. Erpek, T.J. Oâ€™Shea, Y.E Sagduyu, Y. Shi, T.C. Clancy, Deep Learning for Wireless Communications (2020). arXiv:2005.06068
N. Akhtar, A. Mian, Threat of adversarial attacks on deep learning in computer vision: a survey. Ieee Access 6, 14410â€“14430 (2018)
D. Krueger,E. Caballero, J.H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. LeÂ Priol, A. Courville, Outofdistribution generalization via risk extrapolation. In: International Conference on Machine Learning, pp. 5815â€“5826 (2021). PMLR
S. KokaljFilipovic, R. Miller, Adversarial examples in RF deep learning: detection of the attack and its physical robustness. arXiv preprint arXiv:1902.06044 (2019)
Q. Hu, F. Gao, H. Zhang, S. Jin, G.Y. Li, Deep learning for channel estimation: interpretation, performance, and comparison. IEEE Trans. Wireless Commun. 20(4), 2398â€“2412 (2020)
A. Winkelbauer, G. Matz, On quantization of loglikelihood ratios for maximum mutual information. In: Proc. IEEE Intl. Workshop on Signal Proc. Adv. in Wireless Commun., pp. 316â€“320 (2015)
G. Zeitler, G. Bauch, J. Widmer, Quantizeandforward schemes for the orthogonal multipleaccess relay channel. IEEE Trans. Commun. 60(4), 1148â€“1158 (2012)
M. Danieli, S. Forchhammer, J.D. Andersen, L.P. Christensen, S.S. Christensen, Maximum mutual information vector quantization of loglikelihood ratios for memory efficient HARQ implementations. In: 2010 Data Compression Conference, pp. 30â€“39 (2010). IEEE
W. Rave, Quantization of loglikelihood ratios to maximize mutual information. IEEE Signal Process. Lett. 16(4), 283â€“286 (2009)
M. Arvinte, A.H Tewfik, S. Vishwanath, Deep loglikelihood ratio quantization. In: 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1â€“5 (2019). IEEE
P.W. Wolniansky, G.J. Foschini, G.D. Golden, R.A. Valenzuela, VBLAST: An architecture for realizing very high data rates over the richscattering wireless channel. In: Proc. URSI Intl. Symp. on Signals, Syst., and Electron., pp. 295â€“300 (1998)
C. Shen, Y. Zhu, S. Zhou, J. Jiang, On the performance of VBLAST with zeroforcing successive interference cancellation receiver. In: IEEE Global Telecommunications Conference, 2004. GLOBECOMâ€™04., vol. 5, pp. 2818â€“2822. IEEE
P. Luethi, A. Burg, S. Haene, D. Perels, N. Felber, W. Fichtner, VLSI implementation of a highspeed iterative sorted MMSE QR decomposition. In: 2007 IEEE International Symposium on Circuits and Systems, pp. 1421â€“1424 (2007). IEEE
B. Hassibi, H. Vikalo, On the spheredecoding algorithm I. Expected complexity. IEEE Trans. Signal Process. 53(8), 2806â€“2818 (2005)
C. Studer, A. Burg, H. Bolcskei, Softoutput sphere decoding: Algorithms and VLSI implementation. IEEE JSAC 26(2), 290â€“300 (2008)
Y. Wu, J. McAllister, Configurable quasioptimal sphere decoding for scalable MIMO communications. IEEE Trans. Circuits Syst. I Regul. Pap. 68(6), 2675â€“2687 (2021)
M. Mohammadkarimi, M. Mehrabi, M. Ardakani, Y. Jing, Deep learningbased sphere decoding. IEEE Trans. Wireless Commun. 18(9), 4368â€“4378 (2019)
H. He, C.K. Wen, S. Jin, G.Y. Li, Modeldriven deep learning for MIMO detection. IEEE Trans. Signal Process. 68, 1702â€“1715 (2020)
J. Ma, L. Ping, Orthogonal AMP. IEEE Access 5, 2020â€“2033 (2017)
O. Sholev, H.H. Permuter, E. BenDror, W. Liang, Neural network MIMO detection for coded wireless communication with impairments. In: Proc. IEEE Wireless Commun. Netwk. Conf., pp. 1â€“8 (2020)
O. Shental, J. Hoydis, Machine LLRning: Learning to softly demodulate. In: Proc. IEEE Global Commun. Conf. Workshops, pp. 1â€“7 (2019)
D. Tse, P. Viswanath, Fundamentals of Wireless Communication (Cambridge University Press, Cambridge, 2005)
J. Chen, A. Dholakia, E. Eleftheriou, M.P. Fossorier, X.Y. Hu, Reducedcomplexity decoding of LDPC codes. IEEE Trans. Commun. 53(8), 1288â€“1299 (2005)
B. Hanin, Which neural net architectures give rise to exploding and vanishing gradients? In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 580â€“589 (2018)
R.T. Kobayashi, T. AbrÃ£o, Ordered MMSESIC via sorted QR decomposition in ill conditioned largescale MIMO channels. Telecommun. Syst. 63(2), 335â€“346 (2016)
R.W. Heath, N. GonzalezPrelcic, S. Rangan, W. Roh, A.M. Sayeed, An overview of signal processing techniques for millimeter wave MIMO systems. IEEE J. Select. Top. Signal Process. 10(3), 436â€“453 (2016)
H. Sampath, P. Stoica, A. Paulraj, Generalized linear precoder and decoder design for MIMO channels using the weighted MMSE criterion. IEEE Trans. Commun. 49(12), 2198â€“2206 (2001)
A. Kipnis, G. Reeves, Gaussian approximation of quantization error for estimation from compressed data. IEEE Trans. Inf. Theory 67(8), 5562â€“5579 (2021)
D. Arthur, S. Vassilvitskii, kmeans++ the advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACMSIAM Symposium on Discrete Algorithms, pp. 1027â€“1035 (2007)
D.P. Kingma, J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al.: Tensorflow: A system for largescale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 16), pp. 265â€“283 (2016)
M. Biguesh, A.B. Gershman, Trainingbased MIMO channel estimation: a study of estimator tradeoffs and optimal training signals. IEEE Trans. Signal Process. 54(3), 884â€“893 (2006)
Maini, P., Wong, E., Kolter, Z.: Adversarial robustness against the union of multiple perturbation models. In: International Conference on Machine Learning, pp. 6640â€“6650 (2020). PMLR
Smith, L.N.: A disciplined approach to neural network hyperparameters: Part 1â€“learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820 (2018)
C. Thommesen, The existence of binary linear concatenated codes with reedsolomon outer codes which asymptotically meet the GilbertVarshamov bound. IEEE Trans. Inf. Theory 29(6), 850â€“853 (1983)
Zimmermann, E., Milliner, D.L., Barry, J.R., Fettweis, G.: Optimal LLR clipping levels for mixed hard/soft output detection. In: IEEE GLOBECOM 20082008 IEEE Global Telecommunications Conference, pp. 1â€“5 (2008). IEEE
3GPP: Study on Channel Model for Frequency Spectrum Above 6 GHz. (2019). 3GPP
R. Narasimhan, Error propagation analysis of VBLAST with channelestimation errors. IEEE Trans. Commun. 53(1), 27â€“31 (2005)
N.I. Miridakis, T.A. Tsiftsis, On the joint impact of hardware impairments and imperfect CSI on successive decoding. IEEE Trans. Veh. Technol. 66(6), 4810â€“4822 (2016)
N. Shlezinger, R. Fu, Y.C. Eldar, DeepSIC: Deep soft interference cancellation for multiuser MIMO detection. IEEE Trans. Wireless Commun. 20(2), 1349â€“1362 (2020)
Acknowledgements
The authors would like to thank Ian Roberts for providing early feedback for the manuscript.Â The authors would also like to thank the editor and theÂ anonymous reviewers for their constructive feedback during the review process.
Funding
MA, SV, and JT have been supported by grant ONR N000141912590. The funding body has not played any role in the design of the study, analysis, interpretation of data, or writing the manuscript.
Author information
Authors and Affiliations
Contributions
MA designed and implemented the training and evaluation software pipelines for the algorithms, and contributed to writing all sections. MA and JT formulated, proved, and wrote the theoretical results section. All authors contributed to writing and revising the introduction and conclusion sections and to defining the problem formulation. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
1.1 Proof of TheoremÂ 1
Let the QR decomposition of \({\mathbf {H}}\) be given by \({\mathbf {H}} = {\mathbf {Q}} {\mathbf {R}}\), where \({\mathbf {Q}}\) satisfies \({\mathbf {Q}}^\mathrm {H} {\mathbf {Q}} = {\mathbf {I}}_{N_\mathrm {t}}\) and \({\mathbf {R}}\) is a square upper triangular matrix. This decomposition exists for any arbitrary channel, even if the matrix \({\mathbf {H}}\) is rankdeficient. Then, after leftside multiplication with \({\mathbf {Q}}^{\text {H}}\), the system in (1) becomes
Given that the rows of \({\mathbf {Q}}\) are orthonormal, it follows that \(\left\Vert {\mathbf {y}}  {\mathbf {H}} {\mathbf {x}}\right\Vert _2^2 = \left\Vert \tilde{{\mathbf {y}}}  {\mathbf {R}} {\mathbf {x}}\right\Vert _2^2\), for any \({\mathbf {x}}\). Then, by replacing all the norms in (7), it follows that
Thus, there exists a surjective function that maps the features \(\tilde{{\mathbf {y}}}/\sigma _n^2\) and \({\mathbf {R}}/\sigma _n^2\) to the matrix of soft bits \(\mathbf {\Lambda }^{\text {(ML)}}\) by implementing the equation above. In this case, the dimension of the domain of the function is given by the cost of storing these features. As \(\tilde{{\mathbf {y}}}\) is a complexvalued vector of length \(N_\mathrm {t}\), it can be stored using \(2N_\mathrm {t}\) realvalued features. \({\mathbf {R}}\) is a complexvalued upper triangular matrix of size \(N_\mathrm {t} \times N_\mathrm {t}\) with realvalued elements on its diagonal, and it can be stored using \(N_\mathrm {t}\) and \(2\frac{(N_\mathrm {t}  1)N_\mathrm {t}}{2}\) realvalued features for its diagonal and offdiagonal terms, respectively. Thus, we have that
realvalued features are sufficient to reconstruct \(\mathbf {\Lambda }^{\text {(ML)}}\) exactly. Thus, there exists a surjective function \(f:{\mathbb {R}}^{N_\mathrm {t} (N_\mathrm {t} + 2)} \rightarrow {\mathbb {R}}^{K \times N_\mathrm {t}}\) such that \(f(\tilde{{\mathbf {y}}}, {\mathbf {R}}) = \mathbf {\Lambda }^{\text {(ML)}}\). The representation size ratio of this function is equal to \(R_\mathrm {low, ML} = \frac{KN_\mathrm {t}}{N_\mathrm {t}(N_\mathrm {t}+2)} = \frac{K}{N_\mathrm {t}+2}\), proving the lower bound.
1.2 Proof of TheoremÂ 2
Let the singular value decomposition of \({\mathbf {H}}\) be given by \({\mathbf {H}} = {\mathbf {U}} {\mathbf {S}} {\mathbf {V}}^\text {H}\), where \({\mathbf {U}}\) and \({\mathbf {V}}\) both satisfy \({\mathbf {U}}^\text {H} {\mathbf {U}} = {\mathbf {V}}^\text {H} {\mathbf {V}} = {\mathbf {I}}_{N_\mathrm {t}}\) and \({\mathbf {S}}\) is a realvalued, diagonal matrix of size \(N_\mathrm {t} \times N_\mathrm {t}\).
Given that the transmitted symbols are \({\mathbf {s}} = {\mathbf {V}} {\mathbf {x}}\), the system in (1) takes the form:
Then, as \({\mathbf {U}}\) is orthonormal, leftmultiplying with \({\mathbf {U}}^\mathrm {H}\) leads to the postprocessed received symbols:
where \(\tilde{{\mathbf {n}}}\) is still i.i.d. Gaussian noise when \({\mathbf {U}}\) is orthonormal. Finally, as \({\mathbf {S}}\) is diagonal, it follows that the soft bits corresponding to the ith transmitted symbol only depend on the equation:
According to [24], the soft bits for each transmitted symbol in a scalar channel can be derived from a threedimensional realvalued feature representation by explicitly storing the complexvalued \({\tilde{y}}\) and the complexvalued \(s_i / \sigma _n^2\). Thus, the soft bit matrix \(\mathbf {\Lambda }^{\text {(ML)}}\) can be represented using a \(3N_\mathrm {t}\)dimensional feature representation when the channel is diagonalized. The representation size ratio of this function is equal to \(R_\mathrm {up, ML} = \frac{KN_\mathrm {t}}{3N_\mathrm {t}} = \frac{K}{3}\).
Given that \(N_\mathrm {t} \ge 1\), we have that \(R_\mathrm {up}\) represents an upper representation size ratio bound for \(K \ge 3\) and arbitrary channels \({\mathbf {H}}\), because at least the subset of soft bits derived from diagonalized channels cannot be represented using fewer features without further assumptions.
1.3 Proof of TheoremÂ 3
The proof is by induction. Considering (12) and assuming that \(n_i\) is Gaussian, then \({\tilde{n}}_i\) is also Gaussian, as \({\mathbf {n}}\) and \({\mathbf {Q}}\) are independent. The closedform expression of the kth LLR of the \(N_\mathrm {t}\)th symbol is given by
Then, the above equation is a function \(f_{N_\mathrm {t}}({\tilde{y}}_{N_\mathrm {t}} / \sigma _{{\tilde{n}}_{N_\mathrm {t}}}^2, {\underline{r}}_{\mathrm {t}, \mathrm {t}} / \sigma _{{\tilde{n}}_{N_\mathrm {t}}}^2) = L_{:, N_\mathrm {t}}^{(\mathrm {MMSESIC})}\). Given that \({\tilde{y}}_{N_\mathrm {t}}\) is a complex scalar, \({\underline{r}}_{N_\mathrm {t}, N_\mathrm {t}}\) is a real scalar, and \(\sigma _{{\tilde{n}}_{N_\mathrm {t}}}^2\) is real scalar, it follows that the LLR vector corresponding to the last spatial stream can be exactly represented by a threedimensional real vector. This proves the case \(N_\mathrm {t} = 1\), where sequential detection is completed.
The equation for estimating the LLR values corresponding to the ith symbol, given the previous \(N_\mathrm {t}  i\) estimates given by
Letting \({\hat{y}}_{i} = {\tilde{y}}_{i}  \sum _{j=i+1}^{N_\mathrm {t}} {\underline{r}}_{i, j} x_{j}^{(\mathrm {MMSESIC})}\), and treating the remaining interference as Gaussian, we obtain the compact model in (12) as
and there exists a function \(f_i({\hat{y}}_{i} / \sigma _{{\hat{n}}_i}^2, {\underline{r}}_{\mathrm {i}, \mathrm {i}} / \sigma _{{\hat{n}}_i}^2) = L_{:, i}^{(\mathrm {MMSESIC})}\) for each i.
It follows that, given estimates for the previous \(N_\mathrm {t}  i\) symbols, the LLR values of the ith symbol can be exactly represented by a vector with three real values, regardless of the modulation order. Thus, by induction, there exists a function \(f:{\mathbb {R}}^{3N_\mathrm {t}} \rightarrow {{\mathbb {R}}^{K \times N_\mathrm {t}}}\) such that \(f(\hat{{\mathbf {y}}}, \mathrm {diag}(\underline{{\mathbf {R}}})) = {\mathbf {L}}^{\text {(MMSESIC)}}\). Using that \(\Lambda _{i,k} = \tanh \frac{L_{i,k}}{2}\) is a bijective function of L, for all i,Â k, it follows that \(R_\mathrm {up, MMSESIC} = \frac{KN_\mathrm {t}}{3N_\mathrm {t}} = \frac{K}{3}\).
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Arvinte, M., Vishwanath, S., Tewfik, A.H. et al. Joint highdimensional soft bit estimation and quantization using deep learning. J Wireless Com Network 2022, 51 (2022). https://doi.org/10.1186/s1363802202129z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s1363802202129z