Suppose that the message **C**_{
t
} is sent at each block. Since errors occur during transmission of the actual transmitted signal matrix **S**_{
t
} due to channel fading and noise, after differential decoding, assume that the message **E**_{
t
} in each block is detected. It follows that \({{\mathbf {E}}_{t}}{\mathbf {E}}_{t}^{H} = {{\mathbf {I}}_{{n_{T}}}}\), where **I**_{
n
} is the *n*_{
T
}×*n*_{
T
} identity matrix. In order to measure the difference between **C**_{
t
} and **D**_{
t
}, we define \({{\mathbf {D}}_{t}} = {{\mathbf {E}}_{t}}{\mathbf {C}}_{t}^{H}\). So the matrix distance between **C**_{
t
} and **E**_{
t
} can be expressed as \(\text {trace}\left \{ {{\text {Re}} \left ({{{\mathbf {I}}_{{n_{T}}}} - {{\mathbf {D}}_{t}}} \right)} \right \}\). When no error occurs, \(\phantom {\dot {i}\!}{{\mathbf {D}}_{t}} = {{\mathbf {I}}_{{n_{T}}}}\), so \(\text {trace}\left \{ {{\text {Re}} \left ({{{\mathbf {I}}_{{n_{T}}}} - {{\mathbf {D}}_{t}}} \right)} \right \} = 0\). Since \({{\mathbf {D}}_{t}}{\mathbf {D}}_{t}^{H} = {{\mathbf {I}}_{{n_{T}}}}\), it follows that matrix **D**_{
t
} has the same orthogonal property as the message matrix **C**_{
t
} and the actual transmitted signal matrix **S**_{
t
}.

Recall that the HR-DSM transmitter transmits the matrices **S**_{
t
} and **S**_{t−1} instead of directly transmitting the message matrices **C**_{
t
}. Due to the influence of fading and noise, suppose that while **S**_{
t
} and **S**_{t−1} are transmitted, **Q**_{
t
} and **Q**_{t−1} are actually received which causes that the differentially decoded message matrices **C**_{
t
} to become the error message matrices **E**_{
t
}. Obviously, **Q**_{
t
}=**E**_{
t
}**Q**_{t−1}=**E**_{
t
}**C**_{t−1} and \({{\mathbf {Q}}_{t}}{\mathbf {Q}}_{t - 1}^{H} = {{\mathbf {E}}_{t}}\).

Let *η*_{
c
} and *η*_{
e
} be the decision variables for transmission matrices **C** and **E**, respectively. Besides, let *P*(**C**→**E**|**H**) be the pairwise error probability of deciding **E** when **C** is transmitted for a given channel realization **H**. Then, *P*(**C**→**E**|**H**) can be expressed as

$$\begin{array}{*{20}l} P\left({{\mathbf{C}} \to {\mathbf{E}}|{\mathbf{H}}} \right) &= P\left({{\eta_{e}} - {\eta_{c}} > 0|{\mathbf{H}}} \right) \\ &= P\left({\text{trace}\left\{ {{\text{Re}} \left({{\Phi_{t - 1,t}}} \right)} \right\} > 0|{\mathbf{H}}} \right), \end{array} $$

(9)

where

$$\begin{array}{*{20}l} {\eta_{c}} &= \text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{Y}}_{t}}{\mathbf{Y}}_{t - 1}^{H}{{\mathbf{S}}_{t - 1}}{\mathbf{S}}_{t}^{H}} \right)} \right\},\\ {\eta_{e}} &= \text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{Y}}_{t}}{\mathbf{Y}}_{t - 1}^{H}{{\mathbf{Q}}_{t - 1}}{\mathbf{Q}}_{t}^{H}} \right)} \right\} \end{array} $$

(10)

and

$$\begin{array}{*{20}l} {\Phi_{t - 1,t}} = {\eta_{c}} - {\eta_{e}}. \end{array} $$

(11)

Substituting Eq. (2) into Eqs. (10) and (11), we have

$$\begin{array}{*{20}l} &{}\text{trace}\left\{ {{\text{Re}} \left({{\Phi_{t - 1,t}}} \right)} \right\} \simeq \text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{H}}_{t - 1}}{\mathbf{H}}_{t}^{H}\left({{{\mathbf{D}}_{t}} - {{\mathbf{I}}_{{n_{T}}}}} \right)} \right)} \right\}\\ &{}+ \text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{S}}_{t}}{{\mathbf{H}}_{t}}{\mathbf{N}}_{t - 1}^{H}{{\mathbf{Q}}_{t - 1}}{\mathbf{Q}}_{t}^{H} + {{\mathbf{N}}_{t}}{\mathbf{H}}_{t - 1}^{H}{\mathbf{C}}_{t - 1}^{H}{{\mathbf{Q}}_{t - 1}}{\mathbf{Q}}_{t}^{H}} \right.} \right.\\ &{}\left. {\left. { - {{\mathbf{S}}_{t}}{{\mathbf{H}}_{t}}{\mathbf{N}}_{t - 1}^{H}{{\mathbf{S}}_{t - 1}}{\mathbf{S}}_{t}^{H} - {{\mathbf{N}}_{t}}{\mathbf{H}}_{t - 1}^{H}{\mathbf{S}}_{t - 1}^{H}{{\mathbf{S}}_{t - 1}}{\mathbf{S}}_{t}^{H}} \right)} \right\}. \end{array} $$

(12)

Note that the second-order noise terms in Eq. (12) are ignored since they are quite small compared to other noise terms when SNR is large enough. Let

$$\begin{array}{*{20}l} \Delta & = \text{trace}\left\{ {{\text{Re}} \left({{\mathbf{\Phi}_{t - 1,t}}} \right)} \right\}\\ &= - \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right)\rho + \text{trace}\left\{ {{\text{Re}} \left(\Theta \right)} \right\}, \end{array} $$

(13)

where *ρ* is defined as

$$\begin{array}{*{20}l} \rho = \text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{I}}_{{n_{T}}}} - {{\mathbf{D}}_{t}}} \right)} \right\}, \end{array} $$

(14)

and

$$\begin{array}{*{20}l} \Theta &= {{\mathbf{S}}_{t}}{{\mathbf{H}}_{t}}{\mathbf{N}}_{t - 1}^{H}{{\mathbf{Q}}_{t - 1}}{\mathbf{Q}}_{t}^{H} + {{\mathbf{N}}_{t}}{{\mathbf{H}}_{t-1}^{H}}{\mathbf{C}}_{t - 1}^{H}{{\mathbf{Q}}_{t - 1}}{\mathbf{Q}}_{t}^{H}\\ &\,\,\,\,\,\,\,-{{\mathbf{S}}_{t}}{{\mathbf{H}}_{t}}{\mathbf{N}}_{t - 1}^{H}{{\mathbf{S}}_{t - 1}}{\mathbf{S}}_{t}^{H} - {{\mathbf{N}}_{t}}{{\mathbf{H}}_{t-1}^{H}}{\mathbf{S}}_{t - 1}^{H}{{\mathbf{S}}_{t - 1}}{\mathbf{S}}_{t}^{H}. \end{array} $$

(15)

For given transmission matrices **C** and **E**, **Q**_{t−1},**Q**_{
t
},**S**_{t−1} and **S**_{
t
}, and *ρ* can be considered as deterministic quantities. Therefore, we can easily show that *E*[trace{Re(*Θ*)}]=0. Taking the expectation of both sides of Eq. (13), we have:

$$\begin{array}{*{20}l} E\left[ \Delta \right] &= - \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right)E\left[ \rho \right] + E\left[ \text{trace}\left\{ {{\text{Re}} \left(\Theta \right)} \right\}\right]\\ &=- \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right)\rho. \end{array} $$

(16)

The computation of the variance of *Δ* is more complicated, because some terms in Eq. (15) are correlated, although most of the terms are assumed to be mutually independent. It is proved in the “Appendix” section that the variance of *Δ* is given by:

$$\begin{array}{*{20}l} &\text{Var}\left[ \Delta \right]\\ &= \text{Var}\left[ \mathrm{trace\left\{ {{\text{Re}} \left(\Theta \right)} \right\}} \right]\\ &= 8\left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right){N_{0}}\\ & - 4{N_{0}}\left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right)\text{trace}\left\{ {{\text{Re}} \left({{{\mathbf{D}}_{t}}} \right)} \right\}, \end{array} $$

(17)

which can be simplified to

$$\begin{array}{*{20}l} \text{Var}\left[ \Delta \right] = 2\rho \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right){N_{0}}. \end{array} $$

(18)

From Eqs. (9), (13), (16), and (18), it follows that

$$\begin{array}{*{20}l} &\Pr \left({{\mathbf{C}} \to {\mathbf{E}}|{\mathbf{H}}} \right)\\ &\,\,\,\,\,= \Pr \left({{\eta_{e}} - {\eta_{c}} > 0|{\mathbf{H}}} \right)\\ &\,\,\,\,\,= \Pr \left({\Delta > 0|{\mathbf{H}}} \right)\\ &\,\,\,\,\,= Q\left({\sqrt {\gamma \rho \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right)/4}} \right), \end{array} $$

(19)

where *Q* denotes the Gaussian tail function, and *γ*=*E*_{
s
}/*N*_{0} is the SNR per symbol. Defining the instantaneous SNR as

$$\begin{array}{*{20}l} {\gamma_{b}} = \gamma \left({\sum\limits_{\scriptstyle i = 1,\ldots,{n_{R}}\,\hfill\atop \scriptstyle j = 1,\ldots,{n_{T}}\hfill} {{{\left| {{h_{ij}}} \right|}^{2}}}} \right), \end{array} $$

(20)

and using the alternative form of the Gaussian Q-function [25], we can write

$$\begin{array}{*{20}l} \Pr \left({{\mathbf{C}} \to {\mathbf{E}}|{\mathbf{H}}} \right) &= Q\left({\sqrt {\left({\rho /2} \right){\gamma_{b}}}} \right)\\ &= \frac{1}{\pi }\int\limits_{0}^{\pi /2} {\exp \left({ - \frac{{\left({\rho /2} \right){\gamma_{b}}}}{{2{{\sin }^{2}}\theta }}} \right)} d\theta. \end{array} $$

(21)

Averaging Eq. (21) over all realizations of the channel matrix **H**, we obtain the PEP as

$$\begin{array}{*{20}l} \Pr \left({{\mathbf{C}} \to {\mathbf{E}}} \right) = \int\limits_{0}^{\infty} {Q\left({\sqrt {\left({\rho /2} \right){\gamma_{b}}}} \right)} p\left({{\gamma_{b}}} \right)d{\gamma_{b}}. \end{array} $$

(22)

where *p*(*γ*_{
b
}) is the PDF of *γ*_{
b
} given in [26].

Let **u** represent a sequence with *q* information bits and \({\hat {\mathbf {u}}}\) denotes an error sequence with the same number of information bits. The bit error probability *P*_{
b
} of the proposed HR-DSM scheme is union-bounded by [27, 28]:

$$\begin{array}{*{20}l} {P_{b}} \le \frac{1}{{2q}}\sum\limits_{{\mathbf{C}} \ne {\mathbf{E}}} {\Pr \left({{\mathbf{C}} \to {\mathbf{E}}} \right) \cdot w\left({{\mathbf{u}},{\hat{\mathbf{u}}}} \right)}, \end{array} $$

(23)

where \(w\left ({{\mathbf {u}},{\hat {\mathbf {u}}}} \right)\) is the Hamming distance between sequences **u** and \({\hat {\mathbf {u}}}\). The PEP Pr(**C**→**E**) is given by Eq. (22).