### Appendix 1

#### Preliminary results

We first present four lemmas and their proof steps in this section. These results will be useful for the proofs of Theorem 1 in the next section.

##### Lemma 2

By assuming quasi-static and flat Rayleigh-fading channels, the probability of each situation *A*_{
k
} with *k*-qualified relays broadcasting ACK1 signals can be expressed as P\left({A}_{k}\right)\doteq {\rho}^{-(2-k){M}_{r}(1-\stackrel{~}{r})}, where

\stackrel{~}{r}={lim}_{\rho \to \infty}\frac{\stackrel{~}{R}\left(\rho \right)}{log\rho}.

##### Lemma 3

By assuming quasi-static and flat Rayleigh-fading channels and {\alpha}_{1}={\beta}_{1}=\sqrt{\frac{1}{2}} for simplicity, the probability of the event *E* that both relays can successfully perform *ZF* detection and broadcast ACK2 signals, as expressed in Equation (2), can be bounded as

1-\frac{4{\gamma}^{{M}_{r}-1}}{({M}_{r}-1)!}\le P\left(E\right)\le 1-\frac{2{\gamma}^{{M}_{r}-1}}{({M}_{r}-1)!},

(A.1)

where \gamma =\frac{{2}^{\stackrel{~}{R}}-1}{\rho {\alpha}_{1}^{2}{\delta}^{2}}, *δ*^{2} is the variance of each channel coefficient. Furthermore, the probability of the event \overline{E} can be obviously obtained as P\left(\overline{E}\right)\doteq {\rho}^{-({M}_{r}-1)(1-\stackrel{~}{r})}.

##### Lemma 4

When *X*_{1} and *X*_{2} are subjected to exponential distribution, the probability can be approximated as *P*(*X*_{1}*X*_{2}<*a*)≈−*a* ln*a*, where *a*>0 and is sufficiently small.

##### Lemma 5

Assuming that z=max\{{z}_{1},\cdots \phantom{\rule{0.3em}{0ex}},{z}_{{M}_{r}}\}, where *z*_{
i
} is exponentially distributed with unit variance, the expectation of *e*^{−cz}(*c*>0) can be revealed as \epsilon \left[{e}^{-\mathit{\text{cz}}}\right]=\frac{{M}_{r}!}{{\mathrm{\Pi}}_{i=1}^{{M}_{r}}(c+i)},\epsilon [\xb7] denotes the expectation of a random variable.

The proofs steps of the above lemmas are provided in the following section.

#### Proof of Lemma 2

Denote *B* as the event that a relay R_{
j
} is not qualified, i.e., it can be expressed as the event: B\triangleq \bigcup _{i=1}^{2}\left\{log(1+\rho \underset{1}{\overset{2}{\alpha}}|{\mathbf{h}}_{i,\mathit{\text{Rj}}}{|}^{2})<\stackrel{~}{R}\right\}. The probability of the event *B* can be easily calculated as P\left(B\right)\doteq {\rho}^{-{M}_{r}(1-\stackrel{~}{r})}. Thus, the probability of each situation *A*_{
k
} can be expressed as follows:

P\left({A}_{k}\right)\doteq \left(\begin{array}{l}2\\ k\end{array}\right){[1-P(B\left)\right]}^{k}{\left[P\right(B\left)\right]}^{2-k}\doteq {\rho}^{-(2-k){M}_{r}(1-\stackrel{~}{r})}.

(A.2)

#### Proof of Lemma 3

Consider a signal model in which the *r*_{2}-th antenna of R_{2} transmits a message {\stackrel{~}{s}}_{1} with a power lever {\beta}_{1}^{2} while the first source S_{1} broadcasts a message {\stackrel{~}{s}}_{2} with a power lever {\alpha}_{1}^{2}. We set {\alpha}_{1}={\beta}_{1}=\sqrt{\frac{1}{2}} for ease of explanation. Such a signal model can be found in most time slots with odd numbers in situation *A*_{2} event *E*. At this time, the signal vector received by R_{1} can be written as {\mathbf{r}}_{R1}={\mathbf{H}}_{R1}\stackrel{~}{\mathbf{s}}+{\mathbf{w}}_{1}, where {\mathbf{r}}_{R1},{\mathbf{w}}_{1}\in {\mathcal{C}}^{{M}_{r}}, {\mathbf{H}}_{R1}=\left[\begin{array}{ll}{\beta}_{1}{\mathbf{h}}_{2,{r}_{2}}& {\alpha}_{1}{\mathbf{h}}_{R1}\end{array}\right], \stackrel{~}{\mathbf{s}}={[{\stackrel{~}{s}}_{1},{\stackrel{~}{s}}_{2}]}^{T}, and **w**_{1} is the noise vector. This transmission model can be recognized as a special MIMO system with two transmit, and *M*_{
r
} receive antennas when ZF detection is applied. Denote the event as {E}_{R1}\triangleq \left\{log\left(1\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}\frac{\rho}{{\left[{\left(\underset{R1}{\overset{H}{\mathbf{H}}}{\mathbf{H}}_{R1}\right)}^{-1}\right]}_{k,k}}\right)>\stackrel{~}{R},\phantom{\rule{0.3em}{0ex}}\forall \phantom{\rule{0.3em}{0ex}}k\in \{1,2\}\right\}. Applying ZF detection at the receiver and according to [26], the probability of *E*_{R 1} can be expressed as

P\left({E}_{R1}\right)=1-\frac{2{\gamma}^{{M}_{r}-1}}{({M}_{r}-1)!},

(A.3)

when *ρ* is sufficiently large, where \gamma =\frac{{2}^{\stackrel{~}{R}}-1}{\rho {\alpha}_{1}^{2}{\delta}^{2}}. Note that the similar ZF decoding can be applied at R_{2} when R_{1} and S_{2} are transmitting messages at the same time, and *E*_{R 2} is similarly defined as {E}_{R2}\triangleq \left\{log\left(1+\frac{\rho}{{\left[{\left(\underset{R2}{\overset{H}{\mathbf{H}}}{\mathbf{H}}_{R2}\right)}^{-1}\right]}_{k,k}}\right)>\stackrel{~}{R},\forall k\in \{1,2\}\right\}, whose probability is the same as *P*(*E*_{R 1}), i.e.,

\begin{array}{l}P\left({E}_{R2}\right)=1-\frac{2{\gamma}^{{M}_{r}-1}}{({M}_{r}-1)!}.\end{array}

(A.4)

Then, the probability of the event *E* defined in Equation (2) can be revealed as P\left(E\right)=P\left({E}_{R1}\bigcap {E}_{R2}\right), where the events *E*_{R 1} and *E*_{R 2} are not strictly independent. Since reciprocal channel is assumed, the channel vectors {\mathbf{h}}_{1,{r}_{1}} and {\mathbf{h}}_{2,{r}_{2}} have one common element which is the channel coefficient between the *r*_{1}-th antenna of R_{1} and the *r*_{2}-th antenna of R_{2}, so that it is difficult to obtain the accurate value of *P*(*E*). However, when we notice that *P*(*E*) can be bounded as *P*(*E*_{R 1})+*P*(*E*_{R 2})−1≤*P*(*E*)≤*P*(*E*_{R 1}), we have

\begin{array}{l}P\left({E}_{R1}\right)-\frac{2{\gamma}^{{M}_{r}-1}}{({M}_{r}-1)!}\le P\left(E\right)\le P\left({E}_{R1}\right).\end{array}

(A.5)

According to Equation (A.3), Lemma 3 can be proved.

#### Proof of Lemma 4

Assume *X*_{1} and *X*_{2} to be independently exponentially distributed with unit variance without loss of generality. Let D=\left\{\right({x}_{1},{x}_{2})\in {\mathbb{R}}^{2+}|{x}_{1}{x}_{2}<a\} and {f}_{{x}_{1}{x}_{2}}({x}_{1},{x}_{2})={f}_{{x}_{1}}\left({x}_{1}\right){f}_{{x}_{2}}\left({x}_{2}\right) which is the joint density of {*x*_{1},*x*_{2}}, where {f}_{{x}_{i}}\left({x}_{i}\right) denotes *X*_{
i
}’s probability density function (PDF) {f}_{{x}_{i}}={e}^{-{x}_{i}}, *i*=1,2. Then, the probability *P*(*X*_{1}*X*_{2}<*a*) can be expressed as

\begin{array}{lcl}{\mathrm{\Phi}}_{a}& =& P\left({X}_{1}{X}_{2}<a\right)=\int {\int}_{D}{f}_{{x}_{1}{x}_{2}}({x}_{1},{x}_{2})\mathrm{d}{x}_{1}\mathrm{d}{x}_{2}\\ =& {\int}_{0}^{+\infty}{e}^{-x}(1-{e}^{-a/x})\mathrm{d}x\\ =& \sum _{i=1}^{\infty}\frac{{(-1)}^{i+1}}{i!}{a}^{i}{\int}_{0}^{+\infty}\frac{{e}^{-x}}{{x}^{i}}\mathrm{d}\mathrm{x.}\end{array}

(A.6)

Now, the function Φ_{
a
}(*u*) is first defined as

{\mathrm{\Phi}}_{a}\left(u\right)=\sum _{i=1}^{\infty}\frac{{(-1)}^{i+1}}{i!}{a}^{i}{B}_{i}\left(u\right),

(A.7)

where {B}_{i}\left(u\right)={\int}_{u}^{+\infty}\frac{{e}^{-x}}{{x}^{i}}\mathrm{d}x. Then, the *improper integral* Φ_{
a
} can be calculated as {\mathrm{\Phi}}_{a}={lim}_{u\to {0}^{+}}{\mathrm{\Phi}}_{a}\left(u\right).

Moreover, from [29] (Equation 3.351.4), *B*_{
i
}(*u*) can be calculated by the exponential integral function as

\begin{array}{lcl}{B}_{i}\left(u\right)& =& {(-1)}^{i}\frac{\text{Ei}(-u)}{(i-1)!}\\ +{e}^{-u}\sum _{n=0}^{i-2}\frac{{(-1)}^{n}{u}^{-(i-1-n)}}{(i-1)(i-2)\cdots (i-1-n)},\end{array}

(A.8)

where *i*≥2, and exponential integral function can be shown as \text{Ei}(-u)=ln\left(u\right)+{\sum}_{k=1}^{\infty}\frac{{(-u)}^{k}}{k\xb7k!}. Now that, *B*_{
i
}(*u*) can be obtained as

\phantom{\rule{-15.0pt}{0ex}}\frac{{B}_{i}\left(u\right)}{{e}^{-u}}\phantom{\rule{0.3em}{0ex}}=\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}\left\{\phantom{\rule{0.3em}{0ex}}\begin{array}{ll}-ln\left(u\right)\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}o\left(1\right)& \text{if}\phantom{\rule{1em}{0ex}}\phantom{\rule{0.3em}{0ex}}i=1\\ \frac{{(-1)}^{i}ln\left(u\right)}{(i-1)!}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\sum}_{n=0}^{i-2}\frac{{(-1)}^{n}{u}^{-(i-1-n)}}{(i-1)(i-2)\cdots (i-1-n)}+o\left(1\right)& \text{if}\phantom{\rule{1em}{0ex}}\phantom{\rule{0.3em}{0ex}}i\ge 2\end{array}\right.\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}},

(A.9)

where *o*(1)→0 when *u*→0^{+}. By substituting *B*_{
i
}(*u*) into Equation (A.7) and rearranging the infinite series, Φ_{
a
}(*u*) can be rewritten as

{\mathrm{\Phi}}_{a}\left(u\right)={e}^{-u}\sum _{n=0}^{\infty}{\varphi}_{n}\left(u\right)+o\left(1\right),

(A.10)

where

\begin{array}{ll}{\varphi}_{n}\left(u\right)=& \frac{-{a}^{n+1}}{n!(n+1)!}ln\left(u\right)\\ +\sum _{i=n+2}^{\infty}\frac{{(-1)}^{i+n+1}{a}^{i}{u}^{-(i-1-n)}}{i!(i-1)(i-2)\cdots (i-1-n)}.\end{array}

(A.11)

Let *v*=*a* *u*^{−1}, *ϕ*_{
n
}(*v*) can be expressed as

{\varphi}_{n}\left(v\right)=\frac{-{a}^{n+1}lna}{n!(n+1)!}+{a}^{n+1}{F}_{n}\left(v\right),

(A.12)

where

\phantom{\rule{-14.0pt}{0ex}}{F}_{n}\left(v\right)=\frac{lnv}{n!(n+1)!}+\sum _{i=n+2}^{\infty}\frac{{(-1)}^{i+n+1}{v}^{i-1-n}}{i!(i-1)(i-2)\cdots (i-1-n)}.

(A.13)

Following similar proof steps of Corollary 1 in [15], the limit {lim}_{v\to +\infty}{F}_{n}\left(v\right) can be proved to exist. Hence, by recalling Equation (A.10), Φ_{
a
} can be expressed as

{\mathrm{\Phi}}_{a}={lim}_{u\to {0}^{+}}{\mathrm{\Phi}}_{a}\left(u\right)=\sum _{n=0}^{\infty}\left[\frac{-{a}^{n+1}lna}{n!(n+1)!}+{C}_{n}{a}^{n+1}\right].

(A.14)

For a sufficiently large *ρ*, Φ_{
a
} can be approximately calculated as Φ_{
a
}≈−*a* ln*a*.

#### Proof of Lemma 5

It is not difficult to obtain the PDF of *z* as {f}_{z}\left(z\right)={M}_{r}{e}^{-z}{(1-{e}^{-z})}^{{M}_{r}-1}, *z*>0, since z=max\{{z}_{1},\cdots \phantom{\rule{0.3em}{0ex}},{z}_{{M}_{r}}\} and {f}_{{z}_{r}}\left({z}_{r}\right)={e}^{-{z}_{r}}, *r*∈{1,⋯,*M*_{
r
}}, so that

\begin{array}{lcr}\epsilon \left[{e}^{-\mathit{\text{cz}}}\right]& =& {\int}_{0}^{\infty}{e}^{-\mathit{\text{cz}}}{f}_{z}\left(z\right)\mathrm{d}z={M}_{r}{\int}_{0}^{1}{t}^{c}{(1-t)}^{{M}_{r}-1}\mathrm{d}\mathrm{t.}\end{array}

(A.15)

Using binomial theorem, *ε*[*e*^{−cz}] can be written as

\begin{array}{lcl}\epsilon \left[{e}^{-\mathit{\text{cz}}}\right]& =& {M}_{r}\sum _{r=0}^{{M}_{r}-1}{\int}_{0}^{1}{C}_{{M}_{r}-1}^{r}{(-1)}^{r}{t}^{c+r}\mathrm{d}t\\ =& {M}_{r}\sum _{r=0}^{{M}_{r}-1}{C}_{{M}_{r}-1}^{r}\frac{{(-1)}^{r}}{c+r+1}\end{array}

(A.16)

where {C}_{p}^{q}=\frac{p!}{(p-q)!q!}, *p* and *q* are positive integers, and *p*≥*q*. Splitting each term on the right side, Equation (A.16) can be calculated as

\begin{array}{lcl}\frac{\epsilon \left[{e}^{-\mathit{\text{cz}}}\right]}{{M}_{r}}& =& \left(\frac{1}{c+1}-\frac{1}{c+2}\right)+\left(-1\right)\xb7\underset{{C}_{{M}_{r}-2}^{1}}{\underset{\u23df}{\left({M}_{r}-2\right)}}\\ \times \phantom{\rule{0.3em}{0ex}}\left(\phantom{\rule{0.3em}{0ex}}\frac{1}{c\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}2}\phantom{\rule{0.3em}{0ex}}-\phantom{\rule{0.3em}{0ex}}\frac{1}{c\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}3}\phantom{\rule{0.3em}{0ex}}\right)\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}\cdots \phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\left(-1\right)}^{r}\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}\xb7\phantom{\rule{0.3em}{0ex}}\underset{{C}_{{M}_{r}-2}^{r}}{\underset{\u23df}{\phantom{\rule{0.3em}{0ex}}\left(\phantom{\rule{0.3em}{0ex}}{C}_{{M}_{r}-1}^{r}\phantom{\rule{0.3em}{0ex}}-\phantom{\rule{0.3em}{0ex}}{C}_{{M}_{r}-2}^{r-1}\phantom{\rule{0.3em}{0ex}}\right)}}\\ \times \left(\frac{1}{c+r+1}-\frac{1}{c+r+2}\right)\phantom{\rule{0.3em}{0ex}}+\cdots +\phantom{\rule{0.3em}{0ex}}{\left(-1\right)}^{{M}_{r}-2}\\ \times \left(\frac{1}{c+{M}_{r}-1}-\frac{1}{c+{M}_{r}}\right)\\ =& \sum _{r=0}^{{M}_{r}-2}{C}_{{M}_{r}-2}^{r}\frac{{\left(-1\right)}^{r}}{\left(c+r+1\right)\left(c+r+2\right)}\end{array}

(A.17)

Furthermore, by iteratively repeating the similar process in Equation (A.17), the expectation of *e*^{−cz} can be finally obtained as

\epsilon \left[{e}^{-\mathit{\text{cz}}}\right]=\frac{{M}_{r}!}{{\mathrm{\Pi}}_{i=1}^{{M}_{r}}(c+i)}.

### Appendix 2

#### Proof of theorem 1

As shown in [7, 22], the ML error probability can be tightly bounded by the outage probability at high SNR, so the outage probability will be analyzed in this section. According to [22] and the protocol description in Section 2, we can define the outage event of the proposed ARDF protocol as \mathcal{O}\triangleq \bigcup _{k=0}^{2}{\mathcal{O}}_{{A}_{k}}, where {\mathcal{O}}_{{A}_{2}}\triangleq {\mathcal{O}}_{{A}_{2},\overline{E}}\bigcup {\mathcal{O}}_{{A}_{2},E}. Here, {\mathcal{O}}_{{A}_{k}} denotes the outage event in the situation *A*_{
k
} at the destination, {\mathcal{O}}_{{A}_{2},\overline{E}} and {\mathcal{O}}_{{A}_{2},E} are similarly defined. Thus, the overall outage probability of the proposed protocol can be expressed as

\begin{array}{ll}\phantom{\rule{-15.0pt}{0ex}}P\left(\mathcal{O}\right)& =\sum _{k=0}^{1}P\left({\mathcal{O}}_{{A}_{k}}\right)P\left({A}_{k}\right)+P({A}_{2},\overline{E})P\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}+P({A}_{2},E)P\left({\mathcal{O}}_{{A}_{2},E}\right)\phantom{\rule{2em}{0ex}}\\ \le \sum _{k=0}^{1}P\left({\mathcal{O}}_{{A}_{k}}\right)P\left({A}_{k}\right)\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}P\left(\overline{E}\right)P\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}P\left(E\right)P\left({\mathcal{O}}_{{A}_{2},E}\right).\phantom{\rule{2em}{0ex}}\end{array}

(B.1)

The probabilities of each situation *A*_{
k
} and the event *E* have been presented in Lemma 2 and Lemma 3, respectively. Moveover, the outage probability in each situation will be analyzed in the terms of the MAC capacity region. Based on the definition in Equation (1), the source data rate constrains for the proposed-ARDF protocol can be further calculated as follows:

\begin{array}{l}\left|\mathcal{S}\right|Q\stackrel{~}{R}<log\left(det\left(\mathbf{I}+\rho \sum _{l\in \mathcal{\mathcal{L}}}{\mathbf{h}}_{l}\underset{l}{\overset{H}{\mathbf{h}}}\right)\right),\forall \mathcal{S}\subseteq \{1,2\}\end{array}

(B.2)

where \left|\mathcal{S}\right| denotes number of users in , *Q* denotes the number of codewords transmitted by each source in one cooperative frame, and **h**_{
l
} is a channel vector, both the structure of **h**_{
l
} and the set are a function of and the details of their relationship to will be discussed in the next few subsections. The outage events occur when any constraint in Equation (B.2) is not met, and the highest outage probability achieved by each constraint is the dominant factor [22]. In the following subsections, different values of each parameter in Equation (B.2) will be considered for different situations.

#### Situation *A*_{0}

\mathcal{S}\subseteq \{1,2\}, *Q*=*L*, and **h**_{
l
} denotes the *l*-th column vector of {\mathbf{H}}_{{A}_{0}} in Table 1. When \left|\mathcal{S}\right|=1, is assumed to be {1} without loss of generality, so \mathcal{\mathcal{L}}=\{1,3,\cdots \phantom{\rule{0.3em}{0ex}},2L-1\}, and the outage probability at the destination in such a case can be calculated as

\begin{array}{ll}\phantom{\rule{-15.0pt}{0ex}}{P}_{1}\left({\mathcal{O}}_{{A}_{0}}\right)& =P\left\{\left[1\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}\rho ({X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\alpha}_{2}^{2}{X}_{2})\right]\phantom{\rule{0.3em}{0ex}}{\left[1\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}\rho ({\alpha}_{1}^{2}{X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\alpha}_{2}^{2}{X}_{2})\right]}^{L-1}\right.\phantom{\rule{2em}{0ex}}\\ \phantom{\rule{1em}{0ex}}<\left(\right)close="\}">{2}^{L\stackrel{~}{R}}& \stackrel{\left(a\right)}{}P\left\{{\left[1+\rho ({\alpha}_{1}^{2}{X}_{1}+{\alpha}_{2}^{2}{X}_{2})\right]}^{L}{2}^{L\stackrel{~}{R}}\right\}\phantom{\rule{2em}{0ex}}\end{array}\n \n \n \u2264\n P\n \n \n \n \n \alpha \n \n \n 1\n \n \n 2\n \n \n \n \n X\n \n \n 1\n \n \n \n \n \n \n \n 2\n \n \n \n \n R\n \n ~\n \n \n \n \u2212\n 1\n \n \n \rho \n \n \n ,\n \n \n \alpha \n \n \n 2\n \n \n 2\n \n \n \n \n X\n \n \n 2\n \n \n \n \n \n \n \n 2\n \n \n \n \n R\n \n ~\n \n \n \n \u2212\n 1\n \n \n \rho \n \n \n \n \n \n \n \n \n \n \n \n \u2264\n \n \n \n (\n b\n )\n \n \n \n P\n \n \n \n \n X\n \n \n 1\n \n \n \n \n \n \n \n 2\n \n \n \n \n R\n \n ~\n \n \n \n \u2212\n 1\n \n \n \n \n \alpha \n \n \n 1\n \n \n 2\n \n \n \rho \n \n \n \n \n P\n \n \n \n \n X\n \n \n 2\n \n \n \n \n \n \n \n 2\n \n \n \n \n R\n \n ~\n \n \n \n \u2212\n 1\n \n \n \n \n \alpha \n \n \n 2\n \n \n 2\n \n \n \rho \n \n \n \n \n \n \n \n \n \n \u2250\n \n \n \rho \n \n \n \u2212\n 2\n (\n 1\n \u2212\n \n \n r\n \n ~\n \n )\n \n \n ,\n \n \n \n

(B.3)

where *X*_{
i
}=|*h*_{
Si
}|^{2}, *i*=1,2; (*a*) holds since 0<{\alpha}_{1}^{2}<1, and (*b*) holds since *X*_{1} is independent of *X*_{2}. Otherwise, \mathcal{S}=\{1,2\} and \mathcal{\mathcal{L}}=\{1,2,\cdots \phantom{\rule{0.3em}{0ex}},2L\},

{P}_{2}\left({\mathcal{O}}_{{A}_{0}}\right)=P\left\{log\left(det({\mathbf{I}}_{2L+1}+\rho {\mathbf{H}}_{{A}_{0}}\underset{{A}_{0}}{\overset{H}{\mathbf{H}}})\right)<2L\stackrel{~}{R}\right\}.

(B.4)

In order to make the analysis more tractable, a (2*L*+1)×(2*L*+1) square matrix {\stackrel{~}{\mathbf{H}}}_{{A}_{0}} is first defined as {\stackrel{~}{\mathbf{H}}}_{{A}_{0}}\triangleq [{\mathbf{H}}_{{A}_{0}},\mathbf{0}], where the zero column vector is (2*L*+1)-dimensional. It is easy to show that {\mathbf{H}}_{{A}_{0}}{\mathbf{H}}_{{A}_{0}}^{H}={\stackrel{~}{\mathbf{H}}}_{{A}_{0}}{\stackrel{~}{\mathbf{H}}}_{{A}_{0}}^{H} and [{\mathbf{I}}_{2L+1}+\rho {\stackrel{~}{\mathbf{H}}}_{{A}_{0}}{\stackrel{~}{\mathbf{H}}}_{{A}_{0}}^{H}] is a tridiagonal matrix. According to [30], the determinant of the tridiagonal matrix can be shown iteratively as

{D}_{n}=[1+\rho ({x}_{n}+{y}_{n-1}\left)\right]{D}_{n-1}-{\rho}^{2}{x}_{n}{y}_{n-1}{D}_{n-2},

where {D}_{n}=det[{\mathbf{I}}_{2L+1}+\rho {\stackrel{~}{\mathbf{H}}}_{n}{\stackrel{~}{\mathbf{H}}}_{n}^{H}], and {\stackrel{~}{\mathbf{H}}}_{n} denotes the *n*×*n* top-left submatrix from {\stackrel{~}{\mathbf{H}}}_{{A}_{0}}, *x*_{
n
} and *y*_{
n
} are the *n*-th element on the principle diagonal and subdiagonal of {\stackrel{~}{\mathbf{H}}}_{{A}_{0}}, respectively. By using such a property and note that *x*_{2L+1}=0, the following inequality can be obtained

\begin{array}{ll}{D}_{2L+1}\ge & \left(1+\rho {X}_{1}\right){\left(1+\rho {\alpha}_{1}{X}_{1}\right)}^{L-1}{\left(1+\rho {\alpha}_{1}{X}_{2}\right)}^{L}\\ +{\alpha}_{2}^{2L-1}{\left({\rho}^{2}{X}_{1}{X}_{2}\right)}^{L}.\end{array}

(B.5)

From Equation (B.5) and Lemma 4, {P}_{2}\left({\mathcal{O}}_{{A}_{0}}\right) can be upper bounded as

\begin{array}{ll}{P}_{2}\left({\mathcal{O}}_{{A}_{0}}\right)& \le P\left\{{\left({\rho}^{2}{\alpha}_{2}^{2}{X}_{1}{X}_{2}\right)}^{L}<{2}^{2L\stackrel{~}{R}}\right\}\phantom{\rule{2em}{0ex}}\\ \approx -\frac{{2}^{2\stackrel{~}{R}}}{{\alpha}_{2}^{2}{\rho}^{2}}ln\left(\frac{{2}^{2\stackrel{~}{R}}}{\underset{2}{\overset{2}{\alpha}}{\rho}^{2}}\right)\phantom{\rule{2em}{0ex}}\\ \doteq {\rho}^{-2(1-\stackrel{~}{r})}.\phantom{\rule{2em}{0ex}}\end{array}

(B.6)

Thus, P\left({\mathcal{O}}_{{A}_{0}}\right)\stackrel{\u0307}{\le}{\rho}^{-2(1-\stackrel{~}{r})} can be easily obtained by combining {P}_{1}\left({\mathcal{O}}_{{A}_{0}}\right) and {P}_{2}\left({\mathcal{O}}_{{A}_{0}}\right).

#### Situation *A*_{1}

\mathcal{\mathcal{L}}=\mathcal{S}\subseteq \{1,2\}, *Q*=1, **h**_{
l
} denotes the *l*-th column vector of {\mathbf{H}}_{{A}_{1}} in Table 1. When \left|\mathcal{S}\right|=1, assume that \mathcal{S}=\left\{1\right\}, {P}_{1}\left({\mathcal{O}}_{{A}_{1}}\right) can be easily obtained as

\begin{array}{lcl}{P}_{1}\left({\mathcal{O}}_{{A}_{1}}\right)& =& P\left\{{X}_{1}+{\alpha}_{2}^{2}{X}_{2}+{G}_{1}<\frac{{2}^{\stackrel{~}{R}}-1}{\rho}\right\}\\ \stackrel{\u0307}{\le}& {\rho}^{-2(1-\stackrel{~}{r})}\prod _{m=1}^{{M}_{r}}P\left\{|{g}_{1,m}{|}^{2}<\frac{{2}^{\stackrel{~}{R}}-1}{\rho}\right\}\\ \doteq & {\rho}^{-({M}_{r}+2)(1-\stackrel{~}{r})},\end{array}

(B.7)

where {G}_{j}=|{g}_{j,{r}_{j}}{|}^{2}\triangleq max\{|{g}_{j,1}{|}^{2},\cdots \phantom{\rule{0.3em}{0ex}},|{g}_{j,{M}_{r}}{|}^{2}\},j=1,2. Otherwise, \mathcal{S}=\{1,2\}, {P}_{2}\left({\mathcal{O}}_{{A}_{1}}\right) is

{P}_{2}\left({\mathcal{O}}_{{A}_{1}}\right)=P\left\{log\left(det({\mathbf{I}}_{2}+\rho \underset{{A}_{1}}{\overset{H}{\mathbf{H}}}{\mathbf{H}}_{{A}_{1}})\right)<2\stackrel{~}{R}\right\}.

(B.8)

The determinant of [{\mathbf{I}}_{2}+\rho {\mathbf{H}}_{{A}_{1}}^{H}{\mathbf{H}}_{{A}_{1}}] can be first calculated as

\begin{array}{ll}det\left({\mathbf{I}}_{2}+\rho {\mathbf{H}}_{{A}_{1}}^{H}{\mathbf{H}}_{{A}_{1}}\right)=& \prod _{i=1}^{2}\left[1+\rho ({X}_{1}+{G}_{1}+{\alpha}_{i}^{2}{X}_{2})\right]\\ -{\rho}^{2}{\left|{\alpha}_{1}{\alpha}_{2}{X}_{2}+{g}_{M1}^{\ast}{h}_{S1}\right|}^{2},\end{array}

(B.9)

where

\begin{array}{lcr}|{\alpha}_{1}{\alpha}_{2}{X}_{2}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{g}_{M1}^{\ast}{h}_{S1}{|}^{2}\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}& \le & \phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}{\left({\alpha}_{1}{\alpha}_{2}{X}_{2}\right)}^{2}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{1}{X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}2{\alpha}_{1}{\alpha}_{2}{X}_{2}\sqrt{{G}_{1}{X}_{1}}\\ \le & \phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}{\left({\alpha}_{1}{\alpha}_{2}{X}_{2}\right)}^{2}\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{1}{X}_{1}\phantom{\rule{0.3em}{0ex}}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\alpha}_{1}{\alpha}_{2}{X}_{2}\left({X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{1}\right)\phantom{\rule{0.3em}{0ex}}.\phantom{\rule{0.3em}{0ex}}\end{array}

(B.10)

So that det\left({\mathbf{I}}_{2}+\rho {\mathbf{H}}_{{A}_{1}}^{H}{\mathbf{H}}_{{A}_{1}}\right)\phantom{\rule{0.3em}{0ex}}>\phantom{\rule{0.3em}{0ex}}{\rho}^{2}\left[{G}_{1}^{2}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}(1\phantom{\rule{0.3em}{0ex}}-\phantom{\rule{0.3em}{0ex}}{\alpha}_{1}{\alpha}_{2}){X}_{1}{X}_{2}\right] can be obtained, where 0<{\alpha}_{1}{\alpha}_{2}\le \frac{1}{2}, and {P}_{2}\left({\mathcal{O}}_{{A}_{1}}\right) can be upper bounded as

\begin{array}{lcl}{P}_{2}\left({\mathcal{O}}_{{A}_{1}}\right)& \le & P\left\{{G}_{1}^{2}+(1-{\alpha}_{1}{\alpha}_{2}){X}_{1}{X}_{2}<\frac{{2}^{2\stackrel{~}{R}}}{{\rho}^{2}}\right\}\\ \le & P\left\{{X}_{1}{X}_{2}<\frac{{2}^{2\stackrel{~}{R}}}{(1-{\alpha}_{1}{\alpha}_{2}){\rho}^{2}}\right\}P\left\{{G}_{1}<\frac{{2}^{\stackrel{~}{R}}}{\rho}\right\}\\ \doteq & {\rho}^{-({M}_{r}+2)(1-\stackrel{~}{r})},\end{array}

(B.11)

where the last relationship is based on Lemma 4. Hence, P\left({\mathcal{O}}_{{A}_{1}}\right)\stackrel{\u0307}{\le}{\rho}^{-({M}_{r}+2)(1-\stackrel{~}{r})} can be obtained.

#### Situation *A*_{2}event \overline{E}

\mathcal{\mathcal{L}}=\mathcal{S}\subseteq \{1,2\}, *Q*=1, **h**_{
l
} denotes the *l*-th column vector of {\mathbf{H}}_{{A}_{2},\overline{E}} in Table 1. When \left|\mathcal{S}\right|=1, {P}_{1}\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)\doteq {\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})} can be easily obtained by following the similar steps in Equation (B.7). Otherwise, \mathcal{S}=\{1,2\}, {P}_{2}\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right) can be written as

{P}_{2}\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)=P\left\{log\left(det({\mathbf{I}}_{2}+\rho \underset{{A}_{2},\overline{E}}{\overset{H}{\mathbf{H}}}{\mathbf{H}}_{{A}_{2},\overline{E}})\right)<2\stackrel{~}{R}\right\}.

(B.12)

In the above equation, det\left({\mathbf{I}}_{2}+\rho {\mathbf{H}}_{{A}_{2},\overline{E}}^{H}{\mathbf{H}}_{{A}_{2},\overline{E}}\right)={K}_{1}-{K}_{2}, where

\begin{array}{lcl}{K}_{1}& =& \prod _{i=1}^{2}[1+\rho ({X}_{1}+{G}_{1}+{G}_{2}+{\alpha}_{i}^{2}{X}_{2}\left)\right]\end{array}

(B.13)

\begin{array}{lc}\ge & {\rho}^{2}\left[{({X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{2}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{1})}^{2}+{X}_{2}({X}_{1}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{2}\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{G}_{1})\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}{\alpha}_{1}{\alpha}_{2}{X}_{2}^{2}\right],\\ {K}_{2}& =& {\rho}^{2}{\left|{\alpha}_{1}{\alpha}_{2}{X}_{2}+{g}_{M1}^{\ast}{h}_{S1}+{g}_{M2}^{\ast}{g}_{M1}\right|}^{2}\\ \le & {\rho}^{2}{\left[\left({\alpha}_{1}{\alpha}_{2}{X}_{2}\right)+\sqrt{{G}_{1}}(\sqrt{{X}_{1}}+\sqrt{{G}_{2}})\right]}^{2}\\ \le & {\rho}^{2}\left[{\left({\alpha}_{1}{\alpha}_{2}{X}_{2}\right)}^{2}+{\alpha}_{1}{\alpha}_{2}{X}_{2}({X}_{1}+{G}_{2}+2{G}_{1})\right.\\ +\left(\right)close="]">2{G}_{1}({X}_{1}+{G}_{2})& ,\end{array}\n

(B.14)

so that det\left({\mathbf{I}}_{2}+\rho {\mathbf{H}}_{{A}_{2},\overline{E}}^{H}{\mathbf{H}}_{{A}_{2},\overline{E}}\right)>{\rho}^{2}\left[{G}_{1}^{2}+{G}_{2}^{2}+(1-{\alpha}_{1}\right.\left(\right)close="]">\n \n \n \n \alpha \n \n \n 2\n \n \n )\n \n \n X\n \n \n 1\n \n \n \n \n X\n \n \n 2\n \n \n \n can be obtained. By using Lemma 4 and following the similar steps in Equation (B.11), {P}_{2}\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right) can be upper bounded as

\begin{array}{lcl}{P}_{2}\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)& \le & P\left\{{G}_{1}^{2}+{G}_{2}^{2}+(1-{\alpha}_{1}{\alpha}_{2}){X}_{1}{X}_{2}<\frac{{2}^{2\stackrel{~}{R}}}{{\rho}^{2}}\right\}\\ \doteq & {\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})}.\end{array}

(B.15)

Therefore, it can be shown that P\left({\mathcal{O}}_{{A}_{2},\overline{E}}\right)\stackrel{\u0307}{\le}{\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})}.

#### Situation *A*_{2}event *E*

The outage probability in situation *A*_{2} event *E* is difficult to be obtained using the above method, but the upper bound can be calculated by following the similar analysis in [17]. Firstly, model 4 in Table 1 is assumed to be a symmetric 2*L*-user multiple-access system where the codeword *x*(*n*) in **x**_{2L} is transmitted by S_{
n
}, and such an assumption will make the analysis tractable. For the two-user case considered in this paper, the performance would not be worse than the performance of the former one. So that \mathcal{\mathcal{L}}=\mathcal{S}\subseteq \{1,2,\cdots \phantom{\rule{0.3em}{0ex}},2L\}, *Q*=1, **h**_{
l
} denotes the *l*-th column vector of **H** in Equation (3), and there are (2^{2L}−1) source data constrains in Equation (B.2). For each constraint, there exists a probability that the channel condition cannot satisfy it, and the highest outage probability is the dominant factor and achieve the system’s DMT.

In order to calculate the outage probability of each constraint, a (*m*+3)×*m* MIMO channel is first considered as **y**_{
m
}=**F**_{
m
}**s**_{
m
}+**w**_{
m
}, where {\mathbf{s}}_{m}\in {\mathcal{C}}^{m},{\mathbf{y}}_{m},{\mathbf{w}}_{m}\in {\mathcal{C}}^{m+3} and **F**_{
m
} is the (*m*+3)×*m* top-left submatrix from **H** in Equation (3). According to [17], the outage probability achieved by this system for every 1≤*m*≤2*L* is the same as the highest outage probability for each constraint in Equation (B.2). When *m*=1, {P}_{1}\left({\mathcal{O}}_{{A}_{2},E}\right)\doteq {\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})} can be easily obtained, following the similar steps in Equation (B.7).

When *m*>1, following the similar DMT analysis for the inter-symbol interference (ISI) channel in [31] and the proof steps in [17], the average error probability can be upper bounded by

{P}_{e}\le 2(m+3)\xb7\epsilon [exp(-{c}_{1}\overline{\lambda}{\rho}^{1-\stackrel{~}{r}}\left|\mathbf{f}{|}^{2}\right)],

(B.16)

where *c*_{1} is a constant, \mathbf{f}={[{h}_{S1},{h}_{S2},{g}_{1,{r}_{1}},{g}_{2,{r}_{2}}]}^{T}, *ε*[·] denotes the expectation of a random variable, and \overline{\lambda}=\underset{\mathbf{f}\in {\mathcal{C}}^{4}}{inf}{\lambda}_{min}\left(\frac{{\mathbf{F}}_{m}}{\left|\mathbf{f}\right|}\right)>0, *λ*_{min}(·) denotes the minimum singular value of a matrix. By using Lemma 5, *P*_{
e
} can be upper bounded as

\begin{array}{lcl}{P}_{e}& \le & 2(m+3){\left(\frac{1}{{c}_{1}\overline{\lambda}{\rho}^{1-\stackrel{~}{r}}+1}\right)}^{2}{\left[\frac{{M}_{r}!}{{\mathrm{\Pi}}_{i=1}^{{M}_{r}}({c}_{1}\overline{\lambda}{\rho}^{1-\stackrel{~}{r}}+i)}\right]}^{2}\\ \doteq & {\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})}.\end{array}

(B.17)

By observing the fact that P\left({\mathcal{O}}_{{A}_{2},E}\right)\stackrel{\u0307}{\le}{P}_{e}[7], the outage probability of model 4 in situation *A*_{2} event *E* can be upper bounded as

P\left({\mathcal{O}}_{{A}_{2},E}\right)\stackrel{\u0307}{\le}{\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})}.

Now that recalling Equation (B.1), Lemma 2, and Lemma 3 and considering all the situations, the overall outage probability can be written as

\phantom{\rule{-10.0pt}{0ex}}P\left(\mathcal{O}\right)\stackrel{\u0307}{\le}{\rho}^{-min\left\{\right(2{M}_{r}+2\left)\right(1-\stackrel{~}{r}),(3{M}_{r}+1\left)\right(1-\stackrel{~}{r}\left)\right\}}\doteq {\rho}^{-(2{M}_{r}+2)(1-\stackrel{~}{r})}.

(B.18)

Depending on the variable-rate strategy in Equation (34) of [8] and integrating the four transmission modes, the target transmission data rate *R* BPCU can be expressed as

\phantom{\rule{-14.0pt}{0ex}}R\phantom{\rule{0.3em}{0ex}}=\phantom{\rule{0.3em}{0ex}}P\left({A}_{0}\right)\frac{\left(2L\right)\stackrel{~}{R}}{2L+1}+P\left({A}_{1}\right)\frac{\stackrel{~}{R}}{2}+P({A}_{2},\overline{E})\frac{2\stackrel{~}{R}}{5}+P({A}_{2},E)\frac{\left(2L\right)\stackrel{~}{R}}{2L\phantom{\rule{0.3em}{0ex}}+\phantom{\rule{0.3em}{0ex}}3}.

(B.19)

One can also refer to [8] to get the mapping criterion from *R* to \stackrel{~}{R}. It is not difficult to prove the inequality P\left(\overline{E}\right)+P\left({A}_{2}\right)-1\le P({A}_{2},\overline{E})\le P\left(\overline{E}\right), so P({A}_{2},\overline{E})\doteq P\left(\overline{E}\right) can be obtained in the large-SNR region. From Equation (A.2) and Lemma 3 and substituting *R*=*r* log*ρ*, \stackrel{~}{R}=\stackrel{~}{r}log\rho, P({A}_{2},E)=P\left({A}_{2}\right)-P({A}_{2},\overline{E}) into Equation (B.19), *r* can be revealed as

\begin{array}{ll}r\doteq & \stackrel{~}{r}\left\{\frac{2L}{2L+1}\underset{1}{\overset{2}{P}}+{P}_{1}(1-{P}_{1})+\frac{2}{5}{P}_{2}\right.\\ +\left(\right)close="\}">\frac{2L}{2L+3}[{(1-{P}_{1})}^{2}-{P}_{2}]& ,\end{array}\n

(B.20)

where {P}_{1}={\rho}^{-{M}_{r}(1-\stackrel{~}{r})},{P}_{2}={\rho}^{-({M}_{r}-1)(1-\stackrel{~}{r})}. Following the analysis in Claim 3 of [8] and the similar steps in [14], d\left(r\right)=(2{M}_{r}+2){\left(1-\frac{2L+3}{2L}r\right)}^{+} can be proved by substituting \stackrel{~}{r}=\frac{2L+3}{2L}r into Equation (B.18).