Performance Analysis of Relay-Aided Hybrid FSO/RF Cooperation Communication System over the Generalized Turbulence Channels with Pointing Errors and Nakagami-m Fading Channels

Abstract

To improve the performance of fee-space optical communication systems, this paper analyzes the performance of a relay-aided hybrid fee-space optical (FSO)/radio frequency (RF) cooperation system based on a selective combination and decoding forward transmission scheme. In this system, the FSO sub-link experienced Málaga turbulence with pointing errors and the RF sub-link suffered Nakagami-m fading. Firstly, the probability density function (PDF) and cumulative distribution function (CDF) of the end-to-end output signal-to-noise ratio (SNR) of the relay-aided hybrid FSO/RF system are derived. Then, using the extended generalized bivariate Meijer’s G-function (EGBMGF) and the approximate analytical formula of the generalized Gauss–Laguerre integral, mathematical expressions of the end-to-end average bit error rate (ABER) and outage probability of the relay-aided hybrid FSO/RF system with different subcarrier intensity modulation and different detection schemes are derived. Through a simulation analysis of the system, the results show that compared with the other three modulation technologies, the hybrid FSO/RF direct link and relay-aided hybrid FSO/RF system with coherent binary phase shift keying (CBPSK) modulation have the best bit error performance. Compared with direct detection, the hybrid direct link and relay-aided hybrid system with coherent detection can significantly improve the communication performance. Increasing the RF fading parameter m can further improve the bit error and outage performance of the hybrid direct link and relay-aided hybrid system; the hybrid direct link can significantly mitigate the degradation of communication performance in the FSO system caused by pointing errors, and the relay-aided hybrid system can further improve the communication performance; under weak turbulence conditions, the impact of pointing errors on the performance of the relay-aided hybrid system can even be ignored. The greater the total number of paths in the relay-aided hybrid system, the better the communication performance of the system; however, the more hops, the worse the performance of the system. The outage probability of the hybrid direct link and relay-aided hybrid system are very sensitive to the decision threshold, and the larger the decision threshold, the worse the outage performance. The transmission distance of different hybrid direct links has little impact on the performance of hybrid direct links and relay-aided hybrid systems. Improving the signal-to-noise ratio of RF sub-links significantly improves the performance of hybrid direct links and relay-aided hybrid systems.

1. Introduction

In recent years, free-space optical (FSO) communication systems have combined the advantages of microwave communication and optical fiber communication. They have the characteristics of large transmission capacity, fast network construction, being license-free, good security, and so on. They are usually used as the “last-kilometer” of telecommunications or LAN links between buildings [1]. However, the performance of terrestrial FSO communication systems is not only affected by atmospheric channels, such as the weather (cloud, fog, rain, snow, haze, etc.) and optical turbulence, but also affected by the pointing errors [2,3].

In order to mitigate the impact of the atmospheric environment on the performance of terrestrial FSO communication systems and improve the reliability and availability of terrestrial FSO communication systems, a hybrid FSO/radio frequency (RF) transmission system combining an FSO link and a millimeter-wave RF link was proposed [4]. Usman et al. [5] proposed a hybrid FSO/RF transmission scheme based on hard switching (only one link transmits information at the same time) and derived the expressions of the outage probability, average bit error rate (ABER), and ergodic channel capacity (ECC) of the system. Shrivastava et al. [6] proposed a hybrid FSO/RF system with a switching scheme according to the channel state information (CSI), and deduced the closed-form expressions of the ABER and outage probability under a strong turbulent environment. Shakir [7] derived new closed-form expressions of the BER and outage probability of a hybrid FSO/RF parallel communication system based on selective combination technology. The results showed that the performance of the hybrid FSO/RF parallel system was better than that of only the FSO system or only the RF system under different atmospheric environments.

In order to improve the transmission distance and coverage of terrestrial FSO communication systems, a multi-hop network based on FSO communication was proposed [8]. Zedini et al. [9] analyzed the performance of the multi-hop FSO system based on amplify-and-forward (AF) transmission and the channel state information (CSI) assisted or fixed gain relay, and derived new closed-form expressions for the end-to-end ABER and ECC of the system under different binary modulation schemes. Wang et al. [10] proposed a multi-hop FSO communication system based on decode-and-forward (DF) transmission and orthogonal frequency division multiplexing (OFDM), and derived closed expressions of the outage probability and ABER of the system.

In order to further improve the transmission performance of multi-hop networks, a parallel multi-hop cooperative communication network was proposed, that is, a relay-aided system [11]. Gao et al. [12] deduced the mathematical expressions of the ABER and outage probability of the relay-aided multi-hop network based on binary phase shift keying (BPSK) modulation under Gamma–Gamma turbulence with pointing errors. Wang et al. [13] deduced the end-to-end ABER and outage probability of the multi-hop parallel FSO system under the Málaga turbulent channel with pointing errors and a DF relay scheme, and analyzed the performance of the system under different turbulent intensity, weather conditions, pointing errors, and relay-aided structures. Androutsos et al. [14] minimized the mathematical complexity in analyzing outage and error performance by emulating a multi-hop terrestrial FSO communication system with various modulation formats as a single dual-hop scheme. Mohammad et al. [15,16] proposed a multi-user multi-hop hybrid FSO/RF communication system, and deduced and analyzed the mathematical expressions of the system performance.

Thus, motivated by the above-mentioned facts and considering its military and disaster recovery applications, we first proposed a new communication scheme combining the hybrid FSO/RF parallel transmission and the multi-hop parallel relay-aided communication in Ref. [17], that is, the multi-hop parallel hybrid FSO/RF cooperation system or the relay-aided hybrid FSO/RF system. Compared with previous work, the innovative content of this article includes the following points:

• First, we used the gamma–gamma turbulence model in our previous work, while we used the more general Málaga turbulence model in this paper, resulting in a new performance expression.
• Second, due to model limitations, we only derived and analyzed the system performance under CBPSK and DBPSK modulation methods in the previous work. In this paper, we derived and analyzed the system performance under four modulation methods (CBPSK, DBPSK, CBFSK, NBFSK).
• Third, we only derived and analyzed the system performance under direct detection in the previous work. In this paper, we derived and compared the system performance under direct detection and coherent detection.
• Finally, we derived and analyzed the relationship between the system outage probability and signal-to-noise ratio in the previous work. In this paper, we derived and analyzed the relationship between the system outage probability and the decision threshold.

The remainder of this paper, considers direct detection (DD) and heterodyne detection (HD), the PDF and CDF of the output SNR of only the FSO direct link and only the RF direct link are given, respectively. Furthermore, the CDF of the output SNR of the hybrid FSO/RF parallel link in the one-hop or direct link is obtained. Then, the end-to-end performance of the multi-hop parallel hybrid FSO/RF cooperative system based on the DF protocol is analyzed, and new mathematical expressions of the end-to-end ABER and outage probability of the system under four binary subcarrier modulation schemes are derived. Finally, through numerical analysis, the effects of the turbulence, pointing errors, modulation mode, detection technique, RF fading, and relay-aided structure on the hybrid FSO/RF direct link and the relay-aided hybrid system are compared and analyzed.

2. System and Channel Model

Figure 1 shows a schematic of the multi-hop parallel hybrid FSO/RF cooperative communication system. On each branch of the relay-aided hybrid system, the signal is separated into two identical signals after subcarrier modulation, and the signals are transmitted through the FSO and RF transmitter, respectively. According to the principle of maximum SNR, any regenerative repeater will select the maximum SNR signal from the two received signals for demodulation, regeneration, and forwarding.

For the relay-aided hybrid system with an ($N,M$) structure, the system has N cooperation paths and one direct path. For any cooperation path, there are M-hops from the source node to the destination node. Only one relay is allowed to process the received signals at one time based on the symbol-wise DF relaying method. Based on the best path selection scheme, a cooperative path is chosen to implement the transmission of the digital signal from the source node to the destination node.

2.1. One-Hop FSO Sub-Link

The Málaga distribution has unified most of the existing turbulence models, such as gamma–gamma distribution, log-normal distribution, $\kappa$ distribution, and so on [18]. Considering the Málaga turbulence with pointing errors, the PDF of the output instantaneous SNR ${\gamma }_{}^{\mathrm{FSO}}$ of the one-hop FSO sub-link is [19]:

$$f_{{\gamma }_{}^{\mathrm{FSO}}}\left({\gamma }_{}^{\mathrm{FSO}}\right)=\frac{{\xi }_{}^{2}A}{2^r{\gamma }_{}^{\mathrm{FSO}}}\sum _{k=1}^{\beta }{b}_k{G}_{1,3}^{3,0}\left(B{\left(\frac{{\gamma }_{}^{\mathrm{FSO}}}{{\mu }_{}^{\mathrm{FSO},\mathsf{r}}}\right)}^{\frac{1}{r}}\left|\begin{array}{c}1+{\xi }_{}^2\hfill \\ {\xi }_{}^2,\alpha ,k\hfill \end{array}\right.\right),$$

(1)

where $G_{·,·}^{·,·}(·)$ is the Meijer-G function, $\xi$ is the ratio of the equivalent beam radius of the receiver plane to the jitter standard deviation, $B={\xi }^2\alpha \beta (g\beta +{\mathsf{\Omega }}^{\prime })/\left[\left({\xi }^2+1\right)\left(g\beta +{\mathsf{\Omega }}^{\prime }\right)\right]$, and $b_k=a_k{\left[\alpha \beta /(g\beta +{\mathsf{\Omega }}^{\prime })\right]}^{-\left(\alpha +k\right)/2}$,

$$\left\{\begin{array}{c}A=\frac{2{\alpha }^{\alpha /2}}{g^{1+\alpha /2}\mathsf{\Gamma }\left(\alpha \right)}{\left(\frac{g\beta }{g\beta +\mathsf{\Omega }}\right)}^{\beta +\frac{\alpha }{2}}\hfill \\ a_k=\left(\begin{array}{c}\beta -1\hfill \\ k-1\hfill \end{array}\right)\frac{{\left(g\beta +\mathsf{\Omega }\right)}^{1-\frac{k}{2}}}{(k-1)!}{\left(\frac{\mathsf{\Omega }}{g}\right)}^{k-1}{\left(\frac{\alpha }{\beta }\right)}^{k/2}\hfill \end{array}\right.,$$

(2)

where $\alpha$ is a positive parameter, $\beta$ is a natural number, $\mathsf{\Gamma }(·)$ is the Gamma function. $g=2b_0(1-\rho )$, $2b_0$ represents the average power of the total scattering component, $\rho$ ($0\le \rho \le 1$) represents the scattering power coupled with the line-of-sight component, $\mathsf{\Omega }$ represents the average power of mutual coupling components, ${\mu }_{}^{\mathrm{FSO},\mathsf{r}}$ represents the average SNR of the FSO sub-link, and r is the parameter that defines the detection technology, that is, $r=1$ represents HD and $r=2$ represents DD. When $r=1$,

$${\mu }_{}^{\mathrm{FSO},1}={\mu }_{\mathrm{heterodyne}}={\overline{\gamma }}^{\mathrm{FSO}};$$

(3)

when $r=2$,

$${\mu }_{}^{\mathrm{FSO},2}={\mu }_{\mathrm{IM}/\mathrm{DD}}=\frac{\alpha {\xi }^2{\left({\xi }^2+1\right)}^{-2}\left({\xi }^2+2\right)\left(g+{\mathsf{\Omega }}^{\prime }\right)}{\left(\alpha +1\right)\left[2g\left(g+2{\mathsf{\Omega }}^{\prime }\right)+\mathsf{\Omega }{}^{\prime 2}\left(1+\frac{1}{\beta }\right)\right]}{\mu }_{}^{\mathrm{FSO},1}.$$

(4)

Integrate Equation (1) with the formula $F_{\gamma }\left(\gamma \right)={\int }_0^{\gamma }{f}_{\gamma }\left(t\right)dt$, and the CDF of ${\gamma }_{}^{\mathrm{FSO}}$ can be expressed as [19]:

$$F_{{\gamma }_{}^{\mathrm{FSO}}}\left({\gamma }_{}^{\mathrm{FSO}}\right)=D\sum _{k=1}^{\beta }{c}^k{G}_{r+1,3r+1}^{3r,1}\left(E\frac{{\gamma }_{}^{\mathrm{FSO}}}{{\mu }^{\mathrm{FSO},\mathsf{r}}}\left|\begin{array}{c}1,{\kappa }^1\hfill \\ {\kappa }^2,0\hfill \end{array}\right.\right),$$

(5)

where $D={\xi }^{2}A/\left[2^r{\left(2\pi \right)}^{r-1}\right]$, $c^k=b_k{r}^{\alpha +k-1}$, $E=B^r/r^{2r}$, ${\kappa }^1=({\xi }^2+1)/r,\dots ,({\xi }^2+r)/r$ (include r items), and ${\kappa }^2={\xi }^2/r,\dots ,({\xi }^2+r-1)/r,\alpha /r,\dots ,(\alpha +r-1)/r,k/r,\dots ,(k+r-1)/r$ (include $3r$ items).

2.2. One-Hop RF Sub-Link

In the one-hop RF sub-link with Nakagami-m fading channel, the PDF of the SNR ${\gamma }_{}^{\mathrm{RF}}$ is [20]:

$$f_{{\gamma }_{}^{\mathrm{RF}}}\left({\gamma }_{}^{\mathrm{RF}}\right)={\left(\frac{m}{{\overline{\gamma }}_{}^{\mathrm{RF}}}\right)}^m\frac{{\gamma }^{m-1}}{\mathsf{\Gamma }\left(m\right)}exp\left(-\frac{m{\gamma }_{}^{\mathrm{RF}}}{{\overline{\gamma }}_{}^{\mathrm{RF}}}\right)={\left(\frac{m}{{\overline{\gamma }}_{}^{\mathrm{RF}}}\right)}^m\frac{{\gamma }^{m-1}}{\mathsf{\Gamma }\left(m\right)}{G}_{0,1}^{1,0}\left[\frac{m{\gamma }_{}^{\mathrm{RF}}}{{\overline{\gamma }}_{}^{\mathrm{RF}}}\left|\begin{array}{c}-\hfill \\ 0\hfill \end{array}\right.\right],$$

(6)

where ${\overline{\gamma }}_{}^{\mathrm{RF}}$ is the average SNR of the one-hop RF sub-link, and m ($m\ge 0.5$) is the fading parameter of the one-hop RF sub-link. The CDF of the SNR ${\gamma }_{}^{\mathrm{RF}}$ can be obtained by integration,

$$F_{{\gamma }_{}^{\mathrm{RF}}}\left({\gamma }_{}^{\mathrm{RF}}\right)=\frac{1}{\mathsf{\Gamma }\left(m\right)}{G}_{1,2}^{1,1}\left(\frac{m{\gamma }_{}^{\mathrm{RF}}}{{\overline{\gamma }}_{}^{\mathrm{RF}}}\left|\begin{array}{c}1\hfill \\ m,0\hfill \end{array}\right.\right).$$

(7)

2.3. One-Hop Hybrid FSO/RF Link Based on Selective Combination Scheme

In the one-hop hybrid FSO/RF sub-link with the selective combination scheme, the receiver detects two branch signals at the same time and selects the signal with the maximum SNR to demodulate. Then, the output SNR ${\gamma }_{}^{\mathrm{SC}}$ of the selective combiner on the one-hop hybrid link can be expressed as [15]:

$${\gamma }_{}^{\mathrm{SC}}=max({\gamma }_{}^{\mathrm{FSO}},{\gamma }_{}^{\mathrm{RF}}).$$

(8)

The CDF of the SNR ${\gamma }_{}^{\mathrm{SC}}$ can be expressed as [15]:

$$F_{{\gamma }_{}^{\mathrm{SC}}}\left(\gamma \right)=\mathrm{Pr}\left(\max \right({\gamma }_{}^{\mathrm{FSO}},{\gamma }_{}^{\mathrm{RF}})\le \gamma )=\mathrm{Pr}({\gamma }_{}^{\mathrm{FSO}}\le \gamma ,{\gamma }_{}^{\mathrm{RF}}\le \gamma )=F_{{\gamma }_{}^{\mathrm{FSO}}}\left(\gamma \right)F_{{\gamma }^{\mathrm{RF}}}\left(\gamma \right).$$

(9)

Adding subscripts (x, y) to all variables in the Málaga turbulence model and Nakagami fading channel, when the subscript is (x, y) = (i, j), it represents the j-th hop hybrid FSO/RF link of the i-th path, when the subscript is (x, y) = (s, d), it represents the direct hybrid FSO/RF link. Substituting Equations (5) and (7) into Equation (9), the CDF of the selective combiner output SNR ${\gamma }_{s,d}^{\mathrm{SC}}$ of the direct channel and the CDF of the selective combiner output SNR ${\gamma }_{i,j}^{\mathrm{SC}}$ of the j-th hop of the i-th path can be obtained as follows:

$$F_{{\gamma }_{x,y}^{\mathrm{SC}}}\left(\gamma \right)=\frac{D_{x,y}}{\mathsf{\Gamma }\left(m_{x,y}\right)}\sum _{k=1}^{{\beta }_{x,y}}{c}_{x,y}^k{G}_{1,2}^{1,1}\left(\frac{m_{x,y}\gamma }{{\overline{\gamma }}_{x,y}^{\mathrm{RF}}}\left|\begin{array}{c}1\hfill \\ m_{x,y},0\hfill \end{array}\right.\right)G_{r+1,3r+1}^{3r,1}\left(E_{x,y}\frac{\gamma }{{\mu }_{x,y}^{\mathrm{FSO},\mathsf{r}}}\left|\begin{array}{c}1,{\kappa }_{x,y}^1\hfill \\ {\kappa }_{x,y}^2,0\hfill \end{array}\right.\right).$$

(10)

According to formula (07.34.16.0003.01) in Ref. [21], the product of two Meijer-G functions in Equation (10) can be replaced by the extended generalized bivariate Meijer’s G-function (EGBMGF),

$$F_{{\gamma }_{x,y}^{\mathrm{SC}}}\left(\gamma \right)=\frac{D_{x,y}}{\mathsf{\Gamma }\left(m_{x,y}\right)}\left[\sum _{k=1}^{{\beta }_{x,y}}{c}_{x,y}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{x,y},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{x,y}^1\hfill \\ {\kappa }_{x,y}^2,0\hfill \end{array}\right|\left.\frac{m_{x,y}\gamma }{{\overline{\gamma }}_{x,y}^{\mathrm{RF}}},\frac{E_{x,y}\gamma }{{\mu }_{x,y}^{\mathrm{FSO},r}}\right)\right].$$

(11)

3. End-to-End System Performance Analysis

For the relay-aided hybrid FSO/RF cooperative system with DF protocol, the equivalent SNR ${\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}$ of the i-th path can be expressed as [13]:

$${\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}=min({\gamma }_{i,1}^{\mathrm{SC}},{\gamma }_{i,2}^{\mathrm{SC}},\cdots ,{\gamma }_{i,M}^{\mathrm{SC}}).$$

(12)

Considering that each branch of the relay-aided hybrid FSO/RF system is independent and identically distributed, the CDF of the equivalent SNR ${\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}$ of the i-th path can be expressed as [13]:

$$F_{{\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}}\left(\gamma \right)=1-{\left[1-F_{{\gamma }_{i,j}^{\mathrm{SC}}}\left(\gamma \right)\right]}^M.$$

(13)

Based on the best path selection scheme, the receiver of the destination node selects the highest SNR signal from the i-th path (the SNR ${\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}$, $i=1,\dots ,N$) and the direct path (the SNR ${\gamma }_{s,d}^{\mathrm{SC}}$) signals for output. Therefore, the equivalent output SNR ${\gamma }_{\mathrm{eq}}$ at the destination node can be expressed as:

$${\gamma }_{\mathrm{eq}}=max({\gamma }_{s,d}^{\mathrm{SC}},{\gamma }_{\mathrm{eq}}^{\prime }),$$

(14)

where ${\gamma }_{\mathrm{eq}}^{\prime }=\underset{i=1,\dots N}{max}{\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}$, and the CDF of ${\gamma }_{\mathrm{eq}}^{\prime }$ can be written as:

$$\begin{array}{cc}\hfill F_{{{\gamma }^{\prime }}_{{\mathrm{eq}}_{\mathsf{i}}}}\left(\gamma \right)& =\left[F_{{\gamma }_{{\mathrm{eq}}_{\mathsf{i}}}}\left(\gamma \right)\right]{}^N={\left[1-{\left[1-F_{{\gamma }_{i,j}^{\mathrm{SC}}}\left(\gamma \right)\right]}^M\right]}^N\hfill \\ \hfill & ={\left\{1-{\left\{1-\frac{D_{i,j}}{\mathsf{\Gamma }\left(m_{i,j}\right)}\left[\sum _{k=1}^{{\beta }_{i,j}}{c}_{i,j}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{i,j},0 \hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{i,j}^1 \hfill \\ {\kappa }_{i,j}^2,0 \hfill \end{array}\right|\left.\frac{m_{i,j}\gamma }{{\overline{\gamma }}_{i,j}^{\mathrm{RF}}},\frac{E_{i,j}\gamma }{{\mu }_{i,j}^{\mathrm{FSO},r}}\right)\right]\right\}}^M\right\}}^N.\hfill \end{array}$$

(15)

Then, the CDF of the output SNR of the destination node is

$$\begin{array}{cc}\hfill F_{{\gamma }_{\mathrm{eq}}}\left(\gamma \right)& =F_{{\gamma }_{s,d}^{\mathrm{SC}}}\left(\gamma \right)\times F_{{{\gamma }^{\prime }}_{{\mathrm{eq}}_{\mathsf{i}}}}\left(\gamma \right)=F_{{\gamma }_{s,d}^{\mathrm{SC}}}\left(\gamma \right)\times {\left[1-{\left[1-F_{{\gamma }_{i,j}^{\mathrm{SC}}}\left(\gamma \right)\right]}^M\right]}^N\hfill \\ \hfill & =\frac{D_{s,d}}{\mathsf{\Gamma }\left(m_{s,d}\right)}\left[\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} -\hfill \\ -\hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}\gamma }{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}\gamma }{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)\right]\hfill \\ \hfill & \times {\left\{1-{\left\{1-\frac{D_{i,j}}{\mathsf{\Gamma }\left(m_{i,j}\right)}\left[\sum _{k=1}^{{\beta }_{i,j}}{c}_{i,j}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} -\hfill \\ -\hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{i,j},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{i,j}^1\hfill \\ {\kappa }_{i,j}^2,0\hfill \end{array}\right|\left.\frac{m_{i,j}\gamma }{{\overline{\gamma }}_{i,j}^{\mathrm{RF}}},\frac{E_{i,j}\gamma }{{\mu }_{i,j}^{\mathrm{FSO},r}}\right)\right]\right\}}^M\right\}}^N.\hfill \end{array}$$

(16)

3.1. Average Bit Error Rate

For the relay-aided hybrid FSO/RF cooperation system, four binary modulation schemes are used for data transmission in any FSO or RF sub-link. According to Refs. [22,23], the expression of the ABER can be expressed as:

$$P_{\mathsf{b}}=\frac{q^p}{2\mathsf{\Gamma }\left(p\right)}{\int }_0^{\infty }{\left(\gamma \right)}^{p-1}exp(-q\gamma )F_{\gamma }\left(\gamma \right)d\gamma ,$$

(17)

where p and q are the ABER parameters used to describe four binary modulation schemes, as shown in

Table 1. Parameters p and q for the various binary modulation schemes.

Binary Modulation Scheme	p	q
Coherent binary phase shift keying (CBPSK)	0.5	1
Differential binary phase shift keying (DBPSK)	1	1
Coherent binary frequency shift keying (CBFSK)	0.5	0.5
Non-coherent binary frequency shift keying (NBFSK)	1	0.5

.

By substituting Equation (16) into Equation (17), the ABER of the relay-aided hybrid system (N, M) can be obtained:

$$\begin{array}{cc}\hfill P_{\mathsf{b}}^{\mathrm{multi}}& =\frac{q^p}{2\mathsf{\Gamma }\left(p\right)}{\int }_0^{\infty }{\left(\gamma \right)}^{p-1}exp(-q\gamma )\frac{D_{s,d}}{\mathsf{\Gamma }\left(m_{s,d}\right)}\left[\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1 \hfill \\ {\kappa }_{s,d}^2,0 \hfill \end{array}\right|\left.\frac{m_{s,d}\gamma }{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}\gamma }{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)\right]\hfill \\ \hfill & \times {\left\{1-{\left\{1-\frac{D_{i,j}}{\mathsf{\Gamma }\left(m_{i,j}\right)}\left[\sum _{k=1}^{{\beta }_{i,j}}{c}_{i,j}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{i,j},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{i,j}^1\hfill \\ {\kappa }_{i,j}^2,0\hfill \end{array}\right|\left.\frac{m_{i,j}\gamma }{{\overline{\gamma }}_{i,j}^{\mathrm{RF}}},\frac{E_{i,j}\gamma }{{\mu }_{i,j}^{\mathrm{FSO},r}}\right)\right]\right\}}^M\right\}}^{N}d\gamma .\hfill \end{array}$$

(18)

According to formula (3.6.1) in Refs. [24,25], ${\int }_a^{\infty }(x-a){}^{c}exp\left(-b(x-a)\right) f\left(x\right)dx$ $\approx {\displaystyle \sum _{\tau =1}^n}{w}_{\tau }f\left(x_{\tau }\right)$ is the approximate expression of the generalized Gauss–Laguerre integral, where $w_{\tau }$ is the weight function and $x_{\tau }$ is the special point of abscissa. Therefore, Equation (18) can be simplified as:

$$\begin{array}{cc}\hfill P_{\mathsf{b}}^{\mathrm{multi}}& =\frac{q^p}{2\mathsf{\Gamma }\left(p\right)}\frac{D_{s,d}}{\mathsf{\Gamma }\left(m_{s,d}\right)}\sum _{\tau =1}^n{w}_{\tau }\left[\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}{x}_{\tau }}{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}{x}_{\tau }}{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)\right]\hfill \\ \hfill & \times {\left\{1-{\left\{1-\frac{D_{i,j}}{\mathsf{\Gamma }\left(m_{i,j}\right)}\left[\sum _{k=1}^{{\beta }_{i,j}}{c}_{i,j}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{i,j},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{i,j}^1\hfill \\ {\kappa }_{i,j}^2,0\hfill \end{array}\right|\left.\frac{m_{i,j}{x}_{\tau }}{{\overline{\gamma }}_{i,j}^{\mathrm{RF}}},\frac{E_{i,j}{x}_{\tau }}{{\mu }_{i,j}^{\mathrm{FSO},r}}\right)\right]\right\}}^M\right\}}^N.\hfill \end{array}$$

(19)

According to Ref. [26] and $q=1$, $x_{\tau }$ is the $\tau$-th root of the generalized Laguerre polynomial $L_n^{(-(1/2)}\left(x\right)$, and the corresponding weight coefficient is $w_{\tau }=\mathsf{\Gamma }\left[n+(1/2)\right]x_t/$ $\left\{n!{(n+1)}^2{\left[L_{n+1}^{(-1/2)}\left(x_t\right)\right]}^2\right\}$.

According to formula (2.1) in Ref. [27], by substituting Equation (11) into Equation (17), the ABER of the direct link from the source node to the destination node can be obtained:

$$\begin{array}{cc}\hfill P_{\mathsf{b}}^{\mathrm{SD}}& =\frac{q^p{D}_{s,d}}{2\mathsf{\Gamma }\left(p\right)\mathsf{\Gamma }\left(m_{s,d}\right)}\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{\int }_0^{\infty }{\left(\gamma \right)}^{p-1}exp(-q\gamma )G_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}\gamma }{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}\gamma }{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)d\gamma \hfill \\ \hfill & =\frac{D_{s,d}}{2\mathsf{\Gamma }\left(p\right)\mathsf{\Gamma }\left(m_{s,d}\right)}\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{1,0;1,1;3r,1}\left(\left.\begin{array}{c}p\hfill \\ -\hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}}{q{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}}{q{\mu }_{s,d}^{\mathrm{FSO},r}}\right).\hfill \end{array}$$

(20)

3.2. Outage Probability

The outage probability refers to the probability that the end-to-end output SNR is lower than a specific threshold ${\gamma }_{\mathrm{th}}$. Therefore, the outage probability of the system in this paper can be expressed as [13]:

$$P_{\mathrm{out}}=Pr(\gamma <{\gamma }_{\mathrm{th}})={\int }_0^{{\gamma }_{\mathrm{th}}}{f}_{\gamma }\left(\gamma \right)d\gamma =F_{\gamma }\left({\gamma }_{\mathrm{th}}\right).$$

(21)

Substituting Equation (16) into Equation (21), the outage probability of the relay-aided hybrid FSO/RF cooperation system (N, M) can be obtained:

$$\begin{array}{cc}\hfill P_{\mathrm{out}}^{\mathrm{multi}}& =\frac{D_{s,d}}{\mathsf{\Gamma }\left(m_{s,d}\right)}\left[\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}{\gamma }_{\mathrm{th}}}{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}{\gamma }_{\mathrm{th}}}{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)\right]\hfill \\ \hfill & \times {\left\{1-{\left\{1-\frac{D_{i,j}}{\mathsf{\Gamma }\left(m_{i,j}\right)}\left[\sum _{k=1}^{{\beta }_{i,j}}{c}_{i,j}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c} - \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{i,j},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{i,j}^1\hfill \\ {\kappa }_{i,j}^2,0\hfill \end{array}\right|\left.\frac{m_{i,j}{\gamma }_{\mathrm{th}}}{{\overline{\gamma }}_{i,j}^{\mathrm{RF}}},\frac{E_{i,j}{\gamma }_{\mathrm{th}}}{{\mu }_{i,j}^{\mathrm{FSO},r}}\right)\right]\right\}}^M\right\}}^N.\hfill \end{array}$$

(22)

Substituting Equation (11) into Equation (21), the outage probability of the direct hybrid FSO/RF link can be obtained:

$$P_{\mathrm{out}}^{\mathrm{SD}}=\frac{D_{s,d}}{\mathsf{\Gamma }\left(m_{s,d}\right)}\left[\sum _{k=1}^{{\beta }_{s,d}}{c}_{s,d}^k{G}_{0,0;1,2;r+1,3r+1}^{0,0;1,1;3r,1}\left(\left.\begin{array}{c}- \hfill \\ - \hfill \end{array}\right|\right.\left.\begin{array}{c}1\hfill \\ m_{s,d},0\hfill \end{array}\right|\left.\begin{array}{c}1,{\kappa }_{s,d}^1\hfill \\ {\kappa }_{s,d}^2,0\hfill \end{array}\right|\left.\frac{m_{s,d}{\gamma }_{\mathrm{th}}}{{\overline{\gamma }}_{s,d}^{\mathrm{RF}}},\frac{E_{s,d}{\gamma }_{\mathrm{th}}}{{\mu }_{s,d}^{\mathrm{FSO},r}}\right)\right].$$

(23)

4. Numerical Results and Discussions

Under different turbulence conditions, pointing errors, RF fading parameters, modulation modes, detection schemes, and relay-aided structures, the performance of the hybrid FSO/RF direct link and the relay-aided hybrid system is numerically analyzed. By calculating the approximate value of the generalized Gauss–Laguerre quadrature function with $\tau =30$, the ABER of the relay-aided hybrid system can be obtained according to Equation (19). To simplify the simulation analysis, it is assumed in this section that the atmospheric turbulence intensity is consistent for all FSO links in the relay-aided system (but the atmospheric scintillation index also varies depending on the transmission distance), and the fading parameters of all RF links are also the same. The structure parameters ($N=2$, $M=3$), ($N=2$, $M=5$), and ($N=4$, $M=3$) have been selected to avoid entanglement. According to the meaning of the scintillation index, the turbulence parameters set for the 1 km FSO link under the Málaga turbulence are selected as $(\alpha =5,\beta =3,\rho =0.25)$, $(\alpha =3,\beta =2,\rho =0.75)$, and $(\alpha =2,\beta =1,\rho =0.9)$, respectively, for weak, moderate, and strong turbulence conditions. For weak turbulence, the parameters of the Málaga turbulence are $(\alpha =0.36,\beta =1,\rho =0.95)$ and $(\alpha =0.0291,\beta =1,\rho =0.95)$, respectively, for 2 km and 3 km FSO links. Other parameters for the Málaga turbulence include $\mathsf{\Omega }=1.3265$, $b_0=0.1079$, and ${\phi }_A-{\phi }_B=\pi /2$.

Assume that the average SNR of each bit in the FSO and RF sub-links is equal, that is, ${\overline{\gamma }}^{\mathrm{FSO}}={\overline{\gamma }}^{\mathrm{RF}}$, and the distance of the direct link is 1 km for Figure 2, Figure 3 and Figure 4. For strong turbulence, for the pointing errors parameter $\xi =1$ and fading parameter $m=2$, Figure 2 describes the relationship between the ABER performance and the SNR of the hybrid FSO/RF direct link under different modulation and detection schemes. It can be seen from Figure 2 that under the same detection scheme, the ABER performance of the hybrid FSO/RF direct link from high to low is: CBPSK, DBPSK, CBFSK, and NBFSK. Compared with frequency modulation, the ABER performance of the hybrid direct link under phase modulation is better, while the ABER performance of the hybrid direct link under coherent modulation is better than that under incoherent modulation. For example, when the heterodyne detection scheme is adopted and the SNR is 20 dB, the ABERs of the hybrid direct link with CBPSK, DBPSK, CBFSK, and NBFSK modulation are $6.159\times {10}^{-6}$, $1.876\times {10}^{-5}$, $3.854\times {10}^{-5}$, and $1.158\times {10}^{-4}$, respectively. The ABER of the hybrid FSO/RF direct link with the heterodyne detection scheme is lower than that with the direct detection scheme. For example, when the CBPSK modulation scheme is adopted and the SNR is 20 dB, the ABERs of the hybrid direct link under heterodyne detection and direct detection are $6.159\times {10}^{-6}$ and $3.449\times {10}^{-5}$, respectively.

With CBPSK modulation and the fading parameter $m=2$, Figure 3 depicts the relationship between the ABER and the SNR of the hybrid FSO/RF direct link under different turbulence intensity, detection schemes, and pointing errors. Comparing Figure 3a,b, it is easy to find that the ABER of the hybrid FSO/RF direct link with the heterodyne detection scheme is better than that of the direct detection scheme for any turbulence intensity and pointing errors. For example, for any turbulence intensity and pointing errors, the ABER of the hybrid direct link with the heterodyne detection is lower than ${10}^{-5}$ when the SNR is 20 dB in Figure 3a, and the ABER of the hybrid direct link with the direct detection is higher than ${10}^{-5}$ in Figure 3b. Although compared with only the FSO link, the performance of the hybrid FSO/RF link has been greatly improved, it can be seen from Figure 3 that the turbulence intensity and the pointing errors still have a significant impact on the ABER of the hybrid direct link. For example, in Figure 3a, with the heterodyne detection and 16 dB SNR, the ABERs of the hybrid direct link with the pointing errors parameter $\xi =1$ and $\xi =3$ under moderate turbulence are $4.001\times {10}^{-5}$ and $1.445\times {10}^{-5}$, respectively, and the ABERs of the hybrid direct link with the pointing errors parameter $\xi =1$ and $\xi =3$ under weak turbulence are $3.879\times {10}^{-5}$ and $1.249\times {10}^{-5}$, respectively.

With CBPSK modulation and $\xi =1$, Figure 4 shows the relationship between the ABER and the SNR of the hybrid direct link under different turbulence intensity, detection schemes, and RF fading parameters. It can be seen from Figure 3 that the RF fading parameter has a greater impact on the ABER performance of the hybrid FSO/RF direct link. When this increases, the ABER of the system increases. For example, with a 20 dB SNR and moderate turbulence, the ABERs of the hybrid direct link with $m=2$ and $m=4$ under heterodyne detection are $2.992\times {10}^{-6}$ and $2.13\times {10}^{-8}$, respectively, and the ABERs of the hybrid direct link with $m=2$ and $m=4$ under direct detection are $2.823\times {10}^{-5}$ and $1.228\times {10}^{-7}$, respectively.

Assuming that the average SNR of each bit in the FSO and RF sub-links is equal, that is, ${\overline{\gamma }}^{\mathrm{FSO}}={\overline{\gamma }}^{\mathrm{RF}}$, and the distance of each hop and direct link is 1 km for Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9, under CBPSK modulation with heterodyne detection, with the RF fading parameter $m=2$ and weak turbulence conditions, Figure 5 shows the relationship between the ABER and the SNR of the relay-aided hybrid FSO/RF system with different pointing errors and relay-aided structures. Figure 5 shows that the pointing errors have no impact on the ABER performance of the relay-aided hybrid system, that is, the relay-aided hybrid system can compensate for the adverse impact of pointing errors on the system. According to Figure 5, it can also be seen that different structures of the relay-aided hybrid system have different abilities to improve the ABER performance, that is, the ABER performance will be improved with the increase in the path, and will be deteriorated with the increase in the number of hops. This is because when the transmission path increases, the probability of receiving a large SNR signal at the receiving end of the system increases, resulting in the improvement of the ABER performance. When the number of hops increases, the cumulative ABER of the signal received by the receiving end of the system will increase, so the overall ABER of the system will decline. For example, when the SNR is 20 dB, the ABERs of the relay-aided hybrid systems with ($N=2$, $M=3$), ($N=2$, $M=5$), and ($N=4$, $M=3$) structures are $7.98\times {10}^{-12}$, $2.206\times {10}^{-11}$, and $6.925\times {10}^{-16}$, respectively. Comparing Figure 4 and Figure 5, it can be further found that the relay-aided hybrid system can significantly improve the ABER performance of the system compared with the hybrid direct link.

With CBPSK modulation, heterodyne detection, the RF fading parameter $m=2$, and the pointing errors parameter $\xi =1$, Figure 6 depicts the relationship between the ABER and the SNR of the relay-aided hybrid FSO/RF system under different turbulence intensity and relay-aided structures. Figure 6 shows that the relay-aided hybrid FSO/RF system with a suitable structure can better compensate for the negative impact of turbulence and improve the ABER performance. For example, when the SNR is 20 dB, the ABERs of the relay-aided hybrid system with ($N=2$, $M=3$), ($N=2$, $M=5$), and ($N=4$, $M=3$) structures under strong turbulence are $4.088\times {10}^{-11}$, $1.129\times {10}^{-10}$, and $6.197\times {10}^{-15}$, respectively, and the ABERs under moderate turbulence are $8.742\times {10}^{-12}$, $2.417\times {10}^{-11}$, and $7.723\times {10}^{-16}$, respectively.

When the pointing errors parameter $\xi =3$, the RF fading parameter $m=2$, and the decision threshold ${\gamma }_{\mathrm{th}}=10$ dB, Figure 7 shows the relationship between the outage probability and the SNR of the relay-aided hybrid FSO/RF system and hybrid direct link with heterodyne detection and direct detection, respectively. According to Figure 7, it is easy to see that compared with the hybrid direct link, the relay-aided hybrid system can significantly improve the outage performance. In Figure 7, we can also see that a different turbulence intensity still has a certain impact on the relay-aided hybrid FSO/RF system. Different structures of the relay-aided hybrid system have different effects on the outage performance of the system, that is, the outage probability of the system will increase with the increase in hops and decrease with the increase in paths. Comparing Figure 7a,b, it can be seen that the outage probability of the relay-aided hybrid system and the hybrid direct link with heterodyne detection is better than that with direct detection.

When the pointing errors parameter $\xi =1$, decision threshold ${\gamma }_{\mathrm{th}}=10$ dB, and with strong turbulence, Figure 8 shows the relationship between the outage probability and the SNR of the relay-aided hybrid system with different structures and the hybrid direct link under different RF fading parameters. It can be seen from Figure 8 that the relay-aided hybrid system can effectively improve the outage performance compared with the hybrid direct link with different RF fading parameters m. The outage probability of the relay-aided hybrid FSO/RF system and the hybrid direct link will decrease with the increase in m.

When the pointing errors parameter $\xi =1$, RF fading parameter $m=2$, and with strong turbulence, Figure 9 shows the relationship between the outage probability and the SNR of the relay-aided hybrid system with different structures and the hybrid direct link at different decision thresholds. It can be seen from Figure 9 that the outage probability of the relay-aided hybrid system and the hybrid direct link is very sensitive to the decision threshold. Compared with the hybrid direct link, the relay-aided hybrid system with different decision thresholds can effectively improve the outage performance. The outage probability of the relay-aided hybrid system and the hybrid direct link will decrease as the decision threshold decreases.

Assume that the average SNR of each bit in the FSO and RF sub-links is equal, that is, ${\overline{\gamma }}^{\mathrm{FSO}}={\overline{\gamma }}^{\mathrm{RF}}$, the distance of each hop is 1 km and the direct link is 2 or 3 km. With weak turbulence, CBPSK modulation, heterodyne detection, the RF fading parameter $m=2$, the pointing errors parameter $\xi =1$, and the decision threshold ${\gamma }_{\mathrm{th}}=10$ dB, Figure 10 shows the relationship between the performance metrics and the SNR of the relay-aided hybrid FSO/RF system under different direct distances and relay-aided structures. Similar to the previous conclusion, it can be seen from Figure 10 that as the direct distance increases, the performance of the hybrid FSO/RF system will slightly decrease, that is, the transmission distance has little impact on the performance of hybrid direct links and relay-aided hybrid systems. The hybrid direct communication system can effectively improve the reliability of the communication system. Furthermore, we can see that the relay-aided hybrid system can significantly improve the system performance, and as the number of paths increases, the system’s performance can be significantly improved, while an increase in hops slightly reduces the system’s performance.

Assume that the average SNR of each bit in the FSO and RF sub-links is unequal, that is, ${\overline{\gamma }}^{\mathrm{FSO}}\ne {\overline{\gamma }}^{\mathrm{RF}}$, the distance of each hop is 1 km and the direct link is 2 or 3 km. With weak turbulence, CBPSK modulation, heterodyne detection, the RF fading parameter $m=2$, the pointing errors parameter $\xi =1$, and the decision threshold ${\gamma }_{\mathrm{th}}=10$ dB, Figure 11 shows the relationship between the performance metrics and the SNR of the relay-aided hybrid FSO/RF system under different direct distances, the SNR of the RF sub-link, and relay-aided structures. From Figure 11, it can be clearly seen that the changes in the SNR of the RF sub-links have a significant impact on the systems of hybrid direct links and relay-aided hybrid systems. Further, compared with Figure 10, we can find that the performance of the hybrid direct link and the relay-aided hybrid system with different transmission distances under the same conditions is significantly different after the SNR of the RF sub-link changes.

5. Conclusions

This paper studies a relay-aided hybrid FSO/RF cooperation system to further improve the performance of the FSO system. By analyzing the end-to-end instantaneous SNR of the hybrid direct link and the relay-aided hybrid system with the subcarrier modulation scheme, new mathematical expressions of the ABER and outage probability are derived. Through system simulation, the effects of different turbulence intensity, pointing errors, modulation modes, detection schemes, and RF fading parameters on the hybrid direct link and the relay-aided hybrid system are compared and analyzed.

According to the research and analysis of this paper, we can obtain some important conclusions. Compared with the hybrid direct link, the relay-aided hybrid system can further improve the performance of the system over atmospheric turbulence and pointing errors. By selecting the appropriate subcarrier modulation mode and detection scheme, the communication performance of the hybrid direct link and the relay-aided hybrid system can be improved. For example, the performance of the system with CBPSK modulation and the heterodyne detection scheme is the best in this paper. For the relay-aided hybrid FSO/RF system, the communication performance of the system can be improved by increasing the number of paths and reducing the number of hops, but the increase in the number of hops can increase the transmission distance and coverage. The influence of the RF fading parameter m on the hybrid direct link and the relay-aided hybrid system is obvious. The relay-aided hybrid system can greatly compensate for the deterioration of the system performance caused by pointing errors. The change in direct distance has little impact on the performance of hybrid direct links and relay-aided hybrid systems, while the change in the SNR of RF sub-links has a significant impact on their performance.