An Underdetermined Convolutional Blind Separation Algorithm for Time–Frequency Overlapped Wireless Communication Signals with Unknown Source Number

Abstract

It has been challenging to separate the time–frequency (TF) overlapped wireless communication signals with unknown source numbers in underdetermined cases. In order to address this issue, a novel blind separation strategy based on a TF soft mask is proposed in this paper. Based on the clustering property of the signals in the sparse domain, the angular probability density distribution is obtained by the kernel density estimation (KDE) algorithm, and then the number of source signals is identified by detecting the peak points of the distribution. Afterward, the contribution degree function is designed according to the cosine distance to calculate the contribution degrees of the source signals in the mixed signals. The separation of the TF overlapped signals is achieved by constructing a soft mask matrix based on the contribution degrees. The simulations are performed with digital signals of the same modulation and different modulation, respectively. The results show that the proposed algorithm has better anti-aliasing and anti-noise performance than the comparison algorithms.

1. Introduction

1.1. Research Background and Motivation

Blind source separation (BSS), defined as the task of separating the source signals from the observed signals when the source signals and the mixing mode are not known exactly, enables the separation of the mixed signals and the extraction of the source signals without a priori knowledge. As a kind of front-end technology in the field of signal processing, the BSS is of great importance in communication anti-jamming and improving spectrum utilization [1]. Unfortunately, the classical independent component analysis (ICA) and many extended algorithms [2,3,4] are based on the main assumption that the number of observed signals is greater than or equal to the number of source signals, which cannot realize the blind signal separation in underdetermined cases [5]. In practical communication scenarios, the number of observed signals is usually less than the number of source signals on account of system cost or environmental constraints, thus the study of underdetermined blind source separation (UBSS) is of more practical significance. At present, the main approach to tackle the issue of UBSS is based on sparse component analysis (SCA) [6]. The main strategy of the SCA method is solving the UBSS by exploiting the sparsity of the source signal or sparsity in the transform domain. However, with the dramatic increase in RF devices, it is almost inevitable that signals will be overlapped in the time and frequency domains. For communication signals that are not completely sparse in the time and frequency domains, it has been a challenge to solve the UBSS problem using the SCA method. It is of practical importance to separate the TF overlapped signals in underdetermined cases. This paper aims to improve the SCA algorithm for the effective separation of TF overlapped wireless communication signals with unknown source numbers.

1.2. Literature Review

The classical SCA algorithm mainly consists of the two-stage method and the TF masking method. The two-stage method first estimates the mixing matrix and then uses signal sparsity to recover the source signal. Zhen [7] utilized the sparse coding technique to discover the 1-D subspace from the set of TF points, then estimated the mixing matrix by grouping vectors in the subspace through hierarchical clustering, and finally used the least-squares method to achieve source signal recovery; Fang [8] firstly estimated the mixing matrix based on the hyperplane normal vector clustering algorithm, and then used the classical algorithm called orthogonal matching pursuit in compressed sensing to realize the separation of the linear frequency modulated radar mixed signals; Fu [9] realized the recovery of the source signal using a new subspace complementary matching pursuit algorithm. The algorithm reduced the computational complexity by selecting multiple atoms per iteration and also used $L_2$ parametric minimization instead of $L_0$ parametric minimization in order to improve the recovery accuracy.

The other is the TF masking method, i.e., the source signal is separated by designing a TF mask matrix. Yu [10] first detected the peak energy point of the mixed signals and then calculated the cosine distance between all energy points and the peak energy point, afterward obtained the binary mask based on a threshold to realize the reconstruction of the vibration signal; Cobos [11] proposed a maximum a posteriori decision criterion to design a binary mask matrix for the separation of mixed speech signals; Kumar [12] first detected the active single-source points by calculating the mixing ratio, and then found the cluster centers by a dynamic routing algorithm as a way to design a binary mask.

The above algorithms usually assume that the signal is sparse in some domains, such as the time domain, frequency domain, etc. However, in the wireless communication scenarios, the signals are inevitably overlapped in both time and frequency domains. In such cases, it may cause a reduction in separation accuracy or even complete failure when the classical SCA-based method is applied directly to the separation of TF overlapped communication signals. The two-stage method could not guarantee the estimation accuracy of the mixing matrix and the performance of the source signal recovery, while reconstructing the source signals through the TF binary mask will bring certain distortions [13,14,15]. Obviously, for the communication mixed signals with TF aliasing, better separation performance can be obtained by reasonably estimating the TF soft mask to reconstruct the source signals [16].

1.3. Contribution and Paper Organization

Under the framework of the TF masking method, we propose a novel underdetermined convolutional mixing UBSS algorithm for TF overlapped communication signals with an unknown number of sources. Firstly, the observed signals are converted to the sparse domain to obtain the linear clustering characteristic. Consequently, the effects of multipath and time delay are eliminated by calculating short-time spectral power; Secondly, the short-time spectral power ratio of the observed signals is converted into the angular distribution, and the number of source signals can be estimated through the KDE algorithm; Then the contribution degree of the source signal in the observed signal is obtained by the designed contribution degree function. Finally, the TF soft mask is constructed based on the contribution degree, which is employed to separate the mixed signals. The main contribution of the paper can be summarized in the following:

• A source number estimation method based on the KDE algorithm followed by detecting the peak points of the distribution is proposed, which can effectively estimate the number of sources before the signal separation.
• We design a contribution degree function based on the cosine distance, by which the contribution degree of the source signal to the mixed signal can be obtained. The contribution degree clearly reflects the mixing degree of the source signal at each frequency point, and it is of significant importance to improve the separation accuracy.
• The TF soft mask matrix is constructed based on the contribution degree function. Compared with the traditional binary mask, the TF soft mask can improve the separation accuracy for TF overlapped signals.
• The proposed blind source separation method for TF overlapped communication signal extends the application of the SCA algorithm, which reduces the requirement of signal sparsity for the classical SCA blind source separation algorithm.

The rest of this paper is structured as follows. The problem statement is given in Section 2, which includes the introduction of the effect on separation based on the classical SCA methods caused by TF aliasing. Subsequently, Section 3 describes the process of the algorithm in detail, which mainly consists of two parts: source number estimation and source signal recovery. Afterward, the separation performance of this algorithm is tested by the simulations in Section 4. Lastly, the conclusions are drawn in Section 5.

2. Problem Statement

In the wireless communication propagation model, multipath and time delay of signals are inevitable, which can be modeled as the convolutive mixture of the source signals [17]. All variables involved in the problem statement are described in

Table 1. Variable definitions involved in the problem statement.

Variable	Meaning
$\mathit{s}(\mathit{t})$	The source signal.
$\mathit{x}(\mathit{t})$	The received signal.
${\mathit{A}}_{\mathit{q}}$	The mixing matrix corresponding to the qth wireless propagation path of the signal.
$a_{ki}^q$	The amplitude attenuation of the ith source signal to the kth sensor through the qth path.
${\tau }_{ki}^q$	The time delay of the ith source signal to the kth sensor through the qth path.
$X_k(t,f)$	The STFT of the signal $x_k(t)$ received by the kth sensor.
$P_k(f)$	The power of the kth received signal.

. Without considering the noise, the mathematical expression of the convolutional mixture model can be expressed as

$$\mathit{x}(\mathit{t})={\displaystyle \sum _{\mathit{q}=1}^{\mathit{Q}}{\mathit{A}}_{\mathit{q}}\mathit{s}(\mathit{t})}$$

(1)

where $\mathit{s}(\mathit{t})$ represents the source signal; $\mathit{x}(\mathit{t})$ indicates the received mixed signal; $\mathit{Q}$ is the number of wireless propagation paths; ${\mathit{A}}_{\mathit{q}}$ denotes the mixing matrix corresponding to the qth path. Considering the effect of time delay. ${\mathit{A}}_{\mathit{q}}$ can be expressed as

$${\mathit{A}}_q=\left[\begin{array}{cccc}{a}_{11}^q\delta (t-{\tau }_{11}^q)& a_{12}^q\delta (t-{\tau }_{12}^q)& \cdots & a_{1n}^q\delta (t-{\tau }_{1n}^q)\\ a_{21}^q\delta (t-{\tau }_{21}^q)& a_{22}^q\delta (t-{\tau }_{22}^q)& \cdots & a_{2n}^q\delta (t-{\tau }_{2n}^q)\\ ⋮& ⋮& a_{ki}^q\delta (t-{\tau }_{ki}^q)& ⋮\\ a_{m1}^q\delta (t-{\tau }_{m1}^q)& a_{m2}^q\delta (t-{\tau }_{m2}^q)& \cdots & a_{mn}^q\delta (t-{\tau }_{mn}^q)\end{array}\right]$$

(2)

where $\delta (\mathbf{·})$ represents the impulse response, $a_{ki}^q$ and ${\tau }_{ki}^q$ are the amplitude attenuation and time delay of the ith source signal to the kth sensor through the qth path, respectively. In order to obtain the sparse characteristic of the signals, the signal $X_k(t,f)$ received by the kth sensor after a short-time Fourier transform (STFT) [18] can be written as

$$X_k(t,f)={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{ki}^q}}{S}_i^q(t-{\tau }_{ki}^q,f)$$

(3)

Take the BPSK signal as an example; the TF expression is described as

$$X_k(t,f)={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{ki}^q}}\widehat{h}(f\pm f_i)e^{\mp j2\pi f_i(t-{\tau }_{ki}^q)}$$

(4)

where $f_i$ denotes the center frequency of the ith source signal. After performing the modulus operation on $X_k(t,f)$ [19], the power $P_k(f)$ of the kth received signal can be expressed as

$$P_k(f)={\displaystyle {\int }_{-\infty }^{+\infty }\left|X_k(t,f)\right|}dt=\left|{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{ki}^q}}\widehat{h}(f\pm f_i)\right|{\displaystyle {\int }_{-\infty }^{+\infty }\left|e^{\mp j2\pi f_i(t-{\tau }_{ki}^q)}\right|}dt=\left|{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{ki}^q}}\widehat{h}(f\pm f_i)\right|$$

(5)

where $|\mathbf{·}|$ expresses modulus operation. When the spectrum of the source signal is non-aliased, there is only the component of the source signal $s_i(t)$ in the interval $\left[f_i-\Delta ,f_i+\Delta \right]$ ($\Delta >0$) of the received signal. In this case, the power ratio $P_{\alpha }(f)/P_{\beta }(f)$ of any two of the received signals ${\mathit{X}}_{\alpha }$ and ${\mathit{X}}_{\beta }$ is

$$\begin{array}{c}{P}_{\alpha }(f)/P_{\beta }(f)\\ =\frac{\left|{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{\alpha i}^q}}\widehat{h}(f\pm f_i)\right|}{\left|{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{q=1}^Q{a}_{\beta i}^q}}\widehat{h}(f\pm f_i)\right|}=\frac{\left|{\displaystyle \sum _{q=1}^Q{a}_{\alpha i}^q}\right|}{\left|{\displaystyle \sum _{q=1}^Q{a}_{\beta i}^q}\right|}\begin{array}{ccc},& f\in \left[f_i-\Delta ,f_i+\Delta \right]& \end{array}\end{array}$$

(6)

Consequently, the short-time spectral power points of the signal $s_i\left(t\right)$ will be clustered on the line with a slope of $P_{\alpha }(f)/P_{\beta }(f)$. Figure 1 represents the scatter plot of the power ratio of the two received signals after the convolutive mixture of the four BPSK signals. Due to the sparsity of the observed signals in the power domain, the received signals exhibit linear clustering property. There are four linear clusters in Figure 1 corresponding to the four source signals, and thus the mixed signals can be effectively separated by the traditional single-source-point (SSP) detection method [10]. However, in practical communication systems, it is almost impossible to avoid TF aliasing for communication signals. When the spectrum of the source signal is overlapped, the sparsity of the observed signals in the power domain cannot be guaranteed, thus weakening the linear clustering property of the observed signals (Figure 2). The traditional SSP method will not accurately recover the source signal.

3. UBSS Algorithm Based on TF Soft Mask

To tackle the above challenge, we propose a new blind separation strategy based on TF soft mask. The process consists of two main stages. The first step is to determine the number of sources through the KDE algorithm, followed by peak detection. After that, the contribution degree of the source signal in the observed signal is calculated by the contribution degree function to construct the TF soft mask matrix, which is used to separate the TF aliasing signals. All variables involved in the section are described in

Table 2. Variable definitions in this section.

Variable	Meaning
F	The set of frequencies after removing the low power points.
$\theta$	The angular distribution.
$G_{\kappa }(\theta )$	The angular probability density distribution of $\theta$.
$Y_i(t,f)$	The STFT of the ith signal to be reconstructed.
$M_i(t,f)$	The corresponding ith TF soft mask matrix that needs to be constructed.
${\eta }_u^i$	The contribution degree of the ith source signal to the mixed signal.
$d_u^i$	The corresponding cosine distance between the power point and the ith endpoint.
$y_i(t)$	The ith source signal to be reconstructed.

.

3.1. Source Number Estimation

In the problem of UBSS, the source number estimation is an indispensable step. The accurate estimation of the number of sources is the prerequisite for precise source signal recovery [20,21]. In order to estimate the number of sources accurately, the KDE algorithm is used to calculate the angular probability density distribution of the mixed signal in the sparse domain, followed by detecting the number of peak points of the distribution. In order to eliminate the interference of noise on the signal, a threshold $\epsilon$ needs to be set to screen out the low power points. The set of frequencies $F$ after removing the low power points can be obtained by

$$F=\left\{f|{\displaystyle \sum _{k=1}^m{P}_k(f)}>\epsilon \underset{f}{\max }[{\displaystyle \sum _{k=1}^m{P}_k(f)}]\right\}$$

(7)

where $\epsilon$ is the empirical parameter, which is taken as 0.1 in this paper. In the later expressions, $f_u$ is used to denote the uth element of $F$. Afterward, convert the filtered power points into angular distribution $\theta$:

$${\theta }_u=\arctan \left[P_{\alpha }(f_u)/P_{\beta }(f_u)\right]$$

(8)

The statistical histogram after converting to an angular distribution is shown in Figure 3. As seen from the figure, the angular distribution is more densely distributed around the intervals 0.07, 0.52, 1.15, and 1.48 than the other intervals. Thus the angular distribution $\theta$ is modeled based on the KDE algorithm [22]:

$$G_{\kappa }(\theta )=\frac{1}{l}{\displaystyle \sum _{u=1}^l{K}_{\kappa }(\theta -{\theta }_u)}$$

(9)

where $G_{\kappa }(\theta )$ denotes the angular probability density distribution of $\theta$; $l$ is the length of the vector $\theta$; $K_{\kappa }(·)$ represents the kernel function with a smoothing factor $\kappa$ of the bandwidth. In this paper, we choose the Gaussian kernel as the kernel function and take $\kappa$ as 0.25. The angular probability density distribution obtained by the KDE algorithm is shown in Figure 4. The number of peak points of the angular probability density distribution is the number of sources to be estimated. As we can see from Figure 4, there are four peak points marked by red circles, which means that the number of estimated source signals $\widehat{n}$ is four.

3.2. Source Signal Recovery by TF Soft Mask

Ideally, the reconstruction of the source signal is achieved by inverse short-time Fourier transform (ISTFT) in the frequency band $\left[f_i-\Delta ,f_i+\Delta \right]$ 10. However, the traditional clustering method is only a kind of hard segmentation of power points to obtain the TF binary mask. When the frequency spectrum is overlapped, the short-term spectral power of the observed signals is the superposition of the short-term spectral power of the source signals. The integrity of the source signal reconstruction cannot be guaranteed through the method of hard segmentation. Thus a matching soft mask matrix $M(t,f)$ is required to fully recover the source signal.

$$Y_i(t,f)=X(t,f)\odot M_i(t,f)$$

(10)

where $\odot$ represents the Hadamard product; $Y_i(t,f)$ denotes the ith signal to be reconstructed, and $M_i(t,f)$ is the corresponding TF soft mask matrix that needs to be constructed.

In this paper, the construction of the TF soft mask matrix is performed by designing a contribution degree function based on the cosine distance. It can be seen from Figure 2 that among all the power points inside the ith linear cluster, the endpoint is subject to the least interference. That is, when $f_u=f_i$, the contribution degree ${\eta }_u^i$ of the ith source signal to the mixed signal can be taken as 1, and the cosine distance between the endpoint and itself is 1 as well ($f_i$ is the frequency corresponding to the endpoint of the ith linear clustering cluster). When $f_u$ is further away from $f_i$, the contribution degree ${\eta }_u^i$ will be decreased. The corresponding cosine distance $d_u^i$ between the power point and the ith endpoint is decreased as well. Extremely, ${\eta }_u^i$ is taken as 0 when $f_u$ is the frequency of the nearest adjacent endpoint. Hence the contribution degree ${\eta }_u^i$ can be characterized by the cosine distance between all power points and the endpoint of the ith linear cluster. A monotonically increasing contribution function is designed to characterize the positive correlation between contribution degree ${\eta }_u^i$ and cosine distance $d_u^i$:

$${\eta }_u^i=\{\begin{array}{c}1,\begin{array}{ccc}\begin{array}{ccccc}& & & & \begin{array}{ccc}& & \end{array}\end{array}& d_u^i=1& \end{array}\\ {\left(\frac{d_u^i-d_i{}^{\prime }}{1-d_i{}^{\prime }}\right)}^{\gamma },\begin{array}{ccc}& \begin{array}{cc}\begin{array}{cc}& \end{array}& \end{array}& 0<d_u^i<d_i{}^{\prime }\end{array}\\ 0,\begin{array}{ccccc}\begin{array}{ccc}& & \end{array}& & & & \begin{array}{cc}& otherwise\end{array}\end{array}\end{array}$$

(11)

where $d_i{}^{\prime }$ denotes the maximum of the cosine distance between the endpoint of the ith linear cluster and the endpoints of the remaining linear clusters; $\gamma$ denotes the scaling factor of the contribution function, whose different values will affect the separation performance of the algorithm. In Section 4.1, we will discuss the impact of $\mathit{\gamma }$ on separation performance. The cosine distance of the uth power point from the endpoint of the ith linear cluster $d_u^i$ is obtained as

$$d_u^i=CD\left\{\left[\begin{array}{cc}{P}_{\alpha }(f_i),& P_{\beta }(f_i)\end{array}\right]\begin{array}{cc},& \end{array}\left[\begin{array}{cc}{P}_{\alpha }(f_u),& P_{\beta }(f_u)\end{array}\right]\right\}$$

(12)

where $CD$ denotes the cosine distance between two points. The frequencies corresponding to the endpoints of the clustering clusters can be detected by seeking the first $\widehat{n}$ peak points in the sum of the power data marked in red Figure 5. The points marked in red are the endpoints of the detected linear clusters to be determined in Figure 6. Therefore, the contribution degree of the uth point to the ith linear cluster can be calculated by the contribution degree function given above.

Based on the contribution degree ${\eta }_u^i$ obtained from Equation (11), the ith TF soft mask matrix ${\mathit{M}}_i$ is constructed by the following formula:

$${\mathit{M}}_i={\left[\begin{array}{cccc}{\eta }_1^i& {\eta }_2^i& \cdots & {\eta }_F^i\\ {\eta }_1^i& {\eta }_2^i& \cdots & {\eta }_F^i\\ ⋮& ⋮& \ddots & ⋮\\ {\eta }_1^i& {\eta }_2^i& \cdots & {\eta }_F^i\end{array}\right]}_{T\times F}$$

(13)

where $F$ is the number of STFT frequencies; $T$ is the number of time series of STFT. $\widehat{n}$ soft mask matrices can be constructed in this way.

The ith source signal $y_i(t)$ to be reconstructed is

$$y_i(t)=ISTFT\left[X(t,f)\odot M_i(t,f)\right]$$

(14)

where $X(t,f)$ indicates the sum of the observed signals.

The flowchart of the proposed strategy is shown in Figure 7. Firstly, the angular distribution $\theta$ is converted from the power ratio by Equation (8); Secondly, the number of source signals $\widehat{n}$ is estimated through the KDE algorithm followed by the peak density detection; Afterwards, the contribution degree $\eta$ of the source signal in the observed signal is calculated by the contribution degree function based on the cosine distance $d$. Finally, the TF soft mask matrix $\mathit{M}$ is constructed based on the contribution degree $\eta$, and the separated TF signals are reconstructed.

4. Experimental Results and Analysis

Our simulations are aimed at investigating the anti-aliasing performance and anti-noise performance of the proposed algorithm. To measure the degree of spectrum aliasing of the source signals, we define the degree of spectrum aliasing $\lambda$:

$$\lambda =\frac{B_o}{B}\times 100\%$$

(15)

where $B$ represents the source signal spectrum bandwidth, $B_o$ and represents the maximum aliasing width between the source signal spectrum and the adjacent signal spectrum.

Since the “boundary effect” of STFT has an impact on the separation performance [10], we selected the Kaiser window $(\beta =6.865)$ as the window function. The evaluation criteria of separation performance were applied in our simulations, which include the source-to-distortion ratio (SDR), the source-to-interferences ratio (SIR), the source-to-artifacts ratio (SAR) [23], and the separated bit error rate (BER). In order to obtain bitstream information, the coherent demodulation method was used to demodulate the separated source signals. The mixed signals were separated by the SCA algorithm proposed in this paper and the literatures [8,10]. The comparison algorithm in [8] belonged to the classic two-stage method, and [10] used the TF masking method to separate the source signal.

4.1. Simulation 1: The Selection of Scaling Factor $\gamma$ in the Contribution Degree Function

In this simulation, we verified the impact of the scaling factor $\gamma$ on the separation performance of the proposed algorithm. The simulation was conducted under the condition of additive white Gaussian noise with a signal-to-noise ratio (SNR) of 16.4 dB. It is assumed that the three signals with different modulation (2ASK/BPSK/QPSK) are mixed instantaneously, in which the symbol rate of the signals are 5 × 10⁶, 5 × 10⁶, 10 × 10⁶ B, respectively. The modulation carrier frequencies are 13.5, 20, and 26.5 MHz, respectively, with the $\lambda$ of each signal being 35%. The mixing matrix $\mathit{A}$ is

$$\mathit{A}=\left[\begin{array}{ccc}0.21& 0.63& 0.89\\ 0.92& 0.58& 0.22\end{array}\right]$$

(16)

The separation performance with different $\gamma$ is shown in Figure 8; the bigger value is, the better the separation performance is. It can be seen that, with $\gamma$ increasing, the SIR decreased significantly while the SAR and SDR remained almost constant. Thus we take $\gamma$ as 0.5 in the following simulation experiments. Figure 9 and Figure 10 show the waveforms of the source signals and their mixed signals, respectively. Comparatively, Figure 11 shows the waveforms of the signals separated by the proposed algorithm with $\gamma =0.5$. It can be seen that the waveforms of the separated signals are basically consistent with the source signals. Although the waveforms have some distortion at the junction of bit 0 and bit 1, which is caused by the “boundary effect” of STFT, it does not affect the subsequent demodulation and judgment. Figure 12 represents the waveforms after coherent demodulation. The bit data can be recovered by setting the appropriate judgment level.

4.2. Simulation 2: The Anti-Aliasing Performance Simulation

In order to analyze the anti-aliasing performance for signals with different modulation methods, two sets of signals were used as the source signals, which respectively were three signals with the same BPSK modulation and three signals with different modulation (2ASK/BPSK/QPSK). The symbol rate of a single source signal is 5 × 10⁶ B. The modulation carrier frequencies were designed according to different $\lambda$. In this simulation, it is assumed that the source signals propagate in two channels before being received by the sensors, and the SNR is 15 dB. The first channel of signals arrived instantaneously, and the corresponding mixing matrix is

$${\mathit{A}}_1=\left[\begin{array}{ccc}0.23& 0.32& 0.52\\ 0.44& 0.47& 0.13\end{array}\right]$$

(17)

The second channel of signals arrived with delay; the signal propagation delay matrix and the corresponding mixing matrix are

$$\mathit{T}=\left[\begin{array}{ccc}0.1& 0.05& 0.2\\ 0.18& 0.1& 0.04\end{array}\right]$$

(18)

$${\mathit{A}}_2=\left[\begin{array}{ccc}0.22& 0.42& 0.37\\ 0.41& 0.17& 0.28\end{array}\right]$$

(19)

The SDR and BER of the separated signals with the increase in $\lambda$ are shown in Figure 13 and Figure 14. From the simulation results, it can be seen that in the environment with the SNR of 15 dB, as the source signal spectrum aliasing gradually intensifies, the separation performance of the proposed algorithm and the comparison algorithms are reduced to varying degrees. However, for signals with the same BPSK modulation (Figure 13), the average $\mathrm{SDR}$ of the separated signals by the proposed algorithm is increased by 1.05 and 2.35 dB compared with the comparison algorithms. For signals with different modulation (Figure 14), the average $\mathrm{SDR}$ of the separated signals by the proposed algorithm is increased by 0.68 and 2.48 dB. The BER is significantly lower than that of the comparison algorithms. As a result, the anti-aliasing performance of the algorithm in this paper is excellent.

4.3. Simulation 3: Simulation of Algorithm Anti-Noise Performance

In this simulation, the carrier frequency of each of the two groups of source signals was set to 14 MHz, 20 MHz, and 26 MHz, and the $\lambda$ was fixed at 40% correspondingly. In the environment of the SNR from 5 to 30 dB, the method in simulation two was used to mix the source signals. The SDR and BER of the separated signals with the increase in the SNR are shown in Figure 15 and Figure 16:

It can be seen from Figure 15 and Figure 16 that, with the SNR increases, the separation performance of the algorithm proposed in this paper and the comparison algorithms have been effectively improved. However, compared with the comparison algorithms, the SDR and BER of the separated signals by the proposed algorithm are higher and lower, respectively, indicating that the signal separation performance is better. Especially for signals with the same BPSK modulation (Figure 15), when the SNR is 10, the BER of the separated signals is lower than 10⁻². For signals with different modulation (Figure 16), the BER drops below 10⁻² when the SNR reaches 20 dB.

5. Conclusions

In this paper, we have discussed an underdetermined mixed blind signal separation method with the application of TF overlapped communication signals. The classical SCA methods to solve the UBSS problem require high sparsity of the signal. For TF overlapped signals, the classical SCA methods cannot be applied to such problems. To tackle this issue, we propose a novel method for separating TF overlapped signals based on the SCA algorithm. The core idea of the method is to construct a reasonable TF soft mask matrix by designing a contribution degree function. In the proposed method, we utilize the KDE algorithm to obtain the angle probability density distribution, followed by which the source number is estimated by detecting the peak points of the distribution. Then the contribution degree function is designed to generate the contribution degree, based on which the TF soft mask matrix is constructed to obtain the reconstructed TF signal. The sparsity requirement of the algorithm is reduced by constructing a reasonable TF soft mask matrix, which makes the separation algorithm suitable for separating TF overlapped signals. Compared with the classical SCA algorithm, the most significant contribution of the proposed algorithm is to expand the application scope of the SCA algorithm, which no longer requires the signal to have sparsity in the TF domain. In addition, the source estimation process prior to separation enables it to adapt to the case in which the number of sources is unknown. The algorithm’s anti-aliasing ability and anti-noise ability are simulated and verified by simulation, which demonstrates that the proposed method enables to overcome the poor performance of blind separation for TF aliasing communication signals by classical separation algorithms.