Integrated Sensing and Communication via Orthogonal Time Frequency Space Signaling with Hybrid Message Passing Detection and Fractional Parameter Estimation

Abstract

For the orthogonal time frequency space (OTFS) modulation, we generally multiplex symbols on a new type of carrier waveform in the delay-Doppler (DD) domain. These two parameters can be used to infer the range (R) and velocity (V) of the communication user and sensing target; thus, it is natural for the OTFS to be implemented in integrated sensing and communication (ISAC). A framework for ISAC based on OTFS modulation is proposed in this paper, in which the matched filter scheme with fractional parameter estimation is implemented for radar sensing. In addition, the hybrid message passing (MP) detection algorithm is developed for OTFS symbol demodulation. According to the simulation results, fractional DD shifts associated with multiple targets can be accurately obtained through the proposed framework. Meanwhile, the bit error rate under the proposed detector is less than ${10}^{-4}$ when the signal-to-noise ratio is high enough.

1. Introduction

The wireless systems of the future are poised to facilitate wireless communication of superior quality as well as to provide remarkably precise sensing services. The consensus is widespread that sensing capabilities within the forthcoming generation of wireless networks will hold substantially greater significance compared to the networks currently in operation [1,2,3,4].

The majority of scholarly investigations are directed towards orthogonal frequency division multiplexing (OFDM) as a means to achieve the amalgamation of integrated sensing and communication (ISAC) [5,6,7,8,9,10,11,12]. The integration of sensing and communication in the context of ISAC using OTFS signals holds significant promise for addressing challenges in dynamic and multipath-rich environments. ISAC with OTFS signals provides a unified framework where sensing and communication functionalities are seamlessly integrated. This integration allows for joint optimization of resources and enhanced performance in scenarios where both tasks are critical. The DD domain representation in the OTFS provides a powerful way to characterize the environment. In sensing applications, parameters such as target range, velocity, and direction can be directly related to specific regions in the DD domain. OFDM boasts diverse merits, including advantages such as reduced detection intricacy and heightened resilience in radar target detection. Nonetheless, the new requirements for communication and sensing systems pose new challenges for OFDM modulation. For instance, the challenge posed by the increased peak-to-average-power ratio (PAPR) in OFDM systems may hinder power efficiency, particularly concerning carriers at higher frequencies. Additionally, instances of high mobility bring about the deterioration of sub-carrier orthogonality due to the pronounced Doppler effect, thereby potentially leading to considerable degradation in communication performance. Moreover, the channel response in scenarios characterized by substantial mobility exhibits notable fluctuations across distinct coherence regions. In such scenarios, OFDM requires frequent channel estimation to obtain accurate channel state information, leading to a significant increase in signaling overhead. These difficulties serve as a source of inspiration for researchers, propelling them to devise an innovative modulation technique with the capacity to guarantee resilient communication performance in situations characterized by significant mobility.

OFDM and OTFS modulation are different modulation and signal processing techniques designed to address distinct challenges in communication systems. While OFDM is widely used in various current systems, including 4G and 5G cellular networks, OTFS modulation is specifically designed to excel in high-mobility scenarios and challenging channel conditions.

• Channel Characteristics: OTFS modulation is explicitly designed to handle time-varying channels, making it well suited to high-mobility scenarios. It excels in scenarios with rapid changes in the channel, such as vehicular communication or high-speed rail.
• Mobility and Doppler Tolerance: OTFS modulation is designed to handle scenarios with high Doppler shifts. Its DD domain representation naturally separates signals, enabling effective mitigation of Doppler effects.
• Multipath Handling: OTFS modulation excels in multipath environments due to its ability to separate signals in the DD domain, providing superior performance in challenging propagation conditions. OTFS modulation may have an increased computational complexity compared to OFDM.
• Accuracy in Time-Varying Channels: OTFS modulation excels in accuracy in time-varying channels, offering robust communication and sensing capabilities in high-mobility environments.

The newly unveiled OTFS modulation has garnered considerable attention on account of its prospective capability to enable reliable communication in contexts marked by elevated mobility and multi-path propagation, as pointed out in [13,14,15,16,17].

The OTFS modulation framework is designed to address challenges in communication systems operating in highly dynamic environments with severe multipath and time-varying channel conditions. Under environmental conditions with high interference levels or fluctuating signal strength, the OTFS modulation framework can also demonstrate its advantages. For example:

• Multipath Fading and Time Variability

  (a)

  Performance Improvement: OTFS modulation is particularly well suited to environments with severe multipath fading. By transforming the received signals from the time-delay domain to the DD domain, OTFS modulation can effectively separate multipath components. This transformation enhances the system’s ability to combat fading and improves communication performance.

  (b)

  Resilience to Time Variability: OTFS modulation is designed to handle time-varying channels, making it robust in scenarios where the channel conditions change rapidly. This is crucial in dynamic environments, such as those encountered in vehicular communication or mobile communication, where the channel characteristics can vary over short time intervals.

• Interference Rejection

  (a)

  Spatial Separation: The DD domain representation in OTFSs provides a form of spatial separation for different signals. This can aid in the rejection of interference, as signals from different directions in the DD domain can be more easily distinguished.

  (b)

  Improved Detection in Noisy Environments: The ability of OTFS modulation to focus on distinct DD signatures allows for improved detection in the presence of interference and noise. The framework’s capacity to isolate signals in the DD domain contributes to enhanced interference rejection capabilities.

The OTFS modulation framework shows promise in addressing challenges posed by varying environmental conditions, including interference and fluctuating signal strengths. Its unique approach of focusing on the DD domain provides advantages in multipath fading scenarios and dynamic channel conditions, making it a candidate for applications requiring reliable communication in challenging environments. In contrast to conventional approaches that operate in the time-frequency (TF) domain, the OTFS modulation strategy conveys information symbols (ISs) across the two-dimensional (2D) DD domain, as explained in [14,18,19,20,21,22]. In this domain, wireless channel paths with distinct DD shifts represent resolvable paths. Significant benefits are associated with employing the DD domain for channel characterization. In communication scenarios featuring pronounced mobility, the DD domain channel typically exhibits sparsity and quasi-static behavior in contrast to the highly dynamic properties in the TF domain, as emphasized in references [23,24,25,26]. Additionally, the OTFS boasts a reduced PAPR [25,27,28,29,30], simplifying implementation through more efficient power amplifiers, unlike its OFDM counterpart. In traditional OFDM systems, a high PAPR requires power amplifiers to operate with a larger dynamic range, leading to increased power requirements and impacting the overall energy efficiency of the system. OTFS modulation utilizes a unique signal representation in the DD domain, which may impact the PAPR characteristics differently compared to traditional OFDM. The energy efficiency of the OTFS modulation framework may be influenced by the computational complexity of its signal processing algorithms. Adaptive techniques for power control and modulation parameters may be employed to manage power levels dynamically, contributing to energy efficiency. Notably, the DD shifts, which correspondingly infer the velocity and range of targets, are pivotal, making OTFS modulation a logical selection for achieving ISAR (inverse synthetic aperture radar) objectives.

Recent scholarly endeavors have examined radar sensing using the OTFS framework, as evidenced in [31,32]. A maximum likelihood (ML) algorithm for estimating R and V in the OTFS is introduced by the authors in [31]. The results demonstrate that the OTFS signal has superior communication effectiveness compared to OFDM while preserving the radar estimation performance limit. Furthermore, Ref. [32] proposed a matched-filter (MF) algorithm to estimate the R and V parameters, leveraging the structure of the OTFS effective channel (EC). The findings indicate that OTFS-based speed estimations of targets outperform those of OFDM.

Nevertheless, the potential of employing OTFS modulation for radar sensing remains largely untapped. To illustrate, the complexity of the algorithm introduced in [31] increases almost threefold with respect to the size of the OTFS frame, indicating a relatively elevated computational load. Conversely, the investigation carried out in [32] solely accounts for integer-valued DD shifts. Nonetheless, this approach might prove impractical in real-world wireless networks, where temporal resources are constrained, leading to inadequate Doppler resolution. As a result, the inclusion of fractional Doppler components becomes inevitable. Furthermore, achieving precise sensing performance mandates an exploration of radar sensing with limited frequency resources, even though integer delay suffices for communication design in most cases. Consequently, considering both fractional DD indices becomes imperative for effective radar sensing. These concerns collectively drive the motivation to develop an approach that facilitates radar sensing through OTFS signaling, accommodating fractional DD shifts.

The practical implementation of OTFS modulation is facilitated by its compatibility with conventional OFDM and pulse-shaped OFDM transceiver structures. This is achieved through the straightforward addition of pre-processing and post-processing blocks. The choice of shaping pulses in a communication or sensing system can significantly impact the overall performance and the spectral efficiency, especially in the context of sensing where accurate channel estimation and signal recovery are crucial. Shaping pulses are waveforms that are used to shape the transmitted signal and, when appropriately chosen, can enhance various aspects of the sensing performance. By using pulses with well-designed spectral characteristics, it is possible to optimize the use of available bandwidths. The choice of shaping pulses influences the duration and bandwidth of the transmitted signal. Depending on the sensing requirements and the characteristics of the environment, it may be desirable to use pulses with specific durations and bandwidths.

In addition, to achieve full diversity in TF (time frequency) considerations, effective detection methods are essential. Implementing the maximum a posteriori algorithm (MAP) for each symbol, as detailed in reference [33,34,35,36,37,38], represents an optimal approach for data detection within interference channels. The MAP algorithm can be represented by an intuitive cyclic factor graph. However, it is crucial to note that the MAP algorithm becomes computationally infeasible since the number of computation-interfering elements grows exponentially. This exponential growth renders it impractical for scenarios involving multiple signal paths, which requires a low-complexity detector.

In an effort to mitigate this computational burden, numerous approximate methods have been developed. For instance, in reference [39], an alternative MAP algorithm is introduced, which leverages Gaussian approximation (GA) to reduce the complexity of these procedures. Reference [9] explores the use of detectors founded on approximate message passing (AMP). Furthermore, reference [40] introduces a unitary approximate message passing detector, which has the same detection performance as the minimal mean square error (MMSE). Lastly, reference [41] presents a detector based on variational Bayes (VB), which has significantly lower computational demands while ensuring convergence with significantly lower computational demands.

This paper introduces a method for estimating fractional DD method. First, the radar receiver captures the echoes reflected by targets in the DD domain and subsequently carries out 2D matched filtering involving the DD domain symbols between the receiver and transmitter. With the correlation matrix in hand, we deploy a difference-based technique that entails computing the discrepancy between the indices corresponding to the highest and second highest magnitudes in the correlated matrix. This calculation allows us to determine the fractional DD indices. For the OTFS symbol detection part, a novel hybrid MP detection algorithm has been introduced. When fractional Doppler effects are present, the channel responses are dispersed across the entire Doppler spectrum. This dispersion leads to more interfering symbols and a high computational complexity. However, only a subset of path gains corresponding to a specific Doppler frequency shifts rather than the whole dispersion, which will significantly affect the transmission of the OTFS signal [42,43,44,45,46]; therefore, a selective approach is employed. Specifically, data detection is implemented based on the paths with the highest gains, which are named main paths, associated with the dispersed Doppler indices. We employed a non-approximated MP algorithm for the primary propagation paths, whereas a GA-based MP algorithm was used for the secondary propagation paths. Through our simulation results, we illustrate its effectiveness regarding accuracy.

2. System Model

In this section, we first introduce the fundamental principles underlying OTFS modulation. Subsequently, we expound upon the input–output correction of OTFS symbols within the DD domain [47,48,49,50,51,52].

2.1. Introduction of OTFS Modulation

We utilize the symbols $N_T\in \mathbb{N}$ and $M_S\in \mathbb{N}$ to signify the quantity of time slots and subcarriers allocated to each OTFS frame, respectively, with $N_T$ belonging to the set of positive integers. The discrete DD plane can be succinctly depicted as follows:

$$\mathbf{\Gamma }=\left\{\left(\frac{k}{N_{T}T},\frac{l}{M_S\Delta f}|k\in [0,N_T-1],l\in [0,M_S-1]\right)\right\}.$$

(1)

The duration of a single time slot can be represented by T, and the separation between subcarriers is denoted as $\Delta f$. Consequently, the entire duration of an OTFS frame is given by $N_{T}T$, while the occupied bandwidth of the system is $M_S\Delta f$. In terms of resolution, we attain a delay resolution of $\frac{1}{M_S\Delta f}$ and a Doppler shift resolution of $\frac{1}{N_{T}T}$.

The cross-ambiguity function (CAF) between the pulse-shaping filters (PSFs) at the transmitter $g_{tx}\left(t\right)$ and the receiver $g_{rx}$ can be expressed as follows:

$$A_{g_{rx}{g}_{tx}}\left(t,f\right)\stackrel{\Delta }{=}{\int }_t{g}_{tx}\left(t^{\prime }\right)g_{rx}^{*}(t^{\prime }-t)e^{j2\pi \nu (t^{\prime }-t)}d{t}^{\prime }.$$

(2)

To ensure clarity in our discourse, we focus our analysis on situations in which both the transmitter and receiver operating within the OTFS framework are endowed with ideal PSFs. As elucidated in [53], these filters exhibit bi-orthogonal characteristics, which is delineated as follows:

$$\begin{array}{cc}\hfill & \hfill A_{g_{rx}{g}_{tx}}{(t,f)|}_{t=(n-n^{\prime })T-\frac{l_{\tau }}{M_S\Delta f},(m-m^{\prime })\Delta f-\frac{k_{\nu }}{N_{T}T}} \\ \hfill & \hfill =\delta \left[n\right]\delta \left[m\right]q_{{\tau }_{max}}(t-nT)q_{{\nu }_{max}}(f-m\Delta f),\end{array}$$

(3)

where

$$q_a\left(t\right)=\left\{\begin{array}{c}\hfill 1-a<x<a\\ \hfill 0otherwise.\end{array}\right.$$

While the OTFS framework presents promising capabilities for addressing challenges in wireless communication and sensing systems, there are several practical challenges that need to be considered during implementation in real-world scenarios, including:

• Computational Complexity: The OTFS framework involves complex mathematical transformations, including the conversion from the time-delay domain to the DD domain. These operations can be computationally intensive, especially in real-time applications or resource-constrained environments.
• Channel Estimation and Tracking: Accurate channel estimation and tracking are crucial for the success of the OTFS framework. In dynamic environments with rapidly changing channel conditions, developing robust algorithms for real-time channel estimation and tracking becomes a challenge.
• Synchronization: Achieving precise synchronization is essential for accurate transformation between domains in the OTFS framework. Synchronization challenges may arise in scenarios with high mobility, multipath propagation, and varying signal delays.
• Adaptability to Non-Ideal Conditions: Real-world environments often introduce non-ideal conditions such as frequency-selective fading, non-line-of-sight (NLOS) scenarios, and interference. The OTFS framework needs to be adaptable to such conditions, requiring advanced signal processing techniques and algorithms.

While the OTFS framework shows promise, its successful implementation in real-world wireless communication and sensing systems requires careful consideration of these practical challenges.

2.2. DD Domain Input–Output Relationship in OTFS Modulation

We will now delve into a comprehensive elucidation of the OTFS transmission process. To begin with, a sequence of information bits is subjected to a mapping procedure, resulting in a symbol set situated within the DD domain, as described in previous references [54,55,56,57,58]. This symbol set is formally denoted as $\left\{X_{\mathrm{DD}}[k,l]\right|k=0,\dots ,N_T-1,$ $l=0,\dots ,M_S-1\}$. It is essential to underscore that both the indices k and l correspond to the DD references, respectively. Furthermore, it is worth noting that $X_{\mathrm{DD}}$ belongs to the modulation alphabet set $\mathbb{A}$, defined as $\mathbb{A}=\left\{a_i\in \mathbb{C}|i=1,2,\dots ,|A|\right\}$, encompassing a range of modulation schemes, including quadrature amplitude modulation (QAM). Subsequently, the OTFS symbols undergo a transformation using the inverse symplectic (IS) FFT. This transformation process will be further elaborated upon in the subsequent discussion.

$$X_{\mathrm{TF}}[n,m]=\sum _{k=0}^{N_T-1}\sum _{l=0}^{M_S-1}{X}_{\mathrm{DD}}[k,l]e^{j2\pi \left(\frac{nk}{N_T}-\frac{ml}{M_S}\right)},$$

(4)

for $n=0,\dots ,N_T-1$ and $m=0,\dots ,M_S-1$.

In the next step, the modulator, which functions in the TF domain, processes the symbols $X_{\mathrm{TF}}[n,m]$, transforming them into a continuous-time waveform represented as $s\left(t\right)$. This conversion is realized by harnessing the transmitter’s pulse-shaping function (PSF) $g_{tx}\left(t\right)$, making use of the Heisenberg transform technique as detailed in reference [59].

$$s\left(t\right)=\sum _{n=0}^{N_T-1}\sum _{m=0}^{M_S-1}{X}_{\mathrm{TF}}[n,m]g(t-nT)e^{j2\pi m\Delta f(t-nT)}.$$

(5)

Following its propagation through a time-varying channel, as elucidated in reference [60], the signal, initially denoted as $s\left(t\right)$, reaches the receiver. So, we can obtain the received signal, designated as $r\left(t\right)$:

$$r\left(t\right)=\int \int h(\tau ,\nu )s(t-\tau )e^{j2\pi \nu (t-\tau )}d\tau d\nu +w\left(t\right).$$

(6)

Within this equation, the term $w\left(t\right)$ signifies the existence of additive white Gaussian noise (AWGN), represented as ${\mathcal{N}}_0$. On the other hand, $h(\tau ,\nu )\in \mathbb{C}$ denotes a complex baseband channel impulse response, where $\tau$ and $\nu$ represent the channel’s reaction to an impulse with DD shift ($\tau$ and $\nu$, i.e.). The equation is

$$h(\tau ,\nu )=\sum _{i=1}^P{h}_i\delta (\tau -{\tau }_i)\delta (\nu -{\nu }_i).$$

(7)

In this context, our notation includes $P\in \mathbb{Z}$ to represent the number of targets present. We further employ $h_i$, ${\tau }_i$, and ${\nu }_i$ to signify the attributes of the ith target, specifically the reflection DD shift and channel coefficient, respectively. Additionally, the velocity and range of the ith target are denoted by $R_i$ and $v_i$. Within this framework, we can express the corresponding Doppler frequency ${\nu }_i\in \mathbb{R}$ and delay ${\tau }_i\in \mathbb{R}$ in a mathematical formulation.

$${\tau }_i=\frac{2R_i}{c}=\frac{l_{{\tau }_i}}{M_S\Delta f},{\nu }_i=\frac{2f_c{v}_i}{c}=\frac{k_{{\nu }_i}}{N_{T}T}.$$

(8)

Subsequently, we establish a notation where $f_c$ denotes the carrier frequency and c represents the speed of light. Furthermore, we introduce the utilization of $l_{{\tau }_i}\in \mathbb{R}$ and $k_{{\nu }_i}\in \mathbb{R}$ as the indices characterizing Doppler and delay properties for the ith target. In the context of this study, we expand our analysis to encompass fractional elements of Doppler and delay. These indices, specifically $l_{{\tau }_i}=l_i+{\iota }_i$ and $k_{{\nu }_i}=k_i+{\kappa }_i$, incorporate the following elements: $l_i\in \mathbb{Z}$ and $k_i\in \mathbb{Z}$, while ${\iota }_i\in \mathbb{R}$ and ${\kappa }_i\in \mathbb{R}$ signify the real numbers representing the fractional aspects.

Upon reception at the receiver, the incoming signal, denoted as $r\left(t\right)$, is subjected to a transformation to the TF domain. This transformation is executed through the application of the Wigner transform, as elaborated in reference [59]. It is paramount to underscore that this Wigner transform can equivalently be construed as the CAF, denoted as $A_{g_{rx},r}(t,f)$. Furthermore, it may be formally defined as:

$$\begin{array}{cc}\hfill & Y_{\mathrm{TF}}(t,f)=A_{g_{rx},r}(t,f)\hfill \\ \hfill & \stackrel{\Delta }{=}\int g_{rx}^{*}(t^{\prime }-t)r\left(t^{\prime }\right)e^{-j2\pi f(t^{\prime }-t)}d{t}^{\prime }.\hfill \end{array}$$

(9)

Upon performing the sampling of $Y(t,f)$, we derive the discrete output in the following manner:

$$Y_{\mathrm{TF}}[n,m]=Y_{\mathrm{TF}}{(t,f)|}_{t=nT,f=m\Delta f},$$

(10)

for $m=0,\dots M_S-1$ and $n=0,\dots ,N_T-1$. Consequently, through the application of the SFFT to $Y_{\mathrm{TF}}[n,m]$, we obtain the discrete output symbols $Y_{\mathrm{DD}}[k,l]$ in the DD domain.

$$Y_{\mathrm{DD}}[k,l]=\frac{1}{\sqrt[]{N_T{M}_S}}\sum _{n=0}^{N_T-1}\sum _{m=0}^{M_S-1}{Y}_{\mathrm{TF}}[n,m]e^{-j2\pi \left(\frac{nk}{N_T}-\frac{ml}{M_S}\right)}.$$

(11)

In cases where an ideal PSF is employed, as elaborated in reference [61], the IO relationship between the OTFS symbols can be described as:

$$\begin{array}{cc}\hfill Y_{\mathrm{DD}}[k,l]=\sum _{n=0}^{N_T-1}\sum _{m=0}^{M_S-1}& X_{\mathrm{DD}}[n,m]h_{\omega }[k-n,l-m]\hfill \\ \hfill & +Z_{\mathrm{DD}}[k,l].\hfill \end{array}$$

(12)

Here, $Z_{\mathrm{DD}}$ adheres to a complex Gaussian distribution characterized by a mean of zero and a covariance of ${\sigma }^2\mathit{I}$, representing the the DD domain effective noise. Additionally, $h_{\omega }[k,l]$ is the EC in the DD domain, acquired via the following means:

$$h_{\omega }[k,l]=\sum _{i=1}^P{h}_i\omega (k-k_{{\nu }_i},l-l_{{\tau }_i})e^{-j2\pi {\nu }_i{\tau }_i}.$$

(13)

In this context, the function $\omega (k-k_{{\nu }_i},l-l_{{\tau }_i})$ serves as the sampling function. The parameters ${\nu }_i$ and ${\tau }_i$ are defined in accordance with Equation (8). When both the TF transmitter and receiver opt to employ a rectangular window, as extensively elucidated in reference [53], the sampling function takes on a more streamlined and simplified form, as follows:

$$\begin{array}{cc}\hfill & \omega (k-k_{{\nu }_i},l-l_{{\tau }_i})=\mathcal{G}(k-k_{{\nu }_i})\mathcal{F}(l-l_{{\tau }_i}),\hfill \\ \hfill & \mathcal{G}(k-k_{{\nu }_i})\stackrel{\Delta }{=}\frac{1}{N_T}\sum _{k^{\prime }=0}^{N_T-1}{e}^{-j2\pi \frac{(k-k_{{\nu }_i})k^{\prime }}{N_T}},\hfill \\ \hfill & \mathcal{F}(l-l_{{\tau }_i})\stackrel{\Delta }{=}\frac{1}{M_S}\sum _{l^{\prime }=0}^{M_S-1}{e}^{j2\pi \frac{(l-l_{{\tau }_i})l^{\prime }}{M_S}}.\hfill \end{array}$$

(14)

As expounded in reference [61], the non-zero components of the EC in the DD domain exhibit uniform localization patterns when both the rectangular and ideal PSFs are utilized. The primary distinction lies in the introduction of an additional phase offset. Consequently, the algorithms delineated in this study can be readily and seamlessly adapted to suit scenarios where a rectangular PSF is employed.

3. OTFS-Based Radar Sensing

In this section, we introduce a target detection method and the 2D MF-based technique to effectively estimate the parameters.

3.1. Two-Dimensional Matched-Filter Operation

In this subsection, our objective is to provide a detailed explanation of the 2D MF operation for integer parameter estimation. When we initially examine the representation of the received OTFS symbol matrix in the DD domain, the direct identification of target elements within ${\mathbf{Y}}_{\mathrm{DD}}$ (received data in the DD domain) becomes challenging due to the presence of IS. These symbols are influenced by the time-varying channel, which leads to the emergence of overlapping responses in the received DD domain signals. These responses arise from both the channel characteristics and the superimposed IS. In this context, we draw inspiration from pulse compression radar sensing and implement a 2D MF-based estimator. This estimator is likened to performing simultaneous pulse compression in both the DD dimensions, which, in turn, allows us to effectively capture the DD parameters.

To represent the matrix associated with the 2D MF as $\mathbf{V}$, the diverse DD indices of the MF coefficient are denoted as:

$$V[k,l]=\sum _{n=0}^{N-1}\sum _{m=0}^{M_S-1}{Y}_{\mathrm{DD}}^{*}[n,m]X_{\mathrm{DD}}\left[{[n-k]}_{N_T},{[m-l]}_{M_S}\right].$$

(15)

To provide a more lucid explanation of the proposed approach, we have devised a simplified radar sensing example using OTFS principles. In this illustrative scenario, we assume the existence of five targets, denoted as $P=5$. The symbols were randomly generated from a four-QAM scheme. We set the parameters $M_S$ to 32 and $N_T$ to 16, with specific DD indices assigned to each target, represented as $\{4,9,3\}$ and $\{7,1,4\}$, respectively. The resulting symbol matrix in the DD domain, under this particular configuration, is visually represented in Figure 1. As illustrated, the matrix $Y_{\mathrm{DD}}$ exhibits a dense population, primarily attributable to the overlapping responses originating from the DD domain symbols. The task of target identification within this matrix is inherently intricate. However, through the application of a 2D MF computation, the targets become more localized. This process can be likened to a specialized form of pulse compression in the DD domain, analogous to its role in radar sensing. Ultimately, this improves the retrieval of responses in the DD domain.

3.2. GLRT-Based Target Detection

As we encounter the challenge of not knowing the precise number of targets when working with the 2D MF matrix, we have devised a step-by-step algorithm to identify them systematically. Once a target is located, we can simultaneously estimate its crucial characteristics. We continue this process until no more targets can be found.

Let us rewrite Equation (15) in a different form using matrices before we get into the details of the target detection algorithm. This will simplify the detection process and enhance comprehension. We achieve this by substituting Equations (13) and (12) into Equation (15) and performing straightforward calculations.

$$\mathbf{v}=\sum _{i=1}^P{h}_i^{*}{e}^{j2\pi \frac{l_{{\tau }_i}{k}_{{\nu }_i}}{M_S{N}_T}}\tilde{\mathbf{X}}{\mathbf{W}}_{({\tau }_i,{\nu }_i)}^{\mathrm{H}}{\mathbf{x}}^{*}+{\tilde{\mathbf{X}}}^{\mathrm{T}}{\mathbf{z}}^{*},$$

(16)

in which the vector ${\mathbf{x}}^{*}$ has an i-th entry in the matrix $\mathbf{X}$ at row $k_1$ and column $l_1$, and $\tilde{\mathbf{X}}$ is a rearranged matrix of transmitted symbols, which are circulant shifts of the symbols in $\mathbf{X}$. The specific element of the matrix $\mathbf{X}$ located at row $k_1$ and column $l_1$ is represented as follows:

$$\tilde{X}[i,j]=x[{[k_1-k_2]}_{N_T},{[l_1-l_2]}_{M_S}].$$

(17)

In this context, where we define $i=k_1{M}_S+l_1$ and $j=k_2{M}_S+l_2$, $k_{(·)}\in [0,N_T-1]$ and $l_{(·)}\in [0,M_S-1]$ represent row and column indices of the entry in $\mathbf{X}$. The element at position $(k{M}_S+l)$ in the EC vector $\mathbf{h}$ corresponds to the entry in the EC matrix ${\mathbf{h}}_{\omega }$ at row k and column l, and the EC sampling function is represented in matrix form, i.e., ${\mathbf{W}}_{({\tau }_i,{\nu }_i)}^{\mathrm{H}}$. The element in the matrix $\mathbf{W}$ at $i=k_1{M}_S+l_1$ and $j=k_2{M}_S+l_2$ is defined as follows:

$$\tilde{W}[i,j]=\omega (k_2-k_1-k_{{\nu }_i},l_2-l_1-l_{{\tau }_i}),$$

(18)

where $\omega (\nu ,\tau )$ is precisely defined as outlined in Equation (14).

We conduct a binary hypothesis test to obtain the number of targets. In this test, ${\mathcal{H}}_0$ means that the ith target is not present, and ${\mathcal{H}}_1$ means that the ith target is present. The observations can be described as follows:

$$\mathbf{v}=\left\{\begin{array}{ccc}\hfill & \tilde{\mathbf{X}}{\mathbf{z}}^{*}\hfill & \hfill ,\mathrm{under}{\mathcal{H}}_0\\ \hfill & h^{*}{e}^{j2\pi \frac{l_{\tau }{k}_{\nu }}{M_S{N}_T}}\tilde{\mathbf{X}}{\mathbf{W}}_{(\tau ,\nu )}^{\mathrm{H}}{\mathbf{x}}^{*}\hfill & \\ \hfill & +\tilde{\mathbf{X}}{\mathbf{z}}^{*}\hfill & \hfill ,\mathrm{under}{\mathcal{H}}_1\end{array}\right..$$

(19)

To address Equation (19), we consider h, $\tau$ and $\nu$ as deterministic but unknown variables, and use the generalized likelihood ratio test (GLRT) [62]. The inspection is as follows:

$$\Lambda \left(\mathbf{v}\right)=\underset{h,\tau ,\nu }{max}\frac{p\left(\mathbf{v}\right|{\mathcal{H}}_1;h,\tau ,\nu )}{p\left(\mathbf{v}\right|{\mathcal{H}}_0)}\stackrel{{\mathcal{H}}_1}{\underset{{\mathcal{H}}_0}{\gtrless }}\eta .$$

(20)

Here, where $\eta$ serves as the threshold, we make the assumption that $\mathbf{z}$ adheres to a complex Gaussian distribution characterized by parameters $\mathcal{CN}(0,{\sigma }^2)$. Consequently, we can signify that the product ${\tilde{\mathbf{X}}}^{\mathrm{T}}{\mathbf{z}}^{*}$ also conforms to a complex Gaussian distribution, with parameters $\mathcal{CN}(0,\frac{{\sigma }^2}{M_S{N}_T})$. Furthermore, by introducing the term $\tilde{h}=h^{*}{e}^{j2\pi \frac{l_{{\tau }_i}{k}_{{\nu }_i}}{M_S{N}_T}}$, the GLRT takes the following expression:

$$\Lambda \left(\mathbf{v}\right)=\frac{e^{-\frac{1}{{\sigma }^2}{min}_{h,\tau ,\nu }{\Vert \mathbf{v}-\tilde{h}\tilde{\mathbf{X}}{\mathbf{W}}_{(\tau ,\nu )}^{\mathrm{H}}{\mathbf{x}}^{*}\Vert }^2}}{e^{-\frac{1}{{\sigma }^2}{\Vert \mathbf{v}\Vert }^2}}\stackrel{{\mathcal{H}}_1}{\underset{{\mathcal{H}}_0}{\gtrless }}\tilde{\eta }.$$

(21)

When $\tau$ and $\nu$ are constant, the value of $\tilde{h}$ in Equation (21) is as follows:

$$\tilde{h}\approx \frac{{\mathbf{x}}^{\mathrm{T}}\mathbf{W}{\tilde{\mathbf{X}}}^{\mathrm{H}}\mathbf{v}}{{\Vert \mathbf{x}\Vert }^2}.$$

(22)

This approximation arises from the matrix $\frac{{\tilde{\mathbf{X}}}^{\mathrm{T}}{\tilde{\mathbf{X}}}^{*}}{M_S{N}_T}$ exhibiting an approximate identity property. The approximately equal sign converges to an equal sign as $M_S{N}_T$ approaches infinity. For a more comprehensive proof, we refer interested readers to the detailed explanation provided in reference [32]. By adopting the procedure outlined in reference [32] with a phase offset of zero, we establish the approximate identity property of $\frac{{\tilde{\mathbf{X}}}^{\mathrm{T}}{\tilde{\mathbf{X}}}^{*}}{M_S{N}_T}$. Considering the parameter $\tilde{h}$, we can formulate the GLRT:

$${\displaystyle \underset{\tau ,\nu }{max}\frac{\mid {\mathbf{x}}^{\mathrm{T}}\mathbf{W}(\tau ,\nu ){\tilde{\mathbf{X}}}^{\mathrm{H}}\mathbf{v}{\mid }^2}{{\sigma }^2{\Vert \mathbf{x}\Vert }^2}\stackrel{{\mathcal{H}}_1}{\underset{{\mathcal{H}}_0}{\gtrless }}\tilde{\eta }.}$$

(23)

Within this framework, the cell-averaging constant false alarm rate (CA-CFAR) technique is employed to ascertain an adaptive threshold, $\tilde{\eta }=log\left(\eta \right)$. The methodology for calculating $\tilde{\eta }(\nu ,\tau )$ is elucidated in the reference authored by Raviteja et al. (2018) [61]. Targets are discerned at locations where the peak magnitudes surpass this specified threshold, with the resulting DD parameters being considered as the attributes linked to these identified targets.

3.3. Fractional Parameter Estimation

Upon examining the peaks within the DD plane denoted as $\mathbf{\Gamma }$, we are limited to discerning only the integer components of the DD indices [63,64,65,66]. This constraint implies that the fractional segments of these respective indices remain undisclosed, thereby diminishing the precision of the sensing outcomes, as elaborated upon in references [67,68,69,70]. To rectify these concerns, this subsection will present an uncomplicated technique for estimating the fractional parameters.

Upon a closer examination of the 2D MF matrix illustrated in Figure 2, it becomes evident that targets with fractional indices exhibit a phenomenon known as power leakage (PL) into neighboring DD bins. A thorough exploration of the origins of this PL resulting from fractional indices is available in [53,71]. Building upon this insight, we introduce a distinctive method for estimating these fractional indices differentially, as detailed below.

To calculate the fractional component of the Doppler index of the i-th target, we first assume that the delay or Doppler shift is different for each target. Following the 2D MF operation, the row indices associated with the highest and second highest magnitudes within the $l_i$-th column of the DD matrix, denoted as $k_1^{\prime }$ and $k_2^{\prime }$, are:

$$\begin{array}{cc}\hfill k_1^{\prime }& =\underset{k\in \left\{⌈-N_T/2⌉,\dots ,⌈N_T/2⌉-1\right\}}{arg\; max}\left|V[k,l_i]\right|,\hfill \\ \hfill k_2^{\prime }& =\underset{k\in \left\{⌈-N_T/2⌉,\dots ,⌈N_T/2⌉-1\right\}\setminus \left\{k_{{\nu }_1}^{\prime }\right\}}{arg\; max}\left|V[k,l_i]\right|.\hfill \end{array}$$

(24)

Upon establishing the values of $k_1^{\prime }$ and $k_2^{\prime }$, a significant observation comes to light. In noise-free scenarios, the genuine Doppler index, represented as $k_i+{\kappa }_{{\nu }_i}$ and linked to the i-th target, logically falls within the interval delineated by $k_{{\nu }_1}^{\prime }$ and $k_{{\nu }_2}^{\prime }$. It is worth emphasizing that $k_i$ coincides with $k_{{\nu }_1}^{\prime }$, and the discrepancy between $k_{{\nu }_2}^{\prime }$ and $k_{{\nu }_1}^{\prime }$ precisely equals 1. Moreover, the proportion between the magnitudes of the MF coefficients associated with Doppler indices $k_{{\nu }_1}^{\prime }$ and $k_{{\nu }_2}^{\prime }$ at the same delay index is reasonably estimated in the following manner:

$$\frac{\left|V\right[k_{{\nu }_1}^{\prime },l_i\left]\right|}{\left|V\right[k_{{\nu }_2}^{\prime },l_i\left]\right|}\approx \frac{|k_{{\nu }_2}^{\prime }-k_{{\nu }_1}^{\prime }-{\kappa }_{{\nu }_i}|}{|-{\kappa }_{{\nu }_i}|}.$$

(25)

The error of the approximation above is $\mathcal{O}\left(\frac{1}{M_S{N}_T}\right)$.

As a result, we can calculate the fractional Doppler value as:

$${\kappa }_{{\nu }_i}=\frac{(k_{{\nu }_2}^{\prime }-k_{{\nu }_1}^{\prime })\left|V[k_{{\nu }_2}^{\prime },l_i]\right|}{|V[k_{{\nu }_1}^{\prime },l_i]|+|V[k_{{\nu }_2}^{\prime },l_i]|}.$$

(26)

Likewise, employing the same approach described above for fractional delay taps, we obtain:

$${\iota }_{{\tau }_i}=\frac{(l_{{\tau }_2}^{\prime }-l_{{\tau }_1}^{\prime })\left|V[k_i,l_{{\tau }_2}^{\prime }]\right|}{\left|V\right[k_i,l_{{\tau }_1}^{\prime }\left]\right|+\left|V\right[k_i,l_{{\tau }_2}^{\prime }\left]\right|}.$$

(27)

4. Hybrid Message Passing Detector

We will formulate the probability model for the proposed symbol detection algorithm in this section. Subsequently, we provide an in-depth explanation of the hybrid MP algorithm.

The DD domain input–output relationship takes on the vectorized form:

$$\begin{array}{c}\hfill \mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{w}.\end{array}$$

(28)

In the context at hand, we have several key variables. Firstly, the received signal vector, denoted as $\mathbf{y}$, belongs to the complex vector space ${\mathbb{C}}^{M_S{N}_T\times 1}$. Secondly, the EC matrix, denoted as $\mathbf{H}$ (which is considered as known at the receiver when performing the symbol detection), belongs to the complex matrix space ${\mathbb{C}}^{M_S{N}_T\times M_S{N}_T}$. It is noteworthy that $\mathbf{H}$ is sparse and characterized by its elements $H[d,c]$. Moving on, the information vector $\mathbf{x}$ is an element of the complex vector space ${\mathbb{C}}^{M_S{N}_T\times 1}$, with each component $x\left[c\right]$ taking values from the set $\mathcal{A}$ and satisfying $1\le c\le M_S{N}_T$. Finally, we introduce the additive white Gaussian noise vector $\mathbf{w}$, which is an element of the complex vector space ${\mathbb{C}}^{M_S{N}_T\times 1}$. To elaborate further, the structure of $\mathbf{H}$ is expressed as follows:

$$\mathbf{H}=\sum _{i=0}^{P-1}\sum _{j=-N_i}^{N_i}{\mathbf{I}}_N\left(-{\left[j-k_{{\nu }_i}\right]}_N\right)\otimes \left[{\mathbf{I}}_M\left(l_{{\tau }_i}\right)\times \frac{1}{N}\left(\frac{\left.e^{-j2\pi \left(-j-{\kappa }_{{\nu }_i}\right.}\right)-1}{e^{-j\frac{2\pi }{N}\left(-j-{\kappa }_{{\nu }_i}\right)}-1}\right)h_i{e}^{-j2\pi {\nu }_i{\tau }_i}\right].$$

(29)

In our framework, we have two circular shifting matrices denoted as ${\mathbf{I}}_N$ and ${\mathbf{I}}_M$. ${\mathcal{R}}_d$ and ${\mathcal{C}}_c$ are defined as the sets whose the d-th row and c-th column are non-zero elements in the EC $\mathbf{H}$, respectively. By examining the composition of the channel matrix, it becomes evident that the cardinality of both sets denoted as $|{\mathcal{R}}_d|$ and $|{\mathcal{C}}_c|$ equals a constant value, which we denote as S.

The transmitted signals are estimated by the MAP rule. Referring to Equation (10), this estimation procedure can be expressed as:

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}=\underset{\mathbf{x}\in {\mathcal{A}}^{N_T{M}_S}}{argmax}P(\mathbf{x}\mid \mathbf{y},\mathbf{H}).\end{array}$$

(30)

Consider that each of the transmitted symbols in the vector $\mathbf{x}$ is assumed to be independently drawn from the constellation set $\mathcal{A}$ with equal probability, which is equivalent to a uniform prior for $\mathbf{x}$. Furthermore, given the approximate independence of the components of vector $\mathbf{y}$, we can approximate the a posteriori probability, denoted as $P(x\left[c\right]=a_j|\mathbf{y},\mathbf{H})$, as follows:

$$\begin{array}{cc}\hfill P\left(x\left[c\right]=a_j\mid \mathbf{y},\mathbf{H}\right)& =\frac{1}{Q}P\left(\mathbf{y}\mid x\left[c\right]=a_j,\mathbf{H}\right)\approx \prod _{d\in {\mathcal{C}}_c}P\left(\mid y\left[d\right]\mid x\left[c\right]=a_j,\mathbf{H}\right).\hfill \end{array}$$

(31)

The elements $\widehat{x}\left[c\right]$ and $x\left[c\right]$, which belong to the vectors $\widehat{\mathbf{x}}$ and $\mathbf{x}$, respectively, satisfy the condition $1\le c\le M_S{N}_T$.

The representation encapsulated in Equation (31) can be interpreted via a factor graph, as illustrated in Figure 3. This schematic elucidation provides a sophisticated methodology for ascertaining the marginal distribution, capitalizing on the fact that the variables are independent conditionally. Within this graph, each rectangular node, termed a function node (FN), denotes a specific function, such as $f_d=p\left(y_d\right|{\mathbf{x}}_d)$, where ${\mathbf{x}}_d=\left\{x\left[e\right]|e\in {\mathcal{R}}_d\right\}$. On the flip side, every circular node, emblematic of a variable node (VN), aligns with a unique variable. Given the inherent sparsity of the EC matrix $\mathbf{H}$, the interconnections between variables and FCs remain notably finite.

Owing to the impact of the fractional Doppler, it is discernible that the quantity of VNs linked to each FN significantly surpasses the available path count. As a result, the detection procedure for fractional Doppler systems attains a notably elevated level of complexity.

We introduce the concept of ${\mathcal{I}}_d^i$, which represents the non-zero elements in ${\mathcal{R}}_d$ that are related to $h_i$. Similarly, we define ${\mathbf{x}}_d^i$ as the VNs from ${\mathbf{x}}_d$ that are associated with $h_i$. There are $2N_{T_i}+1$ elements for both ${\mathcal{I}}_d^i$ and ${\mathbf{x}}_d^i$. To mitigate the complexity of OTFS symbol detection, we select the element whose absolute value is the highest from ${\mathcal{I}}_d^i$ as the main path, as it affects symbol detection most. The main paths within ${\mathcal{R}}_d$ are denoted as a set $\widetilde{{\mathcal{I}}_d}$, and the remaining elements of ${\mathcal{R}}_d$ as $\overline{{\mathcal{I}}_d}$. The non-approximate MP algorithm is employed for $\widetilde{{\mathcal{I}}_d}$, while a GA-based MP algorithm is used for $\overline{{\mathcal{I}}_d}$ in the context of symbol detection. Consequently, the collections of VNs are referred to as ${\tilde{\mathbf{x}}}_d$ and ${\overline{\mathbf{x}}}_d$, respectively. The partial interconnections between the VNs and FNs are delineated in Figure 4.

The detection algorithm is utilized to realize the factor graph model for calculating the marginal distribution. We begin by considering ${\psi }_{e/d}\left(x\right)={\mu }_{x\to f_{e/d}}\left(x\right)$ and ${\varphi }_{e/d}\left(x\right)={\mu }_{f_d\to x}\left(x\right)$ as the iteration’s value from node $x\left[c\right]$ to node $f_d$ and from node $f_d$ to node $x\left[c\right]$, respectively. Here, define ${\tilde{\varphi }}_{e/d}\left(x\right)={\tilde{\mu }}_{f_d\to x\left[c\right]}\left(x\left[c\right]\right)$ and ${\tilde{\psi }}_{e/d}\left(x\right)={\tilde{\mu }}_{x\left[c\right]\to f_d}\left(x\left[c\right]\right)$ as the previous iteration’s value of ${\mu }_{f_d\to x\left[c\right]}\left(x\left[c\right]\right)$ and ${\mu }_{x\left[c\right]\to f_d}\left(x\left[c\right]\right)$, respectively. Specifically, the message transmitted towards a VN $x_c$ from an FN $f_d$ and the message passing procedure from $x\left[c\right]$, a VN, to $f_d$, an associated factor node, can be established as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\varphi }_{e/d}\left(x\left[c\right]\right)=\sum _{{\mathbf{x}}_d\sim x\left[c\right]}p(y\left[d\right]\mid {\mathbf{x}}_d)\prod _{e\in {\mathcal{R}}_d\sim c}{\psi }_{e/d}\left(x\left[e\right]\right).\hfill \\ {\psi }_{e/d}\left(x\left[c\right]\right)=p\left(x\left[c\right]\right)\prod _{e\in {\mathcal{C}}_c\sim d}{\varphi }_{e/d}\left(x\left[c\right]\right).\hfill \end{array}\right.\end{array}$$

(32)

In order to enhance the performance, we incorporate a “damping factor”, denoted as $\Delta$, where $\Delta$ falls within the range of (0,1], during the message update process ${\mu }_{f_d\to x\left[c\right]}\left(x\left[c\right]\right)$. Consequently, Equation (32) can be reformulated as:

$$\begin{array}{cc}\hfill \varphi \left(x\right[c\left]\right)=& \sum _{{\mathbf{x}}_d\sim x\left[c\right]}p(y\left[d\right]\mid {\mathbf{x}}_d)\prod _{e\in {\mathcal{R}}_d\sim c}\psi \left(x\left[e\right]\right)\hfill \\ \hfill & +\Delta ·\tilde{\varphi }\left(x\right).\hfill \end{array}$$

(33)

We can see that the damping factor has obvious effects on the detection performance in the simulation outcomes.

Likewise, the message passing procedure from $x\left[c\right]$, a VN, to $f_d$, an associated factor node, is established as follows:

Being estimated through an approximate distribution, $b\left(x\left[c\right]\right)$, which is referred to as the “belief”, $x\left[c\right]$ can be reformulated as:

$$\begin{array}{c}\hfill b\left(x\left[c\right]\right)=p\left(x\left[c\right]\right)\prod _{e\in {\mathcal{C}}_c}{\varphi }_{e/d}\left(x\left[c\right]\right).\end{array}$$

(34)

In line with the hybrid MP algorithm, ${\mu }_{x\left[e\right]\to f_d}\left(x\left[e\right]\right)$ is divided into two distinct components, as described by:

$$\begin{array}{cc}\hfill {\psi }_{e/d}\left(x\left[e\right]\right)& =\sum _{i=1}^Q{\mu }_{e,d}^{\left[i\right]}\delta \left(x\left[e\right]-{\alpha }_i\right),e\in \widetilde{{\mathcal{I}}_d};\hfill \\ \hfill {\psi }_{e/d}\left(x\left[e\right]\right)& ={\psi }_{e/d}^{\mathcal{G}}\left(x\left[e\right]\right),e\in \overline{{\mathcal{I}}_d},\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill m_{x\left[e\right]}& ={\mathbb{E}}_{p_{x\left[e\right]}}\left[x\left[e\right]\right]=\sum _{i=1}^Q{\mu }_{e,d}^{\left[i\right]}{\alpha }_i\hfill \\ \hfill v_{x\left[e\right]}& ={\mathbb{E}}_{p_{x\left[e\right]}}\left[x{\left[e\right]}^2\right]-{\left|m_{x\left[e\right]}\right|}^2\hfill \\ \hfill & =\sum _{i=1}^Q{\mu }_{e,d}^{\left[i\right]}{\left|{\alpha }_i\right|}^2-{\left|m_{x\left[e\right]}\right|}^2,\hfill \end{array}$$

(36)

where ${\mu }_{e,d}^{\left[i\right]}$ represents the probability and ${\mu }_{x\left[e\right]\to f_d}^{\mathcal{G}}\left(x\left[e\right]\right)$ signifies a function. Upon inserting the expression from Equation (35) into Equation (32), we derive Equation (37), which is displayed at the beginning of the following page.

$$\begin{array}{cc}\hfill & \varphi \left(x\right[c\left]\right)\hfill \\ \hfill & =\int \sum _{{\mathbf{x}}_d\sim x\left[c\right]}\frac{1}{\pi N_{T_0}}exp\left(\frac{{\left|a\right|}^2}{N_{T_0}}\right)\hfill \\ \hfill & \prod _{e\in \widetilde{{\mathcal{I}}_d}\sim c}{\mu }_{e,d}^{\left[i\right]}\delta \left(x\left[e\right]-{\alpha }_i\right)\hfill \\ \hfill & \times \prod _{e\in \overline{{\mathcal{I}}_d}\sim c}exp\left(-\frac{{\left|x\left[e\right]-m_{x\left[e\right]}\right|}^2}{v_{x\left[e\right]}}\right)\mathrm{d}{\mathbf{x}}_d\sim x\left[c\right]+\Delta ·\tilde{\varphi }\left(x\left[c\right]\right)\hfill \\ \hfill & =\sum _{i=1}^Q\cdots \sum _{i=1}^Q\frac{1}{\pi N_{T_0}}exp\left(\frac{{\left|a-{\sum }_{e\in \overline{I}\left(d\right)\sim c}H[d,e]m_{x\left[e\right]}\right|}^2}{N_{T_0}+{\sum }_{e\in \overline{I}\left(d\right)\sim c}{\left|H[d,e]\right|}^2{v}_{x\left[e\right]}}\right)\hfill \\ \hfill & \times \prod _{e\in \widetilde{{\mathcal{I}}_d}\sim c}{\mu }_{e,d}^{\left[i\right]}\delta \left(x\left[e\right]-{\alpha }_i\right)+\Delta ·\tilde{\varphi }\left(x\left[c\right]\right).\hfill \end{array}$$

(37)

Following this, we employ the message ${\mu }_{f_d\to x\left[c\right]}\left(x\left[c\right]\right)$ to adapt the message ${\mu }_{x\left[c\right]\to f_d}\left(x\left[c\right]\right)$. $a=y\left[d\right]-{\sum }_{e\in \widetilde{{\mathcal{I}}_d}\sim c}H[d,e]x\left[e\right]-H[d,c]x\left[c\right]$. Consequently, we establish the belief $b\left(x\left[c\right]\right)$. The ISl $x\left[c\right]$ can be detected by applying the a posteriori rule, i.e.,

$$\begin{array}{c}\hfill \widehat{x}\left[c\right]=arg\underset{x\left[c\right]\in \mathcal{A}}{max}b\left(x\left[c\right]\right).\end{array}$$

(38)

From Equation (37), it is evident that there are Q states in the detection procedure of each data symbol, and the summation needs to encompass all symbols within ${\mathbf{x}}_d$ except $x\left[c\right]$. Consequently, the summation operations requires ${\left|Q\right|}^P$ operations. Considering that the operational term is insignificantly small in comparison to the exponentially growing term ${\left|Q\right|}^P$, the complexity is effectively $\mathcal{O}\left({\left|Q\right|}^P\right)$. It is worth noting that the computation complexity would be an overwhelming $\mathcal{O}\left({\left|Q\right|}^{P(2N_{T_i}+1)}\right)$ without the Gaussian approximation, rendering it nearly impossible for implementation.

Hybrid detection schemes may involve more complex algorithms and processing steps compared to simpler, single-method detection schemes. Integrating multiple detection methods introduces challenges in terms of system implementation. The coordination and synchronization of different algorithms, especially in real-time applications, can contribute to increased an design and implementation complexity. The primary motivation for adopting a hybrid detection scheme is often to enhance robustness. By combining multiple methods, the system becomes more adaptable to diverse operating conditions, variations in the signal environment, and potential challenges like noise, interference, or fading. Hybrid schemes can lead to an improved detection accuracy compared to individual methods. Each detection method may excel in specific scenarios or under certain conditions, and their combination can provide a more comprehensive and reliable detection capability.

5. Results

In this section, we use Monte Carlo simulations to conduct in-depth research on the evaluation of target estimation and symbol detection performance in different scenarios. We conduct these simulations with an average over ${10}^4$ OTFS frames. For each OTFS frame, we configure the parameters with $M_S=32$ and $N_T=16$, indicating the presence of 32 sub-carriers and 16 time slots within the TF domain. The ISs are four-QAM modulated from the data bits generated randomly. In addition, simulation results show that the proposed hybrid MP detector has a high effectiveness. The effectiveness of the hybrid message-passing algorithm is substantiated, and comprehensive comparisons are made, specifically in relation to conventional detection algorithms encompassing Gaussian-approximation-based MP methods. A uniform maximum iteration limit, denoted as $I_{max}$, is uniformly established as 10 for all algorithms under consideration.

We investigate two different simulations, one involving four targets and the other involving six targets. Our analysis involves the computation of the ratio between the root mean square error (RMSE) and the corresponding resolution in estimating both R and V. We vary the SNR and determine the RMSEs in velocity estimation and range estimation concerning the SNR. The results of these simulations are visually presented in Figure 5 and Figure 6.

As depicted in Figure 5, a noticeable trend emerges when there are four targets: the RMSE associated with the estimated velocity significantly diminishes when the fractional Doppler index is determined through (26). This observation underscores the efficacy of our proposed algorithm. Conversely, when the scenario involves six targets, the RMSE performance deteriorates at identical SNR levels. This degradation is attributed to the heightened interference arising from the overlapping responses of each symbol, rendering the approximation in (25) less precise. Consequently, the accuracy of velocity estimation suffers.

Similarly, as illustrated in Figure 6, when the simulation entails four targets, the RMSE in range estimation is reduced, and the results of the target estimation substantially deteriorate only if the integer part of the indices is considered. This phenomenon aligns with the increased interference and inaccuracies associated with the estimation process in scenarios with a greater number of targets.

When considering only the integer components, it becomes evident that the RMSE in range estimation remains largely unaffected with respect to varying SNRs. This phenomenon arises due to the limited resolutions of R and V within each grid. The average minimum errors align with half of the resolution values, a correspondence that is consistent with our simulation findings.

In the simulations of the proposed hybrid MP detector, we employ a random selection of DD parameters for each path, ensuring the fractional Doppler indices. Note that, for both the Gaussian-approximation-based MP and hybrid MP algorithms, the damping factor is set to 0.7.

We initiate our analysis by examining the influence of the parameter $N_i$. Specifically, we apply both the Gaussian-approximation-based MP and the hybrid MP algorithms to various values of $N_{T_i}$ under specific SNR conditions, where the $\mathrm{SNR}=18\mathrm{dB}$ and $P=6$. The acquired findings are depicted graphically in Figure 7.

Remarkably, a substantial improvement in the bit error rate (BER) is discernible as $N_{T_i}$ increases. As $N_{T_i}$ reaches 2 and beyond, it becomes apparent that the performance of the Gaussian-approximation-based MP stabilizes, while the hybrid MP’s performance continues to improve. Furthermore, it is worth noting that, across all values of $N_i$, the hybrid MP consistently outperforms the Gaussian-approximation-based MP.

Further, we shift our focus to the BER performance comparison of the Gaussian-approximation-based MP algorithm and the hybrid MP with different values of P. The outcomes of this analysis are illustrated in Figure 8, maintaining an $\mathrm{SNR}=18\mathrm{dB}$ and fixing $N_{T_i}=2$.

It is evident from the results that the hybrid MP consistently outperforms the GA-based MP when the system involves more than four paths. For scenarios with precisely four paths, both the Gaussian-approximation-based MP and the hybrid MP exhibit similar performances, characterized by a BER of approximately $5\times {10}^{-4}$. However, when there are more paths, the performance of the Gaussian-approximation-based MP deteriorates, while the hybrid MP’s performance remains robust. This observation underscores the superior effectiveness of the hybrid MP in mitigating multipath interference.

The underlying reason for this disparity lies in the convergence behavior of the two algorithms. The Gaussian-approximation-based MP tends to converge towards a local optimum due to the presence of a substantial number of loops. In contrast, the hybrid MP algorithm efficiently leverages diversity gain, leading to its superior performance in such scenarios.

In Figure 9, we present the BER performance of the hybrid MP algorithm across various SNRs with the same $P=8$ and $N_{T_i}=2$. We compare this performance with that of the MMSE detector, the GA-based MP, and the hybrid MP without any damping factor.

The results indicate that the hybrid MP exhibits robust performance across the entire SNR range. Particularly noteworthy is the performance of the hybrid MP without the “damping factor”, which closely matches that of the Gaussian-approximation-based MP. The inclusion of the “damping factor” significantly improves the performance, reducing the BER from $5\times {10}^{-4}$ to $2\times {10}^{-5}$. This enhancement is especially notable.

It is worth mentioning that a previous study suggested that a “damping factor” might not be necessary for the hybrid algorithm. However, the presence of the fractional Doppler leads to interference energy being spread across more paths, necessitating the incorporation of a “damping factor” to enhance effectiveness.

6. Discussion

The adaptability of the OTFS framework to future technological advancements and standards depends on its inherent flexibility, the willingness of the research community to evolve the framework, and its ability to integrate with emerging technologies. The ability of the OTFS framework to integrate with emerging wireless communication standards is crucial for its long-term adaptability. Compatibility with new modulation schemes, protocols, and frequency bands enhances its relevance in evolving wireless networks. The OTFS framework is designed to support multiple access schemes. It can be more adaptable to diverse communication scenarios. The ability to accommodate different access methods allows it to align with future network architectures and requirements. As wireless networks evolve, the scalability and performance optimization of the OTFS framework will become increasingly important. Continuous efforts to enhance scalability and optimize performance ensure that the framework remains viable in large-scale and high-performance communication systems. The ability of the OTFS framework to address emerging use cases, such as Internet of Things (IoT), Industry 4.0, and beyond, is a key factor in its adaptability. Anticipating and accommodating diverse communication scenarios ensures continued relevance.

There are some limitations to our method. The scalability of the OTFS framework in handling a large number of simultaneous targets or users depends on various factors. Efficient parallelization of algorithms can enhance the system’s ability to handle a large number of simultaneous targets or users. The OTFS framework can potentially support multiple access schemes, allowing it to handle multiple targets or users simultaneously. Differentiation and isolation of signals in the DD domain contribute to scalability. The ability of OTFS modulation to adapt to dynamic channel conditions is beneficial for scalability. It enables the framework to handle variations in the number of targets or users and changes in the communication environment. The computational complexity of the OTFS framework may become a limiting factor as the number of simultaneous targets or users increases. Processing overheads may scale with square or higher orders of the number of targets, impacting real-time performance. Accurate channel estimation and tracking become more challenging with a larger number of simultaneous targets or users. The complexity of tracking and adapting to dynamic channels may impact the scalability of the system. In scenarios with a high density of simultaneous targets or users, interference and crosstalk between signals may increase. The ability of OTFS modulation to distinguish between closely spaced targets may be limited, affecting system performance.

The OTFS framework is designed to be robust in the face of challenging channel conditions in rapidly changing multipath environments. Its unique approach to signal representation in the DD domain offers advantages in scenarios where conventional modulation schemes may struggle. The OTFS framework excels in representing signals in the DD domain, providing a more natural and robust representation of the channel characteristics. This representation allows for the separation of signals arriving at different delays and Doppler shifts, even in environments with severe multipath conditions. OTFS modulation can mitigate the effects of frequency-selective fading, a common challenge in multi-path environments. By transforming signals to the DD domain, OTFS modulation provides a way to disentangle frequency-selective fading components, leading to an improved communication performance. The OTFS framework can be designed to incorporate dynamic channel tracking algorithms. These algorithms adapt to changes in the channel conditions, continuously updating channel estimates to maintain a robust communication performance in environments with varying multi-path characteristics.

Accurate channel estimation is crucial for obtaining channel state information (CSI) for reliable detection. In the OTFS, the instantaneous delay shift and Doppler shift of channel information are discrete on the DD grid. The time delay resolution and Doppler resolution depend on the bandwidth and duration, respectively. In actual communication scenarios, the bandwidth is large enough to provide sufficient latency resolution, but due to the low latency requirements of future communication, the duration may be relatively small. Therefore, the fractional Doppler needs to be considered. In the case of the integer Doppler, the effective channels in the DD domain are sparse. On the contrary, for the fractional Doppler case, the effective DD main channel is distributed across all Doppler indices, which reduces the sparsity of the effective channel and may therefore reduce the channel estimation performance. Estimating the fractional Doppler enables us to improve channel estimation by reconstructing effective DD domain channels using estimated Doppler frequency shifts, which is essential for reliable detection. Furthermore, this can improve the accuracy of positioning. When OTFS modulation assists sensing, accurate estimation of mobile user speed can be obtained through accurate estimations of Doppler frequency shifts.

7. Conclusions

In this paper, we proposed a new method to estimate the fractional DD indices of off-grid targets using a 2D correction-based approach. The DD indices of off-grid targets appear as fractional values in the DD domain. To address this, we presented a novel method based on differences to estimate the fractional components of these parameters. According to the experimental results, the proposed algorithm can accurately estimate the R and V of off-grid targets. In addition, we also designed a hybrid MP-based detector for communication, which has a lower BER performance than existing detection methods according to the simulation results.

By integrating the OTFS framework with existing communication and sensing systems, adaptation of signal processing modules can be achieved. The OTFS framework relies on a unique signal processing approach. Adaptation of existing signal processing modules is necessary to incorporate the transformations necessary for OTFS modulation. OTFS modulation may have specific requirements for error correction in the DD domain and adapt decoding algorithms to handle signals transformed into the DD domain. The OTFS framework can be used thorough simulations and testing to evaluate the performance of an integrated system, and based on performance evaluations, it can iterate through optimization cycles to fine-tune parameters, algorithms, and configurations for an optimal performance. Meanwhile, it can evaluate the latency in the context of real-time requirements and optimize processing pipelines to meet the latency constraints of the overall system.