GIT: A Transformer-Based Deep Learning Model for Geoacoustic Inversion

Abstract

Geoacoustic inversion is a challenging task in marine research due to the complex environment and acoustic propagation mechanisms. With the rapid development of deep learning, various designs of neural networks have been proposed to solve this issue with satisfactory results. As a data-driven method, deep learning networks aim to approximate the inverse function of acoustic propagation by extracting knowledge from multiple replicas, outperforming conventional inversion methods. However, existing deep learning networks, mainly incorporating stacked convolution and fully connected neural networks, are simple and may neglect some meaningful information. To extend the network backbone for geoacoustic inversion, this paper proposes a transformer-based geoacoustic inversion model with additional frequency and sensor 2-D positional embedding to perceive more information from the acoustic input. The simulation experimental results indicate that our proposed model achieves comparable inversion results with the existing inversion networks, demonstrating its effectiveness in marine research.

1. Introduction

Acoustic wave propagation in shallow water is highly susceptible to the geoacoustic characteristics of the seabed that play a crucial role in sound transmission processes. The influence of seafloor properties on acoustic propagation mechanisms has been widely studied [1], with particular attention given to shallow sea environments characterized by complex and variable ocean conditions. In such environments, the reflection and absorption processes of the seabed significantly affect acoustic signal propagation [2]. Understanding the acoustic parameters of the seafloor is vital for both sonar equipment performance evaluation and acoustic field modelling. However, obtaining these geoacoustic characteristics efficiently remains challenging and has, hence, garnered much attention within the marine science community [3].

There are two primary approaches used for determining underwater acoustic parameters: experimental measurements, including in situ measurements during sea trials [4,5], and acoustic inversion techniques. The first approach can only obtain the local bottom characteristics of the target area, making it impractical and resource-intensive to acquire a large range of seafloor acoustic parameters. In contrast, acoustic inversion techniques process data received from sonar systems and are an efficient means of measuring bottom characteristics. These methods enable a quick and cost-effective acquisition of geoacoustic parameters from measured acoustic data, and have made significant contributions to marine research, such as matched-field inversion (MFI) [6], time-of-arrival analysis [7], and modal dispersion techniques [8]. Among these techniques, matched-field inversion (MFI) is the most widely used method. Vertical linear arrays (VLA), towed linear arrays, bottom horizontal linear arrays (HLA) or distributed sensor networks (DSN) are developed for source localization by comparing the energy of the measured signal with that generated by an acoustic forward model simulation. Bayesian approaches [9] that use posterior probability density estimates to infer parameters have also gained attention in uncertain ocean environments. While no single geoacoustic inversion system is superior to others, the efficiency of these methods is usually their most appealing feature.

Motivated by the development of deep learning, its excellent techniques have been increasingly applied in marine acoustics, leading to remarkable achievements and superior performance compared to traditional approaches [10,11]. Various neural networks have been adopted for geoacoustic inversion. This is a functional approximation problem that estimates objective geoacoustic characteristics, such as the sound speed profile, layer thickness, and density of sediments, efficiently.

Liu et al. [12] proposed a deep convolutional neural network architecture for multi-range vertical array processing, demonstrating its efficacy in geoacoustic inversion. Zhu et al. [13] developed a deep-reinforcement-learning-based inversion method to estimate the shear wave velocity, exhibiting better efficiency and accuracy compared to heuristic algorithms. Shen et al. [14] introduced a radial basis function neural network (RBFNN) to MFI, achieving comparable performance to conventional methods. Alfarraj and AlRegib [15] proposed a semi-supervised framework based on a mixed convolutional and recurrent neural network for acoustic impedance inversion, delivering satisfactory results. Yang and Ma [16] investigated a supervised deep fully convolutional neural network (FCN) for marine seismic velocity, which performed better than conventional methods. These studies have demonstrated the superior performance of deep-learning-based inversion methods in geoacoustic inversion. However, such methods have primarily utilized traditional neural network backbones, including convolutional neural networks (CNNs) and fully convolutional networks (FCNs).

While these networks are effective, they possess certain limitations that result in practical bottlenecks. For example, CNNs are biased towards local information due to their use of kernel sliding, and, hence, potentially discard global information for an acoustic signal [17]. Considering these limitations, our research aims to enhance the architecture of deep learning models for geoacoustic inversion.

The present study presents a novel geoacoustic inversion model known as the geoacoustic inversion transformer (GIT). The transformer architecture, originally designed for natural language processing (NLP) [18], has been shown to outperform convolutional neural networks (CNNs) in various domains [19,20]. By leveraging the multi-head self-attention mechanism (MHSA) intrinsic to transformers, the proposed model can process sonar received signals from a more comprehensive global perspective, thus improving its performance in acoustic input analysis [17]. Furthermore, this study proposes a 2-D frequency and sensor positional embedding that captures the position information from both the sensor and frequency domains. It is expected that the proposed GIT can achieve satisfactory inversion performance due to its strong feature representation ability. This study showcases the potential of utilizing transformer architecture in the field of geoacoustic inversion, which remains an underexplored area of research.

The remainder of this paper is organized as follows: Section 2 outlines the problem formulation of geoacoustic inversion. Section 3 presents the proposed inversion framework. The numerical simulation environment and experimental results are presented in Section 4. This section includes a comparison of the proposed model performance with another inversion model. Section 5 gives the study conclusions.

2. Problem Formulation

Equation (1) gives the mathematical representation of a true acoustic observation:

$$y=M\left(x\right)+e,$$

(1)

where y represents the measured signal at the receiver array, x is a vector that represents the model parameters, M is the nonlinear physical model that maps x to y, and e is the measurement noise. Due to the complicated propagation mechanism, the true physical model $M(·)$ is unknown in most marine scenarios. In this case, an alternative method is to use an approximate accurate model that can be expressed as:

$$\overline{y}=\overline{M}\left(\overline{x}\right)+\overline{e}.$$

(2)

The forward acoustic propagation model $\overline{M}$ that represents a numerical model of the true function M is a well-known approach to simulate the acoustic environment in geoacoustic inversion. Both the geoacoustic properties of the ocean floor and the geometrical parameters are contained in the input vector x. The problem of geoacoustic inversion can be expressed as follows: Given an observation of the system output $\overline{y}$ and the forward model $\overline{M}$, find the geoacoustic parameters $\overline{x}$.

$$\overline{x}={\overline{M}}^{-1}\left(\overline{y}\right).$$

(3)

In this approach, the task of geoacoustic inversion can be accomplished if suitable $\overline{M}$ exists. Because the mapping $\overline{M}$ is extremely complex, finding such an approximation of $\overline{M}$ is difficult. However, deep learning neural networks and data-based methods can be adapted to approximate $\overline{M}$ successfully.

3. Methodology

In this section, data preprocessing of the received acoustic signal is presented, followed by a description of the proposed inversion transformer with FS positional embedding.

3.1. Data Preprocessing for Tokenization

Due to multiple reflections of sound waves between the seafloor and surface, the acoustic data received by the sonar system in shallow water contains rich information about the geoacoustic characteristics of the ocean bottom. To extract this implicit information, conventional fast Fourier transform (FFT) analysis is first used on the time-domain to obtain the complex pressure $\mathbf{p}\left(f\right)={\left[p_1\left(f\right),p_2\left(f\right),\dots ,p_L\left(f\right)\right]}^T$. The complex pressure $\mathbf{p}\left(f\right)$ at the specific frequency f can be formulated as:

$$\mathbf{p}\left(f\right)=S\left(f\right)\mathbf{g}(f,\mathbf{r})+\epsilon ,$$

(4)

where $S\left(f\right)$ represents the source term, $\mathbf{g}(f,\mathbf{r})$ means the Green function, and $\epsilon$ is the noise. In practice, the magnitudes of the source frequency spectrum between the simulation data and the real data may be different such that the source magnitudes used to generate the synthetic data are flat across the frequency band. For this reason, the complex pressure $\tilde{\mathbf{p}}\left(f\right)$ is normalized by the L2-norm to reduce the influence of mismatch caused by the source terms [21].

$$\tilde{\mathbf{p}}\left(f\right)=\frac{\mathbf{p}\left(f\right)}{\sqrt{{\sum }_{l=1}^L{\left|p_l\left(f\right)\right|}^2}}=\frac{\mathbf{p}\left(f\right)}{{\parallel \mathbf{p}\left(f\right)\parallel }_2}=\frac{S\left(f\right)\mathbf{g}(f,\mathbf{r})+\epsilon }{{\parallel S\left(f\right)\mathbf{g}(f,\mathbf{r})+\epsilon \parallel }_2}.$$

(5)

If the acoustic data is clean without noise (i.e., $\epsilon =0)$, the normalization Equation (5) can eliminate the effect of the source magnitude. In this case, the above equation can be rewritten as:

$$\tilde{\mathbf{p}}\left(f\right)=\frac{\mathbf{g}(f,\mathbf{r})}{{\parallel \mathbf{g}(f,\mathbf{r})\parallel }_2}.$$

(6)

In Equation (6), the complex pressure is independent of the source term $S\left(f\right)$. The influence of the source magnitude can also be effectively suppressed in the case of acoustic data with a reasonably high SNR. Multiplying the complex pressure with its conjugate transposition gives the normalized sample covariance matrix (SCM), expressed by Equation (6) [22].

$$\tilde{\mathbf{C}}\left(f\right)=\tilde{\mathbf{p}}\left(f\right){\tilde{\mathbf{p}}}^H\left(f\right).$$

(7)

The $L\times L$ SCM is complex in the frequency domain, but the neural network usually receives real numerical values such that the input must be transformed to allow the real-valued model input. For each frequency point, the SCM can be separated into the real and imaginary parts, which are used as two channels of the input feature of size $2\times L\times L$.

$$\mathbf{C}=[Re\left(\tilde{\mathbf{C}}\left(f\right)\right),Im\left(\tilde{\mathbf{C}}\left(f\right)\right)].$$

(8)

Consider a broadband signal which has numbers of frequency points, the SCMs at different frequency points are stacked along the first dimension, and the input is a matrix $\mathbf{C}$ of size $(F\times 2,L\times L)$ and can be expressed as:

$$\mathbf{C}=\left[Re\left(\tilde{\mathbf{C}}\left(f_1\right)\right),Im\left(\tilde{\mathbf{C}}\left(f_1\right)\right),\dots ,Re\left(\tilde{\mathbf{C}}\left(f_F\right)\right),Im\left(\tilde{\mathbf{C}}\left(f_F\right)\right)\right],$$

(9)

where F represents the number of frequency points in a broad signal. To eliminate the effect of unit and scale differences between individual features, to treat each dimensional feature equally, min-max normalization is both applied on the data and the ground-truth. Equation (10) expresses the normalization.

$$z=\frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)},$$

(10)

where x is the feature or the ground-truth label and z is the normalized item.

The CNNs and transformers are two major classes of deep learning models that have been used extensively in various domains. Unlike CNNs, which process the normalized input $\mathbf{C}$ directly using kernel sliding, transformers employ a tokenization procedure to preprocess $\mathbf{C}$ and obtain feature tokens that serve as the basic units in transformer models. To exploit both the frequency and sensor domain information simultaneously, a simple convolutional kernel with a kernel size of (2, L) and stride of (2, L) is deployed before the transformer layers. The first dimension of the convolution kernel is set to 2 to integrate the information on the real and imaginary parts of the $\mathbf{C}$. As a result, the input $\mathbf{C}$ is transformed into $F\times L$ tokens with a preset embedding size that encapsulates all useful information pertaining to the frequency and sensor.

3.2. Frequency and Sensor 2-D Positional Embedding

In standard transformers, a positional embedding is added to each token embedding to enable the model to perceive sequential information. However, in geoacoustic inversion, both the distribution of frequency points and the sonar sensor position affect the energy and phase of the acoustic signal, making it imperative to jointly consider both factors. In this case, this study proposes an incorporation of 2-D positional embeddings in both the frequency and sensor domains to simultaneously capture the position and frequency information.

For each domain, a learnable positional embedding is attached to each token that is initialized as follows:

$$\begin{array}{cc}\hfill & P{E}_{(pos,2i)}=sin\left(pos/{10000}^{2i/dim}\right)\hfill \\ \hfill & P{E}_{(pos,2i+1)}=cos\left(pos/{10000}^{2i/dim}\right),\hfill \end{array}$$

(11)

where $pos$ means the order of the frequency or sensor, $i\in [1,dim/2]$, and $dim$ is the embedding dimension. For broad signals with F frequency points and L sonar receivers, the token embedding is attached with an (F, $dim$) frequency positional embedding in the first dimension, and an (L, $dim$) sensor positional embedding in the second dimension. As a result, meaningful feature tokens with 2-D order information are obtained as input that would be fed into the transformer.

3.3. Transformer Model Based on MHSA

The transformer architecture comprises of multiple encoder and decoder blocks. To reduce the complexity of the model, only the encoder blocks are utilized in the proposed GIT model shown in Figure 1. The model employs N transformer encoders that incorporate the MHSA mechanism to compute the correlation between tokens. To facilitate the regression task of geoacoustic inversion, a regression token is inserted at the beginning of the token sequence following the [CLS] token similar to vision transformers [19]. The first encoder receives feature tokens ${\mathbf{X}}_0$ of size ($F\times L+1$, $dim$). Then the MHSA layer in the N transformer encoder generates three attention matrices, a query matrix ${\mathbf{Q}}_{\mathbf{h}}$, a key matrix ${\mathbf{K}}_{\mathbf{h}}$, and the corresponding value matrix ${\mathbf{V}}_{\mathbf{h}}$, to extract features from ${\mathbf{X}}_{\mathbf{n}-1}$.

$$\begin{array}{c}{\mathbf{Q}}_{\mathbf{h}},{\mathbf{K}}_{\mathbf{h}},{\mathbf{V}}_{\mathbf{h}}={\mathbf{X}}_{\mathbf{n}-\mathbf{1}}{\mathbf{W}}_{\mathbf{h}}^{\mathbf{Q}},{\mathbf{X}}_{\mathbf{n}-\mathbf{1}}{\mathbf{W}}_{\mathbf{h}}^{\mathbf{K}},{\mathbf{X}}_{\mathbf{n}-\mathbf{1}}{\mathbf{W}}_{\mathbf{h}}^{\mathbf{V}}\\ Attention\left({\mathbf{Q}}_{\mathbf{h}},{\mathbf{K}}_{\mathbf{h}},{\mathbf{V}}_{\mathbf{h}}\right)=softmax\left(\frac{{\mathbf{Q}}_{\mathbf{h}}{\mathbf{K}}_{\mathbf{h}}^{\mathbf{T}}}{\sqrt{d_{attention}}}\right){\mathbf{V}}_{\mathbf{h}}.\end{array}$$

(12)

where ${\mathbf{W}}_{\mathbf{h}}^{\mathbf{Q}}$, ${\mathbf{W}}_{\mathbf{h}}^{\mathbf{K}}$, and ${\mathbf{W}}_{\mathbf{h}}^{\mathbf{V}}$ are both learnable matrices to measure the attention weights, $h\in [1,H]$ represents which self-attention head is to be calculated, and the embedding dimension of the attention weight is $d_{attention}=dim/H$.

The Softmax activation and division operation would not change the feature dimension, after the MHSA layers, and the feature maps remain the shape of ($F\times L+1$, $dim$). A multi-layer perceptron (MLP) layer is utilized following the MHSA calculation. This MLP layer is primarily composed of a fully connected layer that is enhanced with a GELU activation function, benefiting the transformer training process. A dropout layer is employed to mitigate the risk of overfitting during model training. In this case, we have the final feature map with rich information to describe the token relations from the perspectives of both the frequency and receiver domains.

As mentioned above, the regression token is preset at the beginning of the token sequences. It serves as a representative of all the tokens in the set and efficiently integrates information from other items in the token sequence. This integration allows for the projection of embeddings into the number of inverted parameters that play an essential role in the GIT. By training with the optimizer, the model assigns appropriate values to all parameters through multi-task learning.

3.4. Multi-Task Learning with Transformer

The task of geoacoustic inversion consists of various parameters with distinct metric spaces and scales. As a result, using one GIT to independently invert each parameter is unsuitable and costly. To address this challenge, we adopt the multi-task learning approach [23] in our model training. The aim is to employ one GIT network to invert all objective parameters simultaneously.

In this case, the multi-task learning method treats the inversion of each parameter as a subtask. By using adaptive loss weights, we compute the mean squared error (MSE) loss between the predicted values for all parameters and their true values. After gradient descent, the task loss weights are automatically adjusted based on the uncertainty of each task’s homoskedasticity. This approach takes advantage of the similarities and dissimilarities between tasks, resulting in enhanced accuracy and prediction performance.

To automatically balance the weight of each regression loss, we formulate the total loss as follows:

$$L_{\mathrm{total}}=\sum _i{w}_i{L}_i,$$

(13)

where i represents the number of subtasks, and $w_i$ is the adaptive weight on the corresponding task $L_i$. This approach enables us to converge inverted parameters with differing scales using one GIT for greater efficiency.

4. Experiments and Analysis

In this section, the objective inversion parameters are first listed, and the simulation environment is described, which is based on the Kraken acoustic model [24]. Furthermore, the experimental setup is given for reproducibility. Finally, we provide a comprehensive analysis of the experimental results to demonstrate the effectiveness of our proposed GIT for geoacoustic inversion.

4.1. Numerical Simulations

Following the SwellEx96 S5 experiment [25], the environment is modeled as a range-independent waveguide shown in Figure 2, which consists of the waveguide, a sediment layer, a mudstone layer, and the halfspace ocean bottom. To generate a large number of replicas for GIT model training, the boundaries between these layers are treated as discontinuities in parameter values, which allows for an accurate representation of real-world underwater environments. Specifically, the shallow point source, located 54 m below the sea surface, emits white noise with the frequency band –3300 Hz; the acoustic signal $x_o$ is received by a 21-element vertical line array (VLA), which is placed in the shallow water. For comprehensive analysis, 13 frequencies are selected from the range of [49, 388] Hz for inversion. The deterministic parameters are the seawater density 1.0 g/cm${}^3$, the seawater depth 220 m, the sediment layer attenuation 0.2 dB/kmHz, the mudstone layer depth 800 m with the density of 2.06 g/cm${}^3$, the mudstone layer attenuation 0.06 dB/kmHz, the halfspace compressional speed 5200 m/s with density of 2.66 g/cm${}^3$, and the halfspace attenuation 0.02 dB/kmHz. In particular, the objective parameters for inversion are the sediment layer depth $h_s$, the sediment density ${\rho }_s$, the top and bottom sound speed of the sediment layer $c_{st}$, $c_{sb}$, and the top sound speed of the mudstone layer $c_{mt}$.

4.2. Inversion Procedure

Starting with generating the synthetic data, our inversion procedure contains the following steps: training and testing, followed by performance comparison with the CNN-based methods, and parameter sensitivity assessment to determine the most influential factors in achieving high inversion performance. The simulated signal waveform selected from one replica and its corresponding spectrum can be seen in Figure 3.

1.

The proposed GIT model is trained using ${10}^5$ replicas, sampled for a uniform distribution for each parameter in the interval, which is shown in

Table 1. Intervals of objective parameters for inversion.

Parameters	Interval
$h_s$ (m)	[30, 40]
$c_{st}$ (m/s)	[1480, 1800]
$c_{sb}$ (m/s)	[1480, 1800]
${\rho }_s$ (g/cm${}^3$)	[1.5, 2.2]
$c_{mb}$ (m/s)	[1800, 2600]

.

Due to the large data size, the inversion parameters of the training data are believed to cover the whole parameter interval.

2.

Multi-task learning is used to optimize the neural network for multiple objective losses, along the direction of minimizing the MSE loss function for each subtask i, which is expressed as:

$$L_i=\frac{1}{N}\sum _N{\left(y-y_t\right)}^2$$

(14)

where y is the network prediction on the geoacoustic parameters, $y_t$ contains the true values of the candidate parameter, while the other parameters vary randomly in the training procedure. Convergence can be achieved when $L_{total}$ reaches a certain level, or would not decrease any more.

3.

To improve the robustness, Gaussian white noise is added to the training data with signal-to-noise-ratios (SNRs) randomly from 10 to 50 dB, which can be formulated as:

$$SNR=10{log}_{10}\frac{{\sum }_{l=1}^L{∥{\mathbf{p}}_l∥}_2^2}{FL{\sigma }^2},$$

(15)

4.

For the testing data, it consists of ${10}^4$ samples (SNR = 20 dB), which is also generated using the parameters in Table 1.

4.3. Experimental Setup

In our work, all the experiments were conducted on a workstation with the detailed test platform and environment shown in

Table 2. The test platform and environment.

Item	Values
2 × CPU	Intel Xeon Platinum 8377C
4 × GPU	Nvidia GeForce RTX 3090
Memory	128 GB
Python	3.9
PyTorch	1.8.0

.

During training, the batch size is set to 128 for efficiency. A total of 400 epochs with a learning rate 0.001 is given for GIT training, and is optimized by the Adam algorithm [26]. For the transformer configurations, the embedding dimension $dim$ is 384, the number of Encoders N is 3, and the number of attention heads is 12. From Figure 4, which shows the loss curves, it is demonstrated that our model is not over-fitted and can successfully work for geoacoustic inversion.

4.4. Experimental Results

In this section, we evaluate the performance of our proposed GIT in comparison with the current state-of-the-art neural networks for acoustic inversion. Additionally, we conduct a comparative study between the 2-D FS positional embedding model and the original positional embedding model to demonstrate its effectiveness. Moreover, we investigate the parameter sensitivity concerning the number of transformer Encoders N and the number of attention heads H to analyze their influence on the inversion performance.

4.4.1. Performance Comparison with CNNs

Owing to the powerful feature extraction ability with convolution kernels, the existing inversion methods mainly make use of the CNNs and their variants. Specifically, a convolutional and recurrent neural networks mixed model (CRNN) [15] and a deep fully convolutional neural network (FCN) [16], which has been widely used in the area of acoustic inversion, are chosen for the comparison experiments. Moreover, it is noted that these networks are modified to accept the input $\mathbf{C}$ and trained from scratch in a supervised manner. The mean absolute error (MAE) is selected as the evaluation metric.

The results shown in

Table 3. Inversion performance comparisons with CNNs on the objective parameters.

Methods	MAE	MAE_${\mathit{h}}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{t}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{b}}$	MAE_${\mathit{\rho }}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{m}\mathit{t}}$
CRNN	93.2	0.167	9.33	28.6	0.0243	55.1
FCN	64.2	0.131	4.92	22.6	0.0175	36.6
GIT (proposed)	56.7	0.101	4.26	22.1	0.0128	30.2

demonstrate that the proposed method, GIT, outperforms the other two methods, CRNN and FCN, in terms of the MAE for all six objective parameters. Specifically, CRNN had the highest MAE values of all methods for most of the objective parameters, indicating its lower accuracy compared to FCN and GIT. However, it still outperformed the other methods in terms of $h_s$. FCN showed better performance than CRNN by achieving lower MAE values for all the objective parameters except $h_s$. It achieved a significant improvement in MAE values compared to CRNN with a relatively simpler network architecture. However, its values were still higher than GIT. GIT (proposed) was the best performing method among all the methods presented in the table. It obtained the lowest MAE values for all the objective parameters, indicating its high accuracy compared to CRNN and FCN. Thus, GIT has strong potential as an effective method for inverting the objective parameters. GIT achieved an MAE of 56.7, which was lower than the MAEs of CRNN (93.2) and FCN (64.2). Moreover, when considering the individual objective parameters, GIT consistently achieved lower MAEs than the other two methods. This suggests that GIT was more accurate in estimating the objective parameter values than CRNN and FCN.

In summary, CRNN performed the worst among the inversion methods, while FCN and GIT showed better performance in terms of lower MAE values. FCN outperformed CRNN for most of the objective parameters except for $h_s$, while GIT provided the best performance in terms of lower MAE values for all the objective parameters.

4.4.2. Performance Impact on the FS Positional Embedding

As mentioned in Section 3.2, a 2-D Frequency and sensor positional embedding approach to jointly capture the information on the frequency and sensor domain is used. We compare it with a randomly initialized 1-D positional embedding transformer with the same experimental configuration. As can be seen in

Table 4. Effect of positional embedding methods on the objective parameters.

Position Embedding	MAE	MAE_${\mathit{h}}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{t}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{b}}$	MAE_${\mathit{\rho }}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{m}\mathit{t}}$
1-D	66.8	0.129	4.67	27.1	0.0125	34.9
FS	56.7	0.101	4.26	22.1	0.0128	30.2

, we find the proposed FS positional embedding can describe more useful information, and, thus, it performs better than 1-D positional embedding.

Specifically, based on the results presented in Table 4, it can be seen that using the FS positional embedding method resulted in lower MAE across all parameters when compared to the 1-D method. For example, the MAE value for the first parameter decreased from 66.8 to 56.7 when switching from the 1-D to the FS method. Similarly, for the second parameter $h_s$, the MAE value decreased from 0.129 to 0.101. This trend continued for the remaining parameters as well. This suggests that the proposed FS positional embedding can capture both the frequency and sensor position information to benefit the geoacoustic inversion. Overall, these results suggest that the FS positional embedding method may be a more effective approach for improving the accuracy of the inversion for the various parameters tested in this study.

4.5. Parameter Sensitivity

As major parameters for the GIT, we further analyzed the parameter sensitivity of the embedding dimension $dim$, the number of heads H, and the number of transformer encoders N.

As shown in

Table 5. Inversion performance varying with the number of encoders N on the objective parameters.

Number of Encoders N	MAE	MAE_${\mathit{h}}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{t}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{b}}$	MAE_${\mathit{\rho }}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{m}\mathit{t}}$
2	67.8	0.134	4.54	27.9	0.0141	35.3
3	56.7	0.101	4.26	22.1	0.0128	30.2
4	55.6	0.111	4.30	21.2	0.0125	30.0
5	52.6	0.108	3.93	20.0	0.0120	28.5
6	75.2	0.157	5.15	32.5	0.0171	37.4

, increasing the number of encoders generally improved the accuracy of inversion, which indicates a deeper GIT has better generalization performance. For example, the MAE value decreased from 67.8 to 52.6 as the number of encoders increased from two to five. However, using six encoders led to a significant increase in MAE values, indicating that too many encoders may not be optimal.

Table 6. Inversion performance varying with the number of attention heads on the objective parameters.

Number of Heads H	MAE	MAE_${\mathit{h}}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{t}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{b}}$	MAE_${\mathit{\rho }}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{m}\mathit{t}}$
3	59.2	0.109	5.17	21.7	0.0249	32.2
6	65.3	0.125	5.74	25.7	0.0165	33.7
12	56.7	0.101	4.26	22.1	0.0128	30.2
24	56.3	0.108	4.28	22.0	0.0139	30.0

demonstrates the variation in inversion performance concerning the number of attention heads H. It is clear from the table that the number of attention heads had a significant impact on the inversion performance. Specifically, when using only three attention heads, the MAE values were relatively high, indicating poor performance. However, as the number of attention heads increased to 6, 12 and 24, the MAE values decreased, which suggests better inversion performance. Notably, by increasing the number of attention heads from 3 to 12, there was a significant decrease in the MAE values for all parameters. However, using 24 attention heads, although the values remained low, there seemed to be no significant improvement over 12 attention heads.

Overall, the results suggest that using more attention heads to capture more perspectives of the sample data can improve the inversion performance, but there may be a saturation point beyond which any further increase in the number of attention heads might not result in a significant improvement in inversion performance.

We also conducted experiments on the inversion performance varying with the embedding dimension $dim$.

Table 7. Inversion performance varying with the embedding dimension on the objective parameters.

Embedding Dimension $\mathit{d}\mathit{i}\mathit{m}$	MAE	MAE_${\mathit{h}}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{t}}$	MAE_${\mathit{c}}_{\mathit{s}\mathit{b}}$	MAE_${\mathit{\rho }}_{\mathit{s}}$	MAE_${\mathit{c}}_{\mathit{m}\mathit{t}}$
96	59.2	0.120	5.17	23.8	0.0165	36.5
192	67.3	0.125	5.74	25.7	0.0215	36.3
384	56.7	0.101	4.26	22.1	0.0128	30.2
768	67.6	0.108	4.26	27.9	0.0120	35.3

demonstrates that the inversion performance varied significantly with different objective parameters and embedding dimensions. For instance, the best performance in terms of the MAE was obtained when the embedding dimension was 384, whereas the worst score was observed at an embedding dimension of 192. Similarly, the highest score for MAE_$h_s$ was seen at an embedding dimension of 96, and the lowest value was observed at 384.

In all, the embedding dimension and the number of encoders parameter were more sensitive to the GIT than the number of attention heads H, which indicates the depth and width of GIT greatly influence the inversion performance.

4.6. Model Complexity

To show the real-time prediction performance of the GIT, besides MAE, we report the total number of parameters (No. Params), the average prediction time (Avg. time) with the standard deviation, and the frames per second (FPS) to evaluate the time-space complexity, which is shown in

Table 8. Time-space complexity comparison of the models.

Model	No. Params (M)	Avg. Time (ms)	FPS
CRNN	0.237	1341.944 ± 33.171	0.75
FCN	31.043	8.805 ± 0.102	113.57
GIT	4.467	1.804 ± 0.246	554.46

.

It is evident from the table that the proposed GIT model outperformed both CRNN and FCN in terms of the average inference time and FPS. In particular, the GIT model showed significantly faster average inference time at 1.804 ± 0.246 ms, compared to CRNN (1341.944 ± 33.171 ms and FCN (8.805 ± 0.102 ms). Moreover, the GIT model was able to process 554.46 frames per second, which was considerably higher than both CRNN (0.75 FPS) and FCN (113.57 FPS).

The GIT model also strikes a balance between the parameter size and efficiency as it has fewer parameters than the FCN model but more than the CRNN model. This suggests that the GIT model can perform well while avoiding excessive computational costs associated with using larger models, such as FCN.

5. Conclusions

In this study, we propose the GIT, a transformer-based geoacoustic inversion model. By leveraging the multi-head self-attention (MHSA) mechanism, our model can effectively capture global information from the acoustic signals. Additionally, to incorporate positional information from both frequency and sonar sensors, we utilize a 2-D FS positional embedding on the tokens for MHSA calculation. Our experimental results demonstrate the effectiveness of the GIT in geoacoustic inversion by outperforming CNN-based models. However, there is still significant potential for further research based on our work. Due to the complexity of real-world data, we intend to apply domain-adaptive techniques to bridge the gap between simulated and real data. This will serve as the primary direction for future work.