A Novel Multi-Feature Fusion Model Based on Pre-Trained Wav2vec 2.0 for Underwater Acoustic Target Recognition

Abstract

Although recent data-driven Underwater Acoustic Target Recognition (UATR) methods have played a dominant role in marine acoustics, they suffer from complex ocean environments and rather small datasets. To tackle such challenges, researchers have resorted to transfer learning in an effort to fulfill UATR tasks. However, existing pre-trained models are trained on audio speech data, and are not suitable for underwater acoustic data. Therefore, it is necessary to make further optimization on the basis of these models to make them suitable for the UATR task. Here, we propose a novel UATR framework called Attention Layer Supplement Integration (ALSI), which integrates large pre-trained neural networks with customized attention modules for acoustic. Specifically, the ALSI model consists of two important modules, namely Scale ResNet and Residual Hybrid Attention Fusion (RHAF). First, the Scale ResNet module takes the Constant-Q transform feature as input to obtain relatively important frequency information. Next, RHAF takes the temporal feature extracted by wav2vec 2.0 and the frequency feature extracted by Scale ResNet as input and aims to better integrate the time–frequency features with the temporal feature by using the attention mechanism. The RHAF module can help wav2vec 2.0, which is trained on speech data, to better adapt to underwater acoustic data. Finally, the experiments on the ShipsEar dataset demonstrated that our model can achieve recognition accuracy of $96.39\%$. In conclusion, extensive experiments confirm the effectiveness of our model on the UATR task.

1. Introduction

The Underwater Acoustic Target Recognition (UATR) task uses acoustic signals to identify underwater targets, which plays an important role in ocean information sensing. It has two branches: active sonar image recognition [1,2] and passive ship-radiated noise recognition [3,4]. Passive ship-radiated noise recognition uses passive sonar systems to determine the direction and type of underwater targets, which have the characteristics of good invisibility and motility. However, passive ship-radiated noise recognition still faces many challenges such as low Signal-to-noise Ratio (SNR) and the antagonism of targets. Therefore, many studies focus on ship-radiated noise recognition [5,6,7,8].

Ship-radiated noise recognition commonly uses time–frequency analysis techniques such as feature extraction. These techniques include Short-term Fourier Transform (STFT) [9], Mel-spectrogram [10,11,12,13,14,15], Mel Frequency Cepstral Coefficients (MFCC) [16], and Gammatone Frequency Cepstral Coefficients (GFCC) [17]. However, some researchers are exploring alternative features for UATR tasks because the time–frequency features currently used lack sufficient time resolution. These works can be roughly divided into two categories: (1) Finding alternative acoustic features. The idea of this type of method is to explore more efficient time–frequency analysis methods such as Boltzmann machines [18,19] and wavelet transforms [20,21]. (2) Feature fusion methods. The idea of this type of method is to improve the feature representation of data through multi-feature fusion [5,22,23,24]. In particular, Zakaria proposed a method to integrate temporal and frequency information [25]. This method address the issue of insufficient temporal resolution in time–frequency features. Nonetheless, the feature fusion method used in these works is simple concatenation, which makes it easy to confuse the information between different features.

Although the methods mentioned above achieve great performance on ship-radiated noise recognition, they have still been plagued with the following issues: (1) Insufficient time resolution. Existing methods still rely on time–frequency features, resulting in time-domain information loss. Thus, using the temporal features in the raw wave as a supplement of time–frequency features can significantly improve the model’s performance, which has also been proved in [25]. (2) Shallow feature fusion. Existing feature fusion methods still use simple fusion strategies such as concatenation, which results in the confusion of information. Nevertheless, the attention mechanism [26,27] is an efficient feature fusion strategy that can achieve the deep integration of features by capturing the correlation between different features. This mechanism has proven to be effective in multimodal tasks [28,29,30,31].

Therefore, we proposed an Attention Layer Supplement Integration (ALSI) model for the UATR tasks. This architecture overcomes the issues mentioned above by fine-tuning the pre-trained wav2vec 2.0 model [32] and multi-feature fusion. Since wav2vec 2.0 is trained on a large amount of speech data, it has a strong feature extraction ability for sound signals. On the basis of wav2vec 2.0, the downstream UATR mission model is further designed, so that it can adapt to the underwater acoustic data. Specifically, temporal embedding encoded by the pre-trained wav2vec 2.0 model fuses with time–frequency features at different granularities. Next, the fused embeddings are fed into a classifier to complete the UATR tasks.

Briefly, our contributions can be summarized as follows:

(1)

We propose an ALSI model, which can integrate high-resolution temporal information and frequency information to complete UATR tasks efficiently.

(2)

To enhance the information fusion efficiency of ALSI and the adaptability of wav2vec 2.0 on underwater acoustic data, we propose the Scale ResNet module to compress time-domain information and the Residual Hybrid Attention Fusion (RHAF) module to integrate different feature embeddings.

(3)

We conduct extensive experiments and meticulous analysis on a widely used public dataset. The combination of features and the model design produced strong performance, surpassing some existing research.

The rest of the article is arranged as follows: Section 2 summarizes the relevant work, Section 3 provides a detailed explanation of the proposed model, Section 4 presents the experimental design and analyzes the experimental results, Section 5 discusses the results and Section 6 draws conclusions.

2. Related Works

Ship-radiated noise recognition is one of the most popular and long-standing topics in the field of UATR. Feature extraction is a necessary step in ship-radiated noise recognition. Thus, many works focus on feature selection and fusion. Below we introduce their details.

Time–frequency feature extraction is still a necessary step in most of the existing works. Due to STFT being one of the basic methods of time–frequency analysis, Wang proposed an AMNet model to complete the UATR task by using an STFT spectrogram [33]. However, STFT is ineffective in representing the low-frequency information of ship-radiated noise. To better characterize the low-frequency information of ship-radiated noise, a CFTANet model was proposed to enhance low-frequency features by concatenating the Mel-spectrogram sub-band [34]. Just like the Mel-spectrogram, GFCC is also a time–frequency analysis method that can characterize the low-frequency information of ship-radiated noise [35]. Therefore, Feng proposed a WA-DS fusion model that integrates GFCC to enrich the character representation of ship-radiated noise [36]. In a word, many works choose Mel-spectrogram and GFCC as time–frequency features to complete UATR tasks [15,17,37,38], as they can more efficiently represent the low-frequency characteristics of ship-radiated noise compared to STFT.

Although these feature extraction methods could effectively improve the recognition performance, a single time–frequency spectrogram cannot provide sufficient information. Thus, researchers began to use feature fusion methods to enrich feature representation and these methods can be roughly divided into two categories: (1) Fusion of time–frequency features. Most feature fusion algorithms use Fourier Transform-based features (STFT, Mel-spectrogram, MFCC, and GFCC are all calculated on the basis of Fourier Transform) [4,5,39,40]. For example, Log Mel, MFCC and optimized feature based on Center loss were integrated in [41]. Zhu proposed integrating Mel-spectrogram with Constant Q Transform (CQT) [42] and achieving great performance [4]. This kind of feature fusion method also appears in [43]. CQT is a wavelet-based time–frequency feature that can represent the low-frequency information of ship-radiated noise more effectively than Fourier Transform-based features. However, due to insufficient high-frequency resolution, CQT is rarely used alone in UATR tasks and needs to be used together with Fourier Transform-based features. (2) Fusion of temporal features and time–frequency features. The idea of this type of method is to use temporal features extracted from raw waves to supplement temporal information for time–frequency features. A more representative work is [25], mentioned above.

Nevertheless, there are still insufficient temporal information issues with these methods. Thus, we proposed a multi-feature fusion method that integrates the temporal feature, CQT feature, and Mel-spectrogram. Firstly, integrating the temporal feature with the CQT feature provides a solution for insufficient temporal information. Secondly, integrating the Mel-spectrogram with the CQT feature can solve the CQT feature’s insufficient high-frequency resolution issue mentioned above.

3. Method

We propose an ALSI model to enhance the recognition accuracy of ship-radiated noise, as in Figure 1. First, ALSI takes the raw wave as the input and converts it into time series features and time–frequency features. These features are then fused at different granularities through two integration layers consisting of RHAF blocks. Finally, the embedding after fusion is sent into the classifier to complete the UATR tasks.

3.1. Time–Frequency Feature Extraction

ALSI takes the original waveform and two time–frequency features as input. The time–frequency feature includes the CQT feature and the Mel-spectrogram.

We randomly selected a 5 s sound clip from each of the 12 sound classes in the public dataset ShipsEar [44] for analysis. To make it clear that the difference between the two features is clear, the data are normalized. As in Figure 2a, The CQT feature has a high resolution of low-frequency information while retaining the textural properties. It can also reduce impulse noise, resulting in a cleaner spectrogram. As in Figure 2b, the textural detail of the Mel-spectrogram is precise, but the high-frequency pulse component still exists. Thus, CQT features play a more important role in feature fusion, and the Mel-spectrum serves as a supplement feature. The CQT feature $X_{CQT}$ is calculated as

$$X_{CQT}(k,n)={\displaystyle \sum _{j=n-⌊\frac{N_k}{2}⌋}^{n+⌊\frac{N_k}{2}⌋}}x\left(j\right)a_k^{*}(j-n+\frac{N_k}{2}),k=1,2,3,\dots ,K,$$

(1)

where, k represents the number of CQT filters, K is the index of CQT filter banks, $N_k$ denotes window length, $a_k^{*}\left(n\right)$ indicates the complex conjugate of $a_k\left(n\right)$, which can be calculated as

$$a_k\left(n\right)=\frac{1}{N_k}\omega \left(\frac{n}{N_k}\right)exp\left[-i2\pi n\frac{f_k}{f_s}\right],$$

(2)

where, $f_k$ is the center frequency of $k^{th}$ CQT filter, and $f_s$ is the sample rate. The window function is represented by $\omega (·)$.

Another time–frequency feature is the Mel-spectrogram, which can be calculated as follows:

$$M={\displaystyle \sum _{k=0}^{N-1}}{\left|X\left(k\right)\right|}^2{F}_i\left(k\right),0\le i\le B,$$

(3)

where, $X\left(k\right)$ represents the Fast Fourier Transform (FFT) of input radiated noise, and B is the number of Mel filters, $F_i(·)$ represents the Mel filter banks.

After feature extraction, the CQT feature and Mel feature will be organized into tensors and fed into the ALSI model, respectively. The tensor dimension of the CQT feature is $E_c\in {\mathbb{R}}^{b\times t\times f_c}$, and the tensor dimension of the Mel-spectrogram is $E_m\in {\mathbb{R}}^{b\times t\times f_m}$, in which b represents the batch size, and t represents the frame length, $f_c,f_m$ denote frequency bins of the CQT feature and Mel-spectrogram, respectively.

3.2. Scale ResNet Block

To obtain frequency-domain information, we proposed a Scale ResNet block to compress time-domain information in the CQT feature, as shown in Figure 3.

The Scale ResNet block takes the CQT feature as input and is then processed by a convolutional Neural Network (CNN) layer first. The feature maps obtained from the previous CNN layer are sent to the Residual Block for the further compression of time-domain information. We design three such Residual Blocks to compress the time-domain information in the CQT feature progressively. After then, the CQT feature is embedded from $E_c\in {\mathbb{R}}^{b\times t\times f}$ into frequency embedding $E_f\in {\mathbb{R}}^{b\times f}$ through a CNN layer and an adaptive average pooling layer. This frequency embedding is able to describe the frequency component of the ship-radiated noise. Finally, the frequency embedding compressed by the Scale ResNet block will be involved in the next step of feature fusion.

3.3. Residual Hybrid Attention Fusion Block

The RHAF module receives two different embeddings encoded by preceding network layers as input. This module combines Multi-head Self Attention (MHSA) and Multi-head Cross Attention (MHCA) to focus on the correlation between sequences better, as shown in Figure 4.

We denote the first input embedding of the RHAF module as major embedding $E_{mj}\in {\mathbb{R}}^{b\times l}$ and denote the second input embedding as minor embedding $E_{mi}\in {\mathbb{R}}^{b\times f}$. Where l denotes embedding length. To begin with, $E_{mj}$ and $E_{mi}$ will be transformed into $E_{mj}^{\prime }\in {\mathbb{R}}^{b\times l\times d}$ and $E_{mi}^{\prime }\in {\mathbb{R}}^{b\times l\times \times d}$ by a group of linear layers. And then $E_{mj}^{\prime },E_{mi}^{\prime }$ is encodded into embedding $Q:E_{m{j}_q}^{\prime }\in {\mathbb{R}}^{b\times l\times d_q},K:E_{m{j}_k}^{\prime }\in {\mathbb{R}}^{b\times l\times d_k},V:E_{m{j}_v}^{\prime },E_{m{i}_v}^{\prime }\in {\mathbb{R}}^{b\times l\times d_v}$ by another group of linear layers, where $d_q=d_k=d_v$ is the dimension of embedding Q, K, and V. For the MHSA part, Q, K, and V are encoded from the same embedding, so that MHSA can obtain the correlations between different values within major embedding. The attention map of MHSA can be calculated as follows:

$$Energy=\frac{E_{m{j}_q}^{\prime }{E}_{m{j}_k}^{\prime T}}{\sqrt{d_q}},$$

(4)

$$A_s=Softmax\left(Energy\right)·E_{m{j}_v}^{\prime }.$$

(5)

For the MHCA part, the attention map is generated by $E_{m{j}_q}$ and $E_{m{j}_k}$, and then $E_{m{i}_v}$ is multiplied. Therefore, MHCA enables the extraction of correlation information between different embeddings. The attention map of MHCA can be calculated as follows:

$$A_c=Softmax\left(Energy\right)·E_{m{i}_v}^{\prime }.$$

(6)

To merge both MHSA and MHCA, we introduce a hyper-parameter $\mu$ to perform a weighted sum of the two attention results:

$$A=A_s+\mu A_c.$$

(7)

To let modules learn features more effectively, we introduce a residual connection during the attention process. This residual connection can be calculated as

$$Attentio{n}_{output}=E_{mj}+A.$$

(8)

The last step is to concatenate the outputs of two embeddings and use a Multi-layer Perceptron (MLP) to reshape the output to be the same size as the input.

For an RHAF block, Q and K come from the same embedding, while the V comes from two input embeddings. As a result, the RHAF block can obtain correlations not only within the same embedding but also between different sequences. Drawing inspiration from how the human brain integrates information, it often first identifies important features within a single piece of information, then compares the relevance of these important features in multiple pieces of information, and finally integrates these crucial features. Therefore, we have designed a form of mixed attention using MHSA and MHCA to imitate this process, with a hyperparameter $\mu$ controlling the weight of integrating important features.

3.4. Multi-Stage Supplement Integration

To improve underwater radiated noise recognition accuracy, we propose an ALSI model based on the multi-feature fusion method, as shown in Figure 1. The ALSI takes the raw wave as input and processes it into one temporal feature and two time–frequency features: the CQT feature and the Mel-spectrogram. The temporal feature is encoded by wav2vec 2.0. And then, the time–frequency features are processed by Scale ResNet and ResNet18 for further fusion. The ALSI model has three branches of fusion, the details of which are as follows:

• Macro fusion. In this fusion branch, we aim to integrate frequency and temporal embedding. Since wav2vec 2.0 already provides high-resolution temporal features, the time-domain information in the CQT features is unimportant in this fusion branch. Therefore, we compress the time-domain information in the CQT feature using Scale ResNet. Scale ResNet encodes the CQT feature into frequency embedding, which will then be integrated with temporal embedding. There are two RHAF blocks in the Macro fusion branch. These two blocks take frequency embedding and temporal embedding as input. Then, we combine these two RHAF blocks’ outputs into one embedding and use an MLP to encode it into another embedding for the next stage of fusion.
• Fine-grained fusion. In this branch of integration, detailed textural features in the CQT spectrogram participate in the fusion process. The Fine-grained fusion branch uses an intact CQT spectrogram to extract textural feature embedding using ResNet18. And then, the embedding from the previous fusion branch and the textural feature embedding together form the input for the second fusion branch. Similar to the Macro fusion, the Fine-grained fusion also consists of two RHAF blocks. These two blocks take the textural and previous fused embedding as input. Then, we combine these two RHAF blocks’ outputs into one embedding and also use an MLP to encode it into another embedding for the final fusion branch.
• Comprehensive integration. In this branch, ResNet18 is first used to extract textural features from the Mel-spectrogram and encode them as embeddings. The Mel-spectrogram can provide more detailed textural features than the CQT feature. The textural embeddings are then concatenated with the output of the embedding from the previous branch and integrated using an MLP for information fusion. It is important to note that RHAF is not used in this branch because there is a semantic gap between the Mel-spectrogram and the CQT spectrogram. Since a large amount of CQT information has already been integrated into the existing embeddings, attending to the Mel-spectrogram at this point would lead to semantic confusion and potentially degrade the model’s performance.

Finally, after three branches of integration, we obtain an embedding that collects high-resolution temporal information and frequency information. The last step of ALSI is to use a classifier to complete UATR tasks.

3.5. Model Learning

We denote the output of ALSI as $z_i,i=1,\cdots ,N$, where N is the number of categories. Then, the output will be transformed into probabilities for each class using softmax:

$$Pred=softmax\left(z_i\right)=\frac{e^{z_i}}{{\displaystyle \sum _{c=1}^N}{e}^{z_c}}.$$

(9)

Due to it being a classification task, the cross-entropy is selected as the loss function:

$$H(P,Q)=-\sum _{i}P\left(i\right)·logQ\left(i\right),$$

(10)

where P represents the one-hot label, and Q denotes the prediction result. The loss will be used to update the gradients of each layer in the network through backpropagation and then optimize the weights of every node. Thus completes one epoch of the learning process.

4. Experiments

In this section, we first introduce the experiment setup and dataset. Then, the experimental results between our model and multiple baseline models are analyzed. Furthermore, we discuss the effectiveness and superiority of each module of our model by ablation experiments.

4.1. Experiments Setup

We have designed the experiments to demonstrate the model performance from the following aspects: (1) Comparing with the existing work, we discuss the model’s performance; (2) We verify the impact of model performance when fusing different time–frequency features and temporal feature; (3) We use ablation experiments to validate the effectiveness of the RHAF module; (4) A hyper-parameter $\mu$ in RHAF controls the weight of integrating. Experiments will investigate how the model performance is affected by $\mu$. (5) The experiment will analyze the performance of models with different structures.

The dataset used in the above experiments is ShipsEar. Specifically, we do not use the conventional five divisions for dataset partitioning. Instead, we treat each type of ship-radiated noise as a separate category. To ensure data balance, we select the data for training and testing the model from the following categories: Fishboat (Class 0), Motorboat (Class 1), Mussel boat (Class 2), Natural ambient noise (Class 3), Ocean liner (Class 4), Passengers (Class 5), and RORO (Class 6). The sample size for each category is shown in the

Table 1. Sample size and label information of each category.

Category	Label	Size
Fishboat	Class 0	1683
Motorboat	Class 1	1751
Mussel boat	Class 2	1667
Natural ambient noise	Class 3	1420
Ocean liner	Class 4	1875
Passengers	Class 5	2078
RORO	Class 6	1513 1

¹ Each sample is a two-second wav file with a sample rate of 22,050 Hz.

. The experiments are conducted on a server with two Nvidia RTX3090 GPUs, each with 22 GB of VRAM (Nvidia, Santa Clara, CA, USA). The operating system used is Ubuntu 18.04, with Python version 3.9 and PyTorch version 2.0.0.

4.2. Results and Analysis

4.2.1. Model Performance

To evaluate the model performance,

Table 2. The performance between the proposed model in this paper and existing work.

Model	Feature (s)	Accuracy	Precision	Recall	F1-Score
Yamnet [16]	MFCC	$78.72\%$	$68.94\%$	$83.58\%$	0.7264
VGGish [45]	Log-Mel	$86.57\%$	$83.30\%$	$85.82\%$	0.8427
ADCNN [46]	Raw Wave	$93.58\%$	$92.36\%$	$97.49\%$	0.9469
CRNN9 [10]	3D Log-Mel	$91.17\%$	$77.64\%$	$95.43\%$	0.8463
LSTM-based [9]	STFT	$94.77\%$	$91.32\%$	$\mathbf{98.14}\%$	0.9449
DRW-AE [21] *	Wavelet	$94.49\%$	-	-	-
ResNet18 [47] *	3D fusion features	$94.30\%$	-	-	-
ALSI (ours)	Raw Wave, CQT, Log-Mel	$\mathbf{96.37}\%$	$\mathbf{96.74}\%$	$96.36\%$	0.9607

* No other results were mentioned in the literature.

compares the performance of the ALSI model proposed in this paper with existing work. Based on the results, our proposed model performs better than most of the existing work. Significantly, because there are many false positive examples in the recognition results of the LSTM-based [9] model, the recall rate is inflated and the accuracy is reduced. Specifically, the confusion matrix in Figure 5 shows that some data in the Motorboat and Passengers classes are confused with other categories. While very few misclassifications occur in the other categories. Analyzing the convergence of the model from Figure 5b, due to sufficient features, the model converges quickly. Figure 6 shows the histogram of classification precision, recall, and F1-score for each type of ship-radiated noise. The graph shows that the accuracy for each category is quite similar, and the model can effectively classify most of the data with only a small amount of data.

4.2.2. Performance with Different Input Features

We first conducted ablation experiments to validate the effectiveness of the features we selected. We test four conditions for the UATR task: (1) Only the temporal information provided by wav2vec 2.0; (2) Fusion of the temporal information from wav2vec 2.0 and CQT features; (3) Fusion of the temporal information from wav2vec 2.0 and Mel-spectrogram feature; (4) Fusion of the temporal information from wav2vec 2.0, CQT features, and Mel-spectrogram features.

In

Table 3. Ablation experiments of different input features.

Temporal	Frequency (CQT)	Frequency (Mel)	Accuracy	Acc Diff 1
✓ 2	-	-	$82.60\%$	$14.38\%$
✓	✓	-	$94.34\%$	$5.840\%$
✓	-	✓	$90.54\%$	$8.730\%$
✓	✓	✓	$\mathbf{96.37}\%$	$\mathbf{3.540}\%$

¹ Accuracy difference between training and validation. ² ✓ indicates that the feature is used in the model.

, the test results show that the best performance is achieved when all three features are input at the same time, because of the sufficient feature. The fusion of temporal information with CQT features performs worse than the former. The fusion of temporal information with Mel-spectrogram features and using only temporal information perform unsatisfactorily.

Secondly, due to the many advantages mentioned earlier, CQT contains less noise than the Mel-spectrogram, making it more suitable for providing frequency information in this task. As the results show, models incorporating CQT in the fusion process perform better than models incorporating Mel-spectrogram. In addition, as shown in Table 3, the CQT feature fused with the temporal feature performs better than the Mel-spectrogram fused with the temporal feature. This is because CQT contains less noise than Mel-spectrogram, making it more effective.

Additionally, corresponding t-SNE plots are also drawn based on the output results of the proposed model with different input features, as shown in Figure 7. By comparing them, it can be observed that adding more features enriches the feature representation. Comparing Figure 7a to other figures, we can see that increasing the feature input can greatly enhance the distinctiveness of the embedding. By comparing Figure 7b with Figure 7c, it can be seen that the embedding with added CQT has better distinctiveness than the embedding obtained by adding the Mel-spectrogram. Finally, comparing Figure 7b–d shows that the embedding with multiple feature fusion has the highest distinctiveness.

Moreover, over-fitting also affects the model performance. Therefore, we verify that integrating more features with the temporal features can restrain over-fitting through ablation experiments. The last column of Table 3 shows that over-fitting is more severe when only using wav2vec2.0, with a difference in accuracy between training and validation of up to 0.14. However, this was effectively mitigated after adding time–frequency features. Combining three types of features has the best effect on restraining over-fitting. Since CQT can express frequency information more clearly, the difference in training and validation accuracy compared to combining the Mel-spectrogram is relatively tiny. Thus, it can be concluded that increasing the feature inputs can reduce over-fitting while improving the model’s recognition performance.

4.2.3. Model Performance with Different Fusion Methods

To investigate the impact of feature fusion techniques on the model, three ablation experiments were performed in three different scenarios: (1) retaining the RHAF of the first stage and replacing the RHAF of the second stage with feature concatenation; (2) retaining the RHAF of the second stage and replacing the RHAF of the first stage with feature concatenation; (3) retaining RHAF in both stages.

Comparing the first two rows of

Table 4. Ablation experiments of different fusion methods ¹.

Level 1	Level 2	Level 3	Accuracy
✓ 2	-	✓	$90.45\%$
-	✓	✓	$57.26\%$
✓	✓	✓	$\mathbf{96.37}\%$

¹ When MLP is used in all fusion phases, the model does not converge and is not listed in the table. ² ✓ indicates that the module is included in the model.

, it can be seen that fusion in the first stage is the primary source of performance, and using fusion in the second stage alone cannot provide sufficient performance support. It can only be a finishing touch on top of the fusion in the first stage. Finally, merging the three features using RHAF can improve the model performance.

4.2.4. Performance with Different Hyper-Parameters

In the previous section, we introduced a hyper-parameter $\mu$ in RHAF, which played a significant role in the feature fusion process. To investigate the selection of this hyper-parameter $\mu$, we designed ablation experiments specifically for it.

Due to the two-stage RHAF in the model, we take 1, 2, and 3 for $\mu$ in the two stages to verify their performance changes, as shown in

Table 5. Performance with different hyper-parameters.

Level 1	Level 2	Accuracy
1	1	$94.38\%$
1	2	$94.13\%$
1	3	$93.88\%$
2	1	$93.03\%$
2	2	$\mathbf{96.37}\%$
2	3	$93.96\%$
3	1	$91.42\%$
3	2	$95.06\%$
3	3	$94.47\%$

. The experimental results demonstrate that performance is better when $\mu$ is simultaneously selected as 2 in both stages. This is because when $\mu$ is selected as 1 in the first stage, insufficient weight is provided to the fused parameters during the fusion process, which are added to the fused features. When $\mu$ is selected as 3, it causes the fused features to dominate and weaken the role of the fused features, leading to a decrease in performance. Therefore, selecting 2 in the first stage is currently the best choice. Similarly, the selection of $\mu$ in the second stage can also be explained.

4.2.5. Performance with Different Model Structures

The structure designed in the text is a form of serial connection that integrates features. A parallel fusion model was designed to investigate the impact of model structure on performance. Since this model added too much ambiguous attribute information to the temporal features at once, causing the features to become confused in representing the data, the model did not converge, as seen in

Table 6. Performance with different model structures.

Structure	Accuracy
Parallel	$14.14\%$ (Not converge)
Concatenation	$\mathbf{96.37}\%$

. This indicates that adding information to a feature should be gradual and not rushed. Since the designed model did not converge, the model structure is not elaborated on here.

5. Discussion

Through the ablation experiments above, what and how these features are fused and which hyper-parameters are chosen in the fusion model are all factors that affect the model’s performance. First, choosing low-noise and high-resolution features is essential, and CQT is one such feature. Thus, integrating frequency information from the CQT feature with temporal embedding extract by wav2vec2.0 can further improve model performance. Meanwhile, textural features in the time–frequency spectrogram are also important. Therefore, integrating embeddings extracted from the CQT feature and Mel-spectrogram by ResNet18 can supplement detailed feature information. Secondly, integrating embedding with different semantics by adding them directly will hurt the model performance. Thus, we designed an RHAF block. The ablation experiment shows that the RHAF block is efficient, as it better focuses on correlations between time–frequency and temporal features. Moreover, different features have different importance during the fusion period, so choosing an appropriate set of hyper-parameters can balance the relationship between attribute features. Overall, the experiments verified that our proposed ALSI model performs well on the UATR task.

6. Conclusions

The article proposes an ALSI model based on pre-training models. This model integrates temporal information provided by wav2vec2.0 and time–frequency information provided by the CQT feature and Mel-spectrogram. To begin with, we selected a Scale ResNet temporal information compression module. Simultaneously, an RHAF feature fusion module is proposed to address the fusion of multiple features and improve the adaptability of wav2vec 2.0 on underwater acoustic data. Finally, the experimental results show that the proposed model performs well on the ShipsEar dataset, achieving a recognition accuracy of $96.36\%$, surpassing existing work. The efficiency of the proposed modules is validated through ablation experiments. The optimal selection of hyper-parameters and the optimal design of the model structure are explored. The next step will involve improving the fusion module of the model to enhance the model’s ability to represent features.