A Mapping Method Fusing Forward-Looking Sonar and Side-Scan Sonar

Abstract

In modern ocean exploration, forward-looking sonar (FLS) provides real-time 2D imaging of the seabed ahead, but its detection range is relatively limited. Conversely, side-scan sonar (SSS) enables large-scale imaging of the seabed during movement but struggles to effectively image areas directly beneath the sensor. Integrating FLS and SSS offers a promising solution by leveraging their complementary strengths to achieve comprehensive seabed mapping. However, no prior research has explored this fusion approach. This paper presents a novel method for FLS and SSS fusion mapping. Firstly, a novel sonar image enhancement method based on equalization is proposed, enabling simultaneous enhancement and grayscale unification of two sonar images. Additionally, an effective area extraction approach for FLS images, grounded on the approximate erosion method, is introduced to produce high-quality FLS mapping. Furthermore, by examining the data distribution in FLS and SSS mappings, the standard deviation of these datasets is utilized to refine the grayscale distribution of FLS mapping, thereby enhancing the grayscale distribution similarity between the two mapping results. Finally, FLS map data are seamlessly integrated into the gaps of the SSS map, resulting in a fused, comprehensive seabed representation. Large-scale experiments demonstrate that the proposed method effectively combines the strengths of FLS and SSS, producing complete and detailed seabed topography maps. Simultaneously, numerous ablation experiments are conducted to evaluate the impact of various parameters on fusion mapping, providing guidelines for selecting the optimal parameters. This fusion approach, thus, holds significant practical value for ocean exploration and seabed mapping applications.

1. Introduction

In modern ocean exploration, the attenuation of radio waves in water significantly limits the range of observation using optical cameras. Sonar sensors, by contrast, offer long detection distances, high accuracy, and superior image resolution, making them indispensable for underwater observation. Consequently, sonar technology has become a cornerstone in various marine applications, including seabed topography imaging, coral monitoring, specialized target detection, and fishery resource management [1,2]. Additionally, extensive research has been conducted in sonar data processing, encompassing underwater target recognition [2] and sonar image segmentation [3,4].

Among the diverse sonar technologies, side-scan sonar (SSS) stands out for its wide application in ocean observation. SSS is typically mounted on ocean-going platforms such as surface vessels and unmanned underwater vehicles (UUVs), utilizing the continuous movement of these platforms to achieve large-scale seabed imaging. It is renowned for its long detection range, high image resolution, and exceptional imaging quality. However, its reliance on vehicle movement often introduces severe distortions in imaging results. To address these distortions, geocoding is employed to construct maps and restore the original geometry. This geocoding process integrates vehicle location information with SSS data to produce accurate representations [5]. Despite its advantages, SSS suffers from a critical limitation: its transducers, installed on both sides of the platform, cannot effectively image the seabed directly beneath the vehicle, resulting in incomplete seabed coverage.

To mitigate this limitation, current research predominantly focuses on image stitching techniques. These methods employ overlapping scan routes to combine results, using the overlapping regions to fill in areas directly below the vehicle that are missed in single scan routes. However, the accuracy limitations of vehicle navigation systems often lead to misalignments in SSS-generated maps, necessitating advanced image stitching techniques to resolve these discrepancies.

Research on SSS image stitching has largely concentrated on image matching methods, which are crucial for achieving high-quality stitching results. For instance, Zhou et al. proposed a combination matching method for SSS images that leverages phase information and deep convolutional features, achieving robust matching performance [6]. Similarly, Zhang et al. introduced a strip image stitching technique based on curvelet transforms and resolution constraints, delivering superior fusion of overlapping regions and effectively eliminating blurry areas in the directly below section of SSS maps [7]. However, this method imposes a significant limitation: the distance between adjacent scan routes must be smaller than the detection range of the SSS. This requirement reduces the efficiency of ocean scanning and fails to meet the speed demands of modern ocean surveying tasks.

Forward-looking sonar (FLS) is a widely used sensor in ocean scanning tasks, capable of performing real-time 2D imaging of the underwater environment. This capability facilitates manual observation and makes FLS a valuable tool for ocean mapping, an increasingly significant area of research. Current research on FLS-based mapping can be broadly categorized into two areas: sonar simultaneous localization and mapping (SLAM) and 3D terrain reconstruction.

In the domain of sonar SLAM, Xu et al. integrated inertial navigation data with FLS images to achieve fusion-based positioning [8]. They introduced an inertial-assisted sliding window optimization framework to address challenges such as incorrect feature matching and insufficient feature points. Similarly, McConnell et al. employed a convolutional neural network (CNN) to generate synthetic overhead images, which were subsequently registered to global overhead imagery for SLAM. Their method incorporated these transformations into a pose SLAM factor graph, achieving robust SLAM performance [9]. Furthermore, an exploration framework for underwater robots operating in cluttered environments has been proposed, utilizing imaging sonar-based SLAM. This framework comprises multiple components, including path generation, location recognition prediction, belief propagation, and utility evaluation using virtual maps [10].

For 3D terrain reconstruction, FLS-based techniques often leverage structure-from-motion (SFM) methodologies to combine multiple FLS images. Franchi et al. proposed a probabilistic 3D occupancy mapping framework tailored for UUV, enabling reconstruction of the seabed’s topographic structure [11]. Another approach applied an enhanced shape-from-shading (SFS) algorithm, employing the Oren and Nayar diffuse reflection model, for 3D reconstruction of sonar images. This method was validated using simulation and sea trial data, achieving reconstructions closely aligned with the actual sizes of underwater objects [12]. Arnold et al. designed a neural network capable of predicting signed distances and directions to the nearest surface for each voxel, innovatively integrating sparse Doppler velocity log (DVL) distance measurements to train a dense 3D reconstruction model [13]. Additionally, a method known as acoustic structure from motion (ASFM) was developed to reconstruct 3D scenes from multiple 2D sonar images while simultaneously localizing the sonar [14].

Despite these advancements, FLS mapping remains constrained by its small detection range and low resolution. Existing FLS-based mapping methods are limited to a narrow operational range (typically under 100 m), as most ocean environments lack abundant features necessary for effective matching. This feature-scarcity significantly hampers the practical application of FLS SLAM.

Analysis of FLS and SSS data highlights their complementary characteristics. SSS offers a large detection range and high resolution but cannot effectively image the seabed directly beneath the sonar. Conversely, FLS excels in imaging areas directly in front and below the sonar but suffers from a limited detection range and lower resolution. Combining FLS and SSS through fusion mapping offers a compelling solution, leveraging their respective strengths to achieve more comprehensive and effective seabed mapping.

This study utilizes an unmanned boat to simultaneously collect FLS and SSS data. To unify the grayscale distribution of the two sonar images, a normalization-based sonar image equalization method is applied. The sonar attitude and positioning data from the navigation system are then incorporated to geocode the FLS and SSS images separately. Subsequently, the standard deviations of the two datasets in their overlapping regions are computed, and the grayscale distribution of the FLS map is adjusted accordingly. Finally, the adjusted FLS map data are integrated into the gaps of the SSS map, resulting in a fused and comprehensive seabed map.

The organization of this paper is as follows: Section 2 analyzes the original images of FLS and SSS and introduces the method proposed in this paper based on the characteristics of sonar images. Section 3 introduces the principle and implementation process of the proposed method in detail. Section 4 conducts ablation experiments on the adjustable parameters and then demonstrates the reliability and practicality of our method by fusion mapping in large-scale sonar data. Section 5 discusses the image filtering and fusion methods used in this paper, comparing the fusion map results with those obtained using other methods. Section 6 summarizes the work of this paper.

2. Statement of the Problem

When a vehicle scans the ocean using SSS, the SSS transducer emits fan-shaped sound waves of a specific frequency into its vertical plane. As these sound waves propagate through the water, they gradually attenuate. Upon encountering the seabed, part of the sound energy is reflected back toward the sonar along the original path. The sonar system measures the intensity of these echoes at fixed time intervals, capturing information about the seabed.

The received echo intensities are arranged chronologically based on their time of arrival to create a 1D sonar ping. By combining multiple sequential pings side by side, a 2D SSS image is generated. Figure 1 illustrates a typical example of an SSS image, highlighting its structure and imaging characteristics.

Analysis of a typical SSS image reveals its high resolution; however, it also suffers from significant grayscale imbalance. In original SSS images, the brightness of the outermost regions often approaches zero, making it difficult to effectively capture and reflect seabed features. Additionally, SSS images exhibit a water column area, which prevents effective imaging of the region directly beneath the sonar. Although geocoding can eliminate the water column area, the resulting images from the left and right sides remain challenging to seamlessly splice.

To address this limitation, this study incorporates imaging results from FLS to fill in the area directly beneath the SSS coverage. This integration is critical for achieving full-coverage scanning in a single pass, enhancing both mapping quality and operational efficiency.

When operating in the ocean, FLS emits sound waves into a square-cone-shaped area directly in front of the sonar, forming multiple beam planes arranged at equal intervals. The system continuously records the intensity of echoes received from each beam plane at fixed time intervals. These echoes are then processed to generate a 2D ocean topography image. Figure 2 illustrates a typical FLS image, highlighting its capability to complement SSS imaging by covering areas directly in front and below the sonar.

Figure 2 shows that the effective imaging region of the FLS on the seabed is concentrated in a specific portion of the image, while other areas lack significant features and serve as background. The effective imaging area represents the FLS’s coverage of the seabed directly in front of the sonar. As the vehicle advances, this effective imaging area overlaps with the images generated by subsequent SSS scans. This overlap enables the integration of FLS imaging data with SSS data for fusion mapping.

3. Method

To achieve the fusion mapping of FLS and SSS, it is essential to collect data from both systems simultaneously. Additionally, the relative positions of the SSS and FLS sensors must remain fixed throughout the operation. The configuration of the unmanned boat and the method for mounting the sonar systems designed for this study are illustrated in Figure 3.

As illustrated in Figure 3, when the unmanned ship moves forward, the area of poor imaging directly below the SSS overlaps with the effective imaging area of the FLS. This overlap allows the FLS imaging data to compensate for the areas with poor imaging beneath the SSS, thereby enabling the fusion mapping of SSS and FLS data.

As demonstrated by the typical FLS and SSS images in Section 2, the two types of sonar images differ significantly in terms of brightness and imaging characteristics. To address these differences, the original SSS and FLS images must undergo grayscale equalization, ensuring consistent grayscale levels. Following equalization, geocoding is performed to align the images spatially. Ineffective imaging areas in both sonar images are then removed to optimize geocoding accuracy. Finally, the images are fused based on the complementary characteristics of the two sonar systems to achieve a unified sonar fusion map.

The flowchart of the proposed method is presented in Figure 4.

The proposed method begins by applying a normalized image equalization technique to unify the grayscale levels of FLS and SSS images. The equalized SSS images are then geocoded to generate an initial map. For the FLS images, the effective imaging area is identified using an approximate erosion method, and the grayscale-equalized data within this area are also geocoded to produce a corresponding map. During the fusion stage, the standard deviations of the overlapping regions between the FLS and SSS maps are proposed to adjust the grayscale distribution of the FLS data. Subsequently, FLS map data are seamlessly integrated into the gaps of the SSS map, resulting in a fused, comprehensive seabed representation.

The proposed method consists of three main components: (1) a sonar image grayscale equalization method based on normalization, (2) sonar geocoding, and (3) sonar fusion mapping. Each component is detailed in the subsequent sections.

3.1. Grayscale Equalization and Geocoding for SSS

Analysis of the original images from the two sonar systems reveals common challenges, including uneven grayscale distribution, high noise levels, and blurred imaging. Furthermore, the imaging results for the same object vary significantly due to the differing detection angles of the two sonars. To facilitate effective fusion, it is necessary to preprocess the sonar images uniformly, ensuring that the grayscale distributions of the FLS and SSS images are comparable. However, existing sonar image enhancement research focuses exclusively on single-sonar characteristics, and no prior work addresses the unification of grayscale distributions across multiple sonar images.

In real seabed environments, the offshore seabed predominantly consists of flat sandy areas shaped by prolonged seawater erosion. Special features such as reefs, shipwrecks, and sand dunes are scattered across this sandy background. Consequently, the common sandy seabed can be treated as the acoustic baseline, allowing normalization of the original sonar images to the acoustic characteristics of the sandy seabed. In this context, special objects on the seabed are regarded as distinct patterns against the sandy background. This normalization approach is applicable to both sonar systems, achieving grayscale equalization and unification of their respective images.

This study introduces a sonar image equalization method based on normalization, drawing inspiration from the Retinex-based approach [15]. The method involves the following steps:

• The original sonar image is mean-filtered to approximate the sandy background.
• The filtered image is used to normalize the original sonar image, effectively separating the background from special features.
• The normalized image is scaled by a brightness parameter, $\kappa$, and mapped to the grayscale range [0, 255], resulting in a grayscale-equalized image.

The brightness parameter, $\kappa$, plays a critical role in the proposed method and will be subjected to detailed analysis through ablation experiments. The process of SSS image equalization and geocoding is illustrated in Figure 5.

As shown in Figure 5, the mean filtering step applied to the SSS image uses a filter size denoted as $M^{SSS}$, which is a critical parameter influencing the effectiveness of sonar mapping. A detailed ablation experiment evaluating the impact of $M^{SSS}$ will be presented in the experimental section.

The geocoding method for SSS begins with the flat seafloor assumption. Using this assumption, a triangulated model calculates the spatial position corresponding to each pixel in the SSS image. The coordinate system definitions used in the geocoding process are illustrated in Figure 6.

The world coordinate system (WCS) in Figure 6 is defined as follows: the X-axis points northward, the Y-axis points eastward, and the Z-axis points toward the center of the Earth. Its origin is typically chosen as the point where the FLS begins operation, and the X-Y plane of the WCS is at sea level.

The coordinate system of SSS (SSSCS) in Figure 6 is defined as follows: it is fixed to the SSS device, and its origin is at the center of the sonar transmitter. Its X-axis points forward from the sonar, the Y-axis points to the right of the sonar, and the Z-axis points downward from the sonar.

When the SSS is mounted on a carrier, the geographic coordinates, $P_{SSS}$, and the rotation matrix, $R_{SSS}$, of the SSS device’s center can be determined using the navigation system data. These parameters are provided as known inputs:

• $R_{SSS}$: The rotation matrix defining the orientation of the SSSCS relative to the WCS.
• $P_{SSS}$: The displacement vector between the SSSCS and the WCS.

Additionally, the carrier is equipped with a seabed altimeter, which provides the relative distance, $H$, between the SSS device and the seabed. For any point on the SSS image, its corresponding distance from the sonar center is denoted as $r$. The spatial point on the flat seabed corresponding to this image point is defined as $P_r$.

To further describe the geometry, we define the angle ${\theta }_r$ between the line connecting $P_r$ and $P_{SSS}$ and the vertical Z-axis. The calculation of ${\theta }_r$ is as follows:

$${\theta }_r=arccos{\displaystyle \frac{H}{r}}$$

(1)

When the pixel is on the left side of SSS image, ${\theta }_r$ is set to a negative value. When the point is on the right side of SSS image, ${\theta }_r$ is set to a positive value. At the same time, Equation (1) needs to ensure that $r$ is greater than $H$, so that the water column area is eliminated during the mapping process. The calculation method of $P_r$ is as follows:

$$P_r=P_{SSS}+R_{SSS}\times \left[\begin{array}{c}0\\ r\times sin{\theta }_r\\ r\times cos{\theta }_r\end{array}\right]$$

(2)

Then, an SSS map is constructed by mapping all points on the SSS image to a geographic coordinate system using Equations (1) and (2). Finally, the 2D map result is obtained by eliminating the Z coordinate of $P_r$.

As shown in the 2D mapping result in Figure 5d, a gap appears in the center of the image due to the inability of SSS to effectively image the area directly beneath it. This gap can be eliminated by fusing FLS and SSS to build a map.

3.2. Grayscale Equalization and Geocoding for FLS

From the typical FLS image, it can be observed that the left half and the edges of the image predominantly consist of noise and do not represent effective seabed features. In contrast, the central portion of the image provides clear and effective imaging of the seabed. Therefore, to perform geocoding on FLS images, the effective imaging area must first be extracted. This extracted area is then geocoded to generate the FLS map.

The detailed process for this method is illustrated in Figure 7.

Figure 7 illustrates the FLS image processing and mapping method proposed in this paper. The steps are as follows:

• The original FLS image is filtered using the mean filtering method.
• The original FLS image is normalized by the filtered image to create a grayscale-balanced image, which is then mapped to the pixel space [0, 255] using the brightness parameter, $\kappa$.
• The brightness parameter, $\kappa$, and the approximate erosion method are applied to extract the effective imaging area from the FLS image.
• The effective imaging area is combined with the normalized image, and sonar navigation data are used to geocode the image, resulting in the final FLS mapping output.

A critical parameter in this process is the filter size, $M^{FLS}$, of the mean filtering method. The experimental section will include a detailed ablation study to analyze the impact of this parameter on the results.

This paper proposes an approximate erosion method to extract the effective area of the FLS image. The calculation and implementation of this method are detailed in Figure 8.

The operation of the approaching erosion method is as follows: First, we set a binary image with the same size as the image (corresponding to Figure 8b) and set all elements to 1. The pixel in the binary image is defined as ${Pe}_{(x,y)}$. The pixel in the FLS image is defined as ${Ps}_{(x,y)}$. Then, we set $x_{up}=0$ and $x_{down}=W_{width}$. For each row on the image (its vertical coordinate is $y$), the subsequent calculation process is as follows:

Method: approaching erosion method

For each $y$:

While:

$x_{up}=x_{up}+1$

If $\frac{\sum {Ps}_{(x_{up}:x_{up}+10, y)}}{10}<\kappa$:

${Pe}_{(x_{up},y)}=0$

Else:

Break

While:

$x_{down}=x_{down}-1$

If ${\displaystyle \frac{\sum {Ps}_{(x_{down}:x_{down}-10, y)}}{10}}<\kappa$:

${Pe}_{(x_{down},y)}=0$

Else:

Break

One of the results derived from the approaching erosion method is shown in Figure 8b, demonstrating its effectiveness in accurately identifying the effective imaging area in FLS images. To evaluate the brightness parameter, $\kappa$, used in this method, detailed ablation experiments will be conducted in the experimental section.

When a vehicle equipped with an FLS scans the seabed, its navigation system provides essential data, including attitude, positioning, and seabed height. Based on the flat seabed assumption, the geocoding of the FLS image can be calculated. The coordinate relationships involved in the FLS imaging process are illustrated in Figure 9.

The coordinate system of FLS (FLSCS) in Figure 9 is defined as follows: it is fixed to the FLS device, and its origin is at the center of the sonar transmitter. Its X-axis points forward from the sonar, the Y-axis points to the right of the sonar, and the Z-axis points downward from the sonar.

For a pixel point, $P_{pixel}={\left[x_{pixel},y_{pixel}\right]}^T$, on an FLS image in Figure 9, its corresponding 3D coordinate in the seabed is $P_n={\left[X_n,Y_n,Z_n\right]}^T$. The rotation matrix, $R_{FLS}$, and a displacement vector, $P_{FLS}={\left[X_{FLS},Y_{FLS},Z_{FLS}\right]}^T$, of FLS is provided by the navigation system. For the pixel points in the effective area of FLS image, the corresponding azimuth, ${\theta }_n$, and range, $r_n$, can be calculated from the coordinates of this pixel point. The calculation method is as follows:

$$\left\{\begin{array}{c}{r}_n={\displaystyle \frac{x_{pixel}D}{W_{width}}}\\ {\theta }_n={\displaystyle \frac{y_{pixel}\alpha }{H_{height}}}\end{array}\right.$$

(3)

where $D$ is the detection range of the FLS. α is the beam opening angle of the FLS, which can be obtained by consulting the sonar manual. $W_{width}$ and $H_{height}$ are the width and height of the FLS image. After calculating the range, $r_n$, and azimuth, ${\theta }_{kn}$, corresponding to the pixel point, the beam surface equation of the spatial point corresponding to the pixel point can be constructed. Then, the normal vector, $V_{kn}={[{Vx}_n,{Vy}_n,{Vz}_n]}^T$, of the beam surface where the spatial point is located can be expressed as follows:

$$\left[\begin{array}{c}{Vx}_n\\ {Vy}_n\\ {Vz}_n\end{array}\right]=R_k\left[\begin{array}{c}cos{\theta }_n\\ sin{\theta }_n\\ 0\end{array}\right]$$

(4)

According to the 3D plane calculation method, the beam surface where the spatial point corresponding to the pixel point is located can be expressed as follows:

$${Vx}_n\left(X_n-X_{FLS}\right)+{Vy}_n\left(Y_n-Y_{FLS}\right)+{Vz}_n\left(Z_n-Z_{FLS}\right)=0$$

(5)

At the same time, the distance constraint equation can be constructed based on the distance, $r_n$, of the pixels in the FLS image as follows:

$${\left(X_n-X_{FLS}\right)}^2+{\left(Y_n-Y_{FLS}\right)}^2+{\left(Z_n-Z_{FLS}\right)}^2={r_n}^2$$

(6)

When the seafloor is assumed to be flat, $Z_n$ in the above equations is equal to the seafloor height provided by the altimeter. Therefore, using Equations (5) and (6), the 3D spatial position corresponding to each pixel in the FLS image can be calculated. By mapping all the pixels within the effective imaging area of the equalized FLS image into 3D space, the geocoding process for the FLS is completed.

3.3. Sonar Fusion Mapping Method

After completing the grayscale equalization and mapping of the FLS and SSS, the two maps can be fused to complement each other. There are significant differences between FLS images and SSS images. This paper utilizes a normalization-based method to process the two sonar images to ensure that both maps have the same average grayscale. For the subsequent map fusion task, the filtering parameter, $M^{SSS}$, is set to 20, the filtering parameter, $M^{FLS}$, is set to 50, and the brightness parameter, $\kappa$, is set to 40. These settings are used to generate the maps for both sonars. The FLS map is then overlaid onto the SSS map, and the result is shown in Figure 10.

Figure 10 shows that there are still noticeable differences between the FLS and SSS maps after grayscale normalization. This is because the two sonar systems not only have different average grayscale values but also exhibit differences in the standard deviation of grayscale distribution. To visually demonstrate the differences in grayscale distribution, sampling is performed at the positions marked by the green lines in Figure 10. The 1D pixel sequences from the sampling are shown in Figure 11.

Figure 11 shows that the brightness variation in the FLS map is smaller than that in the SSS map, indicating that the standard deviation of the FLS map data in the overlapping region is smaller. We conducted a statistical analysis of the pixel histograms for both maps in the overlapping region, excluding data segments with pixel values of 0. The histogram is shown in Figure 11b.

From the histogram, it can be observed that the pixel distributions of both sonar maps are close to a Gaussian distribution, with the mean value corresponding to the brightness parameter, $\kappa$. Therefore, the standard deviation of the two datasets can be used to measure their similarity. The standard deviation for these SSS data is 21.27, and the standard deviation for these FLS data is 9.18.

When the grayscale distributions of the two sonar maps are more similar, the fusion map appears smoother. Thus, the standard deviation can be used to adjust the grayscale distribution of the FLS data to make the two maps more similar. In the map fusion process, a flat seabed region corresponding to overlapping data (e.g., the blue square in Figure 10) is selected, and the standard deviation for both sonar maps in that region is calculated. The grayscale distribution of the FLS data can be adjusted using the following method:

$${Pixel}_{fls}^{new}=\kappa +\left({Pixel}_{fls}^{old}-\kappa \right)\times {\displaystyle \frac{{Std}_{sss}}{{Std}_{fls}}}$$

(7)

where ${Pixel}_{fls}^{old}$ represents the pixel values in the unadjusted FLS map. ${Pixel}_{fls}^{new}$ represents the pixel values in the adjusted FLS map. ${Std}_{sss}$ is the average standard deviation calculated from the SSS map data corresponding to the blue region in Figure 10, and ${Std}_{FLs}$ is the average standard deviation calculated from the FLS map data in the same blue region of Figure 10. The distributions of the adjusted SSS and FLS data are shown in Figure 12.

Figure 12 shows that the adjusted FLS data distribution is now closer to the SSS data distribution. The standard deviation of the adjusted FLS data is 20.8, while the standard deviation of the SSS data is 21.3. This indicates that the two sonar maps are now more similar, resulting in a more natural fusion of the maps. The adjusted fused map is shown in Figure 13.

Figure 13a shows that the grayscale differences in the FLS map become more pronounced after standard deviation adjustment. The object features in the new FLS map are clearer, making the fusion map region appear more defined and natural.

There are many optical image fusion methods that can be used in sonar map fusion, such as the fixed-weight fusion method and distance-weighted fusion method. Since this paper adjusts the grayscale distribution to make the two sonar maps more similar, the proposed fusion method fills the middle gap in the SSS map with FLS map data. This approach ensures that the excellent mapping results from the SSS map are preserved as much as possible. The fusion map obtained using our method is shown in Figure 13b. Additionally, the discussion section will present experimental results from different fusion methods.

4. Experiments

To evaluate the reliability and effectiveness of the proposed method, we conducted real-world scanning missions using a self-designed surface unmanned vessel. The unmanned vessel was equipped with two sonars (FLS and SSS), a GPS locator, a strapdown inertial navigation unit (IMU), and a seafloor altimeter. These instruments allowed simultaneous recording of navigation system outputs and sonar data. By aligning these datasets in time, tight coupling of the various data sources was achieved, enabling precise geocoding and fusion mapping. The unmanned vessel used in the experiments is depicted in Figure 14.

The FLS mounted on the unmanned vessel is an Oculus M750d, while the SSS is a Hydro ES900. The IMU and altimeter are tasked with collecting the carrier’s attitude and the seabed height, respectively. Additionally, the GPS provides high-precision real-time longitude and latitude information. These instruments collectively supply the necessary data to validate the effectiveness of the proposed method. The two types of sonar are shown in Figure 15, with their corresponding performance specifications listed in

Table 1. Performance of ES900 side scan sonar.

Parameters	Indicators	Parameters	Indicators
Manufacturer	Hydro-Tech (China)	Horizontal beamwidth	0.2°
Operating frequency	900 kHz	Horizontal resolution	1 cm
Range (maximum)	75 m	Vertical resolution	0.17 m
Vertical beamwidth	50°	Update rate (maximum)	25 Hz

and

Table 2. Performance of M750d forward-looking sonar.

Parameters	Indicators	Parameters	Indicators
Manufacturer	Blueprint (UK)	Beam separation	0.25°
Operating frequency	1.2 MHz	Horizontal aperture	130°
Range (maximum)	40 m	Vertical aperture	20°
Angular resolution	0.6°	Update rate (maximum)	40 Hz

.

The experiments were conducted in the Yuanyao Wharf area, located in Shandong Province, China. This setting provided a practical environment for testing and demonstrating the reliability of the fusion mapping approach.

After collecting real-world ocean data using the unmanned vessel, this section presents ablation experiments on the key parameters of the proposed method. The method includes three adjustable parameters, each of which significantly influences the fusion mapping results. By adjusting these parameters, the fusion mapping can be tailored to achieve optimal results.

In the sonar image grayscale normalization method, the filter parameters are closely related to the grayscale distribution of the sonar maps. Section 4.1 and Section 4.2 conduct ablation experiments on the filter parameters. Meanwhile, the brightness parameter, $\kappa$, is an important parameter that is related to the brightness of the fusion map. Section 4.3 conducts an ablation experiment on parameter $\kappa$. Finally, Section 4.4 demonstrates the practicality of the proposed method through a large-scale fusion mapping experiment.

4.1. Filter Parameter of FLS Image $M^{FLS}$

To ensure the fused map appears more seamless, it is crucial to harmonize the grayscale distributions of the two sonar maps. In Section 3.3, data from the positions indicated by the green line in Figure 10 were sampled to analyze the similarity in grayscale distributions between the FLS and SSS maps.

In this experiment, the same sampling method is applied to pixel data from the three lines depicted in Figure 16. This enables a comprehensive analysis of the grayscale similarity between the two sonar maps in the overlapping regions, offering further insights into the effectiveness of the proposed method in achieving seamless fusion.

In this ablation experiment, the filter parameter, $M^{SSS}$, of the image is set to 20, and the brightness parameter, $\kappa$, is set to 40. When the filter parameter for the FLS image is varied, single ping sampling is performed on Ping 1 in Figure 16, and the resulting data distributions are shown in Figure 17.

Figure 17 demonstrates that as $M^{FLS}$ decreases, the grayscale distribution of the FLS map becomes more uniform. However, this also leads to a smaller standard deviation, resulting in less clarity in object imaging. As the filter size increases, the grayscale distribution of the FLS map becomes more uneven. The brightness of the FLS data at the center becomes significantly higher than that of the FLS data at the edges, and it also exceeds the brightness of the SSS data. This results in brightness stratification in the overlapping region of the fused map. The standard deviations of these FLS data are summarized in

Table 3. Standard deviation of FLS mapping data with different $M^{FLS}$.

${\mathit{M}}^{\mathit{F}\mathit{L}\mathit{S}}$	Standard Deviations
	Ping 1	Ping 2	Ping 3	Average Value
10	7.81	8.12	7.93	8.04
50	9.18	9.03	9.21	9.19
100	11.01	11.55	11.92	11.47
200	16.36	15.93	16.21	16.41

.

As presented in Table 3, as the filter size increases, the standard deviation of the FLS map data also increases. However, both excessively large and small variances can lead to inconsistencies in the grayscale distributions of the FLS and SSS maps, ultimately degrading the fusion map quality. Based on the calculated average standard deviation, the FLS data are adjusted according to Equation (7). The FLS map is then directly overlaid onto the SSS map, and the resulting sonar map is presented in Figure 18.

Figure 18 demonstrates that when $M^{FLS}$ is set to 10, the FLS map appears the smoothest, but the features of the seabed objects become blurred. When $M^{FLS}$ is set to 100, noticeable brightness stratification appears between the FLS and SSS maps, with the center of the FLS map becoming brighter. When $M^{FLS}$ is set to 200, the brightness stratification between the FLS and SSS maps becomes more pronounced. When $M^{FLS}$ is set to 50, the grayscale distribution of the FLS map is closest to that of the SSS map, and the features of the objects remain clearly visible. Therefore, the filter parameter, $M^{FLS}$, is recommended to be set to 50 in this study.

4.2. Filter Parameter of SSS Image, $M^{SSS}$

The same analysis can be performed for the filter parameter of the SSS data. First, $M^{FLS}$ is set to 50, and the brightness parameter, $\kappa$, is set to 40. When $M^{SSS}$ is set to different values, single-ping sampling is performed at the Ping 1 position in Figure 16 for both map datasets, and the resulting data distributions are shown in Figure 19.

Figure 19 shows that as $M^{sss}$ decreases, the grayscale distribution of the SSS map becomes more uniform. However, this can also lead to a smaller standard deviation, resulting in less clear object imaging. As $M^{SSS}$ increases, the grayscale distribution of the SSS map becomes more uneven, with bright pixel regions appearing in the center of the SSS map. This causes larger pixel differences in the overlapping region between the SSS and FLS maps. The standard deviations of these SSS map data are summarized in

Table 4. Standard deviation of FLS mapping data with different $M^{SSS}$.

${\mathit{M}}^{\mathit{S}\mathit{S}\mathit{S}}$	Standard Deviations
	Ping 1	Ping 2	Ping 3	Average Value
5	18.36	18.14	17.75	18.15
20	20.83	21.15	20.66	20.91
40	22.63	22.45	22.94	22.46
60	23.52	23.99	24.12	23.78

.

Table 4 shows that as $M^{SSS}$ increases, the standard deviation of the SSS map data also increases. However, excessively large or small standard deviations can cause significant differences in the grayscale distributions of the FLS and SSS maps, leading to poor fusion map quality. Fusion maps generated by different $M^{SSS}$ are shown in Figure 20.

Figure 20 shows that when $M^{SSS}$ is small, the grayscale distribution in the fusion map region is smooth, but the object imaging in the fusion map is unclear. When $M^{SSS}$ is large, the object imaging in the SSS map becomes clear, but significant differences appear in the fusion region, resulting in poor fusion map quality. Therefore, this paper recommends setting $M^{SSS}$ to 20, as this value ensures clear object imaging while maintaining a smooth grayscale distribution in the fusion region.

4.3. Brightness Parameter, $\kappa$

The brightness parameter, $\kappa$, is a critical factor in the proposed method. It simultaneously regulates the brightness of the fusion maps and the extraction capability of the effective area of the FLS. Therefore, the ablation experiment in this section explores various fusion mapping results by adjusting the brightness parameter, $\kappa$. The experimental results of the ablation experiment on $\kappa$ are presented in Figure 21.

It can be seen from Figure 21 that the brightness parameter, $\kappa$, has a significant impact on controlling image brightness. At the same time, it can affect the details of the fusion mapping effect of FLS and SSS. When $\kappa$ is small, the image brightness is low, and the effective area extracted from the FLS image is larger. However, this can lead to noise in the fusion area, reducing the clarity of features. When $\kappa$ is large, the image brightness is high, and the effective area extracted from the FLS image is smaller. This can result in greater differences in the fusion area, potentially affecting the seamless integration of FLS and SSS data. Thus, the brightness parameter, $\kappa$, should be carefully adjusted based on specific application requirements to optimize the fusion results.

4.4. Large-Scale Fusion Mapping Experiment

To further demonstrate the applicability of the proposed method, fusion experiments are conducted using sonar data from two large-scale scanning areas. The experimental parameters are set as $M^{SSS}=20$, $M^{FLS}=50$, and $\kappa =40$.

To highlight the advantages of the proposed fusion method, this study compares the original SSS mapping results with the fusion mapping results. The comparison clearly illustrates the improvements achieved by the fusion approach. The visual results of this comparison are presented in Figure 22.

The data shown in Figure 22 cover a distance of more than 800 m, demonstrating that the proposed method maintains high stability and reliability in large-scale ocean scanning tasks. The zoomed-in view highlights targets that were not detected in the original SSS map but are clearly visible in the fusion map.

To verify the stability of the proposed method under different marine conditions, a seabed scanning experiment is conducted using an unmanned vessel and sonar equipment in Sanya, China. The results of the fused mapping using the collected sonar data are shown in Figure 23. Figure 23 demonstrates the effectiveness of the FLS data in filling gaps in the SSS mapping, enabling imaging of the seabed directly beneath the sonar. In the fused mapping results, objects that could not be detected by the SSS mapping are revealed.

These results illustrate that the proposed method effectively resolves the blind spot directly below the SSS scan. Furthermore, the range of the fusion mapping significantly exceeds that achievable with FLS mapping alone. By combining the strengths of both sonar systems, the fusion mapping method demonstrates substantial practical value, providing a comprehensive and enhanced seabed mapping solution.

5. Discussions

This paper introduces a framework for FLS and SSS map fusion, with experiments demonstrating the effects of different parameters on the fusion map. The mean filtering method was employed as the primary filtering technique, and the sonar fusion method involved filling gaps in the SSS map with data from the FLS map. However, alternative image filtering and fusion methods are available, which may offer varying degrees of effectiveness. This section explores these alternative approaches and their impact on the fusion map results.

5.1. Different Sonar Image Filtering Methods

Sonar image filtering is a critical step in the proposed framework, as it directly influences the quality and clarity of the resulting fusion map. The selection of an appropriate filtering technique is, therefore, a key consideration. Several classical image filtering methods exist, including bilateral filtering, mean filtering, median filtering, and Gaussian filtering. Each method has unique properties and applications in image enhancement and noise reduction. In this section, we evaluate the effects of different filtering methods on the fusion map. The results of the fusion maps generated using these alternative filtering techniques are presented in Figure 24.

As shown in Figure 24, the fusion map quality varies significantly depending on the filtering method: The fusion map based on bilateral filtering performs poorly because this method preserves edge features, which are subsequently suppressed by the normalization operation used in the proposed method. This method yields better results but introduces highlighted regions along the edges of the fusion area, which disrupt the overall uniformity. The fusion map generated with Gaussian filtering suffers from unclear features and poor overall fusion quality. The fusion map based on mean filtering, as used in this paper, provides the best results, achieving a balanced and high-quality fusion map.

5.2. Comparative Experiment of Different Fusion Methods

There are many pixel-level image fusion methods, such as fixed-weight fusion, maximum-value fusion, and distance-weighted fusion methods. Since this study focuses on sonar fusion mapping, where image fusion occurs dynamically during the continuous generation of mapping data, global image-based fusion methods are unsuitable.

The proposed method first applies a grayscale adjustment to harmonize the grayscale distributions of the two maps. Subsequently, the gaps in the SSS map are filled with data from the FLS map. This approach maximizes the preservation of effective features from both sonar systems.

In this experiment, the weight for the fixed-weight fusion method is set to 1/2. Pixel fusion in the overlapping area is achieved based on this weight.

The distance-weighted fusion method uses the centerline of the SSS map as the starting point. The farther the FLS pixel is from the centerline, the lower its fusion weight. The pixel closest to the centerline has a fusion weight of 1, and the pixel farthest away has a fusion weight of 0. Conversely, the farther the SSS pixel is from the centerline, the higher its fusion weight. The pixel closest to the centerline has a fusion weight of 0, and the farthest pixel has a fusion weight of 1. Pixel fusion in the overlapping area is achieved based on this weight.

The maximum-value fusion method achieves fusion mapping by retaining the maximum value of pixels in the overlapping area. This method can highlight the features of the fusion area. The fusion maps generated by different fusion methods are shown in Figure 25.

Figure 25 shows that the fusion map generated by the fixed-weight fusion method has a smooth grayscale distribution in the fusion region, but the features of the SSS map are blurred. Similarly, the fusion map generated by the distance-weighted fusion method also has a smooth grayscale distribution in the fusion region, but it also causes the SSS map features to be blurred, which may not be advantageous for subsequent manual observation and analysis.

The fusion map generated by the maximum-value fusion method exhibits noticeable stratification effects, resulting in poor fusion quality. In contrast, the fusion map generated by the proposed method has a smooth grayscale distribution in the fusion region while maintaining clear imaging of seabed objects, achieving the best fusion results.

To provide a comprehensive evaluation of the fusion map quality across different methods, four no-reference image quality assessment metrics are utilized:

• Image Standard Deviation: Measures the contrast of the image.
• Image Entropy: Reflects the amount of information and detail in the image.
• Average Gradient: Evaluates the sharpness and clarity of the image.
• Spatial Frequency Response (SFR): an important indicator for measuring image clarity, especially in edge and detail areas.

The greater the above four evaluation indicators, the higher the image quality. The evaluation metrics for the different fusion methods are summarized in

Table 5. Evaluation of Fusion Map Results Using Different Fusion Methods.

Methods	Standard Deviation	Entropy	SFR	Mean Gradient
Fixed-weight fusion	28.925	6.061	41.18	100.87
Distance-weighted fusion	28.631	6.061	41.14	102.285
Maximum-value fusion	30.187	6.133	42.32	104.743
This study	30.484	6.136	44.27	109.34

, highlighting the superior performance of the proposed method in terms of these metrics.

It can be observed from Table 5 that the evaluation metrics for fusion maps generated by different methods are relatively similar. This is primarily because the SSS map occupies the majority of the fusion map. However, the method proposed in this paper consistently achieves the best performance across all evaluation metrics, demonstrating its superiority and effectiveness.

5.3. Future Work

The proposed method first performs grayscale equalization on SSS images and FLS images, followed by geographic encoding of the sonar images using positioning data from the vehicle’s navigation system. During the fusion mapping process, the quality of the positioning data will affect the final mapping results. For instance, it may cause misalignment between the two sonar maps or lead to distortion in the sonar images. Therefore, it is crucial to use high-precision navigation and positioning equipment for data collection during the task.

In the practical installation process, both the sonars and the navigation system are large in size, making it impossible to install them at the same location. As a result, the relative position between the center of the sonar transducer arrays and the navigation system’s positioning center needs to be measured, and this relative position is used to compensate for positional errors in the sonar mapping results. Additionally, the orientations of the two sonar sensors cannot be perfectly aligned, and it is difficult to accurately measure the orientation error. To address these issues, there are two recommended solutions: (1) minimizing the distance between the sonar sensors; (2) building a high-precision mounting platform to improve the consistency of the sonar sensor alignment. These measures can help reduce misalignment in fusion mapping caused by installation errors.

Both SSS and FLS generate large volumes of data, and their scanning frequencies are relatively high. For example, the ES900 SSS and M750d FLS can transmit sound waves at frequencies greater than 20 Hz. Therefore, the proposed method used currently relies on post-processing for computation. The geographic encoding step of this method is the most time-consuming. However, each pixel of the sonar image is encoded individually during the computation. As a result, GPU programming can be introduced for parallel processing, enabling real-time fusion mapping in a real-time operating system on board a ship.

6. Conclusions

This paper introduces a novel method for fusion mapping of FLS and SSS. Initially, the grayscale values of the two sonar images are equalized using an image equalization method based on normalization. Subsequently, the two sonar images are geocoded by integrating the navigation data. Next, the grayscale distribution of the FLS map is adjusted based on the standard deviation of the two map results, ensuring consistency in their grayscale distributions. Finally, the FLS map data are merged into the gaps of the SSS map, achieving seamless fusion mapping.

This study is the first to explore FLS and SSS fusion mapping. By leveraging the advantages of FLS and SSS while mitigating their individual limitations, the proposed method delivers superior ocean mapping results. It ensures full coverage and a large operational range in a single ocean scanning task, significantly improving the efficiency of seabed exploration. As a result, the proposed method has substantial practical value for oceanographic studies and related applications.

The main contributions of this work are as follows:

• For the first time, this study proposes a framework for the fusion mapping of FLS and SSS data. The framework achieves high-quality seabed mapping and enhances the efficiency of ocean scanning tasks.
• A sonar image grayscale equalization method based on normalization is introduced. This method is applied to both FLS and SSS images, ensuring consistent grayscale levels across the two datasets and enabling smooth grayscale transitions in the fused map.
• This paper proposes a novel method for extracting effective areas from FLS images using an approximate erosion technique. Subsequently, geocoding the pixels within the extracted effective areas yields superior FLS mapping results.
• An analysis of the grayscale distribution of FLS and SSS map data is performed, and a method is proposed to adjust the grayscale distribution of the FLS map based on standard deviations. This approach enhances the clarity of object imaging in the overlapping regions of the fused map.