Inverse Synthetic Aperture Radar Image Multi-Modal Zero-Shot Learning Based on the Scattering Center Model and Neighbor-Adapted Locally Linear Embedding

Abstract

Inverse Synthetic Aperture Radar (ISAR) images are extensively used in Radar Automatic Target Recognition (RATR) for non-cooperative targets. However, acquiring training samples for all target categories is challenging. Recognizing target classes without training samples is called Zero-Shot Learning (ZSL). When ZSL involves multiple modalities, it becomes Multi-modal Zero-Shot Learning (MZSL). To achieve MZSL, a framework is proposed for generating ISAR images with optical image aiding. The process begins by extracting edges from optical images to capture the structure of ship targets. These extracted edges are used to estimate the potential locations of the target’s scattering centers. Using the Geometric Theory of Diffraction (GTD)-based scattering center model, the edges’ ISAR images are generated from the scattering centers. Next, a mapping is established between the edges’ ISAR images and the actual ISAR images. Neighbor-Adapted Local Linear Embedding (NALLE) generates pseudo-ISAR images for the unseen classes by combining the edges’ ISAR images with the actual ISAR images from the seen classes. Finally, these pseudo-ISAR images serve as training samples, enabling the recognition of test samples. In contrast to the network-based approaches, this method requires only a limited number of training samples. Experiments based on simulated and measured data validate the effectiveness.

1. Introduction

In the field of Remote Sensing, the Inverse Synthetic Aperture Radar (ISAR) has become a key technology for Radar Automatic Target Recognition (RATR) [1,2]. ISAR enables high-resolution imaging of moving targets over long distances, in all weather conditions. This capability allows ISAR to effectively detect and recognize ship targets in marine environments [3].

However, in ISAR applications, the targets are usually non-cooperative, resulting in a lack of training data for certain classes. A class that has only descriptions but no training data is termed an unseen class. The recognition that includes unseen classes is known as Zero-Shot Learning (ZSL), which identifies the unseen classes by acquiring knowledge from seen classes [4,5]. In most cases, the samples from seen and unseen classes are distinct [6]. When ZSL incorporates multiple modalities to provide knowledge for unseen classes, it is referred to as Multi-modal Zero-Shot Learning (MZSL) [7]. In MZSL, the principal modality for unseen classes is unavailable, so knowledge is obtained from a secondary modality and the seen classes. To better utilize the knowledge from seen class samples, relevance needs to be built between the seen and unseen classes. These methods typically fall into two categories: embedding-based methods [8,9,10,11] and instance-based methods [6,12,13]. For instance-based methods, a common approach is to synthesize pseudo instances for unseen classes, referred to as the synthesizing method. The synthesis of pseudo instances requires auxiliary information. In other words, most instance-based methods rely on spaces defined by auxiliary information, such as feature space, attribute space, and semantic space, to recognize unseen samples [14,15]. Specifically, the synthetic images are commonly generated using semantic information and optical images, with popular methods including the Generative Adversarial Network (GAN) [16,17,18] and the Auto-Encoder (AE) [19,20,21,22]. The deep unfolding networks are also used for multi-modal image generation [23,24,25]. When the pseudo instances of unseen classes resemble real samples, the classifier trained on these instances becomes skilled at recognizing the test samples. In the end, the recognition of unseen classes is assisted by semantic information. Among these methods, the network-based methods demand more training time and complexity, while the AE-based ones are faster [26]. However, the AE-based approaches always require an additional attribute matrix [21]. Furthermore, synthetic images may still contain errors.

Taking inspiration from MZSL, the principal modality is ISAR images, and pseudo-ISAR images can be generated using secondary modality auxiliary information. Some methods incorporate the auxiliary information for ISAR image recognition, particularly in scenarios where the number of training samples is limited, and certain classes are unseen. For instance, Ma et al. [27] employ electromagnetic data based on the target’s 3D models to enhance accuracy. Nevertheless, acquiring the electromagnetic data generated from 3D models is complex, and ISAR image datasets are often small. As a result, the network-based methods suffer from overfitting. This leads to pseudo instances with distorted structures or non-existent scatterers, ultimately reducing accuracy. Hence, this paper explores the method without utilizing networks.

ISAR images are significantly influenced by the ship’s structure, which can be derived from optical images. Therefore, we chose optical images as the secondary modality to predict the ship’s structure and generate the pseudo-ISAR images. This choice was also because the optical images are more accessible than the 3D models and provide richer information than semantic information [9]. Consequently, optical images can generate pseudo instances both conveniently and precisely.

The similarity between optical and ISAR images contributes to the generation process. The stronger scatterers usually correspond to sharp parts, whereas the weaker scatterers are associated with flat surfaces or shadows, reflecting the underlying structure. This structural information can also be observed in optical images. However, differences also exist between optical images and ISAR images. For instance, the color of optical images is unrelated to scatterers and ISAR images. Since color might disrupt the recognition, images without color but containing structural details are needed. These images can then assist in the ISAR image MZSL.

To establish a connection between the structure and ISAR images, several models have been proposed to find solutions for practical geometries, such as the damped exponential model [28], GTD-based scattering center model [29], and Attribute Scattering Center model [30]. In the case of ISAR, the ship target is in the high-frequency optical region. The ISAR echo is the sum of echoes from multiple simple scattering centers [31]. For instance, the missile target resembling a cone could be represented by three scattering centers [32,33]. Similarly, a ship target could be described by several scattering centers with different parameters, which are determined by the structure of the ship [34]. Inspired by these methods, we employ optical images to extract the structure, analyze the scattering centers, and ultimately synthesize the pseudo-ISAR images to assist in ISAR image MZSL. However, a major challenge remains: utilizing the structure of optical images to estimate the parameters of scattering centers. To address this, we propose Neighbor-Adapted LLE (NALLE), an adaptation of the Locally Linear Embedding (LLE) method [35], with the assumption that similar ship structures correspond to similar scattering centers in ISAR images [34]. Overall, we propose a framework for ISAR image MZSL based on the scattering center model and NALLE.

This paper presents an ISAR image MZSL method, including the following steps: First, detect the edges of the optical images to extract the structure of targets and reduce the impact of color. Then, assuming that all edges are potential scattering centers, generate the ISAR images of the contour using the GTD-based scattering centers model. Subsequently, based on the training data of seen classes, establish a mapping between the ISAR images of edges and the real ISAR images. Next, with this mapping and the ISAR images of edges from unseen classes, synthesize the pseudo-ISAR images through NALLE. Finally, use the pseudo-ISAR images as the training samples for the unseen classes, completing the ISAR image MZSL process.

The contributions are summarized as follows:

• We extract the structure of targets from optical images and reduce the influence of color. Based on this structure, we use the GTD-based scattering centers model to generate the ISAR images of edges.
• By employing NALLE, we modify the factors affecting the amplitude of the scattering centers, which are ignored in the edges’ ISAR image. This adjustment makes the pseudo-ISAR images more similar to the real ISAR images, improving the accuracy of MZSL.
• We introduce a framework for generating ISAR images with optical images aiding in achieving MZSL. This framework avoids network-based methods, ensuring interpretability. The algorithms within this framework are changeable and highly flexible.

The paper is organized as follows: Section 2 introduces the signal model of ISAR. Section 3 presents the framework for ISAR image generation based on the scattering center model and NALLE. Section 4 covers the experimental results and discusses the findings. Finally, Section 5 draws the conclusion.

2. ISAR Signal Model

In this section, we briefly introduce the ISAR signal model. When a radar emits a Linear Frequency Modulated signal, the received signal from an ideal point target is represented as [36]

$$s(\widehat{t},t_m)=A·\mathrm{rect}\left({\displaystyle \frac{\widehat{t}-2R\left(t_m\right)/c}{T_p}}\right)·exp\left(j\pi \gamma {\left(\widehat{t}-{\displaystyle \frac{2R\left(t_m\right)}{c}}\right)}^2\right)·exp\left(j2\pi f_c\left(\widehat{t}-{\displaystyle \frac{2R\left(t_m\right)}{c}}\right)\right),$$

(1)

where A is the scattering coefficient, $\widehat{t}$ is the fast time, $t_m$ is the slow time, c is the speed of light, $R\left(t_m\right)$ is the distance between the radar and the target, $T_p$ is the pulse duration, $f_c$ is the carrier frequency, $\gamma$ is the chirp rate, and $\mathrm{rect}\left(u\right)=\left\{\begin{array}{cc}1,& \left|u\right|\le 1/2\\ 0,& \left|u\right|>1/2\end{array}\right.$ is the range envelope function.

The target can be regarded as the aggregation of multiple scattering centers for high-resolution radar systems. The echoes can be represented as

$$\begin{array}{c}\hfill s(\widehat{t},t_m)=\sum _i{A}_i·\mathrm{rect}\left({\displaystyle \frac{\widehat{t}-2R\left(t_m\right)/c}{T_p}}\right)·exp\left(j\pi \gamma {\left(\widehat{t}-{\displaystyle \frac{2R\left(t_m\right)}{c}}\right)}^2\right)·exp\left(j2\pi f_c\left(\widehat{t}-{\displaystyle \frac{2R\left(t_m\right)}{c}}\right)\right),\end{array}$$

(2)

where $A_i$ is the scattering coefficient for the i-th scattering center.

After de-chirping and removing the Residual Video Phase (RVP) [37], the result in the Range-Doppler domain can be expressed as [38]

$$\begin{array}{c}\hfill S_{\mathrm{RD}}(f_r,f_a)=\sum _i{A}_i·T_p·T_m·\mathrm{sinc}\left(T_p(f_r-2x_i\gamma /c)\right)·\mathrm{sinc}\left(T_m(f_a-f_d)\right),\end{array}$$

(3)

where $f_r$ is the frequency in the fast-time domain, $f_a$ is the frequency in the slow-time domain, $T_p$ is the pulse duration, $T_m$ is the coherent time, $x_i$ is the distance of the i-th scattering center, and $f_d$ is the Doppler frequency of the scattering center. Therefore, the position of each scattering center can be solved using Equation (3).

3. Proposed Method

In this section, we introduce the proposed algorithm, ISAR image MZSL based on the scattering center model and NALLE.

3.1. The Framework of the Algorithm

This paper proposes a framework that utilizes optical images for ISAR image MZSL. This framework is divided into four main parts. The first part is preprocessing. The second part focuses on generating edges’ ISAR images with optical images, and the third part creates pseudo-ISAR images using NALLE. The last part involves the feature extraction, training the classifier with the pseudo-ISAR images, and classifying the test samples.

The primary flowchart is illustrated in Figure 1. The preprocessing steps follow the process described in our previous work [36]. First, the feature points are located. Next, the coefficients for the affine transform are calculated. Finally, both optical and ISAR images are reconstructed using the affine transform, resulting in the reconstructed images where the components are generally positioned in specific areas. For optical image preprocessing, it is essential to set the background to zero, different from the approach for ISAR images. The remaining steps for optical image preprocessing are consistent with those for ISAR images.

After the preprocessing, aiming to acquire the structure of the target from the optical images, the edges of the target are detected using the Canny operator. Subsequently, the scattering center model is applied, treating the extracted structure as potential scattering centers, and the edges’ ISAR images are generated.

Then, the edges’ ISAR images and the real ISAR images are divided into patches, and the mapping is determined between the edges’ ISAR patches and the real ISAR images’ patches of seen classes. Next, NALLE is utilized to generate the pseudo-ISAR images using the optical images of unseen classes.

Finally, the classifier is trained using training data and the pseudo-ISAR images of unseen classes. The recognition results can be obtained.

We elaborate on each step of the procedure in the following sections.

3.2. Generate ISAR Images of Edges from Optical Images

As the ship target is in the high-frequency optical region, the influence among the scattering centers is limited [31]. Due to this limitation, the scattering centers of the target can be considered as independent components. The backscattering field is the sum of the fields from several simple scattering centers. For example, a missile target resembling a cone can be represented by three scattering centers [32]. Since the scattering centers are affected by the structure of the target, the structure becomes one of the major factors in determining the echoes.

This algorithm utilizes the ship’s edge as the potential scattering center and generates the ISAR echoes corresponding to these edges. This process employs scattering center models. Several well-known models exist, with the Attribute Scattering Center (ASC) model [39] being widely recognized. The ASC model shows that the equivalent scattering centers are predominantly distributed at the edge of the target [34,40,41]. Inspired by this, we mainly extract the edge structure as the location for potential scattering centers.

However, in previous studies, the 3D models are used to estimate the parameters in the ASC model. It is challenging to estimate some of these parameters from optical images, especially the orientation angle of the scattering centers. As a compromise, we disregard the influence of the orientation angle, adopting the GTD-based scattering center model [32]. Then, we obtain

$$\begin{array}{cc}\hfill E(f,\phi )=& \sum _i{A}_i·{\left(j{\displaystyle \frac{f}{f_c}}\right)}^{{\alpha }_i}·\exp \left(j{\displaystyle \frac{4\pi f}{c}}(x_{i}cos\phi +y_{i}sin\phi )\right),\hfill \end{array}$$

(4)

where f is the frequency, $f_c$ is the center frequency, $A_i$ is the amplitude of the i-th scattering center, c is the speed of light, $\phi$ is the azimuth angle, $x_i$ and $y_i$ are the coordinates describing the location of the scattering center in the range and Doppler-range, respectively. The typical values for the parameter ${\alpha }_i$ with different scatting geometry are presented in

Table 1. Parameters ${\alpha }_i$ for Discriminate Scattering Geometries.

${\mathit{\alpha }}_{\mathit{i}}$	Scattering Geometries
1	Flat plate; Corner reflector
1/2	Singly curved surface
0	Point scatterer; Doubly curved surface, Straight edge specular
−1/2	Curved edge diffraction
−1	Cone diffraction

. Subsequently, the echoes are estimated, and the Range-Doppler algorithm is adopted to generate the ISAR images.

However, the edges extracted from the optical images cannot be directly considered as the potential scattering centers to generate the pseudo-ISAR images. An optical image is provided to illustrate this problem in Figure 2a. To calculate the edge of the structure, the Canny operator [42] is utilized. But the color in the image creates false edges, as shown in Figure 2b. These false edges introduce errors in estimating the scattering centers, resulting in a distorted structure in the generated ISAR images. To address this, the unreliable edges within the target are discarded, reducing the impact of the false edges caused by lighting and color variations. Nonetheless, this approach also removes the scattering centers associated with the structures. Additionally, this simple model struggles to reflect the coupling, the shadowing, and other complex scattering phenomena [43]. These issues are addressed in the following steps.

3.3. Generate Pseudo-ISAR Images with NALLE

The LLE algorithm relies on simple geometric intuitions. Suppose we have N points in a D-dimensional space, lying on or near a locally linear patch of a manifold. If the smooth manifold is well-sampled, each data point can be reconstructed from its neighbors using linear coefficients. Assume that other points lie on or near a smooth nonlinear manifold of d-dimensions, with a linear mapping from the D-dimensional to the d-dimensional space ($d<D$ and typically $d\ll D$). By design, the reconstruction linear coefficients are invariant to these transformations. This means that the same linear coefficients effective in D-dimensions should also be effective in reconstructing data in d-dimensions.

The LLE algorithm comprises three steps: First, find the K neighbors for a point in the D-dimensional space. Second, calculate the weight of the best reconstruction for the point with the K neighbors. Lastly, utilize this weight to reconstruct the point in the d-dimensional embedding space.

Nevertheless, obtaining well-sampled data for finding neighbors is challenging in ISAR images due to a limited number of samples. To address this, the images are divided into small patches. This division offers two advantages: it reduces dimensionality and increases the number of neighboring patches with similar structures. This transforms the problem from identifying similar ships to identifying similar structures, which makes it feasible to discover similar patches across various ships.

After the affine transform in preprocessing [44], the structures in optical images approximately align with the scattering centers in ISAR images. This leads to the optical image patches being similar to those in ISAR images at the same location. The mapping between the edges’ and real ISAR image patches can be established by matching corresponding locations.

For the seen classes, let $X_O$ denote the optical images and $X_I$ denote the ISAR images. For the unseen classes, let $Y_O$ denote the optical images and $Y_I$ denote the ISAR images. Each image is divided sequentially into patches of size $p\times q$, and each patch is transformed into a vector, written as $x_O^m,x_I^m,y_O^n,y_I^n\in {\mathbb{R}}^{1\ast pq}$, respectively, where $m=1,2,\dots ,M;n=1,2,\dots ,N$ represent the index number of optical and ISAR images, respectively.

After reconstructing all $y_I^n$ patches, one straightforward approach is to calculate the mean of adjacent patches to reconstruct the $Y_I$.

In the initial step of LLE, the K nearest neighbors are selected based on Euclidean distance. However, this distance metric has two limitations. After the aforementioned processing, numerous patches have small values, making them sensitive to noise. This leads to mismatches among image patches and reduces reconstruction similarity. Additionally, Euclidean distance fails to use prior knowledge: the targets in the optical and the ISAR images are aligned in the same position after preprocessing. The patches’ location could serve as a metric to address these drawbacks.

Hence, we propose a modified distance metric, termed Neighbor-Adapted, which combines Euclidean distance with a Gauss function as a factor to reflect the patches’ location. A lower value indicates high similarity and close location between the two patches in the images. For the i-th patch in the image with coordinates $(u_i,v_i)$, the weight with Gauss $W_{Gauss}$ is designed as

$$\begin{array}{cc}\hfill W_{Gauss}& ={\displaystyle \frac{T_{avg}}{G(u,v)}}=T_{avg}·exp\left({\displaystyle \frac{{\left((u-u_i)/u_{max}\right)}^2+{\left((v-v_i)/v_{max}\right)}^2}{2{\sigma }^2}}\right),\hfill \end{array}$$

(5)

where $\sigma$ is the variance, $G(u,v)$ is the Gauss function, $T_{avg}$ is the mean intensity value among all the patches, $(u,v)$ is the coordinate of the patch, $(u_i,v_i)$ are the coordinates of the i-th patch in the image, and $u_{max}$, $v_{max}$ are the maximum coordinates of the patch. The mean intensity value $T_{avg}$ is included to reduce the influence of the Euclidean distance term on the patches with strong intensity.

The second step in NALLE involves finding the K neighbors $x_O^m$ in $X_O$, represented as ${\mathbf{X}}_O^m$. To determine the weight $w_{mn}$ for each patch, minimizing the reconstruction error is essential:

$${\epsilon }^n\left(\omega \right)={∥y_O^n-\sum _m{w}_{mn}{x}_O^m∥}_2^2$$

(6)

where the weight $w_{mn}$ is subject to constraint $\left\{\begin{array}{cc}{w}_{mn}=0,& x_O^m\notin {\mathbf{X}}_O^m\\ {\displaystyle \sum _m}{w}_{mn}=1,& x_O^m\in {\mathbf{X}}_O^m.\end{array}\right.$

This forms a Constrained Least Squares Problem. By incorporating the constraint ${\displaystyle \sum _m}{w}_{mn}=1$ into Equation (6), we have

$${\epsilon }^n\left(\omega \right)={∥\sum _m{w}_{mn}\left(x_O^m-y_O^n\right)∥}_2^2={\mathbf{w}}_n^{\mathrm{T}}·{\mathbf{C}}_n·{\mathbf{w}}_n,$$

(7)

where ${\mathbf{w}}_n=[w_{1n},w_{2n},\dots ,w_{mn},\dots ,w_{Mn}]$, ${\mathbf{C}}_n$ is an $M\times M$ matrix, and the $(l,m)$-th element is given by $C_{lm}={(x_O^l-y_O^n)}^{\mathrm{T}}·(x_O^m-y_O^n).$

This problem can be solved by using the Lagrange multiplier method. The closed-form solution is [45]

$${\mathbf{w}}_n={\displaystyle \frac{{{\mathbf{C}}_n}^{-1}\mathbf{1}}{{\mathbf{1}}^{\mathrm{T}}{{\mathbf{C}}_n}^{-1}\mathbf{1}}}.$$

(8)

Then, the $x_I^m$ is found by mapping from ${\mathbf{X}}_O^m$ to ${\mathbf{X}}_I^m$. Finally, $y_I^n$ is reconstructed using the ${\mathbf{X}}_I^m$ and the weight ${\mathbf{w}}_m$, written as

$$y_I^n=∥\sum _n{w}_{mn}{x}_I^m∥.$$

(9)

After generating all $y_I^n$ patches, the pseudo-ISAR image $Y_I$ is synthesized by averaging adjacent patches based on $y_I^n$.

Figure 3 illustrates the generation of a patch $y_I^n$ in an ISAR image from a corresponding patch $y_O^n$ in the optical image. In this scenario, K is set to 3. In the left portion of the figure, $x_O^{m1}$ to $x_O^{m7}$ represent seven optical image patches for the seen classes. Specifically, $x_O^{m1}$ to $x_O^{m3}$ are the three nearest neighbors of optical patch $y_O^n$ for the unseen class. With the help of $x_O^{m1}$ to $x_O^{m3}$ and $y_O^n$, the matrix ${\mathbf{C}}_n$ is calculated with Equation (7). On the right side, $x_I^{m1}$ to $x_I^{m7}$ represent seven ISAR image patches for the seen classes, which corresponding to the optical image patches. Using the matrix ${\mathbf{C}}_n$, the weight ${\mathbf{w}}_n$ is calculated, and $y_I^n$ is synthesized using Equation (9). The steps of NALLE can be described as follows:

• Find the K nearest neighbors of $y_O^n$ in $X_O$, denoted as ${\mathbf{X}}_O^m:=[x_O^{m1},\dots ,x_O^{mK}]$.
• Reconstruct $y_O^n$ using these K neighbors. Calculate the matrix ${\mathbf{C}}_n$ to obtain the weight ${\mathbf{w}}_n$ that minimizes the reconstruction error ${\epsilon }^n\left(\omega \right)$.
• Locate the ${\mathbf{X}}_I^m$ utilizing the mapping with ${\mathbf{X}}_O^m$, and reconstruct the ISAR image patch $y_I^n$ by applying the weight $w_{mn}$ to the $x_I^m$ in ${\mathbf{X}}_I^m$.

3.4. MZSL with Pseudo-ISAR Images

As mentioned in the introduction, MZSL can be divided into embedding-based and instance-based categories. In this paper, we adopt an instance-based MZSL, which treats MZSL as a recognition problem with missing training data. In the previous section, pseudo-ISAR images are generated. In the MZSL framework, these images can be utilized as the samples for the unseen class. Since the pseudo-ISAR images are similar to the real ISAR images, the classifier can correctly recognize images, including those from the unseen class. After generating the pseudo-ISAR images, each optical image corresponds to a pseudo-ISAR image generated by NALLE. Since the optical images are located identically after preprocessing, the attitude of the pseudo-ISAR images remains consistent. As a result, these images can directly serve as training samples for the unseen classes without preprocessing.

To accomplish the MZSL, the final step is training the classifier with the generated samples. Before training the classifier, the feature extraction is necessary. For ISAR images, one classical feature is the superstructure curve for ISAR images, which is convenient to extract after preprocessing. However, this feature may struggle to reflect the intensity characteristics in ISAR images. Another typical feature, Principal Component Analysis (PCA), is adopted to capture the internal characteristics of the target and the intensity of the scattering centers. Without PCA, the superstructure curve fails to effectively represent the internal characteristics of pseudo-ISAR images. Therefore, the PCA-generated features are combined with the superstructure curve into a concatenated feature. This integration improves recognition accuracy by better reflecting image similarity. Due to the limited number of samples and the high dimensions, a Support Vector Machine is employed for classification [46].

In summary, for the training set, existing ISAR images are used as training samples for the seen classes. In contrast, the unseen classes employ the pseudo-ISAR images generated by the proposed method. All samples utilize a concatenated feature comprising the superstructure curve and the PCA-generated features. Finally, SVM is employed to classify the test samples.

4. Results

4.1. Experiments with Simulated Data

The simulated ISAR images are generated using the computational electromagnetics software with 3D models. The corresponding optical images are rendered based on these 3D models with color added in a similar attitude. The parameters for ISAR simulation are as follows: center frequency $f_c$ = 8.5 GHz, bandwidth B = 150 MHz, pulse interval stepping angle of 0.01 degrees. The elevation angle varies from 0 to 10 degrees, and the azimuth angle ranges from 2.5 to 17.5 degrees. Each ISAR image contains 1024 range points and 128 cross-range points.

Three different ship models are selected for optical and ISAR imaging. Each ship model generates 154 ISAR images and 308 optical images. The samples are shown in Figure 4, with each row representing Target 1, 2, and 3. Each column displays the 3D model, ISAR image, and optical image, respectively.

The training samples for seen classes include two types of ships, with each type having 10 ISAR images, totaling 20 ISAR images. The unseen class includes another type of ship that has no ISAR images as training samples. The training dataset also contains 10 optical images per target type, totaling 30 optical images. The images are divided into $16\times 16$ patches along both rows and columns.

We designate Target 3 as the unseen class, and the results of each step in the proposed method are shown in Figure 5. Figure 5a shows the optical image with an attitude similar to the ISAR image in Figure 5e. Figure 5b shows the target after preprocessing, and Figure 5c displays the ISAR image of edges generated by the scattering center models based on Figure 5b. Figure 5d is the pseudo-ISAR image synthesized by the NALLE algorithm. Comparing Figure 5b,e, the structures’ positions of the optical image align closely with the scattering centers of the ISAR image after the affine transform in the preprocessing. Geometric structures in the optical image, such as the ship island and masts, correspond with the scattering centers in the ISAR image. Comparing Figure 5c,e, the edges’ ISAR image is similar in shape to the ISAR image. However, amplitude differences are present; for instance, the bow appears strong in the optical image but weak in the ISAR image. After applying the NALLE (Figure 5d), the major scattering centers are similar in both images. However, certain areas, such as the low-intensity region in the middle of the ship, still differ from the ISAR image.

To validate the effectiveness of the proposed method, various methods are used for comparison. We initially explored employing network-based methods for comparative analysis. However, current SAR/ISAR MZSL methods predominantly rely on “large-scale” datasets for ISAR targets, with training samples ranging from hundreds to tens of thousands across studies (though this scale remains limited for neural networks). Despite the adaptability of deep learning frameworks (e.g., CNNs [47,48,49,50], and GANs [51,52,53,54]), significant challenges persist in handling extremely limited samples for some ISAR scenarios. In contrast, the traditional model-driven methods enable introducing the priors directly. Long et al. [55] achieved 3D reconstruction using the optical and ISAR images with precisely known target motion parameters. However, this condition might be infeasible for non-cooperative marine targets. This paper adopts the traditional machine learning methods, i.e., simulating the ISAR echoes with the GTD model, generating pseudo-ISAR images with NALLE, and recognizing the images using SVM. Thus, this paper employs methods without networks as contrast tests. Algorithm 1: Generate pseudo-ISAR images as training samples by using optical images directly as inputs for NALLE. Algorithm 2: Use ISAR images of edges as training samples. Algorithm 3: Generate the edges’ ISAR images from optical images containing all edges. Then, apply NALLE to synthesize the pseudo-ISAR images as training samples. Algorithm 4: Generate the pseudo-ISAR images as training samples by using the LLE method without location information. Supervised method: Use real ISAR images as training samples for the unseen class. Apart from the generation of pseudo-ISAR images, the supervised method follows the same steps as our approach, including feature extraction and classification.

The experiments for each algorithm use Target 3 as the unseen class. Training data include 10 optical images for each target type, totaling 30 optical images. Additionally, 10 ISAR images for each of Targets 1 and 2 constitute the seen classes, totaling 20 ISAR images. All algorithms use the same training dataset. All the experiments are repeated with various training datasets, and the average accuracy is then calculated. The results are presented in

Table 2. Accuracy with Target 3 as unseen class for simulated data (mean ± std).

Methods	Target 1	Target 2	Target 3 (Unseen Class)	Overall
Algorithm 1	0.936 ± 0.130	0.976 ± 0.021	0.731 ± 0.278	0.881 ± 0.082
Algorithm 2	0.985 ± 0.012	0.983 ± 0.020	0.432 ± 0.372	0.800 ± 0.118
Algorithm 3	0.979 ± 0.021	0.972 ± 0.032	0.779 ± 0.137	0.910 ± 0.037
Algorithm 4	0.973 ± 0.013	0.945 ± 0.032	0.831 ± 0.101	0.917 ± 0.031
Proposed method	0.984 ± 0.013	0.954 ± 0.025	0.896 ± 0.068	0.945 ± 0.023
Supervised	0.995 ± 0.006	0.984 ± 0.015	0.981 ± 0.009	0.987 ± 0.008

.

For quantitative evaluation, four metrics are adopted: Cross Entropy (CE) [56], Peak Signal-to-Noise Ratio (PSNR), Root Mean Square Error (RMSE) [57], and Structural Similarity Index (SSIM) [58] to verify the quantity performance.

Table 3. Quantitative results for simulated data. The “↑” indicate better performance with a higher metric, while the “↓” represent the opposite.

Methods	PSNR (dB)↑	RMSE↓	CE↓	SSIM↑
Algorithm 1	16.8	36.9	0.249	0.770
Algorithm 2	16.7	37.3	0.141	0.709
Algorithm 3	19.1	28.6	0.178	0.738
Algorithm 4	19.5	27.3	0.253	0.709
Proposed method	19.9	26.1	0.165	0.778

presents the simulated data results. The maximum PSNR is 19.9 dB for the proposed method. The proposed method produces the minimum RMSE value of 26.1. The proposed method achieves the second-lowest CE value (0.165), while the lowest CE value is 0.141 with Algorithm 2. Maximum SSIM (0.779) is obtained by our method.

This algorithm includes three key hyperparameters: the number of neighbors K and the patch size $p\times q$, where p represents the height of the patch and q represents its width. These hyperparameters influence the generation of pseudo-ISAR images, and the analysis of their values is provided later.

With all other parameters fixed, the range of K is set from 1 to 8. Recognition is performed using pseudo-ISAR images as training data, with accuracy as the evaluation metric. The experiment is repeated seven times independently, recording both the accuracy for the unseen class and the overall accuracy. The mean and standard deviation of the accuracy are calculated. Mean accuracy is displayed in Figure 6a, and standard deviation in Figure 6b.

Similar to the selection of K, all other parameters are fixed, and the range of patch heights p is set to {4, 8, 16, 32}, while patch widths q are set to {8, 16, 32, 64}. We repeat the experiment seven times and compute the mean and standard deviation of the accuracy for both the unseen class and all targets. The mean accuracy for the unseen class is presented in Figure 7a, while the mean accuracy for all targets is shown in Figure 7b. Standard deviation of accuracy is depicted in Figure 7c,d for the unseen class and all targets, respectively. To enhance clarity, the axes for p and q in Figure 7a,b are displayed in descending order.

4.2. Experiments with Measured Data

The measured data are obtained with an X-band radar, comprising 260 ISAR images of three ships. This dataset contains 66 ISAR images of a barque, 42 of a geared bulk carrier, and 152 of a gearless bulk carrier. The optical images, sourced from public websites, include 66 optical images of the barque, 42 of the geared bulk carrier, and 61 of the gearless bulk carrier. Sample images are shown in Figure 8. The barque and gearless bulk carrier both include three ISAR image sequences with different attitudes, and the geared bulk carrier includes two ISAR image sequences. The training dataset is similar to the simulated data.

Unlike the simulated data, the optical images in the measured data require background removal. The subsequent steps closely follow those for the simulated data. After preprocessing, the optical and ISAR images are aligned. An image of the gearless bulk carrier is chosen to demonstrate the proposed method in Figure 9. As shown in Figure 9b,e, the shapes are similar after preprocessing between the real optical and ISAR images. Nevertheless, the result in Figure 9c indicates that the amplitude mismatches persist when compared to the measured data. After applying the NALLE, in Figure 9d, the amplitude of the pseudo-ISAR image is similar to the real ISAR image while preserving its shape, benefiting the classification.

The gearless bulk carrier is selected as the unseen class for algorithm accuracy testing, as it has a larger sample size than other categories. Similar to the simulated data, six algorithms, including the proposed method and supervised classification, are utilized for classification. The results are presented in

Table 4. Accuracy with gearless bulk carrier as unseen class for measured data (mean ± std).

Methods	Barque	Geared Bulk Carrier	Gearless Bulk Carrier (Unseen Class)	Overall
Algorithm 1	0.985 ± 0.015	0.969 ± 0.023	0.719 ± 0.095	0.827 ± 0.060
Algorithm 2	0.961 ± 0.037	0.983 ± 0.012	0.632 ± 0.056	0.772 ± 0.032
Algorithm 3	0.981 ± 0.014	0.963 ± 0.019	0.737 ± 0.085	0.835 ± 0.054
Algorithm 4	0.989 ± 0.014	0.983 ± 0.012	0.801 ± 0.075	0.878 ± 0.041
Proposed method	0.987 ± 0.016	0.983 ± 0.012	0.852 ± 0.034	0.908 ± 0.017
Supervised	0.983 ± 0.016	0.983 ± 0.012	0.952 ± 0.014	0.965 ± 0.010

.

Following the simulated data analysis, the same four metrics (CE, PSNR, RMSE, and SSIM) are evaluated and presented in

Table 5. Quantitative results for measured data. The “↑” indicate better performance with a higher metric, while the “↓” represent the opposite.

Methods	PSNR (dB)↑	RMSE↓	CE↓	SSIM↑
Algorithm 1	18.1	32.0	0.216	0.766
Algorithm 2	18.1	31.9	0.192	0.757
Algorithm 3	19.6	27.0	0.157	0.749
Algorithm 4	19.8	26.3	0.167	0.736
Proposed method	19.8	26.3	0.105	0.767

for measured data. The proposed method achieves the maximum PSNR of 20.2 dB and the minimum RMSE of 25.0. Our method has the lowest CE value (0.105). It also achieves the highest SSIM (0.767).

Following the approach used for the simulation data, we analyze the hyperparameters K, p, and q for the measured data. Keeping other hyperparameters constant, the range of K is varied from 1 to 8. The experiments are repeated seven times, using recognition accuracy as the evaluation metric. The mean and standard deviation of accuracy are calculated for both the unseen class and all targets. The mean accuracy is presented in Figure 10a, while the standard deviation is shown in Figure 10b.

With K fixed, the range of patch height p is set to {4, 8, 16, 32}, while the range of patch width q is {8, 16, 32, 64}. Each experiment is repeated seven times. For each trial, the accuracies for both the unseen class and all targets are recorded, and the mean and standard deviation are calculated. The mean accuracy is displayed in Figure 11a,b for both the unseen class and all targets. The standard deviation of accuracy is illustrated in Figure 11c for the unseen class and in Figure 11d for all targets. Notably, the axes for p and q in Figure 11a,b are arranged in descending order to enhance visualization.

5. Discussion

5.1. Experiments with Simulated Data

Recognition results are presented in Table 2, covering Algorithms 1 to 4, the proposed method, and the supervised classification. Results for Algorithm 1 indicate that differences exist between optical and ISAR images. Although the location of structures corresponds to the scattering centers, differences and color disturbances hinder the matching using NALLE, leading to discrepancies between the pseudo-ISAR images and the actual ISAR images. Algorithm 2 estimates the location of the scattering centers. However, the amplitude is difficult to estimate from optical images. Some instances of Target 3 are misidentified as Target 1 due to the similar shape between the two targets’ ISAR images, resulting in lower accuracy. For Algorithm 3, the amplitude of pseudo-ISAR images is close to the simulated ISAR images. Accuracy increases to 77.9% when these pseudo-ISAR images are used to train the classifier. However, false edges caused by color disturbances in optical images introduce mismatches, leading to a loss of accuracy. Algorithm 4 uses the LLE method without location information. Matching low-intensity patches may introduce errors. This results in accuracy that is 6.5% lower than the proposed method. Our proposed method achieves an accuracy of 89.6% for the unseen class and an average accuracy of 94.5%, as it accurately estimates the location and amplitude of the scattering centers. There is still room for improvement, as the accuracy of supervised recognition attains 98.1%.

In Table 3, the highest PSNR is 19.9 dB for the proposed method, demonstrating the best signal preservation. The proposed method yields the smallest RMSE with 26.1. The generated ISAR images at the pixel-level are the most similar to the real ISAR images. The proposed method achieves the inferior CE, indicating a similar probability distribution between the pseudo-ISAR images and the real ISAR images. The highest structural similarity is achieved by the proposed method, with an SSIM value of 0.779. Overall, the proposed method outperforms other approaches in three of four metrics, PSNR, RMSE, and SSIM, and ranks second in CE. The proposed method establishes a balanced compromise and demonstrates superior quantitative performance on simulated data.

Since Target 3 is the unseen class, Figure 6a clearly shows that the mean accuracy for both the unseen class and all targets performs better at K = 3, K = 4, and K = 5. Mean accuracy for Target 3 achieves its highest value at K = 5, reaching 89.6% for the unseen class and 94.5% for all targets. The highest mean accuracy for all targets is 94.6% at K = 4, which is very close to the result at K = 5. Notably, the standard deviation of accuracy is only 0.068 for the unseen class at K = 5, and 0.023 for all targets. Based on these factors, we select K = 5 for the simulated data.

In Figure 7a,b, it is evident that when both p and q are set to 16, the highest accuracy is achieved for the unseen class and all targets, reaching 89.7% and 94.5%, respectively. Additionally, both exhibit the lowest standard deviations, 0.064 and 0.022, respectively. Based on these experiments, we choose a patch size of 16 × 16, and set K to 5 for the simulation data.

5.2. Experiments with Measured Data

In Table 4, for the measured data, the accuracy is lower than that of the simulated data. One reason is that the real optical images contain more details than simulated images, resulting in mismatches. The accuracy of Algorithm 1 is 71.9% with the unseen class. Algorithm 2 can only approximate the location of scattering centers, resulting in discrepancies with the real ISAR image. The accuracy for the unseen class stands at only 63.2%. Furthermore, Algorithm 3 performs worse than in the simulated data, with an accuracy of 73.7%, as the details in real optical images are more complex than those in the simulated images. Algorithm 4 is close to the proposed method, with an accuracy of 80.1%. Our method achieves the highest accuracy for the unseen class at 85.2%, with an average accuracy of 90.8%. The supervised classification achieves 95.2% for the unseen class and an average accuracy of 96.5%. These results signify that our method effectively transforms the optical images into pseudo-ISAR images using the measured data. Furthermore, there is still room for further improvement.

In Table 5, the highest PSNR indicates a better quality in the pseudo-ISAR images, and the minimal RMSE (25.0) reflects the closest pixel-level agreement with the original ISAR images. Our proposed method has the best CE value of 0.106, which means the probability distribution of the value is close to the origin ISAR images. Although Algorithm 1 obtains a similar SSIM, it performs worst in the other three metrics. While Algorithm 4 obtains a similar PSNR and RMSE, it has the worst SSIM and the third-best Cross Entropy. In conclusion, our method demonstrates superior performance on measured data.

Figure 10a clearly shows that the best accuracy is achieved when K = 6, with 85.5% for the unseen class and 91.0% for all the targets. However, in Figure 10b, the standard deviation significantly increases when $K>6$. Specifically, at K = 6, the standard deviation is 0.044 for the unseen class and 0.025 for all targets. A higher standard deviation indicates decreased stability in recognition. One possible explanation is the existence of similar optical patches with varying ISAR structures, which complicate the reconstruction of pseudo-ISAR images. As a compromise, we select K = 5, which yields a mean accuracy of 85.2% for the unseen class and 90.8% for all targets, with standard deviations of 0.034 and 0.017, respectively.

Figure 11 shows that the optimal patch size is 8 × 8. At this size, the unseen class achieves the highest mean accuracy of 85.2% with the lowest standard deviation of 0.035, while all targets achieve the highest mean accuracy of 90.7% with the lowest standard deviation is 0.018.

The optimal patch size is 8 × 8 for the measured data, whereas for the simulated data, it is 16 × 16. The reason might be as follows: The scattering centers in the measured data are smaller and contain more details. Additionally, the measured dataset is smaller than the simulation dataset. Larger patches may fail to capture all possible conditions of real ship structures in ISAR images, leading to patches in the test data that are not represented in the training data. Consequently, for the measured data, the selected hyperparameters are a patch size of 8 × 8 and K = 5.

6. Conclusions

This paper presents a framework for ISAR image generation to facilitate MZSL based on the scattering center model and NALLE. This framework generates pseudo-ISAR images for unseen classes with optical images, serving as the training samples for the classifier to achieve MZSL.

The framework includes these steps: first, the structures of ships are extracted from optical images. Using the extracted structures, the scattering centers are estimated. ISAR images of the edges are generated employing the GTD-based scattering center model. Subsequently, following the mapping between the edges’ ISAR images and the actual ISAR images, pseudo-ISAR images are generated using NALLE. Finally, the classifier for MZSL is trained with the pseudo-ISAR images of unseen classes as training samples. Simulated and measured data are used to validate the effectiveness of the proposed method.

Several areas present potential for improvement. One such improvement involves incorporating semantic information into our method to potentially increase accuracy. Another improvement is addressing the low-intensity regions inside the edges’ ISAR images, particularly when the ship island is large, which might affect the generation of the pseudo-ISAR images. Addressing these issues in future research could enhance both robustness and accuracy.