A Zero-Shot Image Classification Method of Ship Coating Defects Based on IDATLWGAN

Abstract

In recent years, the defect image classification method based on deep transfer learning has been widely explored and researched, and the task of source and target domains with the same painting defect image class has been solved successfully. However, in real applications, due to the complexity and uncertainty of ship painting conditions, it is very likely that there are unknown classes of painting defects, and the traditional deep learning model cannot identify a few classes, which leads to model overfitting and reduces its generalization ability. In this paper, a zero-shot Image classification method for ship painting defects based on IDATLWGAN is proposed to identify new unknown classes of defects in the target domain. The method is based on a deep convolutional neural network combined with adversarial transfer learning. First, a preprocessed ship painting defect dataset is used as input for the domain-invariant feature extractor. Then, the domain invariant feature extractor takes domain invariant features from the source and target domains. Finally, Defect discriminators and domain alignment discriminators are employed to classify the known categories of unlabeled defects and unknown categories of unlabeled defects in the target domain and to further reduce the distance between the edge distributions of the source and target domains. The experimental results show that the proposed model in this paper extracts a better distribution of invariant features in the source and target domains compared to other existing transfer learning models. It can successfully complete the migration task and accurately recognize the painting defects of known categories and new unknown categories, which is a perfect combination of intelligent algorithms and engineering practice.

1. Introduction

Ship coating runs through the entire shipbuilding process as one of modern shipbuilding’s three central process pillars [1]. Ship hull surface coating can effectively prevent all parts of the ship from being corroded by the atmosphere and seawater. Still, it also has anti-fouling, beautification, decoration, and other special functions [2]. The coating quality is directly related to the ship’s construction cycle and maintenance costs, as well as an essential factor affecting the hull’s corrosion resistance and the ship’s service life [3]. However, in the process of ship coating construction, due to the joint influence of many factors such as process parameters, internal and external environment, and coating quality, many different types of coating defects such as sagging, blistering, cracking, delamination, etc. will occur [4], These coating defects not only affect the ship’s beauty but also affect the quality of the ship’s coatings, reduce the ship’s anti-fouling and anti-corrosion function, which affects the service life of the ship [5]. Therefore, in the whole process of ship construction, it is crucial to carry out intelligent identification of painting defects and feedback on painting information to improve the quality of ship painting, prolong the service life of ships, and enhance the market competitiveness of shipbuilding enterprises.

Feature extraction is to extract effective feature information from images, which is usually used as the basis of image classification [6,7]. Domestic and foreign research scholars have carried out a lot of studies on image classification algorithms; Liu et al. [8] combined multi-dimensional convolutional layers with an attention mechanism model to realize the satisfactory classification performance of CNNs in the framework of small-sample learning, which was used to solve the small-sample problem in hyperspectral image classification; Shi et al. [9] proposed an efficient binary network search method to design lightweight binary networks, and used it for image classification tasks in CIFAR10 and CIFAR100; Jin et al. [10] mapped visual feature vectors obtained from fine-tuned ConvNeXt network and semantic vectors obtained from BioBert coding into a common metric space, and proposed a new double-weighted metric loss function for metricizing the distance between images and labels. Zheng et al. [11] used the graph Laplace matrix of the learning dictionary to preserve the locality information, used the label information of the atoms to construct the label embedding term, and verified that the optimal coding coefficients obtained from locality-based and label-based reconstruction are effective for image classification. Currently, traditional coating defect identification mainly relies on professionals to detect the coating quality and record the types and grades of defects through their professional knowledge and work experience. This method is slow [12], inefficient [13], and affected by the subjective factors of the inspectors, resulting in low reliability of the detection results. With the popularization and development of modern manufacturing concepts and intelligent manufacturing technology in ship construction, people gradually begin to apply intelligent technologies and algorithms to the ship construction process. Still, there are few reports on the research and application of intelligent detection of painting defects based on image recognition.

In recent decades, deep learning methods represented by convolutional neural networks have attracted extensive attention from relevant researchers engaged in manufacturing industries with their ability to extract [14] effective high-level features end-to-end automatically and have achieved great success in the fields of natural language processing [15], image recognition [16] and speech recognition [17]. However, there are still many shortcomings that need to be addressed. First, deep learning methods rely on large-scale and high-quality labeled training samples. Still, it is difficult to collect enough labeled training data, and the cost of labeling is too high to guarantee the generalization performance of deep learning models in real industrial environments [18]; second, existing deep learning models usually assume that the training sample dataset and the test sample dataset must be independently and identically distributed, which is unrealistic [19]. According to the research of domestic and foreign scholars, deep transfer learning methods can relax this restriction by effectively solving new tasks in the target domain with knowledge learned from the relevant source domain, effectively solving the above problems, and improving the generalization performance of the target model [20]. Kuang et al. [21] proposed a class-imbalanced adversarial transfer learning (CIATL) network for cross-domain troubleshooting by embedding class-imbalanced learning into the adversarial training process and bi-level adversarial transfer learning, including marginal and conditional distribution adaptation performed. The effectiveness and generalization of the method are verified on a planetary gearbox rig. Lu et al. [22] propose transfer subspace learning for cross-resolution image classification, introducing transfer subspace learning techniques and applying low-rank and sparse constraint matrices. The accuracy of the method was verified on various real image datasets. Li et al. [23] proposed a two-stage transfer adversarial network to construct a new deep transfer learning model based on the adversarial learning strategy, followed by an unsupervised convolutional self-encoder model with silhouette coefficients for multiple novel fault detection of rotating machinery, to solve the fault diagnosis migration task with multiple new faults in the target domain. Xu et al. [24] proposed a new geometric migration metric learning method that integrates pairwise constraints, joint distributional adaption, and manifold regularization into a unified optimization function, making full use of their complementary properties to improve SAR ship classification performance. Conducted in both zero-labeled sample (ZLS) and small-labeled sample (SLS) tasks, experiments show that the proposed method outperforms most state-of-the-art methods. Li et al. [25] proposed a deep learning model based on a multi-scale feature extraction module for steel surface defects, deepening the backbone network by improving the Efficient Feature Fusion (EFF) and Bottleneck modules. The effectiveness of the designed module and method is verified on the public NEU-DET dataset.

In summary, deep transfer learning can be broadly categorized into five types: weighted instance-based migration [26], model parameter-based migration [27], relationship-based migration [28], feature-based migration [29], and adversarial-based migration [30] (ADTL). Among them, adversarial-based deep migration methods are most widely used due to their practicality and good migration results [31].

Although machine learning models based on deep transfer learning algorithms have achieved great success in many fields, such as machinery fault diagnosis [32], image classification [33], etc., most of the existing transfer learning methods are based on the assumption that the source and target domains share the same labeling space, namely, the source and target domains share the same number of categories [34]. However, in practical applications, there will likely be unknown class painting defects due to the complexity and uncertainty of ship painting conditions. These newly appeared unknown class painting defects in the target domain cannot be aligned with the source domain samples due to the lack of samples during the training process in the source domain and are often identified as known class defects in the source domain, which reduces the generalization performance of existing deep transfer learning models. Therefore, detecting unknown class painting defects in the target domain is an important task for intelligent inspection of ships.

To address the aforementioned task, an adversarial migration framework model consisting of domain-invariant feature extractor, Defect discriminators, and domain alignment discriminators is proposed to utilize the known class defects in the source domain to accurately classify the unknown class defects in the target domain. The contributions of this paper are summarized as follows:

(1)

This paper proposes a new zero-sample classification method for ship painting defects based on deep adversarial transfer learning based on Wasserstein GAN (IDATLWGAN) for identifying new unknown class painting defects in the target domain.

(2)

In this paper, the Squeeze-and-Excitation (SE) module is introduced in the domain-invariant feature extractor and used in the transfer learning task, which is more capable of using global information to acquire important domain-invariant features selectively and suppress less useful ones.

(3)

Domain alignment discriminators are introduced and used in a deep transfer model, which learns domain-invariant and class-separation features to classify defects accurately through two-stage adversarial training.

The experiments show that the proposed IDATLWGAN model can better perform the migration task and accurately identify known classes of painting defects and new unknown classes of painting defects on the ship painting defects dataset compared to other existent transfer learning models.

2. Theoretical Background

2.1. Classification Domain CNN Structure

Convolutional Neural Networks (CNN) are feed-forward neural networks which is more widely used in pattern classification because they avoid the complex pre-processing of the image and can be directly input to the original image [35]. CNN in the classification field consists of three parts: convolutional layer, pooling layer, and fully connected (FC) layer. The convolutional layer is the core layer used to build CNN, and its role is to extract image features, as defined in Equation (1).

$$y_m^l=\sigma ({\displaystyle \sum x^{l-1}}·k_m^l·\chi +b_m^l)$$

(1)

where $x^{l-1}$ shows the feature map of the $l-1$ layer input, $·\chi$ shows the convolution operation, $k_m^l$ and $b_m^l$ show the convolution kernel and bias unit corresponding to the $l$ layer m-channel, respectively, $\sigma$ shows the activation function, and $y_m^l$ shows the convolution output of the $l$ layer m-channel.

The pooling layer is used to summarize the feature map features and reduce the spatial size of the feature map. It is defined as shown in Equation (2).

$$g_m^l=V_R(y_m^l(U_c))$$

(2)

where $U_c$ shows the location coordinates, $V_R$ shows the pooling operation, and $g_m^l$ shows the pooled output of the $l$ layer m-channel.

The fully connected (FC) layer acts as a “classifier” in the whole CNN. While operations such as convolutional and pooling layers map the original data to the hidden feature space, the fully connected layer maps the learned “distributed feature representation” to the sample labeling space.

2.2. SE Attention Mechanisms

The SE module was proposed by Jie Hu et al. [36]. The relationship between channels is constructed by introducing Squeeze and Excitation operations, where the feature map is compressed into a $c\times 1\times 1$ feature vector $z_c$ by global average pooling in the Squeeze $(F_{sq}(·))$ phase, as shown in Equation (3).

$$z_c=F_{sq}(u_c)=\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^W{u}_c(i,j)}}$$

(3)

In the Excitation $(F_{ex}(·))$ phase, a weight vector $S$ for a channel is learned to be generated using a fully connected layer ($W_1$,$W_2$) and a nonlinear activation function ($ReLU$, $Sigmoid$). as shown in Equation (4).

$$s=F_{ex}(z,W)=\sigma (g((z,W))=\sigma (W_2\delta (W_{1}z))$$

(4)

A hyperparameter r exists between the two fully connected layers, and in this paper, r is taken as 16. vector $z ( c\times 1\times 1 )$ changes its dimension to $( c/r\times 1\times 1 )$ after passing through the first fully connected layer and then to $( c\times 1\times 1 )$ again after passing through the second fully connected layer. The generated weight vectors $S( c\times 1\times 1 )$ are applied to each channel on the feature map $U( c\times h\times w )$ to weigh the features of different channels. In this way, the SE module can adaptively select and emphasize the important features and improve the discriminative ability of the features, thus improving the model’s performance. Its structure is shown in Figure 1.

2.3. Adversarial-Based Domain Adaptation Training

The Generative Adversarial Networks (GAN) designed by GoodFellow [37] contains two models: a generator and a discriminator. During adversarial training, the goal of generator G is to generate real graph deception slices to deceive discriminator D to maximize the classification error. The goal of discriminator D is to distinguish the picture generated by generator G from the actual picture to minimize the classification error. The model is optimized by a minimal-extremum game of G and D. The GAN optimization objective function $V(D, G)$ is denoted as shown in Equation (5).

$$\underset{G}{\min } \underset{D}{\max }V(D,G)=E_{x ~Pdata(x)}[\log D(x)]+E_{z~P_z(z)}[\log (1-D(G(z)))]$$

(5)

where $\log (D(x))$ is the cross entropy between ${[1,0]}^T$ and ${[D(x),1-D(x)]}^T$. $log(1-D(G(z)))$ is the cross entropy between ${[0,1]}^T$ and ${[D(G(z)),1-D(G(z))]}^T$.

In domain adaptation, given a predefined $N_s\in R$ source domain dataset $D_s={\{(x_s^i,y_s^i)\}}_{i=1}^{N_s}$ with labels $y_s^i$ and $N_t\in R$ unlabeled target domain datasets $D_t={\left\{x_t^i\right\}}_{i=1}^{N_t}$, the source and target domains share the same feature space $(D_s,D_t\in {\chi }_{\mathrm{a}})$ and labeling space ${\chi }_{\mathrm{b}}$. Still, the source and target domains have different distributions. The task of the domain adaptation algorithm is to learn a classifier $f:\alpha \to \beta$ that utilizes the source domain dataset $D_s$ with labels $y_s^i$ to predict the labels of the target domain dataset $D_t$.

The domain adaptive neural network (DANN) designed by Yaroslav Ganin [38] et al. introduced the adversarial idea into the field of transfer learning for the first time. In domain-adaptive adversarial training, the domain-invariant feature extractor learns high-level domain-invariant features from $D_s$ and $D_t$. It characterizes them to be passed to the domain discriminator, which constantly updates the difference between $D_s$ and $D_t$ and calculates the loss. The domain discriminator training goal is to classify $D_s$ and $D_t$ accurately. In contrast, the feature extractor is trained with the opposite goal (due to the presence of the gradient inversion layer), creating an adversarial relationship. The difference between $D_s$ and $D_t$ is minimized by backpropagation, and then the domain-invariant feature extractor learns high-level domain-invariant features.

3. Proposed Methodology

3.1. Problem Definition and Symbolic Description

This paper researches the problem of classifying new unknown class painting defects without labels in different labeling spaces. Specifically, supervised training data and unsupervised test data used to drive the painting defect classification model are collected from the shipyard’s painting logbook, construction accounts, and painting process database. Before elaborating on the IDATLWGAN framework, the readability of the content is enhanced by defining the relevant problems through symbols or formulas.

First of all, for deep transfer learning (DTL), there are two basic concepts: domain D and task T. Among them, the domain D consists of two components: the feature space χ and the edge probability distribution $P(X)$, as shown in Equation (6).

$$D=\{\chi ,P(X)\}$$

(6)

where $X=\{x_1,x_2,\cdots ,x_n\}\in \chi$ denotes an n-dimensional vector; $P(X)$ denotes the marginal probability distribution of the feature space χ.

A given predefined $N_s\in R$ source domain dataset $D_s={\{(x_s^i,y_s^i)\}}_{i=1}^{N_s}$ with labels $y_s^i$ and $N_t\in R$ target domain datasets $D_t={\left\{x_t^i\right\}}_{i=1}^{N_t}$ without labels $y_t^i$ (typically, $1\le N_t\ll N_s$) exists. The source and target domains share the same feature space (namely, $D_s,D_t\in \chi$) but have different edge probability distributions (namely, $P_s(D_s)\ne P_t(D_t)$). If $X_s\ne X_t$ and (or) $P_s(D_s)\ne P_t(D_t)$, the source and target domain distributions are different, (namely, $D_s\ne D_t$). Also, task T consists of two parts: the category labeling space Y and the conditional probability distribution $P(Y|X)$, as shown in Equation (7).

$$T=\{Y,P(Y|X)\}$$

(7)

where Y denotes a category labeling space, the conditional probability distribution $P(Y|X)$ denotes the target prediction function $C(·)$ of the feature space.

Since the source and target domain datasets have the same classes in the existing deep transfer learning models, their class labeling space is also the same. However, unlabeled unknown class defects appear in the target domain, thus $Y_s\ne Y_t$. In addition, in this paper, the target domain $D_t$ contains two components, namely, $n_{jk}$ unlabeled known class painting defects samples and $n_{jn}$ unlabeled unknown class painting defects samples. $n_{jk}$ and $n_{jn}$ are satisfied $n_{jk}+n_{jn}=n_t$. Finally, the high-level invariant feature vectors extracted by the DCNN model from the source and target domains are defined as $V_s^i(x_s^i,y_s^i)$ and $V_t^i(x_t^i)$. This paper shows the Important variables and function symbols and their definitions in

Table 1. Important variable and function symbols and their definitions.

Variable and Function Symbols	Define
$D_s$, $D_t$	Source and target domain datasets
$V_s^i(x_s^i,y_s^i)$, $V_t^i(x_t^i)$	High-level domain invariant features of the source and target domains
$P_s(D_s)$, $P_t(D_t)$	Marginal probability distributions for source and target domains
$G(·)$	Domain-invariant feature extractor
$D(·)$	Defect discriminators modeling
$Critic(·)$	Domain alignment discriminators
${\theta }_G$	Set of weight parameters and bias parameters for each layer in the domain-invariant feature extractor
${\theta }_D$	Set of weight parameters and bias parameters for each layer in the Defect discriminators model
${\theta }_{Critic}$	Set of weight parameters and bias parameters for each layer in the domain alignment discriminators
α	Penalty coefficients for the gradient inversion layer
λ	Weight balance coefficients for the Defect discriminators model
η, ρ	Weight balance coefficients for domain alignment discriminators
$\zeta$	Thresholding parameters for Defect discriminator models
α1	The learning rate of domain-invariant feature extractor and Defect discriminator models
α2	The learning rate of domain alignment discriminators

.

3.2. Overall Network Framework

The IDATLWGAN ship painting defect zero-sample intelligent classification method based on IDATLWGAN proposed in this paper consists of domain-invariant feature extractor G, Defect discriminator D, and domain alignment discriminators critic. The model structure is shown in Figure 2. The acquired ship painting defects dataset is first preprocessed (proportional resizing, data set partitioning, data smoothing, and data normalization) in this work. Then, the preprocessed dataset is divided into source and target domain data. The migratable features of the source and target domain data can be represented as $x_s^h=G(x_s^i)$ and $x_t^h=G(x_t^i)$. The feature extractor G consists of convolutional blocks and SE attention blocks, which are designed to extract high-level domain-invariant features from the defect images of the source and target domains, and its structure and network parameters are shown in Figure 3 and

Table 2. Detailed description of domain-invariant feature extractor network parameters.

Layer	Filter Size	Pacemaker	Padding	Output Size
Input layer real samples x	—	—	—	3 × 128 × 128
Conv2D + BN + LeakyReLU	3 × 3	1	1	128 × 128 × 128
Max pooling layer	2 × 2	2	—	128 × 64 × 64
Conv2D + BN + LeakyReLU	3 × 3	1	1	256 × 64 × 64
Max pooling layer	2 × 2	2	—	256 × 32 × 32
Conv2D + BN + LeakyReLU	3 × 3	1	1	512 × 32 × 32
Max pooling layer	2 × 2	2	—	512 × 16 × 16
Conv2D + BN + LeakyReLU	3 × 3	1	1	1024 × 16 × 16
Global average pooling layer	—	—	—	1024 × 1 × 1
FC + ReLU	—	—	—	64 × 1 × 1
FC + sigmoid	—	—	—	1024 × 1 × 1
Output layer real samples x	—	—	—	1024 × 16 × 16

. Defect discriminator D and domain alignment discriminators critic is employed to classify the known unlabeled defects and unknown unlabeled defects in the target domain by the learned high-level domain invariant features and to reduce further the distributional difference between the edge probability distributions $P_s(D_s)$ and $P_t(D_t)$ in the source and target domains. The domain-aligned discriminator critic is connected to the feature extractor G by the FC layer, gradient reversal layer (GRL), and sigmoid function completion.

The training strategy for the IDATLWGAN model is an adversarial learning mechanism, which is a min-max game between domain-invariant feature extractor G, Defect discriminators D, and domain alignment discriminators critic.

3.3. Loss Function

The purpose of iterative adversarial training of the IDATLWGAN model is to minimize the loss function. Its loss function has four constituents, as shown in Equation (8).

$$\underset{{\theta }_{D,}{\theta }_G}{\min }\{L_D(x_s^i,y_s^i)+\lambda L_c(x_t^i)+\eta \underset{{\theta }_{Critic}}{\max }[L_{wd}(x_s,x_t)-\rho L_{grad}]\}$$

(8)

where λ, η, and ρ are the weight balance coefficients of the loss function, the hyperparameters λ and η are used to determine the hyper coefficients of the confusion level of the source and target domains, and 1 and 2 denote the standard cross-entropy loss function of Defect discriminators model D for fast classification of unlabeled known defective samples in the target domain and the binary cross-entropy loss function for identifying new unlabeled unknown painting defects in the target domain, as shown in Equations (9) and (10).

$$L_D(x_s^i,y_s^i)=-\log [D((V_s^i(x_s^i,y_s^i)))]$$

(9)

$$L_c(x_t^i)=-\zeta \log (p(y=n+1|x_t^i))-(1-\zeta )\log (1-p(y=n+1|x_t^i))$$

(10)

where $D((V_s^i(x_s^i,y_s^i)))$ denotes the probability of defective samples in the source domain, and $V_s^i(x_s^i,y_s^i)$ denotes the high-level domain invariant feature vector extracted by the DCNN model from the source domain, as shown in Equation (11).

$$V_s^i(x_s^i,y_s^i)=\left[\begin{array}{l}p(y=1|x_s^i)\\ p(y=2|x_s^i)\\ ⋮\\ p(y=n|x_s^i)\end{array}\right]=\frac{1}{{\displaystyle \sum _{j=1}^n{e}^{z_j}}}\left[\begin{array}{l}{e}^{z_1}\\ e^{z_2}\\ ⋮\\ e^{z_n}\end{array}\right]$$

(11)

where $p(y=n|x_s^i)$ denotes the probability that the input sample $x_s^i$ in the target domain is classified as a known unlabeled defect class $j\in \{1,2,\cdots ,n\}$, $z_j=V_s{(V_s^i(x_s^i,y_s^i))}_j$ denotes the jth value, $V_s^i(x_s^i,y_s^i)$ denotes the normalization term, and n is the number of types of defects in the known class.

Since a known class defect sample will have a higher probability of being recognized as a known class defect than a new unlabeled unknown defect sample, a threshold parameter $\zeta (0<\zeta <1)$ is chosen to determine whether a sample belongs to a known defect sample or a new unknown class defect sample to quantify the pseudo-decision boundaries between the known class defect category and the unknown class defect category. The role of the threshold parameter is shown in Figure 4. If the probability $p(y=n+1|x_t^i)$ exceeds the threshold $\zeta$, the sample is recognized as a new unknown class defective sample. This paper sets the threshold parameter $\zeta$ to 0.5, as shown in Figure 4.

Domain alignment discriminators loss $L_{wd}(x_s,x_t)$ is used to compute the Wasserstein distance between edge probability distributions $P_s(D_s)$ and $P_t(D_t)$, as shown in Equation (12).

$$L_{wd}(x_s,x_t)=E_{x\sim p_s}[f_{{\theta }_{Critic}}(x)]-E_{x\sim p_t}[f_{{\theta }_{Critic}}(x)]=\frac{1}{N_s}{\displaystyle \sum _{x^s\in X^s}{f}_{{\theta }_{Critic}}(f_{{\theta }_e}(x^s))}-\frac{1}{N_t}{\displaystyle \sum _{x^t\in X^t}{f}_{{\theta }_{Critic}}(f_{{\theta }_e}(x^t))}$$

(12)

where $L_{wd}$ denotes the domain alignment discriminative loss of the source domain data $X_s$ and the target data $X_t$.

Inspired by Wasserstein Generative Adversarial Networks-Gradient Penalty (WGAN-GP), this paper adds a gradient penalty term $L_{grad}$ (gradient penalty, GP) in $L_{wd}(x_s,x_t)$, which is used to solve the problem with Lipschitz constraints such as vanishing or exploding gradients, as shown in Equation (13).

$$L_{grad}={({‖{\nabla }_{x^h}f(x^h)‖}_2-1)}^2$$

(13)

where $x^h$ denotes the vector of high-level invariant feature representations of the source and target domains.

In summary, based on the proposed adversarial training network architecture, the optimization objective of domain-invariant feature extractor G is to minimize $L_D(x_s^i,y_s^i)$ and domain alignment discriminators loss $L_{wd}(x_s,x_t)$ and maximize $L_c(x_t^i)$ at the same time. The optimization objective of Defect discriminators model D is to minimize $L_D(x_s^i,y_s^i)$ and $L_c(x_t^i)$ at the same time. The optimization objective of domain alignment discriminators critic is to minimize $L_{wd}(x_s,x_t)-\rho L_{grad}$ and $L_c(x_t^i)$ at the same time. Therefore, the proposed network optimization problem is shown in Equations (14)–(16).

$${\widehat{\theta }}_G=\arg \{\underset{{\theta }_G}{\min }{L}_D(x_s^i,y_s^i),\underset{{\theta }_G}{\min }{L}_{wd}(x_s^i,x_t^i),\underset{{\theta }_G}{\max }{L}_c(x_t^i)\}$$

(14)

$${\widehat{\theta }}_D=\arg \{\underset{{\theta }_D}{\min }{L}_D(x_s^i,y_s^i),\underset{{\theta }_D}{\min }{L}_c(x_t^i)\}$$

(15)

$${\widehat{\theta }}_{Critic}=\arg \{\underset{{\theta }_{Critic}}{\min }[L_{wd}(x_s,x_t)-\rho L_{grad}],\underset{{\theta }_{Critic}}{\min }{L}_c(x_t^i)\}$$

(16)

where ${\widehat{\theta }}_G$, ${\widehat{\theta }}_D$, and ${\widehat{\theta }}_{Critic}$ denote the optimal estimated parameter values for ${\theta }_G$, ${\theta }_D$, and ${\theta }_{Critic}$, respectively. At each training turn, the parameters of the model are updated, as shown in Equations (17)–(19).

$${\theta }_G\leftarrow {\theta }_G-{\alpha }_1{\nabla }_{{\theta }_G}[L_D(x_s,y_s)-\lambda L_c(x_t)+\eta L_{wd}(x_s,x_t)]$$

(17)

$${\theta }_D\leftarrow {\theta }_D-{\alpha }_1{\nabla }_{{\theta }_D}[L_D(x_s,y_s)+\lambda L_c(x_t)]$$

(18)

$${\theta }_{Critic}\leftarrow {\theta }_{Critic}-{\alpha }_1{\nabla }_{{\theta }_{Critic}}[\lambda L_c(x_t)]-{\alpha }_2{\nabla }_{{\theta }_{Critic}}[L_{wd}(x_s,x_t)-\rho L_{grad}(x^h)]$$

(19)

where ${\alpha }_1$ denotes the learning rate of the domain-invariant feature extractor G and Defect discriminator model D, and ${\alpha }_2$ denotes the learning rate of the domain alignment discriminators critic.

3.4. Training Process Optimization and Implementation Details

Based on the above loss function, to solve the parameter updating and minimal maximal optimization problems of the unsupervised transfer learning Back Propagation (BP) algorithm, this paper brings in the Gradient Reversal Layer (GRL), which has two forms, namely, Forward Propagation and Back Propagation, and its mathematical expressions are shown in Equations (20) and (21). For forward propagation, GRL is a constancy mapping. For backward propagation, GRL extracts the gradient from the next layer and multiplies it by a negative hyperparameter −α (α > 0) to pass it to the previous layer. Thus, GRL has no parameters that need to be predefined or trained. In this paper, the GRL module is introduced between domain-invariant feature extractor G and Defect discriminators model D during backpropagation, which ensures that the feature distributions of different domains are indistinguishable as a way of facilitating domain-invariant feature extractor G and Defect discriminators model D to learn high-level domain-invariant features with an adversarial training strategy.

$$G_{\lambda }(x)=x$$

(20)

$$\frac{d_{G_{\lambda }}}{d_x}=-\alpha I$$

(21)

where I is the unit matrix and α denotes the penalty coefficient, as shown in Equation (22). In this paper, it is simplified and set to 1.

$$\alpha =\frac{2}{1+\exp (-\gamma ·p)}-1$$

(22)

where $\gamma$ is the hyperparameter, set to 10 in this paper, and p denotes the relative value of the iteration process, namely, the ratio of the current number of iterations to the total number of iterations.

In addition to this, momentum-based optimization algorithms such as Adam’s algorithm perform worse due to the instability of the discriminator loss, whereas the RMSProp optimizer performs well even in volatile conditions, so we use the RMSProp optimizer for stochastic gradient descent optimization of the model parameters. The number of epochs is 50. In this work, the initial learning rate is set to 0.001, and the decay rate of the learning rate equals 0.1.

3.5. Training Process

The overall training process of the proposed IDATLWGAN model is shown in Algorithm 1, and the whole network training process is divided into the following eight steps:

• STEP1: Data Collection: Original ship painting defect data sets are collected through the shipyard’s painting log, construction ledger, and painting process database.
• STEP2: Data Preprocessing: Image preprocessing consists of four steps, namely, proportional resizing, data set partitioning, data smoothing, and data normalization.
• STEP3: The ship painting defects training set after STEP2 processing is divided into source and target domain data.
• STEP4: The source and target domain defective training sets are used together as input to the domain-invariant feature extractor G. The extractor extracts high-level domain-invariant features from the defective images of the source and target domains.
• STEP5: The Defect discriminator D and domain alignment discriminators critic is employed to classify the known unlabeled defects and unknown unlabeled defects in the target domain by the learned high-level domain-invariant features and to reduce further the distributional differences between the edge probability distributions of the source and target domains.
• STEP6: Update parameters ${\theta }_G$, ${\theta }_D$, and ${\theta }_{Critic}$ of domain-invariant feature extractor G, Defect discriminator model D, and domain alignment discriminators critic, respectively.
• STEP7: Repeat STEP4~STEP6 to iteratively update the parameters of each module through the adversarial training strategy until convergence, and store all the parameters to obtain the trained optimal estimated parameter values ${\widehat{\theta }}_G$, ${\theta }_D$, and ${\theta }_{Critic}$.
• STEP8: All test datasets are used to test and validate the validity of the IDATLWGAN model.

Algorithm 1: The overall training process for the IDATLWGAN model

Require: Source and target domain datasets $D_s$ and $D_t$; small batch size n; learning rate α1, α2; number of updates of domain alignment discriminators critic in each iteration; and weight balance coefficients λ, η, and ρ for each loss function. Initialize hyperparameters for different networks ${\widehat{\theta }}_G$, ${\theta }_D$, and ${\theta }_{Critic}$. While i < Maximum number of iterations or convergence of parameters of each module do    1: ${\{(x_s^i,y_s^i)\}}_{i=1}^n\sim P_s(D_s)$←Randomized small batch sampling from a real ship painting defect source domain dataset.    2: ${\left\{x_t^i\right\}}_{i=1}^n\sim P_t(D_t)$←Random small batch sampling from a real ship painting defects target domain dataset.    3:  For i = 1, …, $n_D$ do    4:  $x_s^h\leftarrow f_{{\theta }_e}(x^s)\sim P_s(D_s)$, $x_t^h\leftarrow f_{{\theta }_e}(x^t)\sim P_t(D_t)$    5:  Sampling from pairs $x_s^h$ and $x_t^h$ yields $x_r^h$    6:  $x^h\leftarrow \{x_s^h,x_t^h,x_r^h\}$    7:  $L_{grad}\leftarrow {({‖{\nabla }_{x^h}f(x^h)‖}_2-1)}^2$    8:  $L_c(x_t^i)\leftarrow -\zeta \log (p(y=n+1|x_t^i))-(1-\zeta )\log (1-p(y=n+1|x_t^i))$    9:   $L_{wd}(x_s,x_t)\leftarrow \frac{1}{N_s}{\displaystyle \sum _{x^s\in X^s}{f}_{{\theta }_{Critic}}(f_{{\theta }_e}(x^s))}-\frac{1}{N_t}{\displaystyle \sum _{x^t\in X^t}{f}_{{\theta }_{Critic}}(f_{{\theta }_e}(x^t))}$    10:   ${\theta }_{Critic}\leftarrow {\theta }_{Critic}-{\alpha }_1{\nabla }_{{\theta }_{Critic}}[\lambda L_c(x_t)]-{\alpha }_2{\nabla }_{{\theta }_{Critic}}[L_{wd}(x_s,x_t)-\rho L_{grad}(x^h)]$    11:  End For    12:  ${\theta }_D\leftarrow {\theta }_D-{\alpha }_1{\nabla }_{{\theta }_D}[L_D(x_s,y_s)+\lambda L_c(x_t)]$    13:  ${\theta }_G\leftarrow {\theta }_G-{\alpha }_1{\nabla }_{{\theta }_G}[L_D(x_s,y_s)-\lambda L_c(x_t)+\eta L_{wd}(x_s,x_t)]$ End While Output Optimal parameter estimates for each module ${\widehat{\theta }}_G$, ${\theta }_D$, and ${\theta }_{Critic}$. End

4. Experiments Setup and Results

4.1. Description of the Dataset

In this work, a total of 6 typical ship painting defect types are analyzed, including three types of painting defects in the wet film state: sagging (SA), orange skin (OS), blistering (BL), and three types of painting defects in the dry film state: cracking (CR), pinholing (PH), delamination (DF). To avoid the problem of category imbalance, which leads to a transfer learning model, results in overfitting of the model, and significantly reduces generalization performance, the number of samples of each painting defect category is set to 600 in this paper. Among them, the amount of sample data in the source domain of various ship painting defect categories is set to 300, while the amount of data in the target domain of the training set and the test set is set to 150.

Table 3. Ship painting defect data set setting.

Defect Class	Sample Image	Source Domain Sample Data Quantities	Quantity of Data in the Target Domain Train Set	Quantity of Data in the Target Domain Test Set	Label
Sagging		300	150	150	SA
Orange skin		300	150	150	OS
Cracking		300	150	150	CR
Blistering		300	150	150	BL
Pinholing		300	150	150	PH
Delamination		300	150	150	DF

lists the details of the ship painting defect dataset. Since the original ship painting defects images collected were of different sizes and proportions, all the images were resized to 3 × 128 × 128 before carrying out the experiments. The preprocessed ship painting defects dataset was randomly divided into a training set and test set in the ratio of 0.7:0.3. In addition to the above preprocessing techniques, data smoothing technique (Gaussian filtering) and data normalization technique are used in this paper to remove the image noise, improve the image quality and ensure that each feature is in the same dimension, respectively, which significantly improves the convergence speed and prediction performance of the neural network. Based on previous research work, this paper uses the min-max normalization technique to scale the defective image pixels from [0, 255] to [0, 1].

4.2. Experimental Environment

The hardware platform used in this experiment is an 11th Gen Intel ® Core TM i7-11800H @ 2.3 GHz (Intel, Santa Clara, CA, USA) with NVIDIA GeForce RTX 3060 Laptop GPU (NVIDIA, Santa Clara, CA, USA) and 16.0 GB RAM. The programming language is Python (version 3.8.4).

4.3. Evaluation Metrics

Different quantitative evaluation metrics are used in the experimental part of the experiment to accurately and comprehensively assess the overall generative performance of the model and measure the performance of the comparison model for the migration task on various defect categories in the ship painting defect dataset. In this paper’s experiments, four usual evaluation metrics in the confusion matrix are used to quantitatively analyze the model performance: accuracy, precision, recall, and F1 score. The binary classification confusion matrix is shown in

Table 4. The binary classification confusion matrix.

Class	Predicted Positive Class	Predicted Negative Class
Actual positive class	TP	FN
Actual negative class	FP	TN

.

Accuracy and recall are complementary; the higher these metrics are, the better. Accuracy is the most intuitive model performance metric, and precision is the ratio of correctly predicted positive observations to total predicted positive observations. The F1 score is the reconciled mean of the lookup accuracy and recall. It considers both precision rate and recall rate and is often used as a statistical metric to evaluate the performance of a classifier. These evaluation metrics are defined as shown below:

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

(23)

$$\mathrm{Pr}\mathrm{ecision}=\frac{TP}{TP+FP}$$

(24)

$$\mathrm{Recall}=TPR=\frac{TP}{TP+FN}$$

(25)

$$\mathrm{F}1\mathrm{-score}=2\times \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}=\frac{2}{2TP+FP+FN}$$

(26)

4.4. Experimental Results and Analysis

4.4.1. Performance Comparison of Different Transfer Learning Models on Different Painting Defect Categories

To further demonstrate the effectiveness and superiority of the method proposed in this paper, different existing transfer learning models such as Transfer Principal Component Analysis (TCA), Joint Distribution Adaptive (JDA), Domain Adaptive Neural Network (DANN), Deep Transfer Learning (DTL), Deep Confrontation Migration Learning Network (DCTLN), and Two-Stage Migration Confrontation Network (TSTAN) are compared with the IDATLWGAN model proposed in this paper. The number of parameters and training time for each model are shown in

Table 5. Number of parameters and training time.

Model	Parameter	Time
TCA	182 k	1124 s
JDA	173 k	973 s
DANN	227 k	572 s
DTL	235 k	721 s
DCTLN	166 k	1028 s
TSTAN	112 k	677 s
IDATLWGAN	91 k	559 s

. Comparison experiments with various defect categories in different transfer learning models in a real ship painting defect dataset are shown in

Table 6. Comparison of F1-Score, Precision and Recall Rates of Different Transfer Learning Models on Various Defect Categories.

Model	F1-Score						Precision	Recall
	SA	OS	BL	CR	PH	DF
TCA	0.695	0.676	0.648	0.712	0.768	0.789	0.634	0.637
JDA	0.487	0.497	0.573	0.539	0.572	0.625	0.519	0.485
DANN	0.893	0.735	0.826	0.867	0.848	0.864	0.745	0.812
DTL	0.805	0.756	0.874	0.882	0.865	0.895	0.796	0.854
DCTLN	0.823	0.628	0.889	0.943	0.874	0.913	0.832	0.897
TSTAN	0.945	0.784	0.876	0.834	0.865	0.876	0.873	0.749
IDATLWGAN	0.932	0.912	0.957	0.935	0.881	0.943	0.994	0.995

. The bold values indicate the maximum values of the evaluation metrics in different migration models. Table 6 shows the average training results of the five-fold cross-validation to reduce the randomness of the data.

Compared with other existing transfer learning models, although the IDATLWGAN model proposed in this paper has lower F1 scores of 0.013 and 0.008 for sagging defects (SA) and cracking defects (CR) compared with TSTAN and DCTLN, respectively, the F1-score are generally higher in other defects. Meanwhile, defects are much higher than in other transfer learning models in other evaluation metrics (precision and recall). It reflects the unstable performance and the low performance of other transfer learning models when facing different defects. Based on the analysis of the above results, it can be concluded that the overall performance of the IDATLWGAN model proposed in this paper is better than that of other transfer learning models on the ship painting defects dataset.

4.4.2. Performance Comparison of Different Transfer Learning Models on Different Migration Tasks

In this section, to validate the stability and reliability of the proposed IDATLWGAN model, two groups of experiments, I and II, are designed, each with six migration tasks, as shown in

Table 7. Detailed description of different migration tasks.

	Task	Source Domain→Target Domain	Source Domain Dataset Defect Class	Target Domain Dataset Defect Class
I	1	A→C	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
	2	C→A	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
	3	B→E	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
	4	E→B	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
	5	C→D	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
	6	D→C	SA, OS, CR, PH	SA, OS, CR, PH, BL, DF
II	7	B→A	SA, OS, CR, PH	SA, OS, CR, PH, BL
	8	B→A	SA, OS, CR, PH	SA, OS, CR, PH, DF
	9	D→E	SA, OS, CR, PH	SA, OS, CR, PH, BL
	10	D→E	SA, OS, CR, PH	SA, OS, CR, PH, DF
	11	F→B	SA, OS, CR, PH	SA, OS, CR, PH, BL
	12	F→B	SA, OS, CR, PH	SA, OS, CR, PH, DF

. Sagging (SA), orange skin (OS), blistering (BL), cracking (CR), pinholes (PH), and delamination (DF) were collected to construct the source and target domain datasets. The collected painting defects were generated at different air temperatures (15 °C, 25 °C, and 35 °C) and relative humidities (50% and 60%) as shown in

Table 8. Detailed description of different ship painting conditions.

Painting Condition	Air Temperature (°C)	Relative Humidity (%)
A	15	50
B	15	60
C	25	50
D	25	60
E	35	50
F	35	60

. In Experiment I, the source domain dataset is labeled SA, OS, CR, and PH, and the target domain dataset has unlabeled SA, OS, CR, PH, and unlabeled unknown classes BL and DF. Six migration tasks were designed in scenarios with the same temperature and different humidities, different temperatures and different humidities, and different temperatures and same humidities, and a cross-validation strategy was used to validate the IDATLWGAN model’s performance, migration task 1 (namely, A→C) represents the migration task from the source domain defective dataset under condition A to the target domain defective dataset under condition C. The other tasks are similar. The other tasks are similar. To further reduce the experiment’s randomness and verify the reliability of the IDATLWGAN model, Experiment II is designed using the cross-validation strategy. In Experiment II, the source domain dataset has labeled SA, OS, CR, and PH; for the target domain dataset, three tasks contain unlabeled SA, OS, CR, PH, and BL, and the other tasks contain unlabeled SA, OS, CR, PH, and DF.

The experimental results of all the transfer learning tasks on the ship defects dataset are summarized in

Table 9. Comparison of accuracy (%) of different transfer learning models on different transfer tasks.

	Migration Model	TCA	JDA	DANN	DTL	DCTLN	TSTAN	IDATLWGAN
I	1	37.65	38.64	51.63	57.34	67.78	84.08	90.56
	2	36.24	41.18	53.15	56.61	69.41	83.23	94.74
	3	40.14	40.56	54.48	58.56	71.22	86.89	92.78
	4	38.78	37.44	49.67	53.00	64.67	84.33	91.89
	5	40.86	40.33	52.29	58.37	66.87	77.87	89.77
	6	36.51	38.72	51.11	55.35	68.73	85.63	90.51
	Average	38.36	39.48	52.05	56.54	68.11	83.67	91.71
II	7	46.56	49.45	58.73	68.18	78.02	87.75	96.03
	8	48.31	46.14	59.41	66.73	76.32	89.98	94.25
	9	45.68	48.21	63.59	63.68	74.21	86.23	93.67
	10	49.31	50.08	58.73	67.74	72.36	88.43	94.21
	11	46.54	46.37	64.67	68.29	73.03	91.71	95.57
	12	44.95	46.08	59.96	65.43	79.37	93.58	92.39
	Average	46.89	47.72	60.85	66.68	75.55	89.61	94.35

and Figure 5, and the bold values indicate the maximum accuracy values in different transfer learning models on different transfer tasks. Compared with the traditional methods such as TCA and JDA, the existing deep learning methods (DANN, DTL, DCTLN) have slightly higher average accuracy due to their relatively good feature extraction ability to recognize a small number of unknown unlabeled category defects. Due to their powerful feature extraction ability, the TSTAN and IDATLWGAN models have high average accuracy. The quantitative experimental results show that although the accuracy of the TSTAN model is higher than that of the IDATLWGAN model by 1.19% on task 12 (namely, F→B), the accuracy on the remaining tasks is still low. The proposed IDATLWGAN model achieves an average accuracy of 91.71 and 94.35 on the ship defects dataset in the two sets of experiments I and II, which are both higher than the other comparative methods and can effectively detect unknown class painting defects on ships without labels in the target domain.

Confusion matrices are often used to visually represent the number of accurate predictions and the number of misclassifications for each class in the test results and to interpret this result at the class level. Figure 6 shows the confusion matrices for all transfer learning methods on Task 4. The horizontal axis represents the true attributes, and the vertical axis represents the predicted state of the data. The main diagonal elements indicate the number of samples correctly categorized by each defective class. In contrast, besides the main diagonal elements, the remaining elements indicate the number of samples incorrectly categorized into other defective classes. TCA and JDA are entirely unable to identify the unlabeled unknown classes BL and DF due to their limited extraction capabilities. They are not able to efficiently isolate defective samples from other classes. DANN, DTL, DCTLN, and TSTAN are poor at recognizing defective categories compared to IDATLWGAN. TSTAN, with limited feature extraction capability, can only achieve about 83.33% accuracy due to the shadow structure. The results show that IDATLWGAN performs well in painting defect classification.

To present the transfer results intuitively, a visual data dimensionality reduction algorithm-t distribution stochastic neighborhood embedding algorithm (t-SNE) is used to present the learned feature distribution. For example, Task 9 compares the transferable characteristics of the TCA, JDA, DANN, DTL, DCTLN, TSTAN, and IDATLWGAN methods. Severe features are aliasing in the transferable features of TCA and JDA. There is a severe mixing of known painting defect samples from the source and target domains with the new unmarked painting defect samples, which are not separated from the painting defect samples. Therefore, when the model is trained using the prior knowledge of the source domain, the unmarked target samples cannot be effectively separated using TCA and JDA. The transferable features learned by DANN, DTL, and DCTLN need to be more effectively separated. Specifically, although the known painting defect samples have been transferred from the source domain to the target domain, the new unmarked painting defect samples are still mixed in the known painting defect samples. TSTAN and IDATLWGAN can not only cluster the known painting defect samples of the source and target domains clearly but also separate the new unmarked painting defect samples from the known painting defect samples, but the IDATLWGAN clearly has fewer new unlabeled painting defect samples mixed with known painting defect samples compared to TSTAN, so IDATLWGAN can effectively transfer features from the source domain to the target domain and detect new unlabeled painting defect samples in the unlabeled target domain. This result also illustrates that IDATLWGAN has higher transfer accuracy and better performance than other methods.

5. Conclusions and Future Work

In this paper, a zero-sample classification method for ship painting defects based on IDATLWGAN is proposed, which can not only effectively identify the known class of unlabeled painting defects from the source domain to the target domain but also accurately identify the unknown class painting defects that are unlabeled in the target domain at the same time, which can effectively prevent the model from overfitting as well as improve the model’s generalization ability. The experimental results show that compared with other existing transfer learning models, the model proposed in this paper extracts a better distribution of invariant features in the source and target domains, and its Accuracy, F-Score, Precision, and Recall values are significantly higher than those of other models. With the above results, using the IDATLWGAN model to classify ship painting defects can complete the migration task and accurately identify the known classes of painting defects and new unknown class painting defects, a perfect combination of intelligent algorithms and engineering practice. It has high engineering research value and application prospects.

Currently, it is proved experimentally that the transfer learning method can be used for ship painting to identify unknown class painting defects, but experiments have not yet been conducted on other scenarios to illustrate the general applicability of the transfer learning method; therefore, in the follow-up work, we will consider diagnosing unknown faults on troubleshooting to validate the effectiveness and robustness of the proposed method.

In addition, for real-time or resource-constrained applications, techniques to optimize the computational efficiency of the proposed model are investigated. Explore the integration of real-time monitoring systems for continuous defect detection during ship painting. Develop methods to provide explanations and interpretations for the model’s decisions.