A Semantic-Preserving Deep Hashing Model for Multi-Label Remote Sensing Image Retrieval

Abstract

Conventional remote sensing image retrieval (RSIR) systems perform single-label retrieval with a single label to represent the most dominant semantic content for an image. Improved spatial resolution dramatically boosts the remote sensing image scene complexity, as a remote sensing image always contains multiple categories of surface features. In this case, a single label cannot comprehensively describe the semantic content of a complex remote sensing image scene and therefore results in poor retrieval performance in practical applications. As a result, researchers have begun to pay attention to multi-label image retrieval. However, in the era of massive remote sensing data, how to increase retrieval efficiency and reduce feature storage while preserving semantic information remains unsolved. Considering the powerful capability of hashing learning in overcoming the curse of dimensionality caused by high-dimensional image representation in Approximate Nearest Neighbor (ANN) search problems, we propose a new semantic-preserving deep hashing model for multi-label remote sensing image retrieval. Our model consists of three main components: (1) a convolutional neural network to extract image features; (2) a hash layer to generate binary codes; (3) a new loss function to better maintain the multi-label semantic information of hash learning contained in context remote sensing image scene. As far as we know, this is the first attempt to apply deep hashing into the multi-label remote sensing image retrieval. Experimental results indicate the effectiveness and promising of the introduction of hashing methods in the multi-label remote sensing image retrieval.

1. Introduction

With the rapid development of Earth observation technology, the volume of remote sensing data has increased exponentially [1,2,3]. Remote sensing images have broad applications such as early warning for natural disasters, emergency response, and urban construction planning [4,5]. It is a challenging task to quickly obtain remote sensing images of interest from a pool of massive remote sensing images, stimulating extensive research interests among the scientific community [6,7].

Early text-based remote sensing image retrieval (TBRSIR) performed retrieval based on pre-annotated textural information like acquisition time, resolution, and sensor type. Considering the tedious and burdensome manpower requirements and ambiguous annotation derived from text-based retrieval, researchers have turned to the content-based remote sensing image retrieval (CBRSIR). The CBRSIR is always composed of two main parts: feature extraction and similarity measurement. Due to the rapid advances of artificial intelligence technology and computer hardware, obtaining deep features through convolutional neural networks has become a substitute for traditional hand-crafted features. It has been proved that convolutional neural networks, pre-trained or finetuned, can greatly improve the retrieval performance of remote sensing images [8,9,10,11,12].

At present, most of the CBRSIR methods are based on a single label. For simple scenes, such as forests and chaparral, a single label is enough to obtain excellent retrieval results, but for complex scenes, such as sparse residential areas, a single label cannot distinguish complex categories within an image. Remote sensing images are usually composed of multiple categories. In this case, a single label, which represents the most significant semantic content of a remote sensing image, might ignore the complex and abundant information contained in that image. As shown in Figure 1, we select two images from the single label dataset UCMD and the multi-label dataset MLRSD as an example, it is difficult to accurately describe complex remote sensing image scenes using a single label. Therefore, in recent years, in order to overcome the limitations of single-label remote sensing image retrieval, efforts have been made towards multi-label retrieval of remote sensing images [13,14,15,16,17,18], and it is suggested that multi-label retrieval dramatically outperforms single-label retrieval.

However, in the current DCNN-based remote sensing image retrieval architecture, the feature dimensions of the remote sensing images is usually very high [12]. The larger the dataset, the greater the memory cost and the longer the time required to search images. For example, 10 million images with a 512-dimensional feature vector require about 20 GB of memory, which is very unfavorable for massive remote sensing image retrieval. In addition, the time-consuming calculation among feature vectors of the image makes it difficult to meet the real-time retrieval requirements in actual applications. Thus, how to achieve real-time retrieval of remote sensing images in the era of massive remote sensing data is still challenging.

Recently, hashing methods have become increasingly attractive in the task of Approximate Nearest Neighbors (ANN) search problems [19]. Through mapping high-dimensional image feature vector compact binary hash codes into Hamming space and calculating Hamming distance by simple bit operation and XOR operation, hashing methods can save calculation time and reduce memory consumption to a great degree. For example, 10 million pictures with 512-dimensional feature vectors using 128-bit hash code only need 160 MB for storage. Due to their advantages of simple structure, low space cost, and flexibility to expand, hashing methods have been widely used for fast large-scale image retrieval. To solve the problem of semantic gap resulted from the process of encoding the high-dimensional data to binary bits, researchers have proposed strategies to maintain semantic similarity. For example, in the natural image field [20], Zhang et al. used an optimized loss function to preserve semantic similarity. However, in the remote sensing field, almost all existing deep hashing networks are based on single-label learning, which is not able to adapt to complex remote sensing scenes. In a bit more detail, remote sensing images are usually composed of multiple land categories, so that a single label describing the most significant semantic content might ignore the complex and abundant information contained in the image. To tackle this dilemma, we attempt to introduce multi-label supervision into the deep hashing framework. Furthermore, we propose pair-wise label similarity loss to fully exploit the multi-label semantic information, including the hard similarity loss represented by cross entropy and soft similarity loss measured by mean square error. Specifically, the hard similarity loss accounts for completely similar or dissimilar sample pairs, while the soft similarity loss considers partially similar sample pairs, which together encourage the deep hashing model to preserve the semantic consistency between the input paired image samples. The purpose of our research is to improve efficiency and reduce storage of multi-label image retrieval without losing accuracy. The main contributions of this paper can be summarized as follows:

(1) We propose a semantic-preserving deep hashing model for multi-label remote sensing image retrieval. As far as we know, this is the first attempt to introduce hashing methods into multi-label remote sensing image retrieval.

(2) We propose paired label similarity loss to make full use of multi-label semantic information, including hard similarity loss represented by cross-entropy and soft similarity loss measured by mean square error. Specifically, the hard similarity loss considers completely similar or dissimilar sample pairs, while the soft similarity loss considers partially similar sample pairs. Together, they encourage the deep hash model to maintain semantic consistency between the input paired image samples.

We conduct comparative experiments with other baseline models, including the IDHN Model [21], the improved ISDH model [22], the DMSH model [23], the DHN model [24], and the DPSH model [25], to assess the effectiveness of our proposed model.

The remaining part of this paper is organized as follows: in Section 2, related work on the multi-label retrieval of natural image, multi-label retrieval of remote sensing images and image retrieval based on deep hashing is summarized and analyzed. In Section 3, the system architecture of our model is described in detail, and the design of loss function based on deep hashing network is emphasized. In Section 4, comparative experiments are conducted to demonstrate the superiority of our modal. In Section 5, discussions and conclusions are presented, followed by future directions toward which further efforts can be made.

2. Related Work

2.1. Multi-Label Retrieval of Natural Images

In the field of multi-label retrieval of natural images, Li et al. [26] proposed a Multi-Label nearest neighbor propagation method for region-based image retrieval. Nasierding et al. [27] studied the application of multi-label classification in image annotation and retrieval and made a comprehensive comparison of these methods. Li et al. [28] proposed a new multi-label image annotation method for image retrieval based on annotated keywords. The experimental results show that multi-label can better describe the image content at the semantic level. Ranjan et al. [29] proposed a multi-label canonical correlation analysis algorithm to solve the cross-modal image retrieval problem of multi-label annotation. In addition, multi-label image retrieval based on hash algorithms has also received attention from researchers. For example, Lai et al. [30] studied multi-label image retrieval based on instance hashing. Zhang et al. introduced a quantified definition to pairwise similarity in multi-label image retrieval to achieve an effective feature learning and hash-code learning [21]. In addition, recently, Rodriguez et al. [31] made a survey of hash learning methods for multi-label image retrieval. The achievements of multi-label retrieval of natural images have inspired researchers from other fields such as remote sensing.

2.2. Multi-Label Retrieval of Remote Sensing Images

Multi-label labeling of remote sensing images is a very time-consuming and labor-intensive task and requires certain professional knowledge, which results in the lack of large-scale multi-label remote sensing datasets and limits the research on multi-label remote sensing image retrieval.

In [13], Chaudhuri B et al. made a multi-label remote sensing dataset based on the UCM remote sensing image dataset available publicly and proposed a semi-supervised graph theory method for the multi-label remote sensing image retrieval. Dai et al. [14,15] combined spectral and spatial description features to design a supervised multi-label remote sensing image retrieval method using sparse reconstruction and evaluated the effectiveness of the proposed method with a new label likelihood measure. Although the work mentioned above achieved good performance compared with single-label retrieval of remote sensing images, they relied on hand-crafted features for retrieval. Recently, Shao et al. [16] extended the 17 categories of MLRSD to densely label the UCM image library to generate a Dense Labeling Remote Sensing Dataset (DLRSD) for more tasks besides image retrieval, such as semantic segmentation. They evaluated the performance of hand-crafted features and CNN features in remote sensing image retrieval based on MLRSD; furthermore, Shao et al. [17] proposed a novel multi-label approach based on FCN, which is the first attempt of introducing deep learning into multi-label retrieval task. J. Kang et al. [18] proposed a new graph relation network (GRN) for multi-label RS scene classification, which can use a graph structure input to network learning to model the relationship between scenes. Recently, Qi et al. [32] constructed a multi-label high-spatial-resolution remote sensing dataset named MLRSNet for semantic scene understanding for the task of image classification and image retrieval. Sumbul et al. [33] proposed BigEarthNet-MM dataset which is made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches. Sumbul et al. [34] proposed a novel triplet sampling method in remote sensing image retrieval by selecting informative triplets, thus making the model learn accurately, and proposed a novel learning strategy to train a deep neural network to automatically predict a graph structure of remote sensing images in [35].

All of the work mentioned above shows that the use of deep learning for multi-label retrieval of remote sensing images can improve the retrieval performance effectively. However, these approaches have not paid much attention to the issue of efficiency and memory cost, which prevents the application for massive remote sensing image data in practical scenarios.

2.3. Image Retrieval Based on Deep Hashing

Recently, some researchers have proposed deep hashing-based image retrieval methods [36,37,38] to evaluate the effectiveness of integrating deep learning with hashing learning. The existing hash learning methods can be divided into unsupervised and supervised hash learning. Typical unsupervised hash methods include Locality Sensitive Hashing (LSH) [39], Iterative Quantization (ITQ) [40], Semantic Hashing (SH) [41], and Discrete Graph Hashing (DGH) [42]. Compared with unsupervised hashing methods, which only use the distribution information in image data for learning the hash function and do not use the label data of the image, supervised hash learning methods use supervised information in the hash learning, which improves the accuracy of the hash algorithm.

A variety of supervised hash learning methods for image retrieval have been proposed. In 2014, Xia et al. [43] first proposed a staged convolutional neural network hashing method (CNNH) for which deep learning was integrated into the hashing method. Compared with hand-crafted features, the performance of learning features was significantly improved. Subsequently, Li et al. [25] proposed Deep Pairwise Supervised Hash (DPSH) to implement end-to-end hash learning. Liu et al. [44] proposed a deep-supervised hash model (GAAH) for remote sensing image retrieval under the framework of Generating Adversarial Networks (GAN). To address the problem of limited labeled data for remote sensing images, Roy et al. [45] proposed a deep hash network based on metric learning (MiLaN). In the above methods, they do not really make use of multi-label information. More specifically, in these methods, when two images have a common label, their similarity is set to 1, otherwise it is set to 0. Given this fact, Zhao et al. [46] proposed the Deep Semantic Sorting Hashing Method (DSRH), which returns similar images in a ranked order. Ranjan et al. [29] presented a multi-label cross-modal retrieval system by introducing multi-label Canonical Correlation Analysis (ml-CCA). Wu et al. [47] proposed a deep uniqueness-aware hashing method, which added a multi-label classification layer after the hash layer to make the learned hash code more discriminate. All of the studies mentioned demonstrate the effectiveness of hashing learning methods in CNN-based image retrieval. Inspired by these works, we combine the advantage of hash learning and deep learning to multi-label retrieval of remote sensing images, remote sensing images are usually composed of multiple land categories, so that a single label describing the most significant semantic content might ignore the complex and abundant information contained in the image, we proposed a semantic-preserving deep hashing model to verify and evaluate the performance of deep hashing-based multi-label remote sensing image retrieval.

3. A Semantic-Preserving Deep Hashing Model

In this section, firstly we introduce the basic idea of deep hashing-based multi-label image retrieval in Section 3.1, and then we describe the system architecture and retrieval process of our multi-label image retrieval model in detail, while we focus on the design of loss function in Section 3.3.

3.1. The Basic Idea of Deep Hashing Based Multi-Label Image Retrieval

The basic idea of deep hashing-based multi-label image retrieval is to learn a feature mapping function by using the similarity information of the multi-label remote sensing image and then to map the input image to a length of q hash code. Then, the obtained hash code is used to calculate the hamming distance to measure the similarity between remote sensing images. Among them, the image similarity supervision information $S=S_{ij}\left\{i,j=1,2,\dots ,n\right\}$ required for deep hashing can be calculated by the cosine distance between the multi-label vectors. The cosine similarity between remote sensing image $I_i$ and remote sensing image $I_j$ is defined as follows:

$$S_{ij}=\frac{\langle L_i,L_j\rangle }{\left|\left|L_i\left||•|\right|L_j\right|\right|}$$

(1)

In Equation (1), $L_i$, $L_j$ refer to the label vector of multi-label remote sensing image, respectively; $\langle L_i,L_j\rangle$ refers to the inner product of the vector $L_i$ and the vector $L_j$, and its value is equal to $L_i{}^T{L}_j$; $\left|\left|\right|\right|$ represents the L2 normal form operation. According to Formula (1), it can be seen that there are three similarities in paired multi-label remote sensing images: completely similar ($S_{ij}=1$), similar parts ($0<S_{ij}<1$), dissimilar ($S_{ij}=0$). For an ANN search task, the hash code $B={\left\{b_i\right|b_i\in {\left\{-1,1\right\}}^{\mathrm{q}}\}}_{i=1}^{\mathrm{n}}=1$ learned by deep hashing is required to retain the similarity of the paired images. The similarity value of paired hash code $b_i$ and $b_j$ is inversely propotional to the hamming distance. That is to say, in the case of a given paired hash code $b_i$ and $b_j$,if $S_{ij}=1$, the hamming distance of the $b_i$ and $b_j$ should be as small as possible, preferably close to 0; if $S_{ij}=0$, the hamming distance of the $b_i$ and $b_j$ should be as large as possible, the best approaching $\mathrm{q}$; if $0<S_{ij}<1$, the hamming distance of $b_i$ and $b_j$ should be between the minimum distance and the maximum distance.

3.2. System Architecture of Deep Hashing Model

The whole system architecture of our model is shown in Figure 2, and is mainly composed of two parts, namely the deep feature extraction module and the hash learning module. While the deep feature extraction module is responsible for generating high-level and abstract image representation through multi-level architecture, the hash learning module is responsible for converting each image to a binary space in a binary sequence. To preserve the similarity information of multi-label remote sensing image pairs and control hashing quality, we improve the pair-wise similarity loss and use quantization loss to limit the output value range of the hash network.

The training process of our model is: the paired image is fed into the model, the high-dimensional features of multi-label remote sensing images are extracted through multi-layer convolution and two fully connected layers, and then the output is fed to a hash layer that connects the fully connected layer FC1 and the fully connected layer FC2 to generate a hash code of q length. Then, the image similarity is used as supervision information to train the model in an end-to-end manner. In the retrieval process, the multi-label remote sensing image is encoded into a binary code, and then the distance between the binary code of the query image and other images is calculated, and finally the ranked search results are returned.

We choose AlexNet as the feature extraction backbone network. We also use VGG16 network to extract image feature to show the scalability of our model. To make full use of the deep features extracted by the first fully connected layer, we further input the first fully connected layer into the hash layer.

Table 1. Parameters of deep hashing multi-label retrieval model.

	Filters	Size	Input	Output
Conv_1	64	11 × 11/4	227 × 227 × 3	56 × 56 × 64
Maxpool_1		3 × 3/2	56 × 56 × 64	27 × 27 × 64
Conv_2	192	5 × 5/1	27 × 27 × 64	27 × 27 × 192
Maxpool_2		3 × 3/2	27 × 27 × 192	13 × 13 × 192
Conv_3	384	3 × 3/1	13 × 13 × 192	13 × 13 × 384
Conv_4	256	3 × 3/1	13 × 13 × 384	113 × 13 × 256
Conv_5	256	3 × 3/1	13 × 13 × 256	13 × 13 × 256
Maxpool_5		3 × 3/2	13 × 13 × 256	6 × 6 × 256
Flatten			1 × 1 × 9216
Fc_6	4096		1 × 1 × 9216	1 × 1 × 4096
Fc_7	4096		1 × 1 × 4096	1 × 1 × 4096
Merge			[FC_6,FC_7]
Fc_8	q		1 × 1 × 8192	1 × 1 × q

gives main parameters of our deep hashing multi-label retrieval model, in which Conv_i represents the i-th convolutional layer, Maxpool_i represents the i-th pooling layer, and Fc_i represents the i-th layer is a fully connected layer. In our model, a hash layer with q bits output is exploited to replace the classification layer in the AlexNet network. In addition, in order to reduce the possible loss of semantic information and make full use of the deep features extracted by the first fully connected layer, the first fully connected layer Fc_6 is connected to the hash layer Fc_8.

To obtain the binary hash code, we use the symbolic function to quantify the unified features and transform the output of the deep hashing network. The formula for generating a binary hash code is as follows:

$$b_{i\_k}=\mathrm{sgn}(u_{i\_k})$$

(2)

In Formula (2), when the k-th element of the vector of $u_i$ is greater than 0, the k-th element of the vector of $b_i$ is 1, otherwise it is 0. In this way, the deep hash network can directly encode remote sensing images and get the corresponding binary code.

3.3. Design of Loss Function

The design of loss function directly affects the quality of the hash code. In our model, the loss function is composed of a paired similarity loss function and a quantization loss.

(1)

Paired similarity loss

Based on the three kinds of similarity, the similarity between image pairs can be divided into hard similarity $L_{hard}$ and soft similarity $L_{soft}$. This means that, when the image $I_i$ and the image $I_j$ are completely similar or dissimilar ($S_{ij}$ = 1 or $S_{ij}=$ 0), the similarity of the image pairs ($I_i$,$I_j$) belongs to hard similarity and cross entropy loss is suitable for loss calculation. Similarly, when the image $I_i$ and the image $I_j$ are partially similar ($0<S_{ij}$ < 1), the similarity of the image pair belongs to soft similarity and the mean square error is used to represent the loss. In recent years, many deep hashing methods have mostly performed definition in a hard-allocation manner. That is, if they share no less than one class label, the pairwise similarity is “1”, and if they do not share any class label, it is “0”. However, this definition of similarity does not reflect the similarity ranking of paired images containing multiple labels. To express the hard similarity and soft similarity uniformly, we introduce parameters $C_{ij}$ to balance the range of the two values in designing the loss function; $C_{ij}$ = 1 refers to cross entropy loss and $C_{ij}$ = 0 refers to mean square error loss. Since the value range of $L_{soft}$ is 2q times the value range of $L_{hard}$, we introduce hyperparameters $\mathsf{\beta }$ to balance the range of them. Inspired by [39], in order to preserve the semantic similarity of the paired multi-label remote sensing images in the Hamming space, we constructed an inner product expression ${\delta }_{ij}=\phi \langle b_i,b_j\rangle$ based on the hash code and applied it to the construction of the deep hashing model in the loss function of the pairwise similarity. In addition, we use the similarity $S_{ij}$ of paired images to weight the paired similarity loss, expressed as $e^{\alpha s_{ij}}$, where $\mathsf{\alpha }$ is the adjustment parameter. Therefore, our paired similarity loss calculation formula is as follows:

$$\begin{array}{cc}\hfill L_{similarity}& =e^{\alpha s_{ij}}\left(C_{ij}{L}_{hard}+\beta \left(1-C_{ij}\right)L_{soft}\right)\hfill \\ \hfill & =e^{\alpha s_{ij}}{\displaystyle \sum _{s_{ij}\in S}[C_{ij}\left(\log \left(1+e^{{\delta }_{ij}}\right)-s_{ij}{\delta }_{ij}\right)+\beta \left(1-C_{ij}\right){(\frac{b_i^{{\rm T}}{b}_j+q}{2}-s_{ij}\cdot q)}^2]}\hfill \end{array}$$

(3)

In Formula (3), $b_i\in {\left\{-1, 1\right\}}^q$ is a hyperparameter that controls the range of the inner product value and prevents gradient disappearance effect caused by the inner product value of the input sigmoid function being too small. We take $\mathsf{\phi }=0.5$, that is ${\delta }_{ij}=0.5b_i{}^T{b}_j$.

(2)

Quantization loss

For the binary representation of hash codes, most existing hash algorithms use activation functions like sigmoid or tanh to restrict the output value range of the hash layer and quantize the loss function to constrain the network output value to be nearby. However, the sigmoid/tanh activation function is saturated. The closer the activation output value is closer to $\pm 1$, the smaller the gradient value passed to the bottom layer by the model training is, which is more likely to result in gradient disappearance. We use a more relaxed activation function (softsign function) $\mathrm{f}\left(\mathrm{x}\right)=\mathrm{x}/\left(1+\left|\mathrm{x}\right|\right)$ to limit the output of the hash layer to within the range of (−1, 1). We replace the binary code $b_i$ with the network output value $u_i$ and employ the quantization loss to constrain the value of each element of $u_i$ in the near the discrete value $\pm 1$. Therefore, ${\delta }_{ij}$ is redefined as $0.5{\mathrm{u}}_{\mathrm{i}},{\mathrm{u}}_{\mathrm{j}}=0.5\langle {\mathrm{u}}_{\mathrm{i}}{}^{\mathrm{T}}{\mathrm{u}}_{\mathrm{j}}\rangle$, the paired quantization loss $L_Q$ can be defined as:

$$L_Q={\displaystyle \sum _{i,j\in n}\left({\Vert \left|u_i\right|-1\Vert }_1+{\Vert \left|u_j\right|-1\Vert }_1\right)}$$

(4)

Here, ${\left|\left|\right|\right|}_1$ is the L1 normal form operation, $\left|\right|$ is the absolute value operation, and 1 is a vector with elements of the same scale and size as $u_i$. The total loss function of the deep hash model can be defined as:

$$\begin{array}{cc}\hfill L& =e^{\alpha s_{ij}}\left(C_{ij}{L}_{hard}+\beta \left(1-C_{ij}\right)L_{soft}\right)+\gamma L_Q\hfill \\ \hfill & ={\displaystyle \sum _{i,j\in n}{e}^{\alpha s_{ij}}[C_{ij}\left(\log \left(1+e^{{\delta }_{ij}}\right)-s_{ij}{\delta }_{ij}\right)+\beta \left(1-C_{ij}\right){(\frac{u_i^{{\rm T}}{u}_j+q}{2}-s_{ij}q)}^2]}\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\gamma ({\Vert \left|u_i\right|-1\Vert }_1+{\Vert \left|u_j\right|-1\Vert }_1)\hfill \end{array}$$

(5)

In a nutshell, our model extracts image features through a convolutional neural network and generates binary codes through a hash layer; finally, we design a new loss function to better maintain the multi-label semantic information of hash learning contained in complex remote sensing image scene so as to improve retrieval performance.

4. Experiments and Analysis

4.1. Dataset and Evaluation Metrics

4.1.1. Dataset

The first dataset we used was DLRSD (Dense Labeling Remote Sensing Dataset) [16]. The label data of each image in the DLRSD image library was a segmentation image. We analyzed the segmentation map to extract the multi-label of the image. DLRSD has richer annotation information, as illustrated in Figure 3.

Table 2. Annotation description of DLRSD dataset.

Label ID	Category	Number of Images	Label ID	Category	Number of Images	Label ID	Category	Number of Images
1	airplane	100	7	dock	100	13	sea	101
2	bare soil	754	8	field	103	14	ship	103
3	building	713	9	grass	977	15	tank	100
4	car	897	10	mobile home	102	16	tree	1021
5	chaparral	116	11	pavement	1331	17	water	208
6	court	105	12	sand	291

shows the 17 categories and corresponding label IDs of the DLRSD image dataset. The UCMerced dataset is the second dataset used in our experiment. Compared with DLRSD, UCMerced dataset containes 17 categories, and each sample number in each category is 100, it is annotated by Chaudhuri et al. [13]. The predecessor of the image dataset UCMerced is the UCM remote sensing image dataset consisting of 2100 images with a spatial resolution of 30 cm/pixel, an image size of 256 × 256.

Table 3. Annotation description of UCMerced dataset.

Label ID	Category	Number of Images	Label ID	Category	Number of Images	Label ID	Category	Number of Images
1	airplane	100	7	dock	100	13	sea	100
2	bare soil	633	8	field	106	14	ship	102
3	building	696	9	grass	977	15	tank	100
4	car	884	10	mobile home	102	16	tree	1015
5	chaparral	119	11	pavement	1305	17	water	203
6	court	105	12	sand	389

shows the categories and corresponding label IDs of the UCMerced dataset.

4.1.2. Evaluation Metrics

Metrics used to evaluate the performance of the multi-label image retrieval model in our experiments include Accuracy, Recall, Precision, F1-measure and Hamming Loss. Hamming loss directly counts the number of incorrect labels. For these five evaluation metrics, the smaller the value of HL, and the larger the value of Accuracy, Precision, Recall and F1, the better the model is. Among them, F1 comprehensively considers the two metrics of Precision and Recall.

4.2. Experimental Setup

We use pytorch and Adam optimizer to train our network. The GPU and CPU configuration of the computer are NVIDIA GRID M60-8Q and quad-core Intel(R) Xeon(R) CPU E5-2687W v4, respectively. A total of 80% of the images are used as the training set, and the remaining 20% images are used for validating the retrieval performance. We use pre-training weights to fine-tune the weight parameters of Conv_1 to Conv_5, Fc_6, and Fc_7, and set the initial learning rate to 10⁻⁵. Hash layer Fc_8 sets the initial learning rate to 10⁻³. In the selection of parameters, commonly used values were set for multi-label retrieval. The model training batch size is set to 48, the learning rate decreases by 0.5 every 500 iterations, and the total training epoch is 80.

4.3. Parameter Analysis

In the deep hashing model, $\mathsf{\alpha },\mathsf{\beta },\mathsf{\gamma }$ are three important hyperparameters, which heavily affect the performance of the model. Among them, $\mathsf{\alpha }$ is the value of the weighting coefficient that affects the pairwise similarity loss; $\mathsf{\beta }$ is used to adjust the difference in the range of mean square error and cross entropy loss; $\mathsf{\gamma }$ is the hyperparameter that controls the quantization loss.

Of these three parameters, $\mathsf{\beta }$ has the greatest impact on metric learning, so we first set $\mathsf{\gamma }=1.0$ and $\mathsf{\alpha }=0.5$; which are the most commonly used values in hashing learning methods for exploring the impact of hyperparameters on the retrieval performance of the deep hash model.

Table 4. Retrieval performance of our deep hashing model under different values (48-bit hash code).

$\mathsf{\beta }$	HL	Accuracy	Precision	Recall	F1
0	0.1044	0.6498	0.7769	0.7597	0.7482
0.1/q	0.090	0.6942	0.8059	0.8016	0.7866
0.5/q	0.0796	0.7269	0.8217	0.8337	0.8129
0.8/q	0.0792	0.7291	0.8196	0.8370	0.8136
1.0/q	0.0779	0.7354	0.8245	0.8452	0.8197
1.2/q	0.0782	0.7301	0.8212	0.8407	0.8151
5.0/q	0.0808	0.7262	0.8153	0.8340	0.8095

lists the retrieval results when $\mathsf{\beta }$ are {0, 0.1/q, 0.5/q, 1.0/q, 1.2/q, 5.0/q}, and here q means the length q of the hash code.

From the retrieval results shown in Table 4, it can be seen that when there is no paired soft similarity loss ($\mathsf{\beta }=0$), the retrieval results of multi-label remote sensing images are the worst. This finding demonstrates that the paired soft similarity loss has a positive effect on multi-label remote sensing image retrieval. At the same time, when $\mathsf{\beta }=1.0/\mathrm{q}$, the retrieval results are the best. Therefore, in subsequent experiments, $\mathsf{\beta }$ are set as $1.0/\mathrm{q}$.

To explore the impact of hyperparameters $\mathsf{\alpha }$ on the multi-label remote sensing image retrieval performance of the proposed model, we then take $\mathsf{\alpha }=\left\{0,0.1,0.5,1.0\right\}$. The experimental results are shown in

Table 5. Retrieval performance of our deep hashing model under different values (48-bit hash code).

$\mathsf{\alpha }$	HL	Accuracy	Precision	Recall	F1
0	0.0797	0.7294	0.8209	0.8394	0.8151
0.1	0.0795	0.7279	0.8218	0.8368	0.8134
0.5	0.0779	0.7354	0.8245	0.8452	0.8197
1.0	0.0787	0.7327	0.8216	0.8443	0.8171

. Through the retrieval results shown in Table 5, the performance of the deep hash model is optimal when $\mathsf{\alpha }=0.5$. Therefore, in the deep hashing model we proposed, the hyperparameter $\mathsf{\alpha }$ in the loss function takes a value of 0.5.

In the case of $\mathsf{\beta }=1.0/\mathrm{q}$ and $\mathsf{\alpha }=0.5$, the value ranges of the paired hard similarity loss and the paired soft similarity loss are (0, $0.5\sqrt{\mathrm{e}}\times \mathrm{q}$) and [0, $\sqrt{e\times \mathrm{q}}$) respectively. At the same time, the role of quantization loss in our deep hashing model is to constrain the network output value range and has a slight impact on the objective function. So we take $\mathsf{\gamma }$= {0, 0.01, 0.1, 1.0} to determine the value of the hyperparameter $\mathsf{\gamma }$.

Table 6. Retrieval performance of our deep hashing model under different values (48-bit hash code).

$\mathsf{\gamma }$	HL	Accuracy	Precision	Recall	F1
0	0.0800	0.7286	0.8224	0.8376	0.8144
0.01	0.0782	0.7342	0.8249	0.8435	0.8183
0.1	0.0785	0.7329	0.8225	0.8415	0.8168
1.0	0.0779	0.7354	0.8245	0.8452	0.8197

lists the retrieval results of the deep hashing model under different $\mathsf{\gamma }$. It can be seen from the experimental results that the retrieval performance of the model is the best when the value $\mathsf{\gamma }$ is 1.0.

4.4. Experimental Results

4.4.1. Comparison with Baseline Models

To evaluate the effectiveness of the loss function designed in our model, we first verify our model without the fully connected jump connection, and we mark it DHMR/fc. Similarly, we mark the whole model htat incorporates the output feature of the first fully connected layer as DHMR. We compared DHMR/fc and DHMR with current representative multi-label hash methods in natural image retrieval, including IDHN Model [21], improved ISDH model [22] (in the experiments, the network is marked as Im-ISDH), DMSH model [23], DHN model [24] and the DPSH model [25]. For fair comparisons, all deep hashing models use AlexNet as the base network, while keeping the parameters of network training consistent. Furthermore, we also extended the model DHMR to the VGG16 network to explore the scalability and compare our method to the state-of-art method in remote sensing image multi-label retrieval.

It can be seen from the sub-graphs in Figure 4 that the accuracy of DHMR/fc is significantly higher than that of other models. This demonstrates the effectiveness of our designed loss function. In Figure 4, DHMR exhibited a better performance compared with other models when the number of hash output bits are {12, 24, 36, 48, 56, 64, 72, 84, 96}. It can be seen from Figure 4a–d that DHMR shows the best retrieval performance under the evaluation metrics: Accuracy, Precision, Recall and F1. This result suggests that connecting the first fully connected layer to the hash layer has a positive effect on the retrieval performance. In addition, with the increase of the number of hash codes, the retrieval performance of each deep hashing algorithm has been improved. This shows that the greater the number of output hash codes, the more information of the original image is contained.

Moreover, among the comparative models, the DMSH model shows the worst retrieval performance, which might be that this model uses L2 paradigm to measure the distance between hash codes and the similarity between blurred image pairs. The reason the Im-ISDH model performs unsatisfying is the unbalanced ratio of the large hard similarity cross-entropy loss and the small soft similarity mean square error loss, which is not conducive to the training of model.

Further comparison of the retrieval performance of different models when bits = 48 and bits = 96 is shown in

Table 7. Retrieval performance of each algorithm when bits = 48 and 96.

	HL	Accuracy	Precision	Recall	F1
DHMR	0.0779	0.7354	0.8245	0.8452	0.8197	bits = 48
DHMR/fc	0.0779	0.7335	0.8238	0.8425	0.8182
IDHN	0.0809	0.7232	0.8193	0.8321	0.8096
Im-ISDH	0.1016	0.659	0.7811	0.7688	0.7546
DMSH	0.1161	0.6372	0.7423	0.7676	0.7388
DHN	0.093	0.693	0.798	0.8062	0.786
DPSH	0.0914	0.6955	0.7992	0.8086	0.7879
DHMR	0.0761	0.7413	0.8303	0.8505	0.8248	bits = 96
DHMR/fc	0.0756	0.7402	0.8303	0.8482	0.8246
IDHN	0.0762	0.7361	0.8259	0.8434	0.8201
Im-ISDH	0.1059	0.6467	0.7704	0.759	0.7449
DMSH	0.113	0.6489	0.7532	0.7789	0.7492
DHN	0.0866	0.7071	0.8093	0.8184	0.7981
DPSH	0.0861	0.7083	0.8117	0.8184	0.7991

. It can be seen that when bits = 48, DHMR shows the best retrieval performance. HL, Accuracy, Precision, Recall and F1 are 0.0779, 0.7354, 0.8245, 0.8452, 0.8197, respectively, which is superior to all of the other five models and DHMR/fc.

Besides, it can be noticed that the performance when bits = 96 is superior to the case when bits = 48. This can be explained by the fact when the bit increased, the image content features are more specific and comprehensive. When bits = 96, DHMR also shows the best retrieval performance. HL, Accuracy, Precision, Recall, and F1 are 0.0761, 0.7413, 0.8303, 0.8505, 0.8248, respectively.

4.4.2. Performance on Different Settings and Comparison with SOTA Model

(1) performance comparision of different settings in DLRSD

We extend the proposed model to VGG16 network to explore its scalability. The comparative models selected in this study include IDHN and MSRCF. The reason we choose IDHN to make comparison is that it has the best performance among the comparative models mentioned above. MSRCF was proposed by Shao et al. in [17]. They used FCN to extract the segmentation of each image as multi-label vectors.

Table 8. Retrieval performance of each algorithm of VGG16 (DLRSD).

	HL	Accuracy	Precision	Recall	F1
DHMR	0.0747	0.7421	0.8315	0.8496	0.8257	bits = 36
DHMR/fc	0.0750	0.7377	0.8291	0.8442	0.8216
IDHN	0.0780	0.7279	0.8313	0.8321	0.8140
DHMR	0.0725	0.7469	0.8374	0.8505	0.8290	bits = 48
DHMR/fc	0.0726	0.7465	0.8375	0.8497	0.8287
IDHN	0.0744	0.7360	0.8373	0.8365	0.8201
MSRCF	0.1051	0.7193	0.7996	0.8830	0.8150

shows the experimental results when bits = 36 and 48. Table 8 shows that by using VGG16 as the backbone network, the performance of DHMR has been significantly improved with 0.5%, 1.2%, 1.3%, 0.5%, 0.9% in HL, Accuracy, Precision, Recall, and F1 when bits = 48, and DHMR is better than IDHN and DHMR/fc. Furthermore, it can be seen from Table 8 that our model is superior to MSRCF, evidenced by increases in HL, Accuracy, Precision, and F1 by 3.26%, 2.76%, 3.78%, 1.4% when bits = 48. These results indicate the effectiveness of combining deep learning and hash learning for multi-label remote sensing image retrieval.

(2) Performance comparision of different settings in UCMerced dataset

Table 9. Retrieval performance of each algorithm of VGG16 (UCMerced).

	HL	Accuracy	Precision	Recall	F1
DHMR	0.0805	0.7253	0.8153	0.8372	0.8128	bits = 36
DHMR/fc	0.0807	0.7221	0.8232	0.8333	0.8135
IDHN	0.0857	0.5385	0.7568	0.7621	0.7355
DHMR	0.0778	0.7276	0.8278	0.8378	0.8139	bits = 48
DHMR/fc	0.0761	0.7269	0.8274	0.8369	0.8134
IDHN	0.0802	0.6350	0.7589	0.7647	0.7378
DAS-RHDIS	--	0.5680	0.6530	0.7000	0.6750

shows the results on the UCMerced data set. From Table 9, we can see that our model DHMR is better than the IDHN model that performed best in the previous experiments, because we add a similarity weighted loss function to better use the similarity between the images. In addition, DHMR is better than DHMR/fc, which shows the effectiveness of the design of jump connection. On the whole, the performance of the model improves with the increase of bits, that is, when bit = 48, the performance is better than bit = 36. This means that the more hash code bits, the more specific the description of the image content characteristics. Finally, we compared with the latest model DAS-RHDIS, which is a method of selecting a representative triplet from multi-label training images [34]. Our model has shown its advantages in the evaluation metrics of Accuracy, Precision, Recall and F1, which have improved 15.96%,17.48%,13.78% and 15.69%. This may be due to the loss function we designed can better maintain the semantic similarity information.

4.4.3. Multi-Label Image Retrieval Instances

To visually show the multi-label remote sensing image retrieval results of different models, we provide the top 5 images returned by DPSH, DHN, DMSH, Im-ISDH, IDHN, DHMR/fc and DHMR from the DLRSD dataset. Figure 5 and Figure 6 show some retrieval examples on three categories including dense residential and sparse residential.

Each row in Figure 5 and Figure 6 represents the top five retrieval images of different models including DPSH, DHN, DMSH, Im-ISDH, IDHN, DHMR/fc, and DHMR model. The left image in each row is the query image. The title on the bottom of the image is the feature category (multi-label true value) contained in the image. The concrete information about categories and label ID can be referred to Table 2.

In Figure 5 and Figure 6, we choose two complex scenarios, i.e., dense residential and sparse residential, as query images. Search results returned by DHMR and DHMR/fc are more similar to the query image than other models. For instance, in Figure 5, the fourth image returned by DMSH belongs to tennis court; in Figure 6, the third image returned by Im-ISDH belongs to storage tanks and the fourth image returned by IDHN belongs to baseball diamond.

5. Conclusions and Prospects

In this paper, we propose a semantic-preserving deep hashing model for multi-label remote sensing image retrieval. Deep learning and hash learning are integrated for improving efficiency and reducing storage in the complex remote sensing images retrieval without losing accuracy, which is of critical importance in the era of big data.

In our model, first we use convolutional neural networks to extract image features, and then jump-connect the first fully connected layer to the hash layer to fully mine the multi-label semantic information contained in remote sensing images. The experimental results demonstrate the effectiveness and superiority of our model. Our attempt to introduce hash learning into the research of multi-label retrieval of remote sensing images is conducive for real-time environmental applications such as environmental detection, emergency response and many other different fields.

Our future work will be directed to applying graph neural network into our hash model to mine the semantic relationship between multi-label of remote sensing images to further improve performance of multi-label remote sensing images retrieval.