Survey of Cross-Modal Person Re-Identification from a Mathematical Perspective

Abstract

Person re-identification (Re-ID) aims to retrieve a particular pedestrian’s identification from a surveillance system consisting of non-overlapping cameras. In recent years, researchers have begun to focus on open-world person Re-ID tasks based on non-ideal situations. One of the most representative of these is cross-modal person Re-ID, which aims to match probe data with target data from different modalities. According to the modalities of probe and target data, we divided cross-modal person Re-ID into visible–infrared, visible–depth, visible–sketch, and visible–text person Re-ID. In cross-modal person Re-ID, the most challenging problem is the modal gap. According to the different methods of narrowing the modal gap, we classified the existing works into picture-based style conversion methods, feature-based modality-invariant embedding mapping methods, and modality-unrelated auxiliary information mining methods. In addition, by generalizing the aforementioned works, we find that although deep-learning-based models perform well, the black-box-like learning process makes these models less interpretable and generalized. Therefore, we attempted to interpret different cross-modal person Re-ID models from a mathematical perspective. Through the above work, we attempt to compensate for the lack of mathematical interpretation of models in previous person Re-ID reviews and hope that our work will bring new inspiration to researchers.

1. Introduction

Person re-identification (Re-ID) [1] originates from multi-target multi-camera tracking [2], the purpose of which is to retrieve pedestrians’ identities from non-overlapping cameras, as shown in Figure 1. Recently, most Re-ID models [3,4] have been trained on visible image datasets. However, single-modal person Re-ID models are not applicable to complex situations in real scenarios. For example, suspects often conduct criminal activities at night, and visible-light imaging equipment has difficulty capturing the appearance information of pedestrians under low-light conditions. By contrast, infrared imaging equipment is not affected by light conditions and can collect clearer pedestrian images at night. Therefore, for person Re-ID across daytime and nighttime scenes, cross-modal retrieval between infrared and visible images can improve the performance of a person Re-ID model; such a model is called an infrared–visible cross-modal person Re-ID model [5]. Further, if a surveillance camera is damaged or is not installed in the place in which a suspect conducts criminal activities, it is necessary to match witnesses’ textual descriptions of the suspect’s characteristics with pedestrian images captured on other occasions; this is called text–image cross-modal person Re-ID [6]. In addition, a sketch of a suspect can also be drawn according to a description from a witness, and the sketch can be matched with pedestrian images captured on other occasions; this is called sketch person Re-ID [7]. Moreover, to overcome the difficulty of person Re-ID caused by changes in pedestrians’ clothing, a depth camera can be used to obtain the structural information of pedestrians, such as their body shapes and motion signatures; this is called visible–depth person Re-ID [8]. Cross-modal person Re-ID expands the application scope of the person Re-ID task.

However, cross-modal person Re-ID suffers from modal differences. Owing to the large differences in pedestrian features in different modalities, it may be that the differences in the modal features of the same pedestrian are larger than those in the identity features of different pedestrians. This will lead to pedestrians with different identities but the same modal being classified into one category and pedestrians with the same identity but different modalities being classified into different categories. Therefore, the key problem to be solved in cross-modal pedestrian Re-ID is the elimination of modal differences and the extraction of modality-invariant features. Taking infrared–visible cross-modal pedestrian Re-ID as an example, the main methods for reducing modal differences are generation-based and feature-alignment-based methods. Methods based on generation use a generative model to transform infrared and visible images into the same style to reduce the modal differences between them. Methods based on feature alignment are used to map images of different modalities to the same feature space and narrow the feature distribution of pedestrians with the same identity at the feature level to extract modality-invariant features.

Although cross-modal person Re-ID models have made some progress, their performance is still inferior to that of single-modal Re-ID models, which means that the research prospects of cross-modal person Re-ID are very promising. However, we find that the current reviews primarily focus on single-modal person Re-ID, and systematic and profound reviews are lacking for cross-modal person Re-ID. Moreover, current mainstream cross-modal person Re-ID methods are based on deep networks, which resemble a black box. Therefore, we attempted to analyze the existing cross-modal person Re-ID studies from a mathematical perspective, hoping to bring new inspiration to researchers. The contributions of this study are as follows.

• A detailed review of cross-modal person Re-ID studies, which fills the gaps in previous works, is performed. Based on the modalities of data, we classify cross-modal person Re-ID into visible–infrared, visible–depth, visible–sketch, and visible–text person Re-ID. Moreover, from the perspective of mitigating modal differences, we classify cross-modal person Re-ID methods into style-transformation-based methods, shared-feature-mapping-based methods, and auxiliary-information-based methods.
• An analysis of cross-modal person Re-ID methods from a mathematical perspective highlights the importance of mathematical constraints. Constrants from mathematical theory can free deep learning networks from uncontrollable defects and make black-box-like neural networks more explanatory.

In the remainder of this paper, we describe our work according to the following organization: Section 2 reviews the history of cross-modal person Re-ID, previous survey works, and datasets; Section 3 reviews current mainstream deep-learning-based methods; Section 4 presents the conclusions.

2. Literature Overview

In 2006, person Re-ID [9,10,11,12,13] was formally proposed as a separate task for the first time [14], instead of only as a subtask of multi-camera tracking [15,16,17]. However, in the early stages of development, both low-level features (e.g., color and texture) and high-level features (e.g., semantic and structural) were extracted by handcrafted methods based on mathematical theories, such as weighted color histograms (e.g., lab color histograms) [18], edgel histograms [19], texture histograms (e.g., local binary pattern texture histograms) [20], maximally stable color regions [21], and recurrent high-structured patches [22]. These feature extraction methods completely rely on manual design and can lead to stable and reliable experimental results. However, their computational process is extremely complicated. With the rapid development of machine learning, end-to-end deep learning networks have shown excellent performance and have greatly reduced the difficulty of feature extraction.

In 2014, Li et al. [23] completed person Re-ID by using a convolutional neural network (CNN) for the first time and achieved a leap from manual feature extraction [24] to deep-learning-network-based feature extraction [25,26] in this field. However, until 2017, most of the studies in the person Re-ID field were conducted based on single-modal visible image datasets. Therefore, person Re-ID models could not realize cross-modal retrieval between visible images captured under normal light conditions and infrared images captured under low-light conditions, which limited the application scenarios of the person Re-ID task. Therefore, Wu et al. [27] proposed a multimodal dataset called $SYSU\text{-}MM01$, which contains a large number of visible and infrared images. Since then, an increasing number of researchers have turned to infrared–visible cross-modal person Re-ID and extended a series of cross-modal person Re-ID tasks, such as visible–depth, visible–sketch, and visible–text person Re-ID tasks, to meet the actual needs. By counting the number of person Re-ID papers published by the three top conferences in the computer vision field, we obtained the results shown in

Table 1. The number of pedestrian Re-ID articles published by the top three conferences.

	ICCV		ECCV		CVPR
	Single-Modal	Multi-Modal	Single-Modal	Multi-Modal	Single-Modal	Multi-Modal
2017	12	1	-		11	-
2018	-		13	1	28	-
2019	29	1	-		19	2
2020	-		21	1	22	2
2021	14	4	-		21	4
2022	-		-		14	4

.

The proportion of cross-modal person Re-ID studies has been increasing since 2018, but related survey work is absent. To fill the gap in previous surveys of person Re-ID works, we primarily analyze cross-modal person Re-ID works and elaborate on the problems caused by completely relying on deep learning networks. Moreover, we highlight the indispensability of mathematical theories.

2.1. Analysis of Previous Reviews

To enable a quick understanding of person Re-ID methods, many surveys have provided a careful and systematic overview of previous representative works. We provide a brief overview of these works in

Table 2. A summary of recent reviews.

Year	Survey	Contribution
2022	Deep Learning for Person Re-Identification: A Survey and Outlook [33]	A comprehensive and in-depth analysis of the pedestrian Re-ID work. Meanwhile, a baseline named AGW and a new assessment method that was introduced to complement mAP were proposed and achieved excellent performance in four types of Re-ID work.
2020	A Brief Survey of Deep Learning Techniques for Person Re-Identification [32]	Summary of the architectures used in pedestrian re-identification work and classification thereof into five categories according to the differences in architectures: a CNN, RNN, deep belief neural network, deep stacking network, and autoencoder.
2020	A Comprehensive Overview of Person Re-Identification Approaches [31]	Summary of the datasets and metric methods used for pedestrian re-identification tasks in detail and classification of the feature extraction means according to the purpose of the task.
2020	A Survey of Open-World Person Re-Identification [30]	Analysis of the open-world pedestrian re-identification work for the first time and clarification of the definition, realization, and development of the former by comparing open-world works with closed-world works. At the same time, the three development directions of generalization, scalability, and tolerance of clothing changes were summarized for open-world work.
2019	A Survey on Deep-Learning-Based Person Re-Identification Systems [29]	The work on deep-learning-based pedestrian re-identification was classified in terms of the differences in the data used and was divided into three categories: picture-based, video-based, and picture-to-video-based.
2019	Deep-learning-based methods for person re-identification: A comprehensive review [28]	Six types of models in the field of pedestrian re-recognition were analyzed in detail: depth-based recognition models, depth-based verification models, distance-metric-based models, part-based models, video-based models, and data-augmentation-based models.

. In 2019, Wu et al. [28] provided a detailed overview of six types of deep models in the field of person Re-ID (i.e., identification, verification, distance metric-based, part-based, video-based, and data-augmentation-based deep models) and concluded that more high-quality training samples should be generated in the future to improve the performance of person Re-ID models while focusing on the relationship between different granularity features. In addition, this survey indicated that integrating target detection with Re-ID and using multimodal data are developmental trends for the future. However, the description of the datasets and classification of related works in this survey were not accurate. In the same year, Almasawa et al. [29] classified person Re-ID into three categories based on the training data: image-based, video-based, and image-to-video-based. In addition, image-based works were further divided into single-modal and multimodal person Re-ID. This work demonstrated person Re-ID datasets well, which enabled researchers to gain a more intuitive understanding of the application scenarios of person Re-ID. The above-mentioned studies did not consider that person Re-ID datasets can be further divided into open-world and closed-world datasets. Then, Leng et al. [30] conducted an in-depth analysis of open-world works, which remedied the shortcomings of the previous studies in this regard. Unlike previous surveys, which focused on closed-world studies, their survey provided a comprehensive review of open-world studies for the first time; this contributed to the application of person Re-ID in the real world. However, this survey did not fully describe the most typical cross-modal person Re-ID in an open world. Similarly, Wang et al. [31] categorized person Re-ID works according to feature extraction methods and metric methods and summarized some of the mathematical theories. However, they also did not focus on cross-modal studies. Mathur et al. [32] reported that CNNs have improved the performance and efficiency of person Re-ID, but that they have also produced some inevitable problems (e.g., difficulty designing optimal features, over-reliance on labels, requiring adequate hardware resources, and overfitting to the training data). They then classified person Re-ID studies according to their methods to avoid the above problems. Ye at el. [33] published an in-depth and systematic survey that illustrated that open-world person Re-ID would be the future direction. They also proposed more effective baseline and metric losses. However, due to the lack of a mathematical foundation, newcomers cannot gain a deeper understanding based on these surveys and will ignore the use of mathematical theories in future work. Therefore, we summarize and analyze the cross-modal person Re-ID from a mathematical perspective.

2.2. Comparison of Datasets

To provide researchers with a more comprehensive understanding of person Re-ID datasets, we classified them from different perspectives. Based on the type of data, we classified them into image-based and video-based datasets. According to the number of modalities contained in the datasets, we classified them into single-modal and multimodal datasets. According to the type of modality, we further classified the multimodal datasets into RGB–infrared (IR), RGB–depth, RGB–text, and RGB–sketch datasets. To the best of our knowledge, this is the most complete compendium of multimodal datasets to date.

2.2.1. Single-Modal Datasets

Single-modal datasets contain only RGB data captured by visible-light cameras. As shown in

Table 3. Single-modal datasets.

Dataset	Year	Res	Identities/Images(Tracks)/Cameras	Types
VIPeR	2007	fixed	632/1264/2	image
iLIDS	2009	vary	119/476/2	image
GRID	2009	vary	250/1275/8	image
PRID2011	2011	fixed	200/1134/2	image
PRID-2011	2011	fixed	200/400/2	video
CUHK01	2012	fixed	971/3884/2	image
CUHK02	2013	fixed	1816/7264/10	image
CUHK03	2014	vary	1467/13,164/2	image
iLIDS-VID	2014	vary	300/600/2	video
Market-1501	2015	fixed	1501/32,668/6	image
MARS	2016	fixed	1261/20,715/6	video
DukeMTMC	2017	fixed	1404/36,411/8	image
Airport	2017	fixed	9651/39,902/6	image
MSMT17	2018	vary	4101/126,441/16	image
Duke-Video	2018	fixed	1812/4832/8	video
Duke-Tracklet	2018	fixed	1788/12,647/8	video
LPW	2018	fixed	2731/7694/4	video
LS-VID	2019	fixed	3772/14,943/15	video

, limited by the conditions of the hardware, the initial person Re-ID datasets [34,35,36,37,38,39,40,41] usually contained a small amount of nonconsecutive image-based data to eliminate the need for high-performance equipment. With the improvement of hardware performance, training person Re-ID models with video datasets [23,42,43,44,45,46,47] is gradually becoming a trend. Video-based datasets for person Re-ID have the following advantages: (1) less workload—the original unfiltered video sequence is used directly without data filtering, which can reduce manual operations; (2) richer information—compared with image-based datasets, video-based datasets with more information allow the network to be fully trained; (3) more temporal information—video-based datasets can completely preserve gait information, which makes it possible to mine temporal information. Nowadays, the deep-learning-based person Re-ID methods outperform methods that use handcrafted features in terms of accuracy. However, it has been found that if person Re-ID models are trained on closed-world datasets, the obtained models tend to have poor generalization when applied to open-world datasets. Taking TCLNet [48] as an example, its rank-1 accuracy could reach 96.2% on the video-based RGB dataset Duke-Video, but only 52.13% on the video-based RGB–IR cross-modal dataset VCM. Considering that cross-modal person Re-ID has wider application possibilities, an increasing number of researchers have begun to explore how to train models with multimodal datasets and extract modality-invariant features.

2.2.2. Multimodal Datasets

Multimodal person Re-ID datasets include RGB–IR [27,49,50], RGB–depth [51,52,53,54], RGB–text [55], and RGB–sketch [56] datasets. Unlike single-modal person Re-ID, cross-modal Re-ID requires cross-retrieval among data from different modalities. For example, for an infrared–visible person Re-ID, visible images are needed to retrieve pedestrians photographed in infrared conditions, and IR images are needed to retrieve pedestrians captured by visible cameras. In addition, to address the lack of pedestrian visual data, text–image Re-ID and sketch Re-ID utilize text and sketches, respectively, to retrieve pedestrians captured under visible-light conditions. To overcome the difficulty in person Re-ID caused by changes in pedestrians’ clothing, a depth camera can be used to obtain the structural information of pedestrians, such as bones. Therefore, in RGB–depth person Re-ID, it is necessary to match the visible and depth images. Parts of multimodal datasets are shown in Figure 2. As can be seen in Figure 2, there are significant modal differences between IR, sketch, depth, text data, and visible images. Therefore, it is a challenging problem to reduce the modal differences in cross-modal person Re-ID.

Cross-modal datasets play a crucial role in the task of cross-modal person Re-ID, and cross-modal person Re-ID datasets are summarized in this paper, as shown in

Table 4. Multimodal datasets.

Dataset	Year	Boxes	Identities/Data/Cameras	Types
RegDB	2017	128 × 64	412/8240/2	image-based RGB–IR
SYSU-MM01	2017	vary	491/303,420/6	image-based RGB–IR
VCM	2022	vary	927/485,122/12	video-based RGB–IR
PAVIS	2012	-	79/79/2	image-based RGB–depth
BIWI	2014	1280 × 960	50/50/2	image-based RGB–depth
IAS-Lab	2014	-	11/11/2	image-based RGB–depth
DPI-T	2016	-	12/300/5	image-based RGB–depth
CUHK-pedes	2016	-	13003/120,618/2	image-based RGB–text
ICFG-PEDES	2021	-	4102/54,522/37.2	image-based RGB–text
SKETCH	2018	-	200/600/3	image-based RGB–sketch

. It can be seen in Table 4 that the number and scale of the cross-modal datasets are relatively small compared with those of the single-modal datasets. For example, the Sketch Re-ID dataset contains only 200 people, each with one sketch and two photos. This limits the performance of cross-modal person Re-ID models. Therefore, expanding the scale of cross-modal datasets is also a basic task that needs to be performed in the field of cross-modal person Re-ID in the future.

3. Summary of Deep-Learning-Based Cross-Modal Person Re-ID Methods

The process of person Re-ID can be roughly divided into three parts: data collection [57,58,59], feature extraction [60,61,62], and pedestrian matching [63,64,65]. In deep-learning-based methods, off-the-shelf target detection tools are typically used for personal data collection. CNNs, recurrent neural networks, deep belief networks, deep stacking networks, and autoencoders have been used for discriminative feature extraction. Representational and metric learning are used for pedestrian matching. In cross-modal person Re-ID, feature extraction and pedestrian matching must be studied according to the characteristics of different modalities.

3.1. Types of Cross-Modal Person Re-ID

3.1.1. Visible–Infrared Re-ID

At present, single-modal visible-light cameras with poor imaging effects at night are gradually being replaced by dual-modal cameras with an adaptively switching imaging mechanism. Dual-modal cameras can use a visible-light camera to capture color images when the light is sufficient and use an IR camera to capture black-and-white images when the light is poor. Although a dual-modal camera can compensate for the limitation that the visible-light camera cannot work properly when the light is insufficient, visible images captured by visible-light cameras and IR images captured by infrared cameras have considerable modal differences, which introduce challenges for the visible–IR person Re-ID. To improve the accuracy of cross-modal person Re-ID, visible–IR person Re-ID should not only solve the problems faced by traditional person Re-ID (such as posture changes, angle changes, light changes, and local occlusion), but also strive to overcome imbalances in color information between the different modalities.

3.1.2. Visible–Depth Re-ID

A depth image is also called a range image, the pixel value of which is the distance from the camera to each point in the scene. A depth image directly reflects the geometric shape of a target object. Therefore, it can be used to easily solve many problems related to 3D objects and is primarily used for information acquisition in the field of motion capture. Similarly, depth images are widely used in person Re-ID to capture the body shapes and sketch information of pedestrians. In this manner, depth and visible images are fused to overcome changes in illumination and color. Ren et al. [66] proposed a multimodal uniform deep-learning method to accomplish person Re-ID. This method combined anthropometric measures and body shape information contained in depth images with appearance information contained in visible images through a multimodal fusion layer to obtain pedestrian features that were robust to noise. To further adapt to different application scenarios, Imani et al. [67] used RGB, depth, and skeleton modules to realize person Re-ID and attempted to find a modal combination that was suitable for different datasets. To overcome the effect of viewpoint variation on depth devices, Wu et al. [68] designed a depth voxel covariance descriptor that was insensitive to rotation and noise. However, the above works only considered how to use depth features to complement traditional visual features. Because environmental and clothing changes have no impact on the distance information, a depth camera can compensate for the poor performance of a visible-light camera in a low-light environment, which also provides a new approach for long-term retrieval tasks. However, it also creates a problem: that of matching depth images with visible images. Hafner et al. [8] proposed a cross-modal distillation method to learn the representation space shared by visible images and depth images and implemented visible–depth person Re-ID for the first time. This is the only visible–depth Re-ID model available.

3.1.3. Visible–Text Re-ID

In a case in which a visual image of pedestrians is missing but a textual description can be obtained, it is necessary to construct a network with the ability to perform text–image cross-modal retrieval. Because there is a large modal gap between textual and image data, text–image Re-ID studies primarily focus on enhancing the connection between the two types of data at different levels [69]. Ding et al. [70] captured the relationship between different parts of the body through a multi-view nonlocal network for better correspondence between body parts and noun phrases. Yu et al. [6] extended the re-ranking commonly used in single-modal retrieval to cross-modal retrieval and enhanced the accuracy by constructing a bidirectional coarse-to-fine framework. Inspired by the interactions between images and sentences, Wang et al. [71] proposed cross-modal adaptive message passing to adaptively control the cross-modal information flow. Niu et al. [72] divided features according to granularity and realized cross-modal fine-grained retrieval through global–global, global–local, and local–local alignments. Kansal et al. [73] proposed an end-to-end hierarchical attention image–text alignment network to solve the problems of alignment uncertainty and text complexity. Additionally, this network determined the potential relationship between image content and textual information at different levels by using a hierarchical attention alignment network. Farooq et al. [74] enhanced the correlation between the two modalities by performing a canonical correlation analysis. In addition, attributes are often used in visible–text person Re-ID. Zha et al. [75] proposed an adversarial attribute–text-embedding network to learn discriminative modality-invariant visual and text features. Simultaneously, a visual attribute graph convolutional network was proposed to learn the visual attributes of pedestrians with better descriptiveness, interpretability, and robustness than those of appearance features. In addition to the above studies, some researchers have tried to reduce the heavier computational requirements of visible–text Re-ID work by building weakly supervised models [76] to eliminate reliance on identity annotation.

3.1.4. Visible–Sketch Re-ID

Some researchers have attempted to convert textual descriptions into sketch images [7] to alleviate the modal gap between visual and textual data. For example, the police always sketch portraits based on descriptions from eyewitnesses and match the sketches with visible images from surveillance systems. In 2018, Lu et al. [56] first proposed the visible–sketch Re-ID task and provided the first sketch Re-ID dataset. Similarly to most cross-modal retrieval tasks, they used generative adversarial networks to extract discriminative modality-invariant features in sketches and visible images. Gui et al. [77] extracted domain-invariant features by using a gradient reverse layer. Thus, it is possible to model photos and sketches in a common embedding space to solve the domain gap problem. To bridge the domain gap, unlike in the above-mentioned studies, Yang et al. [78] proposed instance-level heterogeneous domain adaptation to transfer the instance-level knowledge in an inductive transfer manner. A sketch contains only contour information and does not contain color and background information; therefore, it is particularly important to process redundant information of visible images in visible–sketch cross-modal retrieval, which makes sketch Re-ID more challenging than traditional person Re-ID.

3.2. Common Solutions for Cross-Modal Person Re-ID

In cross-modal person Re-ID, the challenge is to narrow the gap between different modal features. To achieve the cross-modal person Re-ID task, researchers have proposed a series of methods for solving the modal difference problem. These methods can be roughly classified into the following three categories. The first transforms the style of input images before feature extraction [5,79] to create consistency in the image styles under different modalities and transform heterogeneous data into traditional homogeneous data. The second embeds the features of different modalities into the same mapping space [80,81,82] to achieve consistency. The third extracts modality-independent discriminative features, such as gait features, to assist in cross-modal person Re-ID [83,84].

3.2.1. Style-Transformation-Based Method

Owing to modal differences, it is difficult to extract modality-invariant features directly from the original data. Therefore, it is particularly important to perform image-level style transformation in advance to bridge the modal gap.

Translation between Original Modalities

Initially, a generative adversarial network (GAN), which was constructed in 2014 by Goodfellow et al. [85], was most commonly used for style transformation. In 2017, Zheng et al. [39] applied GANs to person Re-ID for the first time and expanded their training set by generating unlabeled samples that did not belong to a known class. By using real and generated data simultaneously, semi-supervised training of the network was realized, which effectively avoided the overfitting of network parameters and further improved the network performance. In contrast to single-modal person Re-ID, cross-modal person Re-ID often uses GANs to generate heterogeneous data to enhance the relationship between different modal data or convert the cross-modal task into a single-modal one. In 2020, Wang et al. [86] first applied a GAN to a cross-modal person Re-ID and proposed a cross-modal image-generation method for person Re-ID. The method entangled modality-invariant features by generating cross-modal image pairs so that the set-level and instance-level modality variations were simultaneously reduced.

Owing to the inconsistency between labels in training and test sets, independent pixel-level and feature-level generative adversarial training causes the two to be mismatched in testing, which severely damages a network’s performance. Therefore, Wang et al. [87] used a GAN to generate sufficiently realistic fake IR images to achieve pixel-level alignment and alleviate modal differences. They then performed feature-level alignment to resolve the intra-modal differences. Finally, desirable fake IR images could be obtained by jointly using the above two alignments.

In general, GANs use generators to generate fake images, which are input into the discriminator along with real images to verify the authenticity of the images. Through adversarial learning between the generator and discriminator, the performance of the generator is continuously improved, and it can generate real enough fakes to complete cross-modal work. In visible–IR cross-modal person Re-ID, a GAN was used to convert visible images into the style of IR images, which also meant that the color information in the visible images could not be fully utilized. Unlike the previous one-way fake IR image generation, Zhong et al. [88] aimed to completely preserve the rich discriminative information in visible images. Therefore, they colored single-channel IR images to learn the correspondence between single-channel IR and three-channel visible images.

However, it is impossible to generate fake images that are the same as real images. If distorted fake images are used for network training, network performance is adversely affected. Simultaneously, unsupervised generation destroys the local structure of a fake image and introduces a large amount of noise. Therefore, some studies have attempted to corrupt color information by using a homogeneous generation method. For example, instead of using a traditional generator, Zhao et al. [89] used a color jitter operation for clothing color transformation, which could learn discriminative color-independent features and mitigate the effect of the generated noise. In addition, inconsistency in the convergence direction of the parameters of the generator and discriminator could lead to difficulties in network optimization. Thus, the Kullback–Leibler divergence constraint [90,91] can be used to replace the discriminator, which can implicitly promote the gradual convergence of the distributions of the generated and real data.

Translation to a New Intermediate Modality

To eliminate the limitations of GANs, such as noise introduction and convergence difficulties, images of different modalities can be transferred to a common modality. Li et al. [92] built an X modality with multimodal information by transferring knowledge from the visible and IR modalities. This self-supervised generation method was more lightweight and efficient than GAN-based models. In addition, Liu et al. [93] generated intermediate modal data by using a homogeneous generation strategy and resolved the problem of the distortion of heterogeneous generation to a certain extent.

Another method generated an intermediate modality by converting visible images into grayscale images. For example, Ye et al. [94] transformed visible data into homogeneous grayscale data to assist cross-modal feature extraction. In this way, intermediate modality generation did not require an additional training process, and the structural information of the visible images could be completely retained. Because the IR domain was not completely consistent with the gray domain, Liu et al. [95] converted both visible and IR data into grayscale data simultaneously, which eliminated the luminance differences among the IR data during conversion.

However, an unconstrained generation process cannot be controlled. If the generated intermediate data are too similar to the original data, modal differences cannot be mitigated. In general, when using only a deep learning method to automatically learn the parameters of the nonlinear layer without the constraints of the underlying mathematical theory, the network often fails to achieve predictability in practical applications, which is a disadvantage of simply improving the network structure. Unlike previous studies that ignored the underlying mathematical theory, Huang et al. [96] did not consider intermediate modality generation as a general step, but treated it as a Markov decision process [97], which allowed the negative impact of random modality mixing to be successfully avoided.

Mathematical Summary

In general, cross-modal person Re-ID works often use GANs containing both a generator (G) and discriminator (D) to generate images that satisfy the training requirements. For example, the data from modality $M_1$ are used to generate the data from modality $M_2$ through the generator module, or the data from different modalities are fused to generate intermediate modality data. In GANs, the generator is used to generate realistic images to deceive the discriminator and continuously optimize the network to make the discriminative loss as large as possible. At the same time, it is desired to simultaneously improve the discriminator so that it will not be affected by the generator and continuously optimize the network to make the discriminative loss as small as possible. In the above generative process, the performance of the generator and discriminator can be simultaneously improved, and finally, the cross-modal task can be converted into a single-modal task by the generator. The optimization goal of GANs is as follows:

$$\underset{G}{min}\underset{D}{max}V\left(D,G\right)={\mathrm{E}}_{x\sim p_{data}\left(x\right)}\left[logD\left(x\right)\right]+{\mathrm{E}}_{z\sim p_z\left(z\right)}\left[\left(1-D\left(G\left(z\right)\right)\right)\right]$$

(1)

where $p_z\left(z\right)$ is the distribution of the input noise variable, and $p_{data}\left(x\right)$ is the distribution of the original data. In addition, the discriminator is updated by ascending its stochastic gradient, while the generator is updated by descending its stochastic gradient.

However, it is harder to optimize GANs due to the inconsistent gradient convergence direction of the discriminator and generator. Therefore, some works [91,98] used the Kullback–Leibler (KL) divergence constraint to replace the discriminator module and implicitly promote the gradual convergence of the distributions of the modal data $M_1$ and modal data $M_2$ by reducing the scatter value.

$$L_{KL}^{M_1{M}_2}=E_i\sum _{c=0}^{C-1}[p(c|f_i^{M_1})log\frac{p(c|f_i^{M_1})}{p(c|f_i^{M_2})}+p(c|f_i^{M_2})log\frac{p(c|f_i^{M_2})}{p(c|f_i^{M_1})}]$$

(2)

where C is the number of categories in the classification task. p is the probability distribution of the data from different modes. E is the expectation of different category scatter values. ${f_i}^{M_1}$ and ${f_i}^{M_2}$ are features of a person with identity i, which come from modal data $M_1$ and $M_2$, respectively.

Furthermore, directly adding feature maps can also allow the data from different modalities to be fused:

$$x^{mix}=m{f}^{M_1}+\left(1-m\right)f^{M_2}$$

(3)

where m is the mixing ratio, and $f^{M_1}$ and $f^{M_2}$ are feature maps under the modalities $M_1$ and $M_2$, respectively. In order to solve the problem of the fusion process being too random, the calculation process of the mixing ratio can be converted into a Markov decision process:

$$\begin{array}{cc}\hfill & \mathrm{State}:\left[f^{M_1};f^{M_2}\right]\hfill \\ \hfill & \mathrm{Action}:m=SigmoidAvgPool\left(Conv\left(\left[f^{M_1};f^{M_2}\right]\right)\right)\hfill \\ \hfill & \mathrm{Reward}:L_A=-Q(A\left(\left[f^{M_1};f^{M_2}\right]\right),\left[f^{M_1};f^{M_2}\right])\hfill \end{array}$$

(4)

where State, Action, and Reward are the basic steps in the Markov decision process; $[f^{M_1},f^{M_2}]$ is the concatenation of the intermediate feature maps. Meanwhile, the network is updated to obtain a smaller $L_A$ and bigger Q, which implies higher rank-k accuracy and mAP for person Re-ID works. It is worth noting that we want to get a value of m that can produce the smallest $L_A$.

3.2.2. Shared-Feature-Mapping-Based Methods

To mitigate the modal gap and achieve cross-modal person Re-ID, many researchers have attempted to design an ingenious deep-based network to map pedestrian features in different modalities into a common distribution space. Currently, the most commonly used network frameworks for cross-modal person Re-ID are single-stream networks, dual-stream networks, and GANs, as shown in Figure 3.

Single-Stream Networks

AlexNet [99], VGG [100], GoogleNet [101], and ResNet [102] are representative models of traditional single-stream networks, which have only one input and one output. Unlike in a traditional single-stream network, the network inputs in cross-modal Re-ID [27] are pedestrian images with different modalities. A single-stream network often has two different branches, but the branches share parameters. Pedestrian images with different modalities are mapped to the same feature space after processing by the same network.

Dual-Stream Networks

Owing to the large differences between different modal features, single-stream networks destroy the discriminative features of different modalities. Therefore, some studies have begun to use dual-stream networks with partially shared parameters to extract features [103,104,105]. In cross-modal person Re-ID, dual-stream networks always have two branches, with independent parameters in shallower layers and shared parameters in deeper layers. In 2018, Ye et al. [106] first applied dual-stream networks to the field of visible–thermal cross-modal person Re-ID. The proposed end-to-end dual-path feature and metric learning framework not only achieved excellent performance, but also provided a good research basis for future work. However, cross-modal person Re-ID not only needs to address the inter-modal differences, but also needs to deal with the intra-modal perspective differences that exist in traditional person Re-ID. Thus, Ye et al. [107] used hierarchical cross-modal metric learning to constrain the dual-stream network to reduce the impact of view changes on feature extraction. Since then, several cross-modal person Re-ID methods based on this architecture have emerged. These methods extract features from different modal data through the unique convolution layer of each modality and make the feature distribution of different modal outputs gradually consistent under the constraint of a shared convolution layer.

In contrast to the aforementioned learning-based representation methods that focus on designing complex networks to extract ideal features, metric-learning-based methods do not directly treat person Re-ID as a classification problem, but only consider the similarities between pedestrian images. For example, Zhang et al. [108] improved the basic dual-stream network from the perspective of metric learning by mapping features to a hypersphere manifold, thereby eliminating the inconsistency of the norm distributions of different modal features. Ye et al. [109] performed a detailed analysis of triplet loss and proposed a more suitable central triplet loss for metric learning in cross-modal person Re-ID. Unlike in the above studies, which only performed angle transformation in metric learning, Hao et al. [110] noted that it is necessary to strengthen the relationship between the classification subspace and the feature embedding subspace in the spherical space. In their study, a spherical softmax was designed to map the original features, and a two-stage training scheme was used to obtain de-correlated features for classification. Tian et al. [111] found that an information bottleneck, which could preserve useful information and reduce redundant information, achieved good performance in computer vision. However, this method relied heavily on the accurate estimation of mutual information. Therefore, they attempted to provide analytical solutions to fit the mutual information without estimating it and successfully alleviated the above difficulties. Similarly, Meshky et al. [112] noticed the important role of mutual information in mitigating modal effects and extracted intermediate features to retain more mutual information. Furthermore, the entropy value was used to weigh the features of different layers by analyzing the information reflected by it (i.e., the higher the entropy is, the lower discrimination of the features is).

GAN-Based Networks

Inspired by GANs, Dai et al. [113] designed an adversarial network to discriminate the originating modalities of the output features and achieved further alignment of different modal feature distributions to maximize the discriminative loss. Since then, an increasing number of studies have begun to use adversarial ideas for feature-level alignment rather than image-level style conversion [114]. In contrast to previous studies that used a simple discriminator for the network, Hu et al. [115] set up an exclusive identity classifier and modality discriminator for modality-related and identity-related branches separately to disentangle modality-related features. Moreover, to mitigate the randomness of a deep-learning-based network, this method set the representational orthogonal decorrelation at the feature and parameter levels and enforced the process of feature extraction more reliably by using mathematical constraints. To simultaneously address the modal and domain gaps, Fu et al. [116] attempted to reduce the domain gap by using an adversarial method. They calculated the moments for the features in the target domain and achieved cross-modal distribution alignment by closing the moments of the same ID.

Mathematical Summary

Theoretically, a Siamese network can map features of different modalities into the same distribution space, but in practical applications, this capability is usually limited. To obtain features with more consistent distributions, researchers often set up auxiliary networks based on mathematical theories to process the features of different modalities so that they can force them to have similar distributions. For example, features can be further mapped to the corner space [108] in which modal differences are smaller:

$$\begin{array}{cc}\hfill \Theta \left(f^{*};W^{*}\right)& =f^{*}{W}^{*}=\left[f^{*}{w}_1^{*},f^{*}{w}_2^{*},\cdots ,f^{*}{w}_c^{*}\right]\hfill \\ \hfill & =\left[\parallel f^{*}\parallel \times \parallel w_1^{*}\parallel \times cos{\theta }_1,\cdots ,\parallel f^{*}\parallel \times \parallel w_c^{*}\parallel \times cos{\theta }_c\right]\hfill \\ \hfill & =\left[cos{\theta }_1,cos{\theta }_2,\cdots ,cos{\theta }_c\right]\hfill \end{array}$$

(5)

where the angle transformation matrix $W^{*}$ projects feature $f^{*}$ from the feature space to the angular space, and c represents the number of classes in the training data. $W^{*}=[w_1^{*},w_2^{*},\cdots ,w_c^{*}]=[\frac{w_1}{\parallel w_1{\parallel }_2},\frac{w_2}{\parallel w_2{\parallel }_2},\cdots ,\frac{w_c}{\parallel w_c{\parallel }_2}]$, $f^{*}=\frac{f}{{\parallel f\parallel }_2}$.

In addition to transformation at the coordinate level, it is also a feasible method to calculate entropy or moment or to orthogonalize the features at the feature level. For example, by reducing the differences in moments, the consistency of samples that have the same IDs and different modalities can be strengthened [116]. In this way, the modality gap can be implicitly alleviated:

$$\begin{array}{cc}\hfill & {\sigma }_c^{M_1}=\frac{1}{N_c^{M_1}}\sum _{i=1}^{N_c^{M_1}}{\left(f_i^{M_1}-{\mu }^{M_1}\right)}^2\hfill \\ \hfill & {\sigma }_c^{M_2}=\frac{1}{N_c^{M_2}}\sum _{i=1}^{N_c^{M_2}}{\left(f_i^{M_2}-{\mu }^{M_2}\right)}^2\hfill \\ \hfill & d_c^{M_1{M}_2}=D\left({\sigma }_c^{M_1},{\sigma }_c^{M_2}\right)\hfill \end{array}$$

(6)

where $N_c^{M_1}$ and $N_c^{M_2}$ represent the numbers of samples of the c-th identity in modality $M_1$ and modality $M_2$, respectively. c denotes the identity. ${\sigma }_c^{M_1}$ and ${\sigma }_c^{M_2}$ denote the second-order moments (variance). ${\mu }^{M_1}$ and ${\mu }^{M_2}$ represent the prototype (the mean within one class). D denotes the distance function.

For a certain class of features, the moment can reflect the stability of the features in terms of distribution, while the entropy can reflect the accuracy of the features in terms of discriminability. Therefore, we can give a weight w to different features according to their entropy. Thus, the modality-irrelevant features can be enhanced and the modality-relevant features can be weakened:

$$\begin{array}{cc}\hfill & H\left(f\right)=-\sum _{j=1}^{C}P\left(j|f\right)log\left[P\left(j|f\right)\right]\hfill \\ \hfill & w=\frac{1+e^{-H\left(f\right)}}{{\displaystyle \sum _{k=1}^M}1+e^{-H\left(f^k\right)}}\hfill \end{array}$$

(7)

where $H\left(f\right)$ denotes the entropy in the class distribution for the input images with feature f. M denotes the batch size. C is the total number of identities. $P\left(j|f\right)$ represents the probability of identity j given descriptor f.

In order to further separate the modality-irrelevant features $f_c^I$ from the modality-relevant features $f_c^R$, they can be orthogonalized:

$$\begin{array}{c}\hfill {\left(f_c^I\right)}^T{f}_c^R=0\end{array}$$

(8)

where c denotes the identity.

Obviously, all of the above operations are done for the features themselves. Otherwise, a more reasonable metric learning method can be designed for the features. For example, a network can be constrained by a more effective loss function, such as the bi-directional center-constrained top-ranking loss:

$$\begin{array}{cc}\hfill L_{bicenter}& =\sum _{c=1}^{K}max\left[{\rho }_1+D(f_c^{M_1},{\mu }_c)-\underset{\forall c\ne k}{min}D(f_c^{M_1},{\mu }_k),0\right]\hfill \\ \hfill & +\sum _{c=1}^{K}max\left[{\rho }_1+D(f_c^{M_2},{\mu }_c)-\underset{\forall c\ne k}{min}D(f_c^{M_2},{\mu }_k),0\right]\hfill \end{array}$$

(9)

where K is the batch size. ${\mu }_c$ and ${\mu }_k$ are d-dimensional vectors representing the center of class c. $f^{M_1}$ and $f^{M_2}$ are features of modalities $M_1$ and $M_1$, respectively.

3.2.3. Auxiliary-Information-Based Methods

Visual features change significantly due to modal influence. It is impossible to achieve the desired accuracy by relying only on a small amount of modality-invariant discriminative visual information for the identity metric. Therefore, some studies have attempted to explore higher-order relations and temporal information, which are robust to modality changes, to assist identification. Thus, the role of visual information is weakened, and the effect of the modality is mitigated.

Higher-Order Relationship Assistance

Yin et al. [117] built an affinity modeling module to further mine the relationships between the global features of different pedestrians and enhance the ability of network cross-modality matching. However, this method focused more on the relationships between different pedestrians, which essentially enhanced the compactness in the same category and the separability between different categories, making it difficult to constrain the network to the extraction of structural features unrelated to appearance features. To address this issue, Huang et al. [118] used 3DConv to extract discriminative modality-invariant relationships between different parts of the body. With the complement of the relationship features between different parts of the body, the ability of the network to discriminate difficult samples with similar appearances but obvious differences in body shape was further improved. In addition to 3DConv, a transformer is also a common tool for extracting structural information of the human body [119]. To mine more accurate structural information, Gao et al. [120] used key points to assist in feature extraction. This method constructed a graph structure based on the actual structure of the human body and mined the intra-modality and inter-modality structural information of the human body by using cross-modal graph matching.

Temporal Information Assistance

Video-based methods are more advantageous than image-based cross-modal person Re-ID methods. This is because more comprehensive appearance features and modality-independent temporal features can be extracted from video, which is conducive to solving the network performance degradation caused by the modal gap. To fully extract the temporal information in videos, researchers have attempted to use optical flow [121,122], recurrent neural networks [123,124], temporal pooling [44,125,126,127], and 3DConv [128,129,130] to aggregate sequence-level features. For example, Aich et al. [131] used 3DConv with different pooling sizes to exploit temporal information. Hou et al. [132] designed a temporal kernel selection module to adaptively capture short- and long-term temporal relationships. Although the current temporal information extraction methods are relatively mature, few cross-modal person Re-ID methods have used modality-invariant temporal information. MITML [50] is the only method that combines temporal information extraction with cross-modal person Re-ID.

Mathematical Summary

Since visual features are highly susceptible to modal changes, some methods have tried to explore the relationships (e.g., the relationships between different features in a spatial distribution, the relationships between different parts of the same person, the movement rules of individuals, etc.) that are robust to modality changes to assist identity discrimination. Usually, the relationship between features $f_i$ and $f_j$ can be better represented by the similarity matrix A, and the value of A at $\left(i,j\right)$ can be calculated with the following equation:

$$\begin{array}{c}\hfill A_{ij}={exp}^{-dist\left(f_i,f_j\right)}\end{array}$$

(10)

where $dist$ denotes the distance measure function.

Unlike in some works that independently performed the calculation of the relationship matrix before the feature output, a graph convolutional network (GCN) integrates the relationship calculation into the process of feature extraction. Each node of a GCN can not only extract local features as ordinary convolutional networks do, but can also update its own state while receiving messages from neighboring nodes so that each local node can contain more comprehensive global information. Therefore, graph convolution is very suitable for extracting the relationships between local blocks of the human body and outputting the high-order relationship characteristics of the human body. Graph convolution can be expressed as:

$$\begin{array}{c}\hfill H^{l+1}=\sigma \left(A{H}^l{W}^l\right)\end{array}$$

(11)

where $H^l$ and $H^{l+1}$ are features of layer l and layer $l+1$, respectively. A is an adjacency matrix, which is used to identify whether certain edges in the graph exist (for example, $A_{ij}$ represents the edge between node i and node j, and if the edge exists, $A_{ij}=1$; otherwise, $A_{ij}=0$). $W_l$ is a weight matrix for layer l, which can represent the relationship between nodes. $\sigma$ is a nonlinear activation function.

Similarly to graph convolutional networks, recurrent neural networks (RNNs) can selectively pass information from the previous layer to the next layer, so they can also act as information aggregators. RNNs can be represented as:

$$\begin{array}{cc}\hfill h_t& =f\left(U{h}_{t-1}+W{x}_t+b\right)\hfill \\ \hfill y& =V{h}_t\hfill \end{array}$$

(12)

where h is the hidden state, $f\left(\right)$ is the nonlinear activation function, and U, W, V and b are the network parameters.

3.3. Loss Functions

In the training stage, the quality of the features is evaluated at the end of the network. In order to classify output features to an exact category, a clear demarcation between different identities is expected. At the same time, it is expected that the distance between the same identity features is closer than that between different identity features. To achieve this goal, suitable loss functions are needed to optimize the network, as shown in Figure 4. According to the types of loss functions, optimization methods can be divided into metric learning, which is represented by triplet loss ($L_{Tri}$), and representation learning, which is represented by identity loss ($L_{ID}$).

Moreover, it is worth noting that deep-learning-based methods usually update the network parameters through gradient descent:

$$\begin{array}{c}{\theta }_{t+1}={\theta }_t-\alpha \frac{1}{N}\sum _{n=1}^N\frac{\partial L\left(y^{\left(n\right)},f\left(x^{\left(n\right)};\theta \right)\right)}{\partial \theta }\hfill \end{array}$$

(13)

where $\theta$ denotes the network’s parameters. t is the number of optimizations. $\alpha$ is the learning rate. x denotes the training samples. y denotes the label. L is the loss function.

3.3.1. Distance Metric

In order to clearly introduce metric learning and representation learning, we briefly introduce their mathematical models. To measure the distance of features, Re-ID work often uses the Euclidean, Manhattan, Chebyshev, and cosine distance, as shown in Figure 5.

Euclidean Distance (L2 Distance)

The Euclidean distance [133] represents the true distance between two points in n-dimensional space, which can be regarded as the linear distance:

$$\begin{array}{c}\hfill D_e\left(X,Y\right)=\sqrt{\sum _{i=1}^n{\left(x_i-y_i\right)}^2}\end{array}$$

(14)

where $X=[x_1,x_2,\cdots ,x_n]$ and $Y=[y_1,y_2,\cdots ,y_n]$.

Manhattan Distance (L1 Distance)

The Manhattan distance [134] represents the sum of the axial distances between two points X and Y in each dimension, which can be regarded as the rectilinear distance:

$$\begin{array}{c}\hfill D_m\left(X,Y\right)=\sum _{i=1}^n\left|x_i-y_i\right|\end{array}$$

(15)

Chebyshev Distance

The Chebyshev distance [135] represents the maximum of the distance in different axes between two points. The Euclidean distance [133] represents the true distance between two points in the n-th dimension, which can be regarded as the linear distance:

$$\begin{array}{cc}\hfill D_c(X,Y)& =max\left(\right|x_1-y_1|,|x_2-y_2|,\cdots ,|x_n-y_n\left|\right)\hfill \\ \hfill & =\underset{p\to \infty }{lim}{\left(\sum _{i=1}^n{|x_i-y_i|}^p\right)}^{\frac{1}{p}}\hfill \end{array}$$

(16)

Obviously, the Manhattan, Euclidean, and Chebyshev distances can be expressed by a unified formula, i.e., the Minkoff distance formula:

$$\begin{array}{c}\hfill D={\left(\sum _{i=1}^n{\left|x_i-y_i\right|}^p\right)}^{\frac{1}{p}}\end{array}$$

(17)

where $p=1$ is the Manhattan distance, $p=2$ is the Euclidean distance, and $p=\infty$ is the Chebyshev distance.

Cosine Distance

Cosine similarity [136] measures the differences between individuals by calculating the cosine of the angle between two vectors. Compared to the above methods, the cosine distance focuses more on the difference between two vectors in terms of direction rather than distance or length:

$$\begin{array}{cc}\hfill cos\left(\theta \right)& =\frac{X·Y}{∥X∥∥Y∥}\hfill \\ \hfill & =\frac{{\displaystyle \sum _{i=1}^n}{x}_i{y}_i}{\sqrt{{\displaystyle \sum _{i=1}^n}{x_i}^2}\sqrt{{\displaystyle \sum _{i=1}^n}{y_i}^2}}\hfill \end{array}$$

(18)

3.3.2. Metric Learning

The purpose of metric learning [137] is to find an appropriate metric that can make the distance of different identities larger than that of the same identity. In this way, features of each identity can cluster in metric space. In person Re-ID, Euclidean distance and its variants are often used to construct metric learning loss, which can promote feature clustering. The commonly used metric learning losses are as follows.

Contrastive Loss

In contrastive loss [138], if the inputs $X_i$ and $X_j$ are a pair of samples of the same identity, ${\delta }_{ij}=1$; conversely, ${\delta }_{ij}=0$. Therefore, the value of the loss will increase when the distance of samples that come from the same identity is larger or the distance of features that come from different identities is smaller:

$$\begin{array}{c}\hfill L_{con}=\left(1-{\delta }_{ij}\right){\left\{max\left(0,\rho -d_{ij}\right)\right\}}^2+{\delta }_{ij}{d_{ij}}^2\end{array}$$

(19)

where $d_{ij}$ denotes the distance of samples $X_i$ and $X_j$. $\rho$ is the threshold value to be set.

Contrastive loss requires a fixed margin, so a very strong assumption that the distribution of each class of samples is the same is implied here. However, this strong assumption is difficult to hold in general. At the same time, it is difficult to ensure that the distance of features from the same ID is smaller than that of those from different IDs if only the distance between a pair of samples is considered.

Triplet Loss

Unlike contrastive loss, the triplet loss [139] usually contains an anchor sample a, a positive sample p with the same identity as that of a, and a negative sample n with a different identity from that of a. Obviously, the value of the triplet loss will descend when the anchor is closer to the positive sample and distant from the negative sample.

$$\begin{array}{c}\hfill L_{Tri}=max(0,\rho +d_{ap}-d_{an})\end{array}$$

(20)

where $\rho$ is the threshold. $d_{ap}$ denotes the distance between the anchor sample and positive sample, and $d_{ap}$ denotes the distance between the anchor sample and the negative sample. However, due to the characteristics of metric learning itself, triplet loss is easily overfitted to the current samples, which makes it difficult for the network to mine discriminative features from unseen samples.

Hard-Sample-Based Triplet Loss

To improve the generalization of the network and force the network to extract more discriminative features rather than overfit to simple samples, hard-sample-based triplet loss is calculated with a hard negative sample and hard positive sample in a batch instead of all samples:

$$\begin{array}{c}\hfill L_{H-Tri}=max(0,\rho +max{d}_{ap}-min{d}_{an})\end{array}$$

(21)

However, not only hard-sample-based triplet loss, but also traditional triplet loss cannot get rid of calculating the loss based entirely on a relative distance. Therefore, it is impossible to avoid both $d_{ap}$ and $d_{an}$ decreasing or increasing at the same time during optimization, which will have a negative impact on the network performance.

Quadruplet Loss

Compared with triplet loss, quadruplet loss [140] adds another negative sample to calculate the distance between two negative samples:

$$\begin{array}{c}\hfill L_Q=max\left({\rho }_1+d_{ap}-d_{a{n}_1},0\right)+max\left({\rho }_2+d_{ap}-d_{n_1{n}_2},0\right)\end{array}$$

(22)

Obviously, in the second term, there is no shared identity between $d_{ap}$ and $d_{n_1{n}_2}$. Because the quadruplet loss uses the absolute distance between positive and negative samples, it can solve the problem of simultaneously increasing or decreasing.

3.3.3. Representation Learning

Representation learning [141] does not directly compare the similarities between different features as metric learning does, but treats the cross-modal person Re-ID task as a classification problem (multi-classification) or verification problem (binary classification). At the same time, for representation learning, the network needs to connect the feature output layer with a fully connected (FC) layer to make the number of output feature dimensions equal to the number of categories through the FC layer.

Classification loss is used to judge the accuracy of classification results. The higher the accuracy is, the smaller the classification loss is. On the contrary, the classification loss is greater. Cross-entropy loss is usually used as the classification loss for cross-modal person Re-ID.

$$\begin{array}{c}\hfill L_{ID}=-\sum _{i=1}^N{q}_{i}log\left(p\left(f_i\right)\right)\end{array}$$

(23)

where N is the number of samples. $q_i$ represents the one-hot vector of the i-th sample. $p\left(f_i\right)$ represents the prediction probability of $f_i$.

Obviously, due to the FC layers, when there are too many parameters in the network, the computation of representation learning will be complex, and it is difficult to obtain a converged network. However, representation learning will have a more stable result compared to that of metric learning because it takes all categories into account.

4. Conclusions

We provided a comprehensive and systematic analysis of cross-modal person Re-ID works and classified them according to the means of alleviating the modality gap. These methods were classified into picture-level style transform-based, shared-embedding-mapping-based, and modality-invariant auxiliary-feature-based methods. By generalizing and comparing different methods, we found that deep-learning-based networks with mathematical constraints tend to show more robust performance and stronger interpretation. Through the above analysis, we have summarized the current problems and future developmental directions of cross-modal person Re-ID as follows:

• Relying exclusively on off-the-shelf networks leads to researchers’ lack of understanding of the computational processes. Thus, most researchers are unfamiliar with the underlying logic of the networks and are unable to judge whether the overall framework of a network is reasonable. Therefore, an understanding of the mathematical principles related to deep-learning-based networks is required, and the characteristics of the network should be considered.
• In the case of large modal differences, it is difficult to obtain accurate retrieval results based only on a simple deep-learning-based network. Therefore, additional theoretical constraints are required to improve the performance. The existing cross-modal person Re-ID methods primarily rely on appearance features with great modal differences to determine identity. Therefore, more detailed analysis and processing of data are needed to extract more discriminative features in the future.
• When the training data are insufficient, deep-learning-based networks tend to overfit to the visible modality, producing ineffective performance in realistic scenarios. Fortunately, handcrafted feature-extraction-based methods are not troubled by this problem. Therefore, combining the mathematical tools in traditional methods and frameworks in deep-learning-based methods will be very important in the future.
• Although the technology of person re-identification has been rapidly developed, downstream tasks based on person re-identification have still not received attention. For example, in medical health, the behavior of patients with the same identity can be identified and analyzed with different cameras according to the person re-identification technology. The pedestrian behavior of the same patient with a mental illness can be monitored with different cameras on the basis of this technology.