Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator

Abstract

Although image recognition technologies are developing rapidly with deep learning, conventional recognition models trained by supervised learning with class labels do not work well when test inputs from untrained classes are given. For example, a recognizer trained to classify Asian bird species cannot recognize the species of kiwi, because the class label “kiwi” and its image samples have not been seen during training. To overcome this limitation, zero-shot classification has been studied recently, and the joint-embedding-based approach has been suggested as one of the promised solutions. In this approach, image features and text descriptions belonging to the same class are trained to be closely located in a common joint-embedding space. Once we obtain the embedding function that can represent the semantic relationship of image–text pairs in training data, test images and text descriptions (prototypes) of unseen classes can also be mapped to the joint-embedding space for classification. The main challenge with this approach is mapping inputs of two different modalities into a common space, and previous works suffer from the inconsistency between the distribution of two feature sets on joint-embedding space extracted from the heterogeneous inputs. To treat this problem, we propose a novel method of employing additional textual information to rectify the visual representation of input images. Since the conceptual information of test classes is generally given as texts, we expect that the additional descriptions from a caption generator can adjust the visual feature for better matching with the representation of the test classes. We also propose to use the generated textual descriptions to augment training samples for learning joint-embedding space. In the experiments on two benchmark datasets, the proposed method shows significant performance improvements of 1.4% on the CUB dataset and 5.5% on the flower dataset, in comparison to existing models.

1. Introduction

With the development of deep learning technology, the classification performance of image data has improved, and the field of application has expanded. This performance improvement was attributed to the developed feature extraction methodology and learning of a massive artificial neural network for numerous data designated by humans. However, the conventional methods for image classification are limited in that they do not work well when the classes in the testing dataset are not identical to those in the training set. Recently, zero-shot learning has been studied to overcome this problem.

Zero-shot learning is the task of recognizing new patterns whose samples are unavailable during training [1,2,3,4,5,6]. In zero-shot image classification, for example, a classifier that has learned using images of several bird species is asked to assign test images of new species to appropriate categories. Some conceptual information about the new species must be provided to the classifier in some form to perform this task. A representative form of the conceptual information is provided as predefined attributes such as the color of the beak, but these kinds of predefined attributes are rarely given in real applications. As an alternative, more natural descriptions such as “this bird has black feathers and a long beak.” have also been used [7,8,9,10,11]. In this case, image–text matching plays an important role in assigning test images to an appropriate class defined by textual descriptions.

Some researchers [7,8] have introduced an embedding-based approach to solving this task, which creates a semantic embedding space containing representation vectors for both image and text. In the training stage, image and text features are extracted, respectively, and the two sets of features are mapped onto a common embedding space. This joint-embedding method trains the image and text features from a single pair to be close. In the test stage, some conceptual information about unseen classes is given as textual descriptions, and the trained model maps the descriptions to the embedding space to create a prototype vector for each unseen class. Then, the zero-shot classification of test images is performed by assigning the embedded feature of the test image to the class of the nearest prototype vector.

The performance of this approach depends on whether the image features in the joint-embedding space are closely mapped to the corresponding class prototype. In other words, the image and text features belonging to the same category must be well-bound around the prototype in the embedding space. However, in zero-shot learning, test images are given from untrained classes, and their distributional properties can be different from those of training classes. This can lead to a large discrepancy between image features and class prototypes defined as textual features, which results in performance degradation.

To deal with this problem, we propose rectifying the image-feature vector by using additional semantic information obtained from a caption generator [12,13,14]. A caption generator can generate a general description sentence that provides meaningful information for the given input images; thus, it can be used as a guide for enhancing the image feature. Therefore, by combining the two vectors, a better representation of the given input is expected than that from a single image feature.

Figure 1 illustrates how the additional text feature given from the image caption can be combined with the original image feature to aid prediction and the comparison with prototype vectors. Furthermore, the generated description can also be used as augmented data in the training stage, improving the generalizability when the training data have insufficient text descriptions. Extensive experiments were conducted on the CUB-bird [15] and flower datasets [16] to verify the efficiency of the proposed model.

In summary, the main contributions of this work are as follows:

• We propose a novel approach for zero-shot image classification using additional semantic information generated from an image-caption generator. Unlike [17,18] studies that have used the caption generator for enhancing visual feature extraction, the proposed model directly uses the generated text information for the rectification of the embedding vector in the semantic space.
• The generated-text-feature vector enhanced the image-embedding vector in the testing stage by providing additional information. We call this vector a rectified vector, and it exhibits significant improvement in classification accuracy through the experiments.
• We also demonstrate that image-caption generators can play an augmentation role in learning a joint-embedding space, especially when the image–description pairs in the training set are insufficient.

The paper is organized as follows. In Section 1, we describe the zero-shot image classification problem and briefly explain the existing prototype-based methods with a limitation and our proposed approach to solve it. In Section 2, the related works, zero-shot image classification, and image captioning are described. In Section 3, the proposed method for rectifying embedding vectors and data augmentation using generated captions are explained. In Section 4, we analyze the quantitative and qualitative results of the proposed method on two benchmark data. Finally, in Section 5, a summary and future work are presented.

2. Related Works

Zero-shot learning: Early zero-shot image classification studies [19,20,21,22] used a two-step strategy to predict the image class. First, when a new image comes in, the image attributes are predicted through the classifier obtained from the seen class in the learning phase. Second, the probability score of the attributes obtained in this way is combined with the prior attribute information of the unseen class and used for final class prediction. When combining the class attribute and attribute label in this two-stage approach, a domain shift problem occurs in which the attributes of two images from different classes overlap, and the performance deteriorates. For example, the shapes of a lion and a dog are very different, but the attribute called tail yields the same correct answer in the image-attribute classifier. This kind of information loss in the first stage leads to wrong predictions in the second stage.

To solve this domain shift problem, some studies [7,23,24,25,26,27,28] have proposed locating the attribute and image in the same embedding space. When the input image and its corresponding attribute are given, their feature vectors in the joint-embedding space should be located closely. In the inference stage, the image class is assigned to the closest prototype by comparing the distance between the prototype composed of attributes and the input image in the same space. To construct the same joint-embedding space, a model embeds attribute data into the image space [7], the image into the attribute space [23,26], and the image and attribute into the intermediate space [21,24,27,28]. Some models [29,30] that predict the class of the unseen image through the relationship between the image and semantic label using embedding are also presented.

Although various approaches using predefined attributes have demonstrated good results, the fundamental problem is that these methods use attribute information. Predefined attributes rarely exist in real data and are expensive because humans must label them. The studies [7,8] suggested using a free-style natural language description instead of an attribute to overcome this problem. Since such descriptions usually exist in image–text pair datasets that can be easily obtained through a web search, it is more natural and general to use the textual description as the conceptual definition of the test classes for zero-shot image classification. In this manner, the related works on image–text matching [17,21,22] have also been applied to zero-shot classification tasks.

Recently, some research such as CLIP [27] and ALBEF [28] achieved outstanding performance for zero-shot recognition using transformer-based models [31,32] with contrastive learning. Those models showed that a large number of image–text pairs can improve zero-shot recognition ability because they help to construct better joint-embedding space. However, those models are difficult to train because of high-computational costs.

Though the embedding-based methods designed for general image–text matching have shown considerable success in zero-shot classification, they focus on the distance between individual image–text pairs and do not consider the class distribution. Since the class distributions of test images (unseen classes) are not observable while learning the embedding space, it is difficult to guarantee that the image features of a test class are well clustered around its corresponding prototype defined as textual features. In this paper, we try to obtain a better representation of test images by using additional textual information that is derived from images.

Image-caption generation: Caption generation is a challenging artificial intelligence problem in which a text caption is generated from a given image. Recently, deep learning methods have achieved notable results [33,34,35,36] for this problem with attention-based models. For the encoded image from a CNN, the image-caption-generation model predicts a sequence of words that describes the image through an RNN. Some works [37,38] have solved this problem by learning image–text matching relationships using large amounts of image–text pairs using a transformer structure [39]. Furthermore, researchers [17,18] have exploited the image-captioning process to solve the image–text matching task. In [17], the image-captioning process sorts the order of cropped local image regions, and the captioning module helps to obtain better image features when local features are fused as a whole feature. Similarly, another study [18] employed the image-captioning process to obtain fine image features. They fed an extracted image feature that is composed of a local feature and a global feature as an input of the image-captioning model; then, the supervision signal of the captioning process caused the extracted feature to obtain better representation.

Although both models are significantly improved over previous studies, they focus on obtaining better representations of image features and do not directly use the generated text. In contrast, the proposed model uses the results of the caption-generator module directly. The direct use of generated-text features improved the problem of discrepancy between image and text distribution in the test stage. In addition, the generated texts can also be applied to data augmentation in the learning of joint-embedding space.

3. Proposed Method

3.1. Overall Structure

We consider a set of training samples $D_{tr}=\left\{\left(I_i,T_i,y_i\right),i=1,\dots N_{tr}\right\}$ with an associated class label $y_i\in {\mathcal{T}}_{tr}$, where $I_i$ is the $i$th training image and $T_i$ is a textual description of $I_i$. The goal of the zero-shot classification task is to predict $y_j\in {\mathcal{T}}_{ts}$ of test image $I_j{}^{ts}$ under the condition that training and testing classes are mutually disjoint, such that ${\mathcal{T}}_{tr}{\displaystyle \cap }{\mathcal{T}}_{ts}=\varnothing$.

The overall structure of the proposed model is presented in Figure 1. In the training stage, when image $I_i{}^{tr}$ and text $T_i{}^{tr}$ are given, the features of each piece of information are extracted through the CNN and long short-term memory (LSTM) network, respectively. These features are mapped onto the joint-embedding space through two fully connected layers to obtain vector representations ${\phi }_I\left(I_i{}^{tr}\right)$ and ${\phi }_T\left(T_i{}^{tr}\right)$. In the joint-embedding space, the semantic relationship of the image and text pair is represented by the distance of the embedding vector pair. For example, an embedding vector of an image of a bird with a white beak and the text-embedding feature of the description “white head” are trained to be located nearby in the joint-embedding space.

In the testing stage, a given $I_j{}^{ts}$ is assumed to belong to an unseen class ${\mathcal{T}}_{ts}$ in the training stage. Conceptual information for the new class must be provided to recognize it, and it is represented as a vector in the learned joint-embedding space, defined as a prototype vector $\mathbb{P}$. Given the prototype vectors ${\mathbb{P}}_k\left(k\in {\mathcal{T}}_{tr}\right)$, the classification of test image $I_j{}^{ts}$ is performed based on the distance between the image-embedding vector ${\phi }_I\left(I_j{}^{ts}\right)$ and prototype vectors ${\mathbb{P}}_k\left(k\in {\mathcal{T}}_{ts}\right)$. This method is similar to the baseline approach [8] in that the joint-embedding model is constructed in an intermediate space. While previous studies [7,8] have used a single image vector ${\phi }_I\left(I_j{}^{ts}\right)$ in the embedding space, we propose using additional textual information to obtain a better representation of the image. For example, when we examine a red bird, a human analyzes its visual appearance and simultaneously thinks, “It is a red bird.” By combining this kind of linguistic thinking with visual representation, we expect to better represent the object in the cognition system. To implement this procedure, we generate a textual description $G\left(I_j{}^{ts}\right)$ expressing the image $I_j$ linguistically by exploiting the image-caption generator and use this description to obtain a rectified embedding vector $R_j$ complementing the image feature. In addition, the generated text description can also be used in the training stage. The image–text data pairs sometimes do not exist or are very limited in a real situation, which results in poor performance because the joint-embedding space is not well-formed. We deal with this problem of insufficient data using the generated texts as an augmented description.

3.2. Learning Joint-Embedding Space

As depicted in Figure 2, the training stage for the joint-embedding space consists of two flows. First, a CNN model is applied to the image data, and the extracted feature is placed on the joint-embedding space through two trainable fully connected layers. The output of the image processing flow is written as ${\phi }_I\left(I_i\right)$. Second, an LSTM network is applied to the textual description of the given input images, and the text features extracted from the trainable LSTM network are again placed in the joint-embedding space through the trainable fully connected layer. The text processing flow output is written as ${\phi }_T\left(T_i\right)$. The vectors from different flows are placed on the same embedding space; thus, their inner product can be calculated as follows:

$$s_{ij}={\phi }_I\left(I_i\right)·{\phi }_T\left(T_j\right).$$

(1)

Then, the matching loss for all image–text pairs ($I_i,T_i$) in a minibatch subset $B$ is defined as follows:

$$L_{mat}={\displaystyle \sum }_{i=1}^{\left|B\right|}{\displaystyle \sum }_{j\ne i}max\left(0,s_{ij}-s_{ii}+margin\right),$$

(2)

where $s_{ij}$ is the score value of the inner product calculated using the mismatched image–text pair $\left(j\ne i\right)$, and the $margin$ is a positive constant value for stable training. Learning with this loss function updates the embedding function to increase the matched score value $s_{ii}$ and decrease the mismatched score value $s_{ij}$ so that the embedding space can represent a semantic correlation between the image and text.

3.3. Learning Caption Generator

Inspired by the human recognition system that employs multi-information inflow in awareness, we deploy linguistic information derived from images for rectifying the embedding vector of ${\phi }_I\left(I_i\right)$. To realize this process, we used an image-captioning approach that translates an image to a textual sentence form [12,33,34,35,36]. Given a test image $I_j{}^{ts}$ as input, a text description $G\left(I_j{}^{ts}\right)$ is generated through the caption-generator module, as illustrated in Figure 1. The attention-based image-caption-generation method [12] is adopted as the generation model. We employ a simple but intuitive model because our main goal of employing the caption-generator module is leveraging semantic textual information from images. Many other elaborate state-of-the-art captioning models [37,38] can also be utilized for generating high-quality captions.

For training of caption-generator image–sentence pairs, $\left\{\left(I_i,T_i\right)|T_i=[w_{i,1},\dots , w_{i,N}]\right\}$ in $D_{tr}$, where $w$ is a word composing the sentences, are used. The loss for generating the sentence is calculated by minimizing the log likelihood of the sequence of words, written as follows:

$$L_{gen}=-{\displaystyle \sum }_{i=1}^{\left|B\right|}{\displaystyle \sum }_{t=0}^{N-1}\log p_{i,t+1}\left(w_{i,t+1}\right),$$

(3)

where $N$ denotes the number of words in a target sentence, $w_{i,t}$ represents the $t$th word in the $i$th sentence of a minibatch, and $p$ is the softmax function for predicting the next timestep word. One of the merits of our proposed model is that we can simultaneously train the caption-generator and joint-embedding modules. In the entire end-to-end training process, the total loss to be minimized is defined as follows:

$$L=L_{mat}+L_{gen}.$$

(4)

3.4. Rectified Embedding Vector for Zero-Shot Classification

In the testing stage of zero-shot classification, the generated caption is transferred to text feature extraction and combined with the image feature on the joint-embedding space to obtain the rectified vector. As displayed in Figure 2, the rectified vector created through this combination is expected to obtain a better representation for matching with the prototype defined using textural descriptions. For a given $j$-th test image $I_j{}^{ts}$, the rectified embedding vector $R_j{}^{ts}$ can be obtained through a weighted combination of the image-embedding feature $I_j{}^{ts}$ and the embedding feature of the generated text ${\phi }_T\left(G\left(I_j{}^{ts}\right)\right)$, which can be written as follows:

$$R_j{}^{ts}=\left(1-\alpha \right){\phi }_I\left(I_j{}^{ts}\right)+\alpha {\phi }_T\left(G\left(I_j{}^{ts}\right)\right),$$

(5)

where $\alpha$ is a weight value in [0, 1] and is empirically adjusted. The rectified vector $R$ is applied to the distance-based classifier with class prototypes. Finally, the predicted class is obtained as follows:

$$y\left(I_j{}^{ts}\right)=\underset{k\in {\mathcal{T}}_{ts}}{argmin} Dist\left(R_j{}^{ts},{\mathbb{P}}_k\right).$$

(6)

We verified the performance of our proposed method using the rectified vector against the benchmark dataset based on Equation (6), and the results show considerable improvement in performance compared with individual vectors, such as the single image and single text vectors.

3.5. Data Augmentation Using Generated Captions

The textual description obtained using the caption generator can also augment insufficient training data in a more realistic situation. In an actual dataset where the image and text exist as a pair, text information for an image is often insufficient for learning the joint-embedding space. We propose a method to augment training data through various sentences created by the caption-generator model to deal with this problem. When a training image is given, it is possible to generate descriptions through the caption generator, which can be added to the training dataset. In that case, the training dataset $D_{tr}\left\{\left(I_i,T_i,y_i\right)\right\}$ is enlarged with the augmented dataset $D_{Aug}\left\{\left(I_i,T_i^{aug},y_i\right)\right\}$, where $T_i^{aug}=G\left(I_i\right)$. The effect of this augmentation is also investigated in the experiments section (Section 4.3).

4. Experimental Results

4.1. Experimental Settings

This section explains the experimental settings and details in the training and testing stages. In image processing, the pre-trained Resnet-152 model [40] is employed as the feature extractor, and the weight of the image-feature extractor is fixed. In text processing, text features are extracted from given descriptions by the trainable LSTM network. For the construction of the intermediate joint-embedding space, two trainable fully connected layers were employed for dimension reduction, which converts the 2048-dimensional image vector output to a 256-dimensional vector and also converts the 1024-dimensional text vector output to a 256-dimensional vector. In addition, we used the Adam optimizer with a learning rate of 0.005 and a gradient clipping of 5.0 to control the learning rate of the LSTM, and the batch size of the training step was 55. For text preprocessing, all words in the training descriptions were used without removing the stop words, and word-to-vector [34] models were used for word embedding to each word. We also used a single NVIDIA RTX 3090 GPU for computation.

The number of descriptions per image also affects classification accuracy. The training dataset has 10 descriptions per image, and we employed the whole descriptions in the training stage. To create prototypes ${\mathbb{P}}_K$ (k ∈ ${\mathcal{T}}_{ts}$) in the testing stage, we used all descriptions for one image in each prototype class, and this setting is the same as in [8] for a fair comparison. For the evaluation metric for the testing dataset, we used Rank-1 accuracy, similar to previous studies [8,10,11].

4.2. Comparion Models

We compare the five models presented below to verify the proposed model. We refer to the experimental results from previous works [8,11].

(1) Word CNN and word CNN + RNN [8] construct joint-embedding using compatible functions. Unlike previous work [28] that solely trained the image encoder, these methods train the image and text encoders simultaneously using a symmetric structured joint-embedding function.

(2) DEM [7] learns discriminative features using the bidirectional LSTM and CNN. This method constructs a joint-embedding space by minimizing the Euclidean distance between the image and text features on the same embedding space. Unlike the other four comparison models, the experiments only employ the CUB dataset.

(3) The IATV [10] approach uses a two-stage strategy for dealing with the image–text matching task. This approach proposes cross-modal entropy loss to construct the joint-embedding space. This approach also proposes a co-attention mechanism to incorporate the sentence into the image feature through the encoder–decoder LSTM network to improve the image representation for this processing.

(4) CMPC + CMPM [9] uses the norm-softmax loss to make embedded features compact. This method regularizes the angular range of the representation features; thus, it has more discriminative image features. Otherwise, it proposes probability-based loss, which performs KL-divergence, minimizing the distance between the probability distribution of the truly matching pairs and predicted matching pairs in the minibatch to avoid sampling processes, such as the traditional bidirectional ranking loss.

(5) TPAMI [11] also employs the norm-softmax cross-entropy loss for the con image representation and uses the KL-divergence, minimizing to construct joint-embedding space as in [4]. It also introduces GAN [41] loss to improve representation and reduce obscurity between text and image features using discriminator loss.

4.3. Experimental Results

The CUB dataset [15] contains 200 bird species and 11,788 images divided into 150 training and 50 testing classes. In the training stage, 8855 image–description pairs corresponding to 150 classes were used. In the testing stage, 2933 images corresponding to 50 unseen classes were used. The experimental results for the CUB dataset are presented in

Table 1. Experimental results on the dataset.

Representation Method	Rank-1 Accuracy (%)
	CUB	Flower
Word CNN [8]	51.0	60.7
Word CNN + RNN [8]	56.8	65.6
DEM [7]	58.3	60.9
IATV [10]	61.5	68.9
CMPC + CMPM [9]	64.3	68.4
TIMAM [11]	67.7	70.6
Proposed method (rectified vector)	69.3	76.1

. The proposed method surpasses other comparison results on rank-1 accuracy of zero-shot image classification. In addition, the experiment was conducted on the flower dataset [16], which includes images of various flowers and their corresponding descriptions and class information. The dataset has 8189 images divided into 82 training and 20 testing classes. In the training stage, 7034 image–description pairs corresponding to 82 classes were used. A total of 10 descriptions were provided for each image, and all sentences were used for training. In the testing stage, 1155 images corresponding to 20 unseen classes were used in the inference process. All input images were resized to 224 × 224. Similar to the CUB dataset, Table 1 reveals that the proposed model achieves fair improvement in zero-shot classification accuracy. Additionally, the results of our proposed model also outperform most of the results of previous works under the zero-shot classification setting with predefined attributes [22,26,27].

We demonstrated the effect of the rectified vector through ablation studies. In

Table 2. Ablation studies.

Method	Rank-1 Accuracy (%)
	CUB	Flower
Proposed method (rectified vector)	69.3	76.1
Only image features (w/o generated text feature)	67.5	72.1
Only text features (w/o image feature)	24.4	43.8

, the method using only image features means that the generated text feature is not combined with the image feature in the testing stage; thus, α = 0 in Equation (5). Compared with that, the proposed method significantly improves classification performance, demonstrating the effect of the rectified vector. We also verify whether the generated caption created a semantically correct sentence. We set a situation where the method using only text features performs the classification task by using the condition α = 1 in Equation (5). In this experiment, classification performance presents meaningful results, indicating that the image-captioning model generates sentences that semantically match the image.

One of the interesting aspects is that the performance of the generated texts affects the improvement of the combined performance. For the flower dataset with relatively good classification performance on the method using only text features, when the two pieces of information were combined, the improved performance rate considerably increased compared to the CUB dataset. Thus, the advanced image-captioning model that can make more accurate descriptions, such as OFA [37] and mPLUG [38], could improve the proposed method in terms of accuracy.

4.4. Image Caption for Training Data Augmentation

When insufficient data exist in the training stage, data augmentation using the generated sentences is expected to have a strong effect on model training. Different settings are required during the training stage to prove this assumption. Unlike the prior setting, we start the experiment with only one description for each training image. Then we increase the number of additional descriptions generated from the caption generator one by one to see the effect of augmentation.

As presented in Figure 3, using only one description (no augmentation) for training results in poor performance because the image-embedding vector cannot be related to various text-embedding vectors. However, through the data augmentation process using the generated caption, the performance improved remarkably in both datasets. From this result, we confirm that this simple approach to textual data augmentation is highly effective and demonstrates powerful performance in a data-insufficient situation.

4.5. Qualitative Study

Figure 4 presents examples of the zero-shot image classification on the CUB dataset. In the table in Figure 4, the upper part shows the prediction result obtained by using only the image-feature vector, and the lower part shows the case of using our proposed rectified vector. In the example of the bird, a Laysan Albatross is characterized by a brown part of the body. A brown creeper, which the upper model predicted as the rank-1 answer, also has a brown part of the body. The upper model here overly relied on the brown body feature of the image, causing incorrect prediction. Our proposed model (lower part) modifies the prediction of the original model by providing additional information from generated sentences so that it does not depend only on large image features. As a result, the provision of additional textual information through the generated description in our proposed model improves classification accuracy. Although the generated sentences are sometimes not grammatically correct (the right in Figure 4), they still can provide meaningful information correcting the wrong prediction.

One limitation of the proposed method is that the caption generator does not always generate good semantic textual information. Figure 5 illustrates some examples of generated sentences. While the sentences of the left examples are semantically matched with the images, the right examples reveal that some words in the generated sentences are inaccurate. This kind of generated text that semantically mismatched with the image may lead to poor performance. The use of a better generation model may reduce this problem and avoid performance degradation due to the misunderstanding of images.

5. Conclusions

In this paper, we proposed a novel zero-shot image-classification model employing additional textual information obtained through a generative image-captioning model. To solve a problem of zero-shot image classification in which classification performance is degraded due to modality inconsistency between image and prototype vector composed of texts, we derived additional text information that linguistically describes images, and combined it with the image feature as a rectified vector in the test stage. This rectified vector resulted in a better representation than using an image feature alone. The utilization of this rectified embedding vector achieves notable performance improvement on two benchmark datasets. In addition, we demonstrate that when the textual descriptions in training data are insufficient, the generated image captions can be used as augmentation data for learning the joint-embedding space.

In future works, various attempts for performance enhancement are possible. The image- and text-feature-extraction models can be improved by using more sophisticated models, such as ConvNet [21] and BERT [42]. Those advanced models could capture more discriminative features in image and text data. In addition, though we utilized a fixed weighted sum for combining image and text features in the current model, it can be elaborated using a trainable combination module. These elaborated methods may lead to better learning of the joint-embedding space and reduce the classification error. Furthermore, the proposed embedding-vector-rectification method can be easily applied to various multimodal tasks such as text–video retrieval. For example, in a text–video search application that uses joint-embedding space, better search results can be expected by combining a video expression with a caption obtained through a video-captioning module. In addition, we currently augment training data from the perspective of the text; it is possible to improve performance by obtaining a richer number of data pairs through advanced image augmentation methods [43].