Military Image Captioning for Low-Altitude UAV or UGV Perspectives

Abstract

Low-altitude unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs), which boast high-resolution imaging and agile maneuvering capabilities, are widely utilized in military scenarios and generate a vast amount of image data that can be leveraged for textual intelligence generation to support military decision making. Military image captioning (MilitIC), as a visual-language learning task, provides innovative solutions for military image understanding and intelligence generation. However, the scarcity of military image datasets hinders the advancement of MilitIC methods, especially those based on deep learning. To overcome this limitation, we introduce an open-access benchmark dataset, which was termed the Military Objects in Real Combat (MOCO) dataset. It features real combat images captured from the perspective of low-altitude UAVs or UGVs, along with a comprehensive set of captions. Furthermore, we propose a novel encoder–augmentation–decoder image-captioning architecture with a map augmentation embedding (MAE) mechanism, MAE-MilitIC, which leverages both image and text modalities as a guiding prefix for caption generation and bridges the semantic gap between visual and textual data. The MAE mechanism maps both image and text embeddings onto a semantic subspace constructed by relevant military prompts, and augments the military semantics of the image embeddings with attribute-explicit text embeddings. Finally, we demonstrate through extensive experiments that MAE-MilitIC surpasses existing models in performance on two challenging datasets, which provides strong support for intelligence warfare based on military UAVs and UGVs.

1. Introduction

On the battlefield, low-altitude UAVs and UGVs are frequently employed in combat surveillance, targeted patrols, and strategic reconnaissance missions [1] to ensure swift responses to potential threats. Image data generated by UAVs and UGVs provide tactical information upon which critical decisions and actions are taken, especially for situational awareness and ISR (intelligence, surveillance, and reconnaissance)-type applications [2]. As a vital pathway to achieving situational awareness, the common operational picture (COP) can be delineated using scene descriptions and reference terms familiar to the commander [3]. Images that have been pre-tagged with relevant keywords or sentences can provide an advantage for the timely sharing of scene information in a COP [4]. Therefore, utilizing image captioning for the sentence-level pre-tagging of military images is a feasible solution to enhance the comprehension of military intelligence.

Image captioning is a multimodal task that generates descriptive sentences for images, which has evolved from traditional retrieval-based and template-based methods to recent deep learning methods. As indicated in Figure 1, image captioning based on deep learning [5,6,7,8] mainly follows an encoder–decoder architecture, where an input image is processed by a visual encoder, and a caption is generated by an autoregressive language decoder. Then, considering a zero-shot and text-only training setup, some works [9,10,11,12,13,14] extend an encoder with visual language (VL) model to obtain an encoder (VL)–decoder architecture. Military image captioning (MilitIC), as an applied branch of image captioning, focuses on generating appropriate descriptive sentences for military-related images. The MilitIC task involves extracting image features, which includes objects, environments, and their interactions, and converting this information into coherent natural language sentences to support military analysis and intelligence decryption. In MilitIC, the related works [15,16,17] transfer generic image-captioning methods to the military domain. They also follow the vanilla encoder–decoder architecture to achieve promising performance.

However, there remain huge challenges in generating precise semantic descriptions from military imageries with complex and diverse military scenarios. In a brief investigation, the following factors are particularly prominent:

• Dataset scarcity: There is no established benchmark dataset in the field of MilitIC. The datasets from military target detection and camouflaged detection are not readily applicable to this field.
• Deficiency of a specific method: there is a scarcity of methods specifically proposed for MilitIC tasks that can effectively highlight military semantics in caption generation.

To address these challenges, we created a MilitIC benchmark dataset MOCO and put forward a novel encoder–augmentation–decoder architecture with the MAE mechanism named MAE-MilitIC. Figure 1c and Figure 2 showcase our dataset and novel architecture, respectively. The MAE-MilitIC leverages both image and text modalities as a guiding prefix for caption generation, in contrast to the other two architectures, which utilize only a single modality. Rather than directly guiding caption generation by an initial image or text embedding in VL joint space, we propose a MAE mechanism to build a semantic subspace, with the aim to close the modality gap between the image and text and enhance the military semantics of the image through explicit military text. For the MAE mechanism in Figure 1d, we first mapped the embeddings of the input image onto a semantic subspace constructed by relevant texts that described the military content that were to be enhanced and recorded a residual vector. Subsequently, this mapped embedding in the semantic subspace was augmented with explicit military text, which enabled a user-specified augmenting power to control the intensity of the augmentation. Finally, the residual was added back to reconstruct a vector close to the “image region” of the VL joint space. It was demonstrated that the embeddings constructed by the MAE served as better guidance for precise caption generation.

In short, we highlight the three major contributions of this paper:

• We created the first open-access MilitIC benchmark dataset, which was named the Military Objects Dataset in Real Combat (MOCO) and not only includes real and complex military scenes but also provides a vast number of rich captions. MOCO is available at https://github.com/Panlizhi/MOCO (accessed on 1 August 2024).
• We propose a novel encoder–augmentation–decoder architecture, MAE-MilitIC, to leverage image–text modalities to guide caption generation, and introduce the MAE mechanism to augment the image embeddings with explicit military text in the semantic subspace.
• Extensive qualitative and quantitative experiments on two challenging datasets demonstrated that the proposed MAE-MilitIC outperformed existing state-of-the-art image-captioning models.

2. Related Works

Benefiting from the rapid advancement of artificial intelligence and deep learning technology, significant progress has been made in image captioning. In the following sections, we separately delve into the recent advancements in captioning for natural images and military images. Furthermore, we provide an overview of the military images dataset.

2.1. Image Captioning

Image captioning is the task of comprehending the image’s content in terms of objects, attributes, and relationships, and expressing the represented semantic content using appropriately formulated sentences.

(1) Traditional methods: The early works on image captioning used retrieval-based methods and template-based methods with hand-crafted natural language generation techniques. Retrieval-based methods [18,19,20,21,22] utilize sentences generated from similar images to describe the query image. These methods involve building a candidate sentence bank, followed by retrieving corresponding sentences based on similarity calculations between the query image and the candidate sentences. Template-based methods [23,24,25,26,27,28,29,30] generate descriptive sentences for a given image through a two-step process of object detection and template filling. These methods match the extracted various features into the pre-defined template slots, thereby creating a concise description for the image.

(2) Deep learning methods: Deep learning-based image captioning mainly follows an encoder–decoder architecture, where an input image is processed by a visual encoder, and a caption is generated by an autoregressive language decoder. Pioneering works [31,32,33,34] employed CNN and RNN/LSTM architectures to extract features and generate descriptions. Additionally, transformers were applied in encoder/decoder methods to enhance the algorithm’s ability for long-range modeling and high parallel computation [5,6,7,8].

Recently, progress in large vision-and-language pre-trained models (such as SimVLM [35] and CLIP [36]) has improved image-captioning works. The pre-trained paradigm enables models to be trained on vast image–text pairs and learn a generic representation. Some studies explored the pre-training model CLIP in image-captioning tasks due to its strong ability to extract and align features from images and text. As a supervised method, ClipCap [37] produces a fixed-length prefix for each caption by mapping CLIP visual embeddings into the space of a generative language model. The training-free methods ZeroCap [9] and MAGIC [10] leverage the excellent visual–text alignment of CLIP to guide the language model toward a desired visual direction and make the caption semantically related to a given image. Some of the text-only training methods, such as CapDec [11], DeCap [12], and ViECap [38], map the visual feature to the text feature and generate accurate captions with fewer image data. Similarly, CLOSE [13] proposed the task of zero-shot cross-modal transfer, where they learn semantics from textual data and transfer them to visual data based on the CLIP alignment space. In addition, DeCap [12], SmallCap [39], and MeaCap [14] use CLIP to build a memory bank, retrieve relevant captions, and combine them for more effective generation prompts. RLHMN [40] and IVRC [41] emphasize hierarchical multi-granularity semantic alignment between visual and text modality, and highlight the caption refinement strategies.

2.2. Military Image Captioning (MilitIC)

The three primary application areas of image captioning are general-purpose image captioning, medical image captioning, and remote sensing image captioning. Research on image-captioning tasks specifically tailored for the military domain is relatively scarce. Military image captioning focuses on generating appropriate descriptive sentences for military-related images and provides strong support for military intelligence generation. Different from the existing military object detection method, which merely captures the location and class of objects, MilitIC is able to produce accurate descriptions of images, including objects, environments, and their interactions. Information on the interaction between military environments and military targets is crucial for situational awareness tasks. The basic application of MilitIC is the automatic generation of intelligence from image data. More advanced applications involve using UAV images for battlefield reconnaissance by adhering to certain human-defined alarm rules for early attack warnings, and even employing image–text bimodality for image–text intelligence retrieval. These applications have significantly enhanced the capabilities of intelligence processing and decision support in military operations.

To our knowledge, the research [15] is the first to propose military image captioning based on deep learning. It presents a proof-of-concept demonstration following the show-and-tell model [32] and extends the Flickr8K dataset [42] with military-specific images to ensure relevance to the Department of Defense domain. The study [16] explored the benefits of using image captioning to support military decision making. It utilized an encoder–decoder model inspired by [32] and trained it with the COCO dataset [43], which is non-military-specific. The work [17] presents a concept that extracts entities and attributes from military multi-modal data in the Visual Genome dataset [44], converts them into knowledge graphs, and reconstructs contextual phrases. Notably, a video-captioning model for the military domain is proposed in [45], which utilized the existing method I3D [46] to extract entity and motion features from a given video, and then employed a GloVe-Transformer [47,48] generating model to generate video captions.

2.3. Military Image Dataset

The size, quality, and availability of datasets have a significant impact on deep learning methods. In the military domain, due to the sensitivity of equipment information, the overall quality of datasets is often not high. This section will provide an overview of the typical military image datasets, which can be seen in

Table 1. Military image datasets. Note that our dataset includes more target categories and captions, and it is open-source resource.

Dataset	Categories	Number of Captions	Task	Open Access
GMTD	Tank, transporter, and rocket launcher	11,036 images, 20,132 targets	Object detection for ground equipment	-
SFD	Pistol and rifle	3140 images, 3140 labels	Object detection for guns	-
COD10K+	Person	2592 images, 2592 labels	Object detection for camouflage	-
MHCD	Person, airplane, vehicle, and warship	3000 images, 3000 labels	Object detection for camouflage	Yes
MADD	Aircraft	14,088 images, 14,088 labels	Object detection for aircrafts	Yes
MOCO (ours)	Person, tank, rocket, fighter jet, helicopter, AV, UGV, boat, and defensive structure	7449 images, 37,245 captions	Image captioning for real combat	Yes

.

The Ground Military Target Dataset (GMTD) [49] is a ground military vehicle dataset for object detection tasks. The Small Firearms Dataset (SFD) [50] includes various small firearms, such as pistols and rifles. This dataset has been collected for the purpose of using small unmanned aerial systems to search for abandoned small firearms. The work [51] newly added camouflaged soldier images in different environments to the COD10K+ dataset. The research [52] provided a Military High-Level Camouflage Object Detection (MHCD) dataset and aimed to identify objects that are extremely embedded in backgrounds. The Military Aircraft Detection Dataset (MADD) [53], which was designed for object detection, includes 49 distinct military aircraft types, with certain types and their variants grouped into a single class.

Our work provided a real combat dataset called MOCO, which includes 7449 images with 37,245 captions, and has the advantage of offering a rich set of captions for real military combat scenarios. This dataset is particularly beneficial for research on military image captioning, as it provides a large and diverse collection of images accompanied by detailed captions. The open-access nature of the MOCO dataset also ensures that it can be widely utilized by the research community, thus fostering innovation and collaboration in the field of military image analysis.

Considering the size and availability of datasets, this work utilized the MADD and MOCO as benchmark datasets for performance validation.

3. MOCO: A Military Objects Dataset in Real Combat for Military Image Captioning

The deficiency of existing benchmark datasets in terms of the image scale, scene type, and descriptive diversity limits the advancement of military captioning approaches. In order to explore the military image captioning for UAVs, we collected a large-scale military target dataset called MOCO. We emphasized images with a low-altitude UAV perspective for their rich detail in capturing military personnel and equipment movements, in contrast with the vague information in high-altitude captures. In close combat, the imaging range and usage scenarios of UAVs and UGVs tend to be similar, and this work collects some datasets from UGV perspectives to provide richer scene information for the captioning method.

Table 1 comprehensively compares existing benchmark datasets and MOCO in terms of the category, scale, task applicability, and availability. Some typical images and captions shown in Figure 2 include military personnel (soldier, combat engineer), combat equipment (tank, boat, UAV, UGV), and the battlefield environment (trench, anti-tank obstacle). Figure 3a illustrates that the MOCO dataset contains images of various sizes that range from 224 × 224 to 1024 × 1024 pixels and include high-resolution 2K images at 1440 × 1440 pixels.

The MOCO dataset not only provides a variety of real combat images but also offers more professional military descriptions. As presented in Figure 3b, high-frequency words in the word cloud include descriptions of military equipment, personnel, and the environment, where a rich vocabulary is provided by extensive professional human captions. The captions can precisely capture the state characteristics of military targets—whether they are moving, parked, or standing—and illustrate key interactions between objects, including firing and operating. This enables more detailed descriptions of military scenarios and allows for fine-grained battlefield perception.

4. MAE-MilitIC: Map Augmentation Embedding to Enhance Semantics for Military Image Captioning

We introduce a novel encoder–augmentation–decoder structure, MAE-MilitIC, which focuses on the inaccurate description issues and semantic augmentation methods in specific tasks, such as military image caption. The general framework of our MAE-MilitIC is illustrated in Figure 4. Adopting ClipCap [37] as the baseline, which has an encoder–decoder structure (shown in Figure 1a), we introduce an MAE mechanism to enhance the semantics of image embeddings and improve the performance of image captioning.

In MAE, we have three main objectives to fulfill: (i) the augmented image embeddings should act as a more precise guide for the image-captioning task compared with the original one (see Equation (5)); (ii) the augmented image embeddings should be more aligned to the attribute-explicit text in a semantic subspace (see Equation (6)); (iii) the augmented image embeddings should ensure modal consistency (see Equation (10)).

4.1. Backbone of MAE-MilitIC

The general framework of our MAE-MilitIC is illustrated in Figure 4. Given an input image I and a text caption T, our goal is to learn the generation of a meaningful caption for an input image, which can be formulated as a conditional probability optimization:

$$\underset{\theta }{max}log{p}_{\theta }\left(T\right|I)$$

(1)

where $\theta$ denotes the model’s trainable parameters.

I and T are taken into the image encoder ${Enc}_{img}$ and text encoder ${Enc}_{txt}$ to obtain the image embeddings $e_I$ and the text embeddings $e_T$ in the VL joint space $\mathfrak{V}$, respectively. The $\mathfrak{V}$ is established through a large VL pre-training, which enables the preliminary alignment of the image and text. Furthermore, we propose the MAE mechanism to construct a semantic subspace $\mathfrak{W}$ to bridge the modality gap and enhance the semantics of the image embeddings for a specific task. The details of the MAE mechanism are provided in the following section.

The text decoder ${Dec}_{txt}$ includes a generative language model for translating embeddings into text captions. Following recent works [37,54], we considered the augmented image embeddings as a prefix for the input of generative language model. Therefore, the generating process with a prefix–text concatenation can be formalized as follows:

$$\underset{\theta }{max}log{p}_{\theta }\left(T\right|e_I)\Rightarrow \underset{\theta }{max}log{p}_{\theta }\left(T^{\prime }\right|(e_I,T))$$

(2)

where $T^{\prime }$ is the predicting text and $(e_I,T)$ represents the concatenation operation of the prefix embeddings and the text embeddings. Since the essential image semantic information is encapsulated in the prefix embedding, the language model can utilize it to predict the next token without considering future tokens. Thus, the generating process of ${Dec}_{txt}$ can be modified to

$$\begin{array}{c}\hfill \underset{\theta }{max}log{p}_{\theta }\left({T_k}^{\prime }|(e_I,{\left[T_i\right]}_{i<k})\right)\end{array}$$

(3)

where ${T_k}^{\prime }$ is the k-th word of the predicting caption.

In order to enhance the image semantics in $\mathfrak{W}$, the MAE module is considered in Equation (3). The specific generating process of ${Dec}_{\mathrm{txt}}$ can be formalized as follows:

$${T_k}^{\prime }={Dec}_{txt}\left(MAE\left({Enc}_{img}\left(I\right)\right),{\left[T_i\right]}_{i<k}\right)$$

(4)

where $MAE\left(\right)$ is an MAE mechanism. We constructed a generation loss ${\mathcal{L}}_{\mathrm{GEN}}$ to guide the generating process of ${Dec}_{\mathrm{txt}}$:

$${\mathcal{L}}_{\mathrm{GEN}}=\sum _{k=1}^{n}CE\left({T_k}^{\prime },T_k\right)$$

(5)

where $T_k$ is the true caption and $CE(,)$ corresponds to the cross entropy.

The optimization goal of MAE-MilitIC includes two parts: one is to maintain the similarity between the generated texts $T^{\prime }$ and the true texts T, as shown in Equation (5); another is to close the gap between the generated text embeddings and the augmented image embeddings in the semantic subspace $\mathfrak{W}$, which is expressed as a mapping augmentation loss ${\mathcal{L}}_{MAE}$:

$${\mathcal{L}}_{MAE}=CosSim(txt2emb\left(T^{\prime }\right),\mathcal{A}\left(m_I\right))$$

(6)

where $CosSim(,)$ is the cosine similarity and $\mathcal{A}\left(m_I\right)$ is the augmented image embeddings in MAE. The $txt2emb\left(\right)$ represents the text-to-embedding conversion operation.

Considering caption generation and semantic augmentation, the total loss function can be formulated as

$$\mathcal{L}=(1-\lambda ){\mathcal{L}}_{GEN}-\lambda {\mathcal{L}}_{MAE}$$

(7)

where $\lambda$ is a tradeoff parameter.

4.2. Map Augmentation Embedding (MAE)

On the basis of the pre-aligned VL joint space $\mathfrak{V}$, an MAE mechanism is proposed to enhance the semantics of image embeddings for a specific task, such as a (military) image-captioning task. We constructed the semantic subspace $\mathfrak{W}$ using relevant texts that describe the semantics we aimed to disentangle and enhance, map $e_I$ and $e_T$ onto $\mathfrak{W}$ to acquire mapped image and text embeddings $m_I$ and $m_T$, and then augment $m_I$ with the semantic information contained in $m_T$.

4.2.1. Semantic Subspace $\mathfrak{W}$

The works [55,56,57] discovered that since the VL joint space is a vector space, an attribute subspace can be constructed by using a set of relevant text prompts as basis vectors, and if an attribute on the image embedding changes, the corresponding semantics in the subspace will be changed. Inspired by this finding, we considered that the disentangled military attributes from text in the subspace can be used to augment the semantics of image embeddings, and the augmented image embeddings will guide the language model to generate more precise captions encompasses specific military knowledge. The results illustrated in Figure 5 and Figure 6 confirmed our proposed viewpoint.

From the MOCO dataset in Figure 2, one can see that the military image primarily consisted of single or multiple elements of military personnel, combat equipment, and the battlefield environment. Thus, we selected basic text prompts from these three elements, which are given in Section 5.2.

After selecting $N=100$ basic text prompts, we obtained N basic vectors $b_n$ through ${Enc}_{txt}$ to form a set of base vectors $B={\left\{b_n\right\}}_{n=1}^N$. We introduced dimension-reduction techniques to find the basis semantic vectors for $\mathfrak{W}$, as the subspace was a semantically meaningful structure. For example, if we aimed to augment the information regarding the battlefield environment, we could use text embeddings from a set of base environmental elements, such as smoke, clouds, rain, mist, grassland, and desert, as the basis semantic vectors.

In order to preserve more basic vectors and eliminate ambiguous semantic information in basis vectors, we employed two dimension-reduction techniques to construct a semantic subspace and use Gram–Schmidt orthogonalization (GSO) and principal component analysis (PCA). GSO applies the Gram–Schmidt process to obtain an orthonormal basis ${\widehat{b}}_n$ to yield a set of base vectors ${\widehat{B}}_{GSO}={\left\{{\widehat{b}}_n\right\}}_{n=1}^N$, which constructs the semantic subspace ${\mathfrak{W}}^{\mathrm{GSO}}$. PCA extracts $\widehat{N}\left(<N\right)$ principal components as the basis set ${\widehat{B}}_{PCA}={\left\{{\widehat{b}}_n\right\}}_{n=1}^{\widehat{N}}$, which constructs the semantic subspace ${\mathfrak{W}}^{\mathrm{PCA}}$. Actually, the space spanned by ${\widehat{B}}_{GSO}$ or ${\widehat{B}}_{PCA}$ is a space approximating $\mathfrak{W}$ that encompasses all the related texts in the target corpus, such as a military corpus.

4.2.2. Mapping Step $\mathcal{M}$

After constructing the semantic subspace $\mathfrak{W}$, we mapped $e_I$ and $e_T$ onto $\mathfrak{W}$ to acquire the mapped image and text embeddings $m_I$ and $m_T$:

$$m_I={\mathcal{M}}_{\mathfrak{V}\to \mathfrak{W}}\left(e_I\right),m_T={\mathcal{M}}_{\mathfrak{V}\to \mathfrak{W}}\left(e_T\right)$$

(8)

where ${\mathcal{M}}_{\mathfrak{V}\to \mathfrak{W}}$ is the mapping operation between $\mathfrak{V}$ and $\mathfrak{W}$. In the subspace, the original vector can be represented as a linear combination of the basis vectors. Then, the operation ${\mathcal{M}}_{\mathfrak{V}\to \mathfrak{W}}$ is achieved by computing a dot product with the following basis vectors:

$$m_I=\sum _{n=1}^N{e}_I{\widehat{b}}_n^{\prime }{\widehat{b}}_n,m_T=\sum _{n=1}^N{e}_T{\widehat{b}}_n^{\prime }{\widehat{b}}_n$$

(9)

where ${(.)}^{\prime }$ indicates the transpose operation. It should be noted that when using the PCA method, N in Equation (9) should be replaced with $\widehat{N}$.

Furthermore, considering that if too many text embeddings are added to image embeddings during the augmentation step, the rich semantic information in the image will be drowned out. Then, we recorded the residual r between $e_I$ and $m_I$:

$$r=e_I-m_I$$

(10)

which was added back on the augmented image embeddings $\mathcal{A}\left(m_I\right)$ and we ensured it was close to the image region rather than the text region.

4.2.3. Augmentation Step $\mathcal{A}$

To enable the MAE to incorporate attribute-explicit information from the mapped text $m_T$ such that it could well serve as a guide for the caption generating process, we enhanced the impact of the intended attributes on $m_I$ while mitigating the influence of unintended attributes on $m_I$.

Mathematically, we first calculated the coefficients ${\alpha }_n$ of $m_I$ and the coefficients ${\beta }_n$ of $m_T$, which was expressed under the basis $\widehat{B}={\left\{{\widehat{b}}_n\right\}}_{n=1}^{\widehat{N}}$ in $\mathfrak{W}$:

$$\begin{array}{c}\hfill {\alpha }_n=m_I{\widehat{b}}_n^{\prime },{\beta }_n=m_T{\widehat{b}}_n^{\prime }\end{array}$$

(11)

Next, to strengthen the influence of semantic information in $m_T$ on $m_I$, we weakened all the components of the mapped image $m_I$, added back the mapped text $m_T$, and introduced the residual r:

$$\begin{array}{c}\hfill \mathcal{A}\left(m_I\right)=(1-\epsilon )m_I+{\displaystyle \frac{\epsilon {\sum }_{n=1}^N{\alpha }_n}{{\sum }_{n=1}^N{\beta }_n}}{m}_T+r\end{array}$$

(12)

where $\epsilon \in {\mathbb{R}}^{+}$ is a hyperparameter to control the augmenting power. In Equation (13), the first term is used to reduce the influence of both the intended and unintended attributes in $m_I$, the second term refers to adding back the influence of intended attributes in $m_T$, and the third term ensures modal consistency between the augmented image embeddings $\mathcal{A}\left(m_I\right)$ and $e_I$. It should be noted that for the second term, we weakened the unintended attributes in $m_T$, which are the components where $m_T$ has a low value, while maintaining the sum of coefficients of $m_I$.

5. Experiments and Results

5.1. Evaluation Metrics

Evaluating the accuracy of automatically generated image captions is challenging due to the diverse ways an image can be described and the potential variations in meaning even when sharing common words. Similar to a language translation task or text summarization task, image captioning uses adapted language metrics to score the similarity of generated captions to references. The following four metrics are employed in this article for the evaluation of image captioning.

BLEU [58] is a popular metric for machine translation evaluation and one of the first metrics used to evaluate image captions. It computes the geometric mean of n-gram precision scores multiplied by a brevity penalty in order to avoid overly short sentences. The BLEU-N score for a candidate caption against a ground truth sentence is calculated using

$$\mathrm{BLEU}=BP\times exp\left(\sum _{n=1}^N{w}_n·log\left(p_n\right)\right)$$

(13)

where $BP$ is the brevity penalty, which ensures that shorter translations are not unfairly favored; $w_n$ are the weights for each n-gram precision; and $p_n$ is the precision of the n-gram, which measures the degree of match between n consecutive terms (i.e., n-grams) in the candidate text and the corresponding n-grams in the reference text. In this work, N takes the values of 1, 2, 3, and 4. For the 1-gram, which is a single word, it can reflect the accuracy of the individual vocabulary. For higher-order n-grams (where $n>1$), they more significantly reflect the fluency and contextual accuracy of the translation.

ROUGE-L [59] is a package of measures initially developed for the evaluation of text summaries. ROUGE-L computes the F-measure based on the longest common subsequence (LCS), i.e., a set of words shared by two sentences that occur in the same order, without requiring consecutive matches.

METEOR [60] is another machine translation metric. It relies on the use of stemmers, WordNet synonyms, and paraphrase tables to identify matches between a candidate sentence and reference sentences.

SPICE [61] is a metric designed for image-captioning evaluation. It measures the quality of generated captions by computing an F-measure based on the propositional semantic content of candidate and reference sentences represented as scene graphs.

The ranges of BLEU, METEOR, ROUGE-L, and SPICE fall between 0 and 1. If the score of the metrics is high, it means that the predicted and the reference sentences marked by humans are highly similar.

5.2. Dataset Settings

We evaluated MAE-MilitIC on two datasets: MOCO and MADD, as detailed in Table 1. The MOCO dataset was split into a training set with 7192 images and 35,960 captions, and a testing set with 257 images and 1285 captions.

In MADD, the labels were customized for military target detection, but were not suitable for military image captioning. Therefore, we established caption sentences by building a transformation mechanism: the object name $\langle object.name\rangle$ was populated in a prompt This image includes $\langle object.name\rangle$. For example, if the label of an aircraft image is F/A-18, then the following caption may be constructed: This image includes an F/A-18 Hornet aircraft. For MADD, the dataset was divided into two parts: 13,088 images and 13,088 captions for training, and 1000 images and 1000 captions for testing.

We selected basic text prompts that include three elements: military personnel, combat equipment, and battlefield environment. Some examples of basic text prompts that form the basis of the semantic subspace are shown in

Table 2. Examples in basic text prompts.

Text Prompts		Main Elements
	Military Personnel	Combat Equipment	Battlefield Environment
Soldiers crossed the frontline under heavy fire.	Soldier		Fire
Snipers positioned on rooftops monitored enemy movements.	Sniper		Rooftop
Unmanned aerial drones conducted reconnaissance missions.		Unmanned aerial drones
UGVs transported supplies to forward positions.		UGV
Fighter jets engaged in combat maneuvers above the foggy mountains.		Fighter jet	Foggy mountains
The jet conducted a reconnaissance mission under the cover of darkness.		Jet	Darkness
The tank engaged enemy forces in the open plains.		Tank	Open plains
The military vehicle traversed the muddy battlefield under a heavy downpour.		Military vehicle	Heavy downpour
The soldiers that operated the anti-aircraft unit targeted enemy aircraft in the sky.	Soldier	Anti-aircraft unit	Sky
Armored vehicles deployed smoke screens to conceal troop movements.		Armored vehicle	Smoke screens
The howitzer launched rounds into the night sky.		Howitzer	Night sky
Cannons positioned on the coastal cliffs targeted ships at sea.		Cannon	Sea
The combat engineers are setting up fences and trenches.	Combat engineer		Trench
Engineering vehicles repaired damaged roadway.		Engineering vehicle	Roadway

.

5.3. Experimental Settings

We performed experiments on a PC with Intel(R) Xeon(R) Gold 6149 CPU @ 3.10 GHz (16 core), 24 GB RAM, and NVIDIA V100. The operating system and deep learning platforms are Ubuntu 20.04 and PyTorch 1.10.0. The configuration settings are as follows.

For the image encoder ${Enc}_{img}$ and text encoder ${Enc}_{txt}$, we employed the corresponding encoders (RN50 × 4) from CLIP [36], which were pre-trained on 400 million image–text pairs. For the text decoder ${Dec}_{txt}$, we used a transformer block combined with a pre-trained language model, following the work [37]. The transformer block included eight multi-head self-attention layers, each with eight heads. The pre-trained language model was GPT-2 (Generative Pre-trained Transformer 2), which employed implementation details from previous work by Wolf et al. [62]. To maximize the text generation capabilities of the pre-trained language model on large-scale data, our work froze the model parameters, and left only the transformer block as the trainable component.

In the training process, we trained for 10 epochs using a batch size of 40. For the optimization, we used AdamW [63] with a weight decay fix, as introduced by Loshchilov et al. [64], with a learning rate of $2\times {10}^{-5}$ and 5000 warm-up steps. The augmenting power $\epsilon$ was 0.1 and the loss weight coefficient $\lambda$ was 0.1. In the testing process, the beam-search algorithm was adopted for sentence generation and the beam size was set to 5. Finally, the length of the generated sentences was set to no longer than 67 words.

5.4. Experimental Results

Considering that the previous MilitIC methods were not open source and had limited training data, we employed general image-captioning methods for comparison with our approach. As two typical methods for encoder–decoder and encoder (VL)–decoder architectures, ClipCap [37] and CapDec [11] show good performance in general image captioning. In the experiments, we compared MAE-MilitIC with ClipCap and CapDec. These methods first produce visual features by CLIP, which is pre-trained over millions of image–text pairs and then utilizes an large language model to generate the captions. ClipCap employs the CLIP visual features as a guide for caption generation. CapDec utilizes the single text modality and introduces a text-only training setup during training. However, the single-modality method is not optimal and fails to leverage the complementary nature of the two modalities for mutual enhancement. Thus, we introduced the MAE-MilitIC to augment the semantics of image embeddings with text embeddings in the semantic subspace, which provides better guidance for caption generation.

Table 3. Quantitative results of MAE-MilitIC and baselines on MOCO. Bold indicates the best result, while underlined indicates the second-best result.

Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	SPICE
ClipCap	0.441	0.275	0.173	0.111	0.158	0.337	0.102
CapDec	0.363	0.184	0.086	0.041	0.111	0.264	0.059
MAE-MilitICGSO	0.494	0.322	0.212	0.141	0.163	0.353	0.113
MAE-MilitICPCA	0.460	0.290	0.181	0.112	0.158	0.336	0.102

shows the quantitative results of three methods on MOCO. The GSO variant of MAE-MilitIC achieved the best results, while the PCA variant yielded the second-best results. As a method of using images for guidance, ClipCap achieved a moderate performance, and as a method of training using text, CapDec exhibited poor captioning performance. The reason was that the last two methods failed to overcome the inherent modality gap present in the VL joint space. Furthermore, as shown in Figure 5, we collected 200 images and 200 textual descriptions in MOCO test data and visualized their embeddings in the VL joint space and semantic subspace. As shown in Figure 5a, the image and text embeddings in the VL joint space almost lie in two non-overlapping regions and exhibit a certain modal gap. In contrast, Figure 5b,c illustrate that the augmentation embeddings, as a better substitute for image embeddings, can be closer to the attribute-explicit text embeddings in the semantic subspace. Therefore, the use of both image and text data for aligning military semantics is necessary, and the aforementioned results also prove this point.

The quantitative results of MAE-MilitIC and baselines on MADD are shown in

Table 4. Quantitative results of MAE-MilitIC and baselines on MADD.

Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	SPICE
ClipCap	0.547	0.443	0.369	0.271	0.230	0.576	0.437
CapDec	0.542	0.445	0.370	0.265	0.241	0.586	0.430
MAE-MilitICGSO	0.553	0.452	0.379	0.282	0.236	0.583	0.449
MAE-MilitICPCA	0.555	0.448	0.371	0.269	0.241	0.578	0.459

. As can be seen, MAE-MilitIC demonstrated an excellent overall caption performance. We note that CapDec obtained much better results on the ROUGE-L metric. The reason was that the caption template in MADD contained the unified subsequence This image includes, while ROUGE-L heavily focused on the longest common subsequence (LCS), which created the illusion that ROUGE-L was better. Therefore, only one single metric that was inflated did not necessarily equate to high-quality caption. The two variants of MAE-MilitIC achieved the best caption generation performance on most metrics. Figure 6 demonstrates that our MAE mechanism could further align the image and text modalities in the military semantic subspace and offer a superior guiding prefix for image captioning.

The visual results of MAE-MilitIC and previous works in MOCO are presented in Figure 7. In this figure, we present the visual results for eight military scenarios from four perspectives, including long-range perspectives (the first, fifth, and sixth images), low-altitude perspectives (the second and fourth images), a partial view (incomplete imaging, the third image), and close-up perspectives (the seventh and eighth images). As can be seen, our MAE-MilitIC can maintain semantic consistency in objects, actions, and environment with the ground truth, and overall successfully depict the image. As a text-only training method, CapDec missed out on major visual content and produced a large number of hallucinations, with the former due to the weak visual ability it inherited from CLIP, and the latter because it did not align vision and text embeddings well. The ClipCap, without semantic enhancement, was not accurate enough to describe military scenarios. For example, in the fifth image of Figure 7, it failed to describe the military high-level semantic concepts of outposts and trenches, and in the third image, it was unable to capture environmental information. The fourth image of Figure 7 illustrates the limited mathematical counting capability of the three methods, which we attributed to a weaker visual encoder that captured only two military soldiers (the corresponding heatmaps are shown in Figure 8). Moreover, as shown in Figure 8, we utilized the relevancy heatmap [65] to demonstrate the consistency between the images and the main caption components, which was generated by the image encoder ${Enc}_{img}$ and text encoder ${Enc}_{txt}$. The results indicate that MAE-MilitIC described the image content accurately and produced almost no hallucinations. Additional visualization results for complex military scenarios can be found in the Appendix A.

In summary, from various image perspectives, the main objects and environments in the MAE-MilitIC captions were well reflected in the images, which indicate that our method could accurately describe the images and avoid the generation of hallucinations. This could be attributed to the semantic subspace effectively preserving the relevant target semantics while weakening the semantics of irrelevant targets.

5.5. Ablation Studies

In this section, we optimized the parameters to identify the best configuration for our method and evaluated the effectiveness of each step in the MAE mechanism.

5.5.1. Parameter Optimization

To achieve the best model performance, we employed the grid search method for parameter optimization. We selected candidate values for $\lambda$ from [0, 0.1, 0.3, 0.5] and for $\epsilon$ from [0.1, 0.3, 0.5]. There were 12 parameter combinations in total. After training in the MOCO dataset, the experimental results are shown in Figure 9a. The best BLEU-1 result was 0.494 when the parameter combination was set to $\lambda =0.1$ and $\epsilon =0.1$. For the MAE-MilitIC based on PCA mapping, the number of principal components $\widehat{N}$ was selected from [5, 10, 20, 30, 40]. As illustrated in Figure 9b, the metrics BLEU-1, METEOR, and ROUGE-L were consistently achieved optimal values when $\widehat{N}$ was set to 10. Thus, we determined that the values of the three parameters were $\lambda =0.1$, $\epsilon =0.1$, and $\widehat{N}=10$.

5.5.2. Ablation of MAE-MilitIC

To explore the impacts of the main components in MAE-MilitIC, i.e., the mapping step, the augmentation step, and the residual item, we conducted comprehensive ablation studies on the MOCO dataset. We evaluated MAE-MilitIC with both GSO and PCA mapping, whose results are provided in

Table 5. Ablation studies of MAE-MilitIC on MOCO.

Methods	Mapping	Augmentation	Residual	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	SPICE
MAE-MilitICGSO	✓	✗	✗	0.335	0.183	0.102	0.057	0.115	0.279	0.068
	✗	✓	✗	0.440	0.276	0.174	0.112	0.160	0.331	0.106
	✓	✓	✗	0.475	0.303	0.193	0.124	0.156	0.346	0.107
	✓	✓	✓	0.494	0.322	0.212	0.141	0.163	0.353	0.113
MAE-MilitICPCA	✓	✗	✗	0.396	0.220	0.123	0.073	0.127	0.299	0.076
	✗	✓	✗	0.440	0.276	0.174	0.112	0.160	0.331	0.106
	✓	✓	✗	0.453	0.278	0.172	0.105	0.150	0.325	0.100
	✓	✓	✓	0.460	0.290	0.181	0.112	0.158	0.336	0.102

.

As we can see, for two variants of MAE-MilitIC, retaining only the augmentation step yielded the most obvious increase in results. This indicates that utilizing both image and text data, as well as employing one modality to augment the other, was effective. Furthermore, as is shown in the fourth row of Table 5, when using both the mapping and augmentation step, the performance continued to improve. By mapping to a semantic subspace instead of a VL joint space, the augmentation embedding yielded better results. This confirmed that the mapping step could highlight clear military semantics. Finally, after adding the residual item to ensure modal consistency between $\mathcal{A}\left(m_I\right)$ and $e_I$, the best performance of caption generation was achieved in the fifth row of Table 5. Moreover, for the results of MAE-MilitIC^PCA, retaining only augmentation yielded more competitive results than keeping both augmentation and mapping, yet it performed worse compared with retaining all three components. It can be inferred that using a mapping step that ignores the modal consistency can harm the method’s performance.

As a summary, both the mapping step, the augmentation step, and the residual item were helpful for the image-captioning task. Our MAE-MilitIC achieved the best performance when generating accurate military image captions.

6. Conclusions

In this study, we endeavored to improve the performance of MilitIC by offering a reliable solution for the advancement of military image understanding and intelligence generation. One of the contributions of our work was to build the first open-access MilitIC benchmark dataset MOCO, which includes detailed images of real combat scenarios and a rich set of captions. Another contribution was the proposal of a novel encoder–augmentation–decoder architecture MAE-MilitIC, which allows the MAE mechanism to leverage image–text modalities for guiding caption generation and to augment image embeddings with explicit military text in a semantic subspace. The efficacy of MAE-MilitIC was assessed across MOCO and MADD datasets by utilizing various baselines for captioning military images.

This research contributes to understanding military images from UAVs and UGVs, and generates accurate captions for challenging scenarios. Future work will leverage the modal alignment and semantic augmentation capabilities of the MAE mechanism to perform text-based image retrieval, which will quickly identify relevant visual patterns in large image data from UAVs and UGVs.