An Image Captioning Algorithm Based on Combination Attention Mechanism

Abstract

With the maturity of computer vision and natural language processing technology, we are becoming more ambitious in image captioning. In particular, we are more ambitious in generating longer, richer, and more accurate sentences as image descriptions. Most existing image caption models use an encoder—decoder structure, and most of the best-performing models incorporate attention mechanisms in the encoder—decoder structure. However, existing image captioning methods focus only on visual attention mechanism and not on keywords attention mechanism, thus leading to model-generated sentences that are not rich and accurate enough, and errors in visual feature extraction can directly lead to generated caption sentences that are incorrect. To fill this gap, we propose a combination attention module. This module comprises a visual attention module and a keyword attention module. The visual attention module helps in performing fast extractions of key local features, and the keyword attention module focuses on keywords that may appear in generated sentences. The results generated by the two modules can be corrected for each other. We embed the combination attention module into the framework of the Transformer, thus constructing a new image caption model CAT (Combination Attention Transformer) to generate more accurate and rich image caption sentences. Extensive experiments on the MSCOCO dataset demonstrate the effectiveness and superiority of our method over many state-of-the-art methods.

1. Introduction

A long-term goal in the development of artificial intelligence is to develop agents that can perceive and understand rich visual environments around us and communicate with us using natural language [1]. Therefore, it is vital to research image descriptions.

Image caption is a typical multimodal learning problem, as shown in Figure 1, which is a process from vision to language and contains several core problems of representation, alignment, translation, and integration. Image caption requires us to combine techniques and knowledge from multiple fields, such as computer vision and natural language processing. Early image caption models are based on templates that generate description sentences by identifying objects, object attributes, and object relationships in images to match language templates. Existing models gradually get rid of the limitations of earlier target recognition methods and template-based text generation, and most of them adopt an encoder-decoder structure. The best-performing models mostly incorporate the attention mechanism in the encoder–decoder structure. However, regardless of whether the attention mechanism is channel-based or space-based, it is a visual attention mechanism. In the domain of image caption, if only visual features are attended to, errors in visual features will directly lead to errors in the generated description statements.

To solve the above problems, we propose a combination attention module that contains a visual attention sub-module and a keyword attention sub-module. The visual attention sub-module can use image spatial features with a high degree of abstraction to guide the model to focus on key regions of the image. The keyword attention sub-module extracts keyword semantic information to guide the generation of richer and more accurate image description statements. We embedded the combination attention module into the Transformer framework and constructed the CAT (Combination Attention Transformer) model to help generate richer and more accurate image description statements. CAT solves the traditional image caption method’s neglect of global background information and keyword information, which often leads to inaccurate and rich description sentences. We evaluated our method on the MSCOCO image caption dataset, and CAT improved the CIDEr score by 0.2% over the current state-of-the-art method.

2. Related Work

2.1. Image Caption

Image caption is a technique that attempts to “translate” images into text using machine learning methods [2,3]. The research history of the image caption problem is only ten years old, but the image caption model has experienced many major changes. Kiros et al. [4] proposed a method to solve the image caption problem using neural networks. A multimodal embedding space model based on a deep visual model and a long short-term memory unit can accomplish retrieval ranking as well as the image caption task. After that, many scholars tried to use deeper neural networks [5] or other network structures such as gated recurrent units (GRU), long short-term memory units (LSTM) [6], bidirectional LSTM [7], and phi-LSTM [8] to solve the image caption problem. Yao et al. [9] proposed an attribute-based long short-term memory network (LSTM-A) that added advanced attribute features to the combined model of CNN and LSTM and used a weakly supervised multi-instance learning method to complete the image caption task. In the last two years, there have been many attempts directed toward introducing attention mechanisms into the field of image caption research [10,11,12], among which the M2 Transformer model [10], designed by Cornia et al., achieved the most advanced image caption results so far. They used a multi-layer attention mechanism to extract features of images and text and used mesh connections to learn the connection between the two features. At the same time, various decision-making methods of reinforcement learning and generative adversarial networks have also been used in the study of image captioning problems [13], increasing the naturalness and diversity of the generated texts.

2.2. Attention Mechanism

The attention mechanism was initially a human-specific means of using limited cognitive resources to quickly sift through large amounts of information. Humans would learn the target areas to focus on by quickly scanning the global image and then they would devote more attentional resources to the target areas to obtain more detailed information about the target of attention.

The attention mechanism was first applied in the field of computer vision [14], and Xu et al. [15] first applied the attention mechanism to the image captioning algorithm. Its inspiration comes from machine translation, which compares image-to-language translation to language-to-language translation. The encoder–decoder framework is used, but the method of processing text is replaced by the method of processing images in the encoder and then the images are inputted into the decoder to generate natural language.

The model based on the attention mechanism can not only log the positional relationship between information but also measure the importance of different information features according to the weight of the information. Dynamic weight parameters are established by making relevant and irrelevant choices for information features to strengthen key information and weaken useless information, thereby improving the efficiency of deep learning algorithms and improving some of the defects of traditional deep learning. However, whether it is a channel-based or a spatial-based attention mechanism, it is a visual attention mechanism. Image understanding methods that use only a visual attention mechanism may not necessarily extract all the features that need attention effectively. Inaccurate visual features may be extracted, so using only a visual attention mechanism may lead to the problem of generated descriptive statements that are inaccurate and not rich.

2.3. Transformer

In 2017, Vaswani et al. [16] proposed Transformer and applied it to the field of machine translation. Transformer consists of an encoder and a decoder, where the encoder is responsible for mapping the input sequence to the hidden layer after positional encoding, and then the decoder maps the hidden layer to the output sequence. The encoder and decoder each consists of several Transformer blocks with the same structure. Each Transformer block consists of a multi-headed attention layer, a feedforward neural network, residual connection, and layer normalization.

Transformer was first applied in the field of natural language processing (NLP) [16]. After that, Devlin et al. [17] proposed Bidirectional Encoder Representations from Transformers (BERT), which is pre-trained on unlabeled text considering the context of each word. Brown et al. [18] pre-trained a Transformer-based model named Generative Pre-trained Transformer 3 with 175 billion parameters on 45TB of compressed plaintext data. It showed excellent performance without any fine-tuning. These Transformer-based models have achieved major breakthroughs in NLP because of their powerful representational capabilities.

Inspired by the powerful representational capabilities of the Transformer, researchers have started to apply the Transformer to object detection [19,20], image processing [21], semantic segmentation [22], and video captioning [23] tasks. Dosovitskiy et al. [24] proposed ViT, which applies a pure transformer directly to a sequence of image patches to classify the entire image, and it achieved state-of-the-art performances on several image recognition benchmarks. Ji et al. [25] proposed a globally enhanced transformer and applied it to solve the image description problem, which achieved exciting results.

Transformer solves many problems by utilizing the encoder–decoder framework, attention mechanism, layer normalization, and feedforward neural network. However, Transformer also has other drawbacks, such as only being able to efficiently process long sequences, while the processing performance of short sequences has not been improved. For the image caption problem, the transformer encodes the object region and then transforms it into a vector representation to guide the decoding of the description statement. This approach only obtains region-level information but not global information. In this paper, we propose a combination attention module that incorporates a visual attention module and a keyword attention module. The visual attention sub-module can use the spatial features of the image with a high degree of abstraction to guide the model to focus on the key regions of the image, and the keyword attention sub-module extracts keyword semantic information to guide the generation of richer and more accurate image description statements. The results generated by both attention sub-modules can be corrected for each other. After that, we embedded the combination attention module into the Transformer framework and built the CAT image description module to generate richer and more accurate image description statements.

3. Methods

3.1. Combination Attention Module

The attention mechanisms used in current image understanding methods are all visual attention mechanisms. This may lead to the incomplete extraction of features that need attention and the extraction of incorrect visual features. Thus, using only visual attention mechanism may lead to the problem of inaccurate and rich generated description statements. Therefore, we propose using a combination attention module. As shown in Figure 2, the combination attention module consists of a keyword attention module and a visual attention module. The keyword attention module allows the model to focus on the important keyword text by associating keywords with visual features to generate richer and more accurate descriptive statements. The visual attention module uses spatial features with a high degree of abstraction to direct the model’s attention to key regions of the image and generate more accurate descriptive sentences. The results generated by the two attention modules can be corrected for each other, resulting in more accurate and richer image caption statements.

3.1.1. Keywords Attention Module

The existing dataset of image caption domain contains only images and their corresponding description statements. To train our proposed combination attention model, we additionally constructed a keyword dataset based on the image caption dataset. In the process of constructing the keyword dataset, we picked out high-frequency words, merged synonyms in descriptive statements, and increased the weight of some words on which the dataset needed to focus on. The process of constructing the keyword dataset is shown in Figure 3.

In Figure 3, $C$ denotes any of the caption sentences in the dataset $\left\{\mathit{I},\mathit{C}\right\}$. Firstly, we constructed a word list ${\mathit{K}}_0=\left\{\left\{W_1,N_1\right\}\right.,\left.\left\{W_2,N_2\right\},\cdots ,\left\{W_m,N_m\right\}\right\}$ from the image caption sentence $C$, where $N_i$ denotes the number of occurrences of word $W_i$ in all descriptive statements $\mathit{C}$, and m denotes the total number of words. We selected the words with $C_i$ greater than $\left|\mathit{C}\right|/1000$ to form ${\mathit{K}}_1$ to pick out the words with high frequency, and then we selected and picked out some words of lexical nature (e.g., coronal words and adverbs) in the word list ${\mathit{K}}_1$, keeping nouns, verbs, and adjectives in order to obtain the keyword word list ${\mathit{K}}_2$. The keyword word list ${\mathit{K}}_3$ was obtained by merging the synonyms in ${\mathit{K}}_2$, e.g., courage and gut were merged into courage. Finally, the keyword dataset ${\mathit{K}}_4$ is obtained by providing appropriate weights to the keywords by utilizing weakly supervised learning and fine-tuning. The dataset $\left\{\mathit{I},\mathit{C}\right\}$ can be expanded into a dataset $\left\{\mathit{I},\mathit{C},\mathit{K}\right\}$ with keyword data by performing the above operations on all description statements in the dataset $\left\{\mathit{I},\mathit{C}\right\}$.

Different keyword weights indicate that the model pays different levels of attention to the words. The greater weight of the words indicates that they are more important, and their impact on the generated description statements is greater. Keyword weighting is calculated as follows:

$${\mathit{a}}_i^{*}={\mathit{W}}_a\mathrm{ReLU}\left({\mathit{W}}_{\mathrm{ka}}{\mathit{k}}_i+{\mathit{W}}_{\mathrm{ha}}{\mathit{h}}_i\right)$$

(1)

$${\mathit{a}}_i=softmax\left({\mathit{a}}_i^{*}\right)$$

(2)

$${\widehat{\mathit{k}}}_i={\displaystyle \sum }_{i=1}^n{a}_i{\mathit{k}}_i$$

(3)

where ReLU is the modified linear unit activation function. ${\mathit{k}}_i$ is the $i$th component of the keyword vector feature matrix, and ${\mathit{W}}_a$ is the weight to be learned. ${\mathit{W}}_{\mathrm{ka}}$ and ${\mathit{W}}_{\mathrm{ha}}$ are weight matrices; ${\mathit{h}}_i$ is the hidden state of the $i$th word; ${\mathit{a}}_i^{*}$ is the keyword weight of the $i$th word; ${\mathit{a}}^{*}=\left\{a_0^{*},a_1^{*},\cdots ,a_n^{*}\right\}$ is the unnormalized weight vector; $n$ is the number of components of keyword feature $\mathit{k}$; $\mathit{a}=\left\{a_0,a_1,\cdots ,a_n\right\}$ is the unnormalized weight vector; ${\widehat{\mathit{k}}}_i$ is the keyword feature value corresponding to the $i$th word.

Figure 4 is a schematic diagram of the keyword attention module. Keywords are representative words in description statements. Introducing keyword attention not only can result in better image features, but it can also guide the generation of description statements to formulate more accurate and richer generated description statements [26].

$\mathit{W}=\left\{w_1,\cdots ,w_n\right\}$ is a sequence of words generated based on visual attention, and $\mathit{K}=\left\{k_1,\cdots ,k_m\right\}$ is a sequence of keywords. The two representations are made to interact with each other to learn a similarity measure that discriminates whether the generated words of visual attention are relative to the word pairs in the keyword list, and then we take the following maximum value:

$$a_i=ma{x}_j\left(w_i\cdot k_j\right)$$

(4)

where $\cdot$ denotes the dot product, $w_i$ denotes the $i$th word in $\mathit{W}$, and $k_j$ denotes the $j$th word in $\mathit{K}$. Then, the word sequence is represented by the $w_i$ and $a_i$ dot product ${\mathit{W}}^{\prime }$.

$${\mathit{W}}^{\prime }=\left\{w_1\cdot a_{1,\cdots },w_n\cdot a_n\right\}$$

(5)

After the softmax operation, we completed the operation of probabilistically encoding the words in the dictionary.

3.1.2. Visual Attention Module

The visual attention module can help in quickly extracting key local features. Early attention mechanisms only feed region information into the encoder to extract their vector representations. These vector representations only contain image information features at the region-of-interest level, even though the vector representations of individual regions are computed hierarchically through all regions in the image. This paper adopts the globally enhanced attention mechanism proposed by Ji et al. [25], which solves the problem of Transformer’s poor global attention to a certain extent.

As shown in Figure 5, to capture the global representation information, we fed both regional feature $V$ and global feature $g$ into the visual attention module. In this way, we captured global information within the feature layer. Specifically, the output $O^l$ of the $l \mathrm{th} (0\le l<L)$ layer was inputted into the visual attention module of the $l+1$ layer, followed by residual connection and layer normalization, which are computed as shown below:

$${\overline{V}}^{l+1}= \mathrm{MultiHead} \left(O^l,O^l,O^l\right)$$

(6)

$$V^{l+1}= \mathrm{Layer} N\mathrm{orm} \left(O^l+{\overline{V}}^{l+1}\right)$$

(7)

where $O^0=\left(V^0;g^0\right)$. ${\overline{V}}^{l+1}$ represents the output of $O^l$ after passing through the multi-head attention layer. The output $V^{l+1}$ of the $l+1$ layer can be obtained by performing residual connections between ${\overline{V}}^{l+1}$ and $O^l$ and then performing layer normalization. The residual connection helps in avoiding gradient disappearances during the training phase. Then, the final feedforward neural network is used to perform additional processing on the output.

$$O^{l+1}= \mathrm{LayerNorm} \left(V^{l+1}+FFN\left(V^{l+1}\right)\right)$$

(8)

The representations of different layers have different meanings. Therefore, we integrated global representations from different layers by pooling to fuse all information. The process of integration also eases the flow of information in the stack.

3.2. Framework

3.2.1. Encoder

The overall architecture of CAT is shown in Figure 6. Firstly, we resized the input image into $X\in R^{H\times W\times 3}$ resolution. After that, we divided the resized image into $N$ patches, $N=\frac{H}{P}\times \frac{W}{P}$, and $P$ is the set patch size. Then, we reshaped each patch as a sequence of one-dimensional patches $X_p\in R^{N\times \left(P^2\cdot 3\right)}$. We used a linear embedding layer to map the patch sequences into the feature space and embedded learnable location information into patch features [27], after which we obtained the final input of CAT, which is denoted as $P_a=\left[p_1,\dots ,p_N\right]$.

The encoder of CAT consists of a stack of $N$ identical modules, with each layer consisting of a combination attention sub-layer and a position feedforward sub-layer. The combined attention sub-layer contains a visual attention module and a keyword attention module, with each corresponding to an independent scaled dot product attention function [16], which allows the model to jointly focus on different subspaces. We used a linear transformation $W^O$ to aggregate the keyword attention module and visual attention module attention results, and this process can be represented by the following equation:

$$CA(Q,K,V)=Concat\left(h_k,h_v\right)W^O$$

(9)

where $Q$, $K$, and $V$ are the query, key, and value matrix, respectively. $Q$ consists of $n_q$ query vectors, and $K$ and $V$ both contain $n_k$ keys and values. $h_k$ denotes the scaled dot product attention function corresponding to the keyword attention module, and $h_v$ denotes the scaled dot product attention function corresponding to the visual attention module.

The scaled dot-product attention function is one of the core components of Transformer, and all intra- and inter-modal interactions between words and image features are modeled by the scaled dot product attention function. It can be computed in the following manner:

$$Attention(Q,K,V)=Softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

(10)

where $d_k$ is the dimension of $Q$, $K$, and $V$.

The following position feedforward sub-layer comprises two linear layers with a GELU activation function and dropout between them to further transform the features.

$$FFN(x)={\mathrm{FC}}_2\left(Dropout\left(\mathrm{GELU}\left({\mathrm{FC}}_1\left(x\right)\right)\right)\right)$$

(11)

In each sub-layer, we used a residual connection around each sub-layer and then performed layer normalization:

$$\left.x^{\mathrm{out} }=LayerNorm\left(x^{\mathrm{in} }+Sublayer\left(x^{in}\right)\right)\right)$$

(12)

where $x^{in}$ and $x^{\mathrm{out} }$ are the input and output of a sub-layer, which can be an attention layer or a feedforward layer, respectively.

3.2.2. Decoder

The decoder of CAT is also formed by stacking $N$ identical modules. Each layer contains a masked self-attention, a combination attention module, and a position feedforward sub-layer. Similarly to the encoder, we used a residual connection around each sub-layer and then performed layer normalization. We added the position information of the word to the features of the word and used the result of addition and the encoder’s output features as the input to the decoder.

The output of the previous decoding layer is utilized to predict the next word by a linear layer with an output dimension equal to the vocabulary size.

3.3. Training Details

Following the standard practice of image captioning [28,29,30], we utilized the word-level cross-entropy loss (XE) to pre-train our model and used reinforcement learning to fine-tune sequence generation. In the pre-training process, given the ground truth sequence $y_{1:T}^{*}$ and the prediction $y_t^{*}$ of the captioning model with parameter $\theta$, the following cross-entropy loss function was minimized.

$$L_{XE}\left(\theta \right)=-{\displaystyle \sum }_{t=1}^{T}log\left(p_{\theta }\left(y_t^{*}\mid y_{1:t-1}^{*}\right)\right)$$

(13)

In the fine-tuning stage, we employed a self-critical sequence training method on sequences sampled using beam search. For decoding, we drew the top $k$ words from the decoder’s probability distribution at each time step and always kept the one with the highest probability. The top $k$ sequences and final gradients for a sample can be computed in the following ways:

$${\nabla }_{\theta }L\left(\theta \right)=-\frac{1}{k}{\displaystyle \sum }_{i=1}^k\left(\left(r\left({\mathit{y}}^i\right)-b\right){\nabla }_{\theta }\log p\left({\mathit{y}}^i\right)\right)$$

(14)

where ${\mathit{y}}^i$ is the $i$th sentence in the beam, and $r\left(\cdot \right)$ can be the score of any evaluation metric; we use the score of CIDEr-D as the reward. $b=\left({{\displaystyle \sum }}_{i}r\left({\mathit{w}}^i\right)\right)/k$ is the baseline. It is calculated as the average of the rewards obtained by the sampling sequence. At the prediction phase, we performed decoding again using beam search and kept the sequence with the highest predicted probability in the last beam.

4. Experiments

4.1. Dataset and Implementation Details

Our experiments were all conducted on the MSCOCO [31] dataset, which contains 123,287 images, including 82,783 training images, 40,504 validation images, and 40,775 test images. Each image is equipped with five ground truth sentences. We reported the results of the offline evaluation on the “Karpathy” test split [32] and the results of the online evaluation on the MSCOCO test server.

We trained our model in end-to-end fashion with the encoder initialized by a pre-trained ViT model. The input image was resized to a resolution of $384\times 384$, the encoder contained 12 layers, and the decoder contained 4 layers. The feature size is 768, and after each attention and feedforward layer, we used a dropout with a probability of 0.9. We fine-tuned the model using self-critical training [30] for 10 epochs with an initial learning rate of 0.01, decaying to $5\times {10}^{-6}$ after the last 2 epochs. During CIDEr-D optimization, we used a fixed learning rate of $5\times {10}^{-6}$. We used the Adam optimizer with a batch size of 32. Using beam search, the beam size was 3.

We used BLEU [33], METEOR [34], ROUGE-L [35], and CIDEr [36] to evaluate our model.

4.2. Performance Comparison

We compared the performance of CAT with SCST [30], LSTM-A [9], Up-Down [28], RFNet [37], GCN-LSTM [38], SGAE [39], and ETA [40]. All the methods mentioned use the image features of CNN or an object detector as input, while our model takes the original image directly as input.

Table 1. Performance comparison on the MSCOCO online test server. All values are reported as percentages (%). All models were fine-tuned by self-critical training.

Model	BLEU-1		BLEU-2		BLEU-3		BLEU-4		METEOR		ROUGE		CIDEr
	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40
SCST	78.1	93.7	61.9	86.0	47.0	75.9	35.2	64.5	27.0	35.5	56.3	70.7	114.7	116.0
LSTM-A	78.7	93.7	62.7	86.7	47.6	76.5	35.6	65.2	27.0	35.4	56.4	70.5	116.0	118.0
UP-Down	80.2	95.2	64.1	88.8	49.1	79.4	36.9	68.5	27.6	36.7	57.1	72.4	117.9	120.5
RF-Net	80.4	95.0	64.9	89.3	50.1	80.1	38.0	69.2	28.2	37.2	58.2	73.1	122.9	125.1
GCN-LSTM	-	-	65.5	89.3	50.8	80.3	38.7	69.7	28.5	37.6	58.5	73.4	125.3	126.5
SGAE	81.0	95.3	65.6	89.5	50.7	80.4	38.5	69.7	28.2	37.2	58.6	73.6	123.8	126.5
ETA	81.2	95.0	65.5	89.0	50.9	80.4	38.9	70.2	28.6	38.0	58.6	73.9	122.1	124.4
CAT	81.6	96.0	66.5	89.6	51.7	82.8	39.7	72.2	29.1	38.5	58.9	74.4	125.4	127.6

shows the performance comparison results in the online MSCOCO test server, where CAT improves CIDEr by 0.2% over the current state-of-the-art method, achieving a CIDEr score of 127.6. The evaluation results in the offline “Karpathy” test split shown in Figure 7, which also demonstrates the validity of our CAT model.

4.3. Ablation Study

To verify the effectiveness of CAT, we conducted ablation studies from the following aspects.

Firstly, we investigated the effects of different resolutions of the input image on the output image’s description statements. We found that increasing the resolution of the incoming image from $224\times 224$ to $384\times 384$ while keeping the patch size at 16 resulted in a large performance improvement (from 110.3 CIDEr score to 116.6 CIDEr score). We attributed this to the fact that the increase in input resolution allows the length of the patch sequence to increas from 196 to 576, thus allowing more specific delineations of the image and providing more features. Then, we investigated the effects of the number of encoding layers and the number of decoding layers on the output’s description statements. The results are shown in

Table 2. Effect of the ablation on the number of coding and decoding layers. All values are reported as percentages (%).

Layer	BLUE-4	METEOR	ROUGE-L	CIDEr
2	38.1	27.9	59.4	125.5
3	39.3	29.1	58.9	127.9
4	39.5	29.2	58.9	128.7
5	39.2	29.2	58.5	127.1
6	39.2	29.1	58.5	127.0

. When the number of layers gradually increased from 2 to 4, the performance also increases gradually. When the number of layers continued to increase, performance degrades. This may be due to the problems of gradient instability, network degradation, and reduced learning ability at certain shallow layers brought about by the increase in the number of layers. Therefore, we kept the number of encoding layers and decoding layers at four for all subsequent experiments. After that, we investigated the effect of our proposed combination attention module on the output’s description statements. We chose the traditional Transformer as the baseline, we then extended the baseline model by adopting the combination attention module, and the performance also significantly improved (from 126.1 CIDEr score to 128.7 CIDEr score).

4.4. Attention Visualization

To better evaluate the effectiveness of our proposed method qualitatively, we visualized the process of generating image captions by the CAT model in Figure 8. The results in Figure 7 show that, at shallow layers, the model mainly focuses on the background. In the middle layers, the model tends to focus on the main object or the “man” in the image. The last layer fully utilizes the global background and focuses on all objects in the image that need attention, namely “man”, “wave”, and “surfboard”. In addition, our method can help the model to focus on the target region that requires extraction in order to generate the correct word through the keyword attention mechanism.

5. Conclusions

In this paper, we proposed a combination attention module and embedded it into the Transformer architecture to build CAT. CAT solves the traditional image caption method’s problem of neglecting global background information and keyword information of the image, and this neglect leads to less accurate and rich generated description statements. Our proposed combination attention module consists of a visual attention module and a keyword attention module. The visual attention module can help with the fast extraction of key local features and capture the global representation within and between layers of image features to provide more comprehensive key feature information. The keyword attention module captures keywords that may appear in the generated sentences, and the results produced by the two modules can be mutually corrected to generate longer, more accurate, and richer caption sentences. The evaluations on the MSCOCO dataset demonstrated the superiority of our method.