Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion
Abstract
:1. Introduction
- Visual feature research: We investigated the impact of visual elements within the model and validated the approach of incorporating these elements into MMT. Our specific findings indicate that adding visual features enhances the performance of MMT in text prediction.
- Multimodal feature fusion: We proposed a method to generate new multimodal features by combining image features with text features using knowledge distillation technology. This method provides new insights into MMT with cross-modal feature fusion.
- DSKP-MMT model: We studied the application of the DSKP-MMT model combined with a pre-trained model in the MMT system. The experimental results clearly demonstrate the superior performance of this method over the existing image-text fusion MMT framework, thereby affirming the effectiveness of our model in leveraging image information to enhance machine translation.
2. Related Works
2.1. MMT Backbone
2.2. Knowledge Distillation
2.3. Image-Text Feature Fusion
3. Method
3.1. Mirrorless MMT Backbone
3.2. Multimodal Feature Generation
3.2.1. Preliminaries
3.2.2. Knowledge Distillation
- (1)
- The student model S is directed to derive crucial visual representation details directly from the source text, thereby establishing an inter-modal semantic connection between textual content and actual images. In each layer of the teacher model T, when provided with a real image , a visual representation is produced, alongside a corresponding reverse hidden representation in the subsequent layer . The pair of representation features and share identical dimensionalities and encompass comparable latent concepts. The loss is defined as the sum of discrepancies between these two representations, supplemented by an auxiliary regularization term, formulated as follows:
- (2)
- The constrained student model S acquires visual perception from the image using the inverse features, thereby mitigating the intra-modal disparity between these features and the actual images. In this process, we input the inverse features into the teacher model T to derive the teacher’s pseudo-visual representation of its cognition. Next, to motivate the student model to thoroughly learn the image distribution, we reduce the gap between the pseudo-visual representation and its corresponding visual representation . Consequently, is described as a blend of the difference and the image space loss, formulated as follows:
- (3)
- To constrain the student model S and enhance the effective information transfer from the teacher model T, we employ a method to progressively close the gap between known and predicted information, thus improving the precise representation capability of features. Specifically, we achieve this task objective by reducing the difference between the inverted hidden representation generated by the student model S in the next layer and the pseudo visual representation predicted by the teacher model. Then we calculate the new loss layer by layer, starting from the information successfully converted from the previous level and the final information predicted, formulated as follows:
3.3. Feature Refinement
3.4. Objective Function
4. Experiments and Analysis
4.1. Dataset
4.2. Image Feature Extraction
4.3. Training Parameters
4.4. Main Results
- (1)
- MMT systems utilizing image constraints typically surpass those lacking them, demonstrating that incorporating visual data can significantly enhance translation quality.
- (2)
- Our BLEU and METEOR scores using the DSKP-MMT model significantly outperform the test results of the MMT system without introducing images. This enhancement demonstrates our model’s capability to integrate multimodal features seamlessly during training and utilize them to facilitate translation in testing phases.
- (3)
- Our DSKP-MMT model not only exceeds the performance of the existing image-constrained MMT system but also rivals the current state-of-the-art image-constrained MMT system (SOTA).
4.5. Ablation Research
- (1)
- By investigating how different similarity functions impact the assessment of divergence between hidden representations within a module, we determined that the norm produces the optimal results, with the remaining performance ordered as Kl-Div > > > Consine.
- (2)
- By comparing the impact of enhanced knowledge distillation at each neural network layer and block, we observed that the best evaluation results were achieved through distillation at the whole-model level. This emphasizes that utilizing information from both initial and final representations, while reinforcing relationships, effectively guides the student model in producing enriched features. This illustrates the superiority of our knowledge distillation module over traditional approaches.
- (3)
- By studying three image feature extraction methods using different CNNs, we determined that the ResNet50 network structure excelled due to its ability to extract the most robust visual feature information among the deep residual networks. Due to the absence of residual connections and the requirement for a substantial number of training samples, the performance of the VGG19 network structure was subpar. Undoubtedly, the lightweight AlexNet network structure caused the most serious translation performance degradation. This demonstrates that advanced image feature extraction models are well-suited for handling the demands of multi-supervised learning.
- (4)
- By discussing the impact of different distillation loss strategies on translation performance, we find that removing enhanced knowledge distillation and enhanced feature refinement leads to severe performance degradation. Multimodal features without visual awareness show poor performance in machine translation, as they are derived solely by passing global text features through FC&Avg Unpool. This phenomenon suggests that our enhanced method highlights the significance of the teacher model’s intermediate hidden states in instructing the student model on comprehending text–image relationships.
4.6. Image Matching Analysis
4.7. Case Study
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Weighted Transformer Network for Machine Translation. Available online: https://arxiv.org/abs/1711.02132 (accessed on 6 November 2017).
- Laskar, S.R.; Paul, B.; Paudwal, S.; Gautam, P.; Biswas, N.; Pakray, P. Multimodal Neural Machine Translation for English—Assamese Pair. In Proceedings of the 2021 International Conference on Computational Performance Evluation (ComPE), Shillong, India, 1–3 December 2021. [Google Scholar]
- Chen, J.R.; He, T.L.; Zhuo, W.P.; Ma, L.; Ha, S.T.; Chan, S.H.G. Tvconv: Efficient translation variant convolution for layout-aware visual processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–20 June 2022. [Google Scholar]
- Chen, S.Y.; Zeng, Y.W.; Cao, D.; Lu, S.F. Video-guided machine translation via dual-level back-translation. Knowl. -Based Syst. 2022, 245, 108598. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 1–11. [Google Scholar]
- Keyframe Segmentation and Positional Encoding for Video-Guided Machine Translation Challenge 2020. Available online: https://arxiv.org/abs/2006.12799 (accessed on 23 June 2020).
- Incorporating Global Visual Features into Attention-Based Neural Machine Translation. Available online: https://arxiv.org/abs/1701.06521 (accessed on 23 January 2017).
- Distilling Translations with Visual Awareness. Available online: https://arxiv.org/abs/1906.07701 (accessed on 18 June 2019).
- A Novel Graph-Based Multi-Modal Fusion Encoder for Neural Machine Translation. Available online: https://arxiv.org/abs/2007.08742 (accessed on 17 July 2020).
- Multi30k: Multilingual English-German Image Descriptions. Available online: https://arxiv.org/abs/1605.00459 (accessed on 2 May 2016).
- Imagination Improves Multimodal Translation. Available online: https://arxiv.org/abs/1705.04350 (accessed on 7 July 2017).
- Zhang, Z.S.; Chen, K.H.; Wang, R.; Utiyama, M.; Sumita, E.; Li, Z.C.; Zhao, H. Neural machine translation with universal visual representation. In Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA, 9 September 2019. [Google Scholar]
- Generative Imagination Elevates Machine Translation. Available online: https://arxiv.org/abs/2009.09654 (accessed on 13 April 2021).
- Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning. Available online: https://arxiv.org/abs/2303.02861 (accessed on 6 March 2023).
- Lei, T.; Bai, J.; Brahma, S.; Ainslie, J.; Lee, K.; Zhou, Y.Q.; Du, N.; Zhao, V.; Wu, Y.X.; Li, B.; et al. Conditional adapters: Parameter-efficient transfer learning with fast inference. Adv. Neural Inf. Process. Syst. 2023, 36, 8152–8172. [Google Scholar]
- Xin, Y.; Du, J.L.; Wang, Q.; Lin, Z.W.; Yan, K. VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, Vancouver, BC, CA, 20–27 February 2024. [Google Scholar]
- Adaptive Feature Fusion: Enhancing Generalization in Deep Learning Models. Available online: https://arxiv.org/abs/2304.03290 (accessed on 4 April 2023).
- Adaptive Ensemble Learning: Boosting Model Performance through Intelligent Feature Fusion in Deep Neural Networks. Available online: https://arxiv.org/abs/2304.02653 (accessed on 4 April 2023).
- Wu, Y.S.; Chen, K.; Zhang, T.Y.; Hui, Y.C.; Berg-Kirkpatrick, T.; Dubnov, S. Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. In Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece , 4–9 June 2023. [Google Scholar]
- LIUM-CVC Submissions for WMT17 Multimodal Translation Task. Available online: https://arxiv.org/abs/1707.04481 (accessed on 14 July 2017).
- Doubly Attentive Transformer Machine Translation. Available online: https://arxiv.org/abs/1807.11605 (accessed on 30 July 2018).
- CUNI System for the WMT18 Multimodal Translation Task. Available online: https://arxiv.org/abs/1811.04697 (accessed on 12 November 2018).
- Doubly-Attentive Decoder for Multi-Modal Neural Machine Translation. Available online: https://arxiv.org/abs/1702.01287 (accessed on 4 February 2017).
- Shi, X.Y.; Yu, Z.Q. Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language. Math. Probl. Eng. 2022, 2022, 5483535. [Google Scholar] [CrossRef]
- Yao, S.W.; Wan, X.J. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, WA, USA, 5–10 July 2020. [Google Scholar]
- Buciluǎ, C.; Caruana, R.; Niculescu-Mizil, A. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Minging, Philadelphia, PA, USA, 20–23 August 2006. [Google Scholar]
- Distilling the Knowledge in a Neural Network. Available online: https://arxiv.org/abs/1503.02531 (accessed on 9 March 2015).
- Fitnets: Hints for Thin Deep Nets. Available online: https://arxiv.org/abs/1412.6550 (accessed on 27 March 2015).
- Yim, J.; Joo, D.; Bae, J.; Kim, J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Gupta, S.; Hoffman, J.; Malik, J. Cross modal distillation for supervision transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Yuan, M.K.; Peng, Y.X. Text-to-image synthesis via symmetrical distillation networks. In Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Republic of Korea, 22–26 October 2018. [Google Scholar]
- Capwap: Captioning with a Purpose. Available online: https://arxiv.org/abs/2011.04264 (accessed on 9 November 2015).
- Latent Variable Model for Multi-Modal Translation. Available online: https://arxiv.org/abs/1811.00357 (accessed on 16 May 2019).
- Ranaldi, L.; Pucci, G. Knowing Knowledge: Epistemological Study of Knowledge in Transformers. Appl. Sci. 2023, 13, 677. [Google Scholar] [CrossRef]
- Ranaldi, L.; Pucci, G. Does the English Matter? Elicit Cross-Lingual Abilities of Large Language Models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), Singapore, 7 December 2023. [Google Scholar]
- A Convolutional Encoder Model for Neural Machine Translation. Available online: https://arxiv.org/abs/1611.02344 (accessed on 25 July 2017).
- Ye, J.J.; Guo, J.J. Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation. Appl. Intell. 2022, 52, 14194–14203. [Google Scholar] [CrossRef]
- Very Deep Convolutional Networks for Large-Scale Image Recognition. Available online: https://arxiv.org/abs/1409.1556 (accessed on 10 April 2017).
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef]
- Zhang, Y.; Jin, R.; Zhou, Z.H. Understanding bag-of-words model: A statistical framework. Int. J. Mach. Learn. Cybern. 2010, 1, 43–52. [Google Scholar] [CrossRef]
- Reed, S.; Akata, Z.; Yan, X.C.; Logeswaran, L. Generative adversarial text to image synthesis. In Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016. [Google Scholar]
- Zhang, H.; Xu, T.; Li, H.S.; Zhang, S.T.; Wang, X.G.; Huang, X.L.; Metaxas, D.N. Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision(ICCV), Venice, Italy, 22–29 October 2017. [Google Scholar]
- Xu, T.; Zhang, P.C.; Huang, Q.Y.; Zhang, H.; Gan, Z.; Huang, X.L.; He, X.D. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, 12–18 July 2020. [Google Scholar]
- From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions. Available online: http://shannon.cs.illinois.edu/DenotationGraph/data/index.html (accessed on 1 February 2014).
- Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description. Available online: https://arxiv.org/abs/1710.07177 (accessed on 19 October 2017).
- Automatic Image Captioning Based on ResNet50 and LSTM with Soft Attention. Available online: https://onlinelibrary.wiley.com/doi/full/10.1155/2020/8909458 (accessed on 21 October 2020).
- He, K.M.; Zhang, X.Y.; Ren, S.Q.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Fairseq: A fast, Extensible Toolkit for Sequence Modeling. Available online: https://arxiv.org/abs/1904.01038 (accessed on 1 April 2019).
- A Visual Attention Grounding Neural Model for Multimodal Machine Translation. Available online: https://arxiv.org/abs/1808.08266 (accessed on 28 August 2018).
- Probing the Need for Visual Context in Multimodal Machine Translation. Available online: https://arxiv.org/abs/1903.08678 (accessed on 2 June 2019).
- Lin, H.; Meng, F.D.; Su, J.S.; Yin, Y.j.; Yang, Z.Y.; Ge, Y.B.; Zhou, J.; Luo, J.B. Dynamic context-guided capsule network for multimodal machine translation. In Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA, 12–16 October 2020. [Google Scholar]
- Gumbel-Attention for Multi-Modal Machine Translation. Available online: https://arxiv.org/abs/2103.08862 (accessed on 24 July 2022).
- Wang, D.X.; Xiong, D.Y. Efficient object-level visual context modeling for multimodal machine translation: Masking irrelevant objects helps grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 19–21 May 2021. [Google Scholar]
Visual Teacher Model | Multimodal Student Model | ||||
---|---|---|---|---|---|
Layer Name | Output Size | 49-Layer | Layer Name | Output Size | 48-Layer |
T-conv1 | 112 × 112 | 7 × 7, 64, stride2 | S-conv1 | 224 × 224 | 8 × 8, 3, stride2 |
T-conv2_x | 56 × 56 | 3 × 3, max pool, stride2 | S-conv2_x | 112 × 112 | 8 × 8, 3, stride2 |
T-conv3_x | 28 × 28 | S-conv3_x | 56 × 56 | ||
T-conv4_x | 14 × 14 | S-conv4_x | 28 × 28 | ||
T-conv5_x | 7 × 7 | S-conv5_x | 14 × 14 | ||
N/A | 1 × 1 | average pool | - | - | - |
Multimodal Feature Generator | N/A | 7 × 7 | 2048-d fc, average unpool |
MultiModel | MMT Systems | EN-DE | EN-FR | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Test2016 | Test2017 | MS COCO | Test2016 | Test2017 | |||||||
B | M | B | M | B | M | B | M | B | M | ||
Image-must | NMTSRC+IMG [23] | 36.5 | 55.0 | - | - | - | - | - | - | - | - |
IMGD [7] | 37.3 | 55.1 | - | - | - | - | - | - | - | - | |
Fusion-conv [20] | 37.0 | 57.0 | 29.8 | 51.2 | 25.1 | 46.0 | 53.5 | 70.4 | 51.6 | 68.6 | |
VMMT [33] | 37.7 | 56.0 | 30.1 | 49.9 | 25.5 | 44.8 | - | - | - | - | |
Trg-mul [20] | 37.8 | 57.7 | 30.7 | 52.2 | 26.4 | 47.4 | 54.7 | 71.3 | 52.7 | 69.5 | |
VAG-NMT [50] | - | - | 31.6 | 52.2 | 28.3 | 48.0 | - | - | 53.8 | 70.3 | |
DS-SUM-L2 [51] | 39.4 | 58.7 | 32.6 | 52.9 | - | - | 60.7 | 76.0 | 54.2 | 71.0 | |
Del+obj [8] | 38.0 | 55.6 | - | - | - | - | 59.8 | 74.4 | - | - | |
Multimodal [25] | 38.7 | 55.7 | - | - | - | - | - | - | - | - | |
GMNMT [9] | 39.8 | 57.6 | 32.2 | 51.9 | 28.7 | 47.6 | 60.9 | 74.9 | 53.9 | 69.3 | |
DCCN [52] | 39.7 | 56.8 | 31.0 | 49.9 | 26.7 | 45.7 | 61.2 | 76.4 | 54.3 | 70.3 | |
Gumbel-att [53] | 39.2 | 57.8 | 31.4 | 51.2 | 26.9 | 46.0 | - | - | - | - | |
OVC+Lm [54] | - | - | 32.3 | 52.4 | 28.9 | 48.1 | - | - | 54.1 | 70.5 | |
Image-free | Transformer [5] | 37.6 | 55.3 | 31.7 | 52.1 | 27.9 | 47.8 | 59.0 | 73.6 | 51.9 | 68.3 |
Multitask [11] | 36.8 | 55.8 | - | - | - | - | - | - | - | - | |
VMMTF [33] | 37.7 | 56.0 | 30.1 | 49.9 | 25.5 | 44.8 | - | - | - | - | |
UVR-NMT [12] | 36.94 | - | 28.63 | - | - | - | 57.53 | - | 48.46 | - | |
ImagiT [13] | 38.5 | 55.7 | 32.1 | 52.4 | 28.7 | 48.8 | 59.7 | 74.0 | 52.4 | 68.3 | |
DSKP-MMT (ours) | 40.42 | 58.15 | 32.87 | 52.44 | 28.83 | 47.73 | 61.49 | 76.7 | 54.33 | 71.37 |
Sim. Func. | Dist. Gran | CNN Back | Dist. Loss | Avg.B | Avg.M | |
---|---|---|---|---|---|---|
base | L2 | Model | ResNet50 | All_Loss | 40.42 | 58.15 |
(1) | L1 | - | - | - | 39.98 | 58.07 |
L∞ | - | - | - | 39.83 | 57.66 | |
Consine | - | - | - | 39.57 | 57.37 | |
Kl-Div | - | - | - | 40.06 | 57.82 | |
(2) | - | Block | - | - | 39.82 | 57.66 |
(3) | - | - | VGG19 | - | 40.06 | 57.82 |
- | - | AlexNet | - | 39.88 | 57.52 | |
(4) | - | - | - | Text_Loss | 39.72 | 57.50 |
- | - | - | KD_Loss | 40.08 | 57.33 | |
- | - | - | F_Loss | 40.28 | 57.74 |
Source Text (EN) | Source Image | Match Image |
---|---|---|
a man on a four-wheeler is flying through the air. | ||
two dogs are wrestling in a grassy field. | ||
a white dog is about to catch a yellow dog toy. |
Image | Text Descriptive (EN-DE, FR) |
---|---|
EN: a crane operates amidst piles of rubble. | |
DE: ein kran arbeitet mitten in einem schutthaufen. | |
FR: une grue travaille au milieu des amas de décombres. | |
DEMMT: ein bagger arbeitet mitten in den Trümmern. | |
DEOurs: ein kran arbeitet mitten in einem schutthaufen. | |
FRMMT: une excavatrice creuser au milieu des amas de décombres. | |
FROurs: une grue travaille au milieu des amas de décombres. | |
EN: two cars are driving on a racetrack. | |
DE: zwei autos fahren auf einer rennstrecke. | |
FR: deux voitures roulent sur un circuit. | |
DEMMT: zwei rennwagen fahren auf der strecke. | |
DEOurs: zwei autos fahren auf einer rennstrecke. | |
FRMMT: deux voitures course roulent sur la piste. | |
FROurs: deux voitures roulent sur un circuit. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tian, E.; Zhu, Z.; Liu, F.; Li, Z.; Gu, R.; Zhao, S. Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion. Electronics 2024, 13, 3084. https://doi.org/10.3390/electronics13153084
Tian E, Zhu Z, Liu F, Li Z, Gu R, Zhao S. Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion. Electronics. 2024; 13(15):3084. https://doi.org/10.3390/electronics13153084
Chicago/Turabian StyleTian, Erlin, Zengchao Zhu, Fangmei Liu, Zuhe Li, Ran Gu, and Shuai Zhao. 2024. "Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion" Electronics 13, no. 15: 3084. https://doi.org/10.3390/electronics13153084