Adaptive Sensing Data Augmentation for Drones Using Attention-Based GAN
Abstract
:1. Introduction
- Integration of Advanced Deep Learning Techniques and a Novel Spatiotemporal Attention Mechanism: We leverage cutting-edge methods to enhance the generation of synthetic data, providing a robust solution for augmenting time-series data. Our novel spatiotemporal attention mechanism excels at capturing both spatial and temporal dependencies across sequences, significantly improving the performance of deep learning models on drone-collected sensor data. By addressing the intricate relationships between spatial and temporal elements, this mechanism ensures a more comprehensive understanding of the data, leading to better predictive accuracy and model reliability [29]. Furthermore, this mechanism is adaptable to various data dimensions and types, making it versatile for different applications. It also facilitates the effective integration of partial convolution layers within our proposed GAN architecture, which helps in handling missing data and irregular time-series patterns, thereby enhancing the overall robustness and functionality of the system.
- Efficiency and Accuracy in Data Augmentation: Our system enables efficient and accurate augmentation of sensor data, allowing for the collection of sensor data at longer intervals without compromising the quality or quantity of the data. By reducing the sensing frequency, we significantly alleviate network and power burdens, which are critical constraints in drone operations. This approach ensures that the drones can operate for extended periods and cover larger areas without frequent returns for data offloading or battery recharges. Consequently, drones can sustain high performance in various applications such as precision agriculture, environmental monitoring, and surveillance by lowering the sensing frequency and maintaining data volume. The augmented data maintain high fidelity to real-world conditions, ensuring that applications relying on comprehensive sensor data can continue to perform optimally [30].
2. Preliminary
2.1. Generative Adversarial Networks
- Conditional GANs (cGANs): These incorporate auxiliary information, such as class labels, to generate class-specific data, making the models more controllable and versatile [31].
- Wasserstein GANs (WGANs): These use the Wasserstein distance as a loss function to stabilize training and mitigate issues such as mode collapse [32].
- CycleGANs: These are designed for image-to-image translation tasks without requiring paired training examples, enabling applications like style transfer and domain adaptation [33].
- StyleGANs: These have advanced the field of high-resolution image synthesis, particularly in generating highly realistic human faces [34].
2.2. Attention Mechanism
- Global Attention: Considers the entire input sequence when computing attention scores, providing a comprehensive context [41].
- Local Attention: Focuses on a smaller, fixed-size window around a particular point in the input sequence, which is useful for tasks where locality is important [41].
- Self-Attention: Also known as intra-attention, this mechanism relates different positions of the same sequence to compute a representation of the sequence. It is a crucial component of Transformer models, which have set new benchmarks in NLP tasks [29].
2.3. Quantile Transformer Preprocessing
3. System Design
3.1. System Overview
3.2. Exploratory Data Analysis for Drone Sensor
3.3. Spatiotemporal Attention Mechanism
Algorithm 1 Algorithm of spatiotemporal attention mechanism |
Require: Input matrix Ensure: Output
|
3.4. Deep Learning Architecture
4. Experiments
4.1. Data Description
4.2. Evaluation
4.2.1. Experiment of Sensor Data Generation
- 1.
- Visual Comparison: To visually compare the real and synthetic data, we plotted the distribution of the actual voltage data alongside the synthetic data. This visual representation helps to identify whether the synthetic data capture the temporal dynamics and patterns present in the original dataset.
- 2.
- Quantitative Metrics: We employed several quantitative metrics to assess the similarity between the real and synthetic data. These metrics include:
- Mean Squared Error (MSE): This metric measures the average squared difference between the real and synthetic voltage values. A lower MSE indicates that the synthetic data is close to the real data.
- Mean Absolute Error (MAE): This metric calculates the average absolute difference between the real and synthetic values, providing another measure of accuracy.
- Pearson Correlation Coefficient (PCC): PCC measures the linear correlation between the real and synthetic data, with values close to 1 indicating a strong positive correlation.
- Cosine Similarity: This metric evaluates the cosine of the angle between two non-zero vectors, providing a measure of similarity that ranges from −1 to 1, where 1 indicates identical orientation.
4.2.2. Comparison with Other Models
4.3. Use Case
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Kamilaris, A.; Prenafeta-Boldú, F.X. The use of drones in agriculture and forestry. Comput. Electron. Agric. 2017, 138, 232–247. [Google Scholar]
- Gupta, L.; Biswas, P.S. A review of unmanned aircraft system (UAS) applications for agriculture. In Proceedings of the IEEE International Conference on Automation Science and Engineering (CASE), Madison, WI, USA, 17–20 August 2013; pp. 296–301. [Google Scholar]
- Kontogianni, F.; Thomaidis, C.; Tsipras, G. Applications of UAVs in agriculture. In Proceedings of the International Conference on Advanced Intelligent Mechatronics (AIM), Auckland, New Zealand, 9–12 July 2018; pp. 718–723. [Google Scholar]
- Lee, S.; Yoo, S.; Lee, J.Y.; Park, S.; Kim, H. Drone positioning system using UWB sensing and out-of-band control. IEEE Sensors J. 2021, 22, 5329–5343. [Google Scholar] [CrossRef]
- Tsouros, D.C.; Bibi, S.; Sarigiannidis, P.G. Use of drones in agriculture: The way forward. Int. J. Sustain. Agric. Manag. Informatics 2019, 5, 54–74. [Google Scholar]
- Moschetti, D.; Mantovani, G. Monitoring plant health with UAVs in agriculture. Eur. J. Remote Sens. 2018, 51, 881–890. [Google Scholar]
- Cha, J.; Park, S.; Kwon, H. Drone delivery systems: State of the art and future directions. Transp. Res. Part C Emerg. Technol. 2018, 92, 784–806. [Google Scholar]
- Mulero-Pázquez, L.; Caballero, F.; Lucena, C. A review of unmanned aerial vehicle (UAV) technology for image-based analysis and quantification of agricultural fields. Agric. Syst. 2017, 153, 130–145. [Google Scholar]
- Zhang, H.; Zhang, H.; Wu, X. A comprehensive review of the applications of drone platforms for precision agriculture. IEEE Access 2019, 7, 153287–153307. [Google Scholar]
- Ren, H.; Liu, Y.; Zhao, F.; Fan, W. Towards multi-sensor fusion based safe landing spot detection using a UAV. Sensors 2020, 20, 658. [Google Scholar]
- Modares, H.; Lewis, F.L.; Pantic, M. Multi-UAV collaborative sensing for field area coverage: A path planning approach. Robot. Auton. Syst. 2017, 87, 54–65. [Google Scholar]
- Liu, H.; Gong, W. UAV-based agricultural remote sensing: Current status and perspectives. Agronomy 2021, 11, 243. [Google Scholar]
- Liu, H.; Ma, C.; Wang, B. High-precision UAV remote sensing applications for precision agriculture: Challenges and opportunities. Remote Sens. 2019, 11, 2532. [Google Scholar]
- Liu, K.; Zhang, W.; Zhao, Y. Real-time data processing for UAV-based crop monitoring: A comprehensive review. Comput. Electron. Agric. 2019, 166, 104988. [Google Scholar]
- Yick, J.; Mukherjee, B.; Ghosal, D. Wireless sensor network survey. Comput. Networks 2008, 52, 2292–2330. [Google Scholar] [CrossRef]
- Pottie, G.J.; Kaiser, W.J. Wireless integrated network sensors. Commun. ACM 2000, 43, 51–58. [Google Scholar] [CrossRef]
- Raghunathan, V.; Schurgers, C.; Park, S.; Srivastava, M.B. Energy-aware wireless microsensor networks. IEEE Signal Process. Mag. 2002, 19, 40–50. [Google Scholar] [CrossRef]
- Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data 2019, 6, 60. [Google Scholar] [CrossRef]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Cambridge, MA, USA, 8–13 December 2014; pp. 2672–2680. [Google Scholar]
- Kingma, D.P.; Welling, M. Auto-encoding variational bayes. arXiv 2014, arXiv:1312.6114. [Google Scholar]
- Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by back-propagating errors. Nature 1986, 323, 533–536. [Google Scholar] [CrossRef]
- Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Cambridge, MA, USA, 8–13 December 2014; pp. 3104–3112. [Google Scholar]
- Doersch, C. Tutorial on variational autoencoders. arXiv 2016, arXiv:1606.05908. [Google Scholar]
- Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv 2014, arXiv:1406.1078. [Google Scholar]
- Yoon, N.; Kim, H. Pioneering AI in Chemical Data: New Frontline with GC-MS Generation. In Proceedings of the 2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Osaka, Japan, 19–22 February 2024; pp. 826–831. [Google Scholar]
- Yoon, N.; Jung, W.; Kim, H.N. DeepRSSI: Generative Model for Fingerprint-Based Localization. IEEE Access 2024, 12, 66196–66213. [Google Scholar] [CrossRef]
- Creswell, A.; White, T.; Dumoulin, V.; Arulkumaran, K.; Sengupta, B.; Bharath, A.A. Generative adversarial networks: An overview. IEEE Signal Process. Mag. 2018, 35, 53–65. [Google Scholar] [CrossRef]
- Esteban, C.; Hyland, S.L.; Rätsch, G. Real-valued (medical) time series generation with recurrent conditional gans. arXiv 2017, arXiv:1706.02633. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 4–9 December 2017; pp. 5998–6008. [Google Scholar]
- Shen, Y.; Tang, J.; Xiong, Y.; Jin, R. Learning to augment time series data: A hierarchical VAE approach. arXiv 2020, arXiv:2005.12489. [Google Scholar]
- Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv 2014, arXiv:1411.1784. [Google Scholar]
- Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein GAN. arXiv 2017, arXiv:1701.07875. [Google Scholar]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
- Karras, T.; Laine, S.; Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 4401–4410. [Google Scholar]
- Antoniou, A.; Storkey, A.; Edwards, H. Data augmentation generative adversarial networks. arXiv 2017, arXiv:1711.04340. [Google Scholar]
- Yu, L.; Zhang, W.; Wang, J.; Yu, Y. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017; Volume 31. [Google Scholar]
- Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv 2014, arXiv:1409.0473. [Google Scholar]
- Hochreiter, S.; Bengio, Y.; Frasconi, P.; Schmidhuber, J. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl. Based Syst. 1998, 6, 107–116. [Google Scholar] [CrossRef]
- Yoon, N.; Lee, D.; Kim, K.; Yoo, T.; Joo, H.; Kim, H. STEAM: Spatial Trajectory Enhanced Attention Mechanism for Abnormal UAV Trajectory Detection. Appl. Sci. 2023, 14, 248. [Google Scholar] [CrossRef]
- Xu, K.; Ba, J.L.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.S.; Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 2048–2057. [Google Scholar]
- Luong, M.T.; Pham, H.; Manning, C.D. Effective approaches to attention-based neural machine translation. arXiv 2015, arXiv:1508.04025. [Google Scholar]
- Yoon, N.; Kim, H. Detecting DDoS based on attention mechanism for Software-Defined Networks. J. Netw. Comput. Appl. 2024, 230, 103928. [Google Scholar] [CrossRef]
- Chan, S.; Santoro, A.; Lampinen, A.; Wang, J.; Singh, A.; Richemond, P.; McClelland, J.; Hill, F. Data distributional properties drive emergent in-context learning in transformers. In Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 28 November–9 December 2022; Volume 35, pp. 18878–18891. [Google Scholar]
- Li, D.; Chen, D.; Jin, B.; Shi, L.; Goh, J.; Ng, S.K. MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks. In Proceedings of the International Conference on Artificial Neural Networks, Munich, Germany, 17–19 September 2019; pp. 703–716. [Google Scholar]
- Li, Y.; Peng, X.; Zhang, J.; Li, Z.; Wen, M. DCT-GAN: Dilated convolutional transformer-based GAN for time series anomaly detection. IEEE Trans. Knowl. Data Eng. 2021, 35, 3632–3644. [Google Scholar] [CrossRef]
- Zhu, F.; Ye, F.; Fu, Y.; Liu, Q.; Shen, B. Electrocardiogram generation with a bidirectional LSTM-CNN generative adversarial network. Sci. Rep. 2019, 9, 6734. [Google Scholar] [CrossRef] [PubMed]
- Sabir, R.; Rosato, D.; Hartmann, S.; Gühmann, C. Signal generation using 1d deep convolutional generative adversarial networks for fault diagnosis of electrical machines. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 3907–3914. [Google Scholar]
- James, S.; Harbron, C.; Branson, J.; Sundler, M. Synthetic data use: Exploring use cases to optimise data utility. Discov. Artif. Intell. 2021, 1, 15. [Google Scholar] [CrossRef]
- Tatachar, A.V. Comparative assessment of regression models based on model evaluation metrics. Int. J. Innov. Technol. Explor. Eng. 2021, 8, 853–860. [Google Scholar]
Campaign | Location | Date | Duration (min) | Samples |
---|---|---|---|---|
ESP2 | River Ter, Torrello, Barcelona, Spain | 31 March 2017 | 47 | 2814 |
ESP4 | Pantà de Sau Reservoir, Barcelona, Spain | 30 March 2017 | 39 | 2374 |
ESP5 | Bayou Ter River, Barcelona, Spain | 31 March 2017 | 60 | 3601 |
GARDA3 | Lake Gardena, Verona, Italy | 9 May 2017 | 40 | 2451 |
ITA1 | Atlantide Fishing Site, Verona, Italy | 20 April 2017 | 121 | 7243 |
ITA6 | Atlantide Fishing Site, Verona, Italy | 7 March 2017 | 28 | 1704 |
Model | Metric | Latitude | Longitude | Altitude | EC | Temp | DO | Voltage | Heading |
---|---|---|---|---|---|---|---|---|---|
Spatiotemporal -Attention GAN | MSE | 0.0061 | 0.0071 | 0.0054 | 0.0040 | 0.0009 | 0.0042 | 0.0120 | 0.0172 |
MAE | 0.0553 | 0.0633 | 0.0564 | 0.0371 | 0.0231 | 0.0524 | 0.0801 | 0.0881 | |
PCC | 0.9722 | 0.9614 | 0.9578 | 0.9894 | 0.9928 | 0.9406 | 0.9669 | 0.8831 | |
Cosine | 0.9944 | 0.9942 | 0.9915 | 0.9969 | 0.9977 | 0.9922 | 0.9878 | 0.9715 |
Model | Metric | Latitude | Longitude | Altitude | EC | Temp | DO | Voltage | Heading |
---|---|---|---|---|---|---|---|---|---|
DCGAN | MSE | 0.0000 | 0.0000 | 12.7468 | 93,772.9686 | 1.8734 | 0.2295 | 0.7258 | 25,369.4697 |
MAE | 0.0004 | 0.0015 | 2.8572 | 239.6136 | 1.1365 | 0.3903 | 0.7071 | 127.7668 | |
PCC | 0.0086 | 0.0010 | 0.0370 | −0.0010 | −0.0224 | −0.0067 | −0.0179 | −0.0161 | |
Cosine | 1.0000 | 1.0000 | 1.0000 | 0.5387 | 0.9931 | 0.9978 | 0.9986 | 0.7159 | |
LSTM-CNN GAN | MSE | 0.0000 | 0.0000 | 8.3709 | 69,270.8039 | 5.6076 | 0.0917 | 0.9790 | 22,981.1240 |
MAE | 0.0004 | 0.0013 | 2.3484 | 221.3266 | 1.7811 | 0.2630 | 0.7669 | 122.8213 | |
PCC | 0.0115 | 0.0120 | 0.0060 | −0.0167 | 0.0112 | 0.0148 | 0.0064 | −0.0171 | |
Cosine | 1.0000 | 1.0000 | 1.0000 | 0.5779 | 0.9853 | 0.9997 | 0.9981 | 0.7801 | |
MAD-GAN | MSE | 0.0000 | 0.0000 | 19.3150 | 127,297.3289 | 2.0403 | 0.2860 | 0.8311 | 38,136.2132 |
MAE | 0.0004 | 0.0016 | 3.7649 | 278.3726 | 1.1939 | 0.4398 | 0.7502 | 158.1357 | |
PCC | 0.0185 | −0.0380 | −0.0195 | −0.0256 | 0.0176 | −0.0400 | 0.0166 | 0.0067 | |
Cosine | 1.0000 | 1.0000 | 1.0000 | 0.3699 | 0.9929 | 0.9973 | 0.9985 | 0.6877 | |
DCT-GAN | MSE | 0.0000 | 0.0000 | 6.2528 | 77,151.9267 | 1.6957 | 0.0799 | 0.7473 | 20,965.9737 |
MAE | 0.0004 | 0.0015 | 1.9984 | 217.4320 | 1.0775 | 0.2253 | 0.7229 | 118.0895 | |
PCC | −0.0055 | 0.0085 | −0.0162 | −0.0164 | 0.0183 | 0.0047 | 0.0044 | −0.0142 | |
Cosine | 1.0000 | 1.0000 | 1.0000 | 0.6205 | 0.9942 | 0.9994 | 0.9988 | 0.7941 | |
Spatiotemporal -Attention GAN (Proposed) | MSE | 0.0061 | 0.0071 | 0.0054 | 0.0040 | 0.0009 | 0.0042 | 0.0120 | 0.0172 |
MAE | 0.0553 | 0.0633 | 0.0564 | 0.0371 | 0.0231 | 0.0524 | 0.0801 | 0.0881 | |
PCC | 0.9722 | 0.9614 | 0.9578 | 0.9894 | 0.9928 | 0.9406 | 0.9669 | 0.8831 | |
Cosine | 0.9944 | 0.9942 | 0.9915 | 0.9969 | 0.9977 | 0.9922 | 0.9878 | 0.9715 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yoon, N.; Kim, K.; Lee, S.; Bai, J.H.; Kim, H. Adaptive Sensing Data Augmentation for Drones Using Attention-Based GAN. Sensors 2024, 24, 5451. https://doi.org/10.3390/s24165451
Yoon N, Kim K, Lee S, Bai JH, Kim H. Adaptive Sensing Data Augmentation for Drones Using Attention-Based GAN. Sensors. 2024; 24(16):5451. https://doi.org/10.3390/s24165451
Chicago/Turabian StyleYoon, Namkyung, Kiseok Kim, Sangmin Lee, Jin Hyoung Bai, and Hwangnam Kim. 2024. "Adaptive Sensing Data Augmentation for Drones Using Attention-Based GAN" Sensors 24, no. 16: 5451. https://doi.org/10.3390/s24165451