A Multi-Task Network: Improving Unmanned Underwater Vehicle Self-Noise Separation via Sound Event Recognition
Abstract
:1. Introduction
2. Related Work
3. Underwater Acoustic Multi-Task Noise Separation and Recognition Method
3.1. Mask-Based Single-Channel Noise Source Separation
3.2. The Mask-Estimator with Sound Event Improved Module
3.2.1. The Compressor and Up-Sampling
3.2.2. Sound Event Improved Module
- (a)
- The DV Module
- (b)
- The SV Module
3.3. Balancing Multi-Task Weights
3.4. Network Architecture
4. Experimental Procedures
4.1. Data Collection and Dataset Setup
4.2. Data Augmentation
4.3. Experiment Configurations
4.4. Evaluation Metrics
5. Result
5.1. Impact of Loss Function Weights on Performance
5.2. Separation Result
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
UUV | Unmanned Underwater Vehicle |
SINR | Signal-to-Interference-plus-Noise Ratio |
DNNs | Deep Neural Networks |
MLN | Multi-Task Learning Network |
References
- Cancilliere, F.M. Advanced UUV Technology. In Proceedings of the IEEE OCEANS’94, Brest, France, 13–16 September 1994; Volume 1, pp. I/147–I/151. [Google Scholar]
- Wibisono, A.; Piran, M.J.; Song, H.K.; Lee, B.M. A Survey on Unmanned Underwater Vehicles: Challenges, Enabling Technologies, and Future Research Directions. Sensors 2023, 17, 7321. [Google Scholar] [CrossRef]
- Liu, F.; Ma, Z.; Mu, B.; Duan, C.; Chen, R.; Qin, Y.; Pu, H.; Luo, J. Review on Fault-tolerant Control of Unmanned Underwater Vehicles. Ocean Eng. 2023, 285, 115471. [Google Scholar] [CrossRef]
- Li, J.; Zhang, G.; Jiang, C.; Zhang, W. Review A survey of maritime unmanned search system: Theory, Applications and Future Directions. Ocean Eng. 2023, 285, 115359. [Google Scholar] [CrossRef]
- Holmes, J.D.; Carey, W.M.; Lynch, J.F. An Overview of Unmanned Underwater Vehicle Noise in the Low to Mid Frequency Bands. J. Acoust. Soc. Am. 2010, 127, 1812. [Google Scholar] [CrossRef]
- Kumar, P.; Ali, M.; Nathwani, K. Self-Noise Cancellation in Underwater Acoustics using Deep Neural Network Frameworks. In Proceedings of the OCEANS 2023, Limerick, Ireland, 5–8 June 2023. [Google Scholar]
- Raanan, B.-Y.; Bellingham, J.; Zhang, Y.; Kemp, M.; Kieft, B.; Singh, H.; Girdhar, Y. Detection of Unanticipated Faults for Autonomous Underwater Vehicles Using Online Topic Models. J. Field Robot. 2018, 5, 705–716. [Google Scholar] [CrossRef]
- Liu, F.; Tang, H.; Qin, Y.; Duan, C.; Luo, J.; Pu, H. Review on Fault Diagnosis of Unmanned Underwater Vehicles. Ocean Eng. 2022, 243, 110290. [Google Scholar] [CrossRef]
- Zhou, A.; Zhang, W.; Li, X.; Xu, G.; Zhang, B.; Ma, Y.; Song, J. A Novel Noise-Aware Deep Learning Model for Underwater Acoustic Denoising. IEEE Trans. Geosci. Remote Sens. 2023, 61, 4202813. [Google Scholar] [CrossRef]
- Liu, J.; Li, Q.; Shang, D. The Investigation on Measuring Source Level of Unmanned Underwater Vehicles. In Proceedings of the 2016 IEEE/OES China Ocean Acoustics (COA), Harbin, China, 9–11 January 2016. [Google Scholar]
- Zimmerman, R.; D’Spain, G.L.; Chadwell, C.D. Decreasing the Radiated Acoustic and Vibration Noise of a Mid-Size AUV. IEEE J. Ocean. Eng. 2005, 1, 179–187. [Google Scholar] [CrossRef]
- Soni, S.; Yadav, R.N.; Gupta, L. State-of-the-Art Analysis of Deep Learning-Based Monaural Speech Source Separation Techniques. IEEE Access 2023, 11, 4242–4269. [Google Scholar] [CrossRef]
- Gu, J.J.; Yao, D.D.; Li, J.F.; Yan, Y.H. A Novel Semi-Blind Source Separation Framework towards Maximum Signal-to-Interference Ratio. Signal Process. 2024, 217, 109359. [Google Scholar] [CrossRef]
- Drude, L.; Haeb-Umbach, R. Integration of Neural Networks and Probabilistic Spatial Models for Acoustic Blind Source Separation. IEEE J. Sel. Top. Signal Process. 2019, 4, 815–826. [Google Scholar] [CrossRef]
- Chen, J.; Liu, C.; Xie, J.W.; An, J.; Huang, N. Time-Frequency Mask-Aware Bidirectional LSTM: A Deep Learning Approach for Underwater Acoustic Signal Separation. Sensors 2022, 15, 5598. [Google Scholar] [CrossRef] [PubMed]
- Ansari, S.; Alatrany, A.S.; Alnajjar, K.A.; Khater, T.; Mahmoud, S.; Al-Jumeily, D.; Hussain, A.J. A Survey of Artificial Intelligence Approaches in Blind Source Separation. Neurocomputing 2023, 561, 126895. [Google Scholar] [CrossRef]
- Yu, Y.S.; Qiu, X.Y.; Hu, F.C.; He, R.H.; Zhang, L.K. An End-to-End Speech Separation Method Based on Features of Two Domains. J. Vib. Eng. Technol. 2024, 12, 7325–7334. [Google Scholar] [CrossRef]
- Song, R.P.; Feng, X.; Wang, J.F.; Sun, H.X.; Zhou, M.Z.; Esmaiel, H. Underwater Acoustic Nonlinear Blind Ship Noise Separation Using Recurrent Attention Neural Networks. Remote Sens. 2024, 4, 653. [Google Scholar] [CrossRef]
- Luo, Y.; Mesgarani, N. Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation. IEEE/ACM Trans. Audio Speech Lang. Process. 2019, 8, 1256–1266. [Google Scholar] [CrossRef] [PubMed]
- Chen, J.; Mao, Q.; Liu, D. Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation. In Proceedings of the INTERSPEECH, Shanghai, China, 25–29 October 2020. [Google Scholar]
- Hu, X.; Li, K.; Zhang, W.; Luo, Y.; Lemercier, J.-M.; Gerkmann, T. Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Virtual, 6–14 December 2021. [Google Scholar]
- Défossez, A.; Usunier, N.; Bottou, L.; Bach, F. Music Source Separation in the Waveform Domain. arXiv 2021, arXiv:1911.13254. [Google Scholar]
- Hung, Y.-N.; Lerch, A. Multitask Learning for Instrument Activation Aware Music Source Separation. arXiv 2020, arXiv:2008.00616. [Google Scholar]
- Lee, K.J.; Lee, B. End-to-End Deep Learning Architecture for Separating Maternal and Fetal ECGs Using W-Net. IEEE Access 2022, 10, 39782–39788. [Google Scholar] [CrossRef]
- Tzinis, E.; Wisdom, S.; Hershey, J.R.; Jansen, A.; Ellis, D.P.W. Improving Universal Sound Separation Using Sound Classification. In Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual, 4–8 May 2020. [Google Scholar]
- Kavalerov, I.; Wisdom, S.; Erdogan, H.; Patton, B.; Wilson, K.; Roux, J.L.; Hershey, J.R. Universal Sound Separation. In Proceedings of the 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, 20–23 October 2019. [Google Scholar]
- Wisdom, S.; Tzinis, E.; Erdogan, H.; Weiss, R.J.; Wilson, K.; Hershey, J.R. Unsupervised Sound Separation Using Mixture Invariant Training. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2020), Virtual, 6–12 December 2020. [Google Scholar]
- Singh, A.; Ogunfunmi, T. An Overview of Variational Autoencoders for Source Separation, Finance, and Bio-Signal Applications. Entropy 2022, 1, 55. [Google Scholar] [CrossRef]
- Wang, D.; Chen, J. Supervised Speech Separation Based on Deep Learning: An Overview. IEEE/ACM Trans. Audio Speech Lang. Process. 2018, 10, 1702–1726. [Google Scholar] [CrossRef] [PubMed]
- Kingma, D.P.; Welling, M. Auto-Encoding Variational Bayes. arXiv 2020, arXiv:1312.6114. [Google Scholar] [CrossRef]
- Li, L.; Kameoka, H.; Makino, S. FastMvAE2: On Improving and Accelerating the Fast Variational Autoencoder-Based Source Separation Algorithm for Determined Mixtures. IEEE/ACM Trans. Audio Speech Lang. Process. 2023, 31, 96–110. [Google Scholar] [CrossRef]
- Casti, P.; Cardarelli, S.; Comes, M.C.; D’Orazio, M.; Filippi, J.; Antonelli, G.; Mencattini, A.; Di Natale, C.; Martinelli, E. S3-VAE: A Novel Supervised-Source-Separation Variational AutoEncoder Algorithm to Discriminate Tumor Cell Lines in Time-Lapse Microscopy Images. Expert Syst. Appl. 2023, 232, 120861.1–120861.15. [Google Scholar] [CrossRef]
- Pal, M.; Roy, R.; Basu, J.; Bepari, M.S. Blind Source Separation: A Review and Analysis. In Proceedings of the 16th International Oriental COCOSDA Conference, Gurgaon, India, 25–28 November 2013. [Google Scholar]
- Heurtebise, A.; Ablin, P.; Gramfort, A. Multiview Independent Component Analysis with Delays. In Proceedings of the 33rd IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2023), Rome, Italy, 17–20 September 2023. [Google Scholar]
- Yoshii, K.; Tomioka, R.; Mochihashi, D.; Goto, M. Beyond NMF: Time-Domain Audio Source Separation without Phase Reconstruction. In Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR 2013), Curitiba, Brazil, 4–8 November 2013; pp. 369–374. [Google Scholar]
- Parekh, J.; Parekh, S.; Mozharovskyi, P.; Richard, G.; d’Alché-Buc, F. Tackling Interpretability in Audio Classification Networks with Non-negative Matrix Factorization. IEEE-ACM Trans. Audio Speech Lang. Process. 2024, 32, 1392–1405. [Google Scholar] [CrossRef]
- Do, H.D.; Tran, S.T.; Chau, D.T. Speech Source Separation Using Variational Autoencoder and Bandpass Filter. IEEE Access 2020, 8, 156219–156231. [Google Scholar] [CrossRef]
- Huang, P.S.; Kim, M.; Hasegawa-Johnson, M.; Smaragdis, P. Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation. IEEE/ACM Trans. Audio Speech Lang. Process. 2015, 12, 2136–2147. [Google Scholar] [CrossRef]
- Hershey, J.R.; Chen, Z.; Le Roux, J.; Watanabe, S. Deep Clustering: Discriminative Embeddings for Segmentation and Separation. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016; pp. 31–35. [Google Scholar]
- Yu, D.; Kolbæk, M.; Tan, Z.H.; Jensen, J. Permutation Invariant Training of Deep Models for Speaker-Independent Multi-Talker Speech Separation. In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017; pp. 241–245. [Google Scholar]
- Kolbæk, M.; Yu, D.; Tan, Z.H.; Jensen, J. Multitalker Speech Separation with Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks. IEEE/ACM Trans. Audio Speech Lang. Process. 2017, 10, 1901–1913. [Google Scholar] [CrossRef]
- Pishdadian, F.; Wichern, G.; Le Roux, J. Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision. IEEE/ACM Trans. Audio Speech Lang. Process. 2020, 28, 2386–2399. [Google Scholar] [CrossRef]
- Seetharaman, P.; Wichern, G.; Venkataramani, S.; Roux, J.L. Class-Conditional Embeddings for Music Source Separation. In Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 301–305. [Google Scholar]
- Karamatlı, E.; Cemgil, A.T.; Kırbız, S. Weak Label Supervision for Monaural Source Separation Using Non-Negative Denoising Variational Autoencoders. In Proceedings of the 2019 27th Signal Processing and Communications Applications Conference (SIU), Sivas, Turkey, 24–26 April 2019. [Google Scholar]
- Grais, E.M.; Plumbley, M.D. Single Channel Audio Source Separation Using Convolutional Denoising Autoencoders. In Proceedings of the 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Montreal, QC, Canada, 14–16 November 2017; pp. 1265–1269. [Google Scholar]
- Bofill, P.; Zibulevsky, M. Underdetermined Blind Source Separation Using Sparse Representations. Signal Process. 2001, 11, 2353–2362. [Google Scholar] [CrossRef]
- Li, L.; Kameoka, H.; Makino, S. Fast MVAE: Joint Separation and Classification of Mixed Sources Based on Multichannel Variational Autoencoder with Auxiliary Classifier. In Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 546–550. [Google Scholar]
- Li, C.; Luo, Y.; Han, C.; Li, J.; Yoshioka, T.; Zhou, T.; Delcroix, M.; Kinoshita, K.; Boeddeker, C.; Qian, Y.; et al. Dual-path RNN for Long Recording Speech Separation. In Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China, 19–22 January 2021; pp. 865–872. [Google Scholar]
Task Type | Equation | Loss Function |
---|---|---|
Recognition | (10) | |
Separation | (14) | |
Total | (15) |
Number | Layer Description | Output Size |
---|---|---|
Encoder | ||
1 | (256, 3600) | |
Compressor | ||
1 | ||
2 | (256, 900) | |
3 | (256, 450) | |
DV | ||
1 | +GLN+ReLU | (1, 450) |
2 | +sigmoid | (1, 128) |
3 | +softmax | (1, 6) |
SV | ||
1 | +GLN+ReLU | (1, 450) |
2 | +sigmoid | (1, 128) |
3 | +softmax | (1, 6) |
4 | +sigmoid | (1, 128) |
5 | +sigmoid | (1, 450) |
6 | +GLN+ReLU | (256, 450) |
Up-sample | ||
1 | ||
2 | (256, 1800) | |
3 | (256, 3600) | |
Decoder | ||
1 | +ReLU | |
2 | +sigmoid | (2, 28,800) |
Number | Model | Sensitivity | Usable Frequency Range | Sampling Frequency | Location | |
---|---|---|---|---|---|---|
Accelerometer | 2 | Three-axis sensor | 100 mV/g | 0.4–15 kHz | 48 kHz | Head and center of the UUV |
Underwater acoustic transducer array | 2 | Eight-element sensors in each array | −200 dB | 5–20 kHz | 48 kHz | UUV flank array |
Source | Period | Number |
---|---|---|
Activation | 2508 | |
Accelerometer | Uplift | 891 |
Float | 2224 | |
Activation | 1952 | |
Hydrophone | Uplift | 777 |
Float | 1822 |
SISNR (dB) | Cross-Entropy | |
---|---|---|
0.1 | 12.10 | 0.40 |
0.3 | 12.65 | 0.39 |
1 | 12.84 | 0.36 |
3 | 13.24 | 0.29 |
10 | 12.95 | 0.28 |
Index | Separation | Recognition | Parameter (M) | |
---|---|---|---|---|
SISNR (dB) | ||||
MLN-N | 11.25 | 31.1% | 0.22 | 2.73 |
MLN-DV | 11.36 | 28.2% | 0.07 | 2.91 |
MLN-SV | 12.86 | 19.0% | 0.10 | 3.05 |
Index | Separation | Recognition | Parameter (M) | |
---|---|---|---|---|
SISNR (dB) | ||||
MLN-SV | 12.86 | 19.0% | 0.10 | 3.05 |
Conv-Tasnet | 11.87 | 20.5% | 0.15 | 2.36 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Shi, W.; Chen, D.; Tian, F.; Liu, S.; Jing, L. A Multi-Task Network: Improving Unmanned Underwater Vehicle Self-Noise Separation via Sound Event Recognition. J. Mar. Sci. Eng. 2024, 12, 1563. https://doi.org/10.3390/jmse12091563
Shi W, Chen D, Tian F, Liu S, Jing L. A Multi-Task Network: Improving Unmanned Underwater Vehicle Self-Noise Separation via Sound Event Recognition. Journal of Marine Science and Engineering. 2024; 12(9):1563. https://doi.org/10.3390/jmse12091563
Chicago/Turabian StyleShi, Wentao, Dong Chen, Fenghua Tian, Shuxun Liu, and Lianyou Jing. 2024. "A Multi-Task Network: Improving Unmanned Underwater Vehicle Self-Noise Separation via Sound Event Recognition" Journal of Marine Science and Engineering 12, no. 9: 1563. https://doi.org/10.3390/jmse12091563