Detection of Anomalies in Natural Complicated Data Structures Based on a Hybrid Approach
Abstract
:1. Introduction
2. Description of the Applied Methods
2.1. Singular Spectrum Analysis
- Transformation of tshe initial one-dimensional series F into a trajectory matrix,
- 2.
- Singular decomposition of the obtained trajectory matrix .
- 3.
- The grouping of the set of elementary matrixes from item 2 on non-intersecting subsets , {}. Assume that , then the resulting matrix , corresponding to group , is determined as .
- 4.
- Matrixes of the grouped decomposition are Hankelized (are averaged over anti-diagonals). Using the correspondence between the Hankel matrixes and the time series, the recovered series are obtained. The initial series is decomposed into a sum of the recovered series, where each value of the initial series is equal to
2.2. Autoencoder Neural Network
2.3. Adaptive Anomaly Detection Algorithm
- 2.
- A threshold function is applied to wavelet coefficients of the time series decomposition,
- 3.
- For the detected anomalies, their intensities at the time instant can be estimated as follows:
2.4. Scheme of Method Realization
3. Data Processing Results
Results of the Estimates of the Confined Dispersion Fraction
4. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Kuznetsov, V.D. Space weather and risks of space activity. Space Tech. Technol. 2014, 3, 3–13. [Google Scholar]
- Badruddin, B.; Aslam, O.P.M.; Derouich, M.; Asiri, H.; Kudela, K. Forbush decreases and geomagnetic storms during a highly disturbed solar and interplanetary period, 4–10 September 2017. Space Weather 2019, 17, 487. [Google Scholar] [CrossRef]
- Gocheva-Ilieva, S.; Ivanov, A.; Kulina, H.; Stoimenova-Minova, M. Multi-Step Ahead Ex-Ante Forecasting of Air Pollutants Using Machine Learning. Mathematics 2023, 11, 1566. [Google Scholar] [CrossRef]
- Dorman, L.I. Space weather and dangerous phenomena on the earth: Principles of great geomagnetic storms forcasting by online cosmic ray data. Ann. Geophys. 2005, 23, 2997–3002. [Google Scholar] [CrossRef]
- Bojang, P.O.; Yang, T.-C.; Pham, Q.B.; Yu, P.-S. Linking Singular Spectrum Analysis and Machine Learning for Monthly Rainfall Forecasting. Appl. Sci. 2020, 10, 3224. [Google Scholar] [CrossRef]
- Yu, H.; Chen, Q.; Sun, Y.; Sosnica, K. Geophysical Signal Detection in the Earth’s Oblateness Variation and Its Climate-Driven Source Analysis. Remote Sens. 2021, 13, 2004. [Google Scholar] [CrossRef]
- Belonin, M.D.; Tatarinov, I.V.; Kalinin, O.M. Factor Analysis in Petroleum Geology; VIEMS: Kaluga, Russia, 1971; p. 56. [Google Scholar]
- Danilov, D.L.; Zhiglyavsky, A.A. Principal Components of Time Series: The Caterpillar Method; Presskom: St. Petersburg, Russia, 1997; p. 308. [Google Scholar]
- Broomhead, D.S.; King, G.P. Extracting qualitative dynamics from experimental data. Phys. Nonlinear Phenom. 1986, 20, 217–236. [Google Scholar] [CrossRef]
- Colebrook, J.M. Continuous plankton records—Zooplankton and environment, northeast Atlanticand North Sea. Oceanol. Acta 1978, 1, 9–23. [Google Scholar]
- Mallat, S.G. A Wavelet Tour of Signal Processing; Academic Press: San Diego, CA, USA, 1999. [Google Scholar]
- Herley, C.; Kovacevic, J.; Ramchandran, K.; Vetterli, M. Tilings of the time-frequency plane: Construction of arbitrary orthogonal bases and feist tiling algorithms. IEEE Trans. Signal Proc. 1993, 41, 3341–3359. [Google Scholar] [CrossRef]
- Chen, S.; Donoho, D. Atomic Decomposition by Basis Pursuit; Technical Report; Stanford University: Stanford, CA, USA, 1995. [Google Scholar]
- Mallat, S.G.; Zhang, Z.F. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 1993, 41, 3397–3415. [Google Scholar] [CrossRef]
- Coifman, R.R.; Wickerhauser, M.V. Entropy-based algorithms for best basis selection. IEEE Trans. Inf. Theory 1992, 38, 713–718. [Google Scholar] [CrossRef]
- Mandrikova, O.; Mandrikova, B.; Rodomanskay, A. Method of Constructing a Nonlinear Approximating Scheme of a Complex Signal: Application Pattern Recognition. Mathematics 2021, 9, 737. [Google Scholar] [CrossRef]
- Kudela, K.; Rybak, J.; Antalová, A.; Storini, M. Time Evolution of low-Frequency Periodicities in Cosmic ray Intensity. Sol. Phys. 2002, 205, 165–175. [Google Scholar] [CrossRef]
- Stamper, R.; Lockwood, M.; Wild, M.N.; Clark, T.D.G. Solar causes of the long-term increase in geomagnetic activity. J. Geophys. Res. 1999, 104, 325. [Google Scholar] [CrossRef]
- Mandrikova, O.; Mandrikova, B. Hybrid Method for Detecting Anomalies in Cosmic ray Variations Using Neural Networks Autoencoder. Symmetry 2022, 14, 744. [Google Scholar] [CrossRef]
- Belov, A.; Eroshenko, E.; Gushchina, R.; Dorman, L.; Oleneva, V.; Yanke, V. Cosmic ray variations as a tool for studying solar-terrestrial relations. Electromagn. Plasma Process. Body Sun Body Earth 2015, 258–284. [Google Scholar]
- Papailiou, M.; Mavromichalaki, H.; Belov, A.; Eroshenko, E.; Yanke, V. Precursor Effects in Different Cases of Forbush Decreases. Sol. Phys. 2011, 276, 337–350. [Google Scholar] [CrossRef]
- Forbush, S.E. On the Effects in the Cosmic Ray Intensity Observed during Magnetic Storms. Phys. Rev. 1937, 51, 1108–1109.10. [Google Scholar] [CrossRef]
- Gololobov, P.Y.; Krivoshapkin, P.A.; Krymsky, G.F.; Gerasimova, S.K. Investigating the influence of geometry of the heliospheric neutral current sheet and solar activity on modulation of galactic cosmic rays with a method of main components. Sol. -Terr. Phys. 2020, 6, 24–28. [Google Scholar]
- Abunina, M.A.; Belov, A.V.; Eroshenko, E.A.; Abunin, A.A.; Oleneva, V.A.; Yanke, V.G.; Melkumyan, A.A. Ring of Station Method in Research of Cosmic Ray Variations: 1. General Description. Geomagn. Aeron. 2020, 60, 38–45. [Google Scholar] [CrossRef]
- Koundal, P. Graph Neural Networks and Application for Cosmic-Ray Analysis. In Proceedings of the 5th International Workshop on Deep Learning in Computational Physics, Dubna, Russia, 28–29 June 2021. [Google Scholar] [CrossRef]
- Piekarczyk, M.; Bar, O.; Bibrzycki, Ł.; Niedźwiecki, M.; Rzecki, K.; Stuglik, S.; Andersen, T.; Budnev, N.M.; Alvarez-Castillo, D.E.; Cheminant, K.A.; et al. CNN-Based Classifier as an Offline Trigger for the CREDO Experiment. Sensors 2021, 21, 4804. [Google Scholar] [CrossRef] [PubMed]
- Ahn, B.-H.; Moon, G.-H.; Sun, W.; Akasofu, S.-I.; Chen, G.X.; Park, Y.D. Universal time variation of the Dst index and the relationship between the cumulative AL and Dst indices during geomagnetic storms. J. Geophys. Res. 2002, 107, 1409. [Google Scholar] [CrossRef]
- Pattanayak, S. Pro Deep Learning with TensorFlow: A Mathematical Approach to Advanced Artificial Intelligence in Python; Apress: Bangalore, India, 2017; p. 398. [Google Scholar]
- Chui, C.K. An Introduction to Wavelets; Wavelet Analysis and Its Applications; Academic Press: Boston, MA, USA, 1992; ISBN 978-0-12-174584-4. [Google Scholar]
- Daubechies, I. Ten Lectures on Wavelets; CBMS-NSF Regional Conference Series in Applied Mathematics; Society for Industrial and Applied Mathematics: Philadelphia, PA, USA, 1992. [Google Scholar]
- Witte, R.S.; Witte, J.S. Statistics, 11th ed.; Wiley: New York, NY, USA, 2017; p. 496. [Google Scholar]
- Real Time Data Base for the Measurements of High-Resolution Neutron Monitor. Available online: https://www.nmdb.eu (accessed on 30 March 2023).
- Kuzmin, Y. Registration of the intensity of the neutron flux in Kamchatka in connection with the forecast of earthquakes. In Proceedings of the Conference Geophysical Monitoring of Kamchatka, Kamchatka, Russia, 20 September–1 October 2006; pp. 149–156. [Google Scholar]
- Schlickeiser, R. Cosmic Ray Astrophysics; Springer GmbH & Co., KG.: Berlin/Heidelberg, Germany, 2002; p. 519. [Google Scholar]
- Institute of Applied Geophysics. Available online: http://ipg.geospace.ru/ (accessed on 30 March 2023).
- Geomagnetic Equatorial Dst Index. Available online: https://wdc.kugi.kyoto-u.ac.jp/dstdir/ (accessed on 30 March 2023).
Component | Confined Dispersion Fraction |
---|---|
com1 | 0.8942 |
com1 + com2 | 0.9288 |
com1 + com2 + com3 | 0.9501 |
com1 + com2 + com3 + com4 | 0.9607 |
com1 + com2 + com3 + com4 + com5 | 0.9655 |
com1 + com2 + com3 + com4 + com5 + com6 | 0.9681 |
com1 + com2 + com3 + com4 + com5 + com6 + com7 | 0.9703 |
Component | Confined Dispersion Fraction |
---|---|
com1 | 0.8262 |
com1 + com2 | 0.8934 |
com1 + com2 + com3 | 0.9247 |
com1 + com2 + com3 + com4 | 0.9353 |
com1 + com2 + com3 + com4 + com5 | 0.9395 |
com1 + com2 + com3 + com4 + com5 + com6 | 0.9427 |
com1 + com2 + com3 + com4 + com5 + com6 + com7 | 0.9458 |
Component | Confined Dispersion Fraction |
---|---|
com1 | 0.7508 |
com1 + com2 | 0.8490 |
com1 + com2 + com3 | 0.9074 |
com1 + com2 + com3 + com4 | 0.9291 |
com1 + com2 + com3 + com4 + com5 | 0.9395 |
com1 + com2 + com3 + com4 + com5 + com6 | 0.9470 |
com1 + com2 + com3 + com4 + com5 + com6 + com7 | 0.9526 |
Component | Confined Dispersion Fraction |
---|---|
com1 | 0.6269 |
com1 + com2 | 0.7581 |
com1 + com2 + com3 | 0.8307 |
com1 + com2 + com3 + com4 | 0.8649 |
com1 + com2 + com3 + com4 + com5 | 0.8835 |
com1 + com2 + com3 + com4 + com5 + com6 | 0.8964 |
com1 + com2 + com3 + com4 + com5 + com6 + com7 | 0.9049 |
Period | Number of Geomagnetic Disturbances and Geomagnetic Storms | Wavelet Function | Moving Time Window Dimension | Result |
---|---|---|---|---|
2013–2015, 2019–2020 | 405 | Coiflet 2 | Detected: 64% Undetected: 36% False alarm: 32 events | |
Detected: 78% Undetected: 22% False alarm: 27 events | ||||
Detected: 87% Undetected: 13% False alarm: 27 events |
Period | Number of Geomagnetic Disturbances and Geomagnetic Storms | Moving Time Window Dimension | Wavelet Function | Results |
---|---|---|---|---|
2013–2015, 2019–2020 | 405 | Coiflet 1 | Detected: 87% Undetected: 13% False alarm: 29 events | |
Coiflet 2 | Detected: 87% Undetected: 13% False alarm: 27 events | |||
Coiflet 3 | Detected: 85% Undetected: 15% False alarm: 28 events | |||
Daubechies 1 | Detected: 84% Undetected: 16% False alarm: 29 events | |||
Daubechies 2 | Detected: 86% Undetected: 14% False alarm: 29 events |
Period | Number of Geomagnetic Disturbances and Geomagnetic Storms | Results of SSA + AADA |
---|---|---|
2013–2015, 2019–2020 | 405 | Detected: 84% |
Undetected: 16% | ||
False alarm: 35 events |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Mandrikova, O.; Mandrikova, B.; Esikov, O. Detection of Anomalies in Natural Complicated Data Structures Based on a Hybrid Approach. Mathematics 2023, 11, 2464. https://doi.org/10.3390/math11112464
Mandrikova O, Mandrikova B, Esikov O. Detection of Anomalies in Natural Complicated Data Structures Based on a Hybrid Approach. Mathematics. 2023; 11(11):2464. https://doi.org/10.3390/math11112464
Chicago/Turabian StyleMandrikova, Oksana, Bogdana Mandrikova, and Oleg Esikov. 2023. "Detection of Anomalies in Natural Complicated Data Structures Based on a Hybrid Approach" Mathematics 11, no. 11: 2464. https://doi.org/10.3390/math11112464