Submit to Entropy Review for Entropy Propose a Special Issue

Journal Menu

Journal Browser

► Journal Browser

Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Keywords
Published Papers

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Signal and Data Analysis".

Deadline for manuscript submissions: closed (31 October 2021) | Viewed by 38133

Share This Special Issue

Special Issue Editors

Prof. Dr. Tokunbo Ogunfunmi

E-Mail Website
Guest Editor

School of Engineering, Santa Clara University, 500 El Camino Real, Santa Clara, CA 95053, USA
Interests: deep learning; machine learning; adaptive filters; signal processing; applications
Special Issues, Collections and Topics in MDPI journals

Prof. Dr. David Luengo

E-Mail Website
Guest Editor

Department of Audiovisual and Communications Engineering, Technical University of Madrid, C/Nikola Tesla s/n, 28031 Madrid, Spain
Interests: biomedical signal processing; machine learning; adaptive Monte Carlo methods; Bayesian computation; statistical inference

Dr. Nithin V. George

E-Mail Website
Guest Editor

Department of Electrical Engineering, Indian Institute of Technology, Gandhinagar, India
Interests: active noise control; adaptive signal processing; assistive listening devices; psychoacoustics
Special Issues, Collections and Topics in MDPI journals

Dr. Danilo Comminiello

E-Mail Website
Guest Editor

Department of Information Engineering, Electronics and Telecommunications (DIET), Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
Interests: deep learning; adaptive filters; machine learning; audio signal processing
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Adaptive signal processing, machine learning, and deep learning which rely on the paradigm of learning from data have become indispensable tools for extracting information, making decisions, and interacting with our environment. The information extraction process is a very critical step in this process. Many of the algorithms deployed for information extraction have largely been based on using the popular mean square error (MSE) criterion. They leverage the significant information contained in the data. The more accurate the process of extracting useful information from the data, the more precise and efficient the learning and signal processing will be.

However, traditional mean square error (MSE) is not the optimal cost measure to use, when the error distribution is non-Gaussian, such as in supervised learning. In such cases, information theoretic learning (ITL)-based cost measures can provide better nonlinear models in a range of problems from system identification and regression to classification. Information theoretic learning (ITL) has initially been applied for such supervised learning applications.

Entropy and information theory have always represented useful tools to deal with information and the amount of information contained in a random variable. Information theory mainly relies on the basic intuition that learning that an unlikely event has occurred is more informative than learning that a likely event has occurred. Entropy gives a measure of the amount of information in an event drawn from a distribution. For this reason, they have been widely used in adaptive signal processing and machine learning to improve performance by designing and optimizing effective and specific models that fit the data even in noisy and adverse scenario conditions.

The presence of strong disturbances in the error signal can severely deteriorate convergence behavior of adaptive filters and in some cases cause the learning algorithms to diverge. Information theoretic learning (ITL) approaches have recently emerged as an effective solution to handle such scenarios.

Examples of several widely adopted measures include mutual information, cross-entropy, minimum error entropy (MEE) criterion, maximum correntropy criterion (MCC) and Kullback–Leibler divergence, among others. Moreover, a wide class of interesting tasks of adaptive signal processing, machine learning, and deep learning take advantage of entropy and information theory, including exploratory data analysis, feature and model selection, sampling and subset extraction, optimizing learning algorithms, clustering sensitivity analysis, representation learning, and data generation.

This Special Issue aims at providing recent developments in the areas of adaptive signal processing, machine learning, and deep learning using information theory and entropy to improve performance in widespread and popular problems and also to provide effective solutions to emerging problems.

The scope of the Special Issue includes theoretical and applications papers pertaining to all problems involving learning from data.

Prof. Dr. Tokunbo Ogunfunmi
Dr. David Luengo
Dr. Nithin V George
Dr. Danilo Comminiello
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

Adaptive signal processing, adaptive filters
Machine listening, deep learning
Information theoretic learning
Generalized maximum correntropy criterion (GMCC)
Maximum correntropy criterion (MCC), cyclic correntropy
Nonlinear adaptive filters
Robust signal processing, robust learning
Impulsive noise
Model selection and feature extraction
Bayesian learning and representation learning

Published Papers (12 papers)

Download All Papers

Order results

Result details

Show export options Show export options

Select all

Export citation of selected articles as:

Editorial

Jump to: Research, Review

5 pages, 188 KiB

Open AccessEditorial

Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory

by Tokunbo Ogunfunmi

Entropy 2022, 24(10), 1430; https://doi.org/10.3390/e24101430 - 8 Oct 2022

Viewed by 1292

Abstract

This Special Issue on “Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory” was birthed from observations of the recent trend in the literature [...] Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

Research

Jump to: Editorial, Review

20 pages, 1977 KiB

Open AccessArticle

Multi-Class Classification of Medical Data Based on Neural Network Pruning and Information-Entropy Measures

by Máximo Eduardo Sánchez-Gutiérrez and Pedro Pablo González-Pérez

Entropy 2022, 24(2), 196; https://doi.org/10.3390/e24020196 - 27 Jan 2022

Cited by 6 | Viewed by 4187

Abstract

Medical data includes clinical trials and clinical data such as patient-generated health data, laboratory results, medical imaging, and different signals coming from continuous health monitoring. Some commonly used data analysis techniques are text mining, big data analytics, and data mining. These techniques can be used for classification, clustering, and machine learning tasks. Machine learning could be described as an automatic learning process derived from concepts and knowledge without deliberate system coding. However, finding a suitable machine learning architecture for a specific task is still an open problem. In this work, we propose a machine learning model for the multi-class classification of medical data. This model is comprised of two components—a restricted Boltzmann machine and a classifier system. It uses a discriminant pruning method to select the most salient neurons in the hidden layer of the neural network, which implicitly leads to a selection of features for the input patterns that feed the classifier system. This study aims to investigate whether information-entropy measures may provide evidence for guiding discriminative pruning in a neural network for medical data processing, particularly cancer research, by using three cancer databases: Breast Cancer, Cervical Cancer, and Primary Tumour. Our proposal aimed to investigate the post-training neuronal pruning methodology using dissimilarity measures inspired by the information-entropy theory; the results obtained after pruning the neural network were favourable. Specifically, for the Breast Cancer dataset, the reported results indicate a 10.68% error rate, while our error rates range from 10% to 15%; for the Cervical Cancer dataset, the reported best error rate is 31%, while our proposal error rates are in the range of 4% to 6%; lastly, for the Primary Tumour dataset, the reported error rate is 20.35%, and our best error rate is 31%. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

26 pages, 409 KiB

Open AccessArticle

Generalizing the Balance Heuristic Estimator in Multiple Importance Sampling

by Mateu Sbert and Víctor Elvira

Entropy 2022, 24(2), 191; https://doi.org/10.3390/e24020191 - 27 Jan 2022

Cited by 8 | Viewed by 2617

Abstract

In this paper, we propose a novel and generic family of multiple importance sampling estimators. We first revisit the celebrated balance heuristic estimator, a widely used Monte Carlo technique for the approximation of intractable integrals. Then, we establish a generalized framework for the combination of samples simulated from multiple proposals. Our approach is based on considering as free parameters both the sampling rates and the combination coefficients, which are the same in the balance heuristics estimator. Thus our novel framework contains the balance heuristic as a particular case. We study the optimal choice of the free parameters in such a way that the variance of the resulting estimator is minimized. A theoretical variance study shows the optimal solution is always better than the balance heuristic estimator (except in degenerate cases where both are the same). We also give sufficient conditions on the parameter values for the new generalized estimator to be better than the balance heuristic estimator, and one necessary and sufficient condition related to

χ^{2}

divergence. Using five numerical examples, we first show the gap in the efficiency of both new and classical balance heuristic estimators, for equal sampling and for several state of the art sampling rates. Then, for these five examples, we find the variances for some notable selection of parameters showing that, for the important case of equal count of samples, our new estimator with an optimal selection of parameters outperforms the classical balance heuristic. Finally, new heuristics are introduced that exploit the theoretical findings. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

14 pages, 587 KiB

Open AccessArticle

A Probabilistic Re-Intepretation of Confidence Scores in Multi-Exit Models

by Jary Pomponi, Simone Scardapane and Aurelio Uncini

Entropy 2022, 24(1), 1; https://doi.org/10.3390/e24010001 - 21 Dec 2021

Cited by 4 | Viewed by 2951

Abstract

In this paper, we propose a new approach to train a deep neural network with multiple intermediate auxiliary classifiers, branching from it. These ‘multi-exits’ models can be used to reduce the inference time by performing early exit on the intermediate branches, if the confidence of the prediction is higher than a threshold. They rely on the assumption that not all the samples require the same amount of processing to yield a good prediction. In this paper, we propose a way to train jointly all the branches of a multi-exit model without hyper-parameters, by weighting the predictions from each branch with a trained confidence score. Each confidence score is an approximation of the real one produced by the branch, and it is calculated and regularized while training the rest of the model. We evaluate our proposal on a set of image classification benchmarks, using different neural models and early-exit stopping criteria. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

20 pages, 56636 KiB

Open AccessArticle

Ultra-Low-Power, High-Accuracy 434 MHz Indoor Positioning System for Smart Homes Leveraging Machine Learning Models

by Haq Nawaz, Ahsen Tahir, Nauman Ahmed, Ubaid U. Fayyaz, Tayyeb Mahmood, Abdul Jaleel, Mandar Gogate, Kia Dashtipour, Usman Masud and Qammer Abbasi

Entropy 2021, 23(11), 1401; https://doi.org/10.3390/e23111401 - 25 Oct 2021

Cited by 4 | Viewed by 2778

Abstract

Global navigation satellite systems have been used for reliable location-based services in outdoor environments. However, satellite-based systems are not suitable for indoor positioning due to low signal power inside buildings and low accuracy of 5 m. Future smart homes demand low-cost, high-accuracy and low-power indoor positioning systems that can provide accuracy of less than 5 m and enable battery operation for mobility and long-term use. We propose and implement an intelligent, highly accurate and low-power indoor positioning system for smart homes leveraging Gaussian Process Regression (GPR) model using information-theoretic gain based on reduction in differential entropy. The system is based on Time Difference of Arrival (TDOA) and uses ultra-low-power radio transceivers working at 434 MHz. The system has been deployed and tested using indoor measurements for two-dimensional (2D) positioning. In addition, the proposed system provides dual functionality with the same wireless links used for receiving telemetry data, with configurable data rates of up to 600 Kbauds. The implemented system integrates the time difference pulses obtained from the differential circuitry to determine the radio frequency (RF) transmitter node positions. The implemented system provides a high positioning accuracy of 0.68 m and 1.08 m for outdoor and indoor localization, respectively, when using GPR machine learning models, and provides telemetry data reception of 250 Kbauds. The system enables low-power battery operation with consumption of <200 mW power with ultra-low-power CC1101 radio transceivers and additional circuits with a differential amplifier. The proposed system provides low-cost, low-power and high-accuracy indoor localization and is an essential element of public well-being in future smart homes. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Graphical abstract

19 pages, 15317 KiB

Open AccessArticle

An Adaptive Deblurring Vehicle Detection Method for High-Speed Moving Drones: Resistance to Shake

by Yan Liu, Jingwen Wang, Tiantian Qiu and Wenting Qi

Entropy 2021, 23(10), 1358; https://doi.org/10.3390/e23101358 - 18 Oct 2021

Cited by 4 | Viewed by 2139

Abstract

Vehicle detection is an essential part of an intelligent traffic system, which is an important research field in drone application. Because unmanned aerial vehicles (UAVs) are rarely configured with stable camera platforms, aerial images are easily blurred. There is a challenge for detectors to accurately locate vehicles in blurred images in the target detection process. To improve the detection performance of blurred images, an end-to-end adaptive vehicle detection algorithm (DCNet) for drones is proposed in this article. First, the clarity evaluation module is used to determine adaptively whether the input image is a blurred image using improved information entropy. An improved GAN called Drone-GAN is proposed to enhance the vehicle features of blurred images. Extensive experiments were performed, the results of which show that the proposed method can detect both blurred and clear images well in poor environments (complex illumination and occlusion). The detector proposed achieves larger gains compared with SOTA detectors. The proposed method can enhance the vehicle feature details in blurred images effectively and improve the detection accuracy of blurred aerial images, which shows good performance with regard to resistance to shake. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

23 pages, 77656 KiB

Open AccessArticle

A Novel Noise Reduction Method of UAV Magnetic Survey Data Based on CEEMDAN, Permutation Entropy, Correlation Coefficient and Wavelet Threshold Denoising

by Yaoxin Zheng, Shiyan Li, Kang Xing and Xiaojuan Zhang

Entropy 2021, 23(10), 1309; https://doi.org/10.3390/e23101309 - 6 Oct 2021

Cited by 12 | Viewed by 2504

Abstract

Despite the increased attention that has been given to the unmanned aerial vehicle (UAV)-based magnetic survey systems in the past decade, the processing of UAV magnetic data is still a tough task. In this paper, we propose a novel noise reduction method of UAV magnetic data based on complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN), permutation entropy (PE), correlation coefficient and wavelet threshold denoising. The original signal is first decomposed into several intrinsic mode functions (IMFs) by CEEMDAN, and the PE of each IMF is calculated. Second, IMFs are divided into four categories according to the quartiles of PE, namely, noise IMFs, noise-dominant IMFs, signal-dominant IMFs, and signal IMFs. Then the noise IMFs are removed, and correlation coefficients are used to identify the real signal-dominant IMFs. Finally, the wavelet threshold denoising is applied to the real signal-dominant IMFs, the denoised signal can be obtained by combining the signal IMFs and the denoised IMFs. Both synthetic and field experiments are conducted to verify the effectiveness of the proposed method. The results show that the proposed method can eliminate the interference to a great extent, which lays a foundation for the further interpretation of UAV magnetic data. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

18 pages, 4892 KiB

Open AccessArticle

Leak Detection in Water Pipes Based on Maximum Entropy Version of Least Square Twin K-Class Support Vector Machine

by Mingyang Liu, Jin Yang and Wei Zheng

Entropy 2021, 23(10), 1247; https://doi.org/10.3390/e23101247 - 25 Sep 2021

Cited by 4 | Viewed by 1951

Abstract

Numerous novel improved support vector machine (SVM) methods are used in leak detection of water pipelines at present. The least square twin K-class support vector machine (LST-KSVC) is a novel simple and fast multi-classification method. However, LST-KSVC has a non-negligible drawback that it assigns the same classification weights to leak samples, including outliers that affect classification, these outliers are often situated away from the main leak samples. To overcome this shortcoming, the maximum entropy (MaxEnt) version of the LST-KSVC is proposed in this paper, called the MLT-KSVC algorithm. In this classification approach, classification weights of leak samples are calculated based on the MaxEnt model. Different sample points are assigned different weights: large weights are assigned to primary leak samples and outliers are assigned small weights, hence the outliers can be ignored in the classification process. Leak recognition experiments prove that the proposed MLT-KSVC algorithm can reduce the impact of outliers on the classification process and avoid the misclassification color block drawback in linear LST-KSVC. MLT-KSVC is more accurate compared with LST-KSVC, TwinSVC, TwinKSVC, and classic Multi-SVM. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

15 pages, 1799 KiB

Open AccessArticle

Passive Tracking of Multiple Underwater Targets in Incomplete Detection and Clutter Environment

by Xiaohua Li, Bo Lu, Wasiq Ali and Haiyan Jin

Entropy 2021, 23(8), 1082; https://doi.org/10.3390/e23081082 - 20 Aug 2021

Cited by 4 | Viewed by 2347

Abstract

A major advantage of the use of passive sonar in the tracking multiple underwater targets is that they can be kept covert, which reduces the risk of being attacked. However, the nonlinearity of the passive Doppler and bearing measurements, the range unobservability problem, and the complexity of data association between measurements and targets make the problem of underwater passive multiple target tracking challenging. To deal with these problems, the cardinalized probability hypothesis density (CPHD) recursion, which is based on Bayesian information theory, is developed to handle the data association uncertainty, and to acquire existing targets’ numbers and states (e.g., position and velocity). The key idea of the CPHD recursion is to simultaneously estimate the targets’ intensity and the probability distribution of the number of targets. The CPHD recursion is the first moment approximation of the Bayesian multiple targets filter, which avoids the data association procedure between the targets and measurements including clutter. The Bayesian-filter-based extended Kalman filter (EKF) is applied to deal with the nonlinear bearing and Doppler measurements. The experimental results show that the EKF-based CPHD recursion works well in the underwater passive multiple target tracking system in cluttered and noisy environments. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

17 pages, 3249 KiB

Open AccessArticle

An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders

by Eleonora Grassucci, Danilo Comminiello and Aurelio Uncini

Entropy 2021, 23(7), 856; https://doi.org/10.3390/e23070856 - 3 Jul 2021

Cited by 11 | Viewed by 2581

Abstract

Variational autoencoders are deep generative models that have recently received a great deal of attention due to their ability to model the latent distribution of any kind of input such as images and audio signals, among others. A novel variational autoncoder in the [...] Read more.

H

, namely the QVAE, has been recently proposed, leveraging the augmented second order statics of

H

-proper signals. In this paper, we analyze the QVAE under an information-theoretic perspective, studying the ability of the

H

-proper model to approximate improper distributions as well as the built-in

H

-proper ones and the loss of entropy due to the improperness of the input signal. We conduct experiments on a substantial set of quaternion signals, for each of which the QVAE shows the ability of modelling the input distribution, while learning the improperness and increasing the entropy of the latent space. The proposed analysis will prove that proper QVAEs can be employed with a good approximation even when the quaternion input data are improper. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

18 pages, 7996 KiB

Open AccessArticle

A Refined Composite Multivariate Multiscale Fluctuation Dispersion Entropy and Its Application to Multivariate Signal of Rotating Machinery

by Chenbo Xi, Guangyou Yang, Lang Liu, Hongyuan Jiang and Xuehai Chen

Entropy 2021, 23(1), 128; https://doi.org/10.3390/e23010128 - 19 Jan 2021

Cited by 14 | Viewed by 3320

Abstract

In the fault monitoring of rotating machinery, the vibration signal of the bearing and gear in a complex operating environment has poor stationarity and high noise. How to accurately and efficiently identify various fault categories is a major challenge in rotary fault diagnosis. Most of the existing methods only analyze the single channel vibration signal and do not comprehensively consider the multi-channel vibration signal. Therefore, this paper presents Refined Composite Multivariate Multiscale Fluctuation Dispersion Entropy (RCMMFDE), a method which extracts the recognition information of multi-channel signals with different scale factors, and the refined composite analysis ensures the recognition stability. The simulation results show that this method has the characteristics of low sensitivity to signal length and strong anti-noise ability. At the same time, combined with Joint Mutual Information Maximisation (JMIM) and support vector machine (SVM), RCMMFDE-JMIM-SVM fault diagnosis method has been proposed. This method uses RCMMFDE to extract the state characteristics of the multiple vibration signals of the rotary machine, and then uses the JMIM method to extract the sensitive characteristics. Finally, different states of the rotary machine are classified by SVM. The validity of the method is verified by the composite gear fault data set and bearing fault data set. The diagnostic accuracy of the method is 99.25% and 100.00%. The experimental results show that RCMMFDE-JMIM-SVM can effectively recognize multiple signals. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Figure 1

Review

Jump to: Editorial, Research

55 pages, 2165 KiB

Open AccessReview

An Overview of Variational Autoencoders for Source Separation, Finance, and Bio-Signal Applications

by Aman Singh and Tokunbo Ogunfunmi

Entropy 2022, 24(1), 55; https://doi.org/10.3390/e24010055 - 28 Dec 2021

Cited by 32 | Viewed by 7215

Abstract

Autoencoders are a self-supervised learning system where, during training, the output is an approximation of the input. Typically, autoencoders have three parts: Encoder (which produces a compressed latent space representation of the input data), the Latent Space (which retains the knowledge in the input data with reduced dimensionality but preserves maximum information) and the Decoder (which reconstructs the input data from the compressed latent space). Autoencoders have found wide applications in dimensionality reduction, object detection, image classification, and image denoising applications. Variational Autoencoders (VAEs) can be regarded as enhanced Autoencoders where a Bayesian approach is used to learn the probability distribution of the input data. VAEs have found wide applications in generating data for speech, images, and text. In this paper, we present a general comprehensive overview of variational autoencoders. We discuss problems with the VAEs and present several variants of the VAEs that attempt to provide solutions to the problems. We present applications of variational autoencoders for finance (a new and emerging field of application), speech/audio source separation, and biosignal applications. Experimental results are presented for an example of speech source separation to illustrate the powerful application of variants of VAE: VAE,

β

-VAE, and ITL-AE. We conclude the paper with a summary, and we identify possible areas of research in improving performance of VAEs in particular and deep generative models in general, of which VAEs and generative adversarial networks (GANs) are examples. Full article

(This article belongs to the Special Issue Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory)

► Show Figures

Journal Menu

Journal Browser

Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Published Papers (12 papers)

Editorial

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI