Research

Jump to: Review

13 pages, 318 KiB

Open AccessArticle

Computation of Kullback–Leibler Divergence in Bayesian Networks

by Serafín Moral, Andrés Cano and Manuel Gómez-Olmedo

Entropy 2021, 23(9), 1122; https://doi.org/10.3390/e23091122 - 28 Aug 2021

Cited by 5 | Viewed by 2287

Abstract

Kullback–Leibler divergence

K L (p, q)

is the standard measure of error when we have a true probability distribution p which is approximate with probability distribution q. Its efficient computation is essential in many tasks, as in approximate computation [...] Read more.

Kullback–Leibler divergence

K L (p, q)

is the standard measure of error when we have a true probability distribution p which is approximate with probability distribution q. Its efficient computation is essential in many tasks, as in approximate computation or as a measure of error when learning a probability. In high dimensional probabilities, as the ones associated with Bayesian networks, a direct computation can be unfeasible. This paper considers the case of efficiently computing the Kullback–Leibler divergence of two probability distributions, each one of them coming from a different Bayesian network, which might have different structures. The paper is based on an auxiliary deletion algorithm to compute the necessary marginal distributions, but using a cache of operations with potentials in order to reuse past computations whenever they are necessary. The algorithms are tested with Bayesian networks from the bnlearn repository. Computer code in Python is provided taking as basis pgmpy, a library for working with probabilistic graphical models. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

36 pages, 960 KiB

Open AccessArticle

Extended Variational Message Passing for Automated Approximate Bayesian Inference

by Semih Akbayrak, Ivan Bocharov and Bert de Vries

Entropy 2021, 23(7), 815; https://doi.org/10.3390/e23070815 - 26 Jun 2021

Cited by 8 | Viewed by 2316

Abstract

Variational Message Passing (VMP) provides an automatable and efficient algorithmic framework for approximating Bayesian inference in factorized probabilistic models that consist of conjugate exponential family distributions. The automation of Bayesian inference tasks is very important since many data processing problems can be formulated [...] Read more.

Variational Message Passing (VMP) provides an automatable and efficient algorithmic framework for approximating Bayesian inference in factorized probabilistic models that consist of conjugate exponential family distributions. The automation of Bayesian inference tasks is very important since many data processing problems can be formulated as inference tasks on a generative probabilistic model. However, accurate generative models may also contain deterministic and possibly nonlinear variable mappings and non-conjugate factor pairs that complicate the automatic execution of the VMP algorithm. In this paper, we show that executing VMP in complex models relies on the ability to compute the expectations of the statistics of hidden variables. We extend the applicability of VMP by approximating the required expectation quantities in appropriate cases by importance sampling and Laplace approximation. As a result, the proposed Extended VMP (EVMP) approach supports automated efficient inference for a very wide range of probabilistic model specifications. We implemented EVMP in the Julia language in the probabilistic programming package ForneyLab.jl and show by a number of examples that EVMP renders an almost universal inference engine for factorized probabilistic models. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

19 pages, 505 KiB

Open AccessArticle

Improved Local Search with Momentum for Bayesian Networks Structure Learning

by Xiaohan Liu, Xiaoguang Gao, Zidong Wang and Xinxin Ru

Entropy 2021, 23(6), 750; https://doi.org/10.3390/e23060750 - 15 Jun 2021

Cited by 3 | Viewed by 1724

Abstract

Bayesian Networks structure learning (BNSL) is a troublesome problem that aims to search for an optimal structure. An exact search tends to sacrifice a significant amount of time and memory to promote accuracy, while the local search can tackle complex networks with thousands [...] Read more.

Bayesian Networks structure learning (BNSL) is a troublesome problem that aims to search for an optimal structure. An exact search tends to sacrifice a significant amount of time and memory to promote accuracy, while the local search can tackle complex networks with thousands of variables but commonly gets stuck in a local optimum. In this paper, two novel and practical operators and a derived operator are proposed to perturb structures and maintain the acyclicity. Then, we design a framework, incorporating an influential perturbation factor integrated by three proposed operators, to escape current local optimal and improve the dilemma that outcomes trap in local optimal. The experimental results illustrate that our algorithm can output competitive results compared with the state-of-the-art constraint-based method in most cases. Meanwhile, our algorithm reaches an equivalent or better solution found by the state-of-the-art exact search and hybrid methods. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

32 pages, 612 KiB

Open AccessArticle

Message Passing-Based Inference for Time-Varying Autoregressive Models

by Albert Podusenko, Wouter M. Kouw and Bert de Vries

Entropy 2021, 23(6), 683; https://doi.org/10.3390/e23060683 - 28 May 2021

Cited by 6 | Viewed by 2911

Abstract

Time-varying autoregressive (TVAR) models are widely used for modeling of non-stationary signals. Unfortunately, online joint adaptation of both states and parameters in these models remains a challenge. In this paper, we represent the TVAR model by a factor graph and solve the inference [...] Read more.

Time-varying autoregressive (TVAR) models are widely used for modeling of non-stationary signals. Unfortunately, online joint adaptation of both states and parameters in these models remains a challenge. In this paper, we represent the TVAR model by a factor graph and solve the inference problem by automated message passing-based inference for states and parameters. We derive structured variational update rules for a composite “AR node” with probabilistic observations that can be used as a plug-in module in hierarchical models, for example, to model the time-varying behavior of the hyper-parameters of a time-varying AR model. Our method includes tracking of variational free energy (FE) as a Bayesian measure of TVAR model performance. The proposed methods are verified on a synthetic data set and validated on real-world data from temperature modeling and speech enhancement tasks. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

28 pages, 1588 KiB

Open AccessArticle

Inference and Learning in a Latent Variable Model for Beta Distributed Interval Data

by Hamid Mousavi, Mareike Buhl, Enrico Guiraud, Jakob Drefs and Jörg Lücke

Entropy 2021, 23(5), 552; https://doi.org/10.3390/e23050552 - 29 Apr 2021

Cited by 2 | Viewed by 2231

Abstract

Latent Variable Models (LVMs) are well established tools to accomplish a range of different data processing tasks. Applications exploit the ability of LVMs to identify latent data structure in order to improve data (e.g., through denoising) or to estimate the relation between latent [...] Read more.

Latent Variable Models (LVMs) are well established tools to accomplish a range of different data processing tasks. Applications exploit the ability of LVMs to identify latent data structure in order to improve data (e.g., through denoising) or to estimate the relation between latent causes and measurements in medical data. In the latter case, LVMs in the form of noisy-OR Bayes nets represent the standard approach to relate binary latents (which represent diseases) to binary observables (which represent symptoms). Bayes nets with binary representation for symptoms may be perceived as a coarse approximation, however. In practice, real disease symptoms can range from absent over mild and intermediate to very severe. Therefore, using diseases/symptoms relations as motivation, we here ask how standard noisy-OR Bayes nets can be generalized to incorporate continuous observables, e.g., variables that model symptom severity in an interval from healthy to pathological. This transition from binary to interval data poses a number of challenges including a transition from a Bernoulli to a Beta distribution to model symptom statistics. While noisy-OR-like approaches are constrained to model how causes determine the observables’ mean values, the use of Beta distributions additionally provides (and also requires) that the causes determine the observables’ variances. To meet the challenges emerging when generalizing from Bernoulli to Beta distributed observables, we investigate a novel LVM that uses a maximum non-linearity to model how the latents determine means and variances of the observables. Given the model and the goal of likelihood maximization, we then leverage recent theoretical results to derive an Expectation Maximization (EM) algorithm for the suggested LVM. We further show how variational EM can be used to efficiently scale the approach to large networks. Experimental results finally illustrate the efficacy of the proposed model using both synthetic and real data sets. Importantly, we show that the model produces reliable results in estimating causes using proofs of concepts and first tests based on real medical data and on images. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

24 pages, 406 KiB

Open AccessArticle

Mixture-Based Probabilistic Graphical Models for the Label Ranking Problem

by Enrique G. Rodrigo, Juan C. Alfaro, Juan A. Aledo and José A. Gámez

Entropy 2021, 23(4), 420; https://doi.org/10.3390/e23040420 - 31 Mar 2021

Cited by 6 | Viewed by 2257

Abstract

The goal of the Label Ranking (LR) problem is to learn preference models that predict the preferred ranking of class labels for a given unlabeled instance. Different well-known machine learning algorithms have been adapted to deal with the LR problem. In particular, fine-tuned [...] Read more.

The goal of the Label Ranking (LR) problem is to learn preference models that predict the preferred ranking of class labels for a given unlabeled instance. Different well-known machine learning algorithms have been adapted to deal with the LR problem. In particular, fine-tuned instance-based algorithms (e.g., k-nearest neighbors) and model-based algorithms (e.g., decision trees) have performed remarkably well in tackling the LR problem. Probabilistic Graphical Models (PGMs, e.g., Bayesian networks) have not been considered to deal with this problem because of the difficulty of modeling permutations in that framework. In this paper, we propose a Hidden Naive Bayes classifier (HNB) to cope with the LR problem. By introducing a hidden variable, we can design a hybrid Bayesian network in which several types of distributions can be combined: multinomial for discrete variables, Gaussian for numerical variables, and Mallows for permutations. We consider two kinds of probabilistic models: one based on a Naive Bayes graphical structure (where only univariate probability distributions are estimated for each state of the hidden variable) and another where we allow interactions among the predictive attributes (using a multivariate Gaussian distribution for the parameter estimation). The experimental evaluation shows that our proposals are competitive with the start-of-the-art algorithms in both accuracy and in CPU time requirements. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

19 pages, 785 KiB

Open AccessArticle

A Robust Solution to Variational Importance Sampling of Minimum Variance

by Jerónimo Hernández-González and Jesús Cerquides

Entropy 2020, 22(12), 1405; https://doi.org/10.3390/e22121405 - 12 Dec 2020

Cited by 3 | Viewed by 2272

Abstract

Importance sampling is a Monte Carlo method where samples are obtained from an alternative proposal distribution. This can be used to focus the sampling process in the relevant parts of space, thus reducing the variance. Selecting the proposal that leads to the minimum [...] Read more.

Importance sampling is a Monte Carlo method where samples are obtained from an alternative proposal distribution. This can be used to focus the sampling process in the relevant parts of space, thus reducing the variance. Selecting the proposal that leads to the minimum variance can be formulated as an optimization problem and solved, for instance, by the use of a variational approach. Variational inference selects, from a given family, the distribution which minimizes the divergence to the distribution of interest. The Rényi projection of order 2 leads to the importance sampling estimator of minimum variance, but its computation is very costly. In this study with discrete distributions that factorize over probabilistic graphical models, we propose and evaluate an approximate projection method onto fully factored distributions. As a result of our evaluation it becomes apparent that a proposal distribution mixing the information projection with the approximate Rényi projection of order 2 could be interesting from a practical perspective. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures

Figure 1

Review

Jump to: Research

27 pages, 510 KiB

Open AccessFeature PaperReview

Probabilistic Models with Deep Neural Networks

by Andrés R. Masegosa, Rafael Cabañas, Helge Langseth, Thomas D. Nielsen and Antonio Salmerón

Entropy 2021, 23(1), 117; https://doi.org/10.3390/e23010117 - 18 Jan 2021

Cited by 4 | Viewed by 4382

Abstract

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to very restricted model classes, where exact or approximate probabilistic inference is feasible. However, developments in variational inference, a general form of approximate probabilistic [...] Read more.

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to very restricted model classes, where exact or approximate probabilistic inference is feasible. However, developments in variational inference, a general form of approximate probabilistic inference that originated in statistical physics, have enabled probabilistic modeling to overcome these limitations: (i) Approximate probabilistic inference is now possible over a broad class of probabilistic models containing a large number of parameters, and (ii) scalable inference methods based on stochastic gradient descent and distributed computing engines allow probabilistic modeling to be applied to massive data sets. One important practical consequence of these advances is the possibility to include deep neural networks within probabilistic models, thereby capturing complex non-linear stochastic relationships between the random variables. These advances, in conjunction with the release of novel probabilistic modeling toolboxes, have greatly expanded the scope of applications of probabilistic models, and allowed the models to take advantage of the recent strides made by the deep learning community. In this paper, we provide an overview of the main concepts, methods, and tools needed to use deep neural networks within a probabilistic modeling framework. Full article

(This article belongs to the Special Issue Bayesian Inference in Probabilistic Graphical Models)

► Show Figures