Next Article in Journal
Information-Dispersion-Entropy-Based Blind Recognition of Binary BCH Codes in Soft Decision Situations
Next Article in Special Issue
Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data
Previous Article in Journal
Application of Maximum Entropy Method to Semiconductor Engineering
Previous Article in Special Issue
A Novel Nonparametric Distance Estimator for Densities with Error Bounds
Article Menu

Export Article

Open AccessArticle
Entropy 2013, 15(5), 1690-1704; doi:10.3390/e15051690

An Estimate of Mutual Information that Permits Closed-Form Optimisation

Department of Computing, Imperial College, London SW7 2RH, UK
Author to whom correspondence should be addressed.
Received: 1 February 2013 / Revised: 18 April 2013 / Accepted: 28 April 2013 / Published: 8 May 2013
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)
View Full-Text   |   Download PDF [740 KB, uploaded 24 February 2015]   |  


We introduce a new estimate of mutual information between a dataset and a target variable that can be maximised analytically and has broad applicability in the field of machine learning and statistical pattern recognition. This estimate has previously been employed implicitly as an approximation to quadratic mutual information. In this paper we will study the properties of these estimates of mutual information in more detail, and provide a derivation from a perspective of pairwise interactions. From this perspective, we will show a connection between our proposed estimate and Laplacian eigenmaps, which so far has not been shown to be related to mutual information. Compared with other popular measures of mutual information, which can only be maximised through an iterative process, ours can be maximised much more efficiently and reliably via closed-form eigendecomposition. View Full-Text
Keywords: mutual information; dimensionality reduction; feature extraction; pattern recognition; machine learning mutual information; dimensionality reduction; feature extraction; pattern recognition; machine learning

Figure 1

This is an open access article distributed under the Creative Commons Attribution License (CC BY 3.0).

Scifeed alert for new publications

Never miss any articles matching your research from any publisher
  • Get alerts for new papers matching your research
  • Find out the new papers from selected authors
  • Updated daily for 49'000+ journals and 6000+ publishers
  • Define your Scifeed now

SciFeed Share & Cite This Article

MDPI and ACS Style

Liu, R.; Gillies, D.F. An Estimate of Mutual Information that Permits Closed-Form Optimisation. Entropy 2013, 15, 1690-1704.

Show more citation formats Show less citations formats

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top