Received: 8 November 2012 / Revised: 15 February 2013 / Accepted: 19 February 2013 / Published: 25 February 2013

Cited by 5 | PDF Full-text (637 KB) | HTML Full-text | XML Full-text
**Abstract**

The Minimum Mutual Information (MinMI) Principle provides the least committed, maximum-joint-entropy (ME) inferential law that is compatible with prescribed marginal distributions and empirical cross constraints. Here, we estimate MI bounds (the MinMI values) generated by constraining sets

[...] Read more.
**T***comprehended by*_{cr}*m*_{cr}
The Minimum Mutual Information (MinMI) Principle provides the least committed, maximum-joint-entropy (ME) inferential law that is compatible with prescribed marginal distributions and empirical cross constraints. Here, we estimate MI bounds (the MinMI values) generated by constraining sets

**T***comprehended by*_{cr}*m*linear and/or nonlinear joint expectations, computed from samples of_{cr}*N iid*outcomes. Marginals (and their entropy) are imposed by single morphisms of the original random variables.*N*-asymptotic formulas are given both for the distribution of cross expectation’s estimation errors, the MinMI estimation bias, its variance and distribution. A growing**T***leads to an increasing MinMI, converging eventually to the total MI. Under*_{cr}*N*-sized samples, the MinMI increment relative to two encapsulated sets**T***⊂*_{cr1}*T*(with numbers of constraints mcr1<mcr2 ) is the test-difference_{cr2}*δH*=*H*_{max 1, N}-*H*_{max 2, N}≥ 0 between the two respective estimated MEs. Asymptotically,*δH*follows a Chi-Squared distribution^{1}/_{2N}*Χ*2 (m_{cr2-}m_{cr1}) whose upper quantiles determine if constraints in**T**_{cr2}/**T***explain significant extra MI. As an example, we have set marginals to being normally distributed (Gaussian) and have built a sequence of MI bounds, associated to successive non-linear correlations due to joint non-Gaussianity. Noting that in real-world situations available sample sizes can be rather low, the relationship between MinMI bias, probability density over-fitting and outliers is put in evidence for under-sampled data. Full article*_{cr1}
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)