*Article* **Entropy, Information, and the Updating of Probabilities**

**Ariel Caticha**

Physics Department, University at Albany-SUNY, Albany, NY 12222, USA; acaticha@albany.edu; Tel.: +1-(518)-442-4592

**Abstract:** This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as a unique tool for updating (a) that is of universal applicability, (b) that recognizes the value of prior information, and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework—the ME method—can handle arbitrary priors and arbitrary constraints. It includes the MaxEnt and Bayes' rules as special cases and, therefore, unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, and also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.

**Keywords:** maximum entropy; Bayesian inference; updating probabilities

## **Citation:** Caticha, A. Entropy, Information, and the Updating of Probabilities. *Entropy* **2021**, *23*, 895. https://doi.org/10.3390/e23070895

Academic Editor: Petr Jizba and Jan Korbel

Received: 17 June 2021 Accepted: 10 July 2021 Published: 14 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).
