A Hyperbolic Sum Rule for Probability: Solving Recursive (“Chicken and Egg”) Problems

Abstract

We prove that the probability of “A or B”, denoted as p(A or B), where A and B are events or hypotheses that may be recursively dependent, is given by a “Hyperbolic Sum Rule” (HSR), which is relationally isomorphic to the hyperbolic tangent double-angle formula. We also prove that this HSR is Maximum Entropy (MaxEnt). Since this recursive dependency is commutative, it maintains the symmetry between the two events, while the recursiveness also represents temporal symmetry within the logical structure of the HSR. The possibility of recursive probabilities is excluded by the “Conventional Sum Rule” (CSR), which we have also proved to be MaxEnt (with lower entropy than the HSR due to its narrower domain of applicability). The concatenation property of the HSR is exploited to enable analytical, consistent, and scalable calculations for multiple hypotheses. Although they are intrinsic to current artificial intelligence and machine learning applications, such calculations are not conveniently available for the CSR, moreover they are presently considered intractable for analytical study and methodological validation. Where, for two hypotheses, we have p(A|B) > 0 and p(B|A) > 0 together (where “A|B” means “A given B”), we show that either {A,B} is independent or {A,B} is recursively dependent. In general, recursive relations cannot be ruled out: the HSR should be used by default. Because the HSR is isomorphic to other physical quantities, including those of certain components that are important for digital signal processing, we also show that it is as reasonable to state that “probability is physical” as it is to state that “information is physical” (which is now recognised as a truism of communications network engineering); probability is not merely a mathematical construct. We relate this treatment to the physics of Quantitative Geometrical Thermodynamics, which is defined in complex hyperbolic (Minkowski) spacetime.

1. Introduction I: Probabilities

1.1. Overview

Surely how to sum probabilities has been known for generations? The formula for “p(A or B)” (“the probability of either or both instances of {A,B} happening”, where “or” is the logico-probabilistic operator: see Equation (2b), below), has been well known since John Venn published his textbook in 1881; the formula was known much earlier (as mentioned by Leonhard Euler in 1768, for instance). Why should we want to replace or improve upon the canonical “Conventional Sum Rule” (CSR) expression?

The problem is that the CSR is not valid where the probabilities of the events of A and B are recursively dependent on each other, or where their causal priority (temporal or logical) is indeterminate. However, the artificial intelligence (AI) and machine learning (ML) applications widely employed today regularly address complex problems involving multiple hypotheses with multiple conditionalities, including recursive ones. Thus, more general methods are needed.

It turns out that the CSR is a special case; using Bayesian methods to derive the general case results in a “Hyperbolic Sum Rule” (HSR). We start from the seminal work of Richard Cox (1946 [1]) and Edwin Jaynes (2003 [2], 1982 [3]), presenting the HSR Theorem in Section 3.1 (Equation (12)), and further summarise the technical relations in Section 3.3 (as proved in the Appendix A).

Bayesian methods make explicit the physicality of probability, in particular, the fact that agents are always involved where unbiassed estimates of the probabilities of events or hypotheses are required. Then, the value of “Maximum Entropy” (MaxEnt) techniques becomes evident. We prove that the HSR is MaxEnt (in Appendix A.3) using standard Lagrange multiplier methods (see, for example, Caticha 2008 [4]).

What sense does it make to say that a probability sum rule is “MaxEnt”? What rôle could thermodynamics be thought to play in calculating probabilities (for entropy is a thermodynamic quantity)? It is usually thought that thermodynamics is primarily a way of handling statistical ensembles, but that this is not the case can be seen from the success of “Quantitative Geometrical Thermodynamics” (QGT) in calculating geometrical parameters of nuclear, molecular and cosmic structures (DNA and the Milky Way: Parker & Jeynes 2019 [5]; nuclear dimensions of helium isotopes: Parker et al., 2022 [6]). Note also that QGT is defined in a hyperbolic space [5], underlining the explicit link with a hyperbolic sum rule for probabilities.

The point is that the probability of a given event can always be thought to be drawn from a probability distribution, and to obtain an unbiassed estimate of this probability one uses a Maximum Entropy distribution. But to specify a distribution, one gives its functional form so that one can ascribe an entropy to the function. To say that the HSR is MaxEnt is to say that the resulting probabilities are the best (the most unbiassed) estimates available, given the current knowledge. It is important to point out that the mathematical objects resulting from the HSR described here are still regular probabilities, with real values between 0 and 1 (inclusive), provided that the value inputs into the formulae are also regular probabilities. This proviso is, of course, the same for both the CSR and the product rule (Bayes’ Theorem).

The motivation for the HSR is that the artificial intelligence (AI) and machine learning (ML) applications widely employed today regularly use Bayesian methods to address complex problems involving multiple hypotheses with multiple conditionalities. We elaborate on this point briefly in Section 4.4. Of course, these problems do not exclude recursive relations. But the CSR is not valid in the presence of recursion, and therefore, such problems need more general analytical methods. Moreover, separately, digital signal processing (DSP) methods are formally closely related to these new methods in probability (see the further discussion in Section 4.3).

The paper is extensive because it introduces a generalised (and far-reaching) sum rule for probabilities, and is structured as follows: In Section 1, we outline the very extensive prior work relating to how we think about probabilities, and in Section 2, we do the same for the Conventional Sum Rule. We discuss the limitations of the Venn diagram in Section 2.2 and informally derive the HSR in Section 2.3 (Equation (8)) before formally deriving it in Section 3.1 (Equation (12)). This informal approach is presented because a physical appreciation of this far-reaching result is needed to promote the DSP and AI applications we have in mind, as indicated in Section 4. Finally, we summarise (Section 5) and conclude (Section 6).

1.2. Probability in Science

Probability has always been extraordinarily difficult to tie down. Edwin Jaynes made fundamental contributions to the subject, and in his monograph (“Probability Theory: The Logic of Science” [2]) spends two chapters on its principles and elementary applications, commenting on its ‘weird’ and ‘complicated’ history. Indeed, Jaynes’ motivation was aimed at helping the interested reader who already has “a previous acquaintance with probability and statistics” to essentially “unlearn” much of what they may have previously learned!

The ubiquity (and longevity) of fallacies in the use of statistics indicates its difficulty (on the misuse of the “p-value” see, for example, Goodman 2008 [7] and Halsey 2019 [8]; and on the persistence of fallacies see Smaldino and McElreath 2016 [9]). A significant part of the problem may be related to the fact that the “probability” of an event is apparently not a property solely of “external reality”: since there must be someone assessing the probability, it must be a function of not only what information already exists about the event but also who knows it. The fact that our estimates of the probability of some event invariably involve our prior knowledge, combined with the fact that all knowledge ultimately involves the properties of recursive statements (by Gödel’s Theorem [10]; this point has recently been elaborated by Jeynes et al. [11], and Jeynes and Parker [12]) mean that some part of this difficulty must be due to the neglecting, in current (simplified) treatments, of recursive dependencies (such as the “Chicken and Egg” problem, on which see Section 2.3 below).

In 1946, R.T.Cox observed, acutely, that the concept of probability mixes two separate concepts (“the idea of frequency in an ensemble and the idea of reasonable expectation” [1]), which are now represented by two schools usually called the “frequentists” and the “Bayesians”. We point out that Enßlin et al. [13] comment that, “The different views on probabilities of frequentists and Bayesians are addressed by … Allen Caldwell [14] [who] builds a bridge between these antagonistic camps by explaining frequentists’ constructions in a Bayesian language”, and we will assume that Caldwell is right and therefore that we can ignore the philosophical distinction between the frequentists and the Bayesians.

Building on Cox’s fundamental work we will here derive a rigorous treatment of recursive probability which is of general applicability. In particular, we will treat probability as a physical quantity grounded in hyperbolic Minkowski spacetime, and obeying all the appropriate physical laws. The close relation of probability to the new Quantitative Geometrical Thermodynamics [5] (QGT, which constructs the quantity info-entropy by treating the bases of information and entropy as Hodge duals) is due to the fact that the (hyperbolic) entropic velocity q′ ≡ dq/dx in QGT is dimensionless and has properties akin to a probability (0 ≤ q′ ≤ 1); noting that q′ is isomorphic to the more familiar kinematic velocity $\stackrel{ .}{x}$ ≡ dx/dt (0 ≤ $\stackrel{ .}{x}$ ≤ c, where c is the speed of light) in hyperbolic (Minkowski) spacetime (see [5]), and which also obeys the well-known velocity addition theorem of Special Relativity (see Equation (1b)).

The concept of probability as a physical quantity (and obeying physical laws) is as fundamental as the well-known statement by Rolf Landauer that “Information is Physical” [15].

We are used to treating both probability and information anthropomorphically (that is, depending on what you or I might be expected to know): here we will establish an impersonal sense of probability, in the same way that Landauer insisted on the (impersonal) Shannon entropy sense of his Information. Note that each of information and entropy both use and require probabilistic quantities in their fundamental definitions; and also note that the consequences of treating information as being a quantity as physical as is energy (for example) has led to important insights, and not only into basic engineering problems of the global internet (for an example of which see Parker et al., 2015 [16]). We expect similar advances to follow from recognising probability as also being an equally physical quantity.

The relationship between probability and the quantification of information using the Shannon entropy is well understood mathematically, but its interpretation as a physical theory has been convoluted. Although the Shannon entropy was quickly recognised as important, Edwin Jaynes’ formulation of Maximum Entropy (MaxEnt) theory, where the Shannon metric plays a key role, was initially controversial and took some decades to achieve acceptance (see Jaynes’ 1978 summary [17]). However, MaxEnt as a powerful scientific and engineering tool has helped considerably to underpin the physicality of information, and therefore also acts as a support to the underlying assertion of this paper (paraphrasing Landauer) that “Probability is Physical”. We will also explore the implications of this assertion.

Since the concept of MaxEnt will be centrally important in this work, it is worth adding Jaynes’ authoritative definition (1982) [3]: “The MaxEnt principle, stated most briefly, is: when we make inferences based on incomplete information, we should draw them from that probability distribution that has the maximum entropy permitted by the information we do have”. Note that distributions are described by functions, and therefore functions can also be said to be “MaxEnt”. We emphasise this since QGT applies specifically also to small systems (see, for example, Parker et al., 2022, [6]) for which statistical mechanics methods cannot apply. It is worth mentioning that this ambiguity between “distributions” and “functions” is analogous to that between “frequentists” and “Bayesians”. The same thing may be described in various ways, and different things may have the same mathematical representation.

1.3. Probability Is Physical

Torsten Enßlin’s treatment [18] of information as a field is interesting in this context: he considers that a “physical field has an infinite number of degrees of freedom since it has a field value at each location of a continuous space” where he is specifically considering imaging problems of astrophysical datasets. Enßlin et al. [13] treat information informally as an anthropomorphic concept: to the question, “What is information?” they answer, “Anything that changes our minds and states of thinking!” But here, we will treat information (and probability) as physical quantities, not as anthropomorphic concepts; and especially noting that infinite quantities are inimical to physical phenomena. In particular, in our QGT treatment, information is formally defined as a physical quantity (albeit in terms making full use of the properties of analytical continuation, which is itself closely related to fundamental physical precepts such as causality [19] and square-integrability [20] ensuring finite physical quantities [21]), so that the number of degrees of freedom is finite and may be very small, as is observed for the geometrical entropy of isotopes of the helium nucleus [6]. Such results for information were already pointed out by Parker and Walker (2004) [22] who investigated the residues of a meromorphic function (that is, a function analytic nearly everywhere) due to the presence of isolated singularities in the complex plane (singularities which are entirely analogous to particles in their behaviour); and who show that the information (entropy) of such a function is simply given by the sum of the residues.

This is immediately applicable to the Schrödinger equation: it is interesting that we will conclude (Equation (8)) that the appropriate Sum Rule for recursive probabilities has a hyperbolic form, and we will draw out the relation of this to the (hyperbolic) entropic velocities in the QGT formalism [23], in which the entropic Uncertainty Principle and entropic isomorphs of the Schrödinger equation may be derived from the entropic Liouville Theorem; all based on the Boltzmann constant as the relevant quantum of entropy. Of course, QGT is constructed in a hyperbolic (complex) Minkowski 4-space (see §18.4 in Roger Penrose’s “Road to Reality” [24]; as another pertinent example, Maxwell’s electro-magnetic field is a hyperbolic version of the Cauchy-Riemann Equations: see Courant and Hilbert [25] vol.II ch.III §2 Equation (8) passim).

The ramifications of this are very wide. Parker and Jeynes [5] have already shown the relevance of QGT to the stability and structure of spiral galaxies (using the properties of black holes), and also that entropy production (Π ≡ d$\mathcal{S}$/dt) is conserved even in relativistic Maximum Entropy systems [26]. These issues up to now have been treated as problems in quantum gravity, and Matt Visser [27] very helpfully reviews conservative entropic forces (in Euclidean space and non-relativistically, although he comments that representing general relativity entropically should be possible). Note also that Visser suggests that the negative entropies that appear in his treatment can be regarded as information, citing Brillouin’s idea of “negentropy” which Parker and Jeynes [5] have shown to be a subtle misapprehension of the relation between information and entropy (which are actually Hodge duals).

We believe that much progress may be made by using the coherent formalism of QGT (defined in hyperbolic space) which has been shown to apply to both quantum mechanical and gravitational systems (that is, at all scales from sub-atomic to cosmic: see [5,6]), and since quantum mechanics is built on probabilities, the demonstration here that the general recursive sum rule for probabilities is hyperbolic is a significant conceptual regularisation. This conclusion is reinforced by Knuth’s demonstration [28] that: “The sum and product rules, which are familiar from, but not unique to, probability theory, arise from the fact that logical statements form a distributive (Boolean) lattice, which exhibits the requisite symmetries”. Moreover, Jaeger [29] reviews a variety of treatments, some of which involve theories of generalised probability, aimed at deriving quantum mechanics from information theory.

The basic isomorphism for the hyperbolic sum rule for probabilities (that we will prove, see Equation (12)) is the (purely mathematical) double-angle identity for the hyperbolic tangent function:

tanh(a + b) ≡ {tanha + tanhb}/{1 + tanha·tanhb}

(1a)

Another interesting and very simple isomorphism is the well-known relativistic sum rule for velocities {u, v}, given by Jackson in his well-known textbook (in the context of a discussion of aberration and the Fizeau experiment; §11.4, Eq.11.28, [30]):

w ≡ Sum(u, v)
w/c = {u/c + v/c}/{1 + uv/c²}

(1b)

where c is the speed of light. Jackson comments that if u = c then also w = c, which is an “explicit statement of Einstein’s second postulate” (c is a constant).

This latter (Equation (1b)) is clearly physical (since c is involved) where the former (Equation (1a)) is a mathematical identity. Note also that in optics the basic formula for the two-layer Fabry–Perot cavity (etalon) is well-known [31]:

r₃ = {r₂ + r₁Φ}/{1 + r₂r₁Φ}
Φ ≡ exp(−2ikΔz)

(1c)

where the overall scattering (reflectivity) coefficient r₃ is due to a pair of sequential Fresnel reflections (r₁, r₂) separated by a distance Δz for a light ray of propagation constant k; and where we note that light is the physical phenomenon par excellence exhibiting the physics of Special Relativity within the context of hyperbolic (Minkowski) spacetime. Corzine et al. [32] demonstrate that this formula is closely related to the hyperbolic addition rule (Equation (1a)) specifically by using a hyperbolic tangent substitution, which dramatically simplifies the use of the formula in real (multilayer) cases—see Section 2.2 for additional and related discussion.

This approach has recently been supported in an interesting way by Skilling and Knuth [33], who conclude: “it must be acknowledged, quantum theory works. So does probability. And the two are entirely mutually consistent.” Their argument shows logical reasons why probability should be regarded as physical.

Skilling and Knuth claim to not be interested (for these purposes) in the distinction between ontology and epistemology. They say ([33], §3.5):

The ontology–epistemology divide is, for quantitation at least, a distinction without a difference. A bit of information carries no flag to inform us whether it was assigned by a conscious agent or by a mechanical switch. Our job in science is to make sense of our observations, not to indulge in empty disputation between isomorphic views. Our goal here is derivation of a calculus fit for general purpose. Ontology and epistemology share the same symmetries, the same rules, and the same assignments. So, they share a common calculus.

which is suggestive of Karen Barad’s (2007 [34]) insistence that the distinction between ontology and epistemology is not a real one, and therefore, speaking strictly we should refer to “onto-epistemology” ([34], p. 43). Skilling and Knuth also say this ([33], §1).

But, if our object can perturb a partner object, then by symmetry the partner object can also perturb our object. We could assign either role to either.

Our calculus, whatever it is, must be capable of representing such interactions … This insight that interactions are basic is the source of “quantum-ness”.

Again, this recalls Barad’s thesis that “the primary ontological unit is the phenomenon” ([34], p. 333). However, when Skilling and Knuth ([33], §4) say, “We start with an identifiable object”, this is directly contradicted by Barad, who asserts that “objects” do not have “an inherent ontological separability” ([34], p. 340); that is, strictly speaking, identifiable objects do not actually exist per se (since everything is entangled with everything else). But Skilling and Knuth are not aiming at philosophical precision, only at a demonstrable computability; for these purposes such fine distinctions do not matter. They are right to avoid metaphysical considerations in scientific work: although when wider social implications are important it may be necessary to consider the metaphysics (see, for example, [11,12]).

However, it turns out that the inescapable human dimension appears to be especially pronounced in probability, in the sense that the very idea of a probability entails one’s personal state of knowledge or ignorance (and Michael Polanyi insisted long ago that all knowledge is necessarily personal [35]). Howson and Urbach [36] have carefully explained why, although Bayesian (and Maximum Entropy) methods are fully (and helpfully) rational: “there seems to be no way of ‘objectively’ defining prior probabilities … this is really no weakness [since] it allows expert opinion due weight, and is a candid admission of the personal element which is there in all scientific work”. Assessing probabilities necessarily entails assessing uncertainties, and this must always involve some value judgments: although we may do our best to speak rationally about such judgments, it cannot be excluded that different people will (rationally) come to different conclusions.

Note that although scientists have a duty to argue rationally, non-scientists also normally behave rationally. Rationality is a property of humans, and only a rather small subset of humans are scientists.

1.4. Maximum Entropy

It is necessary to make a few initial simple remarks about Maximum Entropy (MaxEnt) methods to clarify the discussion. Jaynes said in 1957 [37]: “The guiding principle is that the probability distribution over microscopic states which has maximum entropy subject to whatever is known, provides the most unbiased representation of our knowledge of the state of the system. The maximum-entropy distribution is the broadest one compatible with the given information; it assigns positive weight to every possibility that is not ruled out by the initial data.” In our derivation here of the Hyperbolic Sum Rule we emphasise that the proper application of MaxEnt methods precludes the surreptitious introduction of tacit assumptions (“knowledge”). That is, all the “priors” (prior knowledge that conditions the data) must be stated explicitly (and our ignorance implies that some priors must sometimes be estimated, which may involve personal value judgments).

A fully Bayesian analysis requires that all prior knowledge is explicitly stated, including the “knowledge” that there is no knowledge available, in which case an “unbiassed” estimate is required. This is usually stated in terms of the “Principle of Indifference” (PI), but unfortunately, there is a set of (“Bertrand”) paradoxes which appear to invalidate the PI in some circumstances. But Parker and Jeynes [38] have resolved these paradoxes using QGT methods by supplying the missing prior information (in the form of the scale invariance condition).

In the Appendix A, we explore the Maximum Entropy (MaxEnt) properties of the HSR, including proof that it really is MaxEnt. We first show how to impose the MaxEnt criterion (Appendix A.1) using the Partition Function description, and then show (Appendix A.2) how other simple sum rules may be “MaxEnt” but are inadmissible since they do not always result in formal probabilities. We prove explicitly that the HSR is MaxEnt (Appendix A.3) and also generalise it for multiple recursive hypotheses (Appendix A.4). We show that the Conventional Sum Rule (CSR) is also MaxEnt (Appendix A.5) within its own domain of applicability, and we also generalise the CSR for multiple hypotheses. Both are MaxEnt, that is, neither the HSR nor the CSR ‘smuggle’ in inadvertent or implicit assumptions within their specified contexts; yet, the HSR encompasses a wider domain of physical application that includes the possibility of recursion between phenomena, whereas the CSR a priori excludes recursion (although there may still be various mutual dependencies). The HSR must, therefore, be used where the properties of the recursion are unknown; that is to say, a set of phenomena are known to be correlated, but the mechanism (or ordering) of causation is not known. The entropy of the HSR is shown to be, in general, higher than that of the CSR (Appendix A.5.3). Therefore, in a Bayesian analysis, the HSR contains fewer implicit assumptions or constraints than the CSR–this being a consequence of the HSR also allowing the possibility of recursion, whereas the CSR a priori excludes recursion.

Finally, in Appendix A.6, we show the immediate relevance of this treatment to digital signal processing, in particular, the handling of so-called “infinite impulse response” and “finite impulse response” filters.

It is important here to point out that the fact that an entity is Maximum Entropy does not mean that the entity has no structure (even though MaxEnt necessarily implies “maximum ignorance”). The reality is more nuanced. For example, we have shown that, at a certain scale, the alpha particle is a (MaxEnt) unitary entity (than which exists no simpler) [6]. But of course, at a different scale we may see the alpha’s constituent four nucleons (protons and neutrons).

Also, being MaxEnt does not preclude change: for example, free neutrons decay (with a half-life calculated ab initio from QGT by Parker and Jeynes 2023 [39]). The most extreme known entity that is MaxEnt is the black hole, and black holes necessarily grow (proved by Parker & Jeynes in 2021 [26] and confirmed in 2023 [40]). Nor does the fact that some entities are unconditionally stable mean that the Second Law does not apply to them. On the contrary! The matter radius of the alpha particle is correctly calculated ab initio from QGT [6].

It might be thought that MaxEnt applies only to distributions, but this is a misconception based on the idea that thermodynamic explanations are always statistical (requiring large systems). But Parker and Jeynes (2019) [5] have shown explicitly that small systems can be analysed with QGT methods, functionalising the system entropy and defining the entropic Lagrangian (whose integration over space is called the “Exertion”, isomorphic to the kinematic Action). The appropriate Euler–Lagrange equations exist to minimise the Exertion, such that the Principle of Least Exertion then applies (isomorphic to the ubiquitous Principle of Least Action). The entropic Hamiltonian is obtained from the entropic Lagrangian via the appropriate Legendre transformation, and whose integration over space yields the system entropy, which in turn is maximised via the same Euler–Lagrange equations, yielding a MaxEnt function.

The physical principle of Maximum Entropy embodies the (physical) Second Law of Thermodynamics. Both the CSR and the HSR are MaxEnt in their respective domains, and therefore, each also embodies important aspects of the Second Law. Moreover, the probabilities calculated by either Rule (each with its own particular domain of applicability) refer to the probability either of the events themselves or of our reasonable expectations; these are both physical things since both we and they are physical.

2. Introduction II: Sum Rules

2.1. The Conventional Sum Rule (I)

Consider p(A) which is a representation of either “the probability of event A occurring” or, nearly equivalently (logically speaking), “the probability of hypothesis A being true”. Of course, hypotheses and events are entirely different things, but for convenience, we will here usually speak of “events” without any loss of generality (ignoring the philosophical frequentist/Bayesian distinction, see Section 1.2).

The probability that A or B will happen is conventionally given (by the “Conventional Sum Rule”, CSR) as the sum of the separate probabilities that A and B, respectively, will each happen, minus the probability that both of {A,B} will happen together, (in general, {A,B} are also not independent of each other):

p(A or B) = p(A) + p(B) − p(A and B)

(2a)

This is simply extended to the conditional case treating the probability of A or B given some conditionality C:

p(A or B|C) = p(A|C) + p(B|C) − p(AB|C)

(2b)

where A and B are two events, conditional either on each other or some other event(s); in other words, in this case C represents all the possible conditionalities, including the mutual conditionalities of {A,B}; that is, p(A|B), p(B|A) (“the probability of A given B”, and conversely). Equation (2b) represents the CSR. But if {A,B} are recursively dependent (the “chicken and egg” scenario) Equation (2b) is invalid since C either cannot be specified explicitly or cannot be specified at all.

Intrinsic to Equation (2a,b) is also the corresponding “Product Rule”, more commonly known as Bayes’ Theorem:

p(A and B|C) ≡ p(AB|C)
             = p(A|BC) p(B|C)
           = p(B|AC) p(A|C)

(2c)

Here, Bayes’ Theorem appears in the latter two expressions of Equation (2c), which yield the same result since Bayes’ Theorem is commutative: p(AB) = p(BA). In particular, Bayes’ Theorem represents a choice as to which of the possible conditionalities should take priority. The logical AND function can be calculated in different ways according to which of the conditionalities is considered to be prior.

We emphasise: although we seek here an alternative form of the Sum Rule (Equation (2b)) that takes recursion into account, Bayes’ Theorem (Equation (2c)) remains valid.

Such recursive issues arise when we start considering which of the conditionalities is prior (an issue of causation): there appears to be a temporal dependence between the two conditionalities since one must surely occur earlier in time than the other. But note that Special Relativity teaches us that if two spatially distinct events are causally connected (that is, they lie within each other’s light cone) then temporal priority is always preserved whatever the frame of reference (FoR). However, if the events are not causally connected in spacetime, then for any FoR where the event of A occurs before that of B, another FoR exists where event B occurs before A. If causal connectedness cannot be determined, either due to the events being spacelike separated, or perhaps because no physical mechanism can be identified to causally connect the events, then there is an ambiguity as to which of the events is prior (and of course, correlation does not imply causation).

An example of a recursive system is the multilayer etalon represented by Equation (1c).

2.2. The Venn Diagram

Jaynes showed that the derivation of the product rule (Equation (2c)) from the basic laws of Boolean logic is straightforward [2]. However, performing the same derivation for the “Sum Rule” of probability is less obvious. In particular, it is worth noting that Jaynes specifically avoids the use of the Venn diagram as a means to justify Equation (2b), preferring to interpret propositions in terms not of sets but as probability distributions carrying incomplete information. It is useful to note in this context that there is intense current interest in how to handle incomplete (or erroneous) “information” in a formal Bayesian analysis (see, for example, Zhang et al. [41]).

Jaynes therefore calls the Venn diagram a ‘useful device’ but comments that it can ‘mislead’ [our emphasis]. The Venn diagram is good for representing binary logical operations (see Figure 1) but, in general, correctly represents neither temporality (including aspects of causality or recursiveness) nor probability distributions. We add that it is also not obvious how the Venn diagram (or the CSR of Equation (2b)) could be usefully extended to include multiple hypotheses and/or conditionalities.

Jaynes’ derivation [2] of the “Product Rule” (Equation (2c)) does not need to involve de Morgan’s theorems (which follow from the rules of Boolean logic). But his derivation of the “Sum Rule” (Equation (2b)) adopts the original schema of Cox [1] to also define an auxiliary function S so that v ≡ S(u), where u ≡ p(A|C) such that v ≡ p($\overline{A}$|C) and shows that

S[S(x)] = x for 0 ≤ x ≤ 1

(3a)

that is, the function S(x) is involutory (self-reciprocal):

S(x) = S⁻¹(x)

(3b)

Using this reciprocal property of S to calculate inverse probabilities avoids the problem that there is no quotient operator in Boolean logic. In particular, for completeness we note that Cox employed the following particular function for S in his analysis:

S(x) = 1 − x

(3c)

We can extend this treatment by generalising Cox’s function S to the function Σ:

$$\mathsf{\Sigma }\left(x\right)={\displaystyle \frac{1-x}{1+x}}$$

(4a)

and here we point out some interesting properties of this new function: $\mathsf{\Sigma }\left[\mathsf{\Sigma }\left(x\right)\right]=x$; that is, Σ is involutive, $\mathsf{\Sigma }\left(x\right)={\mathsf{\Sigma }}^{-1}\left(x\right)$. It is also doubly differentiable and the derivative is a continuous, monotonically decreasing function: $\partial \mathsf{\Sigma }/\partial x=-2/{\left(1+x\right)}^2$ for real x, and 0 ≤ x ≤ 1. The inspiration for Equation (4a) comes from Cox’s original more general equation for S as another involutory function (see Equation (3) above) which is also doubly differentiable. In Cox’s case, he also considered the situations where S (and x) are raised by an integer power m, such that S^m(x) = 1 − x^m for integer m; which also conform to his mathematical requirements and Equation (3a,b). For even m > 0 this can be factored into the geometric series:

(1 − x^m) ≡ (1 + x)(1 − x + x² − x³ + … − x^m−1)

(4b)

with the series in parentheses summing to the quantity (1 + x)⁻¹ as m→∞. Note that this treatment is quite general, such that x may be complex, although in this paper we only assume probabilities described by real x, and 0 ≤ x ≤ 1. However, for all m (with integer m ≥ 1), we can also write:

(1 − x^m) ≡ (1 − x)(1 + x + x² + x³ + … + x^m−1)

(4c)

Recognising that for even m the roots of S^m lie symmetrically around the four quadrants of the unit circle centred on zero in the complex plane, allows us to additionally exploit the associated symmetry relation for the nth root, x_n = −x_m/2+n. It is important to emphasise that although results from complex analysis underpin this analysis, in the sum rules discussed in this paper we only ever assume (and employ) probability theoretical values for x where x is real and 0 ≤ x ≤ 1. Therefore, substituting x_n = −x_m/2+n for the m − 1 terms of the series summation in Equation (4c), we can write:

(1 − x^m) ≡ (1 − x)(1 − x + x² − x³ + … − x^m−1)

(4d)

Applying this (counterintuitive) symmetry we see that Equation (4a) is simply the limit of Equation (4d) as m→∞. In this limit the distinction between even and odd m is insignificant. In any case, it is clear that although Cox’s first solution (using m = 1: the quantity S = 1 − x, see Equation (3c)) represents an important term in the factorisation of Σ, it is by no means the only available solution in the general case for the roots of the function Σ.

Note that although the intermediate Equation (4b,c,d) (and the implied complex analysis) are not used in the formal derivation of the HSR (Section 3.1 and Equation (12)), analogous complex quantities do make an appearance in the discussion of how the finite impulse response (FIR) filter offers a good exemplar of the CSR (sketched in Appendix A.6). The standard theory of such filters (which are important for digital signal processing applications) fundamentally relies on the theorems of complex analysis.

2.3. Recursion: Chicken and Egg

Dependencies can be logical or temporal. Causality (a temporal dependence) also implies the possibility of recursion: this is clear from the “Chicken and Egg” conundrum, on which see Simmons et al. [42] (who consider whether hummingbirds pollinate lots of species because there are lots of hummingbirds or whether there are lots of hummingbirds because there are lots of species to pollinate) or Harvey et al. [43] (who consider two forms of the recursive cyclic network to show how subtle recursion relations may be). The chicken entails a prior egg, which entails a prior chicken…

Formally: both p(A|B) > 0 and p(B|A) > 0 may be true together either if {A,B} are independent or if {A,B} are recursively dependent. However, recursion is not present if C in Equation (2b)—as in p(A|C)—is independent of either A or B; that is, if p(C|A or B) = p(C). Such expressions in Equation (2b) as p(A|C) imply that all dependencies of A are subsumed in C; that is, if A depends on B somehow, then there (usually) exists a C that will say this with little loss of generality (Hofer-Szabó et al. [44] show that Reichenbach’s “common cause” does not necessarily exist).

Here, we will informally derive the “Hyperbolic Sum Rule” that we will prove formally in Section 3.1.

There is an apparent dilemma (when applying Bayes’ Theorem) of having to make what appears to be an arbitrary decision as to which conditionality is the prior. Indeed, when considering the classic “Chicken and Egg” dilemma, having argued in one way, we undergo a Gestalt change and suddenly find ourselves equally arguing in the opposite manner! The recursiveness of this dilemma is self-evident; a different prior may be appropriate depending on where the analysis commences. Curiously, the case of the two-layer Fabry–Perot cavity (Equation (1c)) involves optical events that lie on each other’s light cone; that is to say, as a physical phenomenon, it is directly on the boundary of being causally connected or disconnected. In this case, an intrinsic ambiguity of priority exists. If either may be prior then neither must be.

That is, for a recursive dependency of {A,B}, p(A|B) = p(A) and by symmetry, we also must have p(B|A) = p(B). This is a surprising result since {A,B} being independent also entails p(A|B) = p(A): that is to say, the condition “p(A|B) = p(A)” may entail either “{A,B} are independent” or “{A,B} are recursively dependent”.

In any case, issues of the probability of events and their priority are frequently set within a spacetime context, with the associated physical implications. One of the aims of this paper is to establish a solution to this dilemma of choice that conforms to Special Relativity but is still recognisably consistent with conventional probability theory.

Note that even though both expressions p(A or B|C) and p(A and B|C) commute for A and B, Bayes’ Theorem does distinguish the precedence of the conditionalities associated with the primary events.

Thus, Equation (2b) can be expressed in one of two ways, according to which of the primary events A and B is prior: substituting Equation (2c) into Equation (2b), we find two choices offering two possibilities for the logical OR function:

p(A or B|C) = p(A|C) + p(B|C) − p(A|BC) p(B|C)
       = p(A|C) + p(B|C) − p(B|AC) p(A|C)

(2d)

Equation (2d) may be simplified by noting that the component term of Bayes’ Theorem appearing at the end of each CSR expression (each such latter term representing a conditional possibility) can also be assigned a probability associated with the truth of that conditionality. This allows us to informally re-write Equation (2d) as a single statement, by expanding upon the products from Equation (2c):

p(A or B|C) = p(A|C) + p(B|C) − {[p(A|BC) p(B|C)]_b + [p(B|AC) p(A|C)]_a}

(5a)

where the subscripts {a, b} appearing after each set of square bracket expressions informally indicate the assumed prior for each square bracket expression. The expression in each of the pair of square brackets represents the respective Bayes’ Theorem choice. Equation (5a) can be regularised by including the probability associated with the prior, that is, now explicitly including a recursive element. That is to say, in interpreting the subscripts {a,b} of Equation (5a), say ‘a’ (on the RHS) then this (informally) indicates the probability associated with the event A being true and having occurred. So, one might perhaps, in the first instance, simply want to multiply again by the probability of A, p(A|C). However, we are entertaining here the two ideas of conditionality and recursiveness; in which case employing the conditional probability p(A|BC), instead of p(A|C), recognises the ongoing conditionality of A on B, so that the inclusion of p(A|BC) therefore also builds in a desired recursive aspect into Equation (5a). The same reasoning is used mutatis mutandis for the ‘b’ subscript.

Note that Equation (5a) requires that {A,B} are distinct, that is, the possibility A = B is excluded. (But note that Equation (8) is also valid for the case A = B. This is proved formally: see Equation (12).)

Note also that the probability p(A|BC) associated with the prior is not the same as the probability of the prior, which we may state informally as: p(“A is prior to B”). Thus, we explicitly include the probabilities associated with the priors (removing the subscripts {a,b}). In effect, we now assume the possibility of a ‘prior prior’ and the (logical/temporal) ordering of the events {A,B} now also starts to be explicitly taken into account:

p(A or B|C) = p(A|C) + p(B|C) − {[p(A|BC) p(B|C)] p(B|AC) + [p(B|AC) p(A|C)] p(A|BC)}

(5b)

allowing a consistent (and symmetric) simplification which intrinsically includes the possibility of recursion:

p(A or B|C) = {p(A|C) + p(B|C)} {1 − p(A|BC) p(B|AC)}

(5c)

In Equation (5c) we have the probability of “A or B given C“ being {[the probability of A given C] or [the probability of B given C]} reduced by a cross-probability factor F, where

F ≡ −p(A|BC) p(B|AC)

(6a)

which could be thought to represent a “first level” of dependency, that is, the dependence of A on B (and vice versa). But what if we also consider a deeper dependence between these events? This would give the following:

p(A or B|C) = {p(A|C) + p(B|C)}{1 + F + F² + …)}

(6b)

or, continuing until dependence becomes recursion, as follows:

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)=\left\{p\left(\left.A\right|C\right)+p\left(\left.B\right|C\right)\right\}{\displaystyle \sum _{m=1}^{\infty }}\left\{1+{\left[-p\left(\left.A\right|BC\right)p\left(\left.B\right|AC\right)\right]}^m\right\}$$

(6c)

Given that probabilities may not be greater than unity this infinite series can be compactly written as follows:

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)={\displaystyle \frac{p\left(\left.A\right|C\right)+p\left(\left.B\right|C\right)}{1+p\left(\left.A\right|BC\right)p\left(\left.B\right|AC\right)}}$$

(7)

However, Equation (7) may be simplified since, given that in the recursive case it cannot be determined which of {A,B} is prior, we may take p(A|B) = p(A) and p(B|A) = p(B). Hence:

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)={\displaystyle \frac{p\left(\left.A\right|C\right)+p\left(\left.B\right|C\right)}{1+p\left(\left.A\right|C\right)p\left(\left.B\right|C\right)}}$$

(8)

Equation (8) (isomorphic to Equations (1)) states the “Hyperbolic Sum Rule” (HSR) theorem to be proved where {A,B} are recursively dependent. In deriving the form of Equation (8), we asserted that p(A|BC) = p(A|C) and p(B|AC) = p(B|C); that is, due to the recursion between events {A, B}, the probability of A becomes essentially independent of B, and vice versa. That is, Equation (7) can be re-expressed in the strictly hyperbolic form of Equation (8). In effect, this assertion (p(A|B) = p(A), etc.) can be justified as the most likely (MaxEnt) relationships between the conditional probabilities for A and B when we know a priori that A and B may have a recursive dependency, but we do not know the precise (quantitative) relationship between A and B, or indeed, the priority of A or B. In Appendix A.3 we prove that the HSR of Equation (8) is MaxEnt. That is to say, these assertions can also be justified by recourse to the Principle of Indifference (PI) which is the simplest non-informative prior in Bayesian probability (see [36,38]).

Equation (6b) expresses the case where the relation between events {A, B} is not recursive but stops at a countable depth or iteration: the overt conditionality between them is therefore maintained such that p(A|B) ≠ p(A) and p(B|A) ≠ p(B). For example, if one wanted to formalise the psychological aspects of Poker, one might express a “bluff” as the F term in Equation (6b), a “double-bluff” as the F² term, and a “triple-bluff” as the F³ term etc. But the HSR of Equation (8) expresses the recursion needed for the etalon of Equation (1c) (for example). In contrast, it is clear that the CSR of Equation (2b) is the truncated HSR case of Equation (7), where the CSR a priori ignores recursion and its associated higher-order dependencies (either due to specified information, or because their existence is simply not taken into account) and excludes the higher-order (logical/temporal) ordering of propositions. In any case, Bayes’ Theorem (the Product Rule) remains valid.

2.4. The Conventional Sum Rule (II)

Both the HSR (Equation (8)) and the CSR (Equation (2b)) are MaxEnt in each of their respective domains of applicability (proved, respectively, in Appendix A.3 and Appendix A.5), and we need to know when to use the CSR and when the HSR (and why). In particular, the point of this paper is to emphasise the importance of the temporal (or logical) ordering of propositions, and therefore the possibility for recursion, which is what distinguishes between the CSR and HSR: the CSR does not adequately take the temporal ordering of propositions into account, whereas the HSR does. And, this is particularly evident in the Venn diagram of Figure 1 which cannot represent the temporal or logical ordering of the propositions.

As an example, we consider the scenario of a fair coin being tossed twice. We designate the event A as “the first toss gives a head”, and the event B as “the second toss gives a head”, and we wish to compute the probability of the first toss or the second toss giving a head. Such a ‘fair’ coin does not have memory (each toss is independent of any others), so that we have p(A) = p(B) = ½. Here, there is no recursion since p(C|A) = p(C) and p(C|B) = p(C): that is, C is independent of both of {A,B}, and therefore we may simply use the CSR (Equation (2a)): p(A or B|C) = ½ + ½ − ¼ = ¾.

Clearly, applying the HSR (Equation (8)) to obtain p(A or B|C) = 4/5 in this context is manifestly false, precisely because we already know that there is no recursion present. The difference between the hyperbolic (HSR, Equation (8)) and conventional (CSR, Equation (2b)) sum rules is specifically the prior information that must be taken into account to do the calculation. The decisive question is whether or not the possibility that the mutual dependence of {A,B} is recursive can be excluded.

There are other simple cases where memory is involved, such that p(A|B) ≠ p(A). One such case is Cox’s [1] example of randomly drawing black or white balls from a bag. The probability of a result is affected by the results of previous draws, but this is a memory effect, not a recursive one; and here the system only iterates based on the forward propagation of a previous state (i.e., no feedback): see Appendix A.6 for a discussion of FIR (finite impulse response) filters. In general, if p(A|B) and p(B|A) are expressed in non-recursive form the CSR should be used, but if not, then the HSR should be used. The CSR should be used only where recursion is a priori excluded.

3. A General “Hyperbolic Sum Rule” (HSR)

3.1. Proving the HSR Theorem

We seek to prove the “Hyperbolic Sum Rule” of Equation (8) (whose plausibility was described informally in Section 2.3):

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)={\displaystyle \frac{p\left(\left.A\right|C\right)+p\left(\left.B\right|C\right)}{1+p\left(\left.A\right|C\right)p\left(\left.B\right|C\right)}}$$

with the notation defined in Section 2. Consider the function Σ given previously as Equation (4a):

$$\mathsf{\Sigma }\left(x\right)={\displaystyle \frac{1-x}{1+x}}$$

which has the same properties as Cox’s function S, such that v ≡ Σ(u), where u ≡ p(A|C) and therefore v ≡ p($\overline{A}$|C), properties noted by both Jaynes [2] and Cox [1]. Also Σ{Σ(x)} = x just as S{S(x)} = x.

From de Morgan’s Theorem ($\overline{A\mathrm{O}\mathrm{R}B}\equiv \overline{A}\overline{B}$, hence $A\mathrm{O}\mathrm{R}B\equiv \overline{\overline{A}\overline{B}}$), and see also Equation (13) passim in Cox [1]:

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)=\mathsf{\Sigma }\left[\mathsf{\Sigma }\left(p\left(\left.A\right|\overline{B}C\right)\right)\mathsf{\Sigma }\left(p\left(\left.B\right|C\right)\right)\right]$$

(9)

where $\overline{B}$ (the negation of B) is required in the conditioning probability of the first probability term in order to satisfy the appropriate application of the Product Rule in de Morgan’s Theorem.

Substituting Cox’s first-order expression for Σ (that is, Σ = 1 − x) we recover the “Conventional Sum Rule” (CSR, Equation (2b)), after some manipulation. However, by substituting the full bilinear form for Σ (Equation (4a)) into Equation (9):

$$\begin{array}{ll}p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)& =\mathsf{\Sigma }\left[{\displaystyle \frac{\left(1-p\left(\left.A\right|\overline{B}C\right)\right)\left(1-p\left(\left.B\right|C\right)\right)}{\left(1+p\left(\left.A\right|\overline{B}C\right)\right)\left(1+p\left(\left.B\right|C\right)\right)}}\right]\\ & =\mathsf{\Sigma }\left[{\displaystyle \frac{1-p\left(\left.A\right|\overline{B}C\right)-p\left(\left.B\right|C\right)+p\left(\left.A\right|\overline{B}C\right)p\left(\left.B\right|C\right)}{\left(1+p\left(\left.A\right|\overline{B}C\right)\right)\left(1+p\left(\left.B\right|C\right)\right)}}\right]\end{array}$$

(10)

and expanding the outer self-reciprocal Σ function we find:

$$\begin{array}{ll}p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)& ={\displaystyle \frac{1-\left(\frac{1-p\left(\left.A\right|\overline{B}C\right)-p\left(\left.B\right|C\right)+p\left(\left.A\right|\overline{B}C\right)p\left(\left.B\right|C\right)}{\left(1+p\left(\left.A\right|\overline{B}C\right)\right)\left(1+p\left(\left.B\right|C\right)\right)}\right)}{1+\left(\frac{1-p\left(\left.A\right|\overline{B}C\right)-p\left(\left.B\right|C\right)+p\left(\left.A\right|\overline{B}C\right)p\left(\left.B\right|C\right)}{\left(1+p\left(\left.A\right|\overline{B}C\right)\right)\left(1+p\left(\left.B\right|C\right)\right)}\right)}}\\ & ={\displaystyle \frac{2p\left(\left.A\right|\overline{B}C\right)+2p\left(\left.B\right|C\right)}{2+2p\left(\left.A\right|\overline{B}C\right)p\left(\left.B\right|C\right)}}\end{array}$$

(11)

But, when A depends recursively on B, then also B always depends on A, so that p(A|B) = p(A|A); hence p(A|B) = p(A), also p(A|$\overline{B}$) = p(A); and therefore $p\left(\left.A\right|\overline{B}C\right)=p\left(\left.A\right|C\right)$.

Thus:

$$p\left(\left.A\mathrm{O}\mathrm{R}B\right|C\right)={\displaystyle \frac{p\left(\left.A\right|C\right)+p\left(\left.B\right|C\right)}{1+p\left(\left.A\right|C\right)p\left(\left.B\right|C\right)}}$$

(12)

which is Equation (8) as required. Note that p(A|B) = p(A) if {A,B} are either independent (a CSR case) or recursive (the HSR case).

3.2. Some Analytical and Numerical Comparisons

Having derived a “Hyperbolic Sum Rule” (HSR) expression, it is interesting to see how much it diverges from the “Conventional Sum Rule” (CSR) in a simplified scenario that considers the function p(A or B) for two possible and equally probable events {A, B}, that is, p(A) = p(B).

The HSR is well-behaved in that it yields the same results as the CSR in the limit that neither event occurs, that is, p(A) = 0 and p(B) = 0. In this case, we have p(A|BC) = p(B|AC) = 0 (that is, p(A and B|C) = 0 from Bayes’ Theorem). It is straightforward to see in this specific case that p(A or B) = 0 for both sum rules.

The HSR is also well-behaved where p(A) = p(B) = 1. In this case, the CSR reduces to p(A or B) = p(A) + p(B) − p(B) = p(A) = p(B). The HSR also reduces to the same result, but via the progression p(A or B) = 2p(A)/2 = 2p(B)/2 = p(A) = p(B).

For the simple system in which there are two equally probable events {A,B} the conventional and hyperbolic forms of the sum rule converge when the events are either impossible or certain. It is for the intermediate probabilities 0 < p(A) < 1 that the two forms diverge. Figure 2 shows the behaviour of p(A or B); calculated for the simple case of p(A) = p(B). For the CSR the two events {A,B} are assumed to be independent, that is, p(A|B) = p(A) and p(B|A) = p(B), according to the MaxEnt criterion (in the absence of any other information).

From a simplistic Maximum Entropy (MaxEnt) argument, one might expect that the greatest difference between the two sum rules would occur when the events have the most uncertainty, that is, p(A) = ½. For this situation, the CSR returns an overall probability of p_CSR(A or B|C) = ½ + ½ − (½ × ½) = ¾ (75%). In contrast, the HSR returns a slightly higher overall probability of p_HSR(A or B|C) = (½ + ½)/(1 + [½ × ½]) = ⅘ (80%). The presence of recursion (or mutual dependence) increases the probability that both events will occur.

Interestingly, however, Figure 2 shows that the maximum difference between the two sum rules is actually 5.09 percentage points, for p = 45.35% in this simplified case (reflecting the asymmetry of the hyperbolic tangent function). Thus, we see that the difference between the conventional and hyperbolic sum rules may be quite minor: in most cases it would be hard to observe (less than 5 percentage points).

Being MaxEnt (proved in Appendix A.3), an application of the HSR of Equation (12) would be to calculate the sum probability for two events where the relationship between the two events is not known, yet some correlation (including recursion) is reasonably anticipated or cannot be excluded. In contrast, the CSR (also MaxEnt, proved in Appendix A.5.2) explicitly assumes that recursion is not present and also requires knowledge of the conditional probability relationship between the two events: this may be estimated or simply assumed not to exist (since independence is a reasonable MaxEnt assumption). Note that the CSR explicitly excludes recursion, an assumption which should be justified. Thus, the HSR of Equation (12) obviates both the need for any unknown conditional probability, and also avoids any unjustified assumptions relating to recursion.

For example, imagine a scenario of elections occurring on the same future day in neighbouring countries “A” and “B”, with leading candidates “Alice” and “Bob” in each, respectively. Opinion polls consistently suggest that both candidates have a good chance of being elected: 45% in each case: p(A) = p(B) = 0.45. We pose the question (to which we would like a Bayesian answer, if possible): what is the probability that at least one of the two leading candidates will be elected?

In order to apply the conventional sum rule we require the conditional probability that Alice will be elected given Bob is elected, or alternatively, the probability that Bob is elected given Alice is successful (given by Bayes’ Theorem, Equation (2c)). In the CSR context (in the absence of any other information to the contrary) the best we can assume is that these events are essentially independent, in which case p(A|B) = p(A).

The CSR can then be applied to offer the MaxEnt probability of either Alice or Bob being elected (given that the two events are mutually independent):

p(A or B)_CSR = p(A) + p(B) − p(A)p(B) = 0.698

However, the HSR allows us to include the possibility of a mutual (recursive) relationship between these two events (without making any other assumptions which could skew the calculation), giving a MaxEnt (most likely) probability in the absence of any other information (or any implicit assumptions):

p(A or B)_HSR = {p(A) + p(B)}/{1 + p(A)p(B)} = 0.748

Thus, the HSR probability estimation of at least one of the candidates being successful is noticeably higher than that anticipated from the CSR. This is because there are a variety of ways the two elections could potentially influence each other, and the HSR accounts for the possible existence of these mutual recursive interactions, while (in the absence of specific information as given by the appropriate conditional probabilities) the CSR explicitly ignores such possible recursive influences.

It is perhaps surprising that such a difference between the two sum rules has not been noted up to now: surely empirical observation would have shown a systematic discrepancy from the (wrong) current theoretical calculation based purely on the CSR? That no-one has suspected that the CSR is faulty may be due to the fact that for most simple cases (e.g., card games, gambling etc.) recursion is generally absent (and the CSR is correct). Anyway, probability theory originated in the analysis of games of chance where recursion is excluded. For more complex cases there is usually no practical possibility of re-running the observations (elections in the case given above) sufficiently to gain a statistically significant estimate of the associated empirical probabilities. Most real cases have complex (and recursive) conditionalities and dependencies, and most are unrepeatable, meaning that the associated probabilities cannot be determined by conventional (frequentist) methods. Thus, any systematic error in the underlying probability theory is probably unobservable, and it is hardly surprising that it has been unobserved and unsuspected until now.

3.3. Concatenation Rules for Multiple Hypotheses

In Appendix A.4 we prove the following relations:

A version of Bayes Theorem for multiple summed hypotheses {X}_OR as the number of these hypotheses grows large (see Appendix A.4 Equation (A18)):

$$\begin{array}{c}p\left(A\right)p\left(\left.{\left\{X\right\}}_{\mathrm{OR}}\right|A\right)=p\left({\left\{X\right\}}_{\mathrm{OR}}\right)p\left(\left.A\right|{\left\{X\right\}}_{\mathrm{OR}}\right)\mathrm{a}\mathrm{s}\left\{X\right\}\to \infty \end{array}$$

We also exploit the hyperbolic tangent function to prove a Hyperbolic Sum Rule (HSR) for multiple hypotheses with multiple conditionalities (see Appendix A.4 Equation (A24)):

$$p{\left(\left.A\mathrm{O}\mathrm{R}B\right|D\mathrm{O}\mathrm{R}E\right)}_{\mathrm{HSR}}={\displaystyle \frac{p\left(\left.A\right|D\mathrm{O}\mathrm{R}E\right)+p\left(\left.B\right|D\mathrm{O}\mathrm{R}E\right)}{1+p\left(\left.A\right|B\mathrm{O}\mathrm{R}D\mathrm{O}\mathrm{R}E\right)p\left(\left.B\right|A\mathrm{O}\mathrm{R}D\mathrm{O}\mathrm{R}E\right)}}$$

Finally, a generalised HSR with N multiple hypotheses, which sorts odd-ordered conjunctions into the numerator and even-ordered conjunctions into the denominator (see Appendix A.4 Equation (A25)):

$$p{\left(A_1\mathrm{O}\mathrm{R}{A}_2\mathrm{O}\mathrm{R}\dots \mathrm{O}\mathrm{R}{A}_N\right)}_{\mathrm{HSR}}={\displaystyle \frac{{\sum }_{l=1,\mathrm{odd}}^N{\sum }_{m=1}^{m=\frac{N!}{l!\left(N-l\right)!}}{\prod }_{n=P\left\{l,1:N\right\}}^{l}p\left(\left.A_n\right|\left\{A_{n^{\prime }},l-1\right\}\right)}{{\sum }_{l=0,\mathrm{even}}^N{\sum }_{m=1}^{m=\frac{N!}{l!\left(N-l\right)!}}{\prod }_{n=P\left\{l,1:N\right\}}^{l}p\left(\left.A_n\right|\left\{A_{n^{\prime }},l-1\right\}\right)}}$$

We note the close functional isomorphism of Equation (A25) with the transfer function (defined by the Z-transform) of an infinite impulse response (IIR) filter used in digital signal processing (see further in Appendix A.6), which is given by a quotient of summations in both the numerator and denominator:

$$Z\left(z\right)={\displaystyle \frac{{\sum }_{l=0}^L{b}_l{z}^{-l}}{1+{\sum }_{n=1}^N{a}_n{z}^{-n}}}$$

(13)

The parameter z, is given by z ≡ exp(iω), where ω is a frequency in normalised units, such that z is therefore located on the unit circle in the complex plane (see the discussion in Section 2.2 (Equation (4c)) for the self-reciprocal function S^m, whose roots also lie on the unit circle), noting that for convenience the coefficients a_n in Equation (13) are represented negatively as compared to convention. Of particular interest is to note that the conventional IIR filter generally has the first co-efficient of the denominator as unity, just like the HSR of Equation (12).

Equation (A25) may also be written in a form such that the probability distributions are represented (as MaxEnt decaying exponentials probabilities, see Jaynes 1982 [3]) by the appropriate Lagrange multiplier β (noting that different distributions are generated by different choices of β, with an example of this given in a recent treatment of the Wine/Water paradox [38]) (see Appendix A.5 Equation (A28d)):

$$p{\left(A_1\mathrm{O}\mathrm{R}{A}_2\mathrm{O}\mathrm{R}\dots \mathrm{O}\mathrm{R}{A}_N\right)}_{\mathrm{HSR}}={\displaystyle \frac{{\sum }_{l=1,\mathrm{odd}}^N\frac{N!}{l!\left(N-l\right)!}{\left\{\left(\frac{1-e^{-\beta }}{1-e^{-N\beta }}\right)e^{\beta }\right\}}^l\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\beta {\sum }_{n=P\left\{l,1:N\right\}}^{l}n\right\}}{{\sum }_{l=0,\mathrm{even}}^N\frac{N!}{l!\left(N-l\right)!}{\left\{\left(\frac{1-e^{-\beta }}{1-e^{-N\beta }}\right)e^{\beta }\right\}}^l\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\beta {\sum }_{n=P\left\{l,1:N\right\}}^{l}n\right\}}}$$

additionally showing that the HSR is well-behaved for multiple hypotheses as $N\to \infty$.

In Appendix A.5 we prove the following relation for the Conventional Sum Rule for multiple hypotheses (see Appendix A.5.1 Equation (A34)):

$$\begin{array}{c}p{\left(A_1\mathrm{O}\mathrm{R}{A}_2\mathrm{O}\mathrm{R}\dots \mathrm{O}\mathrm{R}{A}_N\right)}_{\mathrm{CSR}}=\\ 1+{\displaystyle \sum _{l=1,\mathrm{o}\mathrm{d}\mathrm{d}}^N}{\displaystyle \sum _{m=1}^{m={\displaystyle \frac{N!}{l!\left(N-l\right)!}}}}{\displaystyle \prod _{n=P\left\{l,1:N\right\}}^l}p\left(\left.A_n\right|\left\{A_{n^{\prime }},l-1\right\}\right)-{\displaystyle \sum _{l=0,\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}}^N}{\displaystyle \sum _{m=1}^{m={\displaystyle \frac{N!}{l!\left(N-l\right)!}}}}{\displaystyle \prod _{n=P\left\{l,1:N\right\}}^l}p\left(\left.A_n\right|\left\{A_{n^{\prime }},l-1\right\}\right)\end{array}$$

This relation for the CSR (Equation (A34)) rests on results depending on the HSR.

4. Discussion

4.1. Probability Is Physical

In most cases, the distinction between the Conventional Sum Rule (CSR) and the Hyperbolic Sum Rule (HSR) is probably unobservable, given the attainable empirical precision. The case of the two parallel elections considered in Section 3.2 has 80% for the HSR compared to only 75% for the CSR, when the two election probabilities involved are both 50% (p(A) = p(B) = 0.5). But the CSR cannot model recursion.

However, the derivation of the HSR uncovers a mathematical relation that is also observed in diverse physical settings. In particular, the HSR is isomorphic not only to the hyperbolic tangent double-angle identity (Equation (1a)) but also to Einstein’s famous velocity addition formula (Equation (1b)) and to the reflectivity expression for an optical etalon (Equation (1c)). Thus, the general (hyperbolic) probability sum rule conforms to a ‘template’ seen in the physical world; that is to say, although being apparently a logical relation it behaves just as many physical phenomena do. Therefore, we can say that the etalon function (Equation (1c)): intrinsically probabilistic in nature, being a wave scattering phenomenon) represents at least a physical embodiment of the HSR. In particular, the physical structure of the etalon with its two scattering interfaces (A and B, say) suggests that the reflection coefficient calculated from the solution to Equation (1c) can also be interpreted as representing the physico-logical answer to the question: what is the probability that a quantum-mechanical particle (that is, a photon) is reflected from interfaces “A OR B”? It is clear that the basic two-faceted etalon physically expresses the logical OR operation.

This raises the interesting philosophical question: is the hyperbolic nature of the HSR a result of the intrinsic hyperbolic character of the natural universe? Affirming that it is implies that probability theory is firmly empirical (and not merely mathematical): both the theory of Special Relativity (Equation (1b)) and the quantum-mechanical scattering probability (Equation (1c)) are consistent with the hyperbolic nature of spacetime. But denying it would imply that it is simply coincidental that the HSR shares fundamental characteristics of the natural universe.

This question actually goes to the root of a deep philosophical debate as to the intrinsic nature of probability theory: is it purely an empirical construct where probabilistic variables only acquire meaning when a statistical measurement takes place? This view is implied by the so called ‘frequentist’ fraternity, who do not ascribe an independent reality to the “laws” of probability; rather they consider that the probability of a hypothesis depends solely upon the frequency of success over a given number of trials. But Bayesians would say that the hyperbolic nature of the HSR implies that probability theory and the underlying theoretical equations form a component part of the full set of universal natural laws; that is to say, they are an intrinsic aspect of the (hyperbolic) universe with their own independent existence.

We should add that the result that the MaxEnt nature of both the HSR of Equation (12) (possibly recursive, see Appendix A.3) and the CSR (non-recursive, see Appendix A.5) indicates that the equilibrium state of physical systems (corresponding to a stable and most-likely configuration) obeys both the kinematical Principle of Least Action and the entropic Principle of Least Exertion ([11]). These fundamental principles underlie all known physical phenomena and therefore indicate that both the HSR and CSR are consistent with the requirements of the physical world. The fact that the HSR is MaxEnt also implies that all the parts of such systems recursively condition all the other parts. That is, in the general case everything is entangled with everything else just as Karen Barad implies [34]. But here we wish to avoid metaphysical discussion (for which see [6,7]).

4.2. Recursion

Why has the issue of recursion apparently not previously been explicitly considered in probability theory? This may simply be an accidental result of history, with probability theory having developed from the gaming scenarios associated with cards and dice, where events are discretely independent and are characterised with shallow conditionalities that do not reach far back into the past. For example, the game of Poker has explicit calculable conditionalities based on cards revealed to the table, cards in your hand, and the hidden cards remaining; the calculable conditionalities do not involve the (fascinating) issues of bluffing, double bluffing, and multiple bluffing. These could in principle be expressed logically (using the formalism sketched above), but are currently treated as psychology.

However, it is interesting to note that Cox (1946 [1]) derives the general result S^m = 1 − x^m for the involutory function S(x), stating that the value for m (assumed an integer) is “arbitrary” and “purely conventional”; indeed, Jaynes writes that the value of m is “actually irrelevant” [2]. For “simplicity of notation” Cox arbitrarily chooses m = 1 (to effectively define Equation (3c)), which is both convenient and intuitive, since it immediately leads to the Conventional Sum Rule associated with the Venn diagram of Figure 1. Thus, Cox’s analysis allows for recursion conceptually, although he apparently appreciated neither this nor the physical interpretation of his function S^m.

But exploiting the hyperbolic insight described here, we can now understand m to represent the degree of mutual dependency leading to recursion; that is, it has both a logical and physical (spatio-temporal) meaning, with real physical implications which are not at all “arbitrary”. In particular, m = 0 is trivial, m = 1 is the simple CSR case, m⁻¹ = 0 is the HSR case. We consider this further in Section 4.3.

Another reason for ignoring recursion is that the appropriate calculus of probabilities requires substantial computing power which has only become available in recent decades. Up to the point of the emergence of artificial intelligence (AI) and machine learning (ML), approximately at the turn of the millennium, there has not been a good reason to develop a theory of recursive probabilities, since before this time the necessary computations were practically intractable. However, as large-scale computing power has evolved over recent decades, this has also enabled the development of the sophisticated (Bayesian) algorithms underpinning AI and ML programming. Such algorithms allow a dispassionate approach to making probabilistic inferences and decision-making in an environment of uncertainty, within a mathematical framework for updating beliefs and reasoning about uncertain events. In addition, recursion provides AI with additional flexibility to handle complex and dynamic problems by recursively decomposing them, while leveraging self-referential architectures and utilising iterative processes.

Recursion can be exploited relatively simply in AI as a means to break down complex problems into smaller, more manageable subproblems. Alternatively, the complexity and power of modern computing also enables the employment of recursive data structures that can be used to represent hierarchical or interconnected relationships. Recursion can also be seen in the inductive reasoning aspect of AI, which involves inferring general rules or patterns from specific observations, so as to learn iteratively from data in order to improve the accuracy of predictions or classifications.

In AI (and the associated field of neural networking) recursion has been employed empirically: the complexity associated with the algorithms and feed-back routines are not generally amenable to analytical (closed-form or tractable mathematical) analysis, such that currently only ‘simulations’ can be used to analyse AI behaviour. It is in this context of recursive AI that the Hyperbolic Sum Rule described here allows the systematic, reliable and consistent handling of multiple recursive hypotheses (or events) according to Bayesian requirements, as discussed in Appendix A.4. We therefore expect the HSR to enable a more analytic (closed-form and, therefore, tractable) analysis of complex AI algorithms to give additional insights into their form of operation and anticipated outputs.

4.3. Two Distinct Sum Rules

We have shown that in different situations different rules apply for summing the probabilities of events (or, nearly equivalently, the probabilities of certain hypotheses being true). We wish to find a relation that allows us to compute p(A or B|C), that is, the probability of (A or B) given C, where {A,B} are events (or hypotheses) and {C} are conditionalities.

There are two distinct cases depending on whether or not {A,B} are recursively dependent. If {A,B} are independent (such as coin flipping, see Section 2.4) or they are not recursively dependent then the “Conventional Sum Rule” (CSR) applies. But if {A,B} are recursively dependent (such as reflection from a multilayer stack, see Equation (1c)) or indeed, if their mutual relationship is not quantitively known (such that recursion cannot be ruled out) then the “Hyperbolic Sum Rule” (HSR) applies. Concatenation formulae for either or both of multiple events {A,B, …} or multiple conditionalities {C,D, …} are readily derived for both the HSR (Appendix A.4), and the CSR (Appendix A.5; also see the summary in Section 3.3 of important results for both HSR and CSR). Therefore, there is no loss of generality from treating only the simple cases.

If p(A|B) > 0 and p(B|A) > 0, with both conditional probabilities being valid independent of any temporal (that is, causal) or logical constraints, then {A,B} are recursively dependent and the HSR applies. Otherwise, the CSR applies. Conversely, the CSR applies if this probability relation holds:

[{p(C|A) = p(C) or p(C|B)} = p(C)] = {p(C|A or B) = p(C)}

Note that in simple cases (see Figure 2) the difference between CSR and HSR is probably too small to be easily observed: even in the fair coin or election cases (Section 2.4 and Section 3.2) the CSR gives 75% where the HSR gives 80%.

It is clear that the CSR and HSR are formally related (see the discussion in Appendix A.5.2 and Equation (4)). It also seems fairly clear that the “truncated expansion” of Equation (4d) has properties related to (for example) the “finite impulse response” (“FIR”) filter that is important in digital signal processing (DSP) applications (further details in Appendix A.6). The case represented by finite m where m ≥ 1 and m⁻¹ > 0 (analogous to the FIR filter) offers an engineering approach to the probabilistic treatment and control of system noise within a finite (and specified) timeframe, an issue of practical importance for stochastic systems. Such practical probabilistic cases, formally CSR (see Appendix A.5), blur the distinction between CSR and HSR, since it seems that such “FIR filtering” may be represented approximately as a “truncated HSR”, with a formalism that is also approximately MaxEnt in its own terms (see Appendix A.3). However, the derivation of the HSR is directly from Equation (4a): the discussion of finite values of m in Equation (4b,c,d) is presented only by way of comment. The DSP engineers have the “infinite impulse response” (“IIR”) filter, which in our representation is isomorphic to the HSR, but the HSR is defined without any (infinite) summations. There is the (possibly) recursive case (HSR) and the (definitely) non-recursive case (CSR).

4.4. Motivation for the HSR: An Example

The artificial intelligence (AI) and machine learning (ML) applications widely employed today regularly use Bayesian methods to address complex problems involving multiple hypotheses each potentially featuring multiple conditionalities. One such problem is the non-trivial one of how to build a Battery Management System for the electric vehicle (EV) sector, seeking to optimise longevity and lifecycle profile, distance between recharging, recyclability, and overall environmental impact; even providing guidance for driving style to help achieve such an optimisation, in advance of the availability of full automotive self-driving technology. This has attracted significant interest (see, for example, Ghalkhani and Habibi [45]), and is supported in a current EU Horizon programme [46], CIRCUBATT: Circular economy innovations for resilient, competitive and sustainable battery technologies.

A battery pack appears to be a simple thing, but there is much complexity associated with it, as well as its optimised exploitation, and safeguarding measures to minimise risk since such intensive energy storage capability remains hazardous. The CIRCUBATT research programme, which aims to develop commercially viable applications, generates very large datasets based on a multitude of space- and time-stamped parameters generated by the dynamic properties of a large base of EVs and their batteries, whose analysis is facilitated by AI/ML tools which rely crucially on recursion as a means to achieve efficient learning, optimisation and decision-making processes.

The architecture of an AI/ML system using Bayesian inference within its processing engine consists of multiple layers (e.g., an input layer, which receives the input data, hidden layers which process the input data, and an output layer), and frequently features recursion as a means to achieve convergence of the output: recursion being exhibited when an algorithm calls a function or process repeatedly until convergence is achieved. Each layer of the AI/ML system receives multiple inputs, which can be considered to be the prior probabilities (hypotheses or initial beliefs), and outputs into the inputs of the next layer. For example, at an intermediate plane, the various states input into that layer can each represent the possibility of hypotheses (A or B or C or …) considered in the intermediate calculation, where “or” here indicates the Sum Rule to effect the summation of the hypotheses: conventionally the Sum Rule would have been the CSR but now the HSR has been shown to be the correct MaxEnt rule to use; particularly in the absence of a definitive set of a priori rules or contextual constraints (i.e., apart from any known initial learning parameters) for the AI/ML learning phase. Having been fed forward through a number of intermediate stages, the AI algorithm often features recursion by feeding the output results back into its input, and repeating the whole process, before converging to a final result.

Analysing such an AI/ML process analytically is practically impossible when the CSR is employed, as indeed is guaranteeing that the calculation will be ‘well behaved’, or is indeed ‘reliable’, for example when AI/ML is used in a safety-critical context such as autonomous vehicles. Currently, in effect, AI calculations can only be analysed by simulation to see what the output will be. A priori prediction of the result of the AI calculation is effectively impossible currently, as is guaranteeing its reliability and safety, or indeed ensuring a degree of explication and predictability. But concatenation of the HSR, and indeed, ‘nesting’ of HSR expressions within a HSR function, in order to analytically model such a multi-stage and recursive AI architecture is in principle both analysable and tractable. This therefore provides a route to the analytical study and assured understanding of complex AI/ML processes and calculations.

Particularly for probabilistic AI/ML schemes, sigmoid or softmax functions are often used for the non-linearity frequently required in the intermediate (hidden) stages to achieve probabilities and classifications, and it is noteworthy that the HSR automatically embodies such a non-linear (tanh) function, thereby already making it inherently suitable for use in such AI/ML processing; again, the MaxEnt feature of the HSR and the tanh function (see Appendix A.3) is also relevant here with the minimal introduction of unwanted, undesired, or unintended extraneous information by the non-linear function into the final solution.

5. Summary and Future Work

We have shown how a Hyperbolic Sum Rule (HSR) for probability can be derived in a way that is consistent with the mathematical and logical requirements as laid out by Cox [1] and Jaynes [2]. In particular, we have proved that the HSR is Maximum Entropy (MaxEnt), and thereby is also consistent with the Second Law of Thermodynamics.

For the simple case of only two equally probable hypotheses (or events) the conventional and hyperbolic versions of the sum rule yield similar (but not identical) results (see Figure 2). However, when large numbers of hypotheses are being considered, we have shown (Appendix A.5) that properties derived from the HSR are needed to obtain an appropriate Conventional Sum Rule (CSR). That is, although the CSR is applicable to certain simple cases, the HSR must be used where recursion cannot be ruled out (and where we need a MaxEnt solution in accordance with Bayesian requirements).

Apart from its mathematical consistency, we should also note empirically that numerous other physical phenomena involving probabilities also obey the HSR, as one might expect considering the (widely acknowledged) hyperbolic nature of the spacetime metric for the universe (see, for example, Penrose 2004 [24]). Therefore, we conclude that the Hyperbolic Sum Rule implies that probability is physical (also supported by the fact that the HSR is MaxEnt). This is because probabilities always refer to phenomena (however idealised). Phenomena belong in the real world, and their value (including measurement) is necessarily assessed by people for their own purposes. It is, therefore, satisfactory that the general MaxEnt case is represented by a formalism isomorphic to formalisms representing probabilities incorporated with, in general, physical systems (such as the etalon, Equation (1c)).

In terms of application, we expect that these methods will make analytically tractable those calculations that involve many hypotheses featuring multiple dependencies (both spatial and temporal). For example, disentangling the highly complex interactions of the different genes within a single genome represents an extremely difficult computational challenge, yet one with a glittering set of potential medicinal benefits. Another entirely different example regards the handling of certain important filters used in modern digital signal processing. Doubtless, others will also emerge.

Another application which exploits the handling of multiple hypotheses featuring highly interrelated, highly complex and recursive dependencies (currently thought either intractable or impracticable for explicit analytical evaluation) is in the emerging field of rule-based expert learning systems for inference. Such applications depend on the use of Bayesian reasoning, and exploit modern computational resources to handle a high degree of multi-modal inputs and outputs, each requiring a very high number of explicit (and implicit) variables (hypotheses) with associated conditional dependencies. However, issues of reliability, predictability and engineering control of AI technology are becoming more salient, since these are critical prerequisites for the systemic, safe, and validated use of AI technology. We are confident that the Hyperbolic Sum Rule will also contribute towards encouraging AI’s development as a scalable and high-valued technological asset.

6. Conclusions

In general, probabilities of events or hypotheses are added according to a “hyperbolic sum rule”. The linear rule that is conventional is applicable only in certain limited circumstances (where it is known that recursion is not present).

The new rule has been proved “Maximum Entropy”, that is, it is applicable to probability problems requiring unbiassed general solutions according to well-known Bayesian principles. It has been used to generalise the conventional rule for multiple hypotheses, to prove that it too is “Maximum Entropy”; albeit the CSR exhibits a systematically lower maximum entropy because of its extra assumptions. Various useful formulae have been derived for applicability to problems involving multiple hypotheses.