The Information Bottleneck’s Ordinary Differential Equation: First-Order Root Tracking for the Information Bottleneck

Abstract

The Information Bottleneck (IB) is a method of lossy compression of relevant information. Its rate-distortion (RD) curve describes the fundamental tradeoff between input compression and the preservation of relevant information embedded in the input. However, it conceals the underlying dynamics of optimal input encodings. We argue that these typically follow a piecewise smooth trajectory when input information is being compressed, as recently shown in RD. These smooth dynamics are interrupted when an optimal encoding changes qualitatively, at a bifurcation. By leveraging the IB’s intimate relations with RD, we provide substantial insights into its solution structure, highlighting caveats in its finite-dimensional treatments. Sub-optimal solutions are seen to collide or exchange optimality at its bifurcations. Despite the acceptance of the IB and its applications, there are surprisingly few techniques to solve it numerically, even for finite problems whose distribution is known. We derive anew the IB’s first-order Ordinary Differential Equation, which describes the dynamics underlying its optimal tradeoff curve. To exploit these dynamics, we not only detect IB bifurcations but also identify their type in order to handle them accordingly. Rather than approaching the IB’s optimal tradeoff curve from sub-optimal directions, the latter allows us to follow a solution’s trajectory along the optimal curve under mild assumptions. We thereby translate an understanding of IB bifurcations into a surprisingly accurate numerical algorithm.

1. Introduction

The Information Bottleneck (IB) describes the fundamental tradeoff between the compression of information on an input X to the preservation of relevant information on a hidden reference variable Y. Formally, let X and Y be random variables defined, respectively, on finite source and label alphabets $\mathcal{X}$ and $\mathcal{Y}$, and let $p_{Y|X}\left(y\right|x)p_X\left(x\right)$ be their joint probability distribution, or $p\left(y\right|x\left)p\right(x)$ for short (without loss of generality, $p\left(x\right)>0$ for every $x\in \mathcal{X}$ and so $p_{Y|X}$ is well-defined). One seeks [1] to maximize the information $I(Y;\hat{X})$ over all Markov chains $Y⟷X⟷\hat{X}$, subject to a constraint on the mutual information $I(X;\hat{X}):={\mathbb{E}}_{p\left(\hat{x}\right|x)p\left(x\right)}log\frac{p\left(\hat{x}\right|x)}{p\left(\hat{x}\right)}$,

$$I_Y\left(I_X\right):=\underset{p\left(\hat{x}\right|x)}{max}\left\{I(Y;\hat{X}):I(X;\hat{X})\le I_X\right\}.$$

(1)

The latter maximization is over conditional probability distributions or encoders $p\left(\hat{x}\right|x)$. The graph of $I_Y\left(I_X\right)$ is the IB curve. We write $T:=|\hat{\mathcal{X}}|$, for a codebook or representation alphabet $\hat{\mathcal{X}}$. An encoder $p\left(\hat{x}\right|x)$ which achieves the maximum in (1) is IB optimal or simply optimal.

Written in a Lagrangian formulation $\mathcal{L}:=I(X;\hat{X})-\beta I(Y;\hat{X})$ with $\beta >0$ (normalization constraints omitted for clarity), [1] showed that a necessary condition for extrema in (1) is that the IB Equations hold. Namely,

$$\begin{array}{cc}\hfill p\left(\hat{x}\right|x)& =\frac{p\left(\hat{x}\right)}{Z(x,\beta )}exp\left\{-\beta D_{KL}\left[p\left(y\right|x)\left|\right|p\left(y\right|\hat{x})\right]\right\},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill p\left(y\right|\hat{x})& ={\displaystyle \sum _x}p\left(y\right|x)p\left(x\right|\hat{x}),\mathrm{and}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill p\left(\hat{x}\right)& ={\displaystyle \sum _x}p\left(\hat{x}\right|x)p\left(x\right).\hfill \end{array}$$

(4)

In these, $Z(x,\beta ):={\sum }_{\hat{x}}p\left(\hat{x}\right)exp\left\{-\beta D_{KL}\left[p\left(y\right|x)\left|\right|p\left(y\right|\hat{x})\right]\right\}$ is the partition function, $D_{KL}$ is the Kullback–Leibler divergence, $D_{KL}\left[p\right|\left|q\right]:={\sum }_{i}p\left(i\right)logp\left(i\right)/q\left(i\right)$, and $p\left(x\right|\hat{x})$ in (3) is defined by the Bayes rule $p\left(\hat{x}\right|x)p\left(x\right)/p\left(\hat{x}\right)$. The IB Equations (2)–(4) are a necessary condition for an extremum of $\mathcal{L}$ also when it is considered as a functional in three independent families of normalized distributions $\left\{p\left(\hat{x}\right|x)\right\}$, $\left\{p\left(y\right|\hat{x})\right\}$ and $\left\{p\left(\hat{x}\right)\right\}$, ref. [1] (Section 3.3), rather than in $\left\{p\left(\hat{x}\right|x)\right\}$ alone. While satisfying them is necessary to achieve the curve (1), it is not sufficient. Indeed, Equations (2)–(4) have solutions that do not achieve curve (1), and so are sub-optimal. This results in sub-optimal IB curves, which intersect or bifurcate as the multiplier $\beta$ varies (see Section 3.4 in [1]).

Iterating over the IB Equations (2)–(4) is essentially Blahut–Arimoto’s algorithm variant for the IB (BA-IB) due to [1], brought below for reference. While the minimization problem (1) can be solved exactly in special cases [2] (Section IV), exact solutions of an arbitrary finite IB problem whose distribution is known are usually obtained nowadays using BA-IB; see [3] (Section 3) for a survey of other computation approaches. Write $B{A}_{\beta }$ for a single iteration of BA-IB. Since $B{A}_{\beta }$ encodes an iteration over the IB Equations (2)–(4), then an encoder $p\left(\hat{x}\right|x)$ is its fixed point, $B{A}_{\beta }\left[p\right(\hat{x}\left|x\right)]=p\left(\hat{x}\right|x)$, if and only if it satisfies the IB Equations. Or equivalently, if $p\left(\hat{x}\right|x)$ is a root of the IB operator,

$$F:=Id-B{A}_{\beta },$$

(5)

in a manner similar to [4]. We shall then call it an IB root. Agmon et al. [4] used a similar formulation of rate-distortion (RD) and its relations in [5] to the IB, to show that BA-IB suffers from critical slowing down near critical points, where the marginal $p\left(\hat{x}\right)$ of a representor $\hat{x}$ in an optimal encoder vanishes gradually. That is, the number of BA-IB iterations required until convergence increases dramatically as one approaches such points.

Formulating fixed points of an iterative algorithm as operator roots can also be leveraged for computational purposes in a constrained optimization problem, as noted recently by [6] for RD. Indeed, let $F(·,\beta )$ be a differentiable operator on ${\mathbb{R}}^n$ for some $n>0$, $F:{\mathbb{R}}^n\times \mathbb{R}\to {\mathbb{R}}^n$, where $\beta$ is a (real) constraint parameter. Suppose now that $(\mathit{x},\beta )$ is a root of F,

$$F(\mathit{x},\beta )=\mathbf{0},$$

(6)

such that $\mathit{x}=\mathit{x}\left(\beta \right)$ is a differentiable function of $\beta$. Write $D_{\mathit{x}}F:={\left(\frac{\partial }{\partial x_j}{F}_i\right)}_{i,j}$ for its Jacobian matrix, and $D_{\beta }F:={\left(\frac{\partial }{\partial \beta }{F}_i\right)}_i$ for its vector of partial derivatives with respect to $\beta$. The point $(\mathit{x},\beta )$ of evaluation is omitted whenever understood. As is often discussed along with the Implicit Function Theorem, e.g., [7], applying the multivariate chain rule to $F\left(\mathit{x}\left(\beta \right),\beta \right)$ in (6) yields an implicit ordinary differential equation (ODE)

$$D_{\mathit{x}}F\frac{d\mathit{x}}{d\beta }=-D_{\beta }F,$$

(7)

for the roots of F. Plugging in explicit expressions for the first-order derivative tensors $D_{\mathit{x}}F$ and $D_{\beta }F$, one can specialize (7) to a particular setting, which allows one to compute the implicit derivatives$\frac{d\mathit{x}}{d\beta }$ numerically. While [6] discovered the RD ODE this way, they showed that (7) can be generalized to arbitrary order under suitable differentiability assumptions. Namely, they showed that the derivatives $\frac{d^l\mathit{x}}{d{\beta }^l}$ implied by $F=\mathbf{0}$ (6) can be computed via a recursive formula, for an arbitrary-order $l>0$. By specializing this with the higher derivatives of Blahut’s algorithm [8], they obtained a family of numerical algorithms for following the path of an optimal RD root (Part I there).

In this work, we specialize the implicit ODE (7) to the IB. Namely, we plug into (7) the first-order derivatives of the IB operator $Id-B{A}_{\beta }$ (5) to obtain the IB ODE, and then use it to reconstruct the path of an optimal IB root, in a manner similar to [6]. This is not to be confused with the gradient flow (of arbitrary encoders) towards an optimal root at a fixed $\beta$ value, described in [9] (Equation (6)) by an ODE, which is a different optimization approach. In contrast, the implicit Equation (7) describes how a root evolves with $\beta$. So, in principle, one may compute an optimal IB root once and then follow its evolution along the IB curve (1). While the discovery of the IB ODE is due to [10], we derive it here anew in a form that is better suited for computational (and other) purposes, especially when there are fewer possible labels $\mathcal{Y}$ than input symbols $\mathcal{X}$, as often is the case. To that end, we consider several natural choices of a coordinate system for the IB in Section 2 and compare their properties. This allows us to make an apt choice for the ODE’s variable $\mathit{x}$ in (7). In Section 3, we present the IB ODE in these coordinates (Theorem 1). This enables one to numerically compute the first-order implicit derivatives at an IB root, if it can be written as a differentiable function in $\beta$. So long as an optimal root remains differentiable, a simple way to reconstruct its trajectory is by taking small steps at a direction determined by the IB ODE. This is Euler’s method for the IB. The error accumulated by Euler’s method from the true solution path is roughly proportional to the step size, when small enough. For comparison, reverse deterministic annealing [11] with BA-IB is nowadays common for computing IB roots. The dependence of its error on the step size is roughly the same as in Euler’s method. This is discussed in Section 4, where we combine Euler’s method with BA-IB to obtain a modified numerical method whose error decreases at a faster rate than either of the above.

However, the differentiability of optimal IB roots breaks where the solution changes qualitatively. Such a point is often called a phase transition in the IB literature, or a bifurcation—namely, a point where there is a change in the problem’s number of solutions; e.g., [12] (Section 2.3) for basic definitions. As noted already by Tishby et al. in [1], their existence in the IB stems from restricting the cardinality of the representation alphabet $\hat{\mathcal{X}}$. Since IB roots are the solutions of the fixed-point Equations (2)–(4), then the gap between achieving the IB curve (1) to merely satisfying these Equations lies in understanding the solution structure of the IB operator (5), or equivalently its bifurcations. While IB bifurcations were analyzed in several works, including [9,13,14] and others, little is known about the practical value of understanding them. In [15,16] it was shown that they correspond to the onset of learning new classes, and in [4] that they inflict a hefty computational cost to BA-IB. Following [6], this work demonstrates that understanding bifurcations can be translated to a new numerical algorithm to solve the IB. To that end, merely detecting a bifurcation along a root’s path does not suffice. Rather, it is also necessary to identify its type, as this allows one to handle the bifurcation accordingly. One can then continue following the path dictated by the IB ODE.

Almost all of the literature on IB bifurcations is based on a perturbative approach, in a manner similar to [17] (Section IV.C). That is, suppose that the first variation

$$\frac{\partial }{\partial ϵ}\mathcal{L}[p\left(\hat{x}\right|x)+ϵ\Delta p\left(\hat{x}\right|x);\beta ]{|}_{ϵ=0}$$

(8)

of the IB Lagrangian $\mathcal{L}$ vanishes, for every perturbation $\Delta p\left(\hat{x}\right|x)$. This condition is necessary for extremality and implies [1] the IB Equations (2)–(4). Then, $\left(p\left(\hat{x}\right|x),\beta \right)$ is said to be a phase transition only if there exists a particular direction $\Delta q\left(\hat{x}\right|x)$ at which $p\left(\hat{x}\right|x)$ can be perturbed without affecting the Lagrangian’s value to second order,

$$\frac{{\partial }^2}{\partial {ϵ}^2}\mathcal{L}[p\left(\hat{x}\right|x)+ϵ\Delta q\left(\hat{x}\right|x);\beta ]{|}_{ϵ=0}=0.$$

(9)

For finite IB problems, condition (8) boils down to requiring that the gradient of $\mathcal{L}$ vanishes, while condition (9) is equivalent to requiring that its Hessian matrix has a non-trivial kernel (as both are conditions on the directional derivatives, e.g., [18]). The works [9,14,15,16] take such an approach, while [13] focuses on one type of IB bifurcations.

While a perturbative approach is common in analyzing phase transitions, it has several shortcomings when applied to the IB, as noted by [10]. First, the IB’s Lagrangian $\mathcal{L}$ is constant on a linear manifold of encoders $p\left(\hat{x}\right|x)$ [9] (Section 3.1), and so condition (9) leads to false detections. While this was considered there and in its sequel [19] by giving subtle conditions on the nullity of the second variation in (9), in practice it is difficult to tell whether a particular direction $\Delta q\left(\hat{x}\right|x)$ is in the kernel due to a bifurcation or due to other reasons, as they note. Second, note that a finite IB problem can be written as an infinite RD problem [20]. As discussed in Section 5, representing an IB root by a finite-dimensional vector leads to inherent subtleties in its computation. Among other things, these may well result in a bifurcation not being detectable under certain circumstances (Section 5.3). To our understanding, many of the difficulties that hindered the understanding of IB bifurcations throughout the years are, in fact, artifacts of finite dimensionality. Third, conditions (8) and (9) do not suffice to reveal the type of the bifurcation, information which is necessary for handling it when following a root’s path. While [19] (Section 2.9) give conditions for identifying the type, these partially agree with our findings and do not suggest a straightforward way for handling a bifurcation.

Rather than imposing conditions on the scalar functional $\mathcal{L}$, our approach to IB bifurcations follows that of [6] for RD. That is, we rely on the fact that the IB’s local extrema are fixed points of an iterative algorithm, and so they also satisfy a vector equation $F=\mathbf{0}$ (6). We shall now consider a toy problem to motivate our approach. “Bifurcation Theory can be briefly described by the investigation of problem (6) in a neighborhood of a root where $D_{\mathit{x}}F$ is singular” [21]. Indeed, recall that if $D_{\mathit{x}}F$ is non-singular at a root $({\mathit{x}}_0,{\beta }_0)$, then by the Implicit Function Theorem (IFT), there exists a function $\mathit{x}\left(\beta \right)$ through the root, $\mathit{x}\left({\beta }_0\right)={\mathit{x}}_0$, which satisfies $F\left(\mathit{x}\left(\beta \right),\beta \right)=\mathbf{0}$ (6) at the vicinity of ${\beta }_0$. The function $\mathit{x}\left(\beta \right)$ is then not only unique at some neighborhood of $({\mathit{x}}_0,{\beta }_0)$, but further, $\mathit{x}\left(\beta \right)$ inherits the differentiability properties of F [21] (I.1.7). In particular, if the operator F is real-analytic in its variables—as with the IB operator (5)—then so is its root $\mathit{x}\left(\beta \right)$. While a bifurcation can occur only if $D_{\mathit{x}}F$ is singular, singularity is not sufficient for a bifurcation to occur. For example, the roots of the operator

$$F(x,y;\beta ):=(x-\beta ,0)$$

(10)

on ${\mathbb{R}}^2$ consist of the vertical line $x=\beta$, $\left\{\right(\beta ,y):y\in \mathbb{R}\}$, for every $\beta \in \mathbb{R}$. For a fixed y, each such root is real-analytic in $\beta$. However, one cannot deduce this directly from the IFT, as the Jacobian $\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right)$ of F (10) is always singular. Note, however, that in this particular example, the x coordinate alone suffices to describe the problem’s dynamics, and so its y coordinate is redundant. One can ignore the y coordinate by considering the “reduction” $\tilde{F}(x;\beta ):=x-\beta$ of F to ${\mathbb{R}}^1$. Further, discarding y also removes or mods-out the direction $\left(\begin{array}{c}0\\ 1\end{array}\right)$ from $ker{D}_{\mathit{x}}F$, which does not pertain to a bifurcation in this case. This results in the non-singular Jacobian matrix $\left(\begin{array}{c}1\end{array}\right)$ of $\tilde{F}$, and so it is now possible to invoke the IFT on the reduced problem. The root guaranteed by the IFT can always be considered in ${\mathbb{R}}^2$ by putting back a redundant y coordinate at some fixed value. In [6], a similarly defined reduction of finite RD problems was used to show that their dynamics are piecewise real-analytic under mild assumptions.

The intuition behind our approach is similar to [20] (Section III), who observed that “in the IB one can also get rid of irrelevant variables within the model”. Nevertheless, the details differ. Mathematically, we consider the quotient $V/W$ of a vector space V by its subspace W. Elements of V are identified in the quotient if they differ by an element of W: $v_1\sim v_2\iff v_1-v_2\in W$, for $v_1,v_2\in V$. This way, one “mods-out” W, collapsing it to a single point in the quotient vector space $V/W$. The resulting problem is smaller and so easier to handle, whether for theoretical or practical purposes (although not needed for our purposes, this can be made precise in terms of the tangent space of a differentiable manifold; cf., Section 3 in [22]). This is how the one-dimensional vector space $ker{D}_{\mathit{x}}F$ in our toy example (10) was reduced to the trivial $ker{D}_{\mathit{x}}\tilde{F}=\left\{\mathbf{0}\right\}$. However, one needs to understand the solution structure, for example, to ensure that the directions in W are not due to a bifurcation. We note in passing that $V/W$ has a simple geometric interpretation as the translations of W in V, in a manner reminiscent of its better-known counterparts of quotient groups and rings; e.g., [23] (Section 10.2). To keep things simple, however, we shall not use quotients explicitly. Instead, the reader may simply consider the sequel as a removal of redundant coordinates, for we shall only remove coordinates that the reader does not care about anyway, as in the above toy example.

To achieve this approach, one needs to consider the IB in a coordinate system that permits a simple reduction as in (10), and to understand its solution structure. We achieve these in Section 5 by exploiting two properties of the IB which are often overlooked. First, proceeding with the coordinates’ exchange of Section 2, the intimate relations [5,20] of the IB with RD suggest a “minimally sufficient” coordinate system for the IB, just as the x axis is for problem (10). Reducing an IB root to these coordinates is a natural extension of reduction in RD [6]. Reduction of IB roots facilitates a clean treatment of IB bifurcations. These are roughly divided into continuous and discontinuous bifurcations, in Section 5.2 and Section 5.3, respectively. While understanding continuous bifurcations is straightforward, the IB’s relations with RD allow us to understand the discontinuous bifurcation examples of which we are aware as a support switching bifurcation in RD, by leveraging [6] (Section 6). A second property is the analyticity of the IB operator (5), which stems from the analyticity of the IB Equations (2)–(4). By building on the first property, analyticity leads us to argue that the Jacobian of the IB operator (5) is generally non-singular (Conjecture 1) when considered in reduced coordinates as above. As an immediate consequence, the dynamics underlying the IB curve (1) are piecewise real-analytic in $\beta$, in a manner similar to RD. Indeed, the fact that there exist dynamics underlying the IB curve (1) in the first place can arguably be attributed to analyticity (see the discussion following Conjecture 1). Combining both properties sheds light on several subtle yet important practical caveats in solving the IB (Section 5.3) due to using finite-dimensional representations of its roots. These subtleties are compatible with our numerical experience. The results here suggest that, unlike RD, the IB is inherently infinite-dimensional, even for finite problems.

Finally, Section 6 combines the modified Euler method of Section 4 with the understanding of IB bifurcations in Section 5, to obtain an algorithm for following the path of an optimal IB root, in Section 6.1. That is, First-order Root Tracking for the IB (IBRT1). For simplicity, we focus mainly on continuous IB bifurcations, as these are the ones most often encountered in practice (see Section 6.3 on the algorithm’s handling of discontinuous bifurcations). The resulting approximations in the information plane are surprisingly close to the true IB curve (1), even on relatively sparse grids (i.e., with large step sizes), as seen in Figure 1. See Section 6.2 for the numerical results underlying the latter. The reasons for this are discussed in Section 6.3, along with the algorithm’s basic properties. Unlike BA-IB, which suffers from an increased computational cost near bifurcations, our IBRT1 algorithm suffers from a reduced accuracy there, in a manner similar to root tracking for RD [6].

With that, we note that there are standard techniques in Bifurcation Theory for handling a non-trivial kernel of $D_{\mathit{x}}F$ at a root. For example, the Lyapunov–Schmidt reduction replaces the high-dimensional problem $F=\mathbf{0}$ (6) on ${\mathbb{R}}^n$ by a smaller but equivalent problem $\Phi =\mathbf{0}$, where $\Phi (·,\beta )$ maps vectors in the (right) kernel of $D_{\mathit{x}}F$ to vectors in its left kernel. To achieve this, it separates the kernel and non-kernel directions of the problem, essentially handling each in turn; e.g., [21] (Theorem I.2.3) or [24] (Section 9.7). This technique is generic, as it does not rely on any particular property of the problem at hand. As such, it is considerably more involved than removing redundant coordinates, which requires an understanding of the solution structure. In contrast, reduction in the IB is straightforward. For the purpose of following a root’s path, carrying on with redundant kernel directions is burdensome, computationally expensive, and sensitive to approximation errors. Applying Lyapunov–Schmidt to our toy problem (10), for instance, reduces $F=\mathbf{0}$ (6) to choosing a continuously differentiable function $\Phi$ on the y-axis there (which is obtained by first solving for $x=\beta$; see the proof of Theorem I.2.3 in [21] for details). However, since y is redundant in this example, then solving for $\Phi$ can provide no useful information on the dynamics of its roots. In [19], a variant of the Lyapunov–Schmidt reduction was used to consider IB bifurcations due to symmetry breaking. While our findings are in partial agreement with theirs for continuous IB bifurcations, they differ for discontinuous bifurcations (see Section 5.2 and Section 5.3).

Notations

Vectors are written in boldface $\mathit{x}$, and scalars in regular font x. A distribution p pertaining to a particular multiplier value $\beta$ of the IB Lagrangian $\mathcal{L}$ is denoted with a subscript, $p_{\beta }$. Blahut–Arimoto’s algorithm for the IB (BA-IB) is brought below as Algorithm 1, with a single iteration over the IB Equations (2)–(4) (in steps 1.4–1.8) denoted $B{A}_{\beta }$. The probability simplex on a set S is denoted $\Delta \left[S\right]$ (see Section 5.1). The support of a probability distribution p on S is $suppp:=\{s\in S:p(s)\ne 0\}$. The source, label, and representation alphabets of an IB problem are denoted $\mathcal{X},\mathcal{Y}$, and $\hat{\mathcal{X}}$, respectively; we write $T:=|\hat{\mathcal{X}}|$. $\delta$ denotes Dirac’s delta function, ${\delta }_{i,j}=1$ if $i=j$, and zero otherwise.

Algorithm 1 Blahut–Arimoto for the Information Bottleneck (BA-IB), [1].
1:	function BA-IB($p_0\left(\hat{x}\right|x);p_{Y|X}{p}_X,\beta$)
Input:
	An initial encoder $p_0\left(\hat{x}\right|x)$, a problem definition $p\left(y\right|x\left)p\right(x)$, and $\beta >0$.
Output:
	A fixed point $p\left(\hat{x}\right|x)$ of the IB Equations (2)–(4).
2:	Initialize $i\leftarrow 0$.
3:	repeat
4:	$p_i\left(\hat{x}\right)\leftarrow {\sum }_x{p}_i\left(\hat{x}\right|x)p\left(x\right)$
5:	$p_i\left(x\right|\hat{x})\leftarrow p_i\left(\hat{x}\right|x)p\left(x\right)/p_i\left(\hat{x}\right)$
6:	$p_i\left(y\right|\hat{x})\leftarrow {\sum }_{x}p\left(y\right|x)p_i\left(x\right|\hat{x})$
7:	$Z_i(x,\beta )\leftarrow {\sum }_{\hat{x}}{p}_i\left(\hat{x}\right)exp\left\{-\beta D_{KL}\left[p\left(y\right|x)\left|\right|p_i\left(y\right|\hat{x})\right]\right\}$
8:	$p_{i+1}\left(\hat{x}\right|x)\leftarrow \frac{p_i\left(\hat{x}\right)}{Z_i(x,\beta )}exp\left\{-\beta D_{KL}\left[p\left(y\right|x)\left|\right|p_i\left(y\right|\hat{x})\right]\right\}$
9:	$i\leftarrow i+1$
10:	until convergence.
11:	end function

2. Coordinates Exchange for the IB

Just as a point in the plane can be described by different coordinate systems, so can IB roots. As demonstrated recently by [6] for the related rate-distortion theory, picking the right coordinates matters when analyzing its bifurcations. The same holds also for the IB. Our primary motivations for exchanging coordinates are to reduce computational costs and to mod-out irrelevant kernel directions, as explained in Section 1. In this Section, we discuss three natural choices of a coordinate system for parameterizing IB roots and the reasoning behind our choice for the sequel before setting to derive the IB ODE in the following Section 3. This work is complemented by the later Section 5.1, which facilitates a transparent analysis of IB bifurcations.

IB roots have been classically parameterized in the literature by (direct) encoders $p\left(\hat{x}\right|x)$, following [1]. Considering the BA-IB Algorithm 1 reveals two other natural choices, illustrated by Equation (11) below. First, an encoder $p\left(\hat{x}\right|x)$ determines a cluster marginal $p\left(\hat{x}\right)$ and an inverse encoder $p\left(x\right|\hat{x})$, via steps 4 and 5 of Algorithm 1 (denoted 1.4 and 1.5, for short), respectively. These can be interpreted geometrically as $p\left(\hat{x}\right)$-weighted points $q_{\hat{x}}\left(x\right)$ in the simplex $\Delta \left[\mathcal{X}\right]$ of X, so long as they are well-defined, $\forall \hat{x}p\left(\hat{x}\right)\ne 0$. No more than $\left|\mathcal{X}\right|+1$ points in the simplex are required to represent an IB root [2]. The latter is readily seen to analyze the IB in these coordinates, although it pre-dates [1] and has generally escaped broader attention. Second, an inverse encoder determines a decoder $p\left(y\right|\hat{x})$, via step 6. Along with the cluster marginal, $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ can be similarly interpreted as $p\left(\hat{x}\right)$-weighted points $r_{\hat{x}}\left(y\right)$ in the simplex $\Delta \left[\mathcal{Y}\right]$ of Y. This choice of coordinates is implied already by Theorem 5 in [1]. Cycling around Equation (11), a decoder $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ determines via steps 7 and 8 a new encoder, which may differ from the one with which we have started. For notational simplicity, we shall usually write $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ rather than $\left(r_{\hat{x}}\left(y\right),p\left(\hat{x}\right)\right)$ for decoder coordinates (similarly for inverse encoder coordinates). [9:04] Leona Kong

(11)

The above allows us to define three BA operators as the composition of three consecutive maps in Equation (11), encoding an iteration of Algorithm 1. When starting at an encoder $p\left(\hat{x}\right|x)$, its output is a newly defined encoder. Similarly, when starting at one of the other two vertices, it sends an inverse encoder $\left(p\left(x\right|\hat{x}),p\left(\hat{x}\right)\right)$ or a decoder pair $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ to a newly defined one. By abuse of notation, we denote all three compositions by $B{A}_{\beta }$, with the choice of coordinate system mentioned accordingly. Indeed, these are representations of a single BA-IB iteration in three different coordinate systems, and so may be considered as distinct representations of the same operator. For completeness, $B{A}_{\beta }$ in decoder coordinates is spelled out explicitly in Equation (A1) in Appendix A. A newly defined encoder (or inverse encoder or decoder) at a cycle’s completion need not generally equal the one at which we have started. These are equal precisely at IB roots, when the IB Equations (2)–(4) hold. Therefore, the choice of a coordinate system does not matter then, and so moving around Equation (11) from one vertex to another yields different parameterizations of the same root, at least when $\forall \hat{x}p\left(\hat{x}\right)\ne 0$. In particular, this shows that the inverse encoders $q_{\hat{x}}$ in $\Delta \left[\mathcal{X}\right]$ of an IB root are in bijective correspondence with its decoders $r_{\hat{x}}$ in $\Delta \left[\mathcal{Y}\right]$, an observation which shall come in handy in Section 5.

Next, we consider how well each of these coordinate systems can serve for following the path of an IB root. The minimal number of symbols $\hat{x}$ needed to write down an IB root typically varies with the constraints, cf., [1] (Section 3.4) or [2] (Section II.A). Therefore, inverse encoder and decoder coordinates are better suited than encoder coordinates for considering the dynamics of a root with $\beta$, as they allow us to consider its evolution via a varying number of points in a fixed space, $\Delta \left[\mathcal{X}\right]$ or $\Delta \left[\mathcal{Y}\right]$, respectively. Indeed, a direct encoder $p\left(\hat{x}\right|x)$ can be interpreted geometrically as a point in the $\left|\mathcal{X}\right|$-fold product $\Delta {\left[\hat{\mathcal{X}}\right]}^{\mathcal{X}}$ of simplices $\Delta \left[\hat{\mathcal{X}}\right]$ [9] (Section 2). So, if a particular symbol ${\hat{x}}^{\prime }$ is not in use anymore, $p\left({\hat{x}}^{\prime }\right)=0$, then one is forced to choose between replacing $\Delta \left[\hat{\mathcal{X}}\right]$ by a smaller space $\Delta [\hat{\mathcal{X}}\setminus \left\{{\hat{x}}^{\prime }\right\}]$ or carrying on with a redundant symbol ${\hat{x}}^{\prime }$. The latter leads to non-trivial kernels in the IB due to duplicate clusters (e.g., Section 3.1 there), making it difficult to tell whether a particular kernel direction pertains to a bifurcation (or to a “perpetual kernel” [9,19]). In contrast, when considered in decoder coordinates, for example, an IB root is nothing but $p\left(\hat{x}\right)$-weighted paths $r_1,\cdots ,r_T$ in $\Delta \left[\mathcal{Y}\right]$, with $\beta \mapsto r_{\hat{x}}\left(\beta \right)$ a path for each $\hat{x}$. And so, once a symbol ${\hat{x}}^{\prime }$ is not needed anymore, then one can discard the path $r_{{\hat{x}}^{\prime }}$ without replacing the underlying space $\Delta \left[\mathcal{Y}\right]$. This permits the clean treatment of IB bifurcations in Section 5.

The computational cost of solving a first-order ODE as in (7) numerically in $\frac{d\mathit{x}}{d\beta }$ depends on $dim\mathit{x}$. Much of this cost is due to computing a linear pre-image under $D_{\mathit{x}}F$, which is of order $O{\left(\dim \mathit{x}\right)}^3$ [25] (Section 28.4); cf., Section 6. Representing an IB root on T clusters in encoder coordinates requires $\left|\mathcal{X}\right|·T$ dimensions (ignoring normalization constraints), in inverse encoder coordinates $\left(\right|\mathcal{X}|+1)·T$ dimensions, and in decoder coordinates $\left(\right|\mathcal{Y}|+1)·T$ dimensions. Thus, the computational cost is lowest in decoder coordinates, at least when there are fewer possible labels $\mathcal{Y}$ than input symbols $\mathcal{X}$.

A priori, one might expect that derivatives with respect to $\beta$ vanish when the solution barely changes, regardless of the choice of coordinate system. For example, at a very large “$\beta =\infty$” value, an obvious IB root is the diagonal encoder (setting $\hat{\mathcal{X}}:=\mathcal{X}$ and $p\left(\hat{x}\right|x):={\delta }_{x,\hat{x}}$), as can be seen by a direct examination of the IB Equations (2)–(4). It consists of one IB cluster of weight (or mass) $p\left(x\right)$ at $p_{Y|X=x}\in \Delta \left[\mathcal{Y}\right]$ for each $x\in \mathcal{X}$, and so one might expect that it would barely change so long as $\beta$ is very large. However, the logarithmic derivative $\frac{dlog{p}_{\beta }\left(\hat{x}\right|x)}{d\beta }$ in encoder coordinates need not vanish even when the derivatives $\frac{dlog{p}_{\beta }\left(y\right|\hat{x})}{d\beta }$ and $\frac{dlog{p}_{\beta }\left(\hat{x}\right)}{d\beta }$ in decoder coordinates do (see Section 3 on logarithmic coordinates), as seen to the right of Figure 2. Indeed, given the derivative in decoder coordinates, one can exchange it to encoder coordinates by

$$\begin{array}{cc}\hfill \frac{dlog{p}_{\beta }\left(\hat{x}\right|x)}{d\beta }=& J_{\mathrm{dec}}^{\mathrm{enc}}\frac{dlog{p}_{\beta }\left(y^{\prime }\right|{\hat{x}}^{\prime })}{d\beta }+J_{\mathrm{mrg}}^{\mathrm{enc}}\frac{dlog{p}_{\beta }\left({\hat{x}}^{\prime }\right)}{d\beta }\hfill \\ & -D_{KL}\left[p\left(y\right|x)\left|\right|p_{\beta }\left(y\right|\hat{x})\right]+{\displaystyle \sum _{{\hat{x}}^{\prime \prime }}}{p}_{\beta }\left({\hat{x}}^{\prime \prime }\right|x)D_{KL}\left[p\left(y\right|x)\left|\right|p_{\beta }\left(y\right|{\hat{x}}^{\prime \prime })\right],\hfill \end{array}$$

(12)

where $J_{\mathrm{dec}}^{\mathrm{enc}}$ and $J_{\mathrm{mrg}}^{\mathrm{enc}}$ are the two coordinate exchange Jacobian matrices of orders $(T·|\mathcal{X}\left|\right)\times (T·|\mathcal{Y}\left|\right)$ and $(T·|\mathcal{X}\left|\right)\times T$, respectively, given by Equations (A68) and (A70) in Appendix B.4.2. And so, $\frac{dlog{p}_{\beta }\left(\hat{x}\right|x)}{d\beta }$ would often be non-zero even if both $\frac{dlog{p}_{\beta }\left(y\right|\hat{x})}{d\beta }$ and $\frac{dlog{p}_{\beta }\left(\hat{x}\right)}{d\beta }$ vanish. This unintuitive behavior of the derivative in encoder coordinates is due to the explicit dependence of the IB’s encoder Equation (2) on $\beta$. This dependence is the source of the last two terms in Equation (12) (see Equation (A73)). The comparison between encoder and inverse encoder coordinates can be seen to be similar. See Appendix B.4 for further details.

In light of the above, we proceed with decoder coordinates in the sequel.

3. Implicit Derivatives at an IB Root and the IB’s ODE

We now specialize the implicit ODE (7) (of Section 1) to the IB, using the decoder coordinates of the previous Section 2. This allows us to compute first-order implicit derivatives at an IB root (Theorem 1) with remarkable accuracy, under one primary assumption—that the root is a differentiable function of $\beta$. While differentiability breaks at IB bifurcations (Section 5), this allows us to reconstruct a solution path from its local approximations in the following Section 4, so long as it holds.

To simplify calculations, we take the logarithm $\left(logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)\right)$ of the decoder coordinates of Section 2 as our variables. Exchanging the $B{A}_{\beta }$ operator to log-decoder coordinates is immediate, by writing $logB{A}_{\beta }[exp\left(logp\left(y\right|\hat{x})\right),exp\left(logp\left(\hat{x}\right)\right)]$. For short, we denote it $B{A}_{\beta }[logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)]$ when in these coordinates, by abuse of notation. Similarly, exchanging the IB ODE (below) back to non-logarithmic coordinates is immediate, via $\frac{d}{d\beta }logp=\frac{1}{p}\frac{d}{d\beta }p$. In Section 6, we shall assume that $p\left(\hat{x}\right)$ never vanishes. To ensure that taking logarithms is well-defined, we require that no decoder coordinate $p\left(y\right|\hat{x})$ vanishes (while it may have a well-defined derivative $\frac{d}{d\beta }p\left(y\right|\hat{x})$ even with a vanished coordinate, calculation details would differ). A sufficient condition for that is that $p\left(y\right|x)>0$ for every x and y (Lemma A1 in Appendix A).

Next, define a variable $\mathit{x}\in {\mathbb{R}}^{T·\left(\right|\mathcal{Y}|+1)}$ as the concatenation of the vector ${\left(log{p}_{\beta }\left(y\right|\hat{x})\right)}_{y\in \mathcal{Y},\hat{x}\in \hat{\mathcal{X}}}$ with ${\left(log{p}_{\beta }\left(\hat{x}\right)\right)}_{\hat{x}\in \hat{\mathcal{X}}}$. Differentiating $\partial /\partial logp$ with respect to log-probabilities is given by $p·\frac{\partial }{\partial p}$, by the chain rule (setting $u:=logp$, $\frac{df\left(p\right)}{du}=\frac{df}{dp}\frac{dp}{du}$, or equivalently $\frac{df}{dlogp}=p·\frac{df}{dp}$; see Appendix B.1 for a gentler treatment). This gives meaning to the Jacobian matrix $D_{\mathit{x}}(·)$ with respect to our logarithmic variable $\mathit{x}$. The Jacobian $D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ of a single Blahut–Arimoto iteration in these log-decoder coordinates is a square matrix of order $T·\left(\right|\mathcal{Y}|+1)$. Its $(T·|\mathcal{Y}\left|\right)\times (T·|\mathcal{Y}\left|\right)$ upper-left block (below) corresponds to perturbations of BA’s output log-decoder $logp\left(y\right|\hat{x})$ due to varying an input log-decoder $logp\left(y^{\prime }\right|{\hat{x}}^{\prime })$. Since we prime input but not output coordinates, this is to say that the columns of this block are indexed by pairs $(y^{\prime },{\hat{x}}^{\prime })$ and its rows by $(y,\hat{x})$ (one could also enumerate $\mathcal{Y}:=\{y_1,\cdots ,y_{\left|\mathcal{Y}\right|}\}$ and $\hat{\mathcal{X}}:=\{{\hat{x}}_1,\cdots ,{\hat{x}}_T\}$ explicitly, replacing $(y,\hat{x})$ and $(y^{\prime },{\hat{x}}^{\prime })$ throughout by $(y_i,{\hat{x}}_j)$ and $(y_k,{\hat{x}}_l)$, respectively, with $i,k=1,\cdots ,\left|\mathcal{Y}\right|$ and $j,l=1,\cdots ,T$). Its $(T·|\mathcal{Y}\left|\right)\times T$ upper-right block corresponds to perturbations in BA’s output log-decoder $logp\left(y\right|\hat{x})$ due to varying an input log-marginal $logp\left({\hat{x}}^{\prime }\right)$. That is, its columns are indexed by ${\hat{x}}^{\prime }$ and rows by $(y,\hat{x})$. Similarly, for the bottom-left and bottom-right blocks, of respective sizes $T\times (T·|\mathcal{Y}\left|\right)$ and $T\times T$. See (A25) ff., in Appendix B.2, and the end-result at Equation (A44) there. Explicitly, when evaluated at an IB root $\left(logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)\right)$, BA’s Jacobian matrix is given by

$$\begin{array}{c}{D}_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }[logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)]=\hfill \\ \left(\begin{array}{cc}\beta ·\sum _{{\hat{x}}^{″},y^{″}}({\delta }_{{\hat{x}}^{″},{\hat{x}}^{\prime }}-{\delta }_{\hat{x},{\hat{x}}^{\prime }})·[1-\frac{{\delta }_{y^{″},y}}{p_{\beta }\left(y\right|\hat{x})}]C{(\hat{x},{\hat{x}}^{″};\beta )}_{y^{\prime },y^{″}}\hfill & \left(1-\beta \right)·\sum _{y^{″}}[1-\frac{{\delta }_{y^{″},y}}{p_{\beta }\left(y\right|\hat{x})}]B{(\hat{x},{\hat{x}}^{\prime };\beta )}_{y^{″}}\hfill \\ \beta ·[{\delta }_{\hat{x},{\hat{x}}^{\prime }}{p}_{\beta }\left(y^{\prime }\right|\hat{x})-B{(\hat{x},{\hat{x}}^{\prime };\beta )}_{y^{\prime }}]& (1-\beta )·[{\delta }_{\hat{x},{\hat{x}}^{\prime }}-A(\hat{x},{\hat{x}}^{\prime };\beta )]\end{array}\right)\hfill \end{array}$$

(13)

where ${\delta }_{i,j}=1$ if $i=j$ and is 0 otherwise. As mentioned above, primed coordinates $y^{\prime }$ and ${\hat{x}}^{\prime }$ index the columns, and un-primed coordinates y and $\hat{x}$ the rows. Indices $y^{″}$ and ${\hat{x}}^{″}$ with more than a single prime are summation variables. $A,B$, and C are a scalar, a vector, and a matrix, each involving two IB clusters. They are defined by

$$\begin{array}{cc}\hfill A(\hat{x},{\hat{x}}^{\prime };\beta ):=& {\displaystyle \sum _{x^{\prime \prime }}}{p}_{\beta }\left({\hat{x}}^{\prime }\right|x^{\prime \prime })p_{\beta }\left(x^{\prime \prime }\right|\hat{x}),\hfill \\ \hfill B{(\hat{x},{\hat{x}}^{\prime };\beta )}_y:=& {\displaystyle \sum _{x^{\prime \prime }}}p\left(y\right|x^{\prime \prime })p_{\beta }\left({\hat{x}}^{\prime }\right|x^{\prime \prime })p_{\beta }\left(x^{\prime \prime }\right|\hat{x}),\mathrm{and}\hfill \\ \hfill C{(\hat{x},{\hat{x}}^{\prime };\beta )}_{y,y^{\prime }}:=& {\displaystyle \sum _{x^{\prime \prime }}}p\left(y\right|x^{\prime \prime })p\left(y^{\prime }\right|x^{\prime \prime })p_{\beta }\left({\hat{x}}^{\prime }\right|x^{\prime \prime })p_{\beta }\left(x^{\prime \prime }\right|\hat{x}).\hfill \end{array}$$

(14)

In these, y indexes B and the rows of C, $y^{\prime }$ the columns of C, and $x^{″}$ is a summation variable. These $(\hat{x},{\hat{x}}^{\prime })$-labeled tensors have only $\left|\mathcal{Y}\right|$ entries along each axis, thanks to our choice of decoder coordinates. A and B can be expressed in terms of C via some obvious relations; see Equation (A32) and below in Appendix B.2. Appendix B.1 elaborates on the mathematical subtleties involved in calculating the Jacobian (13). See also Equation (A45) in Appendix B.2 for an implementation-friendly form of (13).

Together with $D_{\beta }B{A}_{\beta }$ (Equations (A58) and (A57) in Appendix B.3), we have both of the first-order derivative tensors of $B{A}_{\beta }$ in log-decoder coordinates. This allows us to specialize the implicit ODE (7) (of Section 1) to the IB, in terms of our variable $\mathit{x}$. By abuse of notation, we write ${\left(log{p}_{\beta }\left(y\right|\hat{x}),log{p}_{\beta }\left(\hat{x}\right)\right)}_{y,\hat{x}}$ for its $\left|\mathcal{Y}\right|·T+T$ coordinates, and similarly for its derivatives vector $\mathit{v}$ (15) below.

Theorem 1

(The IB’s ODE). Let $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ be an IB root, and suppose that it can be written as a differentiable function $\beta \mapsto \left(p_{\beta }\left(y\right|\hat{x}),p_{\beta }\left(\hat{x}\right)\right)$ in β. If none of its coordinates vanish, then the vector

$$\mathit{v}:={\left(\frac{dlog{p}_{\beta }\left(y\right|\hat{x})}{d\beta },\frac{dlog{p}_{\beta }\left(\hat{x}\right)}{d\beta }\right)}_{y,\hat{x}}$$

(15)

of its implicit logarithmic derivatives is well-defined and satisfies an ordinary differential equation in β,

$$\left(I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }\right)\mathit{v}=\left(\begin{array}{c}-\sum _{x,{\hat{x}}^{\prime \prime }}[1-\frac{p\left(y\right|x)}{p_{\beta }\left(y\right|\hat{x})}]·[{\delta }_{\hat{x},{\hat{x}}^{\prime \prime }}-p_{\beta }\left({\hat{x}}^{\prime \prime }\right|x)]p_{\beta }\left(x\right|\hat{x})D_{KL}\left[p\left(y\right|x)\left|\right|p_{\beta }\left(y\right|{\hat{x}}^{\prime \prime })\right]\\ \\ \sum _{x,{\hat{x}}^{\prime \prime }}[{\delta }_{\hat{x},{\hat{x}}^{\prime \prime }}-p_{\beta }\left({\hat{x}}^{\prime \prime }\right|x)]p_{\beta }\left(x\right|\hat{x})D_{KL}\left[p\left(y\right|x)\left|\right|p_{\beta }\left(y\right|{\hat{x}}^{\prime \prime })\right]\end{array}\right)$$

(16)

where I is the identity matrix of order $T·\left(\right|\mathcal{Y}|+1)$, and the Jacobian matrix $D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ at the given IB root is given by Equation (13). The right-hand side of (16) is indexed as in (15), by $(y,\hat{x})$ at its top and $\hat{x}$ at its bottom coordinates.

While the IB ODE was discovered by [10], it is derived here anew in log-decoder coordinates due to the considerations in Section 2. It is analogous to the RD ODE, due to [6]; Corollary 1 and around (in Section 5.1) provides a relation between these two ODEs. We emphasize that the first assumption of Theorem 1, that the IB root is a differentiable function of $\beta$, is essential. It consists of two parts: (i) that the root can be written as a function of $\beta$, and (ii) that this function is differentiable. These are precisely the assumptions needed to compute the first-order implicit multivariate derivative $\mathit{v}$ (15) at the given root [6] (Section 2.1). Continuous IB bifurcations violate (ii) (Section 5.2), while discontinuous ones violate (i) (Section 5.3). In contrast, the requirement that no coordinate vanishes is a technical one, due to our choice of logarithmic coordinates.

It is not necessary for the Jacobian of the IB operator (5) (to the left of (16)) to be non-singular in order to solve the IB ODE numerically. Nevertheless, non-singularity of the Jacobian will follow from the sequel (see Conjecture 1 in Section 5). With that, the derivatives $\mathit{v}=\frac{d}{d\beta }\left(log{p}_{\beta }\left(y\right|\hat{x}),log{p}_{\beta }\left(\hat{x}\right)\right)$ (15) computed numerically from the IB ODE (16) at an exact root are remarkably accurate, as demonstrated in Figure 3. As in RD [6], calculating implicit derivatives numerically loses its accuracy when approaching a bifurcation because the Jacobian is increasingly ill-conditioned there. For comparison, the BA-IB Algorithm 1 also loses its accuracy near a bifurcation. This is a consequence of BA’s critical slowing down [4], just as with its corresponding RD variant.

Each coordinate of $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ is treated by the IB ODE (16) as an independent variable. However, the normalization of $p\left(y\right|\hat{x})$ imposes one constraint per cluster $\hat{x}$ (and one for the normalization of $p\left(\hat{x}\right)$). Thus, one might expect the behavior of BA’s Jacobian (13) to be determined by fewer than $T·\left(\left|\mathcal{Y}\right|+1\right)$ coordinates, at least qualitatively. This intuition is justified by the following Lemma 1, which allows us to consider the kernel of the IB operator (5) by a smaller and simpler matrix S; see Appendix C for its proof.

Lemma 1.

Given an IB root as above, define a square matrix of order $T·\left|\mathcal{Y}\right|$ by

$$S_{(y,\hat{x}),(y^{\prime },{\hat{x}}^{\prime })}:=\sum _x{p}_{\beta }\left(x\right|\hat{x})[\beta ·\frac{p\left(y\right|x)}{p_{\beta }\left(y\right|\hat{x})}+\left(1-2\beta \right)]p\left(y^{\prime }\right|x)[{\delta }_{\hat{x},{\hat{x}}^{\prime }}-p_{\beta }\left({\hat{x}}^{\prime }\right|x)].$$

(17)

Then, the nullity of the Jacobian $I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ of the IB operator (5) equals that of $I-S$, where I is the identity matrix (of the respective order), and S is defined by (17),

$$dimker\left(I-S\right)=dimker\left(I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }\right).$$

(18)

Specifically, write $\mathit{v}:={\left(v_{y,\hat{x}}\right)}_{y,\hat{x}}$ for a left eigenvector which corresponds to $1\in eigS$. Then, there is a bijective correspondence between the left kernels at both sides of (18), mapping

$$\mathit{v}\mapsto (\mathit{v},\mathit{u}),$$

(19)

where $\mathit{u}:={\left(u_{\hat{x}}\right)}_{\hat{x}}$ is defined by $u_{\hat{x}}:=\frac{1-\beta }{\beta }·{\sum }_y{v}_{y,\hat{x}}$.

In addition to offering a form more transparent than BA’s Jacobian in (13), Lemma 1 also reduces the computational cost of testing $I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ (16) for singularity, by using the smaller $I-S$ (17) in its place. This makes it easier to detect upcoming bifurcations (see Conjecture 1 in Section 5). Further, one can verify directly that the IB ODE (16) indeed follows the right path. Indeed, if the ODE is non-singular, then, by the Implicit Function Theorem, there is (locally) a unique IB root, which is a differentiable function of $\beta$. And so, there is a unique solution path for a numerical approximation to follow. Finally, we note that a relation similar to (18) holds also for eigenvalues of $D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ (13) other than 1. This can be seen either empirically or by tracing the proof of Lemma 1.

In Section 5, we shall proceed with this line of thought of removing redundant coordinates. In the following Section 4, we turn to reconstruct a solution path from implicit derivatives at a point, with bifurcations ignored for now.

4. A Modified Euler Method for the IB

We follow the path of a given IB root away from bifurcation by using its implicit derivatives computed from the IB ODE (16), of Section 3. We follow the classic Euler method for simplicity, modifying it slightly to get the most out of the calculated derivatives. Improvements using more sophisticated numerical methods are left to future work. The detection and handling of IB bifurcations are deferred to the following Section 5, and thus are ignored in this section.

Let $\frac{d\mathit{x}}{d\beta }=f(\mathit{x},\beta )$ and $\mathit{x}\left({\beta }_0\right)={\mathit{x}}_0$ define an initial value problem. In numerical approximations of ordinary differential equations (ODEs), the Euler method for this problem is defined by setting

$${\mathit{x}}_{n+1}:={\mathit{x}}_n+\Delta \beta ·f({\mathit{x}}_n,{\beta }_n),$$

(20)

where ${\beta }_{n+1}:={\beta }_n+\Delta \beta$, and $|\Delta \beta |$ is the step size. The global truncation error ${max}_n{\parallel {\mathit{x}}_n-\mathit{x}\left({\beta }_n\right)\parallel }_{\infty }$ is the largest error of the approximations ${\mathit{x}}_n$ from the true solutions $\mathit{x}\left({\beta }_n\right)$. A numerical method for solving ODEs is said to be of order d if its global truncation error is of order $O\left(\right|\Delta \beta {|}^d)$, for step sizes $|\Delta \beta |$ small enough. Euler’s method error analysis is a standard result, provided as Theorem 2 below. See [26] (Theorem 212A) or [27] (Theorem 2.4), for example. It shows that Euler’s method (20) is of order $d=1$, under mild assumptions, as demonstrated in Figure 4. The immediate generalization of (20) using derivatives until order d is Taylor’s method, which is a method of order d.

Theorem 2

(Euler’s method error analysis). Let an initial value problem be defined on $[{\beta }_0,{\beta }_f]$ by $\frac{d\mathit{x}}{d\beta }=f(\mathit{x},\beta )$ as above (with ${\mathit{x}}_0$ allowed to deviate from $\mathit{x}\left({\beta }_0\right)$), and suppose that f satisfies the Lipschitz condition with some constant $L>0$. Namely, $\parallel f(\mathit{x},\beta )-f({\mathit{x}}^{\prime },\beta ){\parallel }_{\infty }\le L·{\parallel \mathit{x}-{\mathit{x}}^{\prime }\parallel }_{\infty }$ for every $\mathit{x},{\mathit{x}}^{\prime }$ and $\beta \in [{\beta }_0,{\beta }_f]$.

Then, Euler’s method (20) global truncation error satisfies

$$\underset{{\beta }_0\le {\beta }_n\le {\beta }_f}{max}\parallel {\mathit{x}}_n-\mathit{x}\left({\beta }_n\right){\parallel }_{\infty }\le e^{({\beta }_f-{\beta }_0)L}\parallel {\mathit{x}}_0-\mathit{x}\left({\beta }_0\right){\parallel }_{\infty }+\frac{e^{({\beta }_f-{\beta }_0)L}-1}{L}·\frac{1}{2}|\Delta \beta |\underset{{\beta }_0\le \beta \le {\beta }_f}{max}{∥\frac{d^2\mathit{x}\left(\beta \right)}{d{\beta }^2}∥}_{\infty }.$$

(21)

Specializing Euler’s method to our needs, replace $\mathit{x}$ in (20) above by the log-decoder coordinates of an IB root, as in Section 3. So long as an IB root ${\mathit{p}}_{\beta }:=\left(p_{\beta }\left(y\right|\hat{x}),p_{\beta }\left(\hat{x}\right)\right)$ is a differentiable function of $\beta$ in the vicinity of ${\beta }_n$, it can be approximated by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill log{p}_{{\beta }_{n+1}}\left(y\right|\hat{x})& \approx log{p}_{{\beta }_n}\left(y\right|\hat{x})+\Delta \beta ·\frac{dlog{p}_{\beta }\left(y\right|\hat{x})}{d\beta }{|}_{{\mathit{p}}_{{\beta }_n}}\mathrm{and}\hfill \\ \hfill log{p}_{{\beta }_{n+1}}\left(\hat{x}\right)& \approx log{p}_{{\beta }_n}\left(\hat{x}\right)+\Delta \beta ·\frac{dlog{p}_{\beta }\left(\hat{x}\right)}{d\beta }{|}_{{\mathit{p}}_{{\beta }_n}},\hfill \end{array}\end{array}$$

(22)

where $\frac{dlog{p}_{\beta }\left(y\right|\hat{x})}{d\beta }$ and $\frac{dlog{p}_{\beta }\left(\hat{x}\right)}{d\beta }$ are calculated from the IB ODE (16). Thus, applying (22) repeatedly, we obtain an Euler method for the IB. We shall take only negative steps $\Delta \beta <0$ when approximating the IB, due to reasons explained in Section 5.3 (after Proposition 1). In contrast to the BA-IB Algorithm 1, Euler’s method (22) can be used to interpolate intermediate points, yielding a piecewise linear approximation of the root.

The problem of tracking an operator’s root belongs in general to a family of hard-to-solve numerical problems—known as stiff—if the problem has a bifurcation [6] (Section 7.2). See [26] or [27] for example on stiff differential equations. Stopping early in the vicinity of a bifurcation restricts the computational difficulty and permits convergence guarantees. Early stopping in the IB shall be handled later, in Section 5.2. [6] (Theorem 5) proves that Euler’s method convergence guarantees (Theorem 2) hold for the closely related Euler method for RD with early stopping. While Euler’s method may inadvertently switch between solution branches of the IB ODE (16), the latter guarantees ensure that it indeed follows the true solution path between bifurcations, if the step size $|\Delta \beta |$ is small enough and initializing close enough to the true solution (see Section 5 and Section 6.3 on the distinction between IB bifurcations and singularities of the IB ODE (16)). Although we do not dive into these details for brevity, we note that similar convergence guarantees can also be proven here. Alternatively, Euler’s method can be ensured to follow the true solution path by noting that an optimal IB root is (strongly) stable when negative steps $\Delta \beta <0$ are taken; these details are deferred to Section 6.3, as they depend on Section 5.

Following the discussion in Section 2, there is a subtle disadvantage in choosing decoder coordinates as our variables compared to the other two coordinate systems there. Indeed, recall that the IB is defined as a maximization over Markov chains $Y⟷X⟷\hat{X}$. An (arbitrary) encoder $p\left(\hat{x}\right|x)$ defines a joint probability distribution $p\left(\hat{x}\right|x)p\left(y\right|x)p\left(x\right)$ which is Markov. An inverse encoder pair also similarly defines a Markov chain. In contrast, an arbitrary decoder pair $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ need not necessarily define a Markov chain. Rather, by invoking the error analysis of Euler’s method, one can see that Markovity is approximated at an increasingly improved quality as the step-size $|\Delta \beta |$ in (22) becomes smaller. To enforce Markovity, we shall perform a single BA iteration (in decoder coordinates) after each Euler method step. This ensures that the newly generated decoder pair satisfies the Markov condition, as it is now generated from an encoder.

As a side effect, adding a single BA-IB iteration after each Euler method step improves the approximation’s quality significantly. By linearizing $B{A}_{\beta }$ around a fixed point, one can show that deterministic annealing with a fixed number of BA iterations per grid point is a first-order method. Thus, deterministic annealing may arguably be considered a first-order method, as is with Euler’s method. A similar argument shows that adding a single BA iteration after each Euler method step yields a second-order method. However, while a larger number of added BA iterations obviously improves the approximation’s quality, it does not improve the method’s order. See Appendix D for an approximate error analysis. The predicted orders are in good agreement with the ones found empirically, shown in Figure 4. We note that while [6] did not attempt an added BA iteration, they do discuss a variety of other improvements to root tracking (see Section 3.4 in [6]).

5. On IB Bifurcations

For the IB Equations (2)–(4) to exhibit a bifurcation, it is necessary that the Jacobian of the IB operator (5) be singular, as illustrated by Figure 5. However, a priori singularity is not sufficient to detect a bifurcation (cf., Section 3.1 in [9]), nor does this allow one to distinguish between bifurcations of different types. At an IB root, singularities of the IB ODE (16) (Section 3) coincide with those of $Id-B{A}_{\beta }$ (5) (in log-decoder coordinates). Thus, in order to be able to exploit the IB ODE (16), we shall now take a closer look into IB bifurcations. These can be broadly classified into two types: where an optimal root is continuous in $\beta$ and where it is not. As noted after Theorem 1, each type violates an assumption necessary to compute implicit derivatives. Section 5.2 and Section 5.3 provide the means to identify bifurcations, distinguish between their types, and handle them accordingly, mainly for continuous bifurcations. To facilitate the discussion, Section 5.1 considers the IB as a rate-distortion problem, following [20] and others. This allows us to leverage recent insights on RD bifurcations [6], while suggesting a “minimally sufficient” choice of coordinates for the IB. The latter permits a clean treatment of continuous IB bifurcations in Section 5.2. Viewing the IB as an infinite-dimensional RD problem facilitates the understanding of its discontinuous bifurcations, which in turn highlight subtleties in its finite-dimensional coordinate systems (of Section 2). These provide insight into the IB and are also of practical implications (Section 5.3), and so are necessary for our algorithms in Section 6.

5.1. The IB as a Rate-Distortion Problem

We now explore the intimate relation between the IB and RD, following [5,20]. This leads to a “minimally sufficient” coordinate system for the IB, thereby completing the work of Section 2. In this coordinate system, results [6] on the dynamics of RD roots are readily considered in the IB context. This leads to Conjecture 1, that the IB operator (5) in these coordinates is typically non-singular. The discussion here facilitates the treatment of IB bifurcations in the following Section 5.2 and Section 5.3.

First, recall a few definitions. A rate distortion problem on a source alphabet $\mathcal{X}$ and a reproduction alphabet $\hat{\mathcal{X}}$ is defined by a distortion measure$d:\mathcal{X}\times \hat{\mathcal{X}}\to {\mathbb{R}}_{\ge 0}$ (a non-negative function on $\mathcal{X}\times \hat{\mathcal{X}}$ with no further requirements—see Section 2.2 in [28]) and a source distribution $p_X\left(x\right)$. One seeks the minimal rate $I(X;\hat{X})$ subject to a constraint D on the expected distortion $\mathbb{E}\left[d\right(x,\hat{x}\left)\right]$ [29,30],

$$R\left(D\right):=\underset{p\left(\hat{x}\right|x)}{min}\left\{I(X;\hat{X}):{\mathbb{E}}_{p\left(\hat{x}\right|x)p_X\left(x\right)}\left[d\right(x,\hat{x}\left)\right]\le D\right\},$$

(23)

known as the rate-distortion curve. The minimization is over test channels $p\left(\hat{x}\right|x)$. A test channel that attains the RD curve (23) is called an achieving distribution. We say that an RD problem is finite if both of the alphabets $\mathcal{X}$ and $\hat{\mathcal{X}}$ are finite. Using Lagrange multipliers for (23) with $I(X;\hat{X})+\beta \mathbb{E}\left[d(x,\hat{x})\right]$ (normalization omitted for clarity), one obtains a pair of fixed-point equations

$$p\left(\hat{x}\right|x)=\frac{p\left(\hat{x}\right)e^{-\beta d(x,\hat{x})}}{{\sum }_{\hat{x}}p\left(\hat{x}\right)e^{-\beta d(x,\hat{x})}}\mathrm{and}p\left(\hat{x}\right)=\sum _{x}p\left(\hat{x}\right|x)p\left(x\right)$$

(24)

in the marginal $p\left(\hat{x}\right)$ and test channel $p\left(\hat{x}\right|x)$, similar to the IB Equations (2) and (4). Iterating over these is Blahut’s algorithm for RD [8], denoted $B{A}_{\beta }^{RD}$ here. As with the IB (1), $\beta$ parameterizes the slope of the optimal curve (23) also for RD. See [28] or [31] for an exposition of rate-distortion theory.

We clarify a definition needed to rewrite the IB as an RD problem. We define the simplex $\Delta \left[S\right]$ on a (possibly infinite) set S as the collection of finite formal convex combinations ${\sum }_s{a}_s·s$ of elements of S. That is, as the S-indexed vectors ${\left(a_s\right)}_{s\in S}$ (equivalently, as functions mapping each s in S to a real number $a_s$) that satisfy ${\sum }_s{a}_s=1$ and $a_s\ge 0$, with $a_s$ non-zero for only finitely many elements s (the support of ${\left(a_s\right)}_s$). Addition and multiplication are defined pointwise, as in ${\sum }_s{a}_s·s+{\sum }_s{b}_s·s={\sum }_s(a_s+b_s)·s$. $\Delta \left[S\right]$ is closed under finite convex combinations because the sum of finitely supported vectors is finitely supported. When taking $S=\{{\mathit{e}}_1,\cdots ,{\mathit{e}}_n\}$ the standard basis vectors ${\left({\mathit{e}}_i\right)}_j={\delta }_{i,j}$ of ${\mathbb{R}}^n$, then one can identify the formal operations with those in ${\mathbb{R}}^n$, reducing the simplex $\Delta \left[S\right]$ to its usual definition. We write r for an element of $\Delta \left[\mathcal{Y}\right]$. In particular, an element of $\Delta [\Delta [\mathcal{Y}\left]\right]$ is merely a finite convex combination ${\sum }_{\hat{x}}p\left(\hat{x}\right)r_{\hat{x}}$ of distinct probability distributions $r_{\hat{x}}\left(y\right)\in \Delta \left[\mathcal{Y}\right]$ on $\mathcal{Y}$ (note that $\Delta \left[S\right]$ is a set). When setting $\hat{\mathcal{X}}\subset \Delta \left[\mathcal{Y}\right]$ to be a finite subset of distributions, $|\hat{\mathcal{X}}|<\infty$, then $\Delta \left[\hat{\mathcal{X}}\right]$ is a special case of the decoder coordinates of Section 2 (unlike $\Delta \left[\hat{\mathcal{X}}\right]$ here, the decoder coordinates of Section 2 are not required to have their clusters r distinct).

Now, let a finite IB problem be defined by a joint probability distribution $p_{Y|X}{p}_X$, as in Section 1. To write it down as an RD problem [5,20], define the IB distortion measure by

$$d_{IB}(x,r):=D_{KL}\left[p_{Y|X=x}\left|\right|r\right],$$

(25)

for $x\in \mathcal{X}$, $r\in \Delta \left[\mathcal{Y}\right]$, and $p_{Y|X=x}\in \Delta \left[\mathcal{Y}\right]$ the conditional probability distribution at $X=x$. The distortion measure $d_{IB}$ (25) and $p_X$ define an RD problem on the continuous reproduction alphabet $\hat{\mathcal{X}}:=\Delta \left[\mathcal{Y}\right]$. Minimizing the IB Lagrangian $\mathcal{L}$ (in Section 1) is equivalent to minimizing the Lagrangian of this RD problem [20] (Theorem 5). That is, the IB is a rate-distortion problem when considered in these coordinates. IB clusters $r\in \Delta \left[\mathcal{Y}\right]$ assume the role of RD reproduction symbols, while an IB root (considered now as an RD root) is equivalently described either by the probabilities of each cluster—namely, by a point in $\Delta [\Delta [\mathcal{Y}\left]\right]$—or, by a test channel $p\left(r\right|x)$. The astute reader might notice that the IB Equations (2) and (4) are then equivalent to RD’s fixed-point Equations (24), with the decoder Equation (3) implied by the IB’s Markovity. The IB’s Y-information $I(Y;\hat{X})$ equals the expected distortion $\mathbb{E}\left[d_{IB}(x,\hat{x})\right]$ in (23) up to a constant [20] (Section 5), and so is linear in the test channel $p\left(r\right|x)$. Unlike the finite-dimensional coordinate systems of Section 2, this definition of the IB entails no subtleties due to finite dimensionality, such as duplicate clusters (see more below). However, while it allows us to spell out the IB explicitly as an RD problem, handling an infinite reproduction alphabet is difficult for practical purposes. Since no more than $\left|\mathcal{X}\right|+1$ reproduction symbols are needed to write down an IB root [2], this motivates one to consider the IB’s local behavior, with clusters fixed.

So instead, one may require the reproduction symbols of $d_{IB}$ (25) to be in a list ${\left(r_{\hat{x}}\right)}_{\hat{x}\in \hat{\mathcal{X}}}$ indexed by some finite set $\hat{\mathcal{X}}$, with each $r_{\hat{x}}$ in $\Delta \left[\mathcal{Y}\right]$ (the elements $r_{{\hat{x}}_1},\cdots ,r_{{\hat{x}}_T}$ need not be distinct a priori). This defines a finite RD problem, for which $d_{IB}$ (25) is merely an $\left|\mathcal{X}\right|$-by-T matrix. Yet, placing identical clusters in the list ${\left(r_{\hat{x}}\right)}_{\hat{x}}$ inadvertently introduces degeneracy to the matrix $d_{IB}$ (25), as discussed below. In [5] (Section 6), ${\left(r_{\hat{x}}\right)}_{\hat{x}}$ is taken to be the decoders defined by a given encoder $p\left(\hat{x}\right|x)$, as in Equation (11) (Section 2). We shall then refer to $d_{IB}$ (25) as the distortion matrix defined by $p\left(\hat{x}\right|x)$. When $p_{{\beta }_0}\left(\hat{x}\right|x)$ is an optimal IB root then the problem $(d_{IB},p_X)$ defined by it is called the tangent RD problem. Indeed, its RD curve (23) coincides with the IB curve (1) at this point (since an optimal choice of IB clusters is already encoded into $d_{IB}$ (25), then solving the IB boils down to finding the clusters’ optimal weights $p\left(\hat{x}\right)$, which is an RD problem). However, the curves differ outside this point since IB clusters usually vary with $\beta$, while the distortion of the tangent problem was defined at $p_{{\beta }_0}\left(\hat{x}\right|x)$ and so is fixed. By definition (1), it follows that the IB curve is the lower envelope of the curves of its tangent RD problems [5] (Corollary 2). We note that a similar construction can also be carried out in inverse encoder coordinates, cf., [2].

Regardless of the formulation used to rewrite the IB as an RD problem, the associated RD problem has an expected distortion $\mathbb{E}\left[d_{IB}\right]$ of $I(X;Y)-I(\hat{X};Y)$ at an IB root (Section 5 in [20] and Lemma 8 in [5]). That is, the IB is a method of lossy compression that strives to preserve the relevant information $I(\hat{X};Y)$. Due to the Markov condition, information on Y is available only through X. Thus, one may intuitively consider the IB as a lossy compression method of the information on Y that is embedded in X. These intimate relations between the IB and RD suggest that studying bifurcations in either context could be leveraged to understand the other. Bifurcations in finite RD problems are discussed at length in [6] (Section 6). To facilitate the study of IB bifurcations in the sequel (Section 5.2 and Section 5.3) using results from RD, we need a “minimally-sufficient” coordinate system for the IB.

Consider an IB root in decoder coordinates as finitely many $p\left(\hat{x}\right)$-weighted points $r_{\hat{x}}\left(y\right)$ in $\Delta \left[\mathcal{Y}\right]$, as in Section 2. Exchanging to decoder coordinates (Equation (11) there) is well-defined as long as there are no zero-mass clusters, $\forall \hat{x}p\left(\hat{x}\right)\ne 0$. Yet, even then, the points $r_{\hat{x}}$ in $\Delta \left[\mathcal{Y}\right]$ yielded by BA’s steps 4 through 6 (Algorithm 1) need not be distinct. Namely, they may yield identical clusters $r_{\hat{x}}=r_{{\hat{x}}^{\prime }}$ at distinct indices $\hat{x}\ne {\hat{x}}^{\prime }$. This leads to a discussion of structural symmetries of the IB (its degeneracies), which is not of use for our purposes; cf., [9]. To avoid such subtleties, we shall say that an IB root is reduced if it has no zero-mass clusters, $\forall \hat{x}p\left(\hat{x}\right)\ne 0$, and all its clusters are distinct, $\hat{x}={\hat{x}}^{\prime }\iff r_{\hat{x}}=r_{{\hat{x}}^{\prime }}$. A root that is not reduced is called degenerate or degenerately represented. An IB root can be reduced by removing clusters of zero mass and merging identical clusters of distinct indices—see our reduction algorithm in Section 5.2 below. It is straightforward to see from the IB Equations (2)–(4) that reduction preserves the property of being an IB root. Similarly, reducing a root does not change its location in the information plane. So, a root achieves the IB curve (1) if and only if its reduction does. Therefore, reduction decreases the dimension in which the problem is considered while preserving all its essential properties. This allows us to represent an IB root on the smallest number of clusters possible—its effective cardinality—by factoring out the IB’s structural symmetries. See also [13] (2.3 in Chapter 7), upon which this definition is based.

While the purpose of reduction is to mod-out redundant kernel coordinates (Section 1), it highlights the differences between the various IB definitions found in the literature, bringing to light a subtle caveat of finite dimensionality. To see this, note that reduction could have been defined above in terms of the other coordinate systems of the IB. Its definition in inverse encoder coordinates is nearly identical to that above, while defining it in encoder coordinates is a straightforward exercise. Since the coordinate systems of Section 2 are equivalent at an IB root (without zero-mass clusters), the precise definition does not matter then. Each of these parameterizations encodes the coordinates $r\left(y\right)$ of a root’s clusters r using a finite-dimensional vector $\mathit{x}$ (note Equation (11)). This enables one to represent duplicate clusters $\hat{x}\ne {\hat{x}}^{\prime }$ with $r_{\hat{x}}=r_{{\hat{x}}^{\prime }}$, and obliges one to choose the order in which clusters are being encoded into the coordinates of $\mathit{x}$. A finite-dimensional representation $\mathit{x}$ of an IB root is invariant to interchanging clusters $\hat{x}\ne {\hat{x}}^{\prime }$ precisely when they are identical, $r_{\hat{x}}=r_{{\hat{x}}^{\prime }}$. The IB’s functionals (e.g., its X- and Y-information) are invariant to any cluster permutation; cf., [9,19]. Both of these structural symmetries result from using a finite-dimensional parameterization, with the former eliminated by reduction. In contrast, the elements of $\Delta \left[\mathcal{Y}\right]$ are distinct by definition (since $\Delta \left[\mathcal{Y}\right]$ is a set), and so parameterizing the IB by points in $\Delta [\Delta [\mathcal{Y}\left]\right]$ does not permit identical clusters. An element ${\sum }_{r}p\left(r\right)r$ of $\Delta [\Delta [\mathcal{Y}\left]\right]$ assigns a probability mass $p\left(r\right)$ to every point r in $\Delta \left[\mathcal{Y}\right]$, with only finitely many points r supported. Thus, it implicitly encodes all the entries $r\left(y\right)$ of every probability distribution $r\in \Delta \left[\mathcal{Y}\right]$ in a “one size fits all” approach, giving no room for the choices above. This leads us to argue that the IB’s structural symmetries are not an inherent property but rather an artifact of using its finite-dimensional representations. This is best understood in the context of discontinuous bifurcations, in Section 5.3 below. For comparison, both of the IB formulations [2,20] do not impose an a priori restriction on the number of clusters. The latter does not enable one to encode duplicate clusters, while the former does. The formulation [1] ignores these subtleties altogether, and [9,19] consider the IB on a pre-determined number of possibly duplicate clusters.

In rate-distortion, the reduction of a finite RD problem is defined similarly [6] (Section 3.1), by removing a symbol $\hat{x}$ from the reproduction alphabet $\hat{\mathcal{X}}$ and its column $d(·,\hat{x})$ from the distortion matrix once it is not in use anymore (of zero mass). A distortion matrix d is non-degenerate if its columns are distinct, $d(·,\hat{x})\ne d(·,{\hat{x}}^{\prime })$ for all $\hat{x}\ne {\hat{x}}^{\prime }$. Non-degeneracy arises naturally when considering the RD problem tangent to a given IB root $p\left(\hat{x}\right|x)$. Indeed, the distortion matrix $d_{IB}$ (25) defined by $p\left(\hat{x}\right|x)$ has duplicate columns if the root has identical clusters, while the other direction holds under mild assumptions (if the $\left|\mathcal{X}\right|$ vectors $p_{Y|X=x}$ span ${\mathbb{R}}^{\left|\mathcal{Y}\right|}$, then $D_{KL}[p_{Y|X=x}\left|\right|r_{\hat{x}}]=D_{KL}[p_{Y|X=x}\left|\right|r_{{\hat{x}}^{\prime }}]$ for all x implies that $r_{\hat{x}}=r_{{\hat{x}}^{\prime }}$). Under these assumptions, the distortion matrix induced by an IB root $p\left(\hat{x}\right|x)$ is reduced and non-degenerate precisely when $p\left(\hat{x}\right|x)$ is a reduced IB root.

Reduction in RD provides the means to show that the dynamics underlying the RD curve (23) are piecewise analytic in $\beta$ [6], under mild assumptions. Just as in definition (5) of the IB operator, [4] (Equation (5)) similarly define the RD operator $Id-B{A}_{\beta }^{RD}$ in terms of Blahut’s algorithm for RD [8]. By using their Theorem 1, [6] (Section 3.1) observed that reducing a finite RD problem to the support of a given RD root mods-out redundant kernel coordinates if the distortion measure is finite and non-degenerate (the support of $p\left(\hat{x}\right)$ is defined by $suppp\left(\hat{x}\right):=\{\hat{x}:p\left(\hat{x}\right)>0\}$). That is, the Jacobian $D(Id-B{A}_{\beta }^{RD})$ of the RD operator on the reduced problem is then non-singular (in the right coordinate system—see therein), just as with our toy problem (10) in Section 1. By the Implicit Function Theorem, there is therefore a unique RD root of the reduced problem through the given one; this root is real-analytic in $\beta$ (details there). Considering this for the RD problem tangent to a reduced IB root immediately yields the following:

Corollary 1.

Let $p_{{\beta }_0}\left(\hat{x}\right|x)$ be a reduced IB root of a finite IB problem defined by $p_{Y|X}{p}_X$, such that the matrix $p_{Y|X}$ is of rank $\left|\mathcal{Y}\right|$. Then, near ${\beta }_0$, there is a unique function continuous in β, which is a root of the tangent RD problem through $p_{{\beta }_0}\left(\hat{x}\right|x)$; it is real-analytic in β.

Corollary 1 shows that the local approximation of an IB problem (the roots of its tangent RD problem) is guaranteed to be as well-behaved as one could hope for, provided that the IB is viewed in the right coordinate system. Note, however, that the RD root through $p_{{\beta }_0}\left(\hat{x}\right|x)$ of the tangent problem does not in general coincide with the IB root outside of ${\beta }_0$ since the IB distortion $d_{IB}$ (25) varies along with the clusters that define it. However, when the IB clusters are fixed, then one might expect that the Jacobian (13) of $B{A}_{\beta }$ in log-decoder coordinates would be the same as the Jacobian of its RD variant. Indeed, the Jacobian matrix of $B{A}_{\beta }^{RD}$ is the $T\times T$ bottom-right sub-block of the Jacobian (13) of $B{A}_{\beta }$, up to a multiplicative factor. For this, see Equations (5) and (6) in [4], Equations (14) and (13) in Section 3, and (A25) in Appendix B.2.

As in RD, we argue that reduction in the IB also provides the means to show that the dynamics underlying the optimal curve (1) are piecewise analytic in $\beta$. Corollary 1 concludes that, under mild assumptions, through every reduced IB root passes a unique real-analytic RD root. However, its crux is that the Jacobian of the RD operator $Id-B{A}_{\beta }^{RD}$ is non-singular at a reduced root. Due to the IB’s close relations with RD, and since reduction in the IB is a natural extension of reduction in RD, we argue that the same is also to be expected of the IB operator $Id-B{A}_{\beta }$ (5) in decoder coordinates. To see this, note that IB roots are finitely supported [2] (Lemma 2.2(i)), and so one may take finitely supported probability distributions $\Delta [\Delta [\mathcal{Y}\left]\right]$ for the IB’s optimization variable. Thus, the IB’s $B{A}_{\beta }$ operator in decoder coordinates (of Section 2) may be considered as an operator on $\Delta [\Delta [\mathcal{Y}\left]\right]$. Next, consider the RD problem defined by $p_X$ and $d_{IB}$ (25) on the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$, as in [20]. This defines on $\Delta [\Delta [\mathcal{Y}\left]\right]$ also the BA operator $B{A}_{\beta }^{RD}$ for RD. Now that both BA operators are considered on an equal footing, we note the following. First, while $B{A}_{\beta }^{RD}$ iterates over the IB Equations (2) and (4), its IB variant $B{A}_{\beta }$ iterates also over the decoder Equation (3) (plug the IB distortion measure $d_{IB}$ (25) into the Equations (24) defining $B{A}_{\beta }^{RD}$ to see this). The latter Equation (3) is a necessary condition for $Y\to X\to \hat{X}$ to be Markov, and so can be understood as an enforcement of Markovity (in contrast, an arbitrary triplet $(Y,X,\hat{X})$ of random variables only satisfies $p\left(y\right|\hat{x})={\sum }_{x}p\left(y\right|x,\hat{x})p\left(x\right|\hat{x})$). That is, IB roots are RD roots with an extra constraint. Second, by Theorem 1 in [4], reducing $Id-B{A}_{\beta }^{RD}$ from the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$ to a root of finite support renders it non-singular, under mild assumptions. This suggests that reducing $Id-B{A}_{\beta }$ (5) from $\Delta [\Delta [\mathcal{Y}\left]\right]$ to a root’s effective cardinality should also render it non-singular, due to the similarity between these operators, and since reduction in the IB is a natural extension of reduction in RD. In line with the discussion of Section 1 on reduction, we therefore state the following:

Conjecture 1.

The Jacobian matrix $I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ at (16) of the IB operator (5) in log-decoder coordinates is non-singular at reduced IB roots so long as it is well-defined, except perhaps at points of bifurcation.

The intuition behind this conjecture stems from analyticity, as follows. The IB operator $Id-B{A}_{\beta }$ (5) is real-analytic, since each of the Equations 1.4–1.8 defining it (in the BA-IB Algorithm 1) is real-analytic in its variables. For a root ${\mathit{x}}_0$ of a real-analytic operator F, one might expect that, in general, (i) no roots other than ${\mathit{x}}_0$ exist in its vicinity and that (ii) $D_{\mathit{x}}{F|}_{{\mathit{x}}_0}$ has no kernel. That is, unless the operator is degenerate at ${\mathit{x}}_0$ in some manner or ${\mathit{x}}_0$ is a bifurcation. To see this, recall [32] (Section IX.3) that a real-valued function $F_i$ in $\mathit{x}\in {\mathbb{R}}^n$ is real-analytic in some open neighborhood of ${\mathit{x}}_0$ if it is a power series in $\mathit{x}=(x_1,\cdots ,x_n)$, within some radius of convergence (although a strictly positive radius is needed, we omit these details for clarity). For every practical purpose, one may replace $F_i$ by a polynomial in $(x_1,\cdots ,x_n)$ when $\mathit{x}$ is close enough to the base-point ${\mathit{x}}_0$, by truncating the power series. Viewed this way, a root of an operator $F\left(\mathit{x}\right)=\left(F_1\left(\mathit{x}\right),\cdots ,F_n\left(\mathit{x}\right)\right)$ is nothing but a solution of n polynomial equations in n variables. However, a square polynomial system typically has only isolated roots, which is (i). This is best understood in terms of Bézout’s Theorem; see  [33] (6 in IV.4) for example. For (ii), a vector $\mathit{v}$ is in $ker{D}_{\mathit{x}}F$ precisely when it is orthogonal to each of the gradients $\nabla F_i$. However, $\nabla F_i$ is the vector of the first-order monomial coefficients of $x_1,\cdots ,x_n$ in $F_i$. In a general position, these n coefficient vectors $\nabla F_1,\cdots ,\nabla F_n$ are linearly independent, and so $\mathit{v}$ must vanish as claimed. If F is degenerate such that $F_i=F_j$ for particular $i\ne j$, for example, then both points fail, of course. See also Section I.2 of [34] for (i) and (ii). This intuition accords with the comments of [28] (Section 2.4) on RD: “usually, each point on the rate distortion curve […] is achieved by a unique conditional probability assignment. However, if the distortion matrix exhibits certain form of symmetry and degeneracy, there can be many choices of [a minimizer]”. Indeed, the fact that the dynamics underlying the RD curve (23) are piecewise real-analytic [6] (under mild assumptions) can be similarly understood to stem from the analyticity of the RD operator $Id-B{A}_{\beta }^{RD}$.

Subject to Conjecture 1, a Jacobian eigenvalue of the IB operator (5) must vanish gradually as one approaches a bifurcation, causing the critical slowing down of BA-IB [4] (observe that BA’s Jacobian (13) is continuous in the root at which it is evaluated). When an IB root traverses a bifurcation in which its effective cardinality decreases, then it is not reduced anymore. One can then handle the bifurcation by reducing the root anew. To ensure proper handling by the bifurcation’s type, we consider the latter closely in Section 5.2 and Section 5.3 below. In a nutshell, following the IB’s ODE (16) along with a proper handling of its bifurcations is the idea behind our root-tracking algorithm (in Section 6), for approximating the IB numerically.

Conjecture 1 is compatible with our numerical experience. However, we leave its proof to future work. To that end, one could examine closely the smaller matrix S (17) (of Lemma 1 in Section 3), for example. However, even if Conjecture 1 were violated, then one could detect that easily by inspecting the Jacobian’s eigenvalues. Conjecture 1 also implies that IB roots are locally unique outside of bifurcations when presented in their reduced form. Non-uniqueness of optimal roots is detectable by inspecting the Jacobian’s eigenvalues—see Corollary 3 in Section 5.3 and the discussion following it. See also Section 6.3 in [6] for the respective discussion in RD. With that, most of the results in Section 5.2 and Section 5.3 below do not depend on the validity of Conjecture 1.

5.2. Continuous IB Bifurcations: Cluster Vanishing and Cluster Merging

Following [10], we consider the evolution of IB roots which are a continuous function of $\beta$. By representing an IB root in its reduced form (Section 5.1), it is evident that there are two types of continuous IB bifurcations. We provide a practical heuristic (Algorithm 2) for identifying and handling such bifurcations. The discussion here is complemented by Section 5.3 below, which considers the case where continuity does not hold.

The evolution of an IB root in $\beta$ obeys the ODE (16) as long as it can be written as a differentiable function in $\beta$, as in Theorem 1. Considering the root in decoder coordinates, this amounts to an evolution of a T-tuple of points $r_{\hat{x}}$ in $\Delta \left[\mathcal{Y}\right]$ and their weights $p\left(\hat{x}\right)$. These typically traverse the simplex smoothly as the constraint $\beta$ is varied, as demonstrated in Figure 6. We now consider two cases where this evolution does not obey the ODE (16), due to violating differentiability.

Consider an optimal IB root in its reduced form (see Section 5.1). Namely, consider the reduced form of a root that achieves the IB curve (1). Suppose that its decoders $r_{\hat{x}}$ and weights $p\left(\hat{x}\right)$ are continuous in $\beta$. Then, a qualitative change in the root can occur only if either (i) two (or more) of its clusters collide or (ii) the marginal probability $p\left(\hat{x}\right)$ of a cluster $\hat{x}$ vanishes. In either case, the minimal number of points in $\Delta \left[\mathcal{Y}\right]$ required to represent the root decreases. That is, its effective cardinality decreases (a qualitative change where the effective cardinality increases is obtained by merely reversing the dynamics in $\beta$). We call the first a cluster-merging bifurcation and the second a cluster-vanishing bifurcation, or continuous bifurcations collectively. Both types were observed already in [17] (Section IV.C) in the related setting of RD problems with a continuous source alphabet. Among the two, cluster-vanishing bifurcations are more frequent in practice than cluster merging. This can be understood by considering cluster trajectories in the simplex. In a general position, one might expect clusters to seldom be at the same “time” and place (that is, $\beta$ and $r\in \Delta \left[\mathcal{Y}\right]$).

We argue that cluster merging and cluster vanishing are indeed bifurcations, where IB roots of distinct effective cardinalities collide and merge into one. We offer two ways to see this. First, using the inverse encoder formulation of the IB in [2] (Section II.A), one can consider an optimization problem in which the number of IB clusters is constrained explicitly (the inverse encoders of an IB root with no zero-mass clusters are in bijective correspondence with its decoders, as noted in Section 2, and so inverse encoder and decoder coordinates are interchangeable). By the arguments therein, the constrained problem has an optimal root (due to compactness), which achieves the optimal curve of the constrained problem. The latter curve must be sub-optimal if fewer clusters are allowed than needed to achieve the IB curve (1). Thus, whenever the effective cardinality of an optimal root (in the un-constrained problem) decreases, it must therefore collide with an optimal root of the constrained IB problem (by Corollary 3 in Section 5.3 below). This accords with [1] (Section 3.4), which describes IB bifurcations as a separation of optimal and sub-optimal IB curves according to their effective cardinalities. Second, consider the reduced form of an IB root at the point of a continuous bifurcation. Since its effective cardinality decreases there strictly, say from $T_2$ to $T_1$, then the root can be represented on $T_1$ clusters at the bifurcation itself. However, the Jacobian of the IB operator (5) in log-decoder coordinates is non-singular when represented on $T_1$ clusters, as discussed after Proposition 1 (in Section 5.3). Thus, by the Implicit Function’s Theorem, there is a unique IB root on $T_1$ clusters through this point. It exists at both sides of the bifurcation (above and below the critical point). When represented on $T_2$ clusters, however, the latter intersects at the bifurcation with the root of effective cardinality $T_2$, and so the two roots collide and merge there to one. This argument is identical to [6] (Section 6.2), which proves that distinct RD roots collide and merge at cluster-vanishing bifurcations in RD.

At a continuous bifurcation, IB roots of distinct effective cardinalities collide and merge into one, as discussed above. Specifically, one root achieves the minimal value of the IB Lagrangian and so is stable, while the other root is sub-optimal. As we shall now elaborate, continuous IB bifurcations are thus pitchfork bifurcations (e.g., Section 3.4 in [35]), in accordance with [19]. Even though the optimal root is continuous in $\beta$ (by assumption), its differentiability is violated at the point of bifurcation. This can be inferred from the comments following Theorem 1 and seen in Figure 6. Strictly speaking, several copies of the root of larger effective cardinality collide at a continuous bifurcation. When two clusters $r\ne r^{\prime }$ collide in a cluster merging bifurcation, then the root itself is invariant to interchanging their coordinates after the collision but not before it, breaking the IB’s first structural symmetry discussed in Section 5.1. Interchanging the coordinates of r and $r^{\prime }$ (and their marginals) before the collision yields two distinct copies of essentially the same root. For a cluster vanishing bifurcation, the IB’s functionals (e.g., its X- and Y-information) do not depend on the coordinates ${\left(r\left(y\right)\right)}_y$ of a vanished cluster r, rendering these redundant; cf., [9] (Section 3.1). Before the cluster r vanishes, there is one copy of the root for each index $\hat{x}$, with r placed at its $\hat{x}$ coordinates. Considered in reduced coordinates, these coincide to a single copy after the cluster vanishes. This breaks the IB’s second structural symmetry.

With that, we note that cluster-vanishing bifurcations cannot be detected directly by standard local techniques (i.e., considering the derivative’s kernel directions at the bifurcation point), whether considering the Hessian of the IB’s loss function as in [9] or the Jacobian of the IB operator (5) as here. The technical reason for this is as follows, while the root cause underlying it is best understood in the context of discontinuous bifurcations (after Proposition 1 in Section 5.3). Observe that the $I(Y;\hat{X})$ and $I(X;\hat{X})$ functionals do not depend on the coordinates ${\left(r\left(y\right)\right)}_y$ of clusters r of zero mass. Thus, the directions corresponding to these coordinates are always in the kernel regardless of whether evaluating at a bifurcation or not, and so cannot be used to detect a bifurcation (the direction corresponding to a cluster’s marginal is useless when one does not know which coordinates ${\left(r\left(y\right)\right)}_y$ to pick for r). Indeed, with its dynamics in $\beta$ reversed, “a new symbol grows continuously from zero mass” in a cluster-vanishing bifurcation, as [17] (Section IV.C) comments in a related setting. It is then not clear a priori which point in $\Delta \left[\mathcal{Y}\right]$ should be chosen for the new symbol, rendering the perturbative condition at Equation (9) difficult to test. In accordance with this, Ref. [9] (Section 5) offers a perturbative condition for detecting arbitrary IB bifurcations, while ref. [13] (3.2 in Part III) offers a condition for detecting cluster-merging bifurcations by analyzing cluster stability. However, both conditions are equivalent (Appendix F), and so must detect the same type of bifurcations. In contrast, a cluster-splitting (or merging) bifurcation is straightforward to detect because the stability of a particular cluster $\hat{x}$ is a property of the root itself—see Appendix F and the references therein for details.

One may wonder whether bifurcations exist in the IB for the same reason as they do in RD. As in the IB, RD problems typically have many sub-optimal curves [6] (Section 6.1). While (continuous) bifurcations in the IB stem from restricting the effective cardinality [1] (Section 3.4), in RD they stem from the various restrictions that a reproduction alphabet has. For example, a reproduction alphabet $\hat{\mathcal{X}}:=\{r_1,r_2,r_3\}$ of an RD problem may be restricted to the distinct subsets $\{r_1,r_2\}$ and $\{r_2,r_3\}$, usually yielding distinct sub-optimal RD curves (e.g., Figure 6.1 in [6]). In contrast to RD, the IB’s distortion $d_{IB}$ (25) defined by a root’s clusters is determined a posteriori by the problem’s solution rather than a priori by the problem’s definition. As a result, both reasons for the existence of bifurcations coincide. To see this, consider the IB as an RD problem whose reproduction symbols $\hat{\mathcal{X}}$ are a finite subset of $\Delta \left[\mathcal{Y}\right]$ which is allowed to vary (i.e., as if defining the tangent RD problem anew at each $\beta$). Distinct restrictions of a reproduction alphabet $\hat{\mathcal{X}}$ can be forced to agree by altering the symbols themselves, so long as they are of the same size. For example, restricting the set $\{r_1,r_2,r_3\}$ of reproduction symbols to $\{r_1,r_2\}$ is the same as restricting it to $\{r_2,r_3\}$ instead, and then replacing $r_3$ with $r_1\in \Delta \left[\mathcal{Y}\right]$ in the restricted problem (this is not to be confused with cluster permutations, which change the order in which clusters are listed but do not alter the symbols themselves).

The dynamical point of view above, considering an IB root as weighted points traversing $\Delta \left[\mathcal{Y}\right]$, offers a straightforward way to identify and handle continuous IB bifurcations. It is spelled out as our root-reduction Algorithm 2. For cluster-vanishing bifurcations, one can set a small threshold value ${\delta }_1>0$ and consider the cluster $\hat{x}$ as vanished if $p\left(\hat{x}\right)<{\delta }_1$ (Step 2.3), as in [6] (Section 3.1). Similarly, for cluster-merging bifurcations, one can set a small threshold ${\delta }_2>0$ and consider the clusters $\hat{x}\ne {\hat{x}}^{\prime }$ to have merged if $\parallel r_{\hat{x}}-r_{{\hat{x}}^{\prime }}{\parallel }_{\infty }<{\delta }_2$ (Step 2.9). A vanished cluster is then erased (and merged clusters replaced by one), resulting in an approximate IB root on fewer clusters. This not only identifies continuous IB bifurcations but also handles them, since the output of the root-reduction Algorithm 2 is a numerically reduced root, represented in its effective cardinality. To re-gain accuracy, we shall later invoke the BA-IB Algorithm 1 on the reduced root, as part of our root-tracking algorithm (in Section 6). We note that one should pick the thresholds ${\delta }_1$ and ${\delta }_2$ small enough to avoid false detections, and yet not too small so as to cause mis-detections. Mis-detections will be handled later, in Section 6.1, using a heuristic algorithm.

Algorithm 2 Root reduction for the IB
1:	function Reduce root($p\left(y\right|\hat{x}),p\left(\hat{x}\right);{\delta }_1,{\delta }_2$)
Input:
	An approximate IB root $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$ in decoder coordinates,
	a cluster-mass threshold $0<{\delta }_1<1$ and a cluster-merging threshold $0<{\delta }_2<1$.
Output: An approximate IB root $\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$ at its effective cardinality.
2:	for $\hat{x}$ do
3:	if $p\left(\hat{x}\right)<{\delta }_1$ then            ▹ Delete clusters of near-zero mass.
4:	delete the coordinates of $\hat{x}$, from $p\left(\hat{x}\right)$ and $p\left(y\right|\hat{x})$.
5:	end if
6:	end for
7:	$p\left(\hat{x}\right)\leftarrow$ normalize $p\left(\hat{x}\right)$ ▹ Preserve normalization, in case clusters were removed.
8:	for $\hat{x}\ne {\hat{x}}^{\prime }$ do
9:	if $\parallel p\left(y\right|\hat{x})-p\left(y\right|{\hat{x}}^{\prime }){\parallel }_{\infty }<{\delta }_2$ then    ▹ Merge nearly identical points in $\Delta \left[\mathcal{Y}\right]$.
10:	$p\left(\hat{x}\right)\leftarrow p\left(\hat{x}\right)+p\left({\hat{x}}^{\prime }\right)$
11:	delete the coordinates of ${\hat{x}}^{\prime }$, from $p\left(\hat{x}\right)$ and $p\left(y\right|\hat{x})$.
12:	end if
13:	end for
14:	return $\left(p\left(y\right|\hat{x}),p\left(\hat{x}\right)\right)$
15:	end function

Using the root-reduction Algorithm 2 allows one to stop early in the vicinity of a bifurcation when following the path of an IB root. As mentioned in Section 4, early stopping restricts the computational difficulty of root tracking [6]. Further, reducing the root before invoking BA-IB (Algorithm 1) allows us to avoid BA’s critical slowing down [4], since reduction removes the nearly vanished Jacobian eigenvalues that pertain to the nearly vanished (or nearly merged) cluster(s), which are the cause of BA’s critical slowing down. cf., Proposition 1 (Section 5.3) and the discussion around it. See also [6] (Figure 3.1(C) and Section 3.2) for the respective behavior in RD. Finally, we comment that the root-reduction Algorithm 2 can also be implemented in the other two coordinate systems of Section 2.

5.3. Discontinuous IB Bifurcations and Linear Curve Segments

In the previous Section 5.2, we considered continuous IB bifurcations—namely, when the clusters $r_{\hat{x}}\in \Delta \left[\mathcal{Y}\right]$ and weights $p\left(\hat{x}\right)$ of an IB root are continuous functions of $\beta$. By exploiting the intimate relations between the IB and RD (Section 5.1), we now consider IB bifurcations where these cannot be written as a continuous function of $\beta$. In our experience, discontinuous bifurcations are infrequent in practice. However, the theory they evoke has several subtle consequences of practical implications important for computing IB roots (in Section 6). Though, perhaps more importantly, they oblige one to ask what is the IB? We start with several examples before diving into the theory; e.g., Figure 7.

The examples of discontinuous IB bifurcations of which we are aware can be understood in RD context as follows. Consider the IB as an RD problem on the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$, with IB roots parameterized by points in $\Delta [\Delta [\mathcal{Y}\left]\right]$ (see Section 5.1). In RD, the existence of linear curve segments is well-known [28]—e.g., Figure 2.7.6 in the latter and its reproduction in [6] (Figure 6.2). Section 6.5 in [6] offers an explanation of linear segments in terms of a support-switching bifurcation. Namely, a bifurcation where two RD roots of distinct supports exchange optimality at a particular multiplier value ${\beta }_c$. Both roots evolve smoothly in $\beta$ while only exchanging optimality at the bifurcation. At ${\beta }_c$ itself, every convex combination of these two roots is also an RD root. In particular, the optimal RD root cannot be written as a continuous function of $\beta$. The sudden emergence of an entire segment of roots at ${\beta }_c$ can be understood by RD’s convexity and analyticity properties, as follows. The RD curve (23) is parameterized by the slope $-\beta$ of its tangents [28] (Theorems 2.5.1 and 2.5.2). Above and below ${\beta }_c$, specifying the tangent’s slope determines a curve-achieving distribution on the optimal root (the root whose curve is lower at this slope value). Equivalently, the lower convex envelope of these roots in the RD plane coincides with one root above ${\beta }_c$ and with the other below it, as seen in Figure 8 (black). At ${\beta }_c$ itself, specifying the slope determines a distribution on both roots. Thus, the convexity of the RD curve and of the set of achieving distributions implies a linear segment at ${\beta }_c$ (Theorem 2.4.1 in [28] and Theorem 5 below). Finally, this behavior is possible due to analyticity, since the roots of a real-analytic operator $Id-B{A}_{\beta }^{RD}$ are either isolated (typical) or an algebraic curve (atypical) by Bézout’s Theorem—see (i) in the discussion following Conjecture 1.

For one example of linear curve segments in the IB, say that a matrix M decomposes if it can be written (non-trivially) as a block matrix by permuting its rows or columns. In light of the above, we have the following refinement of Theorem 2.6 in [2]:

Theorem 3.

The IB curve (1) has a linear segment at $\beta =1$ if and only if the problem’s definition $p_{Y|X}{p}_X$ decomposes.

Recall that the slope of the IB curve is $1/\beta$ at a multiplier value $\beta$ [1] (Equation (32)). Thus, Theorem 3 equates decomposable problems with linear curve segments of slope 1 (the slope cannot exceed one due to the data processing inequality). Figure 7 provides a simple decomposable example, exhibiting a support-switching bifurcation between its trivial and non-trivial roots. Non-decomposable examples also exist, exhibiting a support-switching bifurcation at lower slope values (higher critical $\beta$’s). For example, a symmetric binary erasure channel exhibits a support-switching bifurcation [2] (Section IV.B), which is manifested by a linear segment of slope $1/{\beta }_c\le 1$, for ${\beta }_c\ge 1$ (switching between the trivial root at $p_Y$ and a bi-clustered root supported on $\left(\begin{array}{c}{\beta }_c^{-1},1-{\beta }_c^{-1},0\end{array}\right)$ and $\left(\begin{array}{c}0,1-{\beta }_c^{-1},{\beta }_c^{-1}\end{array}\right)\in \Delta \left[\mathcal{Y}\right]$; the linear segment of slope ${\beta }_c^{-1}$ is Equation (4.8) there). See [2] (Section IV) for further examples. We argue that in the IB, support-switching bifurcations exhibit the same behavior as in RD. That is, two roots that evolve smoothly in $\beta$ and exchange optimality at the bifurcation. While the sequel can justify this in general, there is a simple way to see this in practice. Namely, following the two roots of Figure 7 through the bifurcation by using BA-IB with deterministic annealing [11] (follow the trivial root of Figure 7 from left to right and the non-trivial one from right to left, through the bifurcation at ${\beta }_c=1$ there). As deterministic annealing usually follows a solution branch continuously, this immediately reveals either root at the region where it is sub-optimal (not displayed).

A support-switching bifurcation evidently has similar characteristics to a transcritical bifurcation (e.g., Section 3.2 in [35]), though it should perhaps be classified as an imperfect transcritical since the roots do not intersect per se as in a classical transcritical. This extends the results of [19], who conclude that IB bifurcations “are only of pitchfork type” (Theorem 5 therein says that the bifurcations detected by their Theorem 3 are degenerate rather than transcritical, concluding that “the bifurcation guaranteed by Theorem 3 is [generically] pitchfork-like”). To see the reason for this discrepancy, note that they employ the mathematical machinery in [36] of bifurcations under symmetry. Since pitchfork bifurcations are “common in physical problems that have a symmetry” [35] (Section 3.4), then detecting only pitchforks by using the above machinery might not come as a surprise. Both [9] and its sequel [19] consider the IB’s symmetry to interchanging the coordinates of identical clusters (Definition 1(1) in [19]). However, this is a structural symmetry of the IB which stems from representing IB roots by finite-dimensional vectors (Section 5.1), and is broken in continuous IB bifurcations (Section 5.2). On the other hand, discontinuous IB bifurcations need not break this symmetry, as can be seen by inspecting the roots of Figure 7 closely (the trivial solution to the left of ${\beta }_c$ there may be given a degenerate bi-clustered representation, which is fully supported on $p_Y$ but has a second cluster $r\ne p_Y$ of zero mass. Neither of its roots then possesses a symmetry to interchanging cluster coordinates, at either side of ${\beta }_c$).

A few convexity results from rate-distortion theory are needed to consider discontinuous bifurcations in general. These have subtle practical implications, which are of interest in their own right.

Theorem 4

(Theorem 2.4.2 in [28]). The set of conditional probability distributions $p\left(\hat{x}\right|x)$ which achieve a point $(D,R(D\left)\right)$ on the rate-distortion curve (23) is convex.

Viewing the IB as an RD problem as in [20] immediately yields an identical result for the IB:

Corollary 2.

The set of IB encoders that achieve a point$(I_X,I_Y)$on the IB curve (1) is convex.

The proof is provided below for completeness. We note that a version of Corollary 2 in inverse encoder coordinates can also be synthesized from the ideas leading to Theorem 2.3 in [2].

Proof of Corollary 2.

Consider a finite IB problem $p_{Y|X}{p}_X$ as an RD problem $\left(d_{IB},p_X\right)$ on the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$, as defined by (25) in Section 5.1. As noted above, its encoders (or test channels) are conditional probability distributions $p\left(r\right|x)$, with $r\in \Delta \left[\mathcal{Y}\right]$, supported on finitely many coordinates $(r,x)$.

Let $p_1\left(r\right|x)$ and $p_2\left(r\right|x)$ be encoders achieving a point $(I_X,I_Y)$ on the IB curve (1). Define their support by $suppp\left(r\right|x):=suppp(r)$, where $p\left(r\right)$ is defined from $p\left(r\right|x)$ via marginalization, as in (4). By Theorem 5 in [20], $p_1\left(r\right|x)$ and $p_2\left(r\right|x)$ may be considered as test channels achieving the curve (23) of the RD problem $\left(d_{IB},p_X\right)$. The reproduction symbols $r\in \Delta \left[\mathcal{Y}\right]$ supporting a convex combination $p_{\lambda }:=\lambda ·p_1+(1-\lambda )·p_2$, $0\le \lambda \le 1$, are contained in the the supports of $p_1$ and $p_2$: $supp{p}_{\lambda }\subseteq supp{p}_1\cup supp{p}_2$. Therefore, $p_{\lambda }$ is finitely supported. Although Berger’s Theorem 4 assumes that the reproduction alphabet is finite, one can readily see that its proof works just as well when the distributions involved are finitely supported. Thus, by Theorem 4, $p_{\lambda }$ achieves the above point on the RD curve (23). Since this point $(I_X,I_Y)$ is on the IB curve (1), then $p_{\lambda }$ is an optimal IB root.    □

The RD curve (23) is the envelope of lines of slope $-\beta$ and intercept ${min}_{p\left(\hat{x}\right|x)}\left(I(X;\hat{X})+\beta \mathbb{E}\left[d(x,\hat{x})\right]\right)$ along the R-axis, e.g., [28]. Thus, Theorem 4 can be generalized by considering the achieving distributions that pertain to a particular slope value rather than to a particular curve point $(D,R(D\left)\right)$—see [6] (Section 6.3).

Theorem 5

(Theorem 20 in [6]). For any $\beta >0$ value, the set of distributions achieving the RD curve (23) that correspond to β is convex.

As with Corollary 2, we immediately have an identical result for roots achieving the IB curve (1):

Corollary 3.

For any $\beta >0$ value, the set of optimal IB encoders that correspond to β is convex.

See also [2] (Section IV) for an argument in inverse encoder coordinates. In particular, note the duality technique leading to (b) and (c) in Theorem 4.1 there. This duality boils down to describing a compact convex set in the plane by its lines of support, as in the observation leading to Theorem 5. Commensurate with the IB being a special case of RD, Corollary 3 can also be proven directly from the IB’s definitions in direct encoder terms  [37]. Note that the requirement that the IB root indeed achieves the curve is necessary. Otherwise, one could take convex combinations with the trivial IB root $p\left(r\right|x)={\delta }_{r,p_Y}$ (which satisfies the IB Equations (2)–(4) for every $\beta >0$, as one can verify directly). This yields absurd results, since the trivial root contains no information on either X or Y.

As in [6] (Section 6.3), the convexity of optimal IB roots (Corollary 3) has several important consequences. For one, unlike the (local) bifurcations we have considered so far, bifurcation theory also has global bifurcations. These are “bifurcations that cannot be detected by looking at small neighborhoods of fixed points” [12] (Section 2.3). From convexity, it immediately follows that

Corollary 4.

There are no global bifurcations in finite IB problems.

Indeed, if at a given $\beta$ value there exists more than one optimal root, then the Jacobian of the IB operator $Id-B{A}_{\beta }$ (5) must have a kernel vector pointing along the line connecting these optimal roots, by Corollary 3.

With that comes an important practical caveat. Corollaries 2 and 3 hold for the IB when parameterized by points in $\Delta [\Delta [\mathcal{Y}\left]\right]$. However, the above kernel vector (which exists due to convexity) may not be detectable if an IB root is improperly represented by a finite-dimensional vector. For example, consider the bifurcation in Figure 7, where a line segment at ${\beta }_c$ connects the trivial (single-clustered) root to the 2-clustered root. Obviously, the bifurcation there cannot be detected by the Jacobian of the IB operator (5) when it is computed on $T=1$ clusters (Jacobian of order $1·\left(\right|\mathcal{Y}|+1)$). Indeed, the root of effective cardinality two cannot be represented on a single cluster, and so the line segment connecting it to the trivial root does not exist in a 1-clustered representation. This is demonstrated in Figure 9, which compares Jacobian eigenvalues at reduced representations to those at 2-clustered representations. The same reasoning gives the following necessary condition:

Proposition 1

(A necessary condition for detectability of IB bifurcations). A bifurcation at ${\beta }_c$ in a finite IB problem which involves roots of effective cardinalities $T_1$ and $T_2$ is detectable by a non-zero vector in $ker(I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{{\beta }_c})$ only if the latter is evaluated at a representation on at least $max\{T_1,T_2\}$ clusters.

Indeed, suppose that $T_1\lneqq T_2$ (the conclusion is trivial if $T_1=T_2$). By definition, a root of effective cardinality $T_2$ does not exist in representations with less than $T_2$ clusters. Thus, there is no bifurcation in a T-clustered representation if $T<T_2$, and so there is then nothing to detect. As a special case of this argument, note that Conjecture 1 (Section 5.1) implies that the Jacobian is non-singular in a $T_1$-clustered representation of the $T_1$-clustered root (namely, at its reduced representation). With that, we have observed numerically that the eigenvalues of $D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ do not depend on the representation’s dimension if computed on strictly more clusters than the effective cardinality (which makes sense considering Theorem 2.6.1 in [28] or Lemma 2.2(i) in [2]). Rather, only the eigenvalues’ multiplicities vary by dimension. We omit practical caveats on exchanging between the coordinate systems of Section 2 for brevity.

The discussion of discontinuous bifurcations naturally leads one to consider the IB as an RD problem on the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$, as in Corollaries 2 and 3, unlike its usual definitions in the literature. When considered this way, IB roots are merely paths $\mathit{p}\left(\beta \right)$ in $\Delta [\Delta [\mathcal{Y}\left]\right]$, following a piecewise smooth trajectory dictated by the IB ODE (16) (which may be considered as a non-autonomous ODE on $\Delta [\Delta [\mathcal{Y}\left]\right]$). Due to Conjecture 1 and the IFT, these paths are isolated outside bifurcations. Two (or more) roots may intersect in a continuous bifurcation. If one of the intersecting roots is optimal, then the other must be of a strictly smaller effective cardinality due to the arguments in Section 5.2. If two distinct roots are optimal simultaneously, then $\Delta [\Delta [\mathcal{Y}\left]\right]$ contains an entire segment of optimal IB roots, due to Corollary 3. Viewing the IB this way also highlights several subtleties in its calculation. First, parameterizing IB roots with $\Delta [\Delta [\mathcal{Y}\left]\right]$ avoids its structural symmetries (Section 5.1). Second, it shows that, a priori, it is possible to follow the path of an optimal root using local techniques (Corollary 4). Third, it highlights that one must compute on enough clusters to detect a bifurcation (Proposition 1). Though obvious in retrospect, this caveat was not given proper attention in the IB literature. Fourth, as we shall now see, cluster-vanishing bifurcations can only be detected by following an optimal root to its collision with a root of smaller effective cardinality. Fifth, this implies (below) that only negative step sizes $\Delta \beta <0$ should be used to follow an optimal root.

The arguments above imply that cluster-vanishing bifurcations cannot be detected directly by considering kernel directions of the IB operator (5) at the bifurcation, as argued in Section 5.2. Indeed, consider a continuous bifurcation, where roots ${\mathit{p}}_1$ and ${\mathit{p}}_2$ of respective effective cardinalities $T_1<T_2$ intersect. These are paths in $\Delta [\Delta [\mathcal{Y}\left]\right]$ that coincide at the bifurcation itself, ${\mathit{p}}_1\left({\beta }_c\right)={\mathit{p}}_2\left({\beta }_c\right)$, and so in particular are of the same effective cardinality $T_1$ there. This is in contrast to the situation in Corollary 3, where two distinct roots are simultaneously optimal at ${\beta }_c$, leading to an entire segment of optimal roots. Asking whether a bifurcation is detectable amounts to considering the evaluation of $kerD(Id-B{A}_{\beta })$ at a finite-dimensional representation (or “projection”) of $\mathit{p}$. The Jacobian $D(Id-B{A}_{\beta })$ of the IB operator (5) is non-singular when evaluated on a $T_1$-clustered representation of ${\mathit{p}}_1\left({\beta }_c\right)$ in log-decoder coordinates, as noted after Proposition 1. We argue that evaluating it on representations with more clusters $T\gneqq T_1$ does not allow one to detect the bifurcation (even if $T\ge T_2$). See Appendix H for a formal argument. Intuitively, this is because picking a degenerate representation amounts to duplicating clusters of the reduced representation or adding clusters of zero mass (see reduction in Section 5.1). Introducing degeneracies to a reduced root adds no information about the problem at hand.

Due to the above, cluster-vanishing bifurcations cannot be detected by following a root ${\mathit{p}}_1$ of effective cardinality $T_1$ through the bifurcation point, but only by following a root ${\mathit{p}}_2$ with $T_2>T_1$ to its collision with ${\mathit{p}}_1$. As discussed after Conjecture 1 (Section 5.1), the Jacobian of $Id-B{A}_{\beta }$ in reduced log-decoder coordinates can then be used to indicate the upcoming collision of ${\mathit{p}}_2$ with ${\mathit{p}}_1$, in addition to the root-reduction Algorithm 2. The exact same arguments as above apply also to cluster-merging bifurcations. However, as noted in Section 5.2 (and Appendix F), the stability of a particular IB cluster $\hat{x}$ is a property of the root itself. Thus, these are detectable by standard local techniques at the point of bifurcation. Unlike continuous bifurcations, discontinuous bifurcations are inherently detectable due to the line segment in $\Delta [\Delta [\mathcal{Y}\left]\right]$ connecting the roots at the bifurcation (Corollary 3), as long as the IB root is represented on sufficiently many clusters (Proposition 1)—see Figure 9. These results make sense, considering that cluster-vanishing bifurcations are more frequent in practice than other types. Intuitively, branching from a suboptimal root ${\mathit{p}}_1$ to an optimal one ${\mathit{p}}_2$ is harder than the other way around, just as learning new relevant information is harder than discarding it. Cases where both directions are equally difficult are the exception, as one might expect. This is consistent with the later discussion in Section 6.3 on the stability of optimal IB roots (Appendix G).

When following the path of a reduced IB root (as in Section 4), one would like to ensure that its bifurcations are indeed detectable by BA’s Jacobian. Due to the caveats involved in detecting bifurcations of either type, it is necessary to follow the path as the effective cardinality decreases rather than increases. As a result, we take only negative step sizes $\Delta \beta <0$, since the effective cardinality of an optimal IB root cannot decrease with $\beta$. To see this, first note that the IB curve $I_Y\left(I_X\right)$ (1) is concave, and so its slope $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\beta $}\right.$ cannot increase with $I_X$. That is, $\beta$ cannot decrease with $I_X$. Second, note that allowing more clusters cannot decrease the X-information ${\sum }_{\hat{x}}p\left(\hat{x}\right)H\left(p\left(x\right|\hat{x})\right)$ achieved by the IB’s optimization variables. Indeed, a T-clustered variable $\left(p\left(x\right|\hat{x}),p\left(\hat{x}\right)\right)$ (not necessarily a root) can always be considered as $(T+1)$-clustered, by adding a cluster of zero mass; cf., the construction of [2] (Section II.A). Thus, the effective cardinality of an optimal root cannot decrease as the constraint $I_X$ on the X-information is relaxed. With both points combined, the effective cardinality cannot decrease with $\beta$, as argued. In contrast to the IB, we note that the behavior of RD problems is more complicated since the distortion of each reproduction symbol is fixed a priori; e.g., Example 2.7.3 and Problems 2.8–2.10 in [28].

Returning to discontinuous IB bifurcations, we proceed with the argument of Section 5.2 when continuity fails. That is, consider the reduced form of an optimal IB root, and suppose that either its decoders or its weights (or both) cannot be written as a continuous function of $\beta$ at ${\beta }_c$. Write $r_{\hat{x}}^{+}$ and $r_{\hat{x}}^{-}$ for its distinct decoders as $\beta \to {\beta }_c^{+}$ and $\beta \to {\beta }_c^{-}$, respectively. Similarly, write $p^{+}\left(\hat{x}\right)$ and $p^{-}\left(\hat{x}\right)$ for its non-zero weights. Consider the tangent RD problem on the reproduction alphabet $\hat{\mathcal{X}}:={\left\{r_{\hat{x}}^{+}\right\}}_{\hat{x}}\cup {\left\{r_{\hat{x}}^{-}\right\}}_{\hat{x}}\subset \Delta \left[\mathcal{Y}\right]$, as in Section 5.1. See also [4] (Section V), upon which this argument is based. By construction, the IB coincides with its tangent RD problem at the two points $\left(r_{\hat{x}}^{+},p^{+}\left(\hat{x}\right)\right)$, and $\left(r_{\hat{x}}^{-},p^{-}\left(\hat{x}\right)\right)$. Since both points achieve the optimal curve at the same slope value $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$/$}\right.{\beta }_c$, then the linear segment of distributions connecting these points is also optimal, by Theorem 5. Alternatively, one could apply Corollary 3 directly to the IB problem. Either way, there exists a line segment of optimal IB roots, which pertain to the given slope value. In summary,

Theorem 6.

Let a finite IB problem have a discontinuous bifurcation at ${\beta }_c\ge 1$. Then, its IB curve (1) has a linear segment of slope $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\beta }_c$}\right.$.

Unless the decoder sets ${\left\{r_{\hat{x}}^{+}\right\}}_{\hat{x}}$ and ${\left\{r_{\hat{x}}^{-}\right\}}_{\hat{x}}$ are identical, then this is a support-switching bifurcation [6] (Section 6.5), as in Figure 7. A priori, the IB roots $\left(r_{\hat{x}}^{+},p^{+}\left(\hat{x}\right)\right)$ and $\left(r_{\hat{x}}^{-},p^{-}\left(\hat{x}\right)\right)$ may achieve the same point in the information plane, in which case the linear curve segment is of length zero. However, we are unaware of such examples. Yet, even if such bifurcations exist, they would be detectable by the Jacobian of BA-IB (when represented on enough clusters), subject to Conjecture 1.

6. First-Order Root Tracking for the Information Bottleneck

Gathering the results of Section 2, Section 3, Section 4 and Section 5, we can now not only follow the evolution of an IB root along the first-order Equation (16), but can also identify and handle IB bifurcations. This is summarized by our First-order Root-Tracking algorithm for the IB (IBRT1) in Section 6.1, with some numerical results in Section 6.2. Section 6.3 discusses the basic properties of IBRT1, and mainly the surprising quality of approximations of the IB curve (1) that it produces, as seen in Figure 1. We focus on continuous bifurcations (Section 5.2), since these are far more frequent in our experience than discontinuous ones and are straightforward to handle (see Section 6.3 on the handling of discontinuous bifurcations).

6.1. The IBRT1 Algorithm 5

To assist the reader, we first present a simplified version of IBRT1 as Algorithm 3, with edge cases handled later by Algorithm 4—clarifications follow. When combined, these two form our IBRT1 Algorithm 5, specified below.

We now elaborate on the main steps of the Simplified First-order Root Tracking for the IB (Algorithm 3), following Root Tracking for RD [6] (Algorithm 3). Its purpose is to follow the path of a given IB root $p_{{\beta }_0}\left(\hat{x}\right|x)$ in a finite IB problem. The initial condition $p_{{\beta }_0}\left(\hat{x}\right|x)$ is required to be reduced and IB-optimal. Its optimality is needed below to ensure that the path traced by the algorithm is indeed optimal. The step-size $\Delta \beta$ is negative, for reasons explained in Section 5.3 (Proposition 1 ff.). The cluster mass and cluster merging thresholds are as in the root-reduction Algorithm 2 (Section 5.2).

Denote $\tilde{\mathit{p}}$ (step 3 of Algorithm 3) for the distributions generated from an encoder (see Equation (11) in Section 2). Algorithm 3 iterates over grid points $\tilde{\mathit{p}}$, with each while iteration generating the reduced form of the next grid point, as follows. On step 6, evaluate the IB ODE (16) at the current root $\tilde{\mathit{p}}$, solving the linear equations numerically. By Conjecture 1 (Section 5.1), the IB ODE has a unique numerical solution $\mathit{v}$ if $\tilde{\mathit{p}}$ is a reduced root and not a bifurcation. Steps 7 and 8 approximate the root at the next grid point at $\beta +\Delta \beta$, by exponentiating Euler method’s step (22) (Section 4). Normalization is enforced on step 9, since it is assumed throughout. Off-grid points can be generated by repeating steps 7 through 9 for intermediate $\Delta \beta$ values if desired. The approximate root at $\beta +\Delta \beta$ is reduced on step 11, by invoking the root-reduction Algorithm 2 (Section 5.2). Note that Algorithm 2 returns its input root unmodified unless reducing it numerically. If reduced, then the root is a vector of a lower dimension—either a cluster mass $p\left(\hat{x}\right)$ has nearly vanished or distinct clusters have nearly merged. To re-gain accuracy, we invoke (on step 14) the Blahut–Arimoto Algorithm 1 for the IB until convergence, on the encoder defined at step 13 by the reduced root. Although BA-IB is invoked near a bifurcation, this does not incur a hefty computational cost due to its critical slowing down [4]—see comments at the bottom of Section 5.2. Invoking BA (on step 14) before reducing (on step 11) would have inflicted a hefty computational cost to BA-IB due to the nearby bifurcation. Finally, a single BA-IB iteration in decoder coordinates is invoked on the approximate root (step 17), whether reduced earlier or not. This enforces Markovity while improving the order of this method (see Section 4, and Figure 4 in particular). Algorithm 3 continues this way (step 4) until the approximate solution is trivial (single-clustered), or $\beta$ is non-positive. In the IB, the trivial solution is always optimal for tradeoff values $\beta <1$. However, here $\beta$ plays the role of the ODE’s independent variable instead. Thus, we allow Algorithm 3 to continue beyond $\beta =1$, as long as $\beta >0$, which is assumed throughout (the condition $\beta >|\Delta \beta |$ on step 4 ensures that the target $\beta$ value of the next grid point is non-negative). This shall be useful for overshooting—see below.

Algorithm 3 Simplified First-order Root-Tracking for the IB
1:	function sIBRT1($p_{Y|X}{p}_X,{\beta }_0,p_{{\beta }_0}\left(\hat{x}\right|x);\Delta \beta ,{\delta }_1,{\delta }_2$)
Input:
	An IB problem definition $p_{Y|X}{p}_X$ with $\forall x{p}_X\left(x\right)>0$.
	A reduced IB-optimal root $p_{{\beta }_0}\left(\hat{x}\right|x)$ at ${\beta }_0$. A step size $\Delta \beta <0$.
	Cluster-mass threshold ${\delta }_1$ and cluster-merging threshold ${\delta }_2$, with $0<{\delta }_i<1$.
Output: Approximations ${\tilde{\mathit{p}}}_{{\beta }_n}$ of the optimal IB roots ${\mathit{p}}_{{\beta }_n}$ at ${\beta }_n:={\beta }_0+n\Delta \beta$.
2:	Initialize $\beta \leftarrow {\beta }_0$ and $results\leftarrow \left\{\right\}$.
2:	Initialize $\tilde{\mathit{p}}:=\left(\tilde{p}\left(\hat{x}\right|x),\tilde{p}\left(x\right|\hat{x}),\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$ from $p_{{\beta }_0}\left(\hat{x}\right|x)$, via Steps 1.4–1.6.
4:	while $\beta >|\Delta \beta |$ and $\left|supp\tilde{p}\left(\hat{x}\right)\right|>1$ do    ▹ See main text on stopping condition.
5:	Append $\tilde{\mathit{p}}$ to $results$.
6:	$\mathit{v}:=\left(\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta },\frac{dlog\tilde{p}\left(\hat{x}\right)}{d\beta }\right)\leftarrow$ solve the IB ODE (16) at $\tilde{\mathit{p}}$.
7:	$\tilde{p}\left(y\right|\hat{x})\leftarrow \tilde{p}\left(y\right|\hat{x})exp\left(\Delta \beta ·\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta }\right)$
8:	$\tilde{p}\left(\hat{x}\right)\leftarrow \tilde{p}\left(\hat{x}\right)exp\left(\Delta \beta ·\frac{dlog\tilde{p}\left(\hat{x}\right)}{d\beta }\right)$ ▹ Exponentiate the linear approximations (22).
9:	$\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)\leftarrow \mathrm{normalize}\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$
10:	$old\_dim\leftarrow dim\tilde{p}\left(\hat{x}\right)$
11:	$\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)\leftarrow \mathrm{R}\mathrm{E}\mathrm{D}\mathrm{U}\mathrm{C}\mathrm{E}\mathrm{R}\mathrm{O}\mathrm{P}\mathrm{T}(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right);{\delta }_1,{\delta }_2)$.    ▹ Algorithm 2.
12:	if $old\_dim\ne dim\tilde{p}\left(\hat{x}\right)$ then ▹ Root was reduced due to bifurcation.
13:	$\tilde{p}\left(\hat{x}\right|x)\leftarrow$ the encoder defined by $\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$, via Steps 1.7–1.8.
14:	$\tilde{\mathit{p}}\leftarrow BA-IB(\tilde{p}\left(\hat{x}\right|x);p_{Y|X}{p}_X,\beta +\Delta \beta )$.
	▹ Ensure accuracy of the reduced root, using BA-IB Algorithm 1 till convergence.
15:	end if
16:	$\beta \leftarrow \beta +\Delta \beta$.
17:	$\tilde{\mathit{p}}\leftarrow B{A}_{\beta }(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right))$ ▹ A single BA-IB iteration in decoder coordinates.
18:	end while
19:	Append $\tilde{\mathit{p}}$ to $results$.
20:	return $results$.
21:	end function

With that, there are caveats in Algorithm 3, which stem from passing too far or close to a bifurcation. For one, suppose that the error accumulated from the true solution is too large for a bifurcation to be detected. The approximations generated by the algorithm will then overshoot the bifurcation. Namely, it will proceed with more clusters than needed until the conditions for reduction are met later on (see Section 6.3 below), as demonstrated by the two sparse grids in Figure 10 (Section 6.2). For another, suppose that the current grid point $\tilde{\mathit{p}}$ is too close to a bifurcation. This might happen due to a variety of numerical reasons, e.g., thresholds ${\delta }_1,{\delta }_2$ too small, or due to the particular grid layout. The coefficients matrix $I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ of the IB ODE (16) (which is the Jacobian of the IB operator (5)) would then be ill-conditioned, typically resulting in very large implicit numerical derivatives $\mathit{v}$ on step 6; cf., Conjecture 1 ff. in Section 5.1. Any inaccuracy in $\mathit{v}$ might then send the next grid point astray, derailing the algorithm from there on (e.g., inaccuracies due to the accumulated approximation error or due to the error caused by computing implicit derivatives in the vicinity of a bifurcation—see Figure 3 (top) in Section 3). Indeed, the derivatives $\frac{d\mathit{x}}{d\beta }=-{\left(D_{\mathit{x}}F\right)}^{-1}{D}_{\beta }F$ defined by the implicit ODE (7) are in general unbounded near a bifurcation of F (in our case, $D_{\mathit{x}}F$ is always non-singular outside bifurcations, due to Conjecture 1 and the use of reduced coordinates). This can be seen in Figure 2 (Section 2) for example, where the derivatives “explode” at the bifurcation’s vicinity. See also [6] (Section 7.2) on the computational difficulty incurred by a bifurcation. While overshooting a bifurcation is not a significant concern for our purposes (see Section 6.3), passing too close to one is. The latter is important, especially when the step size $|\Delta \beta |$ is small. While decreasing $|\Delta \beta |$ generally improves the error of Euler’s method, it also makes it easier for the approximations to come close to a bifurcation, thus potentially worsening the approximation dramatically if it derails. This motivates one to consider how singularities of the IB ODE (16) should be handled.

Algorithm 4 A heuristic for handling singularities of the IB ODE (16)
1:	function Handle singularity($p_{Y|X}{p}_X,\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right),\mathit{v},\beta$)
Input:
	An IB problem definition $p_{Y|X}{p}_X$, with $\forall x{p}_X\left(x\right)>0$.
	An approximate root $\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$ of the given problem, near a singularity of the IB ODE (16).
	Approximate numerical derivatives $\mathit{v}:=\left(\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta },\frac{dlog\tilde{p}\left(\hat{x}\right)}{d\beta }\right)$ at the given root.
	The $\beta >0$ value of the next (output) grid point.
Output: An approximate IB root $\tilde{\mathit{p}}$ at $\beta$ on one fewer cluster.
2:	${\hat{x}}^{\prime },{\hat{x}}^{\prime \prime }\leftarrow$ the two indices $\hat{x}$ of largest ${∥\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta }∥}_{\infty }$ value (norm of y-indexed vectors).
3:	$\tilde{p}\left(y\right|{\hat{x}}^{\prime })\leftarrow \frac{1}{2}·\left(\tilde{p}\left(y\right|{\hat{x}}^{\prime })+\tilde{p}\left(y\right|{\hat{x}}^{\prime \prime })\right)$ ▹ Replace fastest-moving clusters by their mean.
4:	Erase ${\hat{x}}^{\prime \prime }$ from the decoder $\tilde{p}\left(y\right|\hat{x})$.
5:	$\tilde{p}\left({\hat{x}}^{\prime }\right)\leftarrow \tilde{p}\left({\hat{x}}^{\prime }\right)+\tilde{p}\left({\hat{x}}^{\prime \prime }\right)$
6:	Erase ${\hat{x}}^{\prime \prime }$ from the marginal $\tilde{p}\left(\hat{x}\right)$.
7:	$\tilde{p}\left(\hat{x}\right|x)\leftarrow$ the encoder generated from $(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right))$, via Steps 1.7–1.8.
	▹ A new encoder on one cluster less than the input.
8:	$\tilde{\mathit{p}}\leftarrow BA-IB(\tilde{p}\left(\hat{x}\right|x);p_{Y|X}{p}_X,\beta )$. ▹ Re-gain accuracy, by the BA-IB Algorithm 1.
9:	return $\tilde{\mathit{p}}$
10:	end function

Next, we elaborate on our heuristic for handling singularities of the IB ODE (16), Algorithm 4. The inputs of this heuristic are defined as in Algorithm 3. It starts with the assumption that the coefficients matrix $I-D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ of the IB ODE (16) is nearly singular at the current grid point $\tilde{\mathit{p}}$ due to a nearby bifurcation (although a priori the Jacobian $D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}(Id-B{A}_{\beta })$ may be singular also due to other reasons, by Conjecture 1 it is non-singular at the approximations generated so far since they are assumed to be in their reduced form—see Section 5.1). As a result, the implicit derivatives $\mathit{v}$ at $\tilde{\mathit{p}}$ are not to be used directly to extrapolate the next grid point, as explained above. Instead, we use them to identify the two fastest moving clusters, on step 2 of Algorithm 4 (while this can be refined to handle more than two fast-moving clusters at once, that is not expected to be necessary for typical bifurcations). These are replaced by a single cluster (steps 3 through 6), resulting in an approximate root on one fewer cluster. To re-gain accuracy, the BA-IB Algorithm 1 is then invoked (on step 8) on the encoder generated (on step 7) from the latter root, thereby generating the next grid point. If the fastest-moving clusters have merged (in the true solution) by the next grid point, then the output of Algorithm 4 will be an IB-optimal root if its input grid point is so. Namely, the branch followed by the algorithm remains an optimal one. Otherwise, if these clusters merge shortly after the next grid point, then Algorithm 4 yields a sub-optimal branch. However, optimality is re-gained shortly afterward since the sub-optimal branch collides and merges with the optimal one in continuous IB bifurcations (Section 5.3). Figure 10 below demonstrates Algorithm 4. cf., [6] (Section 3.2) on the similar heuristic in root tracking for RD, which may also lose optimality near a bifurcation and re-gain it shortly after.

Algorithm 5 First-order Root Tracking for the IB (IBRT1)
1:	function IBRT1($p_{Y|X}{p}_X,{\beta }_0,p_{{\beta }_0}\left(\hat{x}\right|x);\Delta \beta ,{\delta }_1,{\delta }_2,{\delta }_3$)
Input:
	An IB problem definition $p_{Y|X}{p}_X$ with $\forall x{p}_X\left(x\right)>0$.
	A reduced IB-optimal root $p_{{\beta }_0}\left(\hat{x}\right|x)$ at ${\beta }_0$. A step size $\Delta \beta <0$.
	Thresholds $0<{\delta }_1,{\delta }_2<1$ for the root-reduction Algorithm 2 (cluster mass and merging).
	A threshold $0<{\delta }_3<1$ for eigenvalues’ singularity.
Output: Approximations ${\tilde{\mathit{p}}}_{{\beta }_n}$ of the optimal IB roots ${\mathit{p}}_{{\beta }_n}$ at ${\beta }_n:={\beta }_0+n\Delta \beta$.
2:	Initialize $\beta \leftarrow {\beta }_0$ and $results\leftarrow \left\{\right\}$.
3:	Initialize $\tilde{\mathit{p}}:=\left(\tilde{p}\left(\hat{x}\right|x),\tilde{p}\left(x\right|\hat{x}),\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$ from $p_{{\beta }_0}\left(\hat{x}\right|x)$, via Steps 1.4–1.6.
4:	while $\beta >|\Delta \beta |$ and $\left|supp\tilde{p}\left(\hat{x}\right)\right|>1$ do
5:	Append $\tilde{\mathit{p}}$ to $results$.
6:	$\mathit{v}:=\left(\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta },\frac{dlog\tilde{p}\left(\hat{x}\right)}{d\beta }\right)\leftarrow$ solve the IB ODE (16) at $\tilde{\mathit{p}}$.
7:	$eigs\leftarrow eig\left(I-S\right){|}_{\tilde{\mathit{p}}}$ ▹ Test ODE for singularity, using S (17) from Lemma 1.
8:	if $\left({min}_{v\in eigs}\left|v\right|\right)<{\delta }_3$ then          ▹ ODE is nearly singular.
9:	$\tilde{\mathit{p}}\leftarrow \mathrm{H}\mathrm{A}\mathrm{N}\mathrm{D}\mathrm{L}\mathrm{E}\mathrm{S}\mathrm{I}\mathrm{N}\mathrm{G}\mathrm{U}\mathrm{l}\mathrm{A}\mathrm{R}\mathrm{I}\mathrm{T}\mathrm{Y}(p_{Y|X}{p}_X,\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right),\mathit{v},\beta +\Delta \beta )$
	▹ Handle an otherwise undetected singularity using Algorithm 4.
10:	else
11:	$\tilde{p}\left(y\right|\hat{x})\leftarrow \tilde{p}\left(y\right|\hat{x})exp\left(\Delta \beta ·\frac{dlog\tilde{p}\left(y\right|\hat{x})}{d\beta }\right)$
12:	$\tilde{p}\left(\hat{x}\right)\leftarrow \tilde{p}\left(\hat{x}\right)exp\left(\Delta \beta ·\frac{dlog\tilde{p}\left(\hat{x}\right)}{d\beta }\right)$
13:	$\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)\leftarrow \mathrm{normalize}\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$
14:	$old\_dim\leftarrow dim\tilde{p}\left(\hat{x}\right)$
15:	$\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)\leftarrow \mathrm{R}\mathrm{E}\mathrm{D}\mathrm{U}\mathrm{C}\mathrm{E}\mathrm{R}\mathrm{O}\mathrm{O}\mathrm{T}(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right);{\delta }_1,{\delta }_2)$.
16:	if $old\_dim\ne dim\tilde{p}\left(\hat{x}\right)$ then
17:	$\tilde{p}\left(\hat{x}\right|x)\leftarrow$ encoder defined from $\left(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right)\right)$, via Steps 1.7–1.8.
18:	$\tilde{\mathit{p}}\leftarrow \mathrm{BA}-\mathrm{IB}(\tilde{p}\left(\hat{x}\right|x);p_{Y|X}{p}_X,\beta +\Delta \beta )$.
19:	end if
20:	end if
21:	$\beta \leftarrow \beta +\Delta \beta$.
22:	$\tilde{\mathit{p}}\leftarrow B{A}_{\beta }(\tilde{p}\left(y\right|\hat{x}),\tilde{p}\left(\hat{x}\right))$
23:	end while
24:	Append $\tilde{\mathit{p}}$ to $results$.
25:	return $results$.
26:	end function

The heuristic Algorithm 4 is motivated by cluster-merging bifurcations. In these, the implicit derivatives are very large only at the coordinates $\frac{dlogp\left(y\right|\hat{x})}{d\beta }$ of the points colliding in $\Delta \left[\mathcal{Y}\right]$ (note that cluster masses barely change in the vicinity of a cluster merging, until the point of bifurcation itself). While intended for cluster-merging bifurcations, this heuristic works nicely in practice also for cluster-vanishing ones. To see why, note that one can always add a cluster of zero mass to an IB root without affecting the root’s essential properties, regardless of its coordinates in $\Delta \left[\mathcal{Y}\right]$ (cf., Section 5.1 on reduction in the IB). Therefore, a numerical algorithm may, in principle, do anything with the coordinates $p\left(y\right|\hat{x})\in \Delta \left[\mathcal{Y}\right]$ of a nearly vanished cluster $\hat{x}$, $p\left(\hat{x}\right)\simeq 0$, without affecting the approximation’s quality too much. Thus, for numerical purposes, one may treat a cluster-vanishing bifurcation as a cluster-merging one. Conversely, in a cluster-merging bifurcation, a numerical algorithm may, in principle, zero the mass of one cluster while adding it to the remaining cluster, again without affecting the approximation’s quality too much. To conclude, for numerical purposes, cluster vanishing is very similar to cluster merging. A variety of treatments between these extremities may be possible by a numerical algorithm. Empirically, we have observed that our ODE-based algorithm treats both as cluster-merging bifurcations. To our understanding, this is because our algorithm operates in decoder coordinates, unlike the BA-IB Algorithm 1, for example, which operates in encoder coordinates.

Finally, we combine the simplified root-tracking Algorithm 3 with the heuristic Algorithm 4 for handling singularities, yielding our IBRT1 Algorithm 5. It follows the lines of the simplified Algorithm 3, except that after solving for the implicit derivatives on step 6, we test the IB ODE (16) for singularity. To that end, we propose using the matrix S (17) (from Lemma 1 in Section 3), since its order $T·\left|\mathcal{Y}\right|$ is smaller than the order $T·\left(\right|\mathcal{Y}|+1)$ of the ODE’s coefficients matrix. This might make it computationally cheaper to test for singularity (on steps 7 and 8 of Algorithm 5). Our heuristic Algorithm 4 is invoked (on step 9) if the ODE (16) is found to be nearly singular, otherwise proceeding as in Algorithm 3.

6.2. Numerical Results for the IBRT1 Algorithm 5

To demonstrate the IBRT1 Algorithm 5, we present the numerical results used to approximate the IB curve in Figure 1 (Section 1)—see Section 6.3 below on the approximation quality and the algorithm’s basic properties. This example was chosen both because it has an analytical solution (Appendix E) and because it allows one to get a good idea of the bifurcation handling added (in Section 6.1) on top of the modified Euler method (from Section 4).

We discuss the numerical examples of this Section in light of the explanations provided in the previous Section 6.1. The error of the IBRT1 Algorithm 5 generally improves as the step-size $|\Delta \beta |$ becomes smaller, as expected. The single BA-IB iteration added to Euler’s method (in Section 4) typically allows one to achieve the same error by using much fewer grid points, thus lowering computational costs. For example, the two denser grids in Figure 10 require about an order of magnitude fewer points to achieve the same error compared to Euler’s method for the IB; this can be seen from Figure 4 (Section 4).

In sparse grids, the approximations often pass too far away from a bifurcation for the root-reduction Algorithm 2 to detect it. When overshooting it, the conditions for numerical reduction are generally met later on, as discussed in Section 6.3 below. Decreasing $|\Delta \beta |$ further often leads the approximations too close to a bifurcation, as can be seen in the densest grid of Figure 10. The implicit derivatives are typically very large at the proximity of a bifurcation, while the least accurate there (see Section 6.1). As these might send subsequent grid points off-track, the heuristic Algorithm 4 is invoked to handle the nearby singularity (see inset of Figure 10). As noted earlier, the computational difficulty in tracking IB roots (or root tracking in general) stems from the presence of a bifurcation, manifested here by large approximation errors in its vicinity. While the algorithm’s error peaks at the bifurcation, it typically decreases afterward when overshooting, as seen in Figure 11. The reasons for this are discussed below in Section 6.3.

6.3. Basic Properties of the IBRT1 Algorithm 5 and Why It Works

Apart from presenting the basic properties of the IBRT1 Algorithm 5, the primary purpose of this section is to understand why it approximates the problem’s true IB curve (1) so well, despite its apparent errors in approximating the IB roots. While shown here only in Figure 1 and Figure 10 (in Section 1 and Section 6.2), this behavior is consistent in the few numerical examples that we have tested. We offer an explanation why this may be true in general.

To understand why the IBRT1 Algorithm 5 approximates the true IB curve (1) so well, we first explain why overshooting is not a significant concern, as noted earlier in Section 6.1. To that end, consider the implicit ODE (7)

$$\frac{d\mathit{x}}{d\beta }=-{\left(D_{\mathit{x}}F\right)}^{-1}{D}_{\beta }F,$$

from Section 1. As long as $D_{\mathit{x}}F$ and $D_{\beta }F$ at its right-hand side are well-defined, it defines a vector field on the entire phase space of admissible $\mathit{x}$ values, at least when $D_{\mathit{x}}F$ is non-singular. That is, even for $\mathit{x}$’s which are not roots (6) of F. Ignoring several technicalities, the IB ODE (16) therefore defines a vector field also outside IB roots (although at a reduced root singularities of the IB ODE (16) coincide with IB bifurcations, the IB’s vector field might a priori be singular elsewhere). Indeed, due to Conjecture 1, the Jacobian of the IB operator $Id-B{A}_{\beta }$ (5) is non-singular in the vicinity of a reduced root ($D_{logp\left(y\right|\hat{x}),logp\left(\hat{x}\right)}B{A}_{\beta }$ (13) is continuous in the distributions defining it, and thus so are its eigenvalues, under mild assumptions—cf., Lemma A1 in Appendix A). Now, suppose that ${\mathit{p}}_{\beta }$ is an optimal IB root, and consider a point ${\mathit{p}}^{\prime }\ne {\mathit{p}}_{\beta }$ in its vicinity. An argument based on a strong notion of Lyapunov stability (in Appendix G) shows that ${\mathit{p}}^{\prime }$ flows along the IB’s vector field towards ${\mathit{p}}_{\beta }$ in regions that do not contain a bifurcation, though only if flowing in decreasing$\beta$ as done by our IBRT Algorithm 5. An approximation ${\mathit{p}}^{\prime }$ would then be “pulled” towards the true root. Stability in decreasing (rather than increasing) $\beta$ values is very reasonable, considering that ${\mathit{p}}_{\beta }$ follows a path of decreasingly informative representations as $\beta$ decreases. Indeed, all the paths to oblivion lead to one place—the trivial solution, whose representation in reduced coordinates is unique. As a result, a numerical approximation ${\mathit{p}}^{\prime }$ would gradually settle in the vicinity of the true root ${\mathit{p}}_{\beta }$ as seen in Figure 10 and Figure 11, so long as ${\mathit{p}}_{\beta }$ does not change much and the step-size $|\Delta \beta |$ is small enough. While this explanation obviously breaks near a bifurcation, it does suggest that the approximation error should decrease when overshooting it (see Section 6.1), once the true reduced root has settled down. In a sense, overshooting is similar to being in the right place but at the wrong time.

The above suggests that the IBRT1 Algorithm 5 should generally approximate the true IB curve (1) well, despite its errors in approximating IB roots. To see this, note that while ${\beta }^{-1}$ is the slope of the optimal curve (1) of the IB [1] (Equation (32)), for the IB ODE (16) it is merely an independent “time-like” variable. When solving for the optimal curve (1), one is not interested in an optimal root or in its $\beta$ value, but rather in its image $\left(I(X;\hat{X}),I(Y;\hat{X})\right)$ in the information plane. As a result, achieving the optimal roots but with the wrong $\beta$ values does yield the true IB curve (1), as required. This is the reason that the true curve (1) is achieved in Figure 1 (Section 1) even on sparse grids, despite the apparent approximation errors in Figure 10 and Figure 11 (Section 6.2). With that, we expect the approximate IB curve produced by the IBRT1 Algorithm 5 to be of lesser quality when there are more than two possible labels y. To see why, note that the space $\Delta \left[\mathcal{Y}\right]$ traversed by approximate clusters is not one-dimensional when $\left|\mathcal{Y}\right|>2$, and so it is possible to maneuver around the clusters of an optimal root.

Next, we briefly discuss the basic properties of the IBRT1 Algorithm 5. Its computational complexity is determined by the complexity of a single grid point. The latter is readily seen to be dominated by the complexity $O(T^2·{\left|\mathcal{Y}\right|}^2·(\left|\mathcal{X}\right|+T·\left|\mathcal{Y}\right|\left)\right)$ of computing the coefficients matrix of the IB ODE (16) and of solving it numerically (on step 6). To that, one should add the complexity of the BA-IB Algorithm 1 each time a root is reduced. However, the critical slowing down of BA-IB [4] is avoided since we reduce the root before invoking BA-IB (see Section 5.2). The complexity is only linear in $\left|\mathcal{X}\right|$ thanks to the choice of decoder coordinates. Had we chosen one of the other coordinate systems in Section 2, then solving the ODE would have been cubic in $\left|\mathcal{X}\right|$ rather than linear (see there). The computational difficulty in following IB roots stems from the existence of bifurcations (Section 4), as it generally is with following an operator’s root [6] (Section 7.2).

As noted in Section 4, convergence guarantees can be derived for Euler’s method for the IB when away from bifurcation, in terms of the step-size $|\Delta \beta |$, in a manner similar to [6] (Theorem 5) for RD. These imply similar guarantees for the IBRT1 Algorithm 5, since adding a single BA-IB iteration in our modified Euler method improves its order (see there). These details are omitted for brevity, however.

For a numerical method of order $d>0$ (see Section 4) with a fixed step-size $|\Delta \beta |$ and a fixed computational cost per grid point, the cost-to-error tradeoff is given by

$$error\propto cos{t}^{-d},$$

(26)

as in [6] (Equation (3.6)), when $|\Delta \beta |$ is small enough. See [26] for example. Figure 3.4 in [6] demonstrates for RD that methods of higher order achieve a better tradeoff, as expected, as in the fixed-order Taylor methods they employ. Since computing implicit derivatives of higher orders requires the calculation of many more derivative tensors of $Id-B{A}_{\beta }$ (5) than done here [6] (Section 2.2), we have used only first-order derivatives for simplicity. However, while the vanilla Euler method for the IB is of order $d=1$, the discussion in Section 4 (and Figure 4 in particular) suggests that the order d of the modified Euler method used by the IBRT1 Algorithm 5 is nearly twice than that; cf., Section 6.2 and Appendix D.

With that, we comment on the behavior of the IBRT1 Algorithm 5 at discontinuous bifurcations. Consider the problem in Figure 7 (Section 5.3), for example. When Algorithm 5 follows the optimal 2-clustered root there, the Jacobian’s singularity (in Figure 9) is detectable by it because the step size $\Delta \beta$ is negative (see the discussion in Section 5.3 there). Indeed, due to Conjecture 1 ff., the algorithm can detect discontinuous bifurcations in general. Whether a particular discontinuous bifurcation is detected by Algorithm 5 in practice depends on the details, of course, as with continuous bifurcations (e.g., on the threshold value ${\delta }_3$ for detecting singularity and on the precise grid point layout). Indeed, the details may or may not cause a particular example to be detected by the conditions on steps 7 and 8 (in Algorithm 5). If missed, Algorithm 5 will continue to follow the 2-clustered root in Figure 7 to the left of the bifurcation, where it is sub-optimal, just as BA-IB with reverse deterministic annealing would. Once detected, though, one may wonder whether the heuristic Algorithm 4 works well also for discontinuous bifurcations. The example of Figure 7 has just one single-clustered root to the left of the bifurcation. Thus, the BA-IB Algorithm 1 invoked on step 8 (of Algorithm 4) must converge to it. However, there may generally be more than a single root of smaller effective cardinality to the left of the bifurcation, to which BA-IB may converge. The handling of discontinuous bifurcations is left to future work. Such handling is expected to be easier in the IB than in RD, since, in contrast to RD, the effective cardinality of an optimal IB root cannot decrease with $\beta$ (bottom of Section 5.3). See Problems 2.8–2.10 in [28] for counter-examples in RD. This makes detecting discontinuous bifurcations easier in the IB and is also expected to assist with their handling.

We list the assumptions used along the way for reference. These are needed to guarantee the optimality of the IBRT1 Algorithm 5 at the limit of small step-sizes $|\Delta \beta |$, except at a bifurcation’s vicinity. In Section 1, it was assumed without loss of generality that the input distribution $p_X$ is of full support, $p\left(x\right)>0$ for every x (otherwise, one may remove symbols x with $p_X\left(x\right)=0$ from the source alphabet). The requirement $p\left(y\right|x)>0$ was added in Section 3 as a sufficient technical condition for exchanging to logarithmic coordinates (Lemma A1 in Appendix A), and could perhaps be alleviated in alternative derivations. Together, these are equivalent to having a never-vanishing IB problem definition, $p\left(y\right|x\left)p\right(x)>0$ for every x and y. The algorithm’s initial condition is assumed to be a reduced and optimal IB root, since reduction is needed by Conjecture 1 in Section 5.1. Finally, the given IB problem is assumed to have only continuous bifurcations, except perhaps for its first (leftmost) one. While these assumptions are sufficient to guarantee optimality, we note that milder conditions might do in a particular problem.

7. Concluding Remarks

The IB is intimately related to several problems in adjacent fields [3], including coding problems, inference, and representation learning. Despite its importance, there are surprisingly few techniques to solve it numerically. This work attempts to fill this gap by exploiting the dynamics of IB roots.

The end result of this work is a new numerical algorithm for the IB, which follows the path of a root along the IB’s optimal tradeoff curve (1). A combination of several novelties was required to achieve this goal. First, the dynamics underlying the IB curve (1) obeys an ODE [10]. Following the discussion around Conjecture 1 (in Section 5.1), the existence of such a dynamics stems from the analyticity of the IB’s fixed-point Equations (2)–(4), thus typically resulting in piece-wise smooth dynamics of IB roots. Several natural choices of a coordinate system for the IB were considered, both for computational purposes and to facilitate a clean treatment of IB bifurcations below. The IB’s ODE (16) was derived anew in appropriate coordinates, allowing an efficient computation of implicit derivatives at an IB root. Combining BA-IB with Euler’s method yields a modified numerical method whose order is higher than either.

Second, one needs to understand where the IB ODE (16) is not obeyed, thereby violating the differentiability of an optimal root with respect to $\beta$. To that end, one not only needs to detect IB bifurcations but also needs to identify their type in order to handle them properly. Unlike standard techniques, our approach is to remove redundant coordinates, following root tracking for RD [6] (see Section 1). To achieve a reduction, we follow the arguably better definition of the IB in [20]. Namely, a finite IB problem is an RD problem on the continuous reproduction alphabet $\Delta \left[\mathcal{Y}\right]$. Therefore, the IB may be intuitively considered as a method of lossy compression of the information on Y embedded in X. Viewing a finite IB problem as an infinite RD problem suggests a particular choice of a coordinate system for the IB, which enables reduction in the IB; this extends reduction in RD [6]. Furthermore, this point of view highlights subtleties of finite dimensionality in computing representations of IB roots. To our understanding, these subtleties hindered the understanding of IB bifurcations throughout the years.

Combining the above allows us to translate an understanding of IB bifurcations to a new numerical algorithm for the IB (the IBRT1 Algorithm 5). There are several directions that one could consider to improve our algorithm. Near bifurcations, one could improve its handling of discontinuous bifurcations. While we used implicit derivatives only of the first order for simplicity, higher-order derivatives generally offer a better cost-to-error tradeoff when away from bifurcations. See also [6] (Section 3.4) on possible improvements for following an operator’s root.