Optimizing Distributions for Associated Entropic Vectors via Generative Convolutional Neural Networks

Abstract

The complete characterization of the almost-entropic region yields rate regions for network coding problems. However, this characterization is difficult and open. In this paper, we propose a novel algorithm to determine whether an arbitrary vector in the entropy space is entropic or not, by parameterizing and generating probability mass functions by neural networks. Given a target vector, the algorithm minimizes the normalized distance between the target vector and the generated entropic vector by training the neural network. The algorithm reveals the entropic nature of the target vector, and obtains the underlying distribution, accordingly. The proposed algorithm was further implemented with convolutional neural networks, which naturally fit the structure of joint probability mass functions, and accelerate the algorithm with GPUs. Empirical results demonstrate improved normalized distances and convergence performances compared with prior works. We also conducted optimizations of the Ingleton score and Ingleton violation index, where a new lower bound of the Ingleton violation index was obtained. An inner bound of the almost-entropic region with four random variables was constructed with the proposed method, presenting the current best inner bound measured by the volume ratio. The potential of a computer-aided approach to construct achievable schemes for network coding problems using the proposed method is discussed.

1. Introduction

Given n discrete random variables, for a fixed joint distribution, all their $2^n-1$ (joint) entropies define an entropic vector in the entropy space ${\mathbb{R}}^{2^n-1}$. By varying over all possible joint distributions, the set of all entropic vectors defines the entropic region. The entropic region plays a fundamental role in information theory and network coding. The closure, known as the almost-entropic region, yields rate regions for multi-source coded networks [1,2]. However, the complete characterization of the entropic region encounters challenges even when $n=3$ [3,4,5,6,7], where only the closure, i.e., the almost-entropic region, is fully characterized [3]. When $n=4$, the characterization of the almost-entropic region also becomes extremely difficult and remains open [8].

In the pursuit of the characterization of the entropic region, one crucial problem is to determine whether an arbitrary vector in the entropy space is entropic or not. If all entropic vectors in the entropy space are verified, the entropic region is fully characterized. Additionally, given a point in the rate region of a coded network, the existence of achievable codes relies on the underlying entropic vector [9].

This problem has been tackled from different perspectives in the literature. Information inequalities, which fully characterize all almost-entropic vectors, are shown to be infinitely many [10] and extremely hard to characterize [8,10,11,12,13,14,15,16,17]. The construction of entropic vectors, however, is more feasible and studied through probability distributions, e.g., [6,7,16,18,19,20,21,22,23,24,25,26,27,28], groups, e.g., [29,30,31,32,33,34,35], and matroids, e.g., [36,37,38,39,40]. Regarding the construction of entropic vectors from probability distributions, remarkable research has attempted to numerically obtain probability mass functions (PMFs) for entropic vectors [16,19,20,21,22,23,24,25,26,27], particularly when the alphabet sizes of the involved random variables are bounded. In this way, the entropic nature of a given vector in the entropy space is immediately verified following the definition if the underlying joint PMF is obtained.

Among these approaches, some are interested in quasi-uniform PMFs [19,27] or PMFs supported on small atoms [22,23,24,25], while more general PMFs are considered in [16,20,21,26]. For example, an algorithm is given in [20] to verify binary entropic vectors by recursively constructing PMFs. In [16], the PMFs are numerically obtained with Newton’s method. In [21], the study of the almost-entropic region when $n=4$ is reduced to a three-dimensional tetrahedron and visualized by various maximization procedures with different methods and strategies [21] (Section VII). Recently, a random search algorithm has been introduced [26] to find the nearest entropic vector to a given target vector. This algorithm iteratively tries a few random perturbations on PMFs, striving to decrease the normalized distance between an entropic vector and the target vector.

However, the limitations of the above methods cannot be overlooked. The method in [20] is only feasible for binary entropic vectors. In [16], and [21], formal algorithms and performances of described methods are not systematically discussed in detail. Although [26] presents more analytical and empirical results, the limitation of the randomized algorithm persists, and the convergence performances are not empirically guaranteed. Furthermore, a critical issue in above methods is that only PMFs with relatively small alphabet sizes can be handled. This limitation is vital, since the verification of an underlying entropic vector in the rate region of a coded network requires large alphabets when subpacketizations are necessary for achievable codes.

With these methods, which construct entropic vectors from PMFs, some interesting optimization problems over the entropic region when $n=4$ can be experimented. It is known that the Ingleton inequality [41] completely characterizes the part of the almost-entropic region achieved by linearly representable entropic vectors [13]; thus, it reduces the study to the entropic vectors violating it. To measure the degree of violation from the Ingleton inequality for an entropic vector, the Ingleton score [16,42] and Ingleton violation index [19] are proposed from different angles. Although remarkable efforts have been made to minimize the Ingleton score [16,21,26,35,42] and maximize the Ingleton violation index [19,26], their optimal values remain open. In [16], the Ingleton score is numerically optimized and conjectured to be $-0.089373$, known as the four-atom conjecture [16] (Conjecture 1). In [21], a technique is proposed to transform an entropic vector into another entropic vector, and the transformed vector is optimized to the Ingleton score $-0.09243$, which refutes the four-atom conjecture by [16]. By experimenting with groups [35], the best-known Ingleton score is currently $-0.0925$. The Ingleton violation index is optimized in [19] and, subsequently, in [43], and its best known value is currently $0.0281316$, according to [26].

As the complete characterization of the almost-entropic region for $n=4$ is difficult, inner bounds can be constructed by taking the convex hull of certain acquired entropic vectors, as previously investigated in [16,22,26]. The quality of such an inner bound is measured by the volume ratio, i.e., the percentage of inner-bound polytope volume to the volume of the outer-bound polytope. In [16], several entropic vectors are optimized to form an inner bound with a volume ratio of $53.4815\%$. Using distributions supported on small atoms, [22] finds entropic vectors that yield an inner bound with the volume ratio $57.8\%$. In [26], with their proposed grid method, the current best-known volume ratio is $62.4774\%$, while the largest volume ratio, which requires the complete characterization of the almost-entropic region when $n=4$, is still open.

In this paper, we develop a novel architecture for optimizing PMFs for associated entropic vectors via convolution neural networks (CNNs). Recently, applications of neural networks (NNs) can be found in information theory [44,45,46] and control theory [47,48]. Our motivation arises from the fruitful research on applying NNs to problems in information theory [44,45,46]. More specifically, in [45], a unique mapping that generates conditional PMFs is defined and approximated using NNs. In [46], a model which generates a capacity-achieving input PMF for a given channel in discrete input spaces is proposed. Consequently, we are especially interested in approximating another mapping, which generates joint PMFs for multiple random variables. Sharing similar ideas with [26], the problem of verifying an entropic vector can be performed by minimizing the normalized distance through NN training. Furthermore, we are motivated to modify NNs due to the special structures and complexities of joint PMFs. Additionally, the developed method can be applied immediately to the optimizations of the Ingleton score and Ingleton violation index, and the construction of inner bounds for the almost-entropic region.

The major contributions of this paper are summarized as follows.

• A novel algorithm is proposed to optimize distributions for entropic vectors with NN training. For each target vector, an NN is trained such that the output PMF produces an entropic vector as close to target as possible. In practice, we implement the algorithm with CNNs, which accelerate and enable the algorithm to generate PMFs with large alphabets.
• The effectiveness of our proposed method is verified by empirical results. More specifically, smaller normalized distances and improved convergence performances are achieved by our proposed method, compared to [26]. In addition, with derived theoretical guarantees, by exploiting the proposed algorithm, the state-of-the-art Ingleton score is reconfirmed, and a new tighter lower bound of the Ingleton violation index is obtained. Furthermore, by utilizing the proposed algorithm, we develop another algorithm to construct a new inner bound of the almost-entropic region ($n=4$), yielding the current best inner bound measured by the volume ratio.

This paper is organized as follows. Section 2 introduces preliminaries, notations, and the problem statement. The proposed method and algorithm with derived theoretical guarantees are presented in Section 3, and the implementation with CNNs is demonstrated in Section 4. In Section 5, empirical results exploiting the proposed method for several problems when $n=4$ are presented. Section 6 summarizes the paper in general, and discusses the potential of the proposed method to construct achievable schemes for network coding problems.

2. Preliminaries and Problem Statement

In this section, we provide the preliminaries, notations, and the problem statement of this paper.

2.1. Convex Cones and Convex Polytopes

Given a set $C\subset {\mathbb{R}}^d$, C is a pointed cone if $\mathbf{a}$$\in C$ implies that $t\mathbf{a}\in C$ for all real $t\ge 0$, and $R_{\mathbf{a}}=\{t\mathbf{a},t\in {\mathbb{R}}^{+}\}\subset C$ is a ray of C. Pointed cones are unbounded except $\left\{\mathbf{0}\right\}$, where $\mathbf{0}$ is the origin of Euclidean space. In the rest of the paper, all cones are assumed to be pointed and unbounded. For all real $t\ge 0$, $R_{\mathbf{b}}=\left\{t\mathbf{b}\right\}$ of C is an extreme ray if $\mathbf{b}$ cannot be expressed as the positive linear combination of any ${\mathbf{a}}_{\mathbf{1}},{\mathbf{a}}_{\mathbf{2}}\in C$. A cone C is convex if C is a convex set.

Given a convex set $C\subset {\mathbb{R}}^d$, C is a convex polyhedron if it is the intersection of finitely many halfspaces (i.e., linear inequalities). A convex cone C is a special polyhedron when the halfspaces that define C contain $\mathbf{0}$ simultaneously, and, in this case, the convex cone C is called polyhedral. A convex polyhedron is a convex polytope if it is bounded. In the rest of the paper, all polytopes are assumed to be convex. For a convex cone (or convex polytope), if we specify one extreme ray (or vertex) as the top and others as the bases, then the convex cone (or convex polytope) is often called the pyramid.

For a convex polyhedron C, there are two equivalent types of representations, i.e., the H-representation and the V-representation [49] (Chapter 1). The H-representation is the set of all halfspaces defining C, and the V-representation is the set of all vertices (if C is a convex polytope) or extreme rays (if C is a polyhedral convex cone) defining C. The transformation between the H-representation and the V-representation for a given convex cone or convex polytope can be numerically performed with the implementation [50,51,52] of the double description method [53]. More details about convex cones and convex polytopes can be found in [49].

2.2. Entropic Vectors and Entropic Region

Let $n\ge 2$, consider a discrete random vector $\mathbf{X}\triangleq (X_1,X_2,\dots ,X_n)$ with a finite index set ${\mathcal{N}}_n=\{1,2,\dots ,n\}$, and $\mathbf{X}$ takes values in the finite alphabet $\mathcal{X}\triangleq {\mathcal{X}}_1\times {\mathcal{X}}_2\times \dots \times {\mathcal{X}}_n$ of size $\left|\mathcal{X}\right|=m={\prod }_{i=1}^n{m}_i$. The realization of $\mathbf{X}$ is denoted as $\mathbf{x}$. Let $p\left(\mathbf{x}\right)$ denote the probability of $\mathbf{x}$ such that $p\left(\mathbf{x}\right)=\mathrm{Pr}\{\mathbf{X}=\mathbf{x}\}$. The joint probability mass function (PMF) of $\mathbf{X}$ is denoted as $\mathbf{p}\triangleq {[p\left(\mathbf{x}\right),\mathbf{x}\in \mathcal{X}]}^{\mathsf{T}}\in {\Delta }_{\mathcal{X}}^m$, where ${\Delta }_{\mathcal{X}}^m$ is the m-dimensional probability simplex defined as

$${\Delta }_{\mathcal{X}}^m\triangleq \left\{\mathbf{p}\in {\mathbb{R}}^m|\sum p\left(\mathbf{x}\right)=1,p\left(\mathbf{x}\right)\ge 0,\forall \mathbf{x}\in \mathcal{X}\right\}.$$

(1)

A vector in ${\mathbb{R}}^m$ is a valid PMF if it belongs to the set ${\Delta }_{\mathcal{X}}^m$.

We consider a subset of components of the random vector $\mathbf{X}$; the above quantities can be denoted accordingly. More specifically, we consider the set of random variables ${\mathbf{X}}_{\alpha }\triangleq \{X_i,i\in \alpha \},\alpha \subseteq {\mathcal{N}}_n\setminus \varnothing$, which takes values in the finite alphabet ${\mathcal{X}}_{\alpha }={\prod }_{i\in \alpha }{\mathcal{X}}_i$. The realization of ${\mathbf{X}}_{\alpha }$ is denoted as ${\mathbf{x}}_{\alpha }$. Let $p_{\alpha }\left({\mathbf{x}}_{\alpha }\right)$ denote the probability of ${\mathbf{x}}_{\alpha }$ such that $p_{\alpha }\left({\mathbf{x}}_{\alpha }\right)=\mathrm{Pr}\{{\mathbf{X}}_{\alpha }={\mathbf{x}}_{\alpha }\}$. The marginal PMF of ${\mathbf{X}}_{\alpha }$ is denoted as ${\mathbf{p}}_{\alpha }\triangleq {[p_{\alpha }\left({\mathbf{x}}_{\alpha }\right),{\mathbf{x}}_{\alpha }\in {\mathcal{X}}_{\alpha }]}^{\mathsf{T}}$. Then, the Shannon entropy of ${\mathbf{X}}_{\alpha }$ is

$$H\left({\mathbf{X}}_{\alpha }\right)=-\sum _{{\mathbf{x}}_{\alpha }\in {\mathcal{X}}_{\alpha }}{p}_{\alpha }\left({\mathbf{x}}_{\alpha }\right){\log }_2{p}_{\alpha }\left({\mathbf{x}}_{\alpha }\right),\forall \alpha \subseteq {\mathcal{N}}_n\setminus \varnothing ,$$

(2)

where $0{\log }_{2}0\triangleq 0$. We define $h_{\alpha }\triangleq H\left({\mathbf{X}}_{\alpha }\right)$, which is the entropy function. The vector consisting of entropy functions for all $\alpha \subseteq {\mathcal{N}}_n\setminus \varnothing$ is the entropic vector, as formally defined in the following definition.

Definition 1

(Entropic vectors [54], Chapter 13). For the random vector $\mathbf{X}$ and index set ${\mathcal{N}}_n$, given a joint PMF $\mathbf{p}\in {\Delta }_{\mathcal{X}}^m$, the entropic vector is defined as

$$\begin{array}{c}\hfill \mathbf{h}\triangleq {\left(h_{\alpha },\alpha \subseteq {\mathcal{N}}_n\setminus \varnothing \right)}^{\mathsf{T}},\end{array}$$

(3)

and $\mathbf{h}$ is the associated entropic vector of $\mathbf{p}$, denoted as ${\mathbf{h}}^{\mathbf{p}}$.

We note that, by varying $\alpha$, there are total $2^n-1$ (joint) entropies for the random vector $\mathbf{X}$. Thus, each $\mathbf{h}$ can be viewed as a vector in the $(2^n-1)$-dimensional Euclidean space, which is defined as the entropy space

$${\mathcal{H}}_n\triangleq {\mathbb{R}}^{2^n-1},n\ge 2.$$

(4)

For example, given a random vector $\mathbf{X}=(X_1,X_2,X_3,X_4)$, let each random variable in $\mathbf{X}$ be uniformly i.i.d.-distributed on alphabet $\{0,1\}$; then, the associated entropic vector in the entropy space ${\mathcal{H}}_4={\mathbb{R}}^{15}$ is

$$\begin{array}{cc}\hfill {\mathbf{h}}^{\mathbf{p}}& ={\left(h_1,h_2,h_3,h_4,h_{12},h_{13},h_{14},h_{23},h_{24},h_{34},h_{123},h_{124},h_{134},h_{234},h_{1234}\right)}^{\mathsf{T}}.\hfill \\ \hfill & ={(1,1,1,1,2,2,2,2,2,2,3,3,3,3,4)}^{\mathsf{T}}.\hfill \end{array}$$

(5)

Allowing infinite alphabets, the region in ${\mathcal{H}}_n$ consisting of all entropic vectors is the entropic region ([54], Chapter 13)

$${\Gamma }_n^{\ast }\triangleq \left\{\mathbf{h}\in {\mathcal{H}}_n|\mathbf{h}\mathrm{is}\mathrm{entropic}\mathrm{vector}\right\}.$$

(6)

The closure of ${\Gamma }_n^{\ast }$ is the almost-entropic region${\overline{\Gamma }}_n^{\ast }$ ([54], Chapter 15).

Given a set of random variables, basic inequalities form the set of inequalities implied by the nonnegativity of all Shannon’s information measures. The nonredundant subset of basic inequalities consists of elemental inequalities ([54], Chapter 14), which are defined as the following two types of inequalities for the random vector $\mathbf{X}$

$$\begin{array}{cc}\hfill H\left(X_i\right|{\mathbf{X}}_{{\mathcal{N}}_n\setminus \left\{i\right\}})& \ge 0,i\in {\mathcal{N}}_n,\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill I(X_i;X_j|{\mathbf{X}}_K)& \ge 0,i\ne j,K\subset {\mathcal{N}}_n\setminus \{i,j\}.\hfill \end{array}$$

(8)

The inequalities implied by elemental inequalities are Shannon-type inequalities. The region in ${\mathcal{H}}_n$ that consists of vectors satisfying all elemental inequalities is ([54], Chapter 14)

$${\Gamma }_n\triangleq \left\{\mathbf{h}\in {\mathcal{H}}_n\left|\mathbf{h}\mathrm{satisfies}\right(7\left)\mathrm{and}\right(8)\right\}.$$

(9)

Sometimes ${\Gamma }_n$ is called the polymatroidal region because of the equivalence between the elemental inequalities for vectors in the entropy space and the polymatroidal axioms for polymatroidal rank functions [55].

Although both ${\Gamma }_n^{\ast }$ and ${\Gamma }_n$ are regions within ${\mathcal{H}}_n$, the former stands for the associated entropic vectors defined by valid PMFs, while the latter is formed by vectors satisfying all Shannon-type inequalities. Hence, it is reasonable to question the identity between ${\Gamma }_n^{\ast }$ and ${\Gamma }_n$. It was first discovered by [3,8] that ${\Gamma }_n$ is a loose outer bound of ${\Gamma }_n^{\ast }$. More specifically, it is known that ${\Gamma }_2^{\ast }={\Gamma }_2$, ${\Gamma }_3^{\ast }\ne {\Gamma }_3$ but ${\overline{\Gamma }}_3^{\ast }={\Gamma }_3$, and ${\overline{\Gamma }}_n^{\ast }\subset {\Gamma }_n$ when $n\ge 4$ [3,8]. These relations reveal that there are inequalities tighter than Shannon-type inequalities as the outer bound of the entropic region, i.e., the inequalities hold for all entropic vectors but cannot be implied by elemental inequalities, and these inequalities are often referred to as the non-Shannon-type inequalities ([54], Chapter 15).

We briefly introduce some known structures of ${\Gamma }_n^{\ast }$, ${\overline{\Gamma }}_n^{\ast }$ and ${\Gamma }_n$. Both ${\Gamma }_n$ and ${\overline{\Gamma }}_n^{\ast }$ are pointed convex cones in the nonnegative orthant of ${\mathcal{H}}_n$ [54] (Chapter 13). The region ${\Gamma }_n$ is polyhedral, while ${\overline{\Gamma }}_n^{\ast }$ is not when $n\ge 4$ [10], i.e., the convex cone ${\Gamma }_4$ (${\overline{\Gamma }}_4^{\ast }$) is represented with finitely many (infinitely many) extreme rays or finitely many (infinitely many) linear inequalities. For $n\ge 3$, although the almost-entropic region ${\overline{\Gamma }}_n^{\ast }$ is a convex cone, the region ${\Gamma }_n^{\ast }$ is not convex and ${\Gamma }_n^{\ast }\subset {\overline{\Gamma }}_n^{\ast }$. The complete characterization of ${\overline{\Gamma }}_n^{\ast }$ for $n\ge 4$ is difficult and open. For more details of ${\Gamma }_n^{\ast }$, ${\overline{\Gamma }}_n^{\ast }$, and ${\Gamma }_n$, please refer to [54] (Chapter 13–15).

Given the difficulty of characterizing ${\Gamma }_n^{\ast }$ with infinite alphabets, in order to optimize finite PMFs numerically, it is practical to consider ${\Gamma }_n^{\ast }$ when the random variables have finite alphabet sizes.

Definition 2

(Alphabet-bounded entropic region [54], Chapter 21). Given a random vector $\mathbf{X}$ with an index set ${\mathcal{N}}_n$, taking values on the finite alphabet $\mathcal{X}$ of size $m={\prod }_{i=1}^n{m}_i$ such that $m<\infty$, the alphabet-bounded entropic region is defined as

$${\Gamma }_{n,\mathcal{X}}^{\ast }\triangleq \left\{{\mathbf{h}}^{\mathbf{p}}\in {\mathcal{H}}_n|\mathbf{p}\in {\Delta }_{\mathcal{X}}^m\right\},$$

(10)

where ${\mathbf{h}}^{\mathbf{p}}$ is the entropic vector associated with the PMF $\mathbf{p}$, and ${\Delta }_{\mathcal{X}}^m$ is the probability simplex defined on the finite alphabet $\mathcal{X}$ of size m.

The region ${\Gamma }_{n,\mathcal{X}}^{\ast }$ is the collection of entropic vectors associated with PMFs defined on the finite alphabet $\mathcal{X}$, and is a compact and closed set.

To characterize the closeness of two given vectors, the most straightforward measurement is the angle between the rays determined by them. Here, we adopt the measure of normalized distance, proposed in [26].

Definition 3

(Normalized distance). Given vectors $\mathbf{a},\mathbf{b}\in {\Gamma }_n\setminus \left\{\mathbf{0}\right\}$, the normalized distance between $\mathbf{a}$ and $\mathbf{b}$ is defined as

$$d_{\mathrm{norm}}(\mathbf{a},\mathbf{b})={\displaystyle \frac{\parallel \mathbf{a}-{\mathbf{b}}^{\prime }\parallel }{\parallel {\mathbf{b}}^{\prime }\parallel }},$$

(11)

where $\parallel ·\parallel$ is the $l_2$ norm, and ${\mathbf{b}}^{\prime }\triangleq \arg {\inf }_{\tilde{\mathbf{b}}\in R_{\mathbf{b}}}\parallel \mathbf{a}-\tilde{\mathbf{b}}\parallel$.

We note that the normalized distance is the tangent of the angle between the rays determined by $\mathbf{a}$ and $\mathbf{b}$. Thus, we also have

$$d_{\mathit{norm}}(\mathbf{a},\mathbf{b})={\displaystyle \frac{\parallel \mathbf{a}-\frac{\mathbf{a}·\mathbf{b}}{{\parallel \mathbf{b}\parallel }^2}\mathbf{b}\parallel }{\parallel \frac{\mathbf{a}·\mathbf{b}}{{\parallel \mathbf{b}\parallel }^2}\mathbf{b}\parallel }}.$$

(12)

2.3. Neural Networks

In general, an NN is a computational model, with significant expressive capability for desired functions [56]. Following convention, we now formally define fully connected feedforward multilayer NNs as a family of functions.

Definition 4

(Neural networks [56]). With $l\in \mathbb{N}$ hidden layers of sizes $d_1,d_2,\dots ,d_l\in \mathbb{N}$, fixed input dimension $d_0$ and output dimension $d_{l+1}$, where $d_0,d_{l+1}\in \mathbb{N}$, let ∘ denote the composition of functions, a fully connected feedforward multilayer NN is defined as the following family of functions

$${\mathcal{G}}_l^{(d_0,d_{l+1})}\triangleq \left\{g:{\mathbb{R}}^{d_0}\to {\mathbb{R}}^{d_{l+1}}|g\left({\mathbf{x}}_{\mathbf{0}}\right)=f_{l+1}\circ \sigma \circ f_l\circ \cdots \circ \sigma \circ f_1\left({\mathbf{x}}_{\mathbf{0}}\right)\right\},$$

(13)

where, for $j\in \{1,2,\dots ,l+1\}$, $f_j:{\mathbb{R}}^{d_{j-1}}\to {\mathbb{R}}^{d_j}$ is a linear operation

$$f_j\left({\mathbf{x}}_{j-1}\right)\triangleq {\mathbf{W}}_j{\mathbf{x}}_{j-1}+{\mathbf{b}}_j,$$

(14)

with the weight matrices ${\mathbf{W}}_j\in {\mathbb{R}}^{d_j\times d_{j-1}}$, the data vectors ${\mathbf{x}}_{j-1}\in {\mathbb{R}}^{d_{j-1}}$, and the bias vectors ${\mathbf{b}}_j\in {\mathbb{R}}^{d_j}$. The nonlinear activation function $\sigma (·)$ is performed on vectors element-wise, such that, for $j\in \{1,2,\dots ,l\}$,

$${\left({\mathbf{x}}_j\right)}_i={\left[\sigma \left(f_j\left({\mathbf{x}}_{j-1}\right)\right)\right]}_i,i=1,2,\dots ,d_j.$$

(15)

For an arbitrary number of hidden layers and sizes, fully connected feedforward multilayer NNs with the same fixed input and output dimensions are defined as

$${\mathcal{G}}^{(d_0,d_{l+1})}\triangleq \bigcup _{l\in \mathbb{N}}{\mathcal{G}}_l^{(d_0,d_{l+1})}.$$

(16)

The set of all possible weights and biases of an NN is the parameter space, which is denoted as $\mathsf{\Phi }\subset {\mathbb{R}}^d$ (here, d is defined as the dimension of the parameter space). We sometimes denote an NN function $g\in {\mathcal{G}}^{(d_0,d_{l+1})}$ with parameters $\varphi \in \mathsf{\Phi }$ as $g_{\varphi }$. Exploiting NNs, desired functions can be parameterized with $\varphi$, and approximated by optimizing $\varphi$ with gradient descent methods [57]. Multilayer feedforward NNs are known to be universal approximators for any measurable function as long as the scale of the model is sufficiently large [56].

Activation functions of NNs may vary in forms for different tasks. More specifically, let the input be $\mathbf{y}\in {\mathbb{R}}^{d_y}$ with elements $y_j,j\in \{1,2,\dots ,d_y\}$; the most common activation functions include the sigmoid activation function ${\sigma }_{\mathrm{s}}{\left(\mathbf{y}\right)}_j={\displaystyle \frac{1}{1+\exp (-y_j)}}$ and the ReLU activation function ${\sigma }_{\mathrm{R}}{\left(\mathbf{y}\right)}_j=\max (y_j,0)$. There are many variations of ReLU, including the ELU activation function ${\sigma }_{\mathrm{E}}{\left(\mathbf{y}\right)}_j=\left\{\begin{array}{cc}{y}_j\hfill & ,y_j\ge 0,\hfill \\ e^{y_j}-1\hfill & ,y_j<0.\hfill \end{array}\right.$ A special activation function is the softmax activation function or the softmax layer, which is defined as ${\sigma }_{\mathrm{sm}}(·):{\mathbb{R}}^{d_y}\to {\mathbb{R}}^{d_y}$, where

$${\sigma }_{\mathrm{sm}}{\left(\mathbf{y}\right)}_j={\displaystyle \frac{\exp \left(y_j\right)}{{\sum }_{j=1}^{d_y}\exp \left(y_j\right)}},j=1,2,\dots ,d_y.$$

(17)

We note that the output vector of the softmax layer has special properties, i.e., ${\sigma }_{\mathrm{sm}}{\left(\mathbf{y}\right)}_j\ge 0$ and ${\sum }_{j=1}^d{\sigma }_{\mathrm{sm}}{\left(\mathbf{y}\right)}_j=1$, which make the output vector a valid PMF in (1).

For details on NNs, please refer to [58] (Chapter 20).

2.4. Problem Statement

In this paper, when a target vector ${\mathbf{h}}_t\in {\Gamma }_n$ is given, we are interested in finding an entropic vector ${\mathbf{h}}^{\mathbf{p}}$ in ${\Gamma }_{n,\mathcal{X}}^{\ast }$, such that $d_{\mathrm{norm}}({\mathbf{h}}^{\mathbf{p}},{\mathbf{h}}_t)$ is relatively small. If ${\mathbf{h}}_t$ is entropic, we aim to give the underlying PMF to verify and realize ${\mathbf{h}}_t$. If ${\mathbf{h}}_t$ is not entropic, we aim to give an entropic vector close to ${\mathbf{h}}_t$ and, hopefully, on the boundary of ${\Gamma }_{n,\mathcal{X}}^{\ast }$.

We recall that $d_{\mathrm{norm}}({\mathbf{h}}^{\mathbf{p}},{\mathbf{h}}_t)$ is the tangent of the angle between the corresponding rays, but the tangent is only strictly increasing when the angle is limited in $[0,{\displaystyle \frac{\pi }{2}}]$. Restricting the target vector ${\mathbf{h}}_t$ to be in ${\Gamma }_n$ can satisfy this requirement, since ${\Gamma }_n$ belongs to the nonnegative orthant of ${\mathcal{H}}_n$ and ${\Gamma }_{n,\mathcal{X}}^{\ast }\subset {\Gamma }_n$.

3. Optimizing PMFs for Associated Entropic Vectors via Neural Networks

In this section, we propose the methodology that tackles the problem stated in Section 2.4. More specifically, given a target vector in ${\Gamma }_n$, we propose to train a corresponding NN to identify the entropic vector closest to the target, and, at the same time, provide the corresponding underlying PMF.

In particular, the proposed NN is configured with a final softmax layer to output PMFs, with the input fixed to a specific constant (similar settings can be found in [45,46]). We claim that, for a given target vector, there always exists an NN such that the entropic vector associated with the produced PMF is arbitrarily close to the target in terms of normalized distance. This statement is formalized in the following theorem.

Theorem 1.

For the random vector $\mathbf{X}$ with a fixed and finite alphabet $\mathcal{X}$, given a target ${\mathbf{h}}_t\in {\Gamma }_n$ (entropic or not), we consider the entropic vector ${\mathbf{h}}^{\ast }\in {\Gamma }_{n,\mathcal{X}}^{\ast }$ that yields the minimal normalized distance $D^{\ast }$ to ${\mathbf{h}}_t$. Then, for $\eta >0$, there exists a corresponding NN $g_{\varphi }$ with parameters $\varphi \in \mathsf{\Phi }$ (where $\mathsf{\Phi }\subset {\mathbb{R}}^d$ is the parameter space) which outputs the PMF ${\mathbf{p}}^{\varphi }$ valid in ${\Delta }_{\mathcal{X}}^m$ from a specific constant input $a\in \mathbb{R}$, such that

$$d_{\mathrm{norm}}({\mathbf{h}}^{{\mathbf{p}}^{\varphi }},{\mathbf{h}}_t)<D^{\ast }+\eta ,$$

(18)

where ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ is the associated entropic vector of ${\mathbf{p}}^{\varphi }$.

The proof of Theorem 1 is straightforward based on the observation that the softmax layer can produce any desired PMF, and both the entropic vector and the normalized distance are continuous functions of the PMF. However, for the completeness of the paper, we provide the proof of Theorem 1 in Appendix B.1. The proof can be described as follows. For a given target vector ${\mathbf{h}}_t\in {\Gamma }_n$, let ${\mathbf{p}}^{\ast }$ be one of the underlying PMFs that achieves ${\mathbf{h}}^{\ast }$, i.e., the entropic vector with minimal normalized distance to ${\mathbf{h}}_t$; then, we can find an NN which outputs ${\mathbf{p}}^{\varphi }$ from a fixed constant, and ${\mathbf{p}}^{\varphi }$ approximates ${\mathbf{p}}^{\ast }$ in $l_1$ norm. By the continuity of the Shannon entropy ([54], Section 2.3), for fixed finite alphabets, we can prove that, as ${\mathbf{p}}^{\varphi }$ approximates ${\mathbf{p}}^{\ast }$ in $l_1$ norm, the associated entropic vector ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ approximates ${\mathbf{h}}^{\ast }$ in normalized distance as well.

Algorithm

In the rest of the section, we give a practical algorithm, i.e., Algorithm 1, to minimize the NN loss function, which is defined as the normalized distance interpreted in (12), i.e.,

$$\mathsf{L}(\varphi ,{\mathbf{h}}_t,{\mathbf{h}}^{{\mathbf{p}}^{\varphi }})\triangleq {\displaystyle \frac{\parallel {\mathbf{h}}^{{\mathbf{p}}^{\varphi }}-\frac{{\mathbf{h}}^{{\mathbf{p}}^{\varphi }}·{\mathbf{h}}_t}{\parallel {\mathbf{h}}_t{\parallel }^2}{\mathbf{h}}_t\parallel }{\parallel \frac{{\mathbf{h}}^{{\mathbf{p}}^{\varphi }}·{\mathbf{h}}_t}{\parallel {\mathbf{h}}_t{\parallel }^2}{\mathbf{h}}_t\parallel }}.$$

(19)

Algorithm 1 trains parameters $\varphi$ with a gradient descent method to find an NN that achieves ${\mathbf{h}}^{\ast }$ in Theorem 1, i.e., identify the entropic vector closest to the target and provide the corresponding underlying PMF. The overall architecture of the proposed method is depicted in Figure 1. More specifically, at each iteration, the NN takes a constant as the input, and outputs a joint PMF ${\mathbf{p}}^{\varphi }$. By configuring the last layer of NN as the softmax layer in (17), i.e.,

$${\mathbf{p}}_j^{\varphi }={\sigma }_{\mathrm{sm}}{\left({\tilde{\mathbf{p}}}^{\varphi }\right)}_j={\displaystyle \frac{\exp \left({\tilde{p}}_j^{\varphi }\right)}{{\sum }_{j=1}^m\exp \left({\tilde{p}}_j^{\varphi }\right)}},j=1,2,\dots ,m,$$

(20)

where ${\tilde{\mathbf{p}}}^{\varphi }\in {\mathbb{R}}^m$ is the layer input, the resulting PMF ${\mathbf{p}}^{\varphi }$ is valid in ${\Delta }_{\mathcal{X}}^m$. Subsequently, the associated entropic vector ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ is computed from ${\mathbf{p}}^{\varphi }$ with (2), (3), and (5). The normalized distance between the associated entropic vector and target vector is then evaluated with (19). To optimize NN parameters $\varphi$, the gradient descent method is performed in turn. The training procedures are iterated for a sufficiently large number N.

Algorithm 1: Optimize PMFs for the associated entropic vector closest to the target vector ${\mathbf{h}}_t\in {\Gamma }_{n,\mathcal{X}}$ via NN training

We further implement Algorithm 1 and the NN model in Figure 1 with CNNs, and details are presented in Section 4. With the proposed implementation techniques, the convergence performances of Algorithm 1 for different targets are demonstrated in Section 5.1. Other empirical results on various problems related to the entropic region are demonstrated in Section 5.2 and Section 5.3.

Before ending this section, in the following remark, we discuss the connections between our work with existing literature [44,45,46], which focus on the neural optimization of distributions for problems in information theory as well.

Remark: Recently, the optimization of probability distributions utilizing generative NNs in information theory has been studied and applied to plenty of problems in [44,45,46]. In [44,46], NNs are designed to generate the input distributions of given channels, and optimized to produce a distribution that achieves channel capacity, serving as a part of the joint estimation–optimization architecture for channel capacity. More specifically, over continuous spaces [44], a model named neural distribution transformer (NDT) is proposed; the optimized NDT transfers uniformly distributed samples into data that are distributed according to the desired capacity-achieving distribution. Over discrete alphabets [46], the PMF generator is proposed to optimize the PMF numerically, and capacity-achieving channel input data can be sampled from the optimized PMF. We note that [46] focuses on channels with feedback and memory, and the PMF generator is based on a time-series deep reinforcement learning model. For channels without feedback and memory, the PMF generator in [46] degenerates to a similar architecture as the one proposed in this paper. However, different from sampling data from generated PMFs, the method proposed in this paper directly computes entropic vectors from PMFs, and theoretical guarantees are provided in the context of this task. In [45], NNs are also used to optimize conditional distributions that achieve the rate-distortion function for any given source distribution, in both continuous and discrete spaces. The PMF generator in [45] takes source samples as input, and produces corresponding conditional PMFs. If the source is deterministic in [45], the conditional PMF generator degenerates to the proposed model in this paper as well.

4. Implementation

In this section, we aim to present implementation details of Algorithm 1 and the NN model in Figure 1 with more advanced deep learning techniques. In addition, we demonstrate that the implementation results in significantly larger feasible alphabet sizes for Algorithm 1 than existing methods, which will contribute to constructing achievable schemes for network coding problems, as discussed in Section 6.

4.1. Implementing with Convolutional Neural Networks

We recall that, for a random vector $\mathbf{X}$, if we assume $|{\mathcal{X}}_i|=k_i=k,i\in {\mathcal{N}}_n$ with joint PMFs $\mathbf{p}\in {\Delta }_{\mathcal{X}}^{k^n}$, then every PMF is a $k^n$-dimensional vector. In addition, we notice that Algorithm 1 and the model depicted in Figure 1 require the dimension of the output of the NN to match the dimension of the desired PMF. Hence, the size of the last layer of the NN grows polynomially with the alphabet size k of each random variable, and grows exponentially with the number of random variables n. Moreover, the indexing problem to calculate all marginal PMFs in Algorithm 1 is computational expensive as well with large dimensional PMF vectors.

Therefore, we are motivated to implement Algorithm 1 and the model in Figure 1 with CNNs, which transfer the vector data in NNs to tensors. Furthermore, by virtue of high-performance GPUs, the training of CNNs can be notably accelerated.

More specifically, if we focus on the case when $n=4$, a PMF $\mathbf{p}$ for $\mathbf{X}$ with $|{\mathcal{X}}_i|=k,i\in {\mathcal{N}}_n$ is a $k^4$-dimensional vector such that $\mathbf{p}\in {\Delta }_{\mathcal{X}}^{k^4}$. To implement with CNNs, we naturally utilize the 3d-CNNs (available in PyTorch [59]), whose data in each layer is a four-dimensional cube. By modifying the NN in Figure 1 with 3d-CNNs, the model takes a constant tensor $\mathbf{A}$ as input, where $\mathbf{A}\in {\mathbb{R}}^{1\times 1\times 1\times 1}$, and outputs a PMF tensor ${\mathbf{P}}^{\varphi }\in {[0,1]}^{k\times k\times k\times k}$.

We demonstrate an example of a configuration of the modified model in

Table 1. An example of a configuration when the NN in Figure 1 is implemented with a CNN.

Layer	Output Dimension	Kernel Size	Padding	Activation Function
Input	$1\times 1\times 1\times 1$	-	-	-
Conv. 3d	$k\times k\times k\times k$	$k\times k\times k$	$k-1$	ELU
Conv. 3d	$k\times k\times k\times k$	$3\times 3\times 3$	1	ELU
Conv. 3d	$k\times k\times k\times k$	$3\times 3\times 3$	1	ELU
Flatten	$k^4$	-	-	softmax
Output	$k\times k\times k\times k$	-	-	-

. Our empirical results in Section 4.2 show that the model implemented with a CNN is feasible even when k is large, acquiring additional gain from GPU acceleration.

4.2. Feasibility for Large Alphabets

In the proposed method, despite the fact that the complexity of the output layer in the original NN grows as the alphabet sizes of the random variables increase, after the implementation discussed in Section 4.1, the training of CNNs can be accelerated by high-performance GPUs, resulting in significant gains in the training speed for Algorithm 1. In Figure 2, we present empirical results of the training speed of the proposed method as the alphabet sizes of random variables increase.

To the best of the authors’ knowledge, when $n=4$, the feasible alphabet size of the proposed method (i.e., 32 for every random variable) is larger than prior works. For instance, the largest alphabet size considered in [26] is 5. Alphabet sizes of 10 and 11 are mentioned in [16,21], respectively.

5. Empirical Results

In this section, empirical results of the method proposed in Section 3 are presented, focusing on various problems related to ${\Gamma }_4^{\ast }$ and ${\overline{\Gamma }}_4^{\ast }$. Firstly, we compare our method with the existing random search algorithm [26]. Secondly, we tackle the problems of optimizing the Ingleton score and Ingleton violation index. Lastly, we exploit the proposed method to obtain many entropic vectors that approximate the boundary of ${\Gamma }_4^{\ast }$, and use the convex hull of these entropic vectors to form an inner bound of ${\overline{\Gamma }}_4^{\ast }$.

Before presenting these results, we first provide a brief introduction of regions within ${\mathcal{H}}_4$. When $n=4$, the Ingleton inequality [41] plays an essential role in the characterization of ${\overline{\Gamma }}_4^{\ast }$, and one permutation instance of the Ingleton inequality is defined as follows.

Definition 5

(Ingleton inequality). For a vector $\mathbf{h}\in {\mathcal{H}}_4$, one specific permutation instance of the Ingleton inequality [41] is denoted as the inner product between the coefficients ${\mathbf{I}}_{34}$ and $\mathbf{h}$

$$\begin{array}{cc}\hfill {\mathbf{I}}_{34}\mathbf{h}\triangleq & h_{12}+h_{13}+h_{14}+h_{23}+h_{24}\hfill \\ \hfill & -h_1-h_2-h_{34}-h_{123}-h_{124}\ge 0.\hfill \end{array}$$

(21)

Permuting ${\mathcal{N}}_4$, there exist six instances of the Ingleton inequality in total, which are denoted as $\mathbf{I}\mathbf{h}\ge \mathbf{0},\mathbf{I}\in {\mathbb{R}}^{6\times 15}$.

When $n=4$, the linear rank region is the intersection of ${\mathcal{H}}_4$ with $\mathbf{I}\mathbf{h}\ge \mathbf{0}$ (all six instances of Ingleton inequality), i.e.,

$${\Gamma }_4^{\mathrm{linear}}\triangleq \left\{\mathbf{h}\in {\mathcal{H}}_4|\mathbf{I}\mathbf{h}\ge \mathbf{0}\right\}.$$

(22)

We consider vectors that represent rank functions of linear subspaces (i.e., linearly representable entropic vectors, or entropic vectors that yield rate regions achieved by linear network codes); then, the linear rank region is the closure of the linear hull of such vectors. The region ${\Gamma }_4^{\mathrm{linear}}$ is a polyhedral convex cone, and is a tight linear inner bound of ${\overline{\Gamma }}_4^{\ast }$ [8,13,60]. However, ${\Gamma }_n^{\mathrm{linear}}$ is only completely characterized for $n\le 5$ [15,17]. Nevertheless, when $n=4$, the relation ${\Gamma }_4^{\mathrm{linear}}\subset {\overline{\Gamma }}_4^{\ast }\subset {\Gamma }_4$ holds.

Hence, one is interested in characterizing the part of ${\overline{\Gamma }}_4^{\ast }$ excluding ${\Gamma }_4^{\mathrm{linear}}$ (i.e., the almost-entropic vectors that cannot be linearly representable or yield rate regions, which cannot be achieved by linear network codes). We recall that $\mathbf{I}\mathbf{h}\ge \mathbf{0}$ represents all six instances of the Ingleton inequality. We define

$${\Lambda }_4\triangleq \left\{\mathbf{h}\in {\Gamma }_4|\mathbf{I}\mathbf{h}\ge \mathbf{0}\mathrm{does}\mathrm{not}\mathrm{hold}\right\},$$

(23)

and the entropic part of ${\Lambda }_4$, i.e.,

$${\Lambda }_4^{\ast }\triangleq \left\{\mathbf{h}\in {\Gamma }_4^{\ast }|\mathbf{I}\mathbf{h}\ge \mathbf{0}\mathrm{does}\mathrm{not}\mathrm{hold}\right\}.$$

(24)

We denote the closure of ${\Lambda }_4^{\ast }$ as ${\overline{\Lambda }}_4^{\ast }$; then, it is clear that ${\Lambda }_4^{\ast }\subset {\overline{\Lambda }}_4^{\ast }\subset {\Lambda }_4$. By [60] (Lemma 4), any ray in ${\Lambda }_4$ exactly violates one instance of the Ingleton inequality, i.e., ${\Lambda }_4$ is the union of six symmetric regions, each region corresponds to and violates one instance of the Ingleton inequality, and the six regions are disjointed and share boundary points only. Thus, we may consider ${\Lambda }_4$ and ${\Lambda }_4^{\ast }$ when only one instance of the Ingleton inequality does not hold. More specifically, when ${\mathbf{I}}_{34}\mathbf{h}\le 0$, i.e., the reversed version of (21) holds, we denote ${\Lambda }_4$ and ${\Lambda }_4^{\ast }$ as ${\Lambda }_4^{34}$ and ${\Lambda }_4^{\ast ,34}$, respectively. Analogously, we can consider different instances of the Ingleton inequality to obtain other five outer regions symmetric to ${\Lambda }_4^{34}$ and five regions symmetric to ${\Lambda }_4^{\ast }$.

We denote the closure of ${\Lambda }_4^{\ast ,34}$ as ${\overline{\Lambda }}_4^{\ast ,34}$; then, in a word, the characterization of ${\overline{\Gamma }}_4^{\ast }$ is reduced to ${\overline{\Lambda }}_4^{\ast ,34}$.

The only extreme ray of ${\Lambda }_4^{34}$ violating ${\mathbf{I}}_{34}\mathbf{h}\ge 0$ (as well as the Zhang–Yeung inequality [8]) is denoted as $R_{\mathbf{v}}$, where

$$\mathbf{v}={(1,1,1,1,1.5,1.5,1.5,1.5,1.5,2,2,2,2,2,2)}^{\mathsf{T}}.$$

(25)

The region ${\Lambda }_4^{34}$ is often called the pyramid and is represented by 15 extreme rays, with $R_{\mathbf{v}}$ as the top and the other 14 extreme rays as bases (a complete list of these extreme rays is available in [16] (Section XI)).

In the rest of this section, we focus on the bounded regions as Definition 2 illustrates with $m=k^4$, i.e., each random variable has an equal alphabet size k. The above discussions still hold, as we denote the bounded versions as ${\Lambda }_{4,\mathcal{X}}^{34}$, ${\Lambda }_{4,\mathcal{X}}^{\ast ,34}$ and ${\overline{\Lambda }}_{4,\mathcal{X}}^{\ast ,34}$.

In the following remark, we present the implementation details of Algorithm 1 for experiments in this section.

Remark 1.

In this section, we utilized Algorithm 1 with hyperparameters $n=4,m_1=m_2=m_3=m_4=4$. Overall, the NN model followed from the configuration presented in Table 1 with $k=4$ and either one or three hidden layers. The NN parameters set ϕ was randomly initialized with the default method of PyTorch [59] (the initial parameters can be fixed by fixing the random seed), the optimizer was selected with Adam optimizer [61] with a learning rate γ tuned in the range of ${10}^{-2}$ to ${10}^{-5}$ (based on the number of hidden layers), and we selected the constant $a=1$. The number of iterations N were tuned to be $5\times {10}^4$ (however, the algorithm can converge with significantly smaller iterations in most tasks of this section).

5.1. Comparison with the Random Search Algorithm in [26]

When evaluating the performance of our method, the target was set as $\mathbf{v}$ in (25), which is trivially not entropic due to the Zhang–Yeung inequality [8], i.e., $D^{\ast }>0$ in Theorem 1. We aimed to compare the returned normalized distance by Algorithm 1 with that obtained by the random search algorithm in [26], which is the only existing method for verifying entropic vectors with general form of PMFs. Intuitively, if Algorithm 1 is capable of obtaining entropic vectors closer to $\mathbf{v}$, we are able to approximate finer boundaries of ${\Gamma }_{4,\mathcal{X}}^{\ast }$.

Figure 3 depicts the empirical results of our method. In addition to $\mathbf{v}$ (Target 1), two entropic vectors in [13] (Theorem 4, Target 2) and [16] (Conjecture 1, Target 3), which belong to the region ${\Lambda }_{4,\mathcal{X}}^{\ast ,34}$, are set as targets, and the results are compared to [26] (the numerical data of the results in [26] are available in [62]). The alphabet size of every random variable, i.e., k, is set to 4. In Figure 3, the curves illustrate the convergence performances as the iterations of Algorithm 1 grow, while the shaded regions represent the range of final results obtained by several experiments in [26].

From Figure 3, it is evident that our method outperforms the random search algorithm proposed by [26]. More specifically, for the non-entropic target (Target 1 in Figure 3), the returned normalized distance using our method ($0.01895$) is smaller than the smallest value obtained by [26] ($0.02482$). For entropic targets (Targets 2 and 3 in Figure 3), our method returns negligible normalized distance as expected, thus successfully verifying these entropic targets. In addition, the normalized distances obtained by [26] vary across different experiments, and even increase with larger iterations. For example, with Target 1 in Figure 3, the method in [26] returns a normalized distance of $0.02482$ with 1226 iterations; however, with 10,091 iterations, a larger normalized distance $0.04283$ is obtained [26] (Figure 1).

In contrast, the results obtained by our method are consistent within multiple experiments, empirically presenting better convergence performances than [26].

5.2. Optimizing Ingleton Score and Ingleton Violation Index

When optimizing entropic vectors for $n=4$, it is of great interest to optimize the Ingleton score and Ingleton violation index. The Ingleton score was first proposed in [42] (Conjecture 4.1) and later rigorously defined in [16] (Definition 3).

Definition 6

(Ingleton score [16] (Definition 3)). Given a random vector $\mathbf{X}=(X_1,X_2,X_3,X_4)$, and an entropic vector $\mathbf{h}\in {\Gamma }_4^{\ast }$, the Ingleton score induced by (21) is defined as

$${\mathbb{I}}_{34}\left(\mathbf{h}\right)\triangleq {\displaystyle \frac{{\mathbf{I}}_{34}\mathbf{h}}{h_{1234}}}.$$

(26)

The Ingleton score is defined as $\mathbb{I}\triangleq {\mathbb{I}}_{34}$ due to permutation symmetry, and the optimization problem of $\mathbb{I}$ is

$$\begin{array}{cc}\hfill \underset{\mathbf{h}}{\inf }& \mathbb{I}\left(\mathbf{h}\right),\hfill \\ \hfill s.t.& \mathbf{h}\in {\Gamma }_4^{\ast }.\hfill \end{array}$$

(27)

Similarly, the Ingleton violation index [19] is defined as follows.

Definition 7

(Ingleton violation index [19]). Given a random vector $\mathbf{X}=(X_1,X_2,X_3,X_4)$, and an entropic vector $\mathbf{h}\in {\Gamma }_4^{\ast }$, the Ingleton violation index induced by (21) is defined as

$${\iota }_{34}\left(\mathbf{h}\right)\triangleq -{\displaystyle \frac{{\mathbf{I}}_{34}\mathbf{h}}{\parallel \mathbf{h}\parallel }}.$$

(28)

Due to permutation symmetry, the Ingleton violation index is defined as $\iota \triangleq {\iota }_{34}$, and the optimization problem of ι is

$$\begin{array}{cc}\hfill \underset{\mathbf{h}}{\sup }& \iota \left(\mathbf{h}\right),\hfill \\ \hfill s.t.& \mathbf{h}\in {\Gamma }_4^{\ast }.\hfill \end{array}$$

(29)

One is interested in minimizing $\mathbb{I}$ or maximizing $\iota$ over ${\Gamma }_4^{\ast }$, because their optimal values reveal the largest violations from (21) among entropic vectors in ${\Lambda }_4^{\ast ,34}$, and, thus, are crucial characterizations of ${\Gamma }_4^{\ast }$ and ${\overline{\Gamma }}_4^{\ast }$.

There is a rich history of optimizing $\mathbb{I}$ [16,21,26,35,42] and $\iota$ [19,26]. More specifically, for $\mathbf{h}\in {\overline{\Gamma }}_4^{\ast }$, [21] proposes a decomposition and two linear transformations on $\mathbf{h}$. If we denote the composition of these techniques on $\mathbf{h}$ as $\mathbf{g}\triangleq T\left(\mathbf{h}\right)$, then it is shown that $\mathbf{g}\in {\overline{\Gamma }}_4^{\ast }$, i.e., $\mathbf{g}$ is still almost entropic for any almost-entropic $\mathbf{h}$. In addition, the Ingleton score optimized with $\mathbf{g}$ in [21] is smaller than the one optimized with $\mathbf{h}$, and refutes the four-atom conjecture in [16] (please refer to [21] (Section VIII) for details). Since $T(·)$ plays an important role in optimizing the Ingleton score and Ingleton violation index in this paper, for both completeness and clarity, we state the details of the techniques that compose $T(·)$ in Appendix A for interested readers.

Thus, we are motivated to exploit our method to obtain a better Ingleton score and Ingleton violation index, when the objective $\mathsf{L}$ in Algorithm 1 is replaced by $\mathbb{I}\left(\mathbf{h}\right)$, $\iota \left(\mathbf{h}\right)$, and $\mathbb{I}\left(\mathbf{g}\right)$, $\iota \left(\mathbf{g}\right)$. Although Theorem 1 does not directly imply that the proposed method can be exploited to optimize the Ingleton score and Ingleton violation index, we claim that there always exists an NN that optimizes the Ingleton score and Ingleton violation index to the optimal values to any desired of accuracy as indicated by Theorems 2 and 3. The proofs of Theorems 2 and 3 are provided in Appendix B.2 and Appendix B.3, respectively.

Theorem 2.

For the random vector $\mathbf{X}=(X_1,X_2,X_3,X_4)$ with a fixed and finite alphabet $\mathcal{X}$, we recall the Ingleton score for entropic vectors defined in Definition 6. We consider the entropic vector ${\mathbf{h}}^{\ast }\in {\Gamma }_{4,\mathcal{X}}^{\ast }$, such that ${\mathbf{h}}^{\ast }$ yields the optimal Ingleton score ${\mathbb{I}}^{\ast }$. Then, there exists an NN with parameters $\varphi \in \mathsf{\Phi }$, such that ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ is the associated entropic vector of ${\mathbf{p}}^{\varphi }=g_{\varphi }\left(a\right)$ valid in ${\Delta }_{\mathcal{X}}^m$, and ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ yields the corresponding Ingleton score ${\mathbb{I}}^{\varphi }$, which is consistent with ${\mathbb{I}}^{\ast }$ within any desired degree of accuracy $\kappa >0$, i.e.,

$$|{\mathbb{I}}^{\varphi }-{\mathbb{I}}^{\ast }|<\kappa ,$$

(30)

where $|·|$ is the absolute value for scalars.

Theorem 3.

For the random vector $\mathbf{X}=(X_1,X_2,X_3,X_4)$ with a fixed and finite alphabet $\mathcal{X}$, we recall the Ingleton violation index for entropic vectors defined in Definition 7. We consider the entropic vector ${\mathbf{h}}^{\ast }\in {\Gamma }_{4,\mathcal{X}}^{\ast }$, such that ${\mathbf{h}}^{\ast }$ yields the optimal Ingleton violation index ${\iota }^{\ast }$. Then, there exists an NN with parameters $\varphi \in \mathsf{\Phi }$, such that ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ is the associated entropic vector of ${\mathbf{p}}^{\varphi }=g_{\varphi }\left(a\right)$ valid in ${\Delta }_{\mathcal{X}}^m$, and ${\mathbf{h}}^{{\mathbf{p}}^{\varphi }}$ yields the corresponding Ingleton violation index ${\iota }^{\varphi }$, which is consistent with ${\iota }^{\ast }$ within any desired degree of accuracy $\mu >0$, i.e.,

$$|{\iota }^{\varphi }-{\iota }^{\ast }|<\mu ,$$

(31)

where $|·|$ is the absolute value for scalars.

The optimized numerical results are compared with the state-of-the-art values in

Table 2. Optimized Ingleton scores and Ingleton violation indices.

Objective	Best-Known Results	Our Results
${\inf }_{\mathbf{h}}\mathbb{I}\left(\mathbf{h}\right)$	$-0.089373$ [16]	$-0.0893733002$
${\inf }_{\mathbf{h}}\mathbb{I}\left(\mathbf{g}\right)$	$-0.0925000777$ [35]	$-0.0925001031$
${\sup }_{\mathbf{h}}\iota \left(\mathbf{h}\right)$	$0.028131604$ [26]	$0.0281316527$
${\sup }_{\mathbf{h}}\iota \left(\mathbf{g}\right)$	N/A	0.0288304141

, and the alphabet size of every random variable is set to 4. In Table 2, our results are rounded to ten decimal digits, and the last four digits are significant since the perturbations for small probability values may lead to large absolute errors of entropy functions and score results.

As seen in Table 2, firstly, for the Ingleton score $\mathbb{I}$, the upper bound $-0.0925001031$, which is optimized with $\mathbf{g}$, is returned (with negligible improvement compared to $-0.0925000777$ due to numerical stability). After the first discovery of this value [34,35], the method in [26] obtained a close value of $-0.092499$ as well, and, thus, we reconfirm this upper bound for the third time. Secondly, for the Ingleton violation index $\iota$, we particularly optimize it with $\mathbf{g}$ as well, and a new lower bound $0.0288304$ is obtained, which beats the current best value of $0.0281316$. We observe that $\iota$ actually measures the sine of the angle between the hyperplane ${\mathbf{I}}_{34}\mathbf{h}=0$ and $\mathbf{h}$; thus, the obtained new lower bound finds an entropic vector with a larger violation angle from the Ingleton hyperplane, bringing us closer to the optimal $\iota$ and the complete characterization of ${\Gamma }_4^{\ast }$ and ${\overline{\Gamma }}_4^{\ast }$.

The corresponding convergence performances for results in Table 2 are presented in Figure 4. From Figure 4, one can verify the obtained results listed in Table 2, and observe that the proposed method successfully converges to the best known Ingleton score when optimizing $\mathbb{I}\left(\mathbf{g}\right)$, and converges to the Ingleton violation index, which exceeds the best-known results when optimizing $\iota \left(\mathbf{g}\right)$.

Remark 2.

The pursuit of the Ingleton score $-0.0925$ appears to be difficult with no alternative strategies in the literature. In [26], the search area is constrained using a strategy called hyperplane score reduction with the minimization of normalized distance with relatively large iterations. Additionally, [35] reports that most experiments yield the value $-0.09103635$, while returning $-0.0925000777$ very occasionally, even with millions of searches with massive computation resources in parallel. Similarly, with our method, the direct minimization of $\mathbb{I}\left(\mathbf{g}\right)$ yields the value $-0.09103635$. However, by selecting the entropic point from [13] as the initial point of Algorithm 1 (by setting it as the target and minimizing the normalized distance first), we reproduced the value $-0.0925$ in a single experiment, with less than 10,000 iterations (which approximately take 6 s).

5.3. Inner Bounding the Almost-Entropic Region

The complete characterization of ${\overline{\Gamma }}_4^{\ast }$ is difficult and open. However, as we find many entropic vectors, a linear inner bound is immediately obtained. For example, for an entropic vector $\mathbf{h}\in {\Lambda }_{4,\mathcal{X}}^{\ast ,34}$, if we regard $\mathbf{h}$ as the top vertex, then the convex hull of the top $\mathbf{h}$ and other 14 base vertices (refer to [16] (Section XI) for a complete list of these extreme rays) yields an inner bound for ${\overline{\Lambda }}_{4,\mathcal{X}}^{\ast ,34}$ and ${\overline{\Gamma }}_4^{\ast }$.

In this paper, following [16] (Section XI), the quality of the inner bound is evaluated by the volume ratio. Let ${\tilde{\Lambda }}_{4,\mathcal{X}}^{34}$ denote the numerical inner bound obtained, and $\mathrm{vol}(·)$ denote the polytope volume computation operation; the volume ratio is defined as

$$R\triangleq {\displaystyle \frac{\mathrm{vol}\left({\tilde{\Lambda }}_{4,\mathcal{X}}^{34}\right)}{\mathrm{vol}\left({\Lambda }_{4,\mathcal{X}}^{34}\right)}}\times 100\%.$$

(32)

Intuitively, $\mathbf{h}$ needs to be closer to the boundary of ${\Lambda }_{4,\mathcal{X}}^{\ast ,34}$ to obtain a larger volume ratio R.

Because Algorithm 1 easily produces entropic vectors with large $\mathbb{I}$ and better $\iota$, we are inspired to exploit it to develop a new algorithm, i.e., Algorithm 2, to construct an inner bound with a larger volume ratio, by optimizing entropic vectors that violate other specific hyperplanes.

Algorithm 2: Construct inner bounds for the almost-entropic region with Algorithm 1

Algorithm 2 can be interpreted as follows. First, starting from the 14 base entropic vectors of ${\Lambda }_{4,\mathcal{X}}^{34}$, i.e., $\mathcal{B}$, we construct an initial inner bound with (an) optimized entropic vector(s), i.e., $\mathcal{T}$, the initial inner bound can be represented by the set of vertices $\mathcal{V}=\mathcal{B}\cup \mathcal{T}$. By transferring the V-representation of the initial inner bound with H-representation, we obtain many hyperplanes with the set of coefficients $\mathcal{H}$. Then, for each of these hyperplanes, we minimize their violation score similar to the Ingleton score using Algorithm 2, i.e., we find entropic vectors ${\mathcal{V}}^{\varphi }$ with large violations from the initial inner bound (violating ${\mathbf{I}}_{34}\mathbf{h}\ge 0$ as well). Now, we have extended the vertices of the initial inner bound from $\mathcal{V}$ to $\mathcal{V}\cup {\mathcal{V}}^{\varphi }$, and expanded the initial inner bound to an inner bound with a larger volume ratio. This procedure can be repeated for inner bounds with larger volume ratios.

Remark 3.

Since ${\parallel \mathbf{c}\parallel }_2<\infty$, by Theorem 2 and its proof in Appendix B.2, the theoretical guarantee for Algorithm 2, i.e., minimizing the hyperplane score ${\displaystyle \frac{\mathbf{c}\mathbf{h}}{h_{1234}}}$ exploiting Algorithm 1, immediately follows.

We present the volume ratio of the inner bound obtained with Algorithm 2 in

Table 3. Volume ratios of inner bounds of ${\overline{\Lambda }}_{4,\mathcal{X}}^{\ast ,34}$.

Methods	Volume Ratio R (%)
Newton’s method [16]	$53.4815$
Non-isomorphic supports [22]	$57.8$
Grid approach [26]	$62.4774$
Ours 1	66.1340
	72.0437

¹ For details of the results, refer to Remarks 4 and 5.

, comparing it with all existing results. For reference, the volume ratio of the outer bound obtained with the non-Shannon-type inequalities in [16] (Section VII) (not tight) was $96.4682\%$, and the volume of the trivial inner bound obtained with ${\mathbf{I}}_{34}\mathbf{h}\ge 0$ was 0. From Table 3, one can see that the volume ratio of the inner bound obtained by our method is larger than existing results. Comparing with the state-of-the-art result [26], the ratio $62.4774\%$ is improved to $72.0437\%$, leading to the current best inner bound of ${\overline{\Gamma }}_4^{\ast }$ measured by volume ratio. Thus, our proposed method takes another step forward in the complete characterization of ${\overline{\Gamma }}_4^{\ast }$.

Remark 4.

Here, we give details on how we obtained the two results listed in Table 3. For the $66.1340\%$ result, the set $\mathcal{T}$ consisted of three entropic vectors that yielded the Ingleton score $-0.0925001031$, the Ingleton violation indices $0.028131653$ and $0.0288304141$ in Table 2, respectively, and the entropic vector that yielded the Ingleton score $-0.09103635$, as discussed in Remark 2. Furthermore, after $N=1$ iteration of Algorithm 2, we obtained 152 vertices. After the convex hull operation, there were 121 entropic vectors, which gave us an inner bound with a volume ratio of $66.1340\%$. For the $72.0437\%$ result, the set $\mathcal{T}$ consisted of one entropic vector that yielded the Ingleton score $-0.0925001031$ in Table 2. Furthermore, after $N=2$ iterations of Algorithm 2, we obtained 2585 vertices, and this inner bound yielded an estimated (as illustrated in Remark 5) volume ratio of $72.0437\%$.

Remark 5.

There are several algorithms available for polytope volume computation, as listed in [63]; we chose “Qhull” [64,65] and “lrs” [66,67], which only require the V-representation of a polytope, while providing the convex hull operation as well. However, all existing numerical methods become computationally intractable with high dimension and a large number of vertices (details of their computation complexities can be found in [65,67]). In our case, “Qhull” was able to compute the result $66.1340\%$ from 152 entropic vectors in ${\mathbb{R}}^{15}$, but became intractable when computing another result where the number of entropic vectors was 2585. Nevertheless, “lrs” provides a function called “Estimate”, which allowed us to estimate the computational time and volume result, without diving into the complete computation process. In this manner, it is estimated that the volume ratio computed from 2585 entropic vectors is $72.0437\%$. The estimation process took several days, and it is suggested that the complete computation will take more than one year.

6. Conclusions and Discussion

This paper introduced a novel architecture to parameterize and optimize PMFs to verify entropic vectors with NNs. Given a target vector, an algorithm is proposed to identify the entropic vector closest to it via NN training. Empirical results demonstrate smaller normalized distances, and improved convergence performances. Optimized Ingleton scores are presented, and a new lower bound on the Ingleton violation index was obtained. A state-of-the-art inner bound of ${\overline{\Gamma }}_4^{\ast }$ was constructed, which was measured by the volume ratio. However, there exist computation burdens in the proposed algorithms. Although Algorithm 1 achieves a larger feasible alphabet size than previous works, its efficiency is still constrained by the alphabet size, as Figure 2 shows. Algorithm 2, which requires auxiliary algorithms to calculate the polytope volume, is limited by the auxiliary algorithms due to high dimensionality and the large number of entropic vectors, as Remark 5 discusses.

Future work includes developing a computer-aided approach to construct achievable schemes for network coding problems using the proposed method. The Linear Programming bound [68], which yields Shannon outer bounds for rate regions of network coding problems, is not tight in general due to ${\Gamma }_n\ne {\overline{\Gamma }}_n^{\ast }$. However, with Algorithm 1, the corner points obtained by the Linear Programming bound can be verified to be entropic or not. Furthermore, if a corner point is verified to be entropic, the returned PMF indicates the coding scheme. Similar arguments have been raised in [26] as well, where the network instances are simple, and the largest alphabet size is small. Nevertheless, due to smaller and more consistent normalized distances depicted in Figure 3 and the feasibility for large alphabets discussed in Section 4.2, we believe that our proposed method can be applied to network coding problems with larger alphabet sizes, and/or more random variables.