Information Distances versus Entropy Metric

Abstract

Information distance has become an important tool in a wide variety of applications. Various types of information distance have been made over the years. These information distance measures are different from entropy metric, as the former is based on Kolmogorov complexity and the latter on Shannon entropy. However, for any computable probability distributions, up to a constant, the expected value of Kolmogorov complexity equals the Shannon entropy. We study the similar relationship between entropy and information distance. We also study the relationship between entropy and the normalized versions of information distances.

1. Introduction

Information distance [1] is a universal distance measure between individual objects based on Kolmogorov complexity. It is the length of a shortest program that transforms one object into the other object. Since the theory of information distance was proposed, various information distance measures have been known. The normalized versions of information distances [2] have been introduced for measuring similarity between sequences. The min distance and its normalized version, which do not satisfy the triangle inequality, have been presented in [3,4]. The time-bounded version of information distance [5] has been used for studying the computability properties of the normalized information distances. A safe approximability of the normalized information distance have been discussed in [6]. Since the normalized information distance is uncomputable, two practical distance measures, the normalized compression distance and the Google similarity distance, have been presented [7,8,9,10,11]. These distance measures have been successfully applied to bioinformatics [10], music clustering [7,8,9], linguistics [2,12], plagiarism detection [13], question answering [3,4,14] and many more.

As mentioned in [1], information distance should be contrasted with the entropy metric. The former is based on Kolmogorov complexity and the latter on Shannon entropy. Various relations between Shannon entropy and Kolmogorov complexity are known [15,16,17]. It is well known that for any computable probability distributions the expected value of Kolmogorov complexity equals the Shannon entropy [18,19]. Linear inequalities are valid for Shannon entropy are also valid for Kolmogorov complexity, and vice verse [20]. We also know that various notions are both based on Shannon entropy and Kolmogorov complexity. Hence, many similar relationships between entropy based notions and Kolmogorov complexity based notions have been proposed. Relations between time-bounded entropy measures and time-bounded Kolmogorov complexity have been proposed in [21]. Relations between Shannon mutual information and algorithmic (Kolmogorov) mutual information have been proposed in [18]. Then, relations between entropy based cryptographic security and Kolmogorov complexity based cryptographic security have been studied [22,23,24,25]. One-way functions have been studied on both time-bounded entropy and time-bounded Kolmogorov complexity [26]. However, the relationship between information distance and entropy has not been studied. In this paper, we study the similar relationship between information distance and the entropy metric. We also analyze the validity of the relationship between normalized information distance and the entropy metric.

The rest of this paper is organized as follows: In Section 2, some basic notions are reviewed. In Section 3, we study the relationship between information distance and the entropy metric. In Section 4, we study the relationship between normalized information distance and the entropy metric. Finally, conclusions are stated in Section 5.

2. Preliminaries

In this paper, let $\left|x\right|$ be the length of the string x and $\log (·)$ be the function ${\log }_2(·)$.

2.1. Kolmogorov Complexity

Kolmogorov complexity was introduced independently by Solomonoff [27] and Kolmogorov [28] and later by Chaitin [29]. Some basic notions of Kolmogorov complexity are given below. For more details, see [16,17]. We use the prefix-free definition of Kolmogorov complexity. A string x is a proper prefix of a string y if we have $y=xz$ for $z\ne \epsilon$, where $\epsilon$ is the empty string. A set of strings A is prefix-free if there are not two strings x and y in A such that x is a proper prefix of y. For convenience, we use the prefix-free Turing machine, i.e., Turing machines with a prefix-free domain.

Let F be a fixed prefix-free optimal universal Turing machine. The conditional Kolmogorov complexity $\mathrm{K}\left(y\right|x)$ of y given x is defined by

$$\mathrm{K}\left(y\right|x)=\min \{\left|p\right|:F(p,x)=y\},$$

where $F(p,x)$ is the output of the program p with auxiliary input x when it is run in the machine F.

The (unconditional) Kolmogorov complexity $\mathrm{K}\left(y\right)$ of y is defined as $\mathrm{K}\left(y\right|\epsilon )$.

2.2. Shannon Entropy

Shannon entropy [30] is a measure of the average uncertainty in a random variable. Some basic notions of entropy are given here. For more details, see [16,18]. For simplicity, all random variables mentioned in the paper are outcomes in the sets of finite strings.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, the marginal distributions of X and Y are defined by $f_1\left(x\right)={\sum }_{y}f(x,y)$ and $f_2\left(x\right)={\sum }_{x}f(x,y)$, respectively.

The joint Shannon entropy of X and Y is defined as

$$\begin{array}{c}\hfill \mathrm{H}(X,Y)=-\sum _{x\in X,y\in Y}f(x,y)\log f(x,y).\end{array}$$

(1)

The Shannon entropy of X is defined as

$$\begin{array}{c}\hfill \mathrm{H}\left(X\right)=-\sum _{x\in X}{f}_1\left(x\right)\log f_1\left(x\right)=-\sum _{x\in X,y\in Y}f(x,y)\log f_1\left(x\right).\end{array}$$

(2)

The conditional Shannon entropy with respect to Y given X is defined as

$$\begin{array}{}\mathrm{(3)}& \hfill \mathrm{H}\left(Y\right|X)& =& \sum _{x\in X}{f}_1\left(x\right)H\left(Y\right|X=x)\hfill \mathrm{(4)}& & =& -\sum _{x\in X}{f}_1\left(x\right)\sum _{y\in Y}f\left(y\right|x)\log f\left(y\right|x)\hfill \mathrm{(5)}& & =& -\sum _{x\in X,y\in Y}f(x,y)\log f\left(x\right|y).\hfill \end{array}$$

The mutual information between variables X and Y is defined as

$$\mathrm{I}(X;Y)=\mathrm{H}\left(X\right)-\mathrm{H}\left(X\right|Y).$$

(6)

Kolmogorov complexity and Shannon entropy are fundamentally different measures. However, for any computable probability distributions, up to $\mathrm{K}\left(f\right)+O\left(1\right)$, the Shannon entropy equals the expected value of the Kolmogorov complexity [17,18,19]. Conditional Kolmogorov complexity and conditional Shannon entropy are also related.

The following two Lemmas are Theorem 8.1.1 from [17] and Theorem 5 from [22], respectively.

Lemma 1.

Let X be a random variable over $\mathcal{X}$. For any computable probability distribution $f\left(x\right)$ over $\mathcal{X}$,

$$0\le (\sum _{x}f(x)\mathrm{K}(x)-\mathrm{H}(X))\le \mathrm{K}(f)+O(1).$$

(7)

Lemma 2.

Let $X,Y$ be two random variables over $\mathcal{X}$, $\mathcal{Y}$, respectively. For any computable probability distribution $f(x,y)$ over $\mathcal{X}\times \mathcal{Y}$,

$$0\le (\sum _{x,y}f(x,y)\mathrm{K}(x|y)-\mathrm{H}(X|Y))\le \mathrm{K}(f)+O(1).$$

(8)

The following two Lemmas will be used in the next section.

Lemma 3.

There are four positive integer $a,b,c,d$ such that

$$\frac{1}{2}\max (a,b)+\frac{1}{2}\max (c,d)>\max (\frac{a+c}{2},\frac{b+d}{2}).$$

Proof. 

Let $a>b>0$ and $d>c>0$, then $\frac{1}{2}\max (a,b)+\frac{1}{2}\max (c,d)=\frac{a+d}{2}>\max (\frac{a+c}{2},\frac{b+d}{2})$. ☐

Lemma 4.

There are four positive integer $a,b,c,d$ such that

$$\frac{1}{2}\min (a,b)+\frac{1}{2}\min (c,d)<\min (\frac{a+c}{2},\frac{b+d}{2}).$$

Proof. 

Let $a>b>0$ and $d>c>0$, then $\frac{1}{2}\min (a,b)+\frac{1}{2}\min (c,d)=\frac{b+c}{2}<\min (\frac{a+c}{2},\frac{b+d}{2})$. ☐

3. Information Distance Versus Entropy

A metric on a set $\mathcal{X}$ is a function $d:\mathcal{X}\times \mathcal{X}\to {\mathbb{R}}^{+}$ having the following properties: for every $x,y,z\in \mathcal{X}$

(i)

$d(x,y)=0$ if and only if $x=y$;

(ii)

$d(x,y)=d(y,x)$;

(iii)

$d(x,y)+d(y,z)\ge d(x,z)$.

Here, entropy metric means the metric on the set of all random variables over a set. $d(X,Y)=H\left(X\right|Y)+H(Y\left|X\right)$ is a metric [16]. It is easy to know that $d(X,Y)=\max \left\{H\right(X\left|Y\right),H\left(Y\right|X\left)\right\}$ is also a metric.

Information distance ${\mathrm{E}}_{\max }(x,y)$ [1], the length of a shortest program computing y from x and vice versa, is defined as

$${\mathrm{E}}_{\max }(x,y)=\min \left\{\right|p|:F(p,x)=y,F(p,y)=x\}.$$

In [1], up to an additive logarithmic term, the equality, ${\mathrm{E}}_{\max }(x,y)=\max (K\left(x\right|y),K\left(y\right|x))$, holds. So ${\mathrm{E}}_{\max }(x,y)$ is called the max distance between x and y.

We show the following relationship between max distance and the entropy metric.

Theorem 1.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\max (H\left(X\right|Y),H\left(Y\right|X))\le \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\le H\left(X\right|Y)+H\left(Y\right|X)+2K\left(f\right)+O\left(1\right)$$

Proof. 

First, from Lemma 2, we have

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\ge \sum _{x,y}f(x,y)\mathrm{K}\left(x\right|y)\ge \mathrm{H}\left(X\right|Y),\end{array}$$

(9)

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\ge \sum _{x,y}f(x,y)\mathrm{K}\left(y\right|x)\ge \mathrm{H}\left(Y\right|X).\end{array}$$

(10)

Thus

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\ge \max (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X)).\end{array}$$

(11)

Moreover, from Lemma 2, we get

$$\begin{array}{ccc}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)& \le & \sum _{x,y}f(x,y)(\mathrm{K}\left(x\right|y)+\mathrm{K}\left(y\right|x))\hfill \\ & \le & \sum _{x,y}f(x,y)\mathrm{K}\left(x\right|y)+\sum _{x,y}f(x,y)\mathrm{K}\left(y\right|x)\hfill \\ & \le & \mathrm{H}\left(X\right|Y)+\mathrm{H}(Y\left|X\right)+2K\left(f\right)+O\left(1\right).\hfill \end{array}$$

 ☐

Remark 1.

From the above Theorem, the inequality ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\ge \max (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ holds.

Unfortunately, the inequality ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\le \max (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ does not hold.

For instance, let the joint probability distribution $f(x,y)$ of X and Y be $f(x_1,y_1)=f(x_2,y_2)=0.5$, and let $a=K\left(x_1\right|y_1),$ $b=K\left(y_1\right|x_1),$ $c=K\left(x_2\right|y_2)$ and $d=K\left(y_2\right|x_2)$ such that $a\ne b$ and $d\ne c$. Assume, without loss of generality, that $a>b$ and $d>c$, then from Lemma 3, we have ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)>{\sum }_{x,y}f(x,y)K\left(x\right|y)+{\sum }_{x,y}f(x,y)K\left(y\right|x)$.

This means we will get the inequality ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)>{\sum }_{x,y}f(x,y)K\left(x\right|y)+{\sum }_{x,y}f(x,y)K\left(y\right|x)\ge \max \left(\mathrm{H}\right(X\left|Y\right),\mathrm{H}\left(Y\right|X\left)\right)$ for some cases.

From above results, we know that the relationship ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\approx \max (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ does not hold.

Because the mutual information between X and Y is defined as $\mathrm{I}(X;Y)=\mathrm{H}\left(X\right)-\mathrm{H}\left(X\right|Y),$ we have the following result.

Corollary 1.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\max (H\left(X\right),H\left(Y\right))-I(X;Y)\le \sum _{x,y}f(x,y){\mathrm{E}}_{\max }(x,y)\le H\left(X\right)+H\left(Y\right)-2I(X;Y)+2K\left(u\right)+O\left(1\right)$$

Min distance ${\mathrm{E}}_{\min }(x,y)$ [3,4] is defined as

$${\mathrm{E}}_{\min }(x,y)=\min \left\{\right|p|:F(p,x,z)=y,F(p,y,r)=x,\left|p\right|+\left|z\right|+\left|r\right|\le {\mathrm{E}}_{\max }(x,y)\}.$$

In [3,4], the equality, ${\mathrm{E}}_{\min }(x,y)=\min (K\left(x\right|y),K\left(y\right|x))$, holds, when a term $O\left(\log \right|x|+|y\left|\right)$ is omitted. Then we have the following relationship between min distance and the entropy metric.

Theorem 2.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\sum _{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \min (H\left(X\right|Y),H\left(Y\right|X))+K\left(f\right)+O\left(1\right)$$

Proof. 

From Lemma 1, we have

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \sum _{x,y}f(x,y)\mathrm{K}\left(x\right|y)\le \mathrm{H}\left(X\right|Y)+\mathrm{K}\left(f\right)+O\left(1\right),\end{array}$$

(12)

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \sum _{x,y}f(x,y)\mathrm{K}\left(y\right|x)\le \mathrm{H}\left(Y\right|X)+\mathrm{K}\left(f\right)+O\left(1\right).\end{array}$$

(13)

Thus

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \min (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))+\mathrm{K}\left(f\right)+O\left(1\right).\end{array}$$

(14)

 ☐

Remark 2.

From the above Theorem, the inequality ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \min (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ holds.

Unfortunately, from Lemma 4, we know that the inequality ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\ge \min (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ does not hold.

Thus, the relationship ${\sum }_{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\approx \min (\mathrm{H}\left(X\right|Y),\mathrm{H}\left(Y\right|X))$ does not hold.

Corollary 2.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\sum _{x,y}f(x,y){\mathrm{E}}_{\min }(x,y)\le \min (H\left(X\right),H\left(Y\right))-I(X;Y)+K\left(f\right)+O\left(1\right)$$

Sum distance ${\mathrm{E}}_{sum}(x,y)$ [1] is defined as

$${\mathrm{E}}_{sum}(x,y)=K\left(x\right|y)+K\left(y\right|x).$$

Then we have the following relationship between sum distance and entropy measure.

Theorem 3.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$H\left(X\right|Y)+H\left(Y\right|X)\le \sum _{x,y}f(x,y){\mathrm{E}}_{sum}(x,y)\le H\left(X\right|Y)+H\left(Y\right|X)+2K\left(f\right)+O\left(1\right)$$

Proof. 

First, we have

$$\begin{array}{ccc}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{sum}(x,y)& =& \sum _{x,y}f(x,y)(\mathrm{K}\left(x\right|y)+\mathrm{K}\left(y\right|x))\hfill \\ & =& \sum _{x,y}f(x,y)\mathrm{K}\left(x\right|y)+\sum _{x,y}f(x,y)\mathrm{K}\left(x\right|y)\hfill \end{array}$$

Then, from Lemma 1, we can get

$$\begin{array}{c}\hfill 0\le \sum _{x,y}f(x,y){\mathrm{E}}_{sum}(x,y)-(\mathrm{H}\left(X\right|Y)+\mathrm{H}\left(Y\right|X))\le 2\mathrm{K}\left(f\right)+O\left(1\right).\end{array}$$

(15)

 ☐

Corollary 3.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$H\left(X\right)+H\left(Y\right)-2I(X;Y)\le \sum _{x,y}f(x,y){\mathrm{E}}_{sum}(x,y)\le H\left(X\right)+H\left(Y\right)-2I(X;Y)+2K\left(f\right)+O\left(1\right)$$

From above results we know that, when f is given, up to a additive constant,

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{E}}_{sum}(x,y)=\mathrm{H}\left(X\right|Y)+\mathrm{H}\left(Y\right|X)=H\left(X\right)+H\left(Y\right)-2I(X;Y).\end{array}$$

(16)

4. Normalized Information Distance Versus Entropy

In this section, we establish relationships between entropy and the normalized versions of information distances.

The normalized version ${\mathrm{e}}_{max}(x,y)$ [2] of ${\mathrm{E}}_{max}(x,y)$ is defined as

$${\mathrm{e}}_{max}(x,y)=\frac{\max \left(K\right(x\left|y\right),K\left(y\right|x\left)\right)}{\max \left(K\right(x),K(y\left)\right)}.$$

Theorem 4.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\sum _{x,y}f(x,y){\mathrm{e}}_{\max }(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right)}{\max \left(H\right(X),H(Y\left)\right)}$$

Proof. 

First, we have

$$\begin{array}{c}\hfill {\mathrm{e}}_{max}(x,y)=\frac{\max \left(K\right(x\left|y\right),K\left(y\right|x\left)\right)}{\max \left(K\right(x),K(y\left)\right)}\le \frac{K\left(x\right|y)+K(y\left|x\right)}{K\left(x\right)}.\end{array}$$

(17)

Then

$$\begin{array}{c}\hfill K\left(x\right){\mathrm{e}}_{max}(x,y)\le K\left(x\right|y)+K\left(y\right|x).\end{array}$$

(18)

Therefore,

$$\begin{array}{ccc}\hfill \sum _{x,y}f(x,y)\left[K\left(x\right){\mathrm{e}}_{max}(x,y)\right]& \le & \sum _{x,y}f(x,y)(K\left(x\right|y)+K\left(y\right|x))\hfill \\ & =& \sum _{x,y}f(x,y)K\left(x\right|y)+\sum _{x,y}f(x,y)K\left(y\right|x),\hfill \end{array}$$

From Lemmas 1 and 2, we have

$$\begin{array}{ccc}\hfill H(x)[\sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)]& \le & [\sum _{x,y}f(x,y)][\sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)],\mathrm{by}\mathrm{Lemma}1\\ & \le & \sum _{x,y}f(x,y)[K\left(x\right){\mathrm{e}}_{max}(x,y)]\hfill \\ & \le & \sum _{x,y}f(x,y)K\left(x\right|y)+\sum _{x,y}f(x,y)K\left(y\right|x)\hfill \\ & \le & \mathrm{H}\left(X\right|Y)+\mathrm{H}(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right),\mathrm{by}\mathrm{Lemma}2\hfill \end{array}$$

Then

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right)}{H\left(X\right)}.\end{array}$$

(19)

Similarly, we have

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right)}{H\left(Y\right)}.\end{array}$$

(20)

Thus

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right)}{\max \left(H\right(X),H(Y\left)\right)}.\end{array}$$

(21)

 ☐

Corollary 4.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\sum _{x,y}f(x,y){\mathrm{e}}_{max}(x,y)\le \frac{H\left(X\right)+H\left(Y\right)-2I(X;Y)+2\mathrm{K}\left(f\right)+O\left(1\right)}{\max \left(H\right(X),H(Y\left)\right)}$$

The normalized version ${\mathrm{e}}_{min}(x,y)$ [3,4] of ${\mathrm{E}}_{min}(x,y)$ is defined as

$${\mathrm{e}}_{min}(x,y)=\frac{\min \left(K\right(x\left|y\right),K\left(y\right|x\left)\right)}{\min \left(K\right(x),K(y\left)\right)}.$$

Because ${\mathrm{e}}_{min}(x,y)\le {\mathrm{e}}_{max}(x,y)$, for all $x,y$ [3,4], the following Corollary is straightforward with the above Theorem.

Corollary 5.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\sum _{x,y}f(x,y){\mathrm{e}}_{\min }(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2\mathrm{K}\left(f\right)+O\left(1\right)}{\max \left(H\right(X),H(Y\left)\right)}$$

The normalized version ${\mathrm{e}}_{sum}(x,y)$ [2] of ${\mathrm{E}}_{sum}(x,y)$ is defined as

$${\mathrm{e}}_{sum}(x,y)=\frac{K\left(x\right|y)+K(y\left|x\right)}{K(x,y)}.$$

Theorem 5.

Let $X,Y$ be two random variables with a computable joint probability distribution $u(x,y)$, then

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{sum}(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2K\left(f\right)+O\left(1\right)}{H(X,Y)}\end{array}$$

(22)

Proof. 

Since ${\mathrm{e}}_{sum}(x,y)=\frac{K\left(x\right|y)+K(y\left|x\right)}{K(x,y)},$ then

$$\begin{array}{c}\hfill K(x,y){\mathrm{e}}_{sum}(x,y)=K\left(x\right|y)+K\left(y\right|x).\end{array}$$

(23)

Therefore,

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y)\left[K(x,y){\mathrm{e}}_{sum}(x,y)\right]=\sum _{x,y}f(x,y)K\left(x\right|y)+\sum _{x,y}f(x,y)K\left(y\right|x).\end{array}$$

(24)

From Lemma 1, we have

$$\begin{array}{ccc}\hfill H(X,Y)[\sum _{x,y}f(x,y){\mathrm{e}}_{sum}(x,y)]& \le & [\sum _{x,y}f(x,y)K(x,y)][\sum _{x,y}f(x,y){\mathrm{e}}_{sum}(x,y)]\hfill \\ & \le & \sum _{x,y}f(x,y)\left[K(x,y){\mathrm{e}}_{sum}(x,y)\right]\hfill \\ & =& \sum _{x,y}f(x,y)K\left(x\right|y)+\sum _{x,y}f(x,y)K\left(y\right|x)\hfill \\ & \le & H\left(X\right|Y)+H(Y\left|X\right)+2K\left(f\right)+O\left(1\right)\hfill \end{array}$$

Thus

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{sum}(x,y)\le \frac{H\left(X\right|Y)+H(Y\left|X\right)+2K\left(f\right)+O\left(1\right)}{H(X,Y)}.\end{array}$$

(25)

 ☐

Corollary 6.

Let $X,Y$ be two random variables with a computable joint probability distribution $f(x,y)$, then

$$\begin{array}{c}\hfill \sum _{x,y}f(x,y){\mathrm{e}}_{sum}(x,y)\le \frac{H(X,Y)-I(X;Y)+2K\left(f\right)+O\left(1\right)}{H(X,Y)}.\end{array}$$

(26)

5. Conclusions

As we know, the Shannon entropy of a distribution is approximately equal to the expected Kolmogorov complexity, up to a constant term that only depends on the distribution [17]. We studied whether a similar relationship holds for information distance. Theorem 5 gave the analogous result for sum distance. We also gave some bounds for the expected value of other (normalized) information distances.