1. Introduction
This paper is an extended version of our paper that is published in the International Symposium on Information Theory 2017 (ISIT 2017) [
1].
Let
and
be two finite sets, and let
W be a fixed channel with input alphabet
and output alphabet
. It is well known that the input-output mutual information is continuous on the simplex of input probability distributions. Many other parameters that depend on the input probability distribution were shown to be continuous on the simplex in [
2].
Polyanskiy studied in [
3] the continuity of the Neyman–Pearson function for a binary hypothesis test that arises in the analysis of channel codes. He showed that for arbitrary input and output alphabets, this function is continuous in the input distribution in the total variation topology. He also showed that under some regularity assumptions, this function is continuous in the weak-∗ topology.
If
and
are finite sets, the space of channels with input alphabet
and output alphabet
can naturally be endowed with the topology of the Euclidean metric, or any other equivalent metric. It is well known that the channel capacity is continuous in this topology. If
and
are arbitrary, one can construct a topology on the space of channels using the weak-∗ topology on the output alphabet. It was shown in [
4] that the capacity is lower semi-continuous in this topology.
The continuity results that are mentioned in the previous paragraph do not take into account “equivalence” between channels. Two channels are said to be equivalent if they are degraded from each other. This means that each channel can be simulated from the other by local operations at the receiver. Two channels that are degraded from each other are completely equivalent from an operational point of view: both channels have exactly the same probability of error under optimal decoding for any fixed code. Moreover, any sub-optimal decoder for one channel can be transformed to a sub-optimal decoder for the other channel with the same probability of error and essentially the same computational complexity. This is why it makes sense, from an information-theoretic point of view, to identify equivalent channels and consider them as one point in the space of “equivalent channels”.
In [
5], equivalent binary-input channels were identified with their
L-density (i.e., the density of log-likelihood ratios). The space of equivalent binary-input channels was endowed with the topology of convergence in distribution of
L-densities. Since the symmetric capacity (the symmetric capacity is the input-output mutual information with uniformly-distributed input) and the Bhattacharyya parameter can be written as an integral of a continuous function with respect to the
L-density [
5], it immediately follows that these parameters are continuous in the
L-density topology.
In [
6], many topologies were constructed for the space of equivalent channels sharing a fixed input alphabet. In this paper, we study the continuity of many channel parameters and operations under these topologies. The continuity of channel parameters and operations might be helpful in the following two problems:
If a parameter (such as the optimal probability of error of a given code) is difficult to compute for a channel W, one can approximate it by computing the same parameter for a sequence of channels that converges to W in some topology where the parameter is continuous.
The study of the robustness of a communication system against the imperfect specification of the channel.
In
Section 2, we introduce the preliminaries for this paper. In
Section 3, we recall the main results of [
6] that we need here. In
Section 4, we introduce the channel parameters and operations that we investigate in this paper. In
Section 5, we study the continuity of these parameters and operations in the quotient topology of the space of equivalent channels with fixed input and output alphabets. The continuity in the strong topology of the space of equivalent channels sharing the same input alphabet is studied in
Section 6. Finally, the continuity in the noisiness/weak-∗ and the total variation topologies is studied in
Section 7.
2. Preliminaries
We assume that the reader is familiar with the basic concepts of General Topology. The main concepts and theorems that we need can be found in the Preliminaries Section of [
6].
2.1. Set-Theoretic Notations
For every integer , we denote the set as .
The set of mappings from a set A to a set B is denoted as .
Let
A be a subset of
B. The indicator mapping
of
A in
B is defined as:
If the superset B is clear from the context, we simply write to denote the indicator mapping of A in B.
The power set of B is the set of subsets of B. Since every subset of B can be identified with its indicator mapping, we denote the power set of B as .
Let be a collection of arbitrary sets indexed by I. The disjoint union of is defined as . For every , the i-th-canonical injection is the mapping defined as . If no confusions can arise, we can identify with through the canonical injection. Therefore, we can see as a subset of for every .
Let R be an equivalence relation on a set T. For every , the set is the R-equivalence class of x. The collection of R-equivalence classes, which we denote as , forms a partition of T, and it is called the quotient space of T by R. The mapping defined as for every is the projection mapping onto .
2.2. Topological Notations
A topological space is said to be contractible to if there exists a continuous mapping such that and for every , where is endowed with the Euclidean topology. is strongly contractible to if we also have for every .
Intuitively, T is contractible if it can be “continuously shrinked” to a single point . If this “continuous shrinking” can be done without moving , T is strongly contractible.
Note that contractibility is a very strong notion of connectedness: every contractible space is path-connected and simply connected. Moreover, all its homotopy, homology and cohomology groups of order are zero.
Let be a collection of topological spaces indexed by I. The product topology on is denoted by . The disjoint union topology on is denoted by .
The following lemma is useful to show the continuity of many functions.
Lemma 1. Let and be two compact topological spaces, and let be a continuous function on . For every and every , there exists a neighborhood of s such that for every , we have: 2.3. Quotient Topology
Let
be a topological space, and let
R be an equivalence relation on
T. The quotient topology on
is the finest topology that makes the projection mapping
continuous. It is given by:
Lemma 2. Let be a continuous mapping from to . If for every satisfying , then we can define a transcendent mapping such that for any . f is well defined on . Moreover, f is a continuous mapping from to .
Let and be two topological spaces, and let R be an equivalence relation on T. Consider the equivalence relation on defined as if and only if and . A natural question to ask is whether the canonical bijection between and is a homeomorphism. It turns out that this is not the case in general. The following theorem, which is widely used in Algebraic Topology, provides a sufficient condition:
Theorem 1. [7] If is locally compact and Hausdorff, then the canonical bijection between , and is a homeomorphism. Corollary 1. Let and be two topological spaces, and let and be two equivalence relations on T and S, respectively. Define the equivalence relation R on as if and only if and . If and are locally compact and Hausdorff, then the canonical bijection between and is a homeomorphism.
Proof. We just need to apply Theorem 1 twice. Define the equivalence relation on as follows: if and only if and . Since is locally compact and Hausdorff, Theorem 1 implies that the canonical bijection from to is a homeomorphism. Let us identify these two spaces through the canonical bijection.
Now, define the equivalence relation on as follows: if and only if and . Since is locally compact and Hausdorff, Theorem 1 implies that the canonical bijection from to is a homeomorphism.
Since we identified and through the canonical bijection (which is a homeomorphism), can be seen as an equivalence relation on . It is easy to see that the canonical bijection from to is a homeomorphism. We conclude that the canonical bijection from to is a homeomorphism. ☐
2.4. Measure-Theoretic Notations
If is a measurable space, we denote the set of probability measures on as . If the -algebra is known from the context, we simply write to denote the set of probability measures.
If and is a measurable singleton, we simply write to denote .
For every
, the total variation distance between
and
is defined as:
The push-forward probability measure:
Let P be a probability measure on , and let be a measurable mapping from to another measurable space . The push-forward probability measure of P by f is the probability measure on defined as for every .
A measurable mapping
is integrable with respect to
if and only if
is integrable with respect to
P. Moreover,
The mapping
from
to
is continuous if these spaces are endowed with the total variation topology:
where (a) follows from Property 1 of [
8].
Probability measures on finite sets:
We always endow finite sets with their finest
-algebra, i.e., the power set. In this case, every probability measure is completely determined by its value on singletons, i.e., if
P is a probability measure on a finite set
, then for every
, we have:
If
is a finite set, we denote the set of probability distributions on
as
. Note that
is an
-dimensional simplex in
. We always endow
with the total variation distance and its induced topology. For every
, we have:
Products of probability measures:
We denote the product of two measurable spaces and as . If and , we denote the product of and as .
If
,
and
are endowed with the total variation topology, the mapping
is a continuous mapping (see
Appendix B).
Borel sets and the support of a probability measure:
Let be a Hausdorff topological space. The Borel -algebra of is the -algebra generated by . We denote the Borel -algebra of as . If the topology is known from the context, we simply write to denote the Borel -algebra. The sets in are called the Borel sets of T.
The support of a measure
is the set of all points
for which every neighborhood has a strictly positive measure:
If P is a probability measure on a Polish space, then .
2.5. Random Mappings
Let M and be two arbitrary sets, and let be a -algebra on . A random mapping from M to is a mapping R from M to . For every , can be interpreted as the probability distribution of the random output given that the input is x.
Let be a -algebra on M. We say that R is a measurable random mapping from to if the mapping defined as is measurable for every .
Note that this definition of measurability is consistent with the measurability of ordinary mappings: let
f be a mapping from
M to
, and let
be the random mapping defined as
for every
, where
is a Dirac measure centered at
. We have:
where (a) and (b) follow from the fact that
is either one or zero, depending on whether
or not.
Let
P be a probability measure on
, and let
R be a measurable random mapping from
to
. The push-forward probability measure of
P by
R is the probability measure
on
defined as:
Note that this definition is consistent with the push-forward of ordinary mappings: if
f and
are as above, then for every
, we have:
Proposition 1. Let R be a measurable random mapping from to . If is a -measurable mapping, then the mapping is a measurable mapping from to . Moreover, for every , we have: Corollary 2. If is bounded and -measurable, then the mapping:is bounded and Σ
-measurable. Moreover, for every , we have: Proof. Write (where and ), and use the fact that every bounded measurable function is integrable over any probability distribution. ☐
Lemma 3. For every measurable random mapping R from to , the push-forward mapping is continuous from to under the total variation topology.
Lemma 4. Let be a Polish (This assumption can be dropped. We assumed that is Polish just to avoid working with Moore–Smith nets.) topology on M, and let be an arbitrary topology on . Let R be a measurable random mapping from to . Moreover, assume that R is a continuous mapping from to when the latter space is endowed with the weak-∗ topology. Under these assumptions, the push-forward mapping is continuous from to under the weak-∗ topology.
2.6. Meta-Probability Measures
Let be a finite set. A meta-probability measure on is a probability measure on the Borel sets of . It is called a meta-probability measure because it is a probability measure on the space of probability distributions on .
We denote the set of meta-probability measures on as . Clearly, .
A meta-probability measure MP on
is said to be balanced if it satisfies:
where
is the uniform probability distribution on
.
We denote the set of all balanced meta-probability measures on as . The set of all balanced and finitely-supported meta-probability measures on is denoted as .
The following lemma is useful to show the continuity of functions defined on .
Lemma 5. Let be a compact topological space, and let be a continuous function on . The mapping defined as:is continuous, where is endowed with the weak-∗ topology. Let f be a mapping from a finite set to another finite set . f induces a push-forward mapping taking probability distributions in to probability distributions in . is continuous because and are endowed with the total variation distance. in turn induces another push-forward mapping taking meta-probability measures in to meta-probability measures in . We denote this mapping as , and we call it the meta-push-forward mapping induced by f. Since is a continuous mapping from to , is a continuous mapping from to under both the weak-∗ and the total variation topologies.
Let and be two finite sets. Let be defined as . For every and , we define the tensor product of and as .
Note that since
,
and
are endowed with the total variation topology,
is a continuous mapping from
to
. Therefore,
is a continuous mapping from
to
under both the weak-∗ and the total variation topologies. On the other hand,
Appendix B and
Appendix F imply that the mapping
from
to
is continuous under both the weak-∗ and the total variation topologies. We conclude that the tensor product is continuous under both of these topologies.
3. The Space of Equivalent Channels
In this section, we summarize the main results of [
6].
3.1. Space of Channels from to
A discrete memoryless channel W is a three-tuple where is a finite set that is called the input alphabet of W, is a finite set that is called the output alphabet of W and is a function satisfying .
For every , we denote as , which we interpret as the conditional probability of receiving y at the output, given that x is the input.
Let be the set of all channels having as the input alphabet and as the output alphabet.
For every
, define the distance between
W and
as follows:
We always endow
with the metric distance
. This metric makes
a compact path-connected metric space. The metric topology on
that is induced by
is denoted as
.
3.2. Equivalence between Channels
Let
and
be two channels having the same input alphabet. We say that
is degraded from
W if there exists a channel
such that:
W and
are said to be equivalent if each one is degraded from the other.
Let
and
be the space of probability distributions on
and
, respectively. Define
as
for every
. The image of
W is the set of output-symbols
having strictly positive probabilities:
For every
, define
as follows:
For every , we have . On the other hand, if and , we have . This shows that and the collection uniquely determine W.
The Blackwell measure (denoted
) of
W is a meta-probability measure on
defined as:
where
is a Dirac measure centered at
. In an earlier version of this work, I called
the posterior meta-probability distribution of
W. Maxim Raginsky thankfully brought to my attention the fact that
is called the Blackwell measure.
It is known that a meta-probability measure MP on
is the Blackwell measure of some discrete memoryless channels (DMC) with input alphabet
if and only if it is balanced and finitely supported [
9].
It is also known that two channels
and
are equivalent if and only if
[
9].
3.3. Space of Equivalent Channels from to
Let
and
be two finite sets. Define the equivalence relation
on
as follows:
The space of equivalent channels with input alphabet
and output alphabet
is the quotient of
by the equivalence relation:
Quotient topology:
We define the topology on as the quotient topology . We always associate with the quotient topology .
We have shown in [
6] that
is a compact, path-connected and metrizable space.
If
and
are two finite sets of the same size, there exists a canonical homeomorphism between
and
[
6]. This allows us to identify
with
, where
and
.
Moreover, for every
, there exists a canonical subspace of
that is homeomorphic to
[
6]. Therefore, we can consider
as a compact subspace of
.
Noisiness metric:
For every
, let
be the space of probability distributions on
. Let
be a finite set, and let
. For every
, define
as follows:
The quantity
depends only on the
-equivalence class of
W (see [
6]). Therefore, if
, we can define
for any
.
Define the noisiness distance
as follows:
We have shown in [
6] that
is topologically equivalent to
.
3.4. Space of Equivalent Channels with Input Alphabet
The space of channels with input alphabet
is defined as:
We define the equivalence relation
on
as follows:
The space of equivalent channels with input alphabet
is the quotient of
by the equivalence relation:
For every
and every
, we identify the
-equivalence class of
W with the
-equivalence class of it. This allows us to consider
as a subspace of
. Moreover,
Since any two equivalent channels have the same Blackwell measure, we can define the Blackwell measure of
as
for any
. The rank of
is the size of the support of its Blackwell measure:
A topology on is said to be natural if and only if it induces the quotient topology on for every .
Every natural topology is
-compact, separable and path-connected [
6]. On the other hand, if
, a Hausdorff natural topology is not Baire, and it is not locally compact anywhere [
6]. This implies that no natural topology can be completely metrized if
.
Strong topology on :
We associate
with the disjoint union topology
. The space
is disconnected, metrizable and
-compact [
6].
The strong topology
on
is the quotient of
by
:
We call open and closed sets in as strongly-open and strongly-closed sets, respectively. If A is a subset of , then A is strongly open if and only if is open in for every . Similarly, A is strongly closed if and only if is closed in for every .
We have shown in [
6] that
is the finest natural topology. The strong topology is sequential, compactly generated and
[
6]. On the other hand, if
, the strong topology is not first-countable anywhere [
6]; hence, it is not metrizable.
Noisiness metric:
Define the noisiness metric on
as follows:
is well-defined because
does not depend on
as long as
. We can also express
as follows:
The metric topology on
that is induced by
is called the noisiness topology on
, and it is denoted as
. We have shown in [
6] that
is a natural topology that is strictly coarser than
.
Topologies from Blackwell measures:
The mapping is a bijection from to . We call this mapping the canonical bijection from to .
Since is a metric space, there are many standard ways to construct topologies on . If we choose any of these standard topologies on and then relativize it to the subspace , we can construct topologies on through the canonical bijection.
In [
6], we studied the weak-∗ and the total variation topologies. We showed that the weak-∗ topology is exactly the same as the noisiness topology.
The total-variation metric distance
on
is defined as:
The total-variation topology
is the metric topology that is induced by
on
. We proved in [
6] that if
, we have:
is not natural, nor Baire, hence it is not completely metrizable.
is not locally compact anywhere.
5. Continuity on
It is well known that the parameters defined in
Section 4.1 depend only on the
-equivalence class of
W. Therefore, we can define those parameters for any
through the transcendent mapping (defined in Lemma 2). The following proposition shows that those parameters are continuous on
:
Proposition 4. We have:
is continuous and concave in p.
is continuous.
is continuous and concave in p.
is continuous.
For every code on , is continuous.
For every and every , the mapping is continuous.
Proof. Since the corresponding parameters are continuous on (Proposition 2), Lemma 2 implies that they are continuous on . The only cases that need a special treatment are those of I and Z. We will only prove the continuity of I since the proof of continuity of Z is similar.
Define the relation
R on
as:
It is easy to see that depends only on the R-equivalence class of . Since I is continuous on , Lemma 2 implies that the transcendent mapping of I is continuous on . On the other hand, since is locally compact, Theorem 1 implies that can be identified with , and the two spaces have the same topology. Therefore, I is continuous on . ☐
With the exception of channel composition, all the channel operations that were defined in
Section 4.2 can also be “quotiented”. We just need to realize that the equivalence class of the resulting channel depends only on the equivalence classes of the channels that were used in the operation. Let us illustrate this in the case of channel sums:
Let
and
and assume that
is degraded from
and
is degraded from
. There exists
and
such that
and
. It is easy to see that
, which shows that
is degraded from
. This was proven by Shannon in [
16].
Therefore, if is equivalent to and is equivalent to , then is equivalent to . This allows us to define the channel sum for every and every as for any and any , where is the -equivalence class of .
With the exception of channel composition, we can “quotient” all the channel operations of
Section 4.2 in a similar fashion. Moreover, we can show that they are continuous:
Proposition 5. We have:
The mapping from to is continuous.
The mapping from to is continuous.
The mapping from to is continuous.
For any binary operation ∗ on , the mapping from to is continuous.
For any binary operation ∗ on , the mapping from to is continuous.
Proof. We only prove the continuity of the channel sum because the proof of continuity of the other operations is similar.
Let be the projection onto the -equivalence classes. Define the mapping as . Clearly, f is continuous.
Now, define the equivalence relation
R on
as:
The discussion before the proposition shows that depends only on the R-equivalence class of . Lemma 2 now shows that the transcendent map of f defined on is continuous.
Notice that can be identified with . Therefore, we can define f on through this identification. Moreover, since and are locally compact and Hausdorff, Corollary 1 implies that the canonical bijection between and is a homeomorphism.
Now, since the mapping f on is just the channel sum, we conclude that the mapping from to is continuous. ☐
6. Continuity in the Strong Topology
The following lemma provides a way to check whether a mapping defined on is continuous:
Lemma 6. Let be an arbitrary topological space. A mapping is continuous on if and only if it is continuous on for every .
Since the channel parameters
I,
C,
,
Z,
and
are defined on
for every
(see
Section 5), they are also defined on
. The following proposition shows that those parameters are continuous in the strong topology:
Proposition 6. Let be the standard topology on . We have:
is continuous on and concave in p.
is continuous on .
is continuous on and concave in p.
is continuous on .
For every code on , is continuous on .
For every and every , the mapping is continuous on .
Proof. The continuity of and immediately follows from Proposition 4 and Lemma 6. Since the proofs of the continuity of I and Z are similar, we only prove the continuity for I.
Due to the distributivity of the product with respect to disjoint unions, we have:
and:
Therefore, is the disjoint union of the spaces . Moreover, I is continuous on for every . We conclude that I is continuous on .
Define the relation R on as follows: if and only if and . Since depends only on the R-equivalence class of , Lemma 2 shows that the transcendent map of I is a continuous mapping from to . On the other hand, since is locally compact and Hausdorff, Theorem 1 implies that can be identified with . Therefore, I is continuous on . ☐
It is also possible to extend the definition of all the channel operations that were defined in
Section 5 to
. Moreover, it is possible to show that many channel operations are continuous in the strong topology:
Proposition 7. Assume that all equivalent channel spaces are endowed with the strong topology. We have:
The mapping from to is continuous.
The mapping from to is continuous.
The mapping from to is continuous.
For any binary operation ∗ on , the mapping from to is continuous.
For any binary operation ∗ on , the mapping from to is continuous.
Proof. We only prove the continuity of the channel interpolation because the proof of the continuity of other operations is similar.
Let
be the standard topology on
. Due to the distributivity of the product with respect to disjoint unions, we have:
and:
Therefore, the space is the topological disjoint union of the spaces .
For every , let be the projection onto the -equivalence classes, and let be the canonical injection from to .
Define the mapping
as:
where
n is the unique integer satisfying
.
and
are the
and
-equivalence classes of
and
, respectively.
Due to Proposition 3 and due to the continuity of and , the mapping f is continuous on for every . Therefore, f is continuous on .
Let be the equivalence relation defined on as follows: if and only if and . Furthermore, define the equivalence relation R on as follows: if and only if and .
Since depends only on the R-equivalence class of , Lemma 2 implies that the transcendent mapping of f is continuous on .
Since is Hausdorff and locally compact, Theorem 1 implies that the canonical bijection from to is a homeomorphism. On the other hand, since and are Hausdorff and locally compact, Corollary 1 implies that the canonical bijection from to is a homeomorphism. We conclude that the channel interpolation is continuous on . ☐
Corollary 3. is strongly contractible to every point in .
Proof. Fix . Define the mapping as . H is continuous by Proposition 7. We also have and for every . Moreover, for every . Therefore, is strongly contractible to every point in . ☐
The reader might be wondering why channel operations such as the channel sum were not shown to be continuous on the whole space instead of the smaller space . The reason is because we cannot apply Corollary 1 to and since neither , nor is locally compact (under the strong topology).
One potential method to show the continuity of the channel sum on is as follows: let R be the equivalence relation on defined as if and only if and . We can identify with through the canonical bijection. Using Lemma 2, it is easy to see that the mapping is continuous from to .
It was shown in [
17] that the topology
is homeomorphic to
through the canonical bijection, where
is the coarsest topology that is both compactly generated and finer than
. Therefore, the mapping
is continuous on
. This means that if
is compactly generated, we will have
, and so, the channel sum will be continuous on
. Note that although
and
are compactly generated, their product
might not be compactly generated.