Measure of Similarity between GMMs Based on Autoencoder-Generated Gaussian Component Representations

Abstract

A novel similarity measure between Gaussian mixture models (GMMs), based on similarities between the low-dimensional representations of individual GMM components and obtained using deep autoencoder architectures, is proposed in this paper. Two different approaches built upon these architectures are explored and utilized to obtain low-dimensional representations of Gaussian components in GMMs. The first approach relies on a classical autoencoder, utilizing the Euclidean norm cost function. Vectorized upper-diagonal symmetric positive definite (SPD) matrices corresponding to Gaussian components in particular GMMs are used as inputs to the autoencoder. Low-dimensional Euclidean vectors obtained from the autoencoder’s middle layer are then used to calculate distances among the original GMMs. The second approach relies on a deep convolutional neural network (CNN) autoencoder, using SPD representatives to generate embeddings corresponding to multivariate GMM components given as inputs. As the autoencoder training cost function, the Frobenious norm between the input and output layers of such network is used and combined with regularizer terms in the form of various pieces of information, as well as the Riemannian manifold-based distances between SPD representatives corresponding to the computed autoencoder feature maps. This is performed assuming that the underlying probability density functions (PDFs) of feature-map observations are multivariate Gaussians. By employing the proposed method, a significantly better trade-off between the recognition accuracy and the computational complexity is achieved when compared with other measures calculating distances among the SPD representatives of the original Gaussian components. The proposed method is much more efficient in machine learning tasks employing GMMs and operating on large datasets that require a large overall number of Gaussian components.

1. Introduction

Gaussian mixture models have significant applications in various machine learning (ML) tasks, including computer vision and pattern recognition [1,2,3,4,5,6,7,8], as well as context-based image matching and texture recognition, since an arbitrary probability density function (PDF) could be successfully modeled by GMM, knowing the exact number of modes of that particular PDF. They can also be applied in a broad class of artificial intelligence (AI) tasks, such as automatic speech recognition [9], speaker verification and recognition [10], age and gender recognition [11], or used as input and/or output data representations in deep learning [12], and within an emerging field of variational autoencoders [4]. As many of those systems are complex, i.e., they involve large amounts of high-dimensional multivariate GMM data representatives, developing less computationally demanding and, at the same time, reasonably accurate GMM similarity measures has become a crucial issue.

Current solutions can be divided into two main categories. The first and the main direction is based on the utilization of information distances between PDFs, as characteristics of information sources. This means that information distances can have various levels of computational complexity and can capture different aspects of the underlying PDFs that are driving the generation of observations from the considered information sources. Thus, the choice of a particular information distance is a matter of the level of computational complexity that is considered as acceptable, the availability and the amount of collected data from information sources, and the desired level of information distance effectiveness in solving a particular ML task. Early works are represented by [13,14,15], which are based on Chernoff, Bhattacharyya, and Matusita information distances, and the most effective Kullback–Leibler (KL) divergence [16,17], as given in [2], or [18,19]. The Bhattacharyya distance [14] is a special case of a Chernoff $\alpha$-divergence [20], while the family of Chernoff-$\alpha$ divergences is related to the family of Rényi $\alpha$-divergences by a simple relation described in [20]. On the other hand, Rényi $\alpha$-divergences come from the Rényi entropy that generalizes the notion of the most commonly used Shannon entropy. Thus, for $\alpha =1$, the Rényi entropy results in the Shannon entropy and consequently the Kullback–Leibler (KL) divergence, as the special case of a more general Rényi $\alpha$-divergence, i.e., the corresponding cross-entropy. The most commonly used and adopted solution among the aforementioned solutions is the KL divergence, but the distance effectiveness in measuring the similarity between the considered probability distributions always depends on the specific task and conditions.

Although the KL divergence exists in the close (i.e., analytical) form as the information distance for the case when PDFs are two multivariate Gaussian components, there is no closed-form solution for the case of GMMs. Various methods approximating the KL divergence between GMMs have been developed, approximating the KL divergence as the KL divergence between their Gaussian components, while retaining acceptably low bounds on the approximation error, as well as high recognition accuracy [2,19]. For example, in [7], a straightforward, although unacceptably computationally expensive solution (especially when dealing with huge amounts of data and the large dimensionality of the underlying feature space) is proposed, based on the Monte-Carlo method calculation of the KL divergence between two GMMs. In the same work, lower and upper approximation bounds are delivered, including experiments conducted on synthetic data, as well as in speaker verification tasks. In [2], another approximation of the KL divergence is proposed and applied in an image retrieval task. In [1], the unscented transform is delivered and applied within a speaker recognition task. Besides these, we also highlight the applications of the KL divergence in the specific tasks of game design, where it can be utilized to establish variational autoencoder (VAE) models that generate new game levels [21], or to compare and analyze the game-levels [22], e.g., as the objective function of an evolutionary algorithm to evolve new levels.

The second group of methods relying on Earth mover’s distance (EMD) also emerged [19,23], all based on the baseline Euclidean EMD method proposed in [24], designed to compare the cross bins of the histogram forms of one-dimensional PDFs. Either the KL divergence or some other information distance between the Gaussian components of GMMs is used as the actual ground or geometry-based distance between the symmetric positive definite (SPD) representations of GMM Gaussians obtained along the geodesic of Riemannian geometry of the SPD manifold, denoted as ${\mathrm{Sym}}_{++}\left(d\right)$. Nevertheless, the computational complexity of all those methods is of the order of $O\left(d^3\right)$, due to the computational complexity of covariance matrix inversion, where $d\in \mathbb{R}$ is the dimension of underlying feature space, i.e., it is computationally very expensive in the case of large GMMs, as well as large feature space dimensionality d, which is inherent to real application scenarios. The reduction in computational complexity is crucial in such systems, so various solutions are able to significantly reduce the computational complexity without significantly decreasing the recognition accuracy and addressing the mentioned issue were proposed. Some of these EMD-based measures, as can be seen in [25,26,27], use the graph Laplacian-based procedures, modified to operate on the ${\mathrm{Sym}}_{++}\left(d\right)$ cone instead of ${\mathbb{R}}^d$, for example, the LPP [28] and NPE [29] dimensionality reduction, i.e., manifold learning techniques. Nevertheless, all mentioned embedding-based GMM similarity measures are based on the shallow ML models, as well as linear projection operators, which transform features from the original high-dimensional parameter space in which SPD representatives lie as elements of the $Sy{m}_{++}\left(d\right)$ cone, to a low-dimensional Euclidean space of the transformed parameters, as used in the calculation formulas of GMM measures. Namely, in contrast to PCA, neighborhood preserving embedding (NPE) performs dimensionality reduction by preserving the local neighborhood structure on the data manifold by constructing an adjacency graph, computing the neighborhood graph weights by solving an optimization problem, and finally determining one “optimal” projection that best fits the manifold structure. In comparison to locality preserving projections (LPP), the objectives are different. On the other hand, in [25], graph weights in the applied LPP procedure are calculated using the KL information distance between the positive definite matrices, i.e., by using Lovric’s positive definite embeddings of Gaussian components from all GMMs in a particular ML task. Thus, it is an extension of LPP to the manifold of ${\mathrm{Sym}}_{++}\left(d\right)$, utilized to compute the low-dimensional Euclidean representatives (vectors). Similarly, in [26], the NPE-like procedure, again involving Lovric’s positive definite embeddings, results in obtaining the MAXDET optimization problem [30] that computes the neighborhood graph weights. Finally, the dimensionality reduction projection operator proposed in [26] is obtained in the same manner as in the LPP case. In both cases [25,26], the low-dimensional embeddings are then used to construct the proposed similarity measure between GMMs.

In this paper, a novel GMM-DeepAutoenc similarity measure between GMMs is proposed. It is based on similarities between the lower-dimensional representations (or representatives) of GMM components, obtained by utilizing deep autoencoder architectures, i.e., deep rather than shallow ML encoding approaches. Since nonlinear deep neural networks (DNNs) are used in such autoencoding embedding procedures, these allow representations of the original SPD matrices of GMM components in the low-dimensional and nonlinear submanifolds embedded in a $Sy{m}_{++}\left(d\right)$ cone. Actually, we construct the low-dimensional Euclidean embeddings by the use of nonlinear autoencoder networks, which enable us to better incorporate the nonlinear structure of the manifold (even non-regular) into the feature vectors. In comparison to the previously discussed LPP, NPE, and DPLM-based GMM similarity measures reported in [25,26,27], the low-dimensional representatives used in this paper are obtained by a nonlinear projective procedure performed by the designed autoencoder, which can be broadly interpreted as the mapping to the tangent space of each particular element or sample, instead of having one common projection hyperplane for dimensionality reduction. Thus, at the roughly similar computational complexity, the method proposed in this paper can be expected to achieve higher effectiveness, i.e., a better trade-off between computational complexity and recognition accuracy.

Two different approaches regarding deep autoencoder architectures have been explored and utilized to obtain the low-dimensional representatives corresponding to the individual Gaussian components of GMMs.

The first approach uses a classical deep autoencoder architecture, utilizing the Euclidean norm between the input and output vectors as the model training cost function. Initially, vectorized upper-diagonal SPD matrices corresponding to individual Gaussian components in GMMs are obtained using the embedding procedure reported in [31] (as can also be seen in [19,23]), and fed as the training inputs to the autoencoder. Afterwards, during the encoder deployment, the low-dimensional Euclidean vectors obtained from the middle layer of the autoencoder are exploited to calculate the ground distances in the proposed similarity measure between GMMs in some particular machine learning tasks.

The second approach uses a deep CNN autoencoder (as can be seen in [32,33,34]), to which the same type of embedded SPD representatives of GMM Gaussian components are applied. However, the Frobenious matrix norm between the two-dimensional matrix input and autoencoder matrix output is employed as the cost function used in the learning procedure. Moreover, as a regularizer in the actual training, various pieces of information, as well as Riemannian manifold-based distances between the SPD matrices belonging to ${\mathrm{Sym}}_{++}(d+1)$ and corresponding to the statistics of the computed autoencoder feature maps, are also used, where $d\in \mathbb{R}$ is the dimension of the underlying Euclidean feature space (which is space of the original GMM components). Thus, the low-dimensional Euclidean representatives of original GMM components can be obtained, where the autoencoder embedding procedure is learned by the additional enforcing of similarities among the PDFs, i.e., the statistics of the feature space observations (computed autoencoder feature maps) corresponding to the encoder and the decoder network, respectively. Since the obtained embedded representatives of the original GMM components are low-dimensional Euclidean vectors, the computational complexity of calculating distances between them is of the order of $O\left(d\right)$, while the computational complexity of various measures utilizing ground distances between the original SPD matrices (representatives denoting individual GMM components) is of the order of $O\left(d^3\right)$. As a result, in the machine learning tasks utilizing the proposed autoencoder-based GMM similarity measure (GMM-DeepAutoenc), a significantly better trade-off between the achieved recognition accuracy and the computational complexity can be obtained, when compared with the classical similarity measure methods. This was experimentally confirmed on various texture recognition tasks, as well as image matching datasets. In case of ML tasks that are characterized by the presence of GMMs with a large overall number of Gaussian components, such an approach would be much more efficient when compared with the existing ones. Moreover, when compared with other methods utilizing the low-dimensional representations of Gaussian components, i.e., when compared to shallow ML based embedding frameworks (see, e.g., [25,26,27]), the proposed deep autoencoder-based measure (GMM-DeepAutoenc) obtains better results in the mentioned tasks and for various datasets, especially in cases with a large number of categories.

2. Materials and Methods

2.1. Baseline GMM Similarity Measures

KL divergence, defined as $KL\left(p\right|\left|q\right)={\int }_{{\mathbb{R}}^d}p\left(x\right)ln\frac{p\left(x\right)}{q\left(x\right)}dx$, is the most natural measure between two probability distributions p and q. Unfortunately, it can be expressed in a closed (analytical) form only for a limited number of distributions, and GMM is not one of these. Let us denote two GMMs as $f={\sum }_{i=1}^n{\alpha }_i{f}_i$ and $g={\sum }_{j=1}^m{\beta }_j{g}_j$, with $f_i=\mathcal{N}({\sum }_i,{\mu }_i)$ and $g_j=\mathcal{N}({\sum }_j,{\mu }_j)$ representing the Gaussian components of the corresponding mixtures. Where ${\alpha }_i>0$, ${\beta }_j>0$ are corresponding weights, satisfying ${\sum }_i{\alpha }_i=1$, ${\sum }_j{\beta }_j=1$, while ${\mu }_i,{\mu }_j$ are the mean vectors of $f_i,g_j$, and ${\sum }_i,{\sum }_j$ are their full covariance matrices. There are various proposed measures between GMMs, almost all of which are based on some specific approximation of the KL divergence. The straightforward and most computationally expensive measure (especially in real-world applications where there are high feature space dimensionalities) is based on using the standard Monte Carlo method (see [35]). The idea is to sample the probability distribution f by using i.i.d. samples $x_i$, $i=1,\dots ,N$, such that $E_f\left[ln\frac{f\left(x\right)}{g\left(x\right)}\right]=KL\left(f\right|\left|g\right)$. This is given by:

$$K{L}_{MC}\left(f\right|\left|g\right)\approx \frac{1}{N}\sum _{i=1}^{N}ln\frac{f\left(x_i\right)}{g\left(x_i\right)}.$$

(1)

Many of the proposed measures are based on approximations utilizing KL divergence between two Gaussian components $KL\left(f_i\right|\left|g_j\right)$, which exist in the closed form given by

$$\begin{array}{ccc}\hfill & & KL(f_i\left|\right|g_j)=ln\frac{|{\sum }_{f_i}|}{|{\sum }_{g_j}|}+Tr\left[{\sum }_{g_j}^{-1}{\sum }_{f_i}\right]+{({\mu }_{f_i}-{\mu }_{g_j})}^T{\sum }_{g_j}^{-1}({\mu }_{f_i}-{\mu }_{g_j})-d\hfill \\ \hfill \end{array}$$

(2)

The roughest approximation (by an upper bound) is based on the convexity of the KL divergence [36]. Thus, for two GMMs $f={\sum }_{i=1}^n{\alpha }_i{f}_i$ and $g={\sum }_{j=1}^m{\beta }_j{g}_j$, it holds that the weighted average approximation is given by:

$$KL\left(f\right|\left|g\right)\le K{L}_{WA}\left(f\right|\left|g\right)=\sum _{i,j}{\alpha }_i{\beta }_{j}KL(f_i\left|\right|g_j),$$

(3)

where $f_i=\mathcal{N}({\sum }_i,{\mu }_i)$ and $g_j=\mathcal{N}({\sum }_j,{\mu }_j)$ are the Gaussian components of the corresponding mixtures, while ${\alpha }_i>0$, ${\beta }_j>0$ are the corresponding weights, satisfying ${\sum }_i{\alpha }_i=1$, ${\sum }_j{\beta }_j=1$. We assume that the KL divergences $KL\left(f_i\right|\left|g_j\right)$ between the corresponding Gaussian components exist in the closed-form, e.g., given by (2).

Matching-based (MB) approximation, proposed in [2] as:

$$\begin{array}{ccc}\hfill & & K{L}_{MB}\left(f\right|\left|g\right)\approx \sum _i{\alpha }_i\left[\underset{j}{min}KL(f_i\left|\right|g_j)+log\left(\frac{{\alpha }_i}{{\beta }_j}\right)\right]\hfill \\ \hfill \end{array}$$

(4)

was based on the assumption that the element $g_j$ in g that is the most proximal to $f_i$ dominates the integral $\int f_{i}logg$ in $KL\left(f\right|\left|g\right)$. Motivated by (4), the more efficient matching-based approximation can be obtained by

$$K{L}_{MBS}\left(f\right|\left|g\right)\approx \sum _i{\alpha }_i\underset{j}{min}KL(f_i\left|\right|g_j),$$

(5)

which shows good performance in the case when components in f and g are far apart, but is inappropriate when there is significant overlap between them.

In [37], the unscented transform-based approximation was proposed in order to deal with the described overlapping situations. This is a method for calculating the statistics of a random variable that undergoes a nonlinear transformation. As it holds $KL\left(f\right|\left|g\right)={\int }_{{\mathbb{R}}^d}flnf-{\int }_{{\mathbb{R}}^d}flng$, the unscented transform approximates ${\int }_{{\mathbb{R}}^d}{f}_{i}lng$ as:

$$\begin{array}{ccc}\hfill & & {\int }_{{\mathbb{R}}^d}{f}_{i}lng\approx \frac{1}{2d}\sum _{k=1}^{2d}lng\left(x_{i,k}\right)\hfill \\ \hfill & & x_{i,k}={\mu }_i+{\left(\sqrt{{\sum }_i}\right)}_k,k=1,\dots ,d\hfill \\ \hfill & & x_{i,d+k}={\mu }_i-{\left(\sqrt{{\sum }_i}\right)}_k,k=1,\dots ,d,\hfill \end{array}$$

(6)

where ${\left(\sqrt{{\sum }_i}\right)}_k$ is the k-th column of the matrix square root of ${\sum }_i$. In the case of ${\int }_{{\mathbb{R}}^d}flng$, we obtain:

$$\begin{array}{ccc}\hfill & & {\int }_{{\mathbb{R}}^d}flng\approx \frac{1}{2d}\sum _{i=1}^n{\alpha }_i\sum _{k=1}^{2d}lng\left(x_{i,k}\right),\hfill \end{array}$$

thus acquiring $K{L}_{UC}\left(f\right|\left|g\right)$. In [35], the approximation based on variational bound is proposed as:

$$\begin{array}{ccc}\hfill & & K{L}_{VB}\left(f\right|\left|g\right)=\sum _i{\alpha }_i\frac{{\sum }_{\widehat{i}}{\alpha }_{\widehat{i}}{e}^{-KL\left(f_i\right|\left|f_{\widehat{i}}\right)}}{{\sum }_j{\beta }_j{e}^{-KL\left(f_i\right|\left|g_j\right)}}.\hfill \\ \hfill \end{array}$$

(7)

In order to obtain an approximate KL divergence between two GMMs, all these approximations utilize KL divergences between Gaussian components that are given in the closed-form expression. On the other hand, Earth mover’s distance (EMD)-based GMM distances stem from the Earth mover’s methodology, which emerges in various recognition tasks, such as those reported in [3,24,38]. In [19], textures are classified using EMD to measure the distributional similarity of sets of Gaussian components that represent the categories, while in [23], the ground distances between Gaussian components given by (2) are incorporated within a sparse EMD-based SR-EMD procedure [23]. The authors utilized various ground distances based on the Riemannian geometry, as well as in combination with the Gaussian component embedding reported in [31]. The best performance among these methods was achieved by the Riemannian metric defined on the Riemannian manifold representing the product of Lie groups ${\mathbb{R}}^d$ and ${\mathrm{Sym}}_{++}\left(d\right)$. According to [23], such a ground distance measure between the Gaussian component $f_i$ and $g_j$ can be motivated by the product of the mentioned Lie groups and defined as:

$$\begin{array}{ccc}\hfill & & d_{f_i,g_j}\left(\xi \right)=(1-\xi )a_{f_i,g_j}+\xi b_{f_i,g_j},\hfill \end{array}$$

(8)

where the individual components of measure $d_{f_i,g_j}\left(\xi \right)$ are defined by the following expressions:

$$\begin{array}{ccc}\hfill & & a_{f_i,g_j}={\left({({\mu }_{f_i}-{\mu }_{g_j})}^T({\sum }_{f_i}^{-1}+{\sum }_{g_j}^{-1})({\mu }_{f_i}-{\mu }_{g_j})\right)}^{\frac{1}{2}},\hfill \\ & & b_{f_i,g_j}(\Omega )={\parallel \Omega \left(ln\left({\sum }_{f_i}\right)-ln\left({\sum }_{g_j}\right)\right)\parallel }_F.\hfill \end{array}$$

We note that the matrix $\Omega$ denotes the result of the decomposition of the matrix M that represents the corresponding matrix Mahalanobis norm, i.e., $M={\Omega }^T\Omega$ and:

$$\begin{array}{ccc}\hfill & & b_{f_i,g_j}(\Omega )=tr\left({\left(ln\left({\sum }_{f_i}\right)-ln\left({\sum }_{g_j}\right)\right)}^{T}M\left(ln\left({\sum }_{f_i}\right)-ln\left({\sum }_{g_j}\right)\right)\right).\hfill \end{array}$$

(9)

Thus, when $d_{f_i,g_j}\left(\xi \right)$ is incorporated into a sparse representation EMD procedure (SR-EMD), one obtains the similarity measure between GMMs as the minimum value of the cost function of the optimization problem given by:

$$\begin{array}{ccc}\hfill & & \underset{M,x_{pq}}{min}{\sum }_{p=1}^{m_f}{\sum }_{q=p+1}^{m_g}{\alpha }_{pq}\left(\parallel y_{pq}-A_{pq}{C}_{pq}^{-1}\left(M\right)x_{pq}{\parallel }_{l_2}^2+\rho {\parallel x_{pq}\parallel }_{l_1}\right).\hfill \\ & & s.t.M\succ 0,\forall x_{pq}⪰0\hfill \end{array}$$

(10)

The described SR-EMD problem involves the joint optimization of the ground distance matrix $C_{pq}\left(M\right)$ and vector $x_{pq}$, which formulate an alternating dictionary learning and sparse representation procedure. Thus, the optimal transport vector $x_{pq}$ is found under the constraints vector $y_{pq}=\left[{\alpha }_1,\dots ,{\alpha }_{m_f},{\beta }_1,\dots ,{\beta }_{m_g}\right]$, defined by the weights of all $(m_f+m_g)$ GMM components, and some fixed $\rho >0$. We note that the term $A_{pq}$ represents the $(m_f+n_g)\times m_f{m}_g$ matrix with 0 and 1 entries. In (8)–(10), terms ${\parallel ·\parallel }_{l_1}$, ${\parallel ·\parallel }_{l_2}$, ${\parallel ·\parallel }_F$, represent $l_1$, $l_2$ vector norms, and matrix Frobenius norm F, respectively.

Recently, approaches for GMM similarity measurements based on dimensionality reduction have been proposed, as can be seen in [25,26], where each embedded SPD representative corresponding to some Gaussian component of GMMs (obtained by using [31]) is replaced by its low-dimensional Euclidian representation (projection). These exploit the shallow ML approach based on graph manifold learning (LPP in [25] or NPE in [26]), which is used in order to obtain the linear projectors, but with kernels containing symmetrized KL divergence between SPD representatives, instead of the Euclidian norm. Similarly, in [27], the dimensionality reduction preceding GMMs’ similarity measurements was also applied using the shallow ML approach proposed in [39] in order to obtain the low-dimensional SPD representatives of the Gaussian components. Some standard KL divergence-based methods (for example [1,2,7]) were then used in order to obtain the similarity measures between GMMs, but now with low-dimensional SPDs. We refer to this type of method as DPLM, inspired by the title of [39], where the corresponding dimensionality reduction based on the distance preservation to the local mean (DPLM) was proposed. Finally, the GMM similarity measure in all [25,26,27] is defined using a fuzzy aggregation of Euclidian distances between low-dimensional representatives, or using an EMD-based distance metric (see [38]) between the sets of the corresponding low-dimensional representatives. Concerning the DPLM-based GMM similarity measures that we use as the baselines in Section 3 with experimental results, and also discuss in Section 2.3 regarding computational complexity, we will denote these by $uDPL{M}_{MB}$, $uDLP{M}_{WA}$, and $uDLP{M}_{VB}$ throughout the paper. The prefix u indicates that these are unsupervised approaches, as can be seen in [27], while the suffixes $WA$, $MB$, and $VB$ denote that the similarity between newly obtained GMMs with low-dimensional SPD matrices (Lovric embeddings) is measured using the standard KL divergence approximation measures given by (3), (4), and (7), respectively. From the results comparing CPU times and recognition accuracies in [27], it can be seen that, by the dimensionality reduction applied in [27], a significantly better trade-off between the recognition accuracy and the computational complexity is achieved when compared with the baseline approaches [1,2,7].

2.2. GMM Similarity Measures Based on Autoencoder-Generated Representations

GMM similarity measures based on the dimensionality reduction, as reported in [25,26,27], are all shallow ML methods utilizing linear approximations, i.e., projectors in order to approximate the underlying sub-manifold of manifold ${\mathrm{Sym}}_{++}(d+1)$ on which the parameters of the SPD representatives (Gaussian components) lie, which is an assumption of the underlying approximation problem. Moreover, all these methods, in their own way, learn linear projectors, that fit the sub-manifold defined by the SPD representatives of the Gaussian components of GMMs in some particular ML problem (i.e., the SPD representatives obtained using the Lovric’s embedding proposed in [31]). However, in this work, we propose a novel approach whereby, using the deep autoencoder network architecture, we learn to encode the mentioned SPD representatives of Gaussian components into the middle layer of the autoencoder, and thus obtain the embedding of SPD matrices into the low-dimensional Euclidean space. If provided with a sufficient amount of learning data, the designed nonlinear embedding aims to obtain an even better trade-off between recognition accuracy and computational complexity. As a result, by using a deep autoencoder network architecture, we can model nonlinear, and even non-regular sub-manifold structures in ${\mathrm{Sym}}_{++}(d+1)$.

2.2.1. Autoencoder Architectures

In this work, we use deep autoencoder network architectures to construct the novel GMM similarity measure that is computationally efficient, and at the same time, retains recognition accuracy. The idea is that, by using a deep autoencoder, we can more efficiently “capture” and characterize the nonlinear sub-manifold that contains the parameters of GMMs used in a particular ML application. Thus, the nonlinear dimensionality reduction performed by the encoding process of the autoencoder network that implicitly performs the mentioned sub-manifold characterization. Such a learning process and consequent dimensionality reduction lead to achieving a more computationally efficient GMM similarity measurement at the same or better recognition accuracy, i.e., a better trade-off between these requirements if a sufficient amount of training data are available.

The autoencoder network comprises nonlinear mappings for the encoder and the decoder, learned on the set of input SPD matrices or their vectorized counterparts. After the autoencoder training, we calculate the proposed GMM similarity measure between two GMMs by aggregating the Euclidean distances between the obtained low-dimensional representations of GMM components, i.e., the low-dimensional representatives of input SPD matrices that correspond to Gaussian components. Described aggregation is performed by various fuzzy aggregation measures, as in [25,26,27], or by using the EMD optimal transport problem described in [24]. We note that in the presented experiments, Gaussian components are used to characterize both training and testing samples, which are instances of the categories in the actual ML task.

We use two different types of deep autoencoders. The first one is the classical fully connected, regularized deep autoencoder, Figure 1.

This usually comprises two sections—an encoder that maps the input data into a lower-dimensional representation, and a subsequent decoder that maps the output of the encoder to the same space as the input data. Furthermore, it is usually built to have symmetrical encoder and decoder layers. Namely, if we denote the encoder and the decoder networks by F and G, respectively, where $n=(d+1)(d+2)/2$ is the size of the input layer, and we supply the vectorized upper triangular $vect\left(P\right)$ of SPD representative $P\in {\mathrm{Sym}}_{++}(d+1)$ obtained by the Lovric embedding [31] of Gaussian component $g=\mathcal{N}(x;\mu ,\sum )\in {\left\{f_i\right\}}_1^{n_f}$ of some GMM $f={\sum }_{i=1}^{n_f}{\alpha }_i{f}_i$ in ${\mathbb{R}}^d$, which is given by:

$$g\hookrightarrow P={|\sum |}^{-\frac{1}{d+1}}\left[\begin{array}{cc}\sum +\mu {\mu }^T& \mu \\ {\mu }^T& 1\end{array}\right],$$

(11)

the autoencoder training procedure can be described by the following objective:

$$\begin{array}{ccc}\hfill & & \widehat{F},\widehat{G}={\mathrm{argmin}}_{F,G}{\mathbb{E}}_{P\sim p\left(P\right)}\left[{\parallel (G\circ F)\left(vect\left(P\right)\right)-vect\left(P\right)\parallel }_{l_2}\right]+\lambda \mathcal{R}(F,G),\hfill \end{array}$$

(12)

where $\mathbb{E}[·]$ stands for the mathematical expectation, ∘ for the composition of the functions, ${\parallel ·\parallel }_{l_2}$ denotes the Euclidean $l_2$ norm, while $P\sim p\left(P\right)$ denotes that the SPD representatives P given by the Lovric embedding (11), with PDF given by p, and $\mathcal{R}(F,G)$ is a regularization term used for parameters of F and G. We recall that, for any matrix $B\in {\mathbb{R}}^{m\times n}$ and its vectorized version $vect\left(B\right)\in {\mathbb{R}}^{mn}$, obtained by the serialization of the matrix B along the vertical or horizontal dimensions, it holds that ${\parallel B\parallel }_F={\sum }_{i=1}^m{\sum }_{j=1}^n{b}_{ij}^2={\parallel vect\left(B\right)\parallel }_{l_2}$. The term $\lambda >0$ is a regularization parameter. Thus, we estimate the loss function in (12) as:

$${\widehat{\mathbb{E}}}_{P\sim p\left(P\right)}\left[{\parallel (G\circ F)\left(vect\left(P\right)\right)-P\parallel }_F\right]=\frac{1}{M}\sum _{i=1}^M\left[\parallel (G\circ F)\left(vect\left(P_i\right)\right)-vect\left(P_i\right){\parallel }_{l_2}\right],$$

(13)

where M stands for the overall number of SPD matrices that are constructed as Lovric-embedding matrices $P_i,i=1,\dots ,M$, i.e., the SPD representatives of the Gaussian components of all GMMs that are available for the training of the autoencoder model (e.g., per each data training instance, there is one GMM with several GMM components). For $\mathcal{R}(F,G)$, we use the $l_2$ regularizer defined by:

$$\mathcal{R}(F,G)=\sum _{l=1}^L\sum _{i=1}^{n_l}\sum _{j=1}^{k_l}{\left(w_{ij}^{\left(l\right)}\right)}^2,$$

(14)

where L denotes the overall number of hidden layers (including F and G), $n_l$ and $k_l$ are the output and the input size of layer l, respectively, while $w_{ij}^{\left(l\right)}$ denote all weights of the autoencoder network, corresponding to the l-th layer. Term $\lambda >0$ is considered as fixed in the optimization problem (12).

For the second type of deep autoencoder architecture, we use a deep CNN autoencoder, where we observe the model inputs, i.e., the entries of some particular Lovric embedding representative $P_i$ (2D SPD matrix), as “pixels” of some 2D image (as usual in vision-based CNNs). This embedding framework consists of the encoder and the decoder CNN networks F and G, respectively, as well as the middle layer $h\in {\mathbb{R}}^l$, as shown in Figure 2.

The training procedure can be given by the same optimization problem (12) where we use the Frobenious matrix norm ${\parallel ·\parallel }_F$ instead of the Euclidean ${\parallel ·\parallel }_{l_2}$ in the objective, as we deal with matrices at the input (as well as at the output) of the CNN autoencoder. Nevertheless, it still requires the strong regularization provided by an additional cost term that forces the similarity between the corresponding feature map tensors in the encoder and the decoder, i.e., the outputs of the convolutional layers corresponding to the encoder and the decoder (between the first feature map of the encoder and the last feature map of the decoder, etc.). Namely, we add the feature map regularization (FMR) penalty term ${\mathcal{R}}_{FMR}$ in the form of a similarity measure between the PDFs that model statistics of the computed feature maps in F and G. Thus, we obtain the CNN autoencoder learning problem as follows:

$$\begin{array}{ccc}\hfill \hat{F},\hat{G}& =& {\mathrm{argmin}}_{F,G}{\mathbb{E}}_{P\sim p\left(P\right)}\left[{\parallel (G\circ F)\left(P\right)-P\parallel }_F\right]+\lambda \mathcal{R}(F,G),\hfill \\ & +& {\lambda }_{FMR}{\mathcal{R}}_{FMR}(F,G)\hfill \end{array}$$

(15)

with some fixed ${\lambda }_{FMR}>0$, where ${\parallel ·\parallel }_F$ is the Frobenius norm defined for $A=\left[a_{ij}\right]\in {\mathbb{R}}^{m\times n}$ as ${\parallel A\parallel }_F={\sum }_{i=1}^m{\sum }_{j=1}^n{a}_{ij}^2$.

For measuring the feature map PDFs, i.e., formulating the ${\mathcal{R}}_{FMR}$, we consider some of the information-based, as well as the Riemannian manifold-based distances reported in [19,23]. For the low-dimensional Euclidean representation of SPD instances $P_i\in {\mathrm{Sym}}_{++}(d+1)$ obtained by the Lovric embedding of Gaussian components, we take the vectors obtained from the middle layer of the deep autoencoder networks, i.e., bottleneck $h\in {\mathbb{R}}^l$, as described herein. These nonlinear projection quantities are then used to formulate the novel GMM-Autoenc similarity measures that we invoke.

2.2.2. Ground Distances Regularization

We present ground distances between multidimensional (k-dimensional) Gaussians, that we use as similarity measures in order to regularize the training of an autoencoder network, i.e., to form ${\mathcal{R}}_{FMR}(F,G)$. Let us consider k-dimensional $f=\mathcal{N}(x;{\mu }_1,{\sum }_1)$ and $g=\mathcal{N}(x,{\mu }_2,{\sum }_2)$, which are elements of ${\mathrm{Sym}}_{++}\left(k\right)$, or $k+1$-dimensional SPD Lovric embedding representatives given by (11) as elements of ${\mathrm{Sym}}_{++}(k+1)$, which are both Riemannian manifolds (see [19,23]), equipped with different Riemannian metrics, and inducing geodesic distances. Out of such ground distances that do not take into account the information geometric properties of multidimensional Gaussians, we consider the $l_2$-based distance:

$$d_{l_2}(f,g)=\parallel {\sum }_1-{\sum }_2{\parallel }_F+\eta {\parallel {\mu }_1-{\mu }_2\parallel }_{l_2},$$

(16)

as well as the robust $l_1$-based distance given by:

$$d_{l_1}(f,g)=\parallel {\sum }_1-{\sum }_2{\parallel }_{l_1}+\eta {\parallel {\mu }_1-{\mu }_2\parallel }_{l_1},$$

(17)

where ${\parallel ·\parallel }_{l_1}$ is the robust $l_1$ norm in ${\mathbb{R}}^k$. We note that the distance functions involving the ${\parallel ·\parallel }_{l_2}$ and ${\parallel ·\parallel }_F$ norm in (16) emerge from the assumption that the prior function, i.e., PDF, in the formulation of the regularization problem, is of the Gaussian type. These can be considered as more appropriate ones when there is a large amount of training data, while the ones based on the $l_1$ norm in (17), which correspond to the prior distribution of Laplace type, are usually used if we tend to obtain a more robust estimation.

Considering that there is a closed-form expression for the KL divergence between two arbitrary Gaussians f and g, denoted by $d_{KL}(f,g)=KL\left(f\right|\left|g\right)$, with $KL\left(f\right|\left|g\right)$ given by (2) (where the symbols are replaced by: $d\mapsto k$, $f_i\mapsto f$, $g_j\mapsto g$), for the information-based ground distance, we use the symmetrized version of the KL divergence given by:

$$d_{K{L}_{sym}}(f,g)=\frac{1}{2}(d_{KL}(f,g)+d_{KL}(g,f)).$$

(18)

Although the KL divergence is not symmetric, we note that it does not mean that it is antisymmetric. Namely, we give the following counterexample: for two discrete Bernoulli distributions ${\mathcal{B}}_{\alpha }\left(x\right)$ and ${\mathcal{B}}_{\beta }\left(x\right)$, parameterized by $\alpha =p$ and $\beta =1-p$, respectively, it holds that $KL({\mathcal{B}}_{\alpha }\left|\right|{\mathcal{B}}_{\beta })=KL({\mathcal{B}}_{\beta }\left|\right|{\mathcal{B}}_{\alpha }).$

For the geometrical ground distance, we use Log-Euclidean metric as geodesic distance (see [40]) on ${\mathrm{Sym}}_{++}(k+1)$, forming the efficient geodesic distance invariant to similarity transformation, given by:

$$d_{le,nonsym}(P_f,P_g)={\parallel ln\left(P_f\right)-ln\left(P_g\right)\parallel }_F,$$

for Lovric embedding SPD representatives $P_f,P_g\in {\mathrm{Sym}}_{++}(k+1)$ of f and g, respectively. Actually, we use the symmetrized version:

$$d_{le}(P_f,P_g)=\frac{1}{2}\left(d_{le,nonsym}(P_f,P_g)+d_{le,nonsym}(P_g,P_f)\right),$$

(19)

2.2.3. Forming the Feature Map Regularizer ${\mathcal{R}}_{FMR}(F,G)$

Let us denote the first feature map tensor of the encoder and the last feature map tensor of the decoder of the CNN autoencoder network, obtained for the input matrix $x\in {\mathbb{R}}^{(d+1)\times (d+1)}$, as $F^{\left(enc\right),x}\in {\mathbb{R}}^{n\times m\times r}$, and $F^{\left(dec\right),x}\in {\mathbb{R}}^{n\times m\times r}$, respectively, (both tensors are of the convolution layers type, and are of identical format, size $m\times n$ and depth r, since the encoder and the decoder are assumed to be symmetric). In a general case, for the r-dimensional feature map or layer indexed by q, we will reshape or reformat the $F^{\left(q\right),x}$ 3D tensor into the following 2D matrix: ${\tilde{F}}^{\left(q\right),x}=\left[f_{1,1}^{\left(q\right),x}\left|f_{1,2}^{\left(q\right),x}\right|\dots f_{m,n}^{\left(q\right),x}\right]$, $q\in \{enc,dec\}$, where $f_{i,j}^{\left(q\right),x}\in {\mathbb{R}}^r$, $i\in \{1,\dots ,n\}$, $j\in \{1,\dots ,m\}$ are r-dimensional observations (or values) of the autoencoder feature maps in layer q (generated by the corresponding statistics of the autoencoder layer q with depth r).

Next, under the assumption that the underlying PDFs are r-dimensional multivariate Gaussians, i.e., that the unknown PDFs rendering the r-dimensional observations (values in the described feature map tensors) are normally distributed, $f_{i,j}^{\left(q\right),x}\sim \mathcal{N}({\sum }^{\left(q\right),x},{\mu }^{\left(q\right),x})$, we obtain the maximum likelihood estimates (MLEs) of their covariances and centroids, i.e., ${{\displaystyle \hat{\sum }}}^{\left(q\right),x}$, ${\widehat{\mu }}^{\left(q\right),x}$, $q\in \{enc,dec\}$ from the columns of matrices ${\tilde{F}}^{\left(q\right),x}$, $q\in \{enc,dec\}$ as:

$$\begin{array}{ccc}\hfill & & {\widehat{\sum }}^{\left(q\right),x}=\frac{1}{mn}\sum _{i=1}^m\sum _{j=1}^n(f_{i,j}^{\left(q\right),x}-{\mu }^{\left(q\right),x}){(f_{i,j}^{\left(q\right),x}-{\mu }^{\left(q\right),x})}^T,\hfill \\ & & {\widehat{\mu }}^{\left(q\right),x}=\frac{1}{mn}\sum _{i=1}^m\sum _{j=1}^n{f}_{i,j}^{\left(q\right),x},\hfill \end{array}$$

(20)

Finally, we form the feature map regularizer ${\mathcal{R}}_{FMR}(F,G)$ in (15) as follows:

$${\mathcal{R}}_{FMR}(F,G)={\mathbb{E}}_{P\sim p\left(P\right)}\left[d_{gd}(f^P,g^P)\right]$$

(21)

where $f^P=\mathcal{N}({\sum }^{\left(enc\right),P},{\mu }^{\left(enc\right),P})$, $g^P=\mathcal{N}({\sum }^{\left(dec\right),P},{\mu }^{\left(dec\right),P})$, and the ground distances $gd\in \{l_2,l_1,K{L}_{sym},le\}$ are all defined in the previous section. $P\in {\mathrm{Sym}}_{++}(r+1)$ stands for the SPD matrices generated by $p\left(P\right)$, i.e., by the distribution p of Lovric embedding SPD matrices representing the feature map Gaussians described by statistics in (20). Note that each of these Gaussians P, i.e., $(r+1)\times (r+1)$ SPD matrices made of ${{\displaystyle \hat{\sum }}}^{\left(q\right),x}$ and mean ${\widehat{\mu }}^{\left(q\right),x}$ in (20), correspond to one of the original Gaussian components brought to the input of the autoencoder, i.e., one input x. In other words, estimates in (20) correspond to the autoencoder feature map observations in ${\mathbb{R}}^r$, while the input Gaussian components are in ${\mathbb{R}}^d$. Since it is assumed that there are M such $(d+1)\times (d+1)$ SPD matrices corresponding to the individual components of all GMMs present in the training phase of some particular ML task, we estimate the expectation in (21) as:

$${\mathbb{E}}_{P\sim p\left(P\right)}\left[d_{gd}(f^P,g^P)\right]=\frac{1}{M}\sum _{i=1}^M{d}_{gd}(f^{P_i},g^{P_i}),$$

(22)

Thus, M is the overall number of Gaussians of GMMs used for the autoencoder training, and each of the M input Gaussians results in feature map tensors $F^{\left(q\right),x}$, $q\in \{enc,dec\}$, i.e., their statistics in (20) are represent by the Lovric embedding $P\in {\mathrm{Sym}}_{++}(r+1)$ in (21).

2.2.4. Forming the GMM Similarity Measure Based on Autoencoder-Generated Representations

We form the proposed novel GMM-Autoenc similarity measures by following the approach similar to that described in [26]. Namely, the similarity measure between two particular GMMs is formed by aggregating the non-negative real values that represent the measure of similarity between corresponding components, i.e., their nonlinear embeddings. Actually, we have to compare the transformed “clouds” of lower l-dimensional Euclidean parameter vectors, with $l\ll n$, $n=(d+1)(d+2)/2$, which are computed from the original Gaussian components of two GMMs. In all approaches that we utilize, for the particular $m_f$-component GMM $f={\sum }_{i=1}^{m_f}{\alpha }_i{f}_i$ with $f_i=\mathcal{N}({\mu }_i,{\sum }_i)$, we represent GMM by the unique tupple $F=(v_1,\dots ,v_{m_f},{\alpha }_1,\dots ,{\alpha }_{m_f})$, which consists of the embedding vectors $v_i$ corresponding to the low-dimensional outputs from the middle (bottleneck) layer of the trained embedding autoencoder described in Section 2.2.1. In this process, the belonging mixture weights ${\alpha }_i$ are also taken into account. We note that the autoencoder inputs are the Lovric embeddings of $f_i$, defined by (11), i.e., SPD matrices $P_i$ or their vectorized counterparts.

Thus, the similarity measure between two GMMs, denoted by $f={\sum }_{i=1}^{m_f}{\alpha }_i{f}_i$ and $g={\sum }_{j=1}^{m_g}{\beta }_j{g}_j$, is introduced as the similarity between the corresponding representatives $F=\left(v_1,\dots ,v_{m_f},{\alpha }_1,\dots ,{\alpha }_{m_f}\right)$ and $G=\left(u_1,\dots ,u_{m_g},{\beta }_1,\dots ,{\beta }_{m_g}\right)$, respectively. It is measured in the autoencoder embedding space, as represented by the weighted low-dimensional Euclidean vectors $v_i,u_i\in {\mathbb{R}}^l$. By doing so, we utilize two different approaches. The first one employs a form of fuzzy union or intersection operation (see, e.g., [41]). Namely, we use the weighted max–min operator as follows:

$$\begin{array}{ccc}\hfill & & p_i=min\{{\beta }_j\parallel v_i-u_j{\parallel }_2|j=1,\dots ,m_f\}\hfill \\ \hfill & & a=max\left\{{\alpha }_i{p}_i\right|i=1,\dots ,m_g\}\hfill \\ \hfill & & q_j=min\{{\alpha }_i\parallel v_i-u_j{\parallel }_2|i=1,\dots ,m_g\}\hfill \\ \hfill & & b=max\left\{{\beta }_j{q}_j\right|j=1,\dots ,m_f\}\hfill \\ \hfill & & D_1(F,G)=\frac{1}{2}(a+b),\hfill \end{array}$$

(23)

as well as the maximum of the positive weighted sums

$$\begin{array}{ccc}\hfill & & a=max\{{\alpha }_i\sum _{j=1}^n{\beta }_j\parallel v_i-u_j{\parallel }_2|i=1,\dots ,m_g\}\hfill \\ & & b=max\{{\beta }_j\sum _{i=1}^m{\alpha }_i\parallel v_i-u_j{\parallel }_2|j=1,\dots ,m_f\}\hfill \\ & & D_2(F,G)=\frac{1}{2}(a+b).\hfill \\ \hfill \end{array}$$

(24)

In the following, the invoked GMM measures induced by $D_1$ will be denoted by GMM-Autoenc${}_{1,AE}$, for the classical autoencoder architecture and GMM-Autoenc${}_{1,AECNN}$ for the CNN autoencoder. Similarly, the GMM measures induced by $D_2$ will be denoted by GMM-Autoenc${}_{2,AE}$ and GMM-Autoenc${}_{2_{A}ECNN}$, respectively. Both types satisfy self-similarity, positivity, and are symmetric; however, on the contrary to KL divergence, self-identity is not satisfied.

The second aggregation approach utilizes the EMD distance on the F and G representatives of two GMMs that are mutually compared. It is based on the work proposed in [24] and the EMD transportation problem. Namely, the computed tuples F and G are interpreted as the sets of low-dimensional Euclidean vectors equipped with the weights or masses ${\alpha }_i,i=1\dots m_f$, and ${\beta }_j,j=1\dots m_g$, respectively. Thus, we introduce the measure of similarity between two GMMs f and g to be the measure between weighted “clouds” F and G consisting of the Euclidean vectors in ${\mathbb{R}}^l$ (e.g., see [24]). Taking into account the distance or cost between all pairs of ${\mathbb{R}}^l$ vectors from F and G, i.e., by computing the matrix $\left[d_{ij}\right]$, it follows that the optimal flow $\left[{\zeta }_{ij}\right]$ (with the lowest overall cost) when moving ${\mathbb{R}}^l$ vectors from F to G is obtained as the solution to the following LP type minimization problem:

$$\begin{array}{ccc}& & \underset{{\zeta }_{ij}}{min}\sum _{i=1}^{m_f}\sum _{j=1}^{m_g}{d}_{ij}{\zeta }_{ij},\hfill \\ & & s.t.\hfill \\ & & {\zeta }_{ij}\ge 0,i=1,\dots ,m_f,j=1,\dots ,m_g,\hfill \\ & & \sum _{j=1}^{m_g}{\zeta }_{ij}\le {\alpha }_i,i=1,\dots ,m_f,\hfill \\ & & \sum _{i=1}^{m_f}{\zeta }_{ij}\le {\beta }_j,j=1,\dots ,m_g,\hfill \\ & & \sum _{i=1}^{m_f}\sum _{j=1}^{m_g}{\zeta }_{ij}=1.\hfill \end{array}$$

(25)

Based on the optimal flow $\left[{\zeta }_{ij}\right]$, it follows that the aggregate EMD distance between the sets of the Euclidean vectors in F and G can be computed as:

$$\begin{array}{ccc}\hfill & & D_{EMD}(F,G)=\frac{{\sum }_{i=1}^{m_f}{\sum }_{j=1}^{m_g}{d}_{ij}{\zeta }_{ij}}{{\sum }_{i=1}^{m_f}{\sum }_{j=1}^{m_g}{\zeta }_{ij}}.\hfill \end{array}$$

(26)

Regarding the LP in (25), $\left[d_{ij}\right]$ is the matrix of Euclidean distances between vectors $v_i$ and $u_j$ in ${\mathbb{R}}^l$, i.e., $d_{ij}={\parallel v_i-u_j\parallel }_2$. The first constraint imposed on the solution $\left[{\zeta }_{ij}\right]$ defines the direction of mass or “supplies” flow, i.e., it allows moving supplies from the cloud or set F to the cloud or set G, but not vice versa. We also note that ${\alpha }_i$ as well as ${\beta }_j$ sums up to be one. The second constraint in (25) limits the amount of supplies that can be sent from F to G, by imposing the restriction that the elements from F individually cannot supply more goods in total to the elements of G than the amount corresponding to the value of their weight in F; while the third constraint puts the same type of restrictions on the elements of G (that they cannot receive more supplies in total than the value of their individual weight, i.e., capacity). The fourth constraint in the form of equality serves the purpose to exploit all available flow from the cloud F to the cloud G (i.e., to move the maximum amount of supplies as possible). Once the transportation problem is solved, the optimal values of $\left[{\zeta }_{ij}\right]$ are determined, and the Earth mover’s distance $D_{EMD}(F,G)$ in (26) is defined as the total work or cost (resulting value of the objective function) that is necessary in order to move, by optimal flow $\left[{\zeta }_{ij}\right]$, the maximum amount of supplies possible, from the “cloud” F to the “cloud” G, which is normalized by the total flow (as in the optimal transportation problem).

Moreover, the fact that the EMD distance is a metric (see [24]), implies that the measure of similarity between two GMMs defined by (26) is also a metric. We denote the GMM measures induced by $D_{EMD}$ as GMM-Autoenc${}_{3,AE}$ and GMM-Autoenc${}_{3,AECNN}$, respectively.

2.3. Computational Complexity

In large-scale ML systems relying on GMMs as a way of modeling distributions of interest, the number of high-dimensional Gaussian components is usually large, which can induce a significant computational burden in cases when the similarity between the GMMs is measured. Therefore, it is of interest to have low-computational complexity, and at the same time, preserve the accuracy of performing the ML task, e.g., recognition, which can be achieved by the baseline KL divergence-based GMM similarity measures. Reducing the computational burden is of the utmost importance in providing efficient services based on such GMM-based ML systems.

Let us assume that the full covariance GMMs f and g have the same number of components m. We denote by d the dimension of the original feature space corresponding to some recognition task. The computational complexity of the Monte Carlo approximation $K{L}_{MC}$, defined by relation (1), is estimated as $O\left(Nm{d}^3\right)$, where N is the number of samples. Namely, there are N observations for which two mixtures are calculated, each with m Gaussian components, and the computational complexity for calculating each component is of the order of $O\left(d^3\right)$, so the overall computational complexity is of the order of $O\left(Nm{d}^3\right)$. However, to obtain an efficient KL divergence approximation, the number of i.i.d. samples N must be large, making it inefficient. For the computational complexity of all KL-based measures, $K{L}_{WA}$, $K{L}_{MB}$, and $K{L}_{VB}$, defined in Section 2.1 by relations (3), (4) and (7), respectively, the rationale is as follows. Their complexities are dominated by the term $KL\left(f_i\right|\left|g_j\right)$, and there are $m^2$ such terms, as it is assumed that both mixtures in the pair have m d-variate Gaussian components, so $i,j=1,\dots ,m$. As $KL\left(f_i\right|\left|g_j\right)$ is defined by (2), it can be seen that calculations of these are dominated by the inversion of d-dimensional SPD matrices, which is of the order of $O\left(d^3\right)$. Thus, the overall computation is of the order of $O\left(m^2{d}^3\right)$. We use all these in the experiments as the baseline measures.

Concerning the complexity of similarity measures reported in [27], they are estimated as $O\left(m\tilde{l}{d}^2\right)+O\left(md{\tilde{l}}^2\right)+O\left(m^2{\tilde{l}}^3\right)$ where $\tilde{l}\times \tilde{l}$ is the dimension of low-dimensional SPD representatives. We denote the mentioned similarity measures (unsupervised versions) by $uDPL{M}_{WA}$, $uDPL{M}_{MB}$, and $uDPL{M}_{VB}$ and use those in the experiments as baseline measures.

The measures reported in [25,26] have complexity estimated by $O\left(m{d}^{2}l\right)+O\left(m^{2}l\right)$, where l is the dimensionality of the corresponding low-dimensional embedding. This also holds for GMM measures induced by $D_1$ and $D_2$, as given by (23) and (24), respectively, while $D_{EMD}$, given by (26), induces the GMM similarity measure with the complexity of the order of $O\left(m{d}^{2}l\right)+O\left(m^{2}l\right)+O\left(m^5\right)$ (as can be seen, e.g., in [25,26]).

Since the proposed GMM-Autoenc${}_{i,AE}$, $i=1,2$ has a single layer encoder, its computational complexity is influenced by the described aggregation operators, and can be estimated in the same way, i.e., as $O\left(m{d}^{2}l\right)+O\left(m^{2}l\right)$, where l is the size of the low-dimensional autoencoder embedding (in the case of the architecture with the input layer, hidden layer, and output layer which is used in the experiments). On the other hand, for GMM-Autoenc${}_{i,CNNAE}$, $i=1,2$, if we assume that there are two convolutional feature map layers in the encoder as well as the decoder of the embedding CNN autoencoder, in the case when using $D_1$ and $D_2$ given by (23) and (24), the computational complexity of the CNN-based GMM similarity measure can be estimated as $O\left(2m{w}_1{h}_1{k_1}^2{n}_1\right)+O\left(2m{w}_2{h}_2{k_2}^2{n}_1{n}_2\right)+O\left(2m{w}_3{h}_3{k_3}^2{n}_2{n}_3\right)+O\left(m^{2}l\right)$. Note that $n_i,w_i,h_i,k_i<d$ are the dimensions of the corresponding convolutional layers in the encoder (depth, width, height, and kernel spatial size).

For the proposed EMD-based similarity measure GMM-Autoenc${}_{3,AE}$ induced by (26), the computational complexity is estimated as $O\left(m{d}^{2}l\right)+O\left(m^{2}l\right)+O\left(m^5\right)$. Namely, the term $O\left(m^{2}l\right)$ corresponds to the computational complexity of calculating $\parallel v_i-u_j{\parallel }_{l2}$ between all pairs of l-dimensional representatives of Gaussian components $f_i$ and $g_j$, respectively, $i,j=1,\dots ,m$. The term $O\left(m{d}^{2}l\right)$ comes from the encoding process in the autoencoder network. At the input, we have $d(d+1)/2)$-dimensional vectors that are transformed by matrix multiplication and nonlinearity into l-dimensional output vectors (from the computational complexity perspective nonlinearity can be neglected if considered to be a piecewise affine). The third term, $O\left(m^5\right)$, comes from the computational complexity of the LP problem given by (25), where the number of iterations in the EMD computation is $O\left(2m\right)$, as described in [23]. Regarding the GMM-Autoenc${}_{3,CNNAE}$, which is based on the CNN autoencoder architecture, the computational complexity can be estimated as $O\left(2m{w}_1{h}_1{k_1}^2{n}_1\right)+O\left(2m{w}_2{h}_2{k_2}^2{n}_1{n}_2\right)+O\left(2m{w}_3{h}_3{k_3}^2{n}_2{n}_3\right)+O\left(m^{2}l\right)+O\left(m^5\right)$, which has the similar complexity as the previously described Autoenc${}_{i,CNNAE}$, $i=1,2$, with only one difference, namely that it has an additional term $O\left(m^5\right)$, originating from solving the LP problem in (25).

It can be seen that the computational complexity of the proposed GMM-Autoenc${}_{i,AE}$ for $i=1,2,3$, i.e., when using $D_1$ and $D_2$ and $D_{EMD}$, is similar to the corresponding GMM similarity measures reported in [25,26], which is thus computationally significantly more efficient when compared with $K{L}_{WA}$, $K{L}_{MB}$, and $K{L}_{VB}$. Depending on the relation between $\tilde{l}$ and l, the proposed autoencoder-based measures are also potentially more efficient when compared with unsupervised $uDPL{M}_{WA}$, $uDPL{M}_{MB}$, $uDPL{M}_{VB}$, and the supervised $sDPL{M}_{WA}$, $sDPL{M}_{MB}$, and $sDPL{M}_{VB}$. The computational complexity of the proposed GMM-Autoenc${}_{i,CNNAE}$, $i=1,2,3$ is also significantly more efficient when compared with $K{L}_{WA}$, $K{L}_{MB}$, and $K{L}_{VB}$, but less efficient than GMM-Autoenc${}_{i,AE}$, $i=1,2,3$.

3. Experimental Results

In this section, the network architecture used in the experiments, as well as the experimental results of applications of the proposed novel GMM-Autoenc similarity measures are presented. The comparisons include several baseline measures given in Section 2.1, over various datasets within a texture recognition task.

3.1. Network Architectures

In the case of the fully connected autoencoder utilized in GMM-Autoenc${}_{i,AE}$, $i=\{1,2\}$, we used a simple fully connected architecture consisting of input and output layers with one hidden representation layer between them and the logistic sigmoid transfer function. The training consisting of scaled conjugate gradient descent optimization was driven by the MSE loss function defined in (13), accompanied by the $l_2$ weight regularization term $\mathcal{R}(\mathcal{F},\mathcal{G})$ defined in (14), and the regularization penalty $\lambda =10$. The number of nodes in the input and output layers was determined by the dimensionality of the input and output feature space. The CNN encoder utilized in GMM-Autoenc${}_{i,CNNAE}$, $i=\{1,2\}$ consists of three convolutional layers narrowing down the higher-dimensional input (matrices of size $d\times d$) into lower-dimensional vectors ${\mathbb{R}}^l$, while preserving the relevant clustering information. The depths of CNN layers are $n_1=32$, $n_2=64$, and $n_3=128$, with the square kernel sizes of $k_1=14$, $k_2=7$, and $k_3=3$. The dimensions of the layers’ width $w_i$ and height $h_r$, $r=\{1,2,3\}$ are implicitly determined by the size of input SPD matrices, but we note that the stride value was 2. The bottleneck layer of size ${\mathbb{R}}^l$ is connected to the last CNN layer by one flattening layer and two additional linear layers with a ReLU component in between. ReLU components are also added among convolutional layers, including one batch norm regularizer between the last two convolutional layers. The CNN decoder architecture replicates the above-described pattern in the backward order, decompressing the latent feature space into the original matrix features. For CNN autoencoder training, the Adam gradient descent optimization algorithm is applied. Adaptive optimization algorithms, such as Adam, have shown a better optimization performance in the means of faster convergence, due to the adaptive learning rate, as can be seen in [42]. Moreover, empirical results demonstrate that Adam works well in practice and still achieves a competitive performance when compared with newer methods, as can be seen in [43]. The procedure is performed by using the objective defined in (15) over 30 epochs, with a learning rate of $0.001$, a learning decay rate decay of $1e-05$, and the feature map regularization term ${\mathcal{R}}_{\mathcal{FMR}}(\mathcal{F},\mathcal{G})$ defined in (21), with a regularization penalty value of ${\lambda }_{FMR}=10$. The first regularization term $\mathcal{R}(\mathcal{F},\mathcal{G})$ in (15) refers to network weights and is of the same type as in the case of previously described GMM-Autoenc${}_{i,AE}$. In both cases, all input matrices (from both the training and test data instances) have been normalized to the [0,1] range before the actual training.

3.2. Performances

Experimental results are given for the proposed GMM-Autoenc-based similarity measures invoked in Section 2.2 and Section 2.2.4, compared to baseline KL-based measures $K{L}_{WA}$, $K{L}_{MB}$ and $K{L}_{VB}$ given by (3), (4), and (7), respectively, and proposed in [1,2]; as well as to the baseline EMD-based unsupervised $uDPL{M}_{WA}$, $uDPL{M}_{MB}$, $uDPL{M}_{VB}$ measures, proposed in [27]. Experiments were conducted within the texture recognition tasks on the UIUC texture dataset, the KTH-TIPS image dataset, and the UMD texture dataset. A five-class recognition problem is considered in all experiments. Concerning the UIUC dataset, the following classes are considered: wood, water, granite, marble, and floor, with images of $640\times 480$ pixel resolution. In the case of the KTH-TIPS dataset, the following classes were considered: aluminum foil, brown bread, corduroy, cotton, and cracker, with images of $1280\times 960$ pixel resolution. Finally, for the UMD texture dataset, we used images from the following classes: paint cans, stones, brick walls, apples, and textile patterns, sampled at $1280\times 960$ pixels. Selected samples from all three databases are shown in Figure 3.

Concerning the underlying texture features used for the experiments as texture descriptors, the region covariance descriptors proposed in [30] are employed, since they have already shown good performance in various texture recognition tasks (see for example [19,23,27,30]). For any given $N\times M$ image, they are formed in the following way: patches of size $m\times m$, $m<min\{N,M\}$ are cropped ($m=128$ with the step size of 16 for the UIUC dataset, $m=40$ with the step size of 5 for the KTH-TIPS dataset and $m=256$ with the step size of 32 for the UMD dataset). For every pixel positioned at $(x,y)$, features were calculated as elements of ${\mathbb{R}}^{\widehat{d}}$, $\widehat{d}=5$, i.e., $[I,|I_x|,|I_y|,|I_{xx}|,|I_{yy}\left|\right]$, where I represents illumination, $I_x$ and $I_y$ are the first-order derivatives and $I_{xx}$ and $I_{yy}$ are the second-order derivatives. Covariance matrices were calculated for the mentioned features and their upper triangular part is vectorized into $d=\widehat{d}(\widehat{d}+1)/2=15$-dimensional feature vectors. The parameters of GMMs are then estimated using the EM algorithm (as can be seen in [44]) applied over the pool of feature vectors obtained as previously described. During experiments, for every $n\times m$-size image in all mentioned datasets, the uniform sampling of the $N_{smpl}$ sub-images of the size of $(n/2)\times (m/2)$ is performed in order to augment every database, which is needed to train a nonlinear autoencoder in order to perform the GMM-Autoenc measure (deep autoencoder in the case of GMM-Autoenc${}_{1,AECNN}$), because originally, we do not have a sufficient amount of data in order to train them. Every GMM (or its low-dimensional Euclidean or SPD representatives for the proposed GMM-Autoenc or DPLM GMM similarity measures, respectively) is compared to all other GMMs in the training set, and its class is determined using the KNN method.

In

Table 1. Recognition accuracies for the proposed GMM-Autoenc-based measures when compared with KL-based as well as DPLM-based GMM similarity measures on UIUC database.

GMM Sim. Meas.				Accuracy
		$\mathit{m}$ = 1		$\mathit{m}$ = 5		$\mathit{m}$ = 10
$K{L}_{MB}$		0.82		0.80		0.80
$K{L}_{WA}$		0.82		0.82		0.82
$K{L}_{VB}$		0.82		0.82		0.82
		$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7
$uDPL{M}_{MB}$		0.72	0.81	0.73	0.74	0.79	0.79
$uDPL{M}_{WA}$		0.72	0.81	0.73	0.74	0.80	0.80
$uDPL{M}_{VB}$		0.72	0.81	0.73	0.74	0.80	0.80
		$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30
GMM-Autoenc${}_{1,AE}$		0.75	0.81	0.75	0.76	0.80	0.80
GMM-Autoenc${}_{2,AE}$		0.75	0.81	0.76	0.76	0.80	0.80
GMM-Autoenc${}_{3,AE}$		0.76	0.80	0.76	0.77	0.81	0.81
GMM-Autoenc${}_{1,AECNN}$		0.75	0.80	0.73	0.77	0.81	0.80
GMM-Autoenc${}_{2,AECNN}$		0.76	0.81	0.71	0.78	0.81	0.81
GMM-Autoenc${}_{3,AECNN}$		0.77	0.81	0.71	0.79	0.81	0.80

,

Table 2. Recognition accuracies for the proposed GMM-Autoenc-based measures when compared with KL-based as well as DPLM-based GMM similarity measures on KTH-TIPS database.

GMM Sim. Meas.				Accuracy
		$\mathit{m}$ = 1		$\mathit{m}$ = 5		$\mathit{m}$ = 10
$K{L}_{MB}$		0.78		0.74		0.75
$K{L}_{WA}$		0.78		0.78		0.78
$K{L}_{VB}$		0.78		0.78		0.78
		$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7
$uDPL{M}_{MB}$		0.57	0.73	0.69	0.71	0.63	0.72
$uDPL{M}_{WA}$		0.57	0.73	0.72	0.75	0.64	0.75
$uDPL{M}_{VB}$		0.57	0.73	0.72	0.75	0.63	0.75
		$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30
GMM-Autoenc${}_{1,AE}$		0.71	0.75	0.73	0.75	0.72	0.74
GMM-Autoenc${}_{2,AE}$		0.71	0.74	0.72	0.74	0.72	0.75
GMM-Autoenc${}_{3,AE}$		0.72	0.75	0.73	0.75	0.73	0.76
GMM-Autoenc${}_{1,AECNN}$		0.72	0.75	0.73	0.74	0.73	0.73
GMM-Autoenc${}_{2,AECNN}$		0.73	0.75	0.71	0.75	0.74	0.76
GMM-Autoenc${}_{3,AECNN}$		0.73	0.76	0.72	0.77	0.74	0.77

and

Table 3. Recognition accuracies for the proposed GMM-Autoenc-based measures when compared with KL-based as well as DPLM-based GMM similarity measures on UMD database.

GMM Sim. Meas.				Accuracy
		m = 1		m = 5		m = 10
$K{L}_{MB}$		0.75		0.73		0.72
$K{L}_{WA}$		0.75		0.75		0.75
$K{L}_{VB}$		0.75		0.75		0.75
		$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7	$\tilde{\mathit{l}}$ = 5	$\tilde{\mathit{l}}$ = 7
$uDPL{M}_{MB}$		0.73	0.74	0.72	0.72	0.70	0.72
$uDPL{M}_{WA}$		0.73	0.74	0.73	0.74	0.71	0.75
$uDLP{M}_{VB}$		0.73	0.74	0.73	0.74	0.71	0.75
		$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30	$\mathit{l}$ = 20	$\mathit{l}$ = 30
GMM-Autoenc${}_{1,AE}$		0.74	0.74	0.73	0.73	0.71	0.72
GMM-Autoenc${}_{2,AE}$		0.73	0.74	0.73	0.74	0.73	0.74
GMM-Autoenc${}_{3,AE}$		0.74	0.75	0.74	0.75	0.73	0.75
GMM-Autoenc${}_{1,AECNN}$		0.74	0.75	0.73	0.73	0.72	0.72
GMM-Autoenc${}_{2,AECNN}$		0.75	0.75	0.74	0.75	0.74	0.74
GMM-Autoenc${}_{3,AECNN}$		0.74	0.75	0.74	0.75	0.74	0.75

, the performances of the novel GMM-Autoenc GMM similarity measures are presented when compared with KL-based as well as DPLM-based measures, for the UIUC, KTH-TIPS, and UMD datasets, respectively. In all experiments, the number of Gaussian components used to represent any particular training or testing image was set to $m\in \{1,5,10\}$. For the case of DPLM-based and GMM-Autoenc measures, $K=3$ for the KNN, and the projection dimension for DPLM-based methods was set to $\tilde{l}\in \{5,7\}$. Results in Table 1, Table 2 and Table 3 for the proposed GMM-Autoenc${}_{i,AECNN}$ refer to the model training procedure with FMR regularization based on the log-Euclidean ground distance $gd=le$ given in (19), as we obtained slightly better results with this regularizer when compared with other ground distances.

From the experimental results presented in Table 1, Table 2 and Table 3, which correspond to recognition accuracy, and taking into account the computational complexity analyses given in Section 2.3, it can be seen that all of the proposed novel GMM-Autoenc measures obtained a better trade-off between recognition accuracy and computational complexity for all datasets when compared with the baseline KL-based as well as DPLM-based measures mentioned in Section 2.2. We note that GMM-Autoenc${}_{i,AE}$ variants could be the preferred choice when compared with GMM-Autoenc${}_{i,AECNN}$ models, probably due to the number of training samples in all datasets, which is relatively small for the training of a CNN architecture.

4. Conclusions

In this work, a novel similarity measure between GMMs based on autoencoder-generated Gaussian component representations is proposed. Low-dimensional Euclidean vectors obtained using nonlinear autoencoder embedding demonstrated higher similarity measurement capabilities when compared with the existing baseline models. Two different approaches were explored and utilized to encode the original Gaussian mixture components. The first one focused on the simple architecture of the classical autoencoder, where the vectorized versions of the SPD matrices obtained by the Lovric embedding of GMM Gaussians were used to generate inputs for the fully connected autoencoder. In the second approach, a more complex CNN autoencoder was utilized in combination with the novel regularization training in the feature space. In terms of trade-off between the recognition accuracy and the computational complexity, autoencoder embedding proved to be a better solution when compared with the existing GMM similarity measurement baselines in texture recognition tasks. In general, any domain in which some random processes, data instances, or groups of entities are described by PDFs in the form of Gaussian mixture models can be considered as possible fields of application that would benefit from the proposed computationally efficient GMM similarity measures. Besides ML recognition tasks based on decision rules in the form of minimum distance classifiers, we also foresee various applications in other domains such as clustering, probabilistic modeling, kriging methods, learning regularization, and feature space alignment as possible extensions of the current study and proposed similarity measures. For more effective dimensionality reduction, the utilized autoencoder networks should be trained on large datasets, but this could be performed in various settings, online or offline, or by designing encoders for specific sub-tasks and data modalities. In that sense, described methods are scalable and could be adapted to specific system requirements. Thus, in order to fully explore the capabilities of the CNN-based embedding approach, in future work, we plan to conduct experiments on tasks with much larger datasets and different data modalities.