An Optimal-Transport-Based Multimodal Big Data Clustering

Abstract

Multimodal clustering achieves outstanding performance in various applications by aggregating information from heterogeneous devices. However, previous methods rely on strong-notion distances to fuse crossmodal complementary knowledge, established on a fragile assumption about the existence of a ubiquitous non-negligible intersection between heterogeneous manifolds of modalities. Due to this unstable theoretical basis, previous methods are essentially challenged by limited performance on general multimodal data. To address this challenge, an optimal-transport-based multimodal clustering (OTMC) method is defined as the optimal transport (OT) from multimodal data distributions to clustering distributions, which leverages a weak-topology measure to capture complementary knowledge with clear discriminative structures. OTMC consists of a modality-specific OT delivering private structures and a modality-common OT delivering shared structures, which transports category structures scattered in manifolds of each modality and all modalities to common prototypes, respectively. Furthermore, variational solutions to OTMC are derived by matching the data-prototype joint distribution, which induces the multimodal OT clustering network, to capture discriminative structures. Finally, the experimental results from four real-world datasets demonstrate the superiority of OTMC, helped by never relying on the phantom of heterogeneous manifold intersections. In particular, OTMC obtains 92.15% ACC, 84.96% NMI, and 83.35% ARI on Handwritten, improving by 2.25%, 2.82%, and 3.28%, respectively.

1. Introduction

With the rapid development and widespread applications of emerging computing technologies, data produced by human beings are experiencing explosive growth [1]. For instance, a smart power plant acquires real-time data through various devices, such as flow sensors, current transducers, and power meters, to monitor the working status of power equipment, in which an enormous amount of data with heterogeneous distributions are collected. In smart cities, smart car-detection technology utilizes multiple cameras and microphones to verify the categories of cars, amassing a great volume of information about cars. These extensive data collected from multiple devices are often referred to as multimodal data, in which the data of each modality contain consistent information that describes shared properties and complementary information about different properties among modalities [2,3,4]. The fusion of complementary and consistent information in multimodal data can further facilitate the understanding of patterns hidden in complex phenomena of the real world [5]. Thus, it is of great importance to design innovative computing paradigms to integrate information from multimodal data.

Multimodal clustering (MC), as a fundamental unsupervised task, aims to aggregate knowledge scattered in heterogeneous modalities to recognize intrinsic patterns of data [6,7,8]. Recently, some pioneering deep multimodal clustering methods have been proposed to leverage hierarchical nonlinear embedding features to explore patterns from fusion information, which can be roughly separated into two classes, i.e., deep fusion methods [9,10] and deep clustering methods [11,12]. The former utilizes self-supervision information of data to extract generalized fusion representations to characterize fusion information, and then performs vanilla single-modality clustering methods to mine patterns, whose two-stage learning strategy may disconnect fusion representation learning and clustering partition, causing suboptimal fusion representations for clustering partitioning. To close the gap between the two processes, deep clustering methods combine fusion representation learning and clustering partition into a joint optimization strategy. They utilize the divergences of data structures to guide the fusion of complementary information among modalities, as well as mining clustering structures in an end-to-end manner.

Existing MC methods have made significant progress in the pattern mining of multimodal data. However, most of them ignore the manifold topology of multimodal data, and focus on distance metrics with strong notions that assume strict overlaps between heterogeneous distributions of data in various modalities, such as Kullback–Leibler (KL) divergence and Jensen–Shannon (JS) divergence, leading to the degradation of performance on pattern mining when they are faced with the challenge of general multimodal data applications [13]. Specifically, general multimodal data collected from diverse sources obey heterogeneous distributions and are embedded in their corresponding manifolds, and the manifolds of data in each modality almost have no non-negligible intersection [14]. In that case, the assumption above is not valid everywhere, which leads to failure in measuring the differences between heterogeneous manifolds when integrating complementary information from multimodal data. In other words, distance metrics with strong notions provide no useful gradient information in training deep clustering models, which blurs discriminative structures when fusing representations of multimodal data. Thus, previous methods may not exhibit desirable performance in pattern mining in real-world applications.

To address this challenge, an optimal-transport-based multimodal clustering method (OTMC) is proposed to conduct multimodal clustering structure mining based on the optimal-transport (OT) theory, which ensures clustering-friendly discriminative structures when fusing complementary modality information with heterogeneous distributions. Specifically, multimodal clustering is defined as a map from multimodal data measures to clustering prototype set measures in OTMC, which utilizes the Wasserstein distance to obtain clear discriminative structures from heterogeneous information with no intersection of data distributions in different modalities. Furthermore, OTMC is decomposed into a modality-specific OT (MsOT) and a modality-common OT (McOT) within a consistent constraint, to disentangle the transportation of private and shared information of each modality. In addition, a generative transport plan is designed to derive the variational solution to OT, which can effectively search the map with a minimal transport cost for mining intrinsic patterns. Finally, although designed for all data with heterogeneous distributions in different modalities, e.g., text and video, OTMC is conducted on four real-world benchmark image datasets in extensive experiments to validate its superiority on high-dimension data.

The main contributions of this paper are as follows:

• An innovative weak-notion distance metric-based method is designed to measure differences between the manifold structures of data collected from diverse devices, which ensures the full fusion of complementary information from data with heterogeneous distributions.
• Multimodal clustering is innovatively modeled using an optimal-transport-based multimodal clustering method (OTMC), which can capture fusion information with clear discriminative structures from heterogeneous modalities for mining intrinsic patterns.
• A variational solution is derived to solve OT based on a generative transport plan, which can precisely match the transport map for transporting the multimodal data to clustering prototypes in OTMC.
• Extensive experiments are conducted on four real-world benchmark datasets, which verify the superiority of OTMC compared with other methods in multimodal clustering, helped by never relying on the phantom of heterogeneous manifold intersections. In particular, OTMC obtains 92.15% ACC, 84.96% NMI, and 83.35% ARI on Handwritten, improving by 2.25%, 2.82%, and 3.28%, respectively.

In the rest of this paper, Section 2 gives the related deep fusion methods and deep clustering methods; Section 3 introduces the theoretical framework of OTMC; Section 4 derives the variational generative solution to OTMC and network implementations; Section 5 showcases the experimental results of OTMC in comparison with deep fusion methods and deep clustering methods; and Section 6 concludes this work and points out some possible future research directions for subsequent exploration.

2. Related Work

Clustering is a classical unsupervised learning task. Various effective clustering algorithms like k-means have been proposed and widely utilized as cornerstones of the solutions to problems such as pavement deterioration [15], facility location [16], and the energy management of wireless sensor networks [17]. OTMC learns intrinsic patterns of multimodal data via neural networks, and is closely related to deep fusion methods and deep clustering methods in multimodal clustering.

Deep fusion methods, as two-stage methods, focus on extracting generalized fusion representations of multimodal data, and then leveraging a single-modality clustering algorithm to mine clustering patterns. For example, DCCA [18,19] learned fusion representations of multimodal data via hierarchical nonlinear mappings along with crossmodal correlation maximizing. DMF [20] mined the consensus semantics of multimodal data by leveraging multi-layer transformations subject to non-negative constraints to explore informative spatial bases. AE2-Nets [9] modeled a degradation process from multimodal data to fusion representations within a nested autoencoder architecture to capture consistent and complementary information among modalities. CMIB [10] utilized the rate-distortion principle to explore robust fusion representations by preserving complementary and consistent information and discarding the superfluous information of modalities. MIFN-ML [21] conducted intermediate fusion of biometric and facial representations after manifold-learning-based dimension reduction techniques, which then made a clustering decision for stress detection via a nonlinear mapping. MCBVAD [22] introduced attentions from other views to suppress information from irrelevant views on the basis of maximizing information entropy and mutual information of views, which produced information-fruitful representations to improve k-means clustering results. Further, with the help of a Transformer-based attention learning block, VMC-CD [23] applied cross-view contrastive learning and maximum entropy of categories to obtain discriminative representations for k-means clustering. OSCAMC [24] jointly learned and optimized view-collective matrices, including a combined non-negative embedding matrix, a collective similarity matrix, a joint coefficient matrix, a unified spectral projection matrix, and so on, and then obtained the clustering results from those matrices.

Deep clustering methods integrate complementary information fusion with clustering pattern recognition in an end-to-end architecture, where divergences of data structures are used to supervise model training. MVaDE [25] modeled the generation process from hidden categories to visible data, where the Gaussian distribution was used to constrain the information fusion of each modality. CoMVC [26] utilized crossmodal contrastive learning followed by adaptive linear combination to fuse modality information, and then conducted Cauchy–Schwarz divergence in the fusion representation space for clustering. SDMVC [27] utilized KL-divergence to mine clustering patterns from concatenated complementary information, which uses consistent semantics to constrain the alignments of representation distributions in each modality. CMSC [28] fused modality information on the basis of a consensus subspace which maximized the correlations of modalities, and then performed spectral clustering to mine clustering patterns. DMIM [29] fused complementary information by maximizing the dependence between modalities and mined multimodal data patterns with the help of an over-clustering strategy. DMAC-SI [5] combined invariant semantics representation learning and the Markov decision process of multimodal data clustering into an end-to-end clustering framework, ensuring the flexibility of clustering structure exploration. EMVFC [30] integrated low-dimensional representation learning and cooperative learning in clustering to improve performance, which reduced redundancy features and noise, enhancing the complementary information and correlation.

The deep fusion methods above extracted generalized fusion representations and then performed vanilla single-modality clustering methods to mine patterns, which may disconnect fusion representation learning and clustering partition, causing suboptimal fusion representations for clustering partition. The deep clustering methods above combined fusion representation learning and clustering partition into a joint optimization strategy for mining clustering structures in an end-to-end manner. However, they still relied on strong-notion distances to fuse crossmodal complementary knowledge, which limited their clustering performance. In contrast, optimal transport can define multimodal clustering as a map from multimodal data measures to clustering prototype set measures, which avoids representation-partition disconnections and dependency on the intersection of data distributions in different modalities.

Through the comparison between optimal transport and clustering, it can be seen that a difference between optimal transport and clustering is that optimal transport ex-changes the operation form for the lowest cost. That is, for any two mappings that map data points to cluster centers, optimal transport chooses the one with lowest cost and does not consider how it works. In a sense, optimal transport is more flexible and difficult than clustering. Thus, the solution to optimal transport can be used in clustering. In multimodal clustering, dividing the data into modality-specific and modality-common components is a standard approach. OTMC further distinguishes itself by incorporating the concept of optimal transport into this framework. Specifically, OTMC extends the rationale for optimal transport by partitioning it into modality-common and modality-specific components as well. This allows OTMC to better capture the intrinsic relationships both within and across modalities, offering a more nuanced representation of the data. In addition, there are diverse opinions about how to evaluate the quality of a clustering algorithm so that many metrics are proposed. If the objective of clustering is just minimizing the sum of distances in each cluster in k-means, no more problems appear. So, in this context, our method is a beneficial step toward providing a new direction for clustering, i.e., evaluating clustering quality based on transport cost.

3. OT for Multimodal Clustering

In this section, multimodal clustering is redefined within the optimal-transport architecture, to mine patterns of multimodal data by adaptively learning an optimal transport map between multimodal data and clustering prototypes. Then, rooted in private and common information hidden in multimodal data, OTMC is decomposed into modality-specific OT and modality-common OT, as shown in Figure 1. In addition, the most frequently used notations are listed in

Table 1. Frequently used notations.

Notations	Description
$\mathcal{X},\tilde{\mathcal{X}}$	Multimodal data/reconstructed data space.
${\mathcal{X}}^m,{\tilde{\mathcal{X}}}^m$	$m$-th modality data/reconstructed data space.
$M$	Number of modalities.
$k$	Number of clusters.
$\mu$	Multimodal data measure.
${\mu }_s^m$	$m$-th modality-specific data measure.
$C$	Clustering prototype set.
$c_j$	$j$-th clustering prototype.
$C_s^m$	$m$-th modality-specific clustering prototype set.
$c_j^m$	$m$-th modality, $j$-th clustering prototype.
$\nu$	Clustering prototype measure over all modalities.
${\nu }_s^m$	$m$-th modality-specific clustering prototype measure.
$T$	Clustering transport map from data to prototypes.
$T_s^m$	$m$-th modality-specific transport map.
$T_c$	Common transport map over all modalities.
$W$	Multimodal data partition scheme.
$W_j$	Set of data split into $j$-th cluster.
$\pi$	Coupled measure of measures $\mu$ and $\nu$.
$G$	Generative transport plan from prototypes to data.
$G_s^m$	$m$-th modality-specific generative transport plan.
$Cat$	Categorical prior distribution.
$Ca{t}_s^m$	$m$-th modality-specific categorical prior distribution.
$Cat(\varpi )$	Categorical distribution with parameter $\varpi$.
$E_s^m(\cdot ;{\theta }_s^m)$	Clustering network with parameter ${\theta }_s^m$.
$P^m$	$m$-th modality soft-clustering-assignment matrix.
$G_s^m(\cdot ;{\phi }_s^m)$	Generative network with parameter ${\phi }_s^m$.
$D_w(\cdot ,\cdot )$	OT-based topological reconstruction loss.

.

3.1. Modeling Multimodal Clustering Based on OT

Optimal transport utilizes a weak topology measure to estimate the similarity between two manifolds defined over a space by computing the cost of moving one point to another within the space. Thus, optimal-transport-based multimodal clustering (OTMC) is proposed to aggregate complementary and consistent information in modalities with heterogeneous manifolds, which provides representations with clear discriminate structures without the help of intersections between data manifolds of different modalities. Specifically, let $\mathcal{X}={\left\{{\mathcal{X}}^m\right\}}_{m=1}^M$ denote the measurable space of a multimodal instance with measure $\mu$, where ${\mathcal{X}}^m$ is the m-th modality space in $M$ modalities, and let $C={\left\{c_j\right\}}_{j=1}^k$ denote the prototype set with cluster number $k$ and prototype measure $\nu$, in which $c_j$ denotes the $j$-th clustering prototype. Multimodal clustering $\mathcal{T}$ groups data into corresponding clustering prototypes with minimal pre-defined criteria as follows:

$$\mathcal{T}\iff \underset{W}{\arg \min }{\displaystyle \sum _{j=1}^k{\displaystyle {\int }_{W_j}d(x,c_j)d\mu (x)}},$$

(1)

where $W=\left\{{(W_j,c_j)}_{j=1}^k\right\}$ stands for a clustering scheme and $W_j$ stands for the data partition belonging to the prototype $c_j$.

Multimodal clustering groups data into corresponding clusters by minimizing discrepancies between data and prototypes, which is natural and consistent with the optimal transport that finds an optimal-transport map from data to the corresponding clusters, as follows:

$$\mathcal{T}\iff \underset{W}{\arg \min }{\displaystyle \sum _{j=1}^k{\displaystyle {\int }_{W_j}d(x,c_j)d\mu (x)}}\iff \underset{T_{\#}\mu =\nu }{\arg \min }{\displaystyle {\int }_{\mathcal{X}}d(x,T(x))d\mu (x)},$$

(2)

where $d\left(,\right)$ measures the transport map cost between data and prototypes, and $\nu ={\displaystyle \sum _{j=1}^k{\nu }_j\delta (c-c_j)}$ stands for the prototype measure based on the Dirac function $\delta (\cdot )$, in which ${\nu }_j$ stands for the $j$-th prototype mass. Meanwhile, $T_{\#}\mu =\nu$ indicates that the overall data mass transported to the prototype $c_j$ is the same as the $c_j$ mass, that is, $\mu (T^{-1}(c_j))={\nu }_j$. The cluster partition $W$ is equivalent to the map $T$ when the prototype measure $\nu$ is fixed. That is, the map $T$ projects instances of cluster $W_j$ to the prototype $c_j$, i.e., $W_j=\left\{x_i\right|T(x_i)=c_j,x_i\in \mathcal{X}\}$.

3.2. Decomposition of OTMC

In multimodal data, each modality of the same instance utilizes heterogeneous structures to deliver the information from different aspects. In other words, the data of each modality contain both private and common information. Thus, the optimal-transport-based multimodal clustering method is decomposed into the modality-specific OT (MsOT) that captures the private information, and the modality-common OT (McOT) that catches the common information.

Modality-specific OT $T_s^m(m=1,\dots ,M)$ aims to capture the $m$-th modality private information that contributes to recognizing inherent patterns in each modality. Specifically, $T_s^m$ represents the map from the data of the $m$-th modality space ${\mathcal{X}}^m$ to the $k$ prototype set $C_s^m$ with measure-preserving $T_s^m{}_{\mathrm{\#}}\mu _{\mathrm{s}}^{\mathrm{m}}={\nu }_s^m$, which pushes forward ${\mu }_s^m$ to ${\nu }_s^m$, i.e., $T_s^m:{\mathcal{X}}^m\to C_s^m$. ${\mu }_s^m$ and ${\nu }_s^m$ denote the measure of the space ${\mathcal{X}}^m$ and the set $C_s^m$, respectively. $T_s^m$ can be obtained by minimizing the following total cost based on Equation (2):

$$\mathrm{MsOT}=\underset{T_s^m{}_{\mathrm{\#}}\mu _{\mathrm{s}}^{\mathrm{m}}={\nu }_s^m}{\arg \min }{\displaystyle {\int }_{{\mathcal{X}}^m}{d}_s^m(x,T_s^m(x))d{\mu }_s^m(x)},m=1,\dots ,M,$$

(3)

where $d_s^m:{\mathcal{X}}^m\times C_s^m\to {ℝ}^{+}$ is the transport map cost between data and prototypes in the $m$-th modality. The modality-specific OT explores manifold differences between modality-specific data and clustering prototypes by utilizing a measure with weak topology, to learn an optimal transform map from a local perspective, which effectively captures the private information for mining patterns in each modality.

Modality-common OT $T_c$ catches common information that reveals the consistent semantics of instances among modalities. To be specific, $T_c$ maps data in multimodal space $\mathcal{X}$ to the modality-common prototype set $C_c$ with measure ${\nu }_c$, which means the map $T_c$ pushes forward $\mu$ to ${\nu }_c$. $T_c$ can be obtained by minimizing the cost based on Equation (2):

$$\mathrm{McOT}=\underset{T_{\mathrm{c\#}}\mu ={\nu }_c}{\arg \min }{\displaystyle {\int }_{\mathcal{X}}{d}_c(x,T_c(x))d\mu (x)},$$

(4)

where $d_c:\mathcal{X}\times C_c\to {ℝ}^{+}$ is the transport map cost of modality-common OT $T_c$. Modality-common OT explores manifold differences between multimodal data and modality-common clustering prototypes by utilizing a measure with weak topology to learn an optimal transform map from a global perspective, which effectively captures the common semantic information for mining patterns in multimodal data.

3.3. The Consistent Constraint

Since both modality-specific OT and modality-common OT are used to mine the intrinsic clustering patterns of multimodal data, their transport plan $T_s^m$ and $T_c$ should be consistent. To this end, a consistent constraint $L_{con}$ between ${\nu }_s^m$ and ${\nu }_c$ is introduced to achieve the above objective by keeping the clustering prototypes of modality-specific OT and modality-common OT subject to the same measure:

$$L_{con}=\min {\displaystyle \sum _{m=1}^{M}D({\nu }_s^m,{\nu }_c)},$$

(5)

where $D(\cdot )$ denotes a metric function that evaluates differences between the measures ${\nu }_s^m$ and ${\nu }_c$. Finally, modality-common OT $T_c$ is used for the cluster assignment result.

4. The Variational Generative Solution Network Implementation

In this section, a variational generative solution to OT is derived, and then, it is generalized to multimodal scenarios for optimizing data partitioning within an optimal-transport clustering network.

4.1. Variational Generative Solution

The variational generative solution learns the generative relationship between data and prototypes by constructing the joint distribution $p\left(x,c\right)$, which provides a theoretically accurate analytical solution to OT. Specifically, $p\left(x,c\right)=p\left(c|x\right)p\left(x\right)$, where $p\left(x\right)$ stands for the data distribution and $p\left(c|x\right)$ denotes the cluster conditional distribution. Since the OT that transports the multimodal data to the clustering prototypes is actually finding the optimal prototype $c_j$ for each data point $x_i$, which is fitting the cluster conditional distribution $p\left(c|x\right)$, the joint distribution has the same solution as OT.

To fit the joint distribution over the data measure $\mu$ and the clustering prototype measure $\nu$, the solution is to find an optimal coupling measure ${\pi }^{\ast }$ between $\mu$ and $\nu$ such that $\pi (\cdot ,c)=\mu$ and $\pi (x,\cdot )=\nu$ are satisfied for any corresponding marginal measures. The optimal coupling ${\pi }^{\ast }$ is obtained by finding the infimum of the coupling cost in the set of coupling measures $\mathsf{\Pi }(\mu ,\nu )$ as follows:

$${\pi }^{\ast }=\underset{\pi \in \mathsf{\Pi }(\mu ,\nu )}{\arg \inf }{\displaystyle {\int }_{\mathcal{X}\times C}d(x,c)d\pi (x,c)},$$

(6)

where $d\left(x,c\right)$ stands for the coupling cost between $x$ and $c$. The optimal coupling solution ${\pi }^{\ast }$ is obtained by reaching the infimum of the total coupling cost over the product space $\mathsf{\Pi }(\mathcal{X}\times C)$.

Since $p\left(x,c\right)=p\left(x|c\right)p\left(c\right)$, the fitting of $p\left(x,c\right)$ can be transformed into the learning of the categorical prior distribution $p\left(c\right)$ and the generative conditional distribution $p\left(x|c\right)$. The former models prior knowledge of the data’s categorical distribution, which can be represented by any distribution $Cat$. The latter learns the generation mechanism of the data to better reflect the probabilistic properties of the data, which can be achieved through the generative transport plan $G$ from the clustering prototypes to the data. Then, the categorical distribution $Cat$ and the generative transport plan $G$ can be updated by minimizing the transport cost between the generated data and the prototypes:

$$\underset{G,Cat}{\arg \inf }{\displaystyle {\int }_{c~Cat}d(G(c),c)d\nu (c)},$$

(7)

where $d(\cdot )$ is the ground transport cost between $x$ and $c$. After fitting the categorical prior distribution $p\left(c\right)$ and the generative conditional distribution $p\left(x|c\right)$, $p\left(x,c\right)$ can be obtained via the Bayesian rule.

4.2. The Variational Generative Solution to OTMC

Based on the variational generative solution to OT, the modality-specific OT and the modality-common OT decomposed from OTMC can easily be solved. Specifically, the solution to modality-specific OT can be achieved by solving the generative transport plan $G_s^m$ from the categorical prior distribution $Ca{t}_s^m$ to the data of the $m$-th modality space ${\mathcal{X}}^m$, minimizing the following ground transport cost:

$$\mathrm{MsOT}=\underset{G_s^m,Ca{t}_s^m}{\arg \min }{\displaystyle {\int }_{c~Ca{t}_s^m}{d}_s^m(G_s^m(c),c)d{\nu }_s^m(c)},m=1,\dots ,M,$$

(8)

where $G_s^m:Ca{t}_s^m\to {\mathcal{X}}^m$ is the generative transport map of the $m$-th modality, and $G_s^m$ generates instances from the categorical distribution, which can retain the real underlying structure information under the supervision of multimodal data. The generative transport plan $G_s^m$ and the categorical distribution $Ca{t}_s^m$ can jointly learn the private data structure information and clustering information of the $m$-th modality.

In practice, the integral over the measure ${\nu }_s^m$ is approximated by Monte Carlo sampling, i.e., the integral of the measure ${\nu }_s^m$ over the categorical distribution $Ca{t}_s^m$ is equivalent to the expectation of all clustering prototypes over $Ca{t}_s^m$. Thus, Equation (8) can induce the following objective function of modality-specific OT $L_s^m$:

$$\underset{G_s^m,Ca{t}_s^m}{\mathrm{argmin}}{\mathbb{E}}_{{\mathcal{X}}^m}{\mathbb{E}}_{Ca{t}_s^m}[{\nu }_s^m(c)d_s^m(G_s^m(c),c)],$$

(9)

where $m=1,\dots ,M$ indicates the $m$-th modality, and the objective function $L_s^m$ fully captures inherent information of the $m$-th modality by jointly learning structure differences between data and semantic information within data.

Similarly, the optimization of modality-common OT is equivalent to solving the generative transport plan $G_c$ from the categorical distribution to the data distribution for all modalities:

$$\mathrm{McOT}=\underset{G_c,Ca{t}_c}{\mathrm{argmin}}{\displaystyle {\int }_{c~Ca{t}_c}{d}_c(G_c(c),c)d{\nu }_c(c)},$$

(10)

where $G_c:Ca{t}_c\to \mathcal{X}$ is the generative transport map for all modalities, which can generate data of all modalities from the common categorical distribution $Ca{t}_c$ to capture common semantics of data.

The joint optimization of the generative transport plan $G_c$ and the categorical distribution $Ca{t}_c$ can simultaneously capture the data structure information of all modalities and crossmodal clustering information. The integral over the measure ${\nu }_c$ is approximated by the expectation of all prototypes over $Ca{t}_c$ based on the Monte Carlo sampling, so the objective function of modality-common OT $L_c$ can be written as:

$$\underset{G_c,Ca{t}_c}{\mathrm{argmin}}{\mathbb{E}}_{\mathcal{X}}{\mathbb{E}}_{Ca{t}_c}[{\nu }_c(c)d_c(G_c(c),c)].$$

(11)

The objective function $L_c$ effectively preserves common information in multimodal data by jointly optimizing the structure differences of all modalities and crossmodal semantic information.

Thus, OTMC can be optimized via the following variational solution:

$$\begin{array}{cc}\hfill L=& {\displaystyle \sum _{m=1}^M{\mathrm{argmin}}_{G_s^m,C_s^m}{\mathbb{E}}_{{\mathcal{X}}^m}{\mathbb{E}}_{Ca{t}_s^m}[{\nu }_s^m(c)d_s^m(G_s^m(c),c)]}\hfill \\ & +{\mathrm{argmin}}_{G_c,Ca{t}_c}{\mathbb{E}}_{\mathcal{X}}{\mathbb{E}}_{Ca{t}_c}[{\nu }_c(c)d_c(G_c(c),c)]\hfill \\ & +{\displaystyle \sum _{m=1}^M{D}_{KL}(Ca{t}_s^m,Ca{t}_c)}\hfill \end{array},$$

(12)

where $D_{KL}$ denotes the consistent constraint of categorical prior distributions $Ca{t}_s^m$ and $Ca{t}_c$, which enables MsOT and McOT to have the same transport target, so that the results of the two transport schemes tend to be the same. It is implemented by the KL-divergence penalty between $Ca{t}_s^m$ and $Ca{t}_c$.

4.3. Multimodal OT Clustering Network

Although they attract much attention from researchers, generative adversarial networks (GANs) involve performance-unstable and computation-expensive adversarial training. Hence, to implement OTMC, a multimodal OT clustering network is designed within a deep neural network architecture which contains a generative network and a clustering network, instead of a GAN. The generative network implements the generative transport plan $G$ together with the categorical distribution $Cat(\varpi )$, and the clustering network produces final assignment results from multimodal data.

The clustering network aims to learn the modality-specific and modality-common soft-clustering assignments that are included in MsOT and McOT. Specifically, in modality-specific OT, the clustering network $E_s^m(\cdot ;{\theta }_s^m)$ encodes modality-specific data into corresponding latent features, and utilizes Student‘s $t$-distribution in the last layer of $E_s^m(\cdot ;{\theta }_s^m)$ to calculate the similarity between latent features $z_i^m$ and clustering prototypes $c_j^m$ to obtain the soft-clustering-assignment matrix $P^m$ as follows:

$$\begin{array}{c}{z}_i^m=E_s^m(x_i^m;{\theta }_s^m)\hfill \\ p_{ij}^m={\displaystyle \frac{{(1+||z_i^m-c_j^m|{|}^2/\alpha )}^{-\frac{\alpha +1}{2}}}{{\displaystyle {\sum }_{j^{\prime }}{(1+||z_i^m-c_{j^{\prime }}^m|{|}^2/\alpha )}^{-\frac{\alpha +1}{2}}}}}\hfill \end{array},$$

(13)

where $x_i^m\in {\mathcal{X}}^m$ is the $i$-th instance from the $m$-th modality; $c_j^m\in Ca{t}^m({\varpi }^m)$ is the $j$-th clustering prototype of the $m$-th modality; and $p_{ij}^m$ can be interpreted as the possibility that the $i$-th instance is divided into the $j$-th cluster. To guide the learning of networks, a target distribution $Q^m$ is introduced, and the KL-divergence between $Q^m$ and the soft-assignment matrix $P^m$ is used as the objective function:

$$L_{cluster}^m=D_{KL}(Q^m,P^m),$$

(14)

where $q_{ij}={\displaystyle \frac{p_{ij}^2 /{\displaystyle {\sum }_i{p}_{ij}}}{{\displaystyle {\sum }_{j^{\prime }}\left(p_{i{j}^{\prime }}^2/{\displaystyle {\sum }_i{p}_{i{j}^{\prime }}}\right)}}}$ is the target assignment possibility of the $i$-th instance to the $j$-th clustering prototype. This objective function guides network learning by aligning the soft assignment to the target probability distribution.

For McOT, the objective of the clustering network $E_c(\cdot ;{\theta }_c)$ is $D_{KL}(Q,P)$, where $P$ and $Q$ are the soft-assignment matrix and target probability distribution over all modalities, respectively. According to the consistent constraint $L_{con}$ in Equation (5), each $D(P^m,P)$ is also regarded as an objective component. Hence, the total objective of the clustering networks is

$$L_{cluster}={\displaystyle \sum _{m=1}^M(D(Q^m,P^m)+D(P^m,P))+D(Q,P)},$$

(15)

where $D(\cdot ,\cdot )$ denotes the KL-divergence between two assignment matrices. The first term guides the network to learn the clustering information within each modality, the second term is the consistency constraint of modality-specific and modality-common clustering information, and the last term guides the network to capture the clustering information over all modalities. Finally, the soft-assignment matrix $P$ is used as the cluster assignment matrix.

The generative network is designed to learn the categorical distribution $Cat(\varpi )$ and the generative transport plan $G$ included in MsOT and McOT. Specifically, for MsOT, the generative network $G_s^m(\cdot ;{\phi }_s^m)$ decodes latent features ${\tilde{z}}_i^m$ sampled from the categorical distribution $Cat(\cdot )$ to corresponding data:

$$\begin{array}{c}{z}_i^m=sample(Cat({\varpi }^m))\hfill \\ {\tilde{x}}_i^m=G_s^m(z_i^m;{\phi }_s^m)\hfill \end{array},$$

(16)

where $Cat({\varpi }^m)$ is the categorical distribution of the $m$-th modality with parameter ${\varpi }^m$. To update the parameter ${\varpi }_s^m$ of the generative network and the categorical distribution parameter ${\varpi }^m$, the topological reconstruction loss between the generative data and the original data is used as a guide, namely,

$$L_{rec}^m={\displaystyle \sum _{i=1}^N{D}_w(x_i^m,G_s^m(sample(Cat({\varpi }^m))))},$$

(17)

where $D_w(\cdot ,\cdot )$ represents the topological reconstruction loss based on optimal transport to measure the differences in manifold structures between original and reconstructed data. The network structure and loss of the generative network $G_c(\cdot ;{\phi }_c)$ of McOT are similar to those of MsOT. So, the total objection function of generative networks can be written as follows:

$$L_{rec}={\displaystyle \sum _{m=1}^M{\displaystyle \sum _{i=1}^N{D}_w(x_i^m,G_s^m(z_i^m))}}+D_w(x_i,G_c(z_i)),$$

(18)

where the purpose of the first term is to optimize the categorical distribution and generative network within the modality, and the purpose of the second term is to optimize over all modalities, capturing the data structure information within and between modalities, respectively.

4.4. The Overall Loss

The overall loss function $L$ of the multimodal OT clustering network is defined as follows:

$$L=L_{rec}+\alpha L_{cluster},$$

(19)

where $\alpha$ denotes the trade-off parameter for balancing the topological reconstruction loss $L_{rec}$ and clustering pattern mining loss $L_{cluster}$, and $L$ is used to optimize the multimodal OT clustering network via the stochastic gradient descent strategy, which effectively boosts the performance of multimodal clustering:

$${\theta }_{t+1}={\theta }_t-{\eta }_t{\nabla }_{\theta }L({\theta }_t),$$

(20)

where ${\theta }_t$ and ${\eta }_t$ denote the parameter and the learning rate of the multimodal OT clustering network at time step $t$, respectively, and ${\nabla }_{\theta }L({\theta }_t)$ denotes the gradient at time step $t$. The overall optimization algorithm is shown in Algorithm 1.

Algorithm 1. Multimodal OT clustering network.

Input: Multimodal dataset $X\in \mathcal{X}$ and convergence criteria thr. Output: Soft-clustering-assignment matrix $P$ and the parameters of clustering network $E_s^m(\cdot ;{\theta }_s^m)$$, E_c(\cdot ;{\theta }_c)$, generative network $G_s^m(\cdot ;{\phi }_s^m)$$, G_c(\cdot ;{\phi }_c)$ categorical distribution $Cat({\varpi }_s^m)$$, Cat({\varpi }_c)$. Initialize: modality-specific categorical distributions $Cat({\varpi }_s^m)$ modality-common categorical distribution $Cat({\varpi }_c)$ parameters of modality-specific clustering networks $E_s^m(\cdot ;{\theta }_s^m)$, parameters of modality-common clustering network $E_c(\cdot ;{\theta }_c)$, parameters of modality-specific generative networks $G_s^m(\cdot ;{\phi }_s^m)$, and parameters of modality-common generative network $G_c(\cdot ;{\phi }_c)$ while convergence criteria thr is not reached by Equation (19) do  for each $m=1,\dots ,M$ do   Sample data in the $m$-th modality $X^m$.   Generate the $m$-th modality soft-assignment matrix $P^m$ according to Equation (13).   Update $E_s^m(\cdot ;{\theta }_s^m)$ by minimizing Equation (14).   Sample modality-specific clustering prototypes from $Cat({\varpi }_s^m)$.   Generate the $m$-th modality reconstructed data ${\tilde{x}}_i^m$ according to Equation (16).   Update $G_s^m(\cdot ;{\phi }_s^m)$ and $Cat({\varpi }_s^m)$ by minimizing Equation (17).  end  Sample multimodal data $X$.  Generate modality-common soft-assignment matrix $P$.  Update modality-common clustering network $E_c(\cdot ;{\theta }_c)$ by minimizing Equation (15).  Sample modality-common clustering prototypes from $Cat({\varpi }_c)$.  Generate multimodal reconstructed data.  Update modality-common generative network $G_c(\cdot ;{\phi }_c)$ by minimizing Equation (18).  Update all network parameters by minimizing Equation (19) according to Equation (20). end

5. Experiments

5.1. Experimental Setup

Datasets. Four real-world high-dimensional image datasets are utilized in the experiments to evaluate the performance of OTMC. Specifically, Handwritten has 2000 objects of 0–9, where the pixel features and profile correlations are used as two different modalities [31]. ORL collects 400 objects in terms of intensity features and Gabor features, which are collected under 10 different class conditions, such as lighting and facial expressions, of 40 subjects [32]. LandUse consists of 2100 satellite images distributed in 21 scene categories, where each satellite image is described by LBP and PHOG features [33]. Scene contains 4485 objects of 15 natural scene categories, in which PHOG and GIST features are used for each object [34]. The detailed information of these datasets is listed in

Table 2. The information of datasets.

Dataset	Number	Modality	Class
Handwritten	2000	2 (pixel/profile correlation)	10 (handwritten numbers 0–9)
ORL	400	2 (intensity/Gabor)	10 (face image shooting conditions)
LandUse	2100	2 (LBP/PHOG)	21 (satellite image scene categories)
Scene	4485	2 (PHOG/GIST)	15 (natural scene categories)

.

Comparison methods. The comparison methods are grouped into two categories, deep fusion methods and deep clustering methods. Specifically, the deep fusion methods include DCCA [18], DCCAE [19], AE2-Nets [9], and CMIB [10]. These methods conduct k-means on the fusion features extracted by the corresponding deep architectures. The deep clustering methods include DMF [20], MVaDE [25], SiMVC [26], CoMVC [26], and SDMVC [27]. These methods are directly conducted on multimodal data to obtain the clustering results in an end-to-end manner. Moreover, FeatConcate is also used as a comparison method, which performs k-means on concatenated features.

Evaluation metrics. Three well-known and widely used evaluation metrics, i.e., accuracy (ACC) [35], normalized mutual information (NMI) [36], and adjusted rand index (ARI), are used to measure the performance of all the methods. High values of these metrics indicate ideal clustering performance.

ACC measures the ratio of correct cluster elements to all elements as follows:

$$\mathrm{ACC}={\displaystyle \frac{{\displaystyle \sum _{i=0}^{N-1}\delta (l_i,m(p_i))}}{N}},$$

(21)

where $l_i$ and $p_i$ are the labels and the predictions, respectively; $m()$ and $\delta ()$ are the Kuhn–Munkres function and the Dirac delta function, respectively; and $N$ is the size of the multimodal dataset.

NMI measures the dependence between the label distribution and prediction distribution as follows:

$$\mathrm{NMI}={\displaystyle \frac{2I(L;P)}{H(L)+H(P)}},$$

(22)

where $L$ and $P$ denote the label distribution and the prediction distribution, respectively; and $I()$ and $H()$ denote the mutual information function and the entropy function, respectively.

ARI is a measure on the basis of the rand index (RI) as follows:

$$\mathrm{ARI}={\displaystyle \frac{\mathrm{RI}-E(\mathrm{RI})}{\max (\mathrm{RI})-E(\mathrm{RI})}},$$

(23)

where RI is the rand index, and $E$ denotes the expectation operator.

Implementation details. OTMC is implemented in PyTorch. In the experiments, vector representations of instances are normalized to $\left[0,1\right]$, and then are fed into the network for training. During training, the Adam solver is used with a batch size of 100 to optimize the network, and the learning rate is set to ${10}^{-3}$ for all modules of the network. In addition, all the reported final results are the average results of 5 repeated experiments.

5.2. Clustering Performance Evaluation

Table 3. Average clustering results on Handwritten, ORL, LandUse, and Scene datasets. Best results are marked in bold.

Dataset	Handwritten			ORL			LandUse			Scene
Metric	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI
FeatConcate	0.6104	0.6070	0.5532	0.5710	0.7528	0.4770	0.1232	0.1608	0.0365	0.3076	0.3540	0.1862
DCCA [18]	0.6626	0.6601	0.6136	0.5968	0.7784	0.5020	0.1551	0.2315	0.0443	0.3618	0.3892	0.2087
DCCAE [19]	0.6917	0.6696	0.6327	0.5940	0.7752	0.4993	0.1562	0.2441	0.0442	0.3644	0.3978	0.2147
DMF [20]	0.6962	0.7160	0.5993	0.6733	0.8164	0.5407	0.1450	0.1543	0.0360	0.3393	0.3576	0.1862
AE2-Nets [9]	0.8152	0.7139	0.6667	0.6885	0.8573	0.5637	0.2479	0.3036	0.1035	0.3610	0.4039	0.2208
MVaDE [25]	0.8875	0.8076	0.7765	0.6950	0.8356	0.5643	0.2248	0.2848	0.0936	0.3782	0.3992	0.2178
CoMVC [26]	0.8205	0.8142	0.7559	0.7063	0.8652	0.5353	0.2624	0.3083	0.1085	0.3859	0.4117	0.2231
SiMVC [26]	0.8295	0.7608	0.6985	0.6921	0.8560	0.5256	0.2448	0.2581	0.0957	0.3775	0.3935	0.2259
SDMVC [27]	0.8990	0.8214	0.8007	0.7104	0.8557	0.5942	0.2681	0.2986	0.1201	0.3857	0.4132	0.2259
CMIB [10]	0.8972	0.8178	0.7988	0.7207	0.8823	0.6004	0.2716	0.3016	0.1242	0.3954	0.4177	0.2383
OTMC	0.9215	0.8496	0.8335	0.7650	0.8837	0.6496	0.2814	0.3254	0.1312	0.4181	0.4373	0.2621

demonstrates the numerical clustering results in terms of ACC, NMI, and ARI. The results show that OTMC obtains the best results on four benchmark datasets. Moreover, there are also two observations about the ranking of comparison methods. First, the clustering results produced by deep fusion and deep clustering methods are better than those produced by FeatConcate, which verifies the capability of the deep methods to model correlations of heterogeneous modalities when fusing complementary knowledge. Second, the deep clustering methods outperform the deep fusion methods in clustering pattern mining, since the deep clustering methods obtain clustering-specific fusion knowledge on multimodal data, rather than the generalized fusion knowledge extracted by the deep fusion methods. In summary, OTMC produces the optimal results compared with ten baseline methods. These observations validate the effectiveness of optimal transport between data and clusters. That is, the modality-specific OT and the modality-common OT grounded in the Wasserstein distance can effectively measure divergences hidden in heterogeneous distributions of modalities. In particular, in comparison with FeatConcate, which relies on the strong-notion Euclidean distance, OTMC achieves more improvements on Handwritten than the other three datasets. The potential reason may be the differences in heterogeneous manifold overlaps, i.e., the heterogeneous manifolds in Handwritten have fewer overlaps than those in the other three datasets, which experience feature engineering to enhance the intersection.

Figure 2 shows the results of the Nemenyi tests on the four datasets. It can be seen that the average rank of OTMC is higher than that of the other methods on all four datasets, which statistically indicates the superiority of OTMC.

5.3. Further Evaluation

Ablation analysis. To investigate the effectiveness of each component of OTMC, an ablation analysis is conducted on Handwritten. As shown in

Table 4. Ablation analysis on Handwritten in terms of ACC, NMI, and ARI. Best results are marked in bold.

Methods	ACC	NMI	ARI
modality-specific OT 1	0.7215	0.7276	0.6212
modality-specific OT 2	0.7255	0.7092	0.6141
modality-common	0.9090	0.8306	0.8098
OTMC	0.9215	0.8496	0.8335

, modality-specific OT produces the lowest results, which are similar in two modalities, due to the limited information of each modality. Modality-common OT produces higher results than modality-specific OT in two modalities, because it can correct the inconsistencies based on modality-common knowledge when transporting common information of modalities. OTMC produces the optimal results, since OT can fuse the private and shared information of modalities. The above observations validate the effectiveness of the collaboration between modality-specific OT and modality-common OT in the pattern mining of multimodal data.

Convergence analysis. To verify the convergence of OTMC, Figure 3 shows the losses on four datasets in each epoch of training. In the 0–10 epoch, all of the losses on the four datasets decrease rapidly. Then, in the 11–20 epoch, the speed of decrease becomes slow. And in the 21–30 epoch, the change in loss tends to zero. In general, OTMC can converge after 30-epoch training (black dotted line in Figure 3).

6. Conclusions

A multimodal clustering method based on optimal transport (OTMC) is designed in this paper to mine intrinsic patterns from the fusion information of multimodal data. It utilizes the weak-notion distance to measure the differences among heterogeneous manifolds, which overcomes the fragile assumption on strict overlaps between data manifolds in previous MC methods. In particular, multimodal clustering is defined as the transport mapping between multimodal data and clustering prototypes, which effectively fuses the complementary information in multimodal data. Then, the variational solution to OT is derived based on a generative transport plan, which utilizes fusion information to produce accurate clustering patterns. Afterwards, a multimodal OT clustering network is designed to achieve the above definition. Finally, extensive experiments illustrate the superiority of OTMC.

The possible future directions of this research are three-fold: First, in current OTMC, the transport plan is completely implemented by a deep neural network. Hence, the performance of OTMC heavily depends on the degree of fitting between the deep neural network and the real transport plan, which introduces uncertainty and may be a potential limitation. In the future, a variational solution with more compact margins will be explored to address this dependence and achieve a more precise map for pattern mining. Second, the performance of OTMC on different types of multimodal datasets, especially those with varying degrees of distribution overlap or complementarity across modalities via feature engineering, will be checked to further verify the theoretical basis of OTMC. Third, OTMC will be tried in other multimodal data learning cases such as graph data classification and time series prediction.