An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning

Abstract

In order to develop machine learning and deep learning models that take into account the guidelines and principles of trustworthy AI, a novel information theoretic approach is introduced in this article. A unified approach to privacy-preserving interpretable and transferable learning is considered for studying and optimizing the trade-offs between the privacy, interpretability, and transferability aspects of trustworthy AI. A variational membership-mapping Bayesian model is used for the analytical approximation of the defined information theoretic measures for privacy leakage, interpretability, and transferability. The approach consists of approximating the information theoretic measures by maximizing a lower-bound using variational optimization. The approach is demonstrated through numerous experiments on benchmark datasets and a real-world biomedical application concerned with the detection of mental stress in individuals using heart rate variability analysis.

1. Introduction

Trust in the development, deployment, and use of AI is essential in order to fully utilize the potential of AI to contribute to human well-being and society. Recent advances in machine and deep learning have rejuvenated the field of AI with an enthusiasm that AI could become an integral part of human life. However, a rapid proliferation of AI will give rise to several ethical, legal, and social issues.

1.1. Trustworthy AI

In response to the ethical, legal, and social challenges that accompany AI, guidelines and ethical principles have been established [1,2,3,4] in order to evaluate the responsible development of AI systems that are good for humanity and the environment. These guidelines have introduced the concept of trustworthy AI (TAI), and the term TAI has quickly gained attention in research and practice. TAI is based on the idea that trust in AI will allow AI to realize its full potential in contributing to societies, economies, and sustainable development. As “trust” is a complex phenomenon being studied in diverse disciplines (i.e., psychology, sociology, economics, management, computer science, and information systems), the definition and realization of TAI remains challenging. While forming trust in technology, users express expectations about the technology’s functionality, helpfulness and reliability [5]. The authors in [6] state that “AI is perceived as trustworthy by its users (e.g., consumers, organizations, and society) when it is developed, deployed, and used in ways that not only ensure its compliance with all relevant laws and its robustness but especially its adherence to general ethical principles”.

Academicians, industries, and policymakers have developed in recent times for TAI several frameworks and guidelines including “Asilomar AI Principles” [7], “Montreal Declaration of Responsible AI” [8], “UK AI Code” [9], “AI4People” [4], “Ethics Guidelines for Trustworthy AI” [1], “OECD Principles on AI” [10], “Governance Principles for the New Generation Artificial Intelligence” [11], and “Guidance for Regulation of Artificial Intelligence Applications” [12]. However, it was argued in [13] that AI ethics lack a reinforcement mechanism, and economic incentives could easily override commitment to ethical principles and values.

The five principles of ethical AI [4] (i.e., beneficence, non-maleficence, autonomy, justice, and explicability) have been adopted for TAI [6]. Beneficence refers to promoting the well-being of humans, preserving dignity, and sustaining the planet. Non-maleficence refers to avoiding bringing harm to people and is especially concerned with the protection of people’s privacy and security. Autonomy refers to the promotion of human autonomy, agency, and oversight including the restriction of AI systems’ autonomy, where necessary. Justice refers to using AI for correcting past wrongs, ensuring shared benefits through AI, and preventing the creation of new harms and inequities by AI. Explicability comprises an epistemological sense and an ethical sense. Explicability refers in the epistemological sense to the explainable AI developed by creating interpretable AI models with high levels of performance and accuracy. In the ethical sense, explicability refers to accountable AI.

1.2. Motivation and Novelty

The core issues related to machine and deep learning that need to be addressed in order to fulfill the five principles of trustworthy AI are listed in Table 1.

Table 1. Core issues with TAI principles and solution approaches.

TAI Principle	Issue	Solution Approach
Beneficence	I1: $\begin{array}{c}\mathrm{non-availability\; of\; large}\hfill \\ \mathrm{high-quality\; training\; data}\hfill \end{array}$	transfer learning
	I2: $\begin{array}{c}\mathrm{models\; (intellectual\; properties)}\hfill \\ \mathrm{are\; not\; widely\; available}\hfill \end{array}$	federated learning
Non-maleficence	I3: $\begin{array}{c}\mathrm{leakage\; of\; private\; information}\hfill \\ \mathrm{embedded\; in\; training\; data}\hfill \end{array}$	$\begin{array}{c}\mathrm{privacy-preserving}\hfill \\ \mathrm{data\; release\; mechanism}\hfill \end{array}$
	I4: $\begin{array}{c}\mathrm{leakage\; of\; private\; information}\hfill \\ \mathrm{embedded\; in\; model\; parameters}\hfill \\ \mathrm{and\; model\; outputs}\hfill \end{array}$	$\begin{array}{c}\mathrm{privacy-preserving}\hfill \\ \mathrm{machine\; and\; deep\; learning}\hfill \end{array}$
Autonomy	I5: $\begin{array}{c}\mathrm{user}’\mathrm{s\; inability\; to\; quantify}\hfill \\ \mathrm{model\; uncertainties\; lead\; to}\hfill \\ \mathrm{indecisiveness\; regarding\; the\; level}\hfill \\ \mathrm{of\; autonomy\; given\; to\; AI\; system}\hfill \end{array}$	$\begin{array}{c}\mathrm{analytical\; quantification\; of}\hfill \\ \mathrm{model\; uncertainties}\hfill \end{array}$
Justice	I6: $\begin{array}{c}\mathrm{bias\; of\; training\; data}\hfill \\ \mathrm{toward\; certain\; groups\; of\; people}\hfill \\ \mathrm{leads\; to\; discrimination}\hfill \end{array}$	federated learning
Explicability	I7: $\begin{array}{c}\mathrm{user}’\mathrm{s\; inability\; to\; understand}\hfill \\ \mathrm{model\; functionality\; leads}\hfill \\ \mathrm{to\; mistrust\; and\; obstruction}\hfill \\ \mathrm{in\; establishing\; accountability}\hfill \end{array}$	$\begin{array}{c}\mathrm{interpretable\; machine\; and}\hfill \\ \mathrm{deep\; learning\; models}\hfill \end{array}$

Solution approaches to address issues concerning TAI have been identified in Table 1; however, a unified solution approach addressing all major issues does not exist. Despite the importance of the outlined TAI principles, their major limitation, as identified in [6], concerns the fact that the principles are highly general and provide little to no guidance for how they can be transferred into practice. To address this limitation, a data-driven research framework for TAI was outlined in [6]. However, to the best knowledge of the authors, no previous study presented a unified information theoretic approach to study the privacy, interpretability, and transferability aspects of trustworthy AI in a rigorous analytical manner. This motivated us in this study to develop a novel information theoretic approach for addressing the privacy, interpretability, and transferability aspects of trustworthy AI in a rigorous analytical manner. This study introduces a unified information theoretic approach to “privacy-preserving interpretable and transferable learning”, as represented in Figure 1, for addressing trustworthy AI issues, which is the novelty of this study.

Figure 1. An information theoretic unified approach to “privacy-preserving interpretable and transferable learning” for studying the privacy–interpretability–transferability trade-offs while addressing beneficence, non-maleficence, autonomy, justice, and explicability principles of TAI.

1.3. Goal and Aims

Our goal is to develop a novel approach to trustworthy AI based on the hypothesis that information theory enables taking into account the privacy, interpretability, and transferability aspects of trustworthy AI principles during the development of machine learning and deep learning models by providing a way to study and optimize the inherent trade-offs. The aims focused on the development of our approach are the following:

Aim 1:

To develop an information theoretic approach to privacy that enables the quantification of privacy leakage in terms of the mutual information between sensitive private data and the data released to the public without the availability of prior knowledge about data statistics (such as joint distributions of public and private variables).

Aim 2:

To develop an information theoretic criterion for evaluating the interpretability of a machine learning model in terms of the mutual information between non-interpretable model outputs/activations and corresponding interpretable parameters.

Aim 3:

To develop an information theoretic criterion for evaluating the transferability (of a machine learning model from source to target domain) in terms of the mutual information between source domain model outputs/activations and target domain model outputs/activations.

Aim 4:

To develop analytical approaches to machine and deep learning allowing for the quantification of model uncertainties.

Aim 5:

To develop a unified approach to “privacy-preserving interpretable and transferable learning” for an analytical optimization of privacy–interpretability–transferability trade-offs.

1.4. Methodology

Figure 2 outlines the methodological workflow. For an information theoretic evaluation of the privacy leakage, interpretability, and transferability, we provide a novel method that consists of following three steps:

Figure 2. The proposed methodology to evaluate privacy leakage, interpretability, and transferability in terms of the information leakages.

1.4.1. Defining Measures in Terms of the Information Leakages

The privacy, interpretability, and transferability measures are defined in terms of the information leakages:

• Privacy leakage is measured as the amount of information about private/sensitive variables leaked by the shared variables;
• Interpretability is measured as the amount of information about interpretable parameters leaked by the model;
• Transferability is measured as the amount of information about the source domain model output leaked by the target domain model output.

1.4.2. Variational Membership Mapping Bayesian Models

In order to derive analytical expressions for the defined privacy leakage, interpretability, and transferability measures, the stochastic inverse models (governing the relationships amongst variables) will be required. In this study, the variational membership mappings are leveraged to build the required stochastic inverse models. Membership mappings [14,15] have been introduced as an alternative to deep neural networks in order to address the issues such as determining the optimal model structure, smaller training dataset, and iterative time-consuming nature of numerical learning algorithms [16,17,18,19,20,21,22]. A membership mapping represents data through a fuzzy set (characterized by a membership function such that the dimension of the membership function increases with an increasing data size). A remarkable feature of membership mappings is that these allow an analytical approach to the variational learning of a membership-mappings-based data representation model. Our idea is to employ membership mappings for defining a stochastic inverse model, which is then inferred using variational Bayesian methodology.

1.4.3. Variational Approximation of Information Theoretic Measures

The variational membership-mapping Bayesian models are used to determine the lower bounds on the defined information theoretic measures for privacy leakage, interpretability, and transferability. The lower bounds are then maximized using variational optimization methodology to derive analytically the expressions that approximate the privacy leakage, interpretability, and transferability measures. The analytically derived expressions form the basis of an algorithm that practically computes the measures using available data samples, where expectations over unknown distributions are approximated by sample averages.

1.5. Contributions

The main contributions of this study are following:

1.5.1. A Unified Approach to Study the Privacy, Interpretability, and Transferability Aspects of Trustworthy AI

The study introduces a novel information theoretic unified approach (as represented in Figure 1) to address the

• Issues I1 and I2 of beneficence principle by means of transfer and federated learning;
• Issues I3 and I4 of non-maleficence principle by means of privacy-preserving data release mechanisms;
• Issue I5 of autonomy principle by means of analytical machine and deep learning algorithms that enable the user to quantify model uncertainties and hence to decide the level of autonomy given to AI systems;
• Issue I6 of justice principle by means of federated learning;
• Issue I7 of explicability principle by means of interpretable machine and deep learning models.

1.5.2. Information Theoretic Quantification of Privacy, Interpretability, and Transferability

The most important feature of our approach is that the notions of privacy, interpretability, and transferability are quantified by information theoretic measures allowing for the study and optimization of trade-offs (such as trade-off between privacy and transferability or trade-off between privacy and interpretability) in a practical manner.

1.5.3. Computation of Information Theoretic Measures without Requiring the Knowledge of Data Distributions

It is possible to derive analytical expressions for the defined measures provided that the knowledge regarding the data distributions is available. However, in practice, the data distributions are unknown, and thus, a way to approximate the defined measures is required. Therefore, a novel method that employs recently introduced membership mappings [14,15,16,17,18,19,20,21,22], is presented for approximating the defined privacy leakage, interpretability, and transferability measures. The method relies on inferring a variational Bayesian model that facilitates an analytical approximation of the information theoretic measures through variational optimization methodology. A computational algorithm is provided for practically calculating the privacy leakage, interpretability, and transferability measures. Finally, an algorithm is presented that provides

• Information theoretic evaluation of privacy leakage, interpretability, and transferability in a semi-supervised transfer and multi-task learning scenario;
• An adversary model for estimating private data and for simulating privacy attacks; and
• An interpretability model for estimating interpretable parameters and for providing an interpretation to the non-interpretable data vectors.

1.6. Organization

This text is organized into sections. The proposed methodology relies on the membership mappings for data representation learning. Therefore, Section 2 has been dedicated to the review of membership mappings. An application of membership mappings to solve an inverse modeling problem by developing a variational membership-mapping Bayesian model is considered in Section 3. Section 4 presents the most important result of this study on the variational approximation of information leakage and development of a computational algorithm for calculating information leakage. The measures for privacy leakage, interpretability, and transferability are formally introduced in Section 5. Section 5 further provides an algorithm to study the privacy, interpretability, and transferability aspects in a unified manner. The application of proposed measures to study the trade-offs is also demonstrated through the experiments made on the widely used MNIST and “Office+Caltech256” datasets in Section 6. Section 6 further considers a biomedical application concerned with the detection of mental stress in individuals using heart rate variability analysis. Finally, the concluding remarks are provided in Section 7.

2. Mathematical Background

This section reviews the membership mappings and transferable deep learning from [14,15,22]. For a detailed mathematical study of the concepts used in this section, the readers are referred to previous works [14,15,22].

2.1. Notations

• Let $n,N,p,M\in \mathbb{N}$.
• Let $\mathcal{B}\left({\mathbb{R}}^N\right)$ denote the Borel $\sigma -$ algebra on ${\mathbb{R}}^N$, and let ${\lambda }^N$ denote the Lebesgue measure on $\mathcal{B}\left({\mathbb{R}}^N\right)$.
• Let $(\mathcal{X},\mathcal{A},\rho )$ be a probability space with unknown probability measure $\rho$.
• Let us denote by $\mathcal{S}$ the set of finite samples of data points drawn i.i.d. from $\rho$, i.e.,

  $$\begin{array}{ccc}\hfill \mathcal{S}& :=\hfill & \hfill \left\{{(x^i\sim \rho )}_{i=1}^N\right|N\in \mathbb{N}\}.\end{array}$$

  (1)
• For a sequence $\mathrm{x}=(x^1,\cdots ,x^N)\in \mathcal{S}$, let $\left|\mathrm{x}\right|$ denote the cardinality, i.e., $\left|\mathrm{x}\right|=N$.
• If $\mathrm{x}=(x^1,\cdots ,x^N),\mathrm{a}=(a^1,\cdots ,a^M)\in \mathcal{S}$, then $\mathrm{x}\wedge \mathrm{a}$ denotes the concatenation of the sequences $\mathrm{x}$ and $\mathrm{a}$, i.e., $\mathrm{x}\wedge \mathrm{a}=(x^1,\dots ,x^N,a^1,\dots ,a^M)$.
• Let us denote by $\mathbb{F}\left(\mathcal{X}\right)$ the set of $\mathcal{A}$-$\mathcal{B}\left(\mathbb{R}\right)$ measurable functions $f:\mathcal{X}\to \mathbb{R}$, i.e.,

  $$\begin{array}{ccc}\hfill \mathbb{F}\left(\mathcal{X}\right)& :=\hfill & \hfill \{f:\mathcal{X}\to \mathbb{R}|fis\mathcal{A}-\mathcal{B}\left(\mathbb{R}\right)measurable\}.\end{array}$$

  (2)
• For convenience, the values of a function $f\in \mathbb{F}\left(\mathcal{X}\right)$ at points in the collection $\mathrm{x}=(x^1,\cdots ,x^N)$ are represented as $f\left(\mathrm{x}\right)=(f\left(x^1\right),\cdots ,f\left(x^N\right))$.
• Let ${\zeta }_{\mathrm{x}}:{\mathbb{R}}^{\left|\mathrm{x}\right|}\to [0,1]$ be a membership function satisfying the following properties:

  Nowhere Vanishing:

  ${\zeta }_{\mathrm{x}}\left(\mathrm{y}\right)>0$ for all $\mathrm{y}\in {\mathbb{R}}^{\left|\mathrm{x}\right|}$, i.e.,

  $$\begin{array}{ccc}\hfill \mathrm{supp}\left[{\zeta }_{\mathrm{x}}\right]& =\hfill & \hfill {\mathbb{R}}^{\left|\mathrm{x}\right|}.\end{array}$$

  (3)

  Positive and Bounded Integrals:

  The functions ${\zeta }_{\mathrm{x}}$ are absolutely continuous and Lebesgue integrable over the whole domain such that for all $\mathrm{x}\in \mathcal{S}$, we have

  $$\begin{array}{c}\hfill 0<{\int }_{{\mathbb{R}}^{\left|\mathrm{x}\right|}}{\zeta }_{\mathrm{x}}\mathrm{d}{\lambda }^{\left|\mathrm{x}\right|}<\infty .\end{array}$$

  (4)

  Consistency of Induced Probability Measure:

  The membership function induced probability measures ${\mathbb{P}}_{{\zeta }_{\mathrm{x}}}$, defined on any $A\in \mathcal{B}\left({\mathbb{R}}^{\left|\mathrm{x}\right|}\right)$, as

  $$\begin{array}{ccc}\hfill {\mathbb{P}}_{{\zeta }_{\mathrm{x}}}\left(A\right)& :=\hfill & \hfill \frac{1}{{\int }_{{\mathbb{R}}^{\left|\mathrm{x}\right|}}{\zeta }_{\mathrm{x}}\mathrm{d}{\lambda }^{\left|\mathrm{x}\right|}}{\int }_A{\zeta }_{\mathrm{x}}\mathrm{d}{\lambda }^{\left|\mathrm{x}\right|}\end{array}$$

  (5)

  are consistent in the sense that for all $\mathrm{x},\mathrm{a}\in \mathcal{S}$:

  $$\begin{array}{ccc}\hfill {\mathbb{P}}_{{\zeta }_{\mathrm{x}\wedge \mathrm{a}}}(A\times {\mathbb{R}}^{\left|\mathrm{a}\right|})& =\hfill & \hfill {\mathbb{P}}_{{\zeta }_{\mathrm{x}}}\left(A\right).\end{array}$$

  (6)

  The collection of membership functions satisfying the aforementioned assumptions is denoted by

  $$\begin{array}{ccc}\hfill \mathsf{\Theta }& :=\hfill & \hfill \{{\zeta }_{\mathrm{x}}:{\mathbb{R}}^{\left|\mathrm{x}\right|}\to [0,1]|\left(3\right),\left(4\right),\left(6\right),\mathrm{x}\in \mathcal{S}\}.\end{array}$$

  (7)

2.2. Review of Variational Membership Mappings

Definition 1

(Student-t Membership Mapping [14]). A Student-t membership-mapping, $\mathcal{F}\in \mathbb{F}\left(\mathcal{X}\right)$, is a mapping with input space $\mathcal{X}={\mathbb{R}}^n$ and a membership function ${\mathsf{\zeta }}_{\mathrm{x}}\in \Theta$ that is Student-t like:

$$\begin{array}{ccc}\hfill {\zeta }_{\mathrm{x}}\left(\mathrm{y}\right)& =\hfill & \hfill {\left(1+1/(\nu -2){\left(\mathrm{y}-{\mathrm{m}}_{\mathrm{y}}\right)}^T{K}_{\mathrm{x}\mathrm{x}}^{-1}\left(\mathrm{y}-{\mathrm{m}}_{\mathrm{y}}\right)\right)}^{-\frac{\nu +\left|\mathrm{x}\right|}{2}}\end{array}$$

(8)

where $\mathrm{x}\in \mathcal{S}$, $\mathrm{y}\in {\mathbb{R}}^{\left|\mathrm{x}\right|}$, $\nu \in {\mathbb{R}}_{+}\setminus [0,2]$ is the degrees of freedom, ${\mathrm{m}}_{\mathrm{y}}\in {\mathbb{R}}^{\left|\mathrm{x}\right|}$ is the mean vector, and $K_{\mathrm{x}\mathrm{x}}\in {\mathbb{R}}^{\left|\mathrm{x}\right|\times \left|\mathrm{x}\right|}$ is the covariance matrix with its $(i,j)-$th element given as

$$\begin{array}{ccc}\hfill {\left(K_{\mathrm{x}\mathrm{x}}\right)}_{i,j}& =\hfill & \hfill kr(x^i,x^j)\end{array}$$

(9)

where $kr:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R}$ is a positive definite kernel function defined as

$$\begin{array}{ccc}\hfill kr(x^i,x^j)& =\hfill & \hfill {\sigma }^{2}exp\left(-0.5\sum _{k=1}^n{w}_k{\left|x_k^i-x_k^j\right|}^2\right)\end{array}$$

(10)

where $x_k^i$ is the k-th element of $x^i$, ${\sigma }^2$ is the variance parameter, and $w_k\ge 0$ (for $k\in \{1,\cdots ,n\}$).

Given a dataset $\left\{(x^i,y^i)\right|x^i\in {\mathbb{R}}^n,y^i\in {\mathbb{R}}^p,i\in \{1,\cdots ,N\}\}$, it is assumed that there exist zero-mean Student-t membership mappings ${\mathcal{F}}_1,\cdots ,{\mathcal{F}}_p\in \mathbb{F}\left({\mathbb{R}}^n\right)$ such that

$$\begin{array}{ccc}\hfill y^i& \approx \hfill & \hfill {\left[\begin{array}{ccc}\hfill {\mathcal{F}}_1\left(x^i\right)& \cdots & {\mathcal{F}}_p\left(x^i\right)\hfill \end{array}\right]}^T.\end{array}$$

(11)

Under modeling scenario (11), [22] presents an algorithm (stated as Algorithm 1) for the variational learning of membership mappings.

Algorithm 1 Variational learning of the membership mappings [22]

Require:  Dataset $\left\{(x^i,y^i)|x^i\in {\mathbb{R}}^n,y^i\in {\mathbb{R}}^p,i\in \{1,\cdots ,N\}\right\}$ and maximum possible number of auxiliary points $M_{max}\in {\mathbb{Z}}_{+}$ with $M_{max}\le N$.   1: Choose $\nu$ and $w=(w_1,\cdots ,w_n)$ as in (12) and (14), respectively.   2: Choose a small positive value $\kappa ={10}^{-1}$.   3: Set iteration count $it=0$ and ${M|}_0=M_{max}$.   4: while $\tau (M{|}_{it},1)<\kappa$ do   5:     ${M|}_{it+1}=\lceil 0.9M{|}_{it}\rceil$   6:     $it\leftarrow it+1$   7: end while   8: Set ${M=M|}_{it}$.   9: if $\tau (M,1)\ge \frac{1}{p}{\sum }_{j=1}^p\mathrm{var}\left(y_j^1,\cdots ,y_j^N\right)$ then 10:     ${\sigma }^2=1$ 11: else 12:     ${\sigma }^2=\frac{1}{\tau (M,1)}\frac{1}{p}{\sum }_{j=1}^p\mathrm{var}\left(y_j^1,\cdots ,y_j^N\right)$ 13: end if 14: Compute $\mathrm{a}={\left\{a^m\right\}}_{m=1}^M$ using (13), $K_{\mathrm{x}\mathrm{x}}$ using (9), $K_{\mathrm{a}\mathrm{a}}$ using (15), and $K_{\mathrm{x}\mathrm{a}}$ using (16). 15: Set $\beta =1$. 16: repeat 17:     Compute $\alpha$ using (18). 18:     Update the value of $\beta$ using (19). 19: until ($\beta$ nearly converges) 20: Compute $\alpha$ using (18). 21: return the parameters set $\mathbb{M}=\{\alpha ,\mathrm{a},M,\sigma ,w\}$.

With reference to Algorithm 1, we have following:

• The degrees of freedom associated to the Student-t membership mapping $\nu \in {\mathbb{R}}_{+}\setminus [0,2]$ is chosen as

  $$\begin{array}{ccc}\hfill \nu & =\hfill & \hfill 2.1\end{array}$$

  (12)
• The auxiliary inducing points are suggested to be chosen as the cluster centroids:

  $$\begin{array}{c}\hfill \mathrm{a}={\left\{a^m\right\}}_{m=1}^M=cluster\_centroid({\left\{x^i\right\}}_{i=1}^N,M)\end{array}$$

  (13)

  where $cluster\_centroid({\left\{x^i\right\}}_{i=1}^N,M)$ represents the k-means clustering on ${\left\{x^i\right\}}_{i=1}^N$.
• The parameters $(w_1,\cdots ,w_n)$ for kernel function (10) are chosen such that $w_k$ (for $k\in \{1,2,\cdots ,n\}$) is given as

  $$\begin{array}{ccc}\hfill w_k& =\hfill & \hfill {\left(\underset{1\le i\le N}{max}\left(x_k^i\right)-\underset{1\le i\le N}{min}\left(x_k^i\right)\right)}^{-2}\end{array}$$

  (14)

  where $x_k^i$ is the k-th element of vector $x^i\in {\mathbb{R}}^n$.
• $K_{\mathrm{a}\mathrm{a}}\in {\mathbb{R}}^{M\times M}$ and $K_{\mathrm{x}\mathrm{a}}\in {\mathbb{R}}^{N\times M}$ are matrices with their $(i,j)$-th elements given as

  $$\begin{array}{ccc}\hfill {\left(K_{\mathrm{a}\mathrm{a}}\right)}_{i,j}& =\hfill & \hfill kr(a^i,a^j)\end{array}$$

  (15)

  $$\begin{array}{ccc}\hfill {\left(K_{\mathrm{x}\mathrm{a}}\right)}_{i,j}& =\hfill & \hfill kr(x^i,a^j)\end{array}$$

  (16)

  where $kr:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R}$ is a positive definite kernel function defined as in (10).
• The scalar-valued function $\tau (M,{\sigma }^2)$ is defined as

  $$\begin{array}{ccc}\hfill \tau (M,{\sigma }^2)& :=\hfill & \hfill \frac{Tr\left(K_{\mathrm{x}\mathrm{x}}\right)-Tr\left({\left(K_{\mathrm{a}\mathrm{a}}\right)}^{-1}{K}_{\mathrm{x}\mathrm{a}}^T{K}_{\mathrm{x}\mathrm{a}}\right)}{\nu +M-2}\end{array}$$

  (17)

  where $\mathrm{a}$ is given by (13), $\nu$ is given by (12), and parameters $(w_1,\cdots ,w_n)$ (which are required to evaluate the kernel function for computing matrices $K_{\mathrm{x}\mathrm{x}}$, $K_{\mathrm{a}\mathrm{a}}$, and $K_{\mathrm{x}\mathrm{a}}$) are given by (14).
• $\alpha =\left[\begin{array}{ccc}\hfill {\alpha }_1& \cdots & {\alpha }_p\hfill \end{array}\right]\in {\mathbb{R}}^{M\times p}$ is a matrix with its j-th column defined as

  $$\begin{array}{ccc}\hfill {\alpha }_j& :=\hfill & \hfill {\left(K_{\mathrm{x}\mathrm{a}}^T{K}_{\mathrm{x}\mathrm{a}}+\frac{Tr\left(K_{\mathrm{x}\mathrm{x}}\right)-Tr\left({\left(K_{\mathrm{a}\mathrm{a}}\right)}^{-1}{K}_{\mathrm{x}\mathrm{a}}^T{K}_{\mathrm{x}\mathrm{a}}\right)}{\nu +M-2}{K}_{\mathrm{a}\mathrm{a}}+\frac{K_{\mathrm{a}\mathrm{a}}}{\beta }\right)}^{-1}{\left(K_{\mathrm{x}\mathrm{a}}\right)}^T{\mathrm{y}}_j\end{array}$$

  (18)
• The disturbance precision value $\beta$ is iteratively estimated as

  $$\begin{array}{ccc}\hfill \frac{1}{\beta }& =\hfill & \hfill \frac{1}{pN}\sum _{j=1}^p\sum _{i=1}^N{\left|y_j^i-\widehat{{\mathcal{F}}_j\left(x^i\right)}\right|}^2\end{array}$$

  (19)

  where $\widehat{{\mathcal{F}}_j\left(x^i\right)}$ is the estimated membership-mapping output given as

  $$\begin{array}{ccc}\hfill \widehat{{\mathcal{F}}_j\left(x^i\right)}& =\hfill & \hfill \left(G\left(x^i\right)\right){\alpha }_j.\end{array}$$

  (20)
  Here, $G\left(x\right)\in {\mathbb{R}}^{1\times M}$ is a vector-valued function defined as

  $$\begin{array}{ccc}\hfill G\left(x\right)& :=\hfill & \hfill \left[\begin{array}{ccc}\hfill kr(x,a^1)& \cdots & kr(x,a^M)\hfill \end{array}\right]\end{array}$$

  (21)

  where $kr:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R}$ is defined as in (10).

Definition 2

(Membership-Mappings Prediction [22]). Given the parameters set $\mathbb{M}=\{\alpha ,\mathrm{a},M,$$\sigma ,w\}$ returned by Algorithm 1, the learned membership mappings could be used to predict output corresponding to any arbitrary input data point $x\in {\mathbb{R}}^n$ as

$$\begin{array}{ccc}\hfill \widehat{y}(x;\mathbb{M})& =\hfill & \hfill {\alpha }^T{\left(G\left(x\right)\right)}^T\end{array}$$

(22)

where $G(·)\in {\mathbb{R}}^{1\times M}$ is a vector-valued function (21).

2.3. Review of Membership-Mappings-Based Conditionally Deep Autoencoders

Definition 3

(Membership-Mapping Autoencoder [15]). A membership-mapping autoencoder, $\mathcal{G}:{\mathbb{R}}^p\to {\mathbb{R}}^p$, maps an input vector $y\in {\mathbb{R}}^p$ to $\mathcal{G}\left(y\right)\in {\mathbb{R}}^p$ such that

$$\begin{array}{ccc}\hfill \mathcal{G}\left(y\right)& \stackrel{\underset{\mathrm{def}}{}}{=}\hfill & \hfill {\left[\begin{array}{ccc}\hfill {\mathcal{F}}_1\left(Py\right)& \cdots & {\mathcal{F}}_p\left(Py\right)\hfill \end{array}\right]}^T,\end{array}$$

(23)

where ${\mathcal{F}}_j$ ($j\in \{1,2,\cdots ,p\}$) is a Student-t membership-mapping, $P\in {\mathbb{R}}^{n\times p}(n\le p)$ is a matrix such that the product $Py$ is a lower-dimensional encoding for y.

Definition 4

(Conditionally Deep Membership-Mapping Autoencoder (CDMMA) [15,22]). A conditionally deep membership-mapping autoencoder, $\mathcal{D}:{\mathbb{R}}^p\to {\mathbb{R}}^p$, maps a vector $y\in {\mathbb{R}}^p$ to $\mathcal{D}\left(y\right)\in {\mathbb{R}}^p$ through a nested composition of finite number of membership-mapping autoencoders such that

$$\begin{array}{ccc}\hfill y^l& =\hfill & \hfill ({\mathcal{G}}_l\circ \cdots \circ {\mathcal{G}}_2\circ {\mathcal{G}}_1)\left(y\right),\forall l\in \{1,2,\cdots ,L\}\end{array}$$

(24)

$$\begin{array}{ccc}\hfill l^{*}& =\hfill & \hfill arg\underset{l\in \{1,2,\cdots ,L\}}{min}{\parallel y-y^l\parallel }^2\end{array}$$

(25)

$$\begin{array}{ccc}\hfill \mathcal{D}\left(y\right)& =\hfill & \hfill y^{l^{*}},\end{array}$$

(26)

where ${\mathcal{G}}_l(·)$ is a membership-mapping autoencoder (Definition 3).

CDMMA discovers layers of increasingly abstract data representation with the lowest-level data features being modeled by the first layer and the highest-level data features being modeled by the end layer [15,22]. An algorithm (stated as Algorithm 2) has been provided in [15,22] for the variational learning of CDMMA.

Algorithm 2 Variational learning of CDMMA [15,22]

Require:  Dataset $\mathbf{Y}=\left\{y^i\in {\mathbb{R}}^p|i\in \{1,\cdots ,N\}\right\}$; the subspace dimension $n\in \{1,2,\cdots ,p\}$; maximum number of auxiliary points $M_{max}\in {\mathbb{Z}}_{+}$ with $M_{max}\le N$; the number of layers $L\in {\mathbb{Z}}_{+}$. 1: for $l=1$ to L do 2:     Set subspace dimension associated to l-th layer as $n_l=max(n-l+1,1)$. 3:     Define $P^l\in {\mathbb{R}}^{n_l\times p}$ such that the i-th row of $P^l$ is equal to the transpose of eigenvector corresponding to the i-th largest eigenvalue of a sample covariance matrix of dataset $\mathbf{Y}$. 4:     Define a latent variable $x^{l,i}\in {\mathbb{R}}^{n_l}$, for $i\in \{1,\cdots ,N\}$, as $$\begin{array}{ccc}\hfill x^{l,i}& :=\hfill & \hfill \left\{\begin{array}{cc}{P}^l{y}^i\hfill & \mathrm{if}l=1,\hfill \\ P^l{\widehat{y}}^{l-1}(x^{l-1,i};{\mathbb{M}}^{l-1})\hfill & \hfill \mathrm{if}l>1\end{array}\right.\end{array}$$ (27) where ${\widehat{y}}^{l-1}$ is the estimated output of the $(l-1)$-th layer computed using (22) for the parameters set ${\mathbb{M}}^{l-1}=\{{\alpha }^{l-1},{\mathrm{a}}^{l-1},M^{l-1},{\sigma }^{l-1},w^{l-1}\}$. 5:     Define $M_{max}^l$ as $$\begin{array}{ccc}\hfill M_{max}^l& :=\hfill & \hfill \left\{\begin{array}{cc}\hfill M_{max}& \mathrm{if}l=1,\hfill \\ \hfill M^{l-1}& \mathrm{if}l>1\hfill \end{array}\right.\end{array}$$ (28) 6:     Compute parameters set ${\mathbb{M}}^l=\{{\alpha }^l,{\mathrm{a}}^l,M^l,{\sigma }^l,w^l\}$, characterizing the membership mappings associated to the l-th layer, using Algorithm 1 on dataset $\left\{(x^{l,i},y^i)|i\in \{1,\cdots ,N\}\right\}$ with the maximum possible number of auxiliary points $M_{max}^l$. 7: end for 8: return the parameters set $\mathcal{M}=\{\{{\mathbb{M}}^1,\cdots ,{\mathbb{M}}^L\},\{P^1,\cdots ,P^L\}\}$.

Definition 5

(CDMMA Filtering [15,22]). Given a CDMMA with its parameters being represented by a set $\mathcal{M}=\{\{{\mathbb{M}}^1,\cdots ,{\mathbb{M}}^L\},\{P^1,\cdots ,P^L\}\}$, the autoencoder can be applied for filtering a given input vector $y\in {\mathbb{R}}^p$ as follows:

$$\begin{array}{ccc}\hfill x^l(y;\mathcal{M})& =\hfill & \hfill \left\{\begin{array}{cc}{P}^{l}y,\hfill & \hfill l=1\\ P^l{\widehat{y}}^{l-1}(x^{l-1};{\mathbb{M}}^{l-1})\hfill & \hfill l\ge 2\end{array}\right.\end{array}$$

(29)

Here, ${\widehat{y}}^{l-1}$ is the output of the $(l-1)$-th layer estimated using (22). Finally, CDMMA’s output, $\mathcal{D}(y;\mathcal{M})$, is given as

$$\begin{array}{ccc}\hfill \widehat{\mathcal{D}}(y;\mathcal{M})& =\hfill & \hfill {\widehat{y}}^{l^{*}}(x^{l^{*}};{\mathbb{M}}^{l^{*}})\end{array}$$

(30)

$$\begin{array}{ccc}\hfill l^{*}& =\hfill & \hfill arg\underset{l\in \{1,\cdots ,L\}}{min}{\parallel y-{\widehat{y}}^l(x^l;{\mathbb{M}}^l)\parallel }^2.\end{array}$$

(31)

For a big dataset, the computational time required by Algorithm 2 for learning will be high. To circumvent the problem of large computation time for processing big data, it is suggested in [15,22] that the data be partitioned into subsets and corresponding to each data subset, a separate CDMMA is learned. This motivates the defining of a wide CDMMA as in Definition 6. For the variational learning of wide CDMMA, Algorithm 3 follows from [15,22], where the choice of number of subsets as $S=\lceil N/1000\rceil$ is driven by the consideration that each subset contains around 1000 data points, since processing the data points up to 1000 by CDMMA is not computationally a challenge.

Definition 6

(A Wide CDMMA [15,22]). A wide CDMMA, $\mathcal{WD}:{\mathbb{R}}^p\to {\mathbb{R}}^p$, maps a vector $y\in {\mathbb{R}}^p$ to $\mathcal{WD}\left(y\right)\in {\mathbb{R}}^p$ through a parallel composition of S ($S\in {\mathcal{Z}}_{+}$) number of CDMMAs such that

$$\begin{array}{ccc}\hfill \mathcal{WD}\left(y\right)& =\hfill & \hfill {\mathcal{D}}_{s^{*}}\left(y\right)\end{array}$$

(32)

$$\begin{array}{ccc}\hfill s^{*}& =\hfill & \hfill arg\underset{s\in \{1,2,\cdots ,S\}}{min}{\parallel y-{\mathcal{D}}_s\left(y\right)\parallel }^2,\end{array}$$

(33)

where ${\mathcal{D}}_s\left(y\right)$ is the output of the s-th CDMMA.

Algorithm 3 Variational learning of wide CDMMA [15,22]

Require:  Dataset $\mathbf{Y}=\left\{y^i\in {\mathbb{R}}^p|i\in \{1,\cdots ,N\}\right\}$; the subspace dimension $n\in \{1,2,\cdots ,p\}$; ratio $r_{max}\in (0,1]$; the number of layers $L\in {\mathbb{Z}}_{+}$. 1: Apply k-means clustering to partition $\mathbf{Y}$ into S subsets, $\{{\mathbf{Y}}^1,\cdots ,{\mathbf{Y}}^S\}$, where $S=\lceil N/1000\rceil$. 2: for $s=1$ to S do 3:     Build a CDMMA, ${\mathcal{M}}^s$, by applying Algorithm 2 on ${\mathbf{Y}}^s$ taking n as the subspace dimension; maximum number of auxiliary points as equal to $r_{max}\times \#{\mathbf{Y}}^s$ (where $\#{\mathbf{Y}}^s$ is the number of data points in ${\mathbf{Y}}^s$); and L is the number of layers. 4: end for 5: return the parameters set $\mathcal{P}={\left\{{\mathcal{M}}^s\right\}}_{s=1}^S$.

Definition 7

(Wide CDMMA Filtering [15,22]). Given a wide CDMMA with its parameters being represented by a set $\mathcal{P}={\left\{{\mathcal{M}}^s\right\}}_{s=1}^S$, the autoencoder can be applied for filtering a given input vector $y\in {\mathbb{R}}^p$ as follows:

$$\begin{array}{ccc}\hfill \widehat{\mathcal{WD}}(y;\mathcal{P})& =\hfill & \hfill \widehat{\mathcal{D}}(y;{\mathcal{M}}^{s^{*}})\end{array}$$

(34)

$$\begin{array}{ccc}\hfill s^{*}& =\hfill & \hfill arg\underset{s\in \{1,2,\cdots ,S\}}{min}{\parallel y-\widehat{\mathcal{D}}(y;{\mathcal{M}}^s)\parallel }^2,\end{array}$$

(35)

where $\widehat{\mathcal{D}}(y;{\mathcal{M}}^s)$ is the output of the s-th CDMMA estimated using (30).

2.4. Membership Mappings for Classification

A classifier (i.e., Definition 8) and an algorithm for its variational learning (stated as Algorithm 4) follows from [15,22].

Definition 8

(A Classifier [15,22]). A classifier, $\mathcal{C}:{\mathbb{R}}^p\to \{1,2,\cdots ,C\}$, maps a vector $y\in {\mathbb{R}}^p$ to $\mathcal{C}\left(y\right)\in \{1,2,\cdots ,C\}$ such that

$$\begin{array}{ccc}\hfill \mathcal{C}(y;{\left\{{\mathcal{P}}_c\right\}}_{c=1}^C)& =\hfill & \hfill arg\underset{c\in \{1,2,\cdots ,C\}}{min}{\parallel y-\widehat{\mathcal{WD}}(y;{\mathcal{P}}_c)\parallel }^2\end{array}$$

(36)

where $\widehat{\mathcal{WD}}(y;{\mathcal{P}}_c)$, computed using (34), is the output of the c-th wide CDMMA. The classifier assigns to an input vector the label of that class whose associated autoencoder best reconstructs the input vector.

Algorithm 4 Variational learning of the classifier [15,22]

Require:  Labeled dataset $\mathbf{Y}=\left\{{\mathbf{Y}}_c|{\mathbf{Y}}_c=\left\{y^{i,c}\in {\mathbb{R}}^p|i\in \{1,\cdots ,N_c\}\right\},c\in \{1,\cdots ,C\}\right\}$; the subspace dimension $n\in \{1,\cdots ,p\}$; ratio $r_{max}\in (0,1]$; the number of layers $L\in {\mathbb{Z}}_{+}$. 1: for $c=1$ to C do 2:     Build a wide CDMMA, ${\mathcal{P}}_c={\left\{{\mathcal{M}}_c^s\right\}}_{s=1}^{S_c}$, by applying Algorithm 3 on ${\mathbf{Y}}_c$ for the given n, $r_{max}$, and L. 3: end for 4: return the parameters set ${\left\{{\mathcal{P}}_c\right\}}_{c=1}^C$.

2.5. Review of Membership-Mappings-Based Privacy-Preserving Transferable Learning

A privacy-preserving semi-supervised transfer and multi-task learning problem has been recently addressed in [22] by means of variational membership mappings. The method, as suggested in [22], involves the following steps:

2.5.1. Optimal Noise Adding Mechanism for Differentially Private Classifiers

The approach suggested in [22] relies on a tailored noise adding mechanism to achieve a given level of differential privacy loss bound with the minimum perturbation of the data. In particularly, Algorithm 5 is suggested for a differentially private approximation of data samples and Algorithm 6 is suggested for building a differentially private classifier.

Algorithm 5 Differentially private approximation of data samples [22]

Require:  Dataset $\mathbf{Y}=\left\{y^i\in {\mathbb{R}}^p|i\in \{1,\cdots ,N\}\right\}$; differential privacy parameters: $d\in {\mathbb{R}}_{+}$, $ϵ\in {\mathbb{R}}_{+}$, $\delta \in (0,1)$. 1: A differentially private approximation of data samples is provided as $$\begin{array}{ccc}\hfill y_j^{+i}& =\hfill & \hfill y_j^i+F_{{\mathrm{v}}_j^i}^{-1}(r_j^i;ϵ,\delta ,d),r_j^i\in (0,1)\end{array}$$ (37) $$\begin{array}{ccc}\hfill F_{{\mathrm{v}}_j^i}^{-1}(r_j^i;ϵ,\delta ,d)& =\hfill & \hfill \left\{\begin{array}{cc}\frac{d}{ϵ}log\left(\frac{2r_j^i}{1-\delta }\right),\hfill & \hfill r_j^i<\frac{1-\delta }{2}\\ 0,\hfill & \hfill r_j^i\in [\frac{1-\delta }{2},\frac{1+\delta }{2}],r_j^i\in (0,1)\\ -\frac{d}{ϵ}log\left(\frac{2(1-r_j^i)}{1-\delta }\right),\hfill & \hfill r_j^i>\frac{1+\delta }{2}\end{array}\right..\end{array}$$ (38) where $y_j^{+i}$ is the j-th element of $y^{+i}\in {\mathbb{R}}^p$. 2: return ${\mathbf{Y}}^{+}=\left\{y^{+i}\in {\mathbb{R}}^p|i\in \{1,\cdots ,N\}\right\}$.

Algorithm 6 Variational learning of a differentially private classifier [22]

Require:  Differentially private approximated dataset: ${\mathbf{Y}}^{+}=\left\{{\mathbf{Y}}_c^{+}|c\in \{1,\cdots ,C\}\right\}$; the subspace dimension $n\in \{1,\cdots ,p\}$; ratio $r_{max}\in (0,1]$; the number of layers $L\in {\mathbb{Z}}_{+}$. 1: Build a classifier, ${\left\{{\mathcal{P}}_c^{+}\right\}}_{c=1}^C$, by applying Algorithm 4 on ${\mathbf{Y}}^{+}$ for the given n, $r_{max}$, and L. 2: return ${\left\{{\mathcal{P}}_c^{+}\right\}}_{c=1}^C$.

2.5.2. Semi-Supervised Transfer Learning Scenario

The aim is to transfer the knowledge extracted by a classifier trained using a source dataset to the classifier of the target domain such that the privacy of the source dataset is preserved. Let ${\left\{{\mathbf{Y}}_c^{sr}\right\}}_{c=1}^C$ be the labeled source dataset where ${\mathbf{Y}}_c^{sr}=\{y_{sr}^{i,c}\in {\mathbb{R}}^{p_{sr}}|i\in \{1,\cdots ,N_c^{sr}\}\}$ represents the c-th labelled samples. The target dataset consist of a few labeled samples ${\left\{{\mathbf{Y}}_c^{tg}\right\}}_{c=1}^C$ (with ${\mathbf{Y}}_c^{tg}=\{y_{tg}^{i,c}\in {\mathbb{R}}^{p_{tg}}|i\in \{1,\cdots ,N_c^{tg}\}\}$) and another set of unlabeled samples ${\mathbf{Y}}_{*}^{tg}=\{y_{tg}^{i,*}\in {\mathbb{R}}^{p_{tg}}|i\in \{1,\cdots ,N_{*}^{tg}\}\}$.

2.5.3. Differentially Private Source Domain Classifier

For a given differential privacy parameters: $d,ϵ,\delta$; Algorithm 5 is applied on ${\mathbf{Y}}_c^{sr}$ to obtain the differentially private approximated data samples, ${\mathbf{Y}}_c^{+sr}=\{y_{sr}^{+i,c}\in {\mathbb{R}}^{p_{sr}}|i\in \{1,\cdots ,N_c^{sr}\}\}$, for all $c\in \{1,\cdots ,C\}$. Algorithm 6 is applied on ${\left\{{\mathbf{Y}}_c^{+sr}\right\}}_{c=1}^C$ to build a differentially private source domain classifier characterized by parameters sets ${\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C$.

2.5.4. Latent Subspace Transformation Matrices

For a given subspace dimension $n_{st}\in \{1,2,\cdots ,min(p_{sr},p_{tg})\}$, the source domain transformation matrix $V^{+sr}\in {\mathbb{R}}^{n_{st}\times p_{sr}}$ is defined as with its i-th row equal to the transpose of the eigenvector corresponding to the i-th largest eigenvalue of the sample covariance matrix computed on differentially private approximated source samples. The target domain transformation matrix $V^{tg}\in {\mathbb{R}}^{n_{st}\times p_{tg}}$ is defined as with its i-th row equal to the transpose of the eigenvector corresponding to the i-th largest eigenvalue of the sample covariance matrix computed on target samples.

2.5.5. Subspace Alignment

A target sample is mapped to source-data-space via following transformation:

$$\begin{array}{ccc}\hfill y_{tg\to sr}\left(y_{tg}\right)& =\hfill & \hfill \left\{\begin{array}{cc}{y}_{tg},\hfill & p_{sr}=p_{tg}\hfill \\ {\left(V^{+sr}\right)}^T{V}^{tg}{y}_{tg},\hfill & p_{sr}\ne p_{tg}\hfill \end{array}\right.\end{array}$$

(39)

Both labeled and unlabeled target datasets are transformed to define the following sets:

$$\begin{array}{ccc}\hfill {\mathbf{Y}}_c^{tg\to sr}& :=\hfill & \hfill \left\{y_{tg\to sr}\left(y_{tg}\right)\right|y_{tg}\in {\mathbf{Y}}_c^{tg}\}\end{array}$$

(40)

$$\begin{array}{ccc}\hfill {\mathbf{Y}}_{*}^{tg\to sr}& :=\hfill & \hfill \left\{y_{tg\to sr}\left(y_{tg}\right)\right|y_{tg}\in {\mathbf{Y}}_{*}^{tg}\}.\end{array}$$

(41)

2.5.6. Target Domain Classifier

The k-th iteration for building the target domain classifier, where $k\in \{1,\cdots ,it\_max\}$, consists of the following updates:

$$\begin{array}{ccc}\hfill {\left\{{\mathcal{P}}_c^{tg}{|}_k\right\}}_{c=1}^C& =\hfill & \hfill \mathrm{Algorithm}4\left({\left\{{\mathbf{Y}}_c^{tg\to sr}\cup {\mathbf{Y}}_{*,c}^{tg\to sr}{|}_{k-1}\right\}}_{c=1}^C{,n|}_k,r_{max},L\right)\end{array}$$

(42)

$$\begin{array}{ccc}\hfill {\mathbf{Y}}_{*,c}^{tg\to sr}{|}_k& =\hfill & \hfill \left\{y_{tg\to sr}^{i,*}\in {\mathbf{Y}}_{*}^{tg\to sr}|\mathcal{C}(y_{tg\to sr}^{i,*};{\left\{{\mathcal{P}}_c^{tg}{|}_k\right\}}_{c=1}^C)=c,i\in \{1,\cdots ,N_{*}^{tg}\}\right\}\end{array}$$

(43)

where $\left\{{n|}_1{,n|}_2,\cdots \right\}$ is a monotonically non-decreasing sequence.

2.5.7. source2target Model

The mapping from source to target domain is learned by means of a variational membership-mappings-based model as in the following:

$$\begin{array}{ccc}\hfill {\mathbb{M}}^{sr\to tg}& =\hfill & \hfill \mathrm{Algorithm}1\left(\mathcal{D},M_{max}\right)\end{array}$$

(44)

$$\begin{array}{ccc}\hfill \mathcal{D}& :=\hfill & \hfill \left\{\left(\widehat{\mathcal{WD}}(y;{\mathcal{P}}_c^{+sr}),y\right)|y\in \left\{{\mathbf{Y}}_c^{tg\to sr}\cup {\mathbf{Y}}_{*,c}^{tg\to sr}{|}_{it\_max}\right\},c\in \left\{1,\cdots ,C\right\}\right\}\end{array}$$

(45)

$$\begin{array}{ccc}\hfill M_{max}& =\hfill & \hfill min(\lceil N^{tg}/2\rceil ,1000)\end{array}$$

(46)

where $N^{tg}=\left|\mathcal{D}\right|$ is the total number of target samples, $\widehat{\mathcal{WD}}(·;·)$ is defined as in (34), ${\mathbf{Y}}_c^{tg\to sr}$ is defined as in (40), and ${\mathbf{Y}}_{*,c}^{tg\to sr}$ is defined as in (43).

2.5.8. Transfer and Multi-Task Learning

Both source and target domain classifiers are combined with the source2target model for predicting the label associated to a target sample $y_{tg\to sr}$ as

$$\begin{array}{ccc}\hfill \widehat{c}(y_{tg\to sr};{\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C,{\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C,{\mathbb{M}}^{sr\to tg})& =\hfill & \hfill arg\underset{c\in \{1,2,\cdots ,C\}}{min}\left\{min\left({∥y_{tg\to sr}-\widehat{\mathcal{WD}}(y_{tg\to sr};{\mathcal{P}}_c^{tg})∥}^2,\right.\right.\\ \hfill & & {∥y_{tg\to sr}-\widehat{y}\left(\widehat{\mathcal{WD}}(y_{tg\to sr};{\mathcal{P}}_c^{+sr});{\mathbb{M}}^{sr\to tg}\right)∥}^2,\hfill \\ \hfill & & \left.\left.{∥y_{tg\to sr}-\widehat{\mathcal{WD}}(y_{tg\to sr};{\mathcal{P}}_c^{+sr})∥}^2\right)\right\}.\hfill \end{array}$$

(47)

where $\widehat{y}\left(·;{\mathbb{M}}^{sr\to tg}\right)$ is the output of the source2target model computed using (22).

3. Variational Membership-Mapping Bayesian Models

We consider the application of membership mappings to solve the inverse modeling problem related to $x=f_{t\to x}\left(t\right)$, where $f_{t\to x}:{\mathbb{R}}^q\to {\mathbb{R}}^n$ is a forward map. Specifically, a membership-mappings model is used to approximate the inverse mapping $f_{t\to x}^{-1}$.

3.1. A Prior Model

Given a dataset: $\left\{(x^i,t^i)\right|i\in \{1,\cdots ,N\}\}$, Algorithm 1 can be used to build a membership-mappings model characterized by a set of parameters, say ${\mathbb{M}}^{x\to t}=\{{\alpha }^{x\to t},\mathrm{a},$$M,\sigma ,w\}$ (where $x\to t$ indicates the mapping from x to t has been approximated by the membership mappings). It follows from (22) that the membership-mappings model predicted output corresponding to an input x is given as

$$\begin{array}{ccc}\hfill \widehat{t}(x;{\mathbb{M}}^{x\to t})& =\hfill & \hfill {\left({\alpha }^{x\to t}\right)}^T{\left(G\left(x\right)\right)}^T\end{array}$$

(48)

where $G(·)\in {\mathbb{R}}^{1\times M}$ is a vector-valued function defined as in (21). The k-th element of $\widehat{t}$ is given as

$$\begin{array}{ccc}\hfill {\widehat{t}}_k(x;{\mathbb{M}}^{x\to t})& =\hfill & \hfill \left(G\left(x\right)\right){\alpha }_k^{x\to t}\end{array}$$

(49)

where ${\alpha }_k^{x\to t}$ is the k-th column of matrix ${\alpha }^{x\to t}$.

Expression (49) allows estimating for any arbitrary x the corresponding t using a membership-mappings model. This motivates introducing the following prior model:

$$\begin{array}{ccc}\hfill t_k& =\hfill & \hfill \left(G\left(x\right)\right){\theta }_k+e_k\end{array}$$

(50)

$$\begin{array}{ccc}\hfill {\theta }_k& \sim \hfill & \hfill \mathcal{N}({\alpha }_k^{x\to t},{\mathsf{\Lambda }}_k^{-1})\end{array}$$

(51)

$$\begin{array}{ccc}\hfill e_k& \sim \hfill & \hfill \mathcal{N}(0,{\gamma }^{-1})\end{array}$$

(52)

$$\begin{array}{ccc}\hfill \gamma & \sim \hfill & \hfill \mathrm{Gamma}(a_{\gamma },b_{\gamma })\end{array}$$

(53)

where $k\in \{1,\cdots ,q\}$; $\mathcal{N}({\alpha }_k^{x\to t},{\mathsf{\Lambda }}_k^{-1})$ is the multivariate normal distribution with mean ${\alpha }_k^{x\to t}$ and covariance ${\mathsf{\Lambda }}_k^{-1}$; and $\mathrm{Gamma}(a_{\gamma },b_{\gamma })$ is the Gamma distribution with shape parameter $a_{\gamma }$ and rate parameter $b_{\gamma }$. The estimation provided by membership-mappings model ${\mathbb{M}}^{x\to t}$ (i.e., (49)) is incorporated by the prior model (50)–(53), since

$$\begin{array}{ccc}\hfill \mathbb{E}\left[t_k\right]& =\hfill & \hfill {\widehat{t}}_k(x;{\mathbb{M}}^{x\to t}).\end{array}$$

(54)

3.2. Variational Bayesian Inference

Given the dataset, $\left\{(x^i\in {\mathbb{R}}^n,t^i\in {\mathbb{R}}^q)\right|i\in \{1,2,\cdots ,N\}\}$, the variational Bayesian method is considered for an inference of the stochastic model (50), with priors as (51)–(53). For all $i\in \{1,\cdots ,N\}$ and $k\in \{1,\cdots ,q\}$, we have

$$\begin{array}{ccc}\hfill t_k^i& =\hfill & \hfill \left(G\left(x^i\right)\right){\theta }_k+e_k^i,\end{array}$$

(55)

where ${\theta }_k\sim \mathcal{N}({\alpha }_k^{x\to t},{\mathsf{\Lambda }}_k^{-1})$ and $e_k^i\sim \mathcal{N}(0,{\gamma }^{-1})$. Define ${\mathrm{t}}_k\in {\mathbb{R}}^N$, ${\mathrm{e}}_k\in {\mathbb{R}}^N$, and $R_{\mathrm{x}}\in {\mathbb{R}}^{N\times M}$ as

$$\begin{array}{ccc}\hfill {\mathrm{t}}_k& =\hfill & \hfill {\left[\begin{array}{ccc}\hfill t_k^1& \cdots & t_k^N\hfill \end{array}\right]}^T\end{array}$$

(56)

$$\begin{array}{ccc}\hfill {\mathrm{e}}_k& =\hfill & \hfill {\left[\begin{array}{ccc}\hfill e_k^1& \cdots & e_k^N\hfill \end{array}\right]}^T\end{array}$$

(57)

$$\begin{array}{ccc}\hfill R_{\mathrm{x}}& =\hfill & \hfill {\left[\begin{array}{ccc}\hfill {\left(G\left(x^1\right)\right)}^T& \cdots & {\left(G\left(x^N\right)\right)}^T\hfill \end{array}\right]}^T.\end{array}$$

(58)

For all $k\in \{1,\cdots ,q\}$, we have

$$\begin{array}{ccc}\hfill {\mathrm{t}}_k& =\hfill & \hfill R_{\mathrm{x}}{\theta }_k+{\mathrm{e}}_k\end{array}$$

(59)

$$\begin{array}{ccc}\hfill p({\theta }_k;{\alpha }_k^{x\to t},{\mathrm{\Lambda }}_k)& =\hfill & \hfill \frac{1}{\sqrt{{\left(2\pi \right)}^M\left|{\left({\mathrm{\Lambda }}_k\right)}^{-1}\right|}}exp\left(-0.5{({\theta }_k-{\alpha }_k^{x\to t})}^T{\mathrm{\Lambda }}_k({\theta }_k-{\alpha }_k^{x\to t})\right)\end{array}$$

(60)

$$\begin{array}{ccc}\hfill p({\mathbf{e}}_k;\gamma )& =\hfill & \hfill \frac{1}{\sqrt{{\left(2\pi \right)}^N{\left(\gamma \right)}^{-N}}}exp\left(-0.5\gamma \parallel {\mathbf{e}}_k{\parallel }^2\right)\end{array}$$

(61)

$$\begin{array}{ccc}\hfill p(\gamma ;a_{\gamma },b_{\gamma })& =\hfill & \hfill \left(b_{\gamma }^{a_{\gamma }}/\mathrm{\Gamma }\left(a_{\gamma }\right)\right){\left(\gamma \right)}^{a_{\gamma }-1}exp(-b_{\gamma }\gamma ).\end{array}$$

(62)

Define the following sets:

$$\begin{array}{ccc}\hfill \mathbf{t}& =\hfill & \hfill \{{\mathrm{t}}_1,\cdots ,{\mathrm{t}}_q\}\end{array}$$

(63)

$$\begin{array}{ccc}\hfill \theta & =\hfill & \hfill \{{\theta }_1,\cdots ,{\theta }_q\}\end{array}$$

(64)

and consider the marginal probability of data $\mathbf{t}$ which is given as

$$\begin{array}{ccc}\hfill p\left(\mathbf{t}\right)& =\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma p(\mathbf{t},\theta ,\gamma ).\end{array}$$

(65)

Let $q(\theta ,\gamma )$ be an arbitrary distribution. The log marginal probability of $\mathbf{t}$ can be expressed as

$$\begin{array}{ccc}\hfill log\left(p\right(\mathbf{t}\left)\right)& =\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(p\right(\mathbf{t}\left)\right)\end{array}$$

(66)

$$\begin{array}{ccc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{p(\mathbf{t},\theta ,\gamma )}{q(\theta ,\gamma )}\right)+\int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{q(\theta ,\gamma )}{p(\theta ,\gamma |\mathbf{t})}\right).\end{array}$$

(67)

Define

$$\begin{array}{ccc}\hfill \mathcal{L}\left(q\right(\theta ,\gamma ),\mathbf{t})& :=\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(p(\mathbf{t},\theta ,\gamma )/q(\theta ,\gamma )\right)\end{array}$$

(68)

to express (66) as

$$\begin{array}{ccc}\hfill log\left(p\right(\mathbf{t}\left)\right)& =\hfill & \hfill \mathcal{L}\left(q\right(\theta ,\gamma ),\mathbf{t})+\mathrm{KL}\left(q\right(\theta ,\gamma )\parallel p(\theta ,\gamma \left|\mathbf{t}\right))\end{array}$$

(69)

where $\mathrm{KL}$ is the Kullback–Leibler divergence of $p(\theta ,\gamma |\mathbf{t})$ from $q(\theta ,\gamma )$ and $\mathcal{L}$, referred to as negative free energy, provides a lower bound on the the logarithmic evidence for the data.

The variational Bayesian approach minimizes the difference (in term of $\mathrm{KL}$ divergence) between variational and true posteriors via analytically maximizing negative free energy $\mathcal{L}$ over variational distributions. However, the analytical derivation requires the following widely used mean-field approximation:

$$\begin{array}{ccc}\hfill q(\theta ,\gamma )& =\hfill & \hfill q\left(\theta \right)q\left(\gamma \right)\end{array}$$

(70)

$$\begin{array}{ccc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill q\left({\theta }_1\right)\cdots q\left({\theta }_q\right)q\left(\gamma \right).\end{array}$$

(71)

Applying the standard variational optimization technique (as in [23,24,25,26,27,28,29]), it can be verified that the optimal variational distributions maximizing $\mathcal{L}$ are as follows:

$$\begin{array}{ccc}\hfill q^{*}\left({\theta }_k\right)& =\hfill & \hfill \frac{1}{\sqrt{{\left(2\pi \right)}^M\left|{\left({\widehat{\mathsf{\Lambda }}}_k\right)}^{-1}\right|}}exp\left(-0.5{({\theta }_k-{\widehat{\mathrm{m}}}_k)}^T{\widehat{\mathsf{\Lambda }}}_k({\theta }_k-{\widehat{\mathrm{m}}}_k)\right)\end{array}$$

(72)

$$\begin{array}{ccc}\hfill q^{*}\left(\gamma \right)& =\hfill & \hfill \left({\left({\widehat{b}}_{\gamma }\right)}^{{\widehat{a}}_{\gamma }}/\mathsf{\Gamma }\left({\widehat{a}}_{\gamma }\right)\right){\left(\gamma \right)}^{{\widehat{a}}_{\gamma }-1}exp(-{\widehat{b}}_{\gamma }\gamma )\end{array}$$

(73)

where the parameters $({\widehat{\mathsf{\Lambda }}}_k,{\widehat{\mathrm{m}}}_k,{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma })$ satisfy the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathsf{\Lambda }}}_k& =\hfill & \hfill {\mathsf{\Lambda }}_k+\left({\widehat{a}}_{\gamma }/{\widehat{b}}_{\gamma }\right){\left(R_{\mathrm{x}}\right)}^T{R}_{\mathrm{x}}\end{array}$$

(74)

$$\begin{array}{ccc}\hfill {\widehat{\mathrm{m}}}_k& =\hfill & \hfill {\left({\widehat{\mathsf{\Lambda }}}_k\right)}^{-1}\left({\mathsf{\Lambda }}_k{\alpha }_k^{x\to t}+\left({\widehat{a}}_{\gamma }/{\widehat{b}}_{\gamma }\right){\left(R_{\mathrm{x}}\right)}^T{\mathrm{t}}_k\right)\end{array}$$

(75)

$$\begin{array}{ccc}\hfill {\widehat{a}}_{\gamma }& =\hfill & \hfill a_{\gamma }+0.5qN\end{array}$$

(76)

$$\begin{array}{ccc}\hfill {\widehat{b}}_{\gamma }& =\hfill & \hfill b_{\gamma }+0.5\sum _{k=1}^q\left\{\parallel {\mathrm{t}}_k-R_{\mathrm{x}}{\widehat{\mathrm{m}}}_k{\parallel }^2+Tr\left({\left({\widehat{\mathsf{\Lambda }}}_k\right)}^{-1}{\left(R_{\mathrm{x}}\right)}^T{R}_{\mathrm{x}}\right)\right\}.\end{array}$$

(77)

Algorithm 7 is suggested for variational Bayesian inference of the model.

Algorithm 7 Variational membership-mapping Bayesian model inference

Require:  Dataset $\left\{(x^i\in {\mathbb{R}}^n,t^i\in {\mathbb{R}}^q)|i\in \{1,\cdots ,N\}\right\}$ and maximum possible number of auxiliary points $M_{max}\in {\mathbb{Z}}_{+}$ with $M_{max}\le N$. 1: Apply Algorithm 1 on the dataset to build a variational membership-mappings model ${\mathbb{M}}^{x\to t}=\{{\alpha }^{x\to t},\mathrm{a},M,\sigma ,w\}$. 2: Choose non-informative priors for covariance matrix, i.e., ${\mathsf{\Lambda }}_k={10}^{-3}{I}_M,\forall k\in \{1,\cdots ,q\}$. 3: Choose non-informative priors for noise variance, i.e., $a_{\gamma }={10}^{-3},b_{\gamma }={10}^{-3}$. 4: Initialize ${\widehat{a}}_{\gamma }/{\widehat{b}}_{\gamma }=1$. 5: repeat 6:     update $\{{\widehat{\mathsf{\Lambda }}}_k,{\widehat{\mathrm{m}}}_k|k\in \{1,\cdots ,q\}\},{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma }$ using (74), (75), (76), (77). 7: until convergence. 8: return the parameters set ${\mathbb{BM}}^{x\to t}=\{\{{\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k|k\in \{1,\cdots ,q\}\},{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma }\}$.

The functionality of Algorithm 7 is as follows:

• Step 1 builds a variational membership-mappings model using Algorithm 1 from previous work [22].
• Algorithm 7 choses at step 2 and 3 relatively non-informative priors.
• The loop between step 5 and step 7 applies variational Bayesian inference to iteratively estimate the parameters of optimal distributions until convergence.

Remark 1

(Computational Complexity). The computational complexity of Algorithm 7 is asymptotically dominated by the computation of inverse of $M\times M$ dimensional matrix ${\widehat{\mathsf{\Lambda }}}_k$ in (75) to calculate ${\widehat{\mathrm{m}}}_k$. Thus, the computational complexity of Algorithm 7 is given as $\mathcal{O}\left(M^3\right)$, where M is the number of auxiliary points.

The optimal distributions determined using Algorithm 7 define the so-called Variational Membership-Mapping Bayesian Model (VMMBM) as stated in Remark 2.

Remark 2

(Variational Membership-Mapping Bayesian Model (VMMBM)). The inverse mapping, $f_{t\to x}^{-1}$, is approximated as

$$\begin{array}{ccc}\hfill t_k& =\hfill & \hfill \left(G\left(x\right)\right){\theta }_k+e_k,\end{array}$$

(78)

$$\begin{array}{ccc}\hfill {\theta }_k& \sim \hfill & \hfill \mathcal{N}({\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k^{-1})\end{array}$$

(79)

$$\begin{array}{ccc}\hfill e_k& \sim \hfill & \hfill \mathcal{N}(0,{\gamma }^{-1})\end{array}$$

(80)

$$\begin{array}{ccc}\hfill \gamma & \sim \hfill & \hfill Gamma({\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma })\end{array}$$

(81)

where $k\in \{1,\cdots ,q\}$ and $({\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k,{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma })$ are returned by Algorithm 7.

Remark 3

(Estimation by VMMBM). Given any $x^{*}$, the variational membership-mapping Bayesian model ${\mathbb{BM}}^{x\to t}$ (returned by Algorithm 7) can be used to estimate corresponding $t^{*}$ (such that $x^{*}=f_{t\to x}\left(t^{*}\right)$) as

$$\begin{array}{ccc}\hfill \tilde{t}(x^{*};{\mathbb{BM}}^{x\to t})& =\hfill & \hfill {\left[\begin{array}{ccc}\hfill \left(G\left(x\right)\right){\widehat{\mathrm{m}}}_1& \cdots & \left(G\left(x\right)\right){\widehat{\mathrm{m}}}_q\hfill \end{array}\right]}^T.\end{array}$$

(82)

4. Evaluation of the Information-Leakage

Consider a scenario where a variable t is related to another variable x through a mapping $f_{t\to x}$ such that $x=f_{t\to x}\left(t\right)$. The mutual information $I(t;x)$ measures the amount of information obtained about variable t through observing variable x. Since $x=f_{t\to x}\left(t\right)$, the entropy $H\left(t\right)$ remains fixed independent of mapping $f_{t\to x}$, and thus, the quantity $I(t;x)-H\left(t\right)$ is a measure of the amount of information about t leaked by the mapping $f_{t\to x}$.

Definition 9

(Information Leakage). Under the scenario that $x=f_{t\to x}\left(t\right)$, a measure of the amount of information about t leaked by the mapping $f_{t\to x}$ is defined as

$$\begin{array}{ccc}\hfill I{L}_{f_{t\to x}}& :=\hfill & \hfill I(t;f_{t\to x}\left(t\right))-H\left(t\right)\end{array}$$

(83)

$$\begin{array}{ccc}& =\hfill & \hfill I(t;x)-H\left(t\right).\end{array}$$

(84)

The quantity $I{L}_{f_{t\to x}}$ is referred to as the information leakage.

This section is dedicated to answering the question: How to calculate the information leakage without knowing data distributions?

4.1. Variational Approximation of the Information Leakage

The mutual information between t and x is given as

$$\begin{array}{ccc}\hfill I(t;x)& =\hfill & \hfill H\left(t\right)-H\left(t\right|x)\end{array}$$

(85)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill H\left(t\right)+\int p(t,x)log\left(p\left(t\right|x)\right)\mathrm{d}t\mathrm{d}x\end{array}$$

(86)

$$\begin{array}{ccc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill H\left(t\right)+{⟨log\left(p\left(t\right|x)\right)⟩}_{p(t,x)}\end{array}$$

(87)

where ${⟨g\left(x\right)⟩}_{p\left(x\right)}$ denotes the expectation of a function of random variable $g\left(x\right)$ with respect to the probability density function $p\left(x\right)$; $H\left(t\right)$ and $H\left(t\right|x)$ are marginal and conditional entropies, respectively. Consider the conditional probability of t which is given as

$$\begin{array}{ccc}\hfill p\left(t\right|x)& =\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma p(\theta ,\gamma ,t|x)\end{array}$$

(88)

where $\theta$ is a set defined as in (63). Let $q(\theta ,\gamma )$ be an arbitrary distribution. The log conditional probability of t can be expressed as

$$\begin{array}{ccc}\hfill log\left(p\right(t\left|x\right))& =\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(p\left(t\right|x)\right)\end{array}$$

(89)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{p(\theta ,\gamma ,t|x)}{p(\theta ,\gamma |t,x)}\right)\end{array}$$

(90)

$$\begin{array}{ccc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{p(\theta ,\gamma ,t|x)}{q(\theta ,\gamma )}\right)+\int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{q(\theta ,\gamma )}{p(\theta ,\gamma |t,x)}\right).\end{array}$$

(91)

Define

$$\begin{array}{ccc}\hfill \mathcal{L}\left(q\right(\theta ,\gamma ),t,x)& :=\hfill & \hfill \int \mathrm{d}\theta \mathrm{d}\gamma q(\theta ,\gamma )log\left(\frac{p(\theta ,\gamma ,t|x)}{q(\theta ,\gamma )}\right)\end{array}$$

(92)

to express (91) as

$$\begin{array}{ccc}\hfill log\left(p\right(t\left|x\right))& =\hfill & \hfill \mathcal{L}\left(q\right(\theta ,\gamma ),t,x)+\mathrm{KL}\left(q\right(\theta ,\gamma )\parallel p(\theta ,\gamma |t,x))\end{array}$$

(93)

where $\mathrm{KL}$ is Kullback–Leibler divergence of $p(\theta ,\gamma |t,x)$ from $q(\theta ,\gamma )$. Using (87),

$$\begin{array}{ccc}\hfill I(t;x)& =\hfill & \hfill H\left(t\right)+{⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}+{⟨\mathrm{KL}\left(q\right(\theta ,\gamma )\parallel p(\theta ,\gamma |t,x))⟩}_{p(t,x)}.\end{array}$$

(94)

That is,

$$\begin{array}{ccc}\hfill I{L}_{f_{t\to x}}& =\hfill & \hfill {⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}+{⟨\mathrm{KL}\left(q\right(\theta ,\gamma )\parallel p(\theta ,\gamma |t,x))⟩}_{p(t,x)}.\end{array}$$

(95)

Since Kullback–Leibler divergence is always non-zero, it follows from (95) that ${⟨\mathcal{L}⟩}_{p(t,x)}$ provides a lower bound on $I{L}_{f_{t\to x}}$ i.e.,

$$\begin{array}{ccc}\hfill I{L}_{f_{t\to x}}& \ge \hfill & \hfill {⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}.\end{array}$$

(96)

Our approach to approximate $I{L}_{f_{t\to x}}$ is to maximize its lower bound with respect to variational distribution $q(\theta ,\gamma )$. That is, we seek to solve

$$\begin{array}{ccc}\hfill {\widehat{IL}}_{f_{t\to x}}& =\hfill & \hfill \underset{q(\theta ,\gamma )}{max}{⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}.\end{array}$$

(97)

Result 1

(Analytical Expression for the Information Leakage). Given the model (78)–(81), ${\widehat{IL}}_{f_{t\to x}}$ is given as

$$\begin{array}{ccc}\hfill {\widehat{IL}}_{f_{t\to x}}& =\hfill & -0.5qlog\left(2\pi \right)+0.5q\left\{ϝ\left({\overline{a}}_{\gamma }\right)-log\left({\overline{b}}_{\gamma }\right)\right\}\hfill \\ \hfill & & -\frac{{\overline{a}}_{\gamma }}{2{\overline{b}}_{\gamma }}\sum _{k=1}^q{⟨|t_k-G\left(x\right){\overline{\mathrm{m}}}_k{|}^2⟩}_{p(t,x)}-\frac{{\overline{a}}_{\gamma }}{2{\overline{b}}_{\gamma }}\sum _{k=1}^q{⟨Tr\left({\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}{\left(G\left(x\right)\right)}^{T}G\left(x\right)\right)⟩}_{p\left(x\right)}\hfill \\ \hfill & & -\frac{1}{2}\sum _{k=1}^q\left\{{({\widehat{\mathrm{m}}}_k-{\overline{\mathrm{m}}}_k)}^T{\widehat{\mathsf{\Lambda }}}_k({\widehat{\mathrm{m}}}_k-{\overline{\mathrm{m}}}_k)+Tr\left({\widehat{\mathsf{\Lambda }}}_k{\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}\right)-log\left(\frac{|{\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}|}{|{\left({\widehat{\mathsf{\Lambda }}}_k\right)}^{-1}|}\right)\right\}+\frac{qM}{2}\hfill \\ \hfill & & \hfill -{\widehat{a}}_{\gamma }log\left({\overline{b}}_{\gamma }/{\widehat{b}}_{\gamma }\right)+log\left(\mathsf{\Gamma }\left({\overline{a}}_{\gamma }\right)/\mathsf{\Gamma }\left({\widehat{a}}_{\gamma }\right)\right)-({\overline{a}}_{\gamma }-{\widehat{a}}_{\gamma })\mathsf{\Psi }\left({\overline{a}}_{\gamma }\right)+({\overline{b}}_{\gamma }-{\widehat{b}}_{\gamma })\left({\overline{a}}_{\gamma }/{\overline{b}}_{\gamma }\right).\end{array}$$

(98)

Here, $ϝ(·)$ is the digamma function and the parameters $({\overline{\mathsf{\Lambda }}}_k,{\overline{\mathrm{m}}}_k,{\overline{a}}_{\gamma },{\overline{b}}_{\gamma })$ satisfy the following:

$$\begin{array}{ccc}\hfill {\overline{\mathsf{\Lambda }}}_k& =\hfill & \hfill {\widehat{\Lambda }}_k+\left({\overline{a}}_{\gamma }/{\overline{b}}_{\gamma }\right){⟨{\left(G\left(x\right)\right)}^{T}G\left(x\right)⟩}_{p\left(x\right)}\end{array}$$

(99)

$$\begin{array}{ccc}\hfill {\overline{\mathrm{m}}}_k& =\hfill & \hfill {\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}\left({\widehat{\mathsf{\Lambda }}}_k{\widehat{\mathrm{m}}}_k+\frac{{\overline{a}}_{\gamma }}{{\overline{b}}_{\gamma }}{⟨{\left(G\left(x\right)\right)}^T{t}_k⟩}_{p(t,x)}\right)\end{array}$$

(100)

$$\begin{array}{ccc}\hfill {\overline{a}}_{\gamma }& =\hfill & \hfill {\widehat{a}}_{\gamma }+0.5q\end{array}$$

(101)

$$\begin{array}{ccc}\hfill {\overline{b}}_{\gamma }& =\hfill & \hfill {\widehat{b}}_{\gamma }+\frac{1}{2}\sum _{k=1}^q{⟨|t_k-G\left(x\right){\overline{\mathrm{m}}}_k{|}^2⟩}_{p(t,x)}+\frac{1}{2}\sum _{k=1}^q{⟨Tr\left({\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}{\left(G\left(x\right)\right)}^{T}G\left(x\right)\right)⟩}_{p\left(x\right)}.\end{array}$$

(102)

Proof of Result 1.

Consider

$$\begin{array}{ccc}\hfill \mathcal{L}\left(q\right(\theta ,\gamma ),t,x)& =\hfill & \hfill {⟨log\left(p\right(t|\theta ,\gamma ,x))⟩}_{q(\theta ,\gamma )}+{⟨log\left(p(\theta ,\gamma )/q(\theta ,\gamma )\right)⟩}_{q(\theta ,\gamma )}.\end{array}$$

(103)

It follows from (78) and (80) that

$$\begin{array}{ccc}\hfill log\left(p\left(t_k\right|{\theta }_k,\gamma ,x)\right)& =\hfill & \hfill -0.5log\left(2\pi \right)+0.5log\left(\gamma \right)-0.5\gamma |t_k-G\left(x\right){\theta }_k{|}^2.\end{array}$$

(104)

Since $t={\left[\begin{array}{ccc}\hfill t_1& \cdots & t_q\hfill \end{array}\right]}^T$, we have

$$\begin{array}{ccc}\hfill log\left(p\right(t|\theta ,\gamma ,x))& =\hfill & \hfill -0.5qlog\left(2\pi \right)+0.5qlog\left(\gamma \right)-0.5\gamma \sum _{k=1}^q{|t_k-G\left(x\right){\theta }_k|}^2.\end{array}$$

(105)

Using (105) and (70)–(71) in (103), we have

$$\begin{array}{ccc}\hfill \mathcal{L}\left(q\right(\theta ,\gamma ),t,x)& =\hfill & \hfill -\frac{q}{2}log\left(2\pi \right)+\frac{q}{2}{⟨log\left(\gamma \right)⟩}_{q\left(\gamma \right)}-\frac{{⟨\gamma ⟩}_{q\left(\gamma \right)}}{2}\sum _{k=1}^q{⟨|t_k-G\left(x\right){\theta }_k{|}^2⟩}_{q\left({\theta }_k\right)}\\ & & \hfill +\sum _{k=1}^q{⟨log\left(\frac{p({\theta }_k;{\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k)}{q\left({\theta }_k\right)}\right)⟩}_{q\left({\theta }_k\right)}+{⟨log\left(\frac{p(\gamma ;a_{\gamma },b_{\gamma })}{q\left(\gamma \right)}\right)⟩}_{q\left(\gamma \right)}.\end{array}$$

(106)

Thus,

$$\begin{array}{ccc}\hfill {⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}& =\hfill & -\frac{q}{2}log\left(2\pi \right)+\frac{q}{2}{⟨log\left(\gamma \right)⟩}_{q\left(\gamma \right)}-\frac{{⟨\gamma ⟩}_{q\left(\gamma \right)}}{2}\sum _{k=1}^q{⟨|t_k{|}^2⟩}_{p\left(t\right)}\hfill \\ \hfill & & -\frac{{⟨\gamma ⟩}_{q\left(\gamma \right)}}{2}\sum _{k=1}^q{⟨{\left({\theta }_k\right)}^T{⟨{\left(G\left(x\right)\right)}^{T}G\left(x\right)⟩}_{p\left(x\right)}{\theta }_k⟩}_{q\left({\theta }_k\right)}+{⟨\gamma ⟩}_{q\left(\gamma \right)}\sum _{k=1}^q{⟨{\left({\theta }_k\right)}^T{⟨{\left(G\left(x\right)\right)}^T{t}_k⟩}_{p(t,x)}⟩}_{q\left({\theta }_k\right)}\hfill \\ & & +\sum _{k=1}^q{⟨log\left(\frac{p({\theta }_k;{\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k)}{q\left({\theta }_k\right)}\right)⟩}_{q\left({\theta }_k\right)}+{⟨log\left(\frac{p(\gamma ;a_{\gamma },b_{\gamma })}{q\left(\gamma \right)}\right)⟩}_{q\left(\gamma \right)}.\hfill \end{array}$$

(107)

Now, ${⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}$ can be maximized with respect to $q\left({\theta }_k\right)$ and $q\left(\gamma \right)$ using variational optimization. It can be seen that optimal distributions maximizing ${⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}$ are given as

$$\begin{array}{ccc}\hfill q^{*}\left({\theta }_k\right)& =\hfill & \hfill \frac{1}{\sqrt{{\left(2\pi \right)}^M\left|{\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}\right|}}exp\left(-0.5{({\theta }_k-{\overline{\mathrm{m}}}_k)}^T{\overline{\mathsf{\Lambda }}}_k({\theta }_k-{\overline{\mathrm{m}}}_k)\right)\end{array}$$

(108)

$$\begin{array}{ccc}\hfill q^{*}\left(\gamma \right)& =\hfill & \hfill \left({\left({\overline{b}}_{\gamma }\right)}^{{\overline{a}}_{\gamma }}/\mathsf{\Gamma }\left({\overline{a}}_{\gamma }\right)\right){\left(\gamma \right)}^{{\overline{a}}_{\gamma }-1}exp(-{\overline{b}}_{\gamma }\gamma )\end{array}$$

(109)

where the parameters $({\overline{\Lambda }}_k,{\overline{\mathrm{m}}}_k,{\overline{a}}_{\gamma },{\overline{b}}_{\gamma })$ satisfy (99)–(102). The maximum attained value of ${⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}$ is given as

$$\begin{array}{ccc}\hfill \underset{q(\theta ,\gamma )}{max}{⟨\mathcal{L}\left(q\right(\theta ,\gamma ),t,x)⟩}_{p(t,x)}& =\hfill & -0.5qlog\left(2\pi \right)+0.5q\left\{ϝ\left({\overline{a}}_{\gamma }\right)-log\left({\overline{b}}_{\gamma }\right)\right\}-\frac{{\overline{a}}_{\gamma }}{2{\overline{b}}_{\gamma }}\sum _{k=1}^q{⟨|t_k-G\left(x\right){\overline{\mathrm{m}}}_k{|}^2⟩}_{p(t,x)}\hfill \\ \hfill & & -\frac{{\overline{a}}_{\gamma }}{2{\overline{b}}_{\gamma }}\sum _{k=1}^q{⟨Tr\left({\left({\overline{\mathsf{\Lambda }}}_k\right)}^{-1}{\left(G\left(x\right)\right)}^{T}G\left(x\right)\right)⟩}_{p\left(x\right)}-\sum _{k=1}^q\mathrm{KL}(q^{*}\left({\theta }_k\right)\parallel p({\theta }_k;{\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k))\hfill \\ \hfill & & -\mathrm{KL}(q^{*}\left(\gamma \right)\parallel p(\gamma ;{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma }))\hfill \end{array}$$

where $ϝ(·)$ is the digamma function. After substituting the maximum value in (97) and calculating Kullback–Leibler divergences, we obtain (98).    □

4.2. An Algorithm for the Computing of Information Leakage

Result 1 forms the basis of Algorithm 8 that computes the information leakage using available data samples.

Algorithm 8 Estimation of information leakage, $I{L}_{f_{t\to x}}=I(t;x)-H\left(t\right)$, using variational approximation

Require:  Dataset $\left\{(x^i\in {\mathbb{R}}^n,t^i\in {\mathbb{R}}^q)|x^i=f_{t\to x}\left(t^i\right),i\in \{1,\cdots ,N\}\right\}$. 1: Apply Algorithm 7 on $\left\{(x^i,t^i)|i\in \{1,\cdots ,N\}\right\}$ with $M_{max}=min(\lceil N/2\rceil ,1000)$ (i.e., constraining the maximum possible number of auxiliary points $M_{max}$ below 1000 for computational efficiency) to obtain variational membership-mappings Bayesian model ${\mathbb{BM}}^{x\to t}=\{\{{\widehat{\mathrm{m}}}_k,{\widehat{\mathsf{\Lambda }}}_k|k\in \{1,\cdots ,q\}\},{\widehat{a}}_{\gamma },{\widehat{b}}_{\gamma }\}$. 2: Initialize $\overline{a}/\overline{b}$, e.g., as $\overline{a}/\overline{b}=\widehat{a}/\widehat{b}$. 3: repeat 4:     Update $\{{\overline{\mathsf{\Lambda }}}_k,{\overline{\mathrm{m}}}_k|k\in \{1,\cdots ,q\}\},\overline{a},\overline{b}$ using (99)-(102) where expectations $<·{>}_{p\left(x\right)}$ and $<·{>}_{p(t,x)}$ are approximated via sample averages. 5: until convergence. 6: Compute ${\widehat{IL}}_{f_{t\to x}}$ using (98) where expectations $<·{>}_{p\left(x\right)}$ and $<·{>}_{p(t,x)}$ are approximated via sample averages. 7: return ${\widehat{IL}}_{f_{t\to x}}$ and the model ${\mathbb{BM}}^{x\to t}$.

The functionality of Algorithm 8 is as follows:

• Step 1 applies Algorithm 7 for the inference of a variational membership-mappings Bayesian model.
• The loop between step 3 and step 5 recursively estimates the parameters ($\{{\overline{\mathsf{\Lambda }}}_k,{\overline{\mathrm{m}}}_k|k\in \{1,\cdots ,q\}\},\overline{a},\overline{b}$) using update rules (99)–(102).
• Step 6 computes the information leakage using (98).

Remark 4

(Computational Complexity). The computational complexity of Algorithm 8 is asymptotically dominated by the computation of inverse of $M\times M$ dimensional matrix ${\overline{\mathsf{\Lambda }}}_k$ in (100) to calculate ${\overline{\mathrm{m}}}_k$. Thus, the computational complexity of Algorithm 8 is given as $\mathcal{O}\left(M^3\right)$, where M is the number of auxiliary points.

Example 1

(Verification of Information Leakage Estimation Algorithm). To demonstrate the effectiveness of Algorithm 8 in estimating information leakage, a scenario is generated where $t\in {\mathbb{R}}^{10}$ and $x\in {\mathbb{R}}^{10}$ are Gaussian distributed such that $x=t+\omega$; $t\sim \mathcal{N}(0,5I_{10})$; $\omega \sim \mathcal{N}(0,\sigma I_{10})$ with $\sigma \in [1,15]$. Since the data distributions in this scenario are known, the information leakage can be theoretically calculated and is given as

$$\begin{array}{ccc}\hfill I{L}_{f_{t\to x}}& =\hfill & \hfill 5log(1+5/\sigma )-0.5log\left(\left|\right(2\pi \mathrm{e}5I_{10}\left)\right|\right).\end{array}$$

For a given value of σ, 1000 samples of t and x were simulated and Algorithm 8 was applied for estimating information leakage. The experiments were carried out at different values of σ ranging from 1 to 15.

Figure 3 compares the plots of estimated and theoretically calculated values of information leakage against σ. A close agreement between the two plots in Figure 3 verifies the effectiveness of Algorithm 8 in estimating information leakage without knowing the data distributions.

Figure 3. A comparison of the estimated information leakage values with the theoretically calculated values.

5. Information Theoretic Measures for Privacy Leakage, Interpretability, and Transferability

5.1. Definitions

To define formally the information theoretic measures for privacy leakage, interpretability, and transferability; a few variables and mappings are introduced in Table 2. Definitions 10–12 provide the mathematical definitions of the information theoretic measures.

Definition 10

(Privacy Leakage). Privacy leakage (by the mapping from private variables to noise-added data vector) is a measure of the amount of information about private/sensitive variable $x_{sr}$ leaked by the mapping $f_{x_{sr}\to y_{sr}^{+}}$ and is defined as

$$\begin{array}{ccc}\hfill I{L}_{f_{x_{sr}\to y_{sr}^{+}}}& :=\hfill & \hfill I\left(x_{sr};f_{x_{sr}\to y_{sr}^{+}}\left(x_{sr}\right)\right)-H\left(x_{sr}\right)\end{array}$$

(110)

$$\begin{array}{ccc}& =\hfill & \hfill I\left(x_{sr};y_{sr}^{+}\right)-H\left(x_{sr}\right).\end{array}$$

(111)

Definition 11

(Interpretability Measure). Interpretability (of noise-added data vector) is measured as the amount of information about interpretable parameters $t_{sr}$ leaked by the mapping $f_{t_{sr}\to y_{sr}^{+}}$ and is defined as

$$\begin{array}{ccc}\hfill I{L}_{f_{t_{sr}\to y_{sr}^{+}}}& :=\hfill & \hfill I\left(t_{sr};f_{t_{sr}\to y_{sr}^{+}}\left(t_{sr}\right)\right)-H\left(t_{sr}\right)\end{array}$$

(112)

$$\begin{array}{ccc}& =\hfill & \hfill I\left(t_{sr};y_{sr}^{+}\right)-H\left(t_{sr}\right).\end{array}$$

(113)

Definition 12

(Transferability Measure). Transferability (from source domain data representation learning models (i.e., ${\mathcal{P}}_1^{+sr},\cdots ,{\mathcal{P}}_C^{+sr}$) to the target domain data representation learning models (i.e., ${\mathcal{P}}_1^{tg},\cdots ,{\mathcal{P}}_C^{tg}$)) is measured as the amount of information about source domain feature vector ${\widehat{y}}_{tg}^{sr}$ leaked by the mapping $f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}$ and is defined as

$$\begin{array}{ccc}\hfill I{L}_{f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}}& :=\hfill & \hfill I\left({\widehat{y}}_{tg}^{sr};f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}\left({\widehat{y}}_{tg}^{sr}\right)\right)-H\left({\widehat{y}}_{tg}^{sr}\right)\end{array}$$

(114)

$$\begin{array}{ccc}& =\hfill & \hfill I\left({\widehat{y}}_{tg}^{sr};{\widehat{y}}_{tg}^{tg}\right)-H\left({\widehat{y}}_{tg}^{sr}\right).\end{array}$$

(115)

Here, ${\widehat{y}}_{tg}^{tg}$ represents the target domain feature vector and $f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}:{\mathbb{R}}^{p_{sr}}\to {\mathbb{R}}^{p_{sr}}$ is the mapping from source domain feature vector ${\widehat{y}}_{tg}^{sr}$ to target domain feature vector ${\widehat{y}}_{tg}^{tg}$.

Table 2. Introduced variables and mappings.

Symbol/Mapping	Definition/Meaning
$x_{sr}\in {\mathbb{R}}^{n_{sr}}$	$\begin{array}{c}\mathrm{vector\; representing\; private}/\mathrm{sensitive\; variables\; associated\; to\; source\; domain}\end{array}$
$y_{sr}\in {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{source\; domain\; data\; vector}\end{array}$
$t_{sr}\in {\mathbb{R}}^q$	$\begin{array}{c}\mathrm{vector\; representing\; the\; set\; of\; interpretable\; parameters\; associated\; to}\\ \mathrm{non-interpretable\; data\; vector} y_{sr}\end{array}$
$y_{sr}^{+}\in {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{noise-added\; data\; vector\; (that\; is\; either\; publicly\; released\; or\; used\; for\; the\; training\; of}-(\\ \mathrm{source\; model)\; obtained\; from} y_{sr} \mathrm{via\; Algorithm} 5\end{array}$
$f_{x_{sr}\to y_{sr}^{+}}:{\mathbb{R}}^{n_{sr}}\to {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{mapping\; from\; private\; variables\; to\; noise-added\; data\; vector},\mathrm{i}.\mathrm{e}.,y_{sr}^{+}=f_{x_{sr}\to y_{sr}^{+}}\left(x_{sr}\right)\end{array}$
$f_{t_{sr}\to y_{sr}^{+}}:{\mathbb{R}}^q\to {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{mapping\; from\; interpretable\; parameters\; to\; noise-added\; data\; vector},\mathrm{i}.\mathrm{e}.,y_{sr}^{+}=f_{t_{sr}\to y_{sr}^{+}}\left(t_{sr}\right)\end{array}$
${\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C$	$\begin{array}{c}\mathrm{differentially\; private\; source\; domain\; autoencoders,\; representing\; data\; features\; of}\\ \mathrm{each\; of} C\mathrm{classes},\mathrm{obtained\; via\; Algorithm} 6\end{array}$
$y_{tg}\in {\mathbb{R}}^{p_{tg}}$	target domain data vector
$y_{tg\to sr}\in {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{representation\; of\; target\; domain\; data\; vector} y_{tg} \mathrm{in\; source\; domain\; via\; transformation}\left(39\right)\end{array}$
${\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C$	$\begin{array}{c}\mathrm{target\; domain\; autoencoders},\mathrm{representing\; data\; features\; of}\\ \mathrm{each\; of} C\mathrm{classes},\mathrm{obtained\; via\; Algorithm} 6\end{array}$
$f_{y_{tg}\to c}:{\mathbb{R}}^{p_{tg}}\to \{1,\cdots ,C\}$	$\begin{array}{c}\mathrm{mapping\; assigning\; class\; label\; to\; target\; domain\; data\; vector} y_{tg} \mathrm{via}\left(47\right),\mathrm{i}.\mathrm{e}.,\\ f_{y_{tg}\to c}\left(y_{tg}\right)=\widehat{c}\left(y_{tg\to sr}\left(y_{tg}\right);{\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C,{\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C,{\mathbb{M}}^{sr\to tg}\right)\end{array}$
${\widehat{y}}_{tg}^{sr}\in {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{transformation\; of} y_{tg} \mathrm{to\; source\; domain\; and\; filtering\; through\; the\; autoencoder}\\ \mathrm{that\; represents\; the\; source\; domain\; feature\; vectors\; of\; the\; same\; class\; as\; that\; of} y_{tg},\mathrm{i}.\mathrm{e}.,\\ {\widehat{y}}_{tg}^{sr}=\widehat{\mathcal{WD}}\left(y_{tg\to sr}\left(y_{tg}\right);{\mathcal{P}}_{f_{y_{tg}\to c}\left(y_{tg}\right)}^{+sr}\right)\end{array}$
${\widehat{y}}_{tg}^{tg}\in {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{transformation\; of} y_{tg} \mathrm{to\; source\; domain\; and\; filtering\; through\; the\; autoencoder}\\ \mathrm{that\; represents\; the\; target\; domain\; feature\; vectors\; of\; the\; same\; class\; as\; that\; of} y_{tg},\mathrm{i}.\mathrm{e}.,\\ {\widehat{y}}_{tg}^{tg}=\widehat{\mathcal{WD}}\left(y_{tg\to sr}\left(y_{tg}\right);{\mathcal{P}}_{f_{y_{tg}\to c}\left(y_{tg}\right)}^{tg}\right)\end{array}$
$f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}:{\mathbb{R}}^{p_{sr}}\to {\mathbb{R}}^{p_{sr}}$	$\begin{array}{c}\mathrm{mapping\; from\; source\; domain\; feature\; vector} {\widehat{y}}_{tg}^{sr} \mathrm{to\; target\; domain\; feature\; vector} {\widehat{y}}_{tg}^{tg},\mathrm{i}.\mathrm{e}.,\\ {\widehat{y}}_{tg}^{tg}=f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}\left({\widehat{y}}_{tg}^{sr}\right)\end{array}$

Since the defined measures are in the form of information leakages, Algorithm 8 could be directly applied for practically computing the measures provided the availability of data samples.

5.2. A Unified Approach to Privacy-Preserving Interpretable and Transferable Learning

The presented theory allows us to develop an algorithm that implements privacy-preserving interpretable and transferable learning methodology in a unified manner.

Algorithm 9 is presented for a systematic implementation of the proposed privacy-preserving interpretable and transferable deep learning methodology. The functionality of Algorithm 9 is as follows:

Algorithm 9 Algorithm for privacy-preserving interpretable and transferable learning

Require:  The labeled source dataset: ${\mathbf{Y}}^{sr}={\left\{{\mathbf{Y}}_c^{sr}\right\}}_{c=1}^C$ (where ${\mathbf{Y}}_c^{sr}=\{y_{sr}^{i,c}\in {\mathbb{R}}^{p_{sr}}|i\in \{1,\cdots ,N_c^{sr}\}\}$ represents the c-th labeled samples); the set of private data: ${\mathbf{X}}^{sr}={\left\{{\mathbf{X}}_c^{sr}\right\}}_{c=1}^C$ (where ${\mathbf{X}}_c^{sr}=\{x_{sr}\in {\mathbb{R}}^{n_{sr}}|x_{sr}=f_{x_{sr}\to y_{sr}}^{-1}\left(y_{sr}\right),y_{sr}\in {\mathbf{Y}}_c^{sr}\}$); the set of interpretable parameters: ${\mathbf{T}}^{sr}={\left\{{\mathbf{T}}_c^{sr}\right\}}_{c=1}^C$ (where ${\mathbf{T}}_c^{sr}=\{t_{sr}\in {\mathbb{R}}^q|t_{sr}=f_{t_{sr}\to y_{sr}}^{-1}\left(y_{sr}\right),y_{sr}\in {\mathbf{Y}}_c^{sr}\}$); the set of a few labeled target samples: ${\left\{{\mathbf{Y}}_c^{tg}\right\}}_{c=1}^C$ (where ${\mathbf{Y}}_c^{tg}=\{y_{tg}^{i,c}\in {\mathbb{R}}^{p_{tg}}|i\in \{1,\cdots ,N_c^{tg}\}\}$ is the set of c-th labeled target samples); the set of unlabeled target samples: ${\mathbf{Y}}_{*}^{tg}=\{y_{tg}^{i,*}\in {\mathbb{R}}^{p_{tg}}|i\in \{1,\cdots ,N_{*}^{tg}\}\}$; and the differential privacy parameters: $d\in {\mathbb{R}}_{+}$, $ϵ\in {\mathbb{R}}_{+}$, $\delta \in (0,1)$.   1: A differentially private approximation of source dataset, ${\mathbf{Y}}^{+sr}={\left\{{\mathbf{Y}}_c^{+sr}\right\}}_{c=1}^C$, is obtained using Algorithm 5 on ${\mathbf{Y}}^{sr}$.   2: Differentially private source domain classifier, ${\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C$, is built using Algorithm 6 on ${\mathbf{Y}}^{+sr}$ taking subspace dimension as equal to $min(20,p_{sr})$ (where $p_{sr}$ is the dimension of source data samples), ratio $r_{max}$ as equal to 0.5, and number of layers as equal to 5.   3: Taking subspace dimension $n_{st}=min(\lceil p_{sr}/2\rceil ,p_{tg})$, the source domain transformation matrix $V^{+sr}\in {\mathbb{R}}^{n_{st}\times p_{sr}}$ is defined as with its i-th row equal to the transpose of the eigenvector corresponding to the i-th largest eigenvalue of sample covariance matrix computed on differentially private approximated source samples. The target domain transformation matrix $V^{tg}\in {\mathbb{R}}^{n_{st}\times p_{tg}}$ is defined as with its i-th row equal to the transpose of the eigenvector corresponding to the i-th largest eigenvalue of the sample covariance matrix computed on target samples.   4: For the case of heterogenous source and target domains, the subspace alignment approach is used to transform target samples via (40) and () for defining the sets ${\left\{{\mathbf{Y}}_c^{tg\to sr}\right\}}_{c=1}^C$ and ${\mathbf{Y}}_{*}^{tg\to sr}$.   5: Initial target domain classifier, ${\left\{{\mathcal{P}}_c^{tg}{|}_0\right\}}_{c=1}^C$, is built using Algorithm 4 on labeled target samples, ${\left\{{\mathbf{Y}}_c^{tg\to sr}\right\}}_{c=1}^C$, taking subspace dimension as equal to $min(20,{min}_{1\le c\le C}\left\{N_c^{tg}\right\}-1)$ (where $N_c^{tg}$ is the number of c-th class labeled target samples), ratio $r_{max}$ as equal to 1, and number of layers as equal to 1.   6: The target domain classifier is updated using (42) and (43) until 4 iterations taking the monotonically non-decreasing subspace dimension n sequence as $\{min(5,p_{sr}),min(10,p_{sr}),min(15,p_{sr}),min(20,p_{sr})\}$ and $r_{max=0.5}$.   7: The mapping from source to target domain is learned by means of a model, ${\mathbb{M}}^{sr\to tg}$, defined as in (44).   8: Compute privacy leakage, $I{L}_{f_{x_{sr}\to y_{sr}^{+}}}$, and adversary model, ${\mathbb{BM}}^{y_{sr}^{+}\to x_{sr}}$, via applying Algorithm 8 on $\left\{(y_{sr}^{+},x_{sr})\right|y_{sr}^{+}=f_{x_{sr}\to y_{sr}^{+}}\left(x_{sr}\right),x_{sr}\in {\mathbf{X}}^{sr},y_{sr}^{+}\in {\mathbf{Y}}^{+sr}\}$.   9: Compute interpretability measure, $I{L}_{f_{t_{sr}\to y_{sr}^{+}}}$, and interpretability model, ${\mathbb{BM}}^{y_{sr}^{+}\to t_{sr}}$, via applying Algorithm 8 on $\left\{(y_{sr}^{+},t_{sr})\right|y_{sr}^{+}=f_{t_{sr}\to y_{sr}^{+}}\left(t_{sr}\right),t_{sr}\in {\mathbf{T}}^{sr},y_{sr}^{+}\in {\mathbf{Y}}^{+sr}\}$. 10: Compute transferability measure, $I{L}_{f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}}$, via applying Algorithm 8 on $\left\{\left({\widehat{y}}_{tg}^{tg}\left(y_{tg}\right),{\widehat{y}}_{tg}^{sr}\left(y_{tg}\right)\right)|y_{tg}\in {\left\{{\mathbf{Y}}_c^{tg}\right\}}_{c=1}^C\cup {\mathbf{Y}}_{*}^{tg}\right\}$, where $$\begin{array}{ccc}\hfill {\widehat{y}}_{tg}^{sr}\left(y_{tg}\right)& =\hfill & \hfill \widehat{\mathcal{WD}}\left(y_{tg\to sr}\left(y_{tg}\right);{\mathcal{P}}_{f_{y_{tg}\to c}\left(y_{tg}\right)}^{+sr}\right)\end{array}$$ (116) $$\begin{array}{ccc}\hfill {\widehat{y}}_{tg}^{tg}\left(y_{tg}\right)& =\hfill & \hfill \widehat{\mathcal{WD}}\left(y_{tg\to sr}\left(y_{tg}\right);{\mathcal{P}}_{f_{y_{tg}\to c}\left(y_{tg}\right)}^{tg}\right)\end{array}$$ (117) $$\begin{array}{ccc}\hfill f_{y_{tg}\to c}\left(y_{tg}\right)& =\hfill & \hfill \widehat{c}\left(y_{tg\to sr}\left(y_{tg}\right);{\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C,{\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C,{\mathbb{M}}^{sr\to tg}\right),\end{array}$$ (118) $y_{tg\to sr}\left(y_{tg}\right)$ is defined as in (39), and $\widehat{c}(·)$ is defined by (47). 11: return in the source domain: classifier ${\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C$; privacy leakage $I{L}_{f_{x_{sr}\to y_{sr}^{+}}}$ and adversary model ${\mathbb{BM}}^{y_{sr}^{+}\to x_{sr}}$; interpretability measure $I{L}_{f_{t_{sr}\to y_{sr}^{+}}}$ and interpretability model ${\mathbb{BM}}^{y_{sr}^{+}\to t_{sr}}$. 12: return in the target domain: classifier ${\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C$. 13: return for transfer and multi-task learning scenario: classifiers ${\left\{{\mathcal{P}}_c^{+sr}\right\}}_{c=1}^C$ and ${\left\{{\mathcal{P}}_c^{tg}\right\}}_{c=1}^C$; source2target model ${\mathbb{M}}^{sr\to tg}$; latent subspace transformation matrices $V^{+sr}$ and $V^{tg}$; transferability measure $I{L}_{f_{{\widehat{y}}_{tg}^{sr}\to {\widehat{y}}_{tg}^{tg}}}$.

• Step 2 builds the differentially private source domain classifier following Algorithm 6 from previous work [22].
• Step 6 results in the building of the target domain classifier using the method of [22].
• An information theoretic evaluation of privacy leakage, interpretability, and transferability is undertaken at step 8, 9, and 10, respectively.
• Step 8 also provides the adversary model ${\mathbb{BM}}^{y_{sr}^{+}\to x_{sr}}$, which can be used to estimate private data and thus to simulate privacy attacks;
• Step 9 also provides the interpretability model ${\mathbb{BM}}^{y_{sr}^{+}\to t_{sr}}$, that can be used to estimate interpretable parameters and thus provide an interpretation to the non-interpretable data vectors.

6. Experiments

Experiments have been carried out to demonstrate the application of the proposed measures (for privacy leakage, interpretability, and transferability) to privacy-preserving interpretable and transferable learning. The methodology was implemented using MATLAB R2017b and the experiments have been made on an iMac (M1, 2021) machine with 8 GB RAM.

6.1. MNIST Dataset

The MNIST dataset contains $28\times 28$ sized images divided into a training set of 60,000 images and test set of 10,000 images. The images’ pixel values were divided by 255 to normalize the values in the range from 0 to 1. The $28\times 28$ normalized pixel values of each image were flattened to an equivalent 784-dimensional data vector.

6.1.1. Interpretable Parameters

For an MNIST digits dataset, there exist no additional interpretable parameters other than the pixel values. Thus, we defined corresponding to a pixel values vector $y\in {[0,1]}^{784}$ an interpretable parameter vector $t\in {\{0,1\}}^{10}$ such that the j-th element $t_j=1$, if the j-th class label is associated to y; otherwise, $t_j=0$. That is, interpretable vector t, in our experimental setting, represents the class label assigned to data vector y.

6.1.2. Private Data

Here, we assume that pixel values are private, i.e., $x_{sr}=y_{sr}$.

6.1.3. Semi-Supervised Transfer Learning Scenario

A transfer learning scenario was considered in the same setting as in [22,30] where 60,000 training samples constituted the source dataset; a set of 9000 test samples constituted the target dataset, and the classification performance was evaluated on the remaining 1000 test samples. Out of 9000 target samples, only 10 samples per class were labeled and the remaining 8900 target samples remained as unlabeled.

6.1.4. Experimental Design

Algorithm 9 is applied with the differential privacy parameters as $d=1$ and $\delta$ = 1 × 10${}^{-5}$. The experiment involves six different privacy-preserving semi-supervised transfer learning scenarios with privacy-loss bound values as $ϵ=0.1$, $ϵ=0.25$, $ϵ=0.5$, $ϵ=1$, $ϵ=2$, and $ϵ=10$. For the computation of privacy leakage, interpretability measure, and transferability measure in Algorithm 9, a subset consisting of 5000 randomly selected samples was considered.

6.1.5. Results

The experimental results have been plotted in Figure 4. Figure 4a–c display the privacy–accuracy trade-off curve, privacy–interpretability trade-off curve, and privacy–transferability trade-off curve respectively. The following observations are made:

Figure 4. The plots between privacy leakage, interpretability measure, transferability measure, and accuracy for MNIST dataset.

• As expected and observed in Figure 4f, the transferability measure is positively correlated with the accuracy of source–domain classifier on target test samples.
• Since we have defined the interpretable vector associated to a feature vector as representing the class label, the positive correlations of interpretability measure with the source domain classifier’s accuracy and the transferability measure are observed in Figure 4e and Figure 4f, respectively.
• The results also verify the robust performance of Algorithm 9 under transfer and multi-task learning scenario, since the classification performance in the transfer and multi-task learning scenario, unlike the performance of the source domain classifier, is not adversely affected by a reduction in privacy leakage, interpretability measure, and transferability measure as observed in Figure 4a,e,f.

Table 3 reports the results obtained by the models that correspond to minimum privacy leakage, maximum interpretability measure, and maximum transferability measure. The robustness of transfer and multi-task learning scenario is further highlighted in Table 3. To achieve the minimum value of privacy leakage, the accuracy of source domain classifier must be decreased to 0.1760; however, the transfer and multi-task learning scenario achieves the minimum privacy leakage value with the accuracy of 0.9510. As observed in Table 3, the maximum transferability-measure models also correspond to the maximum interpretability-measure models.

Table 3. Results of experiments on MNIST dataset for evaluating privacy leakage, interpretability, and transferability.

Method	Privacy Leakage	Interpretability Measure	Transferability Measure	Classification Accuracy
$\begin{array}{c}\mathrm{minimum\; privacy\; leakage}\\ \mathrm{transfer\; and\; multi-task\; learning}\end{array}$	−50.72	−2.14	−664.52	0.9510
$\begin{array}{c}\mathrm{minimum\; privacy\; leakage}\\ \mathrm{source\; domain\; classifier}\end{array}$	−50.72	−2.14	−664.52	0.1760
$\begin{array}{c}\mathrm{maximum\; interpretability\; measure}\\ \mathrm{transfer\; and\; multi-task\; learning}\end{array}$	362.83	5.44	451.93	0.9920
$\begin{array}{c}\mathrm{maximum\; interpretability\; measure}\\ \mathrm{source\; domain\; classifier}\end{array}$	362.83	5.44	451.93	0.9950
$\begin{array}{c}\mathrm{maximum\; transferability\; measure}\\ \mathrm{transfer\; and\; multi-task\; learning}\end{array}$	362.83	5.44	451.93	0.9920
$\begin{array}{c}\mathrm{maximum\; transferability\; measure}\\ \mathrm{source\; domain\; classifier}\end{array}$	362.83	5.44	451.93	0.9950

As a visualization example, Figure 5 displays noise-added data samples for different values of information theoretic measures.

Figure 5. An example of a source domain sample corresponding to different levels of privacy leakage, interpretability measure, and transferability measure.

6.2. Office and Caltech256 Datasets

The “Office+Caltech256” dataset has 10 common categories of both Office and Caltech256 datasets. The dataset has four domains: amazon, webcam, dslr, and caltech256. This dataset has been widely used [31,32,33,34] for evaluating multi-class accuracy performance in a standard domain adaptation setting with a small number of labeled target samples. Following [32], the 4096-dimensional deep-net VGG-FC6 features are extracted from the images. However, for the learning of classifiers, the 4096-dimensional feature vectors are reduced to 100-dimensional feature vectors using principal components computed from the data of amazon domain. Thus, corresponding to each image, a 100-dimensional data vector is constructed.

6.2.1. Interpretable Parameters

Corresponding to a data vector $y\in {\mathbb{R}}^{100}$, an interpretable parameter vector $t\in {\{0,1\}}^{10}$ is defined such that the j-th element $t_j=1$, if the j-th class label is associated to y; otherwise, $t_j=0$. That is, interpretable vector t, in our experimental setting, represents the class-label assigned to data vector y.

6.2.2. Private Data

Here, we assume that the extracted image feature vectors are private, i.e., $x_{sr}=y_{sr}$.

6.2.3. Semi-Supervised Transfer Learning Scenario

Similarly to [31,32,33,34], the experimental setup is follows:

• The number of training samples per class in the source domain is 20 for amazon and is 8 for the other three domains;
• The number of labeled samples per class in the target domain is 3 for all the four domains.

6.2.4. Experimental Design

Taking a domain as the source and another domain as the target, 12 different transfer learning experiments are performed on the four domains associated to the “Office+Caltech256” dataset. Each of the 12 experiments is repeated 20 times via creating 20 random train/test splits. In all of the 240 ($=12\times 20$) experiments, Algorithm 9 is applied three times with varying values of privacy-loss bound: first with differential privacy parameters as $(d=1,ϵ=0.01,\delta =1\times {10}^{-5})$, second with differential privacy parameters as $(d=1,ϵ=0.1,\delta =1\times {10}^{-5})$, and third with differential privacy parameters as $(d=1,ϵ=1,\delta =1\times {10}^{-5})$. As Algorithm 9 with different values of privacy-loss bound $ϵ$ will result in different models, the transfer and multi-task learning models that correspond to the maximum interpretability measure and maximum transferability measure are considered for an evaluation.

6.2.5. Reference Methods

This dataset has been studied previously [31,32,33,34,35,36] and thus, as a reference, the performances of the following existing methods were considered:

• ILS (1-NN) [32]: This method learns an Invariant Latent Space (ILS) to reduce the discrepancy between domains and uses Riemannian optimization techniques to match statistical properties between samples projected into the latent space from different domains.
• CDLS [35]: The Cross-Domain Landmark Selection (CDLS) method derives a domain-invariant feature subspace for heterogeneous domain adaptation.
• MMDT [34]: The Maximum Margin Domain Transform (MMDT) method adapts max-margin classifiers in a multi-class manner by learning a shared component of the domain shift as captured by the feature transformation.
• HFA [36]: The Heterogeneous Feature Augmentation (HFA) method learns common latent subspace and a classifier under the max-margin framework.
• OBTL [33]: The Optimal Bayesian Transfer Learning (OBTL) method employs a Bayesian framework to transfer learning through the modeling of a joint prior probability density function for feature-label distributions of the source and target domains.

6.2.6. Results

Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11, Table 12, Table 13, Table 14 and Table 15 report the results, and the first two best performances have been marked.

Table 4. Accuracy (in %, averaged over 20 experiments) obtained in amazon→caltech256 semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	82.6
privacy-preserving maximum transferability-measure model	VGG-FC6	82.6
non-private ILS (1-NN)	VGG-FC6	83.3
non-private CDLS	VGG-FC6	78.1
non-private MMDT	VGG-FC6	78.7
non-private HFA	VGG-FC6	75.5
non-private OBTL	SURF	41.5
non-private ILS (1-NN)	SURF	43.6
non-private CDLS	SURF	35.3
non-private MMDT	SURF	36.4
non-private HFA	SURF	31.0

Table 5. Accuracy (in %, averaged over 20 experiments) obtained in amazon→dslr semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	88.5
privacy-preserving maximum transferability-measure model	VGG-FC6	88.7
non-private ILS (1-NN)	VGG-FC6	87.7
non-private CDLS	VGG-FC6	86.9
non-private MMDT	VGG-FC6	77.1
non-private HFA	VGG-FC6	87.1
non-private OBTL	SURF	60.2
non-private ILS (1-NN)	SURF	49.8
non-private CDLS	SURF	60.4
non-private MMDT	SURF	56.7
non-private HFA	SURF	55.1

Table 6. Accuracy (in %, averaged over 20 experiments) obtained in amazon→webcam semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	89.3
privacy-preserving maximum transferability-measure model	VGG-FC6	89.3
non-private ILS (1-NN)	VGG-FC6	90.7
non-private CDLS	VGG-FC6	91.2
non-private MMDT	VGG-FC6	82.5
non-private HFA	VGG-FC6	87.9
non-private OBTL	SURF	72.4
non-private ILS (1-NN)	SURF	59.7
non-private CDLS	SURF	68.7
non-private MMDT	SURF	64.6
non-private HFA	SURF	57.4

Table 7. Accuracy (in %, averaged over 20 experiments) obtained in caltech256→amazon semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	92.6
privacy-preserving maximum transferability-measure model	VGG-FC6	92.6
non-private ILS (1-NN)	VGG-FC6	89.7
non-private CDLS	VGG-FC6	88.0
non-private MMDT	VGG-FC6	85.9
non-private HFA	VGG-FC6	86.2
non-private OBTL	SURF	54.8
non-private ILS (1-NN)	SURF	55.1
non-private CDLS	SURF	50.9
non-private MMDT	SURF	49.4
non-private HFA	SURF	43.8

Table 8. Accuracy (in %, averaged over 20 experiments) obtained in caltech256→dslr semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	89.1
privacy-preserving maximum transferability-measure model	VGG-FC6	89.1
non-private ILS (1-NN)	VGG-FC6	86.9
non-private CDLS	VGG-FC6	86.3
non-private MMDT	VGG-FC6	77.9
non-private HFA	VGG-FC6	87.0
non-private OBTL	SURF	61.5
non-private ILS (1-NN)	SURF	56.2
non-private CDLS	SURF	59.8
non-private MMDT	SURF	56.5
non-private HFA	SURF	55.6

Table 9. Accuracy (in %, averaged over 20 experiments) obtained in caltech256→webcam semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	87.8
privacy-preserving maximum transferability-measure model	VGG-FC6	87.7
non-private ILS (1-NN)	VGG-FC6	91.4
non-private CDLS	VGG-FC6	89.7
non-private MMDT	VGG-FC6	82.8
non-private HFA	VGG-FC6	86.0
non-private OBTL	SURF	71.1
non-private ILS (1-NN)	SURF	62.9
non-private CDLS	SURF	66.3
non-private MMDT	SURF	63.8
non-private HFA	SURF	58.1

Table 10. Accuracy (in %, averaged over 20 experiments) obtained in dslr→amazon semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	91.9
privacy-preserving maximum transferability-measure model	VGG-FC6	91.9
non-private ILS (1-NN)	VGG-FC6	88.7
non-private CDLS	VGG-FC6	88.1
non-private MMDT	VGG-FC6	83.6
non-private HFA	VGG-FC6	85.9
non-private OBTL	SURF	54.4
non-private ILS (1-NN)	SURF	55.0
non-private CDLS	SURF	50.7
non-private MMDT	SURF	46.9
non-private HFA	SURF	42.9

Table 11. Accuracy (in %, averaged over 20 experiments) obtained in dslr→caltech256 semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	82.9
privacy-preserving maximum transferability-measure model	VGG-FC6	82.9
non-private ILS (1-NN)	VGG-FC6	81.4
non-private CDLS	VGG-FC6	77.9
non-private MMDT	VGG-FC6	71.8
non-private HFA	VGG-FC6	74.8
non-private OBTL	SURF	40.3
non-private ILS (1-NN)	SURF	41.0
non-private CDLS	SURF	34.9
non-private MMDT	SURF	34.1
non-private HFA	SURF	30.9

Table 12. Accuracy (in %, averaged over 20 experiments) obtained in dslr→webcam semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	88.9
privacy-preserving maximum transferability-measure model	VGG-FC6	89.0
non-private ILS (1-NN)	VGG-FC6	95.5
non-private CDLS	VGG-FC6	90.7
non-private MMDT	VGG-FC6	86.1
non-private HFA	VGG-FC6	86.9
non-private OBTL	SURF	83.2
non-private ILS (1-NN)	SURF	80.1
non-private CDLS	SURF	68.5
non-private MMDT	SURF	74.1
non-private HFA	SURF	60.5

Table 13. Accuracy (in %, averaged over 20 experiments) obtained in webcam→amazon semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	92.3
privacy-preserving maximum transferability-measure model	VGG-FC6	92.3
non-private ILS (1-NN)	VGG-FC6	88.8
non-private CDLS	VGG-FC6	87.4
non-private MMDT	VGG-FC6	84.7
non-private HFA	VGG-FC6	85.1
non-private OBTL	SURF	55.0
non-private ILS (1-NN)	SURF	54.3
non-private CDLS	SURF	51.8
non-private MMDT	SURF	47.7
non-private HFA	SURF	56.5

Table 14. Accuracy (in %, averaged over 20 experiments) obtained in webcam→caltech256 semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	81.4
privacy-preserving maximum transferability-measure model	VGG-FC6	81.4
non-private ILS (1-NN)	VGG-FC6	82.8
non-private CDLS	VGG-FC6	78.2
non-private MMDT	VGG-FC6	73.6
non-private HFA	VGG-FC6	74.4
non-private OBTL	SURF	37.4
non-private ILS (1-NN)	SURF	38.6
non-private CDLS	SURF	33.5
non-private MMDT	SURF	32.2
non-private HFA	SURF	29.0

Table 15. Accuracy (in %, averaged over 20 experiments) obtained in webcam→dslr semi-supervised transfer learning experiments. The first and second best performances have been marked.

Method	Feature Type	Accuracy (%)
privacy-preserving maximum interpretability-measure model	VGG-FC6	90.8
privacy-preserving maximum transferability-measure model	VGG-FC6	90.2
non-private ILS (1-NN)	VGG-FC6	94.5
non-private CDLS	VGG-FC6	88.5
non-private MMDT	VGG-FC6	85.1
non-private HFA	VGG-FC6	87.3
non-private OBTL	SURF	75.0
non-private ILS (1-NN)	SURF	70.8
non-private CDLS	SURF	60.7
non-private MMDT	SURF	67.0
non-private HFA	SURF	56.5

Finally, Table 16 summarizes the overall performance of the top four methods. As observed in Table 16, the maximum transferability-measure model remains as best performing in the maximum number of experiments. The most remarkable result observed is that the proposed methodology, despite being privacy-preserving, ensuring the differential privacy-loss bound to be less than equal to 1 and not requiring access to source data samples, performs better than even the non-private methods.

Table 16. Comparison of the methods on “Office+Caltech256” dataset.

Method	Number of Experiments in Which Method Performed Best
privacy-preserving maximum transferability-measure model	6
privacy-preserving maximum interpretability-measure model	5
non-private ILS (1-NN)	5
non-private CDLS	1

6.3. An Application Example: Mental Stress Detection

The mental stress detection problem is considered as an application example of the proposed privacy-preserving interpretable and transferable learning approach. The dataset from [17], consisting of heart rate interval measurements of different subjects, is considered for the study of an individual stress detection problem. In [17], a membership-mappings-based interpretable deep model was applied for an estimation of stress score; however, the current study deals with application of the proposed privacy-preserving interpretable and transferable deep learning method to solve the stress classification problem. The problem is concerned with the detection of stress in an individual based on the analysis of recorded sequence of R-R intervals, ${\left\{R{R}^i\right\}}_i$. The R-R data vector at $i-$th time-index, $y^i$, is defined as

$$\begin{array}{ccc}\hfill y^i& =\hfill & \hfill {\left[\begin{array}{cccc}\hfill R{R}^i& R{R}^{i-1}& \cdots \hfill & R{R}^{i-d}\end{array}\right]}^T.\end{array}$$

(119)

That is, the current interval and history of previous d intervals constitute the data vector. Assuming an average heartbeat of 72 beats per minute, d is chosen as equal to $72\times 3=216$ so that the R-R data vector consists of on average 3-minute-long R-R intervals sequences. A dataset, say ${\left\{y^i\right\}}_i$, is built via (1) preprocessing the R-R interval sequence ${\left\{R{R}^i\right\}}_i$ with an impulse rejection filter [37] for artifacts detection and (2) excluding the R-R data vectors containing artifacts from the dataset. The dataset contains the stress score on a scale from 0 to 100. A label of either “no-stress” or “under-stress” is assigned to each $y^i$ based on the stress score. Thus, we have a binary classification problem.

6.3.1. Interpretable Parameters

Corresponding to a R-R data vector, there exists the set of interpretable parameters: mental demand, physical demand, temporal demand, own performance, effort, and frustration. These are the six components of stress acquired using the NASA Task Load Index [38]. The NASA Task Load Index provides subjective assessment of stress where an individual provides a rating on the scale from 0 to 100 for each of the six components of stress (mental demand, physical demand, temporal demand, own performance, effort, and frustration). Thus, corresponding to each $217—$dimensional R-R data vector, there exists a six-dimensional interpretable parameters vector acquired using the NASA Task Load Index.

6.3.2. Private Data

Here, we assume that heart rate values are private. As instantaneous heart rate is given as $H{R}^i=60/R{R}^i$; thus, information about private data is directly contained in the R-R data vectors.

6.3.3. Semi-Supervised Transfer Learning Scenario

Out of the total subjects, a randomly chosen subject’s data serve as the source domain data. Considering every other subject’s data as the target domain data, the transfer learning experiment is performed independently on each target subject where 50% of the target subject’s samples are labeled, and the remaining unlabeled target samples also serve as test data for evaluating the classification performance. However, only the target subjects, with data containing both the classes and at least 60 samples, were considered for experimentation. There are in total 48 such target subjects.

6.3.4. Experimental Design

Algorithm 9 is applied with $d=1$, $ϵ\in \{0.1,0.5,1,2,5,8,20,50,100,\infty \}$, and $\delta =1\times {10}^{-5}$. Each of the 48 experiments involves 10 different privacy-preserving semi-supervised transfer learning scenarios with privacy-loss bound values as $ϵ=0.1$, $ϵ=0.5$, $ϵ=1$, $ϵ=2$, $ϵ=5$, $ϵ=8$, $ϵ=20$, $ϵ=50$, $ϵ=100$, and $ϵ=\infty$. The following two requirements are associated to this application example:

• The private source domain data must be protected while transferring knowledge from source to target domain; and
• The interpretability of the source domain model should be high.

In view of the aforementioned requirements, the models that correspond to the minimum privacy leakage and maximum interpretability measure amongst all the models obtained corresponding to 10 different choices of differential privacy-loss bound $ϵ$ are considered for detecting stress.

6.3.5. Results

Figure 6 summarizes the experimental results where accuracies obtained by both minimum privacy-leakage models and maximum interpretability-measure models have been displayed as box plots.

Figure 6. The box plots of accuracies obtained in detecting mental stress on 48 different subjects.

It is observed in Figure 6 that the transfer and multi-task learning improves considerably the performance of source domain classifier. Table 17 reports the median values (of privacy leakage, interpretability measure, transferability measure, and classification accuracy) obtained in the experiments on 48 different subjects. The robust performance of transfer and multi-task learning scenario is further observed in Table 17.

Table 17. Results (median values) obtained in stress detection experiments on a dataset consisting of heart rate interval measurements.

Method	Privacy Leakage	Interpretability Measure	Transferability Measure	Classification Accuracy
$\begin{array}{c}\mathrm{minimum\; privacy\; leakage}\\ \mathrm{transfer\; and\; multi-task\; learning}\end{array}$	−3.74	3.47	291.84	0.9647
$\begin{array}{c}\mathrm{minimum\; privacy\; leakage}\\ \mathrm{source\; domain\; classifier}\end{array}$	−3.74	3.47	291.84	0.3411
$\begin{array}{c}\mathrm{maximum\; interpretability\; measure}\\ \mathrm{transfer\; and\; multi-task\; learning}\end{array}$	0.43	23.92	773.36	0.9619
$\begin{array}{c}\mathrm{maximum\; interpretability\; measure}\\ \mathrm{source\; domain\; classifier}\end{array}$	0.43	23.92	773.36	0.3602

As a visualization example, Figure 7 displays the noise-added source domain heart rate interval data for different values of information theoretic measures.

Figure 7. A display of source domain R-R interval data corresponding to different levels of privacy leakage, interpretability measure, and transferability measure.

7. Concluding Remarks

The paper has introduced information theoretic measures for privacy leakage, interpretability, and transferability to study the trade-offs. This is the first study to develop an information theory-based unified approach to privacy-preserving interpretable and transferable learning. The experiments have verified that the proposed measures (for privacy leakage, interpretability, and transferability) can be used to study the trade-off curves (between privacy leakage, interpretability measure, and transferability measure) and thus to optimize the models for the given application requirements such as the requirement of minimum privacy leakage, the requirement of maximum interpretability measure, and the requirement of maximum transferability measure. The experimental results on the MNIST dataset showed that the transfer and multi-task learning scenario improved remarkable the accuracy from 0.1760 to 0.9510 while ensuring the minimum privacy leakage. The experiments on Office and Caltech256 datasets indicated that the proposed methodology, despite ensuring differential privacy-loss bound to be less than equal to 1 and not requiring an access to source data samples, performed better than even existing non-private methods in six out of 12 transfer learning experiments. The stress detection experiments on a real-world biomedical data led to the observation that the transfer and multi-task learning scenario improved the accuracy from 0.3411 to 0.9647 (while ensuring the minimum privacy leakage) and from 0.3602 to 0.9619 (while ensuring the maximum interpretability measure). The considered unified approach to privacy-preserving interpretable and transferable learning involves membership-mappings-based conditionally deep autoencoders, albeit other data representation learning models could be explored. The future work includes the following:

• Although the text has not focused on federated learning, the transfer learning approach could be easily extended to the multi-party system and the transferability measure could be calculated for any pair of parties.
• Also, the explainability of the conditionally deep autoencoders follows, similar to in [17], via estimating interpretable parameters from non-interpretable data feature vectors using variational membership-mapping Bayesian model.
• Furthermore, the variational membership-mapping Bayesian model quantifies uncertainties on the estimation of parameters (of interest), which is also important for a user’s trust on the model.