An Information-Theoretic Approach for Multivariate Skew-t Distributions and Applications ^†

Abstract

Shannon and Rényi entropies are two important measures of uncertainty for data analysis. These entropies have been studied for multivariate Student-t and skew-normal distributions. In this paper, we extend the Rényi entropy to multivariate skew-t and finite mixture of multivariate skew-t (FMST) distributions. This class of flexible distributions allows handling asymmetry and tail weight behavior simultaneously. We find upper and lower bounds of Rényi entropy for these families. Numerical simulations illustrate the results for several scenarios: symmetry/asymmetry and light/heavy-tails. Finally, we present applications of our findings to a swordfish length-weight dataset to illustrate the behavior of entropies of the FMST distribution. Comparisons with the counterparts—the finite mixture of multivariate skew-normal and normal distributions—are also presented.

1. Introduction

Finite mixture models have used in the analysis of heterogeneous datasets due to its flexibility [1]. These models provide important applications in many scientific fields such as density estimation [2], data mining and pattern recognition [3], image processing and satellite imaging [4], medicine and genetics [4,5], fisheries [6,7], astronomy [2], and more. Specifically, Carreira-Perpiñán [2] considered the mixture of normal densities to find the modes of multi-dimensional data via the Shannon entropy. Because no analytical expressions exist for Shannon entropy of normal mixture of densities, the authors of [2] considered bounds to approximates the Shannon entropy [8]. Given that these applications have been developed in the normal mixture of densities context, several calculations of Shannon and Rényi entropies for non-normal distributions exist.

Azzalini and Dalla-Valle [9] and Azzalini and Capitanio [10] introduced the multivariate skew-normal and skew-t distributions as an alternative to multivariate normal distribution to deal with skewness and heavy-tailness in the data, respectively. Lin et al. [11] proposed a development of finite mixture of skew-t (FMST) distributions, and more recently [4] provided an overview of developments of FMST distributions. Arellano-Valle et al. [12] and Contreras-Reyes [13] presented the mutual information and Shannon entropy for multivariate skew-normal and skew-t distributions, respectively. They highlighted that the calculation of mutual information index and Shannon entropy include the negentropy concept represented by a one-dimensional integral. Contreras-Reyes [14] discussed the values of Rényi entropy of multivariate skew-normal and extended skew-normal distributions. Contreras-Reyes and Cortés [6] considered bounds to approximate the Rényi entropy for finite mixture of multivariate skew-normal (FMSN) distributions by using lower and upper bounds. More recently, the authors of [15,16] presented the bounds of Rényi entropy of multivariate skew-normal-Cauchy and skew-Laplace distributions, and [17] considered the Kullback–Leibler divergence based on multivariate skew-t densities [13] as a classifier for an SAR image system.

In this paper, we propose an information-theoretic approach for FMST distributions. The explicit expressions of Shannon and Rényi entropies of skew-t distribution are derived. We give bounds for Rényi entropies of FMST models. An approximate value of these entropies can be calculated. Simulation studies illustrate the behavior of Rényi entropy approximations for a given order $\alpha$, skewness, and freedom degrees of the proposed mixture model. Numerical simulations illustrates the results for several scenarios: symmetry/asymmetry and light/heavy-tails. Finally, a real data application of a swordfish length-weight dataset is revisited from the work in [6], where the skew-normal model is compared with the proposed skew-t one.

The paper is organized as follows. Section 2 gives the propositions, lemmas, and numerical simulations for computation of Shannon and Rényi entropies of multivariate skew-t random variables. Section 3 provides bounds to approximate Shannon and Rényi entropies for FMST distributions and a numerical application of swordfish data. Finally, Section 4 concludes the study.

2. Multivariate Skew-$\mathit{t}$ Distribution

The multivariate t-distribution corresponds to a generalization of the Student-t-distribution. Let $\mathbf{x}\in {\mathbb{R}}^d$ be a random vector, $d\in {\mathbb{Z}}_{+}$, is multivariate t-distributed [18], denoted as $\mathbf{X}\sim T_d(\mathbf{\xi },\mathbf{\Omega },\nu )$, whose probability density function (pdf) is given by

$$t_d(\mathbf{x};\mathbf{\xi },\mathbf{\Omega },\nu )=B_d\left(\nu \right){|\mathbf{\Omega }|}^{-1/2}\left(1+\frac{Q\left(\mathbf{x}\right)}{\nu }\right),$$

(1)

where $Q\left(\mathbf{x}\right)={(\mathbf{x}-\mathbf{\xi })}^{\top }{\mathbf{\Omega }}^{-1}(\mathbf{x}-\mathbf{\xi })$, $\mu \in {\mathbb{R}}^d$ is a location parameter, $\mathbf{\Omega }\in {\mathbb{R}}^{d\times d}$ is a dispersion matrix, $\nu \in \mathbb{R}$ is the degree of freedom parameter,

$$B_{\ell }\left(\nu \right)=\frac{\Gamma \left(\frac{\nu +\ell }{2}\right)}{\Gamma \left(\frac{\nu }{2}\right){\left(\nu \pi \right)}^{\ell /2}},\nu >0,\forall \ell \in \mathbb{Z},$$

(2)

and $\Gamma (·)$ is the gamma function.

The multivariate normal distribution is obtained from the limit $T_d(\mathbf{\xi },\mathbf{\Omega },\nu )\to N_d(\mathbf{\xi },\mathbf{\Omega })$, as $\nu \to \infty$. In the special case where $\nu =1$, $\mathbf{\xi }=\mathbf{0}$ and $\mathbf{\Omega }=I_d$, $t_d(\mathbf{\xi },\mathbf{\Omega },\nu )$ leads to the multivariate Cauchy distribution. The mean and variance of random vector $\mathbf{x}$ are

$$\begin{array}{ccc}\hfill E\left[\mathbf{X}\right]& =& \mathbf{\xi },\nu >1,\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill V\left[\mathbf{X}\right]& =& \left(\frac{B_{-2}\left(\nu \right)}{2\pi }\right)\mathbf{\Omega },\nu >2,\hfill \end{array}$$

(4)

respectively.

An extension of multivariate skew-normal distribution is the skew-t one by incorporating a degree of freedom parameter representing the presence of atypical observations in empirical data [10,12,19]. We considered the definition of skew-t distribution given in [12]. Let $\mathbf{y}\in {\mathbb{R}}^d$ be a multivariate skew-t random variable with location vector $\mathbf{\xi }\in {\mathbb{R}}^d$, dispersion matrix $\mathbf{\Omega }\in {\mathbb{R}}^{d\times d}$, asymmetry vector $\mathbf{\eta }\in {\mathbb{R}}^d$, and $\nu >0$ degrees of freedom. We denoted $\mathbf{y}$ by $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$ if has pdf given by

$$f(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )=2t_d(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\nu )T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{y}\right)}};\nu +d\right),$$

(5)

where $\overline{\mathbf{\eta }}={\mathbf{\Omega }}^{1/2}\mathbf{\eta }$ and $T(\mathbf{x};\nu +d)\equiv T_1(\mathbf{x};0,1,\nu +d)$ is the univariate Student-t cumulative distribution function (cdf) of $\mathbf{x}$.

The multivariate Student-t distribution is a special case of the multivariate skew-t one and is related to a multivariate skew-normal random variable through equation

$$\mathbf{Y}\stackrel{d}{=}\mathbf{\xi }+V^{-1/2}{\mathbf{Z}}_0,$$

(6)

where ${\mathbf{Z}}_0={\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sim S{N}_d(\mathbf{0},I_d,\mathbf{\eta })$ and $V\sim {\chi }^2/\nu$, and they are independent, where $S{N}_d$ and ${\chi }^2$ denotes the d-variate skew-normal [9] and univariate chi-square distribution [10], respectively. The multivariate skew-t distribution yields the multivariate skew-normal and Student-t ones as $\nu \to \infty$ and when $\mathbf{\eta }=\mathbf{0}$, respectively [4].

Considering the m-cumulant function

$$E\left(V^{-m/2}\right)=\frac{B_{-m}\left(\nu \right)}{{\left(2\pi \right)}^{m/2}},m\ge 1,\nu >m,$$

and Equations (2)–(4), the mean vector and covariance matrix of $\mathbf{Y}$ are, respectively, derived by the authors of [10] in the following forms,

$$\begin{array}{ccc}\hfill E\left[\mathbf{Y}\right]& =& \mathbf{\xi }+\frac{B_{-1}\left(\nu \right)}{\pi }\overline{\mathbf{\eta }},\nu >1,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill V\left[\mathbf{Y}\right]& =& \mathbf{\xi }{\mathbf{\xi }}^{\top }+\left(\frac{B_{-2}\left(\nu \right)}{2\pi }\right)\mathbf{\Omega }+{\left(\frac{B_{-1}\left(\nu \right)}{\pi }\right)}^2(\mathbf{\xi }{\mathbf{\delta }}^{\top }+\mathbf{\delta }{\mathbf{\xi }}^{\top }),\nu >2,\hfill \end{array}$$

(8)

where $\mathbf{\delta }=\mathbf{\Omega }\mathbf{\eta }/\sqrt{1+{\mathbf{\eta }}^{\top }\mathbf{\Omega }\mathbf{\eta }}$.

2.1. Entropies

Shannon [20] proposed an entropy to quantify the uncertainty of a system and posteriorly, Rényi [21] generalized the Shannon entropy for any probability distribution related to a discrete or continuous random variable. In this section, we derive these measures for the skew-t case.

Consider a pdf f associated with a random variable $\mathbf{y}\in {\mathbb{R}}^d$, denoted by $H\left(\mathbf{Y}\right)$, the well-known Shannon entropy [22]. It is defined by

$$H\left(\mathbf{Z}\right)=-{\int }_{{\mathbb{R}}^d}f\left(\mathbf{y}\right)logf\left(\mathbf{y}\right)d\mathbf{y}.$$

(9)

Proposition 1.

The Shannon entropy of $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$ is

$$H\left(\mathbf{Y}\right)=H\left(\mathbf{X}\right)-E\left[log\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel W_H}{\sqrt{\nu +d-1+W_H^2}};\nu +k\right)\right\}\right],$$

(10)

where $W_H\sim S{T}_1(0,1,\parallel \overline{\mathbf{\eta }}\parallel ,\nu +d-1)$, $\parallel \overline{\mathbf{\eta }}\parallel ={\overline{\mathbf{\eta }}}^{\top }\overline{\mathbf{\eta }}={({\mathbf{\eta }}^{\top }\mathbf{\Omega }\mathbf{\eta })}^{1/2}$, and $H\left(\mathbf{X}\right)$ is the Student-t Shannon entropy of $\mathbf{X}\sim T_d(\mathbf{\xi },\mathbf{\Omega },\nu )$ given by

$$H\left(\mathbf{X}\right)=\frac{1}{2}log|\mathbf{\Omega }|-log{B}_d\left(\nu \right)+{\Psi }_d\left(\nu \right),$$

(11)

where

$${\Psi }_d\left(\nu \right)=\left(\frac{\nu +d}{2}\right)\left[\psi \left(\frac{\nu +d}{2}\right)-\psi \left(\frac{\nu }{2}\right)\right],$$

(12)

$B_d\left(\nu \right)$ is defined in (2) and $\psi \left(x\right)=\frac{d}{dx}log\Gamma \left(x\right)$ is the digamma function.

The proof for the Shannon entropy of $\mathbf{Y}$ in (10) is shown in [12]. The same entropy, but for $\mathbf{X}$ in (11), was first presented in [23].

An extension of Shannon entropy is the $\alpha$th Rényi order given by [21]. The Rényi entropy of $\mathbf{Y}$ is denoted by $R_{\alpha }\left(\mathbf{Y}\right)$ and defined by

$$R_{\alpha }\left(\mathbf{Y}\right)=\frac{1}{1-\alpha }log{\int }_{{\mathbb{R}}^d}{\left[f\left(\mathbf{y}\right)\right]}^{\alpha }d\mathbf{y},$$

(13)

with $0<\alpha <\infty$, $\alpha \ne 1$, and where normalization to unity as given by ${\int }_{{\mathbb{R}}^d}f\left(\mathbf{y}\right)d\mathbf{y}=1$ [24]. The Shannon entropy is obtained from $R_{\alpha }\left(\mathbf{Y}\right)$ as $\alpha \to 1$ [14,21]. The Rényi entropy is invariant under a location transformation, but not invariant under a scale transformation. For any ${\alpha }_1<{\alpha }_2$, we have $R_{{\alpha }_1}\left(\mathbf{Y}\right)\ge R_{{\alpha }_2}\left(\mathbf{Y}\right)$, and $R_{{\alpha }_1}\left(\mathbf{Y}\right)=R_{{\alpha }_2}\left(\mathbf{Y}\right)$ if and only if the system is uniformly distributed.

In order to compute the Rényi entropy of a multivariate skew-t random variate, we need the following preliminary results that involve the pdf (5).

Lemma 1.

Let $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$. Then,

$$\begin{array}{ccc}& & E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{y}\right)}};\nu +d\right)\right\}\right]\hfill \\ & & =E\left[log\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel X_0}{\sqrt{\nu +d-1+X_0^2}};\nu +d\right)\right\}\left\{2T\left(\frac{\sqrt{\nu +1}\parallel \overline{\mathbf{\eta }}\parallel X_0}{\sqrt{\nu +X_0^2}};\nu +d\right)\right\}\right],\hfill \end{array}$$

where $X_0\sim T_1(0,1,\nu +d-1)$.

Proof. 

We directly get

$$\begin{array}{ccc}& & E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{Y}\right)}};\nu +d\right)\right\}\right]\hfill \\ & & ={\int }_{{\mathbb{R}}^d}log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{y}\right)}};\nu +d\right)\right\}\hfill \\ & & \times 2t_d(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\nu )T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{y}\right)}};\nu +d\right)d\mathbf{y}.\hfill \end{array}$$

Using the change of variables ${\mathbf{z}}_0={\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })$, we get that $Z_0\sim S{T}_d(\mathbf{0},I_d,\mathbf{\eta },\nu )$ and the determinant of Jacobian matrix ${|\mathbf{\Omega }|}^{-1/2}$. Therefore,

$$\begin{array}{ccc}& & E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{Y}\right)}};\nu +d\right)\right\}\right]\hfill \\ & & ={\int }_{{\mathbb{R}}^d}log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{z}}_0\sqrt{\frac{\nu +d}{\nu +{\mathbf{z}}_0^{\top }{\mathbf{z}}_0}};\nu +d\right)\right\}\hfill \\ & & \times {2|\mathbf{\Omega }|}^{-1/2}{t}_d({\mathbf{z}}_0;\mathbf{0},I_d,\nu )T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{z}}_0\sqrt{\frac{\nu +d}{\nu +{\mathbf{z}}_0^{\top }{\mathbf{z}}_0}};\nu +d\right)d\mathbf{y}\hfill \\ & & =E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{Z}}_0\sqrt{\frac{\nu +d}{\nu +{\mathbf{Z}}_0^{\top }{\mathbf{Z}}_0}};\nu +d\right)\right\}\right].\hfill \end{array}$$

Lemma 3 of [12] implies that

$$\begin{array}{ccc}& & E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{Y}\right)}};\nu +d\right)\right\}\right]\hfill \\ & & =E\left[log\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel W_R}{\sqrt{\nu +d-1+W_R^2}};\nu +d\right)\right\}\right],\hfill \end{array}$$

with $W_R\sim S{T}_1(0,1,\parallel \overline{\mathbf{\eta }}\parallel ,\nu +d-1)$ and $\parallel \overline{\mathbf{\eta }}\parallel ={\overline{\mathbf{\eta }}}^{\top }\overline{\mathbf{\eta }}$. The latter result yields the proof.    □

Lemma 1 allows us to compute an expected value of a d-variate random variable $\mathbf{Y}$ as an expected value of a univariate Student-t one $X_0$. This result is also applied in the computation of Shannon entropy of $\mathbf{Y}$; however, the latter depends on a univariate skew-t. Then, we can represent the expected value of (10) with respect to a univariate Student-t random variable as follows,

Proposition 2.

Let $\mathbf{X}\sim T_d(\mathbf{\xi },\mathbf{\Omega },\nu )$ and $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$, the Shannon entropy of $\mathbf{Y}$ can be written as

$$H\left(\mathbf{Y}\right)=H\left(\mathbf{X}\right)-N\left(X_0\right),$$

(14)

where

$$N\left(X_0\right)=E\left[log\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel X_0}{\sqrt{\nu +d-1+X_0^2}};\nu +d\right)\right\}\left\{2T\left(\frac{\sqrt{\nu +1}\parallel \overline{\mathbf{\eta }}\parallel X_0}{\sqrt{\nu +X_0^2}};\nu +d\right)\right\}\right],$$

and $X_0\sim T_1(0,1,\nu +d-1)$.

Proof. 

By computing the natural logarithm and expectation in both sides of pdf (5), we have

$$\begin{array}{ccc}& & E\left[log\left\{f\right(\mathbf{Y};\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu \left)\right\}\right]=E[log\left\{t_d(\mathbf{Y};\mathbf{\xi },\mathbf{\Omega },\nu )\right\}]\hfill \\ & & +E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{Y}\right)}};\nu +d\right)\right\}\right].\hfill \end{array}$$

Using Lemma 1 and Proposition 1 of [25], we obtain

$$H\left(\mathbf{Y}\right)=H\left(\mathbf{X}\right)-E\left[log\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{\Omega }}^{-1/2}(\mathbf{Y}-\mathbf{\xi })\sqrt{\frac{\nu +d}{\nu +Q\left(\mathbf{Y}\right)}};\nu +d\right)\right\}\right].$$

Then, Lemma 1 gives us the required result for this proof.    □

Lemma 1 and Proposition 2 decompose the expected value into two parts; however, it is still necessary to solve the integral (13) that involves parameter $\alpha$. The next Lemma uses the previous results to obtain the skew-t Rényi entropy.

Lemma 2.

Let $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$. Then,

$${\int }_{{\mathbb{R}}^d}{\left[f(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )\right]}^{\alpha }d\mathbf{y}=C_{\alpha ,d}(\mathbf{\Omega },\nu )E\left[{\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel X_0}{\sqrt{\nu +d-1+X_0^2}};\nu +d\right)\right\}}^{\alpha }\right],$$

(15)

where

$$C_{\alpha ,d}(\mathbf{\Omega },\nu )={|\mathbf{\Omega }|}^{-\alpha /2}{\left[B_d\left(\nu \right)\right]}^{\alpha }{B}_{-d}\left(\alpha [\nu +d]\right){\left(\frac{\nu }{\alpha [\nu +d]}\right)}^{d/2},$$

and $X_0\sim T_1(0,1,\beta )$, $\beta =\alpha (\nu +d)-d$, $0<\alpha <\infty$, $\alpha \ne 1$.

Proof. 

Using the change of variables ${\mathbf{z}}_0={\mathbf{\Omega }}^{-1/2}(\mathbf{y}-\mathbf{\xi })$ associated with Jacobian matrix determinant ${|\mathbf{\Omega }|}^{-1/2}$ in Equation (5), we get

$$\begin{array}{ccc}& & {\int }_{{\mathbb{R}}^d}{\left[f(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )\right]}^{\alpha }d\mathbf{y}\hfill \\ & & ={\left(\frac{B_d\left(\nu \right)}{{|\mathbf{\Omega }|}^{1/2}}\right)}^{\alpha }{\int }_{{\mathbb{R}}^d}{\left(1+\frac{{\mathbf{z}}_0^{\top }{\mathbf{z}}_0}{\nu }\right)}^{-\alpha \left(\frac{\nu +d}{2}\right)}{\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }{\mathbf{z}}_0\sqrt{\frac{\nu +d}{\nu +{\mathbf{z}}_0^{\top }{\mathbf{z}}_0}};\nu +d\right)\right\}}^{\alpha }d{\mathbf{z}}_0.\hfill \end{array}$$

By replacing $\beta =\alpha (\nu +d)-d$ and using the change of variables $\mathbf{u}={\mathbf{z}}_0\sqrt{\frac{\beta }{\nu }}$, we obtain

$$\begin{array}{ccc}& & {\int }_{{\mathbb{R}}^d}{\left[f(\mathbf{y};\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )\right]}^{\alpha }d\mathbf{y}\hfill \\ & & ={\left(\frac{B_d\left(\nu \right)}{{|\mathbf{\Omega }|}^{1/2}}\right)}^{\alpha }{\left(\frac{\nu }{\beta }\right)}^{\frac{d}{2}}{\int }_{{\mathbb{R}}^d}{\left(1+\frac{{\mathbf{u}}^{\top }\mathbf{u}}{\beta }\right)}^{-\alpha \left(\frac{u+d}{2}\right)}{\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }\mathbf{u}\sqrt{\frac{\nu +d}{\beta +{\mathbf{u}}^{\top }\mathbf{u}}};\nu +d\right)\right\}}^{\alpha }d\mathbf{u}\hfill \\ & & ={\left(\frac{B_d\left(\nu \right)}{{|\mathbf{\Omega }|}^{1/2}}\right)}^{\alpha }{\left(\frac{\nu }{\beta }\right)}^{\frac{d}{2}}\frac{1}{B_d\left(\nu \right)}E\left[{\left\{2T\left({\overline{\mathbf{\eta }}}^{\top }\mathbf{U}\sqrt{\frac{\nu +d}{\beta +{\mathbf{U}}^{\top }\mathbf{U}}};\nu +d\right)\right\}}^{\alpha }\right],\hfill \end{array}$$

where $\mathbf{U}\sim T_d(\mathbf{0},I_d,\beta )$. The proof is completed by using Lemma 3 of [12].   □

By taking natural logarithm and multiplying by ${(1-\alpha )}^{-1}$ in both sides of Equation (15), the Lemma 2 yields the final expression of skew-t Rényi entropy as is presented in the next Proposition.

Proposition 3.

The Rényi entropy of $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$ is

$$R_{\alpha }\left(\mathbf{Y}\right)=R_{\alpha }\left(\mathbf{X}\right)+\frac{1}{1-\alpha }log\left(E\left[{\left\{2T\left(\frac{\sqrt{\nu +d}\parallel \overline{\mathbf{\eta }}\parallel W_R}{\sqrt{\alpha (\nu +d)-1+W_R^2}};\nu +d\right)\right\}}^{\alpha }\right]\right),$$

(16)

where $R_{\alpha }\left(\mathbf{X}\right)$ is the Student-t Rényi entropy of $\mathbf{X}\sim T_d(\mathbf{\xi },\mathbf{\Omega },\nu )$ given by

$$R_{\alpha }\left(\mathbf{X}\right)=H\left(\mathbf{X}\right)+\frac{1}{1-\alpha }log\left[C_{\alpha ,d}(\mathbf{\Omega },\nu )\right],$$

(17)

$H\left(\mathbf{X}\right)$ is the Student-t Shannon entropy of $\mathbf{X}$ given in (11), $W_R\sim T_1(0,1,\beta )$, $\beta =\alpha (\nu +d)-d$, and $C_{\alpha ,d}(\mathbf{\Omega },\nu )$ is given in Lemma 2.

It is easy to observe that skew-t Shannon entropy is obtained from (16) by taking the limit as $\alpha$ converges to 1.

2.2. Computational Implementation and Numerical Simulations

All numerical computations were made with R software [26]. The integrals of skew-t Rényi entropies of Equation (16) were evaluated using the integrate function of R software’s QUADPACK routine [27]. This method allows to integrate the Student-t cdf of Equation (16) in the interval $(-\infty ,\infty )$. This section illustrates the relationship between parameters $\alpha$, $\mathbf{\eta }$, and $\nu$ with Rényi entropy for $d=1$, 2, 3, and 4 dimensions. Consider $\mathbf{Y}\sim S{T}_d(\mathbf{\xi },\mathbf{\Omega },\mathbf{\eta },\nu )$ with the following cases.

(a) 

$d=1$, $\mathbf{\xi }=\mathbf{0}$, $\mathbf{\Omega }=1.5$, and $\mathbf{\eta }=0.3$.

(b) 

$d=2$, $\mathbf{\xi }=\mathbf{0}$, $\mathbf{\Omega }=\left(\begin{array}{cc}0.7& 0.3\\ 0.3& 3\end{array}\right)$, and $\mathbf{\eta }={\left(\begin{array}{cc}0.3& 2\end{array}\right)}^{\top }$.

(c) 

$d=3$, $\mathbf{\xi }=\mathbf{0}$, $\mathbf{\Omega }=I_3$, and $\mathbf{\eta }={\left(\begin{array}{ccc}0.3& 2& 0.3\end{array}\right)}^{\top }$.

(d) 

$d=4$, $\mathbf{\xi }=\mathbf{0}$, $\mathbf{\Omega }=I_4$, and $\mathbf{\eta }={\left(\begin{array}{cccc}0.3& 2& 0.3& -0.5\end{array}\right)}^{\top }$.

Figure 1 shows the relationship between Rényi entropy of $\mathbf{Y}$ and $\nu =3,\dots ,30$, $\alpha =2,3,4,5,6,8,10$, and $d=1,2,3,4$, related to cases (a)–(d). The Rényi entropies converge to a finite value for all $\alpha$ and $\nu$, and increase for increments of d. The dispersion matrix $\mathbf{\Omega }$ plays an important role in Rényi entropy, mainly by the increment of matrix dimension related to d. Moreover, the Rényi entropy decreases when $\alpha$ increases, as mentioned in the properties of $R_{\alpha }\left(\mathbf{Y}\right)$. The Rényi entropies decrease when $\nu$ increases, yielding the respective Student-t Rényi entropies ($\nu \to \infty$).

Subplot of case (a) considered negative and positive values of $\eta$ ($-6\le \eta \le 6$), producing a symmetry of $R_{\alpha }\left(\mathbf{Y}\right)$ with respect to this parameter. However, as mentioned below, $R_{\alpha }\left(\mathbf{Y}\right)$ decreases for increments of $\nu$. Subplots of cases (b)–(d) cannot represent the behavior of $\mathbf{\eta }$ given this parameter’s dimension $d=2,3,4$ of. For these cases, order $\alpha$ is considered and the Rényi entropy decreases when $\alpha$ increases.

3. Application to Finite Mixtures of Multivariate Skew-$\mathit{t}$ Distributions

Let us consider the definitions of [5,7,28] for FMST distributions. The pdf of an m-component mixture model with parameter vector set $\tilde{\mathbf{\theta }}=(\tilde{\mathbf{\xi }},\tilde{\mathbf{\Omega }},\tilde{\mathbf{\Lambda }},\tilde{\nu })$: $\tilde{\mathbf{\xi }}=({\mathbf{\xi }}_1,\dots ,{\mathbf{\xi }}_m)$, a set of m location vector parameters $\tilde{\mathbf{\Omega }}=({\mathbf{\Omega }}_1,\dots ,{\mathbf{\Omega }}_m)$, a set of m dispersion matrices $\tilde{\mathbf{\Lambda }}=({\mathbf{\eta }}_1,\dots ,{\mathbf{\eta }}_m)$, a set of shape vector parameters $\tilde{\mathbf{\nu }}=({\nu }_1,\dots ,{\nu }_m)$, a set of degree-of-freedom parameters, and with m mixing weights $\mathbf{p}=(p_1,\dots ,p_m)$ is

$$p(\tilde{\mathbf{y}};\tilde{\mathbf{\theta }},\pi )=\sum _{j=1}^m{p}_{j}f({\mathbf{y}}_j;{\mathbf{\theta }}_j),$$

(18)

where $p_j\ge 0$, ${\sum }_{j=1}^m{p}_j=1$, $\tilde{\mathbf{y}}=({\mathbf{y}}_1,\dots ,{\mathbf{y}}_m)$, and $f({\mathbf{y}}_j;{\mathbf{\theta }}_j)$ are defined as in (5) for a known ${\mathbf{\theta }}_j=({\mathbf{\xi }}_j,{\mathbf{\Omega }}_j,{\mathbf{\eta }}_j,{\nu }_j)$, $j=1,\dots ,m$. If $\tilde{\mathbf{Y}}$ has pdf (18), it is denoted as $\tilde{\mathbf{Y}}\sim FMS{T}_d(\tilde{\mathbf{\theta }},\mathbf{p})$.

Let ${\mathbf{S}}_i=(S_{i,1},\dots ,S_{i,m})$ with ${\sum }_{j=1}^m{S}_{ij}=1$ be a set of m latent indicators of observations $\tilde{\mathbf{y}}$, $i=1,\dots ,n$, where j corresponds to a binary index such that $S_{ij}=1$, if ${\mathbf{Y}}_i$ is from group j, and $S_{ij}=0$ otherwise. Then, indicators ${\mathbf{S}}_1,\dots ,{\mathbf{S}}_n$ are independent and each one with multinomial distribution given by

$$p\left(s_i\right)=p_1^{s_{i,1}}{p}_2^{s_{i,2}}\cdots {\left(1-\sum _{j=1}^{m-1}{p}_j\right)}^{s_{i,m}}$$

and denoted as ${\mathbf{S}}_i\sim M(1;p_1,\dots ,p_m)$. Proposition 1 of the work in [28] and (6) allow obtaining a hierarchical representation to each j-th component pdf (see Section 3.1 of [28] for details). This hierarchical representation is useful for parameter estimation based on EM framework for FMST distributions.

The first and second moments of the j-th component ${\mathbf{Y}}_j$, (7) and (8), respectively, get the first two moments of $\tilde{\mathbf{Y}}$:

$$\begin{array}{ccc}\hfill E\left[\tilde{\mathbf{Y}}\right]& =& \sum _{j=1}^m{p}_j\left({\mathbf{\xi }}_j+\frac{B_{-1}\left(\nu \right)}{\pi }{\overline{\mathbf{\eta }}}_j\right),\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill V\left[\tilde{\mathbf{Y}}\right]& =& \sum _{j=1}^m{p}_j\left[{\mathbf{\xi }}_j{\mathbf{\xi }}_j^{\top }+\left(\frac{B_{-2}\left(\nu \right)}{2\pi }\right){\mathbf{\Omega }}_j+{\left(\frac{B_{-1}\left(\nu \right)}{\pi }\right)}^2({\mathbf{\xi }}_j{\mathbf{\delta }}_j^{\top }+{\mathbf{\delta }}_j{\mathbf{\xi }}_j^{\top })+{\mathbf{\mu }}_j{\mathbf{\mu }}_j^{\top }\right],\hfill \end{array}$$

(20)

where

$${\mathbf{\mu }}_j={\mathbf{\xi }}_j+\left(\frac{B_{-1}\left(\nu \right)}{\pi }\right){\overline{\mathbf{\eta }}}_j-E\left[\tilde{\mathbf{Y}}\right],$$

${\overline{\mathbf{\eta }}}_j={\mathbf{\Omega }}_j^{1/2}{\mathbf{\eta }}_j$, and

$${\mathbf{\delta }}_j=\frac{{\mathbf{\Omega }}_j{\mathbf{\eta }}_j}{\sqrt{1+{\mathbf{\eta }}_j^{\top }{\mathbf{\Omega }}_j{\mathbf{\eta }}_j}},j=1,\dots ,m;$$

see, e.g., in [2,6].

Dehesa et al. [24] obtained an upper bound of Rényi entropy using a variational approach that expresses the Rényi entropy of a finite mixture random variable in terms of the dispersion matrix. Specifically, the result provided by [29] allows obtaining a lower bound for Rényi entropy of an FMST in terms of each component (see also [6]), as is presented next.

$$\frac{log\left[E_{\alpha }\left(\tilde{\mathbf{Y}}\right)\right]}{1-\alpha }\le R_{\alpha }\left(\tilde{\mathbf{Y}}\right)\le \frac{d}{2}\left[log\left(\frac{V\left[\tilde{\mathbf{Y}}\right]}{d}\right)+F_{\alpha }\left(d\right)\right],$$

(21)

with

$$E_{\alpha }\left(\tilde{\mathbf{Y}}\right)=e^{(1-\alpha )R_{\alpha }\left({\mathbf{Y}}_m\right)}+\sum _{j=1}^{m-1}\left\{{\left(\sum _{k=1}^j{p}_k\right)}^{\alpha }\left(e^{(1-\alpha )R_{\alpha }\left({\mathbf{Y}}_j\right)}-e^{(1-\alpha )R_{\alpha }\left({\mathbf{Y}}_{j+1}\right)}\right)\right\}$$

and

$$F_{\alpha }\left(d\right)=\left\{\begin{array}{cc}log\left(\frac{\pi \left(\alpha \right(2+d)-d)}{\alpha -1}\right)+\frac{2}{d(\alpha -1)}log\left(\frac{\alpha (2+d)-d}{2\alpha }\right)+\frac{2}{d}log\left\{\frac{\Gamma \left(\frac{\alpha }{\alpha -1}\right)}{\Gamma \left(\frac{\alpha (2+d)-d}{2(\alpha -1)}\right)}\right\},\hfill & \mathrm{if} \alpha >1,\hfill \\ \\ log\left(\frac{\pi \left(\alpha \right(2+d)-d)}{1-\alpha }\right)+\frac{2\alpha }{d(\alpha -1)}log\left(\frac{\alpha (2+d)-d}{2\alpha }\right)+\frac{2}{d}log\left\{\frac{\Gamma \left(\frac{\alpha (2+d)-d}{2(1-\alpha )}\right)}{\Gamma \left(\frac{\alpha }{1-\alpha }\right)}\right\},\hfill & \mathrm{if} \frac{d}{d+2}\le \alpha \le 1,\hfill \\ \\ log\left(2\pi e\right),\hfill & \mathrm{if} \alpha =1.\hfill \end{array}\right.$$

On the left side of the inequality (21), the lower bound depends on Rényi entropy of each mixture component. A proof of this lower bound is available in Lemma 1 of [6], obtained from Proposition 1 (B1) of [29]. We can see that the right side of the inequality (21) depends on the dispersion matrix, the shape parameters, the $\alpha$th order and dimension d. A proof of this upper bound is available in Section 3 of [24]. For the case $\alpha =1$, $F_{\alpha }\left(d\right)$ is related to Shannon entropy of a multivariate standardized normal random variable.

Swordfish Data Analysis

We considered the dataset used in [6], which corresponds to a sample of 486 and 507 length-weight observations of swordfish males and females, respectively. The swordfish data were sampled in the south Pacific off northern Chile during 2011. The observations were obtained using the sampling program of the Instituto de Fomento Pesquero (IFOP, http://www.ifop.cl/). The dataset includes swordfish from 120 to 257 cm and 110 to 299 cm for males and females, respectively.

Following [6], the length-weight nonlinear function $w\left(l\right)=a{l}^b$ is considered to explain the increments in swordfish weight $w\left(l\right)$ in terms of l, where a and b are the theoretical weight at $l=0$ and weight growth rate, respectively [30]. The authors obtained a good fitting of $w\left(l\right)$ after considering a log-transformed $w\left(l\right)$. Then, a two-column matrix formed by these variables is obtained for the clustering procedure. This means no collinearity problem exists, given the nonlinear relationship of length and weight variables. Therefore, the length-weight data are evaluated with the FMST model for $m=1,\dots ,m^{*}$ (where $m^{*}$ is the maximum age by gender) and $d=2$. The FMST parameter estimates were computed using the mixsmsn R software’s library [31].

Figure 2a,b shows the upper bounds for Rényi entropies, for $m^{*}=9$ and 11 for males and females, respectively [6]. Upper bounds are obtained using the right side of inequality (21), where the parameters of each upper bound were replaced by their respective MLEs as plug-in type estimators [6,12,32]. In theses panels, the upper bounds for Rényi entropies are considered because it is necessary to detect the maximum information for given $\alpha$ and m. Importantly, the values increase when the number of components increases for both genders. Only for males, the Rényi entropies increase until $m=8$ and then stabilize. For females, there exists a breakpoint at $m=7$ components; however, the information still increases to $m>7$. For both genders, we can see that information is maximized for $\alpha =2$ (quadratic Rényi entropy) and there also exist some differences between $\alpha$ values. As in Section 2.2, the 3d-subplot illustrates similar behavior, where order $\alpha$ is considered and the Rényi entropy decreases when $\alpha$ increases.

Given that maximum information of the system is given at $\alpha =2$, panel (c) of Figure 2 illustrates the upper and lower Rényi entropies and their average. Lower bounds are obtained using the left side of inequality (21), where the parameters of each lower bound were replaced by their respective MLEs as plug-in type estimators. Females presented the largest averages for each m. For both genders, the averages tend to be similar for all components m. Panel (d) shows the estimated $\nu$ degree of freedom parameters with respect to average Rényi entropy. In general, males presented light-tails ($\nu \gg 30$), whereas females presented heavy-tails ($\nu \le 20$), except in models with $m=3$ and $m=4$ components. For each gender,

Table 1. Summary of FMST models. Upper and lower bounds of $R_{\alpha }\left(\mathbf{Y}\right)$ and their average are computed for $\alpha =2$. For each FMST model and number of clusters m, the AICand BICcriteria are computed.

Gender	m	$\widehat{\mathit{\nu }}$	Upper ${\mathit{R}}_{\mathit{\alpha }}\left(\mathbf{Y}\right)$	Lower ${\mathit{R}}_{\mathit{\alpha }}\left(\mathbf{Y}\right)$	Average ${\mathit{R}}_{\mathit{\alpha }}\left(\mathbf{Y}\right)$	AIC	BIC
Males	2	26.43	18.65	9.98	14.31	7747.17	7809.96
	3	100	19.39	9.05	14.22	7747.83	7844.11
	4	100	19.81	9.27	14.54	7747.76	7877.53
	5	100	20.07	8.44	14.26	7753.50	7916.77
	6	100	20.32	8.87	14.60	7756.28	7953.03
	7	100	20.43	8.38	14.41	7770.71	8000.95
	8	100	20.43	8.01	14.22	7775.86	8039.59
	9	100	20.52	7.82	14.17	7778.41	8075.64
Females	2	20.09	20.10	10.72	15.41	8835.20	8898.62
	3	70.62	20.82	9.88	15.35	8842.99	8940.25
	4	85.97	21.30	9.60	15.45	8838.54	8969.62
	5	16.35	21.45	9.53	15.49	8847.64	9012.55
	6	17.68	21.76	9.25	15.51	8846.07	9044.81
	7	15.99	21.87	9.32	15.60	8850.85	9083.42
	8	17.23	21.91	8.82	15.36	8864.95	9131.34
	9	16.83	21.96	8.72	15.34	8874.23	9174.45
	10	20.02	22.04	8.68	15.36	8892.39	9226.44
	11	11.91	22.08	8.41	15.24	8902.36	9270.24

summarizes these results in detail as described next.

As in [7], the Rényi entropies are compared using the AICand BICcriteria. Inequality (21) is accomplished and the Rényi information increases for parsimonious models (bigger set of m components). The average Rényi entropy rather slowly increases with components $m=2,\dots ,5$, but stabilizes at $m=6$ components for males. A similar phenomenon occurs for females, where the average Rényi entropy is maximum for $m=7$. Figure 3 illustrates the FMST fits for length-weight by gender. The older swordfish lengths present more variability than younger ones. The estimated parameters of FMST model related to males are

$$\begin{array}{l}\pi =(0.274,0.320,0.074,0.178,0.130,0.025),\\ \widehat{\tilde{\mathbf{\xi }}}=\left(\left(\begin{array}{c}167.42\\ 58.10\end{array}\right),\left(\begin{array}{c}155.49\\ 41.65\end{array}\right),\left(\begin{array}{c}197.62\\ 102.32\end{array}\right),\left(\begin{array}{c}179.64\\ 77.11\end{array}\right),\left(\begin{array}{c}138.87\\ 33.15\end{array}\right),\left(\begin{array}{c}213.50\\ 158.73\end{array}\right)\right),\\ \widehat{\tilde{\mathbf{\Omega }}}=(\left(\begin{array}{cc}9.03& 4.74\\ 4.74& 9.57\end{array}\right),\left(\begin{array}{cc}8.91& 3.10\\ 3.10& 7.62\end{array}\right),\left(\begin{array}{cc}11.35& 7.02\\ 7.02& 18.29\end{array}\right),\left(\begin{array}{cc}10.78& 5.35\\ 5.35& 11.21\end{array}\right),\left(\begin{array}{cc}7.58& 3.74\\ 3.74& 4.84\end{array}\right),\\ \left(\begin{array}{cc}20.66& 11.35\\ 11.35& 14.00\end{array}\right)),\\ \widehat{\tilde{\mathbf{\Lambda }}}=\left(\left(\begin{array}{c}0.53\\ 0.72\end{array}\right),\left(\begin{array}{c}-0.92\\ 1.09\end{array}\right),\left(\begin{array}{c}0.94\\ 1.38\end{array}\right),\left(\begin{array}{c}0.79\\ 0.91\end{array}\right),\left(\begin{array}{c}-0.90\\ -0.63\end{array}\right),\left(\begin{array}{c}0.72\\ 0.94\end{array}\right)\right),\end{array}$$

and $\widehat{\nu }=100$.

The estimated parameters of FMST model related to females are

$$\begin{array}{l}\pi =(0.016,0.295,0.095,0.238,0.045,0.217,0.094),\\ \widehat{\tilde{\mathbf{\xi }}}=\left(\left(\begin{array}{c}263.87\\ 291.84\end{array}\right),\left(\begin{array}{c}192.68\\ 82.77\end{array}\right),\left(\begin{array}{c}207.20\\ 119.81\end{array}\right),\left(\begin{array}{c}151.80\\ 41.75\end{array}\right),\left(\begin{array}{c}236.13\\ 211.90\end{array}\right),\left(\begin{array}{c}160.97\\ 51.30\end{array}\right),\left(\begin{array}{c}222.78\\ 156.56\end{array}\right)\right),\\ \widehat{\tilde{\mathbf{\Omega }}}=(\left(\begin{array}{cc}16.69& 13.22\\ 13.22& 53.03\end{array}\right),\left(\begin{array}{cc}11.53& 5.17\\ 5.17& 13.51\end{array}\right),\left(\begin{array}{cc}8.54& 3.76\\ 3.76& 13.97\end{array}\right),\left(\begin{array}{cc}11.65& 6.02\\ 6.02& 7.96\end{array}\right),\left(\begin{array}{cc}16.35& 7.45\\ 7.45& 22.68\end{array}\right),\\ \left(\begin{array}{cc}8.96& 5.35\\ 5.35& 9.73\end{array}\right),\left(\begin{array}{cc}14.71& 8.13\\ 8.13& 22.89\end{array}\right)),\\ \widehat{\tilde{\mathbf{\Lambda }}}=\left(\left(\begin{array}{c}-0.06\\ 0.67\end{array}\right),\left(\begin{array}{c}-0.83\\ 0.64\end{array}\right),\left(\begin{array}{c}1.04\\ 1.20\end{array}\right),\left(\begin{array}{c}-0.79\\ -0.56\end{array}\right),\left(\begin{array}{c}0.97\\ 1.28\end{array}\right),\left(\begin{array}{c}0.84\\ 0.92\end{array}\right),\left(\begin{array}{c}0.75\\ 1.14\end{array}\right)\right),\end{array}$$

and $\widehat{\nu }=20.849$.

Some differences to the FMSN model considered in [6] appear here, where for females the maximum Rényi entropy based on FMSN distributions is obtained for $m=7$.

4. Conclusions and Final Remarks

We derive upper and lower bounds on the Rényi entropy of a multivariate skew-t random variable. Then, we extended these tools to the class of finite mixture of multivariate skew-t densities. Considering the average of these bounds, the approximate value of entropy can be calculated. Both entropies converge to finite value of a multivariate skew-t random variable and its mixture model for any values of $\alpha$ order, $\nu$ degrees of freedom parameter, and dimension d. Given that FMST Rényi entropies are localized between the upper and lower bounds, the average of these bounds can be used as an approximation of the FMST Rényi entropies. In addition, the FMST Rényi entropy bounds provide useful information about the data and could be considered as a criterion to choose the possible number of components in each gender-based group.

We present an application to 2-dimensional length-weight swordfish data. We compared our results with those obtained in [6]. Given that best results of the work in [6] were obtained using the FMSN Rényi entropies, the Rényi entropy bounds of FMST are compared with the FMSN model, rather than the simplest normal model. As in [6], AIC and BIC values increase when the m increases, where the minimum AIC and BIC values correspond to the simplest model with $m=2$ components. This fact is related to data set complexity (high-dimensionality) and parsimonious models (large number of parameters). However, of all these models, the FMST model has smaller AIC and BIC values with respect to those obtained under the FMSN one (see Table 3 of [6]). The latter is produced by the presence of heavy-tails in distributions of female swordfish, as the FMST model is more flexible than the FMSN one.

Finally, we encourage researchers to use the proposed approach for real-world applications and data analysis, such as environmental [32] and biological [7,30] data.