Parameter Estimation of the Dirichlet Distribution Based on Entropy

Abstract

The Dirichlet distribution as a multivariate generalization of the beta distribution is especially important for modeling categorical distributions. Hence, its applications vary within a wide range from modeling cell probabilities of contingency tables to modeling income inequalities. Thus, it is commonly used as the conjugate prior of the multinomial distribution in Bayesian statistics. In this study, the parameters of a bivariate Dirichlet distribution are estimated by entropy formalism. As an alternative to maximum likelihood and the method of moments, two methods based on the principle of maximum entropy are used, namely the ordinary entropy method and the parameter space expansion method. It is shown that in estimating the parameters of the bivariate Dirichlet distribution, the ordinary entropy method and the parameter space expansion method give the same results as the method of maximum likelihood. Thus, we emphasize that these two methods can be used alternatively in modeling bivariate and multinomial Dirichlet distributions.

1. Introduction

In statistics, the method of moments and maximum likelihood are used frequently, details of which can be found in [1,2]. For a long time, their asymptotic properties have been studied in detail [3]. Since the asymptotic distributions of estimators found by these two methods are normal, they have been proven to be very powerful tools for parameter estimation. However, nowadays, alternative estimation methods based on entropy maximization are applied increasingly frequently.

In 1948, [4] defined entropy as a numerical measure of uncertainty, or conversely the information content, associated with a probability distribution $f(x;\theta )$ with parameter $\theta$. It is used to describe a random variable X and is mathematically expressed as

$$I\left[f\right]=-{\int }_{-\infty }^{\infty }f(x;\theta )lnf(x;\theta )dx,{\int }_{-\infty }^{\infty }f(x;\theta )dx=1$$

(1)

for continuous X, where $I\left[f\right]$ can be considered the mean value of $lnf(x;\theta )$. For discrete probability distributions, the integration operator in (1) is simply replaced by the summation operator. Rényi (1961) provided a generalization of Shannon entropy [5]. Rényi entropy is also called $\alpha$-class entropy. For a discrete case, it is defined as

$$H_R=\frac{ln\left({\sum }_{i=1}^K{p}_i^{\alpha }\right)}{1-\alpha }for\alpha >0and\alpha \ne 1$$

(2)

By L’Hôspital’s rule

$$\underset{\alpha \to 1}{lim}\frac{\frac{d}{d\alpha }\left(ln\left({\sum }_{i=1}^K{p}_i^{\alpha }\right)\right)}{\frac{d}{d\alpha }(1-\alpha )}=\underset{\alpha \to 1}{lim}\frac{\frac{{\sum }_{i=1}^K{p}_i^{\alpha }ln{p}_i}{{\sum }_{i=1}^K{p}_i^{\alpha }}}{-1}=-\sum _{i=1}^K{p}_{i}ln{p}_i=H_S$$

(3)

Therefore, Shannon entropy can be evaluated as a special case of Rényi entropy. Another generalization of Shannon entropy was realized by Constantino Tsallis (1988) [5]. Tsallis entropy is also known as $\beta$-class entropy [6]. It is defined as

$$H_T=\frac{1-{\sum }_{i=1}^K{p}_i^{\alpha }}{\alpha -1}for\alpha >0and\alpha \ne 1$$

(4)

By L’Hôspital’s rule,

$$\underset{\alpha \to 1}{lim}\frac{1-{\sum }_{i=1}^K{p}_i^{\alpha }}{\alpha -1}=\underset{\alpha \to 1}{lim}\frac{\frac{d}{d\alpha }(1-{\sum }_{i=1}^K{p}_i^{\alpha })}{\frac{d}{d\alpha }(\alpha -1)}=\underset{\alpha \to 1}{lim}\frac{-{\sum }_{i=1}^K{p}_i^{\alpha }ln{p}_i}{1}=-\sum _{i=1}^K{p}_{i}ln{p}_i$$

(5)

In other words, Tsallis entropy approaches Shannon entropy as $\alpha \to 1$ as well as Rényi entropy. Note that for continuous distributions, the summation signs in defining equations are replaced by integration signs.

Kullback (1959) used entropy and relative entropy as the two key concepts in multivariate statistical analysis [7]. Asymptotic distributions of various entropy measures can be found in [8]. Pardo emphasizes that entropy and relative entropy formulas can be derived as special cases of divergence measures [9].

Entropy-Based Parameter Estimation in Hydrology is the first book to focus on parameter estimation using entropy for a number of distributions frequently used in hydrology [10], including uniform, exponential, normal, two-parameter lognormal, extreme value type I, Weibull, gamma, Pearson, and two-parameter Pareto distributions, among others. Singh also applies entropy theory to some problems of hydraulic and environmental engineering [11,12,13].

The principle of maximum entropy (POME), described by Jaynes as “the least biased estimate possible on the given information”, can be stated mathematically as follows [14]: Given m linearly independent constraints $C_i$ in the form

$$C_i={\int }_a^b{y}_i\left(x\right)f\left(x\right)dx,i=1,2,\dots ,m,$$

(6)

where $y_i\left(x\right)$ are some functions whose averages over $f\left(x\right)$ are specified, the maximum of I, subject to the conditions in Equation (6), is given by the distribution

$$f\left(x\right)=exp\left[-{\lambda }_0-\sum _{i=1}^m{\lambda }_i{y}_i\left(x\right)\right],$$

(7)

where ${\lambda }_i,i=0,1,\dots ,m$ are Lagrange multipliers and can be determined from Equations (6) and (7) along with the normalization condition in Equation (1).

The general procedure for entropy-based parameter estimation involves (1) defining given information in terms of constraints, (2) maximizing entropy subject to given information, and (3) relating parameters to the given information. In this procedure, Lagrange multipliers are related to the constraints on one hand and to the distribution parameters on the other. One can eliminate the Lagrange multipliers and obtain parameter estimations as well.

The parameter space expansion method was developed by Singh and Rajogopal (1986). This method is different from the previous entropy method in that it employs enlarged parameter space and maximizes entropy subject to both the parameters and the Lagrange multipliers [15]. The method works as follows: for the given distribution, first the constraints are defined, and the POME formulation is obtained in terms of the parameters to be estimated and the Lagrange multipliers. After the maximization procedure, the parameter estimations can be obtained.

Entropy-based models have been intensively used for determining parameter estimations in recent years. For example, Song and Kang examined two entropy-based methods that both use the POME for the estimation of the parameters of the four-parameter exponential gamma distribution [16]. Hao and Singh applied two entropy-based methods, also using the POME, for the estimation of the parameters of the extended Burr XII distribution [17]. Singh and Deng revisited the four-parameter kappa distribution, presented an entropy-based method for estimating its parameters, and compared its performance with that of maximum likelihood estimation, methods of moments, and L moments [18]. Gao and Han used the maximum entropy method to apply a concrete solution to a special nonlinear expectation problem in a special parameter space and analyzed the convergence for the maximum entropy solution [19].

The objective of the present paper is to apply ordinary entropy and parameter space expansion to estimate the parameters of a bivariate Dirichlet distribution as an alternative to the known methods, and then to compare them with those estimated by the maximum likelihood method and method of moments.

2. Dirichlet Distribution

The beta distribution plays an important role in Bayesian statistics, especially in modeling the parameters of the Bernoulli distribution [20]. The Dirichlet distribution is a multivariate generalization of the beta distribution. Thus, the Dirichlet distribution and the generalized Dirichlet distribution can both be used as a conjugate prior for a multinomial distribution [21].

Let $X^k=[X_1,X_2,\dots ,X_k]$ be a vector with k components, $X_i\ge 0$ for $i=1,2,\dots ,k$ and ${\sum }_{i=1}^k{x}_i=1.$ Also, $a^k=[a_1,a_2,\dots ,a_k]$, where $a_i>0$ for each i. The probability density function (pdf) of the Dirichlet distribution is given as

$$f\left(x^k\right)=\frac{\Gamma \left(a_0\right)}{{\prod }_{i=1}^k\Gamma \left(a_i\right)}\prod _{i=1}^k{x}_i^{a_i-1},$$

(8)

where $a_0={\sum }_{i=1}^k{a}_i$, $x_i>0$, $x_1+x_2+\cdots +x_{k-1}<1$, and $x_k=1-x_1-\cdots -x_{k-1}$ and $\Gamma$ is the Euler’s gamma function, which is denoted by the formula $\Gamma \left(x\right)={\int }_0^{\infty }{t}^{x-1}{e}^{-t}dt$ or $\Gamma \left(x\right)=(x-1)!$.

It can be noted that marginals of this Dirichlet distribution are beta distributions [22], namely $X_i\sim Beta\left(a_i,\left({\sum }_{j=1}^k{a}_j\right)-a_i\right)$. The moments are given by

$$E\left[X_i\right]=\frac{a_i}{a_0}$$

(9)

$$Var\left[X_i\right]=\frac{a_i(a_0-a_i)}{a_0^2(a_0+1)}$$

(10)

$$Cov(X_i,X_j)=-\frac{a_i{a}_j}{a_0^2(a_0+1)}$$

(11)

$$Cor(X_i,X_j)=\sqrt{\frac{a_i{a}_j}{(a_0-a_i)(a_0-a_j)}}$$

(12)

For further properties, one may refer to [22,23,24].

3. Ordinary Entropy Method

In the ordinary entropy method, there are three steps in parameter estimation: (1) specification of appropriate constraints, (2) derivation of the entropy function of the distribution, and (3) derivation of the relations between parameters and constraints.

3.1. Specification of Constraints

Taking the natural logarithm of Equation (8), we obtain

$$lnf\left(x^k\right)=ln\Gamma \left(a_0\right)-ln\left(\prod _{i=1}^k\Gamma \left(a_i\right)\right)+\sum _{i=1}^{k}ln\left(x_i^{a_i-1}\right)$$

(13)

Multiplying Equation (13) by $[-f\left(x^k\right)]$ and integrating between $[0,1]$ and $[0,1-x_i]$, we obtain the entropy function

$$I\left[f\right]=-\int \dots \int f\left(x^k\right)lnf\left(x^k\right)d{x}_1\dots d{x}_{k-1}=\left[-ln\frac{\Gamma \left(a_0\right)}{{\prod }_{i=1}^k\Gamma \left(a_i\right)}\right]\int \dots \int f\left(x^k\right)d{x}_1\dots d{x}_{k-1}$$

$$-\int \dots \int \sum _{i=1}^{k}ln\left(x_i^{a_i-1}\right)f\left(x^k\right)d{x}_1\dots d{x}_{k-1}$$

(14)

To maximize $I\left[f\right]$ in Equation (14), the following constraints should be satisfied:

$$\int \dots \int f\left(x^k\right)d{x}_1\dots d{x}_{k-1}=1$$

(15)

$$\int \dots \int ln{x}_{i}f\left(x^k\right)d{x}_1\dots d{x}_{k-1}=E\left[ln{x}_i\right],i=1,\dots ,k-1$$

(16)

$$\int \dots \int ln(1-x_1-\dots -x_{k-1})f\left(x^k\right)d{x}_1\cdots d{x}_{k-1}=E[1-x_1-\cdots -x_{k-1}]$$

(17)

3.2. Construction of the Partition Function and Zeroth Lagrange Multiplier

The least biased pdf, $f\left(x^k\right)$ consistent with equations from (15) to (17) and by POME, takes the following form:

$$f\left(x^k\right)=exp[-{\lambda }_0-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})],$$

(18)

where ${\lambda }_0,{\lambda }_1,\dots ,{\lambda }_k$ are Lagrange multipliers. Substituting (18) in (15) yields

$$\int \dots \int exp[-{\lambda }_0-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}=1$$

(19)

Equation (19) gives the partition function as

$$exp\left({\lambda }_0\right)=\int \dots \int exp[-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1},$$

(20)

which may be further simplified as follows:

$$exp\left({\lambda }_0\right)=\int \dots \int \prod _{i=1}^{k-1}{x}_i^{-{\lambda }_i}{(1-x_1-\cdots -x_{k-1})}^{-{\lambda }_k}d{x}_1\dots d{x}_k=\frac{{\prod }_{i=1}^k\Gamma (1-{\lambda }_i)}{\Gamma (k-{\lambda }_1-\dots {\lambda }_k)}$$

(21)

The zeroth Lagrange multiplier ${\lambda }_0$ is obtained from Equation (21) as

$${\lambda }_0=\sum _{i=1}^{k}ln\Gamma (1-{\lambda }_i)-ln\Gamma (k-{\lambda }_1-\dots {\lambda }_k)$$

(22)

The zeroth Lagrange multiplier is also obtained from (20) as

$${\lambda }_0=ln\int \dots \int exp[-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}$$

(23)

3.3. Relation between Lagrange Multipliers and Constraints

Differentiating Equation (23) with respect to ${\lambda }_1,\dots ,{\lambda }_k$, we obtain the derivatives of ${\lambda }_0$ with respect to ${\lambda }_1,\dots ,{\lambda }_k$:

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_1}=-\frac{\int \cdots \int ln{x}_{1}exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}{\int \cdots \int exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}$$

$$=-\int \cdots \int ln{x}_{1}exp[-{\lambda }_0-\sum _{i=1}^{k-1}-{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}$$

$$=-E\left[ln{X}_1\right]$$

(24)

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_2}=-\frac{\int \cdots \int ln{x}_{2}exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}{\int \cdots \int exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}$$

$$=-\int \cdots \int ln{x}_{2}exp[-{\lambda }_0-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}$$

$$=-E\left[ln{X}_2\right]$$

(25)

If it continues like this until $k-1$,

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_{k-1}}=-\frac{\int \cdots \int ln{x}_{k-1}exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}{\int \cdots \int exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}}$$

$$=-\int \cdots \int ln{x}_{k-1}exp[-{\lambda }_0-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-x_1-\cdots -x_{k-1})]d{x}_1\dots d{x}_{k-1}$$

$$=-E\left[ln{X}_{k-1}\right]$$

(26)

Furthermore,

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_k}=-\frac{\int \cdots \int ln(1-\left({\sum }_{i=1}^{k-1}{x}_i\right))exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-\left({\sum }_{i=1}^{k-1}{x}_i\right))]d{x}_1\dots d{x}_{k-1}}{\int \cdots \int exp[-{\sum }_{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-\left({\sum }_{i=1}^{k-1}{x}_i\right))]d{x}_1\dots d{x}_{k-1}}$$

$$=\int \cdots \int ln(1-x_1-\cdots -x_{k-1})exp[-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-\left(\sum _{i=1}^{k-1}{x}_i\right))]d{x}_1\dots d{x}_{k-1}$$

$$=-E\left[ln(1-x_1-\cdots -x_{k-1})\right]$$

(27)

Differentiating Equation (22) with respect to ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k,$ we obtain

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_1}=\frac{\partial ({\sum }_{i=1}^{k}ln\Gamma (1-{\lambda }_i)-ln\Gamma (k-{\lambda }_1-\dots {\lambda }_k))}{\partial {\lambda }_1}=-\psi (1-{\lambda }_1)+\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(28)

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_2}=\frac{\partial ({\sum }_{i=1}^{k}ln\Gamma (1-{\lambda }_i)-ln\Gamma (k-{\lambda }_1-\dots {\lambda }_2))}{\partial {\lambda }_2}=-\psi (1-{\lambda }_2)+\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(29)

Similarly, for the k term,

$$\frac{\partial {\lambda }_0}{\partial {\lambda }_k}=\frac{\partial ({\sum }_{i=1}^{k}ln\Gamma (1-{\lambda }_i)-ln\Gamma (k-{\lambda }_1-\dots {\lambda }_k))}{\partial {\lambda }_k}=-\psi (1-{\lambda }_k)+\psi (k-{\lambda }_1-\dots {\lambda }_k),$$

(30)

where $\psi \left(x\right)$ is the digamma function, which is defined as $\psi \left(x\right)=\frac{d}{dx}ln(\Gamma \left(x\right))$ [25].

By equating (24) and (28), we obtain

$$E\left[ln{X}_1\right]=\psi (1-{\lambda }_1)-\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(31)

Secondly, by equating (25) and (29), we obtain

$$E\left[ln{X}_2\right]=\psi (1-{\lambda }_2)-\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(32)

If we go on until the $(k-1)$ term

$$E\left[ln{X}_{k-1}\right]=\psi (1-{\lambda }_{k-1})-\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(33)

Next, by (27) and (30), we obtain

$$E\left[ln(1-x_1-\dots -x_{k-1})\right]=\psi (1-{\lambda }_k)-\psi (k-{\lambda }_1-\dots {\lambda }_k)$$

(34)

3.4. Relation between Lagrange Multipliers and Parameters

Substituting (22) into (18) yields

$$f\left(x^k\right)=exp[-\sum _{i=1}^{k}ln\Gamma (1-{\lambda }_i)+ln\Gamma (k-{\lambda }_1-\dots {\lambda }_k)-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i-{\lambda }_{k}ln(1-\left(\sum _{i=1}^{k-1}{x}_i\right))]$$

(35)

$$=\frac{\Gamma (k-{\lambda }_1-\dots {\lambda }_k)}{{\prod }_{i=1}^k\Gamma (1-{\lambda }_i)}\prod _{i=1}^k{x}_i^{-{\lambda }_i}$$

A comparison of Equation (35) with Equation (8) shows that

$$a_i=1-{\lambda }_i$$

(36)

3.5. Relation between Parameters and Constraints

The parameters of the Dirichlet distribution are related to the Lagrange multipliers. In turn, these parameters are related to the known constraints by equations from (31) to (34). By eliminating Lagrange multipliers from these sets of equations, we can obtain an alternative way of presentation as shown below:

$$E\left[ln{X}_1\right]=\psi \left(a_1\right)-\psi \left(a_0\right)$$

(37)

$$E\left[ln{X}_2\right]=\psi \left(a_2\right)-\psi \left(a_0\right)$$

(38)

$$⋮$$

$$E\left[ln{X}_{k-1}\right]=\psi \left(a_{k-1}\right)-\psi \left(a_0\right)$$

(39)

$$E\left[ln(1-x_1-\cdots -x_{k-1})\right]=\psi \left(a_k\right)-\psi \left(a_0\right)$$

(40)

3.6. Distribution Entropy

From (14),

$$I\left[f\right]=-\int \cdots \int f\left(x^k\right)lnf\left(x^k\right)d{x}_1\dots d{x}_{k-1}$$

$$=\left[-ln\frac{\Gamma \left(a_0\right)}{{\prod }_{i=1}^k\Gamma \left(a_i\right)}\right]-\int \cdots \int \sum _{i=1}^{k}ln\left(x_i^{a_i-1}\right)f\left(x^k\right)d{x}_1\dots d{x}_{k-1}$$

(41)

$$=\left[ln\frac{{\prod }_{i=1}^k\Gamma \left(a_i\right)}{\Gamma \left(a_0\right)}\right]-\sum _{i=1}^k(a_i-1)ln{x}_i$$

4. Parameter Space Expansion Method

4.1. Specification of Constraints

Following [15], the constraints for this method are Equation (15) and

$$\int \cdots \int ln{x}_i^{a_i-1}f\left(x^k\right)d{x}_1\cdots d{x}_{k-1}=E\left[ln{X}_i^{a_i-1}\right],i=1,\cdots ,k-1$$

(42)

$$\int ...\int ln{(1-x_1-\cdots -x_{k-1})}^{a_k-1}f\left(x^k\right)d{x}_1\cdots d{x}_{k-1}=E\left[{(1-x_1-\cdots -x_{k-1})}^{a_k-1}\right]$$

(43)

4.2. Derivation of the Entropy Function

The pdf that corresponds to the POME and that is consistent with Equation (15), (42), and (43) takes the form

$$f\left(x^k\right)=exp[-{\lambda }_0-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i^{a_i-1}-{\lambda }_{k}ln{(1-x_1-\cdots -x_{k-1})}^{a_k-1}],$$

(44)

where ${\lambda }_0,{\lambda }_1,\cdots ,{\lambda }_k$ are Lagrange multipliers. Substituting (44) into Equation (15) yields

$$exp\left({\lambda }_0\right)=\int ...\int exp[-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i^{a_i-1}-{\lambda }_{k}ln{(1-x_1-\cdots -x_{k-1})}^{a_k-1}d{x}_1\cdots d{x}_{k-1}]$$

$$=\frac{{\prod }_{i=1}^k\Gamma (1-{\lambda }_i(a_i-1))}{\Gamma (k-{\lambda }_1(a_1-1)-\dots {\lambda }_k(a_k-1))}$$

(45)

Substitution of Equation (45) into (44) gives

$$f\left(x^k\right)=\frac{\Gamma (k-{\lambda }_1(a_1-1)-\dots {\lambda }_k(a_k-1))}{{\prod }_{i=1}^k\Gamma (1-{\lambda }_i(a_i-1))}exp[-\sum _{i=1}^{k-1}{\lambda }_{i}ln{x}_i^{a_i-1}-{\lambda }_{k}ln{(1-\left(\sum _{i=1}^{k-1}{x}_i\right))}^{a_k-1}]$$

(46)

A comparison of Equation (46) with Equation (15) shows that ${\lambda }_1=\dots {\lambda }_k=-1$ and taking the logarithm of (46) and multiplying by $[-f\left(x^k\right)]$ and integrating between $[0,1]$ and $[0,1-x_i]$, we obtain the entropy function

$$I\left[f\right]=-ln\Gamma (k-{\lambda }_1(a_1-1)-\dots {\lambda }_k(a_k-1))+\sum _{i=1}^k\Gamma (1-{\lambda }_i(a_i-1))+\sum _{i=1}^k{\lambda }_{i}E\left[ln{x}_i^{a_i-1}\right]$$

(47)

4.3. Relation between Parameters and Constraints

Equating the partial derivatives of (47) with respect to ${\lambda }_1,\dots ,{\lambda }_k,a_1,\dots ,a_k$ to zero, one obtains

$$\frac{\partial I\left[f\right]}{\partial {\lambda }_1}=0=(a_1-1)\psi \left(K_1\right)-(a_1-1)\psi \left(K_2\right)+E\left[ln{X}_1^{(a_1-1)}\right]$$

(48)

$$K_1=(k-{\lambda }_1(a_1-1)-\dots {\lambda }_k(a_k-1)),K_2=(1-{\lambda }_1(a_1-1))$$

$$⋮$$

$$\frac{\partial I\left[f\right]}{\partial {\lambda }_k}=0=(a_k-1)\psi \left(K_1\right)-(a_k-1)\psi \left(K_k\right)+E\left[ln{X}_k^{(a_k-1)}\right]$$

(49)

$$K_k=(k-{\lambda }_k(a_k-1))$$

$$\frac{\partial I\left[f\right]}{\partial a_1}=0={\lambda }_1\psi \left(K_1\right)-{\lambda }_1\psi \left(K_2\right)+{\lambda }_{1}E\left[ln{X}_1\right]$$

(50)

$$⋮$$

$$\frac{\partial I\left[f\right]}{\partial a_k}=0={\lambda }_k\psi \left(K_1\right)-{\lambda }_k\psi \left(K_k\right)+{\lambda }_{k}E\left[ln{X}_k\right]$$

(51)

The simplification of equations from (48) to (51) yields

$$E\left[ln{X}_1\right]=\psi \left(K_2\right)-\psi \left(K_1\right)$$

(52)

$$⋮$$

$$E\left[ln{X}_k\right]=\psi \left(K_k\right)-\psi \left(K_1\right)$$

(53)

These equations provide the parameter estimators of the Dirichlet distribution.

5. Two Other Parameter Estimation Methods

5.1. Method of Moments

The Dirichlet distribution has k parameters like $a_i,i=1,\dots ,k$. Therefore, i moments are needed for the parameter estimation. We have the moments, variances, and covariance formula in (9), (10), and (11). Because of (11),

$$E\left(X_i{X}_j\right)=Cov(X_i,X_j)+E\left(X_i\right)E\left(X_j\right)=\frac{a_i{a}_j}{a_o(a_0+1)}$$

(54)

$$\frac{E\left(X_i{X}_j\right)}{Cov(X_i,X_j)}=-a_0$$

(55)

If we multiply (55) by the negative of (9),

$$\widehat{a_i}=-E\left(X_i\right)\frac{E\left(X_i{X}_j\right)}{Cov(X_i,X_j)}$$

(56)

Due to (55) and (56), the last parameter estimation is

$$\widehat{a_k}=\frac{-E\left(X_i{X}_j\right)(1-E\left(X_1\right)-\dots E\left(X_{k-1}\right)}{Cov(X_i,X_j)}$$

(57)

5.2. Method of Maximum Likelihood Estimation

The likelihood function L, where n is the sample size, is

$$L={\left[\frac{\Gamma \left(a_0\right)}{{\prod }_{i=1}^k\Gamma \left(a_i\right)}\right]}^n\prod _{j=1}^n\left(\prod _{i=1}^k{x}_{ij}^{a_i-1}\right)$$

(58)

Then the log likelihood function, $lnL$, is

$$lnL=nln\Gamma \left(a_0\right)-n\sum _{i=1}^{k}ln\Gamma \left(a_i\right)+\sum _{j=1}^{n}ln\left[\prod _{i=1}^k{x}_{ij}^{a_i-1}\right]$$

(59)

Differentiating Equation (59) with respect to parameters $a_1,\dots ,a_k$, respectively, and equating each derivative to zero yield the following equations:

$$E\left[ln{X}_1\right]=\psi \left(a_1\right)-\psi \left(a_0\right)$$

(60)

$$E\left[ln{X}_2\right]=\psi \left(a_2\right)-\psi \left(a_0\right)$$

(61)

$$⋮$$

$$E\left[ln{X}_{k-1}\right]=\psi \left(a_{k-1}\right)-\psi \left(a_0\right)$$

(62)

$$E\left[ln(1-x_1-\cdots -x_{k-1})\right]=\psi \left(a_k\right)-\psi \left(a_0\right)$$

(63)

These results are the same as those found by the ordinary entropy method and parameter space expansion method.

The maximum likelihood (ML) estimation method provides singular point estimates for model parameters while overlooking the residual uncertainty inherent in the estimation process. Conversely, the Bayesian estimation method adopts a different approach, yielding posterior probability distributions encompassing the entire spectrum of model parameters. This is achieved by integrating the observed data with prior distributions [26]. Broadly speaking, when contrasted with ML estimation, Bayesian parameter estimation within a statistical model has the potential to yield a robust and stable estimate. This is primarily due to its ability to incorporate the accompanying uncertainty into the estimation process, a particularly valuable attribute when dealing with limited amounts of observed data [27]. The Dirichlet distribution, being a constituent of the exponential family, possesses a corresponding conjugate prior. Nevertheless, due to the intricate nature of the posterior distribution, its practical utility in problem-solving scenarios is limited. Consequently, the task of Bayesian estimation for the Dirichlet distribution, in a general context, lacks analytical tractability. To achieve this objective, Zao employed an approximation approach to model the parameter distribution within the Dirichlet distribution. Specifically, they approximated it with a multivariate Gaussian distribution, leveraging the expectation propagation (EP) framework [28]. Furthermore, there are some studies in reliability engineering which estimate parameters with the determination of quantiles by the application of the maximum likelihood method, such as [29].

6. Simulation and Comparison of Parameter Estimation Methods

Simulation from the Dirichlet distribution can be performed in two steps: the probability integral theorem states that the distribution function of any continuous distribution is uniform on $(0,1)$. Then, by the inverse distribution function of gamma, one may simulate a number of independent gamma variates as needed. In other words, first one may simulate k independent gamma variates $X_1,X_2,\dots ,X_n$ such that $X_i\sim Gamma({\alpha }_i,1),i=1,2,\dots ,k$ and calculate $Y_j=\frac{X_j}{{\sum }_{i=1}^k{X}_i}$; $j=1,2,\dots ,k$. Then the random vector $(Y_1,Y_2\dots ,Y_k)$ fits the Dirichlet distribution, having parameter vector $({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_k)$ [30]. This procedure can easily be realized even by Microsoft Excel.

In the present study, we first simulated 1000 $(X,Y)$ pairs from the Dirichlet distribution for some arbitrary parameters ${\alpha }_1,{\alpha }_2$ and ${\alpha }_3$. We obtained estimates obtained by four methods but since the maximum likelihood estimators and estimators obtained by the ordinary entropy method and parameter space expansion method are all the same, the comparison is between moment estimates and the rest. Then we repeated this experiment 5000 times. The summarizing statistics are as shown in Table 1:

Table 1. Results of some simulations (1000 runs, 5000 runs).

1000 runs/5000 runs	${\mathit{\alpha }}_1=3$	${\mathit{\alpha }}_2=2$	${\mathit{\alpha }}_3=4$
MOM	2.79/2.92	1.86/1.94	3.81/3.88
APE	6.75/2.35	6.99/2.9	4.73/2.94
MLE	3.06/3.23	2.21/2.11	4.64/4.32
APE	2.07/7.85	10.5/5.98	16.24/8.1
1000 runs/5000 runs	${\mathbf{\alpha }}_{\mathbf{1}}=\mathbf{4}$	${\mathbf{\alpha }}_{\mathbf{2}}=\mathbf{0}.\mathbf{25}$	${\mathbf{\alpha }}_{\mathbf{3}}=\mathbf{2}$
MOM	3.96/3.94	0.25/0.24	1.96/1.97
APE	0.99/1.26	2.82/0.05	1.53/1.18
MLE	4.28/4.02	0.26/0.22	2.03/1.96
APE	7.01/0.57	4.27/9.69	1.85/1.79
1000 runs/5000 runs	${\mathbf{\alpha }}_{\mathbf{1}}=\mathbf{0}.\mathbf{5}$	${\mathbf{\alpha }}_{\mathbf{2}}=\mathbf{3}$	${\mathbf{\alpha }}_{\mathbf{3}}=\mathbf{2}$
MOM	0.48/0.5	3.1/3.09	2.09/2.04
APE	3.91/1.94	3.36/3.13	4.67/2.09
MLE	0.61/0.5	3.39/3.1 1	2.32/2.06
APE	22.48/0.58	13.01/3.92	16.1/3.34
1000 runs/5000 runs	${\mathbf{\alpha }}_{\mathbf{1}}=\mathbf{3}$	${\mathbf{\alpha }}_{\mathbf{2}}=\mathbf{3}$	${\mathbf{\alpha }}_{\mathbf{3}}=\mathbf{4}$
MOM	3.01/3.04	3.05/2.99	4.13/4.04
APE	0.49/1.63	1.82/0.36	3.38/1.04
MLE	3.22/2.86	3.41/2.88	4.61/3.61
APE	7.57/4.38	13.67/3.71	15.48/9.64
1000 runs/5000 runs	${\mathbf{\alpha }}_{\mathbf{1}}=\mathbf{13}$	${\mathbf{\alpha }}_{\mathbf{2}}=\mathbf{2}$	${\mathbf{\alpha }}_{\mathbf{3}}=\mathbf{0}.\mathbf{75}$
MOM	14.04/12.99	2.1/1.98	0.83/0.73
APE	8/0.02	5.34/0.56	11/1.45
MLE	12.52/13.11	1.96/2.02	0.75/0.69
APE	3.64/0.87	1.89/1.34	0.72/7.85

Note that the maximum likelihood estimates (MLEs) are obtained by Excel Solver. In general, moment estimates and MLEs are close to each other. Absolute percentage errors (APEs) are calculated by the following formula:

$$APE=\frac{100\ast |parameter-estimate|}{parameter}$$

(64)

Then, it can be inferred that one measure does not dominate all the time, i.e., there are some instances in which moment estimators perform better than the others, and there are other instances in which maximum likelihood estimators do it better. In any case, it is definite from the table that increasing the number of simulations increases precision considerably.

Note that, by the central limit theorem, moment estimators are expected to be distributed normally for a large number of observations (or simulations) since a moment estimator considers a sum of random observations (or the sum of some power of these random observations). Maximum likelihood estimators also have the asymptotic normality property with lower variances. For the Dirichlet distribution, we found that the entropy estimators mentioned above and maximum likelihood estimators are identical. Therefore, entropy estimators also have the asymptotic normality property.

Finally, we note that the selection of $({\alpha }_1,{\alpha }_2,{\alpha }_3)$ is quite arbitrary just for addressing the fact that maximum likelihood estimators (and maximum entropy estimators) are better (i.e., show lower sampling variability as compared to moment estimators). Actually, this was not the case all the time in the simulations. Since, in our study, the initial estimates of maximum likelihood (and maximum entropy) are provided by the method of moments and since the initial estimates provided by moments are close enough to the actual parameters, a great improvement in sampling variability may not be achieved. This is probably due to the nature of nonlinear estimation. To have a better picture, first simulating a random vector $({\alpha }_1,{\alpha }_2,{\alpha }_3)$ several times, then trying to calculate moment estimates and then, based on these initial estimates, moving forward to maximum likelihood estimates may be meaningful.

7. Conclusions

In the present study, parameter estimations of the Dirichlet distribution are obtained by four methods. For a Dirichlet distribution with three parameters, parameter estimates found by entropy methods (that we considered here) and by maximum likelihood are almost the same. Maximum likelihood estimators are consistent, most efficient, sufficient, tend to normality (as the sample size increases), and are invariant under functional transformations [31]. Therefore, parameter estimators found by the entropy methodology have the same appealing properties as the maximum likelihood estimators. Based on the fact that the sample moment will tend to be more concentrated for the corresponding population moment for larger samples, a sample moment can also be used to estimate population moments [1]. In general, moment estimators are asymptotically normally distributed and consistent. However, their variance may be larger than that of estimators derived by other methods [32]. Yet, it may be a good idea to start nonlinear estimation either for maximum likelihood or entropy maximization methods with initial moment estimates. In the present study, we started with moment estimates of a Dirichlet distribution with arbitrarily selected parameters to demonstrate that better parameter estimates (i.e., estimates both with lower bias and lower sampling variability) can be achieved. The simulation part of this work can be enlarged by determining parameters randomly by further simulations for further generalizations.