On the Optimal Point of the Weighted Simpson Index

Abstract

In this short communication, following a brief introduction, we undertake a comprehensive analytical study of the weighted Simpson index. Our primary emphasis concerns the precise determination of the optimal point (minimizer) coordinates and of the minimum value of the index, a differentiable convex function, which is related to the harmonic mean concept. Furthermore, we address and solve the inversion problem and show the tight connection between both approaches. Last, we give some insights and final remarks on this subject.

1. Introduction

The Simpson index concerning a population distributed among k categories or classes is defined as:

$$S={\sum }_{i=1}^k{p}_i^2,$$

where $p_i$ denotes the probability (or proportion of occurrences) of class i. So, one has $p_i\ge 0$ and ${\sum }_{i=1}^k{p}_i=1$, and therefore S is defined on a $k-1$ simplex. This index equals the probability that two elements taken at random from the population of interest belong to the same category or class. The value of Simpson’s index ranges from $1/k$ to 1, with 1 representing no diversity; so, the larger the value of $S$, the lower the diversity. The name “Simpson index” roots from the influential 1949 paper by Edward Hugh Simpson entitled “Measurement of Diversity” [1], wherein he introduced what he called a measure of concentration defined in terms of population constants, with the minimum concentration equaling the maximum diversity. The Simpson index became a widely used quantitative metric in ecological and biodiversity studies as a tool for assessing and quantifying the diversity and evenness of species within ecological communities. It also applies to other biological problems, including biomedical sciences, such as measuring diversity concerning immunity in response to viral infections (e.g., [2]).

However, it is also acknowledged that the original mathematical concept formulation was used in cryptanalysis as far back as the 1920s and 30s—therein named probability of monographic coincidence—by the American cryptanalysts William Friedman and Solomon Kullback (e.g., [3]). It is relevant to note that the Italian statistician Corrado Gini had already applied the quantity $1-{\sum }_{i=1}^k{f}_i^2$ as early as 1912. He defined the index with relative frequencies $f_i$ computed from large samples, referring to it as an index of mutability for disconnected (qualitative) variables [4]. This quantity became later known as the “Gini-Simpson index”, a name adopted in the 1980s by the eminent statistician C. R. Rao (e.g., [5,6]), who restated it with probabilities as $G={\sum }_{i=1}^k{p}_i(1-p_i)=1-{\sum }_{i=1}^k{p}_i^2$. For instance, Jian et al. [7] consider a “Livelihood Simpson Index”, which in fact is a Gini–Simpson index. Obviously, given a probability distribution $\left(p_1, \dots ,p_k\right)$, the Simpson and Gini–Simpson indices correspond to complementary events, verifying $S+G=1$.

The use of a weighted version of the Simpson index appears to have been first reported in 1992, when Nowak and May [8] conceived the effective immune response against the virus population composed of different strains in the context of HIV infections, which was then revisited two years later [9].

Some refer to the weighted Simpson index when they are actually dealing with the weighted Gini–Simpson index (e.g., [10,11]). The weighted Gini–Simpson index is defined as ${GS}_w={\sum }_{i=1}^k{w}_i{p}_i(1-p_i)$, a concave function, differentiable in the interior of a simplex, with an identifiable maximum value for which a method to determine the optimal point (maximizer) was framed based on the fact that one is dealing with feasibility values associated with the constraints of the simplex [12]—namely, that the optimal coordinates must verify $p_i^{*}\ge 0$—which were not taken into account in [11] and may lead to miscalculations.

Yet, Kasulo and Perrings used a price-weighted Simpson index to assess scenarios relative to the connection between the diversity of catch in a multi-species fishery and profit-maximizing regimes [13], and Ma used a symmetric form of the weighted Simpson index for ranking alternatives under a decision-making procedure involving mixed attribute values [14].

Despite the many recent publications addressing diversity, including phylogenetic diversity, and dissimilarity indices—either in biology (e.g., [15,16,17]) or, following the pioneer approach of Patil and Taillie [18], in the social sciences (e.g., [19,20]), mostly associated with developments relative to Hill’s numbers [21] and Rényi entropies [22]—it seems that an analytical study concerning the optimal point (minimizer) and the optimal value (minimum) of the weighted Simpson index has not yet been published.

In this short communication, we undertake a comprehensive analytical study of the weighted Simpson index focusing on the optimization problem. We do not deal with statistical developments, which could be addressed in the future, presumably within the scope of information measures (e.g., [23]).

Concerning the structure of this paper, in Section 2, we solve both the optimization problem and the inverse problem and discuss different normalization procedures of the index in relation to the results obtained and, in Section 3, we present some final remarks.

2. The Weighted Simpson Index

Herein, we define the weighted Simpson index, concerning a population distributed among k categories or classes, as:

$$S_w=\sum _{i=1}^k{w}_i{p}_i^2$$

(1)

where $p_i$ denotes the probability (or proportion of occurrences) of class i and $w_i>0$ is a weight assigned to that class, altogether defining a vector of positive real values ${w=\left(w_1,\dots ,w_k\right)}^t$. For now, we have decided not to impose any extra conditions on the weights, leaving this matter to be discussed later. Our current focus is on understanding the broader context.

Weights allow one to consider various features for the classes. In the context of biodiversity, these features may be related, for example, to environmental benefits, conservation importance, and the vulnerability or economic value of species, a subject that was already emphasized at least since the beginning of the 1980s, and exemplified with biomass and other importance values [24].

2.1. Optimizing the Weighted Simpson Index

Now, we will address the optimization problem concerning the weighted Simpson index $S_w$ for fixed weights $w_1,\dots ,w_k.$ We recall that the lower the value of $S_w$, the greater the diversity of a composition associated with a positive valuation of the different classes or corresponding events. In general, for a weighted index, diversity is not maximized by the uniform distribution of classes. So, it is natural to ask the following: which distribution of the classes minimizes $S_w$ and what is its minimum value? In the following proposition, we derive this optimal distribution and the minimum value of $S_w$ and determine the maximum value of $S_w$ (this one indicating a minimum diversity’s value, what occurs in a vertex of the simplex) as well.

Proposition 1.

Given  ${w=\left(w_1,\dots ,w_k\right)}^t$a vector of positive real numbers and the weighted Simpson index $S_w={\displaystyle {\sum }_{i=1}^k}{w}_i{p}_i^2$ defined on the simplex  ${\Delta }^{k-1}=\left\{\left(p_{1,}\dots ,p_k\right)\in {\mathbb{R}}^k: p_i\ge 0, {\sum }_{i=1}^k{p}_i=1\right\}$,

(a) 

the maximum value of  $S_w$ is given by $\underset{i}{\mathit{max}}{w}_i$;

(b) 

the minimum value of  $S_w$ is given by

$${ S}_w^{*}=\frac{1}{{\sum }_{i=1}^k\frac{1}{w_i}}$$

(2)

and corresponds to the distribution

$$p_j^{*}=\frac{1}{w_j\left({\sum }_{i=1}^k \frac{1}{w_i}\right)} \mathrm{for} j=1,\dots ,k.$$

(3)

Proof. 

The weighted Simpson index $S_w$ is a (continuous) convex function defined in the compact domain ${\mathsf{\Delta }}^{k-1}$and thus the extreme value theorem ensures the existence of the (global) maximum and minimum values of the index. So, $S_w$ attains its absolute maximum at the boundary of ${\mathsf{\Delta }}^{k-1}$ and its absolute minimum at the interior of ${\mathsf{\Delta }}^{k-1}$. Clearly, the maximum is attained when all except one of the $p_i$ are zero, and therefore one has $\mathit{max}{S}_w=\underset{i}{\mathit{max}}{w}_i$.

The minimum can be assessed with the method of Lagrange multipliers for finding the critical points of $S_w$ subject to the equality constraint ${\sum }_{i=1}^k{p}_i=1$. As $S_w$ is differentiable in the interior of ${\mathsf{\Delta }}^{k-1}$, one can build the Lagrangian function:

$$L_w\left(p_1,\dots ,p_k;\mathsf{\lambda }\right)=S_w\left(p_1,\dots ,p_k\right)+\mathsf{\lambda }\left({\sum }_{i=1}^k{p}_i-1\right).$$

Equating partial derivatives to zero, we get:

$$\left\{\begin{array}{c}\frac{\partial L_w}{\partial p_1}=0\\ ⋮\\ \frac{\partial L_w}{\partial p_k}=0\\ \frac{\partial L_w}{\partial \mathsf{\lambda }}=0\end{array}\right.⟺\left\{\begin{array}{c}2w_1{p}_1+\lambda =0\\ ⋮\\ 2w_k{p}_k+\lambda =0\\ {\sum }_{i=1}^k{p}_i-1=0 \end{array}\right.$$

(4)

From the first $k$ equations in (4), we conclude that for any specific $j\in \left\{1, 2,\dots ,k\right\}$, the following equivalence holds:

$$w_j{p}_j=-\lambda /2 ⟺ p_j=-\frac{\lambda }{2w_j} .$$

Then, using the constraint ${\sum }_{j=1}^k{p}_j=1$ one has $-\frac{\mathsf{\lambda }}{2}$ ${\sum }_{i=1}^k\frac{1}{w_i}=1 ⟺ w_j{p}_j$ ${\sum }_{i=1}^k\frac{1}{w_i}=1$ and, recalling the argument of a convex function, we obtain the optimal coordinates of the minimum point given by:

$$p_j^{*}=\frac{1}{w_j\left({\sum }_{i=1}^k \frac{1}{w_i}\right)} \mathrm{for} j=1,\dots k.$$

Now one can evaluate the minimum value of the weighted Simpson index (1) as follows:

$$S_w^{*}=S_w\left(p_1^{*}, \dots , p_k^{*}\right)={\sum }_{j=1}^k{w}_j{\left(\frac{1}{w_j\left({\sum }_{i=1}^k\frac{1}{w_i}\right)}\right)}^2=\frac{1}{{\left({\sum }_{i=1}^k\frac{1}{w_i}\right)}^2}{\sum }_{j=1}^k\frac{1}{w_j}=\frac{1}{{\sum }_{i=1}^k\frac{1}{w_i}}.\square$$

2.2. Some Further Comments on the Minimizer

Note that the minimum value of the weighted Simpson index (2) is related to the harmonic mean of the weights H$\left(w\right)$ by $S_w^{*}=\mathrm{H}\left(w\right)/k$. The harmonic mean $\mathrm{H}$ of $k$ positive real numbers $x_1,\dots ,x_k$ is the reciprocal of the arithmetic mean of the reciprocals of those numbers:

$$\mathrm{H}\left(x_1,\dots ,x_k\right)=\frac{k}{\frac{1}{x_1}+\cdots +\frac{1}{x_k}},$$

and is appropriate for averaging rates over constant numerator units [25]. As a typical example, if a set of investments are invested at different interest rates, and they all give the same income, the unique rate at which all of the capital tied up in those investments must be invested to produce the same revenue as given by the set of investments, is equal to the harmonic mean of the individual rates ([26], p. 240).

The special case of all weights being equal to 1 leads to the uniform distribution: $p_i^{*}=1/k, \forall i$ and to the minimum value $S_w^{*}=1/k$, as can be seen from expressions (3) and (2), and also because in this case, $S_w=S$ with $S$ being the (unweighted) Simpson index whose minimum is $1/k$.

Rewriting the optimal distribution in (3) as:

$$p_j^{*}=\frac{1}{{1+w}_j\left({\displaystyle {\sum }_{i\ne j}}\frac{1}{w_i}\right)} \mathrm{for} j=1,\dots ,k$$

it is straightforward to conclude that the weights are driving forces of the optimal probabilities, operating reciprocally. When the weight attached to a specific class increases and all the others keep invariant, the corresponding optimal probability decreases, and when a weight attached to another class increases, all the others being invariant, the original class increases its optimal probability.

If one considers the valuation of the classes of a distribution in the usual sense of importance assessment with an ordering of positive real numbers, expecting that $v_m>v_n$ would promote the result $p_m^{*}>p_n^{*}$, then one should be aware that the weights associated with the optimal point (3) would not be the values $\left\{v_m,v_n\right\}$, but could possibly be conceived like $w_m=1/v_m$ and $w_n=1/v_n$ instead.

2.3. Normalization

In the realm of indices, normalization comprises the adjustment of the index values to conform to a predetermined range or interval. The use of normalized indices in applications can be important for several reasons. Normalized indices provide a standardized scale, usually ranging from 0 to 1, or 0% to 100%, irrespective of the specific scale or magnitude of the data. This standardization allows for direct comparisons between different datasets or populations and remains consistent across different contexts and scales.

In the case of the weighted Simpson index, this can be done in a classic way defining the normalized weighted Simpson index as:

$$s_w=\frac{S_w-\mathrm{m}\mathrm{i}\mathrm{n}\left(S_w\right)}{\mathrm{m}\mathrm{a}\mathrm{x}\left(S_w\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(S_w\right)}=\frac{S_w{\sum }_{i=1}^k\frac{1}{w_i}-1}{\mathrm{m}\mathrm{a}\mathrm{x}\left(w_i\right){\sum }_{i=1}^k\frac{1}{w_i}-1}.$$

However, this normalization eliminates the effect of the number of classes in the distribution (e.g., species in a community). For example, in the case of all weights equal to 1, the normalized weighted Simpson index of a population with $k$ species uniformly distributed is always 0, and thus independent of the number of species. The fact that normalized indices of diversity can be misleading has already been mentioned by several authors (e.g., [11]).

In the case of a weighted index, it may be relevant to normalize the weights so that the index becomes dimensionless and independent of the order of magnitude of the weights. For the weighted Simpson index, the normalization can be performed, for example, by imposing the condition ${\sum }_{i=1}^k{w}_i=1.$ This procedure can be accomplished by dividing each non-normalized weight $w_i$ by the sum of all the weights. So, the weighted Simpson index with normalized weights corresponding to the weighted Simpson index $S_{w }$is denoted by $S_u=S_{\mathsf{\beta }w}$ with $\mathsf{\beta }={\left({\displaystyle {\sum }_{i=1}^k}{w}_i\right)}^{-1}$ and ${u=\left(u_1,\dots ,u_k\right)}^{\mathrm{t}}$ with $u_i=\mathsf{\beta }{w}_i$.

Proposition 2.

Let  $S_u$ be the weighted Simpson index computed with normalized weights, corresponding to the weighted Simpson index $S_w$. Then:

(a) 

The maximum value of  $S_u$ is given by $\underset{i}{\mathit{max}}{u}_i={\left({\displaystyle {\sum }_{i=1}^k}{w}_i\right)}^{-1}\underset{i}{\mathit{max}}{w}_i$;

(b) 

The minimum value of $S_u$ is given by $S_u^{*}=\frac{1}{{\sum }_{i=1}^k\frac{1}{u_i}}=\frac{1}{{\displaystyle {\sum }_{i=1}^k}{w}_i{\sum }_{i=1}^k\frac{1}{w_i}}$  and corresponds to the distribution $p_j^{*}=\frac{1}{w_j\left({\sum }_{i=1}^k \frac{1}{w_i}\right)}$ for $j=1,\dots k.$

Proof. 

Note that $S_{\mathsf{\beta }w}=\mathsf{\beta }{S}_w$ for any real number $\mathsf{\beta }.$ In fact, $S_{\mathsf{\beta }w}={\displaystyle {\sum }_{i=1}^k}{\mathsf{\beta }w}_i{p}_i^2=\mathsf{\beta }{\displaystyle {\sum }_{i=1}^k}{w}_i{p}_i^2=\mathsf{\beta }{S}_w.$ As $S_u=S_{\mathsf{\beta }w}=\mathsf{\beta }{S}_w$ with $\mathsf{\beta }={\left({\displaystyle {\sum }_{i=1}^k}{w}_i\right)}^{-1}>0$, this normalization procedure does not affect the maximum and minimum points of the index, and the maximum and minimum values of $S_u$ are obtained by multiplying by $\mathsf{\beta }$, respectively, the maximum and minimum values of $S_w$. □

2.4. The Inverse Problem

The inverse procedure relative to the weighted Gini–Simpson index was formulated recently [27]. Now, we consider the analogous problem concerning the weighted Simpson index, stated as the following: given a minimum point of $S_w$ denoted by $\left(p_1^{\mathrm{*}},\dots , p_k^{\mathrm{*}}\right)$, verifying both $0<p_j^{*}<1$ and ${\sum }_{j=1}^k{p}_j^{*}=1$, what would be a set of weights able to generate that solution? The answer to this question is straightforward, as follows:

Proposition 3.

Suppose that  $p^{*}=\left(p_1^{*},\dots , p_k^{*}\right)$, verifying both $0<p_j^{*}<1$  and  ${\sum }_{j=1}^k{p}_j^{*}=1$, is a minimum point of the weighted Simpson index  $S_u$ with normalized weights. Then

$$u_i=\frac{1}{p_i^{*}\left({\sum }_{j=1}^k\frac{1}{p_j^{*}}\right)} \mathrm{for} i=1,\dots ,k,$$

(5)

and the harmonic mean of the weights equals the harmonic mean of the coordinates of the minimum point,  $H\left(u\right)=H\left(p^{*}\right).$

Proof. 

Recalling (3), the weights must be chosen to be inversely proportional to the optimal coordinates $p_j^{*}$ with the proportionality constant equal to $S_u^{*}$ (2), as we can see rewriting:

$$p_j^{\mathrm{*}}=\frac{1}{u_j\left({\sum }_{i=1}^k\frac{1}{u_i}\right)}⟺\frac{1}{p_j^{\mathrm{*}}}=u_j\left({\sum }_{i=1}^k\frac{1}{u_i}\right)⟺\frac{1}{p_j^{\mathrm{*}}}=u_j\frac{1}{S_u^{*}}⟺u_j=\frac{S_u^{*}}{p_j^{\mathrm{*}}}.$$

For normalized weights, using the condition ${\sum }_{j=1}^k{u}_j=1$, one gets ${\sum }_{j=1}^k\frac{S_u^{*}}{p_j^{\mathrm{*}}}=1$ $⟺$ $S_u^{*}=1/\left({\sum }_{j=1}^k\frac{1}{p_j^{\mathrm{*}}}\right)$, and so the weights must be chosen as:

$$u_i=\frac{1}{p_i^{*}\left({\sum }_{j=1}^k\frac{1}{p_j^{\mathrm{*}}}\right)} \mathrm{for} i=1,\dots ,k.$$

Note that from the condition that the sum of the weights equals 1, it follows that ${\sum }_{i=1}^k\frac{1}{u_i}={\sum }_{i=1}^k\frac{1}{p_i^{*}}$ and thus, we can proceed with the equality$\frac{k}{{\sum }_{i=1}^k\frac{1}{u_i}}=\frac{k}{{\sum }_{i=1}^k\frac{1}{p_i^{*}}}$ obtaining the result concerning the corresponding harmonic means H$\left(u\right)=\mathrm{H}\left(p^{*}\right)$. □

For non-normalized weights, there are infinitely many solutions to the inverse problem, parameterized by the minimum $S_w^{*}$. In other words, in that case, for the inverse problem to have a unique solution, it is not sufficient to know the minimum point, and the minimum value $S_w^{*}$ must also be given.

3. Final Remarks

We have presented a detailed analytical study of the optimization problem associated with the weighted Simpson index. The core result is that at the optimal point one has $w_i{p}_i^{*}=w_j{p}_j^{*}$ for $i,j=1,\dots ,k$, and also, that for all i, one gets $w_i{p}_i^{*}=S_w^{*}$, the minimum value of the index. So, there is a trade-off between the weights and the optimal probabilities (or proportions of occurrences) in what could be seen as an equilibrium condition. The fact that Nowak [8,9] has used the weighted Simpson index as a Lyapunov function to assess an antigenic diversity threshold seems compatible with an equilibrium point perspective, which could also be used in a broad sense concerning different problems within several scientific fields. The mathematical results are obtained in simple closed-form solutions and so do not seem prone to unexpected computational difficulties.

Furthermore, for a random variable $W$ with values corresponding to the previous weights ${\left\{w_i\right\}}_{i=1,\dots ,k}$, and the probability function given by $\mathrm{Pr}\left[W=w_i\right]=p_i^{*}$, with $p_i^{*}$ defined as in (3), computing the mean value of $W$ entails $E\left[W\right]={\sum }_{i=1}^k{w}_i{p}_i^{*}=k{S}_w^{*}$, which equals the harmonic mean of the weights, meaning $E\left[W\right]=\mathrm{H}\left(w\right)$.