Clustering Empirical Bootstrap Distribution Functions Parametrized by Galton–Watson Branching Processes

Abstract

The nonparametric bootstrap has been used in cluster analysis for various purposes. One of those purposes is to account for sampling variability. This can be achieved by obtaining a bootstrap approximation of the sampling distribution function of the estimator of interest and then clustering those distribution functions. Although the consistency of the nonparametric bootstrap in estimating transformations of the sample mean has been known for decades, little is known about how it carries over to clustering. Here, we investigated this problem with a simulation study. We considered single-linkage agglomerative hierarchical clustering and a three-type branching process for parametrized transformations of random vectors of relative frequencies of possible types of the index case of each process. In total, there were nine factors and 216 simulation scenarios in a fully-factorial design. The ability of the bootstrap-based clustering to recover the ground truth clusterings was quantified by the adjusted transfer distance between partitions. The results showed that in the best 18 scenarios, the average value of the distance was less than 20 percent of the maximum possible distance value. We noticed that the results most notably depended on the number of retained clusters, the distribution for sampling the prevalence of types, and the sample size appearing in the denominators of relative frequency types. The comparison of the bootstrap-based clustering results with so-called uninformed random partitioning results showed that in the vast majority of scenarios considered, the bootstrap-based approach led, on average, to remarkably lower classification errors than the random partitioning.

1. Introduction

Clustering is a technique for finding groups within data where the elements belonging to the same group (or cluster) are similar according to a specific clustering variable. Although cluster analysis has roots in anthropology, it has quickly spread to various fields, such as biology, medicine, remote sensing, pattern recognition, economics, market research, and social network modeling.

Cluster analysis is usually an unsupervised learning technique, meaning there are no labels or gold standards to rely on. Consequently, it is a very challenging procedure because no straightforward measures exist to evaluate its performance. To grasp the quality of clustering is of the greatest importance [1] and has drawn the attention of researchers in recent years.

An issue in cluster analysis is the stability of the clustering results [1]. Two critical aspects of clustering are the data that are used as input and the clustering method. When the data are a sample from a population (usually the case), typical clustering algorithms like k-means and hierarchical clustering do not account for the sampling variability and variability in the data due to measurement errors. Consequently, the clustering result (a partition) is merely a point estimate.

One way to overcome this limitation is to construct data replications by sampling with replacement from the original dataset. Another way to address this limitation is to make cluster validation via data splitting or subsampling. Ref. [1] provides a detailed explanation of these techniques. In this paper, we focused on the first approach, that is, sampling variability was addressed by incorporating the nonparametric bootstrap [2] into the clustering procedure. For k-means clustering, ref. [3] showed through a simulation study that the nonparametric bootstrap can help obtain confidence information about the cluster centers and recover cluster memberships of data objects.

Another point worth considering is the study of unsupervised learning techniques within the framework of stochastic processes, particularly time series analysis. These methodologies have attracted the attention of researchers due to their ability to describe complex real-life situations, such as those in biological and medical studies [4,5]. This framework allows for using domain-specific models, such as stochastic process models, in the clustering process. The models may help to set the focus for the clustering process. For example, the interest may be to perform clustering based on forecasting properties of a time series [6], or study the statistical properties of the clustering algorithms in the context of stationary ergodic processes, either in distribution or in covariance (see [4] for a review on this topic). Ref. [7] proposed a novel clustering approach that formulates the problem of determining the most homogenous clusters by exploring the stochastic nature of the main clustering variable by means of estimating its best-fit continuous probability distribution.

Multi-type Galton–Watson branching processes have been applied to model various phenomena like the growth of bacterial population, epidemic propagation, and cell kinetics [8,9,10,11]. However, the latter processes have hardly been used in clustering. Nowadays, where data come from different sources within one domain, stochastic process models may be useful for meaningfully combining these data. For example, in epidemiology, common quantities tracked are the reproduction number and prevalence. They both could be incorporated into the branching process model.

Here, we developed a method that unifies the bootstrap and stochastic process model approaches to clustering in a certain way. To the authors’ knowledge, this is a pioneering work in which we carried out a single linkage agglomerative hierarchical clustering in the framework of the three-type Galton–Watson branching processes; the bootstrapping techniques were used to quantify the uncertainty of the estimates obtained from the clustering analysis. The grounding for our work is that the bootstrap is consistent for estimating certain functions of the sample mean (Theorem 23.5, p. 331, [12]), and our work will shed light on how this carries over to clustering in the proposed setting. The method consists of three parts, which are explained in the following paragraph.

First, we assumed we have n three-type Galton–Watson processes. For each process, we defined the parameter of interest as the expected total number of realized individuals in a finite number of generations. Considering that the index case is a categorical random variable, possibly with different distributions across the n models, we defined an estimator for each parameter. The second part involved the bootstrap estimation of the parameter of interest for each process, leading to n Monte Carlo approximated bootstrap distribution functions of the model parameter estimators. The third part involved hierarchical agglomerative variable-group clustering of those distribution functions [13].

Variable-group hierarchical clustering is an extension of the traditional pair-group hierarchical clustering, which allows the handling of tied clusters algorithmically. As for the other two hyperparameters involved in clustering, we have used the supremum metric (as in pp. 150–151 [14], but with the domain being the whole $\mathbb{R}$) to calculate the distance between distribution functions, and we considered the single-linkage method to determine the closest clusters in each step of hierarchical clustering. The single linkage method has been criticized for its chaining tendencies [15], and it can be argued that the supremum distance between distribution functions, being bounded from above by 1, is not well-suited for measuring widely separated distribution functions. However, the reason behind choosing those two hyperparameters is that, in our opinion, it is a promising way to see the implications of the bootstrap consistency for the clustering results. Notably, the consistency of the bootstrap is stated in terms of the supremum metric, and the order-based definition of the single-linkage dissimilarity between clusters appears to fit well with the order-based concept of supremum. Nevertheless, we did not undertake analytical work to study the method.

We performed a simulation study to analyze the performance of the proposed method. We aimed to evaluate how well the partitions obtained by the nonparametric bootstrap distribution functions agree with the comparison partitions obtained based on the transformed multinomial distribution functions. Some of the parameters of the ground-truth sampling distribution functions were sampled from pre-specified distributions (type-specific prevalence), and others were chosen in a fixed manner (e.g., type-specific reproduction numbers for Galton–Watson processes). The performance was quantified with the adjusted transfer distance between set partitions, also called the classification error distance [16]. This distance function has reasonable properties [16] and a known upper limit [17] that facilitates the interpretation of its values, and it can be defined intuitively in terms of admissible transformations that turn one partition into another [18].

The outline of this article is as follows. In Section 2.1.1, we define the model parameter based on a three-type Galton–Watson branching process model with the purpose of using it in the simulation study. In Section 2.1.2, we describe the branching process and the bootstrap-based clustering method that we developed. In Section 2.2, we formulate the questions that we seek to answer in the simulation study. The general overview of the simulation study is in Section 2.3, Section 2.4 and Section 2.5, while the results of the simulation study are given in Section 3. The interpretation of the results, as well as our work’s limitations and possible extensions, are located in Section 4. The entire computer implementation of the study was undertaken in R software, version 4.1.2. The computer code is available in the Supplementary Materials.

2. Simulation Study

The simulation study is structured according to the MADEP framework (M—methods, A—aims, D—data-generating mechanisms, E—estimands, P—performance measures), which differs from the ADEMP structure introduced by [19] only in that we have moved the methods part to the beginning.

The material of this section follows the first author’s previous work [20].

2.1. Methods

In Section 2.1.1, we outline the derivation of the branching process model parameter.

2.1.1. The Model

Let $\mathcal{T}$ be the set $\{1,2,3\}$ that we call the set of types and its subset ${\mathcal{T}}_2=\{1,2\}$. Let ${\mathbb{N}}_0=\mathbb{N}\cup \left\{0\right\}$, where $\mathbb{N}$ is the set of all positive integers.

Let ${\delta }_{ij}=1$ if $i=j$, and ${\delta }_{ij}=0$ if $i\ne j$. Suppose that ${\mathbf{Z}}_0\left({\tau }_0\right)= (Z_0^{\left(1\right)}\left({\tau }_0\right)$, $Z_0^{\left(2\right)}\left({\tau }_0\right)$, $Z_0^{\left(3\right)}\left({\tau }_0\right))$ is a random vector with a constant value ${\mathbf{e}}_{{\tau }_0}=({\delta }_{1{\tau }_0},{\delta }_{2{\tau }_0},{\delta }_{3{\tau }_0})$.

Following [8], we consider a three-type Galton–Watson branching process with the index case of type ${\tau }_0$ from $\mathcal{T}$ as

$$\begin{array}{cc}\hfill & {\mathbf{Z}}_0\left({\tau }_0\right)=(Z_0^{\left(1\right)}\left({\tau }_0\right),Z_0^{\left(2\right)}\left({\tau }_0\right),Z_0^{\left(3\right)}\left({\tau }_0\right))={\mathbf{e}}_{{\tau }_0},\hfill \\ \hfill & {\mathbf{Z}}_{i+1}\left({\tau }_0\right)=(Z_{i+1}^{\left(1\right)}\left({\tau }_0\right),Z_{i+1}^{\left(2\right)}\left({\tau }_0\right),Z_{i+1}^{\left(3\right)}\left({\tau }_0\right))=\stackrel{3}{\sum _{\tau =1}}\stackrel{Z_i^{\left(\tau \right)}\left({\tau }_0\right)}{\sum _{j=1}}{\mathbf{X}}_{ij}^{\left(\tau \right)},i\in {\mathbb{N}}_0,\hfill \end{array}$$

where the sum from $j=1$ to $j=0$ is defined to be the zero vector; ${\left({\mathbf{X}}_{ij}^{\left(\tau \right)}\right)}_{(i,\tau ,j)\in {\mathbb{N}}_0\times \mathcal{T}\times \mathbb{N}}$ is a collection of independent random vectors each with a probability mass function (also called offspring law, or reproduction law)

$$\begin{array}{c}\hfill q_{\tau }(k_1,k_2,k_3)=P\left(\{{\mathbf{X}}_{ij}^{\left(\tau \right)}=(k_1,k_2,k_3)\}\right),(k_1,k_2,k_3)\in D_{\tau }\subset {\mathbb{N}}_0^3\end{array}$$

where $D_{\tau }$ is a set of all triples from ${\mathbb{N}}_0^3$ in which $q_{\tau }(k_1,k_2,k_3)$ takes positive values. Let $\mathbf{m}$ be the following third-order square matrix (the so-called mean matrix of the process) over the set of all non-negative real numbers ${\mathbb{R}}_{+}$,

$$\mathbf{m}=\left(\begin{array}{ccc}{m}_{11}& m_{12}& m_{13}\\ m_{21}& m_{22}& m_{23}\\ 0& 0& m_{33}\end{array}\right),$$

(1)

so that for all ${\tau }_0$ and $\tau$ from $\mathcal{T}$, $m_{{\tau }_0\tau }$ is the expectation of the random variable $Z_1^{\left(\tau \right)}\left({\tau }_0\right)$, denoted by $E\left(Z_1^{\left(\tau \right)}\left({\tau }_0\right)\right)$.

Before proceeding with the definition of the model parameter of interest to us, we highlight some important assumptions in the Galton–Watson branching process model.

• The model is a generation counting process, i.e., it represents the counts of individuals of different types in each generation. One time unit corresponds to one generation.
• Once the time increases by one unit, all the ancestors ‘die’, and each of them is replaced by a random number of descendants (of various types).
• All individuals reproduce independently, i.e., the number of descendants (of various types) of one individual has no bearing on the number of descendants of another individual.
• All individuals of the same type have the same reproduction law, which does not change in time. The reproduction law can only depend on the type of an individual.
• The specific form of the mean matrix $\mathbf{m}$ assumes that the individuals of type 1 and type 2 can produce individuals of all three types. The individuals of type 3 cannot generate individuals of type 1 and 2, and they can only produce individuals of their own type.

For any fixed non-negative integer r, with the next equality, we define a random variable $W_r\left({\tau }_0\right)$ representing the total population size summed over time points from 0 to r and over the first two types, all generated by a single index case (i.e., the individual at time 0) of type ${\tau }_0$:

$$\begin{array}{cc}\hfill W_r\left({\tau }_0\right)& =\left(\stackrel{r}{\sum _{i=0}}{\mathbf{Z}}_i\left({\tau }_0\right)\right){(1,1,0)}^{\top },\hfill \end{array}$$

where the symbol ⊤ means transposition.

For every ${\tau }_0\in \mathcal{T}$, the probability mass function of $W_r\left({\tau }_0\right)$ is denoted by $f_{r,{\tau }_0}(·)$, and we assume that the expected value $E\left(W_r\left({\tau }_0\right)\right)$ is finite. The function $f_{r,{\tau }_0}(·)$ has the support contained in ${\mathbb{N}}_0$ for every ${\tau }_0$.

We consider the prevalence $p_{{\tau }_0}$ of individuals of type ${\tau }_0$ to be positive for every type ${\tau }_0$ from $\mathcal{T}$, and assume that the numbers $p_1$, $p_2$, $p_3$ sum to 1. For fixed r, we consider the mixture probability mass function:

$$\begin{array}{c}\hfill h_r\left(j\right)=\sum _{{\tau }_0\in \mathcal{T}}{p}_{{\tau }_0}{f}_{r,{\tau }_0}\left(j\right),j\in {\mathbb{N}}_0.\end{array}$$

(2)

Suppose that $W_r$ is a random variable with the probability function given by Equation (2). We derive the mean value $\mu$ of the random variable $W_r$ as follows:

$$\begin{array}{cc}\hfill \mu & =\sum _{j=0}^{\infty }j{h}_r\left(j\right)=\sum _{j=0}^{\infty }j\left(\sum _{{\tau }_0\in \mathcal{T}}{p}_{{\tau }_0}{f}_{r,{\tau }_0}\left(j\right)\right)=\hfill \\ \hfill & =\sum _{j=0}^{\infty }\left(\sum _{{\tau }_0\in \mathcal{T}}{p}_{{\tau }_0}j{f}_{r,{\tau }_0}\left(j\right)\right)=\hfill \\ \hfill & =\sum _{{\tau }_0\in \mathcal{T}}\left(p_{{\tau }_0}\sum _{j=0}^{\infty }j{f}_{r,{\tau }_0}\left(j\right)\right)=\hfill \\ \hfill & =\sum _{{\tau }_0\in \mathcal{T}}\left(p_{{\tau }_0}E\left(W_r\left({\tau }_0\right)\right)\right).\hfill \end{array}$$

(3)

To find a formula for $\mu$, we express $E(W_r\left({\tau }_0\right)$ in terms of elements of the matrix $\mathbf{m}$. For that, we use the relationship

$$\begin{array}{c}\hfill E\left({\mathbf{Z}}_i\left({\tau }_0\right)\right)={\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i\end{array}$$

(4)

where $i\in {\mathbb{N}}_0$, and ${\mathbf{m}}^i$ is the i-th power of the matrix $\mathbf{m}$  [8]. By definition of expectation of a random vector, linearity of expectation, and formula (4),

$$\begin{array}{cc}\hfill E\left(W_r\left({\tau }_0\right)\right)& =E\left[\left(\underset{i=0}{\sum ^r}{Z}_i^{\left(1\right)}\left({\tau }_0\right),\underset{n=0}{\sum ^r}{Z}_i^{\left(2\right)}\left({\tau }_0\right),\underset{n=0}{\sum ^r}{Z}_i^{\left(3\right)}\left({\tau }_0\right)\right){(1,1,0)}^{\top }\right]=\hfill \\ \hfill & =\left(\underset{i=0}{\sum ^r}E\left(Z_i^{\left(1\right)}\left({\tau }_0\right)\right),\underset{i=0}{\sum ^r}E\left(Z_i^{\left(2\right)}\left({\tau }_0\right)\right),\underset{i=0}{\sum ^r}E\left(Z_i^{\left(3\right)}\left({\tau }_0\right)\right)\right){(1,1,0)}^{\top }=\hfill \\ \hfill & =\left(\underset{i=0}{\sum ^r}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_1^{\top },\underset{i=0}{\sum ^r}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_2^{\top },\underset{i=0}{\sum ^r}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_3^{\top }\right){(1,1,0)}^{\top }=\hfill \\ \hfill & =\underset{i=0}{\sum ^r}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_1^{\top }+\underset{i=0}{\sum ^r}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_2^{\top }.\hfill \end{array}$$

(5)

To further express (5), we compute ${\mathbf{m}}^i$. For that, we use the fact that the matrix $\mathbf{m}$ is a block-diagonal matrix of the form

$$\left(\begin{array}{cc}{\mathbf{m}}_{2\times 2}& \mathbf{u}\\ {\mathbf{0}}_{1\times 2}& \left(m_{33}\right)\end{array}\right),$$

where

$$\begin{array}{c}\hfill {\mathbf{m}}_{2\times 2}=\left(\begin{array}{cc}{m}_{11}& m_{12}\\ m_{21}& m_{22}\end{array}\right),{\mathbf{0}}_{1\times 2}=\left(\begin{array}{cc}0& 0\end{array}\right),\mathbf{u}=\left(\begin{array}{c}{m}_{13}\\ m_{23}\end{array}\right),\end{array}$$

so that ${\mathbf{m}}^i$, for some two-by-one matrix $\mathbf{v}$, is of the form

$$\left(\begin{array}{cc}{\mathbf{m}}_{2\times 2}^i& \mathbf{v}\\ {\mathbf{0}}_{1\times 2}& \left(m_{33}^i\right)\end{array}\right).$$

Therefore, considering ${\tau }_0=3$, we have ${\displaystyle \underset{i=0}{\sum ^r}}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_1^{\top }={\displaystyle \underset{i=0}{\sum ^r}}{\mathbf{e}}_{{\tau }_0}{\mathbf{m}}^i{\mathbf{e}}_2^{\top }=0$, so $E\left[W_r\left(3\right)\right]=0$, which allows to simplify formula (3) of $\mu$.

On the other hand, to find $E(W_r\left({\tau }_0\right)$ for ${\tau }_0\in {\mathcal{T}}_2$, it suffices to compute ${\mathbf{m}}_{2\times 2}^i$:

$$\begin{array}{c}\hfill E\left(W_r\left({\tau }_0\right)\right)=\sum _{i=0}^r{\mathbf{e}}_{{\tau }_0,2}{\mathbf{m}}_{2\times 2}^i{\mathbf{e}}_{1,2}^{\top }+\sum _{i=0}^r{\mathbf{e}}_{{\tau }_0,2}{\mathbf{m}}_{2\times 2}^i{\mathbf{e}}_{2,2}^{\top },\end{array}$$

where ${\mathbf{e}}_{{\tau }_0,2}$ is a two by one vector $({\delta }_{1{\tau }_0},{\delta }_{2{\tau }_0})$ consisting of the first two elements of ${\mathbf{e}}_{{\tau }_0}$ in the same order as they are in ${\mathbf{e}}_{{\tau }_0}$.

Assuming that the eigenvalues ${\lambda }_1,{\lambda }_2$ of the matrix ${\mathbf{m}}_{2\times 2}$ satisfy ${\lambda }_1\ne {\lambda }_2\ne 1$, then by (p. 7, Equation (2.7), [20]),

$$\begin{array}{c}\hfill \sum _{i=0}^r{\mathbf{e}}_{{\tau }_0,2}{\mathbf{m}}_{2\times 2}^i{\mathbf{e}}_{j,2}^{\top }={\displaystyle \frac{{\delta }_{{\tau }_{0}j}{\lambda }_2-m_{{\tau }_{0}j}}{{\lambda }_2-{\lambda }_1}}\sum _{i=0}^r{\lambda }_1^i+{\displaystyle \frac{m_{{\tau }_{0}j}-{\lambda }_1{\delta }_{{\tau }_{0}j}}{{\lambda }_2-{\lambda }_1}}\sum _{i=0}^r{\lambda }_2^i,j=1,2,\end{array}$$

so that the expected values $E\left(W_r\left({\tau }_0\right)\right),{\tau }_0=1,2,$ in (3) are

$$\begin{array}{c}\hfill E\left[W_r\left({\tau }_0\right)\right]={\displaystyle \frac{{\lambda }_2-(m_{{\tau }_{0}1}+m_{{\tau }_{0}2})}{{\lambda }_2-{\lambda }_1}}\sum _{i=0}^r{\lambda }_1^i+{\displaystyle \frac{(m_{{\tau }_{0}1}+m_{{\tau }_{0}2})-{\lambda }_1}{{\lambda }_2-{\lambda }_1}}{\displaystyle \frac{1-{\lambda }_2^{r+1}}{1-{\lambda }_2}},\end{array}$$

(6)

where

$$\begin{array}{c}\hfill \sum _{i=0}^r{\lambda }_1^i=\left\{\begin{array}{cc}{\displaystyle \frac{1-{\lambda }_1^{r+1}}{1-{\lambda }_1}},\hfill & \mathrm{if}{\lambda }_1\ne 1\hfill \\ r+1,\hfill & \mathrm{if}{\lambda }_1=1\hfill \end{array}\right.,\end{array}$$

$$\begin{array}{c}\hfill {\lambda }_{1;2}={\displaystyle \frac{Tr\left({\mathbf{m}}_{2\times 2}\right)}{2}}\pm \sqrt{{\displaystyle \frac{{(Tr\left({\mathbf{m}}_{2\times 2}\right))}^2}{4}}-det\left({\mathbf{m}}_{2\times 2}\right)},\end{array}$$

(7)

with the trace and the determinant, respectively, being

$$\begin{array}{cc}\hfill & Tr\left({\mathbf{m}}_{2\times 2}\right)=m_{11}+m_{22},\hfill \\ \hfill & det\left({\mathbf{m}}_{2\times 2}\right)=m_{11}{m}_{22}-m_{12}{m}_{21}.\hfill \end{array}$$

We refer to the value

$$\begin{array}{c}\hfill \mu =E\left(W_r\left(1\right)\right)p_1+E\left(W_r\left(2\right)\right)p_2\end{array}$$

(8)

as the model parameter. We will denote it shortly by $c_1{p}_1+c_2{p}_2$, where $c_i=E\left(W_r\left(i\right)\right)$.

2.1.2. Proposed Method

A step-by-step description of the proposed method is as follows.

• Specify n populations of individuals (bacteria or cells in different test tubes, people in various countries, etc.) for which the Galton–Watson branching process model could be applied. In each population, we assume that there are individuals of three types labeled by $1,2,3$ (e.g., there can be mutant cells of type A, mutant cells of type B, and non-mutant cells). The prevalence of individuals of type j in population i we denote by $p_{ij}$. The goal is to find groups of populations such that populations within each group are similar to each other in some sense but more dissimilar to populations in other groups.
• Take a random sample of size N from each of the n populations and record the type of each individual sampled. The result of this step is n samples of types ${\mathbf{t}}_i,i=1,\dots ,n$, where each of the N elements in the vector ${\mathbf{t}}_i$ is either $1,2$ or 3.
• For each population, there is a corresponding three-type Galton–Watson branching process. Specify the number of generations r and the mean matrix $\mathbf{m}$ of interest in the branching process model; we assume that these two choices are the same regardless of the population.
• Compute the coefficients $E\left(W_r\left(1\right)\right),E\left(W_r\left(2\right)\right)$ in the model parameter formula using Equation (5) or Equation (6) (the latter can be used provided the eigenvalues ${\lambda }_1,{\lambda }_2$ of the top-left 2 by 2 submatrix of $\mathbf{m}$ satisfy ${\lambda }_1\ne {\lambda }_2\ne 1$).
• The general idea of this step is, for each of n populations, to apply the nonparametric bootstrap to estimate the distribution function of the model parameter estimator

  $${\widehat{\mu }}_i=c_1{\displaystyle \frac{{\sum }_{j=1}^N{I}_{\left\{1\right\}}\left(T_{ij}\right)}{N}}+c_2{\displaystyle \frac{{\sum }_{j=1}^N{I}_{\left\{2\right\}}\left(T_{ij}\right)}{N}},$$

  where random variable $T_{ij}$ has the categorical distribution $P\left(\{T_{ij}={\tau }_0\}\right)=p_{i{\tau }_0}$, and ${\tau }_0\in \mathcal{T}$, $p_{i1}+p_{i2}+p_{i3}=1$, $I_A$ is the indicator function of the set A. We assume that, for each $i=1,\dots ,n$, the random variables $T_{i1},\dots ,T_{iN}$ are independent. From a sampling theory point of view, this is guaranteed, for example, if the selection method is simple random sampling with replacement (pp. 21–22, [21]). Specifically, let $t_{ij}$ be the j-th entry of the observed sample of types ${\mathbf{t}}_i$. The bootstrap begins by defining ${\widehat{F}}_i$, the empirical categorical distribution based on ${\mathbf{t}}_i$:

  $${\widehat{F}}_i\left(x\right)={\displaystyle \frac{1}{N}}\sum _{j=1}^N{I}_{\{y\in \mathbb{R}:y\le x\}}\left(t_{ij}\right).$$

  (9)
  Choose the number R of bootstrap replications to be used uniformly in all n cases. Then, the Monte Carlo approximation to the nonparametric bootstrap consists of the following steps. Take R bootstrap replications ${\mathbf{t}}_{il}^{*}$ of size N, $l=1,\dots ,R$, from each ${\mathbf{t}}_i$, $i=1,\dots ,n$. Let $t_{ilj}^{*}$ denote the j-th entry in the sample of types ${\mathbf{t}}_{il}^{*}$ of the i-th population and l-th bootstrap replicate. Calculate the model parameter estimate ${\mu }_{il}^{*}$ for each replication ${\mathbf{t}}_{il}^{*}$, $l=1,\dots ,R$:

  $${\mu }_{il}^{*}=c_1{\displaystyle \frac{{\sum }_{j=1}^N{I}_{\left\{1\right\}}\left(t_{ilj}^{*}\right)}{N}}+c_2{\displaystyle \frac{{\sum }_{j=1}^N{I}_{\left\{2\right\}}\left(t_{ilj}^{*}\right)}{N}}.$$
  This leads to approximate bootstrap distribution functions $G_1,\dots ,G_n$ of the model parameter estimator:

  $$G_i\left(x\right)={\displaystyle \frac{1}{R}}\sum _{l=1}^R{I}_{\{y\in \mathbb{R}:y\le x\}}\left({\mu }_{il}^{*}\right).$$

  (10)
• This step comprises the computation of the distance matrix needed for the subsequent clustering process. We suggest computing the distance between population i and j, that is, the entry $(i,j)$ of the distance matrix $D_G$, as the supremum distance between the distribution functions $G_i$ and $G_j$ found in the previous step:

  $$d_S(G_i,G_j)=\underset{x\in \mathbb{R}}{sup}|G_i\left(x\right)-G_j\left(x\right)|.$$
• Apply the hierarchical variable-group agglomerative single linkage clustering using the R software package ‘mdendro’ function ‘linkage’ and using the distance matrix $D_G$ as an input.
• For simplicity, let us assume that the set labels for the populations is $U=\{1,2,\dots ,n\}$. Fix the number k of clusters to retain ($1\le k\le n$). To obtain the partition of U into k clusters (if it exists), we use the so-called ‘cutting the multidendrogram’ method. A more detailed explanation of the method is as follows.
  Recall that a partition ${\mathcal{A}}^{\prime }$ is called to be finer than a partition ${\mathcal{A}}^{\prime \prime }$ (equivalently, ${\mathcal{A}}^{\prime \prime }$ is coarser than ${\mathcal{A}}^{\prime }$), denoted by ${\mathcal{A}}^{\prime }\preccurlyeq {\mathcal{A}}^{\prime \prime }$, if for each cluster A in ${\mathcal{A}}^{\prime }$ there exists a cluster B in ${\mathcal{A}}^{\prime \prime }$ such that A is a subset of B. The set of all partitions of U, denoted by $\mathrm{\Pi }\left(U\right)$, together with the relation ≼ is a lattice. The algorithm starts with the trivial partition ${\mathcal{A}}_1=\{\left\{x\right\}:x\in U\}$. Each time step 2) in the variable-group hierarchical agglomerative clustering algorithm is completed (see [13], pp. 49–50), a new partition of U is obtained by replacing all the clusters that were merged into a strictly larger cluster in that step of the algorithm with the corresponding newly formed superclusters. For example, if initially we had ${\mathcal{A}}_1$, and then after completing part 2) of the algorithm, clusters $\left\{1\right\},\left\{2\right\}$ merged and clusters $\left\{3\right\},\left\{4\right\}$ merged, then ${\mathcal{A}}_2$ will be $\left\{\right\{1,2\},\{3,4\},\{5\},\{6\},\dots ,\{n\left\}\right\}$, i.e., $\left\{1\right\}$ and $\left\{2\right\}$ in ${\mathcal{A}}_1$ were replaced by $\{1,2\}$, similarly $\left\{3\right\}$ and $\left\{4\right\}$ in ${\mathcal{A}}_1$ were replaced by $\{3,4\}$.
  Each time step 2) of the algorithm finishes, we obtain a coarser partition than the previous partition because the new partition is obtained by merging at least two of the clusters of the previous partition. Once the trivial partition $\left\{U\right\}$ is formed, the algorithm stops. Therefore, we can associate a chain of partitions ${\mathcal{A}}_1\preccurlyeq {\mathcal{A}}_2\preccurlyeq \dots \preccurlyeq {\mathcal{A}}_{n^{\prime }}=\left\{U\right\}$ with each completed variable-group algorithm run, and the set of partitions L forming this chain is a sublattice of $\left(\mathrm{\Pi }\right(U),\preccurlyeq )$.
  Let $(T,h_l,h_u)$ be the multivalued tree (as defined by pp. 46, 48, [13]) on U defined by the clustering output obtained in step 7. Then by definition of the clustering algorithm, every partition from L is a subset of T (the latter contains all the clusters involved in the clustering process, but every partition from L contains only some of them). Therefore, the height function $h_l$ is well-defined on all elements of every partition from L. Let $h_{max}$ be an arbitrary fixed number larger than $h_l\left(U\right)$. For each number a in the interval $(0,h_{max})$, denote

  $${\mathcal{H}}_a=\{\mathcal{A}\in L:h_l\left(X\right)<a\mathrm{for}\mathrm{each}X\in \mathcal{A}\}.$$
  Notice that since $h_l\left(X\right)<h_l\left(Y\right)$ whenever $X⊊Y$, then for all $i=1,\dots ,n^{\prime }-1$, ${max}_{A\in {\mathcal{A}}_i}{h}_l\left({\mathcal{A}}_i\right)<{max}_{A\in {\mathcal{A}}_{i+1}}{h}_l\left({\mathcal{A}}_{i+1}\right)$, so for every $a\in (0,h_{max})$, the set ${\mathcal{H}}_a$ with the relation ≼ is a subchain of L consisting of partitions from ${\mathcal{A}}_1$ up to ${\mathcal{A}}_j$ for some $j\in \{1,\dots ,n^{\prime }\}$.
  Hence, there exists the coarsest partition in the set ${\mathcal{H}}_a$ with respect to the relation ≼ for a given value of a, denote it by ${\mathcal{A}}_{\max }^a$. Finally, if there exists $a\in (0,h_{max})$ such that ${\mathcal{A}}_{\max }^a$ contains k clusters (recall that k was fixed at the beginning of step 8), then ${\mathcal{A}}_{\max }^a$ is the output partition. Otherwise, the output partition is not assigned (Figure 1).

Our proposed method is formally presented by Algorithm 1 (based on Sections 1.2, 2, 3.4 [20]), with the main procedure ProposedMet having a subprocedure MCbootstrapDist.

Algorithm 1 Clustering empirical bootstrap distribution functions parametrized by Galton–Watson branching processes

1: Data: $n,R,N,r\in \mathbb{N}$; n vectors ${\mathbf{t}}_i$ from the Cartesian product ${\{1,2,3\}}^N$; $\mathbf{m}\in {\mathbb{R}}_{+}^{3\times 3}$ in the form (1), with the eigenvalues ${\lambda }_1,{\lambda }_2$ of ${\mathbf{m}}_{2\times 2}$ satisfying ${\lambda }_1\ne {\lambda }_2\ne 1$   2: Result: a set $\mathrm{\Pi }=\{{\mathcal{A}}_1,\dots ,{\mathcal{A}}_{n^{\prime }}\}$ of partitions of $U=\{1,...,n\}$, where $n^{\prime }\le n,{\mathcal{A}}_1$ is the trivial partition $\{\left\{x\right\}:x\in U\},{\mathcal{A}}_{n^{\prime }}=\left\{U\right\},{\mathcal{A}}_1\preccurlyeq \dots \preccurlyeq {\mathcal{A}}_{n^{\prime }}$, and the relation ≼ is defined in Section 2.1.2, item 8   3:     4: procedure ProposedMet($\{{\mathbf{t}}_1,\dots ,{\mathbf{t}}_n\},\mathbf{m},r,R$)                                    ▹ Proposed method   5:     $c_{{\tau }_0}=E\left(W_r\left({\tau }_0\right)\right)$ is calculated by (6) for given $\mathbf{m}$ and r for both ${\tau }_0=1$   6:     and ${\tau }_0=2$   7:     for all $i\in \{1,\dots ,n\}$ do   8:         $G_i\leftarrow$MCbootstrapDist(${\mathbf{t}}_i,R,c_1,c_2,N$)   9:     end for 10:     $U\leftarrow \{1,\dots ,n\}$ 11:     $D_G\leftarrow$n-dimensional square matrix with entry $(i,j)$ defined as 12:     $d_S(G_i,G_j)={sup}_{x\in \mathbb{R}}|G_i\left(x\right)-G_j\left(x\right)|$ 13:     $\mathrm{\Pi }\leftarrow$HierClust($U,D_G$)                                                                                     ▹ [13] 14:     return $\mathrm{\Pi }$ 15: end procedure 16:   17: procedure MCbootstrapDist(${\mathbf{t}}_i,R,c_1,c_2,N$)                                                          ▹ [2] 18:     ${\widehat{F}}_i\leftarrow$ empirical distribution function based on ${\mathbf{t}}_i$ given by (9) 19:     for $l=1$ to R do 20:         ${\mathbf{t}}_{il}^{*}\leftarrow$ take a sample of size N from ${\widehat{F}}_i$ 21:         $b_1^{*}\leftarrow$ the number of ones in ${\mathbf{t}}_{il}^{*}$ 22:         $b_2^{*}\leftarrow$ the number of twos in ${\mathbf{t}}_{il}^{*}$ 23:         ${\mu }_{il}^{*}\leftarrow c_1{\displaystyle \frac{b_1^{*}}{N}}+c_2{\displaystyle \frac{b_2^{*}}{N}}$ 24:     end for 25:     $G_i\leftarrow$ empirical distribution function based on ${\mu }_{i1}^{*},\dots ,{\mu }_{iR}^{*}$ given by (10) 26:     return $G_i$ 27: end procedure

2.2. Aims

Our hypothesis was that the method outlined in Section 2.1.2 performs well. We sought evidence for this claim through

(1)

…exploring the performance of the proposed method depending on different combinations of the input values;

(2)

…comparing the performance of the proposed method with a random partitioning method, which consists of choosing a partition of the base set U according to the uniform distribution on the set of all k-class partitions of U. We also sought to evaluate the effect size of the proposed method compared to the random partitioning method.

2.3. Data-Generating Mechanisms and Design

We conducted a simulation study with a fully factorial design and with nine factors (inputs). The inputs and their possible values that we used are given in

Table 1. Inputs for the simulation study and their possible values.

Input	Meaning	Notation	Set of Values
1	number of retained clusters	k	$S_1=\{2,3,4\}$
2	sample size	N	$S_2=\{100,500\}$
3	distribution for sampling prevalence	$dist$	$S_3=\{1,2\}$
4	number of elements to cluster	n	$S_4=\{30,60\}$
5	labels for vectorized form of $\mathbf{m}$	m	$S_5=\{1,2,3\}$
6	number of generations	r	$S_6=\{1,5,15\}$
7	set of type-specific prevalence		$\left\{{\mathcal{P}}_3\right\}$
8	simulations per scenario	$n_{\mathrm{sim}}$	$\left\{144\right\}$
9	number of bootstrap samples	R	$\left\{1000\right\}$

.

In Table 1, m denotes a label for the matrix $\mathbf{m}$ given by (1): $m=1$ corresponds to the case where the eigenvalues of the left upper two by two submatrix ${\mathbf{m}}_{2\times 2}$ of $\mathbf{m}$ satisfy ${\lambda }_2<{\lambda }_1<1$, $m=2$ refers to the case ${\lambda }_2<{\lambda }_1=1$, and $m=3$ to the case $1\ne {\lambda }_2<{\lambda }_1>1$. These three choices were due to the fact that we assumed ${\lambda }_1\ne {\lambda }_2$. We left the case ${\lambda }_1={\lambda }_2$ uncovered to keep the number of factors involved in the study smaller and to reduce the computer code execution time. The exact choices of ${\mathbf{m}}_{2\times 2}$ in the cases $m=1,2,3$ were respectively

$$\begin{array}{c}\hfill \left(\begin{array}{cc}0.6& 0.19\\ 0.27& 0.87\end{array}\right),\left(\begin{array}{cc}0.6& 0.2\\ 0.4& 0.8\end{array}\right),\left(\begin{array}{cc}0.3& 0.9\\ 0.4& 3\end{array}\right).\end{array}$$

It was sufficient to specify only ${\mathbf{m}}_{2\times 2}$ since, as we saw in the end part of Section 2.1.1, the value of the model parameter does not depend on the value of $m_{33}$.

Our pre-study check revealed that the time cost of the simulations with our computer implementation is sensitive to the value of the input n. We found that to complete the study in the planned timeframe, we should limit n to 60. The alternative choice $n=30$ served, on the one hand, to be sufficiently different from the first choice 60 but at the same time far enough from very small values of n (near 1), which we deemed not to have enough practical importance. The choice of possible values of N is based on analogous computational reasons.

In the choice of the number of retained clusters, k was constrained with the inequality $k\le n$. To keep the simulation study design simple and to facilitate performance evaluation, we decided to choose the same set of values of k across all values of n. Hence, we considered $k\le 30$ and made a subjective decision to consider the three smallest values for k just above the trivial value $k=1$.

Type-specific prevalence $\mathbf{p}=(p_1,p_2,p_3)$ was sampled from the fixed set ${\mathcal{P}}_3$ defined by

$$\begin{array}{cc}\hfill & \mathcal{P}=\{0,001+0,002u:u\in \{0,1,2,\dots ,24\left\}\right\},\hfill \\ \hfill & {\mathcal{P}}_3=\{(p_1,p_2,p_3):p_1,p_2\in \mathcal{P},p_1<p_2,p_3=1-p_1-p_2\}.\hfill \end{array}$$

(11)

The distribution for sampling was the uniform distribution

$$\begin{array}{c}\hfill P\left(\{X=i\}\right)={\displaystyle \frac{1}{300}},i\in \{1,\dots ,300\}\end{array}$$

(12)

if $dist=1$, and a truncated geometric distribution (13) if $dist=2$. The idea behind those choices was to have, in the first case, all prevalence triples from ${\mathcal{P}}_3$ be equally likely, and in the second case, the distribution to be concentrated on triples, where the sum of the first two components (i.e., the prevalence of the first two types of individuals) is relatively smaller. To practically use the truncated geometric distribution, we first mapped the set ${\mathcal{P}}_3$ bijectively to the set of the first 300 positive integers $\{1,\dots ,300\}$ in the following way: a triple $\mathbf{y}=(y_1,y_2,y_3)$ succeeds a triple $\mathbf{x}=(x_1,x_2,x_3)$ if and only if the sum of the first two components of $\mathbf{y}$ is larger than the sum of the first two components of $\mathbf{x}$, or if the sum of the first two components in $\mathbf{x}$ and $\mathbf{y}$ is equal, but the second component in $\mathbf{y}$ is larger than in $\mathbf{x}$. Suppose that X is a random variable that takes the value $i\in \{1,\dots ,300\}$ if and only if the i-th triple under the order just defined was sampled from the set ${\mathcal{P}}_3$. The second distribution ($dist=2$) for sampling prevalence was a truncated geometric distribution of the random variable X given by the following probability mass function:

$$\begin{array}{cc}\hfill & P\left(\{X=i\}\right)={\displaystyle \frac{q{(1-q)}^{i-1}}{1-{(1-q)}^{300}}},i\in \{1,\dots ,300\},\hfill \end{array}$$

(13)

where $q=1/10=0.1$. One can check that the random variable X under the truncated geometric distributional assumption with $q=1/10$ really has a negative value of skewness. A reason behind the choice of $q=1/10$ was to ensure that X would have a relatively low expected value. This concentrated the distribution to the domain, corresponding to relatively low prevalence values of type 1 and type 2 individuals. Denoting the expected value of X depending on q by $E\left(X\right)\left(q\right)$, we have ${lim}_{q\to 1}E\left(X\right)\left(q\right)=1$ and ${lim}_{q\to 0+}E\left(X\right)\left(q\right)={\displaystyle \frac{301}{2}}$, but for our choice the expected value $E\left(X\right)(1/10)$ is approximately 10, i.e., the expected value is near the tenth triple in ${\mathcal{P}}_3$.

Regarding the remaining three inputs $r,R$ and $n_{sim}$, we did not use any guidance for designing the possible values of the input r, so its values were chosen as an expert opinion. Fixing the number of bootstrap samples to $R=1000$ works well (p. 214, [22]). The explanation of how we chose the input $n_{sim}$ value is given at the end of Section 2.5.1 below (see inequalities (18)–(20)) since the reasoning depends on the performance measure that we have not yet defined.

We define the set of (simulation) scenarios as

$$\begin{array}{c}\hfill \mathcal{S}=S_1\times S_2\times S_3\times S_4\times S_5\times S_6,\end{array}$$

(14)

where each set $S_i$ is defined in Table 1. Therefore, each scenario from $\mathcal{S}$ is a tuple $(s_1,s_2,s_3,s_4,s_5,s_6)$ of values of respective inputs $1,2,\dots ,6$ from Table 1. Inputs 7, 8 and 9 had a constant value, so we did not use them for the purpose of defining the scenario.

The set $\mathcal{S}$ contains 216 elements (scenarios). For the convenient presentation of results, we ordered all scenarios in $\mathcal{S}$ (lexicographically): a scenario $(y_1,y_2,y_3,y_4,y_5,y_6)$ succeeds a scenario $(x_1,x_2,x_3,x_4,x_5,x_6)$ if and only if there exists an element $i_0$ in $\{1,2,3,4,5,6\}$ such that $y_{i_0}>x_{i_0}$ and $y_i=x_i$ for all i less than $i_0$ (we consider the usual order of natural numbers in each of the sets $S_i$).

2.4. Estimands

Our estimand was a partition $\mathcal{A}=\{A_1,\dots ,A_k\}$ of n elements of the set $U=\{1,\dots ,n\}$ into k clusters. Therefore, all clusters in each partition $\mathcal{A}$ are non-empty, every two clusters in $\mathcal{A}$ are disjoint, and the union of all clusters in $\mathcal{A}$ is U.

2.5. Performance Measures

2.5.1. Average Adjusted Transfer Distance and Its Monte Carlo Error

We wanted to evaluate if our proposed clustering method based on bootstrap empirical distribution functions (10) leads to the same or similar partition of a finite set of individuals U as with clustering using the ground-truth distribution functions of random variables of the form

$$\begin{array}{c}\hfill c_1{\displaystyle \frac{B_1}{N}}+c_2{\displaystyle \frac{B_2}{N}},\end{array}$$

where $c_1,c_2$ have the same meaning as in the definition of the model parameter, and the random vector $(B_1,B_2,B_3)$ has a multinomial distribution with the number of trials N and cell probabilities $(p_1,p_2,p_3)$ as sampled from (11) under assumptions described in Section 2.3:

$$\begin{array}{c}\hfill P\left(\{B_1=k_1,B_2=k_2,B_3=k_3\}\right)={\displaystyle \frac{N!}{k_1!k_2!k_3!}}{p}_1^{k_1}{p}_2^{k_2}{p}_3^{k_3},\end{array}$$

where $k_1,k_2,k_3\ge 0,k_1+k_2+k_3=N$.

The comparison of partitions can performed by computing the distance between partitions.

To measure how well any two partitions of U agree, we used the adjusted transfer distance. The following definition of the adjusted transfer distance was adapted from [17] to fit our need to compare partitions with the same number of clusters. Adjusting refers to the fact that we have divided the usual transfer distance value by the number of elements in the base set.

Let U be a finite set with the number of elements equal to $n\ge 1$. Assume that

$$\begin{array}{c}\hfill \mathcal{A}=\{A_1,\dots ,A_k\},\mathcal{B}=\{B_1,\dots ,B_k\}\end{array}$$

are two partitions of U into k clusters, respectively, such that $\mathcal{A}$ is the underlying ‘true’ partition, and partition $\mathcal{B}$ is used to approximate partition $\mathcal{A}$.

For a natural number k, let $S_k$ be the set of all permutations of k elements, i.e., $S_k$ is a set that contains all bijective functions $\sigma :\{1,\dots ,k\}\to \{1,\dots ,k\}$. We define the adjusted transfer distance $\theta$ between partitions $\{A_1,\dots ,A_k\},\{B_1,\dots ,B_k\}$ of U by

$$\begin{array}{c}\hfill \theta (\{A_1,\dots ,A_k\},\{B_1,\dots ,B_k\})=min\left\{{\displaystyle \frac{n-{\displaystyle \underset{i=1}{\sum ^k}}|A_i\cap B_{\sigma \left(i\right)}|}{n}}:\sigma \in S_k\right\}.\end{array}$$

(15)

In [17], the transfer distance is a function defined for every pair of partitions from the set $\mathrm{\Pi }\left(U\right)$ of all partitions of U. In terms of properties, it is an n-invariant convexely additive metric [16]. Ref. ([23], p. 634) advocates using the transfer distance when the distances under consideration are relatively small or if the number of clusters is less than 5 or 6 (in [23] the adjusted transfer distance is called the “misclassification error distance”). We could follow this recommendation since the number of clusters in our study was always at most 4.

The maximum value of $\theta$ is known: a consequence of Lemma 6 from Ref. by [17] is that if $n\ge 2k-1$ (which is satisfied in our simulation study), then ${\displaystyle \frac{n-⌈\frac{n}{k}⌉}{n}}$ is the maximum value of the range of variation in $\theta$ (here $⌈{\displaystyle \frac{n}{k}}⌉$ is the smallest integer not below ${\displaystyle \frac{n}{k}}$). If $n\ge 2k-1$, the maximum value of $\theta$ either does not depend on n and equals ${\displaystyle \frac{k-1}{k}}$ if k divides n or differs from ${\displaystyle \frac{k-1}{k}}$ by less than ${\displaystyle \frac{1}{n}}$ if k does not divide n. A near-zero value of $\theta$ indicates good performance, and a $\theta$ value near the maximum value of the range of variation is a sign of poor performance.

Algorithm 2 defines the function Theta that is used to calculate the $\theta$ value in the simulation study. The arguments ${\mathrm{\Pi }}_1,{\mathrm{\Pi }}_2$ of Theta are both sets of partitions of U obtained by hierarchical clustering; k is the argument that specifies the cardinality of partitions to look for in ${\mathrm{\Pi }}_1$ and ${\mathrm{\Pi }}_2$. If there is not a partition with cardinality k either in ${\mathrm{\Pi }}_1$ or in ${\mathrm{\Pi }}_2$, then the $\theta$ value is defined as NA (not assigned).

We now describe the performance evaluation in a fixed arbitrary scenario. We performed $n_{sim}=144$ simulations. We denote by $n_{sim}^{\prime }$ the effective number of simulations, that is, the number of simulations that led to a non-NA-adjusted transfer distance value. By ${\theta }_i$, we represent the value of the adjusted transfer distance in the i-th effective simulation. We use as the performance measure the average adjusted transfer distance value defined as

$$\begin{array}{c}\hfill {\overline{\theta }}^{*}={\displaystyle \frac{1}{n_{\mathrm{sim}}^{\prime }}}\underset{i=1}{\sum ^{n_{\mathrm{sim}}^{\prime }}}{\theta }_i.\end{array}$$

(16)

Algorithm 2 Adjusted transfer distance value calculation

Data: $n,n_1^{\prime },n_2^{\prime },k\in \mathbb{N}$ such that $n_1^{\prime }\le n$, $n_2^{\prime }\le n$, $k\le n$; two sets of partitions ${\mathrm{\Pi }}_1=\{{\mathcal{A}}_1,\dots ,{\mathcal{A}}_{n_1^{\prime }}\}$, ${\mathrm{\Pi }}_2=\{{\mathcal{B}}_1,\dots ,{\mathcal{B}}_{n_2^{\prime }}\}$ of $\{1,\dots ,n\}$, where $|{\mathcal{A}}_i|>|{\mathcal{A}}_{i+1}|$ for all $i\in \{1,\dots ,n_1^{\prime }-1\}$, $|{\mathcal{B}}_i|>|{\mathcal{B}}_{i+1}|$ for all $i\in \{1,\dots ,n_2^{\prime }-1\}$ Result: the adjusted transfer distance value of k-class partitions $\mathcal{A}$ and $\mathcal{B}$ respectively in ${\mathrm{\Pi }}_1$ and ${\mathrm{\Pi }}_2$ if such partitions exist; otherwise, the adjusted transfer distance value is not assigned (NA)   function Theta(${\mathrm{\Pi }}_1,{\mathrm{\Pi }}_2,k$)     if exist $\mathcal{A}\in {\mathrm{\Pi }}_1$ and $\mathcal{B}\in {\mathrm{\Pi }}_2$ such that $\left|\mathcal{A}\right|=\left|\mathcal{B}\right|=k$ then         $theta\leftarrow \theta (\mathcal{A},\mathcal{B})$                                                                          ▹$\theta$ defined by (15)     else         $theta\leftarrow \mathrm{NA}$     end if     return $theta$ end function

The value of ${\overline{\theta }}^{*}$ can be seen as a realization of the random variable $\overline{\theta }$, which is obtained by viewing each ${\theta }_i$ as a random variable ${\widehat{\theta }}_i$. We assumed the random variables ${\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_{n_{sim}^{\prime }}$ to be independent and identically distributed. Following (p. 2086, [19]), we estimate the standard deviation of $\overline{\theta }$ using the Monte Carlo standard error of the average adjusted transfer distance estimator $\overline{\theta }$ by

$$\begin{array}{c}\hfill {SE}_{MC}^{*}\left(\overline{\theta }\right)={\displaystyle \frac{s}{\sqrt{n_{sim}}}}=\sqrt{{\displaystyle \frac{1}{n_{\mathrm{sim}}^{\prime }(n_{\mathrm{sim}}^{\prime }-1)}}{\displaystyle \sum _{i=1}^{n_{\mathrm{sim}}^{\prime }}}{\left({\theta }_i-{\overline{\theta }}^{*}\right)}^2},\end{array}$$

(17)

where

$$\begin{array}{c}\hfill s=\sqrt{{\displaystyle \frac{1}{n_{sim}-1}}\sum _{i=1}^{n_{sim}}{\left({\theta }_i-\overline{\theta }\right)}^2}.\end{array}$$

is the empirical standard deviation.

The basis for determining the number of simulations $n_{sim}$ was setting a certain upper bound $b=0.025$ for the half-length of the confidence interval of the mean of $\overline{\theta }$.

Let $z_{1-\alpha /2}$ be the $(1-\alpha /2)$-quantile of the standard normal distribution such that $z_{1-\alpha /2}=2$ (for this choice $\alpha <0.05$). Using the asymptotic normality of $\overline{\theta }$, we obtain

$$\begin{array}{cc}\hfill & z_{1-\alpha /2}{\displaystyle \frac{s}{\sqrt{n_{sim}}}}\le b,\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & n_{sim}\ge {\left({\displaystyle \frac{s{z}_{1-\alpha /2}}{b}}\right)}^2.\hfill \end{array}$$

(19)

Pre-study checks revealed that under various simulation scenarios, usual values of the standard deviation

$$\begin{array}{c}\hfill \tilde{s}=\sqrt{{\displaystyle \frac{1}{n_{\mathrm{obs}}-1}}\sum _{i=1}^{n_{\mathrm{obs}}}{\left({\theta }_i-\overline{\theta }\right)}^2}\end{array}$$

were in the majority of cases bounded by $0.15$ if $n_{\mathrm{obs}}$ was around 10.

Thus, our decision was to fix $s=0.15$. Then by (19),

$$\begin{array}{c}\hfill n_{sim}\ge 144.\end{array}$$

(20)

Using inequality (20) and by trying to minimize the time cost of the study, we set the number of simulations $n_{sim}$ to 144 in each scenario.

2.5.2. Effect Size and Its Monte Carlo Error

In Section 2.5.1, we defined the average adjusted transfer distance ${\overline{\theta }}^{*}$ as a performance indicator of our proposed method discussed in Section 2.1.2 in comparison with its modification, where bootstrap distributions are replaced with certain transformed multinomial distribution functions. As we pointed out from previous literature, the maximum value of $\theta$ for a pair of k-class partitions is known, and that gives us a scale against which to compare the values of ${\overline{\theta }}^{*}$ observed in the simulation study.

In the present subsection, we introduce another way of evaluating the performance. This performance measure is the effect size, which is based on the comparison between ${\overline{\theta }}^{*}$ obtained under a particular simulation scenario with its reference value ${\overline{\vartheta }}^{*}$ (yet to be defined) obtained via random partitioning. Given a set U with n elements, random partitioning means sampling a partition from the set of all k-class partitions of U under the uniform probability distribution assumption:

$$\begin{array}{c}\hfill P\left(\{X=\mathcal{P}\}\right)={\displaystyle \frac{1}{S(n,k)}},\mathcal{P}\in {\mathrm{\Pi }}_k\left(U\right),\end{array}$$

(21)

where $S(n,k)= |{\mathrm{\Pi }}_k\left(U\right)|$ is the Stirling number of the second kind, and X is a random partition.

Since ${\mathrm{\Pi }}_k\left(U\right)$ contains, in general, many elements, e.g., $S(60,4)\approx 5.5·{10}^{34}$, it is essential to have an algorithm that allows us to sample from ${\mathrm{\Pi }}_k\left(U\right)$ without previously enumerating the elements of ${\mathrm{\Pi }}_k\left(U\right)$.

Our approach, implemented as Algorithm 3, is as follows. Let ${\mathcal{Q}}_k$ be the set $\left\{\mathbf{q}=(q_1,\dots ,q_k)\in {\mathbb{N}}^k:q_1\ge \dots \ge q_k,{\sum }_{i=1}^k{q}_i=n\right\}$, and $S_n$ be a set containing all permutations (as n-tuples) of the form $\mathbf{s}=({\omega }_1,\dots ,{\omega }_n)$) of elements of U.

Algorithm 3 Random partitioning (adapted from p. 23, [20])

Data: $n,k\in \mathbb{N}$ such that $k\le n$ Result: a k-class partition $\{A_1,\dots ,A_k\}$ of $U=\{1,\dots ,n\}$   procedure RandPart($n,k$)     ${\mathcal{Q}}_k\leftarrow \left\{(q_1,\dots ,q_k)\in {\mathbb{N}}^k:q_1\ge \dots \ge q_k,{\sum }_{i=1}^k{q}_i=n\right\}$    $(q_1^{*},\dots ,q_k^{*})\leftarrow$ sample one k-tuple from ${\mathcal{Q}}_k$, the probability of selecting a tuple $\mathbf{q}=(q_1,\dots ,q_k)$ is given by $P(\left\{\mathbf{q}\right\}\times S_n\})$     $U\leftarrow \{1,\dots ,n\}$     $S\leftarrow$ set containing all permutations (as n-tuples) of elements of U     $({\omega }_1^{*},\dots ,{\omega }_n^{*})\leftarrow$sample one n-tuple from S (all outcomes equally likely)     define $h\left(x\right)={\sum }_{s=1}^x{q}_s^{*}$ for all $x\in \{1,\dots ,k\}$     $A_1\leftarrow \{{\omega }_1^{*},\dots ,{\omega }_{q_1^{*}}^{*}\}$     $A_j\leftarrow \{{\omega }_{h(j-1)+1}^{*},\dots ,{\omega }_{h\left(j\right)}^{*}\}$ for all $j\in \{2,\dots ,k\}$     return $\{A_1,\dots ,A_k\}$ end procedure

We sample a k-tuple $\mathbf{q}$ from ${\mathcal{Q}}_k$ and an n-tuple $\mathbf{s}$ from $S_n$. We define the sample space to be the Cartesian product ${\mathcal{Q}}_k\times S_n$. We consider the discrete sigma-algebra on ${\mathcal{Q}}_k\times S_n$. Let $X:{\mathcal{Q}}_k\times S_n\to {\mathrm{\Pi }}_k\left(U\right)$ be a random element (that we call a random partition) defined as follows. Take an element $((q_1,\dots ,q_k),({\omega }_1,\dots ,{\omega }_n))$ from the sample space. We read the elements of $({\omega }_1,\dots ,{\omega }_n)$ from left to right and put the leftmost $q_1$ elements into one set, the next $q_2$ elements into another set, …, and the rightmost $q_k$ elements of $({\omega }_1,\dots ,{\omega }_n)$ into yet another set. The obtained sets constitute a partition, that is, the value of X at $((q_1,\dots ,q_k),({\omega }_1,\dots ,{\omega }_n))$. We say that $X(\mathbf{q},\mathbf{s})$ is a partition with the class structure $\mathbf{q}$. The set of all partitions of U with the class structure $\mathbf{q}$ is denoted by ${\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)$.

Our next goal is to define a certain probability measure on ${\mathcal{Q}}_k\times S_n$ that would help us to sample from ${\mathrm{\Pi }}_k\left(U\right)$ according to the uniform distribution (21). Let $c_{\mathcal{P}}(q_1,\dots ,q_k)$ be the number of elements of $S_n$ that leads to the same partition (under construction explained above) for fixed k-tuple $\mathbf{q}$ from ${\mathcal{Q}}_k$, i.e., $c_{\mathcal{P}}\left(\mathbf{q}\right)=\left|A_{\mathcal{P}}\left(\mathbf{q}\right)\right|$, where $A_{\mathcal{P}}\left(\mathbf{q}\right)=\{\mathbf{s}\in S_n:X(\mathbf{q},\mathbf{s})=\mathcal{P}\}$ for fixed $\mathcal{P}\in {\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)$. The value of $c_{\mathcal{P}}\left(\mathbf{q}\right)$ does not depend on $\mathcal{P}$ for fixed $\mathbf{q}$, so we can drop the index $\mathcal{P}$. To express the formula for $c\left(\mathbf{q}\right)$, for each $\mathbf{q}\in {\mathcal{Q}}_k$, let $Q_{\mathbf{q}}$ be the set that contains all components of $\mathbf{q}$ without repetitions. We define the multiplicity, denoted by $m\left(q\right)$, of each element q of $Q_{\mathbf{q}}$ as the number of its occurrences in $\mathbf{q}$. The value of X is invariant to permuting components of $\mathbf{s}$ corresponding to the same class or permuting classes of components of $\mathbf{s}$ containing the same number of components. Therefore,

$$\begin{array}{c}\hfill c\left(\mathbf{q}\right)=\prod _{i=1}^k{q}_i!\prod _{q\in Q_{\mathbf{q}}}m\left(q\right)!.\end{array}$$

If the class structure of $\mathcal{P}$ is $\mathbf{q}$, then the event that a randomly selected partition is $\mathcal{P}$ is $\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)$, or put simply $\{X=\mathcal{P}\}$. For each $\mathbf{q}\in {\mathcal{Q}}_k$, each event of the form $\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)$ corresponds to one and only one partition $\mathcal{P}$ from ${\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)$, namely $X(\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right))$; and conversely, each partition $\mathcal{P}$ from ${\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)$ corresponds exactly to one event $\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)=X^{-1}\left(\mathcal{P}\right)$ (here ⁻¹ is the symbol for the inverse image). For each $\mathbf{q}\in {\mathcal{Q}}_k$, we define ${\mathcal{B}}_{\mathbf{q}}=\{\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right):\mathcal{P}\in {\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)\}$, and $\mathcal{B}$ as the disjoint union $\mathcal{B}={\bigcup }_{\mathbf{q}\in {\mathcal{Q}}_k}{\mathcal{B}}_{\mathbf{q}}$. The bijective correspondences ${\mathcal{B}}_{\mathbf{q}}\ni \left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)\mapsto \mathcal{P}\in {\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)$ can be easily combined to obtain a bijection between $\mathcal{B}$ and ${\mathrm{\Pi }}_k\left(U\right)$. By these bijections, the number of elements in $\mathcal{B}$ is $S(n,k)$, and for each $\mathbf{q}\in {\mathcal{Q}}_k$, the number of elements in $\{\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right):\mathcal{P}\in {\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)\}$ is $\left|\right\{{\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)\left\}\right|$. Hence,

$$\begin{array}{cc}\hfill S(n,k)& =\left|\mathcal{B}\right|=\sum _{\mathbf{q}\in {\mathcal{Q}}_k}|\{\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right):\mathcal{P}\in {\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)\}|=\sum _{\mathbf{q}\in {\mathcal{Q}}_k}\left|\left\{{\mathrm{\Pi }}_k^{\mathbf{q}}\left(U\right)\right\}\right|=\hfill \\ \hfill & =\sum _{\mathbf{q}\in {\mathcal{Q}}_k}{\displaystyle \frac{n!}{c\left(\mathbf{q}\right)}}.\hfill \end{array}$$

Therefore,

$$\sum _{\mathbf{q}\in {\mathcal{Q}}_k}{\displaystyle \frac{n!}{c\left(\mathbf{q}\right)S(n,k)}}=1,$$

which allows us to define the probability of selecting a k-tuple $\mathbf{q}$ from ${\mathcal{Q}}_k$ as follows:

$$\begin{array}{c}\hfill P(\left\{\mathbf{q}\right\}\times S_n)={\displaystyle \frac{n!}{c\left(\mathbf{q}\right)S(n,k)}}.\end{array}$$

Next, we define that the distribution for selecting $\mathbf{s}$ from $S_n$ is the uniform distribution:

$$P(\left\{\mathbf{q}\right\}\times \left\{\mathbf{s}\right\}|\left\{\mathbf{q}\right\}\times S_n)={\displaystyle \frac{1}{n!}}$$

Thus, the probability of obtaining a partition, given the class sizes, is

$$P(\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)|\left\{\mathbf{q}\right\}\times S_n)={\displaystyle \frac{c\left(\mathbf{q}\right)}{n!}}.$$

Therefore, the probability of obtaining a partition is

$$P\left(\{X=\mathcal{P}\}\right)=P(\left\{\mathbf{q}\right\}\times A_{\mathcal{P}}\left(\mathbf{q}\right)|\left\{\mathbf{q}\right\}\times S_n)P(\left\{\mathbf{q}\right\}\times S_n)={\displaystyle \frac{1}{S(n,k)}},$$

which is the inverse of the number of elements in ${\mathrm{\Pi }}_k\left(U\right)$. Therefore, Algorithm 3 implements sampling partitions according to the uniform distribution.

Let $n_{\mathrm{rep}}$ = 12,000. For each possible value of n and k from Table 1, we complete the following steps. We obtain $2n_{\mathrm{rep}}$ partitions with Algorithm 3, let those partitions be ${\mathcal{A}}_1$,…, ${\mathcal{A}}_{2n_{\mathrm{rep}}}$. For each $i\in \{1,\dots ,n_{\mathrm{rep}}\}$, we compute ${\vartheta }_i=\theta ({\mathcal{A}}_{2i-1},{\mathcal{A}}_{2i})$ by (15). We find the sample average ${\overline{\vartheta }}^{*}={\displaystyle \frac{1}{n_{\mathrm{rep}}}}{\displaystyle \sum _{i=1}^{n_{\mathrm{rep}}}}{\vartheta }_i$ to compare it with ${\overline{\theta }}^{*}$ given n and k.

We define the estimate ${\delta }^{*}$ of the effect size $\delta$ by

$$\begin{array}{c}\hfill {\delta }^{*}={\overline{\vartheta }}^{*}-{\overline{\theta }}^{*}.\end{array}$$

In the computation of the effect size estimates, we always assume that both ${\overline{\vartheta }}^{*}$ and ${\overline{\theta }}^{*}$ were determined under the same choice of a pair of numbers $n,k$. From the performance point of view, the larger the positive number of the effect size, the better.

The average adjusted transfer distance estimate ${\overline{\vartheta }}^{*}$ has its corresponding estimator $\overline{\vartheta }={\displaystyle \frac{1}{n_{\mathrm{rep}}}}{\sum }_{i=1}^{n_{\mathrm{rep}}}{\widehat{\vartheta }}_i$, where ${\widehat{\vartheta }}_1,\dots ,{\widehat{\vartheta }}_{n_{\mathrm{rep}}}$ are independent and identically distributed random variables: ${\widehat{\vartheta }}_i=\theta (X_1,X_2)$, where $X_1,X_2$ are independent random partitions in ${\mathrm{\Pi }}_k\left(\mathrm{U}\right)$ both with the uniform distribution.

To quantify the variability in the effect size estimation, we find an estimate for the standard error of the effect size estimator $\widehat{\delta }=\overline{\vartheta }-\overline{\theta }$. For n and k both fixed, the estimators $\overline{\vartheta },\overline{\theta }$ are independent since they are bound to different random experiments that have no relation to each other. Therefore, the standard error (SE) of $\widehat{\delta }$ is

$$\begin{array}{c}\hfill SE\left(\widehat{\delta }\right)=\sqrt{var\left(\widehat{\delta }\right)}=\sqrt{var\left(\overline{\vartheta }\right)+var\left(\overline{\theta }\right)}=\sqrt{{(SE\left(\overline{\vartheta }\right))}^2+(SE\left(\right(\overline{\theta })){)}^2}\end{array}$$

Estimate of the Monte Carlo standard error of $\overline{\vartheta }$ is

$$\begin{array}{c}\hfill {SE}_{MC}^{*}\left(\overline{\vartheta }\right)=\sqrt{{\displaystyle \frac{1}{n_{\mathrm{rep}}(n_{\mathrm{rep}}-1)}}\underset{i=1}{\sum ^{n_{\mathrm{rep}}}}{({\vartheta }_i-{\overline{\vartheta }}^{*})}^2}.\end{array}$$

(22)

By replacing the standard errors of $\overline{\vartheta }$ and $\overline{\theta }$ by their Monte Carlo estimates, the formula for the Monte Carlo standard error of the effect size estimator becomes

$$\begin{array}{c}\hfill {SE}_{MC}^{*}\left(\widehat{\delta }\right)=\sqrt{{\left({SE}_{MC}^{*}\left(\overline{\vartheta }\right)\right)}^2+{\left({SE}_{MC}^{*}\left(\overline{\theta }\right)\right)}^2},\end{array}$$

(23)

where ${SE}_{MC}^{*}\left(\overline{\vartheta }\right),{SE}_{MC}^{*}\left(\overline{\theta }\right)$ are given by (22) and (17), respectively.

The independence assumption of $\overline{\vartheta }$ and $\overline{\theta }$ was practically taken care of by fixing the seed only once at the beginning of the computer program for the whole study (see pp. 2082–2083, [19]).

The reason behind the choice of the value of $n_{\mathrm{rep}}=$ 12,000 was to make the standard error of $\overline{\vartheta }$ (and correspondingly also their estimates) as low as possible while keeping the execution time of the computer program affordable.

2.6. Simulation Algorithm

Algorithm 4 outlines the simulation study process of comparing the partitions obtained with Algorithm 1 with the partitions obtained by considering distribution functions of transformed multinomial random vectors (this is so-called ground truth) instead of empirical bootstrap distribution functions.

The steps of Algorithm 4 are as follows. The outer for-loop (lines 4 to 30) runs over all possible combinations of inputs 2 to 6. Input 1, the number of clusters, is introduced in the later part of the algorithm. First, the number of bootstrap replications is fixed to 1000 (line 5), the number of simulations in each scenario is fixed to 144 (line 6), and the model parameter coefficients are calculated (line 7). The middle-level for-loop (lines 8 to 29) runs over all simulations within a scenario, and it can be viewed as consisting of three parts.

Algorithm 4 Simulation study

1: Data: set of scenarios $\mathcal{S}=S_1\times S_2\times S_3\times S_4\times S_5\times S_6$ defined by (14)   2: Result: $n_{\mathrm{sim}}=144$ adjusted transfer distance values for each scenario in $\mathcal{S}$   3:     4: for all$(N,dist,n,m,r)\in S_2\times S_3\times S_4\times S_5\times S_6$  do   5:     $R\leftarrow 1000$   6:     $n_{\mathrm{sim}}\leftarrow 144$   7:     $c_{{\tau }_0}$ is calculated by (6) for given $\mathbf{m}$ and r for both ${\tau }_0=1$ and ${\tau }_0=2$   8:     for $\mathrm{rep}=1$ to $n_{\mathrm{sim}}$ do   9:         for all $i\in \{1,\dots ,n\}$ do 10:            if $dist=1$ then 11:                ${\mathbf{p}}_i\leftarrow$ sample prevalence according to (12) 12:            else if $dist=2$ then 13:                ${\mathbf{p}}_i\leftarrow$ sample prevalence according to (13) 14:            end if 15:            define $(B_1,B_2,B_3)\sim$ Multinomial($N;{\mathbf{p}}_i$) 16:            $F_i\leftarrow$ distribution function of $c_1{\displaystyle \frac{B_1}{N}}+c_2{\displaystyle \frac{B_2}{N}}$ 17:            ${\mathbf{t}}_i\leftarrow$N observations of a random variable T such that the 18:            j-th coordinate of ${\mathbf{p}}_i$ is the probability that T takes the value j 19:         end for 20:         $U\leftarrow \{1,\dots ,n\}$ 21:         $D_F\leftarrow$n-dimensional square matrix with entry $(i,j)$ calculated as 22:         $d_S(F_i,F_j)={sup}_{x\in \mathbb{R}}|F_i\left(x\right)-F_j\left(x\right)|$ 23:         ${\mathrm{\Pi }}_1\leftarrow$HierClust($U,D_F$) 24:         ${\mathrm{\Pi }}_2\leftarrow$ProposedMet($\{{\mathbf{t}}_1,\dots ,{\mathbf{t}}_n\},\mathbf{m},r,R$) 25:         for all $k\in S_1$ do 26:            consider $\mathbf{s}=(k,N,dist,n,m,r)$ 27:            $thet{a}_{\mathbf{s}}^{\mathrm{rep}}\leftarrow$Theta(${\mathrm{\Pi }}_1,{\mathrm{\Pi }}_2,k$) 28:         end for 29:     end for 30: end for

The first part (lines 9 to 19) is a for-loop, which is basically a preparation for clustering. First, for each $i=1,\dots ,n$, the prevalence sampling is performed according to the distribution specified by input 3 ($dist$). After that, a multinomial random vector is defined with the number of trials equal to the value of input 2 (N) and the cell probabilities defined by the prevalence vector ${\mathbf{p}}_i$ just sampled. Next, distribution function $F_i$ of the transformed multinomial random vector is defined (line 16), and a vector ${\mathbf{t}}_i$ of n observations is obtained by sampling from a categorical distribution with cell probabilities ${\mathbf{p}}_i$.

The second part (lines 20 to 24) performs the clustering. Here U is the set of labels used for clustering, and $D_F$ is the distance matrix, where entry $(i,j)$ is the supremum distance between $F_i$ and $F_j$. The distance matrix $D_F$ is then used to find the ground-truth clustering ${\mathrm{\Pi }}_1$. Clustering ${\mathrm{\Pi }}_2$ is an approximation to ${\mathrm{\Pi }}_1$ and is obtained with Algorithm 1. Both ${\mathrm{\Pi }}_1$ and ${\mathrm{\Pi }}_2$ are sets of nested partitions. Our task was to compare those partitions, and that was performed in the third part.

The third part (lines 25 to 28) performs the performance evaluation. The performance metric $\theta$ value is calculated for each value of the number of clusters k (input 1) based on ${\mathrm{\Pi }}_1$ and ${\mathrm{\Pi }}_2$ obtained under fixed values of all the other inputs 2 to 6. This leads to dependent performance data. It is computationally cheaper and should allow us to study the effect of k better.

3. Results

We studied the performance of our proposed method, outlined in Section 2.1.2, with the simulation study summarized in Section 2.2, Section 2.3, Section 2.4, Section 2.5 and Section 2.6. Recall from Section 2.5 that we considered two main performance measures: average adjusted transfer distance and the effect size. Before we turn to these performance measures, we present some graphs about the data based on which the performance measures were calculated.

Remark 1.

For those Figures below that have the horizontal axis entitled "Scenario", the explanation for the labels (from top to bottom) of top panels is: number of retained clusters (k), sample size (N), distribution for sampling prevalence (1—uniform discrete, 2—truncated geometric), number of populations (n), label for the expectation matrix (m). On the horizontal axis, all the 216 scenarios are grouped into 72 triples of scenarios: each grey vertical column corresponds to three scenarios, where all the other inputs are fixed, but the input r takes values 1, 5, 15. In each gray column, the left column corresponds to $r=1$, the middle column to $r=5$ and the right column to $r=15$. The explanation of how the scenarios for the inputs were ordered is also given at the end of Section 2.3.

Figure 2 shows the empirical scenario-specific distributions of the adjusted transfer distance estimates. The graphs of those 216 distributions are placed vertically in columns and were obtained with Algorithm 4; each distribution is based on $n_{sim}$ simulations. We see that the distributions of the distance values are not identical across scenarios. The distribution of the distances seems to shift further from 0 as the number of retained clusters (k) increases, but we also notice the presence of the effect of the sample taken from each population (N), the distribution used for prevalence sampling, and perhaps also the number of populations n. In several scenarios when $k=2$, the distribution of the distance is concentrated near 0.

With Algorithm 3 in Section 2.5.2 and the paragraph immediately below it, we explained how we applied so-called random partitioning in our study. The distributions of the adjusted transfer distance values obtained in random partitioning are illustrated in Figure 3. These distributions give us a sense of the typical distance values when the two partitions under comparison are independently obtained from the set of all k-class partitions so that each partition is equally likely. Roughly, if both partitions have two clusters, then typical distance values are $0.5$ or just below it, for three clusters, distance values are more concentrated in the range of $0.5$ to $0.6$, and in the case of four clusters, we mostly observe distances $0.58$ to $0.7$. In all six cases considered, the reference distributions had a mean value between $0.42$ and $0.65$, with the mean value increasing both with respect to the number of clusters k and with respect to the number of elements n in the base set used for clustering. Note that the mean values of the six distributions indicated by red dots in Figure 3 were used for calculating effect size estimates.

Recall from the last paragraph of step 8 in Section 2.1.2 that with our proposed method, not always clustering into a pre-specified number of clusters is possible. When we applied Algorithm 4, for the comparison method based on transformed multinomial distribution functions, we also used the same criterion as explained in Section 2.1.2 for determining whether or not a clustering into the desired number of clusters is possible. The function $\theta$ is a function of two variables (partitions). If, for fixed numbers n and k, either partition into k clusters was not possible, then (as explained with Algorithm 2) the value of $\theta$ was not possible to compute, leading to so-called NA values.

Figure 4 depicts the frequencies of NA values of the adjusted transfer distance as they occurred in the process of applying Algorithm 4 depending on the scenario number. Overall, there were 168 NA distance values out of the total of $n_{sim}\mathcal{S}=144·216=$ 31,104 simulations performed, so the proportion of NA values was approximately $0.54\%$ among all simulations, and in none of the scenarios the proportion of NA values was higher than $6/144\approx 4.2\%$. We pay attention to the fact that these NA values did not occur because of missing data in the usual sense. The occurrence of NA values was allowed by design; they were valid outcomes of the simulations since we used the variable-group hierarchical clustering method [13] accompanied by the ‘multidendrogram cutting technique’ for partitioning purposes. Ref. [13] argues that the multidendrogram is unique, but as our results show, when we use their method for partitioning purposes, this uniqueness comes with a price as we do not always obtain the needed number of clusters in a partition. However, the occurrence of NA values still appeared relatively low.

Since subsequent computations required us to use numerical data, our decision was to remove all observations with NA-adjusted transfer distance values. Therefore, all results that follow are conditional on the fact that in applying Algorithm 4, the value of the adjusted transfer distance is numerical.

Figure 5a provides a graphical overview of the values of one of the main performance measures that we computed—the scenario-specific average adjusted transfer distance estimates (${\overline{\theta }}^{*}$). The averages in Figure 5a were obtained from the respective distributions (plotted in Figure 2) resulting from the application of Algorithm 4. It is worth recalling from Section 2.5.1 that the lower the average distance, the better the performance. We observe that ${\overline{\theta }}^{*}$ values across all scenarios lie between $0.04$ and $0.55$. The lowest ${\overline{\theta }}^{*}$ values occur in scenarios where discrete uniform distribution, a sample size of 500, and two retained clusters are used; in those scenarios, the average $\theta$ was always between $0.04$ and $0.10$, which is at most 20% of the maximum possible $\theta$ value $0.5$ in those scenarios. The highest average $\theta$ values occurred in scenarios where truncated geometric distribution and a sample size of 100 were used; then, the average $\theta$ ranged between $0.31$ and $0.55$ and was always more than $58\%$ of the scenario-specific maximum $\theta$ value.

In Figure 5a, the patterns of the effects of each variable are perhaps a bit easier to notice than in Figure 2. Controlling for other inputs, the most notable effects have the following variables. Firstly, the distribution for sampling the prevalence; in all considered scenarios, the performance was worse when the prevalence was sampled with the truncated geometric distribution instead of the uniform distribution. As a reminder, the truncated geometric distribution was concentrated on values, which corresponded to the low total prevalence of the first two types of individuals in the branching process model. Second, the performance steadily worsened when the number of clusters k increased. Thirdly, the higher number of individuals sampled in each population N tended to improve the performance, but for this variable, the effect is not always very clear-cut. The same observations as for N apply to the number of populations n. Finally, the effects of the branching-process-related variables m and r were hardly noticeable. We continue the study of the effect of each variable below.

Figure 5b depicts the values of the second main performance measure—the effect size estimates that we computed to compare our method’s performance with the so-called random partitioning. We calculated the effect size estimate (${\delta }^{*}={\overline{\vartheta }}^{*}-{\overline{\theta }}^{*}$) for each scenario, which is obtained by subtracting the average in Figure 5a in that scenario from the mean value of such reference distribution in Figure 3, which had the same n and k values as in the scenario chosen. Considering Figure 5b, we observe that all effect size estimates are positive, so our method performs better than random partitioning regarding effect size estimates. A remarkable feature of the results we see is that the scenarios appear to divide into two clusters in terms of effect size estimates: one cluster includes all scenarios with the sample size $N=100$ and the truncated geometric distribution for sampling prevalence ($dist=2$) (see blue columns corresponding to scenarios 19–36, 91–108, 163–180); the other cluster includes all the remaining scenarios. For the truncated geometric distribution, together with the sample size of 100, all effect sizes were at most $0.19$. In all other scenarios, effect size estimates lie between $0.24$ and $0.50$, with the highest values occurring in scenarios where the distribution is uniform (discrete), and the number of retained clusters is three or four. This phenomenon of division of scenarios into two clusters (same clusters as in Figure 5b) was also visible in Figure 5a. We interpret the phenomenon through the lens of combined effects: the performance dropped most remarkably when the smaller sample size $N=100$ and the truncated geometric distribution for sampling prevalence were present together; when either of those two factors was present alone, the decline in performance was less noticeable.

Figure 6 allows us to have a closer look at the sensitivity of the performance with respect to each variable separately. The plot shows the range of observed average adjusted transfer distance values for each variable separately over all combinations of the remaining variables (these values were obtained by summarizing the data shown in Figure 5a). The variables $dist$ and N seemed to interact with each other, so we fixed the value of one of those variables to study the effect of the other (plots ‘dist, N = 100’, ‘n, dist = 2’, ‘dist, N = 500’, ‘N, dist = 1’). When we compare the plots in Figure 6, we see that the effect of $dist$ on the average distance range is greater when $N=100$ compared to the case $N=500$. Similarly, the effect of N is larger when $dist=2$ instead of $dist=1$. The third variable among those with the highest influence on the spread of average distance is the number of clusters k, but this does not take into account the fact that the so-called baseline distance increased when k increased, as we saw in Figure 3. To account for this baseline increase in the average distance value, it is useful to study the effect of each of the variables on the effect size estimates (shown in Figure 5b) instead of the average distance estimates (shown in Figure 5a).

Figure 7 shows the distributions of the range of the effect size estimates calculated for each variable separately. The main difference compared to Figure 6 is that the distribution for variable k has shifted to the right, which is also reflected by the fact that the mean value decreased from $0.180$ to $0.061$. At the same time, there was also a slight increase in the mean range value associated with variable n, but this shift was smaller in absolute size: from $0.059$ to $0.091$. As an interpretation, we can say that the increase that we saw in Figure 5a in the average distance with respect to k when controlling for other variables is at least partially credited to the difficulty of getting partitions closer to each other when the number of clusters increases.

4. Discussion

Although bootstrap has been used in cluster analysis in various forms, recent studies have not addressed how the errors made in bootstrap estimation of transformations of the sample mean propagate to the hierarchical clustering results. In an attempt to provide some insights into this, we investigated this problem for hierarchical single linkage clustering of empirical bootstrap distribution functions parametrized by three-type Galton–Watson branching processes. To the authors’ best knowledge, this is the first work of this kind.

Our method relied on a three-type Galton–Watson branching process model, whose assumptions we discussed in Section 2.1.1. Our proposed method is applicable as long as the model’s assumptions are not violated. Actually, it is also valid in a particular case when model assumptions are violated; namely, the usage of the model in our method was through the formulas for the expected number of individuals up to the r-th generation over both types. For the validity of the formulas of this expected value, the assumption that the offspring vectors ${\mathbf{X}}_{ij}^{\left(\tau \right)}$ are independent within a generation is not necessary. Therefore, our method is also applicable in the case of sibling interaction effects. However, this is just one aspect of the model, and overall careful examination is necessary in deciding whether or not the model is adequate to use.

To apply our method in practice, it is necessary to estimate the elements of the mean matrix $\mathbf{m}$. Currently, our method assumes that the mean matrix is the same for all populations. It is possible to modify the method by allowing different mean matrices across populations, but how that would affect the performance is an open question.

The next process involved in the method was a sampling of types and estimating the model parameter. The samples of types of individuals should be random samples to be able to properly apply the nonparametric bootstrap. A sampling technique adequate for obtaining such samples is simple random sampling with replacement. In principle, other sampling techniques and statistical estimation methods could be used, provided they result in the empirical distribution function of the model parameter estimator that is needed in the subsequent step of the method—clustering.

For clustering purposes, we used the supremum distance to compute the distance between distribution functions. One of the properties of the supremum distance is its boundedness from above by 1. In some applications, this may cause clustering problems. For example, if the distributions under consideration have the supports concentrated in non-intersecting segments of the real line, the supremum distance between each distribution function will be one, and obtaining a reasonable clustering is rather hopeless. A solution is to use an unbounded distance function, such as the Wasserstein distance (the integral over the real line of the absolute difference of two distribution functions).

As for the interpretation of our study’s results, the performance of our proposed method was generally less promising when prevalence was distributed according to the truncated geometric distribution instead of the uniform distribution. Moreover, the results showed that when controlling for other inputs, the performance was better when the sample size associated with each process was higher. We can interpret that prevalence being distributed uniformly across processes corresponds to a balance between processes with low and high prevalence; prevalence distributed according to the truncated geometric distribution means that there are relatively more processes with low prevalence than high prevalence. Therefore, the study results suggest that when there is a large proportion of processes with low prevalence, the number of individuals sampled for each process should be increased. Besides the sample size, another variable that, in principle, can be controlled in practice is the number of retained clusters. The results showed that the performance steadily worsened when the number of retained clusters increased from two to four. It points out that if there is not an important reason to prefer to retain four clusters, two or three clusters should be preferred to obtain a better performance.

Type-specific reproduction numbers (labeled by m) and the number of generations (labeled by r) did not affect the overall performance. However, we emphasize that this conclusion does not necessarily mean that these inputs do not affect the resulting partitions. Our performance measure gave us information on how well partitions obtained with the proposed method match the reference partitions. But it was not our focus and we did not study the composition of the resulting partitions themselves, nor how the composition of partitions changed for inputs. These topics can be addressed in future studies.

By using the bootstrap estimation of the model’s parameter, not only is variability in prevalence estimates taken into account, but it also gives insight into asymptotic results regarding the performance of our proposed method. It is well-known that under some conditions, the bootstrap gives asymptotically correct results in estimating transformations of the sample mean. Also, our results showed that when the sample size increased, the performance tended to improve. In light of all this, one can try to prove that our proposed method is asymptotically correct, meaning that the partition obtained with our method approaches the reference partition when the sample size increases. This is left for future work.

Occasional impossibility to calculate the performance measure ($\theta$) value due to ties did not lead to missing $\theta$ data in the ordinary sense—not assigned (NA) value of $\theta$ was a valid outcome of the simulation. We did all the analysis and answered all research questions, except those associated with NA values, by assuming that ties did not prevent partitioning into the desired number of clusters. Although we verified that problematic ties occurred with relatively low frequency, it would be desirable to extend the method so that ties cannot occur.

In conclusion, the results showed that our proposed method works quite well in some circumstances, and in most cases, ties were not a problem for applying our method. Therefore, the consistency of the nonparametric bootstrap carried over to hierarchical single linkage clustering promisingly.