1. Introduction
The phase-type aging model (PTAM) belongs to a class of Coxian Markovian models that were introduced in
Cheng et al. (
2021). The purpose of the PTAM is to provide a quantitative description of the effects of well-known aging characteristics resulting from a genetically determined, progressive, and irreversible process. It provides a means of quantifying the heterogeneity in aging among individuals and identifying the effects of anti-selection (
Cheng 2021;
Cheng et al. 2021).
The PTAM has a unique structure, including a constant transition rate for the aging process and a functional form for the relationship between aging and death (
Cheng et al. 2021). This structure gives rise to flat profile likelihood functions, putting the reliability of parameter estimates into question even if MLEs can be obtained (
Cheng 2021). This problem, which is referred to as the non-estimability issue, was studied in
Raue et al. (
2009). In this context, many estimated values of
m, the number of states, produced nearly the same profile likelihood values, resulting in similar model fitting. Actually, for certain statistical models such as the PTAM, the quality of model fitting is not the only consideration since the model parameters convey biological meaning. If the parameter estimates do not make sense in this context, the research problem cannot be satisfactorily addressed, even if the model fitting quality is found to be adequate. Accordingly, we investigated the problem of flat likelihood functions in the literature, focusing on the field of estimability, also referred to as practical identifiability.
Primarily, estimability must be assessed after identifiability, which can be defined as follows according to
Lehmann and Casella (
1998):
Definition 1. Let be a statistical model with either finite- or infinite-dimensional parameter space Θ. We say that is identifiable if If a model is assessed to be non-identifiable, then it proves unnecessary to consider its estimability (
Miao et al. 2011). The non-identifiability issue relates to properties of the model structure and can only be removed analytically by adding constraints to the model parameters or proposing new model representations (
Hengl et al. 2007;
Raue et al. 2009). On the other hand, the non-estimability issue relates to experimental protocols involving data quality (such as being insufficient or too noisy), algorithm approximation, or other noisy measurements. Thus, one must eliminate the non-identifiability issue before assessing estimability, in which case the remaining unreliability of the parameter estimates can be ascribed to experimental protocols (
Miao et al. 2011). It has been established that the PTAM is identifiable when the number of states is greater or equal to six (
Nie 2022).
We now highlight this paper’s main objectives and contributions with respect to the concept of estimability.
1.1. Defining the Concept of Estimability via an Objective Threshold
In fact, estimability has remained an open problem overall since existing methods for assessing it require the subjective specification of thresholds, which are left to the experimenters to specify. To the best of our knowledge,
Gontier and Pfister (
2020) seems to be the only paper that investigated and tackled the issue of threshold specification. They proposed a new definition of estimability that is based on a model selection perspective, where the subjective threshold is replaced with Bayes factors. The definition advocated in this paper solves this issue as it is based on a parallel but distinct perspective. Rather than eliminating the threshold, we make it an objective quantity that is related to the experiment itself, thus making it objective.
The proposed definition of estimability bears a resemblance to the profile likelihood function method introduced in
Raue et al. (
2009), where the non-estimability issue is defined as occurring when the profile-likelihood-based confidence region is infinite up to a subjective threshold. In order to make the threshold experiment-based and, therefore, objective, one needs to develop a methodology that somehow relates the confidence region to the experimental protocol. The proposed definition achieves this by replacing the profile-likelihood-based confidence region with an innovative confidence region that is based on a carefully designed c.d.f. sensitivity measure, whose purpose is to relate the confidence region to the experimental error by quantifying them into single numbers. This enables one to indirectly compare the confidence region and the experimental error. In this case, the threshold is tailored to the experimental protocol and then becomes objective. Since the measure is solely experiment-based, it quantifies the degree of estimability objectively so that the experimenter knows quantitatively the extent to which the experimental design should be improved in order to make the model estimable. The key to our approach is based on the empirical c.d.f. or ECDF; this is also the case for the Anderson–Darling and Kolmogorov–Smirnov tests, which compare a hypothesized distribution with the ECDF. The ECDF constitutes an objective measure as it represents the distribution of the experimental data.
1.2. Extending the Applicability of the Concept of Estimability to Statistical Inference
The concept of estimability originated in the field of biology, where an ODE model was utilized to model dynamic biological systems. It is also interchangeably referred to as “practical identifiability” in the literature. However, very little research on estimability has been conducted in connection with statistical models.
Gontier and Pfister (
2020) applied the concept of estimability to a statistical model. However, the scope of that paper still remained in system biology as a binomial model was introduced in connection with synaptic transmissions. We address the applicability of the concept of estimability in statistical inference from an objective perspective. Although this is principally illustrated via an application involving the PTAM, the proposed approach is actually applicable in other statistical inference contexts, which are illustrated by means of two additional numerical examples, one involving a discrete distribution and another, a continuous one. This constitutes the second main contribution of this paper.
1.3. Structure of the Paper
This paper is organized as follows. Preliminaries on the PTAM are introduced in
Section 2.
Section 3 provides a literature review on estimability.
Section 4 introduces the proposed definition of estimability, which is validated in
Section 5, where it is established to be innately sound. This definition is implemented to assess the estimability of the PTAM via a simulation study in
Section 6.
Section 7 contains certain remarks and a conclusion.
2. The Phase-Type Aging Model
The phase-type aging model (PTAM) stems from the mortality model introduced by
Lin and Liu (
2007). Phase-type mortality models enable one to link their parameters to biological and physiological mechanisms of aging, which constitutes a definite advantage. Thus, for instance, the longevity risk facing annuity products can be measured more accurately. Experimental results show that the phase-type mortality model with a four-state developmental period and a subsequent aging period achieved very satisfactory fitting results with respect to the Swedish and USA cohort mortality data set (
Lin and Liu 2007). Later on,
Su and Sherris (
2012) applied the phase-type mortality model to an Australian cohort mortality data set.
In a further study of the PTAM,
Cheng et al. (
2021) introduced a parsimonious yet flexible representation that allowed for the modeling of various aging patterns. In addition, an efficient algorithm for evaluating the likelihood of the PTAM was developed in
Cheng (
2023b). The main objective of the PTAM is to describe the human aging process in terms of the evolution of the distribution of physiological ages while utilizing mortality rates as aging-related variables. Thus, although the PTAM can reproduce mortality patterns, it ought not to be treated as a mortality model. In this context, the PTAM is most applicable at human ages beyond the attainment of adulthood, where, relatively speaking, the aging process is the most significant factor that contributes to the variability in lifetimes (
Cheng et al. 2021).
2.1. Preliminaries
Definition 2. Let be a continuous time Markov chain (CTMC) defined on a finite state space , where is the absorbing state and is the set of transient states. Let have as initial distribution over the transient states such that , and let the transition intensity matrix bewhere is an matrix, is the transpose of a null vector, and , with denoting a vector of ones. Define as the time until absorption. Then, T is said to follow a continuous phase-type (CPH) distribution, denoted by of order m, and is defined as the exit vector. Result 1. Given of order m,
It is well-known that, given a CPH distribution of order
m, if
and
has the structure
where
,
, and
, then the resulting distribution constitutes a Coxian distribution with no probability mass at zero
Cox (
1955a,
1955b). A phase diagram such as that displayed in
Figure 1 illustrates the process.
Definition 3. Given that , the PTAM of order m is defined as a Coxian distribution of order m with transition intensity matrix and exit rate vector such thatwhere , , andwhere . This is denoted by . As can be seen from
Figure 2, the PTAM has a phase diagram similar to that of the Coxian distribution shown in
Figure 1 except that the transition rates are constant and the exit rates are functionally related, as specified in (
2).
Note the following:
- (i)
In
Figure 2, each state in the Markov process represents the physiological age—a variable that reflects an individual’s health condition or frailty level. As the aging process proceeds, the frailty level increases until the last state, where the individual’s health conditions have deteriorated to the point of causing death.
- (ii)
The transition rate is assumed to be constant. The exiting rates denoted by are the dying rates or force of mortality. With this setup, an individual is randomly located in a certain state at a given calendar age. This mathematically describes the fact that the individuals involved will have different physiological ages given the same calendar age.
- (iii)
The assumption that dying rates have the structure given in (
2) is somewhat reminiscent of the well-known Box–Cox transformation introduced by
Box and Cox (
1964). The first and last dying rates
and
are included in the model parameters, whereas the remaining in-between rates are interpolated in terms of the parameter
s, which is a model parameter related to the curvature of the exit rate pattern. To verify this,
Figure 3 and
Figure 4 present the effect of
s on the pattern of the exit rates for
and
, respectively. When
, the dying rates have a linear relationship. When
, the rates are concave, and when
, the rates are convex. In particular, when
, the rates behave exponentially. In practice, it is likely that
s is less than one when calibrating to mortality data (
Cheng et al. 2021). That is, the dying rates increase faster than linearly as an individual ages. Throughout this paper, it is assumed that
follows the structure given in (
2) for
.
- (iv)
The value of
needs to be commensurate to the value of
m; otherwise, there would be no need to have numerous
m states when
is small. Accordingly, we let their ratio be a constant; that is,
which can be seen as a reparameterization of the PTAM that involves the five parameters
. We note that such a reparameterization establishes a positive covariance between
and
m, which is more in line with the biological interpretation. From now on, this parameterization of the PTAM is utilized.
2.2. Identifiability of the PTAM
The identifiability of the PTAM was established in
Nie (
2022) for
, where
m denotes the number of states. Illustrative examples of non-identifiable PTAM were also provided for
to clarify the concept. The identifiability, or mathematical uniqueness, guarantees that no other c.d.f.-equivalent representations of the model exist and that the likelihood function has a unique global maximum.
However, identifiability does not imply estimability. Although the model representation is unique, parameter estimates can still be unreliable when the profile likelihood functions are extremely flat, as a large range of different estimates can produce nearly identical likelihood values (
Raue et al. 2009). According to
Cheng (
2021), the profile likelihood functions of the PTAM’s parameters
,
, and
s turn out to be flat, which gives rise to this estimability issue.
3. Literature Review
3.1. Methods for Assessing Estimability
Estimability is also referred to as practical identifiability in the literature. The non-estimability issue arises in the case of flat likelihood functions (
Raue et al. 2009) or, equivalently, the insensitivity of the model c.d.f. with respect to its parameters (
McLean and McAuley 2012), which may be due to either of the following situations:
- (i)
The model c.d.f. is insensitive with respect to parameter changes. Accordingly, this aspect involves model sensitivity.
- (ii)
The effect of one parameter on the c.d.f. may be offset by that of one of the other parameters, this being defined as parameter correlation.
If the model is identifiable, then unreliable parameter estimates, if any, are caused by non-estimability issues that may be due to experimental errors, including data quality (insufficient or too noisy), algorithm approximation, or other noisy measurements (
Miao et al. 2011). Unlike identifiability, estimability is less well-defined, and its characterization has remained an open problem. While it is straightforward to think qualitatively that parameters can be “loosely estimated” under noisy measurements, one would need to define quantitatively what this really means (
Gontier and Pfister 2020;
Raue et al. 2009).
Estimability has been widely studied in system biology, where an ODE model is utilized to model dynamic biological systems. For instance, it is assumed that
where
Four broad types of methodologies are employed. We briefly describe them next.
3.1.1. The Monte Carlo Method
This approach, which was utilized by
Aslett (
2012) in connection with certain lifetime models, involves repeated parameter estimation from a large number of data sets that were simulated by means of the Monte Carlo method. To apply this method, threshold values for parameter uncertainty levels are required to distinguish estimable from non-estimable parameters. Let
be the nominal parameter vector obtained from fitting the model to the original data or from prior knowledge (
Miao et al. 2011). Let
be the vector of the parameter estimates at the
of
M trials, which are based on data simulated from the model having
as its parameter vector. Then, the average relative estimation error (ARE) associated with
, the
element of
, is given by
where
is the
element
,
.
Miao et al. (
2011) defined non-estimability as occurring when the ARE of a parameter is sufficiently high or, equivalently, exceeds a pre-selected threshold
.
3.1.2. Methods Based on the Correlation Matrix or the Fisher Information Matrix
According to
Petersen et al. (
2001), the Fisher information matrix (FIM) associated with the ODE model is given by
where
Rodriguez-Fernandez et al. (
2006) proposed a correlation matrix approach for analyzing the estimability of the ODE model. By the Cramér–Rao Theorem, the covariance matrix can be obtained as
the correlation between
and
being
Similarly,
Quaiser and Mönnigmann (
2009) proposed a total correlation measure. In this instance,
and
are deemed to be non-estimable if their correlation is sufficiently high or, equivalently, exceeds a certain threshold
.
There exist several methods focusing on the FIM.
Dochain and Vanrolleghem (
2001) proposed that the condition number, which is defined as the ratio of the largest eigenvalue to the smallest eigenvalue of the FIM, could also be used to assess estimability. The larger the condition number, the more correlated the parameters and the less estimable they are. The model is then non-estimable if the condition number is sufficiently high or, equivalently, exceeds a certain threshold
. In addition,
Brun et al. (
2001) proposed a collinearity index to measure the parameter correlations. The model is deemed less estimable if the collinearity index is relatively large, which also requires the specification of a threshold.
3.1.3. Methods Based on the Model Sensitivity
Another approach is based on the model sensitivity. As can be seen from (
3), the FIM is obtained in terms of the sensitivity matrix. Thus, the sensitivity matrix may be extracted from the FIM and analyzed specifically. The sensitivity matrix
with observation times
is defined as
Several methods are based on the sensitivity matrix.
Jacquez and Greif (
1985) calculated a sample correlation between the matrix columns. If
is close to one within a certain threshold, then
and
are considered to be non-estimable. Other methods exist, such as the principal components analysis (PCA) technique (
Degenring et al. 2004), the orthogonal method (
Yao et al. 2003), and the eigenvalue method (
Vajda et al. 1989), all of which rely on subjective thresholds.
If the model is sufficiently simplified and involves only a few parameters, the sensitivity function
can be solved analytically, in which case the sensitivity matrix is not needed.
Holmberg (
1982) proposed a visual inspection approach to the sensitivity function. The larger the sensitivity measure of one parameter, the greater the change in the model c.d.f. with respect to the change of that parameter. If the sensitivity functions of certain parameters are linearly dependent, then those parameters are functionally related. The drawback of this approach is that a correlation cannot be quantified based on graphs. Moreover, a visual inspection entails a subjective assessment. Determining whether the graphs of the sensitivity functions are truly dependent depends on the experimenter’s assessment.
3.1.4. Methods Based on Profile Likelihood
Raue et al. (
2009) proposed an explicit definition of estimability that is based on the profile likelihood function. They defined the profile likelihood confidence interval for parameter
as
where
is the profile likelihood function of
and
is a subjectively chosen threshold. Then,
is said to be non-estimable if
is infinite. In other words, given a certain threshold
, there exists a
such that for all
,
holds true. This definition is mathematically clear as it relies on a binary event: whether
is infinite or not. However, a subjective threshold
is still required, as is the case for the other methods.
3.2. Relationships between Identifiability, Estimability, and Sensitivity
Informally, sensitivity refers to the degree to which a model is affected by its parameter values. Graphically, the more sensitive a model is with respect to one parameter, the more noticeably the model’s c.d.f. is affected by changes in that parameter. In this paper, we quantify this concept by introducing the c.d.f. sensitivity measure that is specified in Definition 4.
Identifiability and estimability are related via the concept of sensitivity. From the perspective of sensitivity, if a statistical model
is non-identifiable with the non-identifiable set being
then
f has zero sensitivity if the parameter vector changes within
since the model c.d.f. does not change. Thus, the non-identifiability issue cannot be overcome by improving the experimental design since the model output is then the same for all
x.
On the other hand, if the sensitivity is not zero, then different parameters produce different model c.d.f. values. In this case, different values of x produce different model outputs, and the experiment contributes some information toward parameter inference. The more sensitive the model with respect to one parameter, the more estimable that parameter is. Although, as previously mentioned, different measures, such as those based on correlations, condition numbers, and eigenvalues, may be employed, in each case, a threshold must be set.
It should be emphasized that identifiability and estimability are equally important. If a model is non-identifiable, then the likelihood function has multiple global maxima. In that case, although we are relying on a numerical algorithm that aims to maximize the likelihood function, we may question the reliability of the parameter estimates produced by that algorithm since other maxima may potentially exist. In the case of estimability, although we are certain that the MLEs are unique, we may as well question the reliability of the parameter estimates produced by that algorithm since a wide variety of estimates can yield nearly the same likelihood values.
4. A Methodology for Objectively Assessing Estimability
Based on a thorough review of the literature, very little research on estimability appears to have been conducted in connection with statistical models. To the best of our knowledge, there is only one paper, namely
Gontier and Pfister (
2020), that studied estimability with respect to a statistical model, wherein a new definition of estimability based on a model selection perspective was proposed. A significant contribution of theirs was the elimination of the subjective threshold in (
4) by introducing a Bayes factor into the new definition. Our contribution also aims to address the problem of having to set a subjective threshold. Rather than eliminating it, we shall make the threshold an objective, experiment-based quantity.
In order to do so, a methodology needs to be established that relates the confidence region to the experimental protocol. To achieve this, we rely on the following considerations:
- (i)
The curvature of the likelihood function reflects the sensitivity of the model (c.d.f.) with respect to the parameters (
McLean and McAuley 2012).
- (ii)
The non-estimability issue is defined as occurring when the likelihood-based confidence interval is infinite (
Raue et al. 2009).
In regards to the first consideration, we replace the likelihood-based confidence region introduced in
Raue et al. (
2009) with an innovative confidence region that is based on a carefully designed c.d.f. sensitivity measure. Such a sensitivity measure aims to relate a confidence region to the experimental error by quantifying them. This is indirectly achieved by comparing these quantified numbers under the same measure. In this case, the threshold is tailored to the experimental protocol and, thus, becomes objective. Additionally, in light of the second consideration, we define non-estimability as occurring when the proposed confidence region is infinite.
Several preliminary definitions are needed before defining estimability:
Definition 4. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Then, for , the c.d.f. sensitivity between and with respect to the random sample is defined as Definition 5. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Then, for all real numbers , and are said to be indistinguishable with respect to if their c.d.f. sensitivity with respect to the random sample is no greater than . That is, for , Definition 6. For a given statistical model , consider the procedure utilized for obtaining parameter estimates for that are based on the random sample and the numerical algorithm being implemented. Such a procedure is referred to as experiment Φ.
Definition 7. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Let be an estimated model of f with respect to experiment Φ
and be the empirical c.d.f. (ECDF) obtained from Φ.
Then, the experimental error associated with Φ
is defined as the c.d.f. sensitivity between the estimated model and the ECDF. That is, Definition 8. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Let be an estimated model of f with respect to experiment Φ.
Then, is said to be indistinguishable with respect to Φ
if and are indistinguishable with respect to the experimental error ϵ as defined in (5). That is, for , Definition 9. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Let be an estimated model of f with respect to experiment Φ;
then the set is called a c.d.f. sensitivity-based confidence region (CSCR) with respect to Φ
ifwhere ϵ is as defined in (5). Definition 10. Consider an identifiable statistical model , where Θ is the parameter space with . Given , let be a sub-space of Θ with , and define so that . Let the parameters in be , and let the parameters in be . Let denote the boundary of the domain in the parameter space Θ.
Then, a statistical model is said to be the sub-model of ifwhere is the boundary of and is the associated c.d.f. Definition 11. Consider an identifiable statistical model , where Θ is the parameter space. Then, is said to be the sub-model family of if it comprises all the sub-models of . Namely,where is a sub-model of and is the number of sub-models. This paper’s principal contribution is the definition of the estimability of a statistical model that follows.
Definition 12. Consider an identifiable statistical model with parameter space Θ. Then, is said to be non-estimable with respect to experiment Φ if its CSCR with respect to experiment Φ is infinite. Accordingly, is said to be estimable if its CSCR with respect to experiment Φ is bounded.
Observe that expression (
5) quantifies the experimental error as
. It can also be interpreted as the tolerance level within which the estimated model c.d.f. can vary. The CSCR essentially includes all possible parameters such that the c.d.f. sensitivity of a model having those parameters is less than the experimental error. Clearly, the smaller the experimental error, the smaller the CSCR, this being due to the fact that the experimental error is set as an upper bound in (
6). The next step is to make Definition 12 applicable in practice, as this definition may not be of practical use if utilized directly. This caveat is addressed in the next two theorems.
Theorem 1. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Assume that has a sub-model family . Then, is non-estimable if there exists at least one sub-model that satisfies both of the following conditions:
- (i)
, where ϵ is as defined in (5), and F and are the associated c.d.f. values. - (ii)
contains ∞ or .
Proof. Let a sub-model
satisfy both of the conditions, where
. By Definition 10, we have
Let
be the estimates of
for given values of
. We write it this way to reflect its dependence on
. Assuming that the parameters are estimated under the same numerical algorithm, then for all
x, we have
Subsequently, based on (
7), we have
Denote the
element in
by
. In light of condition (ii),
then becomes either
∞ or
. On applying condition (i), (
8) then implies that for some
j,
Thus, belongs to the CSCR by Definition 9. In that case, the CSCR is infinite given the parameters included in . Then, by Definition 12, is non-estimable with respect to experiment . □
Theorem 2. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Assume that has a sub-model family . Then, is estimable if, for all , conditions (i) and (ii) specified in Theorem 1 cannot be simultaneously satisfied.
Proof. If condition (ii) is not satisfied, then
does not contain
∞ or
. Denote the
element in
by
, and denote the
element in
by
. Then, (
7) implies that for any
and for all
,
such that
,
.
Therefore, no matter whether or not condition (i) is satisfied, is already contained in a bounded region. The CSCR, which is a sub-region of this bounded region, is, of course, bounded as well.
It remains to consider the case when condition (i) is not satisfied while condition (ii) is. In that case, since condition (i) is not verified, we have
Since condition (ii) is satisfied, (
8) then implies that for all
j,
This is equivalent to
Therefore, for each parameter of
that is in the CSCR by Definition 9, its range in absolute value is covered by a bounded interval
. We may then conclude that the entire CSCR is a bounded region. Thus, by Definition 12,
is estimable with respect to experiment
. □
Assuming that the statistical model under consideration is identifiable, the steps to follow for implementing the proposed methodology are enumerated in the next algorithm.
Algorithm 1 The estimability analysis for a statistical model |
- 1:
Determine the sub-models of —a requirement to conduct the proposed estimability analysis. - 2:
Obtain a parameter estimate of , . The estimator should be consistent. It is recommended to proceed as explained in Section 7.1. - 3:
Compute the experimental error , as specified in Definition 7. - 4:
Proceeding as in Step 2, obtain the parameter estimates for all the sub-models of the statistical model. - 5:
Compute the c.d.f. sensitivities between the estimated statistical model and each one of its estimated sub-models using Definition 4. - 6:
Compare all the c.d.f. sensitivities obtained in Step 5 with the experimental error determined in Step 3. If all c.d.f. sensitivities are greater than the , then the statistical model is estimable with respect to the current experiment in light of Theorem 2. Otherwise, it is non-estimable with respect to the current experiment by virtue of Theorem 1. - 7:
If non-estimable, one may consider improving the experimental design by making use of the techniques described in Section 5.3 and then reassess estimability, starting with Step 2.
|
5. Validation of the Proposed Definition
In this section, we substantiate the proposed definition and conclude that it is innately sound. This is achieved by establishing the validity of the associated theoretical results, which happen to be consistent with common sense.
5.1. Validation of the Data Noise, the Algorithm Noise, and the Experimental Error
We first validate the data noise, the algorithm noise, and the experimental error as specified in Definitions 13 and 7. Without any loss of generality, it is assumed that the experimental error originates from the data noise and the algorithm noise (
Chis et al. 2011). The data noise and the algorithm noise are next defined as c.d.f. sensitivity measures:
Definition 13. Consider an identifiable statistical model , where Θ is the parameter space and is the associated c.d.f. Let be an estimated model with respect to experiment Φ.
Denoting the true value of the parameter estimate by , the data noise and the algorithm noise of with respect to experiment Φ
can be defined as the following c.d.f. sensitivities, respectively: We now show that both types of noise tend to zero as the sample size increases. As the sample size goes to infinity,
where
is the true parameter of the model (unknown, of course) and
is the support of
. Note that the limits of
and
are based on asymptotic results on the consistency of the estimator
and the convergence of the ECDF
, respectively. Thus,
is a valid measure of the data noise that behaves as expected.
Now, consider the algorithm noise. As the accuracy of the algorithm tends to perfection,
Thus, as the algorithm is becoming nearly exact, the approximation of
is more accurate, which also agrees with common sense. Accordingly,
is a valid measure of the algorithm noise as well.
After having validated the data noise and the algorithm noise, we may now validate the experimental error. Observe that
Then, as the experimental design tends to perfection, both the data noise and the algorithm noise tend to zero. Consequently, tends to zero by appealing to the so-called “Squeeze Theorem”. It is worth emphasizing that and are unknown to the experimenter since the true value of the parameter estimate is also unknown. Instead, one only knows , which is the output of the numerical algorithm. However, this does not prevent us from establishing that is a valid measure of the experimental error. Otherwise, one would be able to calculate analytically without resorting to numerical algorithms.
5.2. Validation of the c.d.f. Sensitivity-Based Confidence Region
We next validate the definition of the CSCR. Observe that the experimental error,
, is an upper bound in (
6). Thus, as the experimental error decreases, the CSCR shrinks, making the model more estimable, which is consistent with intuition. Now, as in
Section 5.1, consider the argument that the experimental design tends to perfection. Then,
where
is the true (unknown) parameter of the model.
Thus, the CSCR, , collapses to , which is exactly the parameter set for which is non-identifiable. Therefore, if the experimental error goes to zero, then any remaining unreliability associated with the parameter estimates must originate from the non-identifiability issue. This agrees with the statement found in the literature to the effect that it is unnecessary to analyze estimability if the model is non-identifiable since the non-identifiability issue cannot be overcome by improvements in the experimental design. Accordingly, the demonstrated consistency with respect to the CSCR further validates the proposed definition of estimability.
5.3. Validation by Known Techniques
The proposed definition can also be substantiated by making use of known methods for improving estimability. These are discussed next.
5.3.1. Increasing the Sample Size
Increasing the sample size can decrease both
and
. More observations not only decrease the data noise but also decrease the algorithm noise as the likelihood functions then become more and more concave. Then, in light of the arguments presented in
Section 5.1 and
Section 5.2,
decreases, and the CSCR shrinks, which improves estimability. As the sample size tends to infinity,
tends to zero, and the experiment tends to perfection. Thus, the proposed definition is consistent with an increase in the sample size.
5.3.2. Increasing the Convexity of the Log-Likelihood Function
Another way to increase the sample size is to clone the simulated data multiple times. This is called the data cloning method (
Lele et al. 2007,
2010;
Cheng 2023a). With this approach, the likelihood functions become more and more concave, which makes the algorithm approximation more accurate. Accordingly,
decreases. However, this approach only reduces the algorithm noise. It cannot improve
, as
does not change. Thus, the proposed definition is consistent with the data cloning method.
5.3.3. Improving the Algorithm Design
One can also decrease by improving the efficiency of the algorithm approximation. However, the data noise remains unchanged, which is consistent with intuition as the improvement only pertains to the algorithm.
5.3.4. Securing More Complete Information
Estimability can also be enhanced if more information is secured. As the data convey additional information,
decreases. Then, in light of the arguments presented in
Section 5.1 and
Section 5.2,
decreases, and the CSCR shrinks, which improves estimability. Thus, the proposed definition is consistent with obtaining more complete information.
5.4. Illustrative Examples
In
Section 5.1,
Section 5.2 and
Section 5.3, we have theoretically validated our definition of estimability by establishing that it is consistent with common sense. Examples illustrating its applicability to discrete and continuous statistical models follow.
5.4.1. Example 1
Consider the following constrained binomial model:
With the binomial distribution being identifiable, its estimability may be assessed. According to Definition 11,
,
,
,
,
. Then, the sub-model family of
based on
is
since the Poisson distribution is the limiting distribution of the binomial distribution as
with the constraint
.
Let the underlying model have as its parameters and (or ), and consider following three experiments:
Experiment 1: The sample size is .
Experiment 2: The sample size is .
Experiment 3: The sample size is .
The estimability assessment results are presented in
Table 1 and
Figure 5 and
Figure 6.
Table 1 compares the experimental error with the c.d.f. sensitivity between the fitted binomial distribution and its sub-model, which is a direct application of Theorems 1 and 2.
It can be observed that the experimental error decreases as the sample size increases, which again validates the definition of the experimental error. According to Theorem 1, is non-estimable with respect to Experiments 1 and 2 since the c.d.f. sensitivity measure is less than the experimental error. On the other hand, according to Theorem 2, becomes estimable with respect to Experiment 3 since the c.d.f. sensitivity measure is greater.
The above conclusions are further supported by
Figure 5, where the CSCRs for Experiments 1, 2, and 3 are visualized. In line with Theorem 1, the CSCR for Experiments 1 and 2 are both infinite. As the sample size increases, the CSCR shrinks until it becomes bounded in Experiment 3, where
becomes estimable, which is in line with Theorem 2.
A more intuitive way of interpreting the concept of estimability is displayed in
Figure 6. In Experiments 1 and 2, the model is assessed to be non-estimable. This can be interpreted as follows: the inferential power from the data displayed in the histogram is not sufficient to distinguish the estimated p.m.f.s of the underlying model and its sub-model. This is exactly why estimability is also referred to as “practical identifiability” since the shape of the histogram cannot “practically identify” the estimated p.m.f. values of the underlying model and its sub-model. On the other hand, in Experiment 3, the model is assessed to be estimable. Thus, the inferential power is sufficient to distinguish the estimated p.m.f. values. In this case, the shape of the histogram is sufficient to favor the underlying model and dismiss its sub-model.
However, these conclusions cannot be reached by only inspecting
Figure 6, which is why the CSCR associated with the proposed definition is crucial.
5.4.2. Example 2
Consider the following constrained Lomax Pareto Type II (henceforth Pareto) model:
With the Pareto distribution being identifiable, its estimability may be assessed. According to Definition 11,
,
,
,
,
. Then, the sub-model family of
based on
is
since the exponential distribution is the limiting distribution of the Pareto distribution as
with the constraint
.
Let the underlying model have as its parameters and (or ), and consider following three experiments:
Experiment 1: The sample size is .
Experiment 2: The sample size is .
Experiment 3: The sample size is .
The estimability assessment results are presented in
Table 2 and
Figure 7 and
Figure 8.
Table 2 compares the experimental error with the c.d.f. sensitivity between the fitted Pareto distribution and its sub-model, which is a direct application of Theorems 1 and 2. The results yield the same conclusions that were obtained in Example 1.
It should be emphasized that the visualization of the CSCR displayed in
Figure 5 and
Figure 7 is not achievable if the parameter space extends to more than three dimensions, such as in the case of the PTAM, which is considered next. In that case, we have to fully rely on Theorems 1 and 2 and compare the c.d.f. sensitivity with the experimental error. The two-dimensional illustrative examples that were previously presented provide supporting evidence corroborating the validity of Theorems 1 and 2. Another important aspect to point out is that the algorithm utilized to obtain the parameter estimates coincides with that recommended in
Section 7.1. This aspect is discussed further in the remainder of this paper.