Statistics for Continuous Time Markov Chains, a Short Review

Abstract

This review article is concerned to provide a global context to several works on the fitting of continuous time nonhomogeneous Markov chains with finite state space and also to point out some selected aspects of two techniques previously introduced—estimation and calibration—relevant for applications and used to fit a continuous time Markov chain model to data by the adequate selection of parameters. The denomination estimation suits the procedure better when statistical techniques—e.g., maximum likelihood estimators—are employed, while calibration covers the case where, for instance, some optimisation technique finds a best approximation parameter to ensure good model fitting. For completeness, we provide a short summary of well-known important notions and results formulated for nonhomogeneous Markov chains that, in general, can be transferred to the homogeneous case. Then, as an illustration for the homogeneous case, we present a selected Billingsley’s result on parameter estimation for irreducible chains with finite state space. In the nonhomogeneous case, we quote two recent results, one of the calibration type and the other with more of a statistical flavour. We provide an ample set of bibliographic references so that the reader wanting to pursue her/his studies will be able to do so more easily and productively.

1. Introduction

In this review, we will consider Markov chains, in continuous time, with a finite state space. This kind of Markov chain has been used in many modelling studies in actuarial mathematics (see [1] for Markov and semi-Markov models and [2] for demography), in risk and credit management (see [3,4,5,6,7]), in social sciences (see [8] and the articles described in [9]), in biology and medicine (see [10,11,12,13] for simulation studies comparing discrete with continuous time Markov chains and applications to Alzheimer-type dementia, pathology in late life and to normal ageing of the brain in contrast to Alzheimer’s disease, and to ambulatory hypertension monitoring, respectively) and in many other areas.

Our main goal is to provide a summary description of the main ideas—with relevant bibliographic references—whenever these are solidly established and, in the case of nonhomogeneous Markov chains, to enter into the detail of some particular ideas and results susceptible of improvement and further research. As such, we view this contribution as a guide to at least part of the relevant literature on the subject. Let us briefly describe the organisation of this text.

• In Section 2, we mention the main concepts and results on continuous time Markov chains that provide a context and reference for the remainder of this text.
• Section 3, dealing with homogeneous Markov chains, is devoted to a brief presentation of the most important ideas on statistical inference for these chains, mainly for those having a finite state space.
• Section 4 addresses the case of nonhomogeneous Markov chains, and, besides referencing several works on the subject, provides some details of two procedures which enable the determination of parameters of the intensity matrix; the procedure termed calibration covers the case were the departure point for the procedure is some family of discrete time probability transition matrices with numerical entries, and the procedure termed estimation considers data composed of the trajectories of a chain with an absorbing state and with intensities of arbitrary functional form depending on parameters.
• In Section 5, we provide some comment on references on alternative estimation or calibration procedures for Markov chains.
• In Section 6, we present some conclusions drawn from the presentation in this work.

2. Definitions, Notations and Results on Continuous Time Markov Chains

The diversity of the whole universe of Markov chains theories—discrete time, continuous time, homogeneous and nonhomogeneous, with a finite or with an infinite denumerable state space—is fascinating and since it provides sound foundations for modelling in various sciences, it remains a very active area of study. There are many very good references allowing for preliminary study and which provide more in-depth coverage of its most intricate features. Let us briefly detail a selection of these references. Classical introductory readings, although requiring a not insignificant amount of work are [14,15]. Very interesting introductions to the subject with applications in mind are [16,17,18]. For the most mathematically inclined reader, the two invaluable references [19,20] provide an important complement to the first two quoted references; for a deeper understanding of the many subtleties of the theory, the five following references are essential: [21,22,23,24,25].

We next review—following mainly [26,27,28]—the main concepts and results that will be needed to deal with some aspects of the statistical inference for a nonhomogeneous Markov chain process ${\left(Z_t\right)}_{t\ge 0}$ depending on some—possibly vectorial—parameters.

Definition 1

(Continuous time Markov chain). Let $\mathcal{I}:=\{1,2,\dots ,m\}$ be the finite set associated with $\mathbf{\Theta }=\{{\mathit{\theta }}_1,{\mathit{\theta }}_2,\dots ,{\mathit{\theta }}_m\}$. The stochastic process ${\left(Z_t\right)}_{t\ge 0}$ is a continuous time Markov chain with state space $\mathcal{I}$ (or Θ) if the following characteristic Markov property is verified; namely, for all $i_0,i_1,\dots i_n\in \mathcal{I}$ and $0=t_0<t_1<\cdots <t_n<\cdots$, we have that,

$$\begin{array}{cc}\hfill \mathbb{P}& \left[Z_{t_n}=i_n\mid Z_{t_{n-1}}=i_{n-1},\dots Z_{t_1}=i_1,Z_{t_0}=i_0\right]=\hfill \\ \hfill & =\mathbb{P}\left[Z_{t_n}=i_n\mid Z_{t_{n-1}}=i_{n-1}\right].\hfill \end{array}$$

We observe that, by force of the Markov property in Definition 1, the law of a continuous time Markov chain depends only on the following transition probabilities. Let $\mathit{I}$ be the identity matrix with dimension $\#\mathcal{I}$, the Kronecker’s delta, given by:

$${\delta }_i^j=\left\{\begin{array}{cc}0\hfill & i\ne j\hfill \\ 1\hfill & i=j.\hfill \end{array}\right.$$

Definition 2

(Transition probabilities). Let $\mathcal{I}:=\{1,2,\cdots ,m\}$, the finite set associated with $\mathbf{\Theta }=\{{\mathit{\theta }}_1,{\mathit{\theta }}_2,\dots ,{\mathit{\theta }}_m\}$, be the state space of ${\left(Z_t\right)}_{t\ge 0}$, the continuous time Markov chain. The transition probabilities are defined by the characteristic property,

$$\forall i,j\in \mathcal{I},s<t,p(s,i,t,j)=\mathbb{P}\left[Z_t=j\mid Z_s=i\right]andp(t,i,t,j)={\delta }_i^j.$$

Let $\mathcal{L}\left({\mathbb{R}}^{\#\mathcal{I}}\right)$ be the space of the square matrices with coefficients in $\mathbb{R}$. The transition probability matrix function$\mathit{P}:{\mathbb{R}}_{+}\times {\mathbb{R}}_{+}\mapsto \mathcal{L}\left({\mathbb{R}}^{\#\mathcal{I}}\right)$ is defined by,

$$\forall i,j\in \mathcal{I},s<t,\mathit{P}(s,t)={\left[p(s,i,t,j)\right]}_{i,j\in \mathcal{I}} and \mathit{P}(t,t)=\mathit{I}.$$

(1)

The transition probabilities of Markov processes in general satisfy a very important functional equation that results from the Markov property.

Theorem 1

(Chapman–Kolmogorov equations). Consider a nonhomogeneous continuous time Markov chain as given in Definition 1. Let $\mathit{P}$ be its transition probability matrix function as given in Definition 2. We then have:

$$\forall s,u,t,0\le s<u<t,\mathit{P}(s,t)=\mathit{P}(s,u)\mathit{P}(u,t).$$

(2)

The concept of a transition density is essential to the description of conditions allowing for the structural representation of continuous time Markov chains.

Definition 3

(Transition intensities). Let $\mathcal{L}\left({\mathbb{R}}^{\#\mathcal{I}}\right)$ be the space of square matrices with coefficients in $\mathbb{R}$. A function $\mathit{Q}:\mathbb{R}\mapsto \mathcal{L}\left({\mathbb{R}}^{\#\mathcal{I}}\right)$ denoted by:

$$\mathit{Q}\left(t\right)={\left[q(t,i,j)\right]}_{i,j\in \mathcal{I}},$$

is a transition intensity if the following characteristic set of properties are verified; that is, for almost all $t\ge 0$,

(i) 

$\forall i\in \mathcal{I},t\ge 0,q(t,i,i)\le 0$;

(ii) 

$\forall i\in \mathcal{I},t\ge 0,q(t,i,j)-q(t,i,i)\ge 0$;

(iii) 

$\forall i\in \mathcal{I}{\sum }_{j\in \mathcal{I}}q(t,i,j)=0$.

Theorem 2

(Backward and Forward Kolmogorov equations). Suppose that the matrix transition probability $\mathit{P}(s,t)$ is continuous at s, that is:

$$\underset{t\downarrow 0}{lim}\mathit{P}(0,t)=\mathit{I} and \underset{t\downarrow s}{lim}\mathit{P}(s,t)=\underset{t\uparrow s}{lim}\mathit{P}(t,s)=\mathit{I}.$$

(3)

If there exists $\mathit{Q}$ such that:

$$\begin{array}{cc}\hfill \mathit{Q}\left(t\right)& =\underset{k+h\to 0_{+},k\equiv 0\vee h\equiv 0}{lim}{\displaystyle \frac{\mathit{P}(t-k,t+h)-\mathit{I}}{k+h}}=\underset{h\downarrow 0,h>0}{lim}{\displaystyle \frac{\mathit{P}(t,t+h)-\mathit{I}}{h}}=\hfill \\ \hfill & =\underset{k\downarrow 0,k>0}{lim}{\displaystyle \frac{\mathit{P}(t-k,t)-\mathit{I}}{k}},\hfill \end{array}$$

(4)

then, we have the backward Kolmogorov (matrix) equation:

$${\displaystyle \frac{\partial }{\partial s}}\mathit{P}(s,t)=-\mathit{Q}\left(s\right)\mathit{P}(s,t),\mathit{P}(s,s)=\mathit{I},$$

(5)

and the forward Kolmogorov (matrix) equation:

$${\displaystyle \frac{\partial }{\partial t}}\mathit{P}(s,t)=\mathit{P}(s,t)\mathit{Q}\left(s\right),\mathit{P}(t,t)=\mathit{I}.$$

(6)

Remark 1.

The general theory of Markov processes shows that the condition that $\mathit{P}(s,t)$ is continuous in both s and t is sufficient to ensure the existence of the matrix intensities $\mathit{Q}$ given in Formula (4) (see [26] (p. 232)). The determination of the most general conditions for the existence and unicity of solutions of the Kolmogorov equations is an active subject of research with remarkable advances in recent years (see the series of works [29,30,31]).

Corollary 1

(Hostinsky’s representation). Let $\mathit{Q}$ be a transition intensity and the corresponding transition probability matrix $\mathit{P}$ given by Formula (5) or Formula (6). If, for a function $k:[0,+\infty [\mapsto [0,+\infty [$ the following condition is verified:

$$\forall i\in \mathcal{I},u\in [0,+\infty [,-q(u,i,i)\le k\left(u\right)and\forall s,t\ge 0{\int }_s^{t}k\left(u\right)du<+\infty ,$$

(7)

we then have,

$$\begin{array}{cc}\hfill & \mathit{P}(s,t)=\hfill \\ \hfill & =\mathit{I}+{\int }_s^t\mathit{Q}\left(u\right)du+\sum _{n\ge 2}{\int }_s^{t}d{s}_1{\int }_{s_1}^{t}d{s}_2\cdots {\int }_{s_{n-1}}^t\mathit{Q}\left(s_1\right)\mathit{Q}\left(s_2\right)\dots \mathit{Q}\left(s_n\right)d{s}_n,\hfill \end{array}$$

(8)

and,

$$\begin{array}{cc}\hfill & \mathit{P}(s,t)=\hfill \\ \hfill & =\mathit{I}+{\int }_s^t\mathit{Q}\left(u\right)du+\sum _{n\ge 2}{\int }_s^{t}d{t}_1{\int }_s^{t_1}d{t}_2\cdots {\int }_s^{t_{n-1}}\mathit{Q}\left(t_1\right)\mathit{Q}\left(t_2\right)\dots \mathit{Q}\left(t_n\right)d{t}_n,\hfill \end{array}$$

(9)

with the series being absolutely and uniformly convergent for any $s,t$ in a compact interval of $[0,+\infty [$.

Note that Formula (9) is, in fact, an alternative representation of the product integral (see [32] (p. 25)), which is useful in the estimation theory of Markov chains as referred to in Section 5.

Under the condition stated in Formula (7), we have a structural representation of the continuous time Markov chain process ${\left(Z_t\right)}_{t\ge 0}$ (see [26] (p. 229) and also [25] (pp. 279–281)).

Theorem 3

(A discrete time representation of a continuous time Markov chain). Let the intensities satisfy the condition given by Formula (7) in Corollary 1. Then, there exists a sequence of stopping times ${\left({\tau }_n\right)}_{n\ge 1}$—the times of the jumps of the Markov chain—such that with the sequence ${\left(Y_n\right)}_{n\ge 1}$ defined by $Y_n=Z_{{\tau }_n}$, the process defined by:

$$Z_t=\sum _{n=0}^{+\infty }{Y}_{n}1{\mathrm{I}}_{[{\tau }_n,{\tau }_{n+1}[}\left(t\right)=\sum _{n=0}^{+\infty }{Z}_{{\tau }_n}1{\mathrm{I}}_{[{\tau }_n,{\tau }_{n+1}[}\left(t\right)$$

(10)

is a continuous time Markov chain with transition probabilities $\mathit{P}$ given by Definition 2 and transition intensities $\mathit{Q}$ given by Definition 3 and Theorem 2. Moreover, ${\left(Y_n\right)}_{n\ge 1}$ is a Markov process.

Theorem 3 allows the definition of the imbedded Markov chain associated to a continuous time Markov chain ${\left(Z_t\right)}_{t\ge 0}$, that is, a Markov chain process with intensity matrix ${\mathit{Q}}^{\mathit{\theta }}={\left[q{(i,j)}^{\mathit{\theta }}\right]}_{i,j\in \mathcal{I}}$. We state next the definition of the imbedded Markov chain associated to a continuous time Markov chain. In order to recover the notation of [33] (p. 37) (see also [34] (p. 267)), we consider the sequence of stopping times ${\left({\tau }_n\right)}_{n\ge 1}$ given by the jump times and then define the sequence ${\left({\rho }_n\right)}_{n\ge 1}$ staying times, that is, the times in-between jumps by:

$$\left\{\begin{array}{cc}{\rho }_1:={\tau }_1\hfill & \\ {\rho }_n:={\tau }_n-{\tau }_{n-1}\hfill & n\ge 2,\hfill \end{array}\right.$$

(11)

in such a way that ${\tau }_n={\rho }_1+\cdots +{\rho }_n$.

Definition 4

(The imbedded Markov chain). Given ${\left(Z_t\right)}_{t\ge 0}$ the continuous Markov chain process with intensity matrix ${\mathit{Q}}^{\mathit{\theta }}={\left[q{(i,j)}^{\mathit{\theta }}\right]}_{i,j\in \mathcal{I}}$ such that $q{(i,j)}^{\mathit{\theta }}>0$ and with the Markov process ${\left(Y_n\right)}_{n\ge 1}$ defined in Theorem 3, the discrete time Markov process ${(Y_n,{\rho }_n)}_{n\ge 1}$ with state space given by the Cartesian product of the state space of ${\left(Z_t\right)}_{t\ge 0}$—and ${\left(Y_n\right)}_{n\ge 1}$—by $[0,+\infty [$, is the imbedded Markov chain.

The imbedded Markov chain will be used in a result in Section 3 below.

We should also mention [35,36] for the important concept of—what should be called—the reverse imbedding problem; that is, given a discrete time Markov chain, the determination of a continuous time Markov chain such that the given discrete time Markov chain is the imbedded Markov chain in the continuous one. In [37], a discussion concerning the bridging between rigorous mathematical results on the existence of generators—tantamount to a solution to the existence of an imbedded Markov chain given some data—to its computational implementation is presented. A study on different forms of approximating continuous time Markov chains by discrete time ones is given in [38] using different Skorokhod topologies adapted to different embeddings: step functions, linear interpolation, multistep, etc.

3. Homogeneous Markov Chains

The statistical estimation of the parameters of continuous time homogeneous Markov chains is a well-studied subject for which there exist very important references. Let us refer first to the two highly praised, very important works [33,39] and a more recent one with important applications in mind [16]. The book [33] presents the maximum likelihood method for general Markov processes, first, in discrete time, and secondly, in continuous time, for processes that are not diffusions, by considering the imbedded Markov chain process in discrete time—see Definition 4—allowing the exploitation of the results previously obtained in the discrete time set up.

The article [39] contains an extensive list of references and a summary of the main ideas on the subject, which are clearly developed in the book by the same author. Noticing that the estimation procedures apply to irreducible Markov chains with only recurrent and non-null states, we select from [33] (pp. 45–51) the following fundamental result, Theorem 8.1, a result included in a section devoted to homogeneous Markov chains with finite state space. We note that for the notions of recurrence and transience in homogeneous Markov chains, an introductory reference is, for instance [17] (pp. 79–85).

Theorem 4

(Asymptotics of MLE estimators when the length of the one observed trajectory goes to infinity). Suppose that ${\left(Z_t\right)}_{t\ge 0}$ the Markov chain process with intensity matrix ${\mathit{Q}}^{\mathit{\theta }}={\left[q{(i,j)}^{\mathit{\theta }}\right]}_{i,j\in \mathcal{I}}$ and parameter $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_u,\dots ,{\theta }_d)\in \mathbf{\Theta }\subset {\mathbb{R}}^d$ satisfies the following hypothesis.

1. 

The trajectories of ${\left(Z_t\right)}_{t\ge 0}$ are right continuous step functions; we have that ${\sum }_{j\in \mathcal{J}}q{(i,j)}^{\mathit{\theta }}>0$; the transition probabilities satisfy Formula (3) and the intensities satisfy Formula (4) stated in Theorem 2.

2. 

The set $D:=\left\{(i,j):q{(i,j)}^{\mathit{\theta }}>0\right\}$ does not depend on θ; we have that $q{(i,j)}^{\mathit{\theta }}\in C^3\left(\mathbf{\Theta }\right)$ and the matrix,

$${\left[{\displaystyle \frac{\partial }{\partial {\theta }_u}}q{(i,j)}^{\mathit{\theta }}\right]}_{(i,j)\in D,1\le u\le d},$$

has rank d for all $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_u,\dots ,{\theta }_d)\in \mathbf{\Theta }$.

3. 

For each $\mathit{\theta }\in \mathbf{\Theta }$, the Markov chain ${\left(Y_n\right)}_{n\ge 1}$ has only one ergodic set and there are no recurrent states.

Then, with $t_{ij}$ denoting the number of direct jumps from i to j in the sample function observed until time t, with ${\gamma }_i$ denoting the total amount of time the chain is in state i until time t, there exists a consistent solution $\widehat{\mathit{\theta }}\in \mathbf{\Theta }$ of the likelihood equations given by:

$${\displaystyle \frac{\partial }{\partial {\theta }_u}}{L}_t\left(\mathit{\theta }\right)=\sum \left[t_{ij}{\displaystyle \frac{\partial }{\partial {\theta }_u}}ln\left(q{(i,j)}^{\mathit{\theta }}\right)-{\gamma }_i{\displaystyle \frac{\partial }{\partial {\theta }_u}}q{(i,j)}^{\mathit{\theta }}\right]=0.$$

Moreover, with ${\mathit{\theta }}_0\in \mathbf{\Theta }$ being the true parameter value and $\nu \left(t\right):=max\{k:{\tau }_k<t\}$ being the number of jumps that occur until time t, we have that:

(i) 

With the limit being taken in probability, ${lim}_{t\to +\infty }\sqrt{\nu \left(t\right)}\left({\mathit{\theta }}_0-\widehat{\mathit{\theta }}\right)$

(ii) 

With the limit being taken in probability, ${lim}_{t\to +\infty }\sqrt{\nu \left(t\right)}\left(q{(i,j)}^{{\mathit{\theta }}_0}-q{(i,j)}^{\widehat{\mathit{\theta }}}\right)$

(iii) 

With the limit being taken in law (denoted by $\mathcal{L}$):

$$\begin{array}{cc}\hfill & 2\left(\underset{\mathit{\theta }\in \mathbf{\Theta }}{max}{L}_t\left(\mathit{\theta }\right)-L_t\left({\mathit{\theta }}_0\right)\right)=\hfill \\ \hfill & =\sum \left[t_{ij}{\displaystyle \frac{\partial }{\partial {\theta }_u}}ln\left({\displaystyle \frac{q{(i,j)}^{\widehat{\mathit{\theta }}}}{q{(i,j)}^{{\mathit{\theta }}_0}}}\right)-{\gamma }_i{\displaystyle \frac{\partial }{\partial {\theta }_u}}\left(q{(i,j)}^{\widehat{\mathit{\theta }}}-q{(i,j)}^{{\mathit{\theta }}_0}\right)\right]\underset{t\to +\infty }{\overset{\mathcal{L}}{⟶}}{\chi }_r^2.\hfill \end{array}$$

(12)

We observe that if the Markov chain has a sole trap or absorbing state, other estimation procedures have to be devised to take advantage of data composed of a significant number of trajectories all ending in the trap state after a finite number of jumps.

An extension of the regularity conditions needed for the more general estimation results in [33] is given in [40]. A study of the rate of convergence of the maximum likelihood estimator based on the observation of a trajectory, using the Gärtner–Ellis theorem for random processes, is given in [41].

An interesting follow-up appeared in [42] dealing with the situation where it may happen that when estimating the parameters of a stationary Markov chain, the chosen model is incorrect, in the sense that the probability law governing the evolution is not an element of the parametric family stipulated. Interesting convergence results are presented.

The article [43] covers the statistical estimation for Markov jump processes with finite state space in the context of observations at discrete time points—also termed partially observed Markov chains—by studying conditions for the existence and unicity of the maximum likelihood estimators. Examples and simulation studies are presented to illustrate the methodology.

The four works [44,45,46,47] deal with an important subject that may be summarily described as perturbation bounds for homogeneous Markov chains; the results on this subject—see Theorem 5 for an example in the nonhomogeneous case—have important applications, for instance, allowing for control of estimation procedures or for relative indifference in the choice of the intensity matrices.

4. Nonhomogeneous Markov Chains

In this section, we consider nonhomogeneous continuous time Markov chains with a finite state space. Whenever a parameter estimation, or calibration, problem for a model of observations is considered by means of nonhomogeneous continuous time Markov chains, we have to take into preliminary consideration three important facts.

• Generally, we will try to determine the intensities (see Definition 3) that will generate the transition probabilities (see Definition 2), following the classical and very successful modelling approach, tantamount to finding a model of differential equations for the phenomena, in this case, the Kolmogorov equations. In actuarial mathematics, this methodology has been referred to as the Transition Intensity Approach (TIA) (see [48] and…).
• The intensities may have arbitrary functional forms—linear, piecewise constant (see [49] (p. 42)), polynomial, exponential of Gompertz, Makeham or Gompertz–Makeham type (see [49] (pp. 24–25) or [50] (pp. 205–206)) used in health insurance and long-term care modelling—where the functional forms are dependent on parameters that must be fitted to the data either by estimation or calibration.
• The characteristics of the available data relating to observations of phenomena, together with the general—and most often, the qualitative properties expected from the model—are determining for both the choice of the modelling approach and, in the case of modelling by the Kolmogorov equations—the choice of the functional form of the intensities and of the relevant parameters to be estimated or calibrated.

The constructive definition of a nonhomogeneous continuous time Markov chain follows [27] (pp. 242–246) and in a more condensed version [28] (pp. 349–351). Whenever we are faced with a specific estimation-calibration problem, a first decision that conditions all the subsequent decisions is the functional form of the intensities. If there are no imperative reasons justifying a preference of a functional form over others, the choice of the functional form is somehow indifferent by reason of the following. Two sets of intensities may be seen to be perturbations of one another and so results on how the perturbed intensities reflect the corresponding transition probabilities—such as the ones exposed in [51]—are extremely useful. The following result is an example, showing that if a distance between the correspondent probability matrices can be controlled by a distance of the intensity matrices, the choice of the functional form is indifferent.

Theorem 5

(Continuous dependence of the transition probabilities on the transition intensities). Let $∥·∥$ be a matrix norm and let ${\mathbf{Q}}_1(t,\mathit{\theta })$ and ${\mathbf{Q}}_2(t,\mathit{\theta })$ be two matrices of intensities norm-bounded by $M>0$ in $[0,T]$. Define:

$$ϵ\left({\mathbf{Q}}_1,{\mathbf{Q}}_2\right):=\underset{t\in [0,T],\mathit{\theta }\in \mathbf{\Theta }}{sup}∥{\mathbf{Q}}_1(t,\mathit{\theta })-{\mathbf{Q}}_2(t,\mathit{\theta })∥.$$

(13)

Then, we have that,

$$\underset{t\in [x,T],\mathit{\theta }\in \mathbf{\Theta }}{sup}∥{\mathbf{P}}_1(x,t,\mathit{\theta })-{\mathbf{P}}_2(x,t,\mathit{\theta })∥\le ϵ\left({\mathbf{Q}}_1,{\mathbf{Q}}_2\right){\displaystyle \frac{e^{M\left|T-x\right|}-1}{M}},$$

(14)

where ${\mathbf{P}}_1(x,u,\mathit{\theta })$ and ${\mathbf{P}}_2(x,u,\mathit{\theta })$ are the solutions of the Kolmogorov equations—given either in Formula (5) or in Formula (6)—with the matrices of intensities, respectively, ${\mathbf{Q}}_1(t,\mathit{\theta })$ and ${\mathbf{Q}}_2(t,\mathit{\theta })$.

For a proof and an application of this result to a practical problem, see [52].

Remark 2

(Consequences and application of Theorem 5). One of the consequences of this result is that it gives an accrued freedom in the choice of the functional form of the transition intensities, since if two intensity matrices are close to one another—in the supremum norm in Formula (13)—then the correspondent probability function matrices will be close to one another in the same norm. The $x,T$ dependent factor in Formula (14) is not a limitation since this constant, being equivalent to $\left|T-x\right|$ for $T\approx x$, we can apply the result to a decomposition of the time interval $[0,T]$ in a finite number of intervals in such a way that the distance of the probability function matrices is as small as is wanted.

In an alternative comparison result [53], the authors consider the asymptotic behaviour of the log-likelihood ratio as a measure of the deviation between a sequence of real valued random variables and a continuous-state non-homogeneous Markov chain. A class of strong deviation theorems for bivariate functions of the sequence of real valued random variables which are associated with the continuous-state nonhomogeneous Markov chains is established using the super-martingale limit theorem.

4.1. Calibration of Intensities of a Nonhomogeneous Markov Chain

In this section, we describe a methodology termed the calibration of intensities introduced for the purpose of a study in long-term care in [54], a methodology that was further detailed in [55] in a formulation that we reproduce next. Due to the type of applications in mind, the Markov chains under study are supposed to have a finite state space with only one cemetary state.

Let us consider the result presented in Theorem 6. Suppose that we observe a phenomenon in discrete time, for instance, the periodical—say, monthly—recordings of events that may occur in any day of the month. If the state space of the observations is finite or denumerable, the data may allow the phenomenon to be modelled by a discrete time Markov chain, although we know that a more adequate model would be a continuous time Markov chain, possibly even nonhomogeneous. The calibration procedure proposed allows fitting the parameters of a given functional form of intensities by minimising a discrepancy between the transition probabilities of the discrete time model and the transition probabilities of the continuous time model.

Theorem 6

(Calibration of intensities of a continuous chain by discrete observations). Let, for $1\le n\le N$, ${\mathbf{R}}^{{\tau }_n}={\left[r_{ij}^{\left({\tau }_n\right)}\right]}_{i,j=1,\dots ,r}$ be the generic element of a sequence of numerical transition matrices taken at a sequence of increasing dates ${\left({\tau }_n\right)}_{1\le n\le N}$. Consider a set of intensities $\mathbf{Q}(u,\mathit{\lambda })={\left[q(u,i,j,\mathit{\lambda })\right]}_{i,j=1,\dots ,r}$—with $\mathit{\lambda }\in \mathrm{\Lambda }\subset {\mathbb{R}}^d$ being a parameter and Λ being a compact set—satisfying the following conditions:

1. 

For every fixed λ, the functions $q(u,i,j,\mathit{\lambda })$ are measurable as functions of u.

2. 

For every fixed u, the functions $q(u,i,j,\mathit{\lambda })$ are continuous as functions of λ.

3. 

There exists a locally integrable function $M:[0,+\infty [\mapsto [0,+\infty [$, such that for all $\mathit{\lambda }\in \mathrm{\Lambda }$, $i\in \mathcal{I}$, $u\in [0,+\infty [$ and $0\le s\le t$, the following conditions are verified:

$$-q(u,i,i,\mathit{\lambda })\le M\left(u\right)and{\int }_s^{t}M\left(u\right)du<+\infty .$$

(15)

Consider the following.

1. 

We know that there exists $\mathit{P}(s,t,\mathit{\lambda })={\left[p(s,i,t,j,\mathit{\lambda })\right]}_{i,j=1,\dots ,r}$ a probability transition matrix, with entries absolutely continuous in s and t, such that the conditions in Definition 2, the Chapman–Kolmogorov equations in Theorems 1 and 2 are verified.

2. 

For each fixed $s_0$, we can consider the loss function given by,

$$\mathcal{O}(s_0,\mathit{\lambda }):=\sum _{i,j=1,\cdots ,r}\sum _{n=1}^N{\left(p(s_0,i,{\tau }_n,j,\mathit{\lambda })-r_{ij}^{\left({\tau }_n\right)}\right)}^2.$$

(16)

Then, we have that for the optimisation problem, ${inf}_{\mathit{\lambda }\in \mathrm{\Lambda }}\mathcal{O}(s_0,\mathit{\lambda })$, there exists ${\mathit{\lambda }}_0\in \mathrm{\Lambda }$ such that,

$$\mathcal{O}(s_0,{\mathit{\lambda }}_0)=\underset{\mathit{\lambda }\in \mathrm{\Lambda }}{min}\mathcal{O}(s_0,\mathit{\lambda }),$$

(17)

the unique minimum being attained at possibly several distinct points ${\mathit{\lambda }}_0\in \mathrm{\Lambda }$.

Proof. 

See [55] for a proof. □

Remark 3

(On possible applications of Theorem 6). In [54], we applied the result to a sequence of transition matrices ${\left[(r_{ij}^{\left(n\right)}\right]}_{i,j=1,\dots ,r}{)}_{n\ge 1}$ derived from a single observed discrete time transition matrix $\mathit{P}$ and the sequence of powers ${\mathit{P}}^n$, for $n\ge 2$ corresponding to the transition in n steps. We observe that the solution of the optimisation problem in Formula (16) may require heavy computational power since it depends on the numerical solution of the Kolmogorov equations at each step, and the dimension of the optimisation space depends on the number of states and on the number of the intensity parameters to calibrate.

Remark 4

(An open problem related to the calibration procedure in Theorem 6). Let us suppose that the sequence of transition matrices ${\mathbf{R}}^{{\tau }_n}={\left[r_{ij}^{\left({\tau }_n\right)}\right]}_{i,j=1,\dots ,r}$, taken at a sequence of increasing dates ${\left({\tau }_n\right)}_{1\le n\le N}$, in the statement of Theorem 6 corresponds to a sequence of observations of an unknown continuous time nonhomogeneous Markov chain transition matrix. Let ${\mathit{\lambda }}_0^N\in \mathrm{\Lambda }$ be one solution of the optimisation problem given by Formula (17). An open relevant problem is to determine the conditions under which the sequence of transitions matrices given by $\mathit{P}(s,t,{\mathit{\lambda }}_0^N)={\left[p(s,i,t,j,{\mathit{\lambda }}_0^N)\right]}_{i,j=1,\dots ,r}$ for $N\ge 1$, converges to the unknown continuous time nonhomogeneous Markov chain transition matrix.

4.2. Estimation of Nonhomogeneous Markov Chains

A useful approach to estimating the parameters of the non-constant intensities widely used in actuarial mathematics—in the modelling of health insurance and long-term care problems using Markov chains in a broad methodology termed multiple state models—essentially consists of two steps.

• Initially, the intensities are supposed to be constant over a partition of the time interval under study in intervals; this interval may cover one year in the case where the age of the subject matters or cover a period of more than a year in the case where the variation in intensities over the period is assumed to be negligible. The estimation is performed in each interval using the effective methods of maximum likelihood for homogeneous continuous time Markov chains (see Section 3); for this first step, see [56] (pp. 683–690) for details and examples of applications.
• Using additional data of the population for each specific age—such as gender, smoking habits, weight, exercise habits—perform a fitting of the intensity parameters by means of generalised linear models; this complementary methodology is termed graduation of the intensities. See [57] (pp. 126–128) and [58,59] for detailed explanations of the graduation and [60,61] for examples of applications.

Since its inception, this approach has been in use, with variants dealing with more specific problems, in several works; one such example is [62]. In [63], the focus is on the estimation of the annual transition probabilities incorporating external data to the time and jump counts.

In the rest of this section, we describe a methodology proposed and applied in [52] for the estimation of the parameters of the non-constant intensities of arbitrary functional form. The Markov chains under study are assumed to have a finite state space with an absorbing state. This methodology is based on a known algorithmic constructive procedure of nonhomogeneous Markov chains (see [27] (p. 266) and also [28] (pp. 349–351)). The constructive methodology allowing to recover a homogeneous Markov chain, with denumerable state space, is presented in [64] (pp. 473–475). Consider a transition intensity matrix $\mathit{Q}(t,\mathbf{\theta })$ with $\mathit{\theta }\in \mathbf{\Theta }\subset {\mathbb{R}}^d$ a parameter that we intend to estimate.

Definition 5

(Constructive definition of a Markov chain). Consider the entries of a transition intensity matrix,

$$\mathit{Q}(t,\mathit{\theta })={\left[{\mu }_{ij}^{\mathit{\theta }}\left(t\right)\right]}_{i,j=1,\cdots ,d},$$

and define:

$$p^{★}(t,i,j)=\left\{\begin{array}{cc}{\displaystyle \frac{1-{\delta }_i^j}{-{\mu }_{ii}^{\mathit{\theta }}\left(t\right)}}{\mu }_{ij}^{\mathit{\theta }}\left(t\right)\hfill & {\mu }_{ii}^{\mathit{\theta }}\left(t\right)\ne 0\hfill \\ {\delta }_i^j\hfill & {\mu }_{ii}^{\mathit{\theta }}\left(t\right)=0,\hfill \end{array}\right.$$

(18)

where ${\delta }_i^j$ is the Kronecker’s delta. Suppose that $X_0=i$, according to some initial distribution on $\{1,2,\dots ,d\}$.

1. 

We define by induction the jump sequence of the stopping times ${\left({\tau }_n\right)}_{n\ge 0}$ as follows; firstly, we pose ${\tau }_0\equiv 0$.

2. 

Then, the stopping time  ${\tau }_1$, that is, the sojourn time in state i and also the time of the first jump, has an exponential distribution function given by:

$$F_{{\tau }_1}\left(t\right)=\mathbb{P}\left[{\tau }_1\le t\right]=1-exp\left({\int }_0^t{\mu }_{ii}^{\mathit{\theta }}\left(t\right)du\right).$$

(19)

The exponential distribution for  ${\tau }_1$ is a consequence of a general result on the distribution of the sojourn times of a continuous time Markov chain (see Theorem 2.3.15 in [26] (p. 221)).

3. 

Given that the process is in state i, it now may jump to state j at time ${\tau }_1=s_1$ with probability $p^{★}(t,i,j)$ defined in Formula (5), that is,

$$\mathbb{P}\left[X_{s_1}=j\mid {\tau }_1=s_1,X_0=i\right]=p^{★}(s_1,i,j),$$

(20)

and so we have that $X_t=i$ for $0\equiv {\tau }_0\le t<{\tau }_1$.

4. 

Given that ${\tau }_1=s_1$ and $X_{s_1}=j$, we have that ${\tau }_2$, the stopping time of the second jump, has, again, the exponential distribution function given by:

$$F_{{\tau }_2\mid {\tau }_1=s_1}\left(t\right)=\mathbb{P}\left[{\tau }_2\le t\mid {\tau }_1=s_1\right]=1-exp\left({\int }_0^t{\mu }_{jj}^{\mathit{\theta }}(u+s_1)du\right),$$

and we also have that,

$$\mathbb{P}\left[X_{s_2}=k\mid {\tau }_1=s_1,X_0=i,{\tau }_2=s_2,X_{s_1}=j\right]=p^{★}(s_1+s_2,j,k),$$

and, consequently, we have $X_t=j$ for ${\tau }_1\le t<{\tau }_2$.

5. 

We proceed inductively to define the stopping time ${\tau }_3$ and so on.

The following result—compare with Theorem 3—ensures that the preceding construction yields a nonhomogeneous Markov chain with the given intensity matrix.

Theorem 7

(A continuous time Markov chain with a prescribed intensity matrix function). Let the intensity matrix be norm-bounded by a Lebesgue integrable function in $[0,T]$. Then, given the times ${\left({\tau }_0\right)}_{n\ge 1}$, we have that with the sequence ${\left(Y_n\right)}_{n\ge 1}$ defined by $Y_n=X_{{\tau }_n}$, the process defined by:

$$X_t=\sum _{n=0}^{+\infty }{Y}_{n}1{\mathrm{I}}_{[{\tau }_n,{\tau }_{n+1}[}\left(t\right)=\sum _{n=0}^{+\infty }{X}_{{\tau }_n}1{\mathrm{I}}_{[{\tau }_n,{\tau }_{n+1}[}\left(t\right)$$

(21)

is a continuous time Markov chain with transition probability matrix $\mathit{P}$ and transition intensity matrix $\mathit{Q}$.

Proof. 

For a proof of this result in the general case of Markov continuous time Markov processes, see [26] (pp. 221–233). □

We now proceed to detail a methodology to identify the parameter $\mathit{\theta }\in \mathbf{\Theta }$ inspired by the constructive definition of the Markov chain in Definition 5. We assume that we have at our disposal a sample of trajectories of the continuous time nonhomogeneous Markov chain. See Figure 1 taken from [52] as an example, that is schematically a large number of lines—each line is a trajectory—each line giving, by this order, the initial state, the time until the first jump, the second attained state, the time until the second jump, the third attained state, and so on, until eventually the trajectory ends in the absorbing or trap state.

(i) 

Given a state i, we have to find a fitting for the distribution of the random times of $i\to i$ jumps. According to Formula (19), these times have an exponential distribution with density ${\mu }_{ii}^{\mathit{\theta }}\left(t\right)$.

(ii) 

For every other state j, by using $\mathbb{P}\left[X_{s_1}=j\mid {\tau }_1=s_1,X_0=i\right]$, possibly with an approximation, by Formula (20), we can obtain an approximation of $p^{★}(s_1,i,j)$.

(iii) 

By using Formula (18) and the approximation obtained for $p^{★}(s_1,i,j)$ we can obtain an approximation for ${\mu }_{ij}^{\mathit{\theta }}\left(t\right)$.

(iv) 

Finally, we fit an intensity of a chosen functional form to ${\mu }_{ij}^{\mathit{\theta }}\left(t\right)$.

We now describe in detail the procedure for applying the methodology just described.

• Recall that an observed trajectory has the following structure: (first state, time spent in state, second state, time spent in state, third state…). In a finite set of trajectories we have as real data, under the hypothesis of an unbounded time horizon for the observations of the trajectories, the maximum length for all trajectories is finite. We select all the trajectories of length greater than 3 that start at state $i=1$. If the next state is also $i=1$, the time spent in state—in this case, in state $i=1$—is the first part of the sample for obtaining ${\mu }_{11}^{\mathit{\theta }}\left(t\right)$. Select all the trajectories of length greater than 5 for which the second state is $i=1$; this set of trajectories already contains the previously considered set of trajectories, and so, if the third state is also $i=1$, the sum of the times spent in the first state and the time spent in the second state will be the second part of the sample for obtaining ${\mu }_{ii}^{\mathit{\theta }}\left(t\right)$. Repeat, successively, the procedure for all trajectories of length greater than 7, then of length greater than 9, of length greater than 11, until the maximum length of all the trajectories is attained to obtain the full sample for the intensity ${\mu }_{11}^{\mathit{\theta }}\left(t\right)$.
• Fit a smooth kernel distribution to the sample obtained for the intensity ${\mu }_{11}^{\mathit{\theta }}\left(t\right)$.
• Repeat the procedure used for getting the sample for ${\mu }_{11}^{\mathit{\theta }}\left(t\right)$, but this time selecting the transitions $1\to 2$, that is, the transitions from state $i=1$ to state $i=2$. Fit a smooth kernel distribution to this data.
• Now, we look for an estimate of $p^{★}(t,i,j)$ given by Formulas (18) and (20). For that, we will consider rounding the sojourn times—say to unity, in order to have enough observations—and then group all the observations of jumps from the first state according to this rounding. Consider then the observations towards state $i=2$. We will then have that:

  $$\begin{array}{cc}\hfill p^{★}(s_1,1,2)& =\mathbb{P}\left[X_{s_1}=2{\tau }_1=s_1,X_0=1\right]\approx \hfill \\ \hfill & \approx {\displaystyle \frac{\mathbb{P}\left[X_{s_1}=2,s_1-0.5\le {\tau }_1<s_1+0.5\right]}{\mathbb{P}\left[s_1-0.5\le {\tau }_1<s_1+0.5\right]}}=\hfill \\ \hfill & ={\displaystyle \frac{\mathbb{P}\left[s_1-0.5\le {\tau }_1<s_1+0.5X_{s_1}=2\right]·\mathbb{P}\left[X_{s_1}=2\right]}{\mathbb{P}\left[s_1-0.5\le {\tau }_1<s_1+0.5\right]}}\hfill \end{array}$$

  (22)

  and the most left-hand side of the formula will be estimable with the observations by using the smooth kernel distributions.
• Resorting to Formula (18), we can compute values for ${\mu }_{12}^{\mathit{\theta }}\left(t\right)$ and fit a piecewise linear density. That is, using again Formula (18), since ${\mu }_{11}^{\mathit{\theta }}\left(s_1\right)\ne 0$, we have that for an arbitrary time $t=s_1$:

  $${\mu }_{12}^{\mathit{\theta }}\left(s_1\right)\approx {\displaystyle \frac{-{\mu }_{11}^{\mathit{\theta }}\left(s_1\right)}{1-{\delta }_1^2}}{p}^{★}(s_1,1,2)=-{\mu }_{11}^{\mathit{\theta }}\left(s_1\right)·p^{★}(s_1,1,2).$$
• If we consider a set of values of ${\mu }_{12}^{\mathit{\theta }}\left(s_1\right),{\mu }_{12}^{\mathit{\theta }}\left(s_2\right),\dots ,{\mu }_{12}^{\mathit{\theta }}\left(s_k\right)$ and then fit the multidimensional parameter $\mathit{\theta }\in \mathbf{\Theta }$ to these values, using for that purpose a functional form previously selected.
• These procedures are to be repeated in order to obtain the parameters ${\mu }_{2j}^{\mathit{\theta }}$ for $j\ne 2$ and ${\mu }_{3j}^{\mathit{\theta }}$ for $j\ne 3$.
• The intensities ${\mu }_{jj}^{\mathit{\theta }}$ for $j=1,2,3$ are obtained in the usual way as a sum of the remaining intensities in line with index i of the intensity matrix.

Remark 5

(On several open problems related to the estimation methodology proposed). The methodology presented in this section provided results that can be appreciated in detail in [52]. More precisely, we generated data with intensities given by the Gompertz–Makeham functional form, and then, using the methodology just described above, we fitted piecewise linear intensities to the data; next, we computed the errors given by the $L^1$ distance between the integrated transition probabilities from the Gompertz–Makeham functional form and the transition probabilities from the fitted piecewise linear intensities, and we observed that the average error per year is always less than 1%, although with the possibility of one order of magnitude larger errors at some specific years.

We must stress that similarly to what was stated in Remark 4 about the calibration procedure, there are open problems with the estimation procedure stated above and, moreover, the problems are subtle in nature due to the fact that the intensities to be estimated—to wit, ${\mu }_{ij}^{\mathit{\theta }}\left(t\right)$—are functions we are dealing with for estimation in infinite dimensional spaces. The estimation problem may be reduced to a finite dimensional estimation problem as soon as some functional form—for instance, continuous piecewise linear functions—is assumed for the intensities, because then, there will remain only a finite number of degrees of freedom for each intensity—corresponding to the slopes of the linear pieces and the initial value. Under this perspective, the result in Theorem 5 shows that, in fact, the choice of a functional form for the intensities is somehow indifferent, since if two intensities matrices are close to one another, then the corresponding probability transition matrices will be close to one another. A relevant open problem is, then, to determine the most effective functional form for the intensities given an estimation procedure.

5. Comments on Some Additional References

Let us now review some other approaches to procedures of estimation for both homogeneous and nonhomogeneous Markov chains.

• In [65], using the formalism of product integration (see [32,66] for expositions of the concept and [67] for an application to the integrated probability transition functions with fixed points of discontinuity), the authors propose an estimator for the transition probabilities of a nonhomogeneous Markov chain with a finite number of states in the presence of censoring, in particular, when the processes are only observed part of the time. As a first step, an estimator for the integrated transition intensities is obtained using a multivariate counting process, and, in a second step, the product integration is used to obtain the desired estimator for the transition probabilities having useful asymptotic properties. An application of this methodology, using firstly kernel smoothing to estimate the transition intensities, is presented in [68]. An application of the product limit estimator to credit migration matrices is given in [4].
• The work [69] is an extension of the previously referred to Aallen’s work for which the state space has an arbitrary but finite number of both transient and absorbing states. A non-parametric product limit estimator is introduced and shown to be uniformly consistent.
• The Ph.D. thesis [70] develops a method for estimating the parameters of a nonhomogeneous continuous time Markov chain discretely observed by Poisson sampling by using the Dynkin martingale. The estimators are proven to be strongly consistent and asymptotically normal. A simulation study evaluates the performance of the model against the maximum likelihood estimators with continuously observed trajectories.
• The authors of [71] propose a kind of moment estimation procedure to estimate the parameters in the infinitesimal generator of a time-homogeneous continuous-time Markov process: a type of process that may serve as a model of, for instance, high frequency financial data. Several properties of the estimation procedure are proved, such as strong consistency, asymptotic normality, and a characterisation of standard errors.
• The article [72] also develops a calibration procedure by seeking, by means of solving a quadratic objective function minimisation problem subjected to linear constraints, a generator—that is an intensity transition matrix—of a continuous time Markov chain, having a probability transition matrix with a spectrum as close as possible to the spectrum of an estimated discrete time Markov chain transition probability matrix. In the case the chain is embeddable, the procedure returns the generator; if not, the procedure returns a best approximation of the generator, in the sense of the minimisation problem solved. Besides testing the procedure with synthetic data, there is an application to data coming from a time series generated from a model of large scale atmospheric flow.
• For the content of the work [73], the authors say: “In the present work, we focus on Bayesian inference for the partially observed CTMC without using latent variables; we use the likelihood function directly by evaluation of matrix exponentials and perform posterior inference via a Metropolis–Hastings approach, where the generator matrices are fully specified and not constrained”.
• The work [74] is a broad reference on nonhomogeneous Markov chains mainly devoted to the asymptotic behaviour of these chains encompassing some more focused works, such as [75].

In

Table 1. Summary table of the main characteristics of some of the cited works.

H or nH (a)	Articles	F/iF Dt/Ct (b)	TrI/Pr (c)	Est/Cal (d)	App (e)	Cmp/Sim (f)	Data (g)
nH	[65]	F & Ct	TrI	Cst & As	–	–	FtjCo
	[67]	F & Ct	TrI	MLE	–	–	FtjCo
	[69]	F & Dt	TrI	MLE & Cst & As	–	Cmp	TrjsDo
	[68]	F & Ct	TrI & Pr	K & K	H	Cmp	FtjCo
	[70]	iF	TrI	PSam & Cst & As	–	Sim	TrjsDo
	[55]	F & Ct	TrI	Cal	LTC	–	–
	[75]	F & Ct	TrI	MLE	H	–	TrjsDo
	[52]	F & Ct	TrI	nP & Cal	–	Sim	TrjsDo
H	[39]	F & iF & Dt & Ct	TrI & Pr	MLE & Cst & As	–	–	–
	[33]	F & iF & Dt & Ct	TrI & Pr	MLE & Cst & As	–	–	–
	[40]	iF & Dt	Pr	MLE & Cst & As	–	–	TrjsDo
	[42]	iF & Dt	Pr	MLE & Cst & As	–	–	TrjsDo
	[71]	F& iF& Ct & Dt	TrI & Df	ME & Cst & As	–	–	SmpDo
	[73]	F & Ct	TrI	MLE & Bay	–	Sim	TrjsDo
	[41]	iF & Dt	Pr	MLE & MD	–	–	TrjsDo

^(a) H—homogeneous Markov chain; nH—nonhomogeneous Markov chain. ^(b) F—finite state space; iF—infinite state space; Dt—discrete time; Ct—continuous time. ^(c) TrI—estimation of transition intensities, i.e., generator; Pr—estimation of probability densities; Df—estimation of parameters of a diffusion. ^(d) Est—maximum likelihood estimation (MLE), moment estimators (ME), statistical inference on estimators: consistency (Cst), asymptotic properties (As), kernel estimation (K), other non-parametric (nP), Poisson sampling (PSam), moment estimators (MEs); Bay—Bayesian inference; moderate deviations (MD); Cal—calibration of intensities. ^(e) App: with applications, to credit (Crd), to high frequency data (HFD), long-term care (LTC); other health (H). ^(f) Cmp—Comparison studies with other methods; Sim—simulation studies to assess methods. ^(g) Data type: TrjsDo—set of trajectories discretely observed; FtjCo—set of full trajectories observed continuously; SmpDo—set of trajectories sampled at random times.

, we present a comparison summary of the main characteristics of some of the works in statistical estimation, and or, calibration discussed in the text.

6. Conclusions

In this work, we provide a brief review of several important techniques used to fit Markov chain models to phenomena that comprise an evolution of a system taking values in a finite state space; special emphasis was placed on the nonhomogeneous Markov chains models. This review is intended to provide a starting ground for further much needed investigations on the subject. It is manifest that the subject of statistical estimation for nonhomogeneous Markov chains is bound to continue to receive widespread attention due to the range of domains where it finds modelling applications. It is also clear—from this work, in particular—that the methods and procedures designed for the estimation or calibration of parameters of a Markov chain must suit the type of data available; we should highlight two main types of data: either a whole trajectory observed continuously and/or on a special sequence of times, or, in the case of chains having an absorbing state, a sufficiently large set of trajectories of the chain. It is to be expected that the intensive use of computational tools will allow an even wider development of estimation and/or calibration methods.