*2.1. Markov Chains*

Suppose that **X** = {*Xn*, *n* ∈ <sup>Z</sup><sup>+</sup>}, Z<sup>+</sup> := {0, 1, 2, ...} are random variables defined over (<sup>Ω</sup>, F, B(Ω)), and assume that **X** is a Markov chain with transition probability *<sup>P</sup>*(*<sup>x</sup>*, *<sup>A</sup>*), *x* ∈ Ω, *A* ⊂ Ω. Then we have the following definitions:


*<sup>U</sup>*(*<sup>x</sup>*, *A*) = ∑∞*<sup>n</sup>*=<sup>1</sup> *Pn*(*<sup>x</sup>*, *A*) > 0.


$$K\_d(\mathfrak{x}, A) := \sum\_{n=0}^{\infty} P^n(\mathfrak{x}, A) d(n).$$

If there exits a transition kernel *T* satisfying

$$K\_d(\mathbf{x}, A) \ge T(\mathbf{x}, A), \quad \mathbf{x} \in \Omega, \ A \in \mathcal{B}(\Omega), \mathbf{x}$$

then *T* is called the continuous component of *Kd*.


$$
\pi(A) = \int\_{\Omega} \pi(d\mathbf{x}) P(\mathbf{x}, A), \ A \in \mathcal{B}(\Omega),
$$

is called an *invariant measure*.

The following two lemmas will be useful. See Meyn and Tweedie (1996) for the proofs and further details. We denote by *IA*(·) the indicator function of *A*.

**Lemma 1.** *Suppose that X is a weak Feller chain. Let C* ∈ B(Ω) *be a compact set and V a positive function. If*

$$V\left(X\_n\right) - E\left[V\left(X\_{n+1}\right)|X\_n\right] \ge 0,\\ X\_n \in \mathcal{C}',$$

*then there exists an invariant measure, finite on compact sets of* Ω*.*

**Lemma 2.** *Suppose that X is a weak Feller chain. Let C* ∈ B(Ω) *be a compact set, and V a positive function that is finite at some x*0 ∈ Ω*. If*

$$V\left(X\_n\right) - E\left[V\left(X\_{n+1}\right)|X\_n\right] \ge 1 - bI\_{\mathbb{C}}\left(X\_n\right), X\_n \in \Omega,$$

*with b a constant, b* < <sup>∞</sup>*, then there exists an invariant probability measure π on* B(Ω)*.*

**Lemma 3.** *Suppose that X is a ψ-irreducible aperiodic chain. Then the following conditions are equivalent:*

*1. There exists a function f* : Ω → [1, <sup>∞</sup>)*, a set C* ∈ B(Ω)*, a constant b* < ∞ *and a function V* : Ω → [0, <sup>∞</sup>)*, such that*

$$\left| V\left(X\_n\right) - E\left[ V\left(X\_{n+1}\right) \mid X\_n\right] \right| \ge f\left(X\_n\right) - bI\_C\left(X\_n\right), X\_n \in \Omega. \varepsilon$$

#### *2. The chain is positive recurrent with invariant probability measure π, and π* (*f*) < ∞*.*
