Age of Information Cost Minimization with No Buffers, Random Arrivals and Unreliable Channels: A PCL-Indexability Analysis

Abstract

Over the last decade, the Age of Information has emerged as a key concept and metric for applications where the freshness of sensor-provided data is critical. Limited transmission capacity has motivated research on the design of tractable policies for scheduling information updates to minimize Age of Information cost based on Markov decision models, in particular on the restless multi-armed bandit problem (RMABP). This allows the use of Whittle’s popular index policy, which is often nearly optimal, provided indexability (index existence) is proven, which has been recently accomplished in some models. We aim to extend the application scope of Whittle’s index policy in a broader AoI scheduling model. We address a model with no buffers incorporating random packet arrivals, unreliable channels, and nondecreasing AoI costs. We use sufficient indexability conditions based on partial conservation laws previously introduced by the author to establish the model’s indexability and evaluate its Whittle index in closed form under discounted and average cost criteria. We further use the index formulae to draw insights on how scheduling priority depends on model parameters.

1. Introduction

The last decade has witnessed a fast-growing interest in research on the Age of Information (AoI) concept and metric as proposed in Kaul et al. [1,2] and extensions thereof. The AoI quantifies the freshness of information at a receiver in a communication system, as it measures the time elapsed since the most recently received packet was generated at its source. See, e.g., Kosta et al. [3] and Yates et al. [4] for authoritative introductions and surveys. AoI metrics have become increasingly relevant as the demand for real-time applications, where timely information delivery is critical, continues to grow. These include Internet of Things, video streaming, online gaming, remote control, and real-time traffic monitoring.

1.1. AoI Minimization via Transmission Scheduling and Restless Bandit Framework

Much AoI research has addressed the design of transmission scheduling policies to minimize AoI costs in communication networks so that users have access to as much up-to-date information as possible. A relatively simple yet relevant setting for addressing the optimal scheduling problem is that of single-hop wireless communication networks, where information sources generate packets targeted to corresponding users. Packet transmissions are mediated by a base station (BS), i.e., a communications node that connects sources to users through a given set of channels. In such a scenario, the scheduling problem can be formulated as a restless multi-armed bandit problem (RMABP). See, e.g., the work reviewed in Section 1.2.

The RMABP is a versatile Markov decision process (MDP) (see, e.g., Puterman [5] and Bertsekas [6]) modeling framework for optimal dynamic priority allocation, introduced by Whittle in [7] as an extension of the classic multi-armed bandit problem (MABP). It concerns the optimal dynamic selection of up to M out of Nprojects, with $1⩽M<N$. Such projects are generic entities modeled as binary-action MDPs, which can be either active/selected or passive/rested at each time. In the present setting, there are Nprojects representing users and M channels. Selecting a project means allocating a channel for attempting to transmit a packet held at the BS to the corresponding user.

An intuitively appealing and tractable type of policies for the RMABP is provided by priority-index policies (or index policies for short). These attach an index to each project/user, i.e., a scalar function of its state, which is used as a dynamic measure of selection priority: the larger a project’s current index value, the higher the selection priority.

Index policies can be optimal in special models, in particular in the classic MABP (where passive projects do not change state) with $M=1$, as first shown by Gittins and Jones [8], and in Klimov’s model [9]. Yet, they are generally suboptimal for the RMABP, which is computationally intractable (P-space hard, see Papadimitriou and Tsitsiklis [10]). Still, Whittle [7] proposed a now widely popular heuristic index policy for the RMABP, which he conjectured to be asymptotically optimal under the average criterion, as M and N grow to infinity in a fixed ratio. Weber and Weiss [11] showed that such a conjecture does not hold in general but identified sufficient conditions under which it does hold.

Since its introduction, research on the RMABP has generated a vast amount of literature, which was partially reviewed in [12]. Much work has addressed applications of Whittle’s index policy in a wide variety of models, where numerical studies often report a near-optimal performance. Yet, researchers aiming to apply such a policy to a particular RMABP model are confronted with two major roadblocks, widely regarded as difficult: (1) proving that the model is indexable, meaning that it possesses a well-defined Whittle index, which cannot be taken for granted, and (2) devising an efficient index evaluation scheme.

Over the last two decades, the author has developed a framework based on partial conservation laws (PCLs) to overcome such roadblocks in increasingly general settings. See, e.g., Niño-Mora [13,14,15,16,17]. This approach is different from prevailing approaches and still not widely applied. We will apply it here, demonstrating its simplicity and effectiveness for solving a hitherto unsolved problem: proving indexability and evaluating the Whittle index for an AoI optimization model in which such results had not been previously obtained.

1.2. Related Prior Work: AoI Cost Minimization via Whittle’s Index Policy

A number of recent papers have applied the RMABP and Whittle’s index policy for AoI minimization in single-hop wireless communication network models. A major stream of such work considers no-buffer models where untransmitted packets in a time slot are lost. Kadota et al. [18] addressed a no-buffer network model having unreliable channels with a fixed transmission success probability for each user. Packet generation was deterministic: each source generated one packet at the start of each frame. The frames consisted of a fixed number of time slots at which transmissions could be attempted. If a packet transmission failed, it could be reattempted in the following slot of the current frame, if any. The goal was to minimize the (long-run) average linearly weighted AoI cost. Indexability was proven, the Whittle index was obtained in closed form, and, in a simulation study, Whittle’s policy was benchmarked and shown to be nearly optimal and to outperform alternative index policies.

Hsu et al. [19] considered an RMABP model for average linearly weighted AoI minimization incorporating random packet arrivals transmitted through reliable channels. Indexability was established, Whittle’s index was derived in closed form, and the resulting policy was tested. The online setting, where arrival rates are inferred, was also addressed.

Sun et al. [20] and Jhunjhunwala and Moharir [21] proposed the adoption of nonlinear AoI cost metrics to better model the growth of user dissatisfaction with data staleness. Tripathi and Modiano [22] incorporated average cost metrics into an RMABP model with nondecreasing AoI costs in a no-buffer system with deterministic arrivals and unreliable channels. They proved indexability, obtained Whittle’s index in closed form, and tested Whittle’s index policy.

Note that the aforementioned work does not address models incorporating both random packet arrivals and unreliable channels, which is a gap that the present paper fills.

A related stream of work focuses on establishing the asymptotic optimality of Whittle’s index policy in the above models under appropriate assumptions. Maatouk et al. [23] established asymptotic optimality for the model in [18] under a numerically verifiable recurrence condition. Kriouile et al. [24] proved such a result under a less stringent condition.

The no-buffer assumption is replaced in another stream of work by that of one-packet buffers, where the BS stores the last packet generated by a source. Sun et al. [25] considered a one-packet buffer variant of the model in [19], which was shown to result in performance gains. In addition to obtaining and testing Whittle’s index policy, a decentralized index policy was proposed. Sun et al. [26] extended the model in [25] by incorporating both random packet arrivals and unreliable channels. Yet, Ref. [26] did not prove that the model is indexable, as it stated that “it is really hard to establish indexability”. An index was proposed that is meant to approximate the Whittle index as if the model were indexable. The resulting policy was benchmarked in a numerical study, showing that it is near-optimal. Tang et al. [27] extended the model in [25] to incorporate nondecreasing AoI costs. They argued that the model is indexable, derived the Whittle index, and tested it in a numerical study.

The aforementioned work has focused on the average AoI cost minimization criterion. Yet, recent work has argued the relevance of the much less applied discounted cost criterion for such problems. Badia and Munari [28] motivated its use based on applications, e.g., in underwater environments, where the assumption of unlimited operation is unreasonable. In such settings, the discounted criterion, which is well known to model a geometrically distributed time horizon, is more appropriate. In particular, Ref. [28] focused on the performance evaluation of discounted AoI metrics in a model with deterministic arrivals and unreliable channels, assuming that transmissions were attempted with a fixed probability.

Wu et al. [29] considered a discounted AoI minimization model with deterministic arrivals and reliable channels, proving a property of optimal transmission policies. However, they did not consider the design of tractable suboptimal policies with good performance.

Zhang et al. [30] investigated the pricing of AoI updates in a discounted model.

1.3. Contributions over Prior Work

This paper presents the following contributions:

• We prove indexability for a no-buffer AoI optimization model with random packet arrivals, unreliable channels, and general nondecreasing costs, giving closed formulae for its Whittle index, under both discounted and average cost criteria.

  Table 1. Indexability analyses in RMABP AoI optimization models under the average cost criterion.

  Arrivals/Channels		Reliable		Unreliable
  Deterministic	Costs/Buffers	No	Yes	No	Yes
  	Linear	[18]	[25]	[18]	[26] (approx.)
  	Nonlinear	[22]	[27]	[22]	[27]
  Random	Costs/Buffers	No	Yes	No	Yes
  	Linear	[19]	[25]	this paper	[26] (approx.)
  	Nonlinear	this paper	[27]	this paper	[27]

  clarifies the gaps filled by this paper on indexability under the average cost criterion. Note that the “approx.” beside Ref. [26] indicates that a proxy of the Whittle index was considered there, as indexability was not proven. As for indexability under the discounted criterion, to the best of the author’s knowledge, it has not been addressed in an AoI setting. We thus demonstrate the effectiveness of the PCL-indexability approach (based on partial conservation laws (PCLs), see [13,14,15,16,17]) to prove indexability and evaluate the Whittle index by analyzing an AoI model that has not yet yielded to the prevailing approach.
• We provide simplified Whittle index formulae for relevant special cases, in particular linear, quadratic, and threshold-type AoI costs. The resulting expressions allow for a more efficient index evaluation than direct application of the general formulae.
• We analyze the Whittle index formulae to elucidate how a user’s transmission priority depends on model parameters under Whittle’s policy, in particular on the probabilities of packet arrival and successful transmission. The index is nonincreasing in the former; so, other things being equal, Whittle’s policy gives higher transmission priority to users with lower arrival rates. As for the latter probability, the index increases with it for linear costs, thus prescribing higher priority to users for which transmission attempts are more likely to succeed. Yet, for nonlinear costs, the index is generally non-monotonic in that probability.

1.4. Structure of the Paper

The remainder of the paper is organized as follows. Section 2 describes the system model and formulates it as an RMABP. Section 3 reviews the concepts of indexability and Whittle index in the present setting. Section 4 reviews the PCL-indexability approach and applies it to prove indexability and evaluate the Whittle index under the discounted criterion. Section 5 presents a PCL-indexability analysis for the average cost criterion. Section 6 elucidates the dependence of Whittle’s index on model parameters and evaluates the index for special cost functions. Finally, Section 7 concludes the paper. The paper includes two appendices, which contain some of the given proofs.

2. System Model and RMABP Formulation

We consider a system model where N information sources (e.g., monitoring stations) generate status update packets aimed to N corresponding users (e.g., applications or sensors). Time is slotted and time slots are denoted by $t=0,1,2,\dots$. Packet generation is modeled by independent Bernoulli processes: source $n=1,\dots ,N$ generates a packet at the start of a time slot with probability $0<{\lambda }_n⩽1$.

A BS connects sources to users. Transmission capacity is limited, as there are $M<N$ channels, so the BS can attempt to transmit at most M packets per slot among those that have been just generated, if any exist. The BS has no buffering capacity, so any untransmitted packets in a slot are lost.

Channels are unreliable: if the BS attempts to transmit a freshly generated packet from source n to user n, the transmission succeeds with probability $0<{\mu }_n⩽1$ independently across channels and slots. If the packet is transmitted, the BS receives an instantaneous acknowledgment so that it can keep track of the current AoI for each user. Packet generation and transmission error processes are mutually independent.

The state of user n at the beginning of slot t is denoted by $Y_n\left(t\right)\triangleq (B_n\left(t\right),X_n\left(t\right))$, where $B_n\left(t\right)$ equals 1 if a packet has just been generated by source n and is 0 otherwise, and $X_n\left(t\right)$ is the AoI for user n. Thus, $Y_n\left(t\right)$ lies in the state space $\mathcal{Y}\triangleq \{0,1\}\times \mathbb{N}$, where $\mathbb{N}$ is the set of natural numbers with elements $1,2,3,\dots$, as we assume that $X_n\left(t\right)⩾1$, because it takes one time slot for a packet to reach the user.

Figure 1 represents a system with $M=3$ channels and $N=5$ users in a slot where two packets from sources 1 and 3 have been generated. Only two channels can be allocated, which is indicated with gray filling. Figure 2 shows a snapshot of the same system with all channels allocated in a slot, as there are at least as many packets available as channels. Yet, the packet from source 5 will be lost, as it cannot be transmitted in the slot.

The cost incurred by user n in a slot is given by a general nonnegative and nondecreasing function $c_n\left(i_n\right)$ of its AoI $i_n$ at the beginning of the slot.

At the start of each slot, a controller at the BS decides which packets to attempt to transmit, among those just generated, if any, up to a maximum of M.

We shall assume that action choice is based only on the current joint state. This corresponds to considering scheduling policies $\pi$ from the class $\Pi \left(M\right)$ of stationary policies that attempt to transmit up to M packets at each slot. We denote by $A_n\left(t\right)$ the action chosen for user n in slot t, which can be 1 (attempt transmission of an available packet) or 0 otherwise (note that this action must be taken if no packet is available). As in [14], we shall refer to ${\mathcal{Y}}^{\{0,1\}}\triangleq \left\{1\right\}\times \mathbb{N}$ as the set of controllable states for user n, where both actions are available, and to ${\mathcal{Y}}^{\left\{0\right\}}\triangleq \left\{0\right\}\times \mathbb{N}$ as the set of uncontrollable states, where only the passive action is available.

The state transition probabilities for user n under action $a_n$, denoted by $p_{(b_n,i_n),(b_n^{\prime },i_n^{\prime })}^{a_n}\triangleq \mathbb{P}\{Y_n(t+1)=(b_n^{\prime },i_n^{\prime })|Y_n\left(t\right)=(b_n,i_n),A_n\left(t\right)=a_n\}$ for any time slot t, are given by

$$p_{(b_n,i_n),(b_n^{\prime },i_n^{\prime })}^0=\left\{\begin{array}{cc}1-{\lambda }_n\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(0,i_n+1)\hfill \\ {\lambda }_n\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(1,i_n+1)\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(1)

and, for $b_n=1$,

$$p_{(1,i_n),(b_n^{\prime },i_n^{\prime })}^1=\left\{\begin{array}{cc}(1-{\lambda }_n)(1-{\mu }_n)\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(0,i_n+1)\hfill \\ (1-{\lambda }_n){\mu }_n\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(0,1)\hfill \\ {\lambda }_n(1-{\mu }_n)\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(1,i_n+1)\hfill \\ {\lambda }_n{\mu }_n\hfill & \mathrm{if}(b_n^{\prime },i_n^{\prime })=(1,1)\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

Given a discount factor $0<\beta <1$, the expected total discounted cost incurred by policy $\pi \in \Pi \left(M\right)$ starting from the joint state $\mathbf{Y}\left(0\right)=(\mathbf{B}\left(0\right),\mathbf{X}\left(0\right))=(\mathbf{b},\mathbf{i})\in {\{0,1\}}^N\times {\mathbb{N}}^N$, where we use the vector notation $\mathbf{B}\left(t\right)={\left(B_n\left(t\right)\right)}_{n=1}^N$, $\mathbf{X}\left(t\right)={\left(X_n\left(t\right)\right)}_{n=1}^N$, $\mathbf{b}={\left(b_n\right)}_{n=1}^N$ and $\mathbf{i}={\left(i_n\right)}_{n=1}^N$, is

$$F_{(\mathbf{b},\mathbf{i})}^{\pi }\triangleq {\mathbb{E}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left[\sum _{t=0}^{\infty }\sum _{n=1}^N{c}_n\left(X_n\left(t\right)\right){\beta }^t\right].$$

(3)

In (3), ${\mathbb{E}}_{(\mathbf{b},\mathbf{i})}^{\pi }$ is the expectation under policy $\pi$, conditioned on starting from $(\mathbf{b},\mathbf{i})$. The discounted cost problem is to find an admissible policy (i.e., in $\Pi \left(M\right)$) minimizing the cost objective in (3) for any initial joint state $(\mathbf{b},\mathbf{i})$.

We shall also consider the average cost problem, which is to find an admissible scheduling policy minimizing the average expected cost per slot, given by

$${\overline{F}}_{(\mathbf{b},\mathbf{i})}^{\pi }\triangleq \underset{T\to \infty }{\lim \sup }\frac{1}{T}{\mathbb{E}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left[\sum _{t=0}^T\sum _{n=1}^N{c}_n\left(X_n\left(t\right)\right)\right].$$

(4)

Note that we restrict attention to stationary policies, even though we have not proven that they are optimal among the wider class of history-dependent policies. Conditions ensuring such a result in MDP models with countable states and unbounded costs are given, e.g., in Section 6.10 (discounted criterion) and 8.10 (average criterion) of Puterman [5]. Yet, since such issues are only of ancillary interest to this paper, we do not pursue them here, except for certain single-user subproblems (see Proposition 2).

The above problems are formulated as RMABPs by identifying projects with users and taking the active action on a project with attempting to transmit a packet for a user.

Given the intractability of computing an optimal policy, we aim to design suboptimal policies that can be implemented with low complexity and perform well. Index policies are appealing in this setting; they are based on defining an index ${\alpha }_n:{\mathcal{Y}}^{\{0,1\}}\to \mathbb{R}$ for each user n, a scalar function of its controllable states $(1,i_n)$. In a slot starting in $(\mathbf{b},\mathbf{i})$, where $\mathbf{b}$ and $\mathbf{i}$ are as in (3), the policy attempts to transmit up to M available packets, if any exist, giving higher priority to users with larger nonnegative index values and breaking ties arbitrarily.

This raises the issue of how to design easily computable index policies that perform well, if possible. Regarding ease of calculation, a simple choice is the greedy index policy, based on the index ${\alpha }_n(1,i_n)\triangleq c_n\left(i_n\right)$. A more sophisticated idea for constructing index policies was proposed by Whittle [7], as discussed below.

3. Indexability and Whittle Index

This section discusses indexability and the Whittle index, adapting the approach in Whittle [7] to the nuances of the present model. We focus on the discounted cost problem,

$$\underset{\pi \in \Pi \left(M\right)}{\mathrm{minimize}}{F}_{(\mathbf{b},\mathbf{i})}^{\pi },$$

(5)

where $F_{(\mathbf{b},\mathbf{i})}^{\pi }$, defined in (3), is the system’s cost metric, giving the expected total $\beta$-discounted cost under policy $\pi$ starting from $(\mathbf{b},\mathbf{i})$. We define $F_{(\mathbf{b},\mathbf{i})}^{*}$ to be the optimal cost.

We start by formulating a relaxation of optimization problem (5) by replacing the class of admissible scheduling policies $\Pi \left(M\right)$, which can allocate up to K channels per slot, with a larger class of policies. In particular, we use the class of policies $\Pi \left(N\right)$, which can allocate up to N channels per slot, and further impose the additional constraint

$$G_{(\mathbf{b},\mathbf{i})}^{\pi }⩽M_{\beta },$$

(6)

where $M_{\beta }\triangleq M/(1-\beta )$ and

$$G_{(\mathbf{b},\mathbf{i})}^{\pi }\triangleq {\mathbb{E}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left[\sum _{t=0}^{\infty }\sum _{n=1}^N{A}_n\left(t\right){\beta }^t\right].$$

We shall refer to $G_{(\mathbf{b},\mathbf{i})}^{\pi }$ as the system’s work metric, which gives the expected total discounted number of transmission attempts.

Note that (6) is implied by the sample-path constraints implicit in $\Pi \left(M\right)$, namely,

$$\sum _{n=1}^N{A}_n\left(t\right)⩽M,t=0,1,\dots$$

The resulting relaxed problem is

$$\underset{\pi \in \Pi \left(N\right),\left(6\right)}{\mathrm{minimize}}{F}_{(\mathbf{b},\mathbf{i})}^{\pi }.$$

(7)

Note that, by construction, the optimal cost $F_{(\mathbf{b},\mathbf{i})}^R$ of (7) is a lower bound on $F_{(\mathbf{b},\mathbf{i})}^{*}$.

We now attach a nonnegative Lagrange multiplier, or, more properly, since we are dealing with an inequality-constrained optimization problem, a Karush–Kuhn–Tucker (KKT) multiplier, $\nu ⩾0$, to constraint (6) in (7), and formulate the Lagrangian function

$${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left(\nu \right)\triangleq F_{(\mathbf{b},\mathbf{i})}^{\pi }+\nu \left(G_{(\mathbf{b},\mathbf{i})}^{\pi }-M_{\beta }\right),$$

(8)

and the Lagrangian relaxation

$$\underset{\pi \in \Pi \left(N\right)}{\mathrm{minimize}}{\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left(\nu \right).$$

(9)

Note that multiplier $\nu$ represents a charge per transmission attempt. Again, by construction, the optimal cost ${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left(\nu \right)$ of problem (9) gives in turn a lower bound on $F_{(\mathbf{b},\mathbf{i})}^R$ for any $\nu ⩾0$, which follows from the elementary weak duality relation

$${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left(\nu \right)⩽F_{(\mathbf{b},\mathbf{i})}^{\pi },\mathrm{for} \mathrm{any}\pi \in \Pi \left(N\right)\mathrm{satisfying}\left(6\right)\mathrm{and}\nu ⩾0.$$

(10)

The Lagrangian dual problem is to find a multiplier yielding the best lower bound:

$$\underset{\nu ⩾0}{\mathrm{maximize}}{\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left(\nu \right).$$

(11)

Note that (11) is a concave optimization problem—since function ${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left(\nu \right)$ is concave in $\nu$, being a minimum of linear functions—which facilitates its computational solution.

We would like to ensure strong duality, meaning that ${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left({\nu }^{*}\right)=F_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}$ for some ${\pi }^{*}\in \Pi \left(N\right)$ satisfying (6) and ${\nu }^{*}⩾0$, and hence, $F_{(\mathbf{b},\mathbf{i})}^R=F_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}$ and ${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{**}=F_{(\mathbf{b},\mathbf{i})}^R$, where ${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{**}={\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left({\nu }^{*}\right)$ is the optimal value of problem (11). For such a purpose, we adapt the concept of complementary slackness (CS) from inequality-constrained optimization. Given a policy $\pi \in \Pi \left(N\right)$ and a multiplier $\nu ⩾0$, we say that they satisfy CS if

$$\nu \left(G_{(\mathbf{b},\mathbf{i})}^{\pi }-M_{\beta }\right)=0.$$

(12)

The next result gives a sufficient condition for strong duality, which can be used to compute the bound $F_{(\mathbf{b},\mathbf{i})}^R$.

Proposition 1.

Suppose that ${\pi }^{*}\in \Pi \left(N\right)$ solves (9) with ${\nu }^{*}⩾0$, and that they satisfy CS. Then,

(a)

${\pi }^{*}$ solves the relaxed problem (7);

(b)

${\nu }^{*}$ solves the Lagrangian dual problem (11);

(c)

Strong duality holds.

Proof. 

The results follow straightforwardly from the above discussion and the identities

$$\begin{array}{cc}\hfill F_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}-{\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left({\nu }^{*}\right)& =F_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}+{\nu }^{*}\left(G_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}-M_{\beta }\right)-{\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left({\nu }^{*}\right)={\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{{\pi }^{*}}\left({\nu }^{*}\right)-{\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{*}\left({\nu }^{*}\right)=0.\hfill \end{array}$$

□

The solution of the Lagrangian relaxation (9) is further facilitated by the fact that it naturally decouples into the single-user subproblems

$$\underset{{\pi }_n\in {\Pi }_n}{\mathrm{minimize}}{F}_{n,(b_n,i_n)}^{{\pi }_n}+\nu G_{n,(b_n,i_n)}^{{\pi }_n},$$

(13)

for $n=1,\dots ,N$, where ${\Pi }_n$ denotes the stationary transmission policies for the subsystem corresponding to user n in isolation with a single channel. Note that

$$F_{n,(b_n,i_n)}^{{\pi }_n}\triangleq {\mathbb{E}}_{(b_n,i_n)}^{{\pi }_n}\left[\sum _{t=0}^{\infty }{c}_n\left(X_n\left(t\right)\right){\beta }^t\right]$$

(14)

and

$$G_{n,(b_n,i_n)}^{{\pi }_n}\triangleq {\mathbb{E}}_{(b_n,i_n)}^{{\pi }_n}\left[\sum _{t=0}^{\infty }{A}_n\left(t\right){\beta }^t\right]$$

(15)

are the user’s cost metric and work metric, respectively, as defined in Niño-Mora [14]. This follows from considering decoupled policies $\pi ={\left({\pi }_n\right)}_{n=1}^N$, where ${\pi }_n$ is used on user n, so

$$F_{(\mathbf{b},\mathbf{i})}^{\pi }=\sum _{n=1}^N{F}_{n,(b_n,i_n)}^{{\pi }_n}\mathrm{and}{G}_{(\mathbf{b},\mathbf{i})}^{\pi }=\sum _{n=1}^N{G}_{n,(b_n,i_n)}^{{\pi }_n},$$

hence the Lagrangian function in (8) can be decomposed as

$${\mathcal{L}}_{(\mathbf{b},\mathbf{i})}^{\pi }\left(\nu \right)=\sum _{n=1}^N\left(F_{n,(b_n,i_n)}^{{\pi }_n}+\nu G_{n,(b_n,i_n)}^{{\pi }_n}\right)-M_{\beta }\nu .$$

(16)

We shall call the model indexable if, for each user n and AoI $j_n\in \mathbb{N}$, there exists a critical transmission attempt charge ${\nu }_{(1,j_n)}^{*}$, which, viewed as an index, characterizes the optimal policies for subproblem (13) under any $\nu \in \mathbb{R}$, as follows: in state $(1,j_n)$, it is optimal—for any initial state $(b_n,i_n)$—to attempt to transmit if and only if ${\nu }_{(1,j_n)}^{*}⩾\nu$, and it is optimal to not attempt to transmit if and only if ${\nu }_{(1,j_n)}^{*}⩽\nu$. We shall refer to ${\nu }_{(1,j_n)}^{*}$ as user n’s Whittle index. Note that we define Whittle’s indexability property as formulated in, e.g., [17,31].

The Whittle index policy for the N-user model gives higher transmission priority to available packets targeted to users with larger nonnegative indices.

4. PCL-Indexability Analysis: Discounted Cost Criterion

This section reviews the PCL-indexability approach in [15,16] and applies it to prove indexability of the single-user subproblems (13) and derive the discounted Whittle index.

4.1. A Verification Theorem for Threshold-Indexability

Consider a single-user subproblem (13), dropping the user label n and writing the AoI cost function as $c_j$. The discounted cost and work performance metrics are given by

$$F_{(b,i)}^{\pi }\triangleq {\mathbb{E}}_{(b,i)}^{\pi }\left[\sum _{t=0}^{\infty }{c}_{X\left(t\right)}{\beta }^t\right],$$

(17)

and

$$G_{(b,i)}^{\pi }\triangleq {\mathbb{E}}_{(b,i)}^{\pi }\left[\sum _{t=0}^{\infty }A\left(t\right){\beta }^t\right],$$

(18)

so the single-user subproblem of concern is formulated as follows:

$$\underset{\pi \in \Pi }{\mathrm{minimize}}{F}_{(b,i)}^{\pi }+\nu G_{(b,i)}^{\pi }.$$

(19)

We shall write the packet arrival and the transmission success probabilities as $\lambda$ and $\mu$, respectively, dropping the user label. We shall further use the parameter $p\triangleq \lambda \mu$, the probability that in a time slot both a packet is generated and a corresponding transmission attempt succeeds, and consider the complementary probability $q\triangleq 1-p$. Note that $p>0$, but q may equal 0, in the case of deterministic arrivals and a reliable channel ($\lambda =\mu =1$).

We make the following standing assumption on the one-slot cost function $c_i$.

Assumption 1.

(i)

$c_i$ is nonnegative and nondecreasing;

(ii)

${\displaystyle \sum _{i=1}^{\infty }{c}_i{q}^i<\infty .}$

Note that part (ii) is a mild growth condition on $c_i$. We next show that Assumption A1 implies the existence of optimal stationary deterministic policies for subproblem (19).

Proposition 2.

Single-user subproblem $\left(19\right)$ is solved optimally by stationary deterministic policies determined by Bellman’s equations.

Proof. 

The result is trivial in the case $q=0$. In the case $q>0$, we apply Theorem 6.10.4 in Puterman [5], which ensures the stated result provided that Assumptions 6.10.1 and 6.10.2 therein hold, which are formulated in terms of an appropriate weight function. Here, we take $w_{(b,i)}\triangleq q^{-i}$ as the weight function, which satisfies Assumption 6.10.1 in [5] because $c_i/w_{(b,i)}=c_i{q}^i$ is bounded under Assumption 1.

To prove Assumption 6.10.2 in [5], we use the sufficient condition provided by Proposition 6.10.5.a in [5]. In the present setting, this reduces to

$$(1-\lambda )(1-\mu )w_{(0,i+1)}+(1-\lambda )\mu w_{(0,1)}+\lambda (1-\mu )w_{(1,i+1)}+\lambda \mu w_{(1,1)}⩽w_{(1,i)}+L,$$

(20)

for some $L>0$. This follows by verifying that the expression

$$\begin{array}{cc}\hfill & (1-\lambda )(1-\mu )w_{(0,i+1)}+(1-\lambda )\mu w_{(0,1)}+\lambda (1-\mu )w_{(1,i+1)}+\lambda \mu w_{(1,1)}-w_{(1,i)}\hfill \\ \hfill & =(1-\lambda )(1-\mu )q^{-(i+1)}+(1-\lambda )\mu q^{-1}+\lambda (1-\mu )q^{-(i+1)}+\lambda \mu q^{-1}-q^{-i}\hfill \\ \hfill & =(1-\mu )q^{-(i+1)}+\mu q^{-1}-q^{-i}\hfill \end{array}$$

is decreasing in i, and hence its maximum value is reached by $i=0$. This allows us to take $L\triangleq \lambda \mu /(1-\lambda \mu )=(1-q)/q>0$ satisfying (20). □

We shall consider a class of structured policies that intuition suggests might be optimal for problem (19) for any transmission attempt charge $\nu \in \mathbb{R}$, and which will later be proven to be, indeed, optimal: threshold policies, defined as follows. For a nonnegative integer $k⩾0$, the k-threshold policy (or k-policy for short) attempts to transmit a packet in controllable state $(1,j)$ if and only if $j>k$. We shall write its performance metrics as $F_{(b,i)}^k$ and $G_{(b,i)}^k$.

We shall further consider marginal performance metrics, giving the increments in cost and work metrics under threshold policies resulting from a change in the initial action. Thus, the marginal cost metric for the k-policy starting from $(1,i)$ is

$$f_{(1,i)}^k\triangleq F_{(1,i)}^{\langle 0,k\rangle }-F_{(1,i)}^{\langle 1,k\rangle },$$

(21)

where $\langle a,z\rangle$ denotes the policy that takes action a in the initial time slot $t=0$ and then follows the k-policy from time $t=1$ onward. Note that $f_{(1,i)}^k$ measures the decrease in cost that results from modifying the k-policy by attempting to transmit initially in state $(1,i)$ compared to not doing so. The marginal work metric is

$$g_{(1,i)}^k\triangleq G_{(1,i)}^{\langle 1,k\rangle }-G_{(1,i)}^{\langle 0,k\rangle },$$

(22)

which measures the corresponding increase in work expended.

If $g_{(1,i)}^k>0$ for every $i\in \mathbb{N}$, we further define the marginal productivity (MP) metric

$$m_{(1,i)}^k\triangleq \frac{f_{(1,i)}^k}{g_{(1,i)}^k},$$

(23)

and the MP index

$$m_{(1,i)}\triangleq m_{(1,i)}^i.$$

(24)

Consider now the following PCL-indexability (PCLI) conditions:

• (PCLI1) Positive marginal work: $g_{(1,i)}^k>0$ for every AoI $i⩾1$ and threshold $k⩾0$.
• (PCLI2) Monotone nondecreasing MP index: $m_{(1,i)}$ is nondecreasing in i.

The following verification theorem formulates, in the present setting, a result proven by the author in prior work for restless bandits in increasingly general settings. See [13] [Corollary 2] (finite-state projects), [14] [Theorem 6.3] (finite-state projects with general work metrics), [15] [Theorem 4.1] (semi-Markov countable-state projects), and [17] [Theorem 1 and Proposition 1(c)] (real-state projects). The result refers to threshold-indexability—indexability consistent with threshold policies—meaning that both the model is indexable and threshold policies are optimal for subproblem (19) for any $\nu \in \mathbb{R}$.

Theorem 1.

Suppose that condition (PCLI1) holds. Then, the model is threshold-indexable if and only if condition (PCLI2) holds, with Whittle index given by the MP index, i.e., ${\nu }_{(1,i)}^{*}=m_{(1,i)}$.

Note that [13,14,15] show that (PCLI1) and (PCLI2) are sufficient conditions for indexability of discrete-state projects, with the Whittle index being given by the MP index. The characterization of threshold indexability in Theorem 1 under (PCLI1) is stated in [17] as a consequence of Theorem 1 and Proposition 1(c) therein.

We emphasize that the PCL-indexability approach does not entail proving first optimality of threshold policies, as in the currently prevailing approach to indexability: both indexability and optimality of threshold policies follow in one fell swoop from Theorem 1.

We next set out to apply Theorem 1 to the present model, which entails verifying PCL-indexability conditions (PCLI1) and (PCLI2).

4.2. Work Metric Analysis and Condition (PCLI1)

We start with condition (PCLI1), namely that the marginal work metric under threshold policies be positive. Recall that the k-policy, with threshold $k⩾0$, takes the active action (attempt to transmit) in state $(1,i)$ if $i>k$, and the passive action otherwise.

We next address evaluation of the work metric $G_{(b,i)}^k$ under the k-policy. Even though in the system model we assumed that the AoI i is a positive integer, for obtaining the $G_{(b,i)}^k$ we include the fictitious value $i=0$.

From the dynamics in (1) and (2) it follows that the work metric $G_{(b,i)}^k$ satisfies the following linear equations: for $i,k⩾0$,

$$G_{(0,i)}^k=\beta (1-\lambda )G_{(0,i+1)}^k+\beta \lambda G_{(1,i+1)}^k,$$

(25)

and

$$G_{(1,i)}^k=\left\{\begin{array}{cc}\beta (1-\lambda )G_{(0,i+1)}^k+\beta \lambda G_{(1,i+1)}^k,\hfill & i⩽k\hfill \\ 1+\beta (1-\lambda )\mu G_{(0,1)}^k+\beta \lambda \mu G_{(1,1)}^k\hfill & \\ +\beta (1-\lambda )(1-\mu )G_{(0,i+1)}^k+\beta \lambda (1-\mu )G_{(1,i+1)}^k,\hfill & i>k.\hfill \end{array}\right.$$

(26)

In the future, to simplify the notation, we will find it convenient to define

$${\sigma }_i\triangleq 1-\beta q-{\beta }^{i+1}p,$$

(27)

and

$${\Gamma }_i^k\triangleq \left\{\begin{array}{cc}{\beta }^{k-i}{\Gamma }_k^k,\hfill & i<k\hfill \\ {\displaystyle \frac{\beta \lambda }{{\sigma }_k},}\hfill & i⩾k,\hfill \end{array}\right.$$

The next result gives closed-form expressions for evaluating the work metric $G_{(b,i)}^k$.

Lemma 1.

We have

$$G_{(0,i)}^k={\Gamma }_i^{k}and{G}_{(1,i)}^k=\left\{\begin{array}{cc}{\Gamma }_i^k,\hfill & i⩽k\hfill \\ 1+\mu {\Gamma }_0^k+(1-\mu ){\Gamma }_i^k,\hfill & i>k.\hfill \end{array}\right.$$

Proof. 

See Appendix A. □

The following result evaluates the marginal work metric $g_{(1,i)}^k$. Note that we use the standard notation $x^{+}\triangleq max(x,0)$.

Lemma 2.

$$\begin{array}{cc}\hfill g_{(1,i)}^k& =1-\mu ({\beta }^{{(k-i)}^{+}}-{\beta }^k)G_{(0,k)}^k=\frac{1-\beta (1-(1-{\beta }^{{(k-i)}^{+}})p)}{1-\beta (1-(1-{\beta }^k)p)}=\frac{1-\beta q-{\beta }^{{(k-i)}^{+}+1}p}{1-\beta q-{\beta }^{k+1}p}\hfill \\ \hfill & =\frac{{\sigma }_{{(k-i)}^{+}}}{{\sigma }_k},i⩾1.\hfill \end{array}$$

Proof. 

We can use the above results to represent $g_{(1,i)}^k$ as

$$\begin{array}{cc}\hfill g_{(1,i)}^k& =G_{(1,i)}^{\langle 1,k\rangle }-G_{(1,i)}^{\langle 0,k\rangle }=1+\mu G_{(0,0)}^k+(1-\mu )G_{(0,i)}^k-G_{(0,i)}^k=1-\mu \left({\Gamma }_i^k-{\Gamma }_0^k\right)\hfill \\ \hfill & =\left\{\begin{array}{cc}1-\mu ({\beta }^{k-i}-{\beta }^k){\Gamma }_k^k,\hfill & i⩽k\hfill \\ 1-\mu (1-{\beta }^k){\Gamma }_k^k,\hfill & i>k\hfill \end{array}\right.=\left\{\begin{array}{cc}{\displaystyle \frac{1-\beta q-{\beta }^{k-i+1}p}{1-\beta q-{\beta }^{k+1}p}=\frac{{\sigma }_{k-i}}{{\sigma }_k},}\hfill & i⩽k\hfill \\ \\ {\displaystyle \frac{1-\beta }{1-\beta q-{\beta }^{k+1}p}=\frac{{\sigma }_0}{{\sigma }_k},}\hfill & i>k.\hfill \end{array}\right.\hfill \end{array}$$

□

We thus obtain that the model satisfies condition (PCLI1).

Lemma 3.

$g_{(1,i)}^k>1-\beta$, and hence condition (PCLI1) holds.

Proof. 

It is immediately apparent from Lemma 2 that $g_{(1,i)}^k$ is decreasing in i for $i⩽k$, and hence

$$g_{(1,i)}^k⩾g_{(1,k)}^k=\frac{1-\beta }{1-\beta (1-(1-{\beta }^k)p)}>1-\beta ,i⩽k.$$

As for $i>k$, Lemma 2 shows that $g_{(1,i)}^k=g_{(1,k)}^k$, which completes the proof. □

4.3. Cost Metric Analysis

We continue by analyzing cost and marginal cost metrics under the k-policy. As above, we will find it convenient to include the fictitious AoI value $i=0$.

We start by analyzing the cost metric, which satisfies the following linear equations:

$$F_{(0,i)}^k=c_i+\beta (1-\lambda )F_{(0,i+1)}^k+\beta \lambda F_{(1,i+1)}^k,i⩾0$$

(28)

and

$$F_{(1,i)}^k=\left\{\begin{array}{cc}{c}_i+\beta (1-\lambda )F_{(0,i+1)}^k+\beta \lambda F_{(1,i+1)}^k,\hfill & i⩽k\hfill \\ c_i+\beta (1-\lambda )\mu F_{(0,1)}^k+\beta p{F}_{(1,1)}^k\hfill & \\ +\beta (1-\lambda )(1-\mu )F_{(0,i+1)}^k+\beta \lambda (1-\mu )F_{(1,i+1)}^k,\hfill & i>k.\hfill \end{array}\right.$$

(29)

In the future, to simplify notation, we will find it convenient to define, for $i⩾1$,

$$C_i\left(q\right)\triangleq \left\{\begin{array}{cc}{\displaystyle \sum _{j=1}^{\infty }{\left(\beta q\right)}^{j-1}{c}_{i-1+j},}\hfill & \mathrm{if}q>0\hfill \\ {\displaystyle c_i,}\hfill & \mathrm{if}q=0.\hfill \end{array}\right.$$

(30)

We further define quantities ${\Phi }_i^k$ as follows. For $i=k$,

$${\Phi }_k^k\triangleq \frac{1}{{\sigma }_k}\left[p\sum _{j=0}^{k-1}{\beta }^{j+1}{c}_j+(1-\beta q)\left(c_k+\beta C_{k+1}\left(q\right)\right)\right];$$

for $i<k$,

$${\Phi }_i^k\triangleq \sum _{j=i}^{k-1}{\beta }^{j-i}{c}_j+{\beta }^{k-i}{\Phi }_k^k;$$

and, for $i>k$,

$$\begin{array}{c}\hfill {\Phi }_i^k\triangleq \left\{\begin{array}{cc}{\displaystyle {\left(\beta q\right)}^{k-i}({\Phi }_k^k-\beta p\frac{1-{\left(\beta q\right)}^{i-k}}{1-\beta q}{\Phi }_0^k-c_k}\hfill & \\ {\displaystyle -\beta \sum _{j=1}^{i-k-1}{\left(\beta q\right)}^{j-1}{c}_{k+j}-\beta p{\left(\beta q\right)}^{i-k-1}{c}_i),}\hfill & \mathrm{if}q>0\hfill \\ {\displaystyle c_i+\beta c_{i+1}+\beta {\Phi }_0^k,}\hfill & \mathrm{if}q=0.\hfill \end{array}\right.\end{array}$$

The following result gives closed-form expressions for evaluating the cost metric $F_{(b,i)}^k$.

Lemma 4.

We have

$$F_{(0,i)}^k={\Phi }_i^k\mathrm{and}{F}_{(1,i)}^k=\left\{\begin{array}{cc}{F}_{(0,i)}^k,\hfill & i⩽k\hfill \\ \mu c_i+\mu F_{(0,0)}^k+(1-\mu )F_{(0,i)}^k,\hfill & i>k.\hfill \end{array}\right.$$

Proof. 

See Appendix B. □

The next result evaluates the marginal cost metric $f_{(1,i)}^k$ in closed form.

Lemma 5.

We have

(a)

$f_{(1,i)}^k=\mu \left({\Phi }_i^k-{\Phi }_0^k-c_i\right);$

(b)

In particular, in the case $i=k$,

$$f_{(1,i)}^i=\frac{\mu }{{\sigma }_i}\left[\beta (1-{\beta }^i)(1-\beta q)C_{i+1}\left(q\right)-(1-\beta )\sum _{j=0}^i{\beta }^j{c}_j\right].$$

Proof. 

See Appendix B. □

4.4. MP Index Analysis and Condition (PCLI2)

We next give a closed formula for the discounted MP index $m_{(1,i)}$ defined in (24).

Lemma 6.

We have

$$m_{(1,i)}=\left\{\begin{array}{c}{\displaystyle \mu \left[\frac{\beta (1-{\beta }^i)(1-\beta q)}{1-\beta }{C}_{i+1}\left(q\right)-\sum _{j=1}^i{\beta }^j{c}_j\right],\mathrm{if}q>0}\hfill \\ \\ {\displaystyle \frac{\beta (1-{\beta }^i)}{1-\beta }{c}_{i+1}-\sum _{j=1}^i{\beta }^j{c}_j,\mathrm{if}q=0.}\hfill \end{array}\right.$$

(31)

Proof. 

The result follows straightforwardly from Lemmas 2 and 5, using (30). □

We are now ready to establish the PCL-indexability condition (PCLI2).

Lemma 7.

Under Assumption 1, $m_{(1,i)}$ is monotone nondecreasing, so condition (PCLI2) holds.

Proof. 

We have, using that $C_i\left(q\right)=c_i+\beta q{C}_{i+1}\left(q\right)$ and $c_i$ is nondecreasing,

$$\begin{array}{cc}\hfill (1-{\beta }^i)C_{i+1}\left(q\right)-(1-{\beta }^{i-1})C_i\left(q\right)& =(1-{\beta }^i)C_{i+1}\left(q\right)-(1-{\beta }^{i-1})\left(c_i+\beta q{C}_{i+1}\left(q\right)\right)\hfill \\ \hfill & =\left(1-{\beta }^i-(1-{\beta }^{i-1})\beta q\right)C_{i+1}\left(q\right)-(1-{\beta }^{i-1})c_i\hfill \\ \hfill & =\left(1-\beta q-{\beta }^{i}p\right)C_{i+1}\left(q\right)-(1-{\beta }^{i-1})c_i\hfill \\ \hfill & ⩾\left(1-\beta q-{\beta }^{i}p\right)\sum _{j=1}^{\infty }{\left(\beta q\right)}^{j-1}{c}_i-(1-{\beta }^{i-1})c_i\hfill \\ \hfill & =\frac{(1-\beta ){\beta }^{i-1}}{1-\beta q}{c}_i.\hfill \end{array}$$

On the other hand,

$$\sum _{j=1}^i{\beta }^j{c}_j-\sum _{j=1}^{i-1}{\beta }^j{c}_j={\beta }^i{c}_i.$$

Hence, we have, for $i⩾2$,

$$m_{(1,i)}-m_{(1,i-1)}⩾\frac{\mu }{1-\beta }\left[\beta (1-\beta q)\frac{(1-\beta ){\beta }^{i-1}}{1-\beta q}{c}_i-(1-\beta ){\beta }^i{c}_i\right]=0,$$

which completes the proof. □

We thus obtain the following result.

Proposition 3.

Under Assumption 1, the discounted model is threshold-indexable, with Whittle index given by the MP index $m_{(1,i)}.$

Proof. 

The result follows from Lemmas 3 and 7 by applying Theorem 1. □

5. PCL-Indexability Analysis: Average Cost Criterion

This section applies Theorem 1 to the present model under the average cost criterion. This entails verifying conditions (PCLI1) and (PCLI2) for average metrics that are counterparts of the discounted metrics above. Note that the PCL-indexability theory under the average criterion, based on Laurent series expansions, is discussed in [13] [Section 5], [14] [Section 6.5], and [15] [Section 5.2], upon which we draw for the following analyses.

5.1. Work Metric Analysis and Condition (PCLI1)

We start with condition (PCLI1), namely the positivity of the marginal work metric under threshold policies. We start by evaluating the average work metric under the k-policy,

$$G_{(b,i)}^k\triangleq \underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{(b,i)}^k\left[\sum _{t=0}^{T}A\left(t\right)\right].$$

(32)

The following result evaluates $G_{(b,i)}^k$ in closed form, which shows that it does not depend on the initial state $(b,i)$; hence, we write it as $G^k$. This follows from standard results relating discounted and average cost metrics in MDPs. See, e.g., [5] [Theorem 8.10.7]. Note that, below, we incorporate the discount factor $\beta$ in the notation for the discounted metrics considered above to distinguish them from their average criterion counterparts. Thus, e.g., we write the discounted work metric as $G_{\beta ,(b,i)}^k$.

Lemma 8.

We have

$$G^k=G_{(b,i)}^k=\underset{\beta \to 1}{lim}(1-\beta )G_{\beta ,(b,i)}^k=\frac{\lambda }{kp+1}.$$

Proof. 

The result follows by using the expressions for $G_{\beta ,(b,i)}^k$ in Lemma 1. □

The next result evaluates the average marginal work metric $g_{(1,i)}^k$ based on the above formulae for its discounted counterpart, which we now write as $g_{\beta ,(1,i)}^k$.

Lemma 9.

We have

$$g_{(1,i)}^k=\underset{\beta \to 1}{lim}{g}_{\beta ,(1,i)}^k=\frac{{(k-i)}^{+}p+1}{kp+1},i⩾1.$$

Proof. 

The result follows by using the expression for $g_{\beta ,(1,i)}^k$ in Lemma 2. □

We thus obtain satisfaction of condition (PCLI1) under the average criterion.

Lemma 10.

$g_{(1,i)}^k>0$, and hence condition (PCLI1) holds.

Proof. 

The result follows directly from Lemma 9. □

5.2. Cost Metric Analysis

We continue by analyzing the average cost and marginal cost metrics under the k-policy for $k⩾0$, starting with the average cost metric, given by

$$F_{(b,i)}^k\triangleq \underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{(b,i)}^k\left[\sum _{t=0}^T{c}_{X\left(t\right)}\right].$$

(33)

In the future, we will find it convenient to define (cf. (30)), for $i⩾1$,

$$C_i\left(q\right)\triangleq \left\{\begin{array}{cc}{\displaystyle \sum _{j=1}^{\infty }{q}^{j-1}{c}_{i-1+j},}\hfill & \mathrm{if}q>0\hfill \\ {\displaystyle c_i,}\hfill & \mathrm{if}q=0,\hfill \end{array}\right.$$

(34)

Let us now define quantities ${\Phi }^k$ as follows: for any $i⩾0$,

$${\Phi }^k\triangleq \underset{\beta \to 1}{lim}(1-\beta ){\Phi }_{\beta ,i}^k=\frac{1}{kp+1}\left[p\sum _{j=0}^{k-1}{c}_j+p\left(c_k+\beta C_{k+1}\left(q\right)\right)\right].$$

The following result evaluates $F_{(b,i)}^k$ in closed form, which shows that it does not depend on the initial state $(b,i)$; hence, we write it as $F^k$. This follows from standard results (see, e.g., [5] [Theorem 8.10.7]). Below, we denote the discounted cost metric as $F_{\beta ,(b,i)}^k$.

Lemma 11.

We have

$$F^k=F_{(b,i)}^k=\underset{\beta \to 1}{lim}(1-\beta )F_{\beta ,(b,i)}^k={\Phi }^k.$$

Proof. 

The result follows by using the formulae for $F_{\beta ,(b,i)}^k$ in Lemma 4. □

The next result gives a closed-form expression for the average marginal cost metric $f_{(1,i)}^i$, which is the only one needed for our analyses. Below, we denote the discounted marginal cost metric as $f_{\beta ,(1,i)}^k$.

Lemma 12.

We have

$$f_{(1,i)}^i=\underset{\beta \to 1}{lim}{f}_{\beta ,(1,i)}^i=\frac{\mu }{ip+1}\left(ip{C}_{i+1}\left(q\right)-\sum _{j=0}^i{c}_j\right).$$

Proof. 

The result follows by using the expression for $f_{\beta ,(1,i)}^i$ in Lemma 5(b). □

5.3. MP Index Analysis and Condition (PCLI2)

We are now ready to address the PCL-indexability condition (PCLI2), namely, the nondecreasingness of the MP index defined by (24), using the marginal metrics for the average cost criterion. The next result gives closed formulae for such an average criterion MP index by drawing on the discounted MP index derived above, which we now denote as $m_{\beta ,(1,i)}$.

Lemma 13.

$$m_{(1,i)}=\mu \left[ip{C}_{i+1}\left(q\right)-\sum _{j=1}^i{c}_j\right]=\left\{\begin{array}{cc}{\displaystyle \mu \left[ip\sum _{j=1}^{\infty }{q}^{j-1}{c}_{i+j}-\sum _{j=1}^i{c}_j\right],}\hfill & \mathrm{if}q>0\hfill \\ \\ {\displaystyle ip{c}_{i+1}-\sum _{j=1}^i{c}_j}\hfill & \mathrm{if}q=0.\hfill \end{array}\right.$$

(35)

Proof. 

The result follows by taking the limit $m_{(1,i)}={lim}_{\beta \to 1}{m}_{\beta ,(1,i)}$ using the formulae in Lemma 6 for $m_{\beta ,(1,i)}$. □

We next establish satisfaction of condition (PCLI2).

Lemma 14.

Under Assumption 1, the average criterion MP index $m_{(1,i)}$ is monotone nondecreasing, so (PCLI2) holds.

Proof. 

The result follows from the nondecreasingness of the discounted MP index in Lemma 7, since $m_{(1,i)}={lim}_{\beta \to 1}{m}_{\beta ,(1,i)}$. □

We can now establish the model’s indexability and evaluate its Whittle index.

Proposition 4.

Under Assumption 1, the model is threshold-indexable under the average criterion, with Whittle index given by the MP index $m_{(1,i)}.$

Proof. 

The result follows from Lemmas 10 and 13 using Theorem 1 (in its average criterion version). □

6. Index Dependence on Model Parameters and Evaluation for Special Cost Functions

This section exploits the Whittle index formulae obtained above for two interrelated purposes: exploring the dependence of the index on model parameters, drawing corresponding insights, and specializing the index formula to relevant special cost functions.

6.1. Monotonicity of the Whittle Index on the Arrival Rate

We have the following result.

Proposition 5.

Both the discounted Whittle index $m_{\beta ,(1,i)}$ and the average Whittle index $m_{(1,i)}$ are nonincreasing in the packet arrival probability λ.

Proof. 

For the discounted criterion, we use the expression of the index in (31), along with

$$\begin{array}{cc}\hfill \frac{d}{d\lambda }{C}_{\beta ,i+1}\left(q\right)& =\frac{d}{d\lambda }\sum _{j=1}^{\infty }{\left(\beta q\left(\lambda \right)\right)}^{j-1}{c}_{i+j}=\frac{d}{d\lambda }\sum _{j=2}^{\infty }{\left(\beta q\left(\lambda \right)\right)}^{j-1}{c}_{i+j}\hfill \\ \hfill & =\beta \left(\sum _{j=2}^{\infty }(j-1){\left(\beta q\left(\lambda \right)\right)}^{j-2}{c}_{i+j}\right)\frac{d}{d\lambda }q\left(\lambda \right)\hfill \\ \hfill & =-\beta \mu \sum _{j=1}^{\infty }j{\left(\beta q\right)}^{j-1}{c}_{i+j+1},\hfill \end{array}$$

to obtain

$$\begin{array}{c}\hfill \frac{d}{d\lambda }{m}_{\beta ,(1,i)}=-\frac{{\left(\beta \mu \right)}^2(1-{\beta }^i)(1-\beta q)}{1-\beta }\sum _{j=1}^{\infty }j{\left(\beta q\right)}^{j-1}{c}_{i+j+1}⩽0.\end{array}$$

For the average criterion, we use the expression of the index in (35), along with

$$\frac{d}{d\lambda }{C}_{i+1}\left(q\right)=-\mu \sum _{j=1}^{\infty }j{q}^{j-1}{c}_{i+j+1},$$

to obtain

$$\frac{d}{d\lambda }{m}_{(1,i)}=-ip{\mu }^2\sum _{j=1}^{\infty }j{q}^{j-1}{c}_{i+j+1}⩽0.$$

□

Proposition 5 provides the following insight regarding the packet transmission priority under Whittle’s index policy: other things being equal, the policy gives higher (or, more precisely, not lower) transmission priority to users whose packets arrive at lower rates. This is consistent with the numerical results in [19] [Section 7.1] on the properties of optimal scheduling policies for the special case with reliable channels under the average criterion.

It is readily verified with examples that, for general cost functions, the Whittle index does not need to be monotonic on the transmission success probability $\mu$. Yet, some monotonicity properties can be identified in the case of special cost functions, as discussed below.

6.2. Whittle Index Formulae for Special Cost Functions

We next consider evaluation of the Whittle index for special cost functions, which yields formulae that can be implemented with greater efficiency than the general formulae above. We shall consider the cases of linear, quadratic, and threshold-type AoI costs.

6.2.1. Linear AoI Cost

We start with the case of a linear cost $c_i\triangleq ci$, with $c>0$. The discounted index reduces to

$$m_{\beta ,(1,i)}=\frac{\beta c\mu }{1-\beta }\left[i-\frac{\beta \left(1-{\beta }^i\right)p}{(1-\beta )(1-\beta q)}\right],$$

(36)

and the average criterion index simplifies to

$$m_{(1,i)}=\frac{c\mu }{2}\left(i^2+\frac{1+q}{p}i\right)=c\mu i\left(\frac{i-1}{2}+\frac{1}{p}\right).$$

(37)

Even though, for general costs, the Whittle index is not necessarily monotonic in the probability of successful transmission $\mu$, in this special case it turns out to be increasing in $\mu$ (for $i⩾2$ under the average criterion, since $m_{(1,1)}=c/\lambda$).

Proposition 6.

For linear costs, the discounted Whittle index $m_{\beta ,(1,i)}$ is increasing in μ. The same holds for the average Whittle index $m_{(1,i)}$ for $i⩾2$.

Proof. 

Under the discounted cost criterion, we have

$$\frac{d}{d\mu }{m}_{\beta ,(1,i)}=\frac{\beta c}{1-\beta }\left[i-\frac{\beta \left(1-{\beta }^i\right)(2-\beta (2-p))p}{(1-\beta ){(1-\beta q)}^2}\right]>0,$$

which follows because

$$\frac{\beta (2-\beta (2-p\left)\right)p}{(1-\beta ){(1-\beta q)}^2}⩽\frac{1}{1-\beta }<\frac{i}{1-{\beta }^i}.$$

(38)

Note that (38) holds because $i/(1-{\beta }^i)$ increases with i, and the left-most expression in (38) attains the value $1/(1-\beta )$ for $p=1$ and increases with p (recall that $q\triangleq 1-p$), since

$$\frac{d}{dp}\frac{\beta (2-\beta (2-p\left)\right)p}{(1-\beta ){(1-\beta q)}^2}=\frac{2\beta (1-\beta )}{{(1-\beta q)}^3}>0.$$

Under the average cost criterion, we have, for $i⩾2$,

$$\frac{d}{d\mu }{m}_{(1,i)}=\frac{ci(i-1)}{2}>0.$$

This proves the result. □

Hence, for linear AoI costs, other things being equal, Whittle’s policy gives higher transmission priority to users for which packet transmissions are more likely to succeed.

We next graphically illustrate the above results for the base instance with linear cost $c_i\triangleq i$ and the parameters $\lambda =0.7$, $\mu =0.8$, and $\beta =0.8$ for the states $(1,1)$, $(1,2)$, and $(1,3)$. Figure 3 plots the corresponding Whittle index vs. the AoI under the discounted and average cost criteria. The plots show that the discounted index grows nearly linearly in the AoI, in contrast to the quadratic growth of the average criterion index.

Regarding the dependence on $\lambda$, Figure 4 plots the discounted and the average criterion Whittle indices vs. $\lambda$ for AoI values $i=1,2,3$. The plot is consistent with Proposition 5, as the Whittle index is decreasing in $\lambda$. A significant difference is that, as $\lambda \to 0$, the discounted index converges to a finite limit, easily seen to be ${lim}_{\lambda \to 0}{m}_{\beta ,(1,i)}=\beta c\mu i/(1-\beta )$, whereas the average criterion index diverges to infinity. Thus, under the average cost criterion, the increase in priority becomes steeper as the packet arrival probability becomes lower.

Consider now the dependence on $\mu$. Figure 5 plots the discounted and the average criterion Whittle indices vs. $\mu$ for AoI values $i=1,2,3$. The plot is consistent with Proposition 6, as the index increases with $\mu$ (except for $i=1$ under the average criterion, where it remains constant as $\mu$ varies). Note that the increase is linear under the average criterion.

6.2.2. Quadratic AoI Cost

We next turn to the case of a quadratic cost function $c_i\triangleq c{i}^2$. Then, the discounted Whittle index has the evaluation

$$m_{\beta ,(1,i)}=\frac{\beta c\mu }{1-\beta }\left[i^2+\frac{2\left(1-\beta +p{\beta }^{i+1}\right)}{(1-\beta )(1-\beta q)}i-\frac{\beta p\left(1-{\beta }^i\right)(3-\beta (1+(1+\beta )q))}{{(1-\beta )}^2{(1-\beta q)}^2}\right],$$

(39)

whereas the average criterion Whittle index reduces to

$$m_{(1,i)}=c\mu \left[\frac{2}{3}{i}^3+\frac{4-{(1+q)}^2}{2p^2}{i}^2+\frac{21-{(3+p)}^2}{6p^2}i\right].$$

(40)

Considering the same instance as above, Figure 6 plots the corresponding Whittle index vs. the AoI under the discounted and average cost criteria. The plots show that the Whittle index growth with the AoI is much steeper under the average criterion (having cubic growth) than under the discounted criterion (having quadratic growth).

As for the dependence on $\lambda$, the plots in Figure 7 are also consistent with Proposition 5. As was the case for the linear cost case, we see that, as $\lambda \to 0$, the discounted index converges to a finite limit, whereas the average criterion index diverges to infinity.

Consider now the dependence on $\mu$. Figure 8 plots the discounted and the average criterion Whittle index vs. $\mu$ for the same instance. The plot reveals a strikingly different behavior of the discounted and average criterion Whittle indices. Thus, for $i=1$, the discounted index appears to first increase and then to decrease in $\mu$. To verify it, note that

$$m_{\beta ,(1,1)}=\frac{\beta (3-\beta q)c\mu }{{(1-\beta q)}^2},$$

and

$$\frac{d}{d\mu }{m}_{\beta ,(1,1)}=\frac{\beta c(3-\beta (4-\beta +(1+\beta )p\left)\right)}{{(1-\beta q)}^3}.$$

Thus, for the given instance, we have

$$\frac{d}{d\mu }{m}_{\beta ,(1,1)}=\frac{100(55-126\mu )}{{(5+14\mu )}^3},$$

and hence $m_{\beta ,(1,1)}$ increases in $\mu$ for $0<\mu <55/126$ and decreases in $\mu$ for $55/126<\mu <1$.

On the other hand, the plot shows that the discounted Whittle index increases with $\mu$ for $i=2$ and for $i=5$. It is readily verified analytically that such is the case in this instance for AoI values $i⩾2$.

In contrast, the plot shows that the average criterion Whittle index $m_{(1,1)}$ decreases with $\mu$. We can readily verify this, noting that

$$\frac{d}{d\mu }{m}_{(1,i)}=c\left[\frac{2i^3}{3}-\frac{i^2}{2}-\left(\frac{1}{6}+\frac{2}{p^2}\right)i\right]=ci\left[\frac{2}{3}{i}^2-\frac{1}{2}i-\left(\frac{1}{6}+\frac{2}{p^2}\right)\right],$$

(41)

and hence

$$\frac{d}{d\mu }{m}_{(1,1)}=-\frac{2c}{p^2}<0.$$

Furthermore, we have, for $0<p<1$,

$$\frac{d}{d\mu }{m}_{(1,2)}=c\left(3-\frac{4}{p^2}\right)<0,$$

and hence $m_{(1,2)}$ also decreases with $\mu$, consistently with the plot.

As for $m_{(1,5)}$, the plot shows that it first decreases and then increases with $\mu$. Using (41), it is readily verified analytically that such behavior holds provided $\lambda >1/\sqrt{7}$, since

$$\frac{d}{d\mu }{m}_{(1,5)}=c\left(70-\frac{10}{{\lambda }^2{\mu }^2}\right),$$

and hence $m_{(1,5)}$ has a unique minimum at ${\mu }^{*}=1/\left(\lambda \sqrt{7}\right)\in (0,1)$ for such $\lambda$.

The same qualitative behavior holds for $m_{(1,i)}$ where $i⩾3$, first decreasing and then increasing with $\mu$, provided that

$$\lambda >\frac{2\sqrt{3}}{\sqrt{4i^2-3i-1}}$$

since in such cases $m_{(1,i)}$ has a unique minimum as a function of $\mu$ at

$${\mu }^{*}=\frac{2\sqrt{3}}{\lambda \sqrt{4i^2-3i-1}}\in (0,1).$$

Note also that as $\mu \to 0$, $m_{\beta ,(1,i)}\to 0$, whereas $m_{(1,i)}\to \infty$.

6.2.3. Threshold-Type AoI Cost

Finally, consider the case of a threshold-type cost function, with $c_i\triangleq c{1}_{\{i>k\}}$. Then, it is readily verified that the discounted Whittle index is given by

$$m_{\beta ,(1,i)}=\left\{\begin{array}{cc}{\displaystyle \frac{\beta c\mu }{1-\beta }(1-{\beta }^i){\left(\beta q\right)}^{k-i},}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle \frac{\beta c\mu }{1-\beta }(1-{\beta }^k),}\hfill & \mathrm{if}i⩾k,\hfill \end{array}\right.$$

(42)

whereas the average criterion index has the evaluation

$$m_{(1,i)}=\left\{\begin{array}{cc}{\displaystyle c\mu i{q}^{k-i},}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle c\mu k,}\hfill & \mathrm{if}i⩾k.\hfill \end{array}\right.$$

(43)

Considering the same instance as above, with $k=10$, Figure 9 plots the Whittle index vs. the AoI under the discounted and average cost criteria. The plots show that the Whittle index growth with the AoI is slightly steeper for $i<k$ under the average criterion than under the discounted criterion. For $i⩾k$, both indices remain constant as the AoI i grows.

Regarding the dependence on $\lambda$, we have

$$\frac{d}{d\lambda }{m}_{\beta ,(1,i)}=\left\{\begin{array}{cc}{\displaystyle -\frac{{\beta }^{2}c{\mu }^2}{1-\beta }(1-{\beta }^i)(k-i){\left(\beta q\right)}^{k-i-1},}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle 0,}\hfill & \mathrm{if}i⩾k,\hfill \end{array}\right.$$

and

$$\frac{d}{d\lambda }{m}_{(1,i)}=\left\{\begin{array}{cc}{\displaystyle -c{\mu }^{2}i(k-i)q^{k-i-1},}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle 0,}\hfill & \mathrm{if}i⩾k.\hfill \end{array}\right.$$

Figure 10 plots the discounted and the average criterion Whittle indices vs. $\lambda$. The plot shows that both indices are decreasing in $\lambda$ for $i<k$ and remain constant for $i⩾k$. Note that the indices decrease linearly with $\lambda$ for $i=k-1$.

As for the dependence of the Whittle index on $\mu$, we have, for the discounted index,

$$\frac{d}{d\mu }{m}_{\beta ,(1,i)}=\left\{\begin{array}{cc}{\displaystyle -c\frac{{\beta }^2(1-{\beta }^i){\left(\beta q\right)}^{k-i-1}}{1-\beta }((k-i+1)p-1),}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle c\frac{\beta (1-{\beta }^k)}{1-\beta },}\hfill & \mathrm{if}i⩾k,\hfill \end{array}\right.$$

and, for the average criterion index,

$$\frac{d}{d\mu }{m}_{(1,i)}=\left\{\begin{array}{cc}{\displaystyle -ci{q}^{k-i-1}((k-i+1)p-1),}\hfill & \mathrm{if}i<k\hfill \\ \\ {\displaystyle ck,}\hfill & \mathrm{if}i⩾k.\hfill \end{array}\right.$$

Hence, for $i<k$, both indices $m_{\beta ,(1,i)}$ and $m_{(1,i)}$ decrease with $\mu$ whenever $(k-i+1)p>1$, i.e., for $\mu >1/\left(\lambda \right(k-i+1\left)\right)$, and increase with $\mu$ for sufficiently low $\mu$ values, in particular for $\mu <1/\left(\lambda \right(k-i+1\left)\right)$. In contrast, for $i⩾k$, both indices increase linearly in $\mu$. Such behavior is illustrated in Figure 11.

7. Discussion

This paper has addressed a broad model for scheduling AoI updates simultaneously incorporating the features of random packet arrivals, unreliable channels, and general nondecreasing AoI costs in a no-buffer setting. Indexability (existence of the Whittle index) had only been established in prior work for special cases of this model and only under the average cost criterion using the prevailing approach for indexability analysis, based on first establishing optimality of threshold policies and then exploiting the monotonicity property of the optimal threshold. We have proven that the broad model addressed is indexable and have evaluated its Whittle index in closed form, both under the discounted and the average criterion, by carrying out a PCL-indexability analysis, an approach that has been developed by the author over the last two decades. By demonstrating its effectiveness in a relevant model that had not previously yielded to the prevailing analysis approach, we aim to bring such an approach to the attention of researchers in the area so that they can use it to analyze other models. We have further drawn qualitative insights on how scheduling priority under Whittle’s index policy depends on models parameters by exploiting the closed formulae for Whittle’s index for general costs and for special cost functions. It remains to numerically test the performance of Whittle’s index policy in the present model. Future research directions include carrying out a PCL-indexability analysis in more complex AoI scheduling models, such as those incorporating buffers.