Relating the Ramsay Quotient Model to the Classical D-Scoring Rule

Abstract

In a series of papers, Dimitrov suggested the classical D-scoring rule for scoring items that give difficult items a higher weight while easier items receive a lower weight. The latent D-scoring model has been proposed to serve as a latent mirror of the classical D-scoring model. However, the item weights implied by this latent D-scoring model are typically only weakly related to the weights in the classical D-scoring model. To this end, this article proposes an alternative item response model, the modified Ramsay quotient model, that is better-suited as a latent mirror of the classical D-scoring model. The reasoning is based on analytical arguments and numerical illustrations.

1. Introduction

Item response theory (IRT; [1]) is the statistical analysis of test items in education, psychology, and other fields of social sciences. Typically, a number of test items are administered to test-takers. The primary interest lies in inferring the ability (performance or trait) based on the test items. IRT models relate observed item responses to unobserved latent traits. Because the latent trait is unobserved, there are many plausible choices for modeling these relationships. The most popular class of IRT models is that of logistic IRT models [2].

Recently, in a series of papers and a well-written summary book, the researcher Dimiter Dimitrov [3,4] suggested the classical D-scoring rule for scoring items that gives difficult items a higher weight while easier items receive a lower weight. Subsequently, the so-called latent D-scoring model has been proposed as a latent analog of the classical D-scoring model [5]. However, it has been shown that this proposed model is statistically equivalent to the widely used two-parameter logistic IRT model [6]. As argued later in this article, the weights implied by this latent D-scoring model are far from perfectly related to the weights in the classical D-scoring model. This article proposes an alternative IRT model, a modified Ramsay quotient model, that can be interpreted as a latent analog ([3]; or a latent mirror, [4]) of the classical D-scoring rule. The usefulness of the Ramsay quotient model is demonstrated based on analytical and numerical arguments.

The remainder of the article is organized as follows. Section 2 briefly reviews IRT models. In particular, it focuses on the two-parameter logistic and the Ramsay quotient model. Section 3 reviews the classical and latent D-scoring model of Dimitrov. In Section 4, a variant of the Ramsay quotient model is proposed intended to serve as an adequate latent D-scoring model. The usefulness of this model is illustrated in Section 5 using five empirical datasets. Finally, the article closes with a discussion in Section 6.

2. Item Response Modeling

Let $\mathit{X}=(X_1,\dots ,X_I)$ be a vector of I binary random variables $X_i$ ($i=1,\dots ,I$) that are also referred to as items or item responses. A unidimensional IRT model [2,7,8] model parametrizes the multivariate distribution $\mathrm{P}(\mathit{X}=\mathit{x})$ for $\mathit{x}=(x_1,\dots ,x_I)\in {\{0,1\}}^I$ as

$$\mathrm{P}(\mathit{X}=\mathit{x})={\int }_{-\infty }^{\infty }\prod _{i=1}^I\left[P_i{(\mathsf{\theta };{\mathbf{\gamma }}_i)}^{x_i}{\left(1-P_i(\mathsf{\theta };{\mathbf{\gamma }}_i)\right)}^{1-x_i}\right]\mathrm{d}{\mathrm{F}}_{\mathsf{\alpha }}\left(\mathsf{\theta }\right),$$

(1)

where F${}_{\mathsf{\alpha }}$ is the distribution function of the latent trait $\mathsf{\theta }$ (also referred to as the ability variable) that depends on a parameter $\mathsf{\alpha }$. The quantity $P_i(\mathsf{\theta };{\mathbf{\gamma }}_i)=\mathrm{P}\left(X_i=1\right|\mathsf{\theta })$ is referred to as the item response function (IRF) for item i that depends on a parameter ${\mathbf{\gamma }}_i$. From (1), it can be seen that items $i=1,\dots ,I$ are conditionally independent given the latent trait $\mathsf{\theta }$. Some identification constraints on item parameters ${\mathbf{\gamma }}_i$ or distribution parameters $\mathsf{\alpha }$ must be imposed to ensure model identification [9].

If the IRT model (1) has been estimated, individual ability estimates $\widehat{\mathsf{\theta }}$ can be estimated by maximizing the log-likelihood function l that gives the most likely ability estimate for $\mathsf{\theta }$ given a vector of item responses $\mathit{x}$. The log-likelihood function is given by (see [1])

$$l\left(\mathsf{\theta }\right)=\sum _{i=1}^I\left[x_{i}log\left[P_i(\mathsf{\theta };{\mathbf{\gamma }}_i)\right]+(1-x_i)log\left[1-P_i(\mathsf{\theta };{\mathbf{\gamma }}_i)\right]\right]$$

(2)

By taking the derivative with respect to $\mathsf{\theta }$ in (2), the ability estimate $\widehat{\mathsf{\theta }}$ fulfills the nonlinear equation

$$\frac{\partial l}{\partial \mathsf{\theta }}(\widehat{\mathsf{\theta }})=\sum _{i=1}^I\left[x_i\frac{P_i^{\prime }(\widehat{\mathsf{\theta }};{\mathbf{\gamma }}_i)}{P_i(\widehat{\mathsf{\theta }};{\mathbf{\gamma }}_i)}-(1-x_i)\frac{P_i^{\prime }(\widehat{\mathsf{\theta }};{\mathbf{\gamma }}_i)}{1-P_i(\widehat{\mathsf{\theta }};{\mathbf{\gamma }}_i)}\right]=0,$$

(3)

where $P_i^{\prime }\left(\mathsf{\theta }\right)=(\partial P_i)/(\partial \mathsf{\theta })$.

Most IRT models take the full item response pattern into account [10,11,12]. It is only for the simple Rasch model (RM, [13], see Section 2.1.1) that the sum score is a sufficient statistic, and not every item response pattern results in a different ability score.

2.1. Two-Parameter Logistic (2PL) Item Response Model

An important class of IRT models is the class of logistic IRT models. Logistic IRT models employ the logistic link function for parameterizing IRFs. The IRFs in the two-parameter logistic (2PL) model [14] are given by

$$P_i\left(\mathsf{\theta }\right)=\frac{exp\left(a_i(\mathsf{\theta }-b_i)\right)}{1+exp\left(a_i(\mathsf{\theta }-b_i)\right)},$$

(4)

where $a_i$ denotes item discriminations and $b_i$ denotes item difficulties. The ability variable $\mathsf{\theta }$ is typically real-valued. Frequently, a normal distribution is chosen in the 2PL model. However, this assumption can be weakened [15].

An important (and well-known) property of the 2PL model is that ${\sum }_{i=1}^I{a}_i{X}_i$ is a sufficient statistic for $\mathsf{\theta }$ [2]. This directly follows because the individual log-likelihood function defined in (2) can be derived as

$$l\left(\mathsf{\theta }\right)=\left\{\sum _{i=1}^I{a}_i{x}_i\right\}\mathsf{\theta }-\sum _{i=1}^I{a}_i{b}_i-\sum _{i=1}^{I}log\left[1+exp\left(a_i(\mathsf{\theta }-b_i)\right)\right]$$

(5)

A frequently employed identification constraint in the 2PL model is $\mathrm{E}\left(\mathsf{\theta }\right)=0$ and var$\left(\mathsf{\theta }\right)=1$. Alternatively, the item discrimination and the item difficulty for one item can be fixed; that is, $a_i=1$ and $b_i=0$ for some $i\in \{1,\dots ,I\}$.

2.1.1. Rasch Model (1PL Model)

The one-parameter logistic (1PL) model (RM; [13]) is obtained by setting all item discriminations in the 2PL model equal to one (i.e., $a_i=1$ for $i=1,\dots ,I$). In this case, the IRF is given by

$$P_i\left(\mathsf{\theta }\right)=\frac{exp\left(\mathsf{\theta }-b_i\right)}{1+exp\left(\mathsf{\theta }-b_i\right)}.$$

(6)

Note that, in (6), a difference between the ability $\mathsf{\theta }$ and the item difficulty $b_i$ is involved. Therefore, the model could also be referred to as the Rasch difference model. Note that ${\sum }_{i=1}^I{X}_i$ is a sufficient statistic for $\mathsf{\theta }$ in the RM.

In the RM, $\mathrm{E}\left(\mathsf{\theta }\right)$ = 0 is a frequently utilized identification constraint. Alternatively, one item difficulty $b_i$ could be set to zero. As a further alternative, the mean of the item difficulties could be set to zero.

2.1.2. Implementation

The 1PL model or the 2PL model can be estimated using marginal maximum likelihood (MML) using an expectation–maximization (EM) algorithm [16]. Alternatively, the estimation could be carried out using Newton–Raphson algorithms. More generally, the R [17] packages mirt [18] (using the function mirt::mirt() in combination with mirt::createItem() and mirt::createGroup() or sirt [19] (using the function sirt::xxirt()) allow users to arbitrarily define IRFs $P_i(\mathsf{\theta };{\mathbf{\gamma }}_i)$ ($i=1,\dots ,I$) and distribution functions F$\mathsf{\alpha }$ in the IRT model (1) that should be estimated.

2.2. Ramsay Quotient Model (RQM)

Ramsay [20] proposed a quotient of a positive ability variable $\mathsf{\xi }$ and a positive item difficulty parameter $B_i$ in their quotient model as an alternative to the difference model of Rasch (see (6)). The IRF in the Ramsay quotient model (RQM; [20]) is defined as

$$\mathrm{P}(X_i=1|\mathsf{\xi })=\frac{exp(\mathsf{\xi }/B_i)}{K_i+exp(\mathsf{\xi }/B_i)}.$$

(7)

Note that the positive item parameter $K_i$ also allows the representation of guessing effects in multiple-choice items [20]. It should be emphasized that the RQM can be rewritten (7) as (see [20])

$$\mathrm{P}(X_i=1|\mathsf{\xi })=\frac{exp\left\{\frac{1}{B_i}\left(\mathsf{\xi }-B_{i}log{K}_i\right)\right\}}{1+exp\left\{\frac{1}{B_i}\left(\mathsf{\xi }-B_{i}log{K}_i\right)\right\}},$$

(8)

which corresponds to a 2PL model with a positively valued ability. The item discrimination and the item difficulty are given by $a_i=1/B_i$ and $b_i=B_{i}log{K}_i$, respectively. Interestingly, the item discrimination is necessarily correlated with item difficulty. Also, note that ${\sum }_{i=1}^I(1/B_i)X_i$ is a sufficient statistic for the latent ability $\mathsf{\xi }$ because (8) is a 2PL model. An interesting model might result in constraining $K_i$ equal across items (i.e., $K_i=K$ for all $i=1,\dots ,I$). Then, the implied item discrimination $a_i$ is indirectly proportional to the item difficulty $b_i$. That is, we get $a_i{b}_i=logK$.

To sum up, the RQM is a particularly constrained 2PL model with a positively valued ability variable $\mathsf{\xi }$. The item difficulty $B_i$ from the RQM enters both item discrimination $a_i$ and item difficulty $b_i$ from the 2PL model. Interestingly, the RQM is related to the diffusion model of Ratcliffe that is utilized in cognitive psychology for modeling response times [21,22]. Furthermore, the RQM shares the simplicity of the Rasch model if used with only one item parameter but provides a simple alternative to handle guessing effects. In contrast to the three-parameter logistic IRT model, a linearly weighted statistic of items for the latent ability $\mathsf{\xi }$ is available for the RQM.

Implementation

To estimate the RQM defined by the IRFs in (7), it is convenient to define $\tilde{\mathsf{\theta }}=log\mathsf{\xi }$ and ${\tilde{b}}_i=log{B}_i$. These parameters are unbounded in contrast to the original definition in the RQM. If a normal distribution for $\tilde{\mathsf{\theta }}$ is assumed, there is a log-normal distribution for $\mathsf{\xi }$ results. The IRF in (8) can then be written as

$$\mathrm{P}(X_i=1|\mathsf{\xi })=\frac{exp\left\{exp(\tilde{\mathsf{\theta }}-{\tilde{b}}_i)\right\}}{K_i+exp\left\{exp(\tilde{\mathsf{\theta }}-{\tilde{b}}_i)\right\}}=\frac{1}{K_{i}exp\left\{-exp(\tilde{\mathsf{\theta }}-{\tilde{b}}_i)\right\}+1}.$$

(9)

Using the last term on the right side of Equation (9) is preferable to avoid numerical overflow.

3. Dimitrov’s D-Scoring Approach

3.1. Classical D-Scoring Method

The classical D-scoring approach was proposed by Dimitrov [3]. Based on I observed binary item responses $X_i$ ($i=1,\dots ,I$), the classical D-score ${\mathfrak{D}}_c$ is defined as

$${\mathfrak{D}}_c=\frac{{\displaystyle \sum _{i=1}^I{\mathrm{\delta }}_i{X}_i}}{{\displaystyle \sum _{i=1}^I{\mathrm{\delta }}_i}},\mathrm{where}{\mathrm{\delta }}_i=1-\mathrm{P}(X_i=1)=\mathrm{P}(X_i=0).$$

(10)

The variable ${\mathfrak{D}}_c$ is a weighted sum score in which the weights ${\mathrm{\delta }}_i$ are given as one minus the p-value ${\pi }_i=\mathrm{P}(X_i=1)$ of one item. Hence, difficult items are upweighted, and easy items are downweighted in the scoring rule (10). Such a rationale for scoring items might be appealing to some practitioners. Of course, it might be reasonable in a high-stakes test to inform test takers before the test administration when implementing such an unconventional scoring rule [23].

3.2. Rational Function Model (RFM) as a Latent D-Scoring Model

Dimitrov and Atanasov [5] (see also [24]) propose an IRT model, the latent D-scoring method that “can be seen as a latent analog to the classical” D-scoring method ([4], p. 64) described in Section 3.1 (see (10)). The model is called the rational function model (RFM), and the IRF for item $i=1,\dots ,I$ is defined by

$$\mathrm{P}(X_i=1|\mathcal{D})=\frac{1}{1+{\left[\frac{1-\mathcal{D}}{\mathcal{D}}\frac{{\mathsf{\beta }}_i}{1-{\mathsf{\beta }}_i}\right]}^{{\mathsf{\alpha }}_i}},$$

(11)

where $\mathcal{D}$ is the latent ability variable taking values between 0 and 1. The parameter ${\mathsf{\beta }}_i$ (with $0<{\mathsf{\beta }}_i<1$) can be interpreted as a difficulty parameter in the RFM, and ${\mathsf{\alpha }}_i$ is a positive shape parameter. Robitzsch [6] pointed out that the RFM is equivalent to the 2PL model (4) by defining the transformed parameters

$$\mathsf{\theta }=log\frac{\mathcal{D}}{1-\mathcal{D}},b_i=log\frac{{\mathsf{\beta }}_i}{1-{\mathsf{\beta }}_i},\mathrm{and}{a}_i={\mathsf{\alpha }}_i.$$

(12)

Hence, the estimation of the RFM can be carried out using software for the 2PL model [6]. The only ambiguity is about specifying the distribution of the latent variable $\mathsf{\theta }$. If an identification constraint is defined for an item (e.g., $a_1=1$ and $b_1=0$), sufficiently flexible distributions for $\mathsf{\theta }$ can be estimated, given the distribution parameters can be empirically identified. The density function of $\mathcal{D}$ can be obtained using a density transformation of $\mathsf{\theta }$ (see Equation (12) in [6]).

As pointed out in [4] (Ch. 12), $\mathsf{\theta }$ and $\mathcal{D}$ are typically highly correlated in empirical applications (e.g., $r>0.999$), although the ability estimators are nonlinearly (but monotonically) related.

Dimitrov ([4], p. xiv) claimed that the latent D-score variable $\mathcal{D}$ “is a latent mirror” of the classical D-score ${\mathfrak{D}}_c$, “with some advantages in the accuracy of estimation” ([4], p. xiv). The authors computed the ability estimates $\widehat{\mathcal{D}}$ and correlated them with ${\mathfrak{D}}_c$. In a real-data example involving an English proficiency test with 100 items administered by the National Center for Assessment in Saudi Arabia, the correlation was 0.974. It was argued that this correlation was still high enough to demonstrate that the classical D-score ${\mathfrak{D}}_c$ is a good proxy of the latent D-score $\widehat{\mathcal{D}}$ [4,5]. We tend to disagree with this reasoning. First, reasonably different IRT models will typically lead to high correlations between their ability estimates, questioning the usefulness of this measure. Second, it is clear that the two scores will differ because they involve different weighting schemes. In the classical D-score ${\mathfrak{D}}_c$, items are weighted by item difficulty ${\mathrm{\delta }}_i$ in the sufficient statistic ${\sum }_{i=1}^I{\mathrm{\delta }}_i{X}_i$, while the ability estimate $\widehat{\mathcal{D}}$ as a nonlinear transformation of the ability estimate $\widehat{\mathsf{\theta }}$ from the 2PL model possesses the sufficient statistic ${\sum }_{i=1}^I{a}_i{X}_i$. Hence, the correlation between ${\mathfrak{D}}_c$ and $\widehat{\mathcal{D}}$ will be primarily a function of the correlation of the weights ${\mathrm{\delta }}_i$ and $a_i$ ($i=1,\dots ,I$) across items. An obvious latent mirror of the classical D-scoring rule is obtained if the 2PL model would be estimated with fixed item discriminations ${\mathrm{\delta }}_i$ (as can also be seen in [25]). Then, the sum score ${\sum }_{i=1}^I{\mathrm{\delta }}_i{X}_i$ would necessarily be a sufficient statistic of the ability $\mathsf{\theta }$. In the 2PL model with fixed item discriminations, the standard deviation of $\mathsf{\theta }$ can be freely estimated.

4. Modified Ramsay Quotient Model and the Classical D-Scoring Rule

In this section, we present an IRT model that directly implements the idea that more difficult items should receive a higher score in the latent variable.

The RQM has the attractive property that it is equivalent to a particular 2PL model (see Section 2.2) in which the difficulty of an item is simultaneously reflected in the item discrimination and the item difficulty parameters. However, the RQM defined in (8) implies that more difficult items (i.e., with a larger $B_i$) become downweighted in terms of item discrimination (i.e., $1/B_i$ becomes smaller). This is the converse of what is defined in the classical D-score ${\mathfrak{D}}_c$ in (10). In the classical D-score computation, more difficult items receive a higher weight.

Nevertheless, one could simply swap the roles of correct and incorrect item responses in the RQM. We apply the RQM from (7) and (9) and define the IRF for an incorrect item response

$$\mathrm{P}(X_i=0|\eta )=\frac{exp(\mathsf{\xi }/B_i)}{K+exp(\mathsf{\xi }/B_i)}=\frac{exp\left\{exp(-\eta -{\tilde{b}}_i)\right\}}{K+exp\left\{exp(-\eta -{\tilde{b}}_i)\right\}},$$

(13)

where $\tilde{b}=log{b}_i$ and $\eta =-log\mathsf{\xi }$. Note that the term “$-\eta$” appears in (13) because the newly defined latent variable $\eta$ should reflect its ability instead of its non-ability, which is parametrized when modeling incorrect item responses (i.e., $X_i=0$) with the RQM. Also, note that we constrained all K parameters to be equal across items in (13) in order to maximally reflect the item difficulty in the item parameter ${\tilde{b}}_i$ (and $B_i=exp\left({\tilde{b}}_i\right)$). Note that the probability of a correct item response is given as

$$\mathrm{P}(X_i=1|\eta )=1-\mathrm{P}(X_i=0|\eta )=\frac{K}{K+exp\left\{exp(-\eta -{\tilde{b}}_i)\right\}}.$$

(14)

According to the 2PL parametrization of the RQM in (8), the IRF in (13) can be written as

$$\mathrm{P}(X_i=0|\eta )=\frac{exp\left\{\frac{1}{B_i}\left(exp(-\eta )-B_{i}logK\right)\right\}}{1+exp\left\{\frac{1}{B_i}\left(exp(-\eta )-B_{i}logK\right)\right\}},$$

(15)

Consequently, ${\sum }_{i=1}^I(1/B_i)(1-X_i)$ is a sufficient statistic for $\mathsf{\xi }=exp(-\eta )$, $\eta$, or any injective transformation of $\eta$. Trivially, ${\sum }_{i=1}^I(1/B_i)X_i$ is also a sufficient statistic for these variables. Because incorrect item responses were applied in the RQM, $B_i$ reflects item easiness. Hence, easy items get downweighted in this sufficient statistic (and therefore in $\eta$ and their monotone transformations). This is the desired property in the classical D-scoring approach.

The application of this variant of the RQM also depends on the joint K item parameter. Of course, one can simultaneously estimate all item parameters ${\tilde{b}}_i$ ($i=1,\dots ,I$) and the variance of $\eta$ in the modified RQM. However, the fit of the IRT model will typically be a function of K. Because our goal is to make the modified RQM maximally equivalent to the classical D-scoring approach, we repeatedly fit the modified RQM on grid values of K (say, from $K=1$ to 100). Then, we choose the model whose associations in estimated weights $1/B_i$ from the modified RQM in (15) and the weights ${\mathrm{\delta }}_i=1-\mathrm{P}(X_i=1)$ are maximal (see the next Section 5).

5. Numerical Examples

We tested our newly proposed latent D-scoring approach based on the modified RQM (see Section 4) with the classical D-scoring approach discussed in Section 3.1 for several datasets containing binary item responses. For each of the datasets, we fitted the RQM assuming a normal distribution for the latent variable $\eta$ with a fixed mean of 0, and we estimated the standard deviation (SD) of $\eta$. We also estimated the difficulty parameter on the log scale, meaning that ${\tilde{b}}_i$ was estimated, and the parameter $B_i$ was computed as $exp\left({\tilde{b}}_i\right)$.

For each dataset, we estimated the modified RQM for a fixed K parameter for $K=1,2,\dots ,100$. Then, we fitted a linear regression model through the origin according to the model

$${\mathrm{\delta }}_i=\frac{1}{B_i}\mathsf{\beta }+e_i\mathrm{for}i=1,\dots ,I,$$

(16)

where ${\mathrm{\delta }}_i$ are the weights used in the classical D-score ${\mathfrak{D}}_c$. Based on the linear regression (16), we computed the determination coefficient $R^2$ that quantifies how well the weights ${\mathrm{\delta }}_i=1-\mathrm{P}(X_i=1)=1-{\pi }_i$ were approximated by the weights $1/B_i$ obtained from the modified RQM as the latent D-scoring model. Because $R^2$ is a function of K, the optimal RQM that is intended to be used for latent D-scoring can be chosen with the corresponding K that maximizes the determination coefficient $R^2\left(K\right)$.

We also specified a linear regression model of weights ${\mathrm{\delta }}_i$ on item discriminations $a_i$ from the 2PL model to assess the similarity of the classical D-scoring rule with the scores from the latent D-scoring model by means of the determination coefficient $R^2$. The regression model was given by

$${\mathrm{\delta }}_i=a_i\tilde{\mathsf{\beta }}+{\tilde{e}}_i\mathrm{for}i=1,\dots ,I.$$

(17)

In order to compare the implications of the different weights from the models, we computed correlations of the classical D-score with linearly weighted item scores, where the weights were obtained from the RQM (i.e., the RQM D-score) and the 2PL model (i.e., the latent D-score), respectively.

All analyses were carried out in the R software (Version 4.3.1, [17]). The estimation of the modified RQM utilized the sirt::xxirt() function in the R package sirt [19]. The R code for model specification can be found at https://osf.io/nrcag (accessed on 26 September 2023).

In the following, results from five publicly available binary item response datasets were reported. All datasets had no missing values.

The first dataset data.si06 from the R package sirt [19] contains 4441 students on 14 items. The data stem from a verbal comprehension test. Figure 1 displays the determination coefficient of the regression (16) of the weights ${\mathrm{\delta }}_i$ of the classical D-score on the weights $1/B_i$ obtained from the modified RQM as a function of the parameter K. There is a maximum $R^2$, which is attained at $K=12$. Figure 2 displays the weights ${\mathrm{\delta }}_i$ on the y axis and the weights $1/B_i$ on the x axis for the optimal value $K=12$. The estimated SD of $\eta$ from the RQM was 0.459. In addition, the fitted regression line from (16) is shown in Figure 2. It can be seen that the obtained $R^2$ of 0.997 implies that weights ${\mathrm{\delta }}_i$ from the classical D-score were well predicted by weights $1/B_i$ from the RQM.

Figure 3 includes the regression (17) of weights ${\mathrm{\delta }}_i$ on the estimated weights $a_i$ from the 2PL model. It can be seen that the weights ${\mathrm{\delta }}_i$ from the classical D-scoring rule are not well predicted by weights $a_i$ estimated in the 2PL model. The determination coefficient from the regression model was only $R^2=0.502$.

The classical D-score was correlated with the RQM score at 0.9995, while the correlation with the 2PL score was much lower at 0.9072.

Table 1. Dataset data.si06 from R package sirt: item parameters and normalized weights from the Ramsay quotient model (RQM) and the two-parameter logistic (2PL) model.

	CTT		RQM				2PL		Normalized Weights
Item	${\mathit{\pi }}_{\mathbf{i}}$	${\mathit{\delta }}_{\mathbf{i}}$	${\tilde{\mathit{b}}}_{\mathbf{i}}$	${\mathit{K}}_{\mathbf{i}}$	${\mathit{B}}_{\mathbf{i}}$	$\mathbf{1}/{\mathit{B}}_{\mathbf{i}}$	${\mathit{a}}_{\mathbf{i}}$	${\mathit{b}}_{\mathbf{i}}$	${\mathit{\delta }}_{\mathbf{i}}$	$\mathbf{1}/{\mathit{B}}_{\mathbf{i}}$	${\mathit{a}}_{\mathbf{i}}$
WV01	0.79	0.21	−0.08	12	0.92	1.08	1.51	−1.23	0.61	0.66	1.15
WV02	0.65	0.35	−0.46	12	0.63	1.59	0.70	−1.00	1.01	0.96	0.53
WV03	0.73	0.27	−0.33	12	0.72	1.39	1.80	−0.86	0.79	0.84	1.37
WV04	0.74	0.26	−0.32	12	0.73	1.37	2.08	−0.85	0.76	0.83	1.58
WV05	0.48	0.52	−0.88	12	0.42	2.40	1.03	0.10	1.53	1.45	0.79
WV06	0.68	0.32	−0.47	12	0.62	1.60	1.87	−0.66	0.92	0.97	1.43
WV07	0.85	0.15	0.24	12	1.27	0.78	1.26	−1.74	0.44	0.48	0.96
WV08	0.62	0.38	−0.63	12	0.54	1.87	1.98	−0.42	1.11	1.13	1.51
WV09	0.93	0.07	1.56	12	4.74	0.21	1.85	−2.02	0.21	0.13	1.41
WV10	0.48	0.52	−0.83	12	0.43	2.30	0.59	0.18	1.53	1.39	0.45
WV11	0.84	0.16	0.19	12	1.21	0.83	1.35	−1.63	0.46	0.50	1.03
WV12	0.73	0.27	−0.26	12	0.77	1.30	0.82	−1.35	0.80	0.79	0.63
WV13	0.46	0.54	−0.92	12	0.40	2.50	1.16	0.16	1.57	1.51	0.88
WV14	0.22	0.78	−1.35	12	0.26	3.88	0.40	3.25	2.27	2.35	0.30

Note. CTT = indices based on classical test theory; normalized weights were computed such that their mean equals 1.

contains estimated item parameters from the RQM with $K=12$ and the 2PL model. In this table, normalized item weights for the sufficient statistics of the different scores were computed that have an average weight of 1. It can be seen that normalized weights based on the RQM and the ${\mathrm{\delta }}_i$ parameters from classical D-scoring were very similar. However, these weights substantially differed from the weights obtained from the 2PL model, which are also displayed in Figure 3.

The second dataset data.numeracy can be found in the R package TAM [26]. It contains 876 students on 15 items. The data resulted from a numerical comprehension test. The upper left panel in Figure 4 displays the determination coefficient of the regression (16) of the weights ${\mathrm{\delta }}_i$ of the classical D-score on the weights $1/B_i$ obtained from the modified RQM. The estimated SD of $\eta$ from the RQM was 0.505. The maximum $R^2$ value of 0.999 was attained for $K=20$. It can be seen in the upper left panel in Figure 4 that the weights ${\mathrm{\delta }}_i$ from the classical D-score were almost perfectly predicted by weights $1/B_i$ from the RQM. The determination coefficient from the regression model of weights ${\mathrm{\delta }}_i$ from classical D-scoring onto weights $a_i$ from the 2PL model was $R^2=0.895$. The classical D-score correlated with the RQM score at 0.9999, while the correlation with the 2PL score was much lower at 0.9909.

The third dataset data.read from the R package sirt contains 328 students on 12 items. It stems from a reading comprehension test involving three reading comprehension text stimuli. Like in the first dataset, weights from the classical D-score were satisfactorily predicted by the weights from the RQM as indicated by the $R^2$ of 0.999 (see the upper-right panel in Figure 4). The optimal K regarding the maximum determination coefficient was $K=12$. The estimated SD of $\eta$ from the RQM was 0.519. The determination coefficient from the regression model of weights ${\mathrm{\delta }}_i$ from classical D-scoring onto weights $a_i$ from the 2PL model was $R^2=0.425$. The classical D-score correlated with the RQM score at 0.9998, while the correlation with the 2PL score was much lower at 0.8742.

The fourth dataset data.pisaMath from the sirt package contains 565 students on 11 items. The data resulted from a PISA mathematics test involving Austrian students. As the lower-left panel in Figure 4 indicates, the weights ${\mathrm{\delta }}_i$ of the classical D-score were well predicted ($R^2=0.999$) by the weights from the modified RQM for the optimal $K=19$. The estimated SD of $\eta$ from the RQM was 0.433. The determination coefficient from the regression model of weights ${\mathrm{\delta }}_i$ from classical D-scoring onto weights $a_i$ from the 2PL model was $R^2=0.954$. The classical D-score correlated with the RQM score at 0.9999, while the correlation with the 2PL score was much lower at 0.9938.

Finally, the fifth dataset Psych101 from the KernSmoothIRT [27,28] package contains 379 students on 100 items. The data stem from an exam from a psychology class. The lower-right panel in Figure 4 shows some notable deviations from the regression line, and the determination coefficient $R^2$ was 0.996 for the optimal value of $K=13$. The estimated SD of $\eta$ from the RQM was 0.246. The determination coefficient from the regression model of weights ${\mathrm{\delta }}_i$ from classical D-scoring onto weights $a_i$ from the 2PL model was $R^2=0.493$. The classical D-score correlated with the RQM score at 0.9997, while the correlation with the 2PL score was much lower at 0.9551.

It can be shown that item weights ${\mathrm{\delta }}_i$ are a nonlinear function of 2PL item parameters $a_i$ and $b_i$ [29]. Therefore, an anonymous reviewer argued that a low correlation between ${\mathrm{\delta }}_i$ and $a_i$ is expected. We agree with such a view. However, such an observation questions the view of why one should ever believe that the classical D-score ${\mathfrak{D}}_c$ approximates the latent D-score $\mathcal{D}$ (or the other way around).

It was pointed out by an anonymous reviewer that the correlations between the classical D-scores and the 2PL scores (i.e., latent D-score) were quite high because they ranged between 0.9072 and 0.9938, with the exception of the third dataset ($r=0.8742$) that had the smallest number of items. However, we disagree with the anonymous reviewer that these correlation sizes would imply that classical and latent D-scores provide similar findings. In contrast, the item weights implied by the different scoring rules are only weakly related.

6. Discussion

This article searched for an IRT model that follows the principle of classical D-scoring of Dimitrov. In this approach, more difficult items receive higher weights in the classical D-score, which is a weighted sum score. It was shown that a variant of the RQM proves useful in this regard. In the RQM, a weighted sum score is a sufficient statistic for the latent variable contained in this model. Furthermore, in our proposed model, more difficult items are strongly weighted in the sum score. We demonstrated the adequacy of the RQM through five example datasets. Because the weights in the classical D-score were very well predicted by weights obtained from the RQM, one can interpret the RQM as a latent mirror of the D-scoring model. Although the weights were not perfectly predicted in all datasets, one can at least say that the RQM serves as a much more appropriate latent analog of the classical D-scoring model than the latent D-scoring model originally proposed by Dimitrov.

The classical and latent D-scoring methods were applied to important fields in educational measurement. Research was conducted in the areas of differential item functioning [30], test equating and linking [31], multistage testing [32], and standard setting [33]. The proposed methodology can be compared in future research with appropriate adaptations that involve the Ramsay quotient model.