Bias and Linking Error in Fixed Item Parameter Calibration

Abstract

The two-parameter logistic (2PL) item response theory (IRT) model is frequently applied to analyze group differences for multivariate binary random variables. The item parameters in the 2PL model are frequently fixed when estimating the mean and the standard deviation for a group of interest. This method is also called fixed item parameter calibration (FIPC). In this article, the bias and the linking error of the FIPC approach are analytically derived in the presence of random uniform differential item functioning (DIF). The adequacy of the analytical findings was validated in a simulation study. It turned out that the extent of the bias and the variance in distribution parameters increases with increasing variance of random DIF effects.

1. Introduction

Item response theory (IRT) models [1,2,3,4,5,6,7] represent a statistical framework to model multivariate discrete random variables. IRT models are frequently employed in the context of educational or psychological research. Let $\mathit{X}=(X_1,\dots ,X_I)$ be the vector of $I\in \mathbb{N}$ dichotomous (i.e., binary) random variables $X_i\in \{0,1\}$ that are typically referred to as (scored) items (or scored item responses). A unidimensional IRT model [8,9] is a statistical model for the probability distribution $\mathsf{P}(\mathit{X}=\mathit{x})$ for $\mathit{x}=(x_1,\dots ,x_I)\in {\{0,1\}}^I$, where

$${\displaystyle \mathsf{P}(\mathit{X}=\mathit{x};\mathit{\delta },\mathit{\gamma })=\int \prod _{i=1}^I\left[{P_i(\theta ;{\mathit{\gamma }}_i)}^{x_i}{\left(1-P_i(\theta ;{\mathit{\gamma }}_i)\right)}^{1-x_i}\right]\varphi (\theta ;\mu ,\sigma )\mathrm{d}\theta .}$$

(1)

In (1), $\varphi$ denotes the density of the univariate normal distribution with mean $\mu$ and standard deviation $\sigma$. The parameters of the distribution for the latent (factor) variable $\theta$, also labeled as trait or ability, define the parameter of interest $\mathit{\delta }=(\mu ,\sigma )$. The vector $\mathit{\gamma }=({\mathit{\gamma }}_1,\dots ,{\mathit{\gamma }}_I)$ collects all estimated item parameters of item response functions (IRF) $P_i(\theta ;{\mathit{\gamma }}_i)=\mathsf{P}\left(X_i=1\right|\theta )$ ($i=1,\dots ,I$). The two-parameter logistic (2PL) model [10,11,12] has the IRF:

$$P_i(\theta ;{\mathit{\gamma }}_i)=\Psi \left(a_i(\theta -b_i)\right),$$

(2)

where $a_i$ and $b_i$ are the item discrimination and item difficulty, respectively, and the logistic distribution function $\Psi$ is given by the following:

$$\Psi \left(x\right)={\displaystyle \frac{1}{1+exp(-x)}}.$$

(3)

In this case, the item parameters in (1) are given as ${\mathit{\gamma }}_i=(a_i,b_i)$. The model parameters in the IRT model (1) are frequently estimated using (marginal) maximum likelihood estimation [13,14].

IRT models are frequently used to compare the distribution of a particular group, such as a country, in a test (i.e., on a set of items) with a reference distribution with respect to the factor variable $\theta$ in the IRT model (1). Fixed item parameter calibration (FIPC, [15,16,17,18,19]) is frequently use. It assumes that item parameters are known (or estimated) in the total sample, and only the distribution parameters must be estimated. The application of the FIPC method using the 2PL model can be assumed to provide consistent distribution parameter estimates if the item parameters also hold for the group under study. However, it could be that the item parameters were group-specific. This property is referred to as differential item functioning (DIF) in the literature (see, for example, [20,21,22,23,24]). It has been shown in the linking literature that DIF produces additional variability in the estimated mean $\mu$ and standard deviation $\sigma$ [25,26,27,28,29,30,31]. Moreover, it has been demonstrated that the occurrence of DIF can introduce bias in the distribution parameters [32,33]. In particular, bias in FIPC due to random DIF can emerge, because a misspecified IRF is assumed that ignores DIF effects. Importantly, bias in distribution parameter estimates emerges, even if DIF effects are random (i.e., average to zero in the population). Moreover, the presence of random DIF also includes additional variability of the FIPC method.

In this article, the performance of FIPC is investigated in the presence of random DIF [34,35,36,37,38,39,40]. An analytical derivation and a simulation study show that biased estimates can result in the presence of random DIF. The analytical derivation utilizes standard techniques from calculus. However, to our knowledge, such a derivation of the bias and the variance of the FIPC estimates in the presence of DIF have not yet been presented in the literature.

The rest of this article is organized as follows. Section 2 presents an analytical derivation of the bias and the variance of the parameter estimate of the FIPC method. Section 3 reports the findings of a simulation study that aims to confirm the analytical findings of Section 2. Finally, the article closes with a discussion in Section 4.

2. Analytical Derivation of Bias and Linking Error

In this section, we compute approximations of the bias and the variance (i.e., the square of the linking error) for the FIPC method in the presence of random uniform DIF. The derivation operates on the assumption of an infinite sample size of persons. Hence, the effects of sampling error (i.e., sampling of subjects) are neglected.

Let ${\mathit{x}}_h=(x_{h1},\dots ,x_{hI})\in {\{0,1\}}^I$ be a vector of item responses. The index h of an item response pattern can be defined as $h={\sum }_{i=1}^I{2}^i{x}_{hi}$, where $h=1,\dots ,2^I$. In total, there are $H=2^I$ item response patterns for I items. The likelihood of the item response pattern ${\mathit{x}}_h$ is given by (see [41])

$$\mathsf{Like}({\mathit{x}}_h,\mu ,\sigma ,\mathit{a},\mathit{b})=\sum _{t=1}^T{f}_t\prod _{i=1}^I{p}_{it}^{x_{hi}}{(1-p_{it})}^{1-x_{hi}}\mathrm{with}{p}_{it}=\Psi \left(a_i(\sigma {\theta }_t+\mu -b_i)\right),$$

(4)

where $f_t$ are known weights that are proportional to the standard normal distribution, and the likelihood (4) is evaluated at a discrete grid ${\theta }_t$ for $t=1,\dots ,T$ of $\theta$ points. The distribution parameters consists of the mean $\mu$ and the standard deviation $\sigma$. $\mathit{a}=(a_1,\dots ,a_I)$ is the vector of item discriminations, and $\mathit{b}=(b_1,\dots ,b_I)$ is the vector of item difficulties. Note that $\mu$ and $\sigma$ also enter item response probabilities in (4) because the prior probability weights $f_t$ follow a standard normal distribution.

The item response data are generated by additionally assuming uniform DIF effects. The probability of a pattern ${\mathit{x}}_h$ is given by the following (see [41]):

$$w_h=w_h\left(\mathit{e}\right)=\mathsf{P}(\mathit{X}={\mathit{x}}_h)=\mathsf{Like}({\mathit{x}}_h,{\mu }^{*},{\sigma }^{*},{\mathit{a}}^{*},{\mathit{b}}^{*}+\mathit{e}),$$

(5)

and the notation ${\mu }^{*}$ and ${\sigma }^{*}$ indicates that the distribution parameters are fixed for data generation. Note that $w_h$ is a function of random uniform DIF effects $e_i$, where $\mathit{e}=(e_1,\dots e_I)$.

The log-likelihood function l is used to obtain parameter estimates $\widehat{\mathit{\delta }}$ for the population parameter $\mathit{\delta }=(\mu ,\sigma )$. The log-likelihood function is given by the following:

$$l(\mathit{\delta },\mathit{e})=\sum _{h=1}^H{w}_{h}log{L}_h=\sum _{h=1}^H{w}_h\left(\mathit{e}\right)log{L}_h\left(\mathit{\delta }\right),\mathrm{where}$$

(6)

$$L_h=L_h\left(\mathit{\delta }\right)=\mathsf{Like}({\mathit{x}}_h,\mu ,\sigma ,{\mathit{a}}^{*},{\mathit{b}}^{*}).$$

(7)

The parameter estimate is obtained as the maximizer of l, which means that $\widehat{\mathit{\delta }}$ is the root of a score equation:

$$\widehat{\mathit{\delta }}=\underset{\mathit{\delta }}{arg\; min}l(\mathit{\delta },\mathit{e})\mathrm{such}\mathrm{that}{l}_{\mathit{\delta }}(\widehat{\mathit{\delta }},\mathit{e})=0.$$

(8)

Importantly, DIF effects only affect probabilities $w_h$, and the DIF effects are not modelled.

We now derive an approximation of the parameter estimate $\widehat{\mathit{\delta }}$ as a function of random uniform DIF effects $\mathit{e}$. A Taylor expansion of partial derivative $l_{\mathit{\delta }}$ in (8) is utilized in the approximation (see [42] for an extensive usage of this technique). We assume independent random uniform DIF effects with $\mathsf{E}\left(e_i\right)=0$ and DIF variance $\mathsf{E}\left(e_i^2\right)={\tau }^2$ (i.e., a standard deviation $\mathsf{SD}\left(e_i\right)=\tau$; see [38,43]) and use a Taylor expansion around $\mathit{\delta }=\widehat{\mathit{\delta }}$ and $\mathit{e}=\mathbf{0}$

$$l_{\mathit{\delta }}(\widehat{\mathit{\delta }},\mathit{e})=l_{\mathit{\delta }}(\mathit{\delta },\mathbf{0})+l_{\mathit{\delta }\mathit{\delta }}(\mathit{\delta },\mathbf{0})(\widehat{\mathit{\delta }}-\mathit{\delta })+\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i}(\mathit{\delta },\mathbf{0})e_i+{\displaystyle \frac{1}{2}}\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}(\mathit{\delta },\mathbf{0})e_i^2.$$

(9)

Because we also have $l_{\mathit{\delta }}(\mathit{\delta },\mathbf{0})=\mathbf{0}$ (i.e., the true parameter $\mathit{\delta }$ is obtained if DIF effects are absent), we arrive at the following:

$$\widehat{\mathit{\delta }}-\mathit{\delta }=-l_{\mathit{\delta }\mathit{\delta }}^{-1}\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i}{e}_i-{\displaystyle \frac{1}{2}}{l}_{\mathit{\delta }\mathit{\delta }}^{-1}\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}{e}_i^2,$$

(10)

while omitting arguments in the corresponding partial derivatives. Then, we can obtain the expected bias (due to $\mathsf{E}\left(e_i\right)=0$ and $\mathsf{E}\left(e_i^2\right)={\tau }^2$) from (10)) by taking the expectation operator as follows:

$$\mathsf{Bias}\left(\widehat{\mathit{\delta }}\right)=-{\displaystyle \frac{{\tau }^2}{2}}{\left({\displaystyle \frac{1}{I}}{l}_{\mathit{\delta }\mathit{\delta }}\right)}^{-1}\left({\displaystyle \frac{1}{I}}\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}\right).$$

(11)

Note that because $l_{\mathit{\delta }\mathit{\delta }}$ and ${\sum }_{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}$ both increases with I, the bias is asymptotically independent of the number of items I. The variance of $\widehat{\mathit{\delta }}$ can be computed as follows:

$$\mathsf{Var}\left(\widehat{\mathit{\delta }}\right)={\displaystyle \frac{1}{I}}{\tau }^2{\left({\displaystyle \frac{1}{I}}{l}_{\mathit{\delta }\mathit{\delta }}\right)}^{-1}\left({\displaystyle \frac{1}{I}}\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i}{l}_{\mathit{\delta }{e}_i}^{\top }\right){\left({\displaystyle \frac{1}{I}}{l}_{\mathit{\delta }\mathit{\delta }}\right)}^{-\top }.$$

(12)

The linking error is is defined as the square root of the variance [26].

We can now more explicitly compute the appearing terms in (11) and (12). The partial derivatives of l with respect to $\mathit{\delta }$ are given by the following:

$$l_{\mathit{\delta }}=\sum _{h=1}^H{w}_h{\displaystyle \frac{\partial log{L}_h}{\partial \mathit{\delta }}}\mathrm{and}{l}_{\mathit{\delta }\mathit{\delta }}=\sum _{h=1}^H{w}_h{\displaystyle \frac{{\partial }^{2}log{L}_h}{\partial \mathit{\delta }\partial {\mathit{\delta }}^{\top }}},$$

(13)

while the partial derivatives (denoted by the operator ∂) with respect to uniform DIF effects $e_i$ can be computed as follows:

$$l_{\mathit{\delta }{e}_i}=\sum _{h=1}^H{\displaystyle \frac{\partial w_h}{\partial e_i}}{\displaystyle \frac{\partial log{L}_h}{\partial \mathit{\delta }}}\mathrm{and}{l}_{\mathit{\delta }{e}_i{e}_i}=\sum _{h=1}^H{\displaystyle \frac{{\partial }^2{w}_h}{\partial e_i\partial e_i}}{\displaystyle \frac{\partial log{L}_h}{\partial \mathit{\delta }}}.$$

(14)

In detail, the estimating equation for $\widehat{\mu }$ is given by the following:

$${\displaystyle \frac{\partial log{L}_h}{\partial \mu }}={\displaystyle \frac{{\displaystyle \frac{\partial L_h}{\partial \mu }}}{L_h}}={\displaystyle \frac{{\displaystyle \sum _{t=1}^T\frac{\partial L_{ht}}{\partial \mu }}}{L_h}}={\displaystyle \frac{{\displaystyle \sum _{t=1}^T{L}_{ht}{G}_{\mu ,ht}}}{L_h}},\mathrm{where}$$

(15)

$$L_{ht}=f_t\prod _{i=1}^I{p}_{it}^{x_{hi}}{(1-p_{it})}^{1-x_{hi}}\mathrm{and}{G}_{\mu ,ht}=\sum _{i=1}^I{a}_i(x_{hi}-p_{it}).$$

(16)

Moreover, we obtain the estimating equation for $\widehat{\sigma }$ as follows:

$${\displaystyle \frac{\partial log{L}_h}{\partial \sigma }}={\displaystyle \frac{{\displaystyle \sum _{t=1}^T{L}_{ht}{G}_{\sigma ,ht}}}{L_h}},\mathrm{where}{G}_{\sigma ,ht}=\sum _{i=1}^I{\theta }_t{a}_i(x_{hi}-p_{it}).$$

(17)

The second-order derivatives in $l_{\mathit{\delta }\mathit{\delta }}$ are given as follows:

$${\displaystyle \frac{{\partial }^{2}log{L}_h}{\partial \mu \partial \mu }}={\displaystyle \frac{{\displaystyle \left(\sum _{t=1}^T{L}_{ht}\left(G_{\mu ,ht}^2+G_{\mu \mu ,ht}\right)\right)L_h-{\left(\sum _{t=1}^T{L}_{ht}{G}_{\mu ,ht}\right)}^2}}{L_h^2}}\mathrm{with}{G}_{\mu \mu ,ht}=-\sum _{i=1}^I{a}_i^2{p}_{it}(1-p_{it}),$$

(18)

$${\displaystyle \frac{{\partial }^{2}log{L}_h}{\partial \sigma \partial \sigma }}={\displaystyle \frac{{\displaystyle \left(\sum _{t=1}^T{L}_{ht}\left(G_{\sigma ,ht}^2+G_{\sigma \sigma ,ht}\right)\right)L_h-{\left(\sum _{t=1}^T{L}_{ht}{G}_{\sigma ,ht}\right)}^2}}{L_h^2}}\mathrm{with}{G}_{\sigma \sigma ,ht}=-\sum _{i=1}^I{a}_i^2{\theta }_t^2{p}_{it}(1-p_{it})\mathrm{and}$$

(19)

$${\displaystyle \frac{{\partial }^{2}log{L}_h}{\partial \mu \partial \sigma }}={\displaystyle \frac{{\displaystyle \left(\sum _{t=1}^T{L}_{ht}\left(G_{\mu ,ht}{G}_{\sigma ,ht}+G_{\mu \sigma ,ht}\right)\right)L_h-\left(\sum _{t=1}^T{L}_{ht}{G}_{\mu ,ht}\right)\left(\sum _{t=1}^T{L}_{ht}{G}_{\sigma ,ht}\right)}}{L_h^2}},$$

(20)

where $G_{\mu \sigma ,ht}=-{\sum }_{i=1}^I{a}_i^2{\theta }_t{p}_{it}(1-p_{it})$. Furthermore, we now compute the derivatives of $w_h$ with respect to $e_i$. We obtain the following for the derivatives evaluated at $e_i=0$:

$${\displaystyle \frac{\partial w_h}{\partial e_i}}=a_i\sum _{t=1}^T{L}_{ht}{p}_{it}^{1-x_{hi}}{(1-p_{it})}^{x_{hi}}{(-1)}^{x_{hi}}\mathrm{and}{\displaystyle \frac{{\partial }^2{w}_h}{\partial e_i\partial e_i}}=-a_i^2\sum _{t=1}^T{L}_{ht}{p}_{it}^{1-x_{hi}}{(1-p_{it})}^{x_{hi}}(1-2p_{it}){(-1)}^{x_{hi}}.$$

(21)

It appears that the bias-determining term in (11) is mainly affected by partial derivatives with respect to the standard deviation $\sigma$. The bias-determining term can be computed as follows:

$$\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}=\sum _{i=1}^I{\displaystyle \frac{{\partial }^{3}l}{\partial \sigma \partial e_i\partial e_i}}=\sum _{i=1}^I\left(\sum _{h=1}^H{\displaystyle \frac{{\partial }^2{w}_h}{\partial e_i\partial e_i}}{\displaystyle \frac{\partial log{L}_h}{\partial \sigma }}\right)=\sum _{h=1}^H{\displaystyle \frac{\partial log{L}_h}{\partial \sigma }}\sum _{i=1}^I{\displaystyle \frac{{\partial }^2{w}_h}{\partial e_i\partial e_i}}.$$

(22)

This expression can be simplified to the following:

$$\sum _{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}=-\sum _{h=1}^H\left[\left(\sum _{t=1}^T{\displaystyle \frac{L_{ht}}{L_h}}\sum _{i=1}^I{a}_i{\theta }_t(x_{hi}-p_{it})\right)\left(\sum _{t=1}^T{L}_{ht}\sum _{i=1}^I{a}_i^2{p}_{it}^{1-x_{hi}}{(1-p_{it})}^{x_{hi}}(1-2p_{it}){(-1)}^{x_{hi}}\right)\right].$$

(23)

It is difficult to obtain clear insights in the expression (23). However, the sign of this term can be reasoned via simplified assumptions. Assume item difficulties $b_i$ are close to $\mu$. First, we investigate the first factor of (23) that involves the following term:

$${\alpha }_{hit}={\theta }_t(x_{hi}-p_{it}).$$

(24)

Assume that $p_{it}>0.5$. Then, we expect $x_{hi}=1$, and ${\theta }_t$ will typically be positive. In this case, the term ${\alpha }_{hit}$ will be positive. If $p_{it}<0.5$, we expect $x_{hi}=0$, and ${\theta }_t$ will typically be negative. Again, ${\alpha }_{hit}$ will be positive. Second, we investigate the second factor:

$${\beta }_{hit}=p_{it}^{1-x_{hi}}{(1-p_{it})}^{x_{hi}}(1-2p_{it}){(-1)}^{x_{hi}}.$$

(25)

If $p_{it}>0.5$, we expect $x_{hi}=1$, and ${\beta }_{hit}$ is positive. If $p_{it}<0.5$, we expect $x_{hi}=0$, which also implies that ${\beta }_{hit}$ is positive. Because there is a negative sign for ${\sum }_{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}$ in (23), we expect a negative contribution of ${\sum }_{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}$. Furthermore, the second-order derivative matrix $l_{\mathit{\delta }\mathit{\delta }}$ is negative definite for a maximizer. Moreover, from (11), we have the total bias contribution $-l_{\mathit{\delta }\mathit{\delta }}^{-1}{\sum }_{i=1}^I{l}_{\mathit{\delta }{e}_i{e}_i}$, which includes an additional negative sign. By taking these three negative signs into account, we reason that the bias in $\widehat{\mathit{\delta }}$ can be expected to be negative.

3. Simulation Study

3.1. Methods

The 2PL model was used as the IRT model to simulate item responses. We fixed the mean $\mu$ and the SD $\sigma$ for the normally distributed ability variable $\theta$ at 0.3 and 1.2, respectively.

The number I of items was fixed at 15 in the simulation. The item discriminations $a_i$ of the 15 items had a mean of 1.007 and a standard deviation of 0.225 and ranged between 0.6 and 1.3. The item difficulties $b_i$ had a mean of 0.000, a standard deviation of 1.272, and ranged between −2.0 and 2.0. The concrete item parameters can be found at https://osf.io/uf2xc (accessed on 4 August 2024). These item parameters were used in the FIPC scaling model.

For data generation using the 2PL model, random uniform DIF effects $e_i$ ($i=1,\dots ,I$) were added to item difficulties $b_i$. The standard deviation $\mathsf{SD}\left(e_i\right)=\tau$ was chosen at seven factor levels: 0 (indicating no DIF), 0.1, 0.2, 0.3, 0.4, 0.5, and 0.6. Note that random DIF effects were simulated in each replication of the simulation study.

Item responses were simulated according to the 2PL model for sample sizes N of 250, 500, 1000, 2000, and an infinite sample size of subjects (denoted by Inf). In the case of an infinite sample size, the probabilities for all $H=2^I$ item response patterns $\mathit{x}\in {\{0,1\}}^I$ according to the data-generating model were directly computed, and no item responses were simulated.

FIPC was applied as the analysis model in all conditions. The item parameters of the 2PL model were fixed at the prespecified $a_i$ and $b_i$ values, as described above.

In each of the 5 (sample size N) × 7 (DIF standard deviation $\tau$) = 35 cells of the simulation study, 20,000 replications were conducted. We computed the empirical bias and the root mean square error (RMSE) for the estimated mean $\widehat{\mu }$ and the estimated standard deviation $\widehat{\sigma }$ to assess the estimation quality of parameter estimates (see [44]).

The R software (version 4.4.1) [45] was used for the entire analysis in this simulation study. The 2PL model was fitted using the R package TAM [46]. Dedicated R functions for this simulation study were written by the author of this article. These functions and replication material for this Simulation study can be found at https://osf.io/uf2xc (accessed on 4 August 2024).

3.2. Results

Table 1. Simulation study: Bias and root mean square error (RMSE) for the estimate of the mean $\mu$ and standard deviation $\sigma$ as a function of the uniform DIF standard deviation $\tau$ and sample size N.

		Bias					RMSE
		N					N
Par	${\displaystyle \mathit{\tau }}$	250	500	1000	2000	Inf	250	500	1000	2000	Inf
$\mu$	0	−0.001	−0.001	0.000	0.000	0.000	$\phantom{-}$0.086	$\phantom{-}$0.061	$\phantom{-}$0.043	$\phantom{-}$0.030	$\phantom{-}$0.000
	0.1	0.000	0.000	0.000	0.000	−0.001	$\phantom{-}$0.091	$\phantom{-}$0.067	$\phantom{-}$0.051	$\phantom{-}$0.041	$\phantom{-}$0.027
	0.2	−0.003	−0.001	−0.002	−0.003	−0.002	$\phantom{-}$0.102	$\phantom{-}$0.081	$\phantom{-}$0.069	$\phantom{-}$0.062	$\phantom{-}$0.054
	0.3	−0.005	−0.004	−0.006	−0.005	−0.006	$\phantom{-}$0.118	$\phantom{-}$0.102	$\phantom{-}$0.091	$\phantom{-}$0.086	$\phantom{-}$0.081
	0.4	−0.010	−0.009	−0.008	−0.010	−0.010	$\phantom{-}$0.139	$\phantom{-}$0.124	$\phantom{-}$0.116	$\phantom{-}$0.111	$\phantom{-}$0.108
	0.5	−0.014	−0.014	−0.015	−0.015	−0.016	$\phantom{-}$0.160	$\phantom{-}$0.148	$\phantom{-}$0.142	$\phantom{-}$0.137	$\phantom{-}$0.134
	0.6	−0.020	−0.020	−0.021	−0.021	−0.022	$\phantom{-}$0.182	$\phantom{-}$0.171	$\phantom{-}$0.166	$\phantom{-}$0.163	$\phantom{-}$0.161
$\sigma$	0	−0.004	−0.002	−0.001	0.000	0.000	$\phantom{-}$0.075	$\phantom{-}$0.053	$\phantom{-}$0.038	$\phantom{-}$0.027	$\phantom{-}$0.000
	0.1	−0.005	−0.003	−0.002	−0.002	−0.001	$\phantom{-}$0.077	$\phantom{-}$0.055	$\phantom{-}$0.040	$\phantom{-}$0.031	$\phantom{-}$0.016
	0.2	−0.009	−0.007	−0.005	−0.005	−0.005	$\phantom{-}$0.081	$\phantom{-}$0.062	$\phantom{-}$0.049	$\phantom{-}$0.041	$\phantom{-}$0.032
	0.3	−0.013	−0.013	−0.011	−0.011	−0.010	$\phantom{-}$0.089	$\phantom{-}$0.071	$\phantom{-}$0.061	$\phantom{-}$0.054	$\phantom{-}$0.048
	0.4	−0.022	−0.021	−0.019	−0.019	−0.018	$\phantom{-}$0.098	$\phantom{-}$0.083	$\phantom{-}$0.074	$\phantom{-}$0.069	$\phantom{-}$0.064
	0.5	−0.033	−0.031	−0.030	−0.028	−0.029	$\phantom{-}$0.109	$\phantom{-}$0.096	$\phantom{-}$0.088	$\phantom{-}$0.083	$\phantom{-}$0.079
	0.6	−0.044	−0.043	−0.042	−0.041	−0.041	$\phantom{-}$0.121	$\phantom{-}$0.109	$\phantom{-}$0.102	$\phantom{-}$0.099	$\phantom{-}$0.096

Note. Par = parameter; Inf = infinite sample size; biases with absolute values larger than 0.010 are printed in bold font.

reports the bias and the RMSE for the estimates of the mean $\mu$ and the standard deviation $\sigma$. Overall, it turned out that the bias was larger for $\sigma$ than for $\mu$. Moreover, the bias increased as a function of the standard deviation $\tau$. The bias exceeded a non-negligible cutoff value of 0.010 (in absolute value) for $\mu$ for $\tau \ge 0.5$ and for $\sigma$ for $\tau \ge 0.3$. Also, the bias was slightly larger for smaller sample sizes compared to large sample sizes or an infinite sample size. For a fixed value of DIF standard deviation $\tau$, the RMSE was the smallest for infinite sample sizes. On the other hand, the RMSE was zero in the case of an infinite sample size of subjects (i.e., $N=\mathrm{Inf}$) and no DIF standard deviation (i.e., $\tau =0$).

The results in Table 1 are also graphically displayed in Figure 1. Notably, bias and RMSE for $\mu$ and $\sigma$ increase with increasing DIF standard deviation $\tau$. The bias is hardly affected by sample size. Moreover, the RMSE is almost perfectly linearly related to $\tau$ in an infinite sample size. Moreover, the RMSE increases with decreasing sample size N.

4. Discussion

Through analytical arguments and a simulation study, this article shows that the FIPC method provides biased estimates in the presence of random uniform DIF. Analytical arguments demonstrated that the bias was a function of the DIF standard deviation $\tau$. The analytically obtained results were also confirmed based on a simulation study. Overall, the bias in the estimated standard deviation $\widehat{\sigma }$ was larger than in the estimated mean $\widehat{\mu }$.

Note that we derived the bias of distribution parameter estimates under a random DIF assumption. The derivation of the bias under fixed DIF could also rely on the Taylor Expansion (9). However, it is likely that only the linear expansion with respect to $e_i$ must be considered because we now cannot assume $\mathsf{E}\left(e_i\right)=0$ as in the random DIF case. Note that identification assumptions must be imposed in the fixed DIF case (e.g., all DIF effects sum to zero; see [47,48,49]). This property illustrates the circularity of the DIF definition in the case of fixed DIF, and general statements of the bias of a linking method like FIPC crucially depend on the chosen identification assumptions imposed by individual researchers [50].

Our derivation of the bias and the variance only involved uniform DIF effects (i.e., DIF effects in item difficulty). However, the Taylor expansion could be extended to include nonuniform DIF effects (i.e., DIF effects in item discriminations). Future research might focus on generalizing the findings to the more general DIF case.

Moreover, we only investigated the 2PL model with random DIF. The findings regarding the FIPC method can most likely be transferred to more complex IRT models, such as the three-parameter logistic (3PL) models [51] or other IRT models [52,53,54].

It should be emphasized that our findings were also observed for the nonlinear Haebara and Stocking–Lord linking methods [32]. These linking methods (see [55,56,57,58] for general treatments) also result in biased mean and standard deviations in the presence of random uniform DIF. In contrast, the mean–geometric mean linking method [55,59] that directly operates on estimated item parameters typically provides unbiased estimates [32]. Except for very small sample sizes, we believe that mean–geometric mean linking should be the preferred scaling (i.e., linking) method in applications if researchers cannot exclude the situation of random DIF.

Our study did not consider the estimation of the variance of distribution parameters in FIPC estimation, which is important for statistical inference. Linking errors for the FIPC method can be obtained by using an item jackknife that repeatedly estimates the IRT model with FIPC while excluding one item or one group of items [60]. Computational shortcuts for item jackknife for the FIPC method using the 2PL model were discussed in [61]. In the statistical inference of distribution parameters, standard errors due to the sampling of persons and linking errors due to randomness in item parameters must be disentangled. Ref. [28] addressed this topic for linking methods, but the estimation technique does not apply to the FIPC method.

By interpreting the findings of FIPC and the Haebara and Stocking–Lord linking methods, it appears that the source of the bias is the attempt to align a misspecified item response function that contains DIF onto an item response function that does not represent the DIF in their item parameters. A misfitting item results in a smaller estimated item discrimination. Because the estimated standard deviation represents an average estimated item discrimination, the bias in the estimated mean and standard deviation can be explained as a consequence of item misfit. Hence, the bias of FIPC also transfers to the recalibration linking technique [62,63] that is used in some large-scale educational studies [64,65].

Our findings also shed some light on the practical applications of linking methods to educational large-scale assessment studies [66], like programme for international student assessment (PISA; [67]). Older PISA studies effectively implemented the FIPC method for country comparisons [68]. More recent PISA studies opted for a partial invariance approach in which only the largest DIF effects and country-specific item parameters were assumed in the 2PL model [68,69,70]. Hence, an item with a large DIF effect in one country is essentially excluded from country comparisons that involve this country [50]. Nevertheless, there is very likely random DIF in the remaining items, which will subsequently (slightly) bias the distribution parameters. Hence, there is no statistical evidence for preferring scaling models with invariant item parameters (i.e., using the FIPC method for a country) or partially invariant item parameters (i.e., using the FIPC method for a subset of items for a country) instead of linking methods that allow for DIF effects in all items. However, identification assumptions are imposed by model choice, and any assumption can, in principle, be defended due to the definitional ambiguity.