Scoring Individual Moral Inclination for the CNI Test

Abstract

Item response theory (IRT) is a modern psychometric framework for estimating respondents’ latent traits (e.g., ability, attitude, and personality) based on their responses to a set of questions in psychological tests. The current study adopted an item response tree (IRTree) method, which combines the tree model with IRT models for handling the sequential process of responding to a test item, to score individual moral inclination for the CNI test—a broadly adopted model for examining humans’ moral decision-making with three parameters generated: sensitivity to moral norms, sensitivity to consequences, and inaction preference. Compared to previous models for the CNI test, the resulting EIRTree-CNI Model is able to generate individual scores without increasing the number of items (thus, less subject fatigue or compromised response quality) or employing a post hoc approach that is deemed statistically suboptimal. The model fits the data well, and the subsequent test also supported the concurrent validity and the predictive validity of the model. Limitations are discussed further.

1. Introduction

1.1. Item Response Theory

Item response theory (IRT) is a modern psychometric framework for estimating respondents’ latent traits (e.g., ability, attitude, and personality) based on their responses to a set of questions in psychological tests. There are many different IRT models, each considering different facets of item characteristics (e.g., [1]). For example, the responses to a multiple-choice item in a math test can be graded, yielding a binary variable Y = 1 if the response is correct and Y = 0 if the response if not correct. The one-parameter logistic (1PL) IRT model [2] or Rasch model [3] assumes that the probability of answering this item correctly is a function of the respondent’s math ability $\theta$ and this item’s difficulty $\beta$, as defined in

$$P\left(Y=1|\theta ,\beta \right)={\displaystyle \frac{1}{1+\exp \left(\theta -\beta \right)}}$$

IRT models allow us to estimate all respondents’ ability based on their responses and also quantify the items’ characteristics, such as item difficulty.

In this paper, we adopted an item response tree (IRTree) method, which combines the tree model with IRT models for handling the sequential process of responding to a test item [4]. The IRTree models have gained increasing attention in psychology and social sciences. For example, they have been used to identify and accommodate different response styles [5], model answer change behaviors [6], and analyze multiple strategies used in problem solving [7,8]. Traditional IRTree models usually involve a single path to each observed score, but this study adopts an extended version of the IRTree method, similar to [8], allowing for two or more paths to the same observed options. This is referred to as the EIRTree model in this paper.

1.2. Moral Dilemmas and CNI Model

Moral dilemmas have a long history in moral philosophy. They have been used as an effective tool to test our moral intuition and reasoning, and establish principles for addressing difficult moral issues in the real world [9]. Think of this scenario:

You are a church official during the Nazi occupation of the Netherlands. You have an official agreement with the occupying forces that protects anyone under your care. You publicly spoke on behalf of a Jewish family, annoying the Nazis. To demonstrate their power, the Nazis demand that you kill the members of the Jewish family. Otherwise, they will execute dozens of Dutch people they have incarcerated for political reasons. Is it acceptable in this case to kill the Jewish family [10]?

When faced with such a problem, responders usually hesitate to give an answer [11], or struggle with the rationale of their decision [12]. In moral philosophy, an answer of “Yes” to the above question would be classified as utilitarian in nature, which means adopting an action that promotes the greatest good (saving more people). The utilitarian principle holds that moral action is right and only right if it leads to the greatest good for the greatest number of people [13]. It reflects how sensitive the individual is to the consequence of the action. Whereas “No” would be considered a deontological response, which means refusing to choose a morally wrong action—killing. Deontology deems action as more important than consequence [14]: certain rights or duties (e.g., do not harm) must be respected in all scenarios. In other words, the deontological preference indicates norm sensitivity.

These moral dilemmas were extensively adopted as a research instrument by moral psychologists subsequent to the work of [15], who designed a series of such dilemmas to investigate the neural underpinnings of and the dilemma characteristic’s influence on the role of reason and emotion in moral judgment. However, one drawback of this pervasively utilized measurement is that all dilemmas here apply only proscriptive moral norms (that is, if participants make a decision based on the moral norm, they should always answer “no”), and the benefits of action always outweigh its drawbacks (answering “yes” will always produce a better overall result than “no”—such as saving a greater number of human lives). This goes against the principles of experimental design in psychology as “the central aspects of utilitarianism and deontology—consequences and norms—are not manipulated” [16] (p. 343). Moreover, the scoring system results in mistakenly treating the utilitarian and deontological approaches to moral judgment as the two ends of a unilateral binary: the more a participant chooses “yes”, the more utilitarian and less deontological they will be scored. This may be caused by confusing philosophical concepts with psychological processes. Though the utilitarian and deontological views are opposing to each other in moral philosophy, the psychological process between individuals picking each moral view when resolving a moral dilemma should not be seen as such binary opposition [16]. Instead, there are several separate independent decision-making processes [17,18] that most people would go through, but to varying degrees or in different directions, such as the sensitivity to moral norm (conceptually align with deontology) and the sensitivity to consequence (conceptually align with utilitarianism) [16].

Gawronski and colleagues [16] designed a multinominal model called the CNI model to address the above problems. Their contribution was mainly two-fold. Firstly, they designed a new set of items, including six scenarios that varied in four editions: (a) the action (“yes” answer) is moral and leads to a larger benefit; (b) the action is moral but leads to a smaller benefit; (c) the action is not moral and leads to a smaller benefit; (d) the action is not moral but leads to a larger benefit. Secondly, the new model dissociated the utilitarian–deontological inclination, yielding C (Sensitivity to Consequences) and N (Sensitivity to Norm) scores separately, plus an I score (Inaction Preference). The I score separates human beings’ general tendency towards inaction when faced with moral problems [19]. This separation is necessary for psychological study on moral judgment. One typical evidence of this human tendency is that people perceived the same harm as more severe if it was caused by action rather than inaction [20].

Considering the previous example of the Nazi occupation dilemma, the answer “No” conflates the tendency towards inaction with deontological decision or sensitivity to norms. People may choose not to take an action (“No”) simply because they find it difficult to decide. Similarly, the “Yes” judgment conflates participants’ predisposition toward action with utilitarian decision or sensitivity to consequence. For people who answer mostly “yes” (utilitarian judgment), besides having a utilitarian inclination, it is possible that they simply choose harmful actions, regardless of their consequences. Thus, it is necessary to dissociate one’s tendency of inaction from both the deontological and utilitarian tendencies, as the CNI model does.

Gawronski et al.’s [16] model offers methodological benefits. For example, with traditional moral dilemma, previous studies found that a high psychopathy level is associated with a preference for utilitarian over deontological judgment [21,22,23]. Further studies found the ascription for this finding might be that psychopathic participants are less likely to adverse action [23] or that they have less reduced negative emotional reactivity to harmful acts [24], which manifests as less deontological. However, the deontological tendency could not be clearly and directly addressed here as the traditional moral dilemma treats utilitarian and deontological tendencies as pure inverses of each other rather than as separated inclinations. Adding that the commonly possessed inaction preference was not taken into consideration in the traditional moral dilemma; it constrained a direct and comprehensive exploration of the fundamental rationale. Using the CNI model, scholars were able to bridge this void, revealing that that higher level of psychopathy is associated with lower levels of consequence sensitivity, norm sensitivity, and inaction preference [10,16,25].

The CNI model has been broadly adopted since its inception. In addition to the studies on moral dilemma judgment and psychopathy, researchers have employed the CNI model to examine the association between moral dilemma judgments and incidental emotion [26], foreign language [27,28], chronic stress [29], political ideology [30], and personality traits [31], just to name a few.

Despite its theoretical contribution and broad adoption, the wider application of the CNI model is hindered by one major methodological limitation: inability to generate individual-level scores [16]. Since the CNI model adopts a multinomial processing tree (MPT), which is a stochastic model that analyzes categorical data to measure cognitive processes in human behavior [32], a relatively large number of trails is required for individual measurement (for a more detailed discussion, see the “Limitation” section in [16]). In other words, parameter estimation in MPT from individual levels using 24 trails (the number of items in the CNI test) was unreliable. This drawback limits the adoption of the CNI model to between-subject designs only, and any correlational studies are not feasible. Moreover, adopting the MPT model for the CNI model estimate may not be robust enough—two studies on them showed a marginal level of statistical significance (i.e., Study 2a) and a statistically significant deviation (i.e., Study 3a) between the actual and predicted probabilities.

Gawronski and colleagues proposed a solution to enhance individual-level parameter estimation by doubling test items [10]. The drawbacks of their solution were that it was implemented in a post hoc manner and that increasing the test length may lower the quality of the responses [33]. Liu and Liao [34] made another attempt to address this limitation by changing the algorithm named CAN. Because the CAN algorithm is theoretically based on the CNI model, which uses MPT, the CAN algorithm also suffers from parameter deviation for a small number of trails. In addition, the CAN algorithm lacks a statistical index for detecting measurement errors. Liu and Liao [34] could not determine the underlying cause for a lack of correlation between the traits and estimated parameters as it remained unclear if this lack of correlation was due to measurement errors or a true lack of correlation.

To overcome the methodological limitations of the MPT model and the CAN algorithm when analyzing the CNI moral dilemmas, the current study adopted an extended item response tree (EIRTree) model (discussed further below). As a measurement model, the EIRTree model is very flexible and has been applied in self-reported public policy decision making [35] and diagnosis of multiple strategies in educational assessments [8]. It allows for the estimation of individual scores directly, and researchers can compare different model specifications. We explored the viability of the EIRTree model with a set of published data on CNI ([10]; https://osf.io/ndf4w/ accessed on 3 July 2023) and tested the concurrent validity of scores obtained from the EIRTree model by examining its correlation with the original CNI model. The predictive validity was also tested via the association between the EIRTree-CNI model parameters and psychopathy.

2. Methods

2.1. The CNI Model

As above introduced, the CNI test includes four types of items which are the products of two dimensions: Norm (Proscriptive or Prescriptive) and Consequence (Larger or Smaller benefits compared to costs). An example of a CNI test item, the church official scenario, can be found in the beginning of Introduction. Figure 1 shows the hypothetical paths of participants’ responses to each of the four types of items based on their drive to the response. This is where the implicit psychological tendencies (C, N, and I) are calculated based on explicit behavioral responses (action or inaction). By aggregating responses from multiple items, this model quantifies the extent to which the responders adopt each of three response patterns: (a) sensitive to consequence, (b) sensitive to norm, and (c) general action or inaction preference. Given different possibilities of thought processes when making a moral decision, we considered two plausible models: C→N→I and N→C→I. From a theoretical perspective, when deciding on a moral dilemma, it is equally possible that an individual first considers consequences or moral norm. However, we choose to follow Gawronski et al.’s [16] statement that “the I parameter should be set as the lowest one in the hierarchy” (p. 349) and examine only the two models rather than all six possible models involving permutations of the three variables.

2.2. The Proposed Approach

IRTree models are a type of IRT models that propose that the answers to polytomously scored items can be parametrized through a series of psychological processes. The categorical outcomes are modeled as a tree structure with sequentially unobserved nodes. Each node represents a subprocess with some internal outcomes that are modeled using IRT models [1]. The observed responses are the leaves, or endnotes, of the tree. Every leaf is produced through a specific path of the tree. Hence, the probability of an observed response corresponds to the probability of passing through the branches that lead to the leaf representing that response.

Take the CNI model of item type 2 (prosmJ) in Figure 1 as an example. There are three nodes, each with two internal outcomes. For the first node, denoted by $Y_1^{*}$, a respondent obtains an internal score of 1 if they choose “Inaction”, or 0 if they do not make a selection but move to the second node. The CNI model assumes that the tendency of sensitivity to consequences, which is unobserved and denoted by ${\theta }_C$, affects the selection of the respondent at the first node. The probability of choosing “Inaction” in the first node of item $j$ is defined using the one-parameter logistic (1PL) IRT model as in

$$P_j\left(Y_1^{*}=1|{\theta }_C,{\beta }_{j1}\right)={\displaystyle \frac{1}{1+\exp \left({\theta }_C-{\beta }_{j1}\right)}},$$

where ${\beta }_{j1}$ denotes item location parameter of the first node of item $j$. Similarly, the tendency of sensitivity to norm (${\theta }_N$ score) affects the selection of the respondent at the second node, and the probability of choosing “Inaction” in the second node of item $j$ is written by

$$P_j\left(Y_2^{*}=1|{\theta }_N,{\beta }_{j2}\right)={\displaystyle \frac{1}{1+\exp \left({\theta }_N-{\beta }_{j2}\right)}},$$

where ${\beta }_{j2}$ denotes the item location parameter of the second node of item $j$. For the third node, we assume that the tendency of inaction preference (${\theta }_I$ score) affects the selection of the respondent, and the probability of choosing “Inaction” is defined by

$$P_j\left(Y_3^{*}=1|{\theta }_I,{\beta }_{j3}\right)={\displaystyle \frac{1}{1+\exp \left({\theta }_I-{\beta }_{j3}\right)}},$$

where ${\beta }_{j3}$ denotes the item location parameter of the third node of item $j$. Based on the above specifications, the probability of choosing “Inaction” and “Action” on item $j$ can be calculated as

$$\begin{array}{ll}P\left(X_j=\mathrm{Inaction}|{\theta }_C,{\theta }_N,{\theta }_I\right)& \\ & =P_j\left(Y_1^{*}=1|{\theta }_C,{\beta }_{j1}\right)+\left[1-P_j\left(Y_1^{*}=1|{\theta }_C,{\beta }_{j1}\right)\right]P_j\left(Y_2^{*}=1|{\theta }_N,{\beta }_{j2}\right)\\ & +\left[1-P_j\left(Y_1^{*}=1|{\theta }_C,{\beta }_{j1}\right)\right]\left[1-P_j\left(Y_2^{*}=1|{\theta }_N,{\beta }_{j2}\right)\right]P_j\left(Y_3^{*}=1|{\theta }_I,{\beta }_{j3}\right)\mathrm{and}\end{array}$$

$$\begin{array}{ll}P\left(X_j=\mathrm{Action}|{\theta }_C,{\theta }_N,{\theta }_I\right)& \\ & =\left[1-P_j\left(Y_1^{*}=1|{\theta }_C,{\beta }_{j1}\right)\right]\left[1-P_j\left(Y_2^{*}=1|{\theta }_N,{\beta }_{j2}\right)\right]\left[1-P_j\left(Y_3^{*}=1|{\theta }_I,{\beta }_{j3}\right)\right]\end{array}$$

respectively. For other types of items and the NCI model, the EIRTree model’s specifications differ, but they can be defined in a similar way.

2.3. Data Analysis

We employed open-source data to test the proposed model ([10] https://osf.io/ndf4w/ accessed on 3 July 2023). Besides measures of moral dilemma judgment, the data also contain other variables such as participants’ psychopathy level. The data consist of four item types: (i) Item type 1: proscriptive norm (progrJ; benefit of action is larger than cost), (ii) Item type 2: proscriptive norm (prosmJ; benefit of action is smaller than cost), (iii) Item type 3: prescriptive norm (pregrJ; benefit of action is larger than cost), and (iv) Item type 4: prescriptive norm (presmJ; benefit of action is smaller than cost). A total of 161 participants responded to 24 items where each item type was measured by 6 items. For each item type, we fitted the EIRTree model for the CNI and NCI.

Our data analysis was conducted in several phases. The initial phase involves transforming the data format from long to wide and recoding each data point. All item responses in the original data were assigned scores of “49” and “21” to represent action and inaction, respectively. These scores were recoded with a “1” for action and a “0” for inaction.

The parameters of the EIRTree models were estimated with the Hamiltonian Monte Carlo algorithm, which is a Markov Chain Monte Carlo (MCMC) method based on No-U-Turn sampler (NUTS) [36,37], with the rstan package (v2.30.0; [38]). The code can be downloaded from https://osf.io/a382w/ (accessed on 17 September 2023).

Model fits for the CNI and NCI models were examined with the chi-square (${\chi }^2$) test statistics proposed by Béguin and Glas [39]. These authors proposed the computation of two different ${\chi }^2$ statistics. The first ${\chi }^2$ statistic (i.e., ${\chi }_0^2$) uses the observed data and the expected frequency distribution while the second ${\chi }^2$ statistic (i.e., ${\chi }_{rep}^2$) is derived from the frequency distribution of the replicated data and the expected frequency distribution. Using this approach, we reject the model when the posterior predictive p-value (PPP), which represents the proportion of replications where ${\chi }_{rep}^2>{\chi }_0^2$, reaches a significantly low value.

Additionally, we conducted a comparative analysis of the two models using the Leave-One-Out cross-validation (LOO) and the Widely Applicable Information Criterion (WAIC) [40,41]. The LOO entails the sequential elimination of individual data points from the dataset, followed by assessing the model’s predictive performance using the omitted data. This procedure yields significant insights into the model’s capacity to generalize to new data. In contrast, WAIC is an information criterion that balances the goodness of fit and complexity of the model. Considering their predictive accuracy and complexity, LOO and WAIC provide valuable tools for selecting the most suitable model.

Further, we conducted a concurrent validity based on Pearson’s bivariate correlations between scores on each new and original CNI model parameter. We followed the guidelines of [42] criteria: 0.10 (small), 0.30 (moderate), and 0.50 (large) to interpret the degree of association between these variables. The predictive validity was also examined by using Pearson bivariate correlations to test the relationship between each parameter of the new CNI model and the psychopathy level.

3. Results

Figure 2 displays the R-hat statistic for each model parameter. All parameters have R-hat values less than 1.01, indicating successful convergence and unbiased, consistent, and reliable MCMC estimates [43]. The chi-squared test-based posterior predictive check resulted in PPP values of 0.5278 and 0.0003 for the C→N→I and N→C→I models, respectively. Thus, the C→N→I model can fit data adequately but the N→C→I model cannot.

Table 1. Model comparison.

	ELPD LOO		ELPD WAIC
	Estimate	SE	Estimate	SE
Model 1 (CNI)	−1651.6	34.2	−1642.3	33.9
Model 2 (NCI)	−3894.1	81.0	−3585.5	75.6

Note. ELPD = expected log pointwise predictive density; SE = standard error; LOO = Leave-One-Out cross-validation; WAIC = Widely Applicable Information Criterion.

compares the two models based on the LOO and WAIC by calculating their expected log pointwise predictive density (ELPD) for the LOO or WAIC to determine their predictive accuracy. It can be observed that the C→N→I model had a higher ELPD, suggesting that it is preferred to the N→C→I model.

Table 2. Item parameters for IRTree models.

		CNI Model Parameters			NCI Model Parameters
Item Type	Item No.	${\mathit{\beta }}_{\mathit{j}1}$	${\mathit{\beta }}_{\mathit{j}2}$	${\mathit{\beta }}_{\mathit{j}3}$	${\mathit{\beta }}_{\mathit{j}1}$	${\mathit{\beta }}_{\mathit{j}2}$	${\mathit{\beta }}_{\mathit{j}3}$
progrJ: Proscriptive Norm— Benefit of action is larger than cost	1	0.37	−0.94	0.71	−0.65	1.63	−1.06
	2	2.37	−1.71	0.60	−0.04	−0.64	−0.71
	3	1.25	−1.14	1.54	−0.99	0.44	−0.67
	4	4.82	−1.19	1.83	0.54	−1.33	1.43
	5	1.76	0.45	−0.36	1.55	1.90	−0.91
	6	5.68	−0.71	−0.82	2.68	−1.45	−0.25
prosmJ: Proscriptive Norm— Benefit of action is smaller than cost	7	0.03	−0.16	2.57	−0.62	0.75	2.28
	8	2.77	−1.93	1.08	1.78	−1.64	0.92
	9	3.14	−0.86	1.54	2.07	−0.50	1.40
	10	1.42	−1.26	0.93	0.75	−0.85	0.70
	11	−0.36	−1.30	1.03	−0.73	−0.77	0.67
	12	0.90	−1.09	0.18	0.24	−0.75	0.24
pregrJ: Prescriptive Norm— Benefit of action is larger than cost	13	−0.70	0.25	−1.84	−1.25	1.24	−1.92
	14	3.73	−0.99	−1.34	2.69	−0.92	−1.47
	15	3.36	−0.24	−0.63	1.96	−0.02	−0.87
	16	−0.14	−0.84	−1.40	−0.77	−0.29	−1.47
	17	0.39	−1.22	−1.48	−0.47	−0.71	−1.70
	18	−1.34	−0.07	−2.35	−1.64	0.48	−2.35
presmJ: Prescriptive Norm— Benefit of action is smaller than cost	19	0.37	1.89	1.04	0.31	2.93	0.91
	20	1.56	−1.60	−0.96	−0.55	0.04	1.05
	21	0.80	−1.34	−1.55	−1.06	0.76	1.10
	22	5.02	−1.98	−1.84	0.12	−2.11	−1.54
	23	0.97	0.15	1.80	0.67	3.19	3.52
	24	4.52	−1.87	−2.13	−0.31	−1.82	−1.02

Note. ${\beta }_{j1}=$ location parameter of the first node of item j; ${\beta }_{j2}=$ location parameter of the second node of item j and ${\beta }_{j3}=$ location parameter of the third node of item j.

and

Table 3. Summary of person location estimates.

	CNI Model				NCI Model
	${\mathit{\theta }}_{\mathit{C}}$	${\mathit{\theta }}_{\mathit{N}}$	${\mathit{\theta }}_{\mathit{I}}$		${\mathit{\theta }}_{\mathit{N}}$	${\mathit{\theta }}_{\mathit{C}}$	${\mathit{\theta }}_{\mathit{I}}$
Mean	−0.26	0.12	0.01	−0.04	0.00	0.01	−0.26
Standard deviation	1.58	1.84	1.10	1.55	1.85	1.09	1.58
Minimum	−2.84	−3.74	−4.04	−3.34	−3.79	−3.87	−2.84
Maximum	4.56	2.75	2.48	3.47	3.49	2.32	4.56

Note. ${\theta }_C=$ person location at node $C$; ${\theta }_N=$ person location at node N and ${\theta }_I=$ person location at node I.

show the estimated location parameters for item type and item number and summarized person location estimates (i.e., tendency of sensitivity to consequences, norm, and inaction preference), respectively. The item location parameters (i.e., ${\beta }_{j1}$, ${\beta }_{j2}$, and ${\beta }_{j3}$) provide information on which item and node respondents are perceived as more challenging or easy. In other words, the item location parameters help determine respondents’ tendency to move from one node to another or make an “Action” or “Inaction” decision. Note that items with higher location parameters may have fewer respondents making an “Action” or “Inaction” decision. Results from person location assess respondents’ tendency to make decisions ($\theta$ score). Respondents with a higher $\theta$ score for sensitive to consequences, sensitive to norm, and inaction preference have higher tendencies to make moral decisions. Let us consider item 1 of progrJ (Proscriptive Norm-Benefits of action is larger than cost) as an example. In the CNI model, the mean person location for C (M = −0.26, SD = 1.58) is below the item location (${\beta }_{11}=0.37$); however, the discrepancy is relatively small (<0.5 logits). This suggests that, on average, respondents have a lower tendency to make an “action” decision on node C (sensitive to consequences) and are more likely to move to node N (sensitive to norm). At node N, the mean person location is 0.12, which is above the item location parameter (${\beta }_{12}=-0.94$). This provides evidence that, on average, respondents will make an “inaction” decision at node N. However, respondents with location at node N less than the item location estimate (i.e., ${\beta }_{12}=-0.94$) will proceed to node I. It should be noted that the average person location at node I is 0.01, which is significantly lower than the item location, ${\beta }_{13}=0.71$. This indicates that the average respondent will make an “Action” decision at node I.

Table 4. Results from Pearson’s bivariate correlations.

	C-Original	N-Original	I-Original	Psychopathy
C1	0.848 ***	0.144	0.146	−0.288 ***
N1	0.163 *	0.826 ***	0.237 **	−0.563 ***
I1	0.147	0.233 **	0.499 ***	−0.258 **

Note. * p < 0.05; ** p < 0.005; *** p < 0.001; C₁, N₁ and I₁ are CNI scores from EIRTree model.

shows the concurrent validity between the CNI scores from the original and the EIRTree models. The result shows that the estimated CNI scores from the EIRTree model have significantly strong positive correlations with the original CNI scores. This supports the concurrent validity of the EIRTree-CNI approach. Additionally, we explored the predictive validity of the CNI scores and psychopathy level (Table 4). The result suggests that psychopathy level is negatively associated with the estimated C, N, and I score from the EIRTree model, supporting predictive validity.

4. Discussion

The current study proposes to use the EIRTree model for the CNI test, which offers several advantages over existing methods. Firstly, it allows researchers to formally assess the fit of the model and the data, providing validity evidence about the internal structures. Secondly, it can estimate the C, N, and I scores for each individual while keeping the assessment relatively short, addressing a long-standing limitation of the CNI test. We applied the EIRTree model to a dataset that had been previously analyzed and tested for both the C→N→I and N→C→I model structures. The results indicate that the EIRTree model with the C→N→I structure fits data better than the model with the N→C→I structure. Thus, the final EIRTree-CNI model is based on the C→N→I structure, and its concurrent and predictive validity were tested and reported.

The performance of the EIRTree-CNI model over the NCI model provides several implications in the field of psychometrics and individual moral judgement that are worth considering. First, the performance of the EIRTree model applied on the CNI test suggests that the proposed model has an improved measurement accuracy. In other words, the model provides an accurate estimate of person and item parameters which help researchers to reliably distinguish between respondents with different levels of moral dilemmas. Second, insights from the CNI model can promote the development and testing of new items or calibration of existing items to scale. Additionally, the performance of the EIRTree-CNI can help researchers and moral dilemmas test developers to develop items that are more discriminating or provide better model data fit to the proposed model. It is important to note that moral decision-making processes can be challenging especially when it conflicts with respondents’ personal value, social and cultural influences, ethical principles, emotional involvement, and moral development as in the stage an individual can perceive and handle moral dilemmas [44,45]. With the EIRTree-CNI model, researchers and practitioners can make informed decisions about the response processes of respondents.

While the study developed a valuable tool for future moral psychological studies, it is imperative to recognize its inherent limitations, which provide avenues for future research to explore and address. Firstly, the current study adopts a one-parameter logistic IRT model for each node because of its simplicity and smaller sample size requirement. However, researchers can use other IRT models, such as a two-parameter logistic IRT model, which considers both node threshold and discrimination parameters and may, thus, improve model–data fit. Second, the EIRTree-CNI’s performance should be further examined using other datasets, along with a systematical investigation of the validity and reliability. Furthermore, a more comprehensive examination of the utility of the EIRTree model for the CNI test under varied conditions is needed. Further studies may consider using simulations to evaluate the accuracy of persons’ CNI scores under different sample sizes. It is expected that larger sample sizes reduce the likelihood of bias and improve measurement precision by increasing the likelihood that estimated parameters will converge to their true values [1]. However, the determination of a sample size at which accurate estimates are obtained is beyond the scope of the current paper.

5. Concluding Remarks

The current study used an item response theory approach to score the CNI test to address its main limitation of the inability to generate individual scores without adding item numbers and employing a post hoc approach. The EIRTree-CNI model generated under such effort opens greater possibility to statistical analyses such as correlation, regression, and traditional psychometric models like classical test theory and item response theory without potential subject fatigue and compromised response quality. The subsequent test supported the model fit, concurrent validity, and predictive validity of the model.