Psychometric Methods: Theory and Practice

Topic Information

Dear Colleagues,

Measurement and quantification are ubiquitous in modern society. The historical foundation of psychometrics arose from the need to measure human abilities through suitable tests. This discipline then underwent rapid conceptual growth due to the incorporation of advanced mathematical and statistical methods. Today, psychometrics not only covers virtually all statistical methods but also incorporates advanced techniques from machine learning and data mining that are useful for the behavioral and social sciences, including but not limited to the handling of missing data, the combination of multiple-source information with measured data, measurement obtained from special experiments, visualization of statistical outcomes, measurement that discloses underlying problem-solving strategies, and so on. Psychometric methods now have a wide range of applicability in various disciplines, such as education, psychology, social sciences, behavioral genetics, neuropsychology, clinical psychology, medicine, and even visual arts and music, to name a few.

The dramatic development of psychometric methods and rigorous incorporation of psychometrics, data science, and even artificial intelligence techniques in interdisciplinary fields have aroused significant attention and led to pressing discussions about the future of measurement.

The aim of this Special Topic is to gather studies on the latest development of psychometric methods covering a broad range of methods, from traditional statistical methods to advanced data-driven approaches, and to highlight discussions about different approaches (e.g., theory-driven vs. data-driven) to address challenges in psychometric theory and practice.

This Special Topic consists of two subtopics: (1) theory-driven psychometric methods that exhibit the advancement of psychometric and statistical modeling in measurement to contribute to the development of psychological and hypothetical theories; and (2) data-driven computational methods that leverage new data sources and machine learning/data mining/artificial intelligence techniques to address new psychometric challenges.

In this issue, we seek original empirical or methodological studies, thematic/conceptual review articles, and discussion and comment papers highlighting pressing topics related to psychometrics.

Interested authors should submit a letter of intent including (1) a working title for the manuscript, (2) names, affiliations, and contact information for all authors, and (3) an abstract of no more than 500 words detailing the content of the proposed manuscript to the topic editors.

There is a two-stage submission process. Initially, interested authors are requested to submit only abstracts of their proposed papers. Authors of the selected abstracts will then be invited to submit full papers. Please note that the invitation to submit does not guarantee acceptance/publication in the Special Topic. Invited manuscripts will be subject to the usual review standards of the participating journals, including a rigorous peer review process.

Dr. Qiwei He
Dr. Yunxiao Chen
Prof. Dr. Carolyn Jane Anderson
Topic Editors

Participating Journals

Journal Name	Impact Factor	CiteScore	Launched Year	First Decision (median)	APC
Behavioral Sciences behavsci	2.5	2.6	2011	28.7 Days	CHF 2200
Education Sciences education	2.5	4.8	2011	29.8 Days	CHF 1800
Journal of Intelligence jintelligence	2.8	2.8	2013	25.8 Days	CHF 2600

MDPI Topics is cooperating with Preprints.org and has built a direct connection between MDPI journals and Preprints.org. Authors are encouraged to enjoy the benefits by posting a preprint at Preprints.org prior to publication:

1. Immediately share your ideas ahead of publication and establish your research priority;
2. Protect your idea from being stolen with this time-stamped preprint article;
3. Enhance the exposure and impact of your research;
4. Receive feedback from your peers in advance;
5. Have it indexed in Web of Science (Preprint Citation Index), Google Scholar, Crossref, SHARE, PrePubMed, Scilit and Europe PMC.

Published Papers (10 papers)

Order results

Content type Publication Date

Result details

Normal Extended Compact

Journals

All Behavioral Sciences Education Sciences J. Intell.

Show export options expand_more Show export options expand_less

Select all

Export citation of selected articles as:

Plain Text BibTeX BibTeX (without abstracts) Endnote Endnote (without abstracts) Tab-delimited RIS

Error

Oops... you haven't selected anything for export.

Ok

clear

19 pages, 1958 KiB  

Open AccessEditor’s ChoiceArticle

Psychometric Modeling to Identify Examinees’ Strategy Differences during Testing

by Clifford E. Hauenstein, Susan E. Embretson and Eunbee Kim

J. Intell. 2024, 12(4), 40; https://doi.org/10.3390/jintelligence12040040 - 29 Mar 2024

Viewed by 1793

Abstract

Aptitude test scores are typically interpreted similarly for examinees with the same overall score. However, research has found evidence of examinee differences in strategies, as well as in the continued application of appropriate procedures during testing. Such differences can impact the correlates of [...] Read more.

Aptitude test scores are typically interpreted similarly for examinees with the same overall score. However, research has found evidence of examinee differences in strategies, as well as in the continued application of appropriate procedures during testing. Such differences can impact the correlates of test scores, making similar interpretations for equivalent scores questionable. This study presents some item response theory (IRT) models that are relevant to identifying examinee differences in strategies and understanding of test-taking procedures. First, mixture IRT models that identify latent classes of examinees with different patterns of item responses are considered; these models have long been available but unfortunately are not routinely applied. Strategy differences between the classes can then be studied separately by modeling the response patterns with cognitive complexity variables within each class. Secondly, novel psychometric approaches that leverage response time information (in particular, response time residuals) in order to identify both inter and intraindividual variability in response processes are considered. In doing so, a general method for evaluating threats to validity is proposed. The utility of the approach, in terms of providing more interpretable performance estimates and improving the administration of psychological measurement instruments, is then demonstrated with an empirical example. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

attachment

Supplementary material:
Supplementary File 1 (ZIP, 363 KiB)

17 pages, 1604 KiB  

Open AccessArticle

Explanatory Cognitive Diagnosis Models Incorporating Item Features

by Manqian Liao, Hong Jiao and Qiwei He

J. Intell. 2024, 12(3), 32; https://doi.org/10.3390/jintelligence12030032 - 11 Mar 2024

Viewed by 2077

Abstract

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, [...] Read more.

Item quality is crucial to psychometric analyses for cognitive diagnosis. In cognitive diagnosis models (CDMs), item quality is often quantified in terms of item parameters (e.g., guessing and slipping parameters). Calibrating the item parameters with only item response data, as a common practice, could result in challenges in identifying the cause of low-quality items (e.g., the correct answer is easy to be guessed) or devising an effective plan to improve the item quality. To resolve these challenges, we propose the item explanatory CDMs where the CDM item parameters are explained with item features such that item features can serve as an additional source of information for item parameters. The utility of the proposed models is demonstrated with the Trends in International Mathematics and Science Study (TIMSS)-released items and response data: around 20 item linguistic features were extracted from the item stem with natural language processing techniques, and the item feature engineering process is elaborated in the paper. The proposed models are used to examine the relationships between the guessing/slipping item parameters of the higher-order DINA model and eight of the item features. The findings from a follow-up simulation study are presented, which corroborate the validity of the inferences drawn from the empirical data analysis. Finally, future research directions are discussed. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

17 pages, 2377 KiB  

Open AccessArticle

A Comparative Study of Item Response Theory Models for Mixed Discrete-Continuous Responses

by Cengiz Zopluoglu and J. R. Lockwood

J. Intell. 2024, 12(3), 26; https://doi.org/10.3390/jintelligence12030026 - 25 Feb 2024

Viewed by 2238

Abstract

Language proficiency assessments are pivotal in educational and professional decision-making. With the integration of AI-driven technologies, these assessments can more frequently use item types, such as dictation tasks, producing response features with a mixture of discrete and continuous distributions. This study evaluates novel [...] Read more.

Language proficiency assessments are pivotal in educational and professional decision-making. With the integration of AI-driven technologies, these assessments can more frequently use item types, such as dictation tasks, producing response features with a mixture of discrete and continuous distributions. This study evaluates novel measurement models tailored to these unique response features. Specifically, we evaluated the performance of the zero-and-one-inflated extensions of the Beta, Simplex, and Samejima’s Continuous item response models and incorporated collateral information into the estimation using latent regression. Our findings highlight that while all models provided highly correlated results regarding item and person parameters, the Beta item response model showcased superior out-of-sample predictive accuracy. However, a significant challenge was the absence of established benchmarks for evaluating model and item fit for these novel item response models. There is a need for further research to establish benchmarks for evaluating the fit of these innovative models to ensure their reliability and validity in real-world applications. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

23 pages, 4874 KiB  

Open AccessArticle

Conditional Dependence across Slow and Fast Item Responses: With a Latent Space Item Response Modeling Approach

by Nana Kim, Minjeong Jeon and Ivailo Partchev

J. Intell. 2024, 12(2), 23; https://doi.org/10.3390/jintelligence12020023 - 16 Feb 2024

Viewed by 1881

Abstract

There recently have been many studies examining conditional dependence between response accuracy and response times in cognitive tests. While most previous research has focused on revealing a general pattern of conditional dependence for all respondents and items, it is plausible that the pattern [...] Read more.

There recently have been many studies examining conditional dependence between response accuracy and response times in cognitive tests. While most previous research has focused on revealing a general pattern of conditional dependence for all respondents and items, it is plausible that the pattern may vary across respondents and items. In this paper, we attend to its potential heterogeneity and examine the item and person specificities involved in the conditional dependence between item responses and response times. To this end, we use a latent space item response theory (LSIRT) approach with an interaction map that visualizes conditional dependence in response data in the form of item–respondent interactions. We incorporate response time information into the interaction map by applying LSIRT models to slow and fast item responses. Through empirical illustrations with three cognitive test datasets, we confirm the presence and patterns of conditional dependence between item responses and response times, a result consistent with previous studies. Our results further illustrate the heterogeneity in the conditional dependence across respondents, which provides insights into understanding individuals’ underlying item-solving processes in cognitive tests. Some practical implications of the results and the use of interaction maps in cognitive tests are discussed. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

32 pages, 24122 KiB  

Open AccessArticle

Biclustering of Log Data: Insights from a Computer-Based Complex Problem Solving Assessment

by Xin Xu, Susu Zhang, Jinxin Guo and Tao Xin

J. Intell. 2024, 12(1), 10; https://doi.org/10.3390/jintelligence12010010 - 17 Jan 2024

Cited by 1 | Viewed by 2273

Abstract

Computer-based assessments provide the opportunity to collect a new source of behavioral data related to the problem-solving process, known as log file data. To understand the behavioral patterns that can be uncovered from these process data, many studies have employed clustering methods. In [...] Read more.

Computer-based assessments provide the opportunity to collect a new source of behavioral data related to the problem-solving process, known as log file data. To understand the behavioral patterns that can be uncovered from these process data, many studies have employed clustering methods. In contrast to one-mode clustering algorithms, this study utilized biclustering methods, enabling simultaneous classification of test takers and features extracted from log files. By applying the biclustering algorithms to the “Ticket” task in the PISA 2012 CPS assessment, we evaluated the potential of biclustering algorithms in identifying and interpreting homogeneous biclusters from the process data. Compared with one-mode clustering algorithms, the biclustering methods could uncover clusters of individuals who are homogeneous on a subset of feature variables, holding promise for gaining fine-grained insights into students’ problem-solving behavior patterns. Empirical results revealed that specific subsets of features played a crucial role in identifying biclusters. Additionally, the study explored the utilization of biclustering on both the action sequence data and timing data, and the inclusion of time-based features enhanced the understanding of students’ action sequences and scores in the context of the analysis. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

22 pages, 548 KiB  

Open AccessArticle

Modeling Sequential Dependencies in Progressive Matrices: An Auto-Regressive Item Response Theory (AR-IRT) Approach

by Nils Myszkowski and Martin Storme

J. Intell. 2024, 12(1), 7; https://doi.org/10.3390/jintelligence12010007 - 15 Jan 2024

Cited by 1 | Viewed by 2265

Abstract

Measurement models traditionally make the assumption that item responses are independent from one another, conditional upon the common factor. They typically explore for violations of this assumption using various methods, but rarely do they account for the possibility that an item predicts the [...] Read more.

Measurement models traditionally make the assumption that item responses are independent from one another, conditional upon the common factor. They typically explore for violations of this assumption using various methods, but rarely do they account for the possibility that an item predicts the next. Extending the development of auto-regressive models in the context of personality and judgment tests, we propose to extend binary item response models—using, as an example, the 2-parameter logistic (2PL) model—to include auto-regressive sequential dependencies. We motivate such models and illustrate them in the context of a publicly available progressive matrices dataset. We find an auto-regressive lag-1 2PL model to outperform a traditional 2PL model in fit as well as to provide more conservative discrimination parameters and standard errors. We conclude that sequential effects are likely overlooked in the context of cognitive ability testing in general and progressive matrices tests in particular. We discuss extensions, notably models with multiple lag effects and variable lag effects. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

attachment

Supplementary material:
Supplementary File 1 (ZIP, 100 KiB)

23 pages, 5897 KiB  

Open AccessArticle

Using IRTree Models to Promote Selection Validity in the Presence of Extreme Response Styles

by Victoria L. Quirk and Justin L. Kern

J. Intell. 2023, 11(11), 216; https://doi.org/10.3390/jintelligence11110216 - 17 Nov 2023

Cited by 2 | Viewed by 2043

Abstract

The measurement of psychological constructs is frequently based on self-report tests, which often have Likert-type items rated from “Strongly Disagree” to “Strongly Agree”. Recently, a family of item response theory (IRT) models called IRTree models have emerged that can parse out content traits [...] Read more.

The measurement of psychological constructs is frequently based on self-report tests, which often have Likert-type items rated from “Strongly Disagree” to “Strongly Agree”. Recently, a family of item response theory (IRT) models called IRTree models have emerged that can parse out content traits (e.g., personality traits) from noise traits (e.g., response styles). In this study, we compare the selection validity and adverse impact consequences of noise traits on selection when scores are estimated using a generalized partial credit model (GPCM) or an IRTree model. First, we present a simulation which demonstrates that when noise traits do exist, the selection decisions made based on the IRTree model estimated scores have higher accuracy rates and have less instances of adverse impact based on extreme response style group membership when compared to the GPCM. Both models performed similarly when there was no influence of noise traits on the responses. Second, we present an application using data collected from the Open-Source Psychometrics Project Fisher Temperament Inventory dataset. We found that the IRTree model had a better fit, but a high agreement rate between the model decisions resulted in virtually identical impact ratios between the models. We offer considerations for applications of the IRTree model and future directions for research. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

17 pages, 660 KiB  

Open AccessArticle

Estimating the Multidimensional Generalized Graded Unfolding Model with Covariates Using a Bayesian Approach

by Naidan Tu, Bo Zhang, Lawrence Angrave, Tianjun Sun and Mathew Neuman

J. Intell. 2023, 11(8), 163; https://doi.org/10.3390/jintelligence11080163 - 14 Aug 2023

Cited by 5 | Viewed by 1779

Abstract

Noncognitive constructs are commonly assessed in educational and organizational research. They are often measured by summing scores across items, which implicitly assumes a dominance item response process. However, research has shown that the unfolding response process may better characterize how people respond to [...] Read more.

Noncognitive constructs are commonly assessed in educational and organizational research. They are often measured by summing scores across items, which implicitly assumes a dominance item response process. However, research has shown that the unfolding response process may better characterize how people respond to noncognitive items. The Generalized Graded Unfolding Model (GGUM) representing the unfolding response process has therefore become increasingly popular. However, the current implementation of the GGUM is limited to unidimensional cases, while most noncognitive constructs are multidimensional. Fitting a unidimensional GGUM separately for each dimension and ignoring the multidimensional nature of noncognitive data may result in suboptimal parameter estimation. Recently, an R package bmggum was developed that enables the estimation of the Multidimensional Generalized Graded Unfolding Model (MGGUM) with covariates using a Bayesian algorithm. However, no simulation evidence is available to support the accuracy of the Bayesian algorithm implemented in bmggum. In this research, two simulation studies were conducted to examine the performance of bmggum. Results showed that bmggum can estimate MGGUM parameters accurately, and that multidimensional estimation and incorporating relevant covariates into the estimation process improved estimation accuracy. The effectiveness of two Bayesian model selection indices, WAIC and LOO, were also investigated and found to be satisfactory for model selection. Empirical data were used to demonstrate the use of bmggum and its performance was compared with three other GGUM software programs: GGUM2004, GGUM, and mirt. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

attachment

Supplementary material:
Supplementary File 1 (ZIP, 276 KiB)

21 pages, 5191 KiB  

Open AccessArticle

Deterministic Input, Noisy Mixed Modeling for Identifying Coexisting Condensation Rules in Cognitive Diagnostic Assessments

by Peida Zhan

J. Intell. 2023, 11(3), 55; https://doi.org/10.3390/jintelligence11030055 - 16 Mar 2023

Cited by 1 | Viewed by 1916

Abstract

In cognitive diagnosis models, the condensation rule describes the logical relationship between the required attributes and the item response, reflecting an explicit assumption about respondents’ cognitive processes to solve problems. Multiple condensation rules may apply to an item simultaneously, indicating that respondents should [...] Read more.

In cognitive diagnosis models, the condensation rule describes the logical relationship between the required attributes and the item response, reflecting an explicit assumption about respondents’ cognitive processes to solve problems. Multiple condensation rules may apply to an item simultaneously, indicating that respondents should use multiple cognitive processes with different weights to identify the correct response. Coexisting condensation rules reflect the complexity of cognitive processes utilized in problem solving and the fact that respondents’ cognitive processes in determining item responses may be inconsistent with the expert-designed condensation rule. This study evaluated the proposed deterministic input with a noisy mixed (DINMix) model to identify coexisting condensation rules and provide feedback for item revision to increase the validity of the measurement of cognitive processes. Two simulation studies were conducted to evaluate the psychometric properties of the proposed model. The simulation results indicate that the DINMix model can adaptively and accurately identify coexisting condensation rules, existing either simultaneously in an item or separately in multiple items. An empirical example was also analyzed to illustrate the applicability and advantages of the proposed model. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

19 pages, 501 KiB  

Open AccessArticle

Is Distributed Leadership Universal? A Cross-Cultural, Comparative Approach across 40 Countries: An Alignment Optimisation Approach

by Nurullah Eryilmaz and Andres Sandoval-Hernandez

Educ. Sci. 2023, 13(2), 218; https://doi.org/10.3390/educsci13020218 - 20 Feb 2023

Cited by 6 | Viewed by 4011

Abstract

Distributed leadership (DL) is defined as the degree of contact and involvement of various people in making choices or carrying out responsibilities, and is an increasingly used concept among researchers, policymakers, and educationalists worldwide. However, few studies have investigated the cross-cultural comparability of [...] Read more.

Distributed leadership (DL) is defined as the degree of contact and involvement of various people in making choices or carrying out responsibilities, and is an increasingly used concept among researchers, policymakers, and educationalists worldwide. However, few studies have investigated the cross-cultural comparability of the distributed leadership scale for school principals, and few have ranked countries according to their levels of distributed leadership. This study employs an innovative alignment optimisation approach to compare the latent means of distributed leadership, as perceived by school principals, across 40 countries, using data from the OECD Teaching and Learning International Survey (TALIS, 2018). We found that South Korea, Colombia, Shanghai (China), and Lithuania had the highest levels of distributed leadership in school decisions, from the perspective of school principals. In contrast, the Netherlands, Belgium, Argentina, and Japan had the lowest levels. Our findings may serve as guidance for education stakeholders over which nations they could learn from in order to enhance school principal distributed leadership. Full article

(This article belongs to the Topic Psychometric Methods: Theory and Practice)

►▼ Show Figures

Figure 1

Show export options expand_more Show export options expand_less

Select all

Export citation of selected articles as:

Plain Text BibTeX BibTeX (without abstracts) Endnote Endnote (without abstracts) Tab-delimited RIS

 

Error

Oops... you haven't selected anything for export.

Ok

clear

Displaying articles 1-10