Incorporating Fine-Grained Linguistic Features and Explainable AI into Multi-Dimensional Automated Writing Assessment

Abstract

With the flourishing development of corpus linguistics and technological revolutions in the AI-powered age, automated essay scoring (AES) models have been intensively developed. However, the intricate relationship between linguistic features and different constructs of writing quality has yet to be thoroughly investigated. The present study harnessed computational analytic tools and Principal Component Analysis (PCA) to distill and refine linguistic indicators for model construction. Findings revealed that both micro-features and their combination with aggregated features robustly described writing quality over aggregated features alone. Linear and non-linear models were thus developed to explore the associations between linguistic features and different constructs of writing quality. The non-linear AES model with Random Forest Regression demonstrated superior performance over other benchmark models. Furthermore, SHapley Additive exPlanations (SHAP) was employed to pinpoint the most powerful linguistic features for each rating trait, enhancing the model’s transparency through explainable AI (XAI). These insights hold the potential to substantially facilitate the advancement of multi-dimensional approaches toward writing assessment and instruction.

1. Introduction

As technology-powered advances are being incorporated into large-scale writing assessments, automated essay scoring (AES) has received increasing attention, offering a viable alternative to the traditionally time-intensive and laborious manual grading processes [1,2,3]. Due to remarkable advances in corpus linguistics [4,5], natural language processing (NLP) [6,7], and deep learning [3,8,9], AES has the benefits of improved consistency, reduced subjectivity, and constructive feedback by exploiting extensive linguistic features or incorporating cutting-edging algorithms [10,11,12,13,14]. Given the importance of AES, it is unsurprising that the investigation into the power of linguistic features characterizing writing quality has become a critical focus within the domains of writing assessment and instruction in the past five decades. Traditionally, feature-based approaches utilize regression analysis to generate essay scores with manually extracted linguistic features [15,16,17]. The advantage of feature-based approaches involves the interpretability and explainability of linguistic features for writing quality [12,18].

However, there continue to be a plethora of issues associated with traditional approaches using handcrafted linguistic features. Firstly, even though the application of corpus-based methods in evaluating essay quality has proliferated in recent years, many of these applications have attached great importance to exploring linguistic features for argumentative writing quality as it is the most common genre type in large-scale writing assessment [19,20,21]. Relatively few studies have investigated linguistic features characterizing the writing quality of other genres. Additionally, feature-based approaches hang on a single holistic score with different constructs of writing quality being ignored [5,22,23,24]. More attention should be paid to explaining the inner workings of linguistic features underlying each trait, ensuring that the AES models are both robust and reflective of diverse writing constructs. Moreover, scholars have resorted to micro-linguistic features [15,25] and aggregated linguistic features derived from averaged combinations of micro-features that are theoretically interconnected [4,26,27] in characterizing the holistic score of writing. The effectiveness of micro-features, aggregated features, or their combined impact in accurately capturing different constructs of writing quality remains an area ripe for further investigation.

Given the above arguments, there is a need to enhance the understanding of fine-grained linguistic features that define writing quality, encompassing both holistic and analytic aspects. Additionally, traditional linear approaches do not sufficiently capture the intricate relationships between linguistic features and different constructs of writing quality. Therefore, this study pivots to non-linear models and provides a refined lens for examining the associations between linguistic features and different constructs of writing quality. Utilizing Principal Component Analysis (PCA) [28] and Explainable AI (XAI) with SHapley Additive exPlanations (SHAP) [29], the present study explores the extent of influence fine-grained linguistic features have on different constructs of writing quality and the underlying reasons, thereby improving the explainability of the proposed AES model and providing diagnostic and multi-dimensional feedback for writing instruction about what kinds of linguistic features should be given more emphasis in writing assessment and instruction.

2. Related Work

2.1. Holistic and Analytic Rating Traits for Writing Assessment

Writing assessment presents multifaceted challenges and relies heavily on subjective judgment, making it both time-consuming and demanding on human resources [30,31,32,33,34]. It is critical that human raters have a thorough understanding of the rating traits for scoring essays. The existing literature strongly advocates for the idea that a clear set of rating traits is critical to the validity of writing assessments. Without a clear framework, even the most experienced raters may be overwhelmed and unable to effectively measure the various dimensions of writing quality [35,36]. Scholars have also highlighted how specific scoring traits can help raters and educators produce reliable scores and maintain consistency in assessments, thereby mitigating the risk of making premature judgments influenced by personal biases [37,38]. Therefore, it is crucial to develop precise rating traits and their associated descriptors, which underpin the validity and reliability of writing assessments.

Holistic and analytic rating traits have been adopted to identify writing proficiency for different purposes [39,40]. Holistic scoring assigns a singular, comprehensive score to represent the quality of a sample essay. In contrast, analytic scoring evaluates essays through multiple sub-scales (e.g., content, language, and organization) that may or may not be averaged together to a total score [41]. Scholars have noted the efficiency of holistic scoring in educational and large-scale assessment settings, highlighting its cost-effectiveness and time-saving attributes [42,43]. As a result, holistic scoring has gained widespread acceptance in the field of writing assessment over recent years. Its utility lies in that raters can assign a single score, sparing them the necessity of multiple readings to grasp the meaning of analytical traits.

Although holistic scoring is widely adopted, it has been criticized for its use in the field of writing assessment and instruction. As critics point out, a notable limitation is that a single score fails to offer detailed feedback to students, particularly those struggling with writing, on the specific areas they need to improve. Additionally, it does not allow evaluators and educators to recognize and value the distinct characteristics that qualitatively differentiate one essay from another [44]. For example, two essays could be awarded the same overall score for vastly different reasons: one might be commended for its coherent structure, while the other could be celebrated for the sophistication of its sentence construction. Such instances illuminate a fundamental weakness: it may achieve substantial inter-rater reliability, but this comes at the potential cost of compromising the validity of writing assessment [33,45,46].

Depending on the purposes of writing assessment, analytic rating traits involve different aspects of writing quality: vocabulary, content, grammar, register, and organization [39,47]. Analytic rating traits are often used to provide diagnostic information on students’ writing proficiency and areas needing improvement [48]. For instance, a student might exhibit a strong command of syntax while struggling with essay organization. Therefore, an ever-increasing number of scholars have attached great importance to analytic rating traits in writing assessment due to their informative details and diagnostic feedback [49]. Despite the resource-intensive nature of analytic scoring, its advantages are compelling: firstly, it tends to yield more reliable scores compared to holistic scoring methods; secondly, it offers particular benefits for learners and students, whose writing skills may develop at varying paces across different constructs; thirdly, it facilitates more targeted and effective rater training and writing instruction; and fourthly, it is less prone to validity concerns, making it a robust choice for assessing writing quality [33,50,51,52]. Thus, the current study incorporates the strengths of holistic and analytic scoring, thereby improving score reliability in multi-dimensional writing assessment and providing diagnostic feedback for different constructs of writing quality.

2.2. Linguistic Features for Writing Quality

Scholars have attached much importance to linguistic features and their interactions with writing quality [53,54,55,56,57,58]. Generally, linguistic features characterizing writing quality can be categorized into three aspects: vocabulary, syntax, and cohesion [59,60]. In terms of lexical level, scholars have demonstrated that first language (L1) writers tend to use sophisticated phrasal constructions, greater lexical variation, less common words and bi-grams, more imageable words, and academic vocabulary [25,60,61,62,63,64,65,66,67]. Similar results have been observed in second language (L2) writing, revealing that high-quality essays are characterized by longer texts, a broader and more sophisticated vocabulary, and the use of less common and familiar words [66,68,69,70,71,72,73,74].

From a syntactic-level perspective, scholars have indicated that syntactic sophistication is positively related to the writing quality of L1 writers [59,60,75,76]. Subsequent studies have revealed a positive correlation with larger noun phrases, more modifiers, and additional words before the main verb and a negative correlation with simple declarative sentences and essay quality [25,77]. However, not all studies have reported a significant relationship between syntactic-level features and writing quality [25,78]. For L2 writers, complex nominal per clause has been identified as an indicator of writing quality. Additionally, the use of clausal subordination [56,72,79] and the frequency of passive voice are also positively correlated with writing quality.

Regarding cohesion, high-quality essays written by L1 writers are characterized by fewer positive connectives, less content word overlap, and increased usage of hedges and contrast expressions [77,80,81,82]. Research on L2 writers, however, yields mixed outcomes: cohesive devices such as semantic and lexical overlap, connectives, and givenness are negatively correlated with writing quality [55,68].

While existing research has considerably advanced our understanding of how linguistic features correlate with overall writing quality, there has been limited investigation of the correlation between linguistic features and different constructs of writing quality. Exploring multi-dimensional rating traits offers insights into linguistic features that enhance writing instruction across different writing constructs and covers a coherent picture of qualified linguistic features for model construction per rating trait. Furthermore, the advancement of computational analytic tools now allows extracting a wide array of fine-grained linguistic features, further enriching multi-dimensional writing assessment and model development.

2.3. Automated Essay Scoring (AES) Systems

The field of AES has experienced remarkable transformations over the last fifty years, fueled by significant breakthroughs in interdisciplinary research spanning NLP, computational linguistics, artificial intelligence (AI), and education [9,83,84,85]. Initially, AES systems depended on primitive models that utilized handcrafted linguistic features. However, they have evolved into more advanced systems, leveraging the power of AI technology to enhance their capabilities and accuracy. The foundation of AES was laid by Page and his team with Project Essay Grader (PEG), designed to streamline essay scoring on a large scale [86]. It focuses on surface-level linguistic features like paragraphs and word counts and combines multiple regression and NLP to predict essay scores [86,87]. However, PEG has faced criticism for overlooking semantic aspects of essays [88]. Moreover, the inner workings of linguistic features for each rating trait and processes involved in generating the holistic score remain undisclosed to the public [89,90].

In contrast to PEG’s reliance on surface linguistic features, the Intelligent Essay Assessor (IEA) places a strong emphasis on the essence and content-driven linguistic features, harnessing the capabilities of Latent Semantic Analysis (LSA) and information retrieval [83,91]. Through statistical analysis, LSA maps words and essays to their semantic relations, revealing topic-based connections with matrix decomposition [92,93,94]. In the model training process, IEA first extracts linguistic features measuring different constructs of writing quality and uses LSA alongside information retrieval to decipher how features blend to characterize diverse writing traits, shifting focus towards evaluating essays’ semantic depth.

E-rater combines multiple regression, the vector space model, and NLP techniques to extract linguistic features from essays [90,95,96]. This multifaceted approach allows E-rater to evaluate both the quality of language and the content of essays, with a notable focus on organizational quality as well. In the model training process, micro-features are initially aggregated into feature scores, with the significance of each feature determined through multiple regression analysis. For content-based feature extraction, both E-rater and IEA utilize vector space models. However, E-rater distinguishes itself by employing a keyword-based approach that excels in managing synonyms and polysemy. In contrast, IEA adopts a singular value decomposition (SVD)-based model, which excels in uncovering semantic relationships between texts and words by decomposing matrices, thereby facilitating topic-based analysis [26,97]. This methodological distinction enhances the E-rater’s ability to interpret nuanced linguistic variations more effectively.

Leveraging a blend of cognitive science, NLP, computational linguistics, and AI, IntelliMetric marks the advent of AI-driven AES tools and is adept at identifying linguistic features for high-quality essays, thereby crafting models that simulate human scoring [27,98]. Employing a multifaceted approach that includes linear regression, Bayesian ridge regression, and LSA for feature selection and model development, IntelliMetric distinguishes itself with an extensive array of over 400 linguistic features spanning semantic, syntactic, and discourse levels. It undergoes training with essays previously scored by human raters, learning to replicate human evaluative techniques and understanding the interplay between rating traits and linguistic features [27].

As writing quality needs to be judged from different constructs, AES systems and models thus need to be trained with different rating traits corresponding to each construct of writing quality. Therefore, the current study aims to investigate the most powerful linguistic features for different constructs of writing quality. With cutting-edge computational linguistic tools, PCA, and XAI with the SHAP model, a wide range of fine-grained linguistic features are extracted and transformed into weighted components to explore linguistic features for the writing quality at both holistic and analytic rating traits, thereby improving the explainability of the proposed AES model. Moreover, both linear and non-linear AES models are set up to capture the relationship between linguistic features and multifaceted constructs of writing quality.

The new trends and research gaps summarized above prompt the current study to examine the following research questions:

• Which types of features characterize writing quality more accurately: micro-features, aggregated features derived from averaged combinations of micro-features, or a combination of both?
• Which kinds of AES models, linear vs. non-linear models, perform better in encapsulating the association between linguistic features and different constructs of writing quality?
• Which linguistic features emerge as the most powerful for different constructs of writing quality, considering both holistic and analytic rating traits?

3. Method

3.1. Dataset

Within the realm of AES, the Automated Student Assessment Prize (ASAP) dataset is the largest open-access dataset for the advancement and benchmark of machine learning algorithms, encompassing over 16,000 student submissions across eight distinct topics [85]. The length of essays varies with the prompt, generally ranging from 150 to 650 words across the collections. Authored by students across various educational levels, these essays were rigorously evaluated and scored by expert human raters. Although numerous studies have investigated the relationship between linguistic features for the argumentative writing quality [17,20,21,55,72], the relationship between linguistic features and different constructs of writing quality of exposition remains under-explored. To address these needs, we concentrate on the seventh corpus of the ASAP dataset, chosen due to its noteworthy attributes that cover four analytic rating traits: Ideas, Organization, Style, and Conventions.

Dataset 7, which includes 1730 narrative essays written by seventh-grade students, calls for expository essays on the topic of Patience. Essays in this collection are concise yet thorough, with an average length of about 250 words. Essays have been evaluated and assigned scores on a 0–3 scale for each rating dimension by the human raters. The final holistic score was derived by summing the individual scores provided by these two human evaluators. We selected this subset due to its comprehensive scoring rubric that spans multiple evaluative dimensions rather than giving a holistic score. It offers significant advantages for our research objective to explore multi-dimensional writing assessment. The distribution of scores across the analytical rating traits by the two raters is illustrated in Figure 1. The results demonstrated that the vast majority of students fall within the intermediate range in terms of writing proficiency.

To assess the consistency and reliability of human grading, which serves as the gold standard for model construction, we computed the Pearson correlation coefficients between scores assigned by two independent evaluators across holistic and analytic rating traits. The inter-rater reliability between human raters at both holistic and analytic rating traits was moderately acceptable, ranging from 0.545 to 0.722, as detailed in

Table 1. Inter-raterreliability between human raters (*** p < 0.001).

Rating Traits	Pearson Correlation Coefficient (r)
Ideas	0.696 ***
Organization	0.576 ***
Style	0.545 ***
Conventions	0.567 ***
Holistic	0.722 ***

.

3.2. Research Instruments

All the essays were processed with advanced computational analytic tools, including Constructed Response Analysis Tool (CRAT) [99], Tool for the Automatic Analysis of Lexical Diversity (TAALED) [100], Tool for the Automatic Analysis of Lexical Sophistication (TAALES) [66], Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC) [101], Tool for the Automatic Analysis of Cohesion (TAACO) [53,102], and GAMET [103]. Linguistic features pertinent to various dimensions of writing quality were extracted using specific tools selected for several key reasons: First, their public accessibility and cost-free nature made them highly accessible. Second, the ease and speed with which these tools can be downloaded and operated provided unmatched convenience for selecting features and training models. Third, the strong construct validity of the linguistic features is supported by their foundation in established linguistic theories. Research has consistently demonstrated that these tools are effective in extracting linguistic features critical to assessing different facets of writing quality [53,66,99,100,101,102,103].

3.2.1. CRAT

CRAT, a user-friendly text analysis tool, offers genre-specific frequency and diversity insights, enriching its utility for a broad spectrum of textual analyses [99]. This freely available, cross-platform tool is operationalized through a graphical user interface, enhancing user accessibility. Its analytical prowess extends to evaluating lexical and phrasal similarities through advanced methodologies such as keywords, synonym overlaps, and LSA [104]. Additionally, it delves into phrasal congruence by scrutinizing the overlap of critical bi-grams and tri-grams and employing part-of-speech–sensitive slot-grams.

3.2.2. TAALED and TAALES

TAALED extracts 38 lexical diversity indices ranging from surface-level features such as the number of tokens and number of unique tokens to deep-level features such as the moving average type-token ratio (TTR) and measure of textual lexical diversity (MTLD) [100]. Six categories of indices, including a series of different tokens, lexical density, TTR, mass index [105], moving average TTR (50-word window), mean segmental TTR (50-word window), the hypergeometric distribution’s D (HD-D) index [106], and MTLD [67], are extracted for measuring lexical diversity.

TAALES extracts 424 indices, which can be categorized into 13 types: word range and frequency, academic words, n-gram range, frequency and strength of association, semantic network, psycholinguistic word information, and other indices [66].

Table 2. Linguistic features measuring lexical resources.

Indices of Lexical Diversity (No.)	Tool	Indices of Lexical Sophistication (No.)	Tool
token (6)	TAALED	word frequency (68)	TAALES
lexical density (2)	TAALED	word range (48)	TAALES
simple TTR (9)	TAALED	academic words (15)	TAALES
mass index (3)	TAALED	n-gram (227)	TAALES
moving average TTR (3)	TAALED	semantic network (14)	TAALES
mean segmental TTR (3)	TAALED	psycholinguistic word information (14)	TAALES
HD-D (7)	TAALED	age of acquisition/exposure (7)	TAALES
MTLD (9)	TAALED	contextual distinctiveness (8)	TAALES
		word recognition norms (8)	TAALES
		word neighbors (14)	TAALES
		other indices (5)	TAALES

illustrates linguistic features for lexical diversity and sophistication. Eight categories of indices regarding lexical diversity and eleven categories concerning lexical sophistication are included to ensure the richness and comprehensiveness of linguistic features for lexical resources.

3.2.3. TAASSC

TAASSC [101] explores more than 350 indices aimed at gauging various aspects of syntactic complexity and sophistication. These metrics are systematically organized into several distinct categories, including L2 Syntactic Complexity Analysis (L2SCA) [107], fine-grained phrasal complexity, clausal complexity, and syntactic sophistication. Please refer to [101] for the complete list of linguistic features measuring syntactic sophistication and complexity.

3.2.4. TAACO

TAACO involves more than 150 linguistic indices concerning three levels of local, global, and text cohesion [53,102]. The linguistic indices can be categorized into five types: connectives, givenness, TTR, lexical, and semantic overlap. There are altogether 25 types of connectives, ranging from basic connectives (and, but) to positive and negative logical connectives (consequently, notwithstanding that) to measure local cohesion. Givenness is a crucial measure of text cohesion, reflecting how information can be inferred from the context provided by preceding sentences. TTR plays a significant role in this domain by quantifying lexical diversity within essays. It does so by calculating the proportion of unique words (types) to the total word count (tokens) [53,102]. As an essential measurement of text cohesion, a series of TTR-based indices regarding the content word TTR, lemma TTR, content lemma TTR, and TTR for bi-grams and tri-grams are included in TAACO.

3.2.5. GMAET

Six categories of errors concerning duplication, grammar, spelling, style, typography, and white space are calculated with Grammar and Mechanics Error Tool (GMAET) [103]. Grammar errors consist of noun errors with agreement, adjective errors related to comparative and superlative errors, adverb word order errors, connector errors, negation errors, fragment errors, and verb errors with verb usage, person, tense, and aspect [103]. Duplication errors refer to word duplication. Spelling errors involve lowercase, missing hyphens, and apostrophes. Style errors examine features including verbosity, redundancy, and inappropriate word selection, which can detract from the clarity and conciseness of writing. Typography errors, on the other hand, involve mistakes related to capitalization and associated punctuation, which are crucial for maintaining the formal structure and readability of the text. Additionally, white space errors, which pertain to improper spacing within the document, can significantly impact the visual presentation and organization of the content.

3.3. Research Procedures

Figure 2 demonstrates the flowchart of research procedures for feature-based approaches. Based on the computational analytic tools, linguistic features were extracted to characterize the writing quality at holistic and analytic rating traits. This pivotal stage enabled the distillation of each essay’s core attributes into quantifiable metrics, laying the foundation for in-depth analysis. To avoid the problems of multicollinearity and over-fitting caused by too many linguistic features, we first included all linguistic features extracted by computational analytic tools for standardization and normally distributed detection. Then, the multicollinearity tests were conducted to ensure the selected linguistic features were not overlapping. PCA was conducted to condense the information of linguistic indices and keep those features that contribute the most to the variance.

Qualified linguistic features were employed to develop AES models for holistic and analytic rating traits. Linear and non-linear AES models were trained to mirror the scoring patterns of human raters by systematically learning the characteristics of qualified linguistic features. To guarantee the robustness and reliability of the training and evaluation processes for these models, a five-fold cross-validation was implemented. This approach divided the dataset such that 60% was allocated for training purposes, 20% served as a validation set to fine-tune model parameters, and the remaining 20% was utilized for testing the models’ performance. The training sets were used to select the linguistic indices for modeling, while the test sets were employed to verify whether the AES models were valid or not. Finally, SHAP values were applied to shed light on how linguistic features affect the writing quality across different constructs, enhancing our understanding of the inner workings of the proposed AES model.

3.3.1. Feature Extraction and Selection

Six computational analytic tools were used to extract linguistic features for model construction. The feature selection was a pivotal step in refining the array of linguistic attributes extracted from essays, guaranteeing that only the most relevant and robust features were incorporated into the model training process. Initially, an extensive set of 2044 features was extracted from each essay, utilizing advanced linguistic tools such as CRAT [99], TAALED [100], TAALES [66], TAASSC [101], TAACO [53], and GAMET [103].

The advantage of combining a great number of computational analytic tools was to ensure the richness and comprehensiveness of the linguistic features for measuring different constructs of writing quality. If these linguistic indices were directly included in the analysis, the model would be complicated and unstable, leading to over-fitting problems. Therefore, several methods were used to remove the redundant features and keep those that contribute the most to writing quality (see

Table 3. Progressive refinement of features during the feature selection phase, illustrating the reduction in the number of features at each step.

Feature Selection Stage	Number of Features
Initial Features Extracted	2044
After Removing Features with Variance < 0.01	934
After Removing Non-Normally Distributed Features	916
After Removing Features with Multicollinearity (r > 0.90)	362

). The non-parametric Kolmogorov–Smirnov test was employed to assess the adherence of the data to a normal distribution. In addition, the variance threshold was used to remove the feature that did not vary much within itself (variance lower than 0.01) [8]. The variance threshold seeks to eliminate variables demonstrating negligible variation within the dataset on the premise that such features are improbable to yield significant revelations regarding paper quality.

The multicollinearity tests were conducted to overcome the overlapping problems between features. These tests aimed to identify instances where two or more indices exhibited a high degree of correlation. In scenarios where indices were found to be highly correlated (r > 0.90), the index demonstrating the strongest correlation with the scores was retained for subsequent analysis, while redundant indices were excluded from the model [4]. After conducting filter methods, the number of linguistic features was reduced from 2044 to 362. The rigorous feature selection process honed the machine learning model’s training on a refined feature set, ensuring precise and stable scoring of essays.

PCA was used to condense the information of linguistic indices and keep those that contribute the most to the variance. Scholars have indicated three standard methods for determining the number of principal components. The first method was an eigenvalue-based method, which followed the Kaiser–Harris criterion and retained components with eigenvalues greater than 1 [108]. The second method was Cattell’s scree test [109]. By plotting the eigenvalues and principal components, the curve of the scree plot was displayed, and the principal components above the maximum change were retained. The last method was the parallel analysis method [110]. A principal component was considered significant and retained for further analysis if its eigenvalue surpassed the corresponding average eigenvalue derived from a collection of random matrices. The decision on the number of principal components to retain was informed by this eigenvalue criterion, in conjunction with the insights gained from the initial two methods employed in the analysis. To ensure the correlation of the indicators contained in each principal component, only the absolute value of the eigen loading greater than 0.35 was included for further analysis [4].

Figure 3 shows the proportion of the total data variance accounted for by each principal component. Seven principal components were retained by taking account of the first two methods [108,109]. Among them, the first principal component explained 10.1% (74 micro-features) of the variance of the total data variation. The last six principal components explained 5.9% (52 micro-features), 4.9% (26 micro-features), 4.0% (14 micro-features), 3.3% (19 micro-features), 2.5% (6 micro-features), and 2.2% (3 micro-features), respectively. Hence, the seven principal components (194 micro-features) explained 32.9% of the total variance of the original data set.

3.3.2. Model Construction and Validation

Micro-features, aggregated features derived from averaged combinations of micro-features, and a combination of those were used to investigate the strengths of characterizing writing quality. With qualified linguistic features, both linear and non-linear models were developed using the following six machine learning algorithms, namely Bayesian Ridge Regression [111,112], Linear Regression [112,113], Stochastic Gradient Descent Regression [114], Random Forest Regression [115], Decision Tree Regression [116], and Support Vector Regression [117]. The first three algorithms were used for linear AES model construction, whereas the last three algorithms were used for non-linear AES model construction.

• Bayesian Ridge Regression: integrates ridge regression and Bayesian inference, providing a probabilistic perspective on linear regression, excelling in data-scarce environments or large feature domains by proficiently approximating a weight distribution.
• Linear Regression: establishes a linear correlation between the input linguistic features and the predicted scores. Its straightforwardness and interpretability render it a foundational benchmark for numerous regression endeavors.
• Stochastic Gradient Descent Regression: iteratively adjusts parameters to reduce the loss function, promoting efficiency and the capacity to scale.
• Random Forest Regression: An ensemble method that consolidates outputs from various decision trees, renowned for its precision, robustness to over-fitting, and adeptness in handling non-linear correlations.
• Decision Tree Regression: builds a tree-like model that bifurcates the dataset into branches predicated on attribute values, offering an intuitive method capable of deciphering intricate patterns.
• Support Vector Regression: applies support vector machine concepts to regression, focusing on maintaining errors within a defined threshold, excelling in high-dimensional contexts and situations where error tolerance is crucial.

Four evaluation metrics concerning $R^2$, Quadratic Weighted Kappa (QWK), Exact Agreement, and Exact-Plus-Adjacent Agreement were used to assess the discrepancy between the predicted scores generated by the model and the scores assigned by human raters.

$R^2$ serves to quantify the percentage of the variance in human raters’ scores that can be accounted for by the linguistic features utilized in a model. If the $R^2$ of the proposed model was 0.60, 60% of the variability observed in the scores assigned by human raters could be explained by the inputs to the model. $R^2$ usually varied from 0% (the model did not explain any of the variations) to 100% (the model explained all the variations). Its formula is given by the following:

$$R^2=1-\frac{{\sum }_{i=1}^n{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2},$$

where $y_i$ represents the actual scores assigned by human raters, serving as a benchmark for the AES model’s accuracy. The model aims to closely match its predictions ${\widehat{y}}_i$ to these actual scores, effectively emulating human scoring behavior. $\overline{y}$ denotes the average of all actual scores, providing a reference point for assessing variations in both actual and predicted scores.

QWK was employed to evaluate the level of agreement between two raters. QWK values range from 0, indicating no match between raters, to 1, which signifies a perfect match. A great majority of studies have used QWK as a gold standard in evaluating the accuracy of AES models [3,6,83,118]. Therefore, the current study adopts QWK as the metric to gauge the accuracy between human raters’ scores and the scores predicted by the model.

$$QWK=1-\frac{2\times {\sum }_{i,j}{w}_{ij}{o}_{ij}}{{\sum }_{i,j}{w}_{ij}{e}_{ij}},$$

where $w_{ij}$ is the weight assigned to the disagreement between raters for items in categories i and j, $o_{ij}$ is the observed frequency of agreement, and $e_{ij}$ is the expected frequency of agreement under chance.

Exact Agreement was designed to pinpoint the proportion of essays for which the scores assigned by human raters and those predicted by the model are precisely the same. This measure offers a straightforward assessment of the model’s accuracy in replicating human scoring decisions without any margin for error.

Exact-Plus-Adjacent Agreement was used to measure the percentage of the number of essays with the same score or within 1 point of the score.

3.4. Visualization of Feature Importance: SHapley Additive exPlanations (SHAP)

XAI with the SHAP model [29], underpinned by game theory principles, was utilized to clarify the impact of linguistic features on different constructs of writing quality. SHAP provides a coherent and theoretically sound approach for attributing the contributions of input variables to a model’s predictions. This method is distinguished by its capacity to illuminate the overall importance of features across the dataset and to offer detailed explanations for individual prediction instances. For each feature, SHAP values quantify its impact on the variance between the specific prediction and the dataset’s mean prediction. Furthermore, an analysis of SHAP values reveals the key features for each evaluative criterion, thereby enhancing the transparency and explainability of the proposed AES model. The summary plot uses the horizontal axis to display SHAP values, showing how each linguistic feature affects the model’s prediction: positive for enhancement and negative for detriment. The vertical axis ranks these features by their impact based on SHAP value aggregation. Color shifts from blue to red reflect the feature’s value in observations (blue for lower values, red for higher), offering a quick visual insight into the distribution and impact of feature values on the model’s output.

4. Results and Discussion

4.1. Micro-Features, Aggregated Features, and Their Combination in Characterizing Writing Quality

In the investigation of the power of different types of features for assessing writing quality, three distinct groups were utilized: micro-features, aggregated features synthesized from the averaged combinations of micro-features, and a comprehensive approach incorporating both micro-features and aggregated features. The selection criteria for linguistic features and components to be included in the further analysis were based on their demonstrated significant correlation with writing quality, accompanied by at least a small effect size (r > 0.100) [4]. To assess the influence of these different feature types on the evaluation of writing quality, we conducted experiments on the aforementioned six models using a validation dataset. We also delved into the influence of different feature types across holistic and analytic rating dimensions: Ideas, Organization, Style, Conventions, and Holistic. The analysis encompassed three scenarios: utilizing the qualified micro-features only (194 features), exploring these seven aggregated features generated by PCA only, and examining a combination of both.

To better examine the difference in characterizing writing quality, we computed the mean of the evaluation metrics of the six methods for comparative analysis to facilitate the comprehension of how micro-features, aggregated features, and their combination contribute to the prediction of writing quality.

Table 4. Comparison between micro-features, aggregated features, and a combination of both.

Evaluation Metrics	Aggregated Features Only	Micro-Features Only	Combined
${\mathrm{R}}^2$	0.517	0.685	0.690
QWK	0.705	0.820	0.822
Exact Agreement	0.225	0.288	0.285
Exact-Plus-Adjacent Agreement	0.470	0.560	0.555

The most optimal results are highlighted in bold for clarity and emphasis.

indicates the effectiveness of different feature sets: micro-features only, aggregated features from the averaged combinations of micro-features, and their combination in characterizing writing quality.

The coefficient of determination $R^2$ showed that the combined feature approach yielded the highest value ($R^2$ = 0.690). This indicated a marginally superior fit to the data when compared to using micro-features ($R^2$ = 0.685) alone and a notably better performance over aggregated features ($R^2$ = 0.517). Paired t-tests were conducted to explore whether significant improvement exists. The results showed that the micro-feature (p < 0.050) and combined feature approaches (p < 0.050) were significantly improved when compared with the aggregated feature approach. However, no significant differences were found between the micro-feature and combined feature approaches (p > 0.050).

In terms of QWK, the results also supported the superiority of the combined features model with a score of 0.822, marginally outperforming the micro-features model (0.820) and significantly surpassing the aggregated features model (0.705). This increment, although slight, indicated that there were no significant differences between the micro-feature approach and the combined feature approach (p > 0.050). As for Exact Agreement and Exact-Plus-Adjacent Agreement, a slight advantage for micro-features in characterizing writing quality has been reported when compared with the combined feature approach (p > 0.050).

Overall, similar results have also indicated that the micro-feature and combined feature approaches provided a robust basis for characterizing writing quality when compared with the aggregated feature approach [4]. Additionally, the results demonstrated no differences between AES models with micro-features and AES models combining micro-features and aggregated components (p > 0.50). Therefore, further analysis will include only micro-features under the aggregated components.

4.2. Comparison between Linear and Non-Linear AES Models in Capturing Writing Quality

Both linear and non-linear methods were used to explore the relationship between linguistic features and essay scores across different rating traits.

Table 5. AES models’ performance based on micro-features across different rating traits (E: Exact Agreement; E + A: Exact-Plus-Adjacent Agreement).

Model	Ideas	Organization	Style	Conventions	Holistic
Bayesian Linear Regression
$R^2$	0.629	0.556	0.592	0.547	0.680
QWK	0.767	0.679	0.731	0.670	0.806
E	0.481	0.506	0.576	0.481	0.178
E + A	0.892	0.917	0.975	0.936	0.503
Linear Regression
$R^2$	0.637	0.542	0.582	0.496	0.672
QWK	0.783	0.685	0.722	0.670	0.811
E	0.433	0.497	0.551	0.503	0.159
E + A	0.911	0.914	0.965	0.917	0.487
Stochastic Gradient Descent Regression
$R^2$	0.614	0.554	0.598	0.515	0.670
QWK	0.775	0.699	0.753	0.668	0.801
E	0.439	0.519	0.576	0.478	0.185
E + A	0.911	0.914	0.981	0.927	0.487
Random Forest Regression
$R^2$	0.777	0.679	0.740	0.725	0.771
QWK	0.863	0.778	0.832	0.819	0.863
E	0.646	0.640	0.720	0.659	0.280
E + A	0.952	0.949	0.987	0.965	0.643
Decision Tree Regression
$R^2$	0.666	0.469	0.518	0.524	0.527
QWK	0.831	0.742	0.762	0.754	0.763
E	0.678	0.682	0.720	0.659	0.557
E + A	0.904	0.898	0.924	0.908	0.618
Support Vector Regression
$R^2$	0.704	0.616	0.685	0.631	0.732
QWK	0.816	0.746	0.770	0.741	0.839
E	0.573	0.621	0.650	0.608	0.376
E + A	0.924	0.930	0.978	0.933	0.586

The most optimal results are highlighted in bold for clarity and emphasis.

demonstrates the performance of linear and non-linear AES models in delineating the relationship between linguistic features and the multifaceted constructs of writing quality.

Following these tests, Random Forest Regression emerged as the most effective model, demonstrating superior performance across all rating traits and evaluation metrics. It achieved the highest ($R^2$) values, indicating a strong explanatory power for variations in writing quality, with notable scores in Ideas (0.777), Organization (0.769), Style (0.740), Conventions (0.725), and Holistic (0.771). Additionally, this model outperformed others regarding QWK, with scores indicating a robust agreement between predicted and actual ratings, peaking at 0.863 in the holistic rating trait. Decision Tree Regression showed remarkable strength in Exact Agreement, particularly in the Ideas and Organization, suggesting its capability to mirror the scoring patterns of human raters precisely. However, the Random Forest model excelled in the Exact-Plus-Adjacent Agreement metric across all the rating traits.

The Random Forest Regression model stood out for its robust capability across multiple dimensions of writing quality, supported by high agreement rates with human raters. The results suggested that non-linear AES models better captured the relationship between linguistic features and writing quality across different rating traits, which could enhance the overall accuracy and reliability of AES tasks, ensuring a more systematic assessment of writing quality that mirrors the complex judgment process of human raters.

The confusion matrices for each rating trait, illustrated in Figure 4 and Figure 5, provided further insights into the accuracy of AES models, revealing the distribution of predictions across the scoring traits. These matrices are crucial for understanding where the model excels and where it may confuse certain score categories. With qualified micro-features, the proposed AES model not only achieves high accuracy but also aligns closely with the nuanced judgment of human raters across multiple dimensions of writing quality.

We also compared the modeling results with other baseline models. Numerous scholars have conducted deep learning techniques to construct AES models for writing assessments.

Table 6. Comparison of QWK values for the holistic rating trait with Random Forest Regression and the baselines.

Approaches	QWK Scores
MN [119]	0.790
HISK + BOSWE and v-SVR [120]	0.804
TSLF-ALL [121]	0.801
Qe-C-LSTM [122]	0.815
Random Forest Regression	0.863

The most optimal results are highlighted in bold for clarity and emphasis.

shows the QWK scores at the holistic rating trait on the ASAP dataset 7. MN [119] utilizes a novel approach by evaluating the relationship between unlabeled essays and a repository of selected essays for score prediction. HISK + BOSWE and v-SVR [120] explored a method that encompassed both surface-level features and deep-level semantic features. TSLF-ALL [121] proposed a dual-phase learning architecture that integrated the advantages of feature-engineered approaches. Qe-C-LSTM [122] involved a method employing an advanced deep convolutional recurrent neural network for enhanced essay scoring accuracy.

The results demonstrated that our proposed approach with Random Forest Regression outperformed the QWK score by 4.8% over the benchmark Qe-C-LSTM. The paired t-test results indicated that the Random Forest Regression approach significantly improved performance (p < 0.050) when compared with those baselines. The observed improvement in performance can be attributed to a comprehensive and coherent representation of linguistic features that characterize various constructs of writing quality. By encompassing a broad spectrum of linguistic features characterizing different constructs of writing quality, the proposed AES model is better equipped to capture the multifaceted characteristics that constitute effective writing.

4.3. Most Powerful Linguistic Features per Rating Trait: Visualization via SHAP Model

To identify the most powerful linguistic features per rating trait, we calculated SHAP values to offer a rigorous quantification of each feature’s impact on the model’s prediction, thereby elucidating the relative importance of various linguistic features in determining different constructs of writing quality. Figure 6 and Figure 7 encapsulate the top 10 influential features for the holistic and analytic rating traits. Detailed information of linguistic features can be found in Appendix A. The variability in the impact of each feature, as indicated by the spread of SHAP values, underscores the complex interplay between different linguistic attributes when assessing writing quality. The insights derived from this analysis can inform educational interventions, automated writing assessment tools, and further research into the features that contribute to effective writing.

Figure 6 illustrates the SHAP values associated with the top 10 linguistic features that characterize writing quality at the holistic rating trait, thereby offering insights into the relative importance of these linguistic features. It was observable that certain features such as (hdd42_cw) and (nsentences) predominantly had a positive SHAP value distribution, indicating that higher values of these features typically correspond to better writing quality. In contrast, (Freq_N_CW) exhibited a predominantly negative distribution of SHAP values, suggesting that higher values of these features may be associated with lower writing quality. The results offered support to previous findings that less proficient writers tended to utilize more high-frequency words compared to their more proficient counterparts [15,55].

The features were arranged in descending order of their mean absolute SHAP values, signifying their average impact across the dataset. HD-D (hdd42_cw), measuring the lexical diversity, had the highest mean absolute SHAP value, indicating it was the most influential in the model’s assessment of writing quality. Additionally, lexical diversity indices, including the measure of textual lexical diversity (mtld_ma_bi_aw & mtld_original_cw) were identified as significant linguistic features for expository writing quality. The findings corroborated earlier research demonstrating that features based on the type-token ratio were key indicators of writing quality, as substantiated by [65,67,105].

Other features including the number of sentences (nsentences), number of words (nwords), word frequencies (Freq_HAL_CW), LSA-based indices (lsa_all_fwd & lsa_2-3_rwd), mean phonological Levenshtein distances (PLD), and neighborhood frequency of content word (Freq_N_CW) were identified as predictive linguistic features for the overall writing quality of exposition. The results offered support to previous studies showing that LSA-based features [81,82] and word frequencies [15,55] are significantly correlated with writing quality. With the XAI technique (SHAP), thorough and transparent linguistic features characterizing the overall writing quality of exposition are provided, which enhances the explainability of the proposed AES model and offers diagnostic and individualized feedback for multi-dimensional writing instruction and assessment.

In terms of Ideas, SHAP values were plotted on the horizontal axis, quantifying the degree to which each feature affected the predictive model’s output (see Figure 7a). The vertical listing of features indicated their relative importance based on the magnitude of their mean absolute SHAP values, with number of words (nwords) appearing as the most predictive feature. Upon examination of the plot, it was apparent that number of words (nwords) was associated with higher SHAP values, suggesting that it was positively related to writing quality for the Ideas dimension. Conversely, the semantic overlap of function words (lsa_all_fwd) was associated with negative SHAP values, indicating a potential association with lower quality in the Ideas dimension. Other features regarding HD-D (hdd42_cw), number of sentences (nsentences), word frequencies (Freq_HAL_CW & COCA_fiction_Frequency_CW & COCA_Fiction_Bigram_Frequency), LSA-based indices (lsa_all_fwd & lsa_2-3_rwd), number of content words in text with frequency score (Brown_Freq_CW_Log), and word concreteness (Brysbaert_Concreteness_Combined_CW) were identified as predictive linguistic features for the writing quality of Ideas.

As for Organization, analysis of the summary plot (see Figure 7b) revealed that features such as number of words (nwords) and HD-D (hdd42_cw) displayed a predominantly positive distribution of SHAP values, implying that these features were associated with a more favorable assessment of the ’Organization’ dimension. On the other hand, features such as word frequencies of content words (Freq_HAL_CW) exhibited a distribution towards negative SHAP values, suggesting a negative association with the writing quality in terms of Organization. Scholars have indicated words that were more frequently used would be acquired easily and earlier by students [59,64,66]. Thus, more frequent uses tended to get lower scores in writing tasks.

With Regard to Style, the results revealed that HD-D (hdd42_cw) had the highest mean absolute SHAP value, indicating it was the most influential in the model’s assessment of writing quality (see Figure 7c). Additionally, linguistic features including number of sentences (nsentences), number of words (nwords), lexical diversity indices (mtld_original_cw & mtld_ma_bi_aw & mtld_original_aw & root_ttr_fw), the similarity between two adjacent sentences (lsa_2_all_sent), number of positive logical connectives (positive_logical), and lexical density (lexical_density_types) were identified as predictive linguistic features for the overall writing quality of exposition. The results aligned with previous research, showing that the richness of vocabulary within a text was an important indicator of expository writing quality [66,71,74].

For Conventions (see Figure 7d), the analysis revealed that the number of sentences (nsentences) and measure of textual lexical diversity (mtld_ma_bi_aw) were the features with a predominantly positive distribution of SHAP values. This suggested that these features were likely to correlate positively with writing quality. In contrast, the feature measuring the number of words in the text with a frequency score (Brown_Freq_AW_Log) displayed a distribution towards negative SHAP values, potentially indicating a negative relationship with writing quality. The findings corroborate earlier research, indicating that less proficient writers tend to employ a higher number of high-frequency words compared to their more proficient counterparts. This pattern suggests a reliance on more common vocabulary among writers with lower proficiency levels [15,55].

The above analysis through SHAP vividly illustrates the multifaceted nature of linguistic features characterizing writing quality. Existing AES models, leveraging machine learning algorithms, are often criticized for their opaque, black-box nature, which hinges critically on the caliber of the training datasets. Integrating XAI methodologies, particularly SHAP, into these algorithms marks a pivotal advancement in demystifying the underlying mechanisms and providing a transparent analysis of linguistic features that characterize different constructs of writing quality.

5. Conclusions

With advances in computational linguistics, NLP, and AI, scholars have provided thought-provoking insights on incorporating feature-based approaches and machine learning algorithms into AES tasks. However, little has been done to explore the relationship between deep-level linguistic features and different constructs of expository writing quality. Additionally, the inner workings of linguistic features in the black box are still unknown. The present study takes full advantage of the interdisciplinary fields combining corpus linguistics, NLP, and XAI. It also provides insightful perspectives on developing linear and non-linear models to examine the relationship between linguistic features and essay scores from holistic and analytic rating scales.

With cutting-edge computational analytic tools and PCA, deep-level linguistic features were extracted to condense and reduce the noise, thereby eliminating redundancy and enhancing data quality. The results demonstrated that the micro-feature and combined feature approach provided a robust basis for characterizing writing quality compared to the aggregated feature approach. Furthermore, linear and non-linear AES models were constructed to explore the complex relationship between linguistic features and writing quality. The results indicated that the proposed non-linear AES model with Random Forest Regression outperformed the QWK score by 4.8% over the benchmark Qe-C-LSTM, which offered support to the idea that the relationship between linguistic features and writing quality was complicated as multi-dimensional constructs were involved. Lastly, SHAP values were plotted to identify the most powerful linguistic features per rating trait, thereby improving the explainability of the proposed model. Linguistic features for different constructs of writing quality can provide diagnostic feedback for writing instruction and revision. Teachers can also carry out diversified and individualized teaching according to the feedback of linguistic features, characterizing the writing quality through holistic and analytic rating traits.

While our findings contribute to the development of more sophisticated, fair, and transparent AES systems, they also highlight the need for a broader representation of writing genres and a more diverse range of educational backgrounds. Broadening the scope of the datasets would enhance the robustness and generalizability of the proposed AES models. This expansion is crucial for achieving a comprehensive understanding of multi-dimensional writing assessment across large-scale educational contexts. Furthermore, future studies could benefit from integrating large language models [7,123], which promise further to improve the accuracy and generalization capabilities of AES systems. Such advancements are essential for keeping pace with the evolving demands of multi-dimensional assessment.