Exploring Engagement, Performance, and Satisfaction in Online Self-Directed Professional Learning Using LMS Logs

Abstract

Online self-directed professional learning plays a crucial role in sustainable career development. This study leverages a high-quality log dataset to thoroughly analyze the learning features of online self-directed professional learners, focusing on their engagement, performance, and satisfaction. The study reveals that the engagement levels among learners are predominantly low, with 56% categorized as low, 33% as medium, and 11% as high. The performance is generally strong, with 47% of learners achieving excellent results, although 4% fall into the poor category. The satisfaction levels are largely neutral (76%), with only 17% of learners expressing satisfaction and 7% feeling delighted. Despite high course ratings, the number of courses purchased remains minimal. The analysis found no significant correlations between engagement, performance, and satisfaction, but noted that purchasing additional courses can enhance engagement. Furthermore, lesson learning shows significant day-to-day fluctuations and minimal linear autocorrelation. The most significant predictor of course performance is the number of questions answered in quizzes. These findings help us to understand the patterns and relationships among these variables to inform future improvements in online learning platforms. Future research should expand LMS log collection to encompass a wider array of learning features for a more thorough analysis, and empirical research should be conducted to investigate potential underlying causes.

1. Introduction

Online self-directed learning offers high flexibility and convenience, serving as a crucial avenue for employees to enhance their professional skills [1]. Famous MOOC platforms such as Coursera and edX offer a wide range of online self-directed professional courses, spanning fields like data analysis, project management, digital marketing, cybersecurity, and more [2]. Specifically, 365DataScience provides an extensive range of career learning courses in data science and artificial intelligence, equipping employees with the expertise and skills necessary for long-term success in a competitive job market. Investigating the learning behavioral features of online self-directed learners, and exploring the types and proportions of their engagement, performance, and satisfaction, is beneficial in gaining deeper insights into their learning habits and experiences. This can facilitate the optimization of career learning courses and enhance learning outcomes.

Learning management systems (LMS) can track online learners’ activities non-intrusively, generating logs that circumvent the subjective biases inherent in self-reported data. These logs are commonly utilized to analyze the behavioral features of online learners [3], measure engagement [4], and predict performance [5]. Specifically, Sun et al. employed log and time-series analysis to investigate the relationship between self-directed learning behaviors and cognitive engagement, performance, and satisfaction in synchronous online courses [6].

Furthermore, the relationship between online learning engagement, performance, and satisfaction has been a focal point for many researchers. Previous research findings have revealed that, in various contexts, online learning engagement enhances performance [7,8], while satisfaction is correlated with both engagement and performance [9,10]. However, researchers have conducted limited exploration into the behavioral features, engagement, performance, and satisfaction of online self-directed learners using LMS logs.

To bridge this research gap, this study conducted a thorough analysis of a log dataset from learners of 365DataScience, a renowned online career learning course provider. This dataset contains logs from 18,344 learners across 40 courses focused on data science and artificial intelligence over a span of 293 days (1 January 2022 to 20 October 2022). The logs encompass various activities, such as lesson learning, quizzes, practice exams, course exams, ratings, and purchases. It is particularly well suited for the exploration of the learning behavioral features, engagement, performance, and satisfaction of online self-directed learners.

We defined three research questions as follows:

• How can meaningful learning behavioral features be extracted from the logs?
• What are the types and proportions of engagement, performance, and satisfaction among online self-directed professional learners? What are the correlations between them?
• What is the differential impact of various learning features on performance?

2. Literature Review

2.1. Using Logs to Investigate Online Learning Engagement and Satisfaction

The analysis of online learning logs has emerged as a pivotal area of interest in learning analytics. Researchers have focused on various aspects, including identifying patterns in learning behavior [11,12,13,14], measuring engagement [4,15,16], discerning cognitive styles [17,18], and providing tailored recommendations for learning resources [15,19] through the analysis of LMS logs. The outcomes of these studies underscore the significance of learners’ LMS logs as repositories of invaluable information. The results of these studies indicate that the behavioral features of learners can be extracted from LMS logs and utilized to explore their situations during the learning process.

Engagement refers to the degree of cognitive, emotional, and behavioral effort exerted by learners during the learning process. LMS logs are commonly used to measure learners’ behavioral engagement.

Golchehreh et al. reviewed 32 carefully selected papers, synthesizing indicators from previous studies that can be used to measure engagement in LMS logs. Their research findings indicated that a total of 27 indicators were categorized into three themes and six categories as follows: (a) log-in and usage (referring to LMS, access to course material), (b) student performance (assignments, assessments), and (c) communication (messaging, forum participation) [20]. These indicators represent the summation of the frequency or duration of learners’ engagement in various learning activities during a specific period. The most commonly used indicator by researchers among these is access to course resources. Additionally, Javeed et al. devised a method to generate engagement scores based on LMS logs. They extracted behavioral features from the logs of undergraduate engineering students’ quizzes, assignments, discussions, and various navigation behaviors on the LMS to measure engagement. They also employed association rule mining to investigate the differences in behavioral patterns between completers and non-completers [21].

Online learning satisfaction pertains to learners’ evaluative opinions and experiential feelings regarding the quality of online learning services. Usually, satisfaction investigation requires learners to self-report based on questionnaires [22,23]. Researchers usually include a single dedicated question in the questionnaire to holistically measure learners’ satisfaction [24,25].

2.2. The Correlation between Online Learning Engagement, Performance, and Satisfaction

Engagement and satisfaction are crucial determinants of performance in online learning, and the relationship among these factors has attracted considerable attention from researchers (Table 1).

Chai et al. validated the positive impact of online learning engagement on performance based on the Theoretical Framework of the Model of Student Differences for Learning in Online Education (MSDLOE) [26]. Specifically, they regarded learners’ contributions to forums as behavioral engagement, while students’ performance was assessed through weighted scores for participation in discussions (5%), video watching (5%), and final course examinations (90%) [7]. Liu et al. employed partial least squares structural equation modeling (PLS-SEM) to validate that behavioral engagement, emotional engagement, and cognitive engagement all positively influence college students’ performance in online learning [8].

Ji et al.’s findings suggest that, in synchronous online second language learning environments, college students’ satisfaction with online learning is influenced by their engagement, preparation, and participation strategies [27]. Similarly, Han et al. also confirmed the positive correlation between engagement and satisfaction in online English learning in a university [10]. Yousaf investigated the impact of interaction and engagement in online learning on satisfaction among university students. Their findings revealed a significant association between students’ engagement in online learning activities, such as online exams, assignments, and tests, and their satisfaction [28]. Additionally, Chan and Lee (2023) demonstrated that, under new learning methods and subjective teacher support, students’ online learning engagement is positively correlated with satisfaction [29]. Rajabalee et al. devised an online learning module to investigate the relationship between engagement, performance, and satisfaction among first-year students across various disciplines. Their findings revealed a significant correlation among engagement, performance, and satisfaction. However, there were no significant differences observed in perceived satisfaction and engagement among students of different disciplines and genders [9].

Table 1. Comparison of relevant literature on the relationship between online learning engagement, performance, and satisfaction.

Literature	Nation	Type of Learner	Source of Data	Sample Size
Chai et al., 2023 [7]	China	University	Self-report Questionnaire	322
Liu et al., 2023 [8]	China	University	Questionnaire	810
Ji et al., 2022 [27]	Korea	University	Self-report Questionnaire	82
Yousaf et al., 2023 [28]	Pakistan	University	Questionnaire	652
Rajabalee and Santally, 2021 [9]	Mauritius	University	Questionnaire	844
Chan and Lee, 2023 [29]	China	University	Questionnaire	186
Han et al., 2021 [10]	China	University	Questionnaire	428

Table 1. Comparison of relevant literature on the relationship between online learning engagement, performance, and satisfaction.

Literature	Nation	Type of Learner	Source of Data	Sample Size
Chai et al., 2023 [7]	China	University	Self-report Questionnaire	322
Liu et al., 2023 [8]	China	University	Questionnaire	810
Ji et al., 2022 [27]	Korea	University	Self-report Questionnaire	82
Yousaf et al., 2023 [28]	Pakistan	University	Questionnaire	652
Rajabalee and Santally, 2021 [9]	Mauritius	University	Questionnaire	844
Chan and Lee, 2023 [29]	China	University	Questionnaire	186
Han et al., 2021 [10]	China	University	Questionnaire	428

In summary, while numerous researchers have explored the correlation between online learning engagement, performance, and satisfaction, the focus has predominantly been on university students, neglecting attention towards online self-direct career learners. Moreover, the data utilized to investigate engagement, performance, and satisfaction primarily stem from self-reports and questionnaires, exhibiting subjective bias. Additionally, there is a lack of investigation into the specific types of engagement, performance, and satisfaction among online self-direct career learners. Therefore, our study aims to fill the aforementioned research gap.

3. Methodology

3.1. Overall Framework of the Investigation

Based on the objectives of this study, we devised a framework to extract lesson learning series and quantitative learning features, exploring engagement, performance, and satisfaction using logs. Subsequently, we investigated the distribution, types, and correlations among engagement, performance, and satisfaction across online self-direct career learners based on the extracted features. The overall framework of the investigation is illustrated in Figure 1.

As shown in Figure 1, the entire investigation can be divided into three phases. In the first phase, intermediate features are extracted from the raw logs using methods such as serialization, averaging, or summarization. In the second phase, the intermediate features are transformed into meaningful features using time series analysis and normalization. The distribution, types, and correlations of the engagement, performance, and satisfaction of online self-directed professional learners are investigated in the third phase. Additionally, we compare the effects of different quantitative learning features on course performance.

3.2. Phase 1: Generation of Intermediate Features

In the first phase, intermediate features are generated from the raw logs through serialization, averaging, and aggregation. Since the raw logs contain the lesson learning time (in minutes) for each day, the lesson learning sequence for each student can be organized into a 293-dimensional vector {L}, where l_i∈{L} represents the lesson learning time on the ith day. A learner may participate in exams, quizzes, and rating for multiple courses over 293 days. We use averages to represent their levels. Similarly, we utilize aggregation to obtain the total purchase numbers of learners and the total number of days on which they participated in courses, exams, and tests over 293 days, respectively.

3.3. Phase 2: Normalization and Time Series Analysis

In the second phase, to ensure that different features share the same scale for subsequent comparison and analysis, we employed min–max normalization (1) to transform the values of all intermediate features into real numbers ranging between 0 and 1. After normalization, we obtained three features representing engagement, i.e., engaged lessons, engaged exams, and engaged quizzes, which, respectively, indicate the number of days on which a learner participated in lesson learning, exams, and quizzes.

Additionally, we obtained three features to represent performance (course exams, practice exams, and quiz ratio), which, respectively, indicated the learner’s scores in course exams and practice exams and the correct ratio in quizzes. Finally, two features were obtained to represent satisfaction (purchases, rating), which, respectively, indicated the number of course purchases and the ratings given to the courses by each learner.

$$X_{norm}={\displaystyle \frac{X-X_{min}}{X_{max}-X_{min}}}$$

(1)

Since the learner’s lesson learning over 293 days had already been represented as a vector, we used time series analysis to extract features from the lesson learning sequence. The extracted features included L_Cor_1, L_Cor_3, and L_Cor_7, representing the autocorrelation coefficients at a lag of 1 day, 3 days, and 7 days, respectively. The coefficient of variation (L_VC) indicates the fluctuation in lesson learning across different days. Skewness (L_Skew) and kurtosis (L_Kurt) depict the distribution of the learning time over 293 days. Additionally, the complexity-invariant distance complexity (L_Cid) reflects the dynamicity and complexity level of the lesson learning sequence.

$$L_{Co{r}_{day}}={\displaystyle \frac{1}{\left(n-day\right){\sigma }^2}}{{\displaystyle \sum }}_{i=1}^{n-day}\left(l_i-\mu \right)\left(l_{i+day}-\mu \right)$$

(2)

$$L_{Cid}=\sqrt{{\displaystyle \sum }_{i=1}^{i=n-1}{\left(l_{i+1}-l_i\right)}^2}$$

(3)

3.4. Phase 3: Distribution, Type, Correlation, and Importance Analysis

In the third phase, to investigate the types of engagement, performance, and satisfaction among online self-directed professional learners, we employed the unsupervised K-means algorithm [30] to cluster the meaningful features generated in the second phase. The elbow rule [31] was utilized to determine the optimal number of clusters. We evaluated the quality of each clustering result using the sum of squared distance of the samples to the cluster centers, also known as the sum of squared errors (SSE) (4), where ${\mu }_i$ denotes the cluster center of cluster $C_i$.

$$SSE={{\displaystyle \sum }}_{i=1}^k{{\displaystyle \sum }}_{x\in c_i}{\left(x-{\mu }_i\right)}^2$$

(4)

Since the engagement, performance, and satisfaction of learners do not adhere to a normal distribution, we utilized the Spearman correlation coefficient (5) to denote their correlation, where d_i represents the difference in rank between the two variables.

$$r=1-{\displaystyle \frac{6\sum d_i^2}{n\left(n^2-1\right)}}$$

(5)

Finally, we employed regression to investigate the impact of various quantitative learning features on the final course performance. The features included as independent variables were the total duration of lesson learning (L_nums), the number of practice exam attempts (P_E_nums), and the number of questions answered in quizzes (Q_Q_nums). Based on our previous research findings [32], nonlinear regressors can better fit online learning datasets. Therefore, we used a decision tree regressor and employed SHAP values to represent the impact of different learning features on course performance. The Shapley value signifies the average marginal contribution of a feature value within potential coalitions. More precisely, the Shapley value of feature value j in the i-th sample, denoted as ${\mathsf{\phi }}_{ij}$, illustrates its contribution to the output of this sample relative to the average output of all samples in the dataset (6). The SHAP value, rooted in game theory, epitomizes the best Shapley value (5). Additionally, $x^{\prime }$ denotes a coalition vector where the presence of values for all features is assumed [33].

$${\phi }_{\mathit{ij}}\left(\mathit{val}\right)={{\displaystyle \sum }}_{S\subseteq {\{x}_{i1}{,x}_{i2}{,\dots ,x}_{\mathit{ip}}{\}\backslash \{x}_{\mathit{ij}}\}}{\displaystyle \frac{\left|S\right|\mathit{!}\left(P\text{-}\left|S\right|\text{-}1\right)\mathit{!}}{p\mathit{!}}}\left(\mathit{val}(S\cup \left\{x_{\mathit{ij}}\right\}\right)\text{-}\mathit{val}\left(S\right))$$

(6)

$${\mathit{val}}_{\mathit{xi}}\left(S\right)=\int \hat{f}\left(x_{i1}{,x}_{i2}{,\dots ,x}_{\mathit{ip}}\right){\mathit{dP}}_{\mathit{xi}\notin S}{\text{-}E}_X(\hat{f}\left(X\right))$$

(7)

$$g\left(x^{\prime }\right){=\phi }_0+{{\displaystyle \sum }}_{j=1}^M{\phi }_j$$

(8)

4. Result

4.1. Descriptive Statistics and Analysis of Meaningful Features

Based on the overall framework of investigation depicted in Figure 1, meaningful features representing engagement, performance, satisfaction, and the lesson learning sequence were extracted during the second phase.

The descriptive statistics and distributions of the features representing engagement, performance, and satisfaction are illustrated in

Table 2. The descriptive statistics of the features representing engagement, performance, and satisfaction.

	Feature	Mean	Std	CV	Min	Max
Engagement	Engaged Lessons	0.26	0.21	81%	0.01	1.00
	Engaged Exams	0.24	0.18	75%	0.02	1.00
	Engaged Quizzes	0.22	0.16	73%	0.02	1.00
Performance	Course Exams	0.70	0.15	21%	0.00	1.00
	Practice Exams	0.74	0.12	16%	0.00	1.00
	Quiz Ratio	0.95	0.08	8%	0.35	1.00
Satisfaction	Purchases	0.15	0.14	93%	0.10	1.00
	Rating	0.94	0.11	12%	0.20	1.00

and Figure 2.

From Table 2, it is evident that, for the three features representing engagement, the average participation rates in courses, exams, and quizzes are 0.26, 0.24, and 0.22, respectively. This indicates a low level of engagement among online self-directed learners, mirroring the trends observed in various other online learning scenarios. Specifically, the range for engagement in courses, exams, and quizzes is 0.99, 0.98, and 0.98, respectively, with coefficients of variation of 81%, 75%, and 73%. This suggests significant disparities in engagement levels among different learners, with some learners being nearly inactive in learning activities (minimum values for courses, exams, and quizzes are 0.01, 0.02, and 0.02 respectively), while others are highly active across various learning tasks.

In terms of the three features of performance, the learners exhibited good performance in course exams, practice exams, and quizzes, with average scores of 0.70, 0.74, and 0.95, respectively. Moreover, the coefficients of variation relative to their engagement were relatively low (21%, 16%, and 8%, respectively), indicating that the vast majority of learners achieved satisfactory performance in online self-directed professional learning. Nonetheless, it is noteworthy that there were still some learners who performed poorly (minimum values for courses, exams and quizzes are 0.15, 0.12, and 0.08 respectively).

In terms of the two features of satisfaction, the learners provided high ratings for the courses (mean = 0.94, CV = 12%), indicating that most learners highly rated the courses that they had participated in. However, the average number of course purchases among learners is low (mean = 0.15), and there exists significant variation among different learners (cv = 93%). This reflects that learners’ behavior of purchasing courses may not be directly related to their actual satisfaction with the courses.

As seen in Figure 2, the majority of learners’ engagement in courses, exams, and quizzes is individually below their respective average levels, while the majority of learners achieve high accuracy in quizzes and provide high ratings for the courses. This further illustrates that most online self-directed professional learners have low engagement but high performance. There is a significant disparity between the number of courses purchased and the ratings given by learners. Among these eight features, only course exams and practice exams follow an approximately normal distribution. Therefore, when exploring the relationship between engagement, performance, and satisfaction, the rank-based Spearman correlation coefficient may be a more suitable indicator.

In

Table 3. The descriptive statistics of the features of the lesson learning sequence.

Feature	Mean	Std	CV	Min	Max
L_Cor_1	0.08	0.27	338%	−1.02	0.72
L_Cor_3	−0.02	0.27	1350%	−1.17	0.64
L_Cor_7	−0.05	0.31	620%	−1.97	1
L_VC	0.81	0.24	30%	0.1	2.09
L_Skew	1.15	0.83	72%	−1.03	5.72
L_Kurt	1.94	3.95	204%	−3.29	42.42
L_Cid	0.41	0.25	61%	0.02	1.34

, the lagged autocorrelation coefficients (L_Cor_1, L_Cor_3, and L_Cor_7) approach zero, with coefficients of variation at 338%, 1350%, and 620%, respectively. This suggests minimal linear autocorrelation in the learners’ lesson learning across consecutive days, implying that the present lesson learning cannot be reliably described or forecasted based on past days. The average values of L_VC and L_Cid for lesson learning also indicate significant differences between learners on consecutive days, with values of 0.81 and 0.41, respectively. This underscores the lack of continuity in lesson learning among online self-directed professional learners.

The values of skewness (L_Skew) for the lesson learning sequence fluctuate between 1.15 and 5.72, indicating a certain degree of right skewness in the distribution of learning. This right skew suggests that the sequence contains a higher frequency of smaller values compared to larger ones. With a coefficient of variation of 72%, the variability in L_Skew across different time periods indicates significant fluctuations in the skewness values.

Furthermore, the values of kurtosis for the sequence (L_Kurt) range from 1.94 to 42.42, indicating significant variability and extreme values in lesson learning across different days. These findings collectively suggest a lack of continuity in lesson learning among online self-directed learners, with substantial fluctuations between different days.

4.2. Types and Proportions of Engagement, Performance, and Satisfaction among Learners

We employed K-means to cluster the three features representing engagement, with the sum of squared errors (SSE) depicted in Figure 3 for cluster numbers (k) ranging from 1 to 10. From Figure 3, it is observed that the SSE ceases to exhibit significant variations beyond a cluster number of 3. Following the elbow rule, we determine the optimal cluster number to be (k = 3), indicating that engagement among online self-directed professional learners can be categorized into three types. A comparison and the proportions of these three types are presented in Figure 4 and Figure 5.

Based on the results in Figure 4, we designate the three types of engagement as low, medium, and high, with proportions of 56%, 33%, and 11%, respectively. This closely aligns with the findings from numerous other investigations in online learning environments, indicating that the majority of learners have low levels of engagement. The three types of learners exhibit significant differences in engagement across the engaged lessons, quizzes, and exams, with the engaged lessons being the most distinguishing feature among the different types.

Next, the learners’ types of performance are similarly explored. The results of the SSE when the cluster number (k) is set from 1 to 10 are depicted in Figure 6, with the SSE ceasing to exhibit significant variations beyond a cluster number of 4. Following the elbow rule, we determine the optimal cluster number to be (k = 4), indicating that the performance of online self-directed professional learners can be classified into four categories. A comparison and the proportions of these four types are presented in Figure 7 and Figure 8.

Based on the results in Figure 7 and Figure 8, we designate the four types of performance as poor, average, good, and excellent, with proportions of 4%, 19%, 30%, and 47%, respectively. This outcome indicates that the vast majority of learners have achieved very good performance. The differences in performance among the four types of learners in terms of course exams, practice exams, and the quiz ratio are illustrated in Figure 7. It is evident that all types of learners perform well in quizzes, with learners classified as poor exhibiting significant discrepancies compared to other types in both course and practice exams. The differences between learners classified as average and good are minimal, with the performance on both course and practice exams falling within the [0.6, 0.8] range, consistent with typical grading conventions for moderate academic performance.

Finally, the results of the exploration of the types of learners’ satisfaction are depicted in Figure 9, Figure 10 and Figure 11. The SSE when the cluster number (k) is set from 1 to 10 is illustrated in Figure 9. Following the elbow rule, we determine the optimal cluster number to be (k = 3), indicating that the satisfaction of online self-directed professional learners can be classified into three categories. A comparison and illustration of these three types are presented in Figure 10 and Figure 11.

Based on the results shown in Figure 10 and Figure 11, we have categorized satisfaction into three types, named neutral, satisfied, and delighted, with proportions of 76%, 17%, and 7%, respectively. Learners with a neutral satisfaction level rated the course on average close to 0.8, while learners in the other categories rated it close to 1. This indicates that the majority of learners are highly satisfied with the online career course. The primary difference among learners with different levels of satisfaction lies in the number of course purchases.

The number of course purchases may be influenced by various factors, such as practical needs and economic conditions. More than 90% of learners purchase a small quantity of courses (mean = 0.13), while those who purchase a larger number (mean = 0.6) also rate the courses close to 1. This suggests that using purchases as a feature for satisfaction is meaningful. In other words, in a scenario where the majority of learners rate the course highly, the quantity of purchases can provide us with additional insights into learners’ satisfaction with the course from various perspectives.

4.3. The Correlation between the Engagement, Performance, and Satisfaction of Learners

Based on the distribution of meaningful features described in Section 4.1, Spearman coefficients were utilized to indicate the correlations among different features. As depicted in Figure 12, there is a significant positive correlation among the three features representing engagement (r = 0.86, 0.84, 0.75). This suggests a strong association among various learning activities in which online self-directed professional learners are engaged. The three features representing performance exhibit a moderate positive correlation. Specifically, there is a relatively strong correlation between practice exams and course exams (r = 0.47) and a considerable correlation between practice exam scores and the quiz ratio (r = 0.44), whereas the correlation between course exam scores and the quiz ratio is weaker (r = 0.26). Conversely, there is no significant correlation between the two features representing satisfaction (r = −0.0043); this means that learners are influenced by many other factors when purchasing courses.

Additionally, there is a certain correlation between purchases and the three features representing engagement (r = 0.14, 0.1, 0.11), which is more pronounced compared to their correlation with ratings. This suggests that financial investment might serve as a significant external motivator in fostering learning engagement among online self-directed professional learners.

Furthermore, there are no notable pairs of features exhibiting significant correlations in Figure 12, which could be attributed to the distribution of the features. As depicted in Figure 2, learners demonstrate low engagement but high performance and ratings. This could potentially explain the lack of significant correlations among the features representing engagement, performance, and satisfaction.

4.4. The Influence of Learning Features on the Course Performance of Learners

To explore the influencing factors regarding course performance, we employed a decision tree regressor to model the relationships between the learning features, which included the number of quiz questions (Q_Q_nums), the total learning time for lessons (L_nums), the number of practice exam attempts (P_E_nums), and course performance. We utilized the SHAP values described in Section 3.4 to quantify the impacts of these features on course performance. The findings of this investigation are presented in Figure 13.

From Figure 13a, it is evident that the three learning features exhibit significant variations in their impacts on course performance, with Q_Q_nums showing the most substantial influence, while P_E_nums has the least impact and L_nums falls between the aforementioned two features. Each data point in Figure 13b represents the impact (SHAP value) of a specific feature on course performance for an individual learner. It is evident that the SHAP values of Q_Q_nums have a wide distribution across different samples, encompassing both positive and negative values. This indicates that the impact of this feature on course performance is bidirectional—a larger Q_Q_nums value can positively influence course performance, while smaller values can have a negative impact. This further underscores that Q_Q_nums is a feature significantly that affects course performance. The SHAP values of L_nums are primarily concentrated in the positive region, indicating that this feature mainly has a positive impact on course performance, implying that an increase in course study time typically enhances course performance. For most learners, the impact of P_E_nums on course performance is minimal (with most data points near 0).

5. Key Findings and Discussion

Based on the investigation in the previous section, we can enumerate the following key findings.

(1) The engagement levels of online self-directed professional learners are relatively low, as indicated by the mean values of engaged lessons, engaged exams, and engaged quizzes, all of which are below 0.3. Furthermore, there are significant disparities among different learners, with a coefficient of variation (CV) averaging 76%. The engagement of learners can be categorized into three types, low, medium, and high, constituting 56%, 33%, and 11% of the sample, respectively. This distribution aligns with findings from numerous other investigations in online learning environments, suggesting a prevalent level of low engagement among online self-directed learners. Among the three types of learning features—lesson learning, quizzes, and exams—the differences between the learner engagement levels are most pronounced in lesson learning. Coupled with subsequent findings that learners generally have good performance (average value above 0.7) and very high course ratings (average value above 0.94), the potential causes of this low engagement may include the following three factors.

Firstly, learners committed to enhancing their career capabilities through online self-directed learning have strong motivation and are capable of efficiently completing course learning. Additionally, compared to learners’ existing career skills, online career courses tend to have lower levels of difficulty and challenge. Lastly, online self-directed professional learners typically engage in part-time studies, with their learning being influenced by various factors such as work and family responsibilities, making it challenging to sustain consistent and planned learning. Therefore, institutions offering online career courses should enhance the courses’ cutting-edge nature and provide personalized learning recommendations and plans based on learners’ professional backgrounds, skill levels, and needs, so as to better align the courses with learners’ actual requirements. Additionally, designing micro-courses could facilitate flexible self-directed learning for learners.

(2) Online self-directed professional learners generally demonstrate good performance (with an average performance score of 0.8 and an average coefficient of variation of 15%). Learners’ performance can be classified into four types, poor, average, good, and excellent, with proportions of 4%, 19%, 30%, and 47%, respectively. Learners with different performance exhibit significant differences in their scores on course exams, practice exams, and quizzes. Specifically, all learners perform relatively well on quizzes (with an average score of 0.95), which may be attributed to quizzes typically covering more fundamental and straightforward knowledge, which is easier for learners to grasp and address. However, significant disparities exist between learners classified as poor and others in terms of course and practice exams. These findings suggest the need for targeted assistance and support tailored to address the difficulties encountered by learners with poor performance in course learning and practice.

(3) Online self-directed professional learners generally rate courses highly (with an average score of 0.94), indicating their approval of the course content and activities. However, the number of courses purchased by learners is quite low (with an average score of 0.15), and there is significant variation (with a coefficient of variation of 93%). This discrepancy suggests that there may not be a direct correlation between learners’ satisfaction with courses and their purchasing. The satisfaction of learners is categorized into three types, neutral, satisfied, and delighted, accounting for 76%, 17%, and 7%, respectively. Despite the majority of learners expressing contentment with the courses, the limited purchasing may stem from the homogeneity of many courses, failing to address the diverse needs of learners. Therefore, the institution should offer a more diverse array of courses to cater to varying learner needs.

(4) There is a significant positive correlation between the three indicators of engagement (with an average correlation coefficient of 0.82), suggesting that online self-directed professional learners’ participation in various learning activities is interrelated. This correlation may be due to the inherent strong connections between learning tasks or because highly engaged learners often possess strong learning motivation, driving them to participate in all types of learning activities.

The three performance indicators exhibit a moderate positive correlation (with an average correlation coefficient of 0.39), but the correlation between course exams and quiz accuracy is relatively weak (r = 0.26). This might be attributed to learners’ ability to master fundamental knowledge and specific skills effectively, while struggling at times in addressing complex professional issues that require integrated problem-solving abilities.

It is noteworthy that there is no significant correlation (r = −0.0043) between the two features representing satisfaction, indicating that online self-directed learners’ satisfaction with a course does not necessarily lead to course purchases. The possible reasons for this include, firstly, the clear external motivation of online self-directed professional learners, who prioritize enhancing specific occupational skills that they urgently need, diverging from a conventional emphasis on systematic learning. Another possible reason is the current homogeneity of online career courses, where learners may be reluctant to invest additional economic and time costs due to perceived repetition.

Additionally, this study identified a certain correlation between purchases and engagement (average r = 0.12), with this correlation being more pronounced than that between ratings and engagement (r = 0.08). This suggests that financial investment plays a crucial role in motivating online self-directed professional learners to participate in the learning process, indicating that learners may place greater value on learning opportunities due to the monetary commitment.

Overall, there is no significant correlation between the engagement, performance, and satisfaction of online self-directed professional learners, which does not entirely align with many previous research findings. However, this discrepancy may be attributed to various unique factors inherent in online self-directed professional learning. These factors include the learners’ strong motivation and solid foundation, enabling them to achieve commendable performance, while external factors such as family and work obligations may hinder their sustained engagement in course learning.

(5) Regardless of whether the time frame is short-term (1 day) or medium-to-long-term (3 days, 7 days), the lagged autocorrelation coefficients of lesson learning sequences approach zero (average value of 0.05). This indicates that there is little linear autocorrelation between lesson learning on consecutive days, suggesting that learning on prior days cannot reliably predict the learning status on the current day. Furthermore, the large coefficients of variation (CV = 0.81), complexity-invariant distance complexity (CID = 0.41), skewness (1.15–5.72), and kurtosis (1.94–42.42) further confirm the substantial variability and extremity of lesson learning among learners across different days. A possible explanation is that, on one hand, online learning offers a high degree of flexibility and autonomy, allowing learners to choose their own study times. This flexibility may lead to intermittent and discontinuous learning sequences. On the other hand, online self-directed professional learners may be susceptible to various external disturbances, preventing them from consistently following a fixed schedule for course learning.

(6) Among the three quantified learning features, the number of questions answered in quizzes (Q_Q_nums) has the most significant impact on course grades (average SHAP value of 0.058). Additionally, the SHAP value distribution is wide, indicating the bidirectional influence of this feature on course performance. Quizzes are typically used to evaluate learners’ immediate understanding, and Q_Q_nums directly reflects learners’ familiarity with the subject matter and the frequency of their reviews. The frequency of practice exams (P_E_nums) has the smallest impact on course performance (average SHAP value of 0.032). This could be attributed to the low discriminative power of practice exams or learners’ less serious attitudes towards them. The impact of lesson learning (L_nums) lies between the aforementioned two features, with its SHAP values primarily concentrated in the positive range, indicating that this feature mainly has a positive effect on course performance. This aligns with the general understanding that the more time spent learning, the more likely one is to achieve better performance.

6. Conclusions and Further Work

6.1. Conclusions

To investigate the learning features, engagement, performance, and satisfaction of online self-directed professional learners, as well as to explore the relationships among them, a methodological framework based on LMS logs was devised and implemented in this study. This approach aimed to overcome the limitations of prior research, which relied on subjective self-reporting as the basis for the analysis of online learning engagement, performance, and satisfaction. We employed time series analysis, aggregation, normalization, and other methods to extract meaningful features from the raw LMS logs. Subsequently, we utilized unsupervised clustering to explore the types and proportions of engagement, performance, and satisfaction among online self-directed professional learners. The impact of the three quantified learning features on course performance was analyzed using SHAP values based on game theory, overcoming the limitations of classical methods such as linear regression and factor analysis, which often struggle to effectively explore nonlinear correlations.

6.2. Limitations

This study also has some limitations. First, the learning features included in the dataset are simple, and complex online learning features such as patterns of different learning activities and attitudes towards questioning and discussion were not effectively captured. This may limit the comprehensiveness of the findings. Secondly, the reasons behind learners exhibiting varying levels of engagement, performance, and satisfaction are speculative, lacking evidence-based validation. Lastly, although SHAP values based on game theory can analyze the nonlinear effects of different learning characteristics on course performance, there is a lack of in-depth investigation into the interactions between these features.

6.3. Further Work

In the future, it is imperative to collect LMS logs containing a broader range of learning features for a more comprehensive investigation. More methods, such as structural equation modeling, can be adapted to enhance the exploration of engagement, performance, and satisfaction among online self-directed professional learners and elucidate their interrelationships. Additionally, empirical research in more diverse courses to delve into the potential causes is essential.