Comparison of Sentiment Analysis Methods Used to Investigate the Quality of Teaching Aids Based on Virtual Simulators of Embedded Systems

Abstract

Virtual simulators of embedded systems and analyses of student surveys regarding their use at the early stage of the process of learning embedded systems, are presented in this article. The questionnaires were prepared in the Polish language, and the answers were automatically translated into English using two publicly available translators. The results of users’ experiences and feelings related to the use of virtual simulators are shown on the basis of detected sentiment using three chosen analysis methods: the Flair NLP library, the Pattern library, and the BERT NLP model. The results of the selected sentiment detection methods were compared and related to users reference answers, which gives information about the methods quality of the methods and their possible use in the automated review analysis process. This paper comprises detailed sentiment analysis results with a broader statistical approach for each question. Based on the students feedback and sentiment analysis, a new version of the TMSLAB v.2 virtual simulator was created.

1. Introduction

Sentiment analysis of student evaluation is a crucial area of research in understanding the opinions and attitudes of students towards various aspects of education [1,2,3,4,5,6]. It comprises students’ attitudes towards courses, teachers, laboratories, and different educational experiences. There is a growing trend in sentiment analysis in the context of student feedback, utilizing Natural Language Processing (NLP) [7,8,9,10], Deep Learning (DL) [11,12], and Machine Learning (ML) [13,14] techniques. The review papers [15,16,17] provide a systematic study on sentiment analysis of students’ feedback and the main research topics, venues, and top papers. The authors of [15] have analyzed 92 relevant papers showing the main trends in NLP, ML, and DL techniques. It was shown that the application of DL is rapidly growing.

The interesting proposal for sentiment analysis provides an article [18] that discusses the real-time identification and analysis of student sentiment in classroom teaching. It has many advantages compared to standard post-processing text analysis, as it can assist teachers in understanding student learning stages in time so that they can take appropriate actions. The literature review on the subject encompasses various methodologies and applications. Another proposal for real-time student feedback analysis can be found in [19]. The highest results were found for four aspects (preprocessing, features, machine learning, and use of the natural class) with Support Vector Machines (SVMs). Another application of SVM with a radial kernel and CNB (Complement Naïve Bayes) to analyze students’ feedback for real-time interventions in the classroom can be found in [20]. Some other proposals for real-time feedback can be found in [21,22].

Several authors were using different methods of sentiment analysis connected with remote and distance learning that were especially used during the COVID-19 pandemic [23,24,25,26]. The authors of [24] deployed a questionnaire to 660 postgraduate students who had access to lecture recordings. Key findings showed that students who accessed lecture recordings reported an enhanced learning experience.

The paper [27] concentrates on the preprocessing method for closed-ended questionnaires. It uses sentiment analysis through polarity. The main goal was to develop a mechanism for analyzing questions and students’ emotions in closed-ended responses. The authors proposed a Quest_SA software tool for this purpose. Another type of sentence whose sentiment can be analyzed are conditional sentences.

In turn, the paper [28] proposes a system for automatically extracting aspects from the available text and their corresponding orientation. The study proposes a supervised aspect-based opinion mining system based on a two-layered LSTM model. Another new method of sentiment analysis of student feedback is proposed in [29]. It proposes an innovative model to find the targets of the given sentence using Bi-Integrated Conditional Random Fields (CRFs).

The natural language processing of student online sentiment was analyzed in [30]. The authors were looking for new and better ways to support and understand the learning experience of their students. The idea was to investigate the attitudes and emotions of students when they were interacting on social media about their course experience. The short, informal texts were also analyzed in [31]. The sentiment features were primarily derived from novel, high-coverage tweet-specific sentiment lexicons.

The fuzzy logic rules are also used to analyze the sentiment and understand students’ opinions and satisfaction about the university and its services. Some original proposals can be found in [32,33]. Also, a neural network approach can be used in sentiment and behavior analysis [34].

Additionally, the sentiment analysis of student evaluations can benefit from the application of sentiment analysis in different languages, as demonstrated by [35,36,37] in the context of Korean, Vietnamese, and Kurdish, respectively. There is also a valuable, from the author’s point of view, example of sentiment analysis methods in Polish [38].

In the article, the authors focused on a comparative study of sentiment analysis methods [39]. The students’ opinions in Polish regarding embedded systems and their simulators used at the Institute of Automation of the Lodz University of Technology [40,41] were analyzed. These opinions contain various emotions and impressions that accompanied the students’ work both during laboratory classes and on their own. Based on the analysis of the described student opinions, it was proposed to implement several improvements to the simulators.

For the comparison of sentiment analysis methods, available tools with pretrained NLP models were picked. There are a large number of these types of tools [42], especially with their availability in the Python language [43]. The following requirements were taken into account when choosing analysis methods:

• Ability to analyze texts composed of many sentences [44];
• At least a five-level sentiment analysis scale returned;
• Availability of ready-to-use, pre-trained NLP models [45].

Based on these criteria, Pattern [46], BERT [36], and Flair [47] were used for sentiment analysis.

Since the survey was conducted in Polish and the NLP models for sentiment analysis are trained mainly based on English [48], automatic translations [49] using two translators were used. The two publicly available translators were compared: Google Translator [50] and MyMemory Translator [51].

As a final result of the statistical processing analysis of detected sentiments in relation to the reference answers provided by the participants of the proposed survey, the best method for determining sentiment in Polish surveys with automated language translation was selected.

2. Virtual Simulators of Didactical Embedded Systems

2.1. Embedded Systems at the Institute of Automatic Control at Lodz University of Technology

The Institute of Automatic Control at the Lodz University of Technology (LUT) [52] has a long tradition of teaching embedded systems [53]. It has an extensive hardware base and covers typical issues of using these systems in real applications. The set of didactic microprocessor modules shown in Figure 1 is a subset of the hardware base and is used in two subjects: Microprocessor Techniques [54] and Microprocessor Systems Software [53].

This didactic set is, in some cases, special because each hardware module has its own virtual simulator [40,41]. These simulators are increasing the flexibility of students’ work and thus improving the quality of teaching [55].

2.2. The TMSLAB Didactical Module Simulator

Virtual simulators of embedded systems are very important for students. The authors of the paper draw that conclusion based upon nearly 30 years of teaching experience at the LUT and recent experiences connected with the COVID-19 pandemic. The teaching goals are clear—providing access to high-quality material to study outside the university in order to improve skills and knowledge. Specialized laboratories limit the opportunity for education without access to hardware resources. Embedded systems programming classes provide a unique opportunity to conduct education outside the university, providing students with appropriate tools, in this case simulators of embedded systems used in the classes. However, these systems have a unique architecture, and developing simulators for them is not an easy task. Therefore, before upgrading or modifying a given tool, knowledge about the opinions and feelings of the students on this subject is needed. At the same time, students constantly emphasize how much simulators make it easier for them to work on laboratory tasks, whether it is in terms of preparation for task problems or in connection with the need to complete work started during classes.

One of the microprocessor modules used in teaching classes is the TMSLAB system [53]. Its current, second version, is based on the Texas Instruments [56] TMS320F28379D MCU and is equipped with intermediate circuits enabling the use of a graphic-text LCD, a 4 × 4 matrix keyboard, and an LED line. The laboratory module evolved from the first version, equipped with the TMS320F2812 MCU system and a similar set of peripherals [41]. With a lot of effort, a system-level simulator [41] was introduced a few years ago for TMSLAB v.1 (Figure 1) and has been used since then in laboratory classes.

The introduction of a new version of the TMSLAB v.2 laboratory hardware module during classes resulted in several differences from the first version. The developed simulator, TMSLAB v.1, was still useful, but students had the impression that it was not suitable for the equipment, which resulted in their different reactions.

Feedback from students was important about the need to further develop the simulator, and therefore it was necessary to conduct an appropriate survey. Finally, at the beginning of 2024, a decision was made to update the simulator, which resulted in the development of its newer version, shown in Figure 2, in the form of the TMSLAB v.2 simulator.

3. Computing Methods and Tools for Sentiment Analysis

3.1. User Feedback as a Guideline for Product Improvements

In addition to the desired didactic effect related to the use of virtual simulators of embedded systems, the authors also aimed to encourage students to use these tools. This required the implementation of some simulator functionality that is not only about elements that lecturers care about but also taking into account users’ opinions. Therefore, simulator developers need to consider components that are needed from a purposeful perspective but also which functionalities are redundant and which are missing.

A good method to obtain information about the last two aspects is through feedback from surveys. The questions they contain may be open or closed; they may concern very narrow and precisely formulated issues; or they may allow greater freedom of expression [27]. As a final result, the goal of the survey is to obtain feedback related to the purpose of conducting it.

3.2. Text Sentiment Analysis

The opinion and sentiment expressed in the respondent statement are very important pieces of feedback. This sentiment guides the interpretation of the user’s answers [33] and allows for a better understanding of his needs and emotions [15]. With a large number of surveys, manual sentiment analysis takes a lot of time and requires the operator’s work. The results of manual analyses may not be repeatable and may depend on the emotional state of the person interpreting the answers [42]. Automated sentiment detection systems are free of these drawbacks, the results of which can be used to automate processes that meet the needs of respondents [17].

Basically, the result of the sentiment analysis should be the classification of the text into one of the following categories: positive or negative. More precise analysis techniques add a neutral (mixed) category, or additionally, positive mixed and negative mixed. Still, other sentiment analysis methods use indicators that are numbers in the range usually [0,1] or [−1,1], specifying a scale from negative to positive for sentiment description and additionally supplementing the sentiment estimation with a parameter of its detection quality [30].

In accordance with the assumptions presented in the introduction, three tools available for research and comparison were chosen. They were accordingly:

• The Flair NLP library [47];
• The Pattern library [46];
• The BERT NLP model [36].

All three tools were used with default parameters and were in versions: Flair (BLSTM—Bidirectional Long Short-Term Memory) v. 0.13.0, Pattern v.3.6 (DbA—Dictionary-based Approach, SLP classifier based on WSJ trained on the John = ”NNP-PERS” dictionary), and Multilingual BERT (mBERT) used in Transformers library v. 4.35.2 (BERT was used with a pretrained model based on English datasets for a multilingual model).

For the purposes of this article, results were unified to give results in a range from one to five, with subsequent sentiment levels assigned to: positive, mixed positive, mixed, mixed negative, and negative sentiment. All the sentiment analysis methods used in the article were scaled to those five levels to give comparable results. Typically, the literature considers a two-point, three-point, and five-point sentiment. The authors decided to use the last variant, which allows for sentiment assessment with greater diversity, providing better comparison possibilities for analysis methods. The libraries used for analysis return not only the probability of sentiment classification but also a five-point sentiment assessment scale. The probability of the determined sentiment result was used as an element of filtering the results, introducing two sub-variants of the comparison of research results.

4. Research Methodology

4.1. The Quality of Teaching Aids Based on Virtual Simulators of the Embedded System Survey

The results presented in this article are based on a survey prepared to verify the suitability of methods and tools used in the subject of Microprocessor Systems Software [53] lectured at the Institute of Automatic Control, Lodz University of Technology [52]. The survey was prepared in such a way as to obtain answers to questions about the quality of tools and materials supporting the teaching process and not discourage students from completing it. This forced us to limit the questions to the minimum necessary set, giving a picture of opinions, thoughts, and emotions related to the use of embedded system simulators. The survey consisted of five open questions and five corresponding closed questions with answers on a point scale. The survey itself was conducted on the university teaching platform WIKAMP [57], which is based on Moodle [58]. Essentially, the students were asked about:

• The need for using software simulators in question 1;
• Thoughts on the TMSLAB module simulator in question 2;
• The significance of creating a simulator of the STMLAB system supporting FreeRTOS in question 3;
• The impressions, feelings, and emotions they experienced when using microprocessor system simulators during laboratory classes in question 4;
• Opinions regarding the plan to introduce an additional microprocessor system with a simulator and an additional programming task within unchanged teaching hours in question 5.

The survey was not obligatory. It was completed by students interested in providing feedback, which increases its reliability. A total of 14 questionnaires were collected from a group of 60 students. This is a satisfactory result, considering the fact that the semester surveys completed by students at the Lodz University of Technology as part of the Microprocessor Systems Software course, regarding the quality of teaching, are usually completed by a much smaller group. Some of the students aversion to completing surveys may be related to the requirement of many forms of surveys at the university level, which results in overloading the students in this area.

4.2. Sentiment Analysis of Open Questions

The open answers to the survey were posted in Polish. The intention at this point was to avoid errors in the quality of the respondents’ answers resulting from the possible insufficiently precise expression of emotions in a foreign language. This forced the use of language translators to translate the answers into English. Two publicly available translators were used. Google Translator [50] and MyMemory [51] translators. The results of using both translation tools are included in the article.

Due to the exhaustive answers of some respondents, the article also presents two variants of sentiment analysis. The first one, in which entire answers were analyzed, and the second one, where the answers were divided into sentences that were subject to separate analysis. The second approach posed a serious challenge to the compared sentiment analysis methods because some sentences, taken from a broader opinion, may not correctly represent its sentiment.

4.3. Survey Reference Questions

The sentiment detected in the survey responses was verified based on the answers provided by the respondents themselves. For each open question, there was one closed question with an answer on a five-point scale. This scale corresponded to the five levels of detected sentiment. On this basis, the sentiment detection error was determined, which was used to compare sentiment analysis methods. This error was calculated according to the following formula:

$$\mathsf{\Delta }S=\left|S_{ref}-\widehat{S}\right|$$

(1)

where ΔS—sentiment error, $S_{ref}—$sentiment reference level, and $\widehat{S}$—sentiment detected level.

Only those survey answers that led to the determination of Equation (1) were used for result interpretation. This means that the statistical analysis conducted in the article was performed only in cases where both the open question and the corresponding closed question were answered.

5. Sentiment Analysis Results

5.1. Sentiment of Survey Answers

All fourteen collected surveys were subjected to sentiment analysis, considering two different language translators, two sets of responses (complete answers and their split into single sentences), and two groups of results: all results and results rejecting low reliability ones. The assumption made for reliable sentiment detection results is that the method returning the sentiment estimation quality indicator returns its value above half of its total range. For all possible combinations, the results of sentiment are shown in Figure 3 for using Google translator and in Figure 4 for using MyMemory translator.

The presented results of the sentiment analysis have been divided according to the question number and presented in statistical terms with the median and average values separately for each of the analysis methods and reference responses that play a verification role. To better illustrate the collected data and their analysis, a broader statistical approach is presented individually for each question in the attachments in Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6, Figure A7, Figure A8, Figure A9, Figure A10, Figure A11, Figure A12, Figure A13, Figure A14, Figure A15 and Figure A16 for all study cases. There are both probability density distributions obtained for sentiment analysis methods and reference responses, as well as histograms that cover this issue as a supplement to the documentation of statistics regarding the obtained research results. The statistical results included in Appendix A to the article act as complete documentation of the survey results without the need to publish confidential results—another form of sharing survey results. They are also an intermediate element in determining the sentiment estimation error (1), the results of which are presented in Section 5.2.

5.2. Sentiment Analysis Errors

The obtained sentiment analysis results were related to the reference responses in accordance with Equation (1). On this basis, sentiment detection errors were determined and presented in Figure 5 and Figure 6 for all study variants.

Similarly to the collected research analysis results, the obtained statistics of sentiment detection errors are presented in more detail in Appendix A in Figure A17, Figure A18, Figure A19, Figure A20, Figure A21, Figure A22, Figure A23 and Figure A24. Those figures cover not only errors for each question, like in Figure 5 and Figure 6, but also for each sentiment level, which gives a broader look at the issue.

6. Discussion

The conducted review study showed general consistency between emotions expressed descriptively and the respondents ratings from reference answers. Every method used tended to correctly identify the sentiment, but each of them was characterized by different accuracy and statistical properties. Summary error statistics obtained as a result of the study conducted for each analysis method and all study criteria are shown in Figure 7 and Figure 8.

Based on the review answers detected sentiment levels (Figure 3 and Figure 4), their accuracy expressed as detection error (Figure 5 and Figure 6), and the summarized error for all obtained results per method (Figure 7 and Figure 8), several interesting conclusions can be drawn:

• Each method produced results that make it useful for sentiment analysis. In no case did the average error value of the results exceed two sentiment levels;
• As was to be expected, the BERT model is characterized by the highest precision in sentiment detection. Regardless of the analysis used, the average error made by this method was the smallest;
• The Flair library produced very highly polarized results and occasionally balanced ones. This resulted in a large impact on the error statistics made by this method with every detection mismatch. The method can be useful to distinguish the emotional state of opinions very clearly;
• The Pattern library gave very balanced results in terms of sentiment level. This leads to a reduction in the sentiment detection error but does not allow for a clear distinction between the emotional potential of the review responses;
• In the case of very short statements (splitting respondents’ answers into single sentences), all methods returned results consistent with expected ones. The BERT model, which was generally the best, lost the most in terms of analysis quality while at the same time signaling that the obtained results are unreliable;
• The method based on the Pattern library gave surprisingly good results, considering that it is very simple [46] compared to the other two;
• The change of the translator did not affect the nature or general interpretation of the results obtained. This leads to the indirect conclusion that the above methods can be successfully used to analyze the sentiment of texts in Polish using language translators;
• From a statistical point of view, the results obtained for complete answers were similar to the results of analyzing responses split into sentences. This gave a surprisingly high accuracy of analyses of very short texts, which may not necessarily contain emotional potential;
• The results of the error analysis indicate that in both cases of automatic translation, the potential impact of losing the nuances by the translation, discussed in literature [59,60], did not affect the accuracy of sentiment determination.

7. Conclusions

This article compares three sentiment analysis methods for short texts and presents their results in the case of analyzing the quality of teaching aids based on virtual simulators of embedded systems. The comparison took into account various criteria of text taken into account in the analysis, as well as two different language translator tools. It was shown that every sentiment analysis tool gave correct results when analyzing texts in Polish with automated language translations. Even limiting the length of the analyzed text to a single sentence, which may not be emotionally representative in statistical terms, gave satisfactory sentiment detection results.

The issue of potential loss of text nuances in automatic translation was also one of the matters investigated in the article. Two different translators and a given methodology for determining sentiment errors were used for this purpose. The effects of the error analysis show that in all the cases of automatic translation, the potential impact of nuances did not affect the accuracy of sentiment determination. This was observed for both full responses and when questionnaire answers were split into sentences. It reflects the high usability of sentiment analysis through automatic translation and opens the way for automatic sentiment analysis of short texts.

The conclusions of the sentiment analysis method are presented to highlight the strengths and weaknesses of each comparison method. The method based on BERT NLP gave the most precise results, while Pattern was more balanced and Flair strengthened the emotional potential of the statement, often giving mostly negative or positive sentiment.

The article describes what factors may influence the results and what aspects were not taken into account in the study. The results obtained from the comparative analysis are largely influenced by the size of the result set and the expected output sentiment. Due to the limited scope of the survey, the total number of respondents was 14. For this reason, some of the research was carried out by breaking down the survey into individual sentences to enhance their sentimental potential. The main limitation in assessing the obtained results was the level of reference sentiment, which gave less reliable comparative results on mixed sentiment. This was due to the fact that for the mixed reference sentiment, the maximum error that could be made in the analysis was two sentiment levels, while for borderline sentiments it was four levels, i.e., twice as much. For this reason, the questions that were prepared for the survey with the intention of the authors were intended to lead to obtaining borderline reference answers.

The paper also compared not only the currently leading methods but also the simplest ones, based on the DbA (Dictionary-based Approach). This method, considering the mechanisms it uses, may be several times faster than BERT and consume fewer resources, leading to its effective use in small embedded systems, but it requires further research.

Based on the students feedback and sentiment analysis, a new version of the TMSLAB v.2 simulator with more realistic graphics, greater control over the output code and data size in the simulated IDE, and a more interactive user interface was prepared. The authors will also provide our students in the future with a newly developed starter project for the STMLAB simulator (based on the STM32F429 MCU QEMU emulator and LCD system-level simulation) with new features in the form of STM LL (low-layer) drivers and FreeRTOS support in the STMLAB simulator. The ability to run this type of software in a simulated environment is desirable and was assessed mainly positively in the survey. As a result of respondents answers sentiment analysis, the intention to introduce an additional microprocessor module into the teaching program was abandoned, which, according to the survey, would have been perceived relatively negatively.