A Method for Evaluating User Interface Satisfaction Using Facial Recognition Technology and a PSO-BP Neural Network

Abstract

User satisfaction serves as a crucial reference for iteratively optimizing software interface designs. This paper introduces a comprehensive measurement model of user satisfaction, employing Notability and Goodnotes for case studies. The proposed model incorporates facial recognition technology to gauge the intensity of users’ facial expressions while interacting with various functions of the target interface. Additionally, an experimental observation method is employed to gather objective data, including task completion time, task success rates, and operational procedures, alongside users’ subjective evaluations. Leveraging objective data as input and subjective ratings as output, a user satisfaction prediction model based on a PSO-BP neural network has been devised. The results demonstrate an impressive accuracy rate of 86.26%, indicating a high accuracy in subjective perception prediction. This model has proven to be effective for measuring user satisfaction and evaluating software interface usability. Moreover, this research contributes to expanding the repertoire of user interface satisfaction evaluation methods, enhancing the objectivity of measurements and surpassing the efficiency of conventional experimental evaluation techniques. The proposed model holds practical significance for software interface usability assessment and optimization design.

1. Introduction

In the digital era, the dissemination of information and interaction are predominantly facilitated through various software user interfaces. The satisfaction derived from interfaces plays a crucial role in the process of information interaction, serving as a key factor in optimizing and refining both software and related products. The concept of customer satisfaction, initially introduced by American scholar R.N. Cardozo [1] in 1965, relates to the psychological state of customers in terms of meeting or exceeding their expectations when acquiring goods and services. According to ISO standard 9241-11 [2], satisfaction is defined as the user’s comfort and acceptability in product usage. Enhanced user satisfaction correlates with improved user experience. Current methodologies for evaluating user satisfaction predominantly rely on subjective assessments, occasionally supported by objective metrics, yet these approaches lack efficiency and accuracy. Current studies assessing user satisfaction through objective data remain deficient. Therefore, this paper proposes a novel approach to measure user satisfaction, integrating facial recognition technology with a particle-swarm-optimization-enhanced backpropagation neural network (PSO-BPNN). Given that this study’s participants are teachers and students, this research selects Notability and Goodnotes as the subjects for experimental evaluation. These applications, predominantly utilized by the educational cohort in China for their routine academic activities, were chosen to streamline sample selection and enhance data accuracy. This study collects objective data on facial expression intensity, task completion times, success rates, operational sequences, and subjective satisfaction scores during the usage of these applications, establishing a PSO-BPNN-based model. This model forecasts the subjective satisfaction with the software interface, leveraging objective data to refine the evaluation’s accuracy and objectivity.

The second section offers a literature review on related studies. Subsequent sections, three and four, detail the research methodology, encompassing data collection and analysis, leading to the development of a predictive model. The fifth and sixth sections analyze and validate the findings. The seventh section concludes the study, suggesting areas for future research and potential solutions.

2. Related Research

Various methods exist for assessing user satisfaction, with subjective evaluation being the primary technique, supplemented by both subjective and objective comprehensive assessments. Currently, the evaluation of interface satisfaction primarily includes user subjective assessments, objective evaluations of behavioral and physiological data, and assessments based on computer algorithms. The subjective approach to evaluating user satisfaction typically involves collecting subjective and experiential data from users via Delphi methods, questionnaire surveys, interview surveys, and think-aloud protocols [3,4,5,6]. These methods are straightforward and form the cornerstone of user satisfaction assessment. However, an exclusive reliance on subjective evaluation invites concerns regarding accuracy and objectivity.

Several studies have employed objective behavioral analysis and physiological data metrics to assess user satisfaction with specific products and software. Utilizing usability testing foundations, objective behavior analysis has emerged as a prevalent method for evaluating software interface satisfaction [7,8]. Several studies have utilized a combination of online reviews [9], Delphi methods, and interview surveys to measure product and interface satisfaction within specific domains, such as user satisfaction with digital resources [10,11]. Online reviews are effective for assessing user satisfaction but are limited to software with extensive user feedback. Physiological measures, notably EEG, eye movement analysis, and facial recognition technology, are widely used in evaluating interface satisfaction. Guo Fu et al. [12,13] demonstrated that EEG technology analysis could be utilized for assessing interface satisfaction, revealing a correlation between satisfaction assessments and neural responses. EEG data are accurate and objective but have a complex and time-consuming experimental process with strict participant criteria, reducing research efficiency. Eye movement experiments are prevalent in evaluating interface satisfaction, with numerous studies establishing a correlation between objective eye movement data and users’ subjective satisfaction evaluations [14,15,16,17]. Eye tracking data primarily reflect visual features and aesthetic preferences, not emotions and feelings. Related studies typically employ eye-tracking data to substantiate the usability of software interfaces, subsequently deriving interface satisfaction evaluations from the established link between usability and satisfaction. However, these studies do not primarily focus on interface satisfaction evaluation. Facial expression recognition technology is another tool for user-satisfaction-related research. Wang Huanhuan et al. [18] proposed using facial expressions as indicators in usability research, offering a basis for assessing users’ subjective satisfaction. Ana Laura Alves et al. [19] applied questionnaires, scales, and FaceReader 9.1 software to explore how age-related colors affect product satisfaction. With the progression of usability research methods, the measurement and analysis of expressions are garnering increased attention. Acquiring physiological data, like EEG and eye movements, necessitates using specialized, costly equipment such as eye trackers and EEG machines. In contrast, gathering expression data is more straightforward and cost-effective, requiring only a standard camera to capture facial expressions. This approach notably reduces experimental costs and enhances testing efficiency. Facial expressions directly mirror users’ emotional states and are closely linked to satisfaction, making facial expression recognition and analysis technologies practical and valuable for evaluating satisfaction. Accordingly, this study employs facial expression recognition technology to assess interface satisfaction.

The rapid progression in computer algorithms has led to the widespread adoption of various prediction models. In recent years, the product design field has also tried to use BP neural networks and other algorithms to predict and assess design solutions [20,21,22]. These researches offer a scientific theoretical foundation for design evaluation. BP neural networks have been effectively utilized for predicting and evaluating user satisfaction. Recent studies, including those by Zhong Jian et al. [23], have leveraged BP neural networks to assess user satisfaction in software, as well as in vehicular and machine interaction products, presenting a fresh perspective on interface satisfaction evaluation. Software user satisfaction involves many complex factors, including user behavior, emotional response, and operational efficiency. There are often complex nonlinear relationships between these factors. Traditional neural networks are adept at handling these nonlinear relationships. However, in practical applications, the prediction results can be unstable and sometimes exhibit low accuracy due to issues with the initial weights and threshold values. To address these challenges, recent studies have integrated particle swarm optimization to improve the predictive models, thereby increasing both their efficiency and accuracy [24,25]. Particle swarm optimization (PSO) is extensively used across engineering fields as a population-based optimization algorithm. It has a fast convergence speed, strong global search capability, high robustness, and does not rely on the characteristic information of the problem itself [26]. Yang Yanpu [27] utilized the particle swarm optimization (PSO) technique to optimize users’ Kansei evaluation matrices, achieving consensus, proposing a perceptual evaluation process for product shape design based on HFLTSs and PSO. His case study on car-charging pile design confirmed that HFLTSs help resolve users’ perceptual uncertainty and hesitancy. Integrating PSO enhanced the consistency of evaluations, ultimately improving the quality of product form assessment. User evaluations of product styling are markedly similar to satisfaction assessments of software interfaces. To improve the accuracy and stability of the BP neural network prediction model, in this paper, we establish a prediction model using the particle swarm optimization-backpropagation (PSO-BP) neural network to predict user satisfaction. Specifically, the PSO algorithm optimizes the connection weights of BP neural networks, overcoming their inherent limitations. This approach enhances the neural network’s ability to search and adjust parameter spaces more effectively, thereby improving the model’s generalization ability and predictive performance in complex environments.

In summary, this study selects the most popular note-taking software interface in education as its research platform, employing facial expression recognition technology and the PSO-BP neural network model to predict user satisfaction.

3. Research Methods

3.1. Study Design

This study conducted two experiments where participants used Notability and Goodnotes to complete various tasks. In the first experiment, objective data and subjective ratings were collected as participants used Notability, aiming to develop a user satisfaction prediction model. The second experiment involved Goodnotes to validate this model. The model employs a particle-swarm-optimization-enhanced BP neural network algorithm. The methodology of this process is depicted in Figure 1.

Additionally, refer to relevant research [28,29,30], a BP neural network prediction model is established, and its results are compared to confirm the superiority of the PSO-BP neural network prediction model.

3.2. Construction of the PSO-BP Neural Network Model

3.2.1. Particle Swarm Optimization

Particle swarm optimization (PSO) simulates birds’ migratory and foraging behavior, initializing particles in the feasible solution space, to represent potential optimal solutions. Each particle, with an initial speed and direction, iteratively seeks the optimal solution by adjusting its motion based on individual and collective best outcomes. In a D-dimensional space with n particles, the position of the i-th particle is represented as X_i = (X_i1, X_i2, …, X_id), and its velocity as V_i = (V_i1, V_i2, …, V_id). The best-known individual position is P_i = (P_i1, P_i2, …, P_id), and the global optimum, shared by the swarm, is P_g = (P_g1, P_g2, …, P_gd). The velocity and position update formulas for a particle are delineated as follows:

$${\mathrm{v}}_{ij}^{t+1}=w{v}_{ij}^t+c_1{r}_1(p_{ij}^t-x_{ij}^t)+c_2{r}_2(p_{gj}^t-x_{ij}^t)$$

(1)

$$x_{ij}^{t+1}=x_{ij}^t+v_{ij}^{t+1}$$

(2)

$$w(\mathrm{t})=w_{\max }-\frac{t(w_{\max }-w_{\min })}{t_{\max }}$$

(3)

In this context, i″ ranges from 1 to n″, each number representing a particle in the swarm; j″ spans from 1 to D″, denoting the dimensionality of each particle. The iteration count is represented by t″, with t_‘max’ signifying the maximum allowed iterations. The learning factors are given as c_‘1’ and c_‘2’. Random factors r_‘1’ and r_‘2’ are distributed within the [0, 1] interval. The inertia weight w″ helps balance global and local search capabilities, w_‘max’ and w_‘min’ defining its upper and lower limits. These elements collectively dictate the velocity and position updates of the particles in the PSO process.

3.2.2. BP Neural Network Algorithm

The architecture of a BP neural network typically consists of one input layer, several hidden layers, and one output layer. It operates by receiving signal inputs (X₁, X₂, …, X_n) at the input layer, calculating the initial weights through the hidden layers, and the output results are then compared with the actual results. Errors are then backpropagated to the input layer, and the gradient descent method adjusts the weights of the hidden layer. After a certain number of training iterations, the final prediction result (Y) is produced. The accuracy of the prediction is affected by various factors, including the initial weights of the BP neural network, the choice of activation function, the number of hidden layers, the neurons in each hidden layer, and the training iterations. Adjusting these parameters enables achieving the desired prediction accuracy.

3.2.3. PSO-BP Neural Network Algorithm

The BP neural network possesses robust nonlinear mapping capabilities. However, its reliance on gradient-descent-based error backpropagation for adjusting network weights and thresholds often results in local optimum solutions, thereby diminishing prediction accuracy. To mitigate this, the PSO-BP neural network integrates the weights and thresholds of the BP neural network into the framework of PSO particles, allowing for iterative optimization of these weights and thresholds by updating the particles’ velocities and positions. This integration enhances the BP neural network’s convergence speed and prediction accuracy. The methodology is depicted in Figure 1 and progresses through the following stages:

(1)

Constructing the PSO-BP neural network, entailing the initialization of network parameters and the random assignment of velocities and positions to the particle swarm.

(2)

Computing the fitness for all particles, employing a fitness function that calculates the mean squared error between the BP neural network’s output and the target values. A lower fitness value signifies reduced error and thus an improved particle position. The fitness function is defined as follows:

$$Fitness=\frac{1}{Z}\sum _{s=1}^Z\sum _{j=1}^C{\left({{\displaystyle W}}_{sj}^d-{{\displaystyle W}}_{sj}\right)}^2$$

(4)

where Z is the number of samples; C is the number of output network neurons; $W_{sj}^d$ is the ideal value of the JTH output network node of the s sample; and $W_{sj}$ is the actual value of the JTH output network node of the STH sample. In the PSO-BP neural network, each particle’s individual best-known position is denoted as Gbest, while the global best-known position across the entire swarm is represented as Zbest. The process involves comparing each particle’s fitness value with its current Gbest. If a particle’s fitness is superior to its existing Gbest, this fitness value is then adopted as the new Gbest for that particular particle. Moreover, if any particle’s Gbest surpasses the current Zbest of the swarm, this Gbest value is then updated to become the new Zbest.

(3)

In accordance with Equations (1) and (2), the PSO-BP neural network iteratively updates the velocity and position of each particle, guided by individual and global optimal positions. Upon reaching the maximum number of iterations, or achieving the desired error accuracy, the PSO-determined optimized initial weights and thresholds are integrated into the neural network for predictive analysis. The network configuration is then saved. If these criteria are not met, the process reverts to the preceding step for further adjustments.

(4)

The final stage involves applying the data from Goodnotes to this trained network model to validate its efficacy and accuracy in a different yet relevant context.

The flowchart of the PSO-BP neural network is depicted in Figure 2.

3.3. Selection and Collection of User Data

Incorporating facial expressions and user interface action data as integral components of the model’s input is necessary. Since user satisfaction is intimately linked to the specific interactions that users engage in during product use, interface operation data are also included as a crucial element of the model’s input. Following the completion of assigned tasks by the users, satisfaction with each function is evaluated through a questionnaire survey (see Appendix A). The subjective evaluations of the users are then compared with the predicted results, serving as the actual benchmark for user satisfaction.

3.3.1. Facial Expression Data

For facial expression analysis, software options include Gface and FaceReader, the latter being extensively utilized in domains such as consumer behavior and psychological research. FaceReader, known for its over 88% accuracy in facial expression recognition as per official data, is the chosen tool in this study. Its analysis interface is depicted in Figure 3. In facial expression analysis, the camera is initially utilized to capture the user’s facial reactions during software interaction. Subsequently, the collected facial expressions undergo facial expression recognition (FER) using FaceReader software 9.0. The software adopts EKMAN’s classification of expressions, categorizing expressions into six types: joy, sadness, anger, surprise, fear, and disgust. Additionally, a category labeled “neutral” is incorporated, and corresponding intensity values for each emotion are assigned based on the facial characteristics observed at each moment.

However, there are ethical issues involved in collecting facial expression data. Before using this technique, a very detailed explanation of the purpose of this study and the intention to use the data was given to each subject. All subjects signed an informed consent form and a confidentiality agreement.

3.3.2. Interface Operation Data

Interface operation efficiency is a critical determinant of user satisfaction. Pearson and Schaik [31] employed accuracy and speed as key metrics to investigate the impact of navigational hypertext link placement and color on search and interactive tasks. Liu Wen et al. [32] developed a usability evaluation index system grounded in the IPO model, where the operational efficacy in the usability assessment of combat command systems was segmented into task completion rate and time. This framework offered insights for refining the usability of combat command software interfaces and enhancing their human–computer interaction. Based on a literature review, this study identified task completion time and rate as pivotal indices for assessing interface operations.

The collected interface operation data encompassed the duration needed by users to complete each task, task completion status, and detailed user interface steps. Task completion time and status were directly derived from experimental records. However, the specific steps for the same task could vary significantly across different systems. To quantify user operation steps, this study utilized the Lostness concept by Smith [33]. This metric incorporates the number of distinct pages N visited, total page visits S (including repeated visits), and the minimum necessary page visits R for task completion. Lostness is calculated using the formula presented in (5), ranging from 0 to 1, with 0 being ideal. A Lostness degree exceeding 0.5 indicates notable navigation difficulty.

$$\begin{array}{c}\mathrm{L}=\sqrt[2]{{\left(\frac{\mathrm{N}}{\mathrm{S}}-1\right)}^2+{\left(\frac{\mathrm{R}}{\mathrm{N}}-1\right)}^2}\end{array}$$

(5)

3.3.3. User Subjective Satisfaction Score

In this study, users’ subjective satisfaction scores were obtained through a questionnaire survey, indicating their actual satisfaction. Following the completion of relevant tasks, satisfaction with each task was evaluated individually on a scale ranging from 1 to 10, with 1 indicating very dissatisfied and 10 very satisfied’ (see Appendix A). During the model construction phase, users’ subjective satisfaction scores were compared with the predicted outcomes, prompting adjustments to the model. Subsequently, in the model validation phase, these scores were compared with the model’s predicted results to assess the model’s accuracy.

4. Research Process

4.1. Selection of Software Functions

Numerous note-taking software options are currently available, with Notability and Goodnotes leading the industry in the Chinese market, scoring 440,000 and 500,000, respectively, on the Apple App Store. Therefore, this study selects these two software platforms for separate sets of experiments. The first set establishes an interface satisfaction prediction model using relevant experimental data from Notability, while the second experiment validates the model’s effectiveness and accuracy using pertinent data from Goodnotes.

During the model-building process, note-taking software functions were selected based on user frequency to ensure the model’s relevance to common usage scenarios. A questionnaire survey was conducted to gauge the frequency of use for various functions within the two note-taking software options. The questionnaire, comprising 10 functions, prompted users to assign scores (ranging from 1 to 10) reflecting their frequency of use: 1 for no use, 2–4 for minimal use (less than monthly), 5–6 for occasional use (monthly to less than weekly), 7–8 for regular use (weekly to less than thrice weekly), and 9–10 for very frequent use (three or more times weekly) (see Appendix B).

Distributed among students and teachers, the survey garnered 73 valid responses. The average scores for each function’s usage frequency were tabulated (refer to

Table 1. The usage frequency score of the main functions of the note-taking software (N = 73).

Feature	Pen	Eraser	Insert Media	Lasso Tool	Highlighter	Text Box	Catalogue	Amplification	Recording	Laser Pointer
Mean	8.92	8.29	6.67	6.48	5.92	5.48	5.32	3.83	3.57	2.92
Standard Deviation	0.96	1.26	1.38	1.35	1.34	1.6	1.6	1.73	1.72	1.86
Variance	0.93	1.59	1.91	1.81	1.80	2.56	2.68	3.00	2.96	3.44

). Analysis of the survey revealed that functions such as pen, eraser, insert media, and lasso were utilized most frequently, each receiving scores exceeding 6. Consequently, these four functions were selected for experimentation and model establishment.

4.2. Participants

Through empirical observation, note-taking software primarily attracts young professionals and students. Participants aged 18 to 35, mainly college students and office workers, were recruited to ensure relevance. To mitigate the influence of prior software experience, only individuals unfamiliar with the software were chosen, resulting in 44 recruits. All participants, with normal physiological functions and diverse app experience, provided informed consent. They were divided evenly into two groups, A and B, with 22 individuals each, to ensure similar age and gender distributions. Group A used Notability for model training, while Group B used Goodnotes to validate the model. Forty-two valid datasets were obtained, with a mean age of 23 years and a standard deviation of 4.56. The gender distribution was balanced, maintaining a 1:1 male-to-female ratio.

4.3. Experimental Environment and Equipment

The experiment used an iPad Pro 2021 with a 12.9-inch display. Both Notability and Goodnotes were pre-installed and verified for functionality prior to testing. Testing occurred in classroom or office settings to simulate real-world usage. Screen recording and video capture were used to collect data on task completion times, outcomes, and procedures, while facial expressions were recorded for analysis. A nearby rest area was provided for participants not involved in testing.

4.4. Experimental Process

Before the experiment began, the video recording equipment and the settings of the Notability and Goodnotes software were adjusted to standardize the initial parameters (e.g., brush shape, brush color) for each participant. To prevent any interference, participants not involved in the experiment were accommodated in a separate classroom. Participants signaled their readiness to the experiment supervisor, and upon confirmation, the experiment commenced. They were presented with four tasks through interactive text prompts on the interaction page:

• using orange dotted handwriting to draw a heart;
• inserting a picture and placing it isometrically into the designated box;
• using the eraser to erase the purple lines;
• changing the handwriting on the picture to black and resizing it to fit into the box.

Figure 4 depicts the interface designs and task instructions for both applications. The illustrations primarily focused on the toolbar area, with the large plain white work area only partially shown. Each task ended when the participant completed it or chose to proceed to the next one, with subsequent tasks automatically displayed. The experiment was concluded after all four tasks were completed.

Upon completion of the experiment, each participant rated the four tested functions on a scale from 1 to 10. Once the ratings were provided, the usability test concluded.

4.5. The User Satisfaction Model Based on Objective Data

Upon completion of the experiment, we found that the improper sitting position of two participants during the experiment resulted in the camera not capturing their frontal expressions; therefore, the data for these two subjects were considered invalid, leaving 42 valid datasets. Group A comprised subjects 1–21, and Group B included subjects 22–42. The objective data from the 21 users in Group A were coded as matrix $I_A$, and the objective data from the 21 users in Group B were coded as matrix $I_B$. Matrix $I_A$ served as the input for training the BP neural network model, as outlined in Equation (6), while Matrix $I_B$ was used for model validation, as detailed in Equation (7). The objective data for the m-th subject were denoted as Im, as shown in Equation (8). The variables i₁ to i₇ represented the intensity of the seven emotions of joy, sadness, anger, surprise, fear, disgust, and neutrality, respectively. The variable i₈ represented the task completion time, i₉ denoted the task completion status (1 for completed, 0 for uncompleted), and i₁₀ reflected the disorientation degree. The user satisfaction ratings, serving as the output values, were encoded as matrix Y_A for Group A subjects, as indicated in Equation (9), and as matrix Y_B for Group B subjects, as specified in Equation (10), with Ym representing the satisfaction rating of the m-th subject.

$$I_A={\left[I_1,I_2,I_3,I_4,I_5,I_6,I_7,I_8,I_9,I_{10},I_{11},I_{12},I_{13},I_{14},I_{15},I_{16},I_{17},I_{18},I_{19},I_{20},I_{21}\right]}^T$$

(6)

$$I_B={\left[I_{22},I_{23},I_{24},I_{25},I_{26},I_{27},I_{28},I_{29},I_{30},I_{31},I_{32},I_{33},I_{34},I_{35},I_{36},I_{37},I_{38},I_{39},I_{40},I_{41},I_{42}\right]}^T$$

(7)

$$\begin{array}{c}{I}_m=\left[i_1,i_2,i_3,i_4,i_5,i_6,i_7,i_8,i_9,i_{10}\right]\end{array}$$

(8)

$$\begin{array}{c}{Y}_A={\left[Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_{10},Y_{11},Y_{12},Y_{13},Y_{14},Y_{15},Y_{16},Y_{17},Y_{18},Y_{19},Y_{20},Y_{21}\right]}^T\end{array}$$

(9)

$$Y_B={\left[{Y_{22},Y_{23},Y_{24},Y}_{25},Y_{26},Y_{27},Y_{28},Y_{29},Y_{30},Y_{31},Y_{32},Y_{33},Y_{34},Y_{35},Y_{36},Y_{37},Y_{38},Y_{39},Y_{40},Y_{41},Y_{42}\right]}^T$$

(10)

The user satisfaction model was established using the PSO-BP neural network. In this study, the input parameters were represented by matrix I_A. Following the BP neural network algorithm, there are 10 input nodes, 1 output node, and 3 hidden layer nodes. The maximum number of iterations was set to M = 50. And based on the characteristics of the data in this study, MAPE was selected as the loss function. According to the PSO algorithm, the initial position and velocity of particles were randomly generated within the allowable range, with the population size selected as three times the dimension (N = 30). Additionally, the parameters w_max = 1, w_min = 0.02, c_1min = 0.05, c_1max = 1.2, c_2min = 0.05, and c_2max = 1.2 were set. The optimized weight threshold of the PSO algorithm was assigned to the BP neural network, enabling the training of the BP neural network to achieve a prediction model with high accuracy.

5. Research Results

5.1. Notability Data Results

Table 2. Notability data results.

Notability (n = 21)	Variables	Mean	Standard Deviation	Variance
Pen	N	9	1.95	3.81
	S	14	7.1	50.38
	R	7	0	0
	Lostness	0.35	0.21	0.04
	Time	59.14	40.13	1610.79
	Complete	0.9	0.29	0.09
	Score	6.81	2.32	5.39
	Neutral	0.43	0.19	0.03
	Pleasant	0.12	0.23	0.05
	Sad	0.17	0.23	0.05
	Angry	0.1	0.12	0.01
	Surprise	0.06	0.1	0.01
	Fear	0.01	0.01	0
	Disgust	0.07	0.1	0.01
Picture	N	6.86	0.99	0.98
	S	7.81	2.44	5.96
	R	6	0	0
	Lostness	0.16	0.14	0.02
	Time	35.81	28.37	804.92
	Complete	1	0	0
	Score	8.29	1.42	2.01
	Neutral	0.48	0.22	0.05
	Pleasant	0.07	0.11	0.01
	Sad	0.22	0.25	0.06
	Angry	0.13	0.15	0.02
	Surprise	0.02	0.02	0
	Fear	0.03	0.05	0
	Disgust	0.05	0.06	0
Eraser	N	5.86	1.17	1.36
	S	7	2.53	6.38
	R	5	0	0
	Lostness	0.2	0.18	0.03
	Time	31.81	16.97	288.06
	Complete	1	0	0
	Score	8.43	1.79	3.2
	Neutral	0.49	0.21	0.04
	Pleasant	0.06	0.1	0.01
	Sad	0.16	0.23	0.05
	Angry	0.08	0.08	0.01
	Surprise	0.03	0.05	0
	Fear	0.02	0.03	0
	Disgust	0.08	0.12	0.01
Lasso	N	10.43	3.06	9.39
	S	24.57	9	81.01
	R	6	0	0
	Lostness	0.67	0.17	0.03
	Time	157.14	63.1	3981.36
	Complete	0.48	0.5	0.25
	Score	4.33	1.86	3.46
	Neutral	0.44	0.19	0.04
	Pleasant	0.14	0.22	0.05
	Sad	0.12	0.18	0.03
	Angry	0.08	0.08	0.01
	Surprise	0.02	0.04	0
	Fear	0.02	0.02	0
	Disgust	0.06	0.08	0.01

presents the task completion times, completion statuses, manipulation steps, Lostness, facial expression intensities, and subjective ratings for Notability’s four tasks.

In general, the higher the value of Lostness, the longer the takes, and the lower the pleasure and surprise level of the expression, the lower the subjective satisfaction score of the task, and conversely, the higher the subjective rating. This dataset will be used to construct the model for this experiment.

5.2. Goodnotes Data Results

Table 3. Goodnotes data results.

Goodnote (n = 21)	Variables	Mean	Standard Deviation	Variance
Pen	N	11.62	3.92	15.38
	S	19	9.54	90.95
	R	7	0	0
	Lostness	0.48	0.23	0.06
	Time	108.48	76.89	5912.82
	Complete	0.57	0.49	0.24
	Score	5.57	2.74	7.48
	Neutral	0.4	0.19	0.04
	Pleasant	0.11	0.18	0.03
	Sad	0.2	0.25	0.06
	Angry	0.15	0.16	0.03
	Surprise	0.05	0.12	0.01
	Fear	0.03	0.03	0
	Disgust	0.03	0.05	0
Picture	N	6.9	0.87	0.75
	S	8.05	2.19	4.81
	R	6	0	0
	Lostness	0.19	0.14	0.02
	Time	33.05	14.4	207.47
	Complete	1	0	0
	Score	7.9	1.6	2.56
	Neutral	0.52	0.21	0.04
	Pleasant	0.09	0.14	0.02
	Sad	0.16	0.23	0.05
	Angry	0.09	0.07	0.01
	Surprise	0.03	0.04	0
	Fear	0.04	0.09	0.01
	Disgust	0.06	0.07	0
Eraser	N	6.57	1.09	1.2
	S	9.76	3.64	13.23
	R	5	0	0
	Lostness	0.36	0.18	0.03
	Time	43.67	29.37	862.7
	Complete	0.9	0.29	0.09
	Score	7.33	1.94	3.75
	Neutral	0.46	0.18	0.03
	Pleasant	0.07	0.11	0.01
	Sad	0.14	0.26	0.07
	Angry	0.13	0.12	0.01
	Surprise	0.03	0.03	0
	Fear	0.02	0.03	0
	Disgust	0.06	0.07	0
Lasso	N	11.43	2.08	4.34
	S	24.1	8.57	73.51
	R	9	0	0
	Lostness	0.52	0.18	0.03
	Time	165.57	86.45	7473.96
	Complete	0.48	0.5	0.25
	Score	5.95	2.26	5.09
	Neutral	0.44	0.16	0.03
	Pleasant	0.07	0.13	0.02
	Sad	0.17	0.24	0.06
	Angry	0.1	0.1	0.01
	Surprise	0.04	0.05	0
	Fear	0.02	0.04	0
	Disgust	0.06	0.07	0.01

presents the task completion times, completion statuses, manipulation steps, Lostness, facial expression intensities, and subjective ratings for Goodnotes’s four tasks.

This set of data is used in two ways. All objective data are used as input items to the model, and subjective score data are used primarily as a comparison of the model’s predicted values of subjective user satisfaction.

5.3. PSO-BP Model Prediction Results

The objective data matrix of Goodnotes was input into the trained model to predict user satisfaction. The predicted results for the four tasks were then compared with the actual user outcomes, as illustrated in Figure 5. It can be found that the final predictions of the model are closer to the true subjective scores.

6. Result Analysis

6.1. Model Verification

To quantify the model’s accuracy, the mean absolute percentage error (MAPE) was calculated. This metric provides a tangible representation of the average deviation. The MAPE formula, presented in Equation (11), uses Y_i to represent the actual satisfaction of the ith subject, ${\overline{y}}_i$ for the predicted satisfaction, and n as the total number of subjects. Upon calculation, the predicted errors of the model for the four enhanced tasks are presented in

Table 4. Prediction error of customer satisfaction.

	Task 1	Task 2	Task 3	Task 4	Average
MAPE	19.81%	12.55%	9.13%	13.47%	13.74%

as follows: 19.81%, 12.55%, 9.13%, and 13.47%, with an average error of 13.74%. Consequently, the average accuracy rate is 86.26%, indicating that the PSO-BP neural network model could predict user satisfaction with high accuracy to a certain extent, affirming the effectiveness of the model.

$$MAPE=\frac{\sum \left|\frac{y_i-{\overline{y}}_i}{y}\right|}{n}\times 100\%$$

(11)

In order to verify the superiority of the user satisfaction model of the PSO-BP neural network, we compared the prediction results of the PSO-BP model with those of the BP neural network. The parameters of the BP neural network prediction model were consistent with those of the PSO-BP neural network. The comparison results show that the average MAPE of the PSO-BP neural network is 13.74%, while the average MAPE of the BP neural network prediction model is 31.79%. The MAPE of the PSO-BP neural network is significantly better than that of the BP neural network, which indicates that the PSO-optimized BP neural network overcame the problems of slow convergence speed and the susceptibility to local minima. Therefore, the prediction performance is greatly improved and more reliable, and it becomes an effective tool for predicting user satisfaction.

6.2. Error Analysis

By establishing the PSO-BP neural network model, user satisfaction in the Goodnotes software can be predicted to a certain extent. However, a degree of discrepancy with actual user satisfaction still exists. The primary reasons for this discrepancy are as follows: (1) Users exhibit individual differences in subjective evaluation, leading to a less than perfect correlation between objective data and subjective scores. Thus, utilizing objective data to predict subjective scores inherently introduces some level of error. (2) Although the facial expression recognition accuracy of FaceReader software exceeds 86%, errors may still occur, leading to inaccuracies in the input objective data. (3) The quantity of training data is relatively small compared to the actual user population. Increasing the volume of training data can enhance prediction accuracy to some extent. (4) This study does not consider the effect of adversarial attacks on facial expression recognition, which misleads the model through maliciously designed input data and may seriously affect its prediction accuracy and reliability. Studies have shown that the use of adversarial training and defense strategies can effectively improve the security and robustness of the model [34]. In the future, we will deeply explore how to apply these adversarial defense methods in facial expression recognition to enhance the credibility and applicability of the model [35].

7. Conclusions and Future Works

The assessment and measurement of interface satisfaction have long served as crucial metrics for software optimization and upgrades. Presently, satisfaction measurement methods predominantly rely on subjective evaluations, sometimes supplemented with objective methods. However, subjective approaches often lack objectivity, while objective methods can face efficiency issues. In this study, we collected objective data, including facial expression intensity, task completion time, task completion status, and user operation processes, through facial expression recognition technology. Furthermore, we optimized the BP neural network model using the particle swarm optimization (PSO) algorithm. Subsequently, we constructed a user satisfaction prediction model based on the PSO-BP neural network, which we validated using two established note-taking software platforms. Our results indicate that the model exhibits a high level of accuracy and reliability, with an average absolute percentage error of 13.74% in predicting satisfaction across four typical functional tasks in Goodnotes, and the MAPE of the BP neural network model is 31.79%. Therefore, the user satisfaction model based on the PSO-BP neural network demonstrates relatively high accuracy and good stability in this field, making it an effective tool for predicting user satisfaction.

In summary, the user interface satisfaction measurement method proposed here integrates facial recognition technology and the PSO-BP neural network, offering both objectivity and high accuracy. Moreover, it surpasses other objective measurement methods, such as EEG and eye movement tracking, in terms of efficiency. This method can be readily applied to general software satisfaction assessments, providing valuable data and guidance for software interface design, optimization, and upgrades. By expanding the scope of software interface user satisfaction evaluation methods, our study contributes to the advancement of usability research in this domain.

There are still shortcomings in this research. This study primarily uses note-taking software as a case study, and whether the conclusions can be applied to other types of software remains to be further analyzed and discussed. Future research will consider including a wider variety of software types as experimental materials to conduct a more extensive analysis of user satisfaction. Secondly, the participants are mainly from first- and second-tier cities with higher education levels, limiting the applicability of the results to other demographic groups. Future studies should include a more diverse sample, considering factors such as gender, occupation, culture, and education, to better understand their impact on user satisfaction. Finally, this study did not explore the possible antagonistic attacks on facial expressions, and subsequent studies will explore these potential factors in depth to enhance the credibility and applicability of the model.

Moving forward, our research will continue to delve into user satisfaction and usability, exploring their influencing factors. We aim to incorporate deep learning algorithms to further enhance the model’s predictive capabilities, ultimately establishing a satisfaction prediction model with broader applicability and higher accuracy.