Case Study of the Experience Capturer Evaluation Tool in the Design Process of an Industrial HMI

Abstract

In the absence of user experience evaluation tools for industrial human–machine interfaces (HMI), a specific tool called eXperience Capturer (XC) has been created. It is a multi-method user-centred tool that evaluates the pragmatic and experiential aspects of employees’ interaction with industrial HMIs during the three phases of experience. In this article, a case study is shown where the XC tool is used in an industrial HMI design process. The results show that evaluation using the XC tool facilitates the creation of a new design that improves the experience of employees during interaction, increasing their autonomy, competence, closeness to the system, safety and stimulation.

1. Introduction

In recent years, the field of human–computer interaction (HCI) has entered a new era, where the main focus is on designing the user experience (UX) [1]. This approach on experience is based on personal and subjective aspects of the interaction, defined as a dynamic story capable of creating emotions [2]. Previously, the study of interaction was based on designing effective and efficient systems, responding to functional needs. However, today, in addition to the pragmatic aspects, digital interface design and development tries to address the emotional and hedonic needs of users.

Reflecting on this new era, today, most of the consumer market products and services follow the UX focus. However, the purely industrial context, and especially that of the machine-tool, has not yet applied these emotional concepts.

Currently, human–machine interfaces (HMI) allow more features than simple machine control to be added. They offer the opportunity to visualize progress during production processes, show energy consumption, give instructions for manual activities, manage productive tasks and give support through an integral management of the changes that could happen throughout the production process. In this new industrial revolution, despite automation, people’s role will continue to have an important presence and they will begin to adopt more strategic decision-making and problem-solving roles. For this, new HMIs must be adapted to users’ needs and be able to obtain the highest performance both from operators and machines [3].

Along these lines, design centred on the UX is presented as a key opportunity. This UX approach facilitates the creation of new products and services that meet users’ needs and encourage the creation of positive experiences. This approach, in addition to increasing task execution efficiency, will generate positive emotions in operators, influencing learning processes, decision making, knowledge and intelligence [4]. Moreover, positive emotions can increase workers’ motivation and involvement [5].

To date, the aforementioned concepts in the machine-tool environment have hardly been worked on, presumably, due to the lack of tools adapted to this context [6]. In order to implement UX design knowledge in the industry, it is necessary to create tools and methods that help companies in new product development. In this context, the evaluation methods to be used throughout the design process assume a very important role. The evaluation allows identifying critical factors in the interaction experience and facilitates the proposal of new solutions.

In light of the present needs, the eXperience Capturer (XC) tool is currently available [7]. This is a new multi-method tool capable of evaluating the interaction experience with an industrial HMI, taking into account both pragmatic and hedonic aspects, combining user and expert-based evaluations during the three evaluation phases (before, during and after the experience).

In this communication, a case study is shown where the XC tool is used in the process of redesigning an industrial HMI for parts manufacture and control in grinding machines.

2. User Experience (UX) Evaluation Methods

UX is considered the consequence of the user’s internal state (predispositions, expectations, needs, emotions, motivations, etc.), the characteristics of the designed system (functionality, complexity, usability, etc.) and the context (or environment) where the interaction takes place (e.g., organizational/social framework, the activity importance, voluntary use, etc.) [8]. As reflected in the definition, UX goes beyond the concepts of system functionality and usability. However, there is a difference in perception regarding the UX between the scientific and academic field and the industrial sector. In the first one, research focuses on more experiential factors, such as hedonic or emotional aspects. In industry, however, efforts focus on the most practical and functional aspects of the experience, such as usability, novelty or the life cycle [6].

In the scientific field, there is a wide variety of UX evaluation methods, largely due to the variety of concepts and factors related to UX. In recent years, several authors [9,10,11,12,13] have tried to review the methods that currently exist for UX evaluation in digital interfaces. Through these reviews, they try to identify the experience dimensions, the period of time or development phase on which the tools are focused, in order to identify needs for future research.

Among the methods included in these reviews (96 methods in total), it can be seen that the majority is directed to web applications, with only one method focused on the industrial context, the XGoals method [14]. This method assesses the extent to which the system meets the psychological needs of users, collecting information through user questionnaires. Thanks to these answers, users’ assessment of their experience with the industrial system can be collected. However, this approach has a number of limitations. On the one hand, it only allows gathering users sensation about the motivations fulfilled in the execution of a series of tasks, but it does not allow assessing the pragmatic aspects of the system, which is essential in this type of environment. On the other hand, the result of the XGoals questionnaire only includes the user assessment but does not show the elements or aspects of the HMI that have influenced that assessment, which makes it difficult to propose solutions to improve user experiences.

Regarding the source of information of the methods analysed in the review, it can be seen that the majority focuses on user evaluations, with only 6 methods that combine both the user and the expert perspective (evaluation based on the assessor’s judgment). Likewise, more than half of the methods (56%) do not consider usability and emotional factors. Regarding the evaluation phase, surprisingly, only 4 methods analyse the interaction in the three evaluation phases, that is, before, during and after the experience. Finally, most of the tools use quantitative methods for evaluation. Only a small number of them used quantitative combined with qualitative methods (9 methods) and only 2 methods mix quantitative, qualitative and biometric methods.

Therefore, after the literature review, it can be concluded that there is a great need for methods that evaluate user experience in interaction with industrial HMIs, in order to propose new solutions that, on the one hand, increase efficiency and effectiveness in tasks execution and on the other hand, generate positive emotions in operators, thus influencing their learning, motivation and intelligence processes.

3. Case Study

The objective of this research is then, to validate the eXperience Capturer tool as a method that facilitates the redesign of an industrial HMI, in order to improve the user interaction experience. For this, the case study described below has been conducted.

3.1. Methodology

The study was carried out using the user-centered agile design (UCAD) methodology [15]. UCAD was born from the union of user-centered design (UCD) and agile software development (ASD) methodologies. The purpose of the methodology is to offer a working procedure to create an optimal software in terms of its function, but based on user’s needs, in order to obtain a friendly and intuitive product. It proposes a structured process, showing the three main phases of analysis, design and development. A series of tools are proposed for each phase, as well as their mode of use when applied to the digital systems of industrial working environments 4.0.

In this case study, for the analysis phase, the XC tool has been used, in order to identify the UX critical factors with the current version of the HMI to be redesigned. Subsequently, once a redesign prototype has been designed, it has been reassessed with the XC tool (Figure 1).

For the application of the XC tool in the analysis phase, an experiment was carried out with 16 participants, of which 9 are men and 7 women ($\stackrel{-}{\mathrm{X}}$ = 28.8; s = 10.02). After the design phase, the new version of the application has been evaluated, with 5 other participants (3 men and 2 women, $\stackrel{-}{\mathrm{X}}$ = 22). In both experiments, users have previous experience in industrial interfaces, but have no knowledge of the assessed application. 50% of the participants in the first experimentation are conducting studies regarding industrial machinery and have sometimes used an industrial interface. The other 50% work in an industrial environment and are already using this type of application. Participants in the second study were students in the subject of the industrial sector work environment with some previous experience with this type of interface.

3.2. The eXperience Capturer Tool

As the shortcomings identified in the study of the state of the art show, there is a need to create tools that evaluate the UX in the digital environments of industry.

The eXperience Capturer (XC) tool covers the above mentioned need. It is a multi-method tool that evaluates emotional and usability aspects in the three phases of the experience, combining quantitative and monitoring methods. The method is based on the user test, where a series of specific tasks must be executed in the digital interface and then this interaction experience is analysed [7]. The tool consists of three main phases: pre-interaction phase, interaction phase and post-interaction phase (Figure 2).

For the tool application in the current case study, in the first phase of each evaluation, information was collected about the user (age, professional activity, previous experience, etc.) and the interaction context (information regarding place, social environment, etc.). In addition, the psychological motivations or needs to be met during the experience have been defined and a heuristic evaluation based on the UXER tool has been performed [16]. The heuristic evaluation is based on expert evaluation, where through the assessors’ judgment a series of heuristics or guidelines are used to identify different usability problems of the digital interfaces.

In the second phase, user activity was monitored while executing a series of specific tasks, in order to detect critical usability factors and emotional responses. The Tobii Pro X2-60 tool from Eye-tracking was used for monitoring. This device collects data from the point where the user is looking, showing where he/she expects to find each element of the interface, and consequently, where he would like to find it. In addition, during tasks execution, the emotional reactions of the participants were qualitatively analysed.

Finally, in the third phase, the users’ assessment of the experiential aspects were collected, measuring their emotional responses. For this, the PANAS-X questionnaire was used [17]. This questionnaire assesses using positive and negative adjectives. This information is then classified under the terms: general positive emotion, general negative emotion, fearfulness, hostility, guilt, sadness, joviality, self-confidence, degree of attention, shyness, tiredness, serenity, surprise, basic positive affect and basic negative affect. The importance of evaluating both positive and negative emotions should be stressed, because an experience with a lack of positive emotions is not necessarily critical, but an experience with negative emotions can be very damaging for operators.

3.3. Analysed Human–Machine Interfaces (HMIs)

The first HMI analysed in this case study is DoGrind [18] by the company Danobatgroup S. Coop (Figure 3). It is an application for grinding machines, designed to program and manufacture mechanical parts, through which employees can control the entire process. Through this application, operators can create and edit tools, add operations and send parts to be manufactured.

The second HMI analysed is the prototype of the DoGrind application redesign, which has been called Do2Grind (Figure 4). This application maintains the same functionalities, but the navigation architecture and aesthetics have been completely renewed. This redesign has been conducted based on the DoGrind evaluation, which has allowed, for instance, the introduction of a friendlier and closer language, step-by-step instructions for certain operations and continuous feedback messages to user actions. In addition, other changes identified in the evaluation have been integrated thanks to eye-tracking monitoring. For instance, the new version allows the users to see the parts within each group by simply clicking on the group’s folder.

4. Results

This section shows the study results in two main parts. The first part includes the evaluation of the DoGrind application. In the second part, the Do2Grind prototype is evaluated and the results obtained are presented.

4.1. Evaluation of the DoGrind Application

A total of 16 individuals took part in the first evaluation. They had no previous experience in the analysed application but had previously used other applications in the industrial context.

In the first phase of the evaluation, the results of the heuristic evaluation show that the system complies with 75 of the 116 heuristics, so more than 30 aspects to be improved, regarding the system usability, have been detected.

In the second phase, during the monitoring of task execution, 12 critical factors have been detected that have generated usability problems and negative emotional reactions in users. One of the identified errors is described and visualised below, which were repeated by all participants (Figure 5).

The user has to create a new part within an already created group. To do so, the user tries to click in the group expecting a reaction in the screen that allows them to create a new part. However, the screen remains the same. The user spends one minute without being able to progress until the research team indicates the steps to proceed. To create a new part, the user has to click the required group and then the button at the bottom right (indicated with the red square) to change the right column options from ‘new group’ to ‘new part’. As the heatmap shows, the employees do not expect the procedure, as all the attention is focused on the required group and the possible reaction on clicking it, but the button at the bottom right has not been looked at at any time. Thanks to this visualization, it is possible to know that, on the one hand, the actual procedure is not working and, on the other hand, which could be the procedure that the employee expects, i.e., the possible solution for the issue.

Finally, using the PANAS-X questionnaire (Figure 6), the users’ assessment of emotions during the experience has been collected. The results show intermediate values, which indicates that the experience has not been very positive. Despite this, in order to draw definitive conclusions, the results of the DoGrind application and the subsequent Do2Grind prototype will be compared.

4.2. Evaluation of the Do2Grind Prototype

The Do2Grind prototype has been evaluated with 5 new users, who as in the previous case, had no previous experience with the application, but had used other applications in the industrial context. Thanks to the evaluation by the 5 users, despite not being a significant number of participants in statistical terms, several aspects to be improved have been identified. After all, as described by Nielsen [19], testing with 5 users allows to identify most of the usability problems in a fast and cost-effective way.

First, through the heuristic evaluation, it has been concluded that the Do2Grind complies with 99 of 116 heuristics, which shows that the new version has improved its usability by 20.7% (

Table 1. Comparison of the heuristic evaluation between DoGrind and Do2Grind.

Evaluated Heuristics	DoGrind	Do2Grind	Increase (%)
General aspects (9)	3	7	44.4
Trust and credibility (6)	2	5	50
Navigation and information architecture (14)	7	13	42.9
Control and feedback (13)	7	10	23.1
Layout and graphic design (17)	12	17	29.4
Error tolerance (5)	1	5	80
Privacy (2)	2	2	0
Homepage (5)	1	4	60
Content and writing (7)	6	6	0
Tasks execution (20)	12	16	20
Forms and data entries (18)	12	14	11.1
Final result (116)	75	99	20.7

).

Second, in the execution of tasks, a total of 3 critical factors have been identified which have increased the execution time and have negatively influenced the participants’ emotions.

Third, users’ emotions have been identified during tasks execution. With the aim of analyzing the improvement that Do2Grind represents with respect to the DoGrind, the results of both applications have been compared (Figure 7). As the graph shows, the results of the participants’ emotional reactions with the new version of the application have been considerably better. The value of the general positive emotion has been higher and in the same way, the general negative emotion of the new application has been much lower. In negative emotions such as fearfulness, hostility, guilt, sadness, shyness, tiredness, surprise and negative affect, the values of the experience with the DoGrind have been higher. Along the same lines, positive emotions such as joviality, self-confidence, serenity and positive affect have been higher in the interaction with the Do2Grind.

5. Conclusions

This article presents a case study in which the new eXperience Capturer (XC) tool is used in the design and development process of the industrial HMI called DoGrind. The XC tool covers the considerable need for new methods that facilitate the incorporation of UX knowledge in industrial environments. In this new industrial era, optimization in communication and interaction between user and machine will be key to take advantage of the opportunities offered by new technologies.

As it has been validated through the case study, the XC tool has facilitated the proposal of a new version of the DoGrind application that improves the user experience of the operators during the interaction. In the first place, the heuristic evaluation has allowed the identification of 30 aspects to improve that have been introduced in the Do2Grind, to improve the usability of the system by 20.7%. Likewise, the 12 critical factors identified during the execution of user tasks in the previous version have made it possible to restructure the application architecture and navigation mode. In the Do2Grind, it has been possible to verify that critical factors have been significantly reduced, with only 3 errors detected. Finally, negative values identified in the evaluation of emotional reactions highlighted that this new design reduces feelings of fearfulness, hostility or guilt. Thanks to this evaluation, a more friendly and closer language has been introduced, instructions have been included on the steps to be taken in certain operations and continuous feedback messages have been offered to user actions.

Therefore, the introduction of the tool in the design and development of industrial HMIs will enable new experiences to be created regarding the interaction between users and machines in which, in addition to increasing their productivity and performance, they will increase autonomy, competence, proximity, safety and operator stimulation.

Finally, as a future research line, it is proposed to integrate the tool into the design and development processes of HMIs of industrial companies.