First Application of Robot Teaching in an Existing Industry 4.0 Environment: Does It Really Work?

Abstract

This article reports three case studies on the usability and acceptance of an industrial robotic prototype in the context of human-robot cooperation. The three case studies were conducted in the framework of a two-year project named AssistMe, which aims at developing different means of interaction for programming and using collaborative robots in a user-centered manner. Together with two industrial partners and a technological partner, two different application scenarios were implemented and studied with an off-the-shelf robotic system. The operators worked with the robotic prototype in laboratory conditions (two days), in a factory context (one day) and in an automotive assembly line (three weeks). In the article, the project and procedures are described in detail, including the quantitative and qualitative methodology. Our results show that close human-robot cooperation in the industrial context needs adaptive pacing mechanisms in order to avoid a change of working routines for the operators and that an off-the-shelf robotic system is still limited in terms of usability and acceptance. The touch panel, which is needed for controlling the robot, had a negative impact on the overall user experience. It creates a further intermediate layer between the user, the robot and the work piece and potentially leads to a decrease in productivity. Finally, the fear of the worker of being replaced by an improved robotic system was regularly expressed and adds an additional anthropocentric dimension to the discussion of human-robot cooperation, smart factories and the upcoming Industry 4.0.

1. Introduction

The last few decades have demonstrated how many sectors of production can benefit from fast, vast and inexpensive industrial robots. However, direct human-robot cooperation is still considered as not safe enough for the human. Furthermore, the wide deployment of industrial robots went along with the replacement of human labor. In order to develop competitive alternatives, the human and the robot should work as a team to achieve more flexible human-robot interaction [1]. Such an approach has to take the User Experience (UX) within these robotic assembly lines into account. Usability aspects play an important role in terms of the feasibility of a robotic system for different application areas, as well as the user acceptance of the system when integrated into daily working routines. Following the definition of Alben [2], UX is a broad concept including cooperation and usability, as well as factors, such as perceived safety, stress and emotion [3]. Especially in the industrial context, UX factors have to be understood as dynamic over time, which makes them a crucial element in the production process [4].

The idea that an industrial robot can take the role of a cooperative and supportive tool for the workers is part of a relatively new paradigm called “Industry 4.0”. This paradigm describes the upcoming fourth industrial revolution and envisions “smart factories” in which humans and robots will work more and more closely together. The first industrial revolution was the mechanization with water and steam power; it was followed by the second industrial revolution: mass production using assembly lines and electrical energy. After that, the digital revolution began, and the use of electronics and IT in the production processes became common. The term “Industry 4.0” is collectively used for technologies and concepts of the value chain organization characterized by strong adjustment of the products under conditions of highly flexible (high volume) production (mass customization) [5]. Its automation technology features self-diagnosis, cognition, self-tuning and self-configuring methods based on the technological concepts of cyber-physical systems, the “Internet of Things” and the “Internet of Services”; it facilitates the vision of the “smart factory” [6,7]. These smart factories of the Industry 4.0 are modular structures and provide cyber-physical systems, which monitor physical processes, create a virtual copy of the physical world and make decentralized decisions in order to become smarter and to better help people in their increasingly complex working routines.

Up to now, robot-based assistive systems are still not widely spread in the manufacturing industry, due to missing research on their full potential and a lack of usability and user acceptance aspects. In this article, we present research performed within the project AssistMe; a two-year project aiming to develop and evaluate novel means of interaction for cooperative robotics from a user-centered perspective. The project consists of three major development cycles. In the first stage, a cooperative robotic system is used “out-of-the-box” and implemented for two different use cases: (1) assembling automotive combustion engines; and (2) polishing continuous casting molds. These implementations were evaluated within three case studies in terms of usability and user acceptance. The goal of these case studies is to identify the major usability and user acceptance problems of off-the-shelf robotic systems within the smart factory context in order to derive suggestions for improvements (i.e., reducing programming complexity, programming duration, system reliability and process quality). These improvements will be implemented and evaluated in the second and third expansion stage later in the project.

The research presented in this article focuses on the first expansion stage and its user-centered approach. In the three case studies, workers cooperated with the robotic prototype in different application contexts, including laboratory conditions (two days), factory conditions (one day) and assembly line conditions (three weeks). By using video analysis, questionnaires and established interview guidelines [8], we ensured a quantitative and phenomenologically-oriented empirical perspective on the usability and user acceptance of an off-the-shelf robotic system implemented for two different application scenarios. Before we go into detail with the three case studies, we will give an overview of relevant related work with respect to assistive robot systems in the industrial context and user-centered development processes in human-robot cooperation.

2. Related Work

Industrial robots are used in the production lines of factories for manufacturing and fast product assembly. The industry is highly dependent on its equipment and, therefore, must be able to adapt to changes to ensure fast and cost-effective production. The following state of the art gives an overview of Human-Robot Interaction (HRI) in today’s industrial environment and the user-centered research and development approaches in the factory context.

2.1. Human-Robot Interaction in the Industrial Environment

Significant core operations in factories of fast product assembly, in addition to various auxiliary functions (e.g., cleaning), are joining, handling, testing and adjusting. The value-adding operations of the primary assembly (joining) can be differentiated from non-value-adding activities of the secondary assembly (handling, testing, adjusting). The work in [9] defines a cognitive assistance system for a visual-instructive assembly guidance based on an environment-dependent statement generation. The work in [10] provides an assistance system for the automation of the secondary assembly process “testing” with a special focus on completeness and correctness. The work in [11] is a major project for the implementation of a software framework for assembling assistive robots with a similar function as in [12,13,14,15] or as addressed in numerous other projects.

However, from the perspective of executing the named primary and secondary operations, manipulation systems play a crucial role in assembling assistance. These systems can be differentiated with respect to their degree of automation [16,17]. Cranes and balancers allow gravity compensation. The work in [18] defines cobots as robots designed for direct physical interaction with the workers while their design is subject to research until today [19]. Cobots are next to the users and can overcome inertia and friction forces, but do not allow pre-programmed (autonomous) movements. Assistive robots work, as opposed to industrial robots, in the same work area as the operator. Therefore, they have safely limited speeds and forces, automatic restart and allow guiding the robot by hand. Robot systems that can be operated without spatial safety areas have gained acceptance in the market since 2010 [20,21]. The work in [22] presents a historical overview of autonomous robot-based manipulation systems and shows their intense research for the last 30 years. The sparse industrial use of autonomous manipulation systems is based on the conservative attitude of the industry (especially with regard to safety and standardization issues), the high system costs, the lack of flexibility, as well as on the low feasibility of basic research results (technology transfer gap, especially in Europe [23]).

Robotic systems used in the industrial context can be programmed via various interfaces and interaction levels [24]. The work in [25] divides programming procedures into automatic (learning, demonstration and instructional-based) and manual (text-based or graphical) methods. “Learning from Demonstration” (LfD) is especially considered to provide an easy and intuitive way to program robot behaviors, potentially reducing development time and costs tremendously. The work in [26] defined criteria to assess the applicability of state of the art LfD frameworks in industry. Furthermore, there is a distinction between online processes (runtime of the robot system, e.g., programming by demonstration [27], manual teach-in) and offline methods (CAD based, etc.). Another categorization can be made on the level of interaction and the interaction medium. The work in [28] gives an overview of physical Human-Robot Interaction (pHRI). The work in [29] outlines the relevant pHRI topics, such as security and standardization [30,31], mechanical and control engineering design, system stability and benchmarks. Current research, for example, is the learning of motor skills by pHRI [32] and the industry-oriented application [33]. Industrial systems allow only the teach-in of discrete individual positions by pHRI and cannot directly imitate smooth movements.

In modern robotics, the prominent input devices for programming are Tangible User Interfaces (TUI). These user interfaces allow the person to interact with the digital representation of the robot through the physical environment. In this setting, the robot has an equivalent in the digital world, and the TUI represents the mechanisms for its interactive control. The spectrum of TUIs ranges from haptic interfaces (often with a force feedback function) in telepresence systems [34] to robotic systems whose surface is equipped with tactile sensors in order to allow the manual control of individual robot segments [35]. The first industrial input devices, which are not available for all robot manufacturers and types, take the orientation of the input device into account in order to derive the orientation and the direction of the tool (but not the position) [36].

To summarize, the programming and usage of assistive robotic systems for collaboration is equivalent to standard industrial robots, when trained robot programmers are the target group. However, programming can be simplified using macros or the possibility of hand guidance during system teach-in for unexperienced users. These aspects are evaluated in the AssistMe project, with an off-the-shelf robotic system. Industrial installations of off-the-shelf collaborative robots can be considered to remain inflexible and to be unintelligent playback machines for movements and process technology, such as intelligent cameras, etc. It remains complicated and almost impossible with commercially available systems to integrate that which renders adaptive behavior. The AssistMe projects wants to enable naive operators to manually teach a robotic arm for their purposes with little pre-knowledge requested. Afterwards, a safe and user-friendly cooperation with the robot in the production process should be possible. The first series of case studies presented in this article therefore aims at identifying the most severe usability and user acceptance aspects that need to be resolved in order to valuably integrate an assistive robotic system into the assembly line.

2.2. User-Centered Development and Evaluation

In the last 20 years, the importance of usability and acceptance has become more important in the field of industrial robotics. Companies, such as KUKA, are investing more and more into User-Centered Development (UCD); usability standards, such as ISO/TR16982:2002 were developed, and user-friendly industrial robots, such as Baxter [37], were brought to the market. Nevertheless, there is yet little user-oriented research concerning assistive robots in the industrial context (e.g., [3,38,39]), including the teach-in of industrial robots and the user-centered development in the industrial development [40]. In addition, the evaluation is another special challenge in and of itself: How can we assess and compare the usability and acceptance of a robotic system for two different application contexts?

It can basically be distinguished between five evaluation methods in human-robot interaction: (1) self-assessment questionnaires, e.g., the Godspeed questionnaire series as a prominent example for a semantic differential especially developed for HRI [41]; (2) interviews [42]; (3) behavior measurements [43]; (4) psychophysiological measurements [44]; and (5) task performance measures [45]. All of these methods have strengths and weaknesses. In order to minimize the weaknesses, a methodological triangulation is often used, and quantitative and qualitative methods are combined. Moreover, the industrial context also includes gender-relevant aspects, and as with most research in human-robot interaction, investigating and exploring these issues relies heavily on the used evaluation methodology and scenario [46].

In order to get a better understanding of the users, their working procedures and the contextual constraints, it is important to actually leave the laboratory and go into the field where the action takes place, in order to know the real context and how the product will be used. The context should be analyzed in the very beginning to know which factors affect the product development, so UCD activities can be better tailored to development phases [13]. To get knowledge of the context, several methods can be applied, e.g., participating in training sessions [23], by means of a contextual inquiry or an ethnographic approach [14].

Harsh environments, such as the cleanroom of a semiconductor factory, have been studied using contextual inquiry [9] to learn about the working routines of maintainers. In harsh environments, the researchers have to make compromises and work-arounds, such as the investigation of the context of a rock-crushing factory where the researchers were confronted with serious problems, like noise and dirt [22]. The actual integration of user-centered methods in the product life cycle can be achieved with different strategies. The process of increasing the understanding of usability should include all stakeholders, and usability goals need to be described in detail for all of them. Furthermore, the understanding of UCD and good design practices can be spread by presenting the results of the usability team on a regular basis. However, the user-centered development methods will only impact the product development life-cycle if the usability professionals speak the same language as the people in the companies. How to increase the awareness of usability matters with industrial partners is described by [40,47]. The latter confronted industrial partners with usability problems via watching a test situation with real users. This approach helped the developers to get a feeling for when the users had troubles interacting with their software.

To summarize, up to now, still, little empirical user-centered research in the industrial context has been performed in order to improve the usability and user acceptance aspects of assistive robotic systems. Studying robots in the assembly lines with actual end users is a challenging endeavor, not only methodology-wise, but also with respect to all of the different stakeholders involved. In the AssistMe project, we follow a user-centered development and evaluation approach, involving all stakeholders (industrial partners, as well as technological partners) in order to flexibly automatize selected production steps in an economically-viable way. Within two years, we plan to iteratively evaluate the same robotic assistance in different stages of expansion for two different use cases. Stage 1 is an off-the-shelf robotic arm from Universal Robotics. Stage 2 will be further enhanced with a 3D sensor and Stage 3 with force feedback. Every stage of expansion will be evaluated together with representative target users from our industrial partners with respect to usability, user experience and acceptance. After every evaluation, the implications for improvement for the next expansion stage will be derived to keep the operators’ point of view in the development process. The AssistMe project thereby follows a very similar user-centered design approach as presented in [40], and the evaluation activities are methodologically grounded in the USUS (Usability, Social Acceptance, User Experience, and Societal Impact) evaluation framework [48].

3. Motivation and Objectives

The motivation and aim of the AssistMe project is the development of innovative haptic and optic concepts for human-robot cooperation in two different application contexts, namely the assembly of automotive combustion engines (see Figure 1), while the other one treats the machining (polishing) of casting molds (see Figure 2). These concepts can be used during set-up (teach-in) and interaction (collaboration in the assembly line) with an assistive robotic system.

The assembly of a combustion engine includes the installation of a cylinder head cover. The installation is carried out manually by stacking the cover with pre-inserted screws onto the motor block and tightening the screws with a manual power tool. The electronic screwdriver of the manual workplace is fitted with a push-start mechanism, electronic control unit and a shut-off clutch and, therefore, starts rotating when pushed onto the screw and stops motion when retracted, respectively, when a predefined torque is reached. The working instruction of the workstation includes several additional process steps. An automatic screw tightening system is expected to provide assistance and to reduce the workload at the workstation for the human worker. A state of the art collaborative robot system [49] is equipped with the power tool (Figure 1) and programed to perform screw tightening operations in the required order and accuracy to meet a defined process quality (screw-in depth, torque, etc.).

The same robotic system was equipped with a polishing tool. Casting molds are crafted from flat material by wire electro discharge machining that leaves eroded surfaces without the required surface finish quality. Manual polishing (Figure 2) by air pressure-driven oscillating polishing machines is extraordinarily labor intensive, non-ergonomic and harmful to health. Prolonged exposure to hand transmitted vibration from powered processes or tools is associated with an increased occurrence of symptoms and signs of disorders in the vascular, neurological and osteoarticular systems of the upper limbs [50]. Setup and programming time are crucial for the use case, since continuous casting molds are usually one of a kind products, manufactured in a lot size of one, with polishing being by far the most labor-intensive production step, causing umpteen hours of labor per mold. Therefore, an assistive system, that is easy to program and setup, is desirable, which can reduce the amount of labor, especially for ergonomic and health reasons.

4. Studies with the Off-The-Shelf System

The three case studies presented in this article were all performed during the first stage of the AssistMe project with the off-the-shelf robotic system from Universal Robots in order to evaluate the industrial feasibility of the state of the art and to create a benchmark in terms of usability and user acceptance measures for comparison studies of the subsequent expansion stages. Our approach is strongly influenced by [40] and prioritized the participation of all partners in all phases of the evaluation (i.e., use case definition, methodology definition, actual evaluation and data interpretation and the deduction of novel interaction paradigms). This close cross-linkage helps us learn from our industrial partners how to conduct studies in their specific environments. Thus, we understood all of the constraints and possibilities coming along with our interdisciplinary cooperation. The whole cooperation with our industrial partners lasted about eight months, consisting of a kick-off meeting (two days), a preparation phase for each use case (one week each), the three case studies (see

Table 1. Overview of the three use cases.

	Use Case A	Use Case B	Use Case C
Description	Teaching of screw positions	Teaching of polishing positions	Cooperative screwing
Environment	Factory	Laboratory	Factory (assembly line)
Number of Participants	5	5	5
Duration	1 day	2 days	3 weeks
Research Instruments	Observation Questionnaires	Observation Questionnaires	Interviews

), the data analysis (eight weeks) and the final reporting (two weeks).

First use case: The assistive system supports the worker as a third hand, which fastens the screws at taught-in positions on a cylinder head. Real parts were used for the user study. Thereby, the study participants should teach trajectories for the robotic arm.

Second use case: Participants should teach the robot trajectories and positions of the polishing tool. Here, several polishing operations are to simulate with different objectives and the subsequent control and improvement by participants until the robot arm provides satisfactory results. For each use case, accurate movement and positioning, as well as force control were crucial aspects that had to be as close to the requirements as possible.

Third use case: The assistive robot system was deployed in the automotive assembly line for three weeks. There, the worker and the robot collaborated on the task of tightening the screws on the engine block (the operator positioned the screws, and the robot tightened them).

4.1. Research Questions

The focus of our research was on two aspects: (1) identification of major usability problems with the off-the-shelf system for the two application contexts when teaching the robotic system in order two develop new interaction paradigms; (2) collecting suggestions for improvement of the human-robot collaboration in terms of user acceptance and adding a building block to the existing knowledge on the dynamics of user experience in human-robot cooperation in the factory context.

Regarding Aspect 1, we were interested in the following research questions:

(1)

Is there a difference between the two application contexts in terms of the usability of the teach-in (set-up) process?

(2)

Will participants’ attitudes towards robots change due to teaching the robotic system?

(3)

How is the workload of teaching the robot perceived by the users with respect to the application context?

Regarding Aspect 2, we were interested in the following research question:

(1)

How is the collaboration with the robot experienced in the assembly line with respect to user acceptance?

In the following, we will describe the procedure for how we integrated the UCD approach into the implementation of the robotic system for the two application contexts.

4.2. Phase 1: Project Kick-Off Meeting

The first phase consisted of a kick-off meeting with every partner of our consortium in order to introduce each other. The industrial partners explained the specific environmental and technical conditions in order to get a better understanding of the application scenarios of the robot. We introduced ourselves and our role within the project as the scientific HRI-focused partner. The consortium consisted of two industrial partners (providing the testbeds for the use cases), one technological partner (responsible for the implementation of the robot, which was bought from a retailer [49]) and ourselves (responsible for the user-centered evaluation). After clarifying each other’s goals and motivations, we started to prepare how to study the robot in two different application scenarios.

4.3. Phase 2: Preparation of Use Cases and Study Material

In the preparation phase, we defined the three different use cases based on the needs and resources of our industrial partners. After that, the study guidelines and questionnaires were specified, and finally, our industrial partners started the recruitment of participants.

(1)

Use cases: The industrial partner together with the technological partner developed the use cases in order to define how the study participants will interact with the robot within a defined working process. The three use cases were based on the usual working processes and described how the participants had to teach the robot’s trajectories and positions: manually or with the teach pendant. A few weeks before the user studies were conducted, the technological partners visited the industrial partners in order to adapt the robot prototype to the specific process requirements and conditions.

(2)

Study guidelines and questionnaires: The choice of the questionnaires and development of the study guidelines was based on the use cases and research questions. We used established and validated questionnaires (System Usability Scale (SUS) [51], Negative Attitude towards Robot Scale (NARS) [52] and NASA Task Load Index (NASA-TLX) [53]; see Section 4.6 for details) and developed one questionnaire on usability and acceptance questions ourselves. The same study guidelines and questionnaires were used in Use Cases A and B. For the field trial of three weeks (Use Case C), we used an additional interview guideline in order to explore the long-term impressions of the workers in a qualitative manner.

(3)

Recruitment: A week before the each study was conducted, the industrial partners recruited the participants. The participants were all employed at the industrial partners’ factories and were experts in the concerned working processes. The recruitment criteria in terms of age, sex, handedness and prior knowledge of robot control were set to reflect their respective distribution in the factories of our industrial partners. The detailed demographic characteristics of the recruited participants are listed in

Table A1. Demographic characteristics of the participants in each study (SD = standard deviation).

	Use Cases A and C (Screw Tightening)	Use Case B (Surface Polishing)
	(n = 5)	(n = 5)
Age (years)
Mean (SD)	45.2 (11.3)	45.4 (5.7)
Gender
Male (n (%))	3 (60.0%)	5 (100.0%)
Female (n (%))	2 (40.0%)	0 (0.0%)
Handedness
Left (n (%))	0 (0.0%)	1 (20.0%)
Right (n (%))	4 (80.0%)	4 (80.0%)
Both (n (%))	1 (80.0%)	0 (0.0%)
Color Blindness
Green-Red or Blue-Yellow (n (%))	0 (0.0%)	0 (0.0%)
Spatial Imagery
Below-average (n (%))	0 (0.0%)	0 (0.0%)
Average (n (%))	4 (80.0%)	5 (100.0%)
Above-average (n (%))	1 (80.0%)	0 (0.0%)
Excellent (n (%))	0 (0.0%)	0 (0.0%)
Computer Skills
No experience (n (%))	1 (20.0%)	1 (20.0%)
Beginner (n (%))	0 (0.0%)	3 (60.0%)
Experienced (n (%))	3 (60.0%)	1 (20.0%)
Expert (n (%))	1 (20.0%)	0 (0.0%)
Computer Usage per Day
None (n (%))	1 (20.0%)	1 (20.0%)
Less than 1 h (n (%))	0 (0.0%)	2 (40.0%)
1–3 h (n (%))	2 (40.0%)	1 (20.0%)
3–6 h (n (%))	0 (0.0%)	1 (20.0%)
More than 6 h (n (%))	2 (40.0%)	0 (0.0%)
Usually-used OS
Windows (n (%))	5 (100.0%)	4 (0.0%)
Unix/Linux (n (%))	0 (0.0%)	0 (0.0%)
Mac (n (%))	1 (20.0%)	0 (0.0%)
Used Applications
Word Processors (n (%))	4 (80.0%)	3 (60.0%)
Spreadsheet Applications (n (%))	2 (40.0%)	1 (20.0%)
E-Mail Applications (n (%))	5 (100.0%)	3 (60.0%)
Graphics Software (n (%))	1 (20.0%)	0 (0.0%)
Music Applications (n (%))	1 (20.0%)	0 (0.0%)
Desktop-Publishing Software (n (%))	0 (0.0%)	1 (20.0%)
Industrial Machine Skills
No experience (n (%))	0 (0.0%)	0 (0.0%)
Beginner (n (%))	2 (40.0%)	4 (80.0%)
Experienced (n (%))	2 (40.0%)	0 (0.0%)
Expert (n (%))	1 (20.0%)	1 (20.0%)
Industrial Robot Skills
No experience (n (%))	1 (20.0%)	3 (60.0%)
Beginner (n (%))	3 (60.0%)	2 (40.0%)
Experienced (n (%))	1 (20.0%)	0 (0.0%)
Expert (n (%))	0 (0.0%)	0 (0.0%)
Industrial Robot Experience
None (n (%))	3 (60.0%)	5 (100.0%)
Less than 3 months (n (%))	0 (0.0%)	0 (0.0%)
3–6 months (n (%))	0 (0.0%)	0 (0.0%)
6–12 months (n (%))	1 (0.0%)	0 (0.0%)
1–3 years (n (%))	0 (0.0%)	0 (0.0%)
More than 3 years (n (%))	1 (20.0%)	0 (0.0%)

(see the Appendix).

4.5. Phase 3: Performing User Studies

The short-term user studies in which participants should teach the robot specific movements (screw positions vs. polishing positions) took place in an automotive factory (Use Case A) and in a laboratory environment (Use Case B). The first study was conducted within one day; the latter within two days. The studies started by introducing every person (participant, experimenters, technological partner). Next, the participant filled in pre-study questionnaires, and then, the aim of the study and how the robot system can be taught was explained by the technical partner. Afterwards, participants were asked to put on a mobile eye-tracker and to teach the robot the same way as the technical partner demonstrated. In order to understand the thought processes of the participants, that is why they are doing something, they were asked to “think aloud” during the study. During the task, the participant could ask the experimenter and the technological partner for help. After the teaching was completed and the robot performed the learned procedure, the participants had to assess how successful the robot will be in executing the taught movements. In a second run, the participants had to repeat the task, but without the possibility to ask anyone for help. The technological partners and the industrial partners were observing the participants in order to get an impression of the product in real use and to help in emergency situations. The procedure of how we conducted these studies is described in detail in the following:

(1)

Pre-study questionnaires: The questionnaires (demographic questionnaire and NARS [52] questionnaire on attitudes towards robots) were filled in by the participants before any interaction with the robot happened.

(2)

Introduction of the robot: Each participant was introduced to the assistive robot, the control mechanisms and the goal of the trial. In order to relieve stress and increase compliance, the participants were assured that the focus of investigation was only the robot’s performance and there were no negative implications for them.

(3)

Conducting the user study: Each participant was audio- and video-taped with two cameras. Thus, every interaction could be recorded from different distances and angles in order to reproduce the subjective perspective, as well as the context of interaction:

(a)

Videotaping of the first-person view: A head-mounted camera (eye-tracker from Pupil Labs [54]) recorded the participants’ interaction with the robotic arm, the work piece and the touch panel during the teaching procedure. The recording of this point-of-view camera provided additional information concerning the participants’ focus shifts for where he/she was approximately looking.

(b)

Videotaping the whole scenario: One camera was used to get an overview of the scenario for the data interpretation of the eye-tracking recordings and for documenting the think-aloud information.

The use of a first-person and a context-oriented camera helped us with generating a holistic perspective during the later analysis of both use case groups (see Figure 3).

(4)

Post-study questionnaires: The questionnaires (NARS [52], SUS [51], NASA-TLX [53] and a self-developed questionnaire on usability and user acceptance) were filled in by the participants performing the teaching task with the assistive robot.

The field trial took place at the assembly line in an automotive factory (Use Case C) with the participants from Use Case A. The robot system was deployed for three weeks and operated at the same time as the worker in order to cooperatively tighten the screws on an engine block.

(1)

Introduction of the robot at the assembly line: First, the robot was introduced at the assembly line in the industrial partner’s factory. The participants were informed about the aim of the study and how to interact with the robotic system (either manually or by using the touch panel; see Figure 4) during the working process. This field test lasted for three weeks and should help the worker with developing profound opinions concerning the daily cooperation with the robot. Our technological partner was only present in case of a system malfunction.

(2)

Post-field test interview: An interview was conducted after the field test by using established interview guidelines [8] in order to ensure a phenomenological-oriented empirical perspective on the usability and user experience. This approach focuses on: (1) general aspects regarding the work with the robot; (2) experiences before the introduction of the robot; (3) first confrontation with the new system/enrollment; (4) special experiences working with the new system; and (5) special aspects of the HRI (e.g., safety). This open and narrative interview technique evokes personal experiences [55], which were recorded in order to reveal opinions, rumors, impressions, emotional issues, concerns and ideas for improvements.

4.6. Data Analysis

For each participant in Use Cases A and B, the footage of the different views was separately analyzed. For the video analysis of the eye-tracking data, a coding scheme with respect to the interaction modes with the robot (manually or touch panel) was created. The expressed thought processes (think-aloud) were transcribed and grouped content-wise for the final report. The analysis of the questionnaire data followed the established procedures for these validated scales:

• SUS (System Usability Scale) [51]: a low-cost standardized usability scale. It is used to quickly measure the effectiveness, efficiency and satisfaction of a system and includes 10 statement-based items. Participants can rate them on a five-point Likert scale. After the system has been evaluated, the overall score for the usability of the investigated system is calculated (the scores on individual items are not meaningful). The overall score is between zero and 100 with scores being grouped as follows: “80–100: users like the system”; “60–79: users accept the system”; “0–59: users do not like the system”. Additionally, the SUS can be separated into a usability and a learnability dimension for a further differentiation of the results [56].
• NARS (Negative Attitude towards Robot Scale) [52]: a psychological scale to measure the negative attitudes of humans against robots. It should become visible which factors are the reasons preventing individuals from interacting with robots. The NARS questionnaire consists of 14 questions, which are grouped into three sub-scales: S1 = negative attitude toward the situations of the interaction with robots; S2 = negative attitude toward the social influence of robots; S3 = negative attitude toward emotions in the interaction with robots.
• NASA-TLX questionnaire [53]: The NASA-TLX is a subjective workload assessment tool. NASA-TLX allows users to perform subjective workload assessments on operator(s) working with various human-machine systems. NASA-TLX is a multi-dimensional rating procedure that derives an overall workload score based on a weighted average of ratings on six sub-scales. These sub-scales include mental demands, physical demands, temporal demands, own performance, effort and frustration. The sub-scales are rated for each task within a 100-point range with five-point steps.

Furthermore, we developed additional questionnaire items in order to get insights concerning the subjective usability and acceptance of the users (see the Appendix). It features a six-point Likert scale (0–5) and concerns the following aspects:

•

Performance expectancy is defined as the degree to which an individual believes that using the system will help him or her to attain gains in job performance. Its estimated value is the mean of Item 1 and the inverted Item 6.

•

Effort expectancy is defined as the degree of ease associated with the use of the system. Its estimated value is the mean of Item 2 and Item 7.

•

Perceived competence describes the user’s belief that the system has the ability to recognize, understand and react adequately to given tasks with a sufficient set of applicable skills. Its estimated value is the mean of Item 3 and Item 8.

•

System trust describes the user’s belief that the system performs with the needed functionality to do a required task. Furthermore, the system has to demonstrate being able to effectively provide help when needed (e.g., through a help menu) and to reliably or consistently operate without failing. Its estimated value is the mean of Item 4 and Item 9.

•

Perceived safety describes to what extent the robotic system is perceived as safe by its users. Its estimated value is the mean of Item 5 and Item 10.

The post-field test interviews (Use Case C) were conducted by the authors of this paper, who are independent of the factory administration and the robot developers. The interviews were recorded and analyzed following an established thematic analysis approach [57]. The aim of the analysis was the structuring and the interpretation of the data and the derivation of findings. In order to avoid predefined categories, we structured the relevant subjective feedbacks and experiences using an exploratory and data-focused method. In the first step, the single feedbacks and experiences were tagged according to their keywords, which made a rough identification of the relevant issues possible. In the second step, the issues were compared with regard to these keywords in order to merge overlapping topics into distinct categories or to clearly differentiate them from each other.

4.7. Final Discussion of the Study Report with the Industrial Partner

A final report was created including the results of the interviews, the questionnaires and the video analysis. This report was presented to all of the consortium partners in order to clarify the strengths and weaknesses of the current prototype and to deliver vital input for a further development iteration.

5. Results

All participants in Case Studies A and B completed both teaching rounds (i.e., with and without the help of the technological partner), except for one individual in use case B, who completed only the first trial. However, every participant was recorded and filled in the post- and pre-trial questionnaires. The demographic background of participants was very similar for Use Cases A and B (see the Appendix, Table A1). The fact that there were more men can be explained by the men-women ratio in the industrial context. Although the number of participants clearly is too small in order to derive statistical significances, we will report trends one can estimate from having a look at the descriptive statistics of Use Cases A and B. For the questionnaire results, we had the following pre-assumptions:

•

NARS: The negative attitude towards the robot decreases after the human-robot interaction. This expectation is based on other studies [58], which showed such a decrease after short-term human-robot interactions.

•

NASA-TLX: All scores are less than 50%, which indicates low workload. There are no differences expected between the application contexts.

•

SUS: The touch panel for teaching the robot is rated lower in terms of usability (due to its off-the-shelf complexity) than the assistive robot itself (manual teaching).

•

Self-developed items on perceived usability and acceptance: Use Case A should be rated slightly better than Use Case B, as the screwing task could be more robustly automized with the off-the-shelf robot than the polishing task.

The results of the NARS mainly show a decrease of the negative attitude towards the robot in all sub-scales after the human-robot cooperation (see

Table 2. Comparison of the Negative Attitude towards Robot Scale (NARS) rating before and after the human-robot interaction before and after each study (0 = “low”, 4 = “high”).

Sub-Scale	NARS Sub-Scale Description	Use Case A (n = 5)		Use Case B (n = 5)
		Pre (SD)	Post (SD)	Pre (SD)	Post (SD)
S1	Negative attitude towards situations and interactions with robots.	1.27 (0.52)	0.75 (0.52)	1.40 (0.51)	1.23 (0.46)
S2	Negative attitude towards social influence of robots.	1.92 (0.54)	1.40 (0.96)	2.08 (0.46)	2.08 (0.46)
S3	Negative attitude towards emotions in interaction with robots.	2.87 (0.42)	1.50 (0.50)	2.33 (0.99)	2.33 (0.31)

). The subscales were calculated as accumulated mean values over several items according to [52].

The results mainly go in line with our assumption that the negative attitude should decrease after the interaction. However, this effect is more observable for Use Case A than B. This could be due to the fact that participants of Use Case B overall considered the robot as less useful for their working routines than in Use Case A (according to the think-aloud data).

The differences between the two application contexts (teaching screw driving and polishing movements) were also observable in the results of the NASA-TLX, the SUS and the additional items concerning subjective usability and acceptance (see

Table 3. Comparison of the NASA-TLX rating before and after the human-robot interaction for both use cases. Each NASA-TLX scale has 21 grades, and the estimates are converted into percentages to facilitate comparison (0% = “very low”, 100% = “very high”).

NASA-TLX Item Description	Use Case A (n = 5)	Use Case B (n = 5)
Mental Demand: How mentally demanding was the task?	51.25%	48.00%
Physical Demand: How physically demanding was the task?	37.50%	20.00%
Temporal Demand: How hurried or rushed was the pace of the task?	20.63%	59.00%
Performance: How successful were you in accomplishing what you were asked to do?	86.88%	71.00%
Effort: How hard did you have to work to accomplish your level of performance?	53.13%	38.00%
Frustration: How insecure, discouraged, irritated, stressed and annoyed were you?	9.38%	27.00%

,

Table 4. Comparison of the System Usability Scale (SUS) ratings for the control panel and manual robot teaching (0 = “low”, 4 = “high”). Subscales according to [56].

Sub-Scale	SUS Sub-Scale Description	Use Case A (n = 5)		Use Case B (n = 5)
		Panel	Robot	Panel	Robot
S1	Usability (item mean)	2.69	3.15	2.00	2.38
S2	Learnability (item mean)	3.38	3.63	1.70	2.20
	Total score (0%–100%)	70.73%	81.15%	48.50%	58.50%
	Score group	Users accept the system	Users like the system	Users do not like the system	Users do not like the system

and

Table 5. Comparison of additional items related to subjective acceptance after each study (0 = “very low”, 5 = “very high”).

Sub-Scale	Subjective Acceptance Sub-Scale Description	Use Case A (SD) (n = 5)	Use Case B (SD) (n = 5)
S1	System Trust (mean of Item 1 and inverted 6)	3.75 (0.71)	2.90 (0.42)
S2	Safety (mean of Items 2 and 7)	4.13 (0.18)	3.00 (0.00)
S3	Perceived Competence of the System Handling (mean of Items 3 and 8)	4.38 (0.18)	3.30 (0.71)
S4	Performance Expectancy (mean of Items 4 and 9)	3.00 (0.00)	2.40 (0.00)
S5	Effort Expectancy (mean of Items 5 and 10)	4.13 (0.18)	3.30 (0.71)
	Total score (0%–100%)	79.50%	61.60%

). For Use Case B, the participants rated the overall performance lower (71.00%) and the task as more frustrating (27.00%) than for Use Case A (see Table 3). In general, as we expected, most of the NASA-TLX scales were below 50%, which indicates the potential of the off-the-shelf robot to be a useful tool for collaboration after some usability improvements.

Furthermore, the manual control of the robot was rated more positively than the touch panel in both use cases as the results of the SUS questionnaire revealed (participants were asked to fill in the SUS twice, once for the robot itself and once only for the teach panel). However, participants in Use Case B rated both control methods less positive than participants in Use Case A (see Table 4).

Similar differences between the application contexts are evident in the additional items concerning subjective impressions. All items were rated slightly higher in Use Case A (see Table 5).

The biggest differences can be observed for the scales safety and perceived competence. This can be interpreted with the additional fact that participants partly blamed the system and partly themselves for the lack of performance in the polishing task.

The gathered qualitative data from the analysis of the think-aloud data from the videos showed that the touch panel in its off-the shelf version is experienced as not feasible and too complex to control the robot for both use cases. A strong tendency was observed in the video to omit the panel as an intermediate device and to directly control the robot using physical force (manually) (see

Table 6. Frequency of user behavior and stated feedback during the short-time user studies. The feedback categories are derived by clustering the video-taped results.

Short-Time Feedback Category	Use Case A (n = 5)	Use Case B (n = 5)	Total (n = 10)
Manual robot guidance is very bulky, especially with one hand.	5	5	10
The menu architecture of the touch panel is overwhelmingly complex.	5	5	10
Some menu buttons are too small to touch accurately.	4	4	8
Tap/swipe/scroll on the touch panel sometimes does not work.	4	3	7
Manual robot guidance is preferred (hand-pushed or via joystick).	3	2	5
The connecting cable of the touch panel is very bulky.	2	1	3
The angle of the robot arm/tool should be controllable (also for manual guidance).	1	2	3
The high number of coordinate axes makes controlling cognitively demanding.	2	0	2
Robot/screwdriver should be prepared so that shorter people can better handle it.	2	0	2
The teaching process should be linear (straightforward) featuring more feedback.	1	1	2
The robot arm should have more sensors to detect edges and obstacles.	0	2	2
The whole teaching process takes too long.	0	2	2
For smaller persons, the visual switch between panel and robot arm is difficult.	1	0	1
The automatic processing of surfaces should be configurable.	0	1	1

).

However, the manual control was also limited with respect to feasibility due to the bulky robot arm. Furthermore, the dissatisfaction with the systems can be explained by the fact that the actual teaching was only a fraction of the whole process (see Figure 5).

Finally, the interview data from the field trial in the automotive factory revealed that the robot’s working pace was perceived as not flexible enough, which bears the risk of re-establishing a rigid production line logic (see

Table 7. Frequency of statements after three weeks of field testing in the assembly line. The feedback categories are derived by clustering the interview results.

Long-Term Feedback Category	Use Case C (n = 5)
The robot will replace my job in the near future.	5
The robot determines my work and speed.	4
The robot should be more adaptive.	4
The robot does not stop at physical contact.	4
To work with the robot was a big transition.	3
The robot is a helpful tool.	2
The robot should have more anthropomorphic features.	2

). Overall, after three weeks of field-testing (Use Case C), all participants were anxious of being replaced by robots in the near future. More details on the interview data can be found here [59].

6. Interpretation and Recommendation

The three conducted user studies covered laboratory conditions, the industrial context and a field test of three weeks in length. We could learn that the cooperation and the user experience strongly depend on the system’s control modes and degree of adaption. This helps us with differentiating the following aspects for a further development iteration:

(1)

Touch panels: Panels are an intermediate device to control the robot. This implies a continuous switching of visual attention between the robot arm, work piece and touch panel. This implies a high cognitive and attention-related workload for the user. Furthermore, only one third of the time was used for actual teaching, and the rest was used for other activities (e.g., menu navigation). Thus, the application of usability guidelines in this real-time system is absolutely necessary (more intuitive, easier to learn, more shortcuts), and the physical handling of the panel must have high priority (smaller, lighter, more sensitive, without a cord). These insights are in line with Pan et al. [24], who emphasize the importance of teach pendants in the factory context, and Fukui et al. [60], who stated that such input devices should be easy to hold and to operate.

(2)

Manual control: Although the robot arm was described as too bulky to use in practice, there is an overall tendency of preferring manual control over control with an intermediate device. There are various possibilities to integrate manual control, including direct hand pushing, via joystick or via gesture recognition. However, this approach needs an increase of sensors and artificial intelligence in order to live up to its potential. These insights are in line with research concerning teaching motor skills by physical HRI [32] and industry-oriented application [35]. Our technological partner drew similar conclusions concerning the need to improve the manual HRI. Direct interaction should be improved by using more sensors, including cameras for the (1) recording of the whole working place and (2) the detection of potential human-robot movement collisions. The teaching process can be significantly improved by using, e.g., a 3D joystick system.

(3)

Impact on “work”: The field test revealed that the robot prototype was not well perceived with regard to its degree of adaptation towards the users’ individual rhythm, speed and working steps. The robot determined the working pace of the operator, due to its lack of sensors and intelligence. We assume that this essential lack in flexibility is at least partly responsible for the other mentioned shortcomings in usability, perceived safety, system trust, general helpfulness and personal working rhythm of the workers. Furthermore, all workers expressed their fear of being replaced by such a robot in the near future. Although the robot was introduced as a tool and cooperative agent, this fear was not reduced after three weeks of personal human-robot interaction. These empirically-derived insights are in line with Bonekamp and Sure [61], whose review of the current literature on the implications of Industry 4.0 on human labor and work organization revealed a rather consistent view, particularly on job redundancies for low-skilled jobs. However, the factory management discussed the results with the work council, who has a veto concerning further technical and social developments.

The goal of our research was to evaluate an off-the-shelf robotic system in three different use cases for two different application contexts. The results showed us what has to be done in order to develop a robot system that acts as a supportive tool for the workers. However, there is a thin line between a supportive robot and a robot that limits the temporal, physical or social needs of workers, and ultimately replaces them. We believe that the cooperative approach would help with easing the transition of working with a new robot and to increase its overall user experience. Further research and field tests are planned in order to get a deeper understanding and improvement of the mentioned shortcomings. However, firstly, the system will be technically revised to the next expansion stage, including additional sensors for posture recognition in order to measure the persons’ movements and waiting positions, which may improve the factor of perceived safety, as well. More details on this can be found in [62].

7. Conclusions

This article reported three case studies with a cooperative off-the-shelf assistive robot prototype in two application contexts: (1) the assembly of automotive combustion engines; and (2) the machining (polishing) of continuous casting molds. We followed a UCD approach, in which we studied how workers, recruited from our industrial partners’ factories, performed the teach-in (set-up) and collaboration with the robot w.r.t. usability and acceptance aspects. The operators worked with the robotic prototype in laboratory conditions (two days), in a factory context (one day) and in an automotive assembly (three weeks). We used a multi-dimensional qualitative and quantitative methodology in order to get insights into how the human-robot collaboration was experienced. The results of the case studies show the urgency of adaptation of assistive robot systems in order to avoid a change of working routine for the workers. This importance of flexibility is also reflected in the ratings of the teach pendant (a touch panel needed to control the robot), which should be completely replaced through direct physical control. Furthermore, all workers were afraid of being replaced by this or similar robotic systems in the near future. This anthropocentric dimension of human-robot cooperation was also revealed in related research [38], which also suggests that hybrid robots (combining a social- and tool-like appearance and behavioral elements) will be potentially preferred by naive users in the industrial context in the future. Our research so far could not show that a more socially-designed robot would ease collaboration, but that a tool-like robot, which still allows human flexibility and socializing with co-workers, due to high usability and intuitiveness, will be more accepted in the long run.

Our industrial partners drew similar conclusions concerning the need to improve the human-robot collaboration, preferring intuitive and manual solutions, which no longer require an additional touch panel. The precision, safety and adaptation can be improved by the employment of further sensors. The discussion and decision loop also includes the work council, who has a veto concerning further technical and social developments.

The importance of the insights gained through the presented research can be considered as high due to the empirical basis of the approach used (three weeks of field testing; real assembly line; representative workers). However, the limitation of the presented research is the small number of participants (three use cases with five participants each). However, five participants for each use case should in this specific project not be considered as a small sample size, as in the factories of our industrial partners, only these five recruited workers are actually collaborating with the robot. Moreover, the user studies for Use Cases A and B do not aim at quantifying acceptance parameters, but at identifying major usability problems and, by that, to develop new interaction concepts. A small participant number of five can be sufficient to identify the most severe usability problems, as was already discussed by [63]. However, the variety of use cases and the multi-dimensional approach used helped us to get a better understanding of the technological and anthropocentric challenges in Industry 4.0 factories where humans and robots will work more and more closely together.