**1. Introduction**

Human–robot collaboration (HRC) involves one or more humans working with one or more robots to accomplish a certain task or a specific goal [1]. Significant research has focused on interaction aspects for designing robotic systems for use by or with humans [2–6]. This research, which focuses on factors that affect HRC [1,7] at different levels of automation, specifically evaluates the influence of workload.

The level of automation (LOA) of the system, defined as the degree to which the robot and the human are involved in the collaborative task [8–11], influences the characteristics of the dynamics of the collaboration, the behavior of the robots, actions to be taken, as well as autonomy of the human in the collaboration [12,13]. Workload addresses the actual and perceived amount of work that the human operator experiences as related to the effort invested in the task [14,15]. It can be described in terms of the elements that constitute the cost of accomplishing the goal for the human operator in the HRC [16]. These elements could be task-related (such as mental, temporal, and physical demands [17], operator-related (such as skill, strategy, experience [18]) or machine-related (such as poorly designed controls, feedback, inappropriate, or inadequate automation [15]. Workload consequences could be reflected in the stress, fatigue or frustration experienced by the human operator [16], depletion of attentional, cognitive or response resources [15], as well as in performance changes [19]. Workload can also be influenced by task complexity as characterized in terms of the stimuli involved in the task for inputs, as well as the behavioral requirements the human operator should emit in order to achieve a specific level of performance [20]. It could depend on the objective complexity derived from the task properties and on the subjective complexity which is influenced by the human operator's perception [21]. The task properties include the component complexity—number of distinct

**Citation:** Gutman, D.; Olatunji, S.; Edan, Y. Evaluating Levels of Automation in Human–Robot Collaboration at Different Workload Levels. *Appl. Sci.* **2021**, *11*, 7340. https://doi.org/10.3390/ app11167340

Academic Editors: Luis Gracia, Carlos Perez-Vidal and Manuel Armada

Received: 30 May 2021 Accepted: 6 August 2021 Published: 10 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

actions that the human operator must execute or number of informational cues that should be processed (e.g., the number and type of subtasks to be managed, [22]); coordinative complexity—nature of relationships between task inputs and task products, the strength of these relationships as well as the sequencing of inputs (e.g., timing, frequency, intensity and location requirements [23]), and dynamic complexity—changes in the states of the environment which the human operator should adapt to [20,24].

The influence of LOA on HRC has been intensively investigated [25]. However, there are limited studies that investigated factors influencing workload in relation to the design of LOA modes suitable for different HRC collaboration contexts [26]. Moreover, research has revealed that the alignment between manufacturing strategy and automation decisions are often ad hoc in nature [27]. The current study therefore aims to examine the influence of different levels of workload when operating at different levels of automation (LOA) in a human–robot collaborative system. This is important when introducing robotics in real life situations.

To evaluate the overall performance and interaction in such HRC contexts, many different measures are commonly applied for the assessment [22,28–30]. However, by evaluating each measure separately, a holistic evaluation is lacking. We therefore specially designed two constructs that compile different evaluation measures. These constructs are useful in assessing the preferences, performance, and perception of the users regarding various aspects of the collaboration with the robot as required in a user-centered design [31–33]. The constructs are quality of task (QoT) execution (the user's performance aspects) and usability (performance aspects along with other user perception aspects such as perceived ease of use). Additionally, user preferences were evaluated.

We design, implement and evaluate LOA modes in a user study involving 80 participants working at different workload conditions. Section 2 presents the study hypotheses, system design, LOA modes, task, and experimental evaluations of the design. Section 3 is devoted to the experimental results. Discussion is presented in Section 4 while Conclusions and suggestions for future work are discussed in the last section.

#### **2. Materials and Methods**
