Tools and Methods for Human Robot Collaboration: Case Studies at i-LABS

Abstract

The collaboration among humans and machines is one of the most relevant topics in the Industry 4.0 paradigm. Collaborative robotics owes part of the enormous impact it has had in small and medium size enterprises to its innate vocation for close cooperation between human operators and robots. The i-Labs laboratory, which is introduced in this paper, developed some case studies in this sense involving different technologies at different abstraction levels to analyse the feasibility of human-robot interaction in common, yet challenging, application scenarios. The ergonomics of the processes, safety of operators, as well as effectiveness of the cooperation are some of the aspects under investigation with the main objective of drawing to these issues the attention from industries who could benefit from them.

1. Introduction

Until the introduction of Industry 4.0 principles, robotics in factories was mostly about machines replacing laborers who were tasked with non-ergonomic duties. The exploitation of robots was almost limited to manipulation of heavy loads or in uncomfortable positions, execution of dangerous tasks due to toxic payloads or environment, and execution of monotonous repetitive operations. Nowadays, thanks to the wide spread of collaborative robots (or cobots), the trend is shifting, especially in small and medium enterprises (SME’s) [1]. In fact, small batch production and high level of product customization make these industrial entities still based on the versatility of human labor. Cobots, for their part, had a chance to easily insert themselves in this productive paradigm for they have been specifically developed for coexistence with people.

In this scenario, the collaboration of the researchers of Università Politecnica delle Marche together with five companies operating in the technology sector resulted in the creation of a laboratory which aims to disseminate the principles of Industry 4.0 in the local industrial fabric. Among them, collaborative robotics plays a crucial role, alongside with the development of tools and strategies for Human Robot Collaboration [2,3]. The aim is that of achieving the seamless team dynamics of an all-human team, and to this goal the last decade of research focused on several aspects [4,5] going from the machines themselves (exteroception [6,7], collision avoidance [8,9], intrinsic safety of the mechanics [10]), to their relation capabilities with people (interaction modalities [11,12], control oriented perception such as gesture recognition [13] and gaze tracking [14,15]).

Many relevant studies of the recent past can be mentioned trying to assess some of the HRC related aspects. Rusch et al. [16] quantified the beneficial impact, under both ergonomic and economic point of view, of preliminary simulated design of HRC scenarios. Papetti et al. [17] proposed a quantitative approach to evaluate a HRC simulated application. Similar topics were analysed in [18] where authors considered such opposing effects on collaborative production lines. However, besides the economic impact, ergonomics analyses surely play a fundamental role for injury hazard management [19,20]. This becomes even more important considering the conclusions drawn by Dafflon et al. [21]: in light of their claims, completely unmanned factories are not reasonably possible mainly due to the un-feasibility of such complex control systems. Therefore, processes involving humans (Human In The Loop, HITL) will be implemented more and more frequently, justifying the effort of developing dedicated simulation environments and evaluation metrics for anthropocentric approaches [22,23,24,25].

However, the perception of operator commands in HITL tasks remains a widely investigated topic [26,27,28], since it still represents a critical point of the HRC process. Robustness, effectiveness, and above all safety are the keywords to be kept in mind. Thus, the interest aroused by strategies not involving contact among machines and workers is well understendable. Dinges et al. [29] considered the use of facial expressions to assess aggravated HRC scenarios; Yu et al. [30] used a multi-sensor approach to sense both postures and gestures to be interpreted as commands. In all of these examples, the key role played by artificial vision is evident, as also featured by authors of [31].

In the following, some of these topics are exemplified by case studies approached by the researchers at i-Labs. The aim is that of providing an impression of the ongoing research developed to pursue the principles of Industry 4.0 for what concerns the cooperation among humans and machines. In particular, a simulated environment is used to evaluate the ergonomics of an industrial work-cell in the first example. The second case considers the use of virtual reality for testing of obstacle avoidance control strategies applied to a redundant collaborative industrial manipulator. The third case study shows how a simple yet effective gesture recognition strategy can be developed to control tasks executed by a dual arm cobot. For the last case study, an obstacle avoidance control algorithm has been implemented on a collaborative robot to validate the effectiveness of the proposed law.

2. Simulation Methods in HRC Design

The method proposed in this section aims to support the human-oriented HRC design by merging the enabling technologies of industry 4.0 with the concepts of human ergonomics, safety, and performance. As shown in Figure 1, it starts with the human work analysis according to objective ergonomic assessments. The XSens™MVN inertial motion capture system is used to measure joint angles and detect awkward postures. The analysis results drive the collaborative robotic cell design; for example, allocating the non-ergonomic tasks to the robot. The preliminary concept is then virtually simulated to identify critical issues from different perspectives (technical constraints, ergonomics, safety, etc.) and improve the interaction modalities. In these two phases, the NX (https://www.plm.automation.siemens.com/global/uk/products/nx/ (accessed on 1 January 2022)) and Tecnomatix (https://www.plm.automation.siemens.com/global/en/products/tecnomatix/ (accessed on 1 January 2022)) Process Simulate by Siemens are respectively used. The simulation contributes to the design optimization so that all the requirements are satisfied. The realization of the physical prototype allows the HRC experimentation, optimization, and validation before being implemented in the real production line. The experimentation phase includes a new ergonomic assessment to quantitatively estimate the potential benefits for the operator.

The industrial case study refers to the drawers’ assembly line of LUBE Industries, the major kitchen manufacturer in Italy. Currently, a traditional robot and a CNC machine make the first part of the production line automated. Then, three manual stations complete the drawer assembly. Figure 2a shows one of the involved operators, equipped with 18 Xsens MTw (Wireless Motion Tracker), performing the tasks. The analysis involved all the labourers qualified for that specific task for a total of 6 operators (3 males, 3 females) all having average anthropometric characteristics. All the participants were informed about the goal of the study and the procedure. Also they were asked to read and sign the consent form.

The human work analysis highlighted a medium-high ergonomic risk (asymmetrical posture and stereotypy) for the operator dedicated to screwing. Before the objective evaluation, this workstation was mistakenly considered the most ergonomic of the three. Accordingly, the HRC design was based on the need to assign the screwing task to the robot to preserve the operator’s health. Figure 2b shows the preliminary concept of the workstation that includes the following elements: conveyor belt; Universal Robot UR10e; collaborative screwdriver with a flexible extension; an aluminum structure where the robot is fixed by four M9 holes; L-shaped squaring system; clamping system; dispenser for feeding the screwdriver, easily and safely accessible by the operator to reload it; two plexiglass fences to reduce collisions risk.

The simulation highlighted several critical issues to be solved. To overcome some safety and technical problems, a new type of screwdriver was designed and implemented, which provides a maximum torque of 12 Nm and weighs 2.3 kg. To improve the line balancing the first two manual stations were aggregated in the cooperative robotic cell. However, the ergonomics simulation (Figure 2c) highlighted potential ergonomics risks for the operator during the drawer rotation.

Then, an idle rotating roller was inserted so that the operator can manually turn the conveyor belt comfortably and safely. The simulation also showed that the new cooperative cell increased the current takt time. Consequently, an autoloader (FM-503H) was introduced to avoid unnecessary cobot movements. It consists of a tilting blade (suction) that loads the screws from a collection basket. Finally, the human-robot interaction has been improved by introducing the Smart Robots (http://smartrobots.it/product/ (accessed on 1 January 2022) ) vision system, which monitors the area of cooperation and enables the robot program according to human gestural commands. It allows the management of a high number of product variants.

In the new HRC workstation (Figure 2d), installed at i-Labs, negligible ergonomics risks, according to RULA (Rapid Upper Limb Assessment) and OCRA methods, were observed. As summarized in

Table 1. Performance comparison between the “as-is” workstation, the preliminary concept of cooperative robotic cell and the final prototype.

	AS-IS	Pre-Simulation HRC	Validated HRC
RULA	$\mathrm{Right}=5$	$\mathrm{Right}=3$	$\mathrm{Right}=2$
	$\mathrm{Left}=3$	$\mathrm{Left}=3$	$\mathrm{Left}=2$
OCRA	$\mathrm{Right}=13.3$	$\mathrm{Right}=5.7$	$\mathrm{Right}=5.7$
	$\mathrm{Left}=1.9$	$\mathrm{Left}=1.9$	$\mathrm{Left}=1.9$
PHR		$\mathrm{Event}1=14.06$	$\mathrm{Event}1=5.62$
		$\mathrm{Event}2=15.62$	$\mathrm{Event}2=0.94$
		$\mathrm{Event}3=20.62$	$\mathrm{Event}3=0.47$
Takt time	57 s	$<57$ s	$<57$ s
Balance	$\sim 70\%$	$\sim 74\%$	$\sim 80\%$

, also the Pilz Hazard Rating (PHR), which is calculated by (1), was reduced to acceptable values.

$$PHR=DPH\times PO\times PA\times FE$$

(1)

where:

• $DPH$ is Degree of Possible Harm
• $PO$ is Probability of Occurrence
• $PA$ is Possibility of Avoidance
• $FE$ is Frequency and/or Duration of Exposure

As shown, the new production line respects the cycle time (57 s) imposed by the CNC machine and the introduction of a collaborative robot led to better line balancing ($+10\%$). The lab test was aimed at recreating the actual working conditions, therefore both a male and a female volunteer (actually two researchers of the i-Labs group) have been asked to take part in the investigation. Also, in this case, their anthropometric characteristics can be considered average.

3. HITL: Gesture Recognition Examples

In the search for effective ways to increase collaboration between workers and robots, new types of communication can be considered. Voice commands, for example, are typical in many systems today, such as vehicles, home automation systems, assistive robotics, etc., but they are not suitable for the industrial environment, where noise and the coexistence of multiple workers in a shared space make their implementation impossible. On the other hand, image sensors, such as standard or RGBD cameras, can be exploited to capture workers’ gestures, which can then be interpreted by artificial intelligence algorithms to generate a command to be sent to the robot.

A distinction should be made between applications that require the use of wearable devices and those that rely on direct hand and body recognition. Regarding the former type, the most common technology for hand gesture recognition are glove devices. As part of their work at i-Labs, the authors developed the Robotely software suite for the specific purpose of overcoming the need for wearing gloves by adopting a hand recognition system using a machine learning approach [32]. As shown in Figure 3, the software exploits MediaPipe libraries in order to recognize 20 landmarks of the hand and identify up to three different hand configurations that can be converted to commands which can be sent to whatever robot or system. A specific test case was carried out at i-Labs: a PC with a standard webcam was used to run Robotely and connected to a Universal Robots UR10e cobot in order to sequence some operations between the operator and the robot.

A second example is shown in Figure 3 where the commercial Smart Robots system is integrated with the ABB YuMi cobot. Smart Robots is a programmable system based on RGBD sensors directly integrated into the robot control software. Again, three different hand gestures can be recognized and converted into commands, such as stop in Figure 4a, calling a program variation such as a quality inspection (Figure 4b), and opening the gripper for part rejection (Figure 4c). In addition, it is possible to define areas of the workspace corresponding to the activation of different program sequences: when the operator’s hand enters a predefined area of the scene (once, or typically twice to avoid inadvertent activation) a program state is changed. In Figure 4d), for example, the operator covers a letter drawn on the table with his hand to order the robot to compose that letter with LEGO bricks.

4. Safe Autonomous Motions: Collision Avoidance Strategies

This section recalls the obstacle avoidance algorithm introduced by authors in [8,9] and presents some experimental results. The research in this field is presented hereby as an advancement in the direction of safe human-robot coexistence. Also, the resulting control algorithm is at the base of the test case presented in the subsequent section, which makes use of virtual reality tools to test and calibrate the empirical parameters introduced by the algorithm.

For a start, it is possible to consider the velocity kinematics of a generic manipulator owning a number of actuators greater or equal to 6 (depending on its degree of redundancy). In matrix form, the velocity kinematics of such a system can be written as $\dot{\mathbf{x}}=\mathbf{J}\dot{\mathbf{q}}$, where $\mathbf{J}$ is the $6\times n$ arm Jacobian (with n number of the joints composing the kinematic chain of the manipulator, $n\ge 6$) and $\dot{\mathbf{q}}$ is the vector of joint rates. In such terms, the vector of variables is $\mathbf{q}={\left[\begin{array}{ccc}{q}_1& \dots & q_n\end{array}\right]}^T$ where $q_1\dots q_n$ are the n joint variables of the arm kinematic chain. With such notation, the Jacobian $\mathbf{J}$ depends on the structure of the manipulator. The inverse of the Jacobian $\mathbf{J}$ of the redundant system can be obtained as a damped inverse:

$${\mathbf{J}}^{*}={\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T+{\lambda }^2\mathbf{I})}^{-1}$$

(2)

where $\lambda$ is the damping factor, modulated as a function of the smaller singular value of the Jacobian matrix (the interested reader is addressed to [8,9] for further details). The inverse ${\mathbf{J}}^{*}$ can be used to compute the joint velocities needed to perform a given trajectory with a Closed-Loop Inverse Kinematic (CLIK) approach:

$$\dot{\mathbf{q}}={\mathbf{J}}^{*}\left(\dot{\mathbf{x}}+\mathbf{K}\mathbf{e}\right)$$

(3)

where $\dot{\mathbf{x}}$ is the vector of planned velocities, $\mathbf{K}$ is a gain matrix (usually diagonal) to be tuned on the application, and $\mathbf{e}$ is a vector of orientation and position errors (${\mathbf{e}}_r$ and ${\mathbf{e}}_p$), defined as:

$$\mathbf{e}=\left[\begin{array}{c}{\mathbf{e}}_r\\ {\mathbf{e}}_p\end{array}\right]=\left[\begin{array}{c}\frac{1}{2}\left(\mathbf{i}\times {\mathbf{i}}_d+\mathbf{j}\times {\mathbf{j}}_d+\mathbf{k}\times {\mathbf{k}}_d\right)\\ \mathbf{P}-{\mathbf{P}}_d\end{array}\right]$$

(4)

In (4) the subscript d stands for desired planned variable, $\mathbf{P}$ is the position of the end-effector, while $\mathbf{i}$, $\mathbf{j}$ and $\mathbf{k}$ are the unit vectors of the end-effector reference frame.

The collision avoidance strategy is then implemented as a further velocity component (to be added to the trajectory joint velocities) capable of distancing the end-effector and the other parts of the robot from a given obstacle. Such contribution is a function of the distance among the obstacle and every body of the robotic system. To this purpose, the bodies have been represented by means of two control points, $\mathbf{A}$ and $\mathbf{B}$ referring to Figure 5. Called $\mathbf{C}$ the centre of a generic obstacle, the distance among it and the segment $\overline{AB}$ can be differently computed in three different scenarios:

• ${\left(\mathbf{C}-\mathbf{A}\right)}^T\left(\mathbf{B}-\mathbf{A}\right)\le 0$, Figure 5a: in this case the obstacle is closer to the $\mathbf{A}$ tip than to any other point of $\overline{AB}$; the distance d among the obstacle center and the line $\overline{AB}$ coincides with the length of $\overline{AC}$.
• $0<{\left(\mathbf{C}-\mathbf{A}\right)}^T\left(\mathbf{B}-\mathbf{A}\right)<|\overline{AB}{|}^2$, Figure 5b: the minimum distance d lies within points $\mathbf{A}$ and $\mathbf{B}$. In this case, it is:

  $$d=\frac{\left|\left(\mathbf{C}-\mathbf{A}\right)\times \left(\mathbf{B}-\mathbf{A}\right)\right|}{|\overline{AB}|}$$

  (5)
• ${\left(\mathbf{C}-\mathbf{A}\right)}^T\left(\mathbf{B}-\mathbf{A}\right)\ge |\overline{AB}{|}^2$, Figure 5c: point $\mathbf{C}$ is closer to any other point, therefore $d=|\overline{BC}|$.

It is worth remarking that for this paper only spherical obstacles were considered, although similar approaches can be developed for objects of any shape starting from the distance primitives here defined.

Now the set of repulsive velocities for the $\mathrm{i}\mathrm{th}$ body of the system can be introduced as:

$${\dot{\mathbf{q}}}_{r,i}=\psi v_r\left(k{\mathbf{J}}_{\mathbf{A},i}^{*}\frac{\left(\mathbf{A}-\mathbf{C}\right)}{|\overline{AC}|}+(1-k){\mathbf{J}}_{\mathbf{B},i}^{*}\frac{\left(\mathbf{B}-\mathbf{C}\right)}{|\overline{BC}|}\right)$$

(6)

where:

• $\psi$ is an activation parameter, function of d, of the obstacle dimension $r_o$ and the length $r_i$ which characterize the body $\mathrm{i}\mathrm{th}$ (as represented in Figure 5, the dimension $r_i$ defines a region around the line $\overline{AB}$ given by the intersection of two spheres centred in $\mathbf{A}$ and $\mathbf{C}$, and a cylinder aligned with $\overline{AB}$, of radius $r_i$). The activation parameter can be whatever function such that $\psi =0$ if $i>r_i+r_o$, and $\psi =1$ if $i\le r_i+r_o$. Actually, such a transition can be made smoother by the adoption of feasible functions (polynomials, logarithmic, etc.).
• $v_r$ is a customized scalar representing the module of the repulsive velocity provided by obstacle to the body.
• k is a parameter which depends on the three cases of Figure 5: in the first case $k=1$ so that only the point $\mathbf{A}$ influences ${\dot{\mathbf{q}}}_{r,i}$; in the second case both $\mathbf{A}$ and $\mathbf{B}$ are considered proportionally to their distance from $\mathbf{C}$, thus $k=1-{\left(\mathbf{C}-\mathbf{A}\right)}^T\left(\mathbf{B}-\mathbf{A}\right)/|\overline{AB}{|}^2$; at last, in the third case $k=0$ so that only the point $\mathbf{B}$ is relevant to ${\dot{\mathbf{q}}}_{r,i}$.
• ${\mathbf{J}}_{\mathbf{A},i}^{*}$ and ${\mathbf{J}}_{\mathbf{B},i}^{*}$ are the damped inverse of the Jacobian matrices of points $\mathbf{A}$ and $\mathbf{B}$.

At this point, the CLIK control law (3) can be completed as:

$$\dot{\mathbf{q}}={\mathbf{J}}^{*}\left(\dot{\mathbf{x}}+\mathbf{K}\mathbf{e}\right)+\sum _{i=1}^m{\dot{\mathbf{q}}}_{r,i}$$

(7)

being m the number of segments used to describe the manipulator.

At last, some experimental results are shown as demonstration of the obstacle avoiding strategy consistency. Going a little into details, the CLIK control law of the KUKA KMR iiwa (a 7 axes redundant collaborative manipulator) was built using 6 different segments, while the two obstacles were modelled as spheres. As shown by the test rig image (Figure 6), the robot carries a collaborative gripper by Schunk which is also considered for the definition of the robot segments (cfr Figure 7). The results are presented in terms of joints positions and rates, and pose and velocity of the robot end-effector (EE). Each variable is referred, for comparison, to the values collected during the execution of the same trajectory without obstacles in the robot workspace. The robot was controlled via Matlab with a cycle frequency of $\sim 150$ Hz. It is worth remarking that the visualization provided in Figure 7 is a plot built on experimental data. The trajectory under investigation is a simple linear motion with constant orientation of the EE. Two obstacles have been put in the robot workspace, one directly on the planned trajectory ($O_1$ in Figure 7) and the other in the space occupied by the robot non-terminal bodies ($O_2$ in Figure 7). The results show how the control law is able to follow the given trajectory even in presence of the two obstacles. As expected, obstacle $O_1$ (met approximately in the time span 2–4.5 s, gray area in graphs of Figure 7) prevents the robot EE from maintaining the desired pose: the control law permits the dodging of $O_1$ deviating the bare minimum from the planned motion. Regarding obstacle $O_2$ (met in the span $5.5$–$7.5$ s, purple areas), the control law exploits the robot redundancy to avoid the contact among the robot elbow and the obstacle while maintaining the EE on the right trajectory and orientation.

5. Virtual Reality Based Design Methods

In this section a virtual implementation of the obstacle avoidance control strategy is presented. Aside the experimentation previously shown, the algorithm has been tested in advance in a simulated interactive scenario to demonstrate the feasibility of the control law, and to tune the parameters used to optimize the robot response to a dynamic obstacle. Such an approach, schematically shown in Figure 8, represents an efficient paradigm for design and virtual testing of HITL applications. Starting in a simulated environment, the interaction workflow can be designed considering also the presence of labourer and their impact on the production. A further optimization step can be added to optimize the interaction among humans and robots exploiting virtual reality. The VR tools permit one to disregard the modeling of labourer behaviour since their presence is played by actual humans who bring in the design process not only the actual expertise of the production line final users, but also the randomness of human actions and movements with further advantages in terms of security features design.

For this purpose, the idea was to create a virtual reality (VR) application in which a real obstacle is inserted in the simulation loop, while the robot is still completely virtual. The advantages of this approach are many: firstly, the algorithms can be modified and fine-tuned ensuring safety for the operators, secondly these changes can be made easily and quickly, reducing the development time of the real application.

The control architecture used for the VR application is shown in Figure 9, while Figure 10 is the experimental rig. The core of the system is the VR engine, running on a standard PC, developed by the SteamVR development suite with Unreal Engine 4 used to model the kinematics of the KUKA robot. A classic HTC VR set is used to equip the operator: the HMD provides an immersive three-dimensional representation of the workspace, which in this case is simply the robot mounted on a stand; the controller, manually managed by the operator, is rigidly connected to a virtual obstacle, shaped like a sphere, which can be moved in the virtual space to interfere with the robot’s motion. The control of the robot is executed by the same PC in a parallel thread developed in Matlab, with a frame rate set to a typical value for communication protocols with robots (e.g.,$125\mathrm{Hz}$ or $250\mathrm{Hz}$, depending on the manufacturer). The frame rate of the VR engine is set to $60\mathrm{Hz}$, which is the standard display refresh rate in VR applications. The communication between the two executed threads is realized by TCP/IP protocol. A classic pick and place task is simulated, thus the robot controller sends joint rotations to the VR engine at each time step, similarly to what is done in real robotic systems. The trajectory of the robot is updated in real time by the collision avoidance algorithm if the obstacle enters in a safety region of the manipulator. The position of the obstacle is known once the VR engine reads the coordinates of the controller held by the operator and sends this information to the robot controller.

6. Concluding Remarks

The paper showed the results obtained by the i-Labs laboratory in the wide field of research on human-robot interaction. Many technologies have been investigated and experimented to horizontally approach the issue of close cooperation among labourers and machines, going from off-line design, to on-line safety oriented control strategies.

The first case study showed a possible approach to the design and optimization of an assembly work-cell. The optimization, oriented at enhancing the labourer ergonomics and safety, allowed the realization of the prototype work-cell for the cobot assisted assembly of furniture pieces. The outcome quality has been quantitatively estimated.

Similar aspects of Human In The Loop operations have also been investigated in the second case study, which was about the realization of a visual based interaction paradigm among humans and robot. In this case, a depth camera was used to recognize the labourer commands which allowed the execution of simple tasks by the robot. This kind of contactless interaction pushed a little bit further in direction of security, which have been the core focus of the third case study.

The third case introduced an innovative control algorithm for active obstacle avoidance, applicable to any cobot. The strategy, based on the method of repulsive fields of velocity, was tested on a redundant industrial cobot to evaluate the possibilities offered in terms of both safety and task execution ability. At last, a Virtual Reality approach for the detail design an exploitation of the obstacle avoidance strategy has been described in the fourth case study.

The technologies which have been separately presented in this manuscript, actually represent some of the enabling technologies in the field of Industry 4.0 and 5.0 which place the human operator at the very center of the production process. Such a knowledge base can be exploited for the enhancement of industrial processes, especially in SMEs, which represent the main matter of the local industrial tissue.