Novel Approach Using Risk Analysis Component to Continuously Update Collaborative Robotics Applications in the Smart, Connected Factory Model

Featured Application

Digital risk analysis of collaborative robotics applications, featuring connections with manufacturing execution systems to support change management.

Abstract

Building on the idea of Industry 4.0, new models of the highly connected factory that leverage factory-generated data to introduce cost-effective automation and involve the human worker for creating higher added value are possible. Within this context, collaborative robots are becoming more common in industry. However, promises regarding flexibility cannot be satisfied due to the challenging process of ensuring human safety. This is because current regulations and standards require updates to the risk assessment for every change to the robotic application, including the parts involved, the robotic components, and the type of interaction within the workspace. This work presents a novel risk analysis software tool that was developed to support change management for adaptive collaborative robotic systems in the connected factory model. The main innovation of this work is the tool’s ability to automatically identify where changes have been made to components or processes within a specific application through its integration with a connected factory architecture. This allows a safety expert to easily see where updates to the risk assessment are required, helping them to maintain conformity with the CE marking process despite frequent changes. To evaluate the benefits of this tool, a user study was performed with an exemplary use-case from the SHOP4CF project. The results show that this newly developed technology for risk assessment has better usability and lower omission errors when compared to existing methods. Therefore, this study underlines the need for tools that can help safety engineers cope with changes in flexible robotics applications and reduce omission errors.

1. Introduction

The manufacturing domain is constantly presented with challenges due to increasing demands for mass customization and leads to a production referred to as “high-mix low-volume” (HMLV) [1]. The fourth industrial revolution, also known as Industry 4.0 (I4.0), promises to solve these challenges through the extraction of information from the production data created by interconnected processes. More specifically, an I4.0 factory will be a pluralism of interconnected processes that stem from new forms of collaboration, such as: machine-to-machine, machine-to-human, and human-to-human [2]. These processes will receive information on ways to efficiently perform manufacturing through the analysis of the production data [3]. Specific examples of the advantages given by the adoption of I4.0 principles can be found in [4]. However, I4.0 concepts are slow to be adopted by European small and medium-sized enterprises (SMEs) [5], especially in the context of robotics [6]. Therefore, ways to further expand their adoption are necessary.

Human–robot collaboration (HRC), a form of human–machine interaction, is one of the I4.0 areas where adoption is limited. HRC is often implemented to make use of the cognitive abilities of humans to augment machines, executing repetitive or physically strenuous tasks. However, ensuring safety is often a barrier to more widespread usage [7]. This is because changes to an application require review, and possibly an update, to the risk analysis, depending on the significance of the change. While a program can be automatically generated to adapt to new parts that are to be manipulated, risk analysis is completely manually generated at this time and requires the oversight of a safety expert. Therefore, in this paper, a methodology and a software component (i.e., risk analysis tool) to mitigate this issue are presented.

The main innovation of this work is the risk analysis tool and its integration within the overall connected factory architecture. This integration means that it can automatically identify where changes to components or processes within a specific collaborative robot (cobot) application have been made. In case of changes to the application that could potentially require an update to the risk analysis documentation (as part of the CE marking process), only the changed areas are highlighted. This allows the safety expert to easily see where updates to the risk assessment are required. The overall proposed workflow still requires a human safety expert to review all the changes to the risk analysis and evaluate whether the risks have changed; however, by marking all changes, the burdensome task of searching and finding these areas is eliminated. The authors believe that this is a particularly useful tool that can be used in industry in the near future.

To present the main innovation of this tool, this article is structured as follows. Initially, an introduction of the state-of-the-art methods of risk analysis in collaborative robotics applications, as well as of smart factories for HMLV productions, is given in Section 1.1 and Section 1.2. Second, the overall architecture proposed by the EU-funded project, SHOP4CF (https://shop4cf.eu/ (accessed on 5 January 2022)) for smart factories of the future, is outlined in Section 2, highlighting specific software components used for process modeling and risk analysis documentation, and for connecting these with a high-level manufacturing execution system (MES). Third, an exemplary use-case, developed within the project to illustrate how the components work in practice and to provide a comparison of the proposed methods against traditional methods through a user test, is presented in Section 3 and Section 4. Finally, the main findings and future directions are summarized in Section 5 and Section 6.

1.1. Smart Factory and Collaborative Robots

One of the most challenging scenarios that the smart factory will face is enabling a quick response to volatile markets through the production of small product batches containing highly customized products. Recent I4.0 technologies and concepts can be used to achieve this goal. For example, data-driven approaches can enable manufacturing and assembly systems to quickly switch between variant parts and products [8]. Additionally, collaborative robots can be employed in assembly tasks to assist humans [9,10]. Despite these possibilities, achieving lean manufacturing in HMLV environments is still a complex and on-going research topic [11].

Should the smart factory utilize collaborative robots, several barriers hindering their adoption need to be addressed [12]. When integrating cobots in scenarios with high variance, differences in design, configuration, and safety settings are necessary. For example, depending on the objects to be manipulated, different end-effectors (EE) and picking strategies might be required [13,14,15]. Consequently, different programming strategies [16] or types of collaboration, such as the ones shown in Figure 1, might be necessary. Furthermore, the human collaborating with the robot might have different requirements in terms of ergonomic needs and workplace safety [17]. This list of differences can become quite long. Therefore, we see that a robotic (collaborative) application meant to address the HMLV challenge will feature high variability in terms of tools, hardware, and software. Thus, being able to track all these components and perform a proper safety analysis, considering time constraints, can be challenging.

1.2. Risk Analysis and Risk Assessment for Collaborative Robotics Applications

Given the flexible nature of freely programmable robots with their ability to change tooling and manipulated parts, as well as the various types of interactions possible with humans sharing the workspace, a risk analysis is typically required to identify possible hazards, estimate the risks to human health and safety, and identify suitable risk mitigation strategies [19]. This is executed according to the ISO12100:2010 “Safety of machinery–General principles for design–Risk assessment and risk reduction” standard [20]. The risk analysis and the evaluation of the risks (which, in this combination are also known as the risk assessment) are conducted by a safety expert, who receives support from the mechanical and electrical engineers responsible for various aspects of designing the HRC application. In addition to simple methods for documenting a risk assessment, such as in a spreadsheet (such as the one documented in this work), there are several commercially available software systems to support the safety expert in performing a risk assessment (https://www.ibf.at/en/ce-software-safexpert/ (accessed on 5 January 2022)). These software programs focus on documenting hazards, estimated risks, and risk mitigation methods, as illustrated in Figure 2. Newer methods from research [21,22,23,24] focus on automated methods to simplify the overall risk analysis process for collaborative robots. Other studies focus on methods to support non-experts in evaluating the effectiveness of risk mitigation measures [25]. The authors in [26] surveyed safety experts researching this area and identified a need for new software tools that balance ease of use with advanced capabilities for automatic hazard identification and risk assessment. Nevertheless, they expressed the opinion that a risk assessment tool cannot fully replace a human expert, especially with regard to liability and responsibility. While a risk assessment is usually executed when designing and commissioning a new HRC application, it is often overlooked in industry that, according to the machinery directive [27], any significant modification to the HRC application requires an update to the risk assessment. In this context, a substantial modification has been performed when the change leads to new hazards and associated risks that cannot be sufficiently mitigated with existing or simple safeguards [28]. Many HRC applications are delivered to an end-user by a contractor and/or system integrator and, at least in Europe, it is not legally binding to deliver a copy of the original risk assessment as part of the overall documentation to the end-user. Even if documentation of a risk assessment is available, it is often not in an electronic form that can be edited. This makes the task of determining whether a change should be considered a significant modification difficult. To the best of the authors’ knowledge, there are no commercially available risk analysis software tools that directly connect with factory information systems (e.g., MES) to perform the risk assessment and support change management. Therefore, the authors see a clear gap in the state-of-the-art methods and tools that help human experts update and maintain risk assessments of collaborative robotics applications, which are highly flexible and change often. By including human experts in the loop, such a system can be used in industry immediately and without hesitancy, as the responsibility for the decisions remain with a human expert.

2. Proposed Smart Factory Methods and Methodology

This section describes the methodology followed to address the issue of risk analysis in cases of frequent changes in a collaborative application. First, Section 2.1 describes the SHOP4CF architecture. Then, Section 2.2 describes the adopted data modeling approach. Finally, Section 2.3 and Section 2.4 describe the developed components for risk analysis based on the architecture and modeling. The architecture, the data modeling approach, and the manufacturing execution system components are all enabling technologies, without which the risk analysis component would not be able to function. While these systems and technologies are not the main focus of this article or the evaluation, the authors feel that their description is useful for the reader to understand the overall context of the risk analysis component and the peripheral systems involved for proper utilization.

2.1. Overall Architecture and Data Models

To address some of the challenges faced by SMEs, the EU-funded Smart Human Oriented Platform for Connected Factories (SHOP4CF) project was created. SHOP4CF aims to create a unique infrastructure for the convenient deployment of human-centric industrial applications. In the project, a team of twenty partners developed a comprehensive software platform containing a wide range of components that cover a broad spectrum of industrial requirements, especially in the context of flexible and data-rich manufacturing.

SHOP4CF aims to find the right balance between the cost-effective automation of repetitive tasks and the involvement of human workers in areas such as adaptability, creativity, and agility, where they create the greatest added value. Furthermore, the project pursues a highly connected factory model to exploit the benefits of all data generated within a factory. The SHOP4CF approach builds on existing work, including the HORSE (http://horse-project.eu/ (accessed on 4 March 2022)) project and the L4MS (http://www.l4ms.eu/ (accessed on 4 March 2022)) project on smart logistics for manufacturing. Initially, the HORSE consortium designed, developed, deployed, and tested the HORSE framework—a reference architecture for cyber-physical systems that integrate smart technologies and provide manufacturing operations management in hybrid-actors settings [29]. The framework is a modular architecture with clear subsystems and interfaces at several levels of aggregation, resulting from a structured, hierarchical system design based on theoretical principles and guidelines [30]. From a functional high-level perspective, it distinguishes between manufacturing activities taking place in a work cell and activities in a production area or even site (across work cells). This distinction is depicted with two levels, global and local. There is also a clear distinction of phases, one regarding design of manufacturing activities (e.g., modeling and parameterization) and one regarding executions of manufacturing activities (e.g., actual product manufacturing)—i.e., the design and execution phases. Following, the L4MS project was a four-year project with the goal to create a one-stop shop for manufacturing SMEs to help them digitalize intra-factory logistics. This was achieved by Open Platform for Innovations in Logistics (OPIL) [31], an open Internet of Things (IoT) platform with different enablers and the common aim of simplifying the development of customized logistics solutions. The OPIL reference architecture consists of three layers: the IoT nodes layer (L1), which is the layer where software components interact with the physical world using well-established technologies (e.g., robot operating system (ROS)); the cyber-physical middleware layer (L2), which allows interoperability among components of the platform and with external ones; and the software systems layer (L3), which consists of software components developed specifically for the logistics sector.

A similar approach was adopted in SHOP4CF. The platform aspect of the architecture represents the organization, from the functional perspective, of the software and hardware necessary for the software components to be operational. Its top-level logical view is illustrated in Figure 3.

The software layer consists of SHOP4CF components, the middleware, containers (i.e., OS-level virtualization), and third-party information systems (i.e., external to SHOP4CF, such as MES, as discussed in Section 2.4) which may exist on a shop floor. The hardware layer consists of servers. In addition, cyber-physical systems and IoT devices of a shop floor may belong to both layers. Regarding the middleware, the chosen platform is FIWARE, as in the L4MS project, due to its open-source capabilities and wide support from other European projects (https://www.fiware.org/about-us/impact-stories/ (accessed on 14 January 2022)). FIWARE focuses on management of context information—i.e., the current state of the surrounding real world, understood as the state of relevant physical and virtual objects (for instance, a virtual object may be a manufacturing task to be executed). The use of context information helps in developing and representing information used in a smart factory use-case. SHOP4CF components exchange information via the FIWARE middleware whenever possible. Only connections that have real-time constraints are organized directly between two involved components (or between a component and IoT), as the FIWARE middleware does not guarantee response times for real-time systems [33].

To address the focus on data-rich manufacturing environments, the SHOP4CF architecture explicitly adds analytics functionality, in addition to the design and execution functionalities of the previous frameworks, to the SHOP4CF platform. Thus, it consists of six main modules, for which interaction through interfaces are shown on the high-level logical software architecture in Figure 4.

Each of the components built within the project realizes a (sub)set of six main modules. Manufacturing scenarios that require specific functionalities are then addressed by an integrated set of components, for which interoperability is secured by the well-defined interfaces of the architecture and the data models. Considering the use-case in this work, the SHOP4CF architecture can be used as follows: a manufacturing process management component (MPMS) [34] covers the design global through process modeling and the execute global through automated process enactment; MES forwards the order information of a modeled process and conveys the information stored in the local and global datastores (DB), which are labeled as SpecG and SpecL in Figure 4. Finally, a risk analysis tool covers the design local by providing safety requirements for the processes that have to be executed. These requirements are incorporated into the process modeling, so that the subsequent actual execution of the activities is safe. This scenario is elaborated in the next sections once details of the components have been presented.

With respect to data modeling, the focus is on the interoperability among components (i.e., the information exchanged between components) and not on the way that a specific component translates and uses that information internally. By following the well-established formal approach from [35], data requirements are translated to concept data models, consisting of definitions of data entities, their attributes, and the relationships between entities. Then, by applying technical constraints (i.e., concrete technical data format) and considering both existing FIWARE data models and the ISA-95 standard [36] specific SHOP4CF, data models are defined (https://shop4cf.github.io/data-models/ (accessed on 14 January 2022)). Within these data models, of utmost importance in this use-case, are the entities known as process definition and process. The process definition, also similar to the process segment of the ISA-95, is a definition describing the necessary steps to achieve a certain goal related to an order (e.g., assemble a door). The process, also similar to the operations segment or operations definition, is a definition describing the steps that should be performed to achieve a certain goal considering specific parameters (e.g., assemble a black door with black windows) and involved resources. These data models can be used to refer to a process and its parametrization. For example, if a process involves a collaborative robot, safety should be proven considering the parametrized process (i.e., type of robot).

2.2. BPMN Modeling

Product variety, stemming from customization and personalization trends, imposes variety on production equipment and processes [37]. Moreover, the introduction of novel technologies and the increased collaboration between humans and robots result in more complex manufacturing and assembly operations. Flexible modeling is thus imperative for tackling production complexity. The process models should allow for clear definitions of tasks, their detailed sequence with interdependencies, visual expressiveness, optimized allocation of resources, and automated enactment. In manufacturing, various modeling notations have been used, such as value stream mapping (VSM) [38], integration definition (IDEF) [39], and unified modeling language (UML) [40]. In recent efforts, the widely adopted business process model and notation (BPMN) [41] has seen increasing interest for modeling manufacturing processes. While IDEF, VSM, and UML satisfy most of the requirements mentioned above, BPMN also shows an important advantage with its execution semantics [42]. Moreover, BPMN can be used as a common language for integration of business and manufacturing processes [43,44].

BPMN is the de facto standard for business process modeling; however, it requires extensions to support manufacturing processes in which physical entities are included (in comparison to business processes where information processing is dominant). A few extensions have already been proposed with the focus on including physical objects and activities in the models [45], or on modeling IoT-aware processes [46,47,48].

With respect to modeling of HRC applications (e.g., product assembly), BPMN is still a promising candidate. Schönberger et al. [49] used BPMN in a new approach called human—robot time and motion (HRTM) for modeling collaborative tasks. HRTM combines the methods time measurement (MTM) approach, which is used for modeling the working steps of human workers, and the robot time and motion (RTM) approach, which is used for modeling the working steps of robots. Froschauer et al. [50] extended HRTM by combining it with ADAPT [51], a universal modeling approach that allows a shift of programming complexity from the end-user to a modeling expert. However, the modeled BPMN workflows do not include an automatic trigger for robots to perform the corresponding command at the right time. Moreover, in a recent work [52], a digital worker assistance system based on BPMN was presented. The notation was used to enable non-professionals to create adaptive task-sharing processes between human workers and cobots.

Considering the advantages that BPMN can bring, and a collaborative use-case, a similar approach to [50] was employed in this work and in the SHOP4CF consortium. However, some additions were added. Froschauer et al. [50] proposed to model a collaborative task by using two pools (one for the robot and one for the operator), modeling the interaction by using time delays whenever one actor had to wait for a task to finish. Our approach considered having just one pool to represent the entire process, using lanes to identify the actors (i.e., robot and operator). The single pool eases the execution of the process model with respect to messages exchanged between actors. In [50], the two separate pools added extra communication messages. Moreover, in their approach, it was not clear how a “communicate” task was linked to a gateway element (on their model of Figure 1). Secondly, in our approach, the use of time delays, which require an actor to wait until the other actor finishes some tasks, was removed, as this requires intelligence and the precise knowledge of the exact task duration values. Such calculations might be difficult or even pointless to perform. Instead, the AND-split and merge points for separating work between actors were chosen. These two changes ended up providing the BPMN model shown in Figure 5. The added modifications enabled a clear definition of the resources involved in process (i.e., pool) and a description of the type of collaboration involved (i.e., parallel, or sequential tasks).

2.3. Manufacturing Execution System Component

To meet the requirements of HMLV production, an MES needs to provide integration functions that enable it to adapt the production quickly and improve the production output. In this case, an MES for dealing with production changes in the context of a collaborative application was created. The MES had to integrate its capabilities to receive a process definition in the form of a BPMN file, process the BPMN file to extract the necessary resources involved in the process, query the available resources, and insert a production order as a process once the available resources had been mapped to the process. Hence, through these steps, the MES acts as a system and parametrizes the received BPMN to make it executable.

To satisfy these requirements, the internal architecture of the MES is based on a database, following the specifications of the top-level logical data architecture within SHOP4CF [32]. This top-level logical data architecture is visualized in Figure 6.

Using this architecture, the MES parses a BPMN file and create database entities, which reflect the BPMN content. This is achieved by creating entities in the MES database that are mapped to the BPMN artifacts. For sake of clarity, an example based on the BPMN process (as shown in Figure 5) is given here. Starting from the top level, initially, the assembly pool was mapped to the MES as a process definition, considering that it represents an entire process. Secondly, the different activities both for the robot and the operator were mapped to several entities of the task definition, representing all the activities that the process definition contains. To highlight this relationship, each task definition was assigned to a process definition in the MES database. Finally, the different resources (operator, robot, gripper, transmission gear, and motor holder) were mapped into the MES database with the resource specification entity. Due to the connection of the activities to the different resources, this cardinality was maintained by linking the resource specification entities with the task definition entities. Through this mapping, it was possible to represent the BPMN content in a structured format.

After parsing the information, the MES then leaves to a user the possibility of initiating a production order based on a process definition. To do so, the MES must first collect the available resources by querying all the entities marked as “available” in the database, and then display the ones matching the types defined in the resource specifications. Through this procedure, the user could then specify which resources to assign for the order according to capabilities (e.g., John operator, UR 10 robot, Zimmer two-finger jaw gripper) and then finally send the order for clearance through the risk analysis component. This is possible through the interaction of the MES and the RA via the FIWARE context broker, as shown by the sequence diagram in Figure 7.

2.4. Risk Analysis Component

The risk analysis component is used when designing and/or adapting an HRC application. The component supports a safety engineer during the hazard identification, risk evaluation, hazard elimination, and risk reduction, as illustrated in Figure 2. It does not automate the process of estimating risks or identifying hazards. Instead, it highlights which changes to an existing system have resulted in changes to the risk profile and need to be reviewed and enforces formal risk analysis and change management processes. The component allows the user to both import the process structure and insert additional data. The user is then guided through the process of estimating the risk and documenting hazards and risk reduction measures. When performing an update, changes that require the user’s attention are automatically detected by making comparisons against the previous configuration. The change detection is based on comparing identifiers and searching for matches between the previous version and the updated one to find added, modified, and deleted elements. The component was designed to allow both manual data entry and to import data from a variety of sources, such as a digital twin, or from a factory MES. The focus of the risk analysis is on mechanical, HRC-related hazards (e.g., crushing, impact) listed in Annex A of ISO 10218-2 [53]. These can be chosen from a pull-down list.

A novel feature of the risk analysis component is that the data are structured by task—e.g., a partial process, action, or operation performed by some actor in the assessed system, such that every hazard is always associated with a task. With this approach, the tasks are connected to resources (which correspond to a physical entity such as robotic equipment, a part of the environment, or a person) and actions (operations that are performed by one or more resources). A basic assumption of this model is that some action (e.g., motion of some dynamic parts—automated or mechanical equipment of the system, interaction with their environment as part of normal or abnormal operation, putting people or materials on site) is always required in order to introduce risks into a state which would otherwise be safe (e.g., when the equipment is standing still, the factory is empty, and no one is around). It is easy to imagine a counterexample case in which the mere presence of the equipment or materials might constitute a hazard (consider for instance the storage of explosives or corrosive chemicals). Such cases would not fit into the model used by the application; however, one could still consider defining “storing dangerous chemical 1” as a task that covers this specific case. This “task-based modeling” of an application fits well with a robot-centric view of an application. Risks that exist due to a robotic application are heavily dependent on specific information, such as the robot’s speed during a defined movement. As such, the proposed level of granularity is considered sufficient for understanding what the robot is doing and allows the safety expert to identify associated hazards and risks. Although the component is also able to document the evaluation of the individual risks (risk assessment) by the safety expert, the main focus of the innovation is on the methods for asset and task definition, as well as hazard identification during the risk analysis phase.

In the SHOP4CF architecture, the risk analysis component is a web server running inside of a container, which has its own database, a communication channel to FIWARE context broker, and a modern, single-page application user interface. The component supports multiple users with varying access levels. In the normal workflow, a safety expert would import a BPMN model to create the process structure (tasks and their resources), then identify common hazards and perform the initial risk assessment.

A risk analysis project is bound to, and continuously monitors, a specific FIWARE process entity during manufacturing. When the process is triggered, the risk analysis component looks up the current dynamic resources assigned to tasks, determined through search queries for FIWARE resource entities recorded by third-party applications, and then creates a configuration in which the changes are automatically detected. Then, it triggers a notification, prompting a review of the changes and an update of the risk assessment. Once that is completed, the component sends the risk assessment result back to the third-party system via the context broker.

The history of changes to the process structure and configuration can be reviewed, providing transparency and traceability for safety-relevant changes. The project data can be exported or imported in JSON format, providing open data portability or integration with additional tools.

3. Smart Factory Use-Case

The methodology described above can be applied in different applications. To enable an evaluation of the described risk assessment tool through multiple users, a use-case from the SHOP4CF project was considered. This section describes how the previously explained method can be integrated in a smart factory use-case. Initially, the description of the risk analysis component usage is given in Section 3.1. Then, the use-case and the considered collaborative cell are described in Section 3.2. Finally, ways in which the risk analysis can be used in the use-case, and how it compares with an existing tool, is presented in Section 3.3.

3.1. Workflow for using the Risk Analysis Component

To understand how the proposed methods could be used for the execution of risk analysis based on a BPMN model and resource specific information coming from the MES, the workflow from the user perspective is reported here. The workflow starts with the user inserting the high-level description of a process via the MPMS interface. Second, the BPMN is shared with the MES and the RA to obtain the process definition. Third, the user inserts an order into the MES and specifies which resources have to be used. Fourth, the RA receives the process definition and the resources, and the user performs the risk analysis using the obtained information. Finally, the result is sent to the MES and MPMS, which can then trigger the execution. The sequence of the steps is shown in Figure 7.

3.2. Hardware Set-Up

For performing the user evaluation of the proposed risk analysis component, as described in the upcoming Section 4, a collaborative robotic cell and its components needed to be considered. Most importantly, the robot, the EE, and the safety laser scanner have to be considered to ensure safe collaboration. The selected robot is a UR10 collaborative robot from Universal Robots^® (UR) with ISO 10218-1 [54] certification for the integration of a power and force limiting (PFL) risk mitigation measure. The selected EE is a self-built device composed of two-finger jaw module and a vacuum module disposed at 45° with respect to the approaching direction. Finally, the selected laser scanner was the microScan3 from Sick^® to allow for the integration of a speed and separation monitoring (SSM) risk mitigation measure. The work cell and the EE can be seen in Figure 8. To ensure a correct configuration of SSM and PFL, collision testing was performed. The collision testing followed the GRI-LIE-1 protocol from the COVR toolkit [55], considering pressure limits in ISO/TS 15066 [56]. Therefore, a safe operation speed of 50 mm/s was obtained to ensure maximum exerted pressures. Then, the robot was programmed to adopt the defined speed and monitor for applied pressures whenever the operator entered the workspace. A detailed description of this procedure is reported by the authors in [57].

3.3. Using Risk Analysis Component for Smart Factory Use-Case

To showcase the risk analysis component in the context of the collaborative cell, the collaborative assembly of a gearbox for medical machines is considered. The use-case consists of several steps that need to be completed by the operator, by the robot, or by both. An excerpt of the use-case description using BPMN can be seen in Figure 5; for a description of the full model please refer to the Supplementary Material (File S1). This use-case was selected due to the peculiarity of its process, where changes are often introduced to cope with manufacturing resource availabilities and small batch production.

To illustrate the usage of the risk analysis component in the use-case, the methods and software previously used at the manufacturing site are described. The method for documenting the risk analysis is to use an Excel spreadsheet with special functions designed in Visual Basic for Applications (VBA) (referred later as “legacy tool”). The spreadsheet is composed of multiple worksheets to define the involved resources (e.g., robot type, safety devices, material), describe the work performed, and to document possible hazards in the application. In case a user wants to perform a risk analysis, a table of the necessary resources must be created. This includes, but it is not limited to, robot type, materials for the process, and end-effector. Secondly, the work steps need to be inserted and matched with the different resources. Finally, the user assesses risk levels based on frequency of exposure, probability of occurrence, and degree of severity, according to [58]. For sake of clarity, an excerpt of the legacy tool can be found in the support material (File S2). Considering the use of the legacy tool for a process revision related to a change of resources, the user needs to go through the following tasks. Initially, the user inserts new resources into the appropriate worksheet. Then, the work steps where the resources are involved need to be checked one by one and the risks need to be re-evaluated. Finally, risk mitigations are checked against the new risk levels.

On the other hand, if the revision process with the RA component is considered, the procedure unfolds as follows. Initially, the user changes the newly added resources in the asset tab and provides information about them. Once the change is performed, the modified resources are highlighted in orange, as shown in Figure 9.

Afterwards, the RA highlights all the steps in which the risk should be revised according to the modified resources. To better guide the user, this is highlighted in pink under each step. Once the risks are highlighted, the user can select all the steps connected to the new asset via a grouping function and will be prompted to edit all the hazards connected, as shown in Figure 10 on the left. Once the hazards are revised, new mitigations are proposed, and risks are evaluated by frequency of exposure, probability of occurrence, and degree of severity. The user can save the new risk evaluation for the steps concerned with the modified resource, as shown in Figure 10 on the right.

4. Evaluation

This section describes how the described components are evaluated through a user test by comparing the legacy tool with the RA. Initially, the usability results are presented in Section 4.1. Then, in Section 4.2, key performance indicators (KPIs) related to the changes, which the users performed in the risk evaluation analysis, are presented.

4.1. Usability

An evaluation investigating the usability of the RA tool compared to the legacy method was performed. The evaluation focused on a situation where an EE and one material were switched due to a product design change. The users were instructed to review the risk assessment, conduct any changes, and then update the risk assessment using one of the available tools (File S3 contains the project of the RA tool and File S2 contains the project with the legacy tool). Afterwards, they were asked to repeat the same process, but with the other tool. At the end of each interaction, the participants were asked to provide their evaluation on the tools’ usability via the standardized system usability scale (SUS) [59] questionnaire. An explanatory image of the process is shown in Figure 11.

A total of eight users participated in the study. To avoid biases, four of them used the legacy tool first and the remaining used the RA tool first. The outcome of the survey can be seen in Figure 12 and in the supporting material (File S4). On average, the legacy tool scored worse in the SUS compared to the RA. More precisely, RA had a score of M = 65 (SD = 18.32) and the legacy tool had a score of M = 35 (SD = 16.03). A Welch’s t-test was conducted to prove statistical difference between the samples. From this, it is possible to see that p < 0.05 (CI = 95%), meaning that the RA tool scored statistically better compared to the legacy one.

4.2. KPIs on the User Updates to the Risk Assessment

After the study, the risks documented by the users were analyzed to compare the performance of the users based on which tool they used. The following KPIs were monitored: true positives (e.g., fields that were changed, which were supposed to have been changed, and for which the updated content was correct), false negatives (e.g., omissions, in which fields that should have been updated, were not), and false positives (e.g., fields that should have been changed, but for which the updated content was incorrect). The authors formulated the hypothesis that the RA tool usage would lead to fewer false negative and false positive answers, as well as a higher percentage of true positives. In other words, the authors expect that the RA tool will completely eliminate errors wherein the safety expert forgets to add a change. The results for the different KPIs measuring these impacts can be seen in Figure 13. Looking at the figure, it is possible to see that, on average, the RA had lower performance in cases of true positive entries and false positive entries. However, RA scored better in the category of false negative entries. To check the difference among distributions, a Welch’s t-test was conducted after ensuring preconditions to the test. The test yielded a statistical significance for the group of false positive entries (p < 0.05, CI = 95%) and false negative entries (p < 0.05, CI = 95%). On one hand, RA had a score of M = 43 (SD = 44) and the legacy tool had a score of M = 3 (SD = 5) for the false positive entries. On the other hand, RA had a score of M = 0 (SD = 0) and the legacy tool had a score of M = 19 (SD = 17) for the false negative entries. No further statistical significance was found for the true positive entries.

5. Discussion

This section discusses the results identified in Section 4. First, a comparison of the usability scores of the RA and legacy tool is given in Section 5.1. Then, the KPIs for evaluating accuracy and the performance of the RA are discussed in Section 5.2.

5.1. Usability

The evaluation showed that the RA component was more usable compared to the legacy tool. We believe that this is related to the design specifications of the tool, such as the use of a task-based modeling method, the traceability of the steps, and finally, the automatic detection of changes. For example, two participants in the group that used the RA tool second spontaneously reported that having the changes highlighted was better compared to the legacy tool, with which the user had to scroll and search to find the changes. However, despite the good ratings given in the RA tool, the SUS score is not yet optimal—considering that the average was below 70. This indicates that the tool might not be accepted by the whole user pool [60]. Therefore, further improvements can be considered for future developments. For example, while changing resources either via the MES or via manual entry, the user must confirm for each step that the resource has been changed and no grouping feature, as in the risk evaluation, is present. This leads to the user having to click on “save change” several times, therefore leading to a repetitive task which could have influenced the SUS score. This is demonstrated by one user in the group that used the RA tool second. The user spontaneously expressed that the repetitive clicking for every step was improper if compared with the grouping of the resources in the subsequent risk assessment. Therefore, integrating the grouping feature across all steps could be one way to improve the acceptance of the RA tool. As an example, if a robot is changed and PFL is used as mitigation, the maximum allowable speed would depend on the robot type. Considering these insights, future versions of the RA will limit the number of clicks required to confirm changes and will integrate the grouping feature across different steps in the risk evaluation workflow. These changes could lead to an SUS score above 70, indicating a wider acceptance by the user pool.

5.2. KPIs on the User Updates to the Risk Analysis

A different outcome was recorded in the area concerning the accuracy of the information. In contrast to our hypothesis, usage of the RA tool led to a higher number of false positive entries. However, by looking at the RA risk evaluations in the user tests (File S4), it is possible to see that the users erred by not updating the information regarding the smallest contact area in the event of an impact. In the RA, this information had to be inserted in a long string. However, in the legacy tool, this information was organized in a specially designed entry field. Given that the current layout requires a user to manually scroll sideways in the field (and the entire text is not shown in multiple lines), it is possible that users did not see this or realize that they needed to look more carefully within a long string to make the correct update. We believe that this design error led to an increased percentage of false positive entries. Despite this, the results showed that the RA tool was less prone to false negative entries compared to the legacy tool. We believe that this lower rate of false negative entries was possibly due to the highlighting feature provided. Considering this information, we believe that the designed RA tool can better support engineers when frequent changes to risk evaluations are necessary and some fine tunings are necessary to reduce the amount of false positive entries. Therefore, the future version of RA will integrate special entry fields for HRC-related information such as the smallest contact area, similar to the legacy tool, to further support the user and reduce the rate of false positive entries. Moreover, we believe that the RA tool could improve the overall compliance to machinery directive requirements by reminding safety experts about a needed review. However, further experiments are needed to prove this last hypothesis.

6. Conclusions

This work presented a novel software tool to perform and update risk assessments for industrial applications featuring human–robot collaboration. The paper first presented the state-of-the-art methods of performing safety risk assessments. It then underlined the need for new tools to support re-evaluations of cobot applications by safety experts in HMLV production facilities in order to fulfil requirements on documentation related to the CE mark. In particular, the authors identified that there are no risk assessment software tools available that connect with a factory information system to ensure a seamless flow of data and facilitate change management. Afterwards, the overall architecture and specific components, developed within the SHOP4CF project to address the identified needs, were explained. These included a method for task modeling, a lightweight MES, and a risk analysis component. Finally, the risk analysis component was evaluated against an already existing tool for performing risk assessments through a user test. The tools and overall workflow proposed by the authors had a higher usability and lower omission errors compared to the legacy tools. Unfortunately, the RA component usage showed a higher number of false positive entries. Analysis of this unexpected outcome indicated that a simple adjustment to the GUI—increasing the size of a field with text to show all the text without requiring sideways scrolling—is necessary to improve this result.

These results may be connected to some work limitations which the authors would like to directly address. First, the tests were limited to a small user pool due to availability and the necessary knowledge of robotics by the users. Therefore, this study might better represent the usability as perceived by an engineering-based user pool. Second, while the MES was not evaluated in terms of usability, we believe that it served its purpose to parametrize defined processes ready for safe execution and its functionality does not affect the evaluation of the risk analysis tool. However, further studies on the MES usability can be investigated. Considering the limitations, we would like to encourage the use of the tool by other practitioners and look forward to gathering more feedback in the near future. The risk analysis tool is publicly available and is freely accessible on the robotics and automation marketplace (RAMP) (https://www.ramp.eu/#/ (accessed on 3 March 2022)).

Future work will focus on the testing of the MES, perceived usability by a larger user pool, and improvement of the risk analysis component GUI according to the identified shortcomings.