Evaluation of Occupational Exoskeletons: A Comprehensive Protocol for Experimental Design and Analysis

Featured Application

This paper proposes a protocol for researchers, ergonomists, and industrial practitioners to design and carry out well-structured experiments that allow for reliable and valid comparisons between different exoskeleton designs or configurations, considering factors such as user characteristics, task demands, environmental conditions, and subjective user perceptions.

Abstract

This paper proposes a modular protocol for the designing of experimental studies to analyze exoskeletons used in industrial settings to support manual material handling (MMH). Despite exoskeleton technologies starting to be highly commercialized and present in workplaces, research still lacks a standardized procedure for analyzing the impact of these devices on workers. The protocol presented in this paper outlines a step-by-step procedure, including the parameters to be collected and analyzed in a research study. Moreover, the approach could be easily adapted to meet the specificity of a wide range of exoskeletons. The main novelty of the protocol is thus to support the experimental design and analysis of studies assessing the overall impact of exoskeletons on workers. To implement the protocol, the selected case study concerned a palletizing task involving the MMH of 12 cardboard boxes, weighing 10 kg. The results from physiological signals and pressure insoles show that the protocol is comprehensive and can be utilized by researchers evaluating different occupational exoskeletons for assistance during MMH (both active and passive), with modifications to specific parts based on the type of exoskeleton being assessed or the variables of interest.

1. Introduction

Exoskeletons have gained considerable interest in various areas due to their potential to enhance human physical performance, reduce physical workloads, and improve ergonomics. Occupational exoskeletons, i.e., exoskeletons designed to support workers’ activities in the workplace, have emerged as a promising technology in the context of Industry 5.0. These wearable devices are designed to assist industrial workers in various tasks, such as lifting, lowering, and manual handling, by providing physical support and reducing the risk of Work-related Musculoskeletal Disorders (WMSDs) [1]. Their adoption during work activities aims to enhance worker safety, productivity, and overall well-being [2]. Industrial research has shown that exoskeletons can positively impact muscle activity, oxygen consumption, perceived exertion, and usability during lifting tasks, improving the overall efficiency of workers [3].

The assessment of occupational exoskeletons has focused on factors such as muscle activity, musculoskeletal effort, contact pressure, and subjective usability, demonstrating the potential benefits and limitations of these devices in the industry [4,5]. In 2018, Huysamen et al. [6] found that exoskeletons are particularly helpful in tasks involving dynamic lifting and lowering, where they can aid in reducing muscle activity, perceived exertion, and contact pressure on the body. More recently, Lazzaroni et al. [7] highlighted the role of exoskeletons in reducing physical workload during manual material handling (MMH), minimizing the risk of injuries, and improving the overall health and safety of industrial workers. Pesenti et al. [8] demonstrated that the use of occupational exoskeletons equipped with sensors for human activity recognition and adaptive payload compensation can further enhance the effectiveness of these devices in industrial contexts. By providing back support and reducing muscular activity, occupational exoskeletons have the potential to reduce the risk of low-back pain and other occupational disorders associated with MMH in industrial settings [9,10,11].

Currently, numerous studies have been conducted to investigate the effectiveness of occupational exoskeletons in the workplace and the potential issues associated with their use [12]. Most studies, for example, use surface electromyography analysis of the major muscle groups involved in performing a particular work task, with or without an exoskeleton [13,14]. Less common are studies dedicated to evaluating specific ergonomic aspects, e.g., usability, accessibility, and acceptance by the users [15]. A parallel research path has highlighted the importance of evaluating exoskeletons in industrial applications with a focus on human factors to ensure their comparability, usability, and ergonomic design [6,16]. Ergonomic indicators such as usability, accessibility, and user acceptance cannot be ignored when humans are required to interact with newly conceived technological devices, which can hardly be understood based on data and knowledge derived from analyses conducted in a controlled laboratory setting [17]. The interaction requires multiple pieces of knowledge related to the user: not only those related to the characteristics of the person for whom a product is designed, but also those that provide information on user preferences and expectations. In this context, the user becomes a fundamental element of the design process. Hence, it is easy to understand how a design focused solely on process and technological requirements can hinder the benefits of these wearable devices. In [18], which identified great variability in the variables and experimental setups when assessing lower limb exoskeletons, a selection of proper Key Performance Indicators (KPIs) from the reviewed papers was obtained and divided into three main categories: “Goal-Level”, “Kinematic-Kinetic”, and “Human-Robot Interaction”. Similarly, ref. [19] stated that current methods suffer from great variability in tests, protocols, subjects, variables, and setup conditions, hindering a clear and coherent analysis of the Physical Human–Exoskeleton Interaction (pHEI). A proper study of the pHEI could represent a crucial step forward in the field of safety evaluation in wearable robots.

Some users reported excessive interface pressure at the points of contact between the exoskeleton and the body, which may lead to an overall perception of discomfort during use [20,21]. Additionally, some studies are questioning the overall impact of exoskeleton use on the health and safety of the workers [22]. There is limited evidence on the long-term effects of these devices on the health of workers, i.e., the deconditioning of the muscles after a prolonged period of use and the physical demands on other body regions, e.g., due to a variation in the load distribution during MMH, may result in further potential risks [5,23]. Also, no standardized guidelines or protocols are available for testing and evaluating the impact of occupational exoskeletons on industrial workers. The lack of specific safety standards and technical problems have been identified as obstacles in the widespread practical use of exoskeletons in industry [16]. Bock et al. [12] suggested that the main parameters measured during the evaluation of occupational exoskeletons are muscle activation by superficial electromyography, physiological parameters, biomechanics, and performance. Botti and Melloni [5] proposed a thorough evaluation of the recent literature on occupational exoskeletons, with an emphasis on the impact of these devices on workers’ health and safety, performance, and users’ subjective perceptions. The authors concluded that caution is warranted and standardized testing protocols should be developed for consistent evaluation, as the evidence regarding the effectiveness of exoskeletons in reducing the risk of WMSDs is lacking. More research is necessary to understand the long-term implications of exoskeleton implementation in industrial contexts and the individual factors that can influence long-term outcomes, such as worker characteristics, task requirements, and exoskeleton design features.

The evaluation of occupational exoskeletons faces challenges due to the lack of standardized procedures. Several studies highlight the absence of consensus on evaluation methods and metrics for exoskeletons, leading to variations in testing conditions across different studies [10,24,25]. This lack of standardization makes it difficult to compare results [26] and draw meaningful conclusions. This paper aims to contribute to the filling of this gap in the lack of standardized procedures and methodologies in the evaluation of occupational exoskeletons by proposing a protocol for researchers, ergonomists, and industrial practitioners to evaluate the impact of occupational exoskeletons in industrial settings, considering factors such as user characteristics, task demands, environmental conditions, and subjective user perceptions. The goal of such a protocol is to advance the understanding of exoskeleton technology, support evidence-based decision-making in developing and adopting these wearable devices, and support improving worker safety, productivity, and overall occupational health in industries where exoskeletons are utilized.

The proposed protocol is based on results from previous studies on occupational exoskeletons. In particular, in [23] the differences in the pressure distributions of feet on the ground were collected and analyzed in two tasks in different industrial contexts; moreover, in a second preliminary study the maximum and average variations in the pressure exerted by the feet on the ground, the variation in the user physiological parameters, and user feedback on comfort and usability were collected during a simulated palletizing task. The importance of executing these preliminary studies, their description, and the results are presented in Section 3 and Section 4 of this paper.

The remainder of this paper is divided into five sections: Section 2 introduces the materials and methods, i.e., the contents of the testing protocol for an experimental study aimed at assessing the impact of occupational exoskeletons on workers; Section 3 proposes the adaptation of the experimental protocol to the study of an occupational exoskeleton supporting manual lifting activities performed in industrial settings; finally, Section 4 and Section 5 discuss the results and introduce the future developments of this research.

2. Materials and Methods

The definition of a testing protocol ensures the standardization of the testing methodologies adopted during the research, thus guaranteeing the repeatability of this study. The testing protocol for an occupational exoskeleton includes four stages: pre-experimental planning, detailed experimental planning, experiment execution, and data analysis (Figure 1).

The first stage, i.e., pre-experimental planning, incorporates the definition of the targets of the study and a list of the variables of interest, such as the type of exoskeleton to be analyzed and the parameters to be collected. Once the type of exoskeleton to be evaluated has been selected, it is possible to choose the task most appropriate for the characteristics of the selected exoskeleton, based on the manual of use and maintenance of the device or the literature. Additionally, conducting one or more preliminary studies allows for a comprehensive awareness of the research objectives and design factors and identifies potential experimental challenges that may arise during the study. The succession of these phases facilitates a more informed and well-prepared approach. Then, the analysis proceeds to the detailed experimental planning. During this stage, the sample of users is selected, together with the tools and equipment required for the research objectives. Also, questionnaires for the users are designed to collect user perceptions and other critical information to understand the interactions between the users and the device. In the third stage, the experiment is conducted as planned, and all relevant parameters are accurately recorded and collected. This stage is performed either in a controlled environment, e.g., a laboratory, or on the field, e.g., in an industrial setting. Finally, a rigorous data analysis is performed in the fourth stage. During this stage, the results of testing are interpreted, raising insightful discussions and conclusions. More details about the contents of each stage are provided as follows.

2.1. Stage I: Pre-Experimental Planning

2.1.1. Target Definition and List of Variables of Interest

The first stage starts with the definition of the target of the study and the identification of the variables of interest, such as the characteristics of the users, the features of the exoskeleton, and the data to be collected and evaluated during its use, e.g., muscle fatigue, muscle activation, kinematics of movements, metabolic consumption and energy expenditure, heart frequency, and the distribution of the pressure of the feet on the ground [13,14,20,27]. Also, qualitative and subjective data, such as the user perceptions about the comfort and usability of the device, are fundamental, as they provide important insights complementing the quantitative data [23].

2.1.2. Preliminary Study

A preliminary study, or pilot study, is an exploratory investigation conducted before the main research, which allows one to gather essential insights, identify potential issues, and refine the research protocol. Preliminary studies are typically conducted on a small scale and may involve a subset of the target population or a simplified version of the main study design. The aim is to assess the feasibility and practicality of the planned research approach, ensuring that the main study can be executed effectively and efficiently. Preliminary studies support the identification of any potential issues, limitations, and deficiencies in the data collection procedures and equipment, allowing for proactive mitigation and refinement before the main study begins. On the contrary, the absence of one or more preliminary studies exposes the research to the risks of unforeseen technical issues that may arise during the experimental execution, potentially hindering data collection efforts and prolonging the overall timeframe required to complete the testing phase.

2.1.3. Selection of Design Factors

The results of a preliminary study suggest the fundamental design factors for the testing protocol that allow the correct execution of the test and avoid executive errors that may affect the evaluation of the data collected during testing. Design factors are controllable factors whose choice and variation within the protocol creation affect the test performance. These include design factors with an impact on the positioning and calibration of the instrumentation, the positioning and fit of the exoskeleton referring to user characteristics, and the design features that allow the device’s familiarization. The recent technical report of the Italian Organization for Standardization, i.e., the UNI/TR 11950 [17], provides a set of recommendations for the evaluation of occupational exoskeletons, suggesting the four phases of a testing protocol for industrial settings. The first phase involves the stakeholders within the organization to ensure effective communication about the device. This phase should include the exoskeleton manufacturer’s or supplier’s involvement and the voluntary participation of the interested users. Secondly, a training phase should include individual sessions between the supplier/manufacturer and the user to provide proper usage instructions and demonstrations. The third phase involves the device usage during usual work activity. The usage of the exoskeleton in the workplace should be phased in gradually to allow for proper acclimation and ensure user acceptance. This can be achieved by incrementally increasing the duration of exoskeleton usage over time, such as extending the number of hours of use during each work shift and over days.

2.2. Stage II: Detailed Experimental Planning

2.2.1. Definition of the Sample (Sample Size and Participants)

An analysis protocol for the investigation of occupational exoskeletons necessitates the recruitment of an adequate sample size of users to ensure that the findings have sufficient statistical power for valid interpretations. The statistical power of a test is the probability of correctly rejecting the null hypothesis when the alternative hypothesis is true. Mathematically, the power is defined as

Power = 1 − β

(1)

where β is the probability of making a Type II error, which is failing to reject the null hypothesis when the alternative hypothesis is true [28]. Such power depends on several factors, including the sample size, the significance level (α), and the effect size [28]. The sample size is the number of users who participate in the experiment. Increasing the sample size increases the power of the test. The significance level (α) directly impacts the ability to detect true effects and avoid false positives. For a fixed sample size, decreasing α decreases the power. The effect size represents the magnitude of the effect to measure, i.e., the phenomenon or relationship under investigation that can be reliably detected [29]. The larger the effect or difference being tested, the higher the power. Varying any of these three factors results in a corresponding increase or decrease in the statistical power of the study. For example, the power increases with the effect size and the sample size, i.e., large effects are easier to detect than small effects and large samples have smaller standard errors than small samples. Finally, power increases with α, i.e., a larger α means that the threshold for declaring a difference as significant is reduced [29]. Figure 2 shows the relationship between the required sample size and the desired statistical power while holding constant the significance level (α) and the anticipated effect size.

A higher power is desirable because it means the test is more likely to detect an existing effect. Typically, researchers aim for a power of 0.8, which describes an 80% or greater probability of detecting an existing effect [29]. Higher values of power are associated with large samples of users. This can lead to execution problems, such as excessive time required for completing the study, numerous resources required for performing the tests, and higher levels of exposure to potential risk factors with no apparent benefits for the research study [29,31].

Data from previous studies in the same field can be used as a reference to estimate the values of the introduced parameters for the power analysis. Otherwise, a pilot study is necessary to estimate the data needed to perform a priori power analysis and to determine the number of users needed for the test [31].

2.2.2. Tools and Equipment

The tools and equipment needed for the research depend on the type of exoskeleton evaluated and the context in which the study is performed. When selecting the tools and instruments to be used in an evaluation study of occupational exoskeletons, the choices depend on the type of exoskeleton being analyzed, e.g., supporting lifting and carrying tasks, high-frequency light load handling, etc. In the case of lift-assist exoskeletons, the instrumentation chosen must allow for the analysis of the relevant parameters of interest. For instance, in assembly tasks it may be useful to analyze the muscle activation of the upper limbs via electromyography, while in logistics material handling it could be appropriate to evaluate the muscle activation of the back erectors [17]. The technologies utilized should also be suitable for the context in which the study is conducted, e.g., in industrial settings or a laboratory environment. For example, the equipment requirements for exoskeletons designed to assist during manual lifting of light loads at high frequency differ from the requirements for exoskeletons supporting manual lifting of heavy loads and carrying. Hence, the study parameters and equipment must be carefully matched to the exoskeleton’s intended use case.

2.2.3. Questionnaires

The aim of creating questionnaires for users participating in experimental studies related to the investigation of the impact of occupational exoskeletons is to gather useful insights and feedback from the users’ perspective. These questionnaires facilitate the understanding of the user experience, perceptions, and potential issues or concerns associated with the exoskeleton technology being evaluated. The scope of these questionnaires can be divided into three main phases: before the study, during the study, and after the study. Questionnaires administered before the study support the assessment of the participants’ initial perceptions, expectations, and attitudes toward exoskeleton technology. This baseline information is essential to comprehend any preconceived notions or biases that users may have, which could potentially affect their experience during the study. Furthermore, it gives an understanding of the users’ familiarity with exoskeletons, their perceived usefulness, and any concerns or hesitations they may have. Questionnaires administered during the study can capture real-time feedback and experiences of the participants as they interact with the occupational exoskeleton. This information can help identify potential usability issues, discomfort, or challenges that users may face while performing tasks wearing the device. It can also provide insights into factors such as perceived effort, fatigue levels, and overall user experience during separate phases of the study. Post-test questionnaires allow participants to provide comprehensive feedback on their overall experience with the exoskeleton technology. Users can report their perceptions on aspects such as comfort, ease of use, perceived benefits, and any limitations or concerns they experienced. This feedback can be useful in identifying areas for improvement, refining the design of the task or workplace, and addressing potential user acceptance barriers.

Users’ perceptions collected through questionnaires provide a direct and complementary source of information on the subjective experiences, preferences, and concerns of the users that quantitative data alone may not fully capture. Researchers and designers, by incorporating users’ feedback throughout the study, can obtain a holistic perspective of the exoskeleton’s performance, usability, and potential impact on the user’s well-being and productivity. Furthermore, user feedback can inform future design iterations, ergonomic considerations, training protocols, and the overall acceptance and adoption of occupational exoskeletons in industrial settings [32].

2.3. Stage III: Experiment Execution

2.3.1. Data Collection

Data collection involves the use of tools capable of collecting, during the experiment execution, critical information for future analysis and showing the variation in critical parameters in real time. This activity allows us to obtain comprehensive and reliable insights into the performance, usability, and impact of the occupational exoskeletons being evaluated during the experimental study.

The primary aim of data collection is to gather both quantitative and qualitative data that support the investigation of specific research questions. In the context of an experimental study on occupational exoskeletons, these data form the basis for analyzing and understanding the effects of these devices on various aspects, such as biomechanical loading, metabolic cost, task performance, user experience, and overall effectiveness in an industrial setting. For example, biomechanical data may include joint kinematics such as angles and ranges of motion, joint kinetics, such as forces and moments, muscle activity, and postural analysis. Physiological data, such as oxygen consumption, heart rate, and skin temperature, support the investigation of metabolic costs, fatigue levels, and thermal comfort. To measure task performance, various data points can be collected and analyzed, such as task completion time, accuracy or quality of task execution, productivity and other output measures, and ergonomic risk factors, such as repetitive movements and awkward postures adopted during the task execution. Finally, the collection of user experience data completes the set of information to be collected to achieve a holistic perspective. These data include perceived exertion, measured through Borg scale ratings, comfort and usability ratings, subjective feedback, and user acceptance. In addition, environmental data complete the study framework, providing more information about the context in which the study is developed. These data include ambient temperature, humidity, lighting conditions, terrain or surface conditions, workspace dimensions, and operational layout. These data are collected across distinct phases of the study, including baseline measurements without the exoskeleton, during exoskeleton-assisted tasks, and potentially during recovery periods. Through this comprehensive approach, it is possible to compare and analyze the effects of the exoskeleton on different parameters.

Finally, the aim and scope of data collection during an experimental study on occupational exoskeletons must be aligned with the research objectives and hypotheses, guaranteeing complete comprehension of the exoskeleton’s performance, usability, and potential impact on the user’s well-being, productivity, and safety in the workplace.

2.3.2. Experimental Implementation

Each phase of the experimental study demands a prominent level of attention. Before the start of each test, a thorough calibration of all the equipment and instrumentation must be conducted to check their proper operational functionality. A rigorous execution checklist supports the collection of all the data required for the study and the proper execution of the testing procedures. If any unforeseen complications arise during the test execution, then the test must be stopped.

2.4. Stage IV: Data Analysis

2.4.1. Results Analysis and Interpretation

This stage aims to critically examine and interpret the findings obtained from the experimental study. Specifically, data analysis allows us to evaluate the significance of the results and identify any potential patterns or trends. The impact of occupational exoskeletons on workers can be assessed by employing relevant statistical methods, e.g., the Mann–Whitney U test and Spearman’s rank correlation coefficient [33,34], and techniques to assess variables, such as biomechanical loads, muscle activity, metabolic cost, and user experience. Then, the analyzed data are interpreted in the context of the study’s objectives and research questions to understand the observed effects of the exoskeletons on the user’s biomechanics, physiological responses, and overall performance. The findings from the study are related to existing models and previous research studies in the field of occupational exoskeletons and ergonomics.

2.4.2. Conclusions and Recommendations

The last step of the procedure in Figure 1 aims to discuss the findings and provide the conclusions of the research study. The practical implications and the potential benefits or limitations of the exoskeletons are discussed, based on the obtained results and findings in the context of the state-of-the-art scientific literature on workplace safety, productivity, and worker well-being. Furthermore, the study conclusions acknowledge and discuss the limitations or the assumptions behind the proposed study, which may impact the applicability of the findings. Based on the interpretation of the results, this stage provides recommendations for potential improvements or modifications to the workplace design to enhance the benefits of exoskeleton technology in industrial settings. Finally, the conclusions identify areas for future research and further investigations that can expand the current study’s findings or address its limitations.

3. Case Study: Evaluation of a Passive Back-Support Exoskeleton

The case study in this section proposes an experimental protocol for testing the impact of an occupational exoskeleton for back support during a manual handling task. The task consists of stacking twelve carboard boxes on four levels of an industrial pallet. More details about the study protocol and setup are provided in the following subsections.

3.1. Stage I: Pre-Experimental Planning

3.1.1. Target Definition and a List of Variables of Interest

This experimental study aims to analyze the impact of an occupational exoskeleton design to support workers during MMH. The investigated exoskeleton is the LAEVO V2, a passive wearable chest and back support device designed by Laevo B.V. (Eindhoven, The Netherlands) to protect against or prevent back injuries at work [35]. The testing protocol for the experimental study is designed to collect data on different parameters and human factors, including the pressure of the feet on the ground, to highlight any differences in the execution of the task; physiological parameters, such as pulse rate and blood oxygen saturation, to understand the fatigue during the task; the postures adopted during task execution to analyze any variation that the exoskeleton could induce; and, finally, the user perceptions about comfort and usability. These data are collected during the execution of the MMH task, both with and without the exoskeleton, to analyze the variations resulting from the use of the exoskeleton technology.

3.1.2. Preliminary Study

Two preliminary studies were conducted to investigate the impact of the exoskeleton [27,36]. These studies had two main objectives: (1) to provide an initial assessment of the analyzed parameters and evaluate their variation due to the exoskeleton technology; (2) to highlight any limitations and defects in the testing procedure. Specifically, the first preliminary study involved six workers performing two MMH tasks at two different companies: a chemical company and a textile company [27]. The MMH task at the chemical company involved lifting twenty bags, each weighing 25 kg, and moving them from a conveyor to a pallet (task A). The MMH task at the textile company involved moving sixteen horizontal rolls of fabric from a basket positioned on a forklift’s forks to a pallet on the ground, where the rolls were placed vertically (task B). During the handling process, the workers rotated each roll from a horizontal position on the forks to a vertical position on the pallet.

The second preliminary study involved six users performing a palletizing task in a laboratory environment [36]. Each user was required to handle twelve boxes, each weighing 15 kg, and place each box on a pallet to create four different deposit levels. Each subject performed the task twice, i.e., the first time without and the second time with exoskeleton support. The panel was divided into two equal groups: one group performed the task first without and after with exoskeleton aid, and the other group performed the task first with and after without exoskeleton aid. The data collected during each trial were the pressures of the feet on the ground, the heart rate, the blood oxygen saturation, and the user perceptions about comfort and usability. The findings of both the preliminary studies revealed distinct variations in the distribution patterns of the pressure of the feet on the ground and the heart rate when the tasks were performed with and without the exoskeleton [27,36].

3.1.3. Selection of Design Factors

The framework in Figure 3 shows the design factors of the experimental protocol for the investigation of the occupational exoskeleton. After explaining the purpose of the test and the correct execution methods to each user, the anthropometric data in Figure 4 will be collected according to the International Standard ISO 7250-1:2017 [37]. Each user will be required to fill out an introductory questionnaire (Questionnaire A, in Appendix A).

The detailed contents of the questionnaire are in Section 3.2.3. The user wears the monitoring equipment, with the support of a researcher. The monitoring equipment consists of a motion capture garment for posture assessment, a set of sensorized insoles for the evaluation of the pressure of the feet on the ground, and a sensorized wristband for measuring the physiological parameters. More details about the monitoring equipment are in Section 3.2.2.

Then, the user performs the MMH task and real-time video recording begins, as well as the registration of the pressure of the feet on the ground, the body postures, and physiological parameters including the pulse rate, the blood oxygenation, the Heart Rate Variability (HRV), the Electrodermal Activity (EDA), and the skin surface temperature. Each user performs the MMH task twice: once wearing the exoskeleton and once without it. The order of task execution (with or without the exoskeleton) is randomized. Therefore, half of the users perform the task first with the exoskeleton and then without, while the other half perform the tasks in the opposite order. This operational approach ensures that the order of task execution does not influence the parameters recorded during the test. After completing the task for the first time, the user is invited to fill out a second questionnaire (Questionnaire B, in Appendix B), reporting the immediate perceptions of discomfort and physical effort perceived during the test execution. The second task begins after a recovery period, during which physiological parameters are monitored to ensure they return to pre-test values. The phases of the task performed with the exoskeleton (left branch in Figure 3) differ from those when the task is performed without the exoskeleton (right branch in Figure 3). A brief training with the exoskeleton is performed before the test starts. This training aims to enable the user to become familiar with the exoskeleton technology by performing a set of simple movements, such as walking, bending the knees, leaning forward from the hips while pointing the hands towards the feet, and tilting forward obliquely while stretching the arms forward.

3.2. Stage II: Detailed Experimental Planning

3.2.1. Definition of the Sample

The study targets both male and female adults, including students and workers who wish to participate voluntarily. It is also aimed at workers who regularly perform MMH tasks in their workplace, as well as those who do not. Participants must be in good health and must not have current or past WMSDs or other disorders affecting their upper limbs, lower limbs, or spine. Anthropometric and demographic data and information are collected for each user. These data include age, height (L, Figure 4), weight, shoe number, shoulder width (A, Figure 4), wrist circumference (B, Figure 4), forearm circumference (C, Figure 4), forearm length (D, Figure 4), thigh circumference (E, Figure 4), arm circumference (F, Figure 4), chest circumference (G, Figure 4), hip width (H, Figure 4), and waist circumference (I, Figure 4). Anthropometric measurements are useful for evaluating the user’s overall muscle tone, with each measurement taken following the International Standard ISO 7250-1:2017 [37].

3.2.2. Tools and Equipment

The present protocol allows for the analysis of any passive exoskeleton for lumbar support. The task designed for the experimental study simulates a palletizing activity, involving the handling of twelve cardboard boxes (44 × 31.5 × 28.5 cm) each weighing 10 kg. The instruments chosen for data collection include a set of sensorized baropodometric insoles, a pulse oximeter, a sensorized wristband, and a motion capture garment. Specifically, the FlexInFit by Sensor Medica [38] insoles are adopted to monitor the pressure of the feet on the ground. These 0.3 mm thick insoles are worn inside the shoes and contain 214 resistive sensors, with a data acquisition frequency of up to 100 Hz and data transmission via Bluetooth. The insoles communicate with the freeStep biomechanical and postural analysis software by Sensor Medica [39]. A standard pulse oximeter is adopted to monitor the blood oxygen saturation throughout the test. The physiological parameters, such as the pulse rate, the HRV, the EDA, and skin surface temperature, are monitored with the EmbracePlus wristband by Empatica [40].

The motion capture garment for posture assessment is the second generation of the Pivot Yoga suit by Turingsense EU Lab [41]. This sensorized garment, consisting of a long-sleeved shirt and pants, provides full-body, real-time 3D joint kinematics. It detects any variations in postural position during the task execution. The garment communicates via Wi-Fi with the Gemma software 1.0., which manages communication and calibration processes, serves as a graphical user interface for visualizing the reconstructed manikin, and allows for the export of the complete set of recorded joint angles in a .csv file.

Figure 5 shows the technological setup of the study with the motion capture garment, sensorized baropodometric insoles, the sensorized wristband, and the passive back-support exoskeleton IX BACK AIR [42] worn by the user.

3.2.3. Questionnaires

Two separate questionnaires, i.e., Questionnaire A and Questionnaire B, were developed to collect user perceptions. Questionnaire A (see Appendix A) serves as an introductory pre-test questionnaire, to be completed before the beginning of the test. It includes initial questions to measure users’ familiarity with exoskeleton technology, their subjective opinions on these devices, and any prior experience using one. Users who have used an exoskeleton before are asked whether it was powered or passive, and which body part it supported (e.g., upper limbs, lower limbs, back). Additionally, Questionnaire A collects information about the participants, including their level of daily physical activity. For workers who regularly handle loads in their workplace, a set of questions investigates the average weight handled, the duration of the MMH activity, and the type of grip used. The primary objective of Questionnaire A is to better characterize the user sample. This allows for an assessment of how a person’s physical activity level or experience in MMH may influence the overall evaluation of an occupational exoskeleton, both in terms of measured parameter variations and perceptions.

Questionnaire B (see Appendix B) is designed to assess user perceptions after testing. It consists of 13 closed questions, where users rate their preferences on a scale from 1 to 5 for 11 questions. This questionnaire is divided into two parts: one to be completed after testing without the exoskeleton’s use, and another after testing with the exoskeleton’s use. The first part includes two questions: one assessing discomfort experienced in different body areas (e.g., neck, shoulders, upper limbs, back, legs, feet, and ankles), and another evaluating physical fatigue during specific task actions (e.g., maintaining an upright position, walking, bending the knees to lift a load, lifting a load without bending the knees). Users rate their discomfort and fatigue levels on a scale from 1 to 5. The second part, completed after testing with the exoskeleton’s use, repeats these two questions to assess discomfort and physical fatigue. This allows for a comparison between responses after testing with and without wearing the exoskeleton. Specifically, one question investigates the Locally Perceived Pressure (LPP) in various contact areas between the exoskeleton and the human body, formulated using an LPP scale as proposed in [43]. The LPP scale was adapted to a 5-point scale to align the scores with other questions and enhance user questionnaire completion ease. Four usability questions were constructed using a five-point Likert scale [44] to evaluate the ease of wearing and using the exoskeleton, its comfort level, and any potential movement limitations introduced by the device. Two additional questions about usage intentions were formulated using a five-point Likert scale to collect information on how frequently users would use the exoskeleton and consider its implementation as an enhancement in their workplace. Lastly, users are asked about changes in their opinion regarding exoskeleton use and whether they perceive the exoskeleton as beneficial for their work activity.

3.3. Stage III: Experimental Execution

3.3.1. Data Collection

The research study based on the proposed protocol will be conducted at the Department of Engineering “Enzo Ferrari” (DIEF) of the University of Modena and Reggio Emilia. The University Ethics Committee has approved the study protocol “Comitato Etico di Ateneo per la Ricerca” (CEAR), Protocol ID 101861. This study will be conducted in compliance with the ethical principles established in the “World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subject” (Helsinki Declaration) [45] and in the “Convention on Human Rights and Biomedicine” (Oviedo Convention) [46]. Therefore, at the start of each test, an informed consent form will be obtained, ensuring each user’s anonymity of their collected data and the option to withdraw from the test at any time.

3.3.2. Experimental Implementation

The experimental implementation follows the sequence of phases outlined in Section 3.1.3. The initial phase involves guiding users through each step of the test, as shown in Figure 3, to ensure the correct execution of the study. The palletizing task involves the MMH of 12 cardboard boxes, each weighing 10 kg. Each subject performs the MMH task twice during the test: once without wearing the exoskeleton and once wearing the exoskeleton. At the beginning of the MMH task, the boxes are positioned on a pallet, at a consistent height of 14.4 cm from the ground (Figure 6, right). The point of the deposit is a second pallet (Figure 6, left) located two meters from the picking pallet. The MMH task includes (1) lifting the box, (2) transporting it to the deposit pallet, (3) placing the box on the pallet, and (4) returning it to the picking pallet. The distance between the pallets ensures the user performs at least three steps, which is the minimum distance required to collect data on the pressure of the feet on the ground with the freeStep software [39].

While the user picks the first six boxes on the front of the right pallet (boxes from 1 to 6 in Figure 6), a researcher moves the remaining six boxes to the front line to maintain a constant transport distance of two meters throughout the test (Figure 6). Finally, the user picks up each box and transports it to the deposit pallet to obtain four levels (L1, L2, L3, L4), following the sequence and pattern illustrated in Figure 7.

This involves varying the deposit height from L1 to L2, as shown in Figure 7. Specifically, the hands of the user will be 14.4 cm from the ground at L1, 42.9 cm at L2, 71.4 cm at L3, and 99.9 cm at L4. To ensure uniformity in the picking distance (distance between hand and foot position), a line is marked on the ground where the user must place the tips of the feet before lifting and depositing the boxes on the pallets.

3.4. Stage IV: Experimental Analysis

Results Analysis and Interpretation

The following section presents the analysis and the interpretation of the results obtained from the preliminary studies conducted, which justify the need to include the design factors and variables in the proposed protocol. Specifically, the first preliminary study [27] investigated the pressure of the feet on the ground and its distribution patterns during two different MMH tasks, i.e., task A and task B. Both tasks were performed in an industrial setting, as described in Section 3.1.2. Figure 8, Figure 9 and Figure 10 show the variations in the maximum values of the pressure of the feet on the ground during the study (Figure 8), the variation in the pressures of the forefoot (Figure 9), and the variation in the pressures of the rearfoot (Figure 10), recorded during the task execution. Positive values for such variations indicate that the recorded value during the test with the exoskeleton was higher than the value without the exoskeleton. Specifically, Figure 8 shows the variations in the maximum values of the pressure of the feet on the ground for task A (Figure 8a) and task B (Figure 8b). The maximum average variation in the left foot during task A was 39%, while for the right foot it was 18% (Figure 8a). For task B, the maximum average variation in the left foot was −38%, and for the right foot it was 14% (Figure 8b).

The maximum variation in the pressure of the left forefoot during task A was 10% and the maximum variation in the pressure of the right forefoot was 14% (Figure 9a). The maximum variation in the pressure of the left forefoot during task B was −20%, and the maximum variation in the pressure of the right forefoot was 11% (Figure 9b). Similarly, the maximum variation in the pressure of the left rearfoot during task A was +/−19%, and the maximum variation in the pressure of the right rearfoot was −22% (Figure 10a). Figure 10b shows that the maximum variation in the pressure of the left forefoot during task B was 45%, and the maximum variation in the pressure of the right rearfoot was −43%. In the second preliminary study, the maximum and average variations in the pressure exerted by the feet on the ground, the variation in the user physiological parameters, and user feedback on comfort and usability were analyzed during a palletizing task set in a laboratory environment. The task consisted of handling twelve cardboard boxes from a pickup location to a deposit pallet [36]. Figure 11 shows the percentage variations in the distributions of the maximum and average pressure of the feet when the deposit level varies. Still, positive values for such variations indicate that the recorded value during the test with the exoskeleton was higher than the value without the exoskeleton. Figure 11a reveals that the maximum pressure variation recorded for the left foot was 4.54%, while for the right foot it was −4.63%. Figure 11b shows that the maximum variation in mean pressure during the tests was 2.98% for the left foot and −4.64% for the right foot. The physiological parameters recorded were pulse rate and blood oxygen saturation. There were no significant variations in oxygen saturation, while changes in the pulse rate compared to resting conditions are shown in Figure 12. The maximum percentage variation in pulse rate during the test with the exoskeleton was 69%, while for the test without the exoskeleton it was 50%. The lowest percentage change was 4% for tests without the exoskeleton and 0% for tests with the exoskeleton. Finally, the questionnaires revealed that only two out of four users had previously used an exoskeleton. Users found the exoskeleton to be comfortable and easy to use, despite some limitations on the movements needed to perform the MMH task.

4. Discussion

The protocol proposed in this paper supports the evaluation of a passive occupational exoskeleton for lumbar support, particularly during MMH at work. It collects parameters such as variations in the distribution of the pressure of the feet on the ground; physiological parameters like blood oxygen saturation, pulse rate, HRV, EDA, skin surface temperature, and postural variations; and perceptions of comfort and usability. A critical aspect of the protocol development is the execution of preliminary studies [47]. These studies test each component of the protocol, identify weaknesses, and correct them before starting the pilot or main study. The significance of preliminary studies is fundamental across all research fields and is emphasized by the US National Institutes of Health for its value in the research process [48]. Experiments evaluating exoskeletons are time-consuming due to the several factors that research studies consider. Such studies commonly involve repeated measures with and without the exoskeletons, including training periods for task execution and proper exoskeleton use [16]. Additionally, each instrument requires proper calibration time before use. For these reasons, conducting preliminary tests is even more crucial, as it prevents further time loss later. Also, in the data analysis phase, having an analysis mode already tested in preliminary studies ensures one can save time and resources in the final evaluation. The findings of the first preliminary study suggest that the exoskeleton produced some impact on the maximum pressure experienced on the feet of the workers (see Figure 8). However, these data do not explain how the forces were distributed during the tasks, and more research is necessary to investigate the causes of these variations. Moreover, the results in Figure 9 and Figure 10 suggest that the use of the exoskeletons during manual lifting produced some impact on the pressure distribution on the distinct parts of workers’ feet. Again, it is not possible to identify a cause of these variations; for better understanding of its nature, a more in-depth and extended analysis of a wider sample of users, together with a more rigorous and standardized definition of the picking and deposit procedure, are necessary steps [27].

The findings of the second preliminary study show differences between the pressures of the right and left foot on the ground (see Figure 11). The left foot presents higher pressure variation values on average for three out of four deposition levels regarding maximum pressure (Figure 11a) and across all four levels concerning average pressure (Figure 11b). As observed in the previous preliminary study, to better understand the pressure variations between the test with the exoskeleton and the test without the exoskeleton, and to determine whether the use of the device causes these variations, more extensive and standardized tests are necessary. These tests should involve lifting and storing loads with feet kept parallel and following the same picking and depositing sequence to ensure a consistent number of steps between tests.

Also, the findings from the analysis of the physiological parameters revealed sharp variations in the values obtained from the tests with the exoskeleton and the tests without the exoskeleton. Figure 12 suggests that the exoskeleton might increase the pulse rate during use. Specifically, three users involved in the second preliminary study experienced significant increases in pulse rate with the exoskeleton. These users performed the first test without the exoskeleton and the second test with the support of the device. This order of test execution could produce an impact on the pulse rate, despite the planned recovery time between the two tests. Another potential cause is the handling rate of the activity. Users might have increased their handling rate due to the support provided by the exoskeleton, resulting in a higher pulse rate. This finding highlights the necessity of including proper recovery time between tests in a protocol for evaluating occupational exoskeletons, particularly when users perform the test twice. Additionally, the results from the preliminary studies confirm the importance of video recording the activity, including the duration, to better understand variations in physiological parameters. The order of test execution emerged as a significant factor for analysis, requiring further evaluation. Finally, the use of questionnaires in the preliminary studies allowed for the assessment of their construction in terms of user comprehensibility. Key points for constructing an effective questionnaire include the complexity of the questions and the overall length.

Finally, the design of a testing protocol for the evaluation of occupational exoskeletons involves various phases and multiple variables. The proposed model can be adapted for laboratory testing of any occupational exoskeleton while maintaining the general structure and adjusting the task to suit the specific exoskeleton being evaluated. The data collection and analysis procedures remain consistent, with only the instrumentation varying.

5. Conclusions

This paper has proposed a practical protocol for researchers, ergonomists, and industrial practitioners to evaluate the impact of occupational exoskeletons. It can be modified according to the specific type of exoskeleton being assessed or the chosen variables of interest. The proposed protocol is divided into four stages: pre-experimental planning, detailed experimental planning, experimental execution, and data analysis. It allows for the immediate identification of the most relevant variables and critical design factors for the protocol, thereby avoiding unnecessary time wastage in subsequent stages. The preliminary studies supporting the proposed protocol highlighted several key factors for the evaluation of the occupational exoskeleton and for the proper construction of the protocol: recording execution times of activities, providing clear instructions for the initial task, and planning a demonstration phase. Furthermore, the standardization of the task ensures consistency among users in test execution, thus preventing the distortion of data and results. Finally, the proposed protocol allows us to achieve more significant results than previous studies because it is focused on understanding the correct execution of the test by users to ensure maximum repeatability and standardization in the evaluation procedure. Future development of this work involves applying the presented protocol to evaluate and compare different occupational exoskeletons supporting material manual handling in industrial settings and a pilot study to determine the dimension of the sample of users to be involved.