User Evaluation of the MOBOT Rollator Type Robotic Mobility Assistive Device ^†

Abstract

In this paper, we report on the evaluation strategy and the results that were obtained from the final end-user evaluation process of an innovative robotic assistive device supporting mobility. More specifically, the paper deals with the evaluation of the MOBOT robotic rollator as regards to the system’s overall performance and its individual assistive characteristics and functionalities, as implemented in respect to (i) the provided cognitive assistance, and (ii) the adopted audio-gestural human-robot communication model. User evaluation was designed on the basis of an extensive survey of scales and methodologies widely reported in the relevant literature. The actual evaluation phase exploited the QUEST 2.0, ATDPA-Device Form, and PYTHEIA scales to measure the subjective satisfaction of the users. The PYTHEIA scale, in particular, was structured in order to fill the gaps that were identified during the study of previously existing tools for measuring assistive device user satisfaction. The scale was applied for the first time during the reported evaluation process. An analysis of the results showed that MOBOT was ranked very high by end users in all of the aspects addressed by the three employed assessment scales, thus providing significant evidence for positive acceptance of any industrialized outcome of the current prototype in the assistive robots market.

1. Introduction

Autonomy is an essential part of independent daily living. Various assistive devices exist that aim to help individuals with physical disabilities in their day-to-day activities. A constant concern has been to make these devices both easily accessible and safe for persons with reduced upper and/or lower limb strength. Moreover, the ageing of the population [1,2] makes the need for such devices an acute issue for the active living of a big proportion of the society, namely the elderly. According to the latest world report on disability [3], it is estimated that more than a billion people struggle with some of its forms. Nearly 200 million of these people experience considerable functional difficulties. This increase is due to the ageing of the population as accompanied by a global rise in chronic health conditions. Furthermore, following the development prospects of the UN 2015 Millennium Development Goals [4] and beyond, people with disabilities must be empowered in order to be able to participate in their communities, receive the required education, find work, and be socialized.

Assistive Technologies (AT) are defined as “any item, piece of equipment, or product, whether it is acquired commercially, modified, or customized, that is used to increase, maintain, or improve the functional capabilities of individuals with disabilities” [5]. The primary goal of AT is to maintain or improve an individual’s functioning and independence so as to facilitate participation and enhance overall well-being. Nowadays, there are a variety of assistive devices and technologies that are available (e.g., wheelchairs, hearing or visual aids, mobility aids). The benefits from AT are numerous and target different categories of needs and provided assistance. We indicatively mention fall prevention, mobility and independence, social connectivity, cognitive support, etc. [6,7,8,9,10], while safety has been found to be among the most important reasons for the use of AT among older adults [11]. Today research in various sectors of robotics focuses on integrating such technologies.

In this context, robotic applications have rapidly expanded from classic industrial applications performing repetitive tasks to applications involving close and even hands-on human-robot interaction (HRI). In many cases, there is a strong physical interaction with the human body. To this end, it is crucial for researchers during the development phase to keep in mind the human physiology and other biomechanical parameters that are pertinent to the human body. In parallel, the continuous progress in Information and Communications Technologies (ICTs) has led to significant innovations in human-computer and HRI, thus providing new and more intuitive ways of multimodal interaction with AT. The use of high-tech sensors (e.g., lasers, ultrasound) and new generation image/motion capturing devices (e.g., Kinect and GoPro) has raised the challenge of making upcoming assistive devices that are able to “sense” humans and their environment. This fact had made it possible for such devices to assist users more efficiently in various ways, such as finding the shortest path to the target.

In this paper, we report on the methodology and the results of evaluating the MOBOT rollator—a complex, innovative robotic rollator-type assistive device—with end users, at its final development phase. The FP7 MOBOT project (www.mobot-project.eu/) aimed at supporting mobility and thus enforcing fitness and vitality by developing intelligent active mobility assistance robots for indoor environments that provide user-centered, context-adaptive and natural support. These robots should, therefore, function proactively by performing autonomous and context-specific monitoring of human activities and by subsequently reasoning on meaningful user behavioral patterns. To this end, they needed to be able to analyze multi-sensory and physiological signals that are related to gait and postural stability, and perform adaptive compliance control for optimal physical support. Thus, a multimodal action recognition system was developed to monitor, analyze, and predict user actions. Computer vision techniques with modalities, such as range sensor images, haptic information, as well as command-level speech and gesture recognition were also implemented. A multimodal human-robot communication system, involving both verbal and non-verbal communication was synthesized into mobility assistance models taking into consideration critical safety requirements. MOBOT was designed as an intelligent rollator-type mobility assistant providing help to individuals who experience problems in stabilizing their body while walking and who also require support in standing up and sitting down.

For the evaluation of the MOBOT rollator different end-user groups participated in the trials that were conducted during the project’s lifecycle to assess and benchmark the overall system [12,13]. This paper presents the methodology followed and the results achieved from the rollator’s final evaluation study.

2. Overview of Assessment Instruments

Although much research has been invested in the field of assistive robotics resulting in several prototypes, few devices have reached the market. Among the factors that prevent this type of AT from being in common use are high production costs, approval by each country’s responsible legal body, and uncertainty as regards to the benefits that are gained from their adoption. Measuring user satisfaction helps to measure and improve the overall quality of products and/or services. Therefore, similar to any other technological device, assistive robots need to undergo thorough subjective assessment as well as objective evaluation procedures.

Subjective assessment yields many possible answers of varying quality and requires professional judgment to give a score [14]. Subjective assessment records the facts that are provided by end users and demonstrate their perception, understanding, and interpretation of what is happening during their interaction with a device, therefore also measuring the degree of their satisfaction. This kind of information is valuable to researchers and developers, since it enables them to adapt product development to actual user needs.

There are reported attempts to categorize both objective and subjective metrics. According to the USUS Evaluation Framework for Human-Robot Interaction [15], usability, social acceptance, user experience, and societal impact are considered as the main categories of evaluation factors, each of them being divided into specific metrics, either objectively or subjectively measured as listed below:

• Usability, comprising effectiveness, efficiency, learnability, flexibility, robustness, and utility.
• Social Acceptance, comprising performance expectancy, effort expectancy, attitude towards using technology, self-efficacy, forms of grouping, attachment, and reciprocity.
• User Experience, comprising embodiment, emotion, human-oriented perception, feeling of security, and co-experience with robots.
• Societal Impact, comprising all effects of the introduction of robotic agents’ consequences for the social life of a specific community (taking into account cultural differences) in terms of quality of life, working conditions and employment, and education.

The authors of [15] suggest that a subset of the above metrics may be investigated by means of end-user questionnaires, thus considered subjective. Their proposed list includes:

• Utility. It refers to how an interface can be used to reach a certain goal or to perform a particular task. The more the tasks that the interface is designed to perform, the higher its utility.
• Performance Expectancy. It is the degree to which an individual believes that using the system will help him or her to attain gains in performance.
• Effort Expectancy. It indicates the extent to which the user perceives that a system will be easy to use.
• Attitude Towards Using Technology. It is the sum of all positive or negative feelings and attitudes about solving working tasks supported by a humanoid robot.
• Self-Efficacy. It relates to a person’s perception of his or her ability to reach a goal.
• Attachment. It is an affection-tie that one person forms between himself or herself and another person or object—a tie that binds them together in space and endures over time.
• Reciprocity. It is the positive or negative response of the individual towards the actions of others.
• Embodiment. It describes the relationship between a system and its environment and can be measured by investigating the different perturbatory channels, such as morphology, which has impact on social expectations.
• Emotion. An emotion is an essential part of social interaction; it has to be incorporated in the design and assessment of robots.
• Feeling of Security. It is important to investigate how to design HRIs in a way that the individual experiences them as safe.
• Co-experience. Co-experience describes experiences with objects regarding how the individual develops his or her personal experience based on social interaction with others.
• Societal Impact. Societal impact describes all effects in which the introduction of robotic agents results or the social life of a specific community—taking into account cultural differences—in terms of quality of life, working conditions, employment, and education.

When preparing the MOBOT rollator end user evaluation, our intention was to identify those reliable and valid instruments, which are developed for the assessment of assistive or rehabilitation robot devices from the user perspective, which could be employed for our purposes. To serve this goal, we first conducted a study on existing scales, as reported in the literature, to be utilised for the assessment of MOBOT.

Astonishingly enough, we discovered that, although the subjective assessment of any AT item is a significant issue towards defining its usability, there is still no standard methodology or instrument to cover all of the aspects that a thorough end-user evaluation process should incorporate. In [16], the reader may find a systematic review of the literature for studies that assessed rehabilitation or assistive robotic devices from 1980 through January 2016 exploiting PubMed, MEDLINE, CIRRIE, and Scopus databases to retrieve relevant material.

A significant fact reported in [16] is that custom-made questionnaires are found to be the most commonly used tool type (66.7%) that are applied for most of the measures used during the development phase of any rehabilitation and assistive robot device. At the same time, the astonishing majority of the reviewed studies (reaching a percentage of 93.9%) used non-validated or reliable instruments to assess the subjective opinion of their intended target group. A deficit can, therefore, be identified in respect to subjective assessment of assistive devices, which, in turn, has a direct impact on comparing results among different studies and research approaches, making a comparison actually not feasible due to the lack of common standards. The absence of standard scales to be widely used reveals further the need for valid, reliable instruments to become available to end users for the subjective assessment of assistive robotic devices.

Parallel to custom-made subjective measures to evaluate end-user satisfaction and assess the rehabilitation or assistive robot device [16], most reported studies recruited very limited numbers of patient subjects, while semi-structured interviews, as those utilized in [17,18], were not considered as an option for the evaluation of the MOBOT rollator. The Canadian Occupational Performance Measure (COPM) [19] and the System Usability Scale (SUS) [20] were also excluded, since the former is only remotely related to the kind of assessment that we intended and the latter measures only very basic issues of the device. The Psychosocial Impact of Assistive Devices Scale (PIADS) [21] is another scale that is present in the bibliography, which, however, emphasizes the psychosocial impact of assistive devices. Thus, it does not target the evaluation of the actual experience of interacting with a robot assistive device, but rather the impact that this interaction has on quality of life.

On the other hand, two scales, the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0) [22] and the Assistive Technology Device Predisposition Assessment-Device Form (ATDPA) [23] proved to have been validated in general AT devices. Both of them were found to be capable of supporting the scope of subjective assessment in respect to rehabilitation and/or assistive robots, although they demonstrate limited coverage as regards to the evaluated aspects that they take into account. The ATDPA assesses the overall user experience with AT, while QUEST 2.0 is a tool that can be used to measure satisfaction from the use of AT. In order for both scales to be used with Greek speaking elderly subjects, a formal translation and validation process preceded their use in the framework of the MOBOT rollator evaluation, as reported in [24,25].

The identified limitations of the aforementioned instruments regarding the full coverage of factors that assess the subjective satisfaction of users with AT could be covered by the introduction of a different tool, which would fill in the spotted gaps. To this end, a new scale, called PYTHEIA, was proposed [26].

The PYTHEIA scale is a self-reported measure for the assessment of rehabilitation and assistive robotics, as well as other AT devices. Its development aimed to tackle the absence of a valid instrument to evaluate assistive robotic devices, both as a whole, and by addressing any of their individual components or functionalities. PYTHEIA was developed in Greek in order to assess the satisfaction of users with any assistive technology device (e.g., robotic, rehabilitation device) and has been extensively tested for its reliability and validity [27]. In the context of the MOBOT rollator, field test studies were conducted with 147 subjects (inpatients and outpatients of a rehabilitation hospital) who were administered the original questionnaire. The scale is applicable in patients with different diseases, ages, and disabilities who use various assistive devices. According to the decided inclusion criteria, all of the participating subjects scored above 17 in the Mini Mental State Examination (MMSE). Intraclass correlation coefficient (ICC = 0.992), Pearson’s correlation (r = 0.984), and Cronbach’s α (α = 0.793) indicated sufficient reliability measures, while test-retest outcome also showed great stability. The paired samples t-test between initial assessment and reassessment indicated no statistically significant differences (p value = 0.059). Finally, various types of validity were also investigated.

According to the acquired results, PYTHEIA proved a stable, valid, and reliable instrument. Thus, it can be used to measure the satisfaction of patients with any assistive device. PYTHEIA’s uniqueness is that it can assess not only general user satisfaction with any assistive device, but also to independently evaluate any individual feature and functionality that the device may have. To this end, it can be used for evaluating also new and experimental developments (e.g., robotic assistive devices) in laboratory environment. Specifically, it can help developers to assess their products in terms of efficiency, comfort, quality, performance, ergonomics, and usefulness as perceived by end users.

3. Methodology

The MOBOT rollator final evaluation study took place in DIAPLASIS rehabilitation hospital in Kalamata, Greece. During this final system evaluation, an important objective was to gain information from end-user experience and assessment regarding the implemented cognitive assistance and the audio-gestural modules/functionalities of the MOBOT rollator type device (Figure 1), as well as its overall performance under real use conditions. A multi-disciplinary team led the evaluation process.

To begin with, two different methodologies were followed to cater for both types of evaluation. As far as the objective evaluation of the MOBOT platform is concerned, various data from the multiple sensors integrated in the MOBOT rollator were collected and processed following the actual evaluation.

Regarding the subjective platform evaluation, an innovative approach was adopted in order to overcome the weaknesses of other known metrics, as indicated above (2). This section may be divided by subheadings. It should provide a concise and precise description of the experimental results, their interpretation as well as the experimental conclusions that can be drawn.

For the purposes of our evaluation study, we decided to make use of the three most reliable and valid questionnaires, namely the QUEST 2.0, the ATDPA—Device Form, and the PYTHEIA questionnaires, the latter was enriched with a series of interview-based questions.

The procedure followed for the evaluation study at DIAPLASIS was approved by the Bioethics and Deontology Committee of the Technological Educational Institute of Athens (approval code: 2832/09-05-2014).

3.1. Subject Recruitment

A long lasting recruitment process took place in order to identify the right subjects that would fit the MOBOT evaluation strategy. Participation was set on a volunteer basis with no other incentives provided. A parameter to be considered was that, due to the health condition of the target group members, there was a significant risk of dropouts because of fatigue when performing the requested tasks.

In order to achieve the desired number of participants, an extended preparatory period of screening and review of the potential subjects via studying their medical files preceded the evaluation phase. In order to eliminate the danger of ending up with less subjects than required for the collection of enough data that would provide significant measurements, both inpatient and outpatient files of the rehabilitation hospital were examined. What is more, other settings (i.e., collaborating geriatric centers) were contacted for further recruiting of possible candidate subjects. The clinical team interviewed all of the selected patients following a screening process of patient files.

The ΜΟΒΟΤ mobility assistant finally underwent a thorough user evaluation process, with thirty (30) subjects being engaged in various rehabilitation programs at DIAPLASIS, which was the sole clinical partner in the reported project. This sample consisted of all the individuals who met the set participation criteria in this restricted framework. The evaluation setup had previously been checked in a pilot evaluation by eight subjects that were different from those who eventually participated. Two main scenarios were examined during the evaluation phase: (i) cognitive assistance, and (ii) audio-gestural HRI.

As far as the inclusion criteria are concerned, the subjects had to:

• be using an assistive device (rollator type) for at least a month;
• have moderate motor impairment (habitual use of rollator, gait speed <0.6 m/s unassisted);
• have moderate impairment [unable to stand up and sit down unassisted on a normal chair (standardized 100% leg length) without problem, 5-chair stand >16.7 seconds or able to stand up from a chair from elevated chair (120% lower leg length)];
• be at least 65 years old; and,
• have no cognitive impairment (Mini–Mental State Examination (MMSE) ≥26–30) or mild to moderate cognitive impairment (MMSE 17–25).

All of the participants who met the above clinical criteria were diagnosed as able to perform the required tests/assessments and signed an informed consent.

3.2. Scenarios

Two main scenarios were tested during the evaluation of the MOBOT platform. The first scenario assessed the cognitive assistance that was provided, while the second one tested the audio-gestural human-robot communication that was implemented on the device.

The objective of the cognitive assistance scenario was to assess user acceptance of a functionality integrated to the MOBOT platform, which provides direction assistance in the form of informative messages, while following a specific trail. MOBOT provided audio assistance to end users at pre-specified “critical” waypoints. Audio cues were repeated as many times as necessary until the user followed the desired direction in order to reach the end point that was specified at the beginning of the trial. Repetitions aimed to provide the necessary cognitive support to end users suffering from light memory loss while walking with the MOBOT rollator in unfamiliar spaces.

Pre-mapping of the space and the used navigation trail (Figure 2), as well as localization of the MOBOT rollator-type device, were required prior initiating the experiments. According to the research hypothesis, the cognitive assistance functionality would assist subjects from Group A (i.e., cognitive impairment group) to navigate in unfamiliar spaces and achieve comparable results as compared to Group B (i.e., cognitive intact group). As reported in [28], the repetition of audio cues may prove stressful for subjects. Indeed, continuous repetition of localization information by the rollator was annoying for some users with no mental impairment. On the other hand, users with various degrees of mental disorders exhibited full satisfaction of the support that they received and the information repetition tempo. This suggests that this feature should allow for a degree of adjustability according to specific user needs.

However, from the part of user evaluation, it was most important to measure user acceptance and preferences in respect to the proposed interaction model, since the aim has been that end users be able to interact with the MOBOT device in a more intuitive, natural way, using oral commands and gestures that are commonly used in everyday life situations [29] and carry unambiguous semantic value. To this end, the adopted human-robot communication model was built as a structured tree of possible multimodal action-reaction exchanges engaging both audio and gestural signals.

Prior to starting the experimental sessions, the session supervisor instructed each participant on the process protocol and on the way that he or she was expected to communicate with the rollator. The supervisor presented to each participant a set of five selected phrases that were to be pronounced by him/her. Each phrase was connected with one specific gestural signal. The reason why it was decided to use five phrase commands was that they corresponded to the basic functionalities of the platform as described in the following section. More specifically, these were: 1. Come near me! 2. I want to stand up. 3. I need help. 4. Where am I? and 5. Go park! Participants had to use oral and gestural communication signals both separately and in combination in order to communicate with the rollator. Successful communication input on the part of human sources triggered an appropriate reaction on the part of the robotic device. All of the commands used in the three communication variations (audio mode, gestural mode, and audio-gestural mode) were tested in the same order of performance by all of the participants in the evaluation study. The feedback of the rollator in the form of oral signals verified the system’s understanding of the command it received each time. In cases of failure, the rollator provided either the wrong feedback or no feedback at all.

Both audio and gestural commands were performed in the same setting. For the gestural communication, participants moved their hands to perform the set command signals (Figure 3), while the system’s responses were provided in the form of oral utterances.

The results of the adopted assessment methodology are presented in the next section.

4. Platform Functionalities and HRI Model

The MOBOT rollator is an assistive device comprising the main frame, the actuated handles, active wheels, user interface, an electronic control unit, and a number of environment and user sensors. The device’s distinct functionalities [30] meet both mechanical design and user requirements, and they are grouped in the following seven categories, each of which demonstrates the ability of the device to:

(1)

Perceive the user and adapt itself to the user. This includes: localization of the user with respect to the rollator exploiting 3D coordinates and the distance between user and device (“distant”, “close” and “in contact” variables); tracking of the articulated human body; detection of walking patterns; recognition of user gestures, actions, plans, and intentions; recognition and interpretation of the user voice commands; monitoring human performance and postural stability as well as detecting unstable configuration and falls; and, recognition of the human physiological state.

(2)

Detect the environment, including being able to detect obstacles, locomotion-specific data (e.g., surface type/slip), environment-specific data (e.g., slopes), and creating a map of the environment.

(3)

Localize itself within the environment map.

(4)

Approach the user from a distance.

(5)

Assist the user. This is achieved by providing physical assistance during sit-to-stand and stand-to-sit transfers; assisting the user while walking in three ways, which include following user intention (“dock” to the user, accelerate, maneuver, decelerate, stop), balancing and stabilizing the user (fall prevention included), and assisting the user while passing through narrow passages and opening/closing doors; assisting the user while standing (in proximity but in no contact mode); following him/her; providing sensorial assistance such as help to avoid static and dynamic positive and negative obstacles, or assistance on slopes; providing cognitive assistance; and, assisting user localization or guiding/navigating him or her.

(6)

Leave the user and go to parking position in autonomous mode.

(7)

Perform autonomous charging.

The features listed above are supported by a series of technologies that were researched in the framework of MOBOT project and have set the framework for the development of the platform’s multimodal communication model. The technologies integrated in the platform include visual action recognition in continuous RGB-D video streams, object detection, human body pose estimation, action/gesture recognition, spoken language recognition, multimodal sensor fusion for audio-visual gesture recognition, and processing of haptic and physiological data [30].

The synergies among the MOBOT modules have been enabled through the integration of related technologies, so that the platform provides walking and cognitive assistance to elderly users with slight walking and cognitive problems. In this respect, the compass towards implementing the capabilities of the platform has been a well-articulated HRI communication model that supports the most natural communication possible between the platform and its user. In doing this, the model takes into account both the state of the integrated technologies, and the ways that the target user group communicate in their everyday activities.

The multimodal dataset [31], acquired at the early stages of the project, and the respective annotation of verbal and gestural communicative signals was designed with the goal of defining a model of multimodal (HRI) as close to natural human behavior as possible. The database was acquired by means of a sensorized passive rollator comprising multimodal input from various sensors, microphones, and cameras, together with a motion capture system that was used to record human limb movements as well as the rollator and subject’s absolute positions in space. The MOBOT corpus is one of the very few and relatively rich available multimodal-multisensorial resources of activities performed by humans in interaction with a device.

The HR communication model was built as a structured tree of possible multimodal action-reaction exchanges, engaging both audio and gestural signals, enriched with a number of cognitive assistance assertions from the part of the platform, which assimilate human reinforcement to elderly individuals while performing a trivial task.

The platform may interact in three modes: (i) hands-on mode, (ii) following mode, and (iii) stand-by mode, while user input focuses on developing a multimodal dialogue strategy. This strategy takes into account the options of (a) communication via body posture in silence and thus complete absence of any other speech or gesture signal, processing information that can be linked with the platform’s action recognition module, and (b) communication via speech and/or gesture signals, processing information that can be linked with the audio and gestural signal recognition.

The system’s ability to learn how to navigate using a map is linked with a number of cognitive support messages, which are sent to the user in the form of oral questions or reinforcement messages, similar to those that are received from the part of human carers during natural human interaction.

Question type messages are linked with the activation and approach of the robotic device, decision making in respect to the route selected each time or obstacle avoidance situations, and demand some verification from the part of the user. Reinforcement type messages encourage users to complete a task successfully.

The scenarios of use, for which action trees are constructed, are extracted from real user needs as defined during the preparation of acquisition of the MOBOT dataset and are depicted in the scenarios that are actually used in creating the MOBOT corpus, and also through end-user evaluation of the intermediate stages of the platform’s prototypes. Furthermore, the underlying notion behind the part of data acquisition that entails the closed set of combinations of the MOBOT audio-gestural commands is that any system attempting to model human actions in terms of interactional behavior needs to have access to knowledge on the structures of human actions. More specifically, assistive systems dealing with human-machine interaction must be able to decompose human behavior activities into measurable and machine detectable features, so as to be able to make decisions and plan support actions on the basis of heterogeneous sensory data [32,33].

In the related recordings, elderly human individuals of varying age, gender, and motor and cognitive abilities performed a variety of assistance requiring tasks in dialogues involving human carers and a passive rollator. The recorded scenarios were based on a use case list, which included actions that a typical end user may need to perform in real life situations. This case list was adapted to the recording environment.

5. Conclusions

In order to assess the results of the evaluation study that took place at DIAPLASIS, three different questionnaires were employed, as already reported in the previous sections. Each participant, after completing the evaluation scenarios, was asked to respond to the Greek version of the QUEST 2.0 and ATD PA questionnaires, as well as to the PYTHEIA questionnaire. In the case of PYTHEIA, participants evaluated the MOBOT system as a whole based on their hands-on experience in Part A, while in Part B they assessed one by one each of the platform’s features. These features related to (i) the cognitive assistance module (navigation assistance) and (ii) the audio-gestural interaction module.

The cognitive assistance and the audio-gestural interaction functionalities were assessed by a total population of 30 subjects who participated in the entire evaluation study and completed the experiments successfully. The participants suffered from different pathologies (indicatively we mention multiple sclerosis, myopathies, and Parkinson’s disease), were of different ages and exhibited diverse health conditions. The mean participant age was 78.6 years old while 56.7% of the subjects were female. Notably, three subjects failed to complete the cognitive assistance experiment.

According to the obtained results, the subjective evaluation of the MOBOT platform by means of the aforementioned three questionnaires showed that users were in general very satisfied with the implemented functionalities on the MOBOT rollator, while the human-robot communication model was found satisfactory and helpful (Figure 4). More specifically, subjects rated the MOBOT with mean = 3.91 in the PYTHEIA scale, mean = 3.82 in ATDPA, and mean = 3.71 in the QUEST 2.0 scale. The highest score option for all three of the Likert scales used in all of the questionnaires was 5 (i.e., PYTHEIA-5 = Extremely satisfied, ATDPA-5 = Satisfied all the time, QUEST-5 = Very satisfied). When responding to Part B of the PYTHEIA questionnaire, subjects assessed the cognitive assistance functionality with 4.46 and the audio-gestural interaction functionality with 4.30, indicating their high level of satisfaction from the experience of using these features. More specifically, the three most important features of the MOBOT platform were ranked as follows in respect to the parameters of ease of use, feeling of security and contribution to improvement of everyday life (Figure 5):

• Ease of use (median = 4, mean = 4.2759, SD = 0.84077, scoring range: 2–5);
• Feeling of security (median = 4, mean = 4.2759, SD = 0.84077, scoring range: 3–5);
• Contribution to the improvement of everyday life (median = 4, mean = 3.7931, SD = 1.04810, scoring range: 1–5).

Finally, apart from the aforementioned scales, the evaluation team also conducted interviews with the participants to get their direct feeling of their experience with the MOBOT rollator. During the trials, most of the participants (95%) expressed their enjoyment regarding the exploited HRI communication modes, including the audio/gestural options that were offered to them to express commands and the respective device’s reactions, as well as the navigation guidelines and localization information provided by the rollator. They mentioned that these features were assessed as extremely important for making them feel comfortable and allowing them to trust the device, especially in the case that they had to navigate unknown areas [34]. They also expressed noticeable satisfaction with respect to the oral and gestural commands leading the MOBOT device to rest positions, which they found to be a considerably helpful feature.

6. Points for Further Discussion

Thorough evaluation of technological devices equally requires the objective and subjective assessment of each product/service. Research on and development of assistive robots belongs to a distinct sector within robotics, which focuses on devices that can be used either in home or institution environment to assist individuals recover from (often severe) physical trauma or support them in activities of daily living.

Focusing on the healthcare sector, quality of care and patient satisfaction are major issues [35]. Thus, the assessment of any service or device from the perspective of the patient is crucial and should also incorporate opinion extraction on issues relating to adaptability of the device, the feeling of safety, the social perception of the user while using the device, the actual dimensions of the device related to judgments about whether it fits well in the environment of the end user, and so on.

The evaluation study conducted at DIAPLASIS aimed to validate the navigation assistance system of the MOBOT rollator-type mobility assistant, as well as the audio-gestural HRI model that is implemented on the device. Adequate test scenarios were developed as a result of collaboration between technical and clinical partners of the MOBOT consortium, which addressed the specific robotic functionalities that were to be tested. Following an appropriate recruitment strategy, a sufficient number of study participants were recruited. These participants were representative of the predefined target user group of the MOBOT rollator-type mobility assistant. The study was carried out successfully without any unexpected events during the execution of test scenarios.

The results on the subjective user satisfaction survey, initially reported in [34], are very positive and clearly suggest that exploitation of functionalities as those implemented at the MOBOT platform provide a clear benefit for users. Among the positive effects to be mentioned is the decrease of the feeling of insecurity that most users experience while walking, which has been recognized to have a positive effect on user quality of life.

The results of the objective performance metrics analysis (e.g., walking distance, the system’s accurate response rate, and various gait parameters) captured by the systems integrated on the robotic platform substantiate the positive results of the user subjective evaluation. These results impressively show the potential added value of the evaluated robotic functionalities for elderly users with slight gaiting and cognitive impairments.

The implemented audio-gestural HRI enables new modes of natural communication between users and assistive robotic devices, and is, therefore, highly relevant to the development of such devices with a focus on the user perspective. However, it should be noted that audio-gestural action detection and recognition still comprise some of the most challenging tasks in the domains of speech recognition and computer vision. This is especially true in the case of frail elderly people, who may perform commands with significant variation.

One critical outcome of the reported evaluation study relates to the decision to use in parallel three different subjective performance metrics, all of which are recognized as valid and reliable. Regarding the newly structured PYTHEIA scale, there is still a need for in-depth analysis of its usability and obtained results.

Finally, an advantage of the reported study has been the fact that it did not focus on specific pathologies, but rather on the functional status of the participants, as described in the inclusion criteria. This allowed for the observation of usability of the evaluated device in respect to the widest possible population that it may address.