Next Article in Journal
Energy-Optimized 3D Path Planning for Unmanned Aerial Vehicles
Previous Article in Journal
Research on the Corrosion Inhibition Effect of Xanthium sibiricum on Reinforced Steel and the Prediction of Reinforced Concrete Performance under a Stray Current and Chloride Environment
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 14
Issue 16
10.3390/app14166987
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Design of Exergaming Platform for Upper Limb Rehabilitation Using Surface Electromyography

by
Nikolaos Panagiotopoulos
1,
Sofia Lampropoulou
2,
Nikolaos Avouris
1 and
Athanassios Skodras
1,*
1
Department of Electrical and Computer Engineering, University of Patras, 26504 Patras, Greece
2
Department of Physiotherapy, University of Patras, 26504 Patras, Greece
*
Author to whom correspondence should be addressed.
Appl. Sci. 2024, 14(16), 6987; https://doi.org/10.3390/app14166987
Submission received: 15 June 2024 / Revised: 26 July 2024 / Accepted: 6 August 2024 / Published: 9 August 2024
(This article belongs to the Section Biomedical Engineering)
Browse Figures
Versions Notes

Abstract

:
This study explores the development and pilot testing of an exergame designed for the rehabilitation of individuals with upper limb deficits. While traditional physiotherapy is effective, it often fails to fully engage patients due to its repetitive nature. This research integrates a novel exergame into physiotherapy regimens, aiming to enhance patient motivation through a gaming experience that complements conventional sessions. The exergame is structured around a narrative-driven adventure, with exercises embedded in gameplay that mirror adjustable physiotherapy routines. It utilizes the Myo armband, a wearable electromyography device, to capture muscle activity and movement. The system, part of a web-based platform, is easily deployable in various settings, including home environments. Comprehensive evaluations with health professionals and neurological patients indicate that the exergame significantly improves patient engagement. This study not only demonstrates the potential of exergames in enhancing traditional therapy but also underscores the importance of patient-centered therapeutic tools.

1. Introduction

Many individuals struggle with activities of daily living due to muscle weakness, movement difficulties, or chronic pain, significantly impacting their quality of life. Physiotherapy plays a crucial role in managing these conditions by providing structured interventions that improve functionality and overall well-being. However, the need for prolonged therapy sessions can lead to patient reluctance due to boredom, which diminishes intrinsic motivation, and financial constraints. In response, the integration of exergames into treatment regimens is being explored.
Exergames, a portmanteau of exercise and game, also referred to as active video games or fitness games, can be defined as video games that require bodily movements for gameplay and serve as a form of physical exercise [1], without compromising their entertaining character. While exergames are typically used for maintaining fitness, their application in rehabilitating motor skills in patient populations has been extensively explored in recent years [2]. Nevertheless, since commercially available exergames designed with the general public in mind do not cater to specific therapeutic requirements, the development and clinical validation of custom games have become essential.
The existing body of research in this field is rich with innovative contributions. Notably, various studies have demonstrated the effectiveness of user engagement through advanced technologies such as smart glasses for mixed [3] and augmented realities [4], projection mapping for interactive floor contact [5], immersive CAVE systems [6], and motion capture suits [7]. There is also a focus on custom, cost-effective, solutions for capturing motion input data, including the use of smartphones with balance boards [8] or gloves [9] to utilize built-in sensors, employing smartphone cameras for 3D pose estimation [10], and integrating inertial measurement units with smartphones for data collection [11]. The literature further investigates tools to improve the performance and monitoring of physical activity. This involves the use of artificial intelligence to ensure precise exercise execution [12], adaptive difficulty settings via interaction classification [13] or decision-making with fuzzy logic [14], and personalized exercise routines recommendations based on user history [15]. Game design has also been a matter of study, as researchers have examined the impact of allowing specialized users to customize exercises [16], the role of narrative in boosting engagement [17], the effectiveness of specific frameworks like Persuasive System Design [18], and the application of general design principles to exergaming [19]. Additionally, significant attention has been given to understanding user reactions to various game attributes. This encompasses studies on the influence of game elements on virtual exercise experiences [20], the integration of psychoeducational content [21], the effects of collaborative play through shared mechanisms [22] or parallel side-by-side actions [23], the incorporation of cognitive challenges [24], and strategies for sustaining long-term engagement [25]. Nevertheless, despite extensive research, there is still a need for experimentation and research in this area. Accessibility is paramount, necessitating innovations such as controller-free interaction to support individuals with physical limitations, as demonstrated in [26,27], and the creation of games that more closely mirror conventional physical therapy sessions. Moreover, the potential for biofeedback through, for example, heart rate [28] and muscle activity monitoring [29] remains underutilized. Finally, exergames often lack cohesive narratives, overly relying on disjointed minigames.
This study aims to address these gaps by developing an exergame that is playable via affordable and easy-to-use technology, capable of providing measurements of physiological parameters, inclusive of gameplay taking into account a protocol of physiotherapy exercises, and has been designed as an actual game rather than a gamified application. Utilizing the Myo armband, a wearable electromyography device, the game is integrated into a broader web-based platform, with design principles driven by human–computer interaction, and is evaluated by individuals with neurological disorders. Building on this foundation, the research presented herein explores how the introduction of storytelling and a strictly defined exercise routine within this exergame affects the motivation of patients during their rehabilitation.

2. Materials and Methods

This section details the foundational requirements identified during the early stages of development and elaborates on the integration of various components essential for creating the intended exergaming platform.

2.1. Requirements Elicitation

The design was driven by the idea that an exergame should be a complementary tool in physical therapy, complementing, rather than replacing, conventional sessions. During the initial requirements gathering phase, three key pillars were identified: inclusivity, i.e., to ensure that the game is accessible to a wide range of users; portability, i.e., to focus on facilitating play primarily at home; and supervision, i.e., to enable health professionals to effectively monitor and guide the rehabilitation process.
To prevent exclusion due to technological skills or economic status, the game should avoid the use of expensive and not widely available devices like those of immersive virtual reality and instead run adequately on low-end computers; ideally be made available for various devices, i.e., mobile phones and tablets; and be easy to set up. In addition, given the wide range of physical disabilities, the game should target a specific part of the body to optimize exercise and allow for specialized technology and design. It is also necessary to provide the physiotherapist or doctor supervising the patient with some kind of information about the patient’s performance, which requires a dedicated interface.
The choice of the body part to focus on was thoroughly considered. In recent years, research has shown that neurological disorders, i.e., disorders of the nervous system, are the leading cause of disability worldwide [30]. Patients with central nervous system pathologies or trauma, such as stroke, Multiple Sclerosis, or basal ganglia dysfunction, experience various forms of abnormal motor behavior and symptoms. Spasticity, weakness, tremor, and ataxia are some of these symptoms, caused by altered inputs from the corticospinal tract and other descending motor pathways to the spinal cord neural circuits [31]. These symptoms lead to motor deficits in the upper limbs, particularly affecting the hands. Hand disability is one of the most significant dysfunctions, as it is involved in almost all everyday activities, causing problems with the patient’s functionality and autonomy [32]. Therefore, it is crucial that every form of rehabilitation, whether traditional physiotherapy or an exergame, includes exercises for the hands [15].
In addition, personae and scenarios were created after discussions with the intended end-users to guide the game’s design, and a series of interviews were conducted with healthcare professionals to obtain recommendations on the therapeutic nature of the game. These recommendations included clearly defined exercise formats with parameters that can be modified by each healthcare professional to suit individual patient needs. For the pilot testing, we implemented a specific protocol, which we will explain in detail later.

2.2. Conceptual Design

The design of the exergaming solution was based on the above analysis. At the heart of the system is the concept of pairing healthcare professionals with patients, enabling a dynamic interaction where the former adjusts game settings based on the latter’s performance, which, in turn, generates valuable health data. A key feature of the system is its flexibility; it supports the reassignment of patient–professional pairs, ensuring that patient care is continuous and informed, even when a patient changes professional. This is achieved through a web application that allows healthcare professionals to monitor their patients’ progress, with the data being securely stored and managed in a database.
Once the web application concept was solidified, integrating the game directly into it emerged as a strategic move, enhancing accessibility and simplifying maintenance. This approach allowed users to access the game on any device with a web browser, with no need for installation. It also streamlined the process for rolling out updates and introduced the possibility of adding social gaming features, enriching the user experience by fostering a community among players. As a result, the overall architecture of the system includes two types of users—practitioners and players—with different interfaces for each of them. Practitioners have access to a dashboard, while players engage with a personalized gaming experience that is implicitly tailored to their health needs through that dashboard.
Finally, for the motion sensing input device, the Myo armband [33,34] was chosen for its effective surface electromyography (sEMG) sensors that capture muscle activity, its inertial sensors for precise motion tracking, and its user-friendly design. The armband is attached directly to the forearm without electrodes or sticky substances; it is lightweight and offers an adjustable diameter for comfort, making it ideal for continuous health monitoring.
The efficient recognition of a variety of hand movements, especially when constrained by real-time requirements, is a topic of significant interest for many researchers, including our team. This process is fundamental to all applications targeting touchless input, such as our exergame, and typically employs artificial intelligence (AI) techniques. The work presented in [35] explores this topic using a Temporal Convolutional Network (TCN) model applied to entire sequences of sEMG signals to recognize 52 hand gestures. The performance of the model is further evaluated through a real-time simulation, where samples of each signal sequence are progressively fed into the system. Our current efforts include developing deep learning algorithms for hand movement recognition, similar to this approach.
However, in this study, we leverage the capability of the Myo armband to detect five distinct poses through pose start and pose end events, a feature not found in comparable devices. This allows us to prioritize the development of the initial pilot application before integrating a customized recognition scheme. To facilitate future integration, we optimized the system infrastructure by developing an additional compact application running on the user’s device. This intermediary application can receive sEMG data, process the data through a neural network model trained to recognize complex gestures, and transmit the gestural data to the web-based game. This approach enhances the capabilities of the system without interfering with the current gaming experience, as it is configured to use the Myo armband events when no model is specified while visualizing the stream of sEMG data on screen.

2.3. Exercise Protocol

The defined hand exercises were considered to improve hand motor control and function, without increasing muscle tone or fatigue. Knowing that spasticity, which is most commonly noted in the flexor muscles of the upper extremities, is velocity-dependent, the muscle contractions should have duration, but they should not be phasic, and thus, the movements, especially the stretches, should not be rapid [36].
Therefore, the exercises included in the exergame were selected with the purpose of opposing the pattern of spasticity, such as wrist extension and finger abduction exercises, improving hand muscle dexterity and functionality by including gripping actions like pinching and enhancing the muscle power and functionality of the whole hand with exercises such as wrist and finger flexion exercises.
Research studying the effectiveness of electronic games in upper extremity rehabilitation suggests the use of different and varied protocols ranging from 15 to 18 h, spread over 3 to 6 weeks [37,38,39]. In line with these aforementioned studies, the exercise protocol for the present study was designed so that the exercises with the exergame lasted 4 weeks, 5 days per week, 1 h per day, totaling 16 h.
According to personalized aims for each patient and depending on the motor function and dexterity of each patient, the duration of the contractions and relaxations could be changed. The purpose is that the contractions should last for at least 10 s so the muscles have time to be activated and react, and they should not be sudden so as not to increase spasticity, but they need to be long enough to improve muscle power and motor control [40]. Repetitions are necessary for motor rehabilitation, which operates through learning- and use-dependent mechanisms, according to European Stroke Organization guidelines [37]. The rest period following the contractions was set to protect the muscles from fatigue, and it was decided that its duration be at least the same as the contraction duration [41].

3. The Exergaming Platform

In this section, we present the developed exergaming platform prototype. The platform consists of several essential elements, including the Myo armband and its accompanying software, the local intermediary application, the database, and the web application, which includes the game for the patient/players, the dashboard for the practitioners to monitor the gameplay, and an application programming interface (API) that allows for automatic data transfer between the game and the database. Player inputs captured by the Myo armband are transmitted to its accompanying software and then relayed to the local intermediary application, which interfaces with the player’s browser. The browser communicates with the game and API modules of the web application on the server. Concurrently, health professionals access the system via a separate browser interface linked to the web application’s dashboard module. Both the dashboard and the API modules interact with the central database. This architecture is illustrated in Figure 1.
The game is structured around basic exercise rules that form the foundation upon which the gameplay is built. The five hand movements deemed critical for rehabilitation that have been selected are accurately detected by the Myo armband: wrist flexion, wrist extension, finger flexion, finger abduction, and pinching. Concisely, the daily exercise routine is divided into several sets, each consisting of a sequence of repetitions of a specific movement. For a movement to be considered valid, it must be held for a predetermined duration. Taking into account the needs of the patients, the routine includes rest periods between sequences of movements. The number of sets, the number of repetitions per set, and the duration of the holds and rests can be adjusted by healthcare professionals using the dashboard.
The game is designed as an adventure game with a storyline to give players a sense of progression, similar to traditional commercial adventure games. The gameplay is divided into four virtual worlds, each containing up to seven levels, reflecting the number of daily exercise routines a therapist might prescribe for a week, supporting a maximum of twenty-eight consecutive days of exercise. To prevent repetitiveness, four home screens are used, one for each week, reflecting the current stage of the player’s exercise plan. These screens feature platforms representing the daily schedule for that week, with the game’s protagonist positioned on the platform corresponding to the current day’s exercise. In addition, the game’s title and introductory scene feature a countdown element linked to the story, showing the number of days left for the hero to reach the final goal, increasing the player’s engagement with the game world. All these elements are illustrated in Figure 2.
In each level of the game (see Figure 3), the hero’s (1) goal is to collect a magical gem (2), which is protected by an impenetrable shield that can only be neutralized by defeating enemies (3) with a weapon similar to the Myo device (4). Including a Myo-like weapon enhances immersion by integrating the real tool into the virtual world of the game and celebrates it as the source of the character’s power, adding depth to the narrative and shifting the player’s perception from wearing it due to weakness to wearing it for strength. The game integrates exercise by requiring the hero to approach enemies within a certain range, mirroring the physical movements, the exercises, that the player must perform. The corresponding exercise is shown through the hint circle (5). Enemies represent exercise sets, with their hit points reflecting the number of repetitions required. The remaining hit points are indicated by the health bar (6). The game mechanics include an attack recharge time, which is the hold time, and an attack impact time, which is the rest time, facilitating the integration of the exercise routine. Feedback on these times is provided through the attack bar (7).
Levels are designed on a grid system, ensuring that the hero, the gem, and the enemies are strategically placed to allow for free movement and moments of relaxed gameplay. Enemies appear in waves, and once one wave is cleared, the next wave appears until the shield protecting the gem is deactivated. The wave display (8) shows information about the current and upcoming waves of enemies. The game adjusts to various workout routines by altering the number of enemies and their hit points. Figure 4 presents panels illustrating the gameplay. The primary user interface element is the hint circle, which changes color to guide the player through exercise movements. It is white when the hero is within the enemy’s range (see Figure 4c), indicating the required movement. It turns blue when the player performs the movement (see Figure 4a), then green upon successful completion of a repetition (see Figure 4b), signifying that the hold time has passed. When the hero is out of range, the circle has a semi-transparent black color and is without any symbols (see Figure 4d).
The design of the game went through several iterations, during which prototypes were shown to typical users and a focus group of experts. In the next section, the summative evaluation of the prototype is discussed.

4. Evaluation Study

Each of the two distinct and interrelated elements of the system, the health professional interface and the exergame, needs to be treated differently when being evaluated. This is partly because of their different end-users and goals and partly because of the supplementary nature of the professional interface and the primary position of the game presented in this study.
For the evaluation of the system, user testing was performed, where the testing of the interface with health professionals was independent of the testing of the game with typical users, i.e., neurological patients. The professional testing was aimed at evaluating a variety of operations and scenarios, including but not limited to managing exercise programs and patient records, analyzing game sessions, and examining the usability of the graphical user interface. On the other hand, in the patient testing, the only goal for the participants was the completion of the game. In addition, the game was also tested by individuals without neurological disorders. However, this testing was solely conducted to gain a deeper understanding of the specific characteristics of the patients.
In general, end-user testing is an effective way of determining a system’s performance in real-world conditions. Conclusions drawn from how users interact with the system are particularly useful for a game aimed at patients, given the special needs of this population. We used a mixed-methods approach that included both quantitative and qualitative data collection and analysis. Specifically, we used system logs and video recordings of the patient and their screen to collect quantitative data about user interactions, while we conducted user satisfaction questionnaires to collect qualitative data in order to capture the subjective user experience and perceptions. The questionnaire consisted of nine questions and used a Likert scale expressing levels of agreement or disagreement with a given statement. Questions included in the questionnaire that was given to the participants are shown in Table 1. The integration of these methods was consistent with the framework for constructing mixed-methods studies presented in [42] and allowed us to triangulate our findings. It is worth noting that the tests were conducted in a clinical setting, and the think-aloud protocol was followed.
The characteristics of the participants in the study are shown in Table 2.
All the patients, referred to as P1–P6, were selected by considering diversity as much as was possible in the context of this study. To be eligible for participation, patients had to meet specific inclusion criteria, as outlined in [38,39]. Firstly, patients needed to possess some degree of active movement in the wrist, hand, and fingers, specifically extension, to handle the input devices effectively. Additionally, they had to have a muscle strength of at least 3 on the Medical Research Council (MRC) scale, allowing them to perform movements against gravity. Patients with severe hypertony, indicated by a score of 3 or more on the Modified Ashworth Scale (MAS), were excluded due to potential difficulties with wrist and finger extension caused by increased spasticity. Cognitive function is also a critical criterion; patients must have scored above 24 on the Mini-Mental State Examination (MMSE) to comprehend instructions and provide informed consent. Furthermore, some degree of motor function in the hand and wrist is required to play the game, with the recommended minimum scores on the Fugl-Meyer Motor-Sensory Assessment being 5 out of 10 for the wrist, 7 out of 14 for the hand, and a passive range of motion exceeding 12 out of 24.
The game given to the patients was loaded with the training program listed in Table 3, which was the result of the interview phase with the physical therapists. The values for the hold time and the rest time are given in seconds.

5. Results

This section presents the evaluation results for the exergaming platform, including feedback on the health professionals’ application, and focuses on the findings from the exergame user studies.

5.1. Health Professionals’ Application

The evaluation of the health professionals’ application showed that it was easy to use and allowed them to manage the game-related data effortlessly, without any significant learning curve. The tasks assigned were conducted smoothly and quickly. In addition, the final graphical interface and navigation design of the application received positive feedback. However, an initial observation highlighted the lack of detailed patient information, such as the specific neurological condition and its duration. In addition, there was a desire among some professionals to be able to view comparative patient outcomes through graphical representations. As a future improvement, the idea of automating the generation of monthly reports on patient sessions and forwarding them to specialists’ personal email accounts was suggested.

5.2. Exergame

The below discussion of the exergame results begins with detailed observations from the user study with patients, followed by an analysis of the questionnaire responses, and concludes with a comparative study between patient and non-patient groups.

5.2.1. Observations from User Study with Patients

Patient evaluations of the exergame provided interesting findings, based on key observations from the game sessions. One critical issue identified was the significant impact of involuntary spasmodic movements on gameplay. Specifically, users with increased spasticity, corresponding to a score of 2 on the MAS, often required more than twice as many attempts to successfully complete an exercise set. The challenge arose because the Myo armband interpreted brief spasms as an end to and restart of movement, generating excess events. This resulted in the game prematurely terminating the current attempt due to its immediate response to the receipt of the pose end event and then promptly restarting the attempt upon receipt of the pose start event. This instantaneous nature, without the presence of a threshold for accepting a state for the current gesture, highlights the sensitivity of the game’s logic in relation to unintentional movements. As a result, the user had to repeat the exercise repetition, which adversely affected engagement.
An additional observation was the unintended effect of one patient’s behavior during play, as shown in Figure 5. After completing an exercise repetition, the patient instinctively clenched her hand, moving directly from finger abduction to finger flexion, bypassing the neutral rest position. This caused the game, due to its method of processing input data, to incorrectly register the closing of the hand as a completed exercise action. This misrecognition, although rare, highlights a flaw in the game’s accuracy in tracking intended exercise movements.
Another observation focused on the overall player experience was that players occasionally did not respond quickly or correctly to feedback from the game. For instance, they frequently kept their hand in position long after subtle signs suggested that the necessary time had elapsed. Furthermore, players sometimes attempted actions without realizing that certain cues meant their movements had no effect, such as being out of range of an enemy. These examples highlight the importance of providing clearer feedback within the game, as unnecessary effort due to such misinterpretations can adversely affect the patient’s performance by causing undue muscle strain.

5.2.2. Questionnaire

In analyzing responses from questionnaires, points were allocated to each answer option, except for question Q4, due to its unique nature, as explained later. The scoring system was as follows: option (a) scored five points, (b) four points, (c) three points, (d) two points, and (e) one point. The first two response options were associated with positive perspectives, whereas the last two indicated negative viewpoints. Thus, a higher cumulative score from all questionnaires for a particular question indicated more favorable feedback from participants.
The average scores of seven out of the nine questions allow for a comparative analysis of user satisfaction across different aspects of the game as per the questionnaire (Figure 6). Question Q4 was excluded from this analysis because it was not posed to the healthy demographic, given its focus on the challenge level tailored by the total number of sequence repetitions—a factor not applicable to healthy individuals. This exclusion also served to inform physiotherapists about the suitability of the exercise program.
Feedback for the open question Q9, answered only by patients, included the following, mostly positive, remarks: “I really enjoyed the game, although it was somewhat demanding, especially when involuntary arm movements occurred.”, “It could be improved by adding more variety to the activities within each level.”, “I have no criticisms; I hope the game continues to develop!”, “Once the game’s concept and objectives became clear, it was easy to handle. Initial step-by-step instructions would have been helpful.”, and “I had a great time playing and have no suggestions for improvement—keep it up!”.

5.2.3. Comparative Study of Patient and Non-Patient Groups

In addition to the study of the patient group described above, the same routines were performed for the study of an equal-size no-patients group. Our primary aim was to gain insights into general trends among the patients rather than to delve into a comparison of the two groups. In reviewing the camera and device screen recordings, there were consistent differences observed between sessions involving patients and those with non-patients. Firstly, there was a noticeable difference in the time taken by each group to complete the same level, showing longer durations for patients (Figure 7). Secondly, there were instances where players appeared to be making the correct gestures, but the game failed to register these movements due to the absence of pose events from the armband. The device occasionally could not recognize a gesture due to inappropriate signals, i.e., weak or noisy signals. This was particularly evident in video recordings where the players’ hand movements were clearly visible and the visualized sEMG data stream was available. The frequency of these failed attempts to initiate a movement repetition was accurately recorded, highlighting a significant discrepancy in the responsiveness of the interface between the two groups (Figure 8).

6. Discussion

The evaluation process yielded insightful conclusions, highlighting the strengths of the proposed approach and identifying areas for improvement. The interface designed for health professionals was well received and praised for its ease of use and successful integration of essential functionality features. As for the exergame, the feedback from players, collected through questionnaires, was overwhelmingly positive. Notably, not a single respondent gave negative feedback, with the majority expressing positive to very positive experiences. The aspect that received the least praise was the audio presentation, including music and sound effects. At this stage, we utilized freely available resources due to budget constraints and a lack of production expertise within our team. However, recognizing the importance of this aspect with respect to user engagement, we plan to invest in professional sound design to address these shortcomings. Conversely, the usability of the game was considered to be its strongest point, indicating widespread approval among participants, despite minor problems encountered during gameplay.
The issues identified indicate the necessary adjustments to the current design of the game. For example, interface components that convey critical exercise information—such as the completion of hold times or the number of repetitions remaining—could benefit from increased prominence through being larger in size and incorporating brighter colors to ensure immediate recognition by the player. Adjustments to the game logic were also suggested, such as introducing a delay before aborting an exercise attempt to allow for involuntary muscle spasms. These suggested changes underline a multi-faceted approach to refining the implementation of the game in several different but clearly outlined aspects.
The differences in the completion times of the patients and healthy users can be attributed to two main factors. Firstly, the increased fatigue experienced by patients often leads to longer rest periods between exercise repetitions and sets. Secondly, frequent instances of movement recognition problems and premature termination of exercise repetitions were observed, often triggered by small palm movements during gestures. In addition to these findings, time tracking allows for the introduction of new features, such as the ability to compare individual completion times against the overall average, or the development of reward mechanisms based on completion times to encourage player effort and improve performance.
The variation in gesture recognition rates is likely to be due to the weaker electrical signals produced by patients with neurological disorders, which affect the performance of the Myo armband. This suggests the need for specialized processing of electromyography signals to improve the user experience. Modifications to input and output data handling could be implemented through the local intermediary application, laying the groundwork for incorporating customized signal processing and efficient information transfer into the game.
It is important to acknowledge some of the limitations of this study. The sample size of patients was not large enough to draw firm conclusions about the user experience. In addition, the limited duration of use of the game provided little insight into its long-term effectiveness as a rehabilitation tool. The inclusion of a control group performing traditional exercises to compare with the game users is also necessary for the assessment of the game’s therapeutic potential.
Future developments could include extending the health professional interface with additional functionality while maintaining the current framework. Targeted enhancements to the game to address the specific problems identified, as discussed above, could be pursued. The introduction of secondary objectives and mechanisms unrelated to hand movements may further increase player satisfaction without compromising the exercise plan. Finally, the development of gesture recognition algorithms and the integration of a pre-trained neural network model through the local intermediary application could solve the problem of weak and noisy signals and extend the range of recognizable movements.

7. Conclusions

This paper examines the integration of exergames into rehabilitation practices, effectively addressing the challenges associated with traditional physiotherapy. The implementation of an exergaming platform not only complements conventional methods but also introduces a more engaging and dynamic treatment approach. The proposed system architecture promotes the widespread accessibility of therapeutic exergaming, making it accessible to a diverse range of users, including those who may face technological or economic limitations.
Furthermore, the system’s design, which allows health professionals to monitor and adjust settings remotely, ensures personalized and continuous care for each patient. This is supported by the Myo armband technology, which integrates seamlessly into the game, providing accurate tracking and feedback on the user’s performance. By embedding the gaming elements into a narrative and structuring exercises around engaging challenges, the platform effectively maintains patient engagement and motivation, despite the potential constraints of its strictly defined routines.
The effectiveness of this approach was validated through testing with both health professionals and neurological patients. The feedback was overwhelmingly positive, highlighting the system’s acceptance by patients, due to ease of use, the engaging nature of the game, and the practical benefits observed in patient adherence and therapy outcomes. These results underscore the potential of similar exergames to enhance the delivery and effectiveness of physiotherapy, offering a more enjoyable and sustainable option for patients.
This research aims to make a contribution to the field of rehabilitation, rooted in the belief that enhancing the lives of individuals not only benefits those directly affected but also yields widespread positive effects for the caregivers and practitioners’ community.

Author Contributions

Conceptualization, A.S.; methodology, S.L., N.A. and A.S.; software, N.P.; validation, N.P., S.L., N.A. and A.S.; formal analysis, N.P., S.L., N.A. and A.S.; investigation, N.P., S.L., N.A. and A.S.; resources, S.L., N.A. and A.S.; data curation, N.P.; writing—original draft preparation, N.P.; writing—review and editing, N.P., S.L., N.A. and A.S.; visualization, N.P., N.A. and A.S.; supervision, N.A. and A.S.; project administration, N.A. and A.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the University Hospital of Patras (Ethics Committee—No. 64/04-02-21).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The original contributions of this study are included in the article; further inquiries can be directed to the corresponding author.

Acknowledgments

The authors are grateful to Elisabeth Chroni of the Neurology Department and the Neuromuscular Diseases Unit (National Center of Excellence for Rare Diseases) of the University Hospital of Patras for the selection of the patients, providing access to clinics, and receiving the approval for the assessment of the developed application by the patients. They would like also to express their gratitude to Angeliki Anastasopoulou for her contributions throughout this work. Her suggestions during the initial design phase shaped the direction of the platform, and her testing and insightful feedback in each iteration greatly enhanced its quality and functionality. Additionally, her support during the evaluation phase with end-users was instrumental to the successful execution of this part of the study.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
CAVECave Automatic Virtual Environment
3DThree-Dimensional
sEMGSurface electromyography
AIArtificial intelligence
TCNTemporal Convolutional Network
APIApplication programming interface
MSMultiple Sclerosis
PDParkinson’s Disease
sSeconds
MRCMedical Research Council
MASModified Ashworth Scale
MMSEMini-Mental State Examination

References

  1. Gao, Z.; Lee, J.E.; Pope, Z.; Zhang, D. Effect of Active Videogames on Underserved Children’s Classroom Behaviors, Effort, and Fitness. Games Health J. 2016, 5, 318–324. [Google Scholar] [CrossRef]
  2. Rüth, M.; Schmelzer, M.; Burtniak, K.; Kaspar, K. Commercial exergames for rehabilitation of physical health and quality of life: A systematic review of randomized controlled trials with adults in unsupervised home environments. Front. Psychol. 2023, 14, 1155569. [Google Scholar] [CrossRef] [PubMed]
  3. Gmez-Portes, C.; Carneros-Prado, D.; Albusac, J.; Castro-Schez, J.J.; Glez-Morcillo, C.; Vallejo, D. PhyRe Up! A System Based on Mixed Reality and Gamification to Provide Home Rehabilitation for Stroke Patients. IEEE Access 2021, 9, 139122–139137. [Google Scholar] [CrossRef]
  4. Cidota, M.A.; Bank, P.J.; Lukosch, S.G. Design Recommendations for Augmented Reality Games for Objective Assessment of Upper Extremity Motor Dysfunction. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 1430–1438. [Google Scholar] [CrossRef]
  5. Amiri, Z.; Sekhavat, Y.A.; Goljaryan, S. StepAR: A personalized exergame for people with multiple sclerosis based on video-mapping. Entertain. Comput. 2022, 42, 100487. [Google Scholar] [CrossRef]
  6. Minissi, M.E.; Landini, G.A.R.; Maddalon, L.; Torres, S.C.; Giglioli, I.A.C.; Sirera, M.; Abad, L.; Gómez-García, S.; Alcañiz, M. Virtual Reality-Based Serious Games to Improve Motor Learning in Children with Autism Spectrum Disorder: An Exploratory Study. In Proceedings of the 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH), Athens, Greece, 28–30 August 2023; pp. 1–6. [Google Scholar] [CrossRef]
  7. Fu, Y.; Li, Q.; Ma, D. User Experience of a Serious Game for Physical Rehabilitation Using Wearable Motion Capture Technology. IEEE Access 2023, 11, 108407–108417. [Google Scholar] [CrossRef]
  8. Baranyi, R.; Rast, L.; Pinter, K.; Aigner, C.; Hoelbling, D.; Grechenig, T. FruitGrind: Analysis, Design and Development of a Serious Game Supporting Knee Rehabilitation Using a Smartphone Attached to a Balance Board. In Proceedings of the 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH), Athens, Greece, 28–30 August 2023; pp. 1–6. [Google Scholar] [CrossRef]
  9. Song, X.; Ding, L.; Zhao, J.; Jia, J.; Shull, P. Cellphone Augmented Reality Game-based Rehabilitation for Improving Motor Function and Mental State after Stroke. In Proceedings of the 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Chicago, IL, USA, 19–22 May 2019; pp. 1–4. [Google Scholar] [CrossRef]
  10. Wang, G.; Zhu, B.; Fan, Y.; Wu, M.; Wang, X.; Zhang, H.; Yao, L.; Sun, Y.; Su, B.; Ma, Z. Design and evaluation of an exergame system to assist knee disorders patients’ rehabilitation based on gesture interaction. Health Inf. Sci. Syst. 2022, 10, 20. [Google Scholar] [CrossRef]
  11. Kontadakis, G.; Chasiouras, D.; Proimaki, D.; Halkiadakis, M.; Fyntikaki, M.; Mania, K. Gamified platform for rehabilitation after total knee replacement surgery employing low cost and portable inertial measurement sensor node. Multimed. Tools. Appl. 2020, 79, 3161–3188. [Google Scholar] [CrossRef]
  12. Martins, T.; Carvalho, V.; Soares, F.; Leão, C. Physioland: A motivational complement of physical therapy for patients with neurological diseases. Multimed. Tools. Appl. 2024, 83, 12035–12057. [Google Scholar] [CrossRef]
  13. Bouatrous, A.; Meziane, A.; Zenati, N.; Hamitouche, C. A new adaptive VR-based exergame for hand rehabilitation after stroke. Multimed. Syst. 2023, 29, 3385–3402. [Google Scholar] [CrossRef]
  14. Sadeghi Esfahlani, S.; Butt, J.; Shirvani, H. Fusion of Artificial Intelligence in Neuro-Rehabilitation Video Games. IEEE Access 2019, 7, 102617–102627. [Google Scholar] [CrossRef]
  15. González-González, C.S.; Toledo-Delgado, P.A.; Muñoz-Cruz, V.; Torres-Carrion, P.V. Serious games for rehabilitation: Gestural interaction in personalized gamified exercises through a recommender system. J. Biomed. Inform. 2019, 97, 103266. [Google Scholar] [CrossRef] [PubMed]
  16. Duval, J.; Thakkar, R.; Du, D.; Chin, K.; Luo, S.; Elor, A.; El-Nasr, M.S.; John, M. Designing Spellcasters from Clinician Perspectives: A Customizable Gesture-Based Immersive Virtual Reality Game for Stroke Rehabilitation. ACM Trans. Access. Comput. 2022, 15, 26. [Google Scholar] [CrossRef]
  17. Eckert, M.; Aglio, A.; Martín-Ruiz, M.L.; Osma-Ruiz, V. A New Architecture for Customizable Exergames: User Evaluation for Different Neuromuscular Disorders. Healthcare 2022, 10, 2115. [Google Scholar] [CrossRef] [PubMed]
  18. Sanpablo, A.I.P.; Armenta-García, J.A.; Muñiz, A.F.; Peñaloza, A.M.; Mendoza-Arguilés, A.; Rodríguez, M.D. Integration of persuasive elements into exergames: Application in the development of a novel gait rehabilitation system for children with musculoskeletal conditions. J. Biomed. Inform. 2022, 132, 104130. [Google Scholar] [CrossRef] [PubMed]
  19. Herne, R.; Shiratuddin, M.F.; Rai, S.; Blacker, D.; Laga, H. Improving Engagement of Stroke Survivors Using Desktop Virtual Reality-Based Serious Games for Upper Limb Rehabilitation: A Multiple Case Study. IEEE Access 2022, 10, 46354–46371. [Google Scholar] [CrossRef]
  20. Ota, T.; Nakamura, T.; Kuzuoka, H. Effects of Gamification and Communication in Virtual Reality Frozen Shoulder Rehabilitation for Enhanced Rehabilitation Continuity. IEEE Access 2023, 11, 50841–50850. [Google Scholar] [CrossRef]
  21. Stamm, O.; Dahms, R.; Reithinger, N.; Ruß, A.; Müller-Werdan, U. Virtual reality exergame for supplementing multimodal pain therapy in older adults with chronic back pain: A randomized controlled pilot study. Virtual Real. 2022, 26, 1291–1305. [Google Scholar] [CrossRef]
  22. Høeg, E.R.; Bruun-Pedersen, J.R.; Cheary, S.; Andersen, L.K.; Paisa, R.; Serafin, S.; Lange, B. Buddy biking: A user study on social collaboration in a virtual reality exergame for rehabilitation. Virtual Real. 2023, 27, 245–262. [Google Scholar] [CrossRef]
  23. Shah, S.H.H.; Karlsen, A.S.T.; Solberg, M.; Hameed, I.A. A social VR-based collaborative exergame for rehabilitation: Codesign, development and user study. Virtual Real. 2023, 27, 3403–3420. [Google Scholar] [CrossRef]
  24. Raygoza-Romero, J.; Gonzalez-Hernandez, A.; Bermudez, K.; Martinez-Garcia, A.I.; Caro, K. Move&Learn: An Adaptive Exergame to Support Visual-Motor Skills of Children with Neurodevelopmental Disorders. In Proceedings of the Conference on Information Technology for Social Good (GoodIT ’21), Rome, Italy, 9–11 September 2021; pp. 169–174. [Google Scholar] [CrossRef]
  25. Elor, A.; Powell, M.; Mahmoodi, E.; Teodorescu, M.; Kurniawan, S. Gaming Beyond the Novelty Effect of Immersive Virtual Reality for Physical Rehabilitation. IEEE Trans. Games 2022, 14, 107–115. [Google Scholar] [CrossRef]
  26. Pereira, M.F.; Prahm, C.; Kolbenschlag, J.; Oliveira, E.; Rodrigues, N.F. A Virtual Reality Serious Game for Hand Rehabilitation Therapy. In Proceedings of the 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH), Vancouver, BC, Canada, 12–14 August 2020; pp. 1–7. [Google Scholar] [CrossRef]
  27. Liao, K.L.; Huang, M.; Wang, Y.; Wu, Z.; Yang, R.; Zhang, J.; Wang, L. A Virtual Reality Serious Game Design for Upper Limb Rehabilitation. In Proceedings of the 2021 IEEE 9th International Conference on Serious Games and Applications for Health(SeGAH), Dubai, United Arab Emirates, 4–6 August 2021; pp. 1–5. [Google Scholar] [CrossRef]
  28. Parente, C.; Galletti, C.; Bottino, A.; Lamberti, F.; Podda, J.; Tacchino, A.; Brichetto, G.; De Michieli, L.; Barresi, G. A Biofeedback-Enhanced Virtual Exergame for Upper Limb Motor-Cognitive Rehabilitation. In Proceedings of the 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH), Athens, Greece, 28–30 August 2023; pp. 1–8. [Google Scholar] [CrossRef]
  29. Garcia-Hernandez, N.; Garza-Martinez, K.; Parra-Vega, V.; Alvarez-Sanchez, A.; Conchas-Arteaga, L. Development of an EMG-based exergaming system for isometric muscle training and its effectiveness to enhance motivation, performance and muscle strength. Int. J. Hum. Comput. Stud. 2019, 124, 44–55. [Google Scholar] [CrossRef]
  30. Feigin, V.L.; Nichols, E.; Alam, T.; Bannick, M.S.; Beghi, E.; Blake, N.; Culpepper, W.J.; Dorsey, E.R.; Elbaz, A.; Ellenbogen, R.G.; et al. Global, regional, and national burden of neurological disorders, 1990–2016: A systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol. 2019, 18, 459–480. [Google Scholar] [CrossRef] [PubMed]
  31. Younger, D.S. Chapter 6—Multiple sclerosis: Motor dysfunction. In Motor System Disorders, Part II: Spinal Cord, Neurodegenerative, and Cerebral Disorders and Treatment, 1st ed.; Younger, D.S., Ed.; Elsevier: Amsterdam, The Netherlands, 2023; Volume 196, pp. 119–147. ISBN 978-0-323-98817-9. [Google Scholar] [CrossRef]
  32. Lotze, M.; Lindberg, P.G. Editorial: Promoting Manual Dexterity Recovery After Stroke. Front. Neurol. 2019, 10, 815. [Google Scholar] [CrossRef] [PubMed]
  33. Visconti, P.; Gaetani, F.; Zappatore, G.; Primiceri, P. Technical Features and Functionalities of Myo Armband: An Overview on Related Literature and Advanced Applications of Myoelectric Armbands Mainly Focused on Arm Prostheses. Int. J. Smart Sens. Intell. Syst. 2018, 11, 1–25. [Google Scholar] [CrossRef]
  34. Rawat, S.; Vats, S.; Kumar, P. Evaluating and exploring the MYO ARMBAND. In Proceedings of the 2016 International Conference System Modeling & Advancement in Research Trends (SMART), Moradabad, India, 25–27 November 2016; pp. 115–120. [Google Scholar] [CrossRef]
  35. Tsinganos, P.; Jansen, B.; Cornelis, J.; Skodras, A. Real-Time Analysis of Hand Gesture Recognition with Temporal Convolutional Networks. Sensors 2022, 22, 1694. [Google Scholar] [CrossRef] [PubMed]
  36. Trompetto, C.; Marinelli, L.; Mori, L.; Pelosin, E.; Currà, A.; Molfetta, L.; Abbruzzese, G. Pathophysiology of Spasticity: Implications for Neurorehabilitation. BioMed Res. Int. 2014, 2014, 354906. [Google Scholar] [CrossRef] [PubMed]
  37. Kwakkel, G.; Stinear, C.; Essers, B.; Munoz-Novoa, M.; Branscheidt, M.; Cabanas-Valdés, R.; Lakičević, S.; Lampropoulou, S.; Luft, A.R.; Marque, P.; et al. Motor rehabilitation after stroke: European Stroke Organisation (ESO) consensus-based definition and guiding framework. Eur. Stroke J. 2023, 8, 880–894. [Google Scholar] [CrossRef] [PubMed]
  38. Gauthier, L.V.; Kane, C.; Borstad, A.; Strahl, N.; Uswatte, G.; Taub, E.; Morris, D.; Hall, A.; Arakelian, M.; Mark, V. Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): Protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis. BMC Neurol. 2017, 17, 109. [Google Scholar] [CrossRef] [PubMed]
  39. Rozevink, S.G.; van der Sluis, C.K.; Garzo, A.; Keller, T.; Hijmans, J.M. HoMEcare aRm rehabiLItatioN (MERLIN): Telerehabilitation using an unactuated device based on serious games improves the upper limb function in chronic stroke. J. NeuroEng. Rehabil. 2021, 18, 48. [Google Scholar] [CrossRef] [PubMed]
  40. Page, P. Current concepts in muscle stretching for exercise and rehabilitation. Int. J. Sports Phys. Ther. 2012, 7, 109–119. [Google Scholar]
  41. Nogueira, D.V.; Silva, S.B.; de Abreu, L.C.; Valenti, V.E.; Fujimori, M.; de Mello Monteiro, C.B.; Tortoza, C.; Ribeiro, W.; Lazo-Osório, R.A.; Tierra-Criollo, C.J. Effect of the rest interval duration between contractions on muscle fatigue. BioMed Eng. OnLine 2012, 11, 89. [Google Scholar] [CrossRef] [PubMed]
  42. Schoonenboom, J.; Johnson, R.B. How to Construct a Mixed Methods Research Design. Köln. Z. Soziol. Sozialpsychol. 2017, 69, 107–131. [Google Scholar] [CrossRef] [PubMed]
Applsci 14 06987 g001
Figure 1. Architecture of the exergaming platform.
Figure 1. Architecture of the exergaming platform.
Applsci 14 06987 g001
Applsci 14 06987 g002
Figure 2. Screenshots illustrating the storytelling elements of the game. (a) Title screen. (b) Frame from the introductory scene. (c) Home screen for the first week. (d) Home screen for the fourth week.
Figure 2. Screenshots illustrating the storytelling elements of the game. (a) Title screen. (b) Frame from the introductory scene. (c) Home screen for the first week. (d) Home screen for the fourth week.
Applsci 14 06987 g002
Applsci 14 06987 g003
Figure 3. Gameplay scene showing key elements numbered for reference.
Figure 3. Gameplay scene showing key elements numbered for reference.
Applsci 14 06987 g003
Applsci 14 06987 g004
Figure 4. Images capturing various stages of the gameplay. (a) During the hold time of the exercise. (b) Just after the hold time has passed but before the player stops the exercise. (c) During the rest time of the exercise. (d) Just before level completion.
Figure 4. Images capturing various stages of the gameplay. (a) During the hold time of the exercise. (b) Just after the hold time has passed but before the player stops the exercise. (c) During the rest time of the exercise. (d) Just before level completion.
Applsci 14 06987 g004
Applsci 14 06987 g005
Figure 5. Example of incorrect movement recognition in the game. While the game screenshot on the left shows a successfully completed finger abduction repetition, the attached camera footage on the right reveals the player performing a finger flexion motion at that moment.
Figure 5. Example of incorrect movement recognition in the game. While the game screenshot on the left shows a successfully completed finger abduction repetition, the attached camera footage on the right reveals the player performing a finger flexion motion at that moment.
Applsci 14 06987 g005
Applsci 14 06987 g006
Figure 6. Mean scores based on Likert scale.
Figure 6. Mean scores based on Likert scale.
Applsci 14 06987 g006
Applsci 14 06987 g007
Figure 7. Mean level completion times (in minutes) for each member of the respective groups.
Figure 7. Mean level completion times (in minutes) for each member of the respective groups.
Applsci 14 06987 g007
Applsci 14 06987 g008
Figure 8. Successful recognition rate comparison.
Figure 8. Successful recognition rate comparison.
Applsci 14 06987 g008
Table 1. Survey questions used to assess player experience and satisfaction with the game.
Table 1. Survey questions used to assess player experience and satisfaction with the game.
Q1Overall, how would you characterize your experience with the game?
(a) Excellent (b) Good (c) Fair (d) Poor (e) Very poor
Q2Did you find the game interesting and fun?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q3Did you feel that the game was an effective way to practice?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q4Was the level of difficulty of the game appropriate for you?
(a) Very easy (b) Ideal (c) A little demanding (d) Demanding (e) Very demanding
Q5Was the game visually appealing?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q6Did you find that the control system was understandable and easy to use?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q7Was the sound experience (music, sound effects, etc.) enjoyable?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q8Would you recommend the game to a friend or family member?
(a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly disagree
Q9Do you have any suggestions on how the game could be improved?
[Please mark any comments in the blank space below.]
Table 2. Characteristics of participants.
Table 2. Characteristics of participants.
P1P2P3P4P5P6
GenderMaleMaleFemaleFemaleMaleMale
Age25–3455+35–4415–2425–3445–54
Video gamingNoNoNoYesYesNo
Disease 1MSPDMSMSMSPD
1 MS—Multiple Sclerosis; PD—Parkinson’s Disease.
Table 3. Exercise routine followed in the game testing.
Table 3. Exercise routine followed in the game testing.
Wrist FlexionWrist ExtensionFinger FlexionFinger AbductionPinching
Sets22122
Repetitions33335
Hold time (s)101515155
Between SetsBetween Repetitions
Rest time (s) 115
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Panagiotopoulos, N.; Lampropoulou, S.; Avouris, N.; Skodras, A. Design of Exergaming Platform for Upper Limb Rehabilitation Using Surface Electromyography. Appl. Sci. 2024, 14, 6987. https://doi.org/10.3390/app14166987

AMA Style

Panagiotopoulos N, Lampropoulou S, Avouris N, Skodras A. Design of Exergaming Platform for Upper Limb Rehabilitation Using Surface Electromyography. Applied Sciences. 2024; 14(16):6987. https://doi.org/10.3390/app14166987

Chicago/Turabian Style

Panagiotopoulos, Nikolaos, Sofia Lampropoulou, Nikolaos Avouris, and Athanassios Skodras. 2024. "Design of Exergaming Platform for Upper Limb Rehabilitation Using Surface Electromyography" Applied Sciences 14, no. 16: 6987. https://doi.org/10.3390/app14166987

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop