Personalized Visualization of the Gestures of Parkinson’s Disease Patients with Virtual Reality

Abstract

Parkinson’s disease is a neurological disorder characterized by motor and non-motor symptoms. Assessment methods, despite the many years of existence of the disease, lack individualized visualization. On the other hand, virtual reality promises immersion and realism. In this paper, we develop an integrated system for visualizing the gestures of Parkinson’s disease patients in a virtual reality environment. With this application, clinicians will have information about the unique motor patterns and challenges they must address in each individual patient’s case, while the collected data can travel and be easily and instantly visualized in any location. At the beginning of this research, the current terms of immersive technologies in conjunction with data visualization and Parkinson’s disease are described. Through an extensive systematic literature review, the technological developments in the field of Parkinson’s data visualization are presented. The findings of the review lead to the experimental procedure and implementation of the application. The conclusions drawn from this work fuel future extensions on the contribution of immersive technologies to various diseases.

1. Introduction

Parkinson’s disease constitutes one of the most widespread age-related neurodegenerative disorders worldwide, with multiple accompanying clinical and societal challenges. The condition is characterized by a number of non-motor symptoms, such as mood disorders, sleep disorders, or autonomic imbalances, that frequently precede the motor symptoms which the disease has been traditionally linked with, like bradykinesia, rigidity, postural changes, and resting tremor [1]. Since its first description by James Parkinson over 200 years ago, we have gained a lot of insight into the pathology and pathophysiology of the disease, but no groundbreaking progress has been made in relation to its etiology or management [2]. We know, for example, that the disease is associated with the loss of the dopaminergic neurons that normally connect the substantia nigra with the striatum, we know that aggregation of dysfunctional proteins exerts cytotoxic effects along the nervous system, and we recently suspect that the disease might start at the peripheral level, in the enteric nervous system. Nevertheless, no causative factors have been yet identified, and the treatment of Parkinson’s disease still hugely relies on the replacement of dopamine in an attempt to compensate for a short period of time for the loss of the dopaminergic neurons [3]. Other treatment approaches, like deep brain stimulation, are also transiently effective in only certain subpopulations of the patients [4]. Among the challenges of advancing the management of the disease, there is a pressing need to develop approaches for assessing symptom progression and severity and estimating prognosis in a personalized manner [5].

Virtual reality (VR) technologies might offer opportunities in this direction. VR has existed conceptually since the 19th century, but it is only today that this technology has seen a resurgence in interest among researchers, as the possibilities it offers are increasingly impressive [6]. This immersive technology stimulates the brain by almost all sensory modalities, visual, auditory, and tactile, creating a simulation of a given experience in the most realistic possible way. In the field of medicine, VR has made its appearance in surgical skill applications, in the cognitive and physical rehabilitation of stroke patients, in mental health applications for patients with anxiety disorders and chronic pain [7], as well as in telemedicine. Since VR has contributed significantly to many areas of science, the question arises whether it could contribute to improved recording, monitoring, and visualization of Parkinson’s disease symptoms [8].

The aim of the presented work is the development and perspectives of a system for the VR visualization of movements and/or exercises performed by patients with Parkinson’s disease. Via VR, it will be feasible for their treating physicians to monitor the patient’s progress remotely. The movements are constructed from raw data, which are provided by inertia and flex sensors embedded in a smart glove. The use of virtual reality may offer new methods for assessing the progress and severity of symptoms, as well as for assessing prognosis in a more personalized way. According to different scenarios, the physician will be able to employ the sensor data visualization along with VR, either for real-time diagnosis or for follow-up assessment, without any cameras. In this study, we will also examine the potential and challenges associated with the integration of virtual reality in the visualization of motion data of patients with Parkinson’s disease in order to provide better and more comprehensive care.

2. Background and Related Work

By exploring the applications that have been put into practice, we first sought to highlight the gaps in the field of virtual reality data visualization for Parkinson’s disease and the current state of knowledge in order to inform future research. This allowed us to create a synergy between virtual reality and visualization related to Parkinson’s disease.

2.1. Data Sources and Search Keywords

To understand the current state-of-the-art personalized visualization of the gestures of Parkinson’s disease patients using virtual reality, a comprehensive search of prominent academic databases, such as ScienceDirect, IEEE Xplore, and PubMed, was performed. The selected query “(Parkinson OR Parkinson gestures) AND Virtual Reality AND visualization” was aimed at capturing a wide range of relevant literature.

2.2. Analysis of the Literature

The analysis of the relevant literature was performed using the PRISMA guidelines, a rigorous design approach that aims to present the results objectively and clearly and helps to ensure that the review is systematic, transparent, and comprehensive. The PRISMA diagram is based on the question “(Parkinson OR Parkinson gestures) AND Virtual Reality AND visualization”.

The following results were found in the last ten years for the question posed to the search engines:

• The PubMed search engine found 51 articles;
• The IEEE Xplore search engine found 17 articles;
• The ScienceDirect search engine found 633 articles.

Of these, 75 duplicate or common content articles were identified. The aim of the research was to identify corresponding visualization applications using virtual reality, and for this purpose, only complete journal articles were selected. In the final evaluation, 30 integrated articles with similar applications and studies that helped identify symptoms of Parkinson’s disease were selected, as none of the articles were related in the experimental part to the corresponding application implemented in the paper.

2.3. Current State-of-the-Art: Conclusions

This literature review brings together findings from several studies on the applications of immersive technologies, such as virtual and augmented reality, in the treatment and evaluation of neurodegenerative diseases. The 30 comprehensive studies selected cover a wide range of applications of these technologies, focusing mainly on Parkinson’s disease (PD).

The findings of the bibliographic review were divided into three main themes. The first subject is where we find applications using immersive or modern technologies for neurodegenerative diseases. In this category interesting approaches were identified such as treatment of muscle tension in patients with Parkinson’s disease, using serious games and virtual reality in combination with an orthosis brachoid [9], a low-cost virtual reality interface to support the assessment and diagnosis of neurodegenerative diseases in a clinical setting [10], a virtual environment that provides misleading visual stimuli to subjects to induce imbalance events [11], use of evidence-based embedded virtual environments that integrate compensatory techniques like feedback and encouragement through guidance with movement learning [12], augmented reality for the assessment of motor dysfunction in the upper limbs of patients with Parkinson’s disease and stroke [13], rehabilitation of patients with Parkinson’s disease through the development of a serious game based on augmented reality [14], a functional capacity assessment tool with virtual reality (VRFCAT-SL) in clinical evaluations of patients with Parkinson’s disease [15], use of virtual reality as a tool to assess balance in patients with Parkinson’s disease [16], effects of visual feedback on the walking abilities of patients with Parkinson’s disease [17], regulatory control of body posture in patients with idiopathic Parkinson’s disease (IPD), using kinematics and frequency analysis [18], use of a virtual reality (VR)-based gait management strategy to improve gait symmetry in Parkinson’s disease patients exhibiting gait freezing [19], use of robotic gait training (RAGT) with a Lokomat device in the treatment of patients with Parkinson’s disease [20], a telerehabilitation program using a mobile virtual reality system to simultaneously train two patients with Parkinson’s disease in their homes [21], diagnosis of neurodegenerative diseases such as Parkinson’s disease using a low-cost virtual reality interface [10], the effect of dopaminergic therapy and visual flow modulation on distance estimation in patients with Parkinson’s disease [22], the use of a virtual reality system to simulate and adapt interior apartment features for people with Parkinson’s disease [23], the effects of upper extremity virtual reality exercises (ULVRE) on upper extremity sensorimotor function in patients with idiopathic Parkinson’s disease [24], the Internet of Medical Things (IoMT) with the development of an advanced system incorporating flexible sensors triboelectricity and deep learning-based data analyses [25], examining how Parkinson’s disease affects coordinated hand–eye movement and corrective response control during grasping movements of virtual objects, using augmented reality and 3D games to encourage patients with Parkinson’s disease and stroke to perform repetitive exercises near their physical limits in a safe environment [26].

The second topic consists of studies, approaches, and proposals on neurodegenerative diseases using immersive and modern technologies. In the findings of this topic, research was identified, such as how new technologies can affect the processing and visualization of medical data, with significant effects on correct diagnosis and treatment of various diseases [27], new perspectives on the medical management of chronic diseases, focusing in particular on physical activity and exercise as central to the management of neurodegenerative disorders affecting the mobility and mental status of the elderly using wearable technologies [28], effectiveness of virtual reality-based rehabilitation compared to conventional rehabilitation in the recovery of motor function in three groups of patients [29], a literature review on the use of virtual reality in the treatment of anxiety disorder in Parkinson’s disease [30], examination of the evolving field of virtual reality in rehabilitation, with a special focus on augmented virtual reality [31], the capabilities and needs of people with Parkinson’s disease in relation to the use of smart glasses [32], the effectiveness of virtual reality telemedical applications (Tele-VR) in improving cognitive performance and social skills in patients with Parkinson’s disease [33].

The third thematic section is general cases of addressing medical problems using immersive technologies. In the studies identified, the findings were as follows. Augmented reality can be integrated into the physical environment of the user, allowing interaction with virtual elements without the use of traditional interfaces such as screens and keyboards, including medical rehabilitation [34], exploiting augmented reality (AR), using free motion and body motion detection technologies, in order to develop interactive games for the uniform, cost-effective, and objective assessment of upper extremity motor dysfunction in various patient groups [35], augmented virtual reality for manipulation and evaluation of different motion learning mechanisms, with emphasis on error- and reward-based motion adaptation [36].

Despite the innovative and interesting approaches identified in the literature review, there is a clear lack of applications for data visualization of patients with neuro cephalic diseases such as Parkinson’s disease, as well as methods for remote and direct assessment of patients by physicians using immersive technologies that can offer analysis and natural realism. The aim of this paper is to visualize data from patients with Parkinson’s disease in a virtual reality environment, resulting in the remote assessment of patients by physicians.

3. Materials and Methods

The experimental protocol aims to visualize patient data related to neurodegenerative diseases, such as Parkinson’s disease. The aim of this project is to provide participants with a virtual simulation in a virtual environment where they can freely observe their movements and exercises. As the procedure is being recorded, the doctors will be able to view their patients’ conditions from a distance by visualizing the exercises they perform in virtual reality. As a result, patient diagnosis and rehabilitation can be completed remotely and without requiring further effort from the patients.

3.1. The SmartGlove

A specially designed glove was used to collect data from Parkinson’s disease patients. SmartGlove is a patent developed by members of the Human–Computer Interaction Laboratory as part of a research project. In this research, it was used to conduct the experiments and collect data. SmartGlove was developed to provide an undo-nominal solution for remote monitoring of motor symptoms of people suffering from neurodegenerative diseases such as Parkinson’s disease. The system consists of a smart glove (SmartGlove), shown in Figure 1, an application for smart devices (mobPark 1.2.59), and a central management platform [37].

SmartGlove consists of three main elements:

• The glove fabric, which includes special conductive areas in the fingers and palm;
• The system-on-chip (SoC), which includes a small-sized computer model with built-in Bluetooth 5.0 for low power consumption;
• The sensors, which record hand movements, finger flexion, finger contact, and a nine-axis inertial measurement unit (IMU) board to monitor hand rotation.

The whole design philosophy is shown in detail in Figure 2.

On the operational side, SmartGlove collects data in collaboration with the mobile tablet app and then uploads these data to a cloud platform, while the data can be transformed into a graphical output.

3.2. The PD Data Visualization Application in Virtual Reality

The main goal of the application is to create an integrated system that will allow doctors to monitor patients with Parkinson’s disease remotely. This system will also integrate the visualization of the patients’ gestures into virtual reality devices, enhancing immersion and providing more detailed visualization of the gestures.

3.2.1. The Application Data

The app was designed and tested on several categories of Parkinson’s disease-related metrics in conjunction with SmartGlove.

Participants were asked to perform certain tasks while wearing SmartGlove. After completing the exercises, the data were transferred to the system, where they were visualized in graphs and exported in different data types such as xlsx, csv, etc. For each participant, the following data were collected:

• Gyroscope measurements (X,Y,Z);
• Accelerometer measurements (X,Y,Z);
• Magnetometer measurements (X,Y,Z);
• Finger sensor measurements (Thumb, Index, Middle, Ring, Pinky).

An example of a graphical representation of a fist is shown in Figure 3.

3.2.2. Application Analysis

The application that was implemented was developed in Unity 3D application development software in version 2021.3.22f1 and was divided into two main parts, depending on the preference and equipment available to each user. In the first part, the application is accessible from a computer, while in the second part, the application works through virtual reality glasses.

This application is not exclusively addressed to specialists in the field but also to young researchers who wish to work in the field or patients who want to see the evolution of their condition. In this context, the main menu of the application contains a learning environment where the user can be informed about both virtual reality and Parkinson’s disease in a very realistic and immediate manner. The user has the option to read details about each topic and watch relevant videos, as shown in Figure 4.

After the update, the user is welcomed to the application and the virtual environment and is asked to select from a set of different types of databases on the corresponding description (e.g., fist data or hand rotation data, etc.). A small sample of these different types of data is shown in Figure 5.

After the user selects the type of data to be loaded into the system, the user is transferred to the central data simulation room.

To create realistic data, a virtual 3D visualization of a hand designed for virtual reality applications was used. The goal was to have as much detail as possible for each finger. The virtual hand consists of 20 separate parts (finger joints), with at least three parts for each finger. Due to its characteristics and its compatibility with virtual reality, this hand was an ideal choice for this application.

For the development of the application, 17 scripts in C# were implemented for data visualization in the virtual hand, and another 3 scripts related to the visualization rooms (see below) and the operation of the informative objects.

Data visualization was performed by the following methodology:

• Initialization of virtual hand positions;
• Transferring user data selection from the previous room;
• Data upload;
• Calculation of new positions based on data;
• Applying new positions to the virtual hand.

In summary, the application integrates the data controlled by the smart glove into the virtual environment and, once categorized according to the type of sensor they come from, they are applied to the 3D virtual hand with accuracy in both position and time. The goal of the application is to reflect the movements made by the participants using the glove identically in the virtual environment. The whole methodology of the application is shown in Figure 6.

In the first steps of the application, the possibility of adapting the data to any virtual hand was considered, this is performed by defining from the beginning the position where the virtual hand starts.

At the start of the application, as described above, the user, in addition to the educational information received, is asked to select which dataset they wish to run based on the corresponding category in the simulation room. The application consists of three rooms in total: the input–update room, the simulation room, and the motion room. Through PlayerPrefs, the possibility is given to transfer the user’s action to different rooms (scenes) so that the corresponding data can be loaded into the simulation codes.

Utilizing the data from the .csv files and the initial hand positions, data normalization and deviation calculations are performed, resulting in the next position (newz). Then, Unity’s Quaternion. Euler method is adopted to create a quaternion representing the rotation around the three basic axes (X,Y,Z) based on the Euler angles. Quaternions are mathematical structures widely used in 3D spaces to represent rotations. They are an extension of polygonometers, with some important properties that make them valuable tools in areas such as graphics, robotics, physics, and a host of other applications.

After the final calculation that takes place via this method, the new position is applied to the virtual hand, modifying the rotation of the transform.

The functionality of the visualization room is based on the analysis of the individual parts of the application. The accelerometer, gyroscope, and magnetometer data are considered and applied to the whole virtual hand (root), affecting all its individual parts (children and nested children). In contrast, the finger data are fed to each finger and each part separately, ensuring the optimal result. The simulation is implemented in slow view, allowing even the most detailed aspect to be highlighted, as shown in Figure 7. During the simulation, the user is able to freely explore the space and observe the virtual hand from any perspective and distance desired.

The application provides two functions. On the one hand, it allows doctors to view their patients’ exercises from a distance and accurately describe them. On the other hand, it has been enhanced with an additional room that allows participants to see their hands in real-time in the virtual world while performing the SmartGlove exercises. This gives the application an advantage because it records the data while participants see their hands in the virtual world, allowing for data comparison. The Leap Motion gadget (Ultraleap, Bristol, UK) makes it possible to view the participants’ hands in real time.

The Leap Motion Controller, a small device that can be placed on a desk or other surfaces, is designed to track hand and finger movements in a three-dimensional space. It uses infrared sensors and cameras to record movements with great precision. Integrating the Leap Motion Controller was a challenge, as issues such as compatibility of the device with SmartGlove, proper placement of the device on the VR headset, and the ability to modify the respective parts of the hand in relation to SmartGlove had to be addressed.

Considering the above, our first action was to 3D print a base for the Leap Motion, which would be mounted on the VR headset. The printing was performed on the 3D printer Ultimaker S3 (Ultimaker, Utrecht, The Netherlands), using PETG material. PETG is a type of polymer plastic with properties that make it ideal for numerous applications.

After printing was successfully completed, as shown in Figure 8, the base was integrated into the VR device. The ideal placement of the base was determined after several tests. It was found that placing it slightly above the center of the VR device ensures full hand coverage without restricting the built-in cameras (Figure 9).

In the last step, the creation of the environment and the representation of the hand was implemented. In the virtual room designed, the user was able to gaze at the skeleton of the hand, allowing detailed recording of the movements of each part of the hand. Additionally, the combination of SmartGlove and Leap Motion worked perfectly during the visualization, as demonstrated in Figure 10, where the comparison appears direct and understandable. Leap Motion was additionally used in the VR device as it offered not only the possibility to visualize the participant’s hand in real-time, but also the recording of the patients’ data simultaneously. As a result, the data can be recovered in case of sampling or transmission failure.

The experimental procedure allowed the recording of both the object in motion and the surrounding space at each stage. This provided the scientist processing the data with the possibility of visually comparing two different conditions.

In summary, there are many benefits to the developed system. From the perspective of the patient, SmartGlove allows them to record their condition even from home, and the addition of virtual reality facilitates accurate exercise execution for participants in the virtual environment. From the standpoint of the doctor, the innovation of combining data visualization in a virtual reality setting with remote patient monitoring allows for meticulous patient assessment and evaluation of their condition.

Overall, virtual reality benefited patients by lowering their levels of stress and euphoria while also providing highly accurate condition monitoring.

4. Results

During the implementation of the application, data from the SmartGlove platform was tested, mainly concerning patients with Parkinson’s disease. After the app testing was completed, the experimental procedure of all described stages was performed with new participants.

The application was tested by recording data from 20 patients with Parkinson’s disease. SmartGlove recorded the participants’ exercises. Each exercise was paused for 10 s before starting the next one. Because the exercises involved repetitive motions, such as making a fist three times, the duration of the exercises was dictated by the patient’s condition. Upon completion of the exercises, the data were uploaded to the SmartGlove platform. These data were imported into both the app and the visualization room, where an accurate representation of the participants’ exercises was presented through a virtual hand. In addition, during data collection, users were able to wear a projection device (VR headset) to monitor their exercises, while the Leap Motion device recorded participants’ movements simultaneously with SmartGlove. The interaction of the participants during their exercises is shown in Figure 11, and part of the experimental procedure is shown in Figure 12.

After the experimental procedure, participants were asked to evaluate the experiment. Initially, they evaluated the usability and wearability of SmartGlove, which scored an average of 9.77/10, which makes it quite easy and comfortable to use. Afterward, participants evaluated the virtual lobby, through which they were able to learn about Parkinson’s disease and also about virtual reality. This room scored an average of 9.87/10 in terms of its ease of use and its usefulness. Finally, participants were able to evaluate the Leap room, which received an average score of 9.97/10, which indicates that the usability of the room is quite high, as well as the level of realism it offers. A summary of the participants’ scores is shown in Figure 13. It is worth mentioning that during the experimental procedure, the patients did not encounter any problems or any form of refusal regarding the use of virtual reality.

In the second stage, the data and the application were evaluated by 7 medical neurologists through a comprehensive questionnaire.

The 11 questions on the questionnaire were broken down into two themes: questions about the general use of virtual reality and questions about the application and visualization of the data. The rating was done with points from 0, which stated NOT AT ALL, to 10, which stated COMPLETELY.

The results of the questionnaire are shown in Figure 14. The findings state that when it comes to the use and interaction of the virtual environment (first theme), the physicians scored an average of 9.57 and stated that they had a pleasant experience using the environment and did not encounter any serious problems. Regarding the visualization of the data and the comparison with the in-person experimental diagnosis process, physicians stated an average of 9.64, meaning that the visualization was completely realistic and that the application offers a solution for the remote diagnosis and evaluation of patients with Parkinson’s disease and other neurodegenerative patients.

5. Conclusions

Personalized visualization of the gestures of patients with Parkinson’s disease using VR presents a new avenue for enhancing the management and understanding of this complex neurological condition. By integrating immersive technologies with personalized data visualization, healthcare professionals and patients can gain deeper insight into the nuances of symptoms, progression, and responses to treatment of Parkinson’s disease.

This work presents an innovative use of virtual reality to understand and monitor Parkinson’s disease, offering new tools and methods that enhance the personalization of treatment and deepen the study of patients’ motor behavior. Its main contribution is that it proposes a new way of understanding and monitoring disease progression through direct visualization and the use of virtual reality, which can positively impact the treatment and management of Parkinson’s patients.

The application developed in this work uses the potential of immersive technologies, such as virtual reality, for the diagnosis and rehabilitation of patients with Parkinson’s disease. SmartGlove was an influential tool for the development of the application, as it collects data from hand movement, as is the Leap Motion device, for visualizing and recording hand movements. Data analysis and processing were important factors for the correct visualization with high accuracy of hand movements. The integration of exercises with real-time visualization of the patients’ hands provided an innovative approach, which was accepted by the patients, as the surroundings provided an atmosphere of relaxation and calmness. Likewise, because medical professionals could wear virtual reality headsets and see their patients’ movements as though they were happening in front of them, they were able to monitor their patients’ conditions remotely and provide helpful recommendations. In short, the dual nature of the application enables participation in a virtual reality environment both for patients, who can see their exercises and their hands in this environment, and for doctors, who can monitor the condition of their patients.

The use of personalized imaging in VR opens up new potential for personalized interventions and rehabilitation strategies. By visualizing patient-specific data in a virtual environment, clinicians can design targeted treatment programs that address individual motor impairments, cognitive challenges, and psychosocial needs. This personalized approach has the potential to improve treatment outcomes and quality of life for these patients.

Personalized imaging empowers patients with Parkinson’s disease to actively participate in their treatment and gain a deeper understanding of their condition. Through immersive VR experiences, individuals can visualize their own movements, track progress over time, and make informed decisions about their care in collaboration with healthcare providers. However, the denial identified by patients with neurodegenerative diseases due to their condition, to the use of immersive technologies and newer methods of diagnosis and rehabilitation, remains to be eliminated.

The application implemented in this paper incorporates the principles of Digital Twins. Digital Twins are a digital replica of a real object or system, updated in real-time, for monitoring, prediction, and optimization, which is embodied in the application that has been implemented. Our future goal is to build an integrated system that can provide feedback and evaluation on patient progress using artificial intelligence and data sources.

The findings of this study highlight the transformative potential of personalized imaging in the field of Parkinson’s disease research and healthcare delivery. By harnessing the potential of virtual reality and personalized data visualization, researchers can gain deeper insights into disease mechanisms, biomarker identification, and treatment efficacy, paving the way for more targeted interventions and improved patient outcomes.