Definition of Motion and Biophysical Indicators for Home-Based Rehabilitation through Serious Games

Abstract

In this paper, we describe Remote Monitoring Validation Engineering System (ReMoVES), a newly-developed platform for motion rehabilitation through serious games and biophysical sensors. The main features of the system are highlighted as follows: motion tracking capabilities through Microsoft Kinect V2 and Leap Motion are disclosed and compared with other solutions; the emotional state of the patient is evaluated with heart rate measurements and electrodermal activity monitored by Microsoft Band 2 during the execution of the functional exercises planned by the therapist. The ReMoVES platform is conceived for home-based rehabilitation after the hospitalisation period, and the system will deploy machine learning techniques to provide an automated evaluation of the patient performance during the training. The algorithms should deliver effective reports to the therapist about the training performance while the patient exercises on their own. The game features that will be described in this manuscript represent the input for the training set, while the feedback provided by the therapist is the output. To face this supervised learning problem, we are describing the most significant features to be used as key indicators of the patient’s performance along with the evaluation of their accuracy in discriminating between good or bad patient actions.

1. Introduction

In recent years, the impact of telemedicine is still growing thanks to the adoption of the concept of “continuity of care” by the health community. Operators have access to all needed information, regardless of where they are located, and this helps to improve access to health services for patients and end users [1].

Societal changes drive the evolution of the healthcare sector, both in Europe and in the rest of the world. Some of the big issues including the ageing population, chronic diseases, and spread of disabilities (e.g., stroke and all degenerative diseases) have a great impact on health care strategies and care solutions. Thanks to the new technological solutions, the patient is increasingly attentive and aware to their state of health, trying to maintain the highest quality of life level, which requires continuous assistance and care [2].

Digital innovation is involved in many areas of daily life. It plays a fundamental role in the field of medical rehabilitation, and more and more facilities consider it as a fundamental part of their therapeutic and commercial offering. On the market, there are several gaming consoles used for fitness exercising. In particular, the Nintendo Wii is used in several centres as a rehabilitation device, to help the recovery of patients in a more enjoyable way [3,4]. Nevertheless, commercial games are not designed for people with disabilities, therefore making the patient experience demotivation when failing to achieve the goals due to their difficulty.

In this scenario, we are disclosing the Remote Monitoring Validation Engineering System (ReMoVES). We have developed a platform that addresses the problem of continuity of care in a smart and cost-effective way. It relies on serious games with which the patient interacts through Microsoft Kinect and Leap Motion. [5].

Many researchers developed similar solutions, but they focused on algorithms to detect body and hand gestures with a certain level of accuracy [6,7,8] and on the development of serious games for health [9]. In this article, which is an extended version of [10], we present a novel approach to automatically evaluating the performance of the patient during the execution of rehabilitation exercises when unsupervised by therapists. The methods we will use to process the data collected during the entire activity session are discussed in detail.

This paper is structured as follows. Section 2 presents the introduction to the system architecture and presents the characteristics of the serious games that have been developed and included in the ReMoVES platform. Section 3 explains the methods to process data acquired during the activity sessions carried out by patients. Section 4 presents some preliminary results with a sample of the data process. Finally, Section 5 presents a discussion of the results.

2. Description of the System

2.1. Overall Architecture

The ReMoVES platform employs three off-the-shelf devices for motion tracking and biophysical data acquisition which are activated during the execution of functional exercises. On the back-end, a cloud architecture was designed and deployed to provide web-services and data processing.

The idea behind the proposed architecture consists of providing a personal rehabilitation program that is performed at home by the patient themselves, while the therapist can track the performances and effectiveness of the training from any Internet-connected device. Among others, the monitoring of eventual compensation movements allows evaluation of whether an exercise is accomplished in the correct way.

In detail, the built-in algorithms aim to provide a clear and concise report to the therapist, in order to facilitate the interpretation of the evolution of therapy. According to Figure 1, the following technologies are deployed by the ReMoVES platform.

The motion and biophysical values originating from the serious games and the peripherals are stored in the database at 2Hz in an integer or float numeric data type (four bytes, single-precision) along with a timestamp to identify when that event occurred (it is generated on the machine running the serious games, not on the cloud back-end since this would be affected by unpredictable delay). The raw measures from the sensors are provided with different data types and are stored without rounding as reported in

Table 1. Details about the motion and biophysical data collected during the activity: unit of measurement, typical range of values and data type in the database.

Description	Unit	Range	Data Type	Notes
Heart Rate	seconds	from 2 to 0.24	float: 5 dec.	inter-beat interval
Electrodermal activity	kohms	from 0 to INF	integer
Score			integer
Joint position	meters	from −5 to +5	float: 7 dec.	for each X, Y, Z coordinates
Yaw, roll, pitch	radians	from 0 to 2$\pi$	float: 7 dec.
Grab strength		from 0 to 1	float: 7 dec.	0 palm open, 1 hand closed

.

• Serious games are digital games that were completely developed in our labs exclusively for the ReMoVES platform. The system currently includes 10 main serious games and 60 variations: they encourage the patient to carry out functional exercises autonomously along with the traditional motion rehabilitation. A representative screenshot for each serious game is given in Figure 1 (from a to i) and in Figure 2. The creation of these activities involved different processes, technologies, and specialists. In the case of serious games, special attention must be paid to the specifications given by physiotherapists and physiatrists, who shared their expertise to define game parameters such as the level of difficulty. The interface was designed using the Unity3D platform, which is a popular engine often used for developing games and virtual reality applications. The games are scripted in C#. All art assets in both 2D and 3D formats were downloaded from different online sources under Creative Commons license [11]. The whole set of games shares a core library that were developed to allow easy and straightforward communication with the Application Programming Interfaces (APIs) exposed by the cloud back-end that will be presented later.
• Microsoft Kinect V2 is a motion sensing input device based on a high-resolution colour camera and an infrared emitter for depth analysis that can simultaneously 3D-track up to 25 fundamental joints of the framed human body. It offers a wide field of view (70 × 60 degrees) and recognition up to 4.5 m from the device [12]. Several studies have demonstrated that spatiotemporal parameters can be validly obtained by Microsoft Kinect V2 [13,14]. The set-up of the interface between the Microsoft Kinect V2 and the Unity3D engine is effortless because the manufacturer provides a Software Development Kit (SDK) and a Unity add-on, which gives developers access to body joint positions and orientations that can be used directly in rehabilitation game development.
• Leap Motion is explicitly targeted at hand gesture recognition and computes the position of the fingertips and the hand orientation. Its interaction zone is limited to a semi-sphere of radius 0.60 m around the device, with a theoretical accuracy of 0.01 mm according to the manufacturer (or 0.7 mm under real conditions as measured by [15]). Examples of clinical studies that have used the Leap Motion controller are [16,17]. The manufacturer provides a Unity Assets bundle to allow a straightforward interface with the development engine.
• Microsoft Band 2 is a physiological sensor used to collect the two signals chosen as a measure of involvement during the rehabilitation program: heart rate (HR) and electrodermal activity (EDA). It also includes a wide range of additional built-in sensors: accelerometer, gyroscope, barometer, GPS, skin temperature sensor, ambient light, and ultraviolet sensor. Microsoft Band 2 offers the opportunity to collect real-time data in a continuous manner for at least 4 h without recharging [18]. Our interface with Microsoft Band 2 is implemented using the Band SDK for the Universal Windows Platform in C#.
• Cloud back-end was developed using state-of-the-art techniques to provide scalable, secure, and efficient data processing and storage. In Figure 1 it is schematically represented as the Data Processing cloud. The architecture of this component is not the subject of this paper, but it plays a fundamental role since it exposes REpresentational State Transfer (REST) APIs for all-around management (authentication, initialization, data storage) of the activity sessions which can take place remotely and simultaneously in different locations (rehabilitation centres, hospitals, or patients’ homes). In detail, the ReMoVES cloud back-end was developed with PHP 5 and is hosted on an Apache/2.4.6 web server running CentOS 7. The JavaScript Object Notation (JSON) standard has been adopted for encrypted (over SSL) message communication between the cloud back-end and other components. Data is stored in a MySQL relational database running on the same machine: it is made up of tables for users management, rehabilitation sessions scheduling and game log storage. The motion and biophysical values originating from the serious games and the peripherals are stored in the database at 2 Hz in an integer or float numeric data type (four bytes, single-precision) along with a timestamp to identify when that event occurred (it is generated on the machine running the serious games, not on the cloud back-end since this would be affected by unpredictable delay). The raw measures from the sensors are provided with different data types and are stored without rounding as reported in Table 1.

2.2. Serious Games

The rehabilitation activity proposed through the ReMoVES platform is used together with the traditional rehabilitation program. The system proposes a set of exercises from which the therapist can choose the most useful ones according to the patient disease and disability level.

These activities are presented to the patient as serious games in the specific sequence defined by the personalised rehabilitation plan designed by the therapist.

The system currently includes 10 different exercises that can be tweaked according to the requirement of the patient: level, duration, movement pattern, speed, and other parameters can be changed for a total of 60 game variations. They are reported in

Table 2. Serious games list including the purpose of the activity, the employed devices, and the indicators that describe the patient performance during the exercise.

Activity Name	Activity Purpose	Devices	Performance Indicators
BreathBall	Relaxation	Microsoft Band 2	Heart Rate, Angular coefficient ${\beta }_{HR}$
OwlNest	Upper limb rehab	Microsoft Kinect V2	Score, Compensation, Barycenter, Target position
ShelfCans	Upper limb rehab	Microsoft Kinect V2	Score, Compensation, Hand trajectory
FlappyCloud	Lower limb rehab	Microsoft Kinect V2	Score, Compensation, Elevation
EquilibriumPaint	Sit to Stand	Microsoft Kinect V2	Repetition rate, Repetition trend, Coronal shift, Shoulder shift
HotAir	Balance shift	Microsoft Kinect V2	Score, CoP variation, Compensation
EndlessZig	Hand rehab	Leap Motion	Score, Repetition rate, Range of Motion: yaw
CityCar	Hand rehab	Leap Motion	Score, Repetition rate, Range of Motion: pitch
WineBottle	Hand rehab	Leap Motion	Score, Repetition rate, Range of Motion: roll
FloatingTrap	Hand rehab	Leap Motion	Score, Repetition rate, Grab strength

and briefly described in the following.

• BreathBall This activity (see Figure 2) helps the patient to relax and focus on their breathing. It is not an interactive game. The patient is led to breathe with a regular rhythm by following a “breath ball” displayed on the monitor. The BreathBall exercise makes the subject comfortable and brings back their heartbeat to a basal value before the actual rehabilitation session starts.
• ShelfCans This serious game (Figure 1a) introduces the patient to a virtual environment similar to a kitchen. With the arm movement, the patient grabs one of the colourful drink cans appearing in the middle of the screen and drags it to the corresponding shelf. This game is appealing because it requires the user to be attentive to drop off the drink can in the correct shelf according to its colour.
• OwlNest The patient is encouraged to reach an on-screen target (Figure 1b) with the arm motion (Reaching Task) in order to achieve a high in-game score. Many colourful owls appear randomly in any position of the screen for a given time-frame: the user should carry them in the nest before they disappear.
• FlappyCloud This is a functional exercise for the lower limb. The leg Abduction–Adduction movement reflects the position of a cloud object in the game screen: the patient should make it move forward without hitting some obstacles (Figure 1c).
• HotAir This is an activity to improve control of the patient’s body balance. The user can control the direction of a hot-air balloon, floating in the sky with the balance shift: in-game scores are collected when it is led towards the bonus targets (Figure 1d).
• EquilibriumPaint This serious game is an interactive version of the Sit to Stand (STS) exercise, typically used in traditional rehabilitation to evaluate patient performance. The user should stand up and sit down repeatedly within a predefined amount of time (usually 30 or 60 s [19]); their trunk must remain erect: an erroneous lateral shift causes the fall of the cans of paint leaning on an unstable wooden beam (Figure 1e).
• WineBottle This exercise mimics a real-world scenario: pouring liquids from a bottle. With the Pronation–Supination movement of the hand, the patient should control the rotation of a bottle of wine appearing on the screen. They must fill a glass over and over again to collect as many points as possible (Figure 1f).
• EndlessZig In this activity, the patient drives a marble along a zigzag path appearing on the screen. Going out of the boundaries causes score loss; similarly, some bonus gems appear on the path. The patient controls the marble movement with Radial–Ulnar deviation (Figure 1g).
• CityCar In this game, the patient drives a car along a randomly-generated road. The user should steer in the presence of curves and crossroads with the movement of Flexion/Extension of the wrist. Penalties are introduced when the user goes off-track (Figure 1h).
• FloatingTrap In this serious game, the patient is led to open their hand and make a fist alternatively. This exercise requires a good level of concentration: in fact, the user moves a floating raft on the left or on right according to the finger Flexion/Extension in order to avoid some objects in the scene (Figure 1i).

3. Features Description

The core of the ReMoVES platform consists of the remote delivery of a report to the therapist, through which she/he can understand if the home-based rehabilitation process proposed to the patient is effective and carried on correctly [10].

To achieve this goal, we are here introducing a study about the most significant indicators which can be processed by some machine learning algorithms for automatic evaluation of a patient’s performance rehabilitation session.

In this section, we focus on the definition of appropriate indicators aiming to highlight the patient engagement and performance during the execution of the functional exercises.

To this end, the therapist feedback that judges the performance of each session carried out by patients was collected at rehabilitation facilities. This enabled the preparation of a labelled dataset which was used to test the significance and validity of the defined features.

In detail, data are displayed in the graphs of this section which are provided as examples dealing with the problem of discriminating between sessions which were correctly performed (labelled by the therapist as “Good”) and the ones which were inadequately executed (labelled as “Other”). Thanks to such examples, we are able to evaluate if the selected features have discriminant power, as expected by their definition. Such a preliminary phase gives an idea about the best feature, although a larger dataset must be created for a more robust statistical evaluation.

Even though more sophisticated methods can be exploited for the feature normalisation step, in the present study we experienced that simple feature scaling and mean normalisation methods were sufficient. The standardisation of each feature range herein has the purpose of giving the same weight to the various indicators. Mean normalisation involves subtracting the average from the values of an input variable, resulting in a new variable with zero mean. At the end, we expect values between −1 and +1.

In the following, the indicators specifically designed for each exercise are described along with an evaluation of their classification power. A summary is provided in Table 2.

3.1. Relaxation

To make the subject comfortable and to return their heartbeat to a basal value before the rehabilitation session, the patient is led to breathe with a regular rhythm by following a “breath ball” displayed on the monitor. The ball changes its size, simulating inspiration and expiration (Figure 2): when it grows, the patient should inhale, then exhale when it shrinks. When a patient had difficulty following the ball rhythm, we asked them to relax and not strain themselves.

During the execution of this exercise, the Heart rate (HR) signal is acquired. The trend analysis is performed, and the Angular coefficient ${\beta }_{HR}$ of the straight line that best approximates the heart beat is computed, in order to understand the general pattern of the signal. The patient can be considered ready for the execution of physical rehabilitation if ${\beta }_{HR}$ is equal to or less than zero. In fact, a positive coefficient means a state of fatigue and excitement, not suitable for the aim of the rehabilitation process.

3.2. Upper Limb

Microsoft Kinect V2 was deployed in this analysis: the patient interacting with digital games was encouraged to reach an on-screen target with the arm motion (reaching task) in order to collect a high in-game score. The main movements at the shoulder joint (glenohumeral joint) tracked during the execution of functional exercises were: Flexion–Extension and Abduction–Adduction.

Two serious games for upper limb are available in the ReMoVES platform: ShelfCans and OwlNest.

During the execution of the rehabilitation exercise, we extract a large amount of data, but we are currently focusing on these features for the upper limb:

• Score: the score collected in-game is an abstract quantity associated to the player performance for their achievements. It acts as a quantitative indicator of success in the game introduced to motivate patients to attain a better score than their opponents/themselves.
  The Score feature is the ratio between the score collected by the user and the theoretical maximum achievable in the game:

  $$\begin{array}{c}Score=scor{e}_{user}/scor{e}_{max}.\end{array}$$

  (1)
• Hand trajectory: analysis of the hand trajectory is an indicator of the precision and the movement control level that the patient can keep during the activity. We calculate the distance between the trajectory (N samples over time $han{d}_i=(x_i,y_i)$, where x and y are the horizontal/vertical coordinates of the coronal plane) and the shortest line passing through the two points of $target=(x_t,y_t)$ (target spawn position) and $origin=(x_o,y_o)$ (Figure 3b):

  $$\begin{array}{c}Handtrajector{y}_{distance}=\frac{1}{N}\sum _{i=1}^n\frac{(y_t-y_o)x_i-(x_t-x_o)y_i+x_t{y}_o-y_t{x}_o}{\sqrt{{(y_t-y_o)}^2+{(x_t-x_o)}^2}}.\end{array}$$

  (2)
• Compensation: compensation is an undesired movement strategy adopted by the patient during the execution of the exercise. In detail, the distance d between the hand and the shoulder (Figure 3a) while performing the movement is evaluated in order to assess whether the patient reaches the targets by performing a correct Flexion–Extension or Abduction–Adduction movement or moves the whole body instead, implying that the hand-shoulder distance remains constant.
  Let us define $\mu$ the average of distances $d_i$ with respect to the N observations during a single session; the Compensation parameter, index of the wrong movement, is then extracted as follows:

  $$\begin{array}{c}\hfill Compensation=-\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(d_i-\mu \right)}^2}.\end{array}$$

  (3)
• Barycenter: this indicator deals with the detection and measurement of a spatial negligence in hemiparetic patients who ignore the targets that are offered to their attention in a certain part of the in-game space. Spatial negligence affects approximately 30–40% of patients with acute stroke and consists of ignoring part of the space as a result of brain damage. The hand coordinates on the coronal plane, acquired over time, are used to populate a visual heatmap from which the therapist can evaluate the neglected areas; the spatial map is presented with a colour scale ranging from dark blue, where no movements were performed, to red, where there was the greatest occurrence (Figure 3c). This indicator is provided by the horizontal offset of the barycenter of the hand position observations over the session duration. In the evaluation of this indicator, it is also important to consider that in some variations of the OwlNest the targets are not equally distributed on both sides (e.g., to force a patient affected by left hemiparesis to focus his attention on the left side of the screen only).

For each serious game we will build a dataset that allows detection of when the patient is achieving good performances during the rehabilitation process. The data collection process takes place in a rehabilitation structure where a therapist provides live feedback about the patient’s performances. This will provide a dataset for supervised learning in the machine learning method.

In the OwlNest game, we are evaluating these relations: Score–Compensation and Barycenter–Target positions (Figure 4). In the former case, a good performance is achieved with a high Score and low Compensation. In the latter, a good performance is achieved when there is a linear dependency between the position of the hand and the position where the target on the screen spawned. With this analysis, we can discern when a shift from 0 in the Barycenter feature is caused by the patient’s syndrome or just by a bias in the positions where in-game targets (the coloured owls) are generated. For example, when most of the targets are generated in the left part of the screen, it is totally correct to collect a negative (left) Barycenter value and this is not an evidence of spatial negligence.

For the ShelfCans game we are evaluating the relations: Score–Compensation and Hand trajectory–Compensation (Figure 5). The first case was already depicted for the OwlNest game. The latter highlights good performances when the hand follows a straight trajectory matching the shortest path as much as possible, while the functional exercise is carried out without strategies of compensation.

3.3. Lower Limb

Microsoft Kinect V2 is deployed in this kind of activity. The user must accomplish the Abduction–Adduction movement with their impaired leg while they are standing. The therapist can suggest to the patient to hold on to a physical support, such as the back of a chair, in order to prevent falls or equilibrium instability. The elevation of the leg and additional body joint positions are tracked with the Microsoft Kinect V2 to detect the balance status of the user’s body: to accomplish the exercise correctly, the patient must keep the trunk erected and straight.

FlappyCloud is currently the only serious game included in the ReMoVES platform that is developed around a functional exercise for the lower limbs.

During the execution of the rehabilitation exercise, we extract a large amount of data, but we are currently focusing on these features for the lower limbs:

• Score: same as previous activities.
• Elevation: it is the maximum elevation of the impaired leg calculated as the angle between legs during the Abduction–Adduction movement. The Elevation feature is the ratio between the maximum angle reached by the user and the maximum reached on average by healthy subjects:

  $$\begin{array}{c}Elevation=elevatio{n}_{user}/elevatio{n}_{healthy}.\end{array}$$

  (4)
• Compensation: to accomplish the exercise correctly, the patient must keep the trunk erected and straight. In this case, the angle between the patient’s trunk and the ideal vertical attitude is considered.

  $$\begin{array}{c}Compensation={\measuredangle }_{trunk,vertical}.\end{array}$$

  (5)

For the FlappyCloud game, we are evaluating these relations: Score–Compensation and Elevation–Compensation (Figure 6). The first case was already depicted for other games. In the latter case, a good performance is achieved when the patient collects high Elevation value during the Abduction–Adduction while keeping their truck erected and straight.

3.4. Sit to Stand

Microsoft Kinect V2 was deployed in this analysis: the patient must stand up and sit down repeatedly, keeping the trunk as straight as possible. The serious game developed around the STS functional exercise is EquilibriumPaint.

Among the possible parameters we extracted, the following features were taken into account for Sit to Stand evaluation:

• Repetition rate: number of repetitions in a given time; it is proportional to the in-game score. The Repetition rate is the ratio between the rate maintained by the user and the average kept by healthy subjects:

  $$\begin{array}{c}Repetitionrate=rat{e}_{user}/rat{e}_{healthy}.\end{array}$$

  (6)
• Repetition trend: the trend of the time required by the user to accomplish a single STS cycle. The analysis of the variability during a single session is, for example, important to detect the fatigue level of the patient. Given i the repetition number and $t_i$ the cycle duration, the Repetition trend is the slope (m) of the linear regression of $t_i$ with respect to i:

  $$\begin{array}{cc}\hfill m,q& =LinearRegression(i,t),\hfill \\ \hfill Repetitiontrend& =m.\hfill \end{array}$$

  (7)
• Coronal shift: verification of the correct strategy used by the patient to stand up, considering the trunk tilt on coronal plane (Figure 7a). Even though the movement trajectory projected in the sagittal plane must show some forward/backward displacement, the correct coronal trajectory must be linear and constant around the zero value.
• Shoulder tilt: to accomplish the exercise correctly, the patient must stand up and sit down without twisting or tilting the shoulders. A correct movement provides very small angles of inclination with a tolerance of about 10 degrees, and therefore the average value must be around 0 degrees (Figure 7b).

For the EquilibriumPaint game, we are evaluating these relations: Repetition rate–Coronal shift and Repetition trend–Shoulder tilt (Figure 8). In the former case, a good performance is achieved when the patient collects high in-game Score while keeping the Coronal shift around zero during the STS exercise. In the latter, a good performance is achieved when the Repetition Trend remains constant (or even increases) during the whole session, while the average value of Shoulder tilt remains around 0 degrees.

3.5. Balance Shift

The Nintendo Wii Balance Board (WBB) has already been validated as a suitable alternative to a laboratory-grade force platform in the measures of Centre of Pressure (CoP) in contexts where a clinical assessment of balance is required without an elevated precision level [20,21]. The ReMoVES platform does not employ the WBB despite its adequate performances in the serious games context; in fact, according to the preliminary studies we have carried out, the Microsoft Kinect V2 can be exploited for the estimation of the oscillation of the CoP on the mediolateral axis and WBB would be redundant and anti-economic. The following procedure will disclose the method for the CoP estimation using the Microsoft Kinect V2:

• Track the position in space of these fundamental body joints provided by Kinect SDK: right ankle ($A_r$), left ankle ($A_l$), the base of the spine ($S_{bottom}$), and the spine at the shoulder height ($S_{top}$) (Figure 9a).
• Compute the position in space of the middle point (M) between the right ankle and the left ankle.
• Compute the difference between the angles related to the shift of lower and higher sections of the trunk:

  $$\begin{array}{cc}\hfill Til{t}_{bottom}& =\measuredangle (A_r,M,S_{bottom})-\measuredangle (A_l,M,S_{bottom}),\hfill \\ \hfill Til{t}_{top}& =\measuredangle (A_r,M,S_{top})-\measuredangle (A_l,M,S_{top}).\hfill \end{array}$$

  (8)
• Calculate the average between the $Til{t}_{bottom}$ and $Til{t}_{top}$ to get the estimated CoP:

  $$CoP=\frac{Til{t}_{bottom}+Til{t}_{top}}{2}.$$

  (9)

We used this experimental setup to evaluate the performance of the proposed algorithm: three subjects repeated a 2-min-long serious game for postural assessment ten times. Data were synchronously recorded from WBB and Microsoft Kinect V2 and the CoP oscillation on the mediolateral were normalised and compared calculating the Pearson correlation coefficient. The average of the coefficients calculated for each test sessions brought the result of ∼0.89. Figure 10 shows a sample of synchronised and normalised data in a 2-min-long test session.

The serious game focused around the balance shift functional exercise is HotAir. During the execution of the rehabilitation exercise, we extract a large amount of data, but we are currently focusing on these features for balance shift:

• Score: same as for upper limb games.
• Centre of pressure variation: during the Balance shift functional exercise, the patient must load weight on the two legs alternately, and thus the variation of CoP on the mediolateral axis is calculated.

  $$\begin{array}{c}Co{P}_{variation}=-\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(Co{P}_i-\mu \right)}^2}.\end{array}$$

  (10)
• Compensation: to accomplish the exercise correctly, the patient must alternate the load on the lower left and right limbs without compensating the movement by shifting the upper part of the body. This erroneous strategy can be detected by computing the difference between $Til{t}_{bottom}$ and $Til{t}_{top}$ (Figure 9c):

  $$\begin{array}{c}Compensation=Til{t}_{bottom}-Til{t}_{top}.\end{array}$$

  (11)

For the HotAir game, we are evaluating these relations: Score–CoP variation and CoP variation–Compensation (Figure 11). In the first case, a good performance is achieved when the patient collects a high in-game Score while their effort in shifting the load between left and right limbs is high. In the second analysis, the correct performance is achieved with high CoP variation and low Compensation.

3.6. Hand Movements

Leap Motion was deployed in this analysis: the digital games encourage the patient to control in-game objects with hand, wrist, and finger motion. The main movements tracked during the execution of functional exercises are: Radial–Ulnar deviation (Figure 12a), Flexion–Extension (Figure 12b), Pronation–Supination (Figure 12c), and Grasping (Figure 12d).

From the motion data, we extract these fundamental features:

• Repetition rate: same as previous activities.
• Maximum angle: maximum angle reached in each of the proposed movement (corresponding to yaw, pitch, and roll). These values are tracked, respectively, in EndlessZig, CityCar, and WineBottle games.
• Grab strength: evaluation of the grasping exercise with Grab strength function provided by Leap Motion SDK that returns values between 0 and 1 respectively when the hand is open or when it is closed in a grabbing pose. This feature is currently tracked in the FloatingTrap serious game.

3.7. Heart Rate and Electrodermal Activity

During daily rehabilitation therapy, it is very important to monitor the patient’s emotional involvement in order to inform the therapist about the proper performance of the assigned plan of care [22]. For this reason, heart rate (HR) and electrodermal activity (EDA) were recorded and analysed to monitor major changes during the different phases of a rehabilitation session. These data were compared with those from motion sensors to monitor the progress of therapy and the patient’s involvement in the rehabilitation session.

The HR is an informative biological parameter that is easy to record and therefore to introduce in the home setting. In other words, a simple bracelet sensor is sufficient to understand the patient’s condition during the rehabilitation program. By analysing the heartbeat signal, it is also possible to detect if the patient is too fatigued because the activity is excessively difficult or the duration established by the therapist is inappropriate. In the literature, there are different works which indicate that HR and heart rate variability (HRV) may be useful for measuring the valence of emotions, like joy, sadness, and fear [23,24,25].

Both HR and EDA were chosen as biophysical signals to be correlated with activity performance in order to detect a particular state of stress during the rehabilitation process.

The EDA is defined as a change in the electrical properties of the skin [26]. The EDA signal is composed of both tonic and phasic components. The slowly varying base signal is the tonic EDA part, also called the skin conductance level (SCL). The faster-changing part (phasic activity) is related to external stimuli or non-specific activation, and it is called skin conductance response (SCR). Recently, EDA has been used as a biofeedback mechanism in order to teach meditation or emotional control and to treat anxiety disorder [27]. The EDA signal was also widely used as an indicator of the actual state, related in particular to changes in the arousal level. Many studies reported that magnitude of electrodermal change and the intensity of emotional experience are almost linearly associated in arousal dimension. For example, in [28], the authors used EDA and HR to evaluate immersive virtual reality environment, finding significant results of correlation between the two signals and the environment.

In this work, by fixing an observation window W, corresponding to each exercise duration, standard statistical parameters, such as mean, variance, maximum and minimum value, and the Pearson correlation coefficient between HR and EDA, were computed.

For the EDA signal, in order to obtain the phasic data by removing the tonic level of the signal, unrelated to arousal, a median filtering process was applied. The median EDA value of the surrounding samples based on a ±4 s interval centred on the current sample was computed for each sample and the average subtracted from the current sample. After obtaining the phasic data, a peak detection algorithm was applied in order to find skin conductance responses to external stimuli that can influence patients during the execution of rehabilitation activity. A value of 0.01 $\mathsf{\mu }$S was chosen as a significant threshold to discriminate skin conductance variations. The same processing method was applied to the HR signal.

To evaluate the general pattern of the two signals, a trend analysis was computed and evaluated by applying the least-squares method. After applying it, the slopes of the straight lines, ${\beta }_{HR}$ and ${\beta }_{EDA}$, were calculated.

As explained above, HR and EDA are, respectively, indexes of valence and arousal. Once the two signals were normalised between −1 and 1, the $\beta$ variable and $\theta$ angle were used to define the emotional index ($Ie$) as follows:

$$\begin{array}{c}H{R}_{norm}=\left\{h{r}_1,h{r}_2,\dots ,h{r}_n\right\},n=\left|HR\right|=\left|EDA\right|,\hfill \\ ED{A}_{norm}=\left\{ed{a}_1,ed{a}_2,\dots ,ed{a}_n\right\},\forall i=1,\dots ,n,\hfill \\ {\theta }_i=arctan\left(h{r}_i,ed{a}_i\right)\hfill \\ \mathrm{IF}{\theta }_i<0\Rightarrow {\beta }_i=\left|{\theta }_i\right|+\frac{\pi }{2},\hfill \\ \mathrm{IF}{\theta }_i\le \frac{\pi }{2}\Rightarrow {\beta }_i=\frac{\pi }{2}-{\theta }_i,\hfill \\ \mathrm{IF}{\theta }_i>\frac{\pi }{2}\Rightarrow {\beta }_i=\frac{5}{2}\pi -{\theta }_i,\hfill \\ \forall i=1,\dots ,ni{e}_i=1-\frac{{\beta }_i}{\pi },\hfill \\ Ie=\left\{i{e}_1,i{e}_2,\dots ,i{e}_n\right\}.\hfill \end{array}$$

(12)

The emotional index was represented as a graphical parameter (Figure 13) by evaluating how strong the emotion was experienced in a given second in relation to all the emotions perceived throughout the whole rehabilitation activity.

The two normalised signals $H{R}_{norm}$ and $ED{A}_{norm}$ were also represented in the Valence–Arousal circumplex chart. The valence (HR variation) is represented on the x-axis and the arousal (EDA variation) is represented on the y-axis (Figure 14). Emotions can in fact be mapped out, classifying arousal (from high to low) and valence (from pleasure to displeasure) experienced during a particular task. In the right part of the graph, positive emotions are related to a patient which is experiencing happiness, serenity, calmness, and relaxation. On the contrary, the left part contains negative emotions like sadness, depression, fatigue, and tension. The aim of this part of the study is to detect and classify the emotions felt by the patient during the rehabilitation session, in order to personalise the plan of care and, for example, to simplify an exercise if the patient, not achieving the goal, feels fatigued. For each second of the acquisition, a point that represents the perceived emotion is plotted. The dots are printed with a colour that varies from dark blue to yellow in order to have a clear indication of the start and end points of the recording, and therefore the trend over time. The classification analysis of the emotions related to the rehabilitation exercises will be the subject of future studies when a larger dataset for the training phase will be collected.

4. Results

The ReMoVES system is in the test phase at the Complex Recovery and Rehabilitation Facility of the Colletta Hospital in Arenzano (Genoa, Italy) and at the Rehabilitation Centre of Don Carlo Gnocchi Foundation in Fivizzano (Massa-Carrara, Italy).

Up to this point, research efforts have been focused on the definition of the indicators aiming to highlight the patient performance during the execution of the functional exercises. This process requires mutual collaboration between ReMoVES platform developers and therapists. Moreover, the feedback given by the patients plays a crucial role in this testing phase: the system was tried by 41 participants (gender: 19 male, 22 female) aged between 30 and 86 (Figure 15) who were hospitalised due to various causes (stroke, fractures, prostheses, and Parkinson’s disease) (Figure 16); an informed consent has been collected from the involved subjects and data have been recorded anonymously.

Our test group was roughly made up of half post-stroke (11 hemiparetic and 8 with general diseases caused by stroke) and half orthopaedic patients, except for one participant affected by Parkinson’s disease and a young tetraplegic man (upper limb was slightly involved). The therapist integrated the patients’ traditional rehabilitation program with a set of additional exercises of their choice from among the ones provided by the ReMoVES platform. In Figure 17, an overview of the overall number of serious games sessions accomplished is provided. In accordance with Section 3.1, the most frequent exercise was the “BreathBall”, since it is usually carried out before the actual rehabilitation session to induce relaxation.

In this paper, we are providing preliminary results. We want to offer an overall glimpse of the method we have defined to introduce machine learning systems for the automatic evaluation of patient activity performance. At the moment, we have performed the initial steps required by the application of supervised learning methods:

• Determine the training examples. We have decided which features are going to be used as training set for each of our serious games.
• Gather a training set. Thus, a set of input objects is gathered and corresponding outputs are also gathered according to care providers. In detail, “labelled” information consist of flagged data by the therapist as “Good” when the patient performance is valid, and “Other” when the patient carried out the exercise erroneously/inadequately.

Unfortunately, despite the conspicuous number of sessions carried out by patients, the amount of data actually available for analysis is limited. In fact, during this initial testing phase, the focus of the operators (technicians and therapists) was on the stability of the system, the user experience, and the definition of the indicators of each individual game instead of on the labelling phase regarding the evaluation of the patient’s performance.

Most of the game sessions in our database are unlabelled and nearly useless for supervised analysis. Figure 18 shows the results of the classification process achieved with the implementation of a logistic regression model to predict whether a patient performed well during the rehabilitation activity through serious games.

The results are not conclusive because of the reduced dataset (we have nearly 20 labelled sessions for each game), but they highlight how it is possible to classify the “Good” sessions from those labelled as “Other”, showing that the choice of the selected features pairs was effective. We will re-analyze the issue with more data available.

5. Discussion

As reported in the paper, the system had a good impact on the rehabilitation procedures of the individuals involved in the study. Patients and therapists were deeply involved in the assessment and improvement of the overall structure of the system, with particular attention paid to usability and ease of use of the platform.

The general aim was to determine which parameters are useful to the therapist in order to evaluate the patients’ performances. Thanks to the professional involvement of the medical and therapeutic personnel, it was possible to set up a rough dataset that will be used by supervised learning algorithms.

Further steps planned are related to an improvement of the usability and an extension of use to a larger and diversified number of patients, in order to make the dataset larger and more solid. To improve the session labelling process a companion web application for the therapist is under development: this tool is accessible from any device—including a smartphone—and displays a minimal user interface with just two buttons corresponding to “Good” and “Other” label. The therapist will no longer have to note down manually his opinion about the patient performance, but he will label the game session in real-time through the companion web application while the patient is performing the rehabilitation activity. This solution will also bring a remarkable increase of data collection since the therapist will be able to provide multiple feedback over time (for example every 20 s), instead of just one overall final opinion. More granularity will also deliver more precise data for classification since the patient performance could be meaningfully different throughout a single game session and a single label for the entire session could be misleading under certain conditions.

Despite the limited data collected, we are confident that our work is going to bring innovation to the rehabilitation processes. For this reason the team is starting to move the system to a domestic and home-based environment, simplifying and lightening the general rehabilitation process in order to give a better service both to patients and therapists and in the near future to expand the possibility of improving real home-based care.