Biofeedback Applied to Interactive Serious Games to Monitor Frailty in an Elderly Population

Abstract

This article proposes an example of a multiplatform interactive serious game, which is an additional tool and assistant used in the rehabilitation of patients with musculoskeletal system problems. In medicine, any actions and procedures aimed at helping the rehabilitation of patients should entail the most comfortable, but at the same time, effective approach. Regardless of how these actions are orientated, whether for rehabilitation following surgery, fractures, any problems with the musculoskeletal system, or just support for the elderly, rehabilitation methods undoubtedly have good goals, although often the process itself can cause all kinds of discomfort and aversion among patients. This paper presents an interactive platform which enables a slightly different approach to be applied in terms of routine rehabilitation activities and this will help make the process more exciting. The main feature of the system is that it works in several ways: for normal everyday use at home, or for more in-depth observation of various biological parameters, such as heart rate, temperature, and so on. The basic component of the system is the real-time tracking system of the body position, which constitutes both a way to control the game (controller) and a means to analyze the player’s activity. As for the closer control of rehabilitation, the platform also provides the opportunity for medical personnel to monitor the player in real time, with all the data obtained from the game being used for subsequent analysis and comparison. Following several laboratory tests and feedback analysis, the progress indicators are quite encouraging in terms of greater patient interest in this kind of interaction, and effectiveness of the developed platform is also on average about 30–50% compared to conventional exercises, which makes it more attractive in terms of patient support.

1. Introduction

T Rehabilitation procedures in various aspects of medicine are one of the most important and time-consuming treatment processes for patients with various diseases [1,2,3]. According to the World Health Organization, there are 2.4 billion people who are living with a health condition that depend on rehabilitative measures. Unfortunately, there is quite a pressing demand for rehabilitation procedures. This is especially true in countries with a low or near-average standard of living, where about 40–50% of patients cannot gain access to rehabilitative medicine. In this regard, the situation with COVID-19 has only exacerbated this situation. Depending on the focus of the procedure, the measures used in the patient’s recovery can vary considerably, from simple physical exercises [4] to the use of various kinds of additional technical aids [5]. This is especially true for the rehabilitation process of people with musculoskeletal system problems, such as injuries, mild, partial, or complete muscle atrophy, deterioration or loss of motor control, or the usual support for the physical condition of the elderly [6,7,8,9,10,11].

The application of different sets of exercises that are used in these procedures can vary depending on the type of the problem and the extent of it. They can also be divided into different categories, depending on the place, time, or environment in which they are applied, the means used during the procedures, and so on. It is also important to understand that different categories of such rehabilitative measures may have different degrees of effectiveness for different characteristics of patients’ problems and are often only applicable in their own specific area [12,13,14]. This can be especially seen in the presence of additional conditions and means.

Over time, each of the methods used as rehabilitation measures has undergone constant checks and improvements, and become more effective. Nevertheless, due to different technical features or the way in which rehabilitation procedures are carried out, there are cases in which patients may experience various kinds of discomfort during their recovery. This discomfort can be caused by various factors from both physical and emotional aspects [15,16,17,18].

For quite a long time now, the medical field has been researching the possibility of applying various additional remedies to the rehabilitation process, among which there is such a trend known as Serious Games [19,20,21]. This is due to the fact that over the past 20 years, computer technology has made impressive leaps in terms of development, which has expanded the technical capabilities of modern computer equipment, as well as its availability. Modern game making technology enables not only almost any real process to be reproduced within a virtual simulation, but also impresses with its graphic and visual level of elaboration and detail.

Therefore, the idea emerges of finding rehabilitative tasks that could include the additional integration of various game mechanics and devices. Because patients may often encounter all sorts of emotional problems during long rehabilitation courses, this game innovation can help to get rid of this aspect [22,23,24,25]. All problems should be understood to mean that long and even more monotonous exercises or activities over time mean that patients become used to the process, which reduces the motivation to do them each subsequent time. Although this situation does not arise often or in all patients, providing routine activities with a bit of interactivity can have a definite positive effect [26,27].

The introduction of interactive game features makes the patient’s recovery process more meaningful [28]. Even though those actions prescribed by the attending physician are obligatory and indisputable, in some cases they are perceived as “I was told so, so it is necessary”. That is certainly true, but it does not always work with certain age groups, such as children and the elderly. In the case of both the former and the latter, engagement is necessary, and the action process itself is important [29]. If a person finds any kind of activity, including rehabilitation activities, attractive and interesting, then he or she will be more motivated to perform them [30].

This makes even the simplest and most boring physical exercises, such as regular exercise, much more appealing to the patient, but the point is that these kinds of game require special conditions under which they can be used in this way. One of the main problems is how to manage such games—in other words, gameplay [31,32,33]. Firstly, it should be as accurate as possible so as to repeat the actions that the patient needs to perform during rehabilitation procedures. Put another way, in-game levels or activities should essentially take the form of a complete or at least partial substitute for the prescribed exercises. Secondly, it is important that the game should somehow recognize the player’s movements and respond to them correctly. If the first case is quite simple to implement, then the second will cover the more complex and important spectrum of these kinds of computer games.

The purpose of this article is to research the effectiveness of using an interactive game platform as an additional tool for rehabilitation and support for the elderly, or patients with musculoskeletal system problems. An additional goal is to compare how effective the system developed is compared to conventional physical exercises, both in terms of rehabilitation and recovery, as well as in terms of motivation and overall moral satisfaction.

This article is divided into five sections. The first is a literature review. This part is Section 2: a review of articles featuring similar solutions that can be compared to the platform presented in this article. Section 3, titled Materials and Methods, includes a description of the tools and algorithms that were used to develop the system. Section 4 presents the results obtained from the system, as well as examples of analysis and comparison of results. Section 5 and Section 6, titled Discussion and Conclusions, provide a direct discussion of the results obtained from the study and their subsequent prospects, as well as a comparison with similar solutions.

2. Literature Review

With a suitable player tracking system, several technical approaches can be used. For example, in such systems as Kinect [34] or VR-technology [35], body recognition is undertaken by means of several cameras and additional controllers, which together form a kind of coordinate system, in which the player is located [36]. Another option is systems based on neural network algorithms, examples of such systems being OpenPose [37] or PoseNet [38]. In short, it cannot be said which method is better or worse, as both categories have their advantages and disadvantages, such as being dependent on image quality accuracy for recognition of the body in systems based on neural networks, or the need for a large amount of additional equipment and special space requirements in VR-systems.

Whatever the case may be, there are many factors to consider when developing a Serious Games product [39,40,41], although it is most important to understand how effective this or that solution will be. In terms of effectiveness, we mean not only the therapeutic and rehabilitative effect, but also the player’s interest. It is this aspect that will be explored in this article.

The goal was to test the effectiveness of interactive computer games in the rehabilitation of patients with musculoskeletal system problems. The game platform being developed will be compared under real conditions to a set of conventional physical exercises. One of the conditions for development will be the need to ensure the minimum requirements of the system to the players. This means that for a basic game, the number of additional tools should be limited to a personal computer and a webcam. This condition will expand the likely audience, thereby allowing the platform to be used not only for medical purposes, but also as an everyday leisure activity.

The platform being developed was also tested in two stages. At the first stage, the players being tested performed the usual physical exercises for a certain time over a fixed number of times. Then the same players played the game developed directly. After the testing phase, both the body recognition system performance and the emotional aspect of each player was then analyzed. The results obtained from this study will help to determine the effectiveness of the system developed in relation to the rehabilitation process, as well as the relevance of the application using this method in general.

Table 1. Comparison between similar approaches.

Reference	Aim of the Article	Hardware Used	Algorithm	Measurements	Results
[42]	Present low-cost full body rehabilitation framework for the generation of 3D immersive serious games	Camera system: ToF camera IR camera HDM device (Virtual Reality helmet)	GRU-RNN Virtual Reality algorithms	Track position of the body in space.	Increase in rehabilitation rates by an average 15%
[43]	Introduce a smart rehabilitation system for the elderly, without requiring physical contact with traditional control systems.	Webcam	TANGO:H 2d representation system	The method used in the study is based on the evaluation of a vision-based gesture interface by measuring effectiveness, efficiency, and satisfaction.	Gestural exercises yielded higher percentages of task completion (>83%) and task effectiveness (>63%). Eye fatigue(x = 3.15; SD = 0.37). Accurate point is above average (x = 2.57; SD = 0.60)
[44]	Improvement in the balance and postural control of adults.	Webcam + markers Vision-based interaction sensor Wii mote	Modified algorithm based on Kinect body tracking technology	Body position Balance indicators	Usage of interaction objects related to patient interests; patients performed the rehabilitation activity 13.5% faster than when the objects did not represent such interests
[45]	Assess the effects on functional outcomes and treatment adherence of wearable technology and serious games currently used in physical rehabilitation of patients following traumatic bone and soft tissue injuries.	-	Comparison methods of serious games with standard therapy	The search yielded 2704 eligible articles, which were screened by 2 independent reviewers.	Serious games seem a safe alternative or addition to conventional physiotherapy following traumatic bone and soft tissue injuries.
[46]	Describe of newly-developed platform of Remote Monitoring Validation Engineering System for motion rehabilitation.	Microsoft Kinect V2 Microsoft Band 2	Leap Motion Cloud back-end	Different aspects of upper and lower limbs movement, balance, heart rate and electrodermal activity, balance shift.	Most games within this system are nearly useless for supervised analysis.
[47]	Research into the effectiveness of SGs in motor rehabilitation for upper limb and movement/balance	-	Rehabilitation games and systems data systematizing and comparison	Meta-analysis including 61 studies reporting randomized controlled trials in which at least one intervention for motor rehabilitation is included.	Overall moderate effect of SGs on motor indexes, d = 0.59, [95% CI, 0.48, 0.71], p < 0.001
[48]	Evaluate the effects of novel immersive virtual reality technology used for serious games (Oculus Rift 2 plus leap motion controller—OR2-LMC) for upper limb outcomes.	Virtual Reality kit	Mixed methods intervention (MMI)	A mixed methods intervention study, with a qualitative design following technology intervention.	Good result in strength improvements, coordination and dexterity, and speed of participants. No side effects.
[49]	Implement and tests of a system for assessment and monitoring movements, which includes the sensors from Kinect and Leap Motion Controller devices	Microsoft Kinect Leap Motion Controller Kinect	Leap Motion	Using additional motion capture tools along with the virtual game environment.	A study of the feasibility and effectiveness of supplementary remedies on the results of rehabilitation of upper limb problems. Verification of further effectiveness.
[50]	The aim is to evaluate the Fietsgame (Dutch for cycling game), which translates existing rehabilitation exercises into fun exercise games	Control system based on Raspberry Pi Kinect v2 IoT platform	Kinect body tracking system	The study is conducted in a rehabilitation center with 9 participants, including 2 physiotherapists and 7 patients. 6 exercise games under the guidance of a physiotherapist. The mean age of the patients was 74.57 years; all the patients were in the recovery process following hip surgery.	The results showed that 75% to 100% of the patients experienced high levels of enjoyment in all the games except the squats game
[51]	Present a rehabilitation system based on a customizable exergame protocol to prevent falls in the elderly population	Web Camera	Self-developed body tracking platform KINOPTIM	System based on depth sensors and exergames. Measuring of physical abilities and emotional reaction	Performance of the postural response is improved by an average of 80%

shows similar studies and their results below. Some of them are compared to the results obtained from our studies in the Discussion section.

Table 1 provides 10 references to articles that describe similar research to the one proposed. It briefly describes the main purpose of the description, and the methods and means used, as well as the result.

After studying the articles in the table, it was decided to use the simplest sensor system (webcam in this proposal). This decision was made in order to make the platform as accessible as possible to older people, both economically and technically. As one of the aims of the study is to create a system that is as accessible as possible, unlike solutions with additional VR peripherals, consoles or other additional equipment, the use of a webcam reduces the list of requirements. In addition, the system becomes accessible not only, for example, in the doctor’s office, but also at the patient’s home.

3. Materials and Methods

3.1. Materials

The game platform is an additional tool to improve the effectiveness of rehabilitation activities, or to spend leisure time. The main purpose of this study is to test the effectiveness of additional interactive tools and means in the field of rehabilitation and support for the elderly, or people who have received various kinds of injuries to the musculoskeletal system. This study includes both the direct development of the interactive platform itself and the process involved in testing it.

The main task will be to study the effectiveness of rehabilitation procedures using the platform being developed in relation to the usual standard physical exercises. In the long term, a positive result is expected to be seen from the tests, namely a greater efficiency in terms of the interactive way of performing medical rehabilitation exercises, and most importantly, a greater motivational indicator for these actions. The main aspects and parameters of the system developed, as well as testing procedures, will be described below.

3.1.1. Hardware Used

The development and testing process took place on multiple platforms, to maximize the possible devices being supported. The equipment included the following samples, which are shown in

Table 2. Equipment for developing and testing.

Element	Characteristics
Desktop PC
CPU	Ryzen 7 3700x, s-AM4 3.6 GHz/32 Mb
GPU	NVidia GeForce RTX 2070 Super, 8 Gb
RAM	DDR4 32 Gb 2888 GHz
HDD	Seagate Barracuda 1 Tb
Web Camera	InnJoo FHD1080p (1920 × 1080) 60 fps
Laptop Lenovo LEGION Y-540-IPS15i
CPU	Intel(R) Core(TM) i7-9750H CPU@ 2.60 GHz
GPU	NVidia GeForce GTX 1660 Ti, 6 Gb
RAM	DDR4 16 Gb 2888 GHz
SSD	INTEL SSD PEKKW010T8L 1Tb
Web Camera	Integrated Web Camera 720p (1080 × 720) 30 fps

.

This system is designed for use on personal computers or laptops running Windows 10 operating systems. A version for Android 5 and higher is being developed parallel to this, although this variation is reduced in some features.

3.1.2. Inclusion and Exclusion Criteria

For this experiment the inclusion criteria are as follows:

• The user must have a personal computer or laptop and a webcam
• The user must be between the ages of 12 and 85
• The user must be in a spacious room during the game to avoid hitting their surroundings
• The playing room should be well lit with natural or voluminous artificial light
• It is desirable that there be at least one other person next to the user while they are playing
• People with vestibular problems and those who are sensitive to light should only use the system in the presence of others
• People who currently have minor musculoskeletal problems or have had them before.

The authors excluded blind people during the tests because they must read the instructions from the tutorials. Additionally, the user must not have had any serious problems or injuries to the musculoskeletal system, such as, for example, complete or partial paralysis. The room in which the system will be used should be well lit with natural or artificial light.

3.1.3. Experimental Features

(a) 

Information about testers

Ten volunteers were selected for the experiment, in the age range from 21 to 42 years. The main data pertaining to the players are shown in

Table 3. Players main parameters.

№	Gender	Age	Height	Weight	Spine or Limbs Problems	Country
1	Male	26	182	72	Left clavicle fracture	Ukraine
2	Female	25	179	89	-	Morocco
3	Male	26	178	77	Left arm fracture	Pakistan
4	Female	42	167	55	Right wrist injury	Spain
5	Male	27	170	65	-	Spain
6	Male	25	178	72	Left shoulder injury	Spain
7	Male	31	173	78	-	Pakistan
8	Male	25	169	69	-	Colombia
9	Female	21	159	50	Right hip fracture	Czech Republic
10	Male	22	176	63	-	Italy

and Figure 1.

As can be seen in Figure 1 and Table 3, the data provided show information on both physical parameters (height, weight and age, and the presence of any injuries) and social parameters. In the case of the latter, specifically which country the player is from, will help to better understand the results obtained by each of the players during the game.

At the same time, information about the presence or absence of injuries will also help to understand whether injuries in the past have an impact on current results. The data in the column “Spine or limb problems” are information about injuries that occurred before the tests, in order to determine the possible effects of these injuries on the players’ performance. At the time of the tests, all patients were healthy and feeling well.

The choice of this set of players was due in large part to the current situation at the time of the study with the pandemic COVID-2019. Ideally, all tests should be conducted with participants within the age range of 60 to 85 years. Nevertheless, the data that were obtained with the set of players provided will help to qualitatively evaluate the results and more accurately adjust the system for the target age category.

(b) 

Testing plan and regulations

As mentioned earlier, the procedure for testing the system was divided into two stages: obtaining players’ performance during normal exercises and obtaining performance during direct play. During both phases, a total of six tests were conducted in each. The testing interval ranged from 5 to 15 min, depending on the test stage. The time interval between tests was 12 h.

In the first stage, the players took turns performing five exercises while standing in front of the camera. The only information available was the task for the exercise, the number of repetitions required, and the distance to the camera in order to obtain the data as accurately as possible. During this phase, the system recorded the values necessary for each specific exercise, as well as the time of the exercise. The values were saved as a data set for each of the categories of interest.

The second stage differs from the first in that the players now directly used the game platform as a guide for action and exercise. All information about the latter was provided through the user interface. In this case, there was also a record of the parameters that were key for each level-exercise.

It is also important to note that this project was approved and accepted by the Deusto University Ethics Committee.

3.2. Methods

This section will describe in detail the main aspects of the game, such as technical information about the game client itself, the data handled, the exercises, the optional tracker systems, etc.

3.2.1. Overall View

In general, the platform developed is a computer game client, which includes a wide range of features. Figure 1 shows a general diagram of the game platform, which includes the following elements.

Three basic conditions had to be fulfilled in order to create this platform: Condition 1—to create a stable system for tracking player movements, Condition 2—to create a gaming platform that allows the player to perform activities, and Condition 3—to ensure that the data collected during the game are processed and stored. Figure 2 is a general schematic representation of the system. Each of the conditions will be discussed below.

(a) 

Gaming platform

The gaming platform is a game client, a computer game. Structurally, it is divided into levels, which are in turn divided into 3 categories:

• Levels of settings
• Levels for the upper body
• Levels for the lower body.

Each of the categories includes several levels of exercises that are responsible for a specific exercise and, respectively, for a specific muscle and joint group. This system has two main functions. The first is visual and involves providing the player with visual information about what is happening now and what the player should do. In addition to the visual instructions for each exercise, the game interface provides the player with a visual representation of their actions, whether it be interactions with objects, the environment, etc. The second function is to collect information regarding the player’s movements and activity. For each level, the system registers certain indicators, whether it be arm movements or general body position. In general, the platform consists of three functional blocks; more information about which will be given in Section 3.2.2, Section 3.2.3, Section 3.2.4 and Section 3.2.5. In this Section 3.2.1 is a description of the game client.

The total game interface consists of 7 levels, which are divided into different categories, depending on the focus and purpose of the exercise, which it simulates. It is important to note that all game levels have been created to simulate different kinds of physical exercise as accurately as possible.

(b) 

Exercises used

The exercises chosen to develop of the system are some of the basic strengthening physical activities. Each was integrated into the game in a specific order, and the choice of each was coordinated with the therapist. There are 5 exercises in total, including the following:

(1)

“The sign of infinity”. This exercise is a simple activity for the upper extremities. The task involves putting your hands with the palms in front of you, bending your hands at the elbows, and describing the “sign of infinity” in the air. This exercise should be repeated 10 times for each hand, keeping in mind that the right hand moves clockwise and the left counterclockwise. The purpose of this exercise is to assess the player’s motor skills and coordination.

(2)

“Rope Pull”. This level is a movement that is exactly the same as a top-down rope pull, repeated 10 times overall. The goal is to assess the overall mobility of the upper extremities, as well as the accuracy and simultaneity of the movements.

(3)

“Inflating the lifejacket”. This exercise is similar to the previous one, but also involves the shoulder girdle. The activity is also aimed at assessing the synchronicity of actions and the general condition of the upper body.

(4)

“Sailing on a yacht”. The essence of this exercise is to steer the boat and pass through 10 control points, by moving the hips left and right. It is necessary to perform 10 repetitions in each direction. The purpose is to assess the general mobility of the joints of the pelvic area, and condition of the lumbar spine, as well as to train balance.

(5)

“Dinghy Control”. Conceptually, this exercise is similar to the previous one. The only difference is that this time, the movement is not with the hips but with the shoulder girdle. In this case, the main purpose of the exercise is to support and warm up the spine and examine its condition.

Before choosing a set of exercises for the platform, this issue was discussed with the therapist. In the process, it was discovered that for the category of elderly people, the most optimal set of activities would be one that could engage all major muscle groups in a single play session. As such, the order and specificity of each exercise allows you to work on all the major groups, from the upper limbs to the back and hips.

Thus, the order of the exercises described in Section 3.2.1 (b) corresponds to the order that was recommended by the therapist and is integrated into the game. In this way, the player engages the different parts of the musculoskeletal system on a “from lesser to greater” basis: upper limbs, shoulder joints, pelvic girdle, lower back. In addition, coordination (The sign of infinity) and balance (Dinghy Control) are trained.

These exercises are intended solely for the general warm-up and support of players whose age range is between 60 and 85 years.

(c) 

Game Interface

The game itself is a visual user shell made in 3D style. The main message and design of the game involves a maritime theme, namely traveling on a personal yacht on the ocean. This is due to the relatively low mobility of the target audience; at the same time, the theme of travel is one of the most appealing. At the first stages of the system development, we also considered the 2D version, but it was decided to abandon this concept in favor of a greater visual immersion.

The game is also divided into 5 main levels, which comprise essentially an interactive visualization of the physical exercises used and several additional levels, such as the main menu, the level of profile creation and the results table. At each exercise level, the player’s body tracking system is connected, allowing the collection of movement data from different parts of the player’s body. For the latter, this system is invisible and works in the background, allowing sole focus on the game. The algorithm for obtaining these data is described further in Section 3.2.5 and Section 3.2.6.

3.2.2. Architecture Explanation

The architecture of the system is quite simple. It consists of 3 interconnected functional blocks, which perform different kinds of functions. The architecture diagram itself is shown in Figure 3.

The order is next. The sensor connection block is responsible for direct communication of additional sensors or devices with the game platform client. This is done either by using the usual USB protocols, Bluetooth connection, or by using additional tools (expansion boards, Arduino boards, Raspberry Pi boards, and so on). The main sensor required, without which the system loses its efficiency and usefulness in general, is a USB webcam. More about this device will be described later.

The server unit is a remote repository where each player’s progress during the game is saved. There is also the possibility of remote access to a special page, which shows the statistics pertaining to the player, their main indicators, and progress in rehabilitation over a certain period. Demonstrating this information is intended solely for the attending physician, but where necessary, access can also be given to the players themselves. Any information received from the player is not confidential but falls under the rule of medical confidentiality.

The Game Client block is the game client itself, which the player or doctor installs on their personal or hospital computer, and through which the player’s medical data are received and processed. It also includes local data storage, which is essentially a backup copy of the data sent to the server. It is important to note that data processing in the game client, as well as sending them to the server, is carried out in real time.

3.2.3. Body Tracking System Description

The main feature of the platform being developed is a real-time motion tracking system. During the first stages of development, several different systems were tested that can produce human recognition in images, with both machine-learning and neural network-based algorithms having been tested. In the end, the choice fell on the OpenPose system. Even though this algorithm was most suitable for the task at hand, it still required a lot of tweaking, and is based on a trained Convolutional Neural Network (CNN). The system works with both single images and video. In the second case, the load on the system is somewhat greater, as a continuous storyboard with a rate of 30 frames per second (fps) is used. During adaptation of the OpenPose system to our platform, several functional changes were made to the algorithm, without changing the neural network itself:

• The hand recognition module has been reduced (fingers, palm).
• The face recognition module (eyes, mouth, nose, ears, eyebrows) was reduced.
• The storyboarding process was optimized, which allowed a stable 30 fps to be obtained (in the original state the rate was 18–24 fps).

It is worth providing more detail on the features of the finalized OpenPose. As mentioned above, during testing it was found that at this stage, all motion tracking functions are redundant. Therefore, the system was simplified which improved performance. Initially, the system was designed for 25 key points for the body, 21 for the arms, and 70 for the face, although in the modified system, there are only 21 points for the body. A view of the tracker is shown in Figure 4.

In a nutshell, the Body Position Algorithm works by selecting each frame individually from the video stream. The neural network then creates a kind of confidence map on the frame, which creates key points on it. From these points, the direction of the found body parts and their connections are calculated. After that, the points are “assembled” depending on the direction, and finally a “body map” is assigned to the frame. The result is a “skeleton” which is created directly in the production environment, and with which further manipulations can be carried out.

As the algorithm uses a storyboard of the received image, the efficiency and accuracy of the system directly depends on the quality of the incoming video (cropped images). The system is, therefore, highly dependent on the ambient light factor. The accuracy and speed of the algorithm is also affected by the resolution of the source image. The higher it is, the more accurately the neural network detects the key points of the body, which increases the overall accuracy, but at the same time, the speed of rendering these points decreases.

Body tracking algorithm was also integrated into the Unity game engine environment with the functions being based on the generation of 3D objects, which are tied to 2D image coordinates. This image is obtained from the output of the neural network. Then the X and Y coordinates are mapped, which allows binding objects within the system.

For the system to work correctly in a game engine, the main task for the algorithm is to ensure the highest possible accuracy at the highest possible framerate. The modifications we made allowed us to achieve 30 frames per second, with a sufficiently high accuracy.

It is important to note that the accuracy is also affected by the frequency of the incoming video stream. At 25, 30 and 60 fps (camera parameters), the rate of change in the position of key body points in subsequent frames decreases as the number of frames increases. Therefore, during the neural network operation for each frame, there may be “oscillations” of these points on the rendered skeleton. In the game itself, this translates into a fluctuation of the coordinates of each of the points being created. This optimization has reduced these oscillations to 0.4–0.9% of the error between frames.

A more detailed description of the features of the OpenPose algorithm is presented in reference [37].

3.2.4. Game Structure

This section will describe the general structure of the levels, as well as a description of each of them individually. It should be said right away that all available game levels were based on the therapist’s recommendations.

As mentioned above, the main task of the platform is to help in the player’s rehabilitation. Therefore, the main point and main challenge was precisely the need to disguise the usual routine physical exercises in such a way that they would be as difficult to recognize as possible.

At this stage in the game there are 5 levels, which are lined up in a clear sequence, and serve a certain function. The structure of each of the levels is quite simple, including the following elements:

• A level guide with a description of the task, and a demonstration of the movement to be performed by the player
• The actual playing process (performing the exercise)
• The result window, where the player sees how well they did, and the transition window to the next activity (level).

Each level is also divided into two parts: the game interface visible to the player, as well as a function block, which is responsible for obtaining game session data. What these data are and how they are obtained will be described later. Now, it is worth going to each of the levels separately.

A sailboat ride was adopted as a game theme. Considering the target audience, and the relaxing and interesting process in general, this solution proved to be the most appropriate both in terms of activity and level of interest, and in terms of adaptability of the proposed exercises.

One of the main features of the game platform is that, thanks to the work done to integrate the tracking system into the Unity environment, the player does not need any kind of controllers. All control in the game is assured by their movement. The control points, which are responsible for the hands, are essentially analogous to the mouse cursor of a personal computer and allow not only performance of the task to control, but also to the game menu. The usual mouse control is also considered standard.

3.2.5. Data Collection Method

As mentioned earlier, the platform has two methods of collecting information about the player. The first involvers the collection of game information (run time, points, reps), while the second involves the collection of information about the activity of the movements of individual body parts, as well as the player. This system takes the form of a built-in algorithm, which works regardless of the difficulty selected by the player and does not imply a shutdown function.

Although this information about movement activity is not confidential, notification about its collection for purely medical needs is provided before the start of the game. The movement data may be subject to medical secrecy rules and will not be available to anyone other than the treating physician and the player themselves.

Now here is more about the functions of the system, with the general principle for obtaining data being shown in Figure 5.

In total, the algorithm receives data on the following parameters:

• Angle between shoulder and left hand forearm
• Angle between shoulder and right hand forearm
• Left hand position
• Right arm position
• Hip position
• Shoulder position
• Head position
• Head rotation angle.

It all works as follows. Inside the Unity environment, each of the key points, which are created by a neural network (OpenPose algorithm), are tied to the coordinates of the 3D space of the game scene. When fixing the player’s movement, these coordinates change, but only in two directions, X and Y, while the Z coordinate always remains unchanged. Using this, each of the game scenes is constructed in such a way that when linking 3D objects to points of the neural network, these objects will also change only 2 coordinates. This method has both advantages and disadvantages.

The main disadvantage of the system is that it cannot correctly track complex movements in 3 planes. For example, the difference between the arm outstretched forward and outstretched to the side is tracked by the difference in distance between the arm outstretched forward and outstretched sideways, tracked by the straight-line distance between the key points of that arm. Thus, technically, there is a distortion of the limb length in the system, which can also affect the angle at certain positions.

An important aspect when gathering the above information on the indicators is that it is affected by the general lighting of the room which the player is in, as well as the camera position. It may be that lighting reduces the accuracy of body recognition by the neural network and at some moments, it may affect the objects inside the game scene in the form of objects “twitching” (each frame from the camera may give different coordinate positions of key points). In the second case, if the webcam is installed incorrectly, when the lens is excessively raised, lowered, or directed more forcefully to one side, the position may be detected with an error. To avoid this, a calibration level was created, whereby the player must set the lens and lighting correctly.

Despite the disadvantages of the system, it is not intended to obtain accurate values, but rather to track the player’s activity. The main purpose of these indicators is to record activity and provide data for further evaluation and comparison of progress.

Next, we shall look at the way the system works. When you load a level and activate the body tracking algorithm, the activity logging system starts recording the key point coordinate readings for all tracked indicators. The recording frequency is 1 coordinate value every 1 centisecond, and the method for obtaining this coordinate for each of the parameters is different.

Table 4. Parameters recording methods.

Parameter	Title 2
Left arm position	Object X. and Y.properties coordinate recording
Right arm position	Object X. and Y.properties coordinate recording
Head rotation angle	Function of coordinates transform + Vector angle calculation
Head position	Object X.properties coordinate recording
Shoulder position	Object X.properties coordinate recording
Hip position	Object X.properties coordinate recording

shows the algorithms for obtaining these values.

The resulting values are written in the form of a data array, which, after the level is completed, is written in a separate file format—json. This method of saving makes it convenient to work with data both in a manual format and for algorithm processing on the server side of the platform. Next, Figure 6 shows an example of the graphical representation of the resulting data.

Figure 6 shows the dataset of the hip position of one person in 3 different cases: Case 1—normal pace of movement; Case 2—the pace of movement is reduced; and Case 3—the player was inactive most of the time. The task was set for 10 repetitions of hip movement (5 to the left and 5 to the right). The amount of time spent on the exercise in both cases was 10 s. The values on the Y-axis are the deviation coordinate values and are shown in the form of positive and negative values, which also characterizes the deviation side: positive values refer to the right deviation, whereas negative values refer to the left deviation. Based on the information provided in Figure 6, it is possible to draw several conclusions in terms of physical activity:

• The timing of the player’s movements, both the total number and each repetition individually
• The maximum value of the deviation amplitude, which indicates the intensity of the exercise
• The activity as a whole, based on the number of peak values of the deviation amplitude
• The evaluative characteristic of motor skills, based on the direction in which there is a greater number of peak values of the deviation amplitude.

Thus, the parameters obtained from a player’s activity make it possible to evaluate both their current activity during a given game session and to obtain an evaluation over time by comparing their motor characteristics. This information, in theory, should help physicians and the patients themselves to monitor the outcome and progress of the rehabilitation process for different degrees and types of musculoskeletal problems.

3.2.6. Data Saving and Interpretation

As a result of each of the tasks, the system stored all the data obtained in a special .json file, the information from which was used to process the results. All indicators obtained are shown in the form of a data array for each of the categories. An example of such a file is shown in

Table 5. Example of retrieved data cluster.

Body Parameter	Point 1	Point 2	Point 3	Point 4	Point 5	Point 6	Point 7	Point 8	Point 9	Point 10
rightAngelData	−3.660	−3.660	−8.564	−8.564	−8.564	−7.300	−5.835	−5.835	−4.547	−4.547
leftAngelData	−82.022	−82.042	−82.006	−81.974	−81.974	−82.679	−82.825	−82.789	−83.523	−83.467
HeadAngelData	179.786	179.780	179.791	179.784	179.805	179.794	179.795	179.789	179.792	179.785
HipsData	8.276	6.507	7.878	7.878	5.987	4.728	6.099	4.788	6.159	6.159
HeadData	179.788	179.786	179.780	179.791	179.784	179.805	179.794	179.795	179.789	179.792
rangeData	6.908	6.965	6.997	7.054	7.043	7.039	7.091	7.139	7.150	7.209
shoulderData	−0.037	0.862	−0.509	−0.509	0.318	0.095	−1.276	−1.336	−2.707	−2.707
leftMoveData	−58.095	−58.103	−58.103	−58.103	−58.128	−58.115	−58.115	−58.138	−58.138	−58.138
rightMoveData	−56.344	−56.381	−56.383	−56.383	−56.145	−55.957	−55.951	−56.100	−56.107	−56.107
averageArmLData	60.127	−57.823	−58.518	−50.405	6.438	−19.626	−49.406	48.971	−53.573	−41.703
averageArmRData	77.494	−49.893	−56.770	15.153	−12.699	−33.412	27.298	−55.278	2.442	−2.859
averageHipsData	95.968	4.780	7.458	8.700	8.086	8.137	8.336	6.683	8.547	8.080
averageShouldersData	47.749	−3.716	−0.577	−0.610	−0.642	−0.603	−1.873	−0.038	−2.035	−1.135

.

There are 13 parameters in total:

• Right- and left-hand angle data—information that allows you to assess the overall movement activity of the player when performing upper limb exercises
• Head angle and head displacement data—allow tracking the degree of rotation of the player’s head. Useful for understanding the approximate direction of the player’s gaze, as well as problems with general motor skills (for problems with the cervical spine
• Range data—show the general dynamics of the player’s position in space. Useful for evaluating general activity, as well as for evaluating the player’s movement during game activities
• Hips and shoulders displacement data—show the dynamic displacement of the player’s shoulders and hips, as well as the degree of this displacement
• Left and right arm displacement data—the activity of the player’s hand movement. Characteristics of upper and lower maximum deflection, smoothness, and accuracy of the movements
• Average arm, hip, and shoulder movements—characteristics that allow the average position of the body to be estimated, as well as individual areas of the body. Useful when researching into temporary or permanent partial atrophy, palsy, or dysfunction of the muscles of a particular area of the body.

It is important to note that all parameters saved in the file refer to the value of the coordinate of a particular point, which is created by the motion tracking system. By default, the screen coordinate system in the Unity environment is [0%, …, 100%] vertically and [0%, …, 100%] horizontally, from the current screen resolution, which is set to 1080p (1920 × 1080 pixels) in the system by default. Given that all parameters are calculated relative to the median line on the X or Y axis, which is 50% and 50%, respectively, it appears that the screen space is divided into 4 segments, in which, depending on the exercise, the coordinates may record both positive and negative values. This indicates only the direction of movement (in the case of the hips or shoulders, “+” means right and “−” means left.

3.2.7. Data Visualization

To assess exercise performance, it is necessary to select a benchmark example by means of which one can judge not only the correctness of a single activity, but also the quality of progress and how close it is to the desired result. In this regard, a “benchmark characteristic” was created for each of the exercises, which will serve as a reference point in the evaluation of results. For this purpose, each exercise was performed 5 times at the correct tempo and with the correct amplitude of movement, naturally as much as possible. Then, a general result was obtained from the 5 results, which became the “reference” for the system being developed. The following were chosen as the key parameters of these signals:

• Time to complete each task
• Amplitude of the received signal (by means of direct Fourier transform);
• Frequency of repetitions
• Repetition period
• Jitter and shimmer.

As an example, how to check and parse the signal parameters with their subsequent evaluation is shown in Figure 7 with the parameters providing referring to the reference result obtained for Exercise 3.

Additionally, to gain a clearer picture and better assessment of the results, it was decided to obtain a signal that corresponds to a very bad version of the exercise—namely, poor timing, incorrect amplitude, and speed of repetitions, as well as the presence of external factors that could disrupt the procedure involved in obtaining results. An example of this exercise is shown in Figure 8.

Even a visual analysis of the two graphs shows that Figure 8 provides a clear difference in amplitude and repetition time. To describe this example more accurately, Figure 9 and Figure 10 are shown below, which show the results obtained from Exercise 3 in the normal view, and immediately after passing the game level.

In Figure 9 you can see that the graph referring to this signal already more accurately resembles the reference signal. We can clearly see the even level of amplitude and time of each repetition.

Figure 10 directly shows the result obtained during the passage of the level on the game platform.

3.2.8. Data Comparison

As described in Section 3.2.5, the signals received must be processed and certain characteristic data must be obtained from them. Therefore, the following parameters must be extracted from each signal:

(a) 

Time parameter:

This is the exercise time in total. This characteristic helps to ascertain how fast or slow the player performed a particular exercise.

(b) 

Amplitude parameter:

This includes the value of the minimum and maximum amplitude of the signal. The main purpose of the measurement is to represent the player’s activity during the exercise, namely to understand the spatial characteristic of the activity. Given that the amplitude value obtained during the analysis of the player’s movements is shown in the percentage value of the deviation of the key points from the central plane lines on the screen, by obtaining this value, it is possible to judge, for example, how high the player raised their arms, how much they deflected when moving their hips, or how actively they moved in front of the camera.

(c) 

Frequency of repetitions:

It is also important to understand the frequency with which the player performed a particular exercise. This parameter can be obtained by several methods, but since the data obtained in the time and graphical representation are a sinusoidal signal, the most convenient method is the application of the direct Fourier transform, the formula for which is shown below.

$$f\left(t\right)=\frac{1}{2\pi }{{\displaystyle \int }}_{-\infty }^{\infty }F\left(j\omega \right)e^{j\omega t}d\omega$$

(1)

Obtaining this parameter will help to keep track of how often the player is performing the exercise. This parameter is also important for understanding the quality of the exercise (whether it is too slow or too fast), and therefore its effectiveness.

(d) 

Shimmer and Jitter:

Shimmer and jitter values are an estimative characteristic of the quality of exercise performance. Thanks to these parameters, we can judge the quality of the periodicity of movements, both in terms of the difference in time of each repetition and the difference in the amplitude of the movements.

Jitter absolute: Variation of fundamental frequency. The average absolute difference between consecutive periods:

$$jitter \left(a\right)= \frac{1}{N-1}{\displaystyle \sum }_i^{N-1}\left|T_i-T_{i+1}\right|$$

(2)

Jitter relative: Average absolute difference between consecutive periods, divided by the average period:

$$jitter \left(r\right)= \frac{\frac{1}{N-1}{{\displaystyle \sum }}_i^{N-1}\left|T_i-T_{i+1}\right|}{\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{T}_i}$$

(3)

where T_i is the extracted period lengths and N is the number of extracted periods.

Shimmer dB: Expressed as the variability of the peak-to-peak amplitude in decibels, average absolute base-10 logarithm of the difference between the amplitudes of consecutive periods.

$$shimmer \left(a\right)= \frac{1}{N-1}{{\displaystyle \sum }}_{i=1}^{N-1}\left|20\log \left(\frac{A_{i+1}}{A_i}\right)\right|$$

(4)

Shimmer relative: Defined as the average absolute difference between the amplitudes of consecutive periods, divided by the average amplitude, and expressed as a percentage.

$$shimmer \left(r\right)= \frac{\frac{1}{N-1}{{\displaystyle \sum }}_i^{N-1}\left|A_i-A_{i+1}\right|}{\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{A}_i}$$

(5)

where A_i are the extracted peak-to-peak amplitude data and N is the number of extracted fundamental frequency periods.

Having obtained all of the above parameters, it is possible to compare the results obtained, as well as judge the quality of the exercise.

Table 6. Example of comparing the results of one iteration in Exercise 3.

Result	Time, s	Min. Amplitude, %	Max. Amplitude, %	Frequency, Hz	Shimmer Absolute, dB	Shimmer Relative	Shimmer, %	Jitter Absolute	Jitter Relative	Jitter, %
LEFT HAND
Etalon	13.01	−68.15	67.52	0.859	11.78	0.029	2.886	0.076	0.064	5.344
Bad	9.46	−87.34	77.69	1.025	25.14	0.318	31.827	0.414	0.512	63.168
Exercise	9.41	−87.37	69.75	1.297	10.59	0.042	4.241	0.067	0.080	9.701
InGame	15.26	−87.34	77.55	0.806	16.29	0.033	3.280	0.164	0.118	8.438
RIGHT HAND
Etalon	13.01	−84.37	66.29	0.8591	8.69	0.029	2.940	0.091	0.076	6.380
Bad	9.46	−85.22	76.35	1.025	25.04	0.373	37.262	0.389	0.477	58.548
Exercise	9.41	−85.04	64.78	1.297	16.67	0.056	5.568	0.084	0.102	12.228
InGame	15.26	−83.75	76.99	0.806	13.00	0.035	3.483	0.196	0.140	9.997

directly shows the results obtained from four cases of Exercise 3.

The data in Table 6 refer to measurements of a single exercise performed once. Figure 11 shows a graphical analysis of the data obtained.

As for the jitter and shimmer parameters, it is worth noting that the lower their values, the better the result obtained in the end. This will mean that the player performed the game exercise as clearly and correctly as possible.

Additionally, the time and amplitude parameters should be compared in a different way. The reference point is the value of the repetition amplitude of the reference signal, in this case, the distance between the maximum and minimum amplitude of each repetition and the entire signal (values in Table 6).

4. Results

4.1. Comparison of Results

In addition to comparing the results obtained from a single exercise, it is also important to understand the dynamics of those results. This is necessary in order to understand the efficiency of the system in repeated use, as well as to ascertain the parameter of overall efficiency.

Table 7. The results obtained from Exercise 4 in its two interpretations: game and usual.

Result	Time, s	Min. Amplitude, %	Max. Amplitude, %	Frequency, Hz	Shimmer Absolute, dB	Shimmer Relative	Shimmer, %	Jitter Absolute	Jitter Relative	Jitter, %
Ordinary Exercise
Etalon	25.39	−19.4	19.7	28.55	0.44	7.91	0.044	4.398	0.590	0.241
Bad	12.41	−11.92	28.67	31.01	0.82	13.54	0.198	19.77	0.229	0.201
Try 1	11.24	−19.04	11.15	24.75	0.91	10.76	0.120	12	0.158	0.140
Try 2	10.94	−9.46	22.86	29.12	0.93	7.91	0.057	5.731	0.142	0.130
Try 3	10.61	−15.16	18.98	27.66	0.96	10.38	0.112	11.241	0.108	0.096
Try 4	10.54	−12.28	17.88	26.72	1.06	4.94	0.055	5.498	0.100	0.095
Try 5	10.44	−10.96	20.92	28.09	0.98	11.04	0.090	8.955	0.082	0.083
Try 6	9.18	−13.81	17.58	26.824	1.12	7.82	0.092	9.233	0.128	0.139
In-Game Exercise
Etalon	19.27	−37.20	41.82	70.35	0.31	15.41	0.084	8.422	0.508	0.155
Bad	29.79	−38.53	38.27	63.08	0.28	17.85	0.178	17.770	2.758	0.484
Try 1	28.00	−45.05	58.12	80.16	0.21	19.83	0.136	13.585	0.330	0.077
Try 2	25.69	−45.06	46.28	76.19	0.23	13.40	0.092	9.168	0.223	0.050
Try 3	25.72	−39.05	49.01	83.95	0.19	18.02	0.182	18.24	0.545	0.127
Try 4	23.17	−42.46	44.82	70.47	0.3	21.70	0.196	19.573	0.368	0.105
Try 5	22.88	−43.91	46.9	74.81	0.26	21.39	0.162	16.153	0.952	0.254
Try 6	22.09	−22.68	26.4935	43.81	0.23	15.76	0.153	15.327	0.778	0.187

shows the results obtained from Exercise 4 over six days with a 12-h interval between exercises for one person. The exercises are performed in normal exercise mode and directly in the game. The comparison is made according to the same parameters that were shown in Table 6. The results presented are the specific case of a particular player. The results of other players are treated in the same way and according to the same methodology as presented below.

Table 7 shows the results obtained from Exercise 4 in two different interpretations: as a game level performance and as a normal physical exercise. Figure 12 shows a visual representation of the results.

The effectiveness of the system can be judged following the first tests. One of the indicators of the system is time, which shows how fast the player completes a level. From this, it is possible to compile statistics on the player’s progress, and this can be done by comparing it to the normal way of performing the exercise. In the example of the results obtained from Exercise 4 in Table 7 and Figure 12, the time value decreases in both cases of normal exercise and the in-game level. In essence, however, the time values for both cases mean a different situation. Figure 13 and

Table 8. Players time parameters.

Result	Time Standard, s	Time Standard Error, %	Time In-Game, s	Time In-Game Error, %	Percentage of Standard Accuracy, %	Percentage of In-Game Accuracy, %
Etalon	25.39	0	19.27	0	100	100
Try 1	11.24	55.73	29.79	45.30	44.27	54.70
Try 2	10.94	56.91	28	33.32	43.09	66.68
Try 3	10.61	58.21	25.69	33.47	41.79	66.53
Try 4	10.54	58.49	25.72	20.24	41.51	79.76
Try 5	10.44	58.88	23.17	18.73	41.12	81.27
Try 6	9.18	63.84	22.88	14.63	36.16	85.37

shows the progression of the results as a percentage.

The data in Table 8 are obtained from the percentage ratio between the values of the reference and received signals. The table shows the difference in the main indicators of the signals in terms of time and accuracy. This is clearly expressed in the fact that the time of the standard execution of Exercise 4 decreases with each new attempt (column “Time Standard, s”). At the same time, the “Time Standard error, %” column shows the percentage of deviation of the obtained time for each try from the “Etalon” value. Time indicators are taken as an example as one of the main evaluation parameters.

Figure 13a,b shows the difference in time needed to complete Exercise 4 for each trial. In the first case, the Try 1 result differs from the reference by 14.15 s, which is 44.27%. This gap then increases with each new trial to 9.18 s, i.e., 63.84%. This indicates that this examinee is trying to finish the exercise quickly (there is no question as to the correctness of the exercise at this point).

The second case is exactly the opposite. This time difference is 10.52 s (45.3%), and the time difference decreases to 3.61 s, which is only 14.63%, unlike in the case of the standard exercise. Based on this result, we can conclude that by choosing an interpretation of the exercise based on the platform developed, the player will have a greater motivation and desire to perform the exercise more correctly.

Even though in both cases the exercise time is reduced, the case of the in-game result shows that the platform developed justifies itself as an additional tool for rehabilitation proposes. This is due to the fact that by introducing a certain kind of activity, such as, in this case, steering the ship through the control points, the person focuses not just on completing the exercise, but also on performing it as correctly as possible. This is also greatly helped by the presence of a visual response to the player’s actions.

The same pattern is observed in the case of the frequency of exercise. In Figure 12, the frequency response of both cases shows that while in the case of the normal exercise, the player tries to perform the exercise faster each time, the in-game case allows to regulate the tempo of the movements at the same level.

The jitter and shimmer values are also indicative of a certain kind of result. Due to the peculiarities of the game level for Exercise 4, there is a need to control the period of each repetition by the game itself, which allows you to bring the time of each repetition to the median value. Calculation these parameters is an important part of the system because they mainly control and regulate the dynamics of the player’s movements, which allows monitoring and regulating the rehabilitation process more accurately.

4.2. Explanations of Results

As can be seen in Figure 12, in general, both interpretations of Exercise 4 evidence a similar progression of results. Nevertheless, it should be noted that with respect to some specifics, Exercise 4 is somewhat different in nature in these two interpretations. While in the case of the simple hip movement, the main goal is the precision of the movement—namely, the most similar values of the amplitude of the movement, the period of each repetition and the total time of the movement—in the game version, the key goals are somewhat different.

Based on the peculiarities of Level 4 (which is the interpretation of the Exercise 4), the number of repetitions is 10, with 5 hip movements on each side, while for normal performance it is 20, with 10 movements on each side. However, this reduction in repetition is compensated by the fact that the player is not limited to this number. Since the goal of the game level is to steer the boat through 10 gates, and not just 20 moves, the player can do more to achieve this. An example of how this works is shown in Figure 13.

Unlike the usual physical interpretation of Exercise 4, the in-game version forces the player to focus more on the quality of the exercise. Considering that the boat is steered directly with the hips, the number of repetitions will be determined by the player’s level of dexterity and control. In the “ideal” performance of this exercise, the movement pattern (Figure 13b) is similar to the physical performance movement pattern (Figure 14a). As you can see in Figure 14c, the first attempt to pass the level required far more movement for the player, although after passing the same level on the 6th attempt (Figure 14d), player significantly improved their score. Instead of 18 moves, they needed 10, which is a 44.4% increase. At the same time, the level completion time also improved, whereby the score improved from 28 s to 22.09, which is 21.1% better than the 1st try.

In the case of a normal exercise, the time score indicates that the exercise was performed faster with each new time, to perform it faster. These rules of analysis also apply to Exercise 5 since they are similar in their nature and the way they are performed. At the same time, the game interpretation of the first three exercises does not differ in mechanics from the physical one, and so in this case, a direct comparison of the indexes given in Table 6 and Table 7 is provided.

5. Discussion

Even though, based on the results, the system shows its effectiveness, additional checks and evaluations are required. Having analyzed those articles that are listed in Table 1, as well as other similar studies, it was concluded that, in general, the effectiveness of introducing additional systems, such as computer games, into the rehabilitation process is quite effective at present. Thanks to the variability of technical and software solutions, it is possible to choose the activity for almost any types of rehabilitation procedure. This includes, for example, assistance with problems with the upper extremities [44,47,49], lower extremities [47,50], or for the whole body [42,43,51]. The variability in the methods used in these studies also allows the rehabilitation process to be tailored as effectively as possible. These include the use of special tracking tools, such as the Kinect system or Virtual reality interfaces [42,44,46,49,50], and neural networks to avoid the need for additional sensors [43,51].

Since the system presented in this article also has its own characteristics, it is possible to compare it to the above solutions. This should be undertaken in terms of several aspects that affect different sides of the platforms.

Technical side: Includes additional peripherals or technical devices. If we refer to that case, then it is worth comparing the additional tools and sensors that were used in different approaches, and what the reason for this decision was. The purpose of the platform described in this article is to create the most convenient and simple system, which does not require any additional handling by the doctor or players. This technical solution already offers certain kinds of advantages, in contrast to the variants where a virtual reality system was used [42,48]: the Kinect platform or Wii [46,49], additional camera systems and additional technical means [42,50]. Disregarding additional sensors offers you an advantage in terms of accessibility, as well as in terms of the longer-term relevance of the platform. For example, Microsoft Kinect and Nintendo Wii have been out of production and without support since 2017 and 2013, respectively.

The platform presented here does not require any additional peripherals other than the webcam, which is a very common and cheap device. This makes the system not only as affordable as possible from the financial standpoint, but also easy to install and use.

Visual side: Undoubtedly, the visual part of the platform has a very important function, namely, to attract the player and motivate them to return to the game. In the case of the platform developed, it was decided to make the player part of a certain little story in which they are directly involved. Unlike other solutions [44,48,49,50] where the player simply performs tasks and activities related only to one general style, or has no thematic activities at all [43,50] our platform offers a small adventure, which makes the system more interesting.

In addition, this kind of exercise presentation, which has a beginning and an end goal, is designed to help the player, mentally. The theme of travel is chosen precisely because of the possible limited mobility of the players, given the age category in addition to the peculiarities of the location, and thus helps to try out activities that are not normally available to a person, even if not to the fullest extent. This, in turn, has a positive effect on their mental state.

Algorithm used: The built-in motion recognition algorithm is based on a neural network, and this solution has both pluses and minuses. Unlike the Kinect technologies [46,49], where the 3D image is built using a system of depth cameras, or the Virtual Reality system [42,48], in which the work is based on a constant reading of the position of sensors inside the helmet and controllers in addition to building a volumetric image with the camera system, solutions based on neural networks depend quite heavily on external parameters. These include the quality of lighting, positioning of the player in front of the camera, and quality of the processed image. In this regard, and in order to obtain the most correct data, it is necessary to introduce and observe additional rules of work with systems based on neural networks [43,51].

Bearing in mind that the main task was to make the platform as accessible and simple as possible, it was decided in this study to take the neural network algorithm as a basis. Consequently, use of the platform allows you to apply the principle of “install–run–use”.

Medical focus and value to physicians: The system proposed is designed to help support and rehabilitate predominantly the elderly, although it can also be applied to patients with minor musculoskeletal system problems. The focus of the choice of exercises and their direction is based on the principle of “a little of everything”, in contrast to the systems aimed at working with a particular part of the body, such as the upper extremities [44,46,48] or the lower extremities [42,50,51].

In terms of the value of the system to medical staff, the platform acts as a tool for systematizing information, which allows the doctor to monitor progress of the patient’s recovery and their overall activity indices more accurately and qualitatively. The main feature is that, unlike some similar systems [43,48,50], direct participation by the attending physician in the game process is not required. All the data obtained can be viewed and examined at any time, thanks to the cloud interface where this information is duplicated, as seen, for example, in the solution [43].

Results and effectiveness: Given the specific nature of the study presented, it is rather difficult to compare it to the results obtained from similar solutions. Although the effectiveness of the gaming platform increases the patient’s outcome from 30% to 50% depending on the level and activity, the value in this case does not show the specific rehabilitative potential of the system. There are several reasons for this.

Firstly, while in studies like ours, the main parameters for assessing the effectiveness of the system were precisely the rehabilitation indicators, and it was the result of rehabilitation actions that was compared and evaluated, in our case, the signals into which the system converts the player’s performance are researched. Therefore, in this case, it is the indicators of the received signals, such as jitter, shimmer, amplitude–frequency response, and so on, that are important for the study presented here. It is on this comparison that the working principle of the platform presented is based.

Secondly, because of the situation regarding the COVID-19 pandemic, it was not possible to conduct tests with the target audience. Bearing in mind that young people (from 21 to 42 years old) took part in the study, the results obtained from them are not entirely accurate, albeit at the same time, proving extremely useful for further research purposes.

Thirdly, which follows on from the first point, based on the results obtained, it turned out to be a wrong solution to study such indicators as the Activities of Daily Living (ADLs), Cognitive Function Scale (CFS), Nottingham Health Profile (NHP) [45] score. The reason for this is that it would be incorrect to compare the performance of healthy young people to results from elderly people.

Fourthly, one of the main tasks at this stage of research was to find out the value of the system’s effectiveness, as well as its capabilities as an additional module for rehabilitation procedures, which, unfortunately, we were unable to fully test at the time of writing this article due to the COVID-19 pandemic.

Emotional component. While testing the platform, most of the players noted that performing physical activities in the form of a computer game was a more interesting experience for them, as opposed to conventional physical activities. In this study, we did not test participants on their emotional state after the game, as the data obtained would not have been entirely correct. The explanation for this is that older people react differently to computer games than the age group that took part in the test. Therefore, the estimated results would have been different.

Nevertheless, the feedback provided a certain kind of information that will help to further develop the system.

6. Conclusions and Future Plans

This article presents an interactive system developed to help rehabilitate and support the elderly, as well as patients with musculoskeletal system problem. The main idea was to create an additional tool that would be as simple as possible and would not require any additional material resources, designed to improve the rehabilitation process, and to make it more interesting. At this stage, the developed platform is an independent gaming platform, which includes a set of tools and algorithms aimed at improving the rehabilitation process itself, as well as aspects of it. During the development and testing of the system, there were certain questions that needed to be addressed, as follows.

Question 1: Technical Basis. Given that the original goal was to create a system that can be used by different users, it was decided to create an interactive gaming platform, in other words, a serious game that the user can install easily and use at any time.

The main point was that the player’s indicators should be recorded in real time with the use of a minimum number of additional means. That is why a body tracking system was used with only a webcam, while providing the maximum level of functionality.

Question 2: Medical rationale. That set of 5 basic exercises that were integrated into the platform form part of a set of rehabilitative activities specified by a physiotherapist. Considering the specific target population, namely, elderly people from 60 to 85 years old, as well as people with minor musculoskeletal system problems, the set of exercises used can support the user in physical terms in a complex way, similar to the usual exercise. The only difference is that the interpretation of these exercises and the presence effect make physical activity more meaningful and visually attractive to the user.

Question 3: How valuable is the system, and how can it be used? In addition to its use in the home as a regular entertainment system, it is also of medical value. Thanks to the built-in player tracking and analysis system, all player results are stored and can be viewed immediately after the game or after a period of time. Furthermore, these results can be used as a record of the rehabilitation process, owing to data systematization. This feature is very useful for doctors because it allows them to monitor the patient’s progress not only in the hospital, but also remotely.

Limitations: Unfortunately, owing to the situation regarding the pandemic, it is not possible at present to compare the results of the effect of the system developed directly on the target audience. This is also because those subjects who participated in this study do not have any problems with the musculoskeletal system, and their ages are lower than the target age. Nevertheless, the results clarify many aspects and allow us to draw conclusions about the effectiveness of the platform.

Future plans: There are also plans to improve the system. In view of the current situation regarding the COVID-19 pandemic, the testing process was unfortunately deprived of the opportunity to obtain results directly from the target audience. Nevertheless, we plan to test elderly people in the near future.

We also plan to expand the set of integrated exercises, and their different gradation in terms of complexity and orientation. At present, research is underway on the use of additional material aids, such as smart bands, balance sensors, and so on. Work is also already underway to connect additional tools and sensors, although first and foremost, the plan is to implement and connect levels that will be aimed at analyzing and training the player’s balance.