Proteo: A Framework for Serious Games in Telerehabilitation

Abstract

Within the context of telerehabilitation, serious games have a significant role, but creating software for serious games is resource demanding. We present Proteo, a modular and open-source framework for developing serious games from scratch. We also present two serious game implementation examples with analysis of end user and therapist/researcher satisfaction. By involving a group of 11 specialized therapists and 9 end users we analyzed the Proteo’s user satisfaction. We found that both groups scored high for the level of involvement, and the therapists scored also high for the level of suitability. More in depth, both groups showed significant differences between positive and negative feelings, with positive feelings scoring higher than negative ones. Finally, the user level of suitability was reported as high while the difficulty of the system and the difficulty of the task were reported as low. Proteo has proven to be a useful tool to develop serious games for telerehabilitation and has been well accepted by the users involved in the evaluation tests.

1. Introduction

Telerehabilitation (TR) can be defined as delivering rehabilitation services to the patient’s home through a remote connection with the therapist, possibly enhanced by additional multimedia technologies [1,2,3]. TR can be provided using a variety of technological tools and modalities, for example, real-time visits with audio, video, or both; asynchronous e-visits, telephone assessments; closed-circuit television to video conferences; internet-based communication software (e.g., teleconference platform) with headsets, webcams, speakers, handheld cameras, microphones, and internet; serious games; or remote evaluations of recorded videos or images [1].

Several systematic reviews have argued that TR is an effective tool for patients with different disabilities, from musculoskeletal conditions, multiple sclerosis, neurological disorders, and stroke to autism and genetic syndromes [4,5,6,7,8,9]. Moreover, it was found that TR services are effective both for children and adults with disabilities, and professionals have also reported high levels of satisfaction and acceptance of TR services [5,10,11,12]. However, it is expected that advancements in technology can increase the usability and effectiveness of TR, but current TR systems do not usually allow for therapists to customize or adapt these systems to the characteristics of the patient and setting, and they are limited to specific patient conditions [13,14,15,16,17,18]. This can be due to different factors such as the complexity of the process, the technical aspects of TR service development, and the heterogeneity of patients with disabilities. Thus, it is necessary to develop and test the suitability and safety of TR modalities that can be customized by the therapist across subgroups of patients.

The last year, which has been deeply impacted by the COVID-19 pandemic, has led to a significant increase in the use of remotely guided rehabilitation and therapy interventions [19,20,21,22,23]. However, patient conditions vary significantly, and the need for a framework to define customized TR systems that can improve the engagement of the involved patients arises, with the aim of facing the enormous variability in patient specificity and needs. Moreover, such a framework should have a modular structure, such that a therapist can add support for specific hardware devices (i.e., eye trackers) or software capabilities (i.e., computer vision, machine learning) which would be useful in allowing the patients interact with the system and to acquire data for further analyses. Consequently, the implementation of TR leads to a double challenge: motivating patients and ensuring an effective system [24,25,26].

In this scenario, we propose Proteo, a framework designed to give therapists and researchers without specific skills in software development the capability of defining TR service customization and developing rehabilitation-oriented serious games, starting from game templates. Moreover, the framework can be applied in rehabilitation centers that could easily develop suites of remotely administered serious games, focusing only on a simplified game implementation, exploiting the framework modules for client/server communication, user management, human–computer interaction, or deep learning techniques.

This framework aims at: (a) the creation of personalized serious games for TR systems; (b) monitoring of the therapeutic goals of the rehabilitation process; (c) simplifying the interaction schemes between the patients and serious games. Moreover, Proteo is open-source and cross-platform, and it is distributed under the MIT license.

In the next sections, after presenting Proteo, we discuss two use cases to analyze the practicability of the serious games creation process and to discuss user satisfaction. In this paper, we present experiments involving therapists and researchers, while we plan to apply Proteo in telerehabilitation programs involving people with disabilities as part of future work.

2. Materials and Methods

Several libraries and frameworks have been proposed to facilitate the programming of visual/artistic applications (e.g., openFrameworks.cc) and games (e.g., stencyl.com accessed on 11 May 2021, defold.com accessed on 11 May 2021, www.yoyogames.com/en/gamemaker, accessed on 11 June 2021, https://gdevelop-app.com/, accessed on 11 June 2021). Libraries such as openFrameworks can benefit from using the C++ language, which has various modules that are specifically designed for artistic purposes with computer vision support, a crucial tool when dealing with serious games. However, openFrameworks is intended for C++ developers, which can be a barrier to those who have no programming skills. On the other hand, applications and libraries for game development (even with a simplified block programming interface) are missing significant tools necessary in the field of telerehabilitation, such as specific computer vision tools aimed at pose estimation, face detection, facial expression recognition etc.

The aim of the proposed platform, Proteo, is hence to give therapists and researchers without specific skills in software development, the capability of defining serious game customizations starting from game templates. Given a game template, which defines the game characteristics, Proteo enables therapists/researchers to define the game logic and the user interaction in a simplified manner, focusing only on what she/he wants to analyze, overcoming technological barriers and concentrating only on defining tasks and collecting data.

Moreover, the framework can be applied in rehabilitation or research centers with software development skills that could easily develop suites of remotely administered serious games, focusing only on a simplified game implementation, exploiting the framework modules for client/server communication, user management, a database for collecting data, human–computer interaction, or deep learning techniques, thus potentially gathering a large amount of data that can enable the exploitation of machine learning techniques [27].

In the following, we present Proteo, an open-source and cross-platform scriptable platform that is highly modular with built-in support for many useful modules for serious game implementation. We analyze two use cases testing Proteo with two different kinds of users: therapist/researchers and end users. In the first case, we ask the therapist/researcher group to customize a serious game after being given a game template through the platform. In the latter, we ask a group of end users to play a serious game developed with Proteo. In both cases we collected data regarding user satisfaction, which will be discussed in the Results section. Moreover, the technical details of the platform are presented in Appendix A.

2.1. Serious Game Implementation with Proteo

The process of implementing a serious game with Proteo involves a multidisciplinary team. As mentioned previously, the aims of such a platform are: (i) to facilitate the implementation of a serious game by providing specific tools and libraries to developers and (ii) to give therapists the possibility to customize serious games in terms of user interaction, data to acquire etc.

Starting from an initial idea, developing a serious game with Proteo involves a precise workflow, as shown in Figure 1.

The overall project is discussed between IT experts and clinicians, defining the gameplay and which parameters the clinicians want to customize. Depending on the proposed activity, these parameters can affect different aspects of the game (e.g., audio, user interaction, acquired data etc.) thus determining which Proteo module is needed. For example, in GymTetris, which we present in the following section, it is necessary to match each of the four possible movements of the tetrominoes with a specific body movement. In this case, the module for the pose estimation is required.

Once these parameters are selected, the developer implements the serious game application in Lua. In addition, the developer implements a simplified Blockly interface so that therapists can customize the serious game in a simplified manner, without changes to the Lua code.

To test Proteo and to collect preliminary regarding about the usage of the framework, we prepared two serious game examples, namely GymTetris and Problem Solving Task, which are presented in the following section.

2.1.1. GymTetris

GymTetris is based on the popular Tetris game (https://en.wikipedia.org/wiki/Tetris, accessed on 11 May 2021) and was conceived to encourage physical exercise. This serious game follows the dynamics of the classic game, but the user moves or rotates the tetrominoes by means of body movements. A scheme of the Proteo modules used to develop the game is shown in Figure 2.

As previously discussed, the game template provides a base that the therapist can customize while considering the needs of the user to which the game will be deployed. In the case of GymTetris, the game can be customized in terms of user interaction and data collection: the therapist can choose which body movements must be performed by the user to interact with the game by using the Blockly [28] editor (Figure 3a). Moreover, the therapist can decide which user data to collect during the game sessions (Figure 3b).

Figure 4 shows a screenshot of the game (you can also find the game implementation on GitHub: https://massimobernava.github.io/proteo/, accessed on 11 June 2021) (for more information, see Supplementary Materials section). To detect user movements, the game incorporates a pose estimation model [29].

2.1.2. Problem Solving Task

Problem Solving Task is a game in which the user has to solve cognitive tasks, such a series completion. Moreover, the user can interact with an avatar represented by an animated character that speaks to foster the user’s attention. In Figure 5, a scheme of the Proteo modules involved in the game is shown, while in Figure 6, a game screenshot is presented.

In Problem Solving Task, four main blocks are provided for game customization (Figure 7):

• Questionnaire page creation;
• Cognitive task page creation;
• Avatar creation;
• User attention trigger.

The questionnaire page is a page where different types of questions can be entered and contains a button that can trigger a customized action (e.g., continue to the next page or save the data and conclude the test).

The task page includes the presentation of a main image and nine images from which the user must choose the correct image to complete the exercise. The block allows the therapist to select which answers are correct and optionally shuffle the answers before displaying them. Depending on the correct or incorrect answer, the therapist selects the consequent action (e.g., an animation by the avatar).

Through the “show page” block, the first page to be displayed is selected, and it is possible to create a sequence of pages to be shown.

The avatar has four possible animations. The first animation is performed cyclically by the avatar (where there is no activity). The other three animations can be activated according to the therapist’s settings by combining them with sounds or spoken sentences.

The last block allows the therapist to select which actions to take if the user is distracted or attentive (e.g., an animation by the avatar).

In this case, the therapist can choose the images needed for a specific task and how to analyze user performance and collect data but can also decide how and if the avatar should interact with the user. Here, the user interaction to choose the correct image to complete the series is conventionally based on conventional mouse interaction, however, the actions of the avatar can be activated by more natural input modalities, for example, by the user’s level of attention. This kind of avatar interactivity has indeed been proven to be very important to enhance human–computer interaction [30], especially for users with an attention deficit [31].

In the following section, we present our findings of the User Satisfaction Evaluation Proteo Questionnaire, both from a group of therapists and end users.

2.2. Sample and Recruitment

A convenience sample of specialized therapists was recruited from the Department of Medicine and in the hospital health units at the University of Messina and the San Raffaele Hospital of Milan to test the Problem Solving Task example for both involvement and suitability. The inclusion criteria were to have at least 3 years of experience with rehabilitation training. No attention was paid to previous experience with the use of computers or of computer apps. Twenty potential participants were approached personally or by phone. Nine of them refused to participate, and eleven were included. The final sample was composed of 11 specialized therapists in cognitive and motor training for people with disabilities. They had a mean age of 31.48 with a range of 26–43; ten participants were female and one was a male. Another convenience sample of end users was also recruited to test the Proteo modular framework to test GymTretis. The nine end users had a mean age of 32.08 with a range of 25–44; four participants were female and five were male.

2.3. Outcome Measures

In this study, the User Satisfaction Evaluation Proteo Questionnaire (USEPQ) was used. USEPQ is based on USEQ [32], but it was specifically designed to test the Proteo system (see

Table 1. The USEPQ (User Satisfaction Evaluation Proteo Questionnaire).

Question	Response
	Not at All–Very Much
Previous knowledge/experience
Q1. Please indicate your level of computer experience	1	2	3	4	5
Q2. Indicate your level of experience with the apps on your computer	1	2	3	4	5
Q3. Have you ever used a system similar to Proteo?	No			Yes
Q4. Indicate your level of knowledge of serious games	1	2	3	4	5
Q5. Indicate your level of knowledge in creating serious games	1	2	3	4	5
Q6. Indicate your level of knowledge of the security/privacy	1	2	3	4	5
Level of involvement
Q7. I enjoyed trying Proteo	1	2	3	4	5
Q8. I felt involved	1	2	3	4	5
Q9. I felt tired	1	2	3	4	5
Q10. I felt bored	1	2	3	4	5
Level of adequacy/suitability
Q11. Were you successful using the system?	1	2	3	4	5
Q12. Were you able to control the system?	1	2	3	4	5
Q13. Is the information provided by the system clear?	1	2	3	4	5
Q14. Do you think that this system will be helpful for telerehabilitation?	1	2	3	4	5
Q15. How difficult did you find the system?	1	2	3	4	5
Q16. How difficult did you find the task?	1	2	3	4	5
Q17. Did you feel discomfort during your experience with the system?	1	2	3	4	5
Q18. If yes, why?

). Essentially, USEPQ extends USEQ, including questions to get responses from therapists on items related to Proteo. The Core Module measures the users’ previous knowledge and experience with applications such as Proteo through different components (see Table 1), the level of involvement with the components (with negative affect and positive affect), and the level of suitability of the system and the tasks, with a total 18 items.

USEQP for therapists includes 18 questions, 16 of them with a response graded on a 5-point Likert Scale (from 1 = “not at all” to 5 “very much”) and a final open question. Following the USEPQ scheme, the first six questions measure the users’ previous knowledge and experience with apps such as Proteo, with computers, the knowledge of serious games, and the capacity to create and develop software. The next four questions measure the involvement of the user (if he enjoys using Proteo, if he feels involved, if he feels bored or tired). The third group, composed of eight questions, evaluates the level of suitability of Proteo. In this group, the first question (Q1) evaluates the successfulness in using Proteo, the second (Q2) evaluates the level of control, the third (Q3) evaluates the clearness, Q4 evaluates the level of helpfulness, Q5 and Q6 evaluate the perceived difficulty of the task and of the system, Q7 evaluates discomfort, and the last question, (Q8), requires a final explanation open answer. The global score of the first subscale ranges from 5 (poor previous knowledge) to 25 (excellent previous knowledge). Before calculating the global score of the second subscale, Q9 and Q10 have to be reversed, and the global score ranges from 4 (poor level of involvement) to 20 (high level of involvement). Before calculating the global score of the third subscale, Q15, Q16, and Q17 have to be reversed, and the global score ranges from 7 (poor suitability) to 35 (excellent suitability).

The USEPQ for end users only includes nine questions (Q1–Q4, Q6–Q10).

2.4. Data Analysis

Because of the ordinal nature of the measures, non-parametric statistical tests were used for data analyses. Hence, the median was used as a measure of central tendency and the interquartile range (IQR) for dispersion. To test for differences across conditions, the Wilcoxon signed-rank test was used for pairwise comparisons, with significance values adjusted by the Bonferroni correction. The effect size was calculated by dividing the absolute (positive) standardized test statistic z by the square root of the number of pairs:

$$d=\frac{z}{\sqrt{n}}.$$

Data were analyzed using IBM Statistics for Mac, Version 20.0 (IBM Corp., Armonk, NY, USA).

3. Results

Results are discussed first with reference to the therapists and later with reference to the end users. The three sections of Proteo related to the users’ previous knowledge, the involvement of the user, and the level of suitability were analyzed for the therapists, and the two sections on previous knowledge and involvement were analyzed for end users.

3.1. Users’ Previous Knowledge

Concerning Users’ previous knowledge, comparisons with the Wilcoxon signed-rank test revealed that Serious game creation knowledge was significantly lower than Computer experience (z = −2.99; p < 0.011, d = 0.84), Apps experience (z = −3.02; p < 0.001, d = 0.94) and Security rules knowledge (z = −2.77; p < 0.011, d = 0.77).

In

Table 2. Medians and interquartile range in the Proteo experience questionnaire components.

Previous Experience	Median	Interquartile Range
Computer experience	3	2–4
Apps experience	3	2–4
Serious game knowledge	2	1–3
Serious game creation	1	1–2
Security rules knowledge	3	2–4
Level of involvement
Level of enjoyment	4	4–5
Level of involvement	4	4–5
Level of tiredness	2	1–3
Level of boredom	1	1–2
Level of suitability
Successfulness	4	3–4
Ability to control	3	3–4
Clarity	4	3–4
Helpfulness	4	3–5
Difficulties with the system	3	2–3
Difficulty with the task	2	2–3

and Figure 8, all conditions (Computer experience, Apps experience, Serious game knowledge, Security rules knowledge) were reported as presenting an intermediate level of Knowledge (out of 3 points). Their level of Serious game creation knowledge was considered low.

3.2. Involvement of the Therapist Users

Regarding the user’s level of involvement (Figure 9), the conditions of positive feeling (level of enjoyment and level of involvement) were reported as high (out of 4 points) while the conditions of negative feeling were reported as low (1 or 2). Concerning engagement (Table 1), in all conditions, pairwise comparisons revealed that enjoyment and involvement were significantly higher than tiredness and boredom (enjoyment vs. tiredness: (z = −2.725; p < 0.009, d = 0.86), enjoyment vs. boredom: (z = −2.971; p < 0.009, d = 0.94), involvement vs. tiredness: (z = −2.625; p < 0.009, d = 0.83), involvement vs. boredom: (z = −2.873; p < 0.004, d = 0.90).

3.3. Level of Suitability

Concerning the user level of suitability (Figure 10), the conditions of successfulness, ability to control, clarity, and helpfulness were reported as high (between 3 and 5), while the difficulty of the system and the difficulty of the task were reported as low. Pairwise comparisons with the Wilcoxon signed-rank test revealed that the first four conditions were significantly lower than the last two: difficulty of the task with successfulness, ability to control, clarity, and helpfulness, respectively: z = −1.951; p < 0.09, d = 0.51; z = −1.54; p < 0.07, d = 0.49, z = −1.799; p < 0.05, d = 0.59 and z = −2.36; p < 0.009, d = 0.75, and the difficulty of the system with successfulness, ability to control, clarity, and helpfulness, respectively: z = −1.83; p < 0.09, d = 0.53; z = −1.49; p < 0.08, d = 0.49, z = −1.83; p < 0.03, d = 0.61 and z = −2.41; p < 0.009, d = 0.81.

3.4. End Users’ Previous Knowledge

Concerning pervious user knowledge (

Table 3. Medians and interquartile range in the Proteo experience questionnaire components.

Previous Experience	Median	Interquartile Range
Computer experience	5	3–5
Apps experience	4	1–5
Serious game knowledge	3	1–5
Security rules knowledge	2	3–5
Level of involvement
Level of enjoyment	4	4–5
Level of involvement	3	3–5
Level of tiredness	1	1–1
Level of boredom	2	1–3

and Figure 11), all conditions (computer experience, apps experience, serious game knowledge, and security rules knowledge) were reported as presenting a high level of knowledge (out of 4/5 points). The level of serious game knowledge was low (2 points). Pairwise comparisons with the Wilcoxon signed-rank test revealed that serious game knowledge was significantly lower than computer experience z = −2.51; p < 0.009, d = 0.83.

3.5. Involvement of the User

Regarding the level of user involvement (Table 2 and Figure 12), the conditions of positive feeling (the level of enjoyment and level of involvement) were reported as high (out of 4 points) while the conditions of negative feeling (boredom and tiredness) were reported as low (1 or 2). Concerning engagement (Table 2), in all conditions Pairwise comparisons revealed that enjoyment and involvement were significantly higher than tiredness and boredom (enjoyment vs. tiredness: (z = −2.99; p < 0.001, d = 0.94), enjoyment vs. boredom: (z = −2.771; p < 0.009, d = 0.87), involvement vs. tiredness: (z = −2.92; p < 0.009, d = 0.90), involvement vs. boredom: (z = −2.673; p < 0.004, d = 0.88).

4. Discussion

We then tested the process of game customization by therapists and analyzed the levels of experience, involvement, and suitability for both therapists and end users. Data from the different components of the User Satisfaction Evaluation Proteo Questionnaire assessed the impact that the two game experiences had on participant engagement. Overall and irrespective of the game mode, participants reported low feelings of boredom and tiredness, i.e., low levels negative affect and highs level of positive affect (involvement and enjoyment). Irrespective of previous experience level, the suitability with the conditions of successfulness, ability to control, clarity, and helpfulness were also reported as high.

More in depth, with reference to the therapists, before starting the study, we wanted to refer to a sample of therapists who were not very experienced in the use of the PC or in the use of the different apps; the results concerning the therapists’ previous knowledge confirmed these data, as the therapists show an intermediate level of knowledge and, as we expected, their experience in creating a serious game was very low. With reference to the therapists’ level of involvement, the condition of positive feeling was reported as high while the condition of negative feeling was reported as low. With reference to users’ previous knowledge, the sample we contacted to participate in the research had a high level of knowledge, and their level of involvement was also very high. We believe that this positive impact of the system on the subjects’ feelings was mainly due the easiness and the fun of the system. This was confirmed by the data related to the level of suitability described by the therapists. As we underlined, the therapists were able to finish the assigned task, and they reported a strong feeling of being in control of the process and that the Proteo app was clear and helpful.

In clinical practice, frameworks such as Proteo can enable simultaneous training of physical, cognitive, and psychological aspects and show great potential to improve patient motivation and adherence, mainly due to the possibility of gamification and personalizing the feedback from TR modalities that can be customized by the therapist across subgroups of patients. In order to analyze the suitability of the system, our results are promising; in order to analyze the safety of Proteo app, we have no data because nobody replied in an affirmative way to the question: “Did you feel discomfort during your experience with the system?” For this reason and because no therapist reported problems during the task, we believe that Proteo is safe. However, in order to systematically adopt Proteo in routine clinical practice, the ease with which the therapists completed the assigned task and the pleasure of creating new tasks are not sufficient for the success of the therapy. It is necessary that the therapists are well-trained. They must be competent when determining what feedback to give and at what moment, they and must be aware of the hierarchy of learning objectives, the transfer of them, and all of the factors that can lead to the autonomy the patient. The therapists also need to be careful not to give patients more help than necessary.

This study has some limitations that should be acknowledged. Regarding the data analysis, our sample size is relatively small for some of the statistical analyses performed, and the results should be considered with caution. However, sample size calculations determined that the sample size was sufficient to evaluate the study objectives.

5. Conclusions

Telerehabilitation systems have been widely used in the recent years, showing their crucial importance during the COVID-19 pandemic. In the field of telerehabilitation, serious games play an important role since they can foster user engagement and hence, enhance rehabilitation outcomes. However, telerehabilitation systems based on serious games are often suited for specific patient conditions, so implementing new games or even customizing existing ones is resource demanding and requires software development skills. Moreover, several libraries exist to ease game development, but they lack in offering advanced user interaction methods, such as those that use computer vision algorithms or eye-tracking devices.

In this paper, we presented Proteo: an open source, modular framework for developing serious games from scratch, providing a high-level interface for serious game development and customization. Proteo provides many built-in modules to cover most common features that a serious game may need, such as client/server communication, user management, data management, advanced human-computer interaction, computer vision, or deep learning techniques.

Proteo has proven to be a useful tool to develop serious games for telerehabilitation and has been well accepted both by the therapists and by the users we involved in the evaluation of suitability questionnaire.

As for future work, we plan to conduct more extensive tests involving a larger group of therapists and rehabilitation researchers. Moreover, we plan to use Proteo in structured telerehabilitation programs involving people with disabilities.

Another step is to explore other varieties of games, such as complex task units (such as with math problems). Finally, we consider it important to understand if the obtained results will be similar with larger group sizes and will also investigate the applicability of Proteo to different settings such as in families, schools, and hospitals.