Next Article in Journal
A Spatial Adaptive Algorithm Framework for Building Pattern Recognition Using Graph Convolutional Networks
Previous Article in Journal
Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments
Previous Article in Special Issue
Sensor Technologies for Caring People with Disabilities
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 19
Issue 24
10.3390/s19245517
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Hand Rehabilitation and Telemonitoring through Smart Toys

by
N. Alberto Borghese
1,*,
Jacopo Essenziale
1,
Renato Mainetti
1,
Elena Mancon
2,
Rossella Pagliaro
2 and
Giorgio Pajardi
2
1
Department of Computer Science, AIS-Lab Laboratory of Applied Intelligent Systems, Università degli Studi di Milano, 20133 Milan, Italy
2
Hand Surgery Unit, San Giuseppe MultiMedica Hospital, 20123 Milan, Italy
*
Author to whom correspondence should be addressed.
Sensors 2019, 19(24), 5517; https://doi.org/10.3390/s19245517
Submission received: 1 October 2019 / Revised: 25 November 2019 / Accepted: 3 December 2019 / Published: 13 December 2019
(This article belongs to the Special Issue Sensor Technologies for Caring People with Disabilities)
Browse Figures
Review Reports Versions Notes

Abstract

:
We describe here a platform for autonomous hand rehabilitation and telemonitoring of young patients. A toy embedding the electronics required to sense fingers pressure in different grasping modalities is the core element of this platform. The system has been realized following the user-centered design methodology taking into account stakeholder needs from start: clinicians require reliable measurements and the ability to get a picture remotely on rehabilitation progression; children have asked to interact with a pleasant and comfortable object that is easy to use, safe, and rewarding. These requirements are not antithetic, and considering both since the design phase has allowed the realization of a platform reliable to clinicians and keen to be used by young children.

1. Introduction

The hand, both morphologically and functionally, represents one of the main elements that characterize humankind and serves as the main tool by which to interact with the environment in daily life. High finger mobility and opposable thumbs allow us to pinch, grab, manipulate, and interact with objects, enabling us to effectively explore the world, proving to be an essential element of the learning process throughout our life cycle, from childhood to old age [1].
During pregnancy, embryogenesis of the upper extremities takes place between the fourth and eighth week after fertilization, and the majority of congenital anomalies of the upper extremity occurs during this period. These occur nearly twice in every 1000 children born [2], and they manifest under several typologies with varying degrees of seriousness. In the most severe cases, when a few or no fingers are present (Oligodactyly), current clinical practice is based on surgically transferring one or more toes to the hand to enable it to grasp objects for interaction. However, to achieve this goal, a long rehabilitation period is required. Rehabilitation is also required, although for a shorter time, for those children who, after domestic or outdoor accidents, suffer from injuries to their hands (from fractures to finger amputation), requiring clinical intervention.
Classical rehabilitation is constituted of a set of exercises aimed to increase finger strength, coordination, speed, and accuracy of movement. Moreover, for transfer surgery, such exercises are aimed also to integrate implanted fingers into a new body scheme [3]. To engage children, exercises are usually carried out through toys (e.g., building blocks), that are manipulated by children according to the exercises goal and under the instructions and supervision of a professional therapist in one-to-one sessions. Often, this requires commuting to rehabilitation centers, inflicting significant organizational and economical stress on children’s families [4].
Information and C ommunication Technology (ICT) has the potential to change this, as it can enable patients to carry out part of the rehabilitation program at home. In recent years, the development of low-cost tracking devices, like Nintendo Wii Balance Board (https://www.nintendo.com/consumer/downloads/wiiBalanceBoard.pdf) and Microsoft Kinect (https://developer.microsoft.com/en-us/windows/kinect), that have shown valid accuracy [5], and the distribution of professional-level game engines, usually without costs for educational institutions (e.g., Unity 3D (https://unity.com/), Unreal (https://www.unrealengine.com/)), has been recognized to have a large potentiality in the rehabilitation field, especially for children for whom videogames are a natural entertainment tool. This has spurred the development of several rehabilitation platforms based on exergames, in which patients move an avatar inside the graphical interactive game environment through the tracking device in use to fulfil some game task. The movements elicited are functional to rehabilitation.
The most advanced platforms of this kind enable autonomous exercising and supervision is offered by clinicians through the data acquired by the platform itself during training [6,7]. This is a form of telemonitoring, which is defined as the use of information and computer technology to remotely monitor some aspect of the user’s clinical state at his/her home. It is relatively new in medicine but it has quickly developed particularly for certain pathologies. For instance, heart is monitored remotely transmitting the patient’s ECG to the hospital, where it is automatically analyzed on the fly; if a possible condition related to heart failure is detected an alarm is triggered to the patient and to clinicians, saving lives [8]. Given its success, telemonitoring is spreading fast on many other clinical domains like autonomous rehabilitation.
Several platforms, generically based on exergames and external tracking devices, both commercial [9,10] and research prototypes (e.g., [11,12,13,14]), have been developed to support autonomous posture [12,15,16,17] and arms [11] rehabilitation, but no platform of this kind, as far as we know, has been developed yet for hand rehabilitation especially in the pediatric age. This domain, in fact, provides a three-fold challenge: tracking the motion of the fingers, measuring grasping force, and providing a system that can engage children. Several platforms are described in literature (see Background section), but none of them can integrate all these aspects. Moreover, most of them do not provide a quick way to tune the exercise difficulty according to the child current motor ability, a fundamental feature to ensure an adequate challenge level.
We show here how a complete hand at-home rehabilitation system, based on exergames, with telemonitoring capability, can be realized using a tablet and an adequately designed pressure sensing device that is used as tracking device inside the exergames. To this aim, a modular flexible architecture that can sense fingers pressure has been designed. It wirelessly transmits pressure data to the host device, typically a smartphone or a tablet, on which the exergames are run. The incorporation of emotional aspects in the design of the sensor case has allowed for high acceptability among children. At the end of each game session, the exercise data are sent to the cloud becoming available to evaluate the progress, to schedule recall visits at the hospital, and to select a proper set of exergames and set their difficulty according to the actual child’s capabilities.
The paper is organized as follows: Section 2 reviews the most promising approaches to hand rehabilitation and telemonitoring. Section 3 describes in detail the platform that is discussed in Section 4. Section 5 draws the final conclusions.

2. Related Research

The first attempts to use technology to support hand rehabilitation come from the robotic field where instrumented gloves [18,19,20,21,22] and exoskeletons [23,24,25,26] have been proposed, also in combination with exergames. These systems have proven to be very useful when dealing with patients who do not have enough muscle power to move their hands on their own. However, they limit the freedom of patients’ movement and alter proprioception and natural behavior of the hand [27]. Moreover, they are bulky, not easy to dress, costly, and are therefore restricted primarily to clinical use with post-stroke patients. Lastly, their dimension makes it difficult to use them in pediatric hand rehabilitation.
For these reasons, sensing devices that have not to be attached to hands are preferred. A possible approach is based on using cameras to survey movement. Motion capture systems, based on using passive markers to robustly identify the finger joints on the camera images, were first proposed [28]. Although this approach is potentially accurate, it is limited to the laboratory domain, as the set-up time required to attach the markers to the hand makes this approach unfeasible at home. Of more interest are markerless approaches to motion capture. Recently, the Leap MotionTM [29,30] and RGB-D cameras (e.g., KinectTM [31]) have been introduced to track the human body. However, such systems require that relatively simple movements, well visible to the camera, be performed. Moreover, their accuracy is not adequate for capturing free grasping movements [32,33]. Lastly, they require that the user lies inside the working volume of the device, while children tend to move a lot while playing. For these reasons, such devices are not a good match for hand rehabilitation, especially at a pediatric age.
Solutions that do not have such problems are based on touch screen devices in general and on tablets in particular. Tablets can detect the movement of one or more fingers on its surface with high accuracy [34,35,36,37,38,39]. Combined with exergames, they allowsguiding the users in exercises aimed to recover hand dexterity. Most of these solutions allow setting an adequate degree of difficulty and in some cases provide the clinicians with a log of children activity. However, a clear definition of an adequate set of rehabilitation exercises and their quantitative evaluation is still largely missing. Moreover, neither force exercises can be provided with these approaches nor grasping of real objects.
More recently, to enable force exercises, a few attempts to embed sensors inside objects have been described. These objects are inspired by the Myogrip [40], Takei [41], Jamar [42], or Vigorimeter [43] handheld dynamometers, that are used to measure the maximum grasping force exerted squeezing their handle. All of these devices are meant for clinical use. A few attempts to provide equivalent measurements at home have been proposed. The Grip-ball system is constituted of an inflated ball containing a pressure sensor and the associated electronics [44]. A similar approach has been adopted in the Domo-grip system [45], in the SqueezeOrb device [46], the latter developed for human-machine interfacing, and in the Lokee smart ball [47] or the Gripable device [48]. However, such devices allow only the palm power grasp and not all the other finer prehension modalities typical of the human grasping repertoire [49].
Moreover, all of these sensing devices maintain a clinical-like structure, in which the tracking object is clearly distinguishable and associated with measurement. This has been recognized to be an obstacle to adoption especially for children and the idea of instrumenting objects of everyday life has been recently pursued. In CogWatch project [50] objects of everyday life (e.g., coffee machines, mugs) have been instrumented with inertial and contact wireless sensors to gather data that can be used to assess the correct sequence of gestures (e.g., preparing a coffee) to early detect cognitive decline. Sen.se [51] has pushed this approach further providing motion sensors that can be attached to any daily life object. However, such approaches do not allow recovering detailed information on hand interaction. A more suitable approach to hand rehabilitation has been pursued in the Caretoy project [52], that has developed wireless cylindrical toys made of elastomer with embedded sensors. The toys are composed of two soft air-filled chambers connected to a pressure sensor; a rigid case inside the toy contains the control unit. We leverage this approach to realize a flexible pressure-sensing device that can be used to measure fingers pressure in different grasping modalities giving to it the aspect of a toy. This device, combined with a tablet device and with suitable exergames, allows supporting the large variety of exercises required for hand rehabilitation.

3. Materials and Methods

3.1. Platform Specifications

To provide a valid solution, the starting point is eliciting the functional specifications by all stakeholders [53], who, in our case, are:
Clinicians (physiotherapists, neurodevelopmental disorders therapists and surgeons): They require a tool that can guide the child through adequate exercises, which can be adapted to child current motor abilities. Moreover, they require remote monitoring of the rehabilitation progression from the hospital.
Patients (children): They have to use the platform for exercising autonomously. The platform should be easy to use and fun at the same time. We aim here to children between 2 and 7 years old for which a maximum force range of 5 kg has been defined.
Caregivers: These are, preferentially, family members. They need to minimize travel to the hospital with their child and maintain a tie with the clinicians at the hospital to tune exercise and optimize recall visits.
We have to co-design the platform with all of these stakeholders. To realize a system that can be useful and functional for the end-users, we have adopted two parallel processes throughout the development phase: the Design Thinking Process [54] and Human-Centered Design [55,56]: the first is based on searching for possible solutions to the problem, keeping the needs of the users at the center of the design process, while goal of Human-Centered Design is to maximize the usability and user experience of the system. Accordingly, we have involved all the users early in the design loop to take care effectively of several key interaction aspects: ergonomics, ease of use, and maintenance of the system, as well as effectiveness.
In the first phase, we have identified a core set of exercises required by observing live sessions and through focus groups with clinicians. These belong to two categories:
Strength exercises: Aim at strengthening fingers muscles to regain the power required to grasp objects. To achieve this, patients are required to grasp objects using different prehension modalities: pinch, palm, or side grasp [49]. Typically, therapists set a specific force, duration and intensity of the exercises and a repetition frequency.
Mobility exercises: Aim at regaining mobility, increasing sensitivity and range of fingers motion. To this aim, patients are required to tap, slide or pinch (e.g., like in the zoom-in/out functionality of many software applications) on a surface using one or multiple fingers. Here, therapists typically set the range of motion, its speed and accuracy [57].
We explicitly remark that the degree of difficulty of each exercise is regulated upon the current child’s motor ability.
The two sets of exercises call for different hardware requirements. The first set requires hardware capable of measuring finger strength when the user interacts with objects with different finger arrangements according to the different grasping modalities. The second set requires hardware capable of precisely track fingers position.
A single instrument cannot satisfy the two requirements in an unobtrusive and ecological way. If mobility exercises can be performed on the tablet screen as shown in [34,35,36,37,38,39], a sensing object that can be used for strength exercises was not available and has been developed as described here.

3.2. Platform Design and Development

3.2.1. Force Sensing Sub-System

The core element of the platform is a specifically designed toy that embeds the dedicated hardware capable of sampling, digitizing and wirelessly transmitting in real-time pressure measurements to the mobile device (typically a tablet or a smartphone) on which the exergames are played (Figure 1). The toy realized here is based on LegoTM that is a typical construction familiar to children of young age, and has the shape of a cottage (Figure 2a) that hides the sensing architecture.
The pressure sensor adopted should be robust enough to resist possible damage when handled by children. To this aim, we have adopted a load cell (13 mm × 13 mm × 50 mm) obtained disassembling a consumer kitchen electronic scale (Zheben, SF-400) with a pressure range of 0–7 kg. The load cell has been encapsulated inside a LegoTM structure and the sensitive part of the cell has been covered by a flat LegoTM red tile (Figure 2c) to indicate clearly where the child has to press. The encapsulated load cell has then been positioned inside the Lego cottage in an area where the children can comfortably interact with.
The processing electronics is housed inside the cottage and connected to the load cell. The force signal is amplified by a differential operational amplifier with an adjustable gain (TI INA125P); a particular configuration has been adopted to maximize stability as shown in Appendix A. The amplifier is connected to the micro-controller of an Arduino board that samples the signal, filters it, and sends it wirelessly through a Bluetooth chip HC-06 transceiver [58] at a frequency of 250 Hertz. Bluetooth 2.0 + EDR (Enhanced Data Rate) has been adopted to limit current sink and therefore power consumption. To maximize attractiveness, children can personalize the cottage with LegoTM characters or other constructions, provided that these do not interfere with the sensing area.
In the configuration shown in Figure 2a,b, the child can activate the sensor by pressing with one finger. To accommodate pinch grasp, a rotating mechanism, made of a LegoTM shaft inserted into two parallel bricks, has been designed and realized (Figure 2c). In this way, the LegoTM case of the load cell is free to rotate and emerges from its housing; the child can thus squeeze the case with two opposite fingers as required by pinching. Different pinch amplitudes can be accommodated by simply making the load cell case thicker by adding flat bricks on the top of it (Figure 2d). Finally, for power grasp (palm grasp), the cell case can be detached from the cottage and used as a free-tracker (Figure 2e). In this way, the entire repertoire of basic prehensions of the human hand is covered.
Two ranges of force have been defined: from 0 to 3 kg and from 0 to 5 kg, according to age and rehabilitation state. The prototype is built on a 30 cm × 30 cm LegoTM base and the weight of the construction is about 0.3 kg, which makes it easily portable. The accuracy and reliability have been fully tested as reported in Appendix B. Linearity and repeatability are in the order of 0.1%, which is 3 g for the 3 kg range and 5 g for the 5 kg range.
Such a high resolution is not required when the child exercises at home where the capability of applying a force above a given threshold is sufficient for training purposes. Under this hypothesis, the tracker can be simplified and a simpler sensing mechanism, based on the same principle on which a clothespin works, has been devised.
The sensing object is constituted of a pet-toy realized through 3D printing (FABTotum Personal Fabricator printer). This has been designed to be attractive to children: a crocodile made of two parts: the upper and lower jaw pivoting around the end of the mouth (Figure 3). The child has to press the tail of the crocodile to make it open its mouth; such movement is resisted by a set of rubber bands applied to the mouth whose strength depends on their number and regulates the force that has to be exerted by the child. A smart button, constituted of a digital contact, a microcontroller, and a radio transmitter, is inserted through a frontal slot inside a lodge created in the bottom jaw of the crocodile, to make battery substitution easy. The button is used to sense when the user has exerted enough pressure to open the mouth. The smart button chosen here is a Camkix Bluetooth Remote Shutter; typically used as a remote shutter of digital cameras, it has the robustness required for use by children. Such a device transmits a digital pulse whenever the contact is open and is compatible with the blue-tooth HID (Human Interface Device) profile; it can be easily interfaced with smartphones and tablets operating system as it is recognized automatically, like regular keyboards or mice, and can therefore be used easily as an input device.

3.2.2. Motion Sensing Sub-System

For the mobility exercises, we leverage the multi-touch display of mobile terminals like smartphones or tablets. Such devices accurately detect the position of one or more fingers at 30 Hz and can therefore easily be used to measure fingers tapping, sliding, or pinching over the device surface (see videos in the Supplementary Materials). This tablet or smartphone will be the same that works as a host for the pressure-sensing device.

3.2.3. Platform Architecture

The sensing devices and the host constitute the platform that is given to the families to support autonomous rehabilitation of their child. It can be regarded as the client component of the typical client-server architecture used in the telerehabilitation domain [6]: the client runs the exergames, acquires the pressure data from the sensing devices, logs them and attaches a time stamp. At the end of the exercise, it transmits these data to a server, that, in turn, processes the data to compute the results and show them to the clinicians. Moreover, the server sends to the client the list of the exergames chosen by the clinicians, along with their difficulty degree (Figure 4).
The server is composed itself of two main components: the data storage and a graphical application that allows clinicians to analyze rehabilitation progression and to configure the rehabilitation sessions. Such a graphical application is implemented as a web-application to allow clinicians to access the data from anywhere, even outside the hospital.
When the application is started on the mobile terminal, the user sees the exercises grouped into a suite: the user can choose among the different exergames assigned by the therapist. The exercises will be already configured at the right difficulty and to be played either with the tablet or with the pressure sensor as a tracker (Figure 5).

3.3. Therapeutic Exergames

A set of exergames were developed, following a methodology specifically designed for therapeutic exergames [53], that consists of three main steps:
Identification of the exercises, all their requirements, constraints and all the parameters that determine their challenge level;
Transformation of the exercises into Virtual Exercises to discuss with clinicians game mechanics and actions;
Introducing all the contour elements that characterize a video-game (characters, graphical elements, color, music, sounds and rewards) as a layer on top of the exercises themselves to maximize entertainment.
The exergames were designed to implement different exercises such that the same game can be played to increase either force or mobility, thus creating a multi to multi mapping between exercises and exergames. We briefly describe here some of the developed exergames. As required, they cover the range of 2–7 years old.

3.3.1. Breaking Eggs

A closed egg is placed in the middle of the screen. The player has to scratch the egg surface to break it. As a reward, a pet animal, different from trial to trial, exits from the egg and appears on the screen; the child can caress the pet, receiving an additional reward from the game: cute little stars or hearths, and pleasant sounds (Figure 6a). This game supports both mobility and strength exercises. The child can break the egg by tapping, sliding, or pinching on the display surface (mobility exercises), or by pressing, pinching, or grasping the sensor (strength exercises). The simple rewarding mechanism of the game has been designed for children 2–3 years old.

3.3.2. Jigsaw

A nine-piece jigsaw is proposed to the child, who can solve it by dragging each piece in the right place with his/her hand. This game supports all the mobility exercises and can be played only through the multi-touch screen (Figure 6b). Different jigsaws are provided, and additional ones can be obtained from images provided by the child himself. It has been designed for children 3–7 years old.

3.3.3. Mouse & Cheese Maze

A mouse is trapped inside a maze and the child has to guide it with his/her hand all the way to a target where it finds a piece of cheese. The maze shape is created procedurally and will be different from trial to trial to increase variability. This game supports all the mobility exercises and can be played only through the multi-touch screen (Figure 6c). It has been designed for children 3–7 years old.

3.3.4. Hot Air Balloon (and Similar Themes)

This is a runner game. The player is in control of a hot air balloon that flies in the sky. The goal is to catch targets (e.g., coins) and avoid distractors (e.g., birds) by moving the balloon up and down by moving his/her fingers according to the specified exercise (Figure 6d). This game supports all the exercises using either the multi-touch screen or the pressure sensing device. Some variants of this exergame are available. In a marine theme, the graphical environment is the sea, the controlled element is a submarine, the targets are coins, and the distractors are fishes. In an outer space theme, the graphical elements represent the stars, the mobile element is an astronaut, and the distractors are aliens. The different themes allow showing to the child games that appear very different one from the other, while they share the same mechanics and guide the child through the same exercises. These games have been designed for children of 4–7 years old.

4. Discussion

To guarantee a good, fast, and complete recovery, rehabilitation after surgery should be continuous and intensive [59], and the system described here goes in this direction: it is meant to be used for a prolonged time and at home, autonomously. For this reason, a careful design phase is required. We have started from the clinical needs and developed the sensing part first. The touch screen of mobile devices has been identified as a suitable means to track the movements required to increase finger mobility. Pressure detection instead has required the design and development of a novel flexible pressure-sensing device.
We have started designing the first prototype to illustrate to clinicians the sensing mechanism [53]. This was constituted of the bare load cell attached to a rigid metal frame (Figure 7a) and to a simple clothespin with a button that detected when the clothespin was closed (Figure 7b). The load cell was considered adequate, as it showed enough robustness for children use; the clothespin mechanism was considered suited to elicit an adequate force from the children, but the grasping region on top was considered somehow slippery and a better shaping was required for grasping.
The second step was to provide a case that can be attractive to children. For this reason, we have embedded the load cell into a LegoTM cottage structure (Figure 7c). This has allowed on one side to hide the sensor inside the cottage, and on the other side to offer to the child an object with whom he/she can be familiar. Following a similar line of thought, the clothespin was transformed into a 3D pet animal. A crocodile was chosen as it has two desirable characteristics that are a good match for the sensor requirements: it has a large head, inside which the trigger sensor can be hidden, and it has a long tail that can be used to induce the grasping movements required by clinicians in an ecological way (Figure 7d). Additionally, the tail has been designed with scales that facilitate a firm grasp. Moreover, it has allowed us to greatly simplify the case, reducing cost and size.
The load-cell based sensing device has first been tested for accuracy, repeatability, and linearity (Appendix B) to assess measurement validity following a procedure similar to that proposed in [45]. The resolution and accuracy are on average an order of magnitude larger than that of dynamometers typically used in the clinics [40,41,42], that have an accuracy of 50 g, with a sensitivity of 10 g [40]. This is due to the fact that such dynamometers have been developed to assess grip force mainly in adults and elders, and have a range that is almost twenty times larger as it reaches 90 kg. The device itself is also of large dimension. Focus on the pediatric hand has allowed miniaturizing the device on one side and increasing its accuracy and resolution on the other, to values that are larger than the target set by clinicians.
A single non-rechargeable button cell (CR2032 3 V) is used in the crocodile sensor. This is dimensioned to operate the device for one month, half an hour a day before the cell discharges and has to be replaced. The opening in the lower jaw (Figure 7d) makes this operation easy. The LegoTM cottage accommodates a larger source of energy inside the case. We use here a pair of rechargeable stylus batteries (1.5 V AA) that are recharged typically at the end of the week. Indeed, recharging modalities that do not require extracting the batteries from their housing (USB, induction) would be preferable, but they require additional room for the charging components and will be part of a subsequent refinement stage.
The two sensing systems were then tested with stakeholders. Clinicians have validated the adequacy of the interaction modalities offered and of the measurements collected. Children, on their side, generally liked the objects; however, their feeling and that of their parents was that the sensing systems were somehow “cold” objects, not really interesting for children per sè.
Eliciting some emotional reaction is fundamental to increase the attractiveness and therefore intrinsic motivation to use the system. To this aim, we have added some emotional content to the objects [60]. We have added LegoTM characters and additional aesthetic LegoTM structures on top of the cottage (Figure 2). Moreover, we have also granted to children the possibility to further personalize the cottage. The crocodile’s surface was pained with a-toxic acrylic painting to provide a colorful and playful final appearance (Figure 3). The degree of pleasantness was largely increased in this last step and children were keen to use the crocodile repeatedly [61].
Although the overall system is quite simple, it contains one of the main requisites of telerehabilitation and telemonitoring systems in general: they can be adjusted to the patient’s current capabilities [6,7]. The combination of the exergames with the sensing devices enables a difficulty progression in the exercises along the three dimensions typically required by clinicians: range, speed, and accuracy. In all exergames, the game pace sets the speed, which is the frequency of spawning targets and distractors. The therapist can then program the range of motion required to consider a movement valid. For instance, for pinch movement, the child has to open the fingers by more than 40 mm, or the pinch force has to be larger than 1 kg. Only valid movements allow moving the avatar inside the game. Finally, the degree of accuracy can be increased by reducing the dimension of the targets to be hit by the game avatar. Similar reasoning applies to the other grasping modalities. Such flexibility in the platform is allowed by a combination of an adequate range and resolution of the sensors with the definition of specific and intuitive game parameters that regulate the exercise difficulty, according to the good practice of serious games design [12].
A single suite of exer-games is provided to guide both exercises on mobility and strength. This allows speeding up game learning and increases the attachment to the graphical characters that are used again and again. The exergames are designed to automatically collect data from the input devices (position or force), as well as game interaction outcomes (targets hit/miss, distractors hit/miss) along with their timestamp. Force data depend on the sensor used. They are values expressed in grams for the cottage sensor device and they are binary values (trigger values) for the crocodile toy; the latter notifies the game that the user applied enough force on the sensor. Position data are obtained by converting fingers position expressed in pixels into millimeters, considering the device-specific screen resolution (PPI: pixels per inch). These data are used by the game itself to evaluate user movement, interact with game elements, and provide feedback in real-time. When a game ends, the data acquired are sent to a remote server in the cloud for storage and analysis by the therapists who can access them through a web application. This modality allows them to review the rehabilitation progression of their patients in the most suitable period of the day. Moreover, this allows creating a loop that starts from the therapist, who assigns the exercises to the patients with a predefined level of difficulty, goes through the exercises that are carried out at home, and goes back to the clinicians with the results that can be used to tune the therapy. Such continuous telemonitoring capability is at the basis of any autonomous rehabilitation platform.
To make therapist evaluation most effective, views of the data with a different level of detail are provided. To get a quick and clear picture of patient progression, data are aggregated on a per-exercise basis according to the methodology described in [62] and average values on speed, accuracy, and range of motion are plotted over time. For instance, a typical plot of pinch progression is shown in Figure 8.
The clinicians can also analyze in-depth a single exercise of a single rehabilitation session through a detailed report like that shown in Figure 9.
This in-depth analysis of the time course of movement and force development along with game interaction outcome allows having a clear picture of how the exergame has been played. All these data, collected over the cohort of patients served by the same hospital, can be processed through adequate machine learning techniques to cluster patients, identify suitable models of rehabilitation progression for each cluster, and optimize the scheduling of hospital recall visits and the therapy program [6].
Game results, such as score, number of target, or distractors hit and missed are also displayed in this report; these values are not used by therapists to assess exercise performance quality, but to establish whether the game is at the right difficulty level. If the game becomes too simple or too difficult, the patient tends to be distressed by the game and quit [12,53]. This information is used to set a proper challenge level for the user with the goal of keeping him/her interested and engaged in the game itself (flow state).
Indeed, the score achieved in a game is dependent not only on the use of hand/fingers, but also on the gamer’s skill that theoretically can improve by continuing with playing. However, this possibility is very remote here as the ability in the game is strictly dependent on the force or motion skills acquired. Therefore, this information is used to set proper level of difficulty of the exer-game, by increasing range, force, or accuracy required to animate the avatar.
Overall, the platform presented here is a general approach in which sensors disappear inside objects and make the objects smart as they can measure some relevant aspect of everyday life in a real setting. Such smart objects are the basis of what can be classified a telehealth rehabilitation service in which the toys developed are integrated inside a cloud-based system that provides the analysis of rehabilitation data and relevant information to clinicians [62].
The platform can acquire a huge amount of data from a large population of patients. This enables machine learning algorithms to browse the data to extract meaningful information at the population level. One of the most promising directions is the identification of homogenous clusters of patients for whom a tailored rehabilitation program can be defined and the frequency of recall visits can be optimized. The same data can allow the comparison of different rehabilitation programs to identify strong and weak elements.
The large population could also be the basis for creating a virtual community of patients and their parents. This can, on one side, support gamification to boost child motivation in performing the exercises (e.g., virtual medals, points, leader boards, ranking, and so forth, cf. [63]), while on the other side it connects all the stakeholders and provides them a place where to exchange information, data, suggestions, and where they can find continuous support from peers.
Additional work can be spent in trying to identifying automatically when the child has made the right movement. In fact, children tend to use either their intact fingers or their intact hand to perform the movements required by the exergames; presently, their parents correct them to perform the right movement. This could be achieved, for instance, by analyzing the pressure or movement time course with adequate classification algorithms. Finally, although the hardware is relatively simple and cheap (at present less than 100 Euros), its cost should be further reduced to enable massive diffusion.

5. Conclusions

We show here how combining electronics with functional and emotional design, a sensor, which can be used for clinical purposes, can be realized such that is appears as an attractive toy to its young users. This allows prolonged natural use that is the goal of such smart objects and opens the way to increased contamination between mechanics, computer science, electronics, and design to produce objects of everyday use that can have transparent sensing capabilities. Such an approach also allows the combining of telerehabilitation and telemonitoring into a single platform, thus making the system particularly suitable for autonomous use at home.

Supplementary Materials

Videos showing the platform working with the tablet, with the cottage sensor and with the crocodile sensor can be viewed at https://gvm.aislab.di.unimi.it/index.php/s/k7iaXdw8IRSt6rT. The password is sensors2019.

Author Contributions

Conceptualization, N.A.B. and R.M.; methodology, E.M., G.P., N.A.B., R.M., J.E. and G.P.; firmware & software, R.M. and J.E.; validation, R.M., E.M., R.P., G.P.; data curation, R.M.; writing—original draft preparation, R.M.; writing—review and editing, N.A.B.; supervision, N.A.B.; project administration, N.A.B.; investigation, R.M., E.M., R.P., G.P.

Funding

This research received no external funding.

Acknowledgments

The authors wish to thank the anonymous reviewers for their comments and criticisms that have allowed to greatly improve the paper.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Amplification Design

Force is sensed through a load cell here. This is typically constituted of a block of steel or aluminum inside which a section less resistant to deformations is made, for instance by creating holes inside the block as it can be clearly seen in Figure 2d. Force applied induces a strain in the block that deforms mainly the less resistant section. At the top and bottom of this section a strain gauge, in the form of foil resistors, is applied to sense strain (Figure A1). To minimize resistance variations due to temperature, resistors are arranged as a full Wheatstone bridge [64]. This produces a differential voltage proportional to the overall strain [65].
The output voltage of the load cell is in the order of few millivolts, and amplification is required to produce a voltage inside the range that can be sampled reliably by a microcontroller. This is usually accomplished by an instrumentation amplifier [66] that combines high input impedance, high differential gain, high common-mode and power supply rejection ratio thanks to matched transistors, low drift and offset [67]. We have adopted here the widely used INA 125 P amplifier (Texas Instrument) [68] because of its low power consumption, and its high linearity over the full measurement range. Therefore a linear model can be used to describe the sensor/amplifier behavior, as classically adopted in sensors domain [69]:
V o u t   =   G x   +   H
where Vout is the output voltage amplified, proportional to the weight applied on the load cell, x is the load cell output voltage, G is the amplifier gain and H its offset. G could be set in the range of 4 to 10,000 through an external resistor, RG (Figure 8) [66,68]:
G   =   4   Ω + 60   k Ω R G
To determine G and H in Equation (A1) an iterative procedure has been adopted here inspired to [70]. When the differential amplifier input voltage is bounded to be only positive (Vin+ ≥ Vin−), like in this case, particular attention has to be paid to input common-mode range [67]. One possibility is to polarize Vin+ and Vin− such that minimum input voltage stays above 1 V. However, this might introduce noise on the measured voltage unless an extremely precise and stable reference, equal at both terminals, is applied. For this reason, we have preferred to bias IAREF to provide a minimum voltage to the output, Vmin, equal to 0.3 V, such that the amplifier’s transistors are on and operate in their linear region [70]. To achieve this we have introduced an external variable resistor RT trimmed as follows.
Sensors 19 05517 g0a1 550
Figure A1. The circuit arrangement of the instrumentation amplifier INA125: it is connected with the load cell full bridge on the left end, and to the microcontroller analogic input to the right end. The trimmed resistor, RG, determines the gain. It has been set here to 30 Ω for the 3 kg range. The trimmed resistor RT provides a polarization of the output to keep the amplifier inside its linear region.
Figure A1. The circuit arrangement of the instrumentation amplifier INA125: it is connected with the load cell full bridge on the left end, and to the microcontroller analogic input to the right end. The trimmed resistor, RG, determines the gain. It has been set here to 30 Ω for the 3 kg range. The trimmed resistor RT provides a polarization of the output to keep the amplifier inside its linear region.
Sensors 19 05517 g0a1
We first set RG (500 Ω trimmed resistor) to its maximum value to obtain the minimum possible amplification. Then we regulate nominal value suggested by Equation 1 and set RT such that Va ≈ 0,5 V and Vout ≈ Vmin when no load is applied on the load cell. We then adjust RG (decreasing its value) to obtain a Vout equal to the full measuring range of the Analog to Digital Converter (ADC) of the microcontroller at full load, Vref equal to 2.5 V in our case. To this aim, the load cell is loaded with the desired maximum weight (FMax), and RG is adjusted until Vout reaches Vref. We then measure again Vout with the unloaded cell, and it will generally differ from Vmin measured before because of the larger IAREF introduced by RG. We tune RT again to bring Vout back to Vmin and RG such that the output reaches Vref at FMax. This procedure converges in a few iterations. The final value of Vout, with the sensor unloaded, represents the H parameter in Equation 1. Such procedure has been carried out with the Vout pin connected to the microcontroller ADC pin to provide the true load impendence, although, due to the very high impedance of the ADC input, no appreciable change on RT is seen with or without the connection.

Appendix B. Characterization of Force Measurements

We have tested experimentally the linearity of the output signal by measuring the output voltage of the amplifier against a set of known weights placed on the sensor. The measurement range upper limit, FMax, was enlarged of about 50 g above maximum measurement range to avoid ceiling effects (FMax = 3050 kg and FMax = 5050 kg respectively). The ADC resolution is:
FMax/(210 − H)
where H represents the offset in Equation (A1) and it was determined experimentally for the two ranges level as H = 102 (for the 3 kg range) and H = 113.32 (for the 5 kg range). From this, it follows that the resolution is of 3.31 g for the 3 kg range and of 5.55 g for the 5 kg range. These values are well above the minimum resolution required for rehabilitation.
To convert the ADC output into pressure values we need to determine offset and gain of the ADC [68]. To this aim, we have measured a set of known weights distributed over the whole measurement range with one weight being just below the upper end of the range to avoid saturation (Figure A2). We have taken ten measurements for each weight and we report their average in Figure A2. The least-squares linear regression is computed and plotted. As expected, the line slope is larger for the widest range (5 kg). The plots show a high degree of linearity over the whole range with an RMS error of 3.25 g and 1.80 g, for 3 kg and 5 kg range, respectively (Table A1 and Table A2). These values are below the ADC resolution. For sake of clarity, we have excluded from the regression the last value of the two tables because of possible saturation, in fact, the maximum weight (3 kg and 5 kg respectively) was slightly underestimated, that might be attributed to saturation of the differential amplifier. However, being this error well below the resolution required by the therapists, we have not further investigated this issue.
Sensors 19 05517 g0a2 550
Figure A2. Linearity test: Objects with known weights (inside the 3 kg range, left; inside the 5 kg range, right) were placed statically over the sensor. The read-out of the sensor was collected through the micro-controller and sent to the host for the analysis. Each value corresponds to measurement averaged over 10 samples. The linear regression obtained (R2 = 0.99998231 for 3 kg and R2 = 0.99999035 for 5 kg) that represent a highly linear response.
Figure A2. Linearity test: Objects with known weights (inside the 3 kg range, left; inside the 5 kg range, right) were placed statically over the sensor. The read-out of the sensor was collected through the micro-controller and sent to the host for the analysis. Each value corresponds to measurement averaged over 10 samples. The linear regression obtained (R2 = 0.99998231 for 3 kg and R2 = 0.99999035 for 5 kg) that represent a highly linear response.
Sensors 19 05517 g0a2
The absence of hysteresis has been assessed through test-retest [71]. To this aim, we have measured the same set of known weights used before going from the smallest to the largest and then from the largest to the smallest for both the 3 kg and 5 kg ranges. Each time, the current weight was removed from above the load cell, and the next weight was placed over the cell and its weight acquired. No parameter was tuned in between each pair of samplings. The average absolute difference between the two measurements was 3.16 g and the RMSE of the second acquisition (3.11 g) was similar to the RMSE of the first acquisition (3.25 g). All these values are below the resolution of the measurement of 3.31 g and 5.55 g respectively (Table A3).
Table
Table A1. Measurement of known weights in the range 0–3 kg.
Table A1. Measurement of known weights in the range 0–3 kg.
Average 10 Bit SampledReal Weight [g]Computed Weight [g]Difference [g]
103.940−0.65−0.65
257.91500505.11+5.11
407.091000995.10−4.9
560.0515001497.54−2.46
714.1120012003.58+2.58
865.3425002500.33+0.33
1014.2330002989.38−10.62
Table
Table A2. Measurement of known weights in the range 0–5 kg.
Table A2. Measurement of known weights in the range 0–5 kg.
Average 10 Bit SampledReal Weight [g]Computed Weight [g]Difference [g]
113.3201.33+1.33
257.71800798.67−1.33
402.8816001600.29+0.29
547.2424002397.44−2.56
693.1132003202.98+2.98
837.3240003999.30−0.70
1015.350004982.11−17.89
Table
Table A3. Test-retest of the sensor. The same weights of Table A1 have been adopted. Maximum weight was 3000 g. The trimmed difference is evaluated between the first and second measurements.
Table A3. Test-retest of the sensor. The same weights of Table A1 have been adopted. Maximum weight was 3000 g. The trimmed difference is evaluated between the first and second measurements.
Real WeightFirst Measurement [g]Second Measurement [g]Absolute Difference [g]
0−0.65−5.755.09
500505.11503.251.85
1000995.101002.697.59
15001497.541501.724.19
20012003.582002.261.32
25002500.332501.611.28
30002989.382990.190.81

References

  1. Lockman, J.J.; McHale, J.P. Object Manipulation in Infancy BT—Action in Social Context: Perspectives on Early Development; Lockman, J.J., Hazen, N.L., Eds.; Springer: Boston, MA, USA, 1989; pp. 129–167. [Google Scholar]
  2. Little, K.J.; Cornwall, R. Congenital Anomalies of the Hand—Principles of Management. Orthop. Clin. 2016, 47, 153–168. [Google Scholar] [CrossRef] [PubMed]
  3. Henry, S.; Wei, F.C. Thumb Reconstruction with Toe Transfer. J. Hand Microsurg. 2010, 2, 72–78. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Langhorne, P.; Duncan, P. Does the organization of postacute stroke care really matter? Stroke 2001, 32, 268–274. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Clark, R.A.; Pua, Y.H.; Fortin, K.; Ritchie, C.; Webster, K.E.; Denehy, L.; Bryant, A.L. Validity of the Microsoft Kinect for assessment of postural control. Gait Posture 2012, 36, 372–377. [Google Scholar] [CrossRef] [PubMed]
  6. Borghese, N.A.; Murray, D.; Paraschiv-Ionescu, A.; de Bruin, E.D.; Bulgheroni, M.; Steblin, A.; Luft, A.; Parra, C. Rehabilitation at Home: A Comprehensive Technological Approach. In Virtual, Augmented Reality and Serious Games for Healthcare 1; Ma, M., Jain, L.C., Anderson, P., Eds.; Springer: Berlin/Heidelberg, Germany, 2014; pp. 289–319. [Google Scholar]
  7. Peretti, A.; Amenta, F.; Tayebati, S.K.; Nittari, G.; Mahdi, S.S. Telerehabilitation: Review of the State-of-the-Art and Areas of Application. JMIR Rehabil. Assist. Technol. 2017, 4, e7. [Google Scholar] [CrossRef] [PubMed]
  8. Gensini, G.F.; Alderighi, C.; Rasoini, R.; Mazzanti, M.; Casolo, G. Value of Telemonitoring and Telemedicine in Heart Failure Management. Card. Fail. Rev. 2017, 3, 116–121. [Google Scholar] [CrossRef] [Green Version]
  9. SilverFit. Available online: https://silverfit.com/en/ (accessed on 4 November 2019).
  10. Jintronix. Available online: http://www.jintronix.com/ (accessed on 4 November 2019).
  11. The Rehabilitation Gaming System. 2016. Available online: http://www.aal-europe.eu/projects/rgs/ (accessed on 25 October 2019).
  12. Pirovano, M.; Mainetti, R.; Baud-Bovy, G.; Lanzi, P.L.; Borghese, N.A. Intelligent Game Engine for Rehabilitation (IGER). IEEE Trans. Comput. Intell. AI Games 2016, 8, 43–55. [Google Scholar] [CrossRef]
  13. Trombetta, M.; Bazzanello Henrique, P.P.; Brum, M.R.; Colussi, E.L.; De Marchi, A.C.B.; Rieder, R. Motion Rehab AVE 3D: A VR-based exergame for post-stroke rehabilitation. Comput. Methods Programs Biomed. 2017, 151, 15–20. [Google Scholar] [CrossRef]
  14. Anton, D.; Goni, A.; Illarramendi, A.; Torres-Unda, J.J.; Seco, J. KiReS: A Kinect-Based Telerehabilitation System. In Proceedings of the 2013 IEEE 15th International Conference on e-Health Networking, Applications and Services, Healthcom, Lisbon Portugal, 9–12 October 2013; pp. 444–448. [Google Scholar]
  15. Rewire Project. 2016. Available online: http://www.rewire-project.eu (accessed on 25 October 2019).
  16. Nestore-Coach. Available online: https://nestore-coach.eu/home (accessed on 4 November 2019).
  17. Lozano-Quilis, J.A.; Gil-Gómez, H.; Gil-Gómez, J.A.; Albiol-Pérez, S.; Palacios-Navarro, G.; Fardoun, H.M.; Mashat, A.S. Virtual Rehabilitation for Multiple Sclerosis Using a Kinect-Based System: Randomized Controlled Trial. JMIR Serious Games 2014, 2, e12. [Google Scholar] [CrossRef]
  18. Ates, S.; Haarman, C.J.W.; Stienen, A.H.A. SCRIPT passive orthosis: Design of interactive hand and wrist exoskeleton for rehabilitation at home after stroke. Auton. Robots 2017, 41, 711–723. [Google Scholar] [CrossRef] [Green Version]
  19. Borboni, A.; Mor, M.; Faglia, R. Gloreha—Hand Robotic Rehabilitation: Design, Mechanical Model, and Experiments. J. Dyn. Syst. Meas. Control 2016, 138, 111003–111012. [Google Scholar] [CrossRef]
  20. Wang, B.; Mcdaid, A.; Biglari-Abhari, M.; Aw, K.C. Design and Development of a Glove for Post-Stroke Hand Rehabilitation. In Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics, AIM, Munich, Germany, 26–30 June 2017. [Google Scholar]
  21. Ma, Z.; Ben-Tzvi, P.; Danoff, J. Hand Rehabilitation Learning System with an Exoskeleton Robotic Glove. IEEE Trans. Neural Syst. Rehabil. Eng. 2016, 24, 1323–1332. [Google Scholar] [CrossRef] [PubMed]
  22. Guo, Y.; Xu, F.; Song, Y.; Cao, X.; Meng, F. A Soft Robotic Glove for Hand Rehabilitation Using Pneumatic Actuators with Variable Stiffness. In Proceedings of the International Conference on Intelligent Robotics and Applications, Shenyang, China; 2019; pp. 119–129. [Google Scholar]
  23. Park, W.; Jeong, W.; Kwon, G.H.; Kim, Y.H.; Kim, L. A Rehabilitation Device to Improve the Hand Grasp Function of Stroke Patients using a Patient-Driven Approach. In Proceedings of the IEEE International Conference on Rehabilitation Robotic, Seattle, WA, USA, 24–26 June 2013. [Google Scholar]
  24. Klug, F.; Hessinger, M.; Koka, T.; Witulla, P.; Will, C.; Schlichting, T.; Endl, C.; Albenstetter, A.; Champagne, P.O.; Gagnon, D.H.; et al. An Anthropomorphic Soft Exosuit for Hand Rehabilitation. IEEE Int. Conf. Rehabil. Robot. 2019, 2019, 1121–1126. [Google Scholar] [PubMed]
  25. Rudd, G.; Daly, L.; Jovanovic, V.; Cuckov, F. A Low-Cost Soft Robotic Hand Exoskeleton for Use in Therapy of Limited Hand–Motor Function. Appl. Sci. 2019, 9, 3751. [Google Scholar] [CrossRef] [Green Version]
  26. Cafolla, D. A personalized flexible exoskeleton for finger rehabilitation: a conceptual design. In IFToMM World Congress on Mechanism and Machine Science, Krakow, Poland; Springer: Cham, Germany, 2019; pp. 73–82. [Google Scholar]
  27. Chang, W.H.; Kim, Y.H. Robot-assisted Therapy in Stroke Rehabilitation. J. Stroke 2013, 15, 174–181. [Google Scholar] [CrossRef]
  28. Alexanderson, S.; O’Sullivan, C.; Beskow, J. Real-time labeling of non-rigid motion capture marker sets. Comput. Graph. 2017, 69, 59–67. [Google Scholar] [CrossRef] [Green Version]
  29. Alimanova, M.; Borambayeva, S.; Kozhamzharova, D.; Kurmangaiyeva, N.; Ospanova, D.; Tyulepberdinova, G.; Gaziz, G.; Kassenkhan, A. Gamification of Hand Rehabilitation Process using Virtual Reality Tools: Using Leap Motion for Hand Rehabilitation. In Proceedings of the 2017 1st IEEE International Conference on Robotic Computing, IRC 2017, Laguna Hills, CA, USA, 10–12 April 2017. [Google Scholar]
  30. Taylor, J.; Curran, K. Using leap motion and gamification to facilitate and encourage rehabilitation for hand injuries: leap motion for rehabilitation. In Handbook of Research on Holistic Perspectives in Gamification for Clinical Practice; IGI Global: London, UK, 2016; pp. 183–192. [Google Scholar]
  31. Proffitt, R.; Sevick, M.; Chang, C.Y.; Lange, B. User-Centered Design of a Controller-Free Game for Hand Rehabilitation. Games Health J. 2015, 4, 259–264. [Google Scholar] [CrossRef] [Green Version]
  32. Valentini, P.P.; Pezzuti, E. Accuracy in fingertip tracking using Leap Motion Controller for interactive virtual applications. Int. J. Interact. Des. Manuf. 2017, 11, 641–650. [Google Scholar] [CrossRef]
  33. Guzsvinecz, T.; Szucs, V.; Sik-Lanyi, C. Suitability of the kinect sensor and leap motion controller-A literature review. Sensors 2019, 19, 1072. [Google Scholar] [CrossRef] [Green Version]
  34. Rand, D.; Schejter-Margalit, T.; Dudkiewicz, I.; Kizony, R.; Zeilig, G.; Kizony, R. The Use of the iPad for Poststroke Hand Rehabilitation; A Pilot Study. In Proceedings of the 2013 International Conference on Virtual Rehabilitation, ICVR 2013, Philadelphia, PA, USA, 26–29 August 2013. [Google Scholar]
  35. Levac, D.; Dumas, H.M.; Meleis, W. A Tablet-Based Interactive Movement Tool for Pediatric Rehabilitation: Development and Preliminary Usability Evaluation. JMIR Rehabil. Assist. Technol. 2018, 5, e10307. [Google Scholar] [CrossRef]
  36. ReHand. Available online: https://rehand.net/ (accessed on 25 October 2019).
  37. Core Hand. Available online: https://www.digitalaffinity.co.uk/mobile/corehand/iphone-ios-app-core-hand.php (accessed on 25 October 2019).
  38. Dexteria—Fine Motor Skill Development. Available online: https://apps.apple.com/us/app/dexteria-fine-motor-skill/id420464455?ign-mpt=uo%3D4 (accessed on 25 October 2019).
  39. MoTrack TherapyTM—The Home Hand Rehabilitation SolutionTM. Available online: https://www.motracktherapy.com/ (accessed on 25 October 2019).
  40. Hogrel, J.Y. Grip strength measured by high precision dynamometry in healthy subjects from 5 to 80 years. BMC Musculoskelet. Disord. 2015, 16, 139. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Takei Digital Dynamometer. Available online: https://www.pandamed.co.uk/product/takei-4501-digital-dynamometer/ (accessed on 4 November 2019).
  42. Richards, L.; Palmiter-Thomas, P. A Critical Review of Tools, Methods, and Clinical Utility for Grip Strength Measurement. Critic. Rev. Phys. Rehabilit. Med. 2017, 29, 1–4. [Google Scholar] [CrossRef]
  43. Neumann, S.; Kwisda, S.; Krettek, C.; Gaulke, R. Comparison of the grip strength using the martin-vigorimeter and the JAMAR-dynamometer: Establishment of normal values. In Vivo 2017, 31, 917–924. [Google Scholar] [PubMed] [Green Version]
  44. Jaber, R.; Hewson, D.J.; Duchêne, J. Design and validation of the Grip-ball for measurement of hand grip strength. Med. Eng. Phys. 2012, 34, 1356–1361. [Google Scholar] [CrossRef] [PubMed]
  45. Hewson, D.J.; Li, K.; Frèrejean, A.; Hogrel, J.Y.; Duchêne, J. Domo-Grip: Functional Evaluation and Rehabilitation using Grip Force. In Proceedings of the 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC’10, Buenos Aires, Argentina, 31 August–4 September 2010; pp. 1308–1311. [Google Scholar]
  46. Pintaric, T.; Kment, T.; Spreicer, W. SqueezeOrb: A Low-Cost Pressure-Sensitive User Input Device. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST, Bordeaux, France, 27–29 October 2008. [Google Scholar]
  47. Lookee® Smart Ball—Hand Exerciser Grip Strengthener & Trainer. Available online: https://www.lookeetech.com/products/lookee-smart-ball-hand-exerciser-grip-strengthener-trainer-with-app-battle-games (accessed on 25 October 2019).
  48. Gripable. Available online: https://www.gripable.co/how-it-works (accessed on 25 October 2019).
  49. MacKenzie, C.L.; Iberall, T. The Grasping Hand; Elsevier: Amsterdam, The Netherlands, 1994; Volume 104. [Google Scholar]
  50. Jean-Baptiste, E.M.D.; Nabiei, R.; Parekh, M.; Fringi, E.; Drozdowska, B.; Baber, C.; Jancovic, P.; Rotshein, P.; Russell, M. Intelligent Assistive System Using Real-Time Action Recognition for Stroke Survivors. In Proceedings of the 2014 IEEE International Conference on Healthcare Informatics, Verona, Italy, 15–17 September 2014; pp. 39–44. [Google Scholar]
  51. Sen.se. Available online: https://www.myrobotcenter.eu/en/sense (accessed on 4 November 2019).
  52. Serio, S.M.; Cecchi, F.; Assaf, T.; Laschi, C.; Dario, P. Design and Development of a Sensorized Wireless Toy for Measuring Infants’ Manual Actions. IEEE Trans. Neural Syst. Rehabil. Eng. 2013, 21, 444–453. [Google Scholar] [CrossRef] [PubMed]
  53. Pirovano, M.; Surer, E.; Mainetti, R.; Lanzi, P.L.; Alberto Borghese, N. Exergaming and rehabilitation: A methodology for the design of effective and safe therapeutic exergames. Entertain. Comput. 2016, 14, 55–65. [Google Scholar] [CrossRef]
  54. Eckman, M.; Gorski, I.; Mehta, K. Leveraging design thinking to build sustainable mobile health systems. J. Med. Eng. Technol. 2016, 40, 422–430. [Google Scholar] [CrossRef]
  55. Borghese, N.A.; Bottini, G.; Sedda, A. Videogame Based Neglect Rehabilitation: A Role for Spatial Remapping and Multisensory Integration? Front. Hum. Neurosci. 2013, 7, 1–3. [Google Scholar] [CrossRef] [Green Version]
  56. Elnaggar, A.; Reichardt, D. Digitizing the hand rehabilitation using serious games methodology with user-centered design approach. In Proceedings of the 2016 International Conference on Computational Science and Computational Intelligence (CSCI) Las Vegas, Nevada, WA, USA, 5–7 December 2016; pp. 13–22. [Google Scholar]
  57. Sullivan, S. Physical Rehabilitation; F.A. Davis Company: Philadelphia, PA, USA, 2019. [Google Scholar]
  58. HC-06 Datasheet. Available online: https://www.olimex.com/Products/Components/RF/BLUETOOTH-SERIAL-HC-06/resources/hc06.pdf (accessed on 4 November 2019).
  59. Langhorne, P.; Coupar, F.; Pollock, A. Motor recovery after stroke: A systematic review. Lancet Neurol. 2009, 8, 745–754. [Google Scholar] [CrossRef]
  60. Norman, D.A. Emotional Design: Why We Love (or Hate) Everyday Things; Basic Civitas Books, 2004. [Google Scholar]
  61. Borghese, N.A.; Mainetti, R.; Essenziale, J.; Cavalli, E.; Mancon, E.M.; Pajardi, G. Hand Rehabilitation with Toys with Embedded Sensors. In Converging Clinical and Engineering Research on Neurorehabilitation II.; Springer: Berlin/Heidelberg, Germany, 2016; Volume 15, pp. 425–430. [Google Scholar]
  62. Borghese, N.A.; Pezzera, M.; Mainetti, R.; Essenziale, J.; Cazzaniga, R.; Reggiori, B.; Mercurio, S.; Confalonieri, P. A Cloud-Based Platform for Effective Supervision of Autonomous Home Rehabilitation through Exer-Games. In Proceedings of the 2018 IEEE 6th International Conference on Serious Games and Applications for Health, SeGAH 2018, Wien, Austria, 16–18 May 2018; pp. 1–6. [Google Scholar]
  63. Pezzera, M.; Tironi, A.; Essenziale, J.; Mainetti, R.; Alberto, B.N. Approaches for increasing patient’s engagement and motivation in exer-games-based autonomous telerehabilitation. In Proceedings of the 2019 IEEE 7th International Conference on Serious Games and Applications for Health (SeGAH), Kyoto, Japan, 5–7 August 2019. [Google Scholar]
  64. Maundy, B.; Gift, S.J.G. Strain gauge amplifier circuits. IEEE Trans. Instrum. Meas. 2013, 62, 693–700. [Google Scholar] [CrossRef]
  65. Ştefănescu, D.M. Wheatstone Bridge—The Basic Circuit for Strain Gauge Force Transducers. In Handbook of Force Transducers: Principles and Components; Springer: Berlin/Heidelberg, Germany, 2011; pp. 347–360. [Google Scholar]
  66. Muller, I.; De Brito, R.; Pereira, C.E.; Brusamarello, V. Load cells in force sensing analysis—Theory and a novel application. IEEE Instrum. Meas. Mag. 2010, 13, 15–19. [Google Scholar] [CrossRef] [Green Version]
  67. Horowitz, P.; Hill, W. The Art of Electronics; Cambridge University Press: Cambridge, UK, 1989. [Google Scholar]
  68. Texas Instruments, INA 125 Datasheet. Available online: http://www.ti.com/lit/ds/symlink/ina125.pdf (accessed on 4 November 2019).
  69. Frosio, I.; Stuani, S.; Borghese, N.A. Autocalibration of MEMS Accelerometer. In Proceedings of the Conference Record-IEEE Instrumentation and Measurement Technology Conference, Sorrento, Italy; 2006. [Google Scholar]
  70. Adafruit Learning System. 2016. Available online: https://learn.adafruit.com/smart-cocktail-shaker/ (accessed on 4 November 2019).
  71. Trochim, W.M.K. The Research Methods Knowledge Base, 2nd ed.; Atomic Dog: Cincinnati, OH, USA, 2006. [Google Scholar]
Sensors 19 05517 g001 550
Figure 1. The modular sensor architecture, embedded inside a toy, is represented here. (a) Force is measured at the prescribed contact point of the object through a load cell. (b) The signal is amplified, (c) sampled and filtered by a micro-controller and (d) transmitted to a remote host, where (e) it is used to animate an avatar inside a graphical game environment (cf. Figure 2a).
Figure 1. The modular sensor architecture, embedded inside a toy, is represented here. (a) Force is measured at the prescribed contact point of the object through a load cell. (b) The signal is amplified, (c) sampled and filtered by a micro-controller and (d) transmitted to a remote host, where (e) it is used to animate an avatar inside a graphical game environment (cf. Figure 2a).
Sensors 19 05517 g001
Sensors 19 05517 g002 550
Figure 2. (a) The toy developed to measure force is shown here. A personalized LegoTM construction hosts the sensor and the associated hardware. Notice that all the hardware components are not visible. The output of the sensing device is wirelessly connected to the tablet on which the game is run. (b) A zoom of the housing of the load cell, covered with LegoTM bricks. (c) A LegoTM shaft allows raising the cell such that pinch movements are also allowed. The flat tile covering the pressure point is highlighted in green. (d) The same structure can be used to accommodate different fingers apertures (depending on age, rehabilitation stage etc.). (e) The sensor and its LegoTM case can be detached from the housing and used as a free-sensor for more complex grasping movements.
Figure 2. (a) The toy developed to measure force is shown here. A personalized LegoTM construction hosts the sensor and the associated hardware. Notice that all the hardware components are not visible. The output of the sensing device is wirelessly connected to the tablet on which the game is run. (b) A zoom of the housing of the load cell, covered with LegoTM bricks. (c) A LegoTM shaft allows raising the cell such that pinch movements are also allowed. The flat tile covering the pressure point is highlighted in green. (d) The same structure can be used to accommodate different fingers apertures (depending on age, rehabilitation stage etc.). (e) The sensor and its LegoTM case can be detached from the housing and used as a free-sensor for more complex grasping movements.
Sensors 19 05517 g002
Sensors 19 05517 g003 550
Figure 3. A remote shutter (a) is inserted inside the lower jaw of the crocodile (b) Force has to be exerted on the tail to open the mouth (c) When the mouth is closed a tooth in the upper jaw presses against the button trigger (grey area, red arrow, d), while when the mouth is open the tooth is raised and the button is releasedThe amount of force is regulated through a rubber band. The crocodile’s size is: 100 mm × 45 mm × 35 mm and weighs about 50 g.
Figure 3. A remote shutter (a) is inserted inside the lower jaw of the crocodile (b) Force has to be exerted on the tail to open the mouth (c) When the mouth is closed a tooth in the upper jaw presses against the button trigger (grey area, red arrow, d), while when the mouth is open the tooth is raised and the button is releasedThe amount of force is regulated through a rubber band. The crocodile’s size is: 100 mm × 45 mm × 35 mm and weighs about 50 g.
Sensors 19 05517 g003
Sensors 19 05517 g004 550
Figure 4. The whole system architecture is shown. It consists of the following components: a mobile terminal as a host on which the exergames are run. The host is connected wireless with the smart toy used as tracker and transmits to the server the data acquired during the interaction. Such data can then be accessed through a web application form any browser by the therapist.
Figure 4. The whole system architecture is shown. It consists of the following components: a mobile terminal as a host on which the exergames are run. The host is connected wireless with the smart toy used as tracker and transmits to the server the data acquired during the interaction. Such data can then be accessed through a web application form any browser by the therapist.
Sensors 19 05517 g004
Sensors 19 05517 g005 550
Figure 5. The exergames prescribed by the clinician are shown. They have already been configured at the correct difficulty level.
Figure 5. The exergames prescribed by the clinician are shown. They have already been configured at the correct difficulty level.
Sensors 19 05517 g005
Sensors 19 05517 g006 550
Figure 6. Different exergames developed for hand rehabilitation: (a) break-egg, (b) puzzle, (c) maze (d) hot air balloon. The mapping between games and exercises is multi to multi. For example, the pinch exercise can be guided by brake-egg games or hot air balloon; hot air balloon, on its side, can support all mobility and strength exercises.
Figure 6. Different exergames developed for hand rehabilitation: (a) break-egg, (b) puzzle, (c) maze (d) hot air balloon. The mapping between games and exercises is multi to multi. For example, the pinch exercise can be guided by brake-egg games or hot air balloon; hot air balloon, on its side, can support all mobility and strength exercises.
Sensors 19 05517 g006
Sensors 19 05517 g007 550
Figure 7. (a) The first prototype of the load cell-based sensing device. No case is provided to the bare sensor. (b) The first prototype of the clothespin based sensing device with an embedded sensor that detects when the clothespin is closed. Such prototypes are enhanced in the lower row. (c) The second prototype of the load-cell based toy capable of detecting the force. All the electronics is enclosed within the two LegoTM cases, which are kept very basic. (d) The clothespin has been turned into a crocodile and the sensor and the electronics are accommodated inside the crocodile lower jaw and the LegoTM case eliminated. No painting of the crocodile is provided.
Figure 7. (a) The first prototype of the load cell-based sensing device. No case is provided to the bare sensor. (b) The first prototype of the clothespin based sensing device with an embedded sensor that detects when the clothespin is closed. Such prototypes are enhanced in the lower row. (c) The second prototype of the load-cell based toy capable of detecting the force. All the electronics is enclosed within the two LegoTM cases, which are kept very basic. (d) The clothespin has been turned into a crocodile and the sensor and the electronics are accommodated inside the crocodile lower jaw and the LegoTM case eliminated. No painting of the crocodile is provided.
Sensors 19 05517 g007
Sensors 19 05517 g008 550
Figure 8. Pinch report on all pinching exercises performed in a 10 weeks period. The amplitude of pinching increases largely in the first month of training and then increases even more as shown by the green line. The same is true for pinch speed. Notice that a clearer picture can be obtained from the trend over time of the performance. Dots represent rehabilitation sessions carried out with the exergames.
Figure 8. Pinch report on all pinching exercises performed in a 10 weeks period. The amplitude of pinching increases largely in the first month of training and then increases even more as shown by the green line. The same is true for pinch speed. Notice that a clearer picture can be obtained from the trend over time of the performance. Dots represent rehabilitation sessions carried out with the exergames.
Sensors 19 05517 g008
Sensors 19 05517 g009 550
Figure 9. Detailed results of one pinching exercise carried out with the LegoTM cottage. Here the force threshold to have a valid pinch was set to 0.5 kg (blue horizontal line). Numerical results are reported in the left panel. About two-thirds of the pinch movements were over-threshold. Each pinch is shown on the bottom. The therapist could review the rehabilitation session by clicking on the play button (bottom-left corner). During the animation, the diameter of the green dot on the upper right panel represents the force exerted during the current time instant. The yellow colored box in the bottom chart represents the elapsed frames from the beginning of the exercise to the current time instant.
Figure 9. Detailed results of one pinching exercise carried out with the LegoTM cottage. Here the force threshold to have a valid pinch was set to 0.5 kg (blue horizontal line). Numerical results are reported in the left panel. About two-thirds of the pinch movements were over-threshold. Each pinch is shown on the bottom. The therapist could review the rehabilitation session by clicking on the play button (bottom-left corner). During the animation, the diameter of the green dot on the upper right panel represents the force exerted during the current time instant. The yellow colored box in the bottom chart represents the elapsed frames from the beginning of the exercise to the current time instant.
Sensors 19 05517 g009

Share and Cite

MDPI and ACS Style

Borghese, N.A.; Essenziale, J.; Mainetti, R.; Mancon, E.; Pagliaro, R.; Pajardi, G. Hand Rehabilitation and Telemonitoring through Smart Toys. Sensors 2019, 19, 5517. https://doi.org/10.3390/s19245517

AMA Style

Borghese NA, Essenziale J, Mainetti R, Mancon E, Pagliaro R, Pajardi G. Hand Rehabilitation and Telemonitoring through Smart Toys. Sensors. 2019; 19(24):5517. https://doi.org/10.3390/s19245517

Chicago/Turabian Style

Borghese, N. Alberto, Jacopo Essenziale, Renato Mainetti, Elena Mancon, Rossella Pagliaro, and Giorgio Pajardi. 2019. "Hand Rehabilitation and Telemonitoring through Smart Toys" Sensors 19, no. 24: 5517. https://doi.org/10.3390/s19245517

APA Style

Borghese, N. A., Essenziale, J., Mainetti, R., Mancon, E., Pagliaro, R., & Pajardi, G. (2019). Hand Rehabilitation and Telemonitoring through Smart Toys. Sensors, 19(24), 5517. https://doi.org/10.3390/s19245517

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop