Prototype Results of an Internet of Things System Using Wearables and Artificial Intelligence for the Detection of Frailty in Elderly People

Abstract

As society moves towards a preventative approach to healthcare, there is growing interest in scientific research involving technology that can monitor and prevent adverse health outcomes. The primary objective of this paper is to develop an Internet of Things (IoT) wearable system based on Fried’s phenotype that is capable of detecting frailty. To determine user requirements, the system’s architecture was designed based on the findings of a questionnaire administered to individuals confirmed to be frail. A functional prototype was successfully developed and tested under real-world conditions. This paper introduces the methodology that was used to analyze the data collected from the prototype. It proposes an interdisciplinary approach to interpret wearable sensor data, providing a comprehensive overview through both visual representations and computational analyses facilitated by machine learning models. The findings of these analyses offer insights into the ways in which different types of activities can be classified and quantified as part of an overall physical activity level, which is recognized as an important indicator of frailty. The results provide the foundations for a new generation of affordable and non-intrusive systems able to detect and assess early signs of frailty.

1. Introduction

The evolution of technology has generated a vast amount of easily accessible personal health data, and this has raised a new concern: how can the current heath data of individuals be monitored to prevent future diseases or complications? Health data are monitored for a variety of reasons, including diabetes assistance and insulin delivery [1], monitoring for atrial fibrillation [2] in patients who have suffered a stroke [3], and support for the diagnosis and treatment of Parkinson’s disease [4]. Additionally, IoT systems have been proposed for real-time health monitoring in elderly adults; these systems use wearable devices and mobile applications, and they could potentially improve healthcare in geriatric residences [5]. The usability of such Internet of Medical Things (IoMT) systems has been evaluated using heuristics, and this has led to significant improvements [6]. In their study, Sim et al. [7] observed an evolution in mobile heath technology usage from monitoring to diagnostic and therapeutic tools. In the same study, it was stated that one of the major challenges in the field of mobile health technology is the discovery and validation of meaningful digital biomarkers, and this is the main aim of the current article.

The availability of new technologies, such as embedded sensors in different accessories, and the capability of artificial intelligence to perform complex tasks should be used to improve health outcomes, especially in geriatric populations. One of the most concerning subjects in geriatric medicine is frailty. As Fried et al. [8] have stated, it can be considered synonymous with disability and comorbidity, and it is highly prevalent in old age, conferring a high risk of disability, hospitalization, and mortality. Frailty is classified as a syndrome which can be prevented or treated if is detected in its early stages [9], but it is also an important component in decision making and the effective prevention and treatment of perioperative complications in elderly patients [10].

Fried et al. [6] proposed a phenotype composed of five criteria for determining frailty: low physical activity, weight loss, exhaustion, slowness, and weakness. Multiple research projects on the detection of frailty using technology have been initiated based on Fried’s phenotype [11,12], while others have developed their own methodology [13]. There remain several research gaps which justify further research in this area:

• Objective measures: Fried relies on subjective measures, which can lead to inaccurate estimations of frailty. This can be improved by using cutting-edge technologies that are able to measure precisely and overcome human error.
• Professional assessment dependency: while most of the criteria require a health care professional’s assessment, because of the nature of frailty, smart devices can assist the elderly in a non-intrusive manner, enabling them to more easily provide correct information for further computation and insight extraction.
• Ease of use: most projects propose complex sensor setups for data collection, and these may be difficult for elderly patients to use.
• Personalization: A new dimension should be taken into consideration, i.e., assessing the dynamic nature of frailty. For instance, various factors, such as changes in social environment, can interfere with systems that have a predefined baseline for health conditions and therefore lack personalization, and this can lead to false outcomes.

The current work is part of a Eureka European funded project (https://cinnamon-project.eu/en/home-page/, accessed on 11 June 2023) and is intended to develop a system able to detect early signs of frailty in elderly people. The main component of this system will be an affordable commercial off-the-shelf (COTS) product. The project has the following objectives:

Research objective 1: conduct research to determine the COTS product and sensors necessary to obtain relevant insights concerning frailty based on an already validated phenotype;

Research objective 2: develop and administer a questionnaire to determine the functional and non-functional requirements that the elderly and their caregivers have of the system;

Research objective 3: design, develop, and test a prototype that can meet the initial requirements and record and process raw sensor data;

Research objective 4: assess insights related to one or more criteria of the selected phenotype based on data recorded by the prototype;

Research objective 5: conduct a clinical study using data collected from elderly subjects to discover patterns and create an artificial intelligence model that can assess frailty;

Research objective 6: demonstrate a proof of concept of a system able to identify early signs of frailty via developed software and self-reporting tools and integrated in a master dashboard to alert caregivers to early signs of frailty in monitored elderly patients.

Following the research design phase, a smartwatch wearable device was found to satisfy the initial requirements of an easy-to-use and non-intrusive system. Fried’s phenotype was used in conjunction with cutting-edge technology to detect frailty in this project.

To detect and assess one of the five Fried criteria, low physical activity, the system is designed to measure different daily activities and compute an average daily energy expenditure. The energy expenditure is determined by human activities that are detected and processed at the system level. Park et al. (2022) [14] demonstrated in their extended study that non-frail adults carried out their daily activities for significantly longer durations and with increased energy expenditure compared with frail adults. According to Silva et al. (2019), there is an association between frailty and an insufficient physical activity level due to excessive sedentary behavior [15], while Pozo-Cruz et al. (2017), using a triaxial accelerometer to objectively assess sedentary time, found that there is a relationship between periods of low physical activity and frailty in older adults [16].

A previous study demonstrated that by using pre-trained decision tree models it is possible to classify user activities in real-time [17], while another study showed that k-nearest neighbors (KNN) produces better results with wearables and concluded that a combination of multiple classifiers constitutes a superior approach [18]. Several studies have validated the use of accelerometers and proven them to be a valuable source of energy expenditure data [19].

In a previous study, a questionnaire was completed by 58 participants who were divided into two categories: respondents monitoring a frail patient and respondents clinically diagnosed as frail [20]. The questionnaire was administered during a short period of time in 2022. The subjects were selected by psychologists involved in the research. The process consisted of two phases: in the first, the selected elderly participants were diagnosed as frail if they had a high Groningen questionnaire score, a valid clinical index for frailty assessment [21]; in the second phase, the participants answered a list of questions to determine the key functionalities of our intended system, e.g., usability, data privacy, cost, and personal security. From the questionnaire results, a list of functional and non-functional requirements was defined: a system able to detect frailty must be easy to use, non-intrusive, reliable, trustworthy (i.e., it must have a high level of security to ensure data privacy and confidentiality), and affordable.

Based on these requirements, an IoT system with a Technical Readiness Level of 4 (technology validated in a laboratory environment) was successfully developed and tested [22]. The system uses high amounts of data and processes it under high-performance and low-cost conditions (in terms of hardware resources and processing power), and the results are available in [22]. The primary data source for the prototype is the raw data obtained from the sensors (accelerometer, gyroscope, orientation, optical) embedded in a smartwatch worn on the wrist. Compared with the five other projects described in [22], our system is easy to use and non-intrusive, and it has robust security ensured by standard and custom mechanisms.

The current article is organized as follows: first, it presents the methodology used for data acquisition. Once different sets of data were successfully recorded, they were analyzed using a data visualization module. The recorded data were divided into three input sets and used to train seven machine learning models. The goal was to check the system’s ability to detect various activities using data from the previously developed system and to identify the minimal data requirements for achieving comparable performance using lighter input sets. After both analyses were conducted, several conclusions were drawn, and the need for future work was emphasized.

2. Data Analysis Methodology

The main component of the IoT system is a Fitbit Versa smart wearable, which records raw data and forwards it to a private server. It was determined to be the main source of data needed to detect early signs of frailty based on Fried’s phenotype [8]. The data acquisition protocol involved recording data concerning the performance of different activities for a limited period. In Figure 1, the general architecture and main data flow are illustrated. The captured data frame has eight values from four built-in input sensors:

• an accelerometer which outputs x, y, and z accelerations;
• a gyroscope sensor, with gx, gy, and gz representing the rotation around each axis;
• an orientation sensor, with $q_0$ representing the scalar part of the quaternion and $q_1$, $q_2$, and $q_3$ representing the factors of the quaternion;
• an optical heart rate sensor, with output W representing the heart rate value.

The data frame is presented in

Table 1. Data frame structure.

activity

x

y

z

gx

gy

gz

$q_0$

$q_1$

$q_2$

$q_3$

W

timestamp

. The system’s input data are ingested by a data processing (DP) module. In this data frame, activity represents the type of activity recorded and timestamp indicates the time the data were recorded. An accelerometer is used for frailty detection and an orientation sensor gives additional insights (this sensor can give output related to the rotation of the individual). The heart rate monitor sensor is used to distinguish between activities with different accelerations and orientation quaternion amplitudes (for example, activities which require short movements of the arms, but which require high energy usage).

As Figure 1 shows, four components of the system can be distinguished:

• The data capture module, represented by the smartwatch with input sensors. This module is represented by a system which had been previously designed, developed, and tested to record raw data obtained from smart watches [22].
• The data processing component, responsible for removing noise and normalizing data for other components. It prepares data in different input formats so that they can be used by the plotter and the analyzer;
• The activity data plotter, represented by a MATLAB module.
• The machine learning algorithms analyzer, represented by a python module which automatically sets parameters for each classifier based on user input, trains models, and tests the performance of each one. This module handles multiple classifiers, including logistic regression, decision tree, random forest, gradient boosting, KNN, support vector machines (SVMs), and Gaussian naive Bayes. It is composed of the following:

  ○

  Parameter setting, which automatically sets the parameters for each classifier based on the user’s input. Parameters play a crucial role in machine learning algorithms as they define the behavior of the models during training. By allowing users to input their desired parameters, the module ensures flexibility and customization.

  ○

  Model training, which takes the specified parameters and trains models for each classifier. Training involves feeding the model with labeled input data (known as the training set) and allowing it to learn patterns and relationships within the data.

  ○

  Performance evaluation, which tests the performance of each model after it has been trained. Performance evaluation is essential to determine how well the models generalize to unseen data. Common evaluation metrics include accuracy, precision, recall, and F1 score.

  ○

  Classifiers, several of which are supported, each with its own characteristics and advantages.

2.1. Data Visualization Using Activity Data Plotter

The activity data plotter takes the input data and plots them automatically using MATLAB script, performing data visualization for a sensor dataset that includes data obtained from the accelerometer, the gyroscope, and the orientation sensor (expressed as quaternions). The script computes the vector magnitude (VM) and Euler’s angles.

• The vector magnitude (VM) is the size of the vector with accelerations as components:

$$VM=\sqrt{x^2+y^2+z^2}$$

(1)

• Euler’s angles (φ, θ, and μ) refers to a set of three angles that are used for describing the orientation or movement of a rigid body (in the current research, that of the user) in a three-dimensional space. These three angles (pitch, roll, and yaw) are determined using the quaternion components of the orientation sensor:

$$\left[\begin{array}{c}\phi \\ \theta \\ \mu \end{array}\right]=\left[\begin{array}{l}\arctan \frac{2(q_0{q}_1+q_2{q}_3)}{1-2({q_1}^2+{q_2}^2)}\\ \arcsin \left(2\left(q_0{q}_2-q_3{q}_1\right)\right)\\ \arctan \frac{2(q_0{q}_3+q_1{q}_2)}{1-2({q_2}^2+{q_3}^2)}\end{array}\right]$$

(2)

The script provides a comprehensive visualization of loaded data in both 2D and 3D spaces, separated for each activity type.

2.2. Automatic Comparison Using Machine Learning Algorithm Analyzer

The Python library sklearn (https://scikit-learn.org/stable/, accessed on 11 June 2023) was chosen for the automatic comparison as it is easy to use and well documented. It is an open-source library which provides tools for different machine learning classifiers and statistical models, and it was used to train different models with a learning to testing data size ratio of 4:1. The following classification algorithms were used for training:

• Logistic regression (LR) classifier
• Decision tree (DT) classifier
• Random forest (RF) classifier
• Gradient boosting (GB) classifier
• K-nearest neighbors (KNN) classifier
• Support vector classifier (SVC)
• Gaussian naive Bayes (GNB) classifier

The logistic regression classifier is a statistical model based on probabilities. This classifier is utilized to classify data into one of two categories. The algorithm estimates the likelihood of a binary result, e.g., whether an activity type is walking or not, based on a set of input characteristics. The maximum likelihood estimation is used by the algorithm to derive a set of coefficients that can be used to map input features to the logarithm of the odds of the binary outcome.

The decision tree classifier is a machine learning method used for both regression and classification tasks. It is defined as a supervised learning method and it creates a tree-like model of decisions, emphasizing their possible outcomes.

The random forest classifier is a machine learning algorithm that utilizes a collection of decision tree classifiers, known as random trees, to classify patterns and data. By training each tree on a random subset of data and employing majority voting, it determines the final class of an input instance. This algorithm effectively handles data with multiple features and delivers precise and reliable results when classifying new data.

While the random forest classifier builds an ensemble of independent decision trees that vote on the final prediction, the gradient boosting classifier sequentially trains decision trees to correct the errors of previous trees. In this way, it avoids using weak learners in its model and achieves a stronger decision model by adjusting the subsequent models to target the remaining errors or gradients of the loss function from the previous predictions.

Like the decision tree classifier, the k-nearest neighbors classifier is a type of machine learning algorithm used for both classification and regression tasks. Its accuracy depends on data quality, and it determines the classification of a new data point by finding the “k” closest labeled datum in the n-dimension space.

Another common learning method is the support vector classifier, a linear algorithm that constructs a set of planes with the data separated into multiple classes. It has the advantage of being suitable for large datasets because it can adjust the parameters of the generated spaces.

The last algorithm we tested was the Gaussian naïve Bayes classifier, which is a classification algorithm that assumes that features in the data follow a Gaussian distribution, i.e., that they are independent of each other. It calculates the mean and standard deviation for each feature in each class during training and uses these statistics to build a probability model. It is efficient and works well with high-dimensional data, but it relies on the assumed feature independence.

2.3. Data Acquisition Protocol

Data were collected for four types of activities:

• Fast walk: the subject is walking at a fast pace.
• Slow walk: the subject is walking at slow pace.
• Resting: the subject is standing still.
• Climbing up stairs: the subject is going up a staircase.

For these activities, five one-minute recording sessions were carried out. A five-minute pause was taken between sessions to avoid data compromise. The sensors (Fitbit Versa built-in sensors: accelerometer, gyroscope, orientation sensor, and optical sensors) were set to record data ten times per second (except the optical sensor, which only recorded one datum per session). The data were forwarded to a NodeJS server to be encapsulated in JSON format and saved on a MongoDB server for further processing. All the data were recorded by the same user.

3. Results

3.1. Data Visualization Using Activity Data Plotter

The first part of the data analysis was performed using the data visualization comparison component of the system. For each activity, 600 datasets were annotated and used as input for the plots. To add stability and increase the reliability of the data, the mean value was calculated for every two sets and used as input, mitigating any random errors of the sensors. The goal was to observe the differences between each activity as represented by the different input sensors. The resulting plots were divided into four sections, each section corresponding to an activity: 1st section—fast walk; 2nd section—slow walk; 3rd section—resting; and 4th section—climbing up stairs.

Figure 2 illustrates the x, y, and z accelerations for each activity type. As can be observed, in the case of resting, the readings are stable and low in amplitude because the force of gravity is measured and the subject is not moving. The slow walk readings are characterized by moderate and regular oscillations, while the fast walk readings have larger oscillations, reflecting the increased speed. It can clearly be observed that it is easy to distinguish between these three activities. In the first part of plot, the x and y accelerations for climbing up stairs do not respect the pattern present in the rest of plot. This is because in the first session the smartwatch was worn inside-out.

Figure 3 shows the vector magnitude for each acceleration. Vector magnitude was chosen as the means to determine the intensity of the vector formed by the accelerometer in different phases. It can be observed that there are significative differences in terms of amplitude for all three axis accelerations. While we have high amplitudes for fast walk in each session, in the case of resting, there were small changes in the position of the smartwatch. Additionally, in Figure 3, there are major differences between fast walk and the other activities, but the similarities in terms of amplitude between slow walk and climbing up stairs require further investigation.

If the data from the accelerometer are plotted in tridimensional space, as in Figure 4, it can be observed that the dispersion of points for each activity have different concentration points. This is the best representation for accurately spotting significant differences between the selected activities. While an overlap between the different points of the activities can be observed, these are a result of the nature of movement. The main insight they provide is that all four activities form different data clusters in three-dimensional space. The most significant observation is that for fast walk, we have a more spread-out pattern. Another meaningful observation is that in 3D space, the points for resting are not what we expected: there is less variation compared with the other activities across all three axes, and they are concentrated in a small region.

Another source which can be used for features extraction is the orientation sensor. After computing Euler’s angles, the differences between slow walk and climbing up stairs led to a new hypothesis: in conjunction with accelerations, it is possible to create a data model which can easily predict a performed action. This hypothesis is based on Figure 5. It can be observed that for resting, where the movement is minimal, the three angles are relatively constant, meaning that the person carried out small body movements during the recording session. The most recognizable activity is climbing up stairs, which has high variations in terms of amplitude for roll and yaw. This is because when a person is climbing up stairs, there is more of a vertical motion compared with that involved in fast walk or slow walk, where the movement takes place along a horizontal plane and resembles the swing of a pendulum.

3.2. Classification Using Machine Learning

After conducting a comprehensive data visualization analysis, the automatic comparison module was fed with input data. This step aimed to assess the behavior and performance characteristics of the different models. The data were split into training data (80%) and testing data (20%) to ensure a robust evaluation of each model’s performance. The purpose of using artificial intelligence algorithms was to check the feasibility of a real-world application.

Based on the source data, three new input sets were defined:

(1)

Input set 1 contained accelerations x, y, and z from the accelerometer, q₀, q₁, q₂, and q₃ from the orientation sensor, gx, gy, and gz from the gyroscope, the heart rate monitor value, and the activity type;

(2)

Input set 2 contained the acceleration vector magnitude, the heart rate monitor value, Euler’s angles (the pitch, yaw, and roll from the orientation sensor), and the activity type;

(3)

Input set 3 contained the acceleration vector magnitude, q₀, q₁, q₂, and q₃ from the orientation sensor, and the activity type.

The reason for defining new input data was to compare how the models performed when they were given fewer inputs and to determine whether they were sufficiently able to analyze lighter datasets in order to compute and predict activity types.

The seven chosen models were trained and tested in order to determine the following metrics: accuracy, i.e., the proportion of correctly classified sets out of the total number of used sets; precision, i.e., how many instances were correctly classified as positive out of the total number of positive instances predicted by the model; recall, i.e., the ratio between the number of correctly classified positive instances and the number of actual positive instances; and the F1 score, i.e., the harmonic mean of the precision and the recall, which helps to assess the overall performance of the model.

The performance metrics for each input set are presented in

Table 2. Input set 1, input set 2, and input set 3 performance comparison.

Model	Accuracy			Precision			Recall			F1 Score
	1	2	3	1	2	3	1	2	3	1	2	3
Logistic regression	0.8095	0.6054	0.7664	0.8044	0.5588	0.774	0.8095	0.6054	0.7664	0.8001	0.5744	0.7574
Decision tree	0.9138	0.907	0.9388	0.9155	0.9067	0.9393	0.9138	0.907	0.9388	0.9144	0.9069	0.939
Random forest	0.9524	0.9252	0.9546	0.952	0.9245	0.9541	0.9524	0.9252	0.9546	0.952	0.9248	0.9541
Gradient boosting	0.9637	0.9252	0.9456	0.9642	0.9241	0.9454	0.9637	0.9252	0.9456	0.9635	0.9244	0.9454
K-nearest neighbors	0.9138	0.8617	0.8503	0.9156	0.8588	0.8479	0.9138	0.8617	0.8503	0.9127	0.86	0.8474
SVC	0.8254	0.8095	0.8095	0.736	0.7219	0.7238	0.8254	0.8095	0.8095	0.7736	0.7573	0.7577
Gaussian NB	0.8866	0.8639	0.8707	0.8914	0.861	0.8669	0.8866	0.8639	0.8707	0.8883	0.8574	0.8679

. As can be observed in Figure 6, where the performance for input set 1 is visually represented, all seven models have a median accuracy over 0.89, with the gradient boosting model reaching 0.96. The working mechanisms of this model, which is more robust to overfitting, result in a more accurate prediction. The same result can be observed for the decision tree and random forest models, which, like the gradient boosting model, use tree-based models for their predictions. Overall, the gradient boosting model outperformed the other models in all metrics, making it the most suitable model for activity type recognition. Because the datasets are time series, meaning that they are recorded one after the other, the other models can be adjusted in order to detect false positive results and correct them during prediction.

Comparing the three input sets, the gradient boosting and random forest classifiers demonstrated remarkable consistency in their performance metrics across diversified datasets. This demonstrates the ability of the models to adapt to data variations. Both classifiers consistently ranked among the highest performers in terms of accuracy, precision, recall, and F1 score metrics. While the k-nearest neighbors classifier achieved decent performance, it fell behind the tree-based models when lighter datasets were provided.

The logistic regression classifier initially achieved decent performance, but it demonstrated the most substantial decrease in performance when lighter datasets were provided, making it unsuitable for such classifications. The Gaussian NB model was also robust.

4. Discussion and Conclusions

The data visualization captured various aspects of raw sensor data, each plot enabling a different interpretation for each activity. The differences in each visual representation enable us to distinguish between different activities, and this means that classification models can provide valuable results.

Machine learning models provide great results in terms of performance for the classification of activity types. The performance of all models on input set 3 was either similar to or better than their performance on input set 2. This suggests that the reduced data in input set 3 still captured the essential patterns needed for prediction, while the data in input set 2 may lack some significant features. The use of input set 1 required more processing power, and this has a direct impact on the hardware specifications of a wearable. It is not necessary to process all the data from the wearable sensors; the same results can be obtained with a low-cost wearable.

The results from both the data visualization and the machine learning comparison indicate the promising potential of a lightweight system which can assess different activities performed by an elderly individual during the course of a day in order to detect frailty. Activity detection can be applied to daily activities and provide energy expenditure insights. Energy expenditure is considered a baseline for physical activity, and it can be used in conjunction with other information to quantify early signs of frailty in the elderly.

It is safe to assert that using a combination of an accelerometer, an orientation sensor, and heart rate data is sufficient to distinguish between different daily activities. The comparison between the initial input data and the lighter datasets derived from it gives another valuable insight: it is not necessary to transfer all the data from a smartwatch to a data processor to extract valuable information about activity types. Effective results can be obtained using the vector magnitude of the accelerations and the orientation sensor quaternions.

More research is needed to overcome the limitations of the prototype. The most concerning of these is the subjectivity and variability of the collected data. Different people may exhibit different patterns of movement during fast walk, and more analysis is needed to see how data varies among individuals with different characteristics (e.g., age, weight, and height). The nature of the data collection is another limitation: to obtain big datasets requires a lot of physical activity in different recording sessions, and this is a challenge.

The frailty-detection process should also involve the measurement and analysis of physiological parameters. Due to technical and financial limitations, our prototype only records heart rate values. Future work should also aim to include other parameters, such as respiratory rate, blood pressure, and blood oxygenation, to detect whether there are connections between these and early signs of frailty.

Collecting data on the movement patterns of individuals, especially in the context of physical activity, may involve capturing sensitive information that these individuals might not be comfortable sharing. These could include details about their health conditions, disabilities, or other personal details that they may not want to disclose or have used against them. Moreover, there is a risk that the data collected through these prototypes could be vulnerable to misuse or unauthorized access. If the collected data are not handled properly, they could be exploited by malicious entities for purposes such as identity theft, targeted advertising, or discriminatory practices. Therefore, it is crucial to address the ethical considerations surrounding data collection in prototype technologies. This includes implementing robust security measures, obtaining informed consent from participants, and ensuring that data are anonymized and used only for their intended purposes. The consideration of these ethical concerns is essential for the responsible development and deployment of prototype technologies as it helps to safeguard individuals’ privacy and maintain trust in the research and development process.

Given that frailty detection is a multidimensional process that involves leveraging various information sources from a user, the models do not need to achieve high accuracy. Due to technical limitations, Further research is necessary to address, according to Fried’s phenotype, exhaustion, slowness, and weakness, using smart devices. The next step in this project is to record data from clinically diagnosed frail individuals and compare their extracted data to data from healthy individuals with the aim of detecting significant differences. This paper demonstrates that when detecting frailty insights using a wearable, it is possible to use artificial intelligence to distinguish between different activity types and compute an overall physical activity score to assess frailty criteria. Future work should investigate whether a individual’s slowness can also be extracted from wearable sensor data.