Modular Open-Core System for Collection and Near Real-Time Processing of High-Resolution Data from Wearable Sensors

Abstract

Wearable devices, such as smartwatches integrating heart rate and activity sensors, have the potential to transform health monitoring by enabling continuous, near real-time data collection and analytics. In this paper, we present a novel modular architecture for collecting and end-to-end processing of high-resolution signals from wearable sensors. The system obtains minimally processed data directly from the smartwatch and further processes and analyzes the data stream without transmitting it to the device vendor cloud. The standalone operation is made possible by a software stack that provides data cleaning, extraction of physiological metrics, and standardization of the metrics to enable person-to-person and rest-to-activity comparisons. To illustrate the operation of the system, we present examples of datasets from volunteers wearing Garmin Fenix smartwatches for several weeks in free-living conditions. As collected, the datasets contain time series of each interbeat interval and the respiration rate, blood oxygen saturation, and step count every 1 min. From the high-resolution datasets, we extract heart rate variability metrics, which are a source of information about the heart’s response to external stressors. These biomarkers can be used for the early detection of a range of diseases and the assessment of physical and mental performance of the individual. The data collection and analytics system has the potential to broaden the use of smartwatches in continuous near to real-time monitoring of health and well-being.

1. Introduction

Smartwatches that integrate sensors for measuring heart rate (HR) and activity are available in the commercial marketplace from multiple vendors. Originally used for fitness and activity tracking, these devices now find applications in monitoring the health and well-being of individuals and groups [1,2]. These wearable, minimally invasive devices provide an alternative pathway to clinical diagnostics that can be operated continuously in near real time [3]. For example, recent studies have demonstrated that the wearables can detect viral respiratory infections, such as influenza or COVID-19, before symptoms manifest themselves [4,5,6,7]. Other studies have shown the potential of wearables for detecting atrial fibrillation and sleep apnea [8,9]. Smartwatches have also been considered for monitoring the real-time performance status of military personnel and first responders [10,11,12].

In studies of wearables, signals recorded by smartwatch-integrated sensors were typically collected by a mobile application (app) provided by the original equipment manufacturer (OEM), which users install on their smartphones (Figure 1). After the smartphone and the watch are paired, the mobile app collects the signals. The data stream is typically transmitted to an OEM-operated cloud server that utilizes proprietary algorithms for further processing [13]. Examples of metrics that are extracted from sensor signals include step counts, HR, respiration rate, stress, and sleep quality [14]. The metrics, their changes over time, and various statistical measures can then be accessed through the mobile app, giving device users the ability to track their physical activity and vital signs. Third-party developers can access this health information for a group of individuals (e.g., a study cohort) once granted permission using an application programming interface (API).

Although the OEM-provided cloud services are essential for the broad use of smartwatches, wearables systems based on OEM-provided cloud services have limitations. For example, real-time health monitoring in remote locations and other resource-constrained environments may be hampered by intermittent Internet connection, which limits communication to the vendor cloud where more advanced health metrics (e.g., readiness) are calculated [11]. While some smartwatches can process and display health data on the watch, baselines of 7+ days are often needed to compute more complex health indicators such as heart rate variability (HRV) anomalies, and these are only available on higher-end devices. These requirements limit the ability to rapidly deploy monitoring systems for military personnel and first responders in tactical operations or emergency situations in which insight into an individual’s current health status is critical. Modifications to the current OEM data collection and analysis paradigm are also needed in sensitive situations in which data can only be stored and processed on approved enterprise servers and cannot leave the controlled network [10,15]. For example, only selected cloud-based solutions are approved for housing the controlled unclassified information (CUI) generated from wearables worn by military, homeland security, and in some cases, first responder personnel.

In addition to the use cases focused on military and first responder personnel, researchers also need higher-resolution data and more insight into the calculations being performed than is traditionally made available through OEM services. Vendor-provided metrics are heavily processed (e.g., averaged over minutes or hours), and data access is traditionally limited to daily summary and windowed information (which can vary in reporting frequency from 15 s to 15 min depending upon the vendor and metric). Algorithm calculations are vendor proprietary and not transparent for interpretation by the end user. For example, sleep metrics (e.g., sleep quality, sleep duration) can vary widely between smartwatch models and compared to gold-standard polysomnography [16]. The availability of higher-resolution data (e.g., the millisecond differences in time between each heartbeat versus average HR in beats per minute) allows researchers to engineer more insightful metrics (e.g., time-domain, frequency-domain, and non-linear HRV metrics) in order to develop new or more advanced analytical models and improve the performance of existing models. Additionally, researchers can utilize open-source algorithms for extracting physiological metrics to provide transparency and understanding that is not typically obtainable from proprietary analytics engines [10,12,17].

In this paper, we describe a system architecture for collecting and processing data from wearable sensors that is independent of OEM cloud services, is compatible with secure local operation, and provides access to high-resolution data. This system can be used for a variety of applications, from monitoring human performance and well-being to remote medical care. We implemented this architecture in a system that collects and processes data from Garmin Fenix series smartwatches [18]. The high-resolution data consist of a series of interbeat intervals (IBIs), which are time intervals between consecutive heart beats recorded with millisecond precision by the photoplethysmography (PPG) sensor integrated into the watch [19]. This data stream is accompanied by the count of steps [20], reported once per minute, which provides a measure of the level of physical activity, as well as respiration rate and blood oxygen saturation (SpO₂). The IBI and step count time series are the basis for the calculation of not only the average HR, but also a variety of HRV metrics [21]. HRV is a source of information regarding the operation of the autonomic nervous system, which regulates the human body’s response to illness and other physical and psychological stressors [22].

The operation of the wearables system is illustrated with examples of datasets collected from volunteers monitored with Garmin Fenix series smartwatches in free-living conditions. We process the datasets using a combination of open-source techniques available from the literature and developed in-house. Among the techniques are algorithms for the identification of artifacts in the collected data, the extraction of HR and HRV metrics, and the standardization of these metrics for each individual with respect to the individual’s activity level. The activity matching allows us to express cardiac metrics in the context of the user’s physiological condition, thus advancing HR monitoring beyond absolute measurements toward relative assessment of one’s cardiac performance.

The rest of the paper is organized as follows: Section 2 describes the architecture of the system, its interfaces to communication networks, hardware requirements, and the software stack. Section 3 presents the methodology for processing the wearables data. Section 4 presents examples of datasets acquired using the system and metrics extracted from the datasets. Section 5 provides a brief review of potential applications of smartwatch-based monitoring for detecting disease and for optimizing human physical and mental performance. Section 6 presents a summary and outlook.

2. System Architecture

The data-collection and processing system, as shown in Figure 2, consists of three modules: (1) wearable devices integrating sensors acquiring physiological and other data; (2) SyncHubs handling data synchronization with the wearables; and (3) a local or cloud-based storage and analytics node (SAN). The SAN can transfer selected data and metrics to other networks or visualization resources (the optional fourth module).

The architecture is modular and can be adapted to support multiple concepts of operations. There is no transmission of data to OEM cloud servers, and the system can operate with no connectivity to Internet.

2.1. Wearables

The system architecture, as shown in Figure 2, is compatible with any wearable device that is provided with a software development kit (SDK) to enable access to raw or minimally processed signals collected by the sensors, either directly or through a smartphone or tablet paired with the device. To date, we have deployed the system with a family of smartwatches and wristbands from Garmin Ltd. (including the Garmin Fenix 6) and with the Samsung Galaxy Watch 3. The wearables integrate two PPG sensors for monitoring HR and SpO₂, and a three-axis accelerometer for detecting movement in the x, y, and z directions. The watch firmware uses accelerometer signals to calculate step counts that are posted every 1 min.

2.2. SyncHubs

SyncHubs are low in size, weight, and power devices, such as smartphones or tablets, which handle data acquisition from wearables in either one-to-one or many-to-one pairing configuration. To date, we have deployed the system using Android smartphones and Android Samsung Galaxy Tab Active3 and S6 Lite tablets. The data transfer and synchronization is facilitated through the custom application, called SIGMA+ Health (S+) for the Defense Advanced Research Projects Agency (DARPA) program that supported the development (Figure 3).

The S+ app builds on the Garmin Companion SDKs for Android and iOS to provide direct integration with supported devices and enable data logging of IBIs, step counts, respiratory rate, and SpO₂. Data acquisition between the S+ app and the wearable is executed via a Bluetooth Low Energy connection that supports persistent pairing for automated background synchronization (sync). The data are stored in an encrypted SQLite database until the app initiates data transmission. During data transmission over HTTPS (Hypertext Transmission Protocol Secure), encrypted Java Script Object Notation (JSON) messages are conveyed to one or more network-addressable SANs using standard web services. These endpoints are modifiable and specific to the cohort.

The initial setup associates the user with a unique cohort through a study PIN and a unique participant ID that does not contain personally identifiable information. The participant ID allows the application to track the user’s data from one or more devices throughout the study. Within the app, users can view the connectivity status of their wearables (as to whether they are paired) and manually trigger a data sync if required. They can also determine the status of an in-progress sync or time since the last sync and successful upload. The SyncHub can also provide the GPS coordinates of the mobile device when needed.

2.3. Storage and Analytics Node

The designated endpoint edge or cloud-based hardware runs the SAN software stack that receives, processes, and exports data transmitted by SyncHubs. The SAN’s software stack is compatible with commodity hardware. To date, we have deployed SANs on AWS cloud clusters and, in a simplified form (simplified user interface), on Panasonic Toughbook laptop computers and Xavier NVIDIA Jetson-based single-board computers.

Figure 4 shows a simplified schematic of the SAN’s architecture as it is implemented in an instance on the AWS cloud.

Users’ physiological data are ingested by the system through a specific receiving IP address, where they are validated by an edge service (the exposed service on the SAN that receives network external traffic) and forwarded to a message queue that is implemented using Apache Kafka^®(Apache Software Foundation, Wakefield, MA, USA). Once in the queue, messages are distributed to internal consumers, such as a database write service and data-processing algorithms. The write service writes data and algorithm outputs to a database to be persistently stored on disk. These data are then available for distribution by the data API, a service that receives and responds to requests for data by internal or external consumers, such as visualization clients or other third-party applications.

3. Data-Processing Methodology

Figure 5 illustrates the methodology that we use for the processing of data acquired from wearables integrating HR and activity-monitoring sensors.

3.1. Data Quality Control

Sensor data acquired through the S+ app are packaged in the JSON format. Each data point has an associated timestamp with millisecond precision and value. The number of steps, SpO₂, and respiration rate are reported in 60 s intervals. IBI values with millisecond precision are reported with each detected beat. The incoming data are segmented into 5 min windows for each data type with each point in the window formatted as [timestamp, value] and aligned based on real time so that the windows start at 5 min marks (e.g., 12:00, 12:05).

To remove artifacts, we specify for each data type a range of acceptable values formatted as [low_accept, high_accept]. If the value is smaller than low_accept or greater than high_accept, the datapoint is flagged. For step count, respiration rate, and SpO₂, the flagged values are removed. For IBI, we flag and remove IBI values where either the value itself or the difference in timestamps are outside of the valid range.

We define the following data quality control (QC) measures to characterize each of the incoming raw (or minimally processed) data streams:

• Raw data fraction, RDF (%), is calculated as the ratio of the number of datapoints collected to the maximum number of datapoints possible in the monitoring period;
• Artifact data fraction, ADF (%), is calculated as the ratio of the number of the artifacts removed by the QC checks from the data stream to the total number of datapoints collected in the monitoring period.

3.2. Metrics Extraction

Metrics (feature) extraction is the process of reducing a data stream to a set of informative and nonredundant metrics. The extracted metrics are used as inputs to models and support the model’s interpretability.

To analyze the variability of physiological metrics over time, we employ a moving window approach, where a window of a user-specified width and step moves through the input signal, time-stamping the position of the window and computing the median or average of the values in the window. In the discussed case, we are using a window width of 5 min and a step of 5 min (i.e., no overlap between neighboring windows).

Table 1. Heart rate variability metrics that can be extracted from IBI time series [21,23].

Metric	Unit	Description
Time Domain
IBI	ms	Median IBI
SDNN	ms	Standard deviation of interbeat intervals
RMSSD	ms	Root mean square of successive differences in interbeat intervals
PNN50	%	Percentage of successive interbeat intervals that differ more than 50 ms
TI	-	HRV triangular index = the integral of the HRV histogram divided by the height of the histogram
Frequency Domain
TP	ms2	Total spectral power
VLF		Log of very-low-frequency power (0–0.04 Hz)
LF	-	Log of low-frequency power (0.04–0.15 Hz)
HF	-	Log of high-frequency power (0.15–1.0 Hz)
Resp	Hz	Respiration rate
Nonlinear Domain
DFA		Short- and long-term fluctuations in detrended fluctuation analysis
SD1	-	Standard deviation of the points perpendicular to the line-of-identity in the Poincare plot
SD2		Standard deviation along the line-of-identity in the Poincare plot
SampEn	-	Sample entropy

lists some of the HRV metrics that can be extracted from IBI time series.

To calculate HRV metrics in the frequency domain, we resample the IBI stream to 5 Hz using the piecewise cubic-Hermite-interpolating polynomial method [24]. If there is a gap in the signal greater than 15 s, we pick the longest continuous segment to resample in the 5 min window. If there are more than three gaps of this length, we do not use this window. After interpolation, the signal is filtered to the low-frequency (LF: 0.04 to 0.15 Hz) and high-frequency regions (HF: 0.15 to 1.0 Hz; we increased the upper limit to 1.0 Hz vs. the typically used 0.40 Hz to account for physically active persons). We compute a value representing the LF and HF power for 60 s and 30 s windows, respectively, using the following formula: value = ln (variance (data window)). Values less than 2.5 and more than 9.0 are eliminated, and the median values are calculated to represent the 5 min window [24].

For respiration rate, SpO₂, and step count, all of which are reported by the S+ app every 1 min, we calculate the average of values in the 5 min window; we require at least two valid data points for the calculation. The window width and the required number of datapoints are configurable.

For each metric, we define the valid data fraction, VDF (%), as the ratio of the number of valid 5 min windows to the total number of 5 min windows in the monitoring period.

3.3. Metrics Standardization

The SAN software stack provides the option of standardizing the extracted metrics with respect to the person’s baseline. Average HR and HRV values vary widely from person to person, and the instantaneous values for a given person depend on whether the individual is resting or is engaged in physical activity. We establish the baseline for each person by collecting the data over several days in this person’s healthy state. The data are collected in the first-in, first-out buffer (Figure 5). Parameters of the baseline are used in the calculation of z-scores, according to the following formula:

$$z{X}_{ij}\left(S\right)=\frac{X_{ij}\left(S\right)-{\mu }_j\left(S\right)}{{\sigma }_j\left(S\right)},$$

where $X_{ij}$ is the datapoint $i$ of metric $j$, S is the number of steps per minute, and ${\mu }_j\left(S\right)$ and ${\sigma }_j\left(S\right)$ are the mean and standard deviation of $X_{ij}$, respectively, calculated from the baseline dataset that is matched to the S of the $X_{ij}$ datapoint.

The activity matching is determined according to the following rules using the current 5 min step count value and further limited using an average of the step counts for the previous 30 min (i.e., two-step match):

• Data with step counts exceeding an UPPER_STEP threshold are considered equivalent;
• Data with step counts below a LOWER_STEP threshold are considered equivalent;
• Data with step counts between the LOWER_STEP and UPPER_STEP thresholds are matched based on step counts within 30% of the current value. Lower limits on the minimum step match (e.g., ±10 steps) and lower bound (e.g., greater than 0 steps) are applied.

After matching, we limit the size of the dataset in the first-in, first out buffer (Figure 5) to the most recent 1440 datapoints (equivalent to 5 days of matched data using a 5 min window) and calculate the mean and standard deviation of the metric for the baseline set. If the minimum amount of required data is present in the matched dataset, a standardized value is calculated. The hyperparameters of the calculations (e.g., the required numbers of datapoints) are configurable.

4. Examples of Datasets and Metrics

Below, we present examples of datasets acquired using the described data-collection and processing system. The goal of this effort was to test system functionality for an extended period (up to 6 months) under realistic conditions and determine if the data collected could be used to track changes in a user’s health in near real time. A total of 47 individuals (23–78 years, mean: 38.9 years; demographic information was not collected for 20 individuals in the initial pilot study) were enrolled into data-collection studies that were approved by the RTI’s internal review board. Individuals provided informed consent and were asked to wear a Garmin Fenix 6 smartwatch as continuously as possible as they went about their normal life and to complete a daily health survey.

As described in Section 3, data streams collected with the S+ app contained the IBI value for each detected heartbeat, and SpO₂, respiration rate, and steps counts every 1 min. Figure 6 presents plots of the original data collected for 1 day selected from the total monitoring period as an example. Figure 7 shows selected HRV metrics extracted from a dataset collected for 1 week; all metrics were calculated in 5 min epochs, as described in Section 3. Also shown in Figure 7 are plots of IBI data quality metrics: RDF, VDF, and ADF averaged hourly.

To illustrate the best-case scenarios for the described data acquisition and processing system, the dataset presented in Figure 6 and Figure 7 was selected from among the best data quality datasets we acquired when testing the system. The RDF and VDF are close to 100% on average, indicating good wear compliance on the part of the user, no issues in synchronizing the data on the watch with the smartphone, and low ADF.

5. Applications

To illustrate one of potential applications of the data-collection and analytics system, Figure 8 gives an example of the use of selected HRV metrics as inputs to the S+ algorithm developed in an earlier study for early detection of respiratory infections [17]. Inputs to this S+ algorithm are standardized IBI, LF, and HF values (i.e., z-scores; see Section 3.3) calculated in 5 min windows, and the output is S+ score, which was found to be correlated with the presence of infection. As seen in the bottom panel of Figure 8, the S+ scores are elevated prior to the device user reporting symptoms of illness. Symptom scores were calculated using the daily health survey that asks participants to provide a 0–3 score for 11 symptoms (0 is no symptoms and 3 is severe symptoms).

Table 2 lists other applications of continuous HRV monitoring, along with references to previously published studies. The presence of correlations between HRV and phenotypical measures related to specific illnesses that these studies demonstrated can enable early detection of physiological anomalies and help guide effective interventions. Other reported applications of HRV monitoring include fitness tracking, sleep quality tracking, and fatigue assessment.

The described data-collection and analytics system is constructed for extendibility to other sensors and algorithms. Of particular relevance, given the strong dependence of HRV metrics on the type and level of physical activity, are algorithms associated with human activity recognition, a rapidly growing field of research that aims to automatically identify and classify human motions and activities from various tracking devices [25,26].

Table 2. Examples of applications of HRV monitoring.

Application	HRV Metrics	Ref.
Coronary artery disease	SD1, SD2	[27]
Atrial fibrillation	IBI	[28]
Obstructive sleep apnea	VLF, LF, HF, TP, SDNN, RMSDD, pNN50, TI, SampEn	[29]
Chronic obstructive pulmonary disease	SDNN, LF, HF	[30]
Viral respiratory infection	IBI, LF, HF, LF/HF	[17]
Mild traumatic brain injury	IBI, LF, HF, LF/HF	[31]
Epilepsy	IBI, LF, HF, LF/HF	[32]
Major depressive disorder	LF, HF	[33]
Chronic fatigue syndrome	RMSSD, SDNN, pNN50, LF, HF	[34]
Sleep quality assessment	RMSSD, SDNN, pNN50, LF, HF	[35]
Fitness tracking: aerobic threshold determination	DFA	[36]

HF = high frequency; HRV = heart rate variability; LF = low frequency; pNN50 = percentage of adjacent intervals; RMSSD = root-mean-squared successive difference of IBIs; SampEn = sample entropy; SDNN = standard deviation of IBIs; TI = triangular index; TP = total power; VLF = very low frequency.

Table 2. Examples of applications of HRV monitoring.

Application	HRV Metrics	Ref.
Coronary artery disease	SD1, SD2	[27]
Atrial fibrillation	IBI	[28]
Obstructive sleep apnea	VLF, LF, HF, TP, SDNN, RMSDD, pNN50, TI, SampEn	[29]
Chronic obstructive pulmonary disease	SDNN, LF, HF	[30]
Viral respiratory infection	IBI, LF, HF, LF/HF	[17]
Mild traumatic brain injury	IBI, LF, HF, LF/HF	[31]
Epilepsy	IBI, LF, HF, LF/HF	[32]
Major depressive disorder	LF, HF	[33]
Chronic fatigue syndrome	RMSSD, SDNN, pNN50, LF, HF	[34]
Sleep quality assessment	RMSSD, SDNN, pNN50, LF, HF	[35]
Fitness tracking: aerobic threshold determination	DFA	[36]

HF = high frequency; HRV = heart rate variability; LF = low frequency; pNN50 = percentage of adjacent intervals; RMSSD = root-mean-squared successive difference of IBIs; SampEn = sample entropy; SDNN = standard deviation of IBIs; TI = triangular index; TP = total power; VLF = very low frequency.

6. Summary and Outlook

We presented a modular architecture that is designed to collect and analyze minimally processed data from wearable devices. Because the system was designed to be standalone and independent of OEM cloud services, its software stack incorporates routines for end-to-end data processing, including data cleaning, metrics extraction, and metrics standardization. When implemented using a local server, data acquisition and data processing are achieved with no connection to the Internet; raw data, metrics, and algorithm results can later be transmitted to designated cloud resources when connectivity is established.

We deployed this system with Garmin Fenix series smartwatches and used smartphones or tablets running the S+ app to acquire high-resolution data. The S+ app transferred the data to either to a local server (e.g., laptop or single-board computer) or a cloud resource (e.g., AWS cloud clusters) where data were stored and analyzed. The system is currently being implemented in conjunction with the S+ algorithm for detecting viral respiratory illness before symptom onset and the algorithm for estimating core body temperature from average HR [17,37,38].

The proposed architecture has the potential to broaden the use of smartwatches in applications such as monitoring military personnel or first responders where communication resources may be limited, and strict security standards are in place. This system is also useful for research studies, where access to high-resolution data can be leveraged to develop and improve the accuracy and interpretability of custom algorithms and analytical models for disease diagnostics and human performance assessment. Future development efforts will be focused on increasing the number of wearables that can be used with the platform, as well as implementing additional algorithms and metrics that target short-term health changes due to exertional heat illness, psychological stress, and cognitive fatigue. The long-term concept is to provide a wearable sensor system that provides capabilities ranging from fitness and performance monitoring to remote, near real-time medical care.