NE-Motion: Visual Analysis of Stroke Patients Using Motion Sensor Networks

Abstract

A large number of stroke survivors suffer from a significant decrease in upper extremity (UE) function, requiring rehabilitation therapy to boost recovery of UE motion. Assessing the efficacy of treatment strategies is a challenging problem in this context, and is typically accomplished by observing the performance of patients during their execution of daily activities. A more detailed assessment of UE impairment can be undertaken with a clinical bedside test, the UE Fugl–Meyer Assessment, but it fails to examine compensatory movements of functioning body segments that are used to bypass impairment. In this work, we use a graph learning method to build a visualization tool tailored to support the analysis of stroke patients. Called NE-Motion, or Network Environment for Motion Capture Data Analysis, the proposed analytic tool handles a set of time series captured by motion sensors worn by patients so as to enable visual analytic resources to identify abnormalities in movement patterns. Developed in close collaboration with domain experts, NE-Motion is capable of uncovering important phenomena, such as compensation while revealing differences between stroke patients and healthy individuals. The effectiveness of NE-Motion is shown in two case studies designed to analyze particular patients and to compare groups of subjects.

1. Introduction

A stroke is a medical condition that results from a prolonged disruption of blood flow to the brain, leading to a loss of function or death [1]. According to Bonita et al. [2], the number of people afflicted by stroke is expected to continue to increase due to demographic changes and inadequate control of risk factors. Therefore, the development of methodologies to assist professionals in the treatment of stroke sequelae is of paramount importance.

The Fugl–Meyer Assessment (FMA) [3] is a clinical bedside instrument to assess abnormal motion in stroke patients. It is considered the gold standard for measuring motor impairment, and is used to guide rehabilitation treatment. The FMA testing protocol evaluates patients as they perform particular movements on both upper extremities (UE), which is the focus of this work, and lower extremities (LE). A trained assessor scores the movements according to a grading scale. The FMA is challenged by the need for a trained assessor and by the subjectivity and coarseness of its grading scale, which both limit the frequent administration of the FMA and the nuanced capture of impairment. In the last decade, FMA scoring has been complemented using data from sensors affixed to the patient’s UE [4,5,6], making it less subjective. These motion sensors generate a large amount of data that can be used to perform tasks other than FMA score computation. For example, the sensor data could be used to generate finer details about movement abnormality and identification of compensatory movements (alternative movement strategies to circumvent impairment). In these cases, the sensor motion data are collected and analyzed from Wearable Sensors Networks (WSNs) [7], which are defined by a set of interconnected sensors dedicated to collecting physiological parameters in the patient’s body. This technology can be used, for example, to diagnose patients through telemedicine applications [6] to analytically detect the compensatory actions of patients in rehabilitation [8], and others [9]. However, computational tools devoted to the visual analysis of a large amount of motion sensor data is quite scarce, especially in the context of assessing the motor function of stroke patients.

In this work, we propose a visual analytic tool to automate the assessment of motor impairment and motor compensation in stroke. Called NE-Motion (Network Environment for Motion Capture Data Analysis), the proposed tool provides resources to visually explore the sensor data and provide greater details on the motion of stroke patients. For example, the proposed tool enables an understanding of the UE segments used by a patient when performing a group of movements and the segments that are more commonly used by stroke patients when performing a particular movement. NE-Motion also enables the comparison of stroke patient and healthy individuals, making it possible to visually identify patients that perform more similarly to healthy individuals. NE-Motion makes use of a graph learning mechanism to transform sensor data into a set of graphs associated to each patient or healthy individual. The graph based representation enables the construction of an elegant and solid mathematical framework to filter out the set of graphs and extract gist information to support the analytical tasks.

In summary, the main contribution of this work are the following:

• A methodology to represent motion sensor data as a set of graphs;
• A mathematical framework to filter the set of graphs so as to extract gist information to support the analytical tasks;
• NE-Motion, a visual analytic tool to assist stroke rehabilitation specialists in identifying impairment patterns and to compare individuals;
• Case studies showing the effectiveness of NE-Motion in answering important questions related to stroke patients.

2. Related Work

The literature about visualization methods for sensor data analysis is quite comprehensive. In this section, we focus on visual analytic methods tailored to handle physiological motion data and techniques devoted to the visual analysis and comparison of temporal data. A more comprehensive discussion about fundamental concepts and analytical techniques to deal with WSNs can be found in Khan and Pathan [9] and Mosenia et al. [10] surveys.

2.1. Visual Analysis of Motion Capture Data

Motion capture data record body movement over time, typically represented as high-dimensional time-varying information [11]. Application fields where motion capture data play a major role are medicine, sports, games, and animation. In all those fields, the large amount of data involved in the analysis are mostly handled with the aid of visualization systems [12].

Good examples of visualization tools devoted to handling motion capture data include the work by Krekel et al. [13], where a visual widget displays a representation of the upper section of a three-dimensional human skeleton jointly with the angles between limb joints, and Motion Browser [14], which synchronize a patient’s video performing the movement with the visualization of patient’s muscle bundles that have the greatest activation during the movement. Dedicated to visualizing lower extremity motions, the visualization system proposed by Nguyen et al. [15] helps medical professionals to examine patients with patellofemoral instability in the knee, using a radial presentation. Wagner et al. [16] propose KAVAGait, a tool to represent modules of physical forces that act on the human foot during a walk. Motion data are not restricted to human beings. FuryExplorer [17], for example, is a visual analytic tool focused on the analysis of horse motion data, relying on glyph-based silhouettes of horses to describe the position of animals.

Gestures comprise another category of motion data. MotionExplorer [18] groups similar gestures and represents them as evolutions of a stick silhouette enclosed in a circle. GestureAnalyser [19] performs hierarchical and semi-supervised clustering of motions, animating the execution of the movement of each individual. The methodology in Motionflow [20] incorporates a oriented graph to indicate the evolution of a movement. Aiming to detect key positions in the course of movements, Bernard et al. [12] developed a visualization system based on clusters of similar positions derived from classifications.

Table 1. Summary of the main characteristics of existing visualization assisted motion capture data analysis. From left to right, the first column contains a reference to the technique, the second column describes the type of motion data handled by each method, the third column highlights the mathematical and computational model each technique relies on, and the right most column shows the visual metaphors employed in the visualization. Notice that NE-Motion differs from other approaches in all aspects: type of data, modeling and visual metaphor.

Work	Motion Data Type	Modeling Approach	Visual Metaphor
Bernard et al. [12] tool	Common human body movements	Supervised and unsupervised learning	Stick-man glyph and labeled horizontal bars
Krekel et al. [13] tool	Movements performed by patients with fractured proximal humerus	Spline and dimensionality reduction	3-D Human pose view, parallel coordinates, and scatter plot
MotionBrowser [14]	Movements performed by obstetrical brachial plexus patients	Entropy, Kullback–Leibler divergence, and K-means clustering	Curve-charts, histogram and video-replay
Nguyen et al. [15] tool	Patellofemoral joint motion on the knee of operated patients	GPU-based feature identification and tracking technique using SIFT	Radial plot and static 3D sweep plot with magic mirror strategy
KAVAGait  [16]	Walk performed by gait abnormalities patients	Fisher’s statistical indicators and filtering strategies	Curve-charts, twin box plots and range sliders
FuryExplorer  [17]	Trotting motion of lame horses	Dimensionality reduction and K–D tree clustering	Horse glyphs, projections and multi-curve charts
MotionExplorer  [18]	Common human body movements	Divisive hierarchical clustering algorithm	Stick-man glyphs, and tree and directed graphs
GestureAnalyser [19]	Gestures performed with the UE	Interactive agglomerative hierarchical clustering and dynamic time warping	Colored tree graphs and animated UE glyphs
Motionflow  [20]	Gestures performed with the UE	Interactive partition-based clustering	Colored tree graphs, treemap and animated UE glyphs
NE-Motion (proposed)	Movements performed by stroke patients using UE	Graph learning, dimensionality reduction and filtering strategies	Arc Visualization, projections, and curve-charts

summarizes the main properties of each technique discussed above. Notice that NE-Motion is the only one to handle UE motion data of stroke patients, being also the only one to rely on graph learning to model the problem. In addition, our tool uses visual metaphors dedicated to the visualization of relationships (arc visualization), individual properties (projections), and time series (curve-chart). In contrast to the works described above, we propose a new methodology to represent the “synchronization” between data from the sensors during the execution of specific movements.

2.2. Visualization and Comparison of Temporal Data

Motion capture data are a special type of time-varying data [21]. A multitude of visualization tools have been developed to explore time-varying data, ranging from pure time series analysis to multifaceted and graph-based pattern analysis.

In the context of pure time series analysis, there are visualization tools to analyze the growth and decrease of time series [22], methods to explore features in different scales [23], techniques devoted to identify periodicity and anomalies [24], interactive schemes to visually identify patterns [25], among many others. Regarding multifaceted time-varying data, works range from focus + context [26] weather analysis [27,28,29] to multiplayer game analysis [30]. In the context of time-varying data defined on graphs, visualization tools to analyze mobility [31], dynamic graphs [32], and crime data [33] have been proposed. The methods above are only examples of visualization methods for time-varying data and a more comprehensive overview can be found in a number of surveys [21,34,35]

Our approach differs from the methods discussed above in two main aspects: NE-Motion focuses on the synchronization of time series while enabling the visual identification of time series that “synchronize” more often.

3. Requirements and Tasks

For almost two years, we interacted with two experts, with an MD and PhD degree, in stroke motion assessment and rehabilitation (AP, HS). From that interaction, we came up with a set of requirements that should be addressed by the analytical tool, which we summarize as follows:

• R1—relation between joint angles: An important issue when analyzing motion sensor data is to figure out which pairs of joint angles have significant relationships in a given patient and which ones can be disregarded. Such analysis must order the data according to their location in the body.
• R2—prevalent joint angle relationships: Because of impairment, stroke patients may not move segments of their body that typically move together in healthy individuals. Stroke patients may also use other body segments to circumvent limited motion in impaired body segments. For instance, if a patient cannot flex his shoulder and extend his elbow, he may flex his trunk in order to move his hand forward in space. This means that certain segments are not moving together that normally would in a healthy individual (impairment), while other segments are moving together that normally would not (compensation). Thus, the pairs of sensors that most strongly correlate in stroke patients versus healthy individuals are important to identify.
• R3—focused analysis: In order to perform certain examinations, experts have to focus on a particular movement, particular individual, and on one side of the body. It is important to be able to easily switch the movement, individual, or body side during the analysis of motions.
• R4—specific patterns: Identifying patterns that occur only in stroke patients or in healthy individuals is also of great relevance. The identification of individuals with unique behavior (outliers) is also a requirement. Those patterns areuite difficult to get from simple correlation analysis, demanding mechanisms tailored to this end.
• R5—contrast individuals: A difficulty faced by the experts is the comparison of individuals when an ordinal scale, such as the FMA, is used. Comparing stroke patients according to their impairment level or comparing them against healthy individuals is important to assess the evolution of a rehabilitation process.

We used these requirements to design a set of visualization and analytical tasks for NE-Motion. The mapping among tasks and requirements is indicated in parenthesis:

• Task 1. Overview: The system must calculate and display the relationships between joint angle data from the sensors, discarding relations that are not of interest. Moreover, the joint angles must be presented in a predefined order according to their location on the body (R1).
• Task 2. Filtering and selection: The tool should be able to focus the analysis on particular movements, particular individuals, and side of the body (R3). These selection and filtering should be promptly available to the users.
• Task 3. Revealing group-level patterns: The tool should be able to detect patterns occurring in particular groups of individuals, for example, highlighting relationships between joint angles that are only observed in stroke patients or healthy individuals (R4).
• Task 4. Sorting: Sorting routines should be implemented to enable the visualization of the most frequent relationships in groups of individuals (R2). This feature will characterize impairment and compensation during motor performance in stroke patients.
• Task 5. Detailing sensor relationships: Users must be able to visualize detailed joint angle information from the sensors in order to identify trends and extreme values (outliers) (R3 and R4).
• Task 6. Comparing individuals: The analytical tool should provide mechanisms to compare individuals based on their motion data. This comparison will enable users to identify the individuals that differ most from others. It will identify the data patterns of stroke patients that differ the most from or closely matches healthy individuals and other stroke patients (R5).
• Task 7. Time series presentation: The tool should allow users to visualize and compare the motion data of a particular participant against an average times series obtained by combining the data from all participants. This feature will enable users to identify typical and spurious behaviors in an individual’s data (R4 and R5).
• Task 8. Global and local analysis: The system must enable global (group of individuals) as well as local (at the individual level) analysis. Global analysis involves comparing the ’average’ behavior of groups of individuals. In contrast, local analysis enables exploring the performance of a particular individual in a given movement and comparing it against a group of individuals (R3, R4 and R5).

Before providing a match between the tasks above and the design decisions for NE-Motion, we describe the data set provided by our expert partners and the mathematical/computational framework that supports NE-Motion.

4. Data Set and Notation

To record the upper extremity (UE) motion, we attached nine inertial measurement unit (IMUs; Noraxon Inc., Scottsdale, AZ) to the upper body (both hands, forearms, arms, spine, and pelvis). Each IMU samples linear acceleration, angular velocity, and magnetic heading at 100 Hz, and the software additionally generates quaternions for each IMU and upper body joint angle values. For the visualization analysis, we used $N_{\mathcal{A}}=20$ relevant joint angles at upper body joints (wrist, elbow, shoulder, thoracic spine, and lumbar spine). We used the time series corresponding to the 20 joint angles, described in details in

Table 2. List of anatomical angles. The system uses a rigid-body skeletal model to convert the IMU measurements into joint and segment angles. Shoulder total flexion is a combination of shoulder flexion/extension and shoulder ad-/abduction. Thoracic angles are computed between the cervical vertebra and the thoracic vertebra. Lumbar angles are computed between the thoracic vertebra and pelvis.

Joint/Segment	Anatomical Angle
Shoulder	Shoulder flexion/extension Shoulder internal/external rotation Shoulder ad-/abduction Shoulder total flexion
Elbow	Elbow flexion/extension
Wrist	Wrist flexion/extension Forearm pronation/supination Wrist radial/ulnar deviation
Thorax	Thoracic flexion/extension Thoracic axial rotation Thoracic lateral flexion/extension
Lumbar	Lumbar flexion/extension Lumbar axial rotation Lumbar lateral flexion/extension

, for each FMA movement.

UE motion was also recorded synchronously with two cameras (60 FPS, $1088\times 704$; Ninox, Noraxon) positioned orthogonally less than 2 m from the subject. Trained coders used the video data to label the FMA items, which simultaneously labeled the IMU data.

To directly examine the relationship between FMA scores and the output of NE-Motion, each individual performed the FMA while we recorded their motion. We examined $N_{\mathcal{M}}=13$ movements of the FMA that entailed arm and wrist motions; the hand items were not examined, given that the IMUs (placed on the back of the hand) did not read out information about the fingers. For data collection, patients are instructed to perform one movement at a time. Specifically, the movements are very simple tasks, such as simulating answering a telephone. There are no repetitions, that is, each individual performs the movement only once. In this case, stroke patients performed the movements on their paretic (affected) side, while healthy individuals performed the movements on both sides of the body. A trained assessor appraised each movement and assigned a score of 0 (for no or incorrect movement), 1 (for partial movement), or 2 (for “normal” movement).

Table 3. Demographic characteristics of patients in the data set. The range (between parenthesis) and average values are shown for the age, years since stroke, and FMA score.

Demographic Property	Stroke	Control
Number of patients	51	18
Age (years)	57.76 (21.26–84.26)	61.55 (41.99–82.96)
Male	23	10
Female	28	8
Ischemic Stroke	42	-
Hemorrhagic Stroke	9	-
Years since stroke	5.34 (0.26–38.44)	-
Paretic side	28 left and 23 right	-
FMA score (points)	43.11 (8–65)	65.475 (62–66)

summarizes the demographic information of the patients in the database. It is noteworthy that this study was carried out in accordance with the recommendations of the Declaration of Helsinki [36,37]. The protocol was approved by the NYU Institutional Review Board.

The data set comprises $N_{\mathcal{P}}^{{\mathrm{stroke}}_L}=28$ of left-paretic stroke patients (impaired on the left side of the body), $N_{\mathcal{P}}^{{\mathrm{stroke}}_R}=23$ of right-paretic stroke patients, and $N_{\mathcal{P}}^{{\mathrm{CRTL}}_R}=N_{\mathcal{P}}^{{\mathrm{CRTL}}_L}=18$ of healthy individuals, totaling $N_{\mathcal{P}}=N_{\mathcal{P}}^{\mathrm{stroke}}+N_{\mathcal{P}}^{\mathrm{CRTL}}=N_{\mathcal{P}}^{{\mathrm{stroke}}_R}+N_{\mathcal{P}}^{{\mathrm{stroke}}_L}+N_{\mathcal{P}}^{{\mathrm{CRTL}}_R}+N_{\mathcal{P}}^{{\mathrm{CRTL}}_L}=87$ individuals. We use $N_{\mathcal{A}}=20$ joint angles. Therefore, the data set contains $N_{\mathcal{T}}$ time series, where $N_{\mathcal{T}}$ is given by the following:

$$N_{\mathcal{T}}=N_{\mathcal{P}}·N_{\mathcal{M}}·N_{\mathcal{A}}=87·13·20=22620.$$

(1)

To settle the notation, consider the following sets:

• Joint Angles: $\mathcal{A}:=\left\{a_1,a_2,...,a_{N_{\mathcal{A}}}\right\};$
• Movements: $\mathcal{M}:=\left\{M_1,M_2,...,M_{N_{\mathcal{M}}}\right\};$
• Participants: $\mathcal{P}:=\underset{\mathcal{P}\mathrm{stroke}}{\underbrace{\{p_1,p_2,...,p_{N_{\mathcal{P}}^{\mathrm{stroke}}}}},\underset{\mathcal{P}\mathrm{CTRL}}{\underbrace{p_{N_{\mathcal{P}}^{\mathrm{stroke}}+1},...,p_{N_{\mathcal{P}}}\}}}$,

where ${\mathcal{P}}^{\mathrm{stroke}}$ is the set of stroke patients and ${\mathcal{P}}^{\mathrm{CTRL}}$ is the set of healthy controls. We divide the sets according to the side of the body that has been recorded, that is, the following:

○

${\mathcal{P}}^{\mathrm{stroke}}:={\mathcal{P}}^{{\mathrm{stroke}}_L}\cup {\mathcal{P}}^{{\mathrm{stroke}}_R}$,

○

${\mathcal{P}}^{\mathrm{CTRL}}:={\mathcal{P}}^{{\mathrm{CTRL}}_L}\cup {\mathcal{P}}^{{\mathrm{CTRL}}_L}$;

The set of time series $\mathcal{T}$ is denoted by:

• Time series: ${\displaystyle \mathcal{T}=\bigcup _{p_i\in \mathcal{P},M_j\in \mathcal{M}}{\mathcal{T}}_{i,j}}$,

where ${\mathcal{T}}_{i,j}:=\left\{t_{i,j,1},t_{i,j,2},...,t_{i,j,N_{\mathcal{A}}}\right\}$ accounts for the set of time series of the individual $p_i$ performing the movement $M_j$.

A main issue to understanding how each individual or groups of individuals behave is to figure out which pairs of joint angles “correlate” (synchronize) during the movements accomplished by the individuals, as such correlations point out which parts of the body are concurrently moving during the movement. In the next section, we present the mathematical foundation of the methodology we developed to tackle the analysis of the movements and individuals.

As each individual may take a different amount of time to perform the same movement, the number of time steps of each time series in $\mathcal{T}$ may be different. Thus, it is necessary to transfer all time series of $\mathcal{T}$ to a space of the same dimension. In this case, we perform a coordinate up-sampling procedure using spline polynomials [38], so that we transfer all time series of $\mathcal{T}$ to the space of dimension $m_{max}$, with $m_{max}$ being of length 1500 since $98.8\%$ of the data has a size equal or lower than this.

Therefore, we will consider all time series as having the same number of time steps.

5. Mathematical Foundation

The proposed methodology relies on Graph Learning (GL) theory [39] to identify and represent the relation between pairs of joint angles. An undirected network, or a graph, $G(V,E)$ is given by a set of vertices $V=\{v_1,v_2,...,v_n\}$ and a set of edges $E\subset V\times V$, where $(v_i,v_j)\in E$ if the vertices $v_i$ and $v_j$ are linked in G. The graph G can be represented by an adjacency matrix $W\in {\mathbb{R}}^{n\times n}$, where an entry $W_{i,j}$ in W is non-zero if and only if $(v_i,v_j)\in E$. Since G is undirected, the matrix W is symmetric.

The GL problem consists of defining the edge set E from a set of vertices V and multivariate data X associated to each vertex $v_i\in V$. In other words, in GL, the set of edges E, and thus W, are unknown and must be computed from V and X. Specifically, the multivariate data X can be represented as a matrix $X={\left[x_1,x_2,...,x_n\right]}^T\in {\mathbb{R}}^{n\times m}$, where the row $x_i\in {\mathbb{R}}^m$ corresponds to a vector associated with the vertex $v_i\in V$ of G.

To compute the set of edges E, GL techniques assume certain premises, being that “smoothness” is one of the most used hypotheses. Intuitively, smoothness seeks to ensure that a pair of vertices $v_i$ and $v_j$ are connected by an edge $(v_i,v_j)\in E$ if the vectors $x_i$ and $x_j$ associated to those vertices are similar to each other, that is, $∥x_i-x_j∥$ must be close to zero. In other words, the edge set E should connect the most similar vertices. In mathematical terms, the resulting edge set must give rise to an adjacency matrix W that minimizes the Dirichlet energy [40].

$$D_W\left(X\right):=\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^n{W}_{i,j}{∥x_i-x_j∥}^2.$$

(2)

Equation (2) tells that $W_{i,j}$ must be zero when $∥x_i-x_j∥$ is large. Moreover, large values of $W_{i,j}$ correspond to edges connecting quite similar vertices ($∥x_i-x_j∥$ close to zero).

To compute an adjacency matrix W (and thus the edge set E) that minimizes Equation (2) while ensuring that W is symmetric, non-negative, and sparse, we rely on the optimization procedure proposed by Kalofolias [41], which consists of solving the following mathematical problem:

$$\begin{array}{cccc}\hfill W^{*}& :=& argmin\hfill & D_W\left(X\right)+g\left(W\right).\\ \hfill & & W\in \mathcal{W}\hfill \end{array}$$

(3)

where $\mathcal{W}=\{W\in {\mathbb{R}}^{n\times n}|W=W^T,W_{i,i}=0,\mathrm{and}{W}_{i,j}\ge 0,\forall i,j\}$. The right most regularization term $g\left(W\right)$ aims to ensure sparsity and connectness and it is defined as follows:

$$g\left(W\right)=-\alpha \left({\mathbf{1}}^T·log\left(W·\mathbf{1}\right)\right)+\gamma ·{∥W∥}_{\mathrm{F}}^2,$$

(4)

where $\mathbf{1}\in {\mathbb{R}}^n$ is a vector with all coordinates equal to 1, $\alpha >0$ and $\gamma \ge 0$ are parameters set as proposed by Kalofolias and Perraudin [42]. The logarithmic term is applied in each entry of the vector $W·\mathbf{1}$ and it enforces each vertex to be connected to other vertices of the graph since an isolated vertex result in row with zeros in W. The second term ${∥W∥}_{\mathrm{F}}^2$ avoids the trivial solution while enforcing sparsity.

Kalofolias [41] demonstrates in his work that the framework derived from Equations (3) and (4) can be modeled to be solved by the primal–dual scalar optimization procedure [43] that is scalable while presenting fast convergence, making this methodology doable even for large scale problems. Specifically, as $D_W+g$ is a lower semi-continuous convex function [44], the optimization problem discussed here is guaranteed to converge to the minimum in a very efficient way.

In the following section, we show how we use GL to build networks whose edges represent the relation between pairs of joint angles.

6. Network Construction

We rely on the GL methodology discussed in Section 5 to build a network for each movement. Therefore, each individual $p_k\in \mathcal{P}$ has a set of associated networks, one for each movement $M_l\in \mathcal{M}$. The vertices of each network correspond to the joint angles, so all networks have the same number of vertices. Each vertex (joint angle) has a time series associated to it and the time series play the role of the multivariate data $x_i$ discussed in Section 5. Given the vertices and the time series associated with them, we compute the edges of the network associated with a movement by solving the GL problem (3). Since the GL results in the edges connecting similar nodes, the edges naturally indicate the pairs of joint angles that most correlate during a movement. Figure 1 illustrates the process we use to compute the set of networks associated with each individual.

As pointed out in requirement R1, some correlations between joint angles are of no interest, so they must not be represented as edges in the network. Specifically, null time series and time series with nearly constant values are of no interest, as those time series are not recording a movement. To avoid edges connecting vertices corresponding to joint angles whose time series are null and nearly constant, we modify the similarity measured (see Equation (2)) to become the following:

$$D_W\left(X\right):=\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^n{W}_{i,j}{Z}_{i,j},$$

(5)

where $Z_{i,j}$ is given by

$$Z_{i,j}:=\alpha z_1(i,j)+\beta z_2(i,j)+\gamma z_3(i,j).$$

(6)

where $z_1(i,j)={∥t_{k,l,i}-t_{k,l,j}∥}^2$ and $t_{k,l,i}$ accounts for the time series associated to joint angle $a_i$ of the individual $p_k$ performing the movement $M_l$. The values $\alpha ,\beta ,$ and $\gamma$ are non-negative parameters that control the importance of each term in $Z_{i,j}$. In this work, we define $(\alpha ,\beta ,\gamma )=(1,1,1)$. The term $z_2(i,j)$ penalizes the relations between null time series and it is given by the following:

$$z_2(i,j):=\left\{\begin{array}{cc}2,\hfill & \mathrm{if} t_{k,l,i} \mathrm{or} t_{k,l,j}\mathrm{are} \mathrm{null},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

The term $z_3(i,j)$ in Equation (6) penalizes constant time series and is defined as follows:

$${\displaystyle z_3(i,j):=e^{-\mathrm{diff}\left(t_{k,l,i}\right)}+e^{-\mathrm{diff}\left(t_{k,l,j}\right)},}$$

(8)

where diff is a function that calculates the sum of the absolute value of the differences between consecutive time series values, that is, the following:

$${\displaystyle \mathrm{diff}\left(x\right):={\displaystyle \sum _{i=1}^{m-1}\left|x_i-x_{i+1}\right|},x\in {\mathbb{R}}^m.}$$

(9)

Algorithm 1 summarizes the steps for creating a graph $G_{k,l}$ associated with the individual $p_k\in \mathcal{P}$ and movement $M_l$. The algorithm takes as input the time series ${\mathcal{T}}_{k,l}$ representing the respective joint angles and it outputs the networks $G_{k,l}(S,E_{k,l})$, that is, the adjacency matrices $W_{k,l}={\left[{\left(W_{k,l}\right)}_{i,j}\right]}_{i,j=1}^{N_{\mathcal{A}}}$. In terms of computational times, the algorithm takes around a tenth of a second to calculate one network and, consequently, as we need to calculate $87·13=1131$ networks, the method takes around $113.1$ s to compute all the networks in a personal computer with 2.4 GHz Intel(R) Core i7 CPU and 16 GB of RAM.

Algorithm 1 Computing a network.

1: Input:${\mathcal{T}}_{k,l}=\{t_{k,l,1},t_{k,l,2},...,t_{k,l,N_{\mathcal{A}}}\}$: set of $N_{\mathcal{A}}$ time series representing the joint angles of individual $p_k$ when performing the movement $M_l$. 2: Output:$W_{k,l}$: adjacency matrix corresponding to the network $G_{k,l}(S,E_{k,l})$. 3: $\mathrm{Z}:=0$                    ▹$0\in {\mathbb{R}}^{N_{\mathcal{A}}\times N_{\mathcal{A}}}$ is the null matrix 4: for$i:=1$ to $N_{\mathcal{A}}$ do 5:     for $j:=i+1$ to $N_{\mathcal{A}}$ do 6:         ${\mathrm{Z}}_{i,j}:=z_1(i,j)+z_2(i,j)+z_3(i,j)$ 7:         ${\mathrm{Z}}_{i,j}:={\mathrm{Z}}_{j,i}$ 8:     end for 9: end for 10: Solve the optimization problem in the Equation (3), given the pairwise similarity measures $\mathrm{Z}={\left({\mathrm{Z}}_{i,j}\right)}_{i,j=1}^{N_{\mathcal{A}}}$, to obtain the adjacency matrix $W_{k,l}$.

6.1. Filtering Networks

In order to accomplish the tasks described in Section 3, the networks associated to an individual or group of individuals must be filtered out to get the information of interest. For instance, if the focus of the analysis is on movement $M_l$ performed by patients with stroke on the left side of the body, the filtering mechanism should return the set of networks associated with that movement for that group of individuals.

We rely on set theory [45] as the building block for the design of a family of filters that support our tasks. Let $G_U$ denote a graph-like structure where the nodes correspond to the joint angles, but the edges are the union of the edges from all networks, that is, the following:

$$G_U=G_U(\mathcal{A},E_U)=G_U\left(\mathcal{A},\bigcup _{p_k\in \mathcal{P},M_l\in \mathcal{M}}{E}_{k,l}\right).$$

(10)

where $\mathcal{A}$ is the vertex set corresponding to joint angles and $E_U$ comprises the set of all edges from all networks. Mathematically, $G_U$ is not a graph in the classical sense since it may have several edges connecting the same pairs of vertices; however, the structure of $G_U$ will serve our purposes. We denote by $\mathsf{\Lambda }$ a filter that extracts from $E_U$ the subset of edges that must be involved in a given analysis. For instance, suppose that the domain of interest is stroke patients affected on the left side of the body (${\mathcal{P}}^{{\mathrm{stroke}}_L}$) and movement $M_2$ ($l=2$). Thus, the filter $\mathsf{\Lambda }$ must be the following:

$$\mathsf{\Lambda }=\left\{\begin{array}{c}″\mathrm{stroke} \mathrm{individuals}\\ \mathrm{movement}{M}_2\\ \mathrm{left} \mathrm{side}″\end{array}\right\}.$$

(11)

Mathematically, the filter is implemented as follows:

$$E_{\mathsf{\Lambda }}=\left\{e\in \bigcup E_{k,l}\subset E_U|p_k\in {\mathcal{P}}^{{\mathrm{stroke}}_L},l=2\right\},$$

(12)

giving rise to

$$G_{\mathsf{\Lambda }}=G_{\mathsf{\Lambda }}\left(\mathcal{A},E_{\mathsf{\Lambda }}\right).$$

(13)

Filtering Edge Functions

Certain tasks demand that edges from $E_U$ are filtered based on the values of a function defined on $E_U$. Suppose, for example, that one wants to figure out which is the edge that is more prevalent in the networks of healthy individuals when performing the movement $M_8$ with the right side of the body. In this case, the filter $\mathsf{\Lambda }$ can be stated as follows:

$$\mathsf{\Lambda }=\left\{\begin{array}{c}″\mathrm{most} \mathrm{prevalent} \mathrm{edge}\\ \mathrm{healthy} \mathrm{individuals}\\ \mathrm{movement} M_8\\ \hfill \mathrm{right} \mathrm{side} \mathrm{of} \mathrm{the} \mathrm{body}″\end{array}\right\}.$$

(14)

To accomplish the filtering we define the function $f_{\mathrm{count}}:E_U\to \mathbb{R}$ as follows:

$$f_{\mathrm{count}}\left(e\right):=\#\left(\left\{E_{k,l}\subset E_U|\begin{array}{c}le\in E_{k,l},\hfill \\ p_k\in {\mathcal{P}}^{{\mathrm{CTRL}}_R},\hfill \\ l=8\hfill \end{array}\right\}\right),$$

(15)

where $\#(·)$ is the number of elements in a set.

The filtered subset of edges $E_{\mathsf{\Lambda }}$ is obtained as follows:

$$E_{\mathsf{\Lambda }}=\left\{e\in \bigcup E_{k,l}|\begin{array}{c}{p}_k\in {\mathcal{P}}^{{\mathrm{CTRL}}_R},\hfill \\ l=8,\hfill \\ f_{\mathrm{count}}\left(e\right)=\mathsf{\Gamma }\left(e\right)\hfill \end{array}\right\},$$

(16)

where $\mathsf{\Gamma }\left(e\right):=max\left\{f_{\mathrm{count}}\left(e\right)|e\in \bigcup E_{k,l}\right\}$.

The filtering scheme presented above provides a unified manner of filtering edges in order to accomplish local and global analysis involved in the tasks described in Section 3.

7. NE-Motion

NE-Motion is a visual analytic system designed to tackle the tasks outlined in Section 3. Each of the visual components described in the following were designed based on thorough discussions with the experts, where the pros and cons of each possible alternative were analyzed until consensus was reached. Several adjustments were made during the course of development to tackle issues and to improve the analytical capability of the proposed visualization tool.

Figure 2 shows the main components of NE-Motion, which are detailed in the following.

Filtering Menu. The filtering menu (component $a_1$ in Figure 2) enables users to interactively define a filter $\mathsf{\Lambda }$, thus addressing tasks $T_2$ and $T_4$ while supporting the accomplishment of task $T_3$. Users can select the following:

• A particular category of individuals: stroke, healthy, or both.
• Side of the body: left or right.
• Choose a particular individual.
• Sort by top K most prevalent relations.
• Edges that take place concurrently on both stroke patients and healthy control individuals or that appear in one category but are not present in the other.

The available filtering options enable many such different analyses as, for example, analyzing the similarities and differences between healthy and stroke affected individuals.

Pairwise Relationship View NE-Motion relies on an arc visualization metaphor [46] to reveal the relation of joint angles resulting from a filtering (edges in $E_{\mathsf{\Lambda }}$). The joint angles (Figure 2 $a_3$) are arranged according to a pre-established order determined by the specialists and pairs of joint angles containing edges in $E_{\mathsf{\Lambda }}$ are connected by an arc (Figure 2 $a_2$). Users can interactively select a particular arc (relation) to get details about the group of individuals where the selected relationship is present. This component addresses task $T_1$ and support task $T_5$.

Detailed Relation View This component, illustrated in Figure 2B, is responsible for highlighting details about the relationship of a pair of joint angles. The detailed relation view is triggered when the user selects a particular arc in the pairwise relations view and it shows the proportion of healthy control and stroke patients that present that relationship in each movement selected by the user (Figure 2 $b_1$). A scatter plot (Figure 2 $b_2$) derived from measurements computed from the time series associated the each joint angle linked by the selected arc is also provided. Specifically, each axis in the scatter plot corresponds to a joint angle; each point represents an individual (green is healthy and red is stroke), and the coordinates of the points are given by the measurement extracted from the time series of the individuals. Users can choose (Figure 2 $b_3$) three different measurements: mean value, entropy, and energy of the time series. In addition, the size of each circle is defined according to the FMA score of the represented individual in which the larger the circle, the lower the FMA score and, therefore, the more impaired the motion. The visualization widget shown in Figure 2 $b_4$ shows the average shape of the time series associated with the pair of joint angles linked by the selected arc. The green time series correspond to healthy individuals and the red ones to stroke patients, as requested by the domain experts. In this case, the average time series of stroke patients is mirrored on the y-axis, enabling the user to compare the kinematic profiles of the joint angle changes over time. In addition, for $b_4$, the tool provides the user with two viewing options: “Proportional” and “Same Length”. The “Proportional” view displays the average time steps of each group’s time series, which may be shorter in healthy subjects and longer in stroke patients. The “Same Length” view displays the time series of both groups in same number of time-steps, enabling the direct comparison of the kinematic profiles. If an individual is selected in the scatter plot by hovering the mouse on the point, the particular time series of that individual is also presented (black curve in the plot), making it possible to compare the individual against the group average. The detailed relation view addresses tasks $T_3,T_5,T_6$ and $T_7$ while supporting $T_8$.

Comparison View This analytical component provides resources to compare individuals in the whole data set. It comprises three main visual widgets. The first widget, illustrated in Figure 2 $c_1$, presents the average and standard deviation curves for the time series associated to each joint angle for both healthy (green) and stroke individuals (red). The time series are ordered in this component according to the joint angle arrangement of component $a_3$, that is, the box topper in $c_1$ refers to the leftmost joint angle in $a_3$. In this case, the time series from the whole set of movements can be analyzed or filtered based on a particular movement (one side of the body must be chosen). When a particular joint angle is selected, a detailed view of that joint angle shows up, as illustrated in Figure 2 $c_2$, allowing to visually compare, through a statistical curve based summary, the time series behavior of stroke patients and healthy control individuals in the chosen joint angle. Figure 3 shows the elements in the detailed view.

In order to provide an overview on how individuals compare to each other, the comparison view enables a multidimensional projection visual widget (Figure 2D). Three different projection schemes are provided (Figure 2 $d_1$): network principal component analysis (PCA), motion curves PCA, and auto-encoder multidimensional scaling (AE-MDS).

The network PCA maps to a 2D space a multidimensional feature vector derived from the set of networks of each individual. Specifically, for each network we extract the following measures: node degree ($f_{\mathrm{deg}}$) [47], closeness centrality ($f_{\mathrm{close}}$) [48]; eigencentrality ($f_{\mathrm{eig}}$) [49], and clustering coefficient ($f_{\mathrm{clus}}$) [50]. Therefore, for each movement, we have a feature vector in ${\mathbb{R}}^{4·N_{\mathcal{A}}}$. If all movements are considered simultaneously, the feature vector is $N_{\mathcal{M}}\left(4N_{\mathcal{A}}\right)$-dimensional.

Motion curves PCA simply consider each normalized time series as high-dimensional data and project the curves in a two-dimensional space using PCA.

AE-MDS projection uses MDS to project an embedding of each individual from a multidimensional space. The embedding is accomplished, using an autoencoder trained with all time series. To increase the amount of training data, we also included the time series in reversed order in the training data. The encoder architecture consists of six layers of 1D convolutions. The first convolutional layer transforms the input to four channels while maintaining its length. The next five layers halve the length and the amount of channels, reaching a dimension of 188, which we use as the embedded representation for each individual. The decoder has a similar architecture to the encoder, except that in the encoder, we use a Relu activation function, while in the decoder, no activation is used. PCA was chosen for projecting network structures and motion curves because it handles high-dimensional data quite well while being computationally efficient. Since the AE embeds the curves in a relatively low dimensional space, MDS can be used, providing an alternative visualization.

An individual can be selected in the projection layout (Figure 2 $d_2$) by hovering the mouse on the corresponding point (the ID of the individual shows up by hovering the mouse). Once selected, the time series associated with that individual are shown in the detailed view. The comparison view was designed to address task $T_6,T_7$ and $T_8$.

All NE-Motion visual components can be dragged over the screen for repositioning, facilitating the visualization and comparison of the displayed information.

Implementation

The network construction is done using PyGSP [51] in Python. Time series are stored in a database using MySQL and queried through PHP and Ajax. NE-Motion visual components are coded in JavaScript and D3.js. The autoencoder is built using TensorFlow.

8. Case Studies

In this section, we present two case studies carried out jointly with the experts that closely collaborated in the development of NE-Motion. The first case study focuses on the visual identification of patterns and comparison between individuals. The second case study shows the utility of NE-Motion to enable a detailed analysis of stroke patients so as to reveal the similarities in their motion.

8.1. Case Study I: Visual Identification of Patterns and Comparison between Individuals

In this case study, we focus on Comparison View and Pairwise Relations View NE-Motion components to visually identify patterns and compare individuals. Specifically, from the Filtering menu we select the movement “flexor synergy” and the left side of the body (Task $T_2$). Starting with their hand on their knee, the flexor synergy item of the FMA requires the individual to raise their hand toward the ear, bringing the elbow up to shoulder height with the thumb pointed toward the ceiling as if they are answering a telephone.

To perform the comparison between individuals, we trigger two NE-Motion visual components: the pairwise relations view and the comparison view as shown, respectively, in Figure 4-A and $C/D$. In this case, for the AE-MDS projection ($d_2$), the color of the circles refers to the subject group (red for stroke patients and green for healthy individuals) and the radii of the circles indicate the impairment level (circles with larger radii refer to higher impairment). The projection shows that there is a segregation between the groups with stroke patients (red circles) concentrated on the bottom right of the projection, and healthy individuals (green circles) concentrated on top of the projection layout (Task $T_6$). We also note that the projection corresponds well with the FMA scores: stroke patients with high FMA scores (i.e., less impairment) are projected close to the healthy cluster in the AE-MDS projection.

In contrast, stroke patients with lower FMA scores (larger diameter) are projected away from the healthy cluster. This indicates that our model, based on an AE neural network, used to embed the joint angle time series was able to properly group individuals according to their impairment level. Consequently, we can note that the projection is able to capture the differences in the movements performed by stroke patients and healthy control individuals. This analysis attests the effectiveness of NE-Motion mathematical and analytical design, making possible to represent individuals with similar level of impairment in an intuitive manner.

Directing our analysis on the individual “stroke_27” (score 8 out of 12 on flexor synergy item and a total score 47 out of 66 on the FMA), we can see in the Comparison View component in Figure 4 that “stroke_27” is projected near other stroke individuals with similar diameters (i.e., similar FMA scores). This trend is observed in both AE-MDS and PCA projection, which demonstrates that the motion data of each group of individuals have well-defined features within each category and that the patient in question has similar time series to other patients with a similar FMA score, as shown in Figure 5.

In addition, when analyzing the motion time series for “stroke_27” in the Comparison View component in Figure 4-C and, more in-depth, in the zoomed view depicted in Figure 6, it is evident that this subject’s joint angle motion curves (shown as black lines) are not smooth and, in some cases, are outside the statistical limits relative to the curves of the other individuals with similar condition (tasks $T_6$, $T_7$ and $T_8$).

Indeed, if we look at its joint angle network presented in the Pairwise Relations View component (task $T_1$), nodes in the thoracic and lumbar regions have a high degree of connectivity with various nodes of the upper body. This connectivity indicates that the truncal motion is associated with limb motion, as highlighted by the bottom dashed rectangles in Figure 4-$a_2$ and $a_3$.

Focusing on the trunk (curves highlighted with the dashed rectangle in Figure 4-c1 and broken out in Figure 6), we can see that for “stroke_27”, the motion of the thoracic spine is more pronounced and different from other individuals. This is an indication that this individual may be using compensatory motions during completion of the “flexor synergy” FMA item.

8.2. Case Study II: Understanding Patterns of Stroke Patients

This second case study focused on exploring the features belonging only to stroke individuals. To this end, from the Filtering menu we selected the FMA item “Shoulder Flexion ${90}^{\circ }$ to ${180}^{\circ }$” on the left side of the body. In this movement, the subject starts with their arm held out in front of them (shoulder flexed to ${90}^{\circ }$ and elbow extended to $0^{\circ }$) and lifts their arm up (shoulder flexed to ${180}^{\circ }$ and elbow extended to $0^{\circ }$). Subjects were instructed to keep their elbow straight at all times during the movement.

To explore the differences between stroke patients and healthy control groups, we filtered out the edges that simultaneously appeared in networks associated with both groups. This filtering can be performed by setting the option “Differences” in the Filtering menu. The filtering operation resulted in a motion graph with only two connections concerning stroke individuals and none for healthy individuals (task $T_3$), as we can see in the NE-Motion Pairwise Relations View component presented in Figure 7. Inother words, NE-Motion shows that only two joint angle synchronizations, unique to stroke patients, occur during the execution of the movement “Shoulder Flexion ${90}^{\circ }$ to ${180}^{\circ }$” with the left side. In this case, we can see that these synchronizations take place between the joint angles “Shoulder Abduction LT, deg” and “Shoulder Flexion LT, deg” and “Shoulder Abduction LT, deg” and “Elbow Flexion LT, deg”.

Activating the Detailed Relation View (Figure 8), we can see that elbow flexion and shoulder abduction synchronization co-occurs on this FMA item in seven stroke individuals (tasks $T_4$ and $T_5$). This indicates the intrusion of an abnormal flexion synergy when the arm is flexed on seven stroke patients ($b_1$ and $b_2$). As an example, we can analyze the details of this relationship (task $T_8$) in the motion curves of individual “stroke_4” (score 0 out off 2 on “Shoulder Flexion ${90}^{\circ }$ to ${180}^{\circ }$” and a total score 27 out of 66 on the FMA) in this component. Specifically, in Figure 8 $b_4$, the elbow flexion and shoulder abduction angles are initially stable when the individual lifts the arm, but then there is an increase in the two angles, considering their respective scales, as the arm is fully lifted above the head (task $T_7$). This synchronous increase in elbow flexion and shoulder abduction suggests the manifestation of intrusive flexion synergies that can be seen in stroke patients lifting the arm weight against gravity. Moreover, we can see in the scatter plot ($b_2$) the distribution according to the mean value (selected in $b_3$) of the time series for the selected edge. Notably, as abduction increases, elbow flexion also increases. This increased elbow flexion (despite instructions to keep the elbow straight) implies the intrusion of a flexion synergy.

8.3. Experts Comments

Reduced impairment, as evidenced by the re-emergence of normal movement patterns including the absence of intrusive synergies, is the signpost of true motor recovery. Impairment limits activity performance, whereas compensation helps achieve it. Although compensatory motions support functionality in the short-term, they blunt beneficial training effects and increase aberrant plasticity in lesioned animals [52,53]. In humans, preventing compensatory motions during training promotes greater reduction in UE impairment [54]. If the presence and degree of impairment and compensation can be identified in a moving patient, treatment strategies can be adjusted to maximally target impairment and minimize compensation.

With knowledge about impairment and compensation occurring in functional movement, one can generate rules for changing a treatment strategy in real time. For example, if compensatory motions start to escalate, a therapist can downgrade task difficulty to lessen compensation and focus on impairment reduction. The future of health care delivery is also unclear: access to inpatient rehabilitation services is decreasing nationally, with a push toward rehabilitation in the home. Given the growing interest in remote monitoring and training by telerehabilitation, the automated capture and quantitation of movement abnormalities could assist remote rehabilitation training.

The visualization tool, though preliminary, may help identify both impairment and compensation in quasi real-time. This visualization could guide the targeting of impairment and compensation in certain limb segments, facilitating the delivery of personalized rehabilitation interventions.

9. Discussion and Limitations

NE-Motion was developed to address a particular application; therefore, it bears some limitations. For instance, NE-Motion might not be appropriated to applications involving a large number of vertices, and some visualization components are not promptly scalable. In fact, the adaptation of NE-Motion to applications involving data sets with hundreds of data streams would demand the redesign of important visualization components, such as the Pairwise Relations View.

We believe, though, that the graph learning network representation proposed in this work can serve other scenarios. Applications such as rehabilitation of sports injuries and athletic performance analysis are two examples, as those applications involve the use of sensors to assess and compare individuals and groups of individuals.

As future work, we intend to extend NE-Motion to applications involving multivariate time series databases, as, for example, electroencephalography (EEG).

10. Conclusions

In this work, we introduced a visual analytics tool to assist in the rehabilitation process of stroke patients. NE-Motion was developed in close collaboration with domain experts, who helped us to translated their analytical needs into the visualization system. We also proposed a graph learning methodology to represent the movements by the joint angles as a set of networks. A solid mathematical framework to filter the set of networks was also proposed. NE-Motion was validated in two case studies using real data and with feedback from the domain experts, which attested to its effectiveness in addressing the analytical tasks.