Smartphone-Based Video Analysis for Guiding Shoulder Therapeutic Exercises: Concurrent Validity for Movement Quality Control

Abstract

Neuromuscular re-education through therapeutic exercise has a determinant role in chronic shoulder pain rehabilitation. Smartphones are an interesting strategy to extend the rehabilitation to a home-based scenario as it can increase the attraction and involvement of users by providing feedback. Objective: To analyze the concurrent validity of a smartphone’s application based on 2D video analysis against the gold-standard 3D optoelectronic system for assessing movement quality during upper limb therapeutic exercises. Methods: Fifteen young adults were evaluated while executing two different shoulder exercises with a smartphone’s 2D video and a 3D optoelectronic system simultaneously in two conditions: (1) with the supervision and instructions of a physiotherapist (guided exercise), and (2) without the feedback of the physiotherapist (non-guided exercise). The data obtained during the guided and non-guided exercises were compared to calculate the movement quality index based on the approximation of the non-guided exercise to the guided exercise for the head, trunk, and shoulder’s range of movement. The agreement of the movement quality index assessed with the smartphone application and the optoelectronic system was carried out through Bland–Altman analysis. Results: The Bland–Altman analysis indicates the range of agreement and bias tendency. This tendency demonstrates that the percentage of difference between the two methods increases as the movement quality index decreases. Conclusions: There is agreement between the movement quality evaluated by a gold-standard method and the developed application, although the proposed method appears to have less sensitivity for evaluating movements with lower quality index.

1. Introduction

Chronic shoulder pain is the third most common pain condition presented in primary health care [1,2,3]. Exercise-based physiotherapy is the first line of approach [3,4,5]. In the clinical context, kinematic re-education in ensured through the supervision of qualified physiotherapists in promoting adequate muscle recruitment and movement patterns during the therapeutic exercise [6]. Continuous monitoring by a specialized professional is essential for greater health gains, but this monitorization may be limited when the rehabilitation process is extended to home. Recent studies have highlighted both the advantages and challenges associated with home-based rehabilitation [4,7,8]. Home-based rehabilitation offers logistical benefits, optimizing resources, reducing travel time, and providing flexibility in appointment scheduling [9]. This rehabilitation modality has shown potential in enhancing users’ performance in daily activities, improving functional capacity, and enhancing overall quality of life. The home environment, rich in context-dependent learning opportunities and the use of familiar objects, increases the likelihood of transferring acquired skills to daily living activities [10]. However, the effectiveness of home-based rehabilitation may be influenced by several factors. The absence of direct clinical oversight, despite efforts to include continuous monitoring, pain education, and feedback in the home setting, could impact the overall success of the intervention. The adherence to prescribed home-based exercise programs has been posed as one of the main challenges in home-based rehabilitation. The non-monitorization of critical parameters such as frequency, intensity, sets, repetitions, rest time, and exercise speed can contribute to this fact [4,11,12,13]. Therefore, further research to optimize home-based implementation is required.

To address these limitations, among the various proposed solutions, one that shows high potential is the use of smartphones. Smartphones could be an interesting strategy as they can increase the attraction and involvement of users in the rehabilitation process. Given that it is a device that is easily used in daily life and available anywhere, it allows the bridging of gaps related to time, space, and costs and, more importantly, allows remote monitoring by the physiotherapist [4,10,14,15,16]. Several studies pointed to the use of smartphones to support rehabilitation as being very promising [17] since it would increase the effectiveness of exercise-based physiotherapy interventions in this setting and, thus, encourage health gains [4].

Two-dimensional (2D) systems that are incorporated in smartphone’s cameras are simple to use, easily accessible, and affordable. However, their performance when compared to other systems, such as three-dimensional (3D) systems, is not yet well established [18]. Motion capture systems, widely used to quantify human movement, and 3D motion capture systems are considered the gold standard in human movement analysis in terms of accuracy and reliability [18]. However, 3D analysis methods are expensive and cannot be used in in home-based settings. On the other hand, the overall performance of 2D motion capture systems is not yet well established, which could justify the lack of broad utilization of these systems in both research and clinical contexts [18].

Given the aforementioned, there is a clear need to provide satisfactory evidence for the validity and reliability of smartphones’ 2D video system as a tool to guide movement rehabilitation through movement quality control. Considering its portability, real-time data responsiveness, and the standardization of its usage, this smartphone application featuring a 2D camera system has the potential to offer benefits to the scientific community, patients, and rehabilitation professionals, ultimately enhancing treatment adherence and overall quality of life. Thus, the present study aims to validate a smartphone application which, through video recording monitors, supervises the execution of shoulder therapeutic exercises and gives the user feedback regarding the movement quality.

2. Materials and Methods

2.1. Study Design

This is a validation study, classified as observational, cross-sectional, and analytical.

2.2. Sample

The target population of this study included Center for Rehabilitation Research’s employees aged between 18 and 35 years [19]. Musculoskeletal and neurological conditions that influence exercise performance, history of persistent pain associated with the shoulder complex, and extreme obesity (BMI greater than 40 kg/m²) were exclusion criteria. Only young adults who consented to be contacted were assessed.

The final sample consisted of 15 participants. Ethical approval was obtained by the institutional Ethics Committee (CE0108C). All participants provided their written informed consent before the data collection began, according to the Declaration of Helsinki.

2.3. Instruments

2.3.1. Sample Selection and Characterization

A questionnaire via Google Forms was used to collect data to characterize the population and the criteria required for participation in the study. This questionnaire included topics related to demographic data (age, height, weight, dominant limb), general health (infectious, systemic, neurological, and/or musculoskeletal diseases) and shoulder pain (presence and frequency of shoulder pain). In addition, it featured a section for the participant to give their consent to be contacted to carry out the physical assessment and the study protocol.

The measuring tape of COMED^® (COMED SAS, Strasbourg, France) has inelastic and flexible characteristics. It was used to measure the height (m) of the participants, being 200 cm in length and bearing graduation every 1 mm [20].

The TANITA scale, model BC-543 Inner Scan TM (Monitoring Your Health, Amsterdam, The Netherlands), was used to assess total body mass (kg) and body mass index (BMI) [21]. Its dimensions are 30 × 30 × 3 cm³, accounting for a mass of 2.52 kg. It has a maximum capacity of 150 kg and an accuracy of 0.1 kg per kg.

The International Physical Activity Questionnaire (IPAQ) was used to characterize the level of physical activity of the participants. This version was validated for the Portuguese population together with the coordinating group in Portugal, Mota, and Sardinha [22]. The questionnaire features a value referred to as the criterion validity considering the accelerometer data of r = 0.49 and a Cronbach α of 0.96 [22].

To assess the participants’ potential interest in using the smartphone app developed, two research questions were made: “Would you find this application useful?” and “Would you see yourself using this application?”. The participants were asked to answer “Yes” or “No”.

2.3.2. Kinematic Data

The joint position of the shoulder, forearm, wrist, head, and trunk segments were assessed using an optoelectronic system, the Qualisys Motion Capture System (Qualisys AB, Göteborg, Sweden) [23], for concurrent validation. The spatial position of the reflector markers, placed on the participant, were collected using twelve infrared cameras, eight Oqus 500 and four Miqus M3, connected to the Qualisys USB Analog Acquisition interface, at a sampling frequency of 100 Hz. Qualisys Track Manager software version 2021.2 (Qualisys AB, Göteborg, Sweden) [23] was used to display, acquire, and analyze kinematic data.

The smartphone-based video analysis was conducted by a smartphone application, designed for Android devices, that leverages the power of machine learning to facilitate shoulder rehabilitation exercises. The app uses the device’s 2D camera to capture video footage of patients performing their exercises. This footage is then processed in real-time using the MediaPipe library [24], a state-of-the-art machine learning solution for computer vision tasks, which recommends a minimum resolution of 640 × 480 pixels and 30 frames per second. Employing pose detection, a computer vision technique that identifies and tracks human body parts, the app analyzes users’ movements during shoulder rehabilitation exercises. Despite the 2D nature of the camera, the app extracts valuable information such as shoulder position, arm angles, movement patterns, and gesture recognition from the detected poses. By analyzing the spatial relationships and temporal dynamics of key body landmarks, the application intelligently interprets the user’s movements, offering a nuanced understanding of shoulder rehabilitation exercises beyond the limitations of a traditional 2D perspective. The key feature of the app is its ability to provide immediate feedback to the user. By analyzing the video footage, the app calculates a metric that reflects how similar the performance of the exercise is when non-guided compared to the exercise guided. This score is based on various factors such as the accuracy of the movements and the completion of the exercise.

MediaPipe is a powerful tool that uses machine learning to process video footage in real-time. One of the key features of MediaPipe is its ability to detect and track landmarks on the human body. These landmarks are specific points of interest that are identified in each frame of the video footage. When a patient performs an exercise, the app uses MediaPipe to identify these landmarks on the patient’s body. Once the landmarks are identified, it tracks their movement across multiple (video) frames. This allows the app to analyze the patient’s movements and calculate a score representing movement quality. For this study, landmarks from the shoulder, forearm, wrist, head, and trunk segment were considered.

2.4. Procedures

2.4.1. Sample Selection and Characterization

Data collection took place at a biomechanical laboratory, the Center for Rehabilitation Research of the School of Health at the Polytechnic Institute of Porto, between 5 and 20 August 2023, in a controlled environment. To avoid inter-rater error, each researcher was always responsible for the same task. Prior to data collection, anthropometric measures, body mass, height, and body mass index (BMI) were recorded for each participant. The participants maintained the orthostatic position on the scale, with bare feet and their upper limbs along the body, facing forward [25].

To identify the level of physical activity, participants were asked to complete the IPAQ.

2.4.2. Data Acquisition

The collections were carried out in one moment.

For the collection of kinematic data with Qualisys Motion Capture System, 32 reflective markers were placed bilaterally in the anatomical references (identified by manual palpation) needed to identify the movement of the assessed body segments: left and right anterior head (L/RALH), left and right posterior head (L/RPLH), left and right lateral part of the acromion (L/RCAJ), deepest point of incisura jugularis (SJN), xiphoid process (SXS), left and right styloid apophysis of the ulna (L/RULN) and radius (L/RRAD), left and right anterior superior iliac spine (L/RIAS), left and right lateral prominence of the greater trochanter (L/RFTC), spinous process of the seventh cervical vertebrae (CV7), second thoracic vertebrae (TV2), midpoint between the inferior angles of the most caudal points of the two scapulae (TV7), first lumbar vertebrae (LV1), fifth lumbar vertebrae (LV2), left and right posterior superior iliac spine (L/RIPS), left and right lateral (L/RLELB) and medial (L/RMELB) epicondyle of the humerus, left and right dorsal second metacarpal head (L/RLH), and left and right fifth metacarpal head (L/RMH) [26,27,28,29,30]. The marker setup is presented in

Table 1. Anterior view of marker setup.

Anterior View
Marker Name	Description
L/RALH	Left/Right anterior head
L/RCAJ	Left/Right acromion
SJN	Deepest point of incisura jugularis
SXS	Xiphoid process, the most caudal point of the sternum
L/RIAS	Left/Right anterior superior iliac spine
L/RFTC	Most lateral prominence of the greater trochanter
L/RRAD	Left/Right distal radius
L/RULN	Left/Right distal ulna

and

Table 2. Posterior view of marker setup.

Posterior View
Marker Name	Description
L/RPLH	Left/Right posterior head
CV7	Spinous process of the seventh cervical vertebrae
TV2	Second thoracic vertebrae
TV7	Midpoint between the inferior angles of the most caudal points of the two scapulae
LV1	First lumbar vertebrae
LV3	Third lumbar vertebrae
LV5	Fifth lumbar vertebrae
L/RIPS	Left/Right posterior superior iliac spine
L/RLELB	Left/Right lateral elbow
L/RMELB	Left/Right medial elbow
L/RLH	Left/Right dorsal 2nd metacarpal head
L/RMH	Left/Right dorsal 5th metacarpal head

.

After placing the reflective markers, participants were instructed on the execution of two therapeutic exercises, namely, a diagonal shoulder exercise (D1) [31] and a multi-joint exercise, including shoulder external rotation at 90° of shoulder abduction (M90) [32] by a specialized physiotherapist. A description of the exercises used are shown in

Table 3. Description of the diagonal shoulder exercise (D1) and multi-joint exercise, including shoulder external rotation at 90° of shoulder abduction (M90) used for the kinematic analysis.

Exercise	Initial Position	Final Position	Description
Diagonal Shoulder Exercise (D1)			The participant is standing, looking forwards with trunk and pelvis in a neutral position. The hand of the right upper limb should be at the level of the hip of the opposite lower limb and rotated inwards. The participant was asked to elevate the upper limb, both in the sagittal and frontal planes, and simultaneously rotate the right hand outwards. In the end, the patient should return to the starting position.
Multi-joint Exercise, Including Shoulder External Rotation at 90° of Shoulder Abduction (M90)			The participant is seated with knees bent at 90° flexion and feet on the floor, looking straight ahead and with the trunk and pelvis in a neutral position. The right elbow should be at 90° flexion, and the hand should be pointing forwards. The participant was asked to rotate the trunk to the right side and simultaneously rotate the right hand, producing the maximum shoulder external rotation. For 3 s, the participant was asked to bring the scapulas together, avoiding tilting the trunk. In the end, the patient should return to the starting position.

. Before data collection, sufficient time was given until the participants became familiar with the experimental setting.

Initially, each participant performed the exercises under the supervision of the physiotherapist to collect data from the movements that served as the basis for the comparative analysis—guided exercises (GE). The order in which the exercises were performed was randomized for each participant. After this collection, each participant watched a demonstration video on the application of the exercises to be performed. Subsequently, each exercise was performed unilaterally with the right upper limb three times, without supervision—non-guided exercises (NGE). A resting time of one minute between repetitions was established to prevent fatigue. While conducting the exercises, the optoelectronic system (considered the gold-standard equipment) recorded the 3D kinematics, and simultaneously, a 2D video was captured using a smartphone.

After performing all the proposed exercises, the researcher showed each participant a video demonstration of the application, exposing all the functionalities and its layout. Then, the potential interest of the participants in using the smartphone app was questioned.

To analyze the movement variation in Qualisys Motion Capture System, angles were calculated for each segment between two lines formed by anatomical markers. For the analysis of the head segment in both exercises, the markers left and right anterior head (L/RALH) and left and right acromion (L/RCAJ) were considered. Regarding the trunk segment, the markers left and right acromion (L/RCAJ) and left and right anterior superior spine (L/RIAS) were considered for both exercises. For the analysis of the shoulder segment, in the D1 exercise, the markers right acromion (RCAJ), right anterior superior spine (RIAS), and right lateral elbow (RLELB) were used. In the M90 exercise, we contemplated the right acromion (RCAJ), right lateral elbow (RLELB), and right distal ulna (RULN) markers for the analysis of the shoulder segment. Before and after calculating each angle, the “fit to 2nd degree curve” filter was applied.

For the smartphone app kinematic analysis, the recorded videos from guided exercises (GE) were directly compared to the non-guided exercises (NGE) for each participant. This analysis was performed (to all video frames) by calculating a similarity score between the landmarks from head, trunk, and shoulder segments of both videos. For the relevant landmarks, the distance between both videos was calculated using the cosine similarity. After comparing both videos, the average similarity was calculated, and if a predefined threshold was reached, that frame was considered similar. To calculate the final score representing movement quality, the similar number of frames was measured against the total, e.g., for a specific landmark, if we had an exercise video with 100 frames and 80 were deemed as similar, the final score would be 80%.

For the optoelectronic systems data, the final score was determined based on a comparison of the range of movement of guided exercises (GE) and non-guided exercises (NGE), which represents the percentage of approximation of the range of motion (ROM) obtained for each segment.

The percentage of approximation of the range of movement (ROM) obtained for each segment with the guided exercises against the non-guided exercises was designed as movement quality index and was assessed by using the following Formula (1):

$$Movement quality index (\%)=1- \frac{{(ROM}_{GE}-{ROM}_{NGE)}}{{ROM}_{GE}}\times 100$$

(1)

After the collection and processing of the data provided by the instruments, they were exported to a Microsoft Excel for Microsoft 365 spreadsheet [33].

2.4.3. Statistical Analysis

For the statistical analysis, the software Predictive Analytics Software Statistics version 28 (SPSS IBM Corporation, Armonk, NY, USA) was used, with a significance level of 0.05 and a confidence interval of 95%.

Since the sample size was less than 30, the distribution of normality could not be assured. Thus, the Shapiro–Wilk test was used to test normality since the number of the sample was less than 50. As the variables did not follow a normal distribution, the median, the 25 and 75 percentiles, and the percentage values were used for the descriptive analysis.

Bland–Altman analysis was used to test the agreement between methods and to identify possible bias tendency. The percentage of the differences between the two methods were plotted against averaged values of the two methods. Separate Bland–Altman plots were created for head, trunk, and shoulder segments for both exercises. A linear regression analysis was calculated to quantify the bias tendency.

3. Results

3.1. Sociodemographic and Clinical Characterisation

As a result of the questionnaire distribution, 19 responses were obtained, of which 4 were excluded. Thus, the final sample consisted of 15 participants. The exclusion criteria are described in Figure 1.

After analyzing the quantitative variables of age and BMI, the median values (P25; P75) of each parameter were used to characterize the population. These values are described in

Table 4. Characterization of the participants regarding age and BMI (Body Mass Index). Descriptive values of median (P25; P75) and p-values from the Shapiro–Wilk test are presented.

	Median (P25; P75)	Valor p
Age (years)	25.00 (23; 27)	0.001
IMC (kg/m2)	23.14 (22.04; 27.34)	0.004

.

Regarding the variable of gender, IPAQ and “Last shoulder pain episode” percentage values (%) of each parameter were used to characterize the population and are described in

Table 5. Characterization of the sample according to gender, the level of physical activity, and the variable “Last shoulder pain episode”. Descriptive percentage values are presented below.

Gender	Women	60%
	Men	40%
Physical Activity Level	Low	20%
	Moderate	26.7%
	High	53.3%
“Last Time You Had Shoulder Pain”	Never	66.6%
	More than 6 months ago	26.7%
	Less than 6 months ago	6.7%

.

Also, regarding upper limb dominance, only one participant reported predominantly using the left upper limb; the other participants were right-handed. Only one participant reported having a diagnosed health condition, which in this case was diabetes mellitus 1.

3.2. Concurrent Validity

As depicted in Figure 2 and

Table 6. Data from Bland–Altman analysis for all kinematic variables.

Segments	Bias		SD of Bias		LoA
	D1	M90	D1	M90	D1	M90
Head	30.88	28.95	20.67	39.74	[−9.63; 71.40]	[−48.94; 106.8]
Trunk	22.20	6.22	14.80	6.37	[−6.80; 51.21]	[−6.27; 18.71]
Shoulder	3.51	−20.46	3.62	31.23	[−3.58; 10.62]	[−81.67; 40.74]

Standard deviation (SD); limits of agreement (LoA).

, the extent of agreement varies according to the exercise and the specific body segment under consideration. To illustrate, when focusing on the head segment during the D1 exercise, a range of limits of agreement spanning from −9.63 to 71.40 can be observed, and an even wider range from −48.94 to 106.8 during the M90 exercise (Figure 2, Table 6). Bland–Altman analysis of this segment in both exercises revealed the presence of bias, which is corroborated by a significant correlation obtained from the regression analysis (p < 0.05). This correlation indicates that the difference (%) between the two methods increases as the movement quality index decreases. Turning our attention to the trunk segment, a more limited variation in the range of limits of agreement can be noted with an evident bias (Figure 2). It is noteworthy, however, that the average values of the movement quality index in the trunk segment were consistently high across both exercises. In the shoulder segment during the D1 exercise, a narrower range of limits of agreement and a low bias value can be observed (Figure 2). Conversely, in the M90 exercise within the same segment, wider limits of agreement and higher bias values can be noted (Figure 2). Notably, no discernible bias trend was found in this segment, as evidenced by the absence of a significant correlation (p > 0.05), which sets it apart from the other segments.

4. Discussion

The purpose of this study was to assess the concurrent validity of a smartphone application employing a 2D video system for supervising shoulder complex exercises in comparison to a gold-standard optoelectronic 3D system. Our hypothesis centered on the feasibility of employing an affordable and portable system as a kinematic tool with reasonable accuracy for guiding home-based rehabilitation exercises when compared to the gold standard.

Our findings indicate that agreement varies depending on the body segment and exercise in question. Starting with the analysis of the HEAD segment, we observed a wide range of limits of agreement in both exercises, representing significant variation in the percentage difference between the two methods. The Bland–Altman analysis also reveals a consistent bias trend. As the movement quality index of each subject decreased, the percentage difference between the two methods increased. However, upon closer examination of the Bland–Altman plot for the HEAD segment during the M90 exercise, it becomes evident that an agreement between methods exists when the movement quality index exceeds 90%, as evidenced by mean differences approaching zero [34]. This result may suggest that there is an alignment between the application and the 3D system when participants perform the guided and non-guided exercises almost identically. Future studies should assess the application in populations more prone to compensations, such as those with associated symptoms, as our participants generally demonstrated good movement quality.

Regarding the trunk segment, which plays a pivotal role in both exercises, we observed narrower ranges of limits of agreement and bias values compared to the head segment. The bias trend was again evident in the Bland–Altman analysis and confirmed by the regression analysis. However, in the context of the D1 exercise and considering the analysis of the trunk segment, the average variation in the movement quality index ranged from 70% to 100%, and for the M90 exercise, it ranged from 90% to 100%. This finding suggests that, overall, participants displayed consistent performance in both guided and non-guided versions of the exercise. Consequently, the observed trend associated with the trunk segment in both exercises may have limited clinical significance, as the small variation in the average movement quality control (x-axis) indicated an optimal kinematic relationship between GE and NGE.

In the shoulder segment during exercise D1, a narrower range of limits of agreement was noted, indicating reduced disparities between the 2D and 3D kinematic analyses. This phenomenon may be linked to the lower bias values observed in this segment and exercise. Furthermore, in this exercise, the shoulder segment displayed a narrower range of variation, with the movement quality index varying between 90% and 100%. The diagonal movement performed by the shoulder complex is recognized as a functional motion commonly employed in numerous daily activities [31,35]. This may explain the participants’ higher proficiency in executing this movement without direct supervision from a physiotherapist, closely mirroring their performance in the guided exercise. This suggests that the observed bias trend may not have clinical significance. However, in the M90 exercise, the shoulder segment, despite showing higher values of limits of agreement and bias compared to the same segment in the D1 exercise, did not exhibit the consistent bias observed in the other segments and showed no statistical differences between methods (p > 0.05). This result reveals that for the principal segment of this exercise—the shoulder—the application demonstrates kinematic agreement with the 3D system.

Globally, the results of the present study seem to suggest that the application could be a valuable tool for supporting home-based rehabilitation in individuals with extensive movement experience and postural control. Future studies are required involving populations with low physical activity levels, limited movement awareness, or even associated chronic pain to test the suitability of this system for these subjects [36].

Some limitations were identified in this study. The fact that the sample was recruited voluntarily led to the existence of selection bias, resulting in a small sample size while hindering the representativeness of the population and decreasing the statistical power, thus compromising the external validity of this study. Regarding the questionnaires used to characterize the sample, all were subject to memory bias since some questions referred to past events.

For future studies, we suggest the recruitment of a larger sample and it would also be relevant to use more stringent inclusion and exclusion criteria to control confounding variables and allow data reproducibility. Given postural variability, it would be interesting to include specific populations (e.g., those with shoulder pain and those without), different tasks, and data collection protocols [18].

To our knowledge, very few studies have validated 2D video systems integrated into smartphones against 3D optoelectronic systems. A study conducted by Ramkumar et al. (2018) [37] aimed to validate a mobile technology for assessing shoulder range of motion. The study involved a comparison between a motion-based machine learning software and a goniometer for only four specific arcs of shoulder motion. While the study showed promising levels of accuracy, it is important to note that it did not employ gold-standard equipment, such as a 3D system. This methodology could potentially have an impact on the study’s findings and results. Most existing research, as indicated by several systematic reviews [38,39] primarily examines tasks such as running or mechanical lifting, rather than those directly related to upper limb functionality. This underscores the innovative nature of our study, and it can represent a starting point for further research into this matter.

5. Conclusions

The smartphone application designed for supervising home-based rehabilitation exercises may be a valuable tool for guiding therapeutic exercise in specific populations, especially those with substantial movement experience and a heightened sense of body awareness and postural control. The results of the present study demonstrate an agreement between the movement quality evaluated by a gold-standard method and the developed application, although the proposed method appears to have less sensitivity for evaluating movements with lower quality index. However, considering that high levels of movement quality index were obtained in the present study, future studies involving shoulder pain conditions with lower levels of movement quality control are required.