Reconfigurable Robotic Exercising Companion

Abstract

Regular exercise plays a crucial role in promoting overall well-being in today’s lifestyle. However, individuals often find it challenging to properly execute exercises, including maintaining correct postures and appropriate movement speeds. Robotic companions have emerged as potential solutions to assist and motivate users during exercise sessions. This research paper proposes a novel robot companion designed for exercise scenarios using a reconfigurable robot. In contrast to existing non-reconfigurable robotic companions, the use of a reconfigurable robot provides added flexibility in generating emotions. The system incorporates a module that utilizes fuzzy logic to evaluate the correctness of exercise performance based on posture variations and movement speeds. The robot generates emotions and provides feedback to users based on the exercise correctness score. The robot expresses emotions through reconfigurations, motion patterns, and variations in robot speed. This emotion-based feedback could be helpful for creating engaging and interactive exercise experiences. Apart from emotion generation, the robot utilizes vocal cues as feedback. Experimental results validate the effectiveness of the proposed system in evaluating exercise correctness and demonstrating meaningful emotion transitions. The findings of this work contribute to the development of innovative robotic companions for improving exercise adherence and overall well-being.

1. Introduction

Regular exercise offers numerous benefits for physical and mental health [1]. With the rise in office jobs and sedentary lifestyles, people spend extended periods sitting and working on laptops, leading to weight gain and obesity. Therefore, incorporating daily exercise is essential. Exercising offers a broad range of benefits, such as disease prevention, increased energy levels, enhanced cognitive function, and stress relief [2]. Many social media platforms provide videos and tips to conduct exercises, such as on YouTube and Instagram. However, it is crucial to note that while people may engage in exercise, they often require proper guidance and motivation to perform exercises correctly, maximizing their impacts on the body. The introduction of robots as exercise trainers becomes impactful in this context.

Robots have emerged as invaluable tools for guiding and motivating individuals during their exercise routines, offering significant advantages over traditional methods of exercise guidance [3,4,5]. One key advantage is their ability to provide consistent and precise instructions, ensuring that individuals perform exercises accurately, thereby minimizing the risk of injury and maximizing the effectiveness of each movement. Furthermore, robots can offer real-time feedback on form and technique, enabling individuals to maintain proper posture and alignment throughout their workouts.

Social assistive robots have emerged as valuable exercise companions, encouraging users to engage in regular physical activity [6]. A robot that monitors user engagement and heart rate while performing upper body exercise has been proposed in [6]. This robot generated appropriate emotions based on user engagement and heart rate using an nth order Markov chain. Another study investigated the effects of exercising alone, receiving instructions from a robot, and exercising conjunctively with the robot [7]. The findings revealed that exercising in conjunction with a humanoid robot increased user motivation and led to longer exercise durations. The impact of different levels of automation on personalizing socially assistive robots in exercise scenarios was explored in a study [8]. The results demonstrated that adaptive robots were perceived as more competent and trustworthy than adaptable robots. A study comparing acknowledgment–feedback versus non-acknowledgment–feedback for a robot exercise instructor and companion has been introduced in [9]. The outcomes indicated that users who exercised co-actively with the robot exhibited significant performance improvements compared to those who were instructed by the robot or exercised alone. These studies highlight the positive influence of social assistive robots in exercise contexts, emphasizing the benefits of companion robots in increasing motivation, personalization, and performance during physical activity.

Furthermore, as the global elderly population grows, promoting active and engaged lifestyles becomes increasingly important for their overall well-being. Recognizing this need, numerous research studies have focused on developing exercising companions specifically tailored for elderly individuals [10,11]. The work [12] introduced a diverse range of human–robot interactive games specifically designed to engage both the elderly and young adults. These interactive games serve as effective tools to encourage physical activity and foster healthy lifestyles across different age groups. Particularly, the use of robot companions for rehabilitation purposes has emerged as an intriguing application in this context [13,14,15]. Integrating robotics in rehabilitation programs showcases the potential of exercise-focused robot companions to aid in the recovery and enhancement of physical well-being in disabled individuals. To personalize the interaction with humans while engaging in the exercises, robotic platforms have been introduced [16]. Furthermore, studies have been conducted to enhance the social interaction in physical training using robots [17].

The existing research discussed above primarily focuses on developing fixed-shape robots as exercise companions. However, there is an unexploited potential for introducing reconfigurable robots for this purpose. Reconfigurable robots have the capability of changing their shape based on the context in which they operate [18,19]. The unique capabilities, applications, and challenges of reconfigurable robots have been introduced in [20]. These robots have been extensively studied in various applications such as cleaning [21], exploration [22], and inspection [23], which demand area coverage. Their use as exercise companions remains limited despite their unique capabilities. Reconfigurable robots present an intriguing opportunity for human–robot interaction in the domain of robotic exercise companions. Through morphology transformation, reconfigurable robots can represent expressions in a more detailed, convincing manner. Their shape-changing abilities can be leveraged to display a wide range of emotions and generate diverse motion patterns, making them highly suitable for assisting in exercise activities.

Motivated by this potential, this paper proposes the development of a reconfigurable exercising companion designed to provide emotional support during exercise scenarios. The novelty of this paper lies in the robot’s ability to adjust its physical configuration and provide emotional feedback based on how well the exercises are performed, aiming to motivate users to improve. The use of reconfigurability for this new application is the major contribution compared to the state of the art, where only non-reconfigurable robots have been used up to now. In this proposed system, the robot delivers exercise instructions while an external laptop displays exercise-related videos for the user. The user’s exercise performance is monitored using the external laptop, and a fuzzy inference system is developed to evaluate the correctness of their movements. Based on the assessed exercise correctness, the robot generates appropriate emotions and motions to enhance the exercise experience. By capitalizing on the reconfigurable nature of the robot, this proposed system aims to create a versatile and interactive exercising companion that can adapt to different exercises and provide personalized emotional support. The incorporation of a fuzzy inference system enables real-time evaluation of the user’s exercise correctness, allowing the robot to generate suitable emotional responses and motion patterns accordingly.

The rest of this paper is outlined as follows. A system overview of the proposed method is given in Section 2. Emotion-based feedback in relation to the performance of exercise is provided in Section 3. Detailed information regarding experimental validation is presented in Section 4. Section 5 provides concluding remarks.

2. System Overview

A functional overview of the proposed system is shown in Figure 1. The proposed system analyzes and evaluates the correctness of an exercise performed by a user and gives verbal and emotion-based feedback to interact with the user. The system extracts dynamic characteristics information related to body movement in order to evaluate performance. The Body Joint Extraction unit gathers vision data as an RGB image through a monocular camera. It outputs the skeletal information as a set of 2D coordinate points representing the specific joints/landmarks of the body. The necessary parameters, such as joint angles and speed, are calculated by the exercise parameter calculation unit based on data points. The set of body joints used for the calculation varies according to the specific exercise the user performs. All the references for each body landmark corresponding to each exercise are stacked in the exercises posture reference database. Calculated parameters are then fed to the exercise correction evaluation unit. This unit uses fuzzy logic to analyze the correctness of an excise based on the incoming parameters. This module is implemented in such a way that it can adapt to different exercises. The parameters required for adapting to different exercises are stored in the parameter reference database. The system evaluates each complete exercise cycle and assigns a correctness score accordingly. The emotion output generation unit determines the type of emotion with its level based on the history of the exercise correctness score. This emotion is then fed to the action manager. The overall coordination of the system is managed by the action manager (AM). It triggers voice responses through the voice response generation unit based on the exercise correctness score. The voice response generation unit is a text-to-speech converter with a set of predefined responses. Furthermore, the AM enables playing a reference video of the exercise at the beginning of the exercise for the user to follow. Up to this level, the whole process is executed inside a PC.

The AM communicates the expression generator in the robot through a wireless link. The expression generator acts upon the requested emotion levels and determines the motion cues and shape changes. The predefined robot shape configurations mapped to a particular emotion with the motion profile to follow stored in the expression database are used in this regard. The reconfiguration controller executes the shape-shifting part of the robot based on the input from the expression generator. The navigation controller handles the maneuvering of the robot according to the given motion profile.

Reconfigurable Robot Platform

The proposed system utilizes the reconfigurable robot Smorphi for emotion expression. Smorphi (www.wefaarobotics.com, accessed on 15 June 2024) is a reconfigurable mobile robot platform consisting of four modules that are connected serially through three hinges for allowing intra-reconfigurability (see Figure 2a). Thus, all four modules act as a single platform. Reconfigurability of the robot is achieved by rotating modules around the hinges to create different shape configurations. This robot can create seven shapes in total, as shown in Figure 2b. Each robot block has a mecanum drive locomotion system to perform holonomic movements required for reconfiguration and navigation.

3. Emotion-Based Feedback per Performance of Exercise

3.1. Perceiving Exercises

The correctness of an exercise depends on several parameters. Each exercise requires the movements of a unique set of body joints in predefined ranges. Not achieving or exceeding these ranges tends to reduce the overall outcome expected from the exercise. This scenario can be used as a measure to evaluate the correctness of an exercise. For example, a situation of a person doing an exercise involved with hand movements can be considered. This exercise mainly involves two body joints (i.e., shoulder and elbow joints). Monitoring the movements of these joints can give an idea of how correct the exercise is. There should be a method to extract the information related to the body joint movement for the proposed system to evaluate the correctness of an exercise. In this regard, the proposed system utilizes a monocular camera to capture the movements of body joints. The locations of the body joints are extracted from 2D image frames using MediaPipe Pose framework (https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker accessed on 15 May 2024).

This joint extraction model returns a total of 33 human body joints as depicted in Figure 3a. Here, each joint has a unique Identity (ID), which is useful for tracking. This information is then used to derive the joint angles that can be used to determine the correctness of the posture during an exercise [24]. Apart from varying the posture, the speed of the movements should be within certain limits for a specific exercise where the movement speed should not be too slow or too fast [25]. Therefore, posture correctness and movement speed are considered to evaluate the correctness of an exercise. The posture correctness is measured as the deviation from the required posture movement range.

Given the 2D body joint coordinates for considered exercise, joint angles can be derived. For a predefined set of exercises, the required joint angles to be calculated are stored in a database with a given reference ID to each joint angle. Let set A consist of the exercises $\{E_1,E_2,\dots E_e\dots E_n\}$. The eth exercise consists of an array of joint angles IDs $[{\theta }_1,{\theta }_2\dots {\theta }_i\dots {\theta }_n]$ to be measured. Equation (1) explains how each exercise data point with relevant angles is stored in the database A. The required angles can be derived using this database for a known exercise.

$$A=\{E_1:[{\theta }_a,{\theta }_b,\dots ],\dots ,E_e:[{\theta }_1,{\theta }_2\dots {\theta }_i\dots {\theta }_n],\dots E_n\}$$

(1)

A specific joint angle can be calculated using another database that stores a stack of body joint IDs required to form a specific joint angle. Let B be the set of the joint angle identities for the eth exercises such that $B=\{{\theta }_1,{\theta }_2,\dots {\theta }_i\dots {\theta }_n\}$. Each joint angle ID points to a set of body joint IDs where ith joint angle can be measured using the locations of the joints $(j_{i-1},j_i,j_{i+1})$. The mapping of joint IDs to each joint in database B is explained in (2).

$$B=\{{\theta }_1:(j_0,j_1,j_2),\dots ,{\theta }_i:(j_{i-1},j_i,j_{i+1}),\dots {\theta }_n\}$$

(2)

The joint angle ${\theta }_i$ can be calculated as in (3), where $(x_i,y_i)$ is the extracted joint coordinates (see Figure 3b).

$${\theta }_i=ta{n}^{-1}{\displaystyle \frac{y_{i+1}-y_i}{x_{i+1}-x_i}}-ta{n}^{-1}{\displaystyle \frac{y_{i-1}-y_i}{x_{i-1}-x_i}}$$

(3)

such that relevant body joint angles are calculated for each captured image frame at time t. Calculation of the joint angle deviation becomes complex if each captured frame is considered, since there should be a database of correct joint angle values for each frame, which is challenging to implement. Instead, the deviation between maximum and minimum values of joint angles acquired per an exercise cycle is considered. In this case, the only requirement is to have a database with possible maximum and minimum joint angles for each joint involved in each exercise. Consider that the minimum and maximum angles of the ith joint of the exercise $E_e$ are ${\theta }_{i:min}$ and ${\theta }_{i:max}$, respectively.

Let the database F consist of reference minimum and maximum values for joint angle data $({\theta }_1,{\theta }_2,\dots {\theta }_i,\dots {\theta }_n)$ for each considered exercise $(E_1,E_2,\dots E_i\dots E_n)$. Then, (4) shows how the data are stored in the database F.

$$F=\{E_1,\dots ,E_e:\{\dots ,{\theta }_i:({\theta }_{i:refmin},{\theta }_{i:refmax}),\dots \},\dots ,E_n\}$$

(4)

Using the reference minimum and maximum joint angles (i.e., ${\theta }_{i:refmin},{\theta }_{i:refmax}$), the posture deviation for the eth exercises at time t, D can be calculated as in (5). Mean square error is utilized as the joint deviation since deviations can be negative and positive.

$$D=\sqrt{\sum {({\theta }_{i:min}-{\theta }_{i:refmin})}^2+{({\theta }_{i:max}-{\theta }_{i:refmax})}^2}\forall i\in e\mathrm{th}$$

(5)

The second parameter to evaluate the correctness of an exercise is the movement speed per cycle (defined as S). Let the image frame $f_o$ indicate the initiation of one exercise cycle, which is acquired at time $t=T_o$. Then, $f_p$ frame represents the end of that cycle at $t=T_p$. Then, the speed per cycle S can be calculated as in (6). The frames corresponding to the minimum and maximum joint angles are used to identify the initiation and completion frames of an exercise cycle.

$$S=1/(T_p-T_o)$$

(6)

Finally, calculated posture deviation (i.e., D) and the movement speed (i.e., S) are used to evaluate the exercise correctness in each cycle.

3.2. Exercise Correctness Score

The goal of this module is to evaluate the correctness of an exercise performed by a user. Although the mathematical modeling of the relationship between perceived parameters (i.e., posture deviation, D and movement speed, S) and exercise correctness is challenging, linguistic explanations based on expert knowledge can provide insights. In this context, fuzzy logic can be employed to model this process, which involves unknown dynamics and linguistic relationships [26]. Fuzzy logic is a universal approximator that can map the nonlinear relationship between input and output spaces through linguistic expressions [27,28]. Additionally, the user parameters perceived by the robot may contain noise due to limitations in the vision-based detection of human body joints. In contrast, fuzzy logic effectively handles imprecise sensory information [29]. Furthermore, the relationships between input parameters and exercise correctness should be adapted to each exercise. Fuzzy logic possesses adaptive abilities, making it well-suited for evaluating exercise correctness based on the perceived information. Hence, fuzzy logic is utilized to develop the module to evaluate the correctness of an exercise performed by a user based on the perceived information.

The inputs of the fuzzy logic module are the posture deviation, D, and movement speed, S. Figure 4a,b depict the corresponding input fuzzy membership functions. Non-singleton fuzzy sets have been chosen to cope with the uncertainties in the inputs. The membership function for D has been defined with the three triangular fuzzy sets; L: Low, M: Medium, and H: High. The membership function for S has been defined with the three triangular fuzzy sets; S: Slow, G: Good, and F: Fast. The ranges D and S should be varied with the exercise a user is conducting since the amount of variation in body joint angles and the expected speed differ from one exercise to another. The ranges of the input membership functions are adapted with $K_e$ and $L_e$ to facilitate the adaptation of the fuzzy logic module for different exercises. The parameters $K_e$ and $L_e$ define the possible posture deviation and the speed range for the eth exercise. These range parameter values are derived by analyzing a set of typical series of exercise cycles, and obtaining the average for each selected parameter. These data are pre-stored in the database and retrieved when they are required.

The output of this fuzzy logic module is the correctness score of an exercise performed by a user (defined by C). The output membership function is given in Figure 4c. The output membership function has five triangular fuzzy sets; VL: Very Low, L: Low, M: Medium, H: High, and VH: Very High. The fuzzy rule base given in

Table 1. Rule base of the fuzzy logic module.

Input Memberships		Deviation D
		L	M	H
Movement speed S	S	H	L	VL
	G	VH	M	L
	F	H	L	VL

maps the input fuzzy sets with the output fuzzy sets. There is a particular speed that is most effective for a specific exercise, and performing it too fast or too slow would not be as impactful. Similarly, deviation in the postures should be minimized to make the exercise perfect. These facts were considered in formulating the rule base based on expert knowledge. The fuzzy sets in the output membership function are defined by equivalently dividing the output range in such a way that the minimum and the maximum output are in the range $[0,100]$. The fuzzy logic module is considered to be a Mamdani-type system with the center of area method for the defuzzification. A C value closer to 100 indicates that the exercise is performed flawlessly, while 0 indicates that the exercise is not performed at a satisfactory level.

3.3. Emotion Generation

Expressing emotions through non-humanoid robots can be challenging [30]. Therefore, adopting a systematic approach to generate emotions effectively is crucial. One effective method is utilizing an emotion space model to represent and assess emotions. Various options are available in this context, ranging from 1D to 4D emotion space models [31]. The application and the desired number of emotions should be accounted for to determine the most suitable model.

In the proposed system, the exercises evaluation unit generates a correctness score in the range $[0,100]$ for each exercise cycle. Emotions can be mapped by setting threshold values at specific positions along the continuum of correctness values, which spans from 0 to 100. This enables the mapping of multiple emotions by defining distinct regions. However, due to the dynamic nature of the scenario and the limitations of the hardware setup, incorporating multiple emotions may result in a less smooth emotion transformation.

Existing literature based on emotion space models reveals various approaches to express emotions using non-humanoid robots such as movement path, speed, color indications, and sound [32,33]. Using multiple unique techniques allows for distinguishing between emotions easily. Thus, the proposed system uses three different methods to express an individual emotion with its level. Those are movement pattern, movement speed, and reconfiguration-based visual indication. As one emotion is the opposite of the other, making the transition recognizable while being smooth is vital. The proposed emotion model is inspired by the valence–arousal model [34]. Valence discriminates between positive and negative emotions, while arousal stands for the energy level of each emotion. Happiness refers to high valence, high arousal expression, whereas sadness refers to low valence, low arousal expression. According to the work [34], movement patterns can be used to represent the valence where jerky motion indicates the negative and a smoother motion pattern is used for the positive expression. Furthermore, speed can be used to indicate the level of energy, which is arousal. Apart from that, the proposed system uses one more technique to distinguish emotions in view of valence. Visual indication of the emotion is a highly efficient way to represent it such that a user can easily understand the expression well. The proposed system uses a sticker arrangement placed on top of the robot to indicate emotion as a facial expression similar to a human. This sticker placement is aligned in such a way that the reconfiguration can produce a unique human-like visual aid. Reconfiguration is a unique feature of Smorphi, the mobile robot that is used in the proposed system. Figure 5 shows the emotion model used in the proposed system. It also shows how the expression modalities are mapped to specific emotions.

The proposed system uses O-shape and S-shape configurations for the emotion expression through reconfiguration based visual aid. Figure 6 shows the reconfiguration-based emotions that can be generated by the robot.

After receiving the exercise’s correctness score at cycle t (i.e., $C_t$), the average correctness score at cycle t (defined as $\widehat{C_t}$) is calculated as in (7) considering the correctness scores received for the immediate past three cycles. This average calculation is performed every three cycles to avoid the high variations in emotion output that could not be handled by reconfiguration as well as to account for the history of exercise performance by a user. Two threshold values are introduced to define the emotion regions based on the range of $\widehat{C_t}$. Algorithm 1 shows the flow of the emotion generation based on the average correctness score. According to the output emotion, the robot motion profile is derived. The robot moves on a jerky path for the sad emotion expression, and for the happy emotion, it follows a smooth circular path. Furthermore, the robot’s movement speed is calculated based on the degree of each emotion level. This process is given in Algorithm 2.

$$\widehat{C_t}={\displaystyle \frac{1}{3}}\sum _{t-2}^t{C}_t$$

(7)

Algorithm 1 Emotion derivation

$\widehat{C_t};$ $upperthreshold;$ $lowerthreshold;$ $emotion\leftarrow none$ $emotionlevel\leftarrow 0$ if  $\widehat{C_t}\ge upperthreshold$  then     $emotion\leftarrow happy$     $emotionlevel\leftarrow \left({\displaystyle \frac{\widehat{C_t}-upperthreshold}{100-upperthreshold}}\right)\ast 100$ else     if $\widehat{C_t}\le lowerthreshold$ then         $emotion\leftarrow sad$         $emotionlevel\leftarrow \left({\displaystyle \frac{lowerthreshold-\widehat{C_t}}{lowerthreshold-0}}\right)\ast 100$     end if end if

Algorithm 2 Motion pattern, robot speed calculation

$emotion;$ $emotionlevel;$ $maxspeed;$ $motionpattern\leftarrow none$ $speed\leftarrow 0$ if  $emotion=happy$  then     $motionpattern\leftarrow circular$     $speed\leftarrow \left({\displaystyle \frac{emotionlevel}{100}}\right)\ast maxspeed$ else     $motionpattern\leftarrow jerky$     $speed\leftarrow \left({\displaystyle \frac{emotionlevel}{100}}\right)\ast maxspeed$ end if

The robot is commanded to follow the motion profile and the speed determined by the algorithms. These motion cues are predefined for both cases of emotions. The navigation controller of the robot derives linear velocity components of the robot in x and y directions at time t (defined by $(v_{x_t},v_{y_t})$). Here, the robot’s movement is holonomic. Hence the angular displacement of the robot is not required for circular loop motions.

Let the motion pattern be circular with a fixed radius and speed is $v_t$ at time t. Let $\varphi$ be the current position on the path. Thus, directional linear velocity components of the robot $(v_{x_t},v_{y_t})$ can be calculated according to (8). The directional velocity components are derived according to (9) for the jerky motion. Here, $\alpha$ is a constant that controls the amount of jerkiness. The wheel angular velocities corresponding to the linear velocity components are calculated based on the kinematic model of the robot. The motors are commanded to follow the derived wheel velocities.

$$\left[\begin{array}{c}{v}_{x_t}\\ v_{y_t}\end{array}\right]=v_t\left[\begin{array}{c}cos\left(\varphi \right)\\ sin\left(\varphi \right)\end{array}\right]$$

(8)

$$\left[\begin{array}{c}{v}_{x_t}\\ v_{y_t}\end{array}\right]=v_t\left[\begin{array}{c}cos\left(\varphi \right)+sin\left(\alpha \varphi \right)\\ sin\left(\varphi \right)+cos\left(\alpha \varphi \right)\end{array}\right]$$

(9)

4. Experiments and Results

4.1. Experimental Setup

Normal squats and sumo squats were considered for the experiments (see Figure 7b,c). These two exercises were often conducted in domestic environments and are essential to verify the correctness of body joint movements. The user maintained a distance from the laptop in such a way that the camera captured the whole body’s joints. The robot was placed in front of the user in order to perform the emotional expression. The experiment setup is depicted in Figure 7a. The setup consisted of a laptop computer that handles the vision-based perception and processing. The built-in front camera of the laptop was used to capture the video. No additional data collection or training was performed as the detection processing module uses pre-trained models. The robot was initiated with the happy configuration without movements. The experiments commenced with a warm greeting and then provided a video guide to follow the specific exercise by the computer. The user then performed the exercise, referring to a guide video on the computer screen. In order to observe and verify the transition of emotions, the user was instructed to perform the exercise incorporating correct and incorrect posture variations. This process allowed for the examination of emotional changes in response to different performance scenarios. The important variables of the system were recorded for each time step throughout the experiment.

4.2. Results and Discussion

The results obtained with respect to the normal squat exercise are displayed in Figure 8. The variations in the emotions and the robot speed with the average correctness score during this experiment are depicted in Figure 9.

Table 2. Normal squat performance results.

Exercise Cycles	Posture Deviation	Cycle Speed	Correctness Score	Average Correctness Score	Emotion	Robot Speed
1	23.12	0.43	66.98	83.57	Happy	0.38
2	13.66	0.53	83.71
3	12.38	0.46	100.00
4	13.82	0.47	98.32	90.31	Happy	0.49
5	16.70	0.46	86.50
6	17.10	0.41	86.10
7	11.00	0.51	88.99	70.40	Happy	0.17
8	15.10	0.38	99.13
9	39.83	0.59	23.08
10	53.76	0.63	9.01	26.15	Sad	−0.23
11	33.14	0.25	45.12
12	36.53	0.60	24.33
13	53.74	0.65	8.68	8.45	Sad	−0.51
14	88.57	0.82	8.33
15	87.81	0.78	8.33

represents the results for the first 15 cycles of normal squat exercise performance. The variation in the parameters during the second cycle is considered for the explanation. Knee and hip flexion are the movements that are required to be monitored during the squat exercises. Therefore, variations in knee and hip angles are considered in evaluating the exercise’s correctness. These were identified from the pre-stored data in the posture reference database. The posture variation captured by the camera during the second cycle is shown in Figure 10.

The maximum knee and hip joint angles captured by the system were 179° and 182°, while the minimum knee and hip angles were recorded as 52° and 61°, respectively. The maximum reference angle defined for knee and hip angles was 180°, while the minimum reference angle was 65° for both. Thus, the posture deviation was observed as 13.78°. According to the fuzzy membership functions that correspond to this exercise, this value has a higher degree of membership for the Low deviation region and a trivial degree of membership for the Medium deviation region. The movement speed was recorded as 0.53, slightly higher than the reference movement speed corresponding to the exercise (reference speed is 0.38). Thus, a correctness score of 83.71 was received for this cycle, suggesting that the user is correctly performing the exercise. Here, the user performed the exercise correctly, and the proposed system successfully interpreted it.

The user intentionally performed some segments of the exercise incorrectly to validate the system’s ability to identify such situations. For example, the user performed the exercise during cycle 15 while having deviated posture movements. Figure 11 depicts the body joint observation recorded in the system. During this cycle, minimum angles of 131° and 122° were observed for hip and knee flexion, respectively. Consequently, the deviation value was calculated as 87.8°. In addition, the observed movement speed was 0.6 m s⁻¹ indicating a too-fast movement speed. As a result, the fuzzy logic system returned a correctness score of 8.33, suggesting that the user was performing the exercise incorrectly.

The variations in the average correctness score and the corresponding emotions of the robot is given in Figure 9. The average correctness value at the end of the third cycle was 83.57. Since the average value is higher than the threshold value of 60, the robot started to express a happy emotion with a circular movement. The movement speed was 0.38 m s⁻¹. The user performed the exercise correctly in the subsequent three cycles, and the average correctness score observed at the end of the sixth cycle was 90.31. As a result of the increased happiness level, the robot speed was raised to 0.49 m s⁻¹ to express that the user is performing well.

At the end of the 9th cycle, the average correctness score was 70.4. This value was higher than the predefined upper threshold, and the robot’s emotion output was happy. However, the robot speed was lowered to 0.3 m s⁻¹ since the average correctness score dropped. According to these parameter values, the robot’s configuration was an O-shape, and it was moving on a circular path with a speed of 0.3 m s⁻¹. The same emotion with the same speed was observed for the subsequent three cycles.

At the end of the 12th cycle, the average correctness was 26.15, which is below the lower threshold of the average correctness score considered for sad emotion transition (i.e., 40). So, the robot reconfigured itself to an S-shape to express a sad emotion. After reconfiguration was performed, the robot moved on a jerky path with a speed of 0.23 m s⁻¹. The robot remained in this shape configuration and motion expression until the end of the 24th cycle. At the end of the 24th cycle, the robot’s emotion changed to happy, and the robot was moving in a circular path in the O-shape. This behavior continued until the end of the first experiment. These observations related to emotion variation suggest that the proposed system is capable of varying the emotion as planned to give feedback to the user.

The second experiment was carried out considering a sumo squat exercise. Similar to the previous experiments, the user performed both correct and wrong exercise cycles to analyze the behavior of the system. Only the knee angle was considered for evaluating the correctness of this exercise since it was the factor defined in the database. Figure 12 shows a cycle of a user accurately performing sumo squats. The third cycle is considered here. The maximum knee flexion has been recorded as 180°, where the leg is fully stretched. As for the minimum angle, 95° was observed. The reference minimum for the knee flexion angle was defined as 100° for this exercise.

Hence, the posture deviation was observed as 7.87°, which has a higher degree of membership for the low category in the fuzzy membership function. The movement speed was observed as 0.73 m s⁻¹, which is closer to the expected speed level for this exercise. Thus, the exercise correctness score received a value of 100. The system has correctly identified the perfectness of the exercise being performed by the user. Figure 13 shows the variation in the exercise correctness score during the second exercise. The cycle number 20 represents a situation where the user incorrectly performed the exercises. The posture deviation was 29.53°, and the movement speed was observed as 0.37 m s⁻¹. These parameters were way off from the expectation. Thus, a correctness score of 13.78 was received, indicating that the user was performing the exercise incorrectly.

Figure 14 shows the variations in robot emotion and average correctness score. Referring to the figure, an emotion transition happened at the 27th cycle. The previous average, calculated at the 24th cycle, was beneath the lower threshold; hence the robot expressed sad emotion with an S-shape configuration while moving in a jerky path with a speed of 0.1 $\mathrm{m}{\mathrm{s}}^{-1}$. In the 27th cycle, the average correctness score climbed above the upper threshold, and the emotion output transformed from sad to happy as in Figure 14. The robot reconfigured from S-shape to O-shape in this regard. The robot’s speed also varied accordingly.

The observation from the two experiments confirms that the system is effective in evaluating the correctness of an exercise being performed by a user. The results also validated that the robot is capable of expressing emotions based on the variation in the exercise correctness. Furthermore, the robot configuration, movement path, and speed are effectively varied for expressing the required emotion.

5. Conclusions

A reconfigurable robotic system that can evaluate the correctness of an exercise being performed by a user and gives emotion-based feedback to the user has been proposed. The system consists of two segments, one for exercise correctness evaluation and another for robot emotion generation.

The exercise correctness evaluation module implemented using fuzzy logic determines the correctness of an exercise based on a set of parameters derived from the skeletal information. A monocular vision-based system is used to acquire skeletal information about a user during an exercise scenario. The posture deviation and the movement speed are considered for evaluating the correctness. The system has been developed in such a way that it can be adapted to various exercises by defining the reference values corresponding to exercises of interest.

The robot’s reconfiguration, movement pattern, and speed varied per the correctness of the exercise to generate robot emotions. The major leap of the proposed system over the existing approaches in emotion expression is its unique emotion generation with the aid of reconfiguration. The system also provides predefined voice feedback to the user, revealing an idea about the correctness of the exercise.

In order to evaluate the performance of the proposed system, experiments have been carried out considering a mixture of correct and wrong exercise scenarios. The experimental results validated the effectiveness of the proposed system in determining the exercise correctness and the transition of emotions per the variation in the exercise correctness. Therefore, the proposed system would be useful as an exercise companion that could interact and provide feedback to a user to improve exercise performance, as it provides interactive emotional feedback that encourages the users to correct their postures.

Currently, the proposed system is only capable of evaluating one target at a time, and the robot may fail to adapt to an environment with more than one human. The positioning of the user is highly important as the vision system should be able to capture the whole body of the user to generate skeletal information. As future work, it is expected that the system will be improved to solve these limitations. Furthermore, the current system has been implemented to express only two emotions, happy and sad, which can be further improved to have more emotional feedback.