User Assessment of a Customized Taekwondo Athlete Performance Cyber–Physical System

Abstract

This study outlines the assessment of the cyber–physical system SPERTA, which was designed to evaluate the real-time performance of Taekwondo athletes. The system conducts performance analyses focusing on speed, acceleration, strength, and identifying and quantifying the athlete’s movements. The research involved administering an online questionnaire to athletes and coaches to evaluate the system’s acceptance and usability. The methodology included using a questionnaire with open and closed questions to assess participant satisfaction and system usability. The results showed a positive response to the system, with participants emphasizing its reliability and ease of use. An analysis of the responses revealed a strong internal consistency, as indicated by the Cronbach’s alpha coefficient, which enhances the research instrument’s reliability. Additionally, the analysis of open-ended questions was conducted through thematic analysis to gain a deeper understanding of participants’ experiences and perceptions of the system. These results highlight the effectiveness of the SPERTA system as a valuable tool for the real-time performance evaluation of Taekwondo athletes, providing insights for future improvements and the development of more effective training techniques.

1. Introduction

The emergence of computer technology in conjunction with the rapid progress in technology has revolutionized the development of electronic systems. This has made the development process faster and, at the same time, more efficient and precise. In addition, the time and cost of production can be significantly reduced. In the course of this development, new methods and approaches have emerged, such as agile development and object-oriented software development [1]. At the same time, there has been remarkable developments in artificial intelligence and the Internet of Things (IoT), which have created a new reality in the development of cyber–physical systems [2]. This is due to the possibility of connecting devices and systems to a global network such as the Internet, creating ever smarter and better-connected systems. In this new scenario, we can respond to the needs of modern society and drive technological innovation based on the continuous development of cyber–physical systems [3].

Taekwondo was officially recognized as a martial art in South Korea in 1955 and consists of a combination of self-defense and sport. Since then, this martial art has seen great popularity all over the world [4].

As in other sports, evaluating and improving the performance of athletes is a constant challenge for coaches. However, the lack of technical solutions specifically designed for Taekwondo makes this challenge even more difficult [5]. To change this reality and close this gap, the SPERTA (Real-time Evaluation System for Top Taekwondo Athletes) cyber–physical system was developed [6].

This study was developed with the support of the Portuguese National Taekwondo Team and with the participation of athletes and coaches from the Taekwondo Academy at the University of Minho (Portugal), where Olympic-level athletes’ train. This paper aims to present the results obtained, regarding acceptance, level of intrusion, and ease of use among athletes, after using the SPERTA cyber–physical system in a real use environment during training sessions. The athletes’ opinions about possible changes or improvements in the developed system were also obtained. The main goal is to reach a system that can be used effectively as an auxiliary tool for evaluating the performance of Taekwondo athletes in real time.

2. Methodology

The paper is structured into six chapters, as outlined in the flowchart depicted in Figure 1. In the initial chapter, an introduction and contextualization of the study are provided. The second chapter outlines the methodology employed in the study, highlighting the research conducted on athletic performance analysis and the system created for performance analysis, SPERTA. The subsequent chapter details the study conducted to evaluate the acceptability and usability of the system, discussing the data collection method and the questionnaire used to assess the acceptability of the system. The fourth chapter presents the results obtained from the survey analysis. The fifth chapter involves a discussion of the findings. The final chapter presents the conclusion and outlines future work.

2.1. Studies on Performance Analysis in Sport

The advancement of electronics and computer technology has given rise to the creation of systems that are capable of carrying out tasks of varying complexity, depending on their intended use. These systems can be viewed as tools that assist users in specific activities [7]. For instance, there are tools that have been developed to enable the analysis and monitoring of the human body by recording and interpreting the movements executed by the system’s users [8,9]. Thanks to this development, it has been possible to carry out a considerable number of research studies in the sports field, contributing to a better performance development of athletes and, in some cases, predicting and preventing the risk of injury to athletes [10].

One of the areas of sport where there is a greater lack of performance assessment systems is in martial arts, which includes Taekwondo [11]. The assessment of athletes’ performances is fundamentally based on the analysis of the movements performed by the athletes. Considering the plurality and complexity of the movements, which make the task of evaluating athletes’ performance demanding, it is of greater value to use technological tools that are able to automate this responsibility of the coach during training. This represents an evolution in training methods that contributes to a faster and more effective response to the need to correct the trainee’s body movements and allows for more effective adaptation and correction of the athlete’s movements [12].

A significant number of studies carried out result from the acquisition of images and videos, in part, due to technological developments in this area [13,14]. Evolution does not always result in positive outcomes, as the enhanced image and video quality of modern devices can make it challenging to capture well-balanced images and videos due to their increased sensitivity to brightness and color variations in the environment [15]. This realization has led to the creation of a new type of camera that utilizes depth sensors to extract information from images. These cameras, known as 3D cameras because of their recording technique, are capable of capturing depth images regardless of environmental colors, lighting orientation, or intensity [16]. One example of such 3D cameras is the Microsoft Kinect (Microsoft, Redmond, WA, USA), which has been utilized in research to classify gestures and track hand movements [17]. This innovation has simplified the process of recording and accessing data in research settings due to its portability, ease of setup, and ability to generate 3D images without the need for markers. With its unique features, this 3D camera can be seamlessly integrated into the development of 3D video motion systems, enabling the analysis of human movement kinematics and the identification of joints and body segments with precision.

Considering Microsoft Kinect, in their study, Zerpa et al. compared the values obtained from the displacement measurements of the Microsoft Kinect with the values obtained from the displacement measurements of another system based on peak motus markers [18]. An analysis of the obtained results showed that the Microsoft Kinect camera effectively combines the characteristics of simplicity in configuration, data acquisition, and analysis compared to the Peak Motus system, as it is a system that does not use markers.

Among the experiments conducted with the Microsoft Kinect camera, Patsadu et al. employ it for human motion recognition by using data mining classification methods in video streaming of twenty human body joint positions. For the research, a system was developed that recognizes gesture patterns such as standing up, sitting down, and lying down. Neural networks with backpropagation, decision trees, support vector machines, and Naive Bayes were used as classification methods for the comparison. The classification approach with the best performance was based on backpropagation neural networks, as it achieved 100% accuracy in recognizing the human gestures studied [19].

Another study, which aimed to introduce a methodology to evaluate the performance of Taekwondo athletes in real time, also used Microsoft Kinect. To achieve this objective, image processing techniques were used to detect, identify, and monitor the frequency of movements of Taekwondo athletes in a training environment [20]. The method used was to accurately identify the movements considering the angles formed between the joints of the human body. These angles are then calculated and compared with the reference values for each movement previously stored in a database.

A recent study presented a new approach to evaluate the real-time performance of Taekwondo athletes during their training sessions using 3D cameras [21]. In the study, the Orbbec Astra 3D camera (Orbbec, Troy, MI, USA) was used due to its unique characteristics compared to the Microsoft Kinect 2 (Microsoft, Redmond, WA, USA). The Orbbec Astra was preferred due to its compact size, lighter weight, and extended range. The result is a system that can store valuable information about the athletes and their movements. The data collected include Cartesian coordinate values in the physical domain, providing detailed information about the joints of the human body. The provision of numerical Cartesian coordinates together with the graphical representation of Cartesian coordinate lines and velocity lines in real time provides comprehensive information about the movements of the athlete’s wrists and ankles and thus valuable information for the athlete and the coach.

As already mentioned, the Microsoft Kinect 3D camera is used extensively in studies on the kinematic analysis of human movements. However, further studies are needed to consolidate and ensure its functionality and reliability in this field [22]. In addition, some studies have highlighted the contribution and development of systems that do not use markers to obtain information for a revolution in human motion analysis, predicting a greater application in human motion capture studies [23,24].

Although they comprise an increasingly used method, 3D cameras are not used as a means of data collection in all motion analysis studies. Some studies take a different approach to obtaining data on the movements performed by people and instead develop systems that use motion sensors. These sensors allow the collection of specific data from specific locations on a person’s body when placed and positioned for this purpose [25]. This implementation perspective provides a motion analysis method that can be useful in analyzing the performances of athletes. Despite the results obtained in some studies, these systems still have limitations that need to be improved according to the literature [26].

Not all studies attempt to analyze the movements of a person’s entire body; some focus on specific body parts. This is the case of Suarez and Murphy, who present a case study in which the movements and position of the hands are analyzed by using different techniques to classify gestures and the position of the hand [27].

Other authors use magneto-inertial technology, as they consider this technology to be a reliable and efficient method of improving athlete performance. They also believe that this technology can be effective in preventing injuries and improving the specificity of training in relation to an athlete’s profile, as the sensors used enable the estimation of temporal, dynamic, and kinematic parameters [28].

Other studies on motion analysis focus exclusively on identification or performance analysis. Some studies also look at the influences that athletes are exposed to during their sporting activities [29]. These scientists use smart sensors and sensor fusion technology to collect valuable data for analysis. These cutting-edge systems are specifically designed for use in the biomedical and sports fields and use advanced techniques and methods to analyze physical data about the human body. They have the ability to bring about revolutionary changes in various areas, such as rehabilitation and improving athletic performance.

In their study, the authors used IMUs as part of their motion sensor approach [30]. The study confirmed that the combination of impact signals with IMUs can be a reliable assessment method. Heart rate monitoring was also included to monitor the physical condition of the athletes. The approach was to integrate a “non-invasive” sensor system into the Taekwondo athletes’ clothing. Pressure sensors and thin-film piezo resistors for force measurement and accelerometers were used to measure impact. Bluetooth technology was chosen as the communication method between the sensors and the computer, but this has a bandwidth limitation during transmission [31].

The SPERTA cyber–physical system not only uses a 3D camera to capture data on the athletes’ movements, but also uses IMUs attached to the athletes’ extremities (wrists and ankles). The use of these IMUs, which combine an accelerometer and a gyroscope, enables the collection of additional movement data, but is mainly aimed at compensating for data loss that occurs during rotational movements due to the locking of the athlete’s skeletal joints. This contributes to greater data consistency, which is necessary for correct and efficient analysis [6].

2.2. The SPERTA System

This study is based on a system specifically designed to evaluate the performance of Taekwondo athletes in real time. In order to meet the requirements defined for the work, a system architecture has been designed that combines a series of elements with specific characteristics whose objective is to create a tool capable of performing what is proposed.

The SPERTA cyber–physical system is an innovative initiative that aims to develop a real-time assessment system for evaluating the performance of Taekwondo athletes. Its main objective is to provide coaches with a reliable and affordable system to evaluate and analyze athletes’ performance in real time. The purpose of this work is to improve training techniques and the overall development of Taekwondo practice.

The overall system, developed as part of SPERTA, consists of a 3D camera, the Orbbec Astra with RGB (red, green, blue) and depth sensors [32], inertial measurement units (IMUs) with an accelerometer and a gyroscope, and a computer for data acquisition and processing. This setup makes it possible to obtain detailed information such as the Cartesian coordinates and speed of the hand and foot movements performed by the athletes and to provide coaches with accurate feedback to correct or improve an athlete’s technique by displaying the data in real time.

The SPERTA system aims to improve training and assessment methods for Taekwondo martial art athletes using technology and movement analysis. The system allows users to capture and analyze movements without the need for markings on the human body, optimizing it for better monitoring and improvement of the athlete’s performance under normal training conditions. Allowing users to identify and record the movements performed, the system displays the values of speed, acceleration, and force of the athlete’s upper and lower limbs in real time. In order to achieve that goal, it collects data on the athlete’s movements, which are analyzed and processed in order to present relevant information in real time. This information consists of the following elements: values of speed, acceleration, and strength of movements; accounting and identification of movements performed by the athlete during the training session; storage, visualization, and export of training data performed by the athlete. The availability of these data in real time, or at any time by accessing the system, allows the coach and athlete to quickly adjust training methods, enabling a rapid evolution of the athlete’s performance. This does not happen using traditional methodologies that depend on visual analysis, sometimes later through video analysis, which is more prone to errors.

Thus, the equipment of the system consists of a computer, preferably portable, compatible with the Windows operating system [33], and a 3D camera sensor. The Orbbec Astra camera [32] was utilized for this research. The computer was equipped with specially designed software for the SPERTA system, enabling data collection and processing. Additionally, the system incorporated IMUs that were specifically developed for this purpose. These IMUs consisted of an ESP8266 wireless microcontroller 802.11 (Wi-Fi) microcontroller development board [34], a GY 521 MPU 6050 accelerometer and gyroscope [35], and a 3.7 V 190 mAh Li-Po battery (refer to Figure 2 and Figure 3).

The development of the SPERTA cyber–physical system was driven by the consideration of cost-effectiveness and practicality in a real-world setting. It was crucial for the system to be minimally intrusive for Taekwondo athletes. To achieve this, careful attention was given to selecting hardware components that were readily available and designing a system that was both practical and user-friendly. This encompassed ensuring a seamless user experience and creating comfortable and lightweight equipment to facilitate ease of use. The positioning of the sensors on the athletes can be observed in Figure 4 and Figure 5.

The equipment utilized by athletes comprises a compact board built on ESP8266 with Wi-Fi capabilities, along with an accelerometer and gyroscope module, a battery, and a power management module (Figure 2). To ensure a systematic arrangement of these components, a container was devised. This container is affixed to the athletes’ wrists and ankles using an elastic band equipped with Velcro fasteners (Figure 4 and Figure 5).

The core of the SPERTA system, which is also the main contribution and innovation of this work—as it is what allows all components to contribute to the correct and efficient functioning of the system—is the software. This has been developed from scratch, enabling the integration of all system components in terms of communication, collection, storage, analysis, and the presentation of data to the user. It also provides an accessible and intuitive user interface for athletes and coaches to interact with the system, adding and/or receiving information, through an application for Windows operating systems. This interface provides the necessary operations and functionalities, thus allowing information to be presented and interpreted correctly.

The SPERTA system was developed with the aim of creating a tool capable of evaluating the performance of Taekwondo athletes in real time. This system is made up of different elements, with different functionalities, which together allow the collection, storage, analysis, and availability of data from athletes’ training sessions. These elements are a 3D camera, motion sensors, and a computer. The 3D camera used is the Orbbec Astra with RGB camera and depth sensor, to capture the athletes’ movements during training. The motion sensors, developed for the system, consist of an ESP8266 board with Wi-Fi along with a GY 521 MPU 6050 sensor (accelerometer and gyroscope) and a 3.7 V 190 mAh Li-Po Battery. We have considered a total of four motion sensors, as they are placed on the extremities of the upper and lower limbs of the Taekwondo athletes. The software was developed from scratch in C# with Visual Studio 2017 IDE; this is a key component of the framework, as it allows the aggregation of all elements. Integrating the Nuitrack™ SDK enables 3D human skeleton tracking; more specifically, it tracks the various joints and provides the coordinates of these joints in a three-dimensional environment. The software delivers and stores athletes’ movement data, performing and presenting the calculation of speed, acceleration, and force values. To perform this task, the framework software receives data from the 3D camera and movement sensors and analyzes the data received, which are saved and made available to athletes and coaches via the application interface, offering valuable feedback for evaluating athletes’ performances.

The SPERTA system’s characteristics are carefully outlined to offer a technological solution aimed at assisting coaches in minimizing the duration required for evaluating an athlete’s performance, thereby facilitating a swifter enhancement of the athlete’s proficiency. By providing instantaneous feedback and precise data, this system empowers coaches to make well-informed choices and modify their training regimens to steer athletes towards more effective training methods, ultimately rapidly enhancing their performance. Consequently, it plays a pivotal role in advancing Taekwondo training and performance evaluation through the integration of technology, motion analysis, and real-time feedback.

Nevertheless, in order to enhance the structure of navigation and data representation, and to accommodate the system’s expanded functionalities, a novel interface was devised with the intention of integrating it into the system at a later stage (Figure 6).

3. Study on the System’s Acceptability

To guarantee precise and uniform testing and data gathering, a set of guidelines was implemented to delineate the procedures for carrying out the tests. This set of guidelines primarily emphasizes the essential equipment and its organization within the testing setting, while also specifying the particular data to be collected and the individuals responsible for the data collection process. By adhering to this set of guidelines, a standardized test configuration was established, ensuring consistency during the execution of the studies.

To meet the equipment requirements, a 3D camera was positioned at a height of one meter above the ground. The athletes were positioned in a profile stance, facing the 3D camera from the right side, and were positioned at a distance of three meters from the camera. Careful attention was given to organizing the space and ensuring appropriate lighting conditions to facilitate accurate data capture. Additionally, to ensure the comfort of the athletes and minimize any interference with their movements, inertial measurement units (IMUs) were strategically placed on their limbs, wrists, and ankles.

It is worth noting that this protocol adheres to the General Regulation on Personal Data Protection of the European Union (GDPR)—Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 [36].

Considering the unique characteristics of Taekwondo, which involves a wide range of movements with a focus on leg techniques, four specific movements were selected for data collection. Due to the large number of technical movements performed in the practice of Taekwondo, four were selected as representatives of these movements. These movements, presented in

Table 1. Taekwondo movements collected.

Jirugui: Frontal punch.
AP Tchagui: One of the legs in the form of a front kick.
Bandal Tchagui: Side kick at face height
Miro Tchagui: Pushing front kick.

, were the Jirugui (punching technique), the Ap Tchagi (front kick), the Bandal Tchagui (side kick), and the Miro Tchagui (pushing front kick).

The participants’ ages spanned from fourteen to forty years old, with varying levels of experience in Taekwondo ranging from one to twenty-five years. A gender distribution of 50% was identified. Moreover, the participants held different levels of graduation, such as DAN (black belt), 2nd KUP (red belt), 3rd KUP (blue belt), 8th KUP (yellow belt), and 9th KUP (white with yellow strip). Despite the small number of athletes, the diversity in experience among them ensured that meaningful results could be achieved.

Before the data collection sessions began, a testing procedure was followed. Athletes were required to provide informed consent, ensuring that they fully understood the nature of their participation. Athletes also received a brief demonstration and explanation of the movements that they would be performing. This was performed to guarantee their correct understanding and active participation. Additionally, adequate warm-up time was provided to the athletes prior to their participation in the data collection sessions

In order to ensure accurate data collection, each athlete performed each of the movements separately, paying maximum attention to maintaining proper posture and technique. To ensure sufficient data, each athlete performed 20 repetitions of each of the selected movements for the study. Between each movement, the athlete rested for 3 min in order to avoid exhaustion, thus ensuring efficient data collection.

3.1. System Data Collection

Data collection plays a vital role in any research study, serving as the fundamental basis for acquiring precise and dependable results.

The data collection sessions were conducted at the training location of the involved Taekwondo athletes. This choice was made to ensure that the system was utilized in a real environment, thereby maximizing the accuracy of the obtained data. Additionally, the utmost consideration was given to minimize any disruption to the athletes’ training routine.

The collected data consist of the values of the Cartesian coordinates of the athletes’ joints, as well as the linear acceleration values of the accelerometer and the angular rotation rate of the gyroscope of the movements performed. So, the input to the deep learning module consists of a set of information on the position of the X, Y, and Z axes (three-dimensional coordinates) of each of the human athletes’ joints, such as knees, ankles, elbows, and wrists. These data are provided by the 3D camera’s depth sensor and developed motion sensors. They are collected by the SPERTA application, developed for the system, which stores them in a database and sends them to the deep learning module to count and identify the athletes’ movements. In Figure 7, a graphical representation of the data that are provided to the deep learning network is presented. The set of frames, the X, Y, and Z coordinates, represents the movement of the right-hand joint during Jirugui movement throughout a sequence of 80 samples. However, to classify the athlete’s movements, data from other joints are considered.

To determine which deep learning methodology would be most suitable for the data typology, preliminary tests were carried out using convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long–short-term memory networks (LSTM), as each of the neural networks has different features regarding the method of analysis and data output. To carry out the tests, each of the models were composed of a layer of 100 units and then a dropout rate of 0.5. Below is a brief description of the features of each of the neural networks studied:

• Convolutional neural networks (CNNs) are feedforward neural networks that use filters and pooling layers and do not have recurrent connections. They are suitable for working with images and videos, for example for facial recognition, medical analysis, and image classification [37].
• Recurrent neural networks (RNNs) are a type of artificial neural network used to analyze time series data; they use recurrent connections to generate output. They are suitable for speech recognition, natural language processing, and tasks such as text summarization, machine translation, and speech analysis. RNNs, due to their architecture, can suffer from problems such as gradient disappearance and explosion [38].
• Long–short-term memory (LSTM) networks are a type of RNN with greater memory power to remember the outputs of each node for a longer period, thus combating the disappearance of gradients or the long-term dependency problem of RNNs. LSTM networks have gates that update and regulate cell states, such as the forget gate, which determines what information in the cell state should be forgotten, helping RNNs remember the critical inputs needed to generate the correct output [39].

After carrying out tests with the described neural networks, the results obtained allowed us to determine that the LSTM model obtained better results. Having been applied in the study, the LSTM model was composed of a first LSTM layer with 100 units, then a dropout of 0.5 rate, followed by action layer with the ReLU function, and, at the end, a Softmax layer with the probabilities per class. This model, due to its analysis features, is better suited to the data under analysis, which is made up of a set of values (frames) over a period of time. The number of frames provided to the deep learning module may vary. This is due to the fact that different movements may have different durations for their execution. Considering this, later, during preprocessing, the data are resampled to a defined static size of 80 frames before inference in the deep learning neural network. The end and beginning of the movements are defined by the software with the acceleration (and deceleration) readings of the athlete’s joints. In other words, when a large variation in acceleration is detected, joint data begin to be collected, and when there is no movement for a certain period, this movement is defined as the end of the movement to be classified. Then, these data will be sent to the deep learning module to be preprocessed, and inferences are made.

Therefore, to evaluate the quality and execution of Taekwondo techniques, the SPERTA system employs deep-learning-based action recognition algorithms. For the system to analyze the temporal dynamics and movement patterns of athletes, LSTM neural networks are used as they allow the SPERTA system to accurately classify and evaluate different Taekwondo movements.

3.2. System Acceptability Survey

The developed system for athletes aims to minimize interference with their training routines and movements, ensuring minimal intrusion. Throughout the development process, careful attention was given to adapting the constituent elements of the framework, including the software interface and movement sensors. These elements will be the primary points of contact and interaction for athletes and coaches when using the system.

This research seeks to determine whether the developed system fulfills its intended purpose of being user-friendly for athletes and coaches. To assess the satisfaction and fulfillment of expectations regarding the system’s usage, an online survey was conducted. Athletes and coaches anonymously completed this survey after using the most up-to-date version of the system [40].

Usability testing involves the observation of a specific user group that represents the platform’s users. Users are requested to complete tasks on a designated interface while a moderator analyzes their actions. The primary goal is to detect any issues in a product or service, enhance the interface, and gain a better understanding of the users’ requirements [41]. Participants are given tasks to complete while their interactions are closely monitored, enabling necessary adjustments to be made before the final development phase [42]. This method underscores the importance of user experience as a fundamental element for the growth and success of a platform. A positive user experience not only fosters user confidence but also encourages sustained usage. The system usability scale (SUS), created by John Brooke, is widely acknowledged as one of the most commonly used techniques for assessing usability. This method involves administering a post-test questionnaire to participants following a usability testing session. The questionnaire comprises 10 questions that are rated on a Likert scale, with scores ranging from 0 to 100 [43]. By calculating the total points obtained by each user from the questionnaire, their usability score can be determined. This score is then interpreted using Brooke’s scaling scheme (Figure 8). According to Jeff Sauro’s research, a score of at least eighty points indicates a highly efficient platform or website [44]. Over the past thirty years, this approach has undergone extensive testing and has consistently demonstrated to be a dependable and effective method for evaluating interface usability [45].

Jeff Sauro has determined that a platform or website must achieve a minimum of eighty points in order to be considered effective in terms of usability [46]. This approach, which has been in use for three decades, has demonstrated its effectiveness and reliability in assessing interface usability.

The questionnaire underwent validation using Cronbach’s alpha, a metric of internal consistency that gauges the degree of interrelatedness among a set of items for a given group. By evaluating response consistency across multiple items in a survey or assessment tool, it ensures that the items effectively capture the same construct. This method is commonly employed in research to evaluate the reliability of a scale or research instrument. A high Cronbach’s alpha value signifies that the items on the scale consistently measure the same underlying construct. The value ranges from 0 to 1, with higher values indicating increased reliability [47].

Upon conducting Cronbach’s alpha analysis on the questionnaire, a coefficient of 0.70 was obtained for the closed questions, while a coefficient of 0.84 was calculated for the open questions. This substantial alpha value indicates that the survey items effectively gauge the acceptability and usability of the system in study. The strong internal consistency demonstrated by Cronbach’s alpha further strengthens the reliability and validity of the research instrument, instilling confidence in the insights gathered for decision making and ongoing improvement efforts.

Regarding the analysis of open-ended questions, the qualitative data obtained from the free-response questions were analyzed using thematic analysis. Thematic analysis is a widely used method for identifying patterns and themes in qualitative data, enabling researchers to systematically organize and interpret textual data. This approach allows us to gain deeper insights into the experiences, perceptions, and suggestions of participants regarding the system [48].

The online survey was constructed and made accessible through the utilization of the Google Forms tool. It comprises a total of seventeen questions, with seven being closed-response questions and ten being free-response questions. The closed questions employed a well-established five-point Likert scale, which was initially developed in 1932 and is currently widely acknowledged. This particular scale enables individuals to express their level of agreement, disagreement, approval, or disapproval across various categories. The closed questions in the survey implemented the five-point Likert scale, ranging from “Totally disagree” to “I totally agree”. The specific questions presented to the athletes in the survey can be found in

Table 2. List of questions applied in the form for athletes and coaches.

Ner	Question
1	Age?
2	Gender?
3	Experience in Taekwondo (in years)?
4	Indicate your degree (Taekwondo belt)?
5	The system configuration and installation process were simple?
6	The system interface was user-friendly and easy to navigate?
7	The instructions provided were clear and understandable?
8	The system accurately captured and recorded my movements?
9	The 3D camera sensor was reliable in capturing motion data?
10	Conditions during testing were adequate?
11	Overall, I found the system easy to use?
12	Do you consider specific movements (Jirugui, Ap Tchagi, Bandal Tchagui, Miro Tchagui) suitable for evaluating the performance of Taekwondo athletes?
13	Did you encounter any difficulties or challenges when performing the movements? If yes, please specify.
14	How confident are you in the system’s accuracy in capturing your movements?
15	Do you consider that the testing environment (location, configuration, etc.) provided an accurate performance assessment? If not, please explain.
16	Do you have any suggestions or improvements that you would like to propose for the testing system or protocol?
17	Overall, how would you rate your experience using the system and applying the test protocol?

, following a predetermined sequence.

4. Results

Ten responses were collected after distributing the form with questions to the athletes and coaches of the Taekwondo Academy at the University of Minho (Portugal). These responses formed the group that was considered for the case study. Among the ten individuals in the group, there was an equal distribution of 50% between genders. The age range of the participants varied between fourteen and forty years old, with experience in Taekwondo ranging from one to twenty-five years.

Additionally, the group had individuals with different levels of graduation, including DAN (black belt), 2nd KUP (red belt), 3rd KUP (blue belt), 8th KUP (yellow belt), and 9th KUP (white with yellow strip). Despite the small number of athletes, the group had a diverse range of experience levels, which was essential for ensuring the significance of the study results.

After analyzing the survey questions, graphs were generated to visually represent the obtained results (Figure 9, Figure 10 and Figure 11). This presentation of the survey results allows for a preliminary interpretation of the data. By observing the graphs, it can be inferred that the number of positive responses for each question is significantly high.

However, a thorough analysis based on appropriate scientific criteria and methodologies for each item of the survey is necessary. This was carried out and presented in the following section.

5. Discussion

The analysis of the closed-ended responses, which included seven questions, revealed that the category “Totally Disagree” did not receive any responses, while the option “I disagree” obtained only 3% of the responses. On the other hand, the category “I do not agree nor disagree” accumulated 9% of the responses, while the options “I agree” and “I totally agree” received 37% and 51% of the responses, respectively (Figure 9). The analysis of the responses by question revealed that the majority of participants agreed or strongly agreed with the statements presented. For example, 60% of the participants agreed that the system’s configuration and installation process was simple, while 40% strongly agreed with this statement. Furthermore, 70% of the participants agreed that the provided instructions were clear and understandable, and 60% agreed that the system accurately captured their movements (Figure 10).

The set of open-ended questions comprises ten questions, with the initial four questions serving the purpose of delineating the cohort of athletes who participated in the online survey. The remaining open-ended questions afford athletes the opportunity to articulate their viewpoints and, if they so desire, propose alterations and enhancements to the system or its implementation methodology.

Thematic analysis was utilized to analyze the open-ended questions in the questionnaire, as it is a method that facilitates both inductive and deductive approaches. This method proves to be beneficial in converting qualitative survey data into a binary interpretation by identifying common themes and categorizing responses as either yes or no. By employing coding and theme development, this method allows for the establishment of clear criteria to determine yes or no responses, ultimately enhancing the reliability and validity of the interpretation [49].

Based on the results obtained in this section of the form (Figure 11), it is feasible to establish the following conclusions. The specific Taekwondo movements evaluated (Jirugui, Ap Tchagi, Bandal Tchagui, Miro Tchagui) were considered suitable by 100% of the participants for assessing athletes’ performance. These movements are fundamental in the discipline and allow for a comprehensive evaluation of athletes’ technical skills. The majority of participants (75%) did not encounter difficulties or challenges when performing the movements with the equipment used. However, some suggested that the equipment be adapted to better fit the arms and legs, and that applying maximum force without a target could be uncomfortable. All participants (100%) expressed confidence in the system’s accuracy in capturing their movements. It is believed that the accuracy will further improve with the continuous development of the program. The testing environment was considered suitable by 100% of the participants, providing appropriate conditions for accurate performance assessment. However, it is suggested that the use of a target bag could further enhance the testing protocol. The majority of participants (75%) proposed improvements, such as more compact and lightweight sensors attached to the extremities to prevent them from coming off during training, and the possibility of using the system in competitive environments. Overall, the experience of using the system and applying the testing protocol was positively evaluated by all participants (100%). The project was considered promising, useful for improving athletes’ performance in various disciplines, and intuitive to use.

Considering the value of 0.70 for alpha Cronbach in the closed-ended questions and the value of 0.84 for alpha Cronbach in the open-ended questions of the applied questionnaire, and incorporating the values obtained in the analysis of the set of closed-ended questions, it can be inferred that the level of acceptance and usability in the use of the developed system is quite positive; this aligns with the defined objective for the system as a tool that is capable of being used to evaluate the performance of Taekwondo athletes in the training environment. The suggested improvements, such as equipment adaptations and the possibility of use in competitions, can further enhance the system. It is important to conduct future studies with larger samples and in different scenarios in order to obtain better validation.

6. Conclusions and Future Work

The current study has demonstrated the effectiveness of the SPERTA system, a cyber–physical system specifically designed to assess the real-time performance of Taekwondo athletes during their training sessions. By conducting a survey among athletes and coaches who used the system, it was possible to evaluate its acceptance and usability. The analysis of survey responses indicated a high level of satisfaction among participants. The majority agreed or strongly agreed with the statements presented on the Likert scale, showing approval regarding the system’s configuration, interface usability, clarity of instructions, accuracy of motion capture, reliability of 3D camera sensors, and overall ease of use. Open-ended response questions provided additional insights into the participants’ experiences. Athletes and coaches emphasized acceptance, ease of use, and overall satisfaction with the system. They expressed confidence in the system’s ability to assess performance in Taekwondo and suggested the inclusion of additional movements for evaluation purposes. Regarding their experience with the system, they reported minimal challenges during movements and considered the system to be a promising tool for performance assessment. The comprehensive methodology of the study, combining multiple-choice and open-ended questions, enabled a deeper understanding to be built of athletes’ experiences with the developed and implemented system. High levels of satisfaction, along with constructive feedback, indicate a favorable reception of the system, highlighting its potential as a valuable tool for assessing performance in Taekwondo in a training environment. Furthermore, there is expressed interest in using the system in competitive settings.

As future work, the intention is to enhance the sensor fixation system, making it more comfortable and less intrusive for athletes. Additionally, miniaturizing the sensors to further reduce their impact on athletes’ performance is desired. Furthermore, conducting a longitudinal study to assess the system’s impact on athletes’ performance evolution over time is planned. Moreover, testing the system on a larger number of athletes and in different scenarios to obtain a more robust validation is intended. Additionally, exploring the application of the system in competitive environments, evaluating its feasibility and acceptance in this context, is also a goal.

By implementing these future works, it is expected to further improve the SPERTA system, making it an even more effective and widely adopted tool for the evaluation and enhancement of Taekwondo athletes’ performance.