A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions

Abstract

Recent developments in video analysis of sports and computer vision techniques have achieved significant improvements to enable a variety of critical operations. To provide enhanced information, such as detailed complex analysis in sports such as soccer, basketball, cricket, and badminton, studies have focused mainly on computer vision techniques employed to carry out different tasks. This paper presents a comprehensive review of sports video analysis for various applications: high-level analysis such as detection and classification of players, tracking players or balls in sports and predicting the trajectories of players or balls, recognizing the team’s strategies, and classifying various events in sports. The paper further discusses published works in a variety of application-specific tasks related to sports and the present researcher’s views regarding them. Since there is a wide research scope in sports for deploying computer vision techniques in various sports, some of the publicly available datasets related to a particular sport have been discussed. This paper reviews detailed discussion on some of the artificial intelligence (AI) applications, GPU-based work-stations and embedded platforms in sports vision. Finally, this review identifies the research directions, probable challenges, and future trends in the area of visual recognition in sports.

1. Introduction

Automatic analysis of video in sports is a possible solution to the demands of fans and professionals for various kinds of information. Analyzing videos in sports has provided a wide range of applications, which include player positions, extraction of the ball’s trajectory, content extraction, and indexing, summarization, detection of highlights, on-demand 3D reconstruction, animations, generation of virtual view, editorial content creation, virtual content insertion, visualization and enhancement of content, gameplay analysis and evaluations, identifying player’s actions, referee decisions and other fundamental elements required for the analysis of a game.

The task of player detection (identification) and tracking is very difficult because of many challenges, which include the similar appearance of subjects, complex occlusions, an unconstrained field environment, background, unpredictable movements, unstable camera motion, issues with calibration of low textured fields, and the editing performed for broadcasting video, lower pixel resolution of players who are distant and smaller in the frame, and motion blur, among others. The simultaneous detection of players and ball and tracking them at once is quite challenging, because of the zigzag movements of the ball and player, change of the ball from player to player, severe occlusion between players and the ball. Hence, this paper presents a survey of detection, classification, tracking, trajectory prediction and recognizing the team’s strategies in various sports. Detection and tracking of players is the only major requirement in some sports such as cycling and swimming. Hence, this study presents a survey of detection, classification, tracking, trajectory prediction and recognizing the team’s strategies in various sports. Detection and tracking of the player is the only major requirement in some sports such as cycling, swimming, among others. As a result, as illustrated in Figure 1, this research classifies all sports into two categories: player-centered and ball-centered sports, with extensive analysis in Section 4.

Figure 1. Classification of different types of sports.

Recent developments in video analysis of sports have a focus on the features of computer vision techniques, which are used to perform certain operations for which these are assigned, such as detailed complex analysis such as detection and classification of each player based on their team in every frame or by recognizing the jersey number to classify players based on their team will help to classify various events where the player is involved. In higher-level analysis, such as tracking the player or ball, many more such evaluations are to be considered for the evaluation of a player’s skills, detecting the team’s strategies, events and the formation of tactical positions such as midfield analysis in various sports such as soccer, basketball, and also various sports vision applications such as smart assistants, virtual umpires, assistance coaches, have been discussed in Section 7. A higher-level semantic interpretation is an effective substitute, especially in situations when reduced human intervention and real-time analysis are desired for the exploitation of the delivered system outputs.

The main task of video summarization or highlight extraction is extracting key events of the game which provides users with an ability to view highlights as per their interests. For this purpose, it is necessary to detect and classify gestures, recognize the actions of the referee/umpire, track players and the ball in key events like the time of goal scoring to analyze and classify different types of shots performed by players. The framework for processing and analyzing task-specific events in sports applications, such as playfield extraction, detection, and tracking of the player/ball is shown in Figure 2, and detailed analyses of playfield extraction are discussed in Section 3.

Figure 2. Framework of processing and analysis of different applications in sports video.

A detailed review of research in the above-mentioned domains is presented in this article and the data were compiled from papers that focus on computer vision-based approaches that were used for each application, followed by inspecting key points and weaknesses, thereby investigating whether these methodologies in their current state of implementation can be utilized in real-time sports video analysis systems.

Features of the Proposed Review

Some of the surveys and reviews published in different sports video processing and their main contributions, i.e., whether the paper discussed (marked as ‘✔’ if discussed in Table 1) hand-crafted and machine learning algorithms, the type of sport, different tasks in sports, whether it provided datasets or not and finally the aim of the review are discussed and summarized in Table 1 and listed below.

• Tan et al. [1] researched badminton movement analysis such as Badminton smashing, badminton service recognition, badminton swing, and shuttle trajectory analysis.
• Bonidia et al. [2] presented a systematic review of sports data mining, which discusses the current research body, themes, the dataset used, algorithms, and research opportunities.
• Rahmad et al. [3] presented a survey on video-based sports intelligence systems used to recognize sports actions. They provided video-based action recognition frameworks used in the sports field and also discussed deep learning implementation in video-based sports action recognition. They proposed a flexible method that classifies actions in different sports with different contexts and features as part of future research.
• Eline and Marco [4] presented a summary of 17 human motion capture systems which reports calibration specs as well as the specs provided by the manufacturer. This review helps researchers to select a suitable motion capture system for experimental setups in various sports.
• Ebadi et al. [5] presented a survey on state-of-the-art (SOTA) algorithms for player tracking in soccer videos. They analyzed the strengths and weaknesses of different approaches and presented the evaluation criteria for future research.
• Thomas [6] proposed an analysis of computer vision-based applications and research topics in the sports field. The study summarized some of the commercially available systems such as camera tracking and player tracking systems. They also incorporated some of the available datasets of different sports.
• Cust et al. [7] presented a systematic review on machine learning and deep learning for sports-specific movement recognition using inertial measurement units and computer vision data.
• Kamble et al. [8] presented an exhaustive survey on ball tracking categorically and reviewed several used techniques, their performance, advantages, limitations, and their suitability for a different sports.
• Shih [9] focused on the content analysis fundamentals (e.g., sports genre classification, the overall status of sports video analytics). Additionally reviewed are SOTA studies with prominent challenges observed in the literature.
• Beal et al. [10] explored AI techniques that have been applied to challenges within team sports such as match outcome prediction, tactical decision making, player investments, and injury prediction.
• Apostolidis et al. [11] suggested a taxonomy of the existing algorithms and presented a systematic review of the relevant literature that shows the evolution of deep learning-based video summarization technologies.
• Yewande et al. [12] explored a review to better understand the usage of wearable technology in sports to improve performance and avoid injury.
• Rana et al. [13] offered a thorough overview of the literature on the use of wearable inertial sensors for performance measurement in various sports.

Table 1. Summary of previous surveys and reviews in different sports.

Articles	Hand Crafted Algorithms	Machine Learning Algorithms	Sport and Application				Discussion about Dataset	Aim of Review
			Sport	Detection	Tracking	Classification and Movement Recognition
[1]	✗		Badminton	✗	✗	✔	✗	Motion analysis
[2]	✗	✔	-	✔	✗	✔	✗	Sport data mining
[3]	✗	✔	-	✔	✗	✔	✔	-
[7]	✗	✔	-	✗	✗	✔	✗	-
[4]	✗	✗	-	✗	✗	✔	✗	Motion Capture
[8]	✔	✔	Soccer	✔	✔	✗	✗	Ball Tracking
[5]	✗	✔	Soccer	✔	✔		✗	Player detection/tracking
[6]	✗	✔	-	✔	✔	✗	✔	Availability of datasets for sports
[9]	✗	✗	-		✗		✗	Content-Aware Analysis
[10]	✗	✔	-	✗	✗	✔	✗	-
[11]	✗	✔	-	✗	✗	✗	✔	Video Summarization
[12]	✔	✗	-	-	-	-	✗	Wearable technology in sports
[13]	✔	✗	-	-	✔	✔	✗	Wearable technology in sports

The proposed survey mainly focuses on providing a proper and comprehensive survey of research carried out in computer vision-based sports video analysis for various applications such as detection and classification of players, tracking players or balls and predicting the trajectories of players or balls, recognizing the team’s strategies, classifying various events on the sports field, etc. and in particular, establishing a pathway for next-generation research in the sports domain. The features of this review are:

• In contrast to recently published review papers in the sports field, this article comprehensively reviews statistics of studies in various sports and various AI algorithms that have been used to cover various aspects observed and verified in sports.
• It provides a roadmap of various AI algorithms’ selection and evaluation criteria and also provide some of the publicly available datasets of different sports.
• It discusses various GPU-based embedded platforms for real-time object detection and tracking framework to improve the performance and accuracy of edge devices.
• Moreover, it demonstrates various applications in sports vision and possible research directions.

The rest of this paper is organized as follows. Section 2 provides statistical details of research in sports. Section 3 presents extraction data vis-a-vis various sports playfields, followed by a broader dimension that covers a wide range of sports and is reviewed in Section 4. Some of the available datasets for various sports along and embedded platforms have been reviewed in Section 5 and Section 6. Section 7 provides various application-specific tasks in the field of sports vision. Section 8 covers potential research directions, as well as different challenges to be overcome in sports studies. Last but not least, Section 9 concludes by describing the final considerations.

2. Statistics of Studies in Sports

Detection of the positions of the players at any given point of time is the basic step for tracking a player, which is also needed for graphics systems in sports for analysis and obtaining pictures of key moments in a game. Equipment and methods used in commercial systems for broadcast analysis vary from those depending on a manual operator clicking on the players’ feet with a calibrated camera image to an automated technique involving segmentation and identification of areas which likely correspond to players. For the performance improvement of sport teams in soccer, volleyball, hockey, badminton among others, analyzing the movements of players individually, and the real-time team formation, may provide a crucial real-time insight for the team coach.

The research articles discussed in this review were obtained from various reputed publishers such as IEEE, Elsevier, MDPI, Springer among many others, and top-tier computer vision conferences such as CVPR, ICCV, and ECCV, ranging from high impact factor online sources in the domain of player/ball/referee detection and tracking in sports, classification of objects in sports, behavior, and performance analysis of players, gesture recognition of referees/umpires, automatic highlight detection, score updating among others. Figure 3 provides overall information of sports research publications in the past five years considered in this comprehensive survey article.

Figure 3. Sports research progress in past five years.

Figure 4 provides the statistics of studies of various sports in various applications such as detecting/tracking the player and ball, trajectory prediction, classification, and video summarization. which are published in various standard journals as presented in Figure 3.

Figure 4. Sports wise research progress.

3. Play Field Extraction in Various Sports

Detection of the sports field plays an important role in sports video analysis. Detection of the playfield region has two objectives. One is to detect the playfield region from non-playfield areas, while the other is to identify primary objects from the background by filtering out redundant pixels such as grass and court lines. This provides a reduced pixel which requires processing and reduction in errors to simplify player or ball detection and tracking phases, event extraction, pose detection, etc. The challenges here include distinguishing the color of the playfield from that of the stadium, lighting conditions and sometimes weather, viewing angles, and the shadows. Therefore, accurate segmentation of the playfield cannot be achieved just by processing the color of the playfield under certain situations and making it constant without updating the statistics throughout the game. There is also an added noise when the player’s clothes matching that of the ground, and there appear shadows at the base of a player from different sources of light. A Gaussian-based background subtraction technique [14], which is implemented using computer vision methods, generates the foreground mask as shown in Figure 5.

Figure 5. Background subtraction model.

Researchers have used a single dominant color for detecting the playfield. Accordingly, some studies have utilized the features of images in which illumination is not affected by transforming the images from RGB space to HIS [15,16,17], YCbCr [18], normalized RGB [19,20,21].

For a precise capture of the movements of the players, tracking the ball and actions of referees on the field or court, it is necessary to calibrate the camera [5,8] and also to use an appropriate number of cameras to cover the field. Though some algorithms are capable of tracking the players, some other objects also need to be tracked in dynamically complex situations of interest for detailed analysis of the events and extraction of the data of the subject of interest. Reference [22] presented an approach to extract the playing field and track the players and ball using multiple cameras in soccer video. In [23,24], an architecture was presented, which uses single (Figure 6a) and multi-cameras (Figure 6c) to capture a clear view of players and ball in various challenging and tricky situations such as severe occlusions and the ball being missing from the frames. To estimate the players’ trajectory and team classification in [25,26] a bird’s eye view of the field is presented to capture players, precisely as shown in Figure 6b. Various positions of the camera for capturing the entire field are presented in [27,28] to detect and track the players/ball and estimate the position of the players.

Figure 6. Camera placements in the playfield. (a) Ceiling-mounted camera [23]. (b) Birds eye view of the field [25,26]. (c) Multiple cameras placed to cover the complete playfield [27].

Morphological operations-based techniques can separate the playfield and non-playfield regions, but they cannot detect the lines in the playfield. The background subtraction-based techniques generate foreground regions by subtracting the background frame from the current frame (i.e., by detecting moving objects in the frame); however, they fail to detect the playfield lines as shown in Figure 5. So, the best way to detect the playfield lines is by labeling the data as playfield lines (as shown in Figure 7a), advertisements (as shown in Figure 7b), and the non-playfield regions as shown in Figure 7c.

Figure 7. Background-labeled samples from the dataset (a) playfield lines, (b) advertisements, (c) non-playfield region.

Training the model using a dataset that is labeled as playfield lines, advertisements, and the non-playfield region as shown in Figure 7 can detect and classify the playfield lines, advertisements, and the non-playfield region, which reduces the detection of false positives and false negatives.

4. Literature Review

In this section, the overview of traditional computer-vision methods implemented for major application specifics in sports (such as detection, event classification/recognition, tracking and trajectory prediction) investigated by the researchers and their significant limitations is discussed.

4.1. Basketball

Basketball is a sport played between two teams consisting of five players each. The task of this sport is to score more points than the opponent. This sport has several activities with the ball such as passing, throwing, bouncing, batting, or rolling the ball from one player to another. Physical contact with an opponent player may be a foul if the contact impedes the players’ desired movement. The advancements in computer vision techniques have effectively employed fully automated systems to replace the manual analysis of basketball sports. Recognizing the player’s action and classifying the events [29,30,31] in basketball videos helps to analyze the player’s performance. Player/ball detection and tracking in basketball videos are carried out in [32,33,34,35,36,37] but fail in assigning specific identification to avoid identity switching among the players when they cross. By estimating the pose of the player, the trajectory of the ball [38,39] is estimated from various distances to the basket. By recognizing and classifying the referee’s signals [40], player behavior can be assessed and highlights of the game can be extracted [41]. The behavior of a basketball team [42] can be characterized by the dynamics of space creation presented in [43,44,45,46,47,48] that works to counteract space creation dynamics with a defensive play presented in [49]. By detecting the specific location of the player and ball in the basketball court, the player movement can be predicted [50] and the ball trajectory [51,52,53] can be generated in three dimensions which is a complicated task. It is also necessary to study the extraction of basketball players’ shooting motion trajectory, combined with the image feature analysis method of basketball shooting, to reconstruct and quantitatively track the basketball players’ shooting motion trajectory [54,55,56,57]. However, it is difficult to analyze the game data for each play such as the ball tracking or motion of the players in the game, because the situation of the game changes rapidly, and the structure of the data is complicated. Therefore, it is necessary to analyze the real-time gameplay [58]. Table 2 summarizes various proposed methodologies used to complete various challenging tasks in basketball sport including their limitations.

Table 2. Studies in basketball.

Studies in Basketball
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[31]	Recognizing actions of basketball players by using image recognition techniques	Bi-LSTM Sequence2Sequence	The metrics used to evaluate the method are the Spearman rank-order correlation coefficient, Kendall rank-order correlation coefficient, Pearson linear correlation coefficient, and Root Mean Squared Error and achieved 0.921, 0.803, 0.932, and 1.03, respectively.	The methodology failed to recognize difficult actions due to which accuracy is reduced. The accuracy of action recognition can be improved with a deep convolutional neural network.
[54]	Multi-future trajectory prediction in basketball.	Conditional Variational Recurrent Neural Networks (RNN)—TrajNet++	The proposed methodology was tested on Average Displacement Error and Final Displacement Error metrics. The methodology is robust if the number it achieves is smaller than 7.01 and 10.61.	The proposed methodology fails to predict the trajectories in the case of uncertain and complex scenarios. As the behavior of the basketball or players is dynamic, belief maps cannot steer future positions. Training the model with a dataset of different events can rectify the failures of predictions.
[58]	Predicting line-up performance of basketball players by analyzing the situation of the field.	RNN + NN	At the point guard (pg) position 4 candidates were detected and at the center (c) position 3 candidates were detected. The total score of pg candidates is 13.67, 12.96, 13.42, 10.39, and where the total score of c candidates is 10.21, 14.08, and 13.48, respectively.	-
[32]	Multiplayer tracking in basketball videos	YOLOv3 + Deep-SORT, Faster-RCNN + Deep-SORT, YOLOv3 + DeepMOT, Faster-RCNN + DeepMOT, JDE	Faster-RCNN provides better accuracy than YOLOv3 among baseline detectors. The joint Detection and Embedding method performs better in the accuracy of tracking and computing speed among multi-object tracking methods.	Tracking in specific areas such as severe occlusions and improving detection precision improves the accuracy and computation speed. By adopting frame extraction methods, in terms of speed and accuracy, it can achieve comprehensive performance, which may be an alternative solution.
[40]	Recognizing the referee signals from real-time videos in a basketball game.	HOG + SVM, LBP + SVM	Achieved an accuracy of 95.6% for referee signal recognition using local binary pattern features and SVM classification.	In the case of a noisy environment, a significant chance of occlusion, an unusual viewing angle, and/or variability of gestures, the performance of the proposed method is not consistent. Detecting jersey color and eliminating all other detected elements in the frame can be the other solution to improve the accuracy of referee signal recognition.
[30]	Event recognition in basketball videos	CNN	mAP for group activity recognition is 72.1%	The proposed model can recognize the global movement in the video. By recognizing the local movements, the accuracy can be improved.
[59]	Analyzing the behavior of the player.	CNN + RNN	Achieved an accuracy of 76.5% for four types of actions in basketball videos.	The proposed model gives less accuracy for actions such as passing and fouling. This also gives less accuracy of recognition and prediction on the test dataset compared to the validation dataset.
[33]	Tracking ball movements and classification of players in a basketball game	YOLO + Joy2019	Jersey number recognition in terms of Precision achieved is 74.3%. Player recognition in terms of Recall achieved 89.8%.	YOLO confuses the overlapped image for a single player. In the subsequent frame, the tracking ID of the overlapped player is exchanged, which causes wrong player information to be associated with the identified box.
[29]	Event classifications in basketball videos	CNN + LSTM	The average accuracy using a two-stage event classification scheme achieved 60.96%.	Performance can be improved by introducing information such as individual player pose detection and player location detection
[49]	Classification of different defensive strategies of basketball payers, particularly when they deviate from their initial defensive action.	KNN, Decision Trees, and SVM	Achieved 69% classification accuracy for automatic defensive strategy identification.	Considered only two defensive strategies `switch’ and `trap’ involved in Basketball. In addition, the alternative method of labeling large Spatio-temporal datasets will also lead to better results. Future research may also consider other defensive strategies such as pick-and-roll and pick-and-pop.
[38]	Basketball trajectory prediction based on real data and generating new trajectory samples.	BLSTM + MDN	The proposed method performed well in terms of convergence rate and final AUC (91%) and proved deep learning models perform better than conventional models (e.g., GLM, GBM).	To improve the accuracy time series the prediction has to consider. By considering factors such as player cooperation and defense when predicting NBA player positions, the performance of the model can be improved.
[51]	Generating basketball trajectories.	GRU-CNN	Validated on a hierarchical policy network (HPN) with ground truth and 3 baselines.	The proposed model failed in the trajectory of a three-dimensional basketball match.
[41]	Score detection, highlights video generation in basketball videos.	BEI+CNN	Automatically analyses the basketball match, detects scoring, and generates highlights. Achieved an accuracy, precision, recall, and F1-score of 94.59%, 96.55%, 92.31%, and 94.38%.	The proposed method is lacking in computation speed which achieved 5 frames per second. Therefore, it cannot be implemented in a real-time basketball match.
[34]	Multi-person event recognition in basketball videos.	BLSTM	Event classification and event detection were achieved in terms of mean average precision, i.e., 51.6% and 43.5%.	A high-resolution dataset can improve the performance of the model.
[44]	Player behavior analysis.	RNN	Achieved an accuracy of 80% over offensive strategies.	The methodology fails in many factors such as complexity of interaction, distinctiveness, and diversity of the target classes and other extrinsic factors such as reactions to defense, unexpected events such as fouls, and consistency of executions.
[39]	Prediction of the 3-point shot in the basketball game	RNN	Evaluated in terms of AUC and achieved 84.30%.	The proposed method fails in the case of high ball velocity and the noisy nature of motion data.

4.2. Soccer

Soccer is played using football, and eleven players in two teams compete to deliver the ball into the other team’s goal, thereby scoring a goal. The players confuse each other by changing their speed or direction unexpectedly. Due to them having the same jersey color, players look almost identical and are frequently possess the ball, which leads to severe occlusions and tracking ambiguities. In such a case, a jersey number must be detected to recognize the player [60]. Accurate tracking [61,62,63,64,65,66,67,68,69,70,71,72] by detection [73,74,75,76] of multiple soccer players as well as the ball in real-time is a major challenge to evaluate the performance of the players, to find their relative positions at regular intervals, and to link spatiotemporal data to extract trajectories. The systems which evaluate the player [77] or team performance [78] have the potential to understand the game’s aspects, which are not obvious to the human eye. These systems are able to evaluate the activities of players successfully [79] such as the distance covered by players, shot detection [80,81], the number of sprints, player’s position, and their movements [82,83], the player’s relative position concerning other players, possession [84] of the soccer ball and motion/gesture recognition of the referee [85], predicting player trajectories for shot situations [86]. The generated data can be used to evaluate individual player performance, occlusion handling [21] by the detecting position of the player [87], action recognition [88], predicting and classifying the passes [89,90,91], key event extraction [92,93,94,95,96,97,98,99,100,101], tactical performance of the team [102,103,104,105,106], and analyzing the team’s tactics based on the team formation [107,108,109], along with generating highlights [110,111,112,113]. Table 3 summarizes various proposed methodologies to resolve various challenging tasks in soccer with their limitations.

Table 3. Studies in Soccer.

Studies in Soccer
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[71]	Player and ball detection and tracking in soccer.	YOLOv3 and SORT	Methodology achieved tracking accuracy of 93.7% on multiple object tracking accuracy metrics with a detection speed of 23.7 FPS and a tracking speed of 11.3 FPS.	This methodology effectively handles challenging situations, such as partial occlusions, players and the ball reappearing after a few frames, but fails when the players are severely occluded.
[72]	Player, referee and ball detection and tracking by jersey color recognition in soccer.	DeepPlayerTrack	The model achieved a tracking accuracy of 96% and 60% on MOTA and GMOTA metrics, respectively, with a detection speed of 23 FPS.	The limitation of this method is that, when a player with the same jersey color is occluded, the ID of the player is switched.
[77]	Tracking soccer players to evaluate the number of goals scored by a player.	Machine Learning and Deep Reinforcement Learning.	Performance of the player tracking model measured in terms of mAP achieved 74.6%.	The method failed to track the ball at critical moments such as passing at the beginning and shooting. It also failed to overcome the identity switching problem.
[94]	Extracting ball events to classify the player’s passing style.	Convolutional Auto-Encoder	The methodology was evaluated in terms of accuracy and achieved 76.5% for 20 players.	Concatenation of the auto-encoder and extreme learning machine techniques will improve classification of the event performance.
[101]	Detecting events in soccer.	Variational Auto- encoder and EfficientNet	Achieved an F1-score of 95.2% event images and recall of 51.2% on images not related to soccer at a threshold value of 0.50.	The deep extreme learning machine technique which employs the auto-encoder technique may enhance the event detection accuracy.
[82]	Action spotting soccer video.	YOLO-like encoder	The algorithm achieved an mAP of 62.5%.	-
[78]	Team performance analysis in soccer	SVM	Prediction models achieved an overall accuracy of 75.2% in predicting the correct segmental and the outcome of the likelihood of the team making a successful attempt to score a goal on the used dataset.	The proposed model failed in identifying the players that are more frequently involved in match events that end with an attempt at scoring i.e., a `SHOT’ at goal, which may assist sports analysts and team staff to develop strategies suited to an opponent’s playing style.
[85]	Motion Recognition of assistant referees in soccer	AlexNet, VGGNet-16, ResNet-18, and DenseNet-121	The proposed algorithm achieved 97.56% accuracy with real-time operations.	Though the proposed algorithm is immune to variations of illuminance caused by weather conditions, it failed in the case of occlusions between referees and players.
[106]	Predicting the attributes (Loss or Win) in soccer.	ANN	The proposed model predicts 83.3% for the winning case and 72.7% for loss.	-
[109]	Team tactics estimation in soccer videos.	Deep Extreme Learning Machine (DELM).	The performance of the model is measured on precision, recall, and F1-score and achieved 87.6%, 88%, and 87.8%, respectively.	Team tactics are estimated based on the relationship between tactics of the two teams and ball possession. The method fails to estimate the team formation at the beginning of the game.
[88]	Action recognition in soccer	CNN-based Gaussian Weighted event-based Action Classifier architecture	Accuracy in terms of F1-score achieved was 52.8% for 6 classes.	By classifying the actions into subtypes, the accuracy of action recognition can be enhanced.
[62]	Detection and tracking of the ball in soccer videos.	VGG – MCNN	Achieved an accuracy of 87.45%.	It could not detect when the ball moved out of play in the field, in the stands region, or from partial occlusion by players, or when ball color matched the player’s jersey.
[95]	Automatic event extraction for soccer videos based on multiple cameras.	YOLO	The U-encoder is designed for feature extraction and has better performance in terms of accuracy compared with fixed feature extractors.	To carry out a tactical analysis of the team, player trajectory needs to be analyzed.
[80]	Shot detection in a football game	MobileNetV2	The MobileNetV2 method performed better than other feature extractor methods.	Extracting the features with the MobileNetV2 and then using 3D convolution on the extracted features for each frame can improve detection performance.
[86]	Predicting player trajectories for shot situations	LSTM	Performance is measured in terms of F1-score and achieved 53%.	The model failed to predict the player trajectory in the case of players confusing each other by changing their speed or direction unexpectedly.
[108]	Analyzing the team formation in soccer and formulating several design goals.	OpenCV is used for back-end visualization.	The formation detection model achieved a max accuracy of 96.8%.	The model is limited to scalability as it cannot be used on high-resolution soccer videos. The results are bounded to a particular match, and it cannot evaluate the tactical schemes across different games. Visualization of real-time team formation is another drawback as it limits the visualization of non-trivial spatial information. By applying state-of-the-art tracking algorithms, one can predominantly improve the performance of tactics analysis.
[60]	Player recognition with jersey number recognition.	Spatial Constellation + CNN	Achieved an accuracy of 82% by combining Spatial Constellation + CNN models.	The proposed model failed to handle the players that are not visible for certain periods. Predicting the position of invisible players could improve the quality of spatial constellation features.
[89]	Evaluating and classifying the passes in a football game.	SVM	The proposed model achieves an accuracy of 90.2% during a football match.	To determine the quality of each pass, some factors such as pass execution of player in a particularly difficult situation, the strategic value of the pass, and the riskiness of the pass need to be included. To rate the passes in sequence, it is necessary to consider the sequence of passes during which the player possesses the ball.
[84]	Detecting dribbling actions and estimating positional data of players in soccer.	Random forest	Achieved an accuracy of 93.3%.	The proposed methodology fails to evaluate the tactical strategies.
[103]	Team tactics estimation in soccer videos.	SVM	The performance of the methodology is measured in terms of precision, recall, and F1-score and achieved 98%, 97%, and 98%.	The model fails when audiovisual features could not recognize quick changes in the team’s tactics.
[93]	Analyzing past events in the case of non-obvious insights in soccer.	k-NN, SVM	To extract the features of pass location, they used heatmap generation and achieved an accuracy of 87% in the classification task.	By incorporating temporal information, the classification accuracy can be improved and also offers specific insights into situations.
[61]	Tracking the players in soccer videos.	HOG + SVM	Player detection is evaluated in terms of accuracy and achieved 97.7%. Classification accuracy using k-NN achieved 93% for 15 classes.	-
[79]	Action classification in soccer videos	LSTM + RNN	The model achieves a classification rate of 92% on four types of activities.	By extracting the features of various activities, the accuracy of the classification rate can be improved.

4.3. Cricket

In many aspects of cricket as well, computer vision techniques can effectively replace manual analysis. A cricket match has many observable elements including batting shots [114,115,116,117,118,119,120,121], bowling performance [122,123,124,125,126,127], number of runs or score depending on ball movement, detecting and estimating the trajectory of the ball [128], decision making on placement of players’ feet [129], outcome classification to generate commentary [130,131], detecting umpire decision [132,133]. Predicting an individual cricketer’s performance [134,135] based upon his past record can be critical in team member selection at international competitions. Such process are highly subjective and usually require much expertise and negotiation decision-making. By predicting the results of cricket matches [136,137,138,139,140] such as the toss decision, home ground, player fitness, player performance criteria [141], and other dynamic strategies the winner can be estimated. The video summarization process gives a compact version of the original video for ease in managing the interesting video contents. Moreover, the video summarization methods capture the interest of the viewer by capturing exciting events from the original video [142,143]. Table 4 summarizes various proposed methodologies with their limitations to resolve various application issues in cricket.

Table 4. Studies in Cricket.

Studies in Cricket
Refs.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[117]	Shot classification in cricket.	CNN—Gated Recurrent Unit	It is evaluated in terms of precision, recall, and F1-score and achieved 93.40%, 93.10%, and 93% for 10 types of shots.	By incorporating unorthodox shots which are played in t20 in the dataset may improve the testing accuracy.
[125]	Detecting the action of the bowler in cricket.	VGG16-CNN	It was evaluated in terms of precision, recall, and F1-score and the maximum average accuracy achieved is 98.6% for 13 classes (13 types of bowling actions).	Training the model with the dataset of wrong actions can improve detection accuracy.
[129]	Movement detection of the batsman in cricket.	Deep-LSTM	The model was evaluated in terms of mean square error and achieved a minimum error of 1.107.	-
[142,143]	Cricket video summarization.	Gated Recurrent Neural Network + Hybrid Rotation Forest-Deep Belief Networks YOLO	The methodology was evaluated in terms of precision, F1-score, accuracy and achieved 96.82%, 94.83%, and 96.32% for four classes. YOLO is evaluated on precision, recall, and F1-score and achieved 97.1%, 94.4%, and 95.7% for 8 classes.	Decision tree classifier performance is low due to the existence of a huge number of trees. Therefore, a small change in the decision tree may improve the prediction accuracy. Extreme Learning Machines have faced the problem of overfitting, which can be overcome by removing duplicate data in the dataset.
[134]	Prediction of individual player performance in cricket	Efficient Machine Learning Techniques	The proposed algorithm achieves a classification accuracy of 93.73% which is good compared with traditional classification algorithms.	Replacing machine learning techniques with deep learning techniques may improve the performance in prediction even in the case of different environmental conditions.
[114]	Classification of different batting shots in cricket.	CNN	The average classification in terms of precession is 0.80, Recall is 0.79 and F1-score is 0.79.	To improve the accuracy of classification, a deep learning algorithm has to be replaced with a better neural network.
[130]	Outcome classification task to create automatic commentary generation.	CNN + LSTM	Maximum of 85% of training accuracy and 74% validation accuracy	Due to the unavailability of the standard dataset for the ball by ball outcome classification in cricket, the accuracy is not up to mark. In addition, better accuracy leads to automatic commentary generation in sports.
[132]	Detecting the third umpire decision and an automated scoring system in a cricket game.	CNN + Inception V3	It holds 94% accuracy in the Deep Conventional Neural Network (DCNN) and 100% in Inception V3 for the classification of umpire signals to automate the scoring system of cricket.	To build an automated umpiring system based on computer vision application and artificial intelligence, the results obtained in this paper are more than enough.
[133]	Classification of cricket bowlers based on their bowling actions.	CNN	The test set accuracy of the model is 93.3% which demonstrates its classification ability.	The model lacks data for detecting spin bowlers. As the dataset is confined to left-arm bowlers, the model misclassifies the right-arm bowlers.
[115]	Recognition of various batting shots in a cricket game	Deep-CNN	The proposed models can recognize a shot being played with 90% accuracy.	As the model is dependent on the frame per second of the video, it fails to recognize when the frames per second increases.
[131]	Automatic highlight generation in the game of cricket.	CNN + SVM	Mean Average Precision of 72.31%	The proposed method cannot clear metrics to evaluate the false positives in highlights.
[133]	Umpire pose detection and classification in cricket.	SVM	VGG19-Fc2 Player testing accuracy of 78.21%	Classification and summarization techniques can minimize false positives and false negatives.
[116]	Activity recognition for quality assessment of batting shots.	Decision Trees, k-Nearest Neighbours, and SVM.	The proposed method identifies 20 classes of batting shots with an average F1-score of 88% based on the recorded movement of data.	To assess the player’s batting caliber, certain aspects of batting also need to be considered, i.e., the position of the batsman before playing a shot and the method of batting shots for a particular bowling type can be modeled.
[136,137]	Predicting the outcome of the cricket match.	k-NN, Naïve Bayesian, SVM, and Random Forest	Achieved an accuracy of 71% upon the statistics of 366 matches.	Imbalance in the dataset is one of the causes which produces lower accuracy. Deep learning methodologies may give promising results by training with a dataset that included added features.
[124]	Performance analysis of the bowler.	Multiple regression	Variation in ball speed has a feeble significance in influencing the bowling performance (the p-value being 0.069). The variance ratio of the regression equation to that of the residuals (F-value) is given as 3.394 with a corresponding p-value of 0.015.	-
[135]	Predicting the performance of the player.	Multilayer perceptron Neural Network	The model achieves an accuracy of 77% on batting performance and 63% on bowling performance.	-

4.4. Tennis

Worldwide, Tennis has experienced gain a huge popularity. This game need a meticulous analysis to reducing human errors and extracting several statistics from the game’s visual feed. Automated ball and player tracking belongs to such class of systems that requires sophisticated algorithms for analysis. The primary data for tennis are obtained from ball and player tracking systems, such as HawkEye [144,145] and TennisSense [28,146]. The data from these systems can be used to detect and track the ball/player [147,148,149,150], visualizing the overall tennis match [151,152] and predicting trajectories of ball landing positions [153,154,155], player activity recognition [156,157,158], analyzing the movements of the player and ball [159], analyzing the player behavior [160] and predicting the next shot movement [161] and real-time tennis swing classification [162]. Table 5 summarizes various proposed methodologies to resolve various challenging tasks in tennis with their limitations.

Table 5. Studies in Tennis.

Studies in Tennis
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[145]	Monitoring and Analyzing tactics of tennis players.	YOLOv3	The model achieved an mAP of 90% with 13 FPS on high-resolution images.	Using a lightweight backbone for detection, modules can improve the processing speed.
[158]	Player action recognition in tennis.	Temporal Deep Belief Network (Unsupervised Learning Model)	The accuracy of the recognition rate is 94.72%	If two different movements are similar, then the model fails to recognize the current action.
[162]	Tennis swing classification.	SVM, Neural Network, K-NN, Random Forest, Decision Tree	Maximum classification accuracy of 99.72% achieved using NN with a Recall of 1. The second-highest classification accuracy of 99.44% was achieved using K-NN with a recall of 0.98.	If the play styles of the players are different but the patterns are the same, in that case, models failed to classify the current swing direction.
[156]	Player activity recognition in a tennis game.	Long Short Term Memory (LSTM)	The average accuracy of player activity recognition based on the historical LSTM model was 0.95, and that of the typical LSTM model was 0.70.	The model lacks real-time learning ability and requires a large computing time at the training stage. The model also lacks online learning ability.
[147]	Automatic detection and classification of change of direction from player tracking data in a tennis game.	Random Forest Algorithm	Among all the proposed methods, model 1 had the highest F1-score of 0.801, as well as the smallest rate of false-negative classification (3.4%) and average accuracy of 80.2%	In the case of non-linear regression analysis, the classification performance of the proposed model is not up to the mark.
[153]	Prediction of shot location and type of shot in a tennis game.	Generative Adversarial Network (GAN) (Semi-Supervised Model)	The performance factor is measured based on the minimum distance recorded between predicted and ground truth shot location.	The performance of the model deviates from the different play styles as it is trained on the limited player dataset.
[159]	Analyzing individual tennis matches by capturing spatio-temporal data for player and ball movements.	For data extraction, a player and ball tracking system such as HawkEye is used.	Generation of 1-D space charts for patterns and point outcomes to analyze the player activity.	The performance of the model deviates from different matches, as it was trained only on limited tennis matches.
[157]	Action recognition in tennis	3-Layered LSTM	The classification accuracies are as follows: Improves from 84.10 to 88.16% for players of mixed abilities. Improves from 81.23 to 84.33% for amateurs and from 87.82 to 89.42% for professionals, when trained using the entire dataset.	The detection accuracy can be increased by incorporating spatio-temporal data and combining the action recognition data with statistical data.
[161]	Shot prediction and player behavior analysis in tennis	For data extraction, player and ball tracking systems such as HawkEye are used and a Dynamic Bayesian Network for shot prediction is used.	By combining factors (Outside, Left Top, Right Top, Right Bottom) together, speed, start location, the player movement assessment achieved better results of 74% AUC.	As the model is trained on limited data (only elite players), it cannot be performed on ordinary players across multiple tournaments.
[148]	Ball tracking in tennis	Two-Layered Data Association	Evaluation results in terms of precision, recall, F1-score are 84.39%, 75.81%, 79.87% for Australian open tennis matchwa and 82.34%, 67.01%, 73.89% for U.S open tennis matches.	The proposed method cannot handle multi-object tracking and it is possible to integrate audio information to facilitate high-level analysis of the game.
[160]	Highlight extraction from rocket sports videos based on human behavior analysis.	SVM	The proposed algorithm achieved an accuracy of 90.7% for tennis videos and 87.6% for badminton videos.	The proposed algorithm fails to recognize the player, as the player is a deformable object of which the limbs perform free movement during action recognition.

4.5. Volleyball

In volleyball, two teams of six players each are placed on either side of a net. Each team attempts to ground a ball on the opposite team’s court and to score points under the defined rules. So, detecting and analyzing the player activities [163,164,165], detecting play patterns and classifying tactical behaviors [166,167,168,169], predicting league standings [170], detecting and classifying spiking skills [171,172], estimating the pose of the player [173], tracking the player [174], tracking the ball [175], etc., are the major aspects of volleyball analysis. Predicting the ball trajectory [59] in a volleyball game by observing the motion of the setter player has been conducted. Table 6 summarizes various proposed methodologies to resolve various challenging tasks in volleyball sport with their limitations.

Table 6. Studies in volleyball.

Studies in Volleyball
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[173]	Group activity recognition by tracking players.	CNN + Bi-LSTM	The model achieved an accuracy of 93.9%.	The model fails to track the players if the video is taken from a dynamic camera. Temporal action localization can improve the accuracy of tracking the players in severe occlusion conditions.
[174]	Recognizing and classifying player’s behavior.	SVM	The achieved recognition rate was 98% for 349 correct samples.	-
[167]	Classification of tactical behaviors in beach volleyball.	RNN + GRU	The model achieves better classification results as prediction accuracies range from 37% for forecasting the attack and direction to 60% for the prediction of success.	By employing a state-of-the-art method and training on a proper dataset that has continuous positional data, it is possible to predict tactics behavior and set/match outcomes.
[175]	Motion estimation for volleyball	Machine Vision and Classical particle filter.	Tracking accuracy is 89%	Replacing methods with deep learning algorithms gives better results.
[168]	Assessing the use of Inertial Measurement Units in the recognition of different volleyball actions.	KNN, Naïve Bayes, SVM	Unweighted Average Recall of 86.87%	By incorporating different frequency domain features, the performance factor can be improved.
[59]	Predicting the ball trajectory in a volleyball game by observing the the motion of the setter player.	Neural Network	The proposed method predicts 0.3 s in advance of the trajectory of the volleyball based on the motion of the setter player.	In the case of predicting the 3D body position data, the method records a large error. This can be overcome by training properly annotated large data on state-of-art-methods.
[164]	Activity recognition in beach volleyball	Deep Convolutional LSTM	The approach achieved a classification accuracy of 83.2%, which is superior compared with other classification algorithms.	Instead of using wearable devices, computer vision architectures can be used to classify the activities of the players in volleyball.
[170]	Volleyball skills and tactics analysis	ANN	Evaluated in terms of Average Relative Error for 10 samples and achieved 0.69%.	-
[165]	Group activity recognition in a volleyball game	LSTM	Group activity recognition of accuracy of the the proposed model in volleyball is 51.1%.	The performance of architecture is poor because of the lack of hierarchical considerations of the individual and group activity dataset.

4.6. Hockey/Ice Hockey

Hockey, also known as Field hockey, is an outdoor game played between two teams of 11 players each. These players use sticks that are curved at the striking end to hit a small and hard ball into their opponent’s goal post. So, detecting [176] and tracking the player/hockey ball, recognizing the actions of the player [177,178,179], estimating the pose of the player [180], classifying and tracking the players of the same team or different teams [181], referee gesture analysis [182,183] and hockey ball trajectory estimation are the major aspects of hockey sport.

Ice hockey is another similar game to field hockey, with two teams with six players each, wearing skates and competing on an ice rink. All players aim to propel a vulcanized rubber disk, the puck, past a goal line and into a net guarded by a goaltender. Ice hockey is gaining huge popularity on international platforms due to its speed and frequent physical contact. So, detecting/tracking the player [184,185,186], estimating the pose of the player [187], classifying and tracking with different identification the players of the same team or different teams, tracking the ice hockey puck [188], and classification of puck possession events [189] are the major aspects of the ice hockey sport. Table 7 summarizes various proposed methodologies to resolve various challenging tasks in hockey/ice hockey with their limitations.

Table 7. Studies in hockey.

Studies in Hockey
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[176]	Detecting the player in hockey.	SVM, Faster RCNN, SSD, YOLO	HD+SVM achieved the best results in terms of accuracy, recall, and F1-score with values of 77.24%, 69.23%, and 73.02%.	The model failed to detect the players in occlusion conditions.
[188]	Localizing puck Position and Event recognition.	Faster RCNN	Evaluated in terms of AUC and achieved 73.1%.	Replacing the detection method with the YOLO series can improve the performance.
[181]	Identification of players in hockey.	ResNet + LSTM	Achieves player identification accuracy of over 87% on the split dataset.	Some of the jersey number classes such as 1 to 4 are incorrectly predicted. The diagonal numbers from 1 to 100 are falsely classified due to the small number of training examples.
[177]	Activity recognition in a hockey game.	LSTM	The proposed model recognizes the activities such as free hits, goals, penalties corners, and long corners with an accuracy of 98%.	As the proposed model is focused on spatial features, it does not recognize activities such as free hits and long corners as they appear as similar patterns. By including temporal features and incorporating LSTM into the model, the model is robust to performance accuracy.
[180]	Pose estimation and temporal-based action recognition in hockey.	VGG19 + LiteFlowNet + CNN	A novel approach was designed and achieved an accuracy of 85% for action recognition.	The architecture is not robust to abrupt changes in the video, e.g., it fails to predict hockey sticks. Activities such as a goal being scored, or puck location, are not recognized.
[187]	Action recognition in ice hockey using a player pose sequence.	CNN+LSTM	The performance of the model is better in similar classes such as passing and shooting. It achieved 90% parameter reduction and 80% floating-point reduction on the HARPET dataset.	As the number of hidden units to LSTM increases, the number of parameters also increases, which leads to overfitting and low test accuracy.
[178]	Human activity recognition in hockey.	CNN+LSTM	An F1-score of 67% was calculated for action recognition on the multi-labeled imbalanced dataset.	The performance of the model is poor because of the improper imbalanced dataset.
[179]	Player action recognition in an ice hockey game	CNN	The accuracy of the actions recognized in a hockey game is 65% and when similar actions are merged accuracy rises to 78%.	Pose estimation problems due to severe occlusions when motions blur due to the speed of the game and also due to lack of a proper dataset to train models, all causing low accuracy.

4.7. Badminton

Badminton is one of the most popular racket sports, which includes tactics, techniques, and precise execution movements. To improve the performance of the player, technology plays a key role in optimizing the training of players; technology determines the movements of the player [190] during training and game situations such as with action recognition [191,192,193], analyzing the performance of player [194], detecting and tracking the shuttlecock [195,196,197]. Table 8 summarizes various proposed methodologies to resolve various challenging tasks in badminton with their limitations.

Table 8. Studies in badminton.

Studies in Badminton
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[195]	Shuttlecock detection problem of a badminton robot.	Tiny YOLOv2 and YOLOv3	Results show that, compared with state-of-art methods, the proposed networks achieved good accuracy with efficient computation.	The proposed method fails to detect different environmental conditions. As it uses the binocular camera to detect a 2D shuttlecock, it cannot detect the 3D shuttlecock trajectory.
[191]	Automated badminton player action recognition in badminton games.	AlexNet+CNN, GoogleNet+CNN and SVM	Recognition of badminton actions by the linear SVM classifier for both AlexNet and GoogleNet using local and global extractor methods is 82 and 85.7%.	The architecture can be improved by fine-tuning in an end-to-end manner with a larger dataset on features extracted at different fully connected layers.
[192]	Badminton activity recognition	CNN	Nine different activities were distinguished: seven badminton strokes, displacement, and moments of rest. With accelerometer data, accurate estimation was conducted using CNN with 86% precision. Accuracy is raised to 99% when gyroscope data are combined with accelerometer data.	Computer vision techniques can be employed instead of sensors.
[193]	Classification of badminton match images to recognize the different actions were conducted by the athletes.	AlexNet, GoogleNet, VGG-19 + CNN	Significantly, the GoogleNet model has the highest accuracy compared to other models in which only two-hit actions were falsely classified as non-hit actions.	The proposed method classifies the hit and non-hit actions and it can be improved by classifying more actions in various sports.
[196]	Tracking shuttlecocks in badminton	An AdaBoost algorithm which can be trained using the OpenCV Library.	The performance of the proposed algorithm was evaluated based on precision and it achieved an average precision accuracy of 94.52% with 10.65 fps.	The accuracy of tracking shuttlecocks is enhanced by replacing state-of-the-art AI algorithms.
[190]	Tactical movement classification in badminton	k-Nearest Neighbor	The average accuracy of player position detection is 96.03 and 97.09% on two halves of a badminton court.	The unique properties of application such as the length of frequent trajectories or the dimensions of the vector space may improve classification performance.

4.8. Miscellaneous

Player detection and tracking is the major requirement in athletic sports such as running, swimming [198,199], and cycling. In sports such as table tennis [200], squash [201,202], and golf [203], ball detection and tracking and player pose detection [204] are challenging tasks. In ball-centric sports such as rugby, American football, handball, baseball, ball/player detection [205,206,207,208,209,210,211] and tracking [212,213,214,215,216,217,218,219,220,221], analyzing the action of the player [23,222,223,224,225,226,227], event detection and classification [228,229,230,231,232], performance analysis of player [233,234,235], referee identification and gesture recognition are the major challenging tasks. Video highlight generation is a subclass of video summarization [236,237,238,239] which may be viewed as a subclass of sports video analysis. Table 9 summarizes various proposed methodologies to resolve various challenging tasks in various sports with their limitations.

Table 9. Studies in various sports.

Studies in Various Sports
Ref.	Problem Statement	Proposed Methodology	Precision and Performance Characteristics	Limitations and Remarks
[211]	Beach sports image recognition and classification.	CNN	The model achieved a recognition accuracy of 91%.	Lightweight networks of deep learning algorithms can improve the recognition accuracy and can also be implemented in real-time scenarios.
[199]	Motion image segmentation in the sport of swimming	GDA + SVM	The performance of the Symmetric Difference Algorithm was measured in terms of recall and achieved 76.2%.	Using advanced optimization techniques such as Cosine Annealing Schedulers with deep learning algorithms may improve the performance.
[200]	Identifying and recognizing wrong strokes in table tennis.	k-NN, SVM, Naïve Bayes	Performs various ML algorithms and achieves an accuracy of 69.93% using the Naïve Bayes algorithm.	A standard dataset can improve the accuracy of recognizing the wrong strokes in table tennis.
[212]	Multi-player tracking in sports	Cascade Mask R-CNN	The proposed Deep Player Identification method studies the patterns of jersey number, team class, and pose-guided partial feature. To handle player identity switching, the method correlates the coefficients of player ID in the K-shortest path with ID. The proposed framework achieves state-of-art performance.	When compared with existing methods, the computation cost is higher and can be considered a major drawback of the proposed framework. To refine 2D detection, temporal information needs to be considered and can be transferred to tracking against a real-time performance such as soccer, basketball, etc.
[215]	Individual player tracking in sports events.	Deep Neural Network	Achieved an Area Under Curve (AUC) of 66%	Tracking by jersey number recognition may increase the performance of the model.
[204]	Skelton-based key pose recognition and classification in sports	Boltzmann machine+CNN Deep Boltzmann machine + RNN	The proposed architecture successfully analyses feature extraction, motion attitude model, motion detection, and behavior recognition of sports postures.	The architecture is bound to individual-oriented sports and can be further implemented on group-based sports, in case of challenges such as severe occlusion, misdetection due to failure in blob detection in object tracking.
[224]	Human action recognition and classification in sports	VGG 16 + RNN	The proposed method achieved an accuracy of 92% for ten types of sports classification.	The model fails in the case of scaling up the dataset for larger classification which shows ambiguity between players and similar environmental conditions. Football, Hockey; Tennis, Badminton; Skiing, Snowboarding; these pairs of classes have similar environmental features; thus, it is only possible to separate them based on relevant actions which can be achieved by state-of-the-art methods.
[237]	Replay and key event detection for sports video summarization	Extreme Learning Machine (ELM)	The framework is evaluated on a dataset that consists of 20 videos of four different sports categories. It achieves an average accuracy of 95.8%, which illustrates the significance of the method in terms of key-event and replay detection for video summarization.	The performance of the proposed method drops in the case of the absence of a gradual transition of a replay segment. It can be extended by incorporating artificial intelligence techniques.
[228]	Event detection in sports videos for unstructured environments with arbitrary camera angles.	Mask RCNN + LSTM	The proposed method is accurate in unsupervised player extraction, which is used for precise temporal segmentation of rally scenes. It is also robust to noise in the form of camera shaking and occlusions.	It can be extended to doubles games with fine-grained action recognition for detecting various kinds of shots in an unstructured video and it can be extended to analyze videos of games such as cricket, soccer, etc.
[229]	Human motion quality assessment in complex motion scenarios.	3-Dimensional CNN	Achieved an accuracy of 81% on the MS-COCO dataset.	Instead of the Stochastic Gradient Descent technique for learning rate, using the Cosine annealing scheduler technique may improve the performance.
[213]	Court detection using markers, player detection, and tracking using a drone.	Template Matching + Particle Filter	The proposed method achieves better accuracy (94%) in the case of two overlapping players	As the overlapping of players, increases the accuracy of detection and tracking decreases due to similar features of players on the same team. The method uses a template matching algorithm, which can be replaced with a deep learning-based state-of-art algorithm to acquire better results.
[214]	Target tracking theory and analyses its advantages in video tracking.	Mean Shift + Particle Filter	Achieves better tracking accuracy compared to existing algorithms such as TMS and CMS algorithms.	If the target scales change then the tracking of players fails due to the unchanged window of the mean-shift algorithm. Furthermore, it cannot track objects which are similar to the background color. The accuracy of tracking players can be improved by replacing them with artificial intelligence algorithms.
[236]	Automatically generating a summary of sports video.	2D CNN + LSTM	Describes a novel method for automatic summarization of user-generated sports videos and demonstrated the results for Japanese fencing videos.	The architecture can be improved by fine-tuning in an end-to-end manner with a larger dataset for illustrating potential performance and also to evaluate in the context of a wider variety of sports.
[227]	Action Recognition and classification	SVM	Achieved an accuracy of 59.47% on the HMDB 51 dataset.	In cases where the object takes up most of the frame, the human detector cannot completely cover the body of the object. This leads to the system missing movements of body parts such as hands and arms. In addition, recognition of similar movements is a challenge for this architecture.

4.9. Overview of Machine Learning/Deep Learning Techniques

There are multiple ways to classify, detect, and track objects to analyze the semantic levels involved in various sports. They pave the way for player localization, jersey number recognition, event classification and trajectory forecasting of the ball in a sports video with a much better interpretation of an image as a whole.

The selected AI algorithm is better if it is tested and benchmarked on different data. To evaluate the robustness of AI algorithms, some metrics are required which measure the performance of particular AI algorithms to enable better selection. Figure 8 depicts the road map of the machine learning algorithms’ general information, methods, and evaluation criteria for a particular task and required libraries/tools for training the model. Figure 9 depicts the roadmap of the deep learning algorithm selection, training, and evaluation criteria for a particular task and required libraries/tools for training the model. Figure 10 shows taxonomy of various deep learning techniques of classification [240,241,242,243,244,245], detection [246] and prediction [247,248,249] algorithms, unsupervised learning [250,251], tracking [252,253,254,255,256,257,258,259,260,261], and trajectory prediction [262,263,264,265,266,267,268,269]. Since various tasks in sports such as classification/detection, tracking, and trajectory prediction show great advantages in various sports. A bi-layered parallel training architecture in distributed computing environments was introduced in [270], which discusses the time-consuming training process of large-scale deep learning algorithms.

Figure 8. Block diagram of the road map to machine learning architecture selection and training.

Figure 9. Block diagram of the road map to deep learning architecture selection and training.

Figure 10. Overview of deep learning algorithms of classification/detection, tracking and trajectory prediction.

5. Available Datasets of Sports

In this section, a brief description of some sports video datasets which are available publicly with annotations is provided. Utilizing these shared datasets provides a platform for comparison of the performance of algorithms with common data for improving the transparency in research in this domain. Additionally, sharing the data among the users (researchers) reduces time-consuming efforts in capturing and annotating large quantities of videos in diversified areas. This allows users to obtain benchmark scores for the algorithms developed.

These shared datasets can be categorized into two types: videos or still images, which are typically taken with moving cameras, particularly of individual athletes or of team sports, for recognition of player actions [96,165,225,226,271,272,273,274], event detection and classification [34,98,275], which are often captured using several setups of static cameras, for detection and tracking of players/balls [276,277,278], pose estimation [279], and sports event summarization [280] of team plays. One dataset is focused on the spectators’ actions in sports rather than those of the players. These datasets which are available for analysis largely perform a great variety of actions. Table 10 describes the available datasets of various sports, the mode of the dataset, annotated parameters, number of frames, and length of the video.

Table 10. Details of the available datasets.

Details of the Available Dataset
Refs.	Sport	Dataset	Mode of the Dataset	Annotated Parameters	Length of the Video and Number of Images
[101]	Soccer	Image type Football Keyword Dataset	Event detection and Classification.	Events such as free kicks, penalty kicks, tackles, red cards, yellow cards.	Dataset was categorized as training, testing, and validation with 5000, 500, and 500 images.
[271]	Basketball	Basketball dataset	Action Recognition	Dribbling, Passing, Shooting	Dataset consists of video of 8 h duration, 3399 annotations and 130 samples of each class.
[121]	Cricket	Video type Cricket Strokes Dataset	Cricket Stroke Localization.	Annotated with strokes played in untrimmed videos.	Highlights dataset and the Generic dataset comprised of Cricket telecast videos at 25 fps.
[272]	Table Tennis	Video type TTNet dataset	Ball detection and Event Spotting	Ball bounces, Net hit, Empty Events	5 Videos of 10–25 min duration for training and 7 short videos for testing
[96,272]	Soccer	Image type SoccerNet and SoccerNetv2 dataset	Action spotting in soccer videos	Goal, Yellow/Red Card, Substitution	Handles a length of video of about 764 h and 6637 moments which are split into three major classes (Substitution, Goal, and Yellow/Red Card).
[165]	Volleyball	Volleyball dataset	Group activity recognition	Person-level actions, Temporal dynamics of a person’s action and Temporal evolution of group activity	1525 frames were annotated.
[273]	Hockey	The Spectators of Hockey (S-HOCK)	Analyzing Crowds at the Stadium	spectator categorization such as position, head pose, posture and action	Video type dataset. 31 s 30 fps
[281]	487 classes of sports	Sports-1M dataset	Sports classification and activity recognition.	Activity labels	5 m 36 s
[276]	Soccer	Player position in soccer video dataset	Player tracking system	Trajectories of players	45 min
[225]	Volleyball	Indoor volleyball dataset	Activity Detection and Recognition	Seven activities such as the serve, reception, setting, attack, block, stand, and defense/move are annotated to each player in this dataset.	23 min 25 fps
[274]	Various jump games, various throw games, bowling, tennis serving, diving, and weightlifting.	Olympic Sports Dataset	Recognition of complex human activities in sports	Different poses in different sports	Video type dataset. It contains 16 sports classes, with 50 sequences per class.
[34]	Basketball	NCAA Dataset	Event Recognition	Event classification, Event detection and Evaluation of attention.	The video type dataset. Length 1.5 h long Annotated with 11 types of events.
[277]	Soccer	ISSIA Soccer Dataset	Objective method of Ground Truth Generation	Player and Ball trajectories	2 min 25 fps
[280]	Basketball	APIDIS Basketball Dataset	sport-event summarization	Basketball events such as the position of players, referees, and the ball.	16 min, 22fps
[279]	Badminton, Basketball, Football, Rugby, Tennis, Volleyball	Martial Arts, Dancing and Sports dataset	3D human pose estimation.	It is annotated with 5 types of actions.	Size of the dataset is 53,000 frames.
[226]	Kicking, golf swing, lifting, diving, riding horses, skateboarding, running, walking, swing-side.	UCF Sports Action Dataset	Action Recognition	Action localization and the class label for activity recognition.	13k clips and 27 h of video data
[278]	Soccer	SSET	Shot segmentation, Event detection, Player Tracking	Far-view shot, Medium-view shot, Close-view shot, Out-of-field shot, and Playback shot.	350 soccer videos of total length 282 h 25 fps

The parameters which are annotated in the ISSIA dataset relate to the positions of the ball, player, and referee in each video from each camera. The images shown in Figure 11 are a few sample frames from the ISSIA dataset.

Figure 11. Instances from the ISSIA dataset [277].

The parameters which are annotated in the TTNet dataset are the ball bouncing moments, the ball hitting the net, and empty events. The images shown in Figure 12 are a few sample frames from the TTNet dataset.

Figure 12. Instances from the TTNet dataset [275].

For the creation of the APIDIS dataset, videos were captured from seven cameras from above and around the court. The events which are annotated in this dataset are player positions, movements of referees, baskets, and the position of the ball. The images shown in Figure 13 are a few samples from the APIDIS dataset.

Figure 13. Instances from the APIDIS dataset [280].

6. GPU-Based Work Stations and Embedded Platforms

To find the target, GPU-constrained devices such as Raspberry Pi, Latte Panda, Odroid Xu4 and Computer Vision were used. The disadvantages of machine learning techniques are that they provide poor or inaccurate results and have issues in predicting an unknown future data, whereas deep learning algorithms provide accurate results and also make predictions from unknown future data. Segmentation, localization and image classification are visual recognition systems which have prominent research contributions.

Among embedded AI computing platforms, NVidia Jetson devices provide low-power computing and high-performance support for artificial intelligence-based visual recognition systems. Jetson modules are configured with OpenCV, cuDNN, CUDA Toolkit, L4T with LTS Linux kernel and TensorRT. The Intel Movidius Neural Compute Stick uses the Intel Movidius Neural Compute SDK in GPU-Constrained devices to deploy AI algorithms.

Wang et al. [203] presented a high-speed stereo vision system that can track the motion of a golf ball at a speed of 360 km/h under indoor lighting conditions. They implemented the algorithm on a field-programmable gate array board with an advanced RISC machine CPU [62] which implemented a deep learning approach to track the soccer ball on NVIDIA GTX1050Ti GPU [43] and a deep learning algorithm on GTX 1080 ti GPU, based on CUDA 9.0 and Caffe to analyze the technical features in basketball video. Table 11 shows the basic comparison between GPU-based devices and GPU-Constraint Devices and possible deep learning algorithms to implement on various devices.

Table 11. Comparison between Jetson Modules and GPU-Constrained Devices.

	Jetson TX1	Jetson TX2	Jetson AGX Xavier	Raspberry Pi Series	Latte Panda	Odroid Xu4
CPU	Quad-core ARM Processor	Dual-core Denver CPU and quad-core ARM 57	64 bit CPU and 8-core ARM	63-bit quad- core ARM	Intel Cherry Train quad-core CPU	Cortex A7 octa-core CPU
GPU	NVIDIA Maxwell with CUDA cores	NVIDIA Pascall with CUDA cores	Tensor cores + 512-core Volta GPU	-	-	-
Memory	4GB Memory	8 GB Memory	16 GB Memory	1 GB Memory	4 GB Memory	Stacked memory of 2 GB
Storage	16 GB Flash Storage	32 GB storage	32 GB storage	Support Micro SD card	64 GB storage	Support Micro SD Card
Possible DL Algorithms to Implement	YOLO v2 and v3, tiny YOLO v3, SSD, Faster–RCNN and Tracking algorithm like YOLO v3 + Deep SORT, YOLO v4, YOLOR			YOLO, YOLO v2 and SSD-MobileNet etc.

A Field Programmable Gate Array (FPGA) has also been used in sports involving 3D motion capturing, object movement analysis and image recognition, etc. Table 12 describes how different researchers performed various studies of sports on hardware platforms such as FPGA and GPU-based devices and their results in terms of performance measures are listed.

Table 12. Performance of various studies on hardware platforms.

Ref.	GPU-Based Work Station	Embedded Platform	Problem Statement	Performance Measures	Result
[29]	NVidia Titan X GPU.	-	Event classifications in basketball videos.	Average Accuracy	58.10%
[38]	NVIDIA GTX 960	-	Basketball trajectory prediction based on real data and generating new trajectory samples.	Measured in terms of AUC	91.00%
[282]	-	FPGA	Recognizing swimming styles of a swimmer.	The result shows the three-level identification system in Average, Minimum and Maximum offset.	4.14%, 2.16%, 5.77%.
[198]	-	-	-	Recall and Specificity	85% and 96.6%
[33]	NVidia GeForce GTX 1080Ti	-	Tracking ball movements and classification of players in a basketball game	Precision and Recall	74.3% and 89.8%
[282]		FPGA	Ball detection and tracking to reconstruct trajectories in basketball.	Accuracy	>90%
[43]	NVidia GTX 1080 ti GPU	-	Analyzing the behavior of the player.	Accuracy	83%
[283]	-	FPGA	Detecting the movement of the ball in basketball.	Average rate vs Frame range	Varies from 12 to 100% for different frame ranges.
[215]	NVidia GTX 1080Ti	-	Individual player tracking in sports events.	Achieved an Area Under Curve (AUC)	66%
[88]	NVidia GeForce GTX 1080Ti	-	Action recognition in soccer	Accuracy in terms of F1-score	52.80%
[284]	-	FPGA	Movement classification in basketball based on Virtual Reality Technology to improve basketball coaching.	Accuracy	93.50%
[62]	NVidia GTX 1050Ti	-	Detection and tracking of the ball in soccer videos.	Accuracy	87.45%
[285]	-	FPGA	Action recognition based on arm movement in basketball.	Accuracy	92.30%

7. Applications in Sports Vision

A fan who is digitally connected becomes the biggest online influencer of sports venues. Teams and stadium owners provide plenty of personalized experiences through their custom apps, mobile phone support for content with offers and live updates of game information using digital boards to increase the engagement of fans and in turn generate opportunities for new revenues [286]. Figure 14 depicts where AI technology can be used within the sporting landscape.

Figure 14. AI technology framework for the sport industry.

Modern artificial intelligence fields are not good sparring partners, but they can be valuable as research tools. One of the most effective methods to improve this is to learn from one’s failures. The suggested method to improve playing abilities is to review games, but how can one detect the mistakes? How can one come up with better alternatives? This challenge is solved by the field of artificial intelligence analysis tools such as AlphaGO [287,288,289], which provide probability distribution of smart moves and their assessment.

7.1. Chabots and Smart Assistants

Recently, sports bodies such as the NHL and NBA have started using virtual assistants to respond to inquiries made by fans in a wide range of topics such as ticketing, arena logistics, parking, and other game-related information. If the bots are not capable, such scenarios are handled by human intervention and they maintain customer services for that.

7.2. Video Highlights

The challenges facing the industry include not just the creation of content but also delivering it to customers through multiple devices and screens for viewing different content at different times. There is a serious demand from fans for in-depth analysis and also for commentary. Many others like action-packed highlights and some behind-the-scenes content as well. Introducing AI enables the solving of challenging tasks in various sports and it provides an exciting viewing experience to the audience, attracting more viewers.

7.3. Training and Coaching

Effective methods for improving the analysis of the performance of athletes and also assisting coaches with team guidance to gauge the tactics of the opponents are gaining popularity.

An application that uses AI contains huge a dataset of game performances and training-related information, which is backed up with the knowledge of several coaches and sports scientists. They act as an accumulated source of current knowledge on the dissemination of the latest techniques, tactics, or knowledge for professional coaches.

With the evolution of knowledge on any tactic or technique, the knowledge base of AI is updated. The accumulated data can be used for training and educating sports coaches, scientists, and also athletes, which in turn leads to improved performance.

7.4. Virtual Umpires

In cricket and tennis, Video Assistant Referees (VAR) and Decision Review Systems (DRS) have used Hawk-eye, slow-motion replays, and some other technologies. However, the catch is that these involve a request from players or team for review when an umpire’s or referee’s decision has some uncertainty and then involves other parties to assist the main umpire. The whole process is time-consuming and takes away the momentum and excitement of the game.

The latest camera technology supporting AI software creates a situation where an umpire’s role is limited to on-field behavior management of players rather than making critical decisions. For example, in the case of tennis, computer vision is used for detecting placement and speed of the ball; therefore, the need for a line umpire is eliminated. The future scope can be an umpire’s earpiece and glasses assisting the decision instantly and eliminating the necessity of reviews.

7.5. AI Assistant Coaches

AI can be way more capable in situations involving dynamic planning and analysis of the scenes where a coach would rely on previous data and experience, and it is not effective enough to frame dynamically changing strategies in comparison to a machine. A future can be imagined in which a machine with AI running alongside the gameplay dynamically predicts and creates strategies, helping the teams to gain an edge over others.

One example where the evolution of AI can be seen is Chess. The evolution of chess technology is shown in Figure 15. The Russian Garry Kasparov who was considered the world’s number 1 for about 19 years with an Elo rating (skill level measurement) 2851 was surpassed by Magnus Carlsen with Elo rating 2882 in 2014.

Figure 15. Evolution of chess technology demonstrates the speed of AI adoption.

In computers, Deep Blue’s rating which was 2700+ was surpassed by Deep Mind’s Alpha Zero with an estimated Elo 3600, which was developed by Google’s sibling DeepMind. It was developed by a reinforcement learning technique called self-play. It took just 24 h to achieve it and proves the capabilities of the machine.

7.6. Available Commercial Systems for Player and Ball Tracking

Hawkeye [290,291,292] is the technology for ball tracking in cricket, tennis, and soccer. The area of primary application is officiating in these sports to enhance broadcast videos. Figure 16 shows the visualization performance of the commercial systems.

Figure 16. Hawkeye technology in cricket, tennis and soccer [290,291,292].

STATS SportVU [293] and ChyronHego TRACAB [294] are the technologies for player tracking in sports. The area of primary application is to track players in various sports, analyze their performance and assist coaches for training. Figure 17 shows the player position and pose estimation using commercial systems. SportsVu is a computer vision technology that provides real-time optical tracking in various sports. It provides in-depth performance of any team, such as tracking every player from both teams to provide comprehensive match coverage, collecting data to provide tactical analysis of the match, and highlighting the performance deviations to reduce injuries in the game.

Figure 17. TRACAB Gen5 Technology for player tracking [294].

8. Research Directions in Sports Vision

Based on the investigation of the available articles in sports, we were able to come out with various research topics and identifies research directions to be taken for further research in sports. They are categorized based on the task specifics in sports applications (such as major sports in which player/ball/referee detection and tracking, pose estimation, trajectory prediction are required) as shown in Figure 18 to provide promising and potential research directions for future computer vision/video processing in sports.

Figure 18. Major task specifics in sports applications.

As sporting activities are dynamic, the accuracy and reliability of single- or multi-player tracking [63,215] in real-time sports video can be enhanced by proposing a framework that learns object identities with deep representations which resolve the problem of identity switching among players [212]. By considering the temporal information, the performance of the tracking algorithm can become robust to overcome problems such as severe occlusions and miss-detection.

The accuracy of classifying different defensive strategies of soccer [49] can be improved by labeling large spatio-temporal datasets and by classifying the actions into subtypes [88]. The performance measures of team tactics analysis [93,95] of soccer videos can be enhanced by analyzing player trajectories. By incorporating the temporal information, the classification accuracy can be improved while it also offers more specific insights into situations such as pass events in the case of non-obvious insights in soccer videos [93]. Accurate pose detection as shown in Figure 19a is still a major challenge to identify whether the player is running, jumping, or walking as shown in Figure 19b, and also to handle the severe occlusions or identity switches among players.

Figure 19. Instances from soccer matches. (a) Detecting body pose and limbs. (b) Handling severe occlusions among players.

To assess the batter’s caliber, certain aspects of batting need to be considered, i.e., position of batsman before playing a shot, and the method of batting shots for a particular bowling type needs to be modeled [116]. Classification and summarization techniques can minimize false positives and false negatives to detect and classify umpire poses [133]. Detecting various moments such as whether the ball hit the bat and precise detection of the player and wicket keeper at the moment of run-outs, as shown in Figure 20a, is still a major issue in cricket. Predicting the trajectory of balls bowled by spin bowlers as shown in Figure 20b can be resolved accurately by labeling large datasets and modeling using SOTA algorithms.

Figure 20. Instances from cricket matches. (a) Precise detection at the moment of run outs. (b) Predicting the trajectory of the ball in-line or out-line, etc.

The recognition accuracy of player actions in badminton games [191] can be improved by SOTA computer vision algorithms and fine-tuning in an end-to-end manner with a larger dataset on features extracted at different fully connected layers. In the implementation of an automatic linesman system in badminton games [295], the algorithm is not robust to the far views of the camera, where illumination conditions heavily impact the system while the speed of the shuttlecock is also a major factor for poor accuracy. So, it is necessary to track the path, which becomes simpler for the referee to decide if a shuttle lands out or in as shown in Figure 21.

Figure 21. Exact spot on which the shuttlecock lands.

8.1. Open Issues and Future Research Areas

Computer vision plays a vital role in the area of sports video processing. To analyze sports events, there are many issues open for research. Calibration and viewpoints of the camera to capture the sports events such as close-up views, far views, and wide views in a degree of occlusions are still issues that have not been satisfactorily addressed.

Detecting the ball in various sports helps to detect and classify various ball-based events such as goals, possession of the ball, and many other events. Due to the size, speed, velocity, and unstructured motion of the ball compared to the players and playfield in various sports, it is still an open issue to detect and track the ball. Various AI algorithms have been developed to achieve better performance in various sports such as soccer [62,84], basketball [33,38], tennis [148], and badminton [195,196] in terms of detecting and tracking concerning various aspects of the ball.

Tracking players and the ball is one of the most open areas for research which includes various issues such as fast and frequent movements of the players, the similar appearance of players due to jersey color in team sports, often partial and full occlusions of players. Various algorithms use linear motions for multi-player tracking, resulting in poor performance but solves data association problems with appearance models. However, this algorithm fails in various conditions such as severe occlusions, the ambiguity of appearance between players, etc. [296,297] applied context information to track the players in soccer and volleyball sports.

8.2. Future Research Trends according to Methodologies in Sports Vision

In this section, we aim to set forth the methodological approach to various components of detection, classification, and tracking in sports. By considering the deep analysis of sports studies, it is clear that the performance of the algorithm depends on the type (annotation parameters) of dataset used, which is carried out based on loss functions and evaluation metrics. The major difficulties in the real-time use of AI algorithms in various research areas of sports are accuracy, computation speed, and size of the model. Considering all these aspects, the development of future trends based on contemporary ideas are presented below.

• Due to the continuous movements of the player, jersey numbers encounter serious deformation, and various image sizes and low resolution make it difficult to read the jersey number [205]. Players’ similar appearances and severe occlusions make it difficult to track and identify players, referees, and goalkeepers reliably, which causes the critical problem of identity switching among players [212]. To solve these challenges, a framework is needed to propose that learning objects’ identities with deep representations and improve tracking using identity information is necessary.
• The algorithms employed to detect and track the state of the ball such as whether it is controlled by a player (dribbling), moving on the ground (passed from one player to another player), or flying in the air to categorize the movement as a rolling pass or lobbing pass are not robust concerning the size, shape, and velocity of the ball and other parameters under different environmental conditions. A few researchers have come forward with novel ideas to deal with the above-mentioned aspects [62,68,69,107,203,210,285]; however, the state-of-the-art research is still at a nascent stage.
• Conventional architectures of detecting, classifying, and tracking are replaced with more promising and potential modern learning paradigms such as Online Learners and Extreme Learning Machines.
• Various activities in sports such as players’ fatigue information can be acquired from wearable sensors and monitoring health conditions of each player in the play-field, tracking players and the ball, predicting the trajectory of the ball, game play analysis and evaluations; identifying players’ actions and other fundamental elements can be accomplished with the help of big data and information technologies. This improves the sports industry’s operational efficiency and leverages its immense potential. Due to advancements in big data analysis [298,299], and the Internet of Things (IoT) [300], personalized care monitoring will become a new direction and breakthrough in the sports industry.

8.3. Different Challenges to Overcome in Sports Studies

• Classification of jersey numbers in sports such as soccer and basketball is quite simple [205] as they have plain jerseys, but in the case of the sports such as hockey and American football, the jerseys are massive and have sharp contours. Therefore, jersey number recognition is quite hard. By implementing proper bounding box techniques and digit recognition methods, better performance of jersey number recognition in every sport can be achieved.
• Action recognition in sports videos [222,223] is explicitly a non-linearity problem, which can be obtained by aligning feature vectors, by providing a massive amount of discriminative video representations. It can provide a method to capture the temporal structure of a video that is not present in the dynamic image space and analyzing salient regions of frames for action recognition.
• Provisional tactical analysis related to player formation in sports such as soccer [102], basketball, rugby, American football, and hockey, and pass prediction [89,90,91,98], shot prediction [81,301], the expectation for goals given a game’s state [82] or possession of the ball, or more general game strategies, can be achieved through AI algorithms.
• Recognition of the fine-grained activity of typical badminton strokes can be performed by using off-the-shelf sensors [192], and it can be replaced with automatic detection and tagging of aspects/events in the game and use of CCTV-grade digital cameras without additional sensors.
• The identity of the player is lost when the player moves out of the frame and, to retain the identity when the players reappear in subsequent frames, the player must be recognized. The key challenges for player recognition are posing variations, i.e., change or rotation of the image in different poses on 2D or 3D perspectives [56,302], which is the most difficult recognition challenge, especially in case of resolution effects, variable illuminations, or lighting effects and severe occlusions.

9. Conclusions

Sports video analysis is an emerging and very dynamic field of research. This study comprehensively reviewed sports video analysis for various applications such as tracking players or balls in sports and predicting the trajectories of players or balls, players’ skill, and team strategies analysis, detecting and classifying objects in sports. As per the requirements of deploying computer vision techniques in various sports, we provided some of the publicly available datasets related to a particular sport. Detailed discussion on GPU-based work stations, embedded platforms and AI applications in sports are presented. We have present various classical techniques and AI techniques employed in sports, their performance, pros, cons, and suitability to particular sports. We list probable research directions, existing challenges, and current research trends with a brief discussion and also widely used computer vision techniques in various sports.

Individual player tracking in sports is very helpful for coaches and personal trainers. Though the sports include particularly challenging tasks such as similarities between players, generation of blurry video segments in some cases, partial or full occlusions between players, the invisibility of jersey numbers in some cases, computer vision is the best possible solution to achieve player tracking.

Classification of jersey numbers in sports such as soccer and basketball is quite simple as they have plain jerseys, but in the case of sports such as hockey and American football, the jerseys are massive and come with sharp contours, due to which jersey number recognition is quite hard. By implementing proper bounding box techniques and digit recognition methods, better performance vis-a-vis jersey number recognition in every sport can be achieved. As the appearance of players varies from sport to sport, an algorithm trained on one sport may not work when it is tested on another sport. The problem may be solved by considering a dataset that contains a small set of samples from every sport for fine-tuning.

In the case of multi-player tracking in real-time sports videos, severe occlusions cause a critical problem of identity switching among the players. The continuous movement of players makes it difficult to read jersey numbers. A player’s similar appearance to another and severe occlusions make it difficult to track and identify players, referees, and goalkeepers reliably. Multiple object tracking in sports is a key prerequisite for the realization of advanced operations in sports, such as player movement and their position in sports, which will give good objective criteria to the team manager for developing a new plan to improve team performance as well as evaluating each player accurately.

Commercially used multi-camera tracking systems of players rely on some mixtures of manual and automated tracking and player labeling. Optical tracking systems are a good approach for tracking players occluding each other or players with a similar appearance. The algorithm may detect false positives from out of the court such as fans wearing team uniforms, as the appearance of fans is similar to that of players. This can be eliminated by estimating the play area or broadcast camera parameters with extra spatiotemporal locations of player positions.

Action recognition in sports videos is explicitly a non-linearity problem, which can be obtained by aligning feature vectors, by providing a massive amount discriminative video representations to capture the temporal structure of the video that is not present in the dynamic image space and analyzing the salient regions of the frames for action recognition.

The algorithms employed so far for detecting and tracking ball movements began with estimating the 3D ball position in trajectory. Employing these methods is very critical, as they include a lot of mathematical relations and require reliable reference objects to construct the path of the trajectory. Kalman filter- and particle filter-based methods are robust concerning the size, shape, and velocity of the ball. However, the methods fail to establish the track when the ball reappears after occlusion. Trajectory-based methods solve the problem of occlusion and are robust in obtaining data regarding missing and merging balls but fail to obtain data regarding the case of the size and shape of the ball. Data association methods are best suited for detecting and tracking small size balls in small courts such as tennis courts but are not suited for challenges in sports such as basketball, soccer, and volleyball. AI algorithms predict the precise trajectories of the ball from a knowledge of previous frames and are immune to challenges such as air friction, ball spin, and other complex ball movements. A precise database that includes different sizes and shapes of the ball has to be introduced to detect the ball position and enable tracking algorithms to perform efficiently.

Detection and tracking of players, balls and assistant referee as well as semantic scene understanding in computer vision applications of sports is still an open research area due to various challenges such as sudden and rapid changes in movements of the players and ball, similar appearance, players with extreme aspect ratios (players have extremely small aspect ratios in terms of height and width when they fall down on the field) and frequent occlusions. The future scope of computer vision research in sports, therefore, is handling limitations more accurately on different AI algorithms.

As the betting process involves financial assets, it is important to decide which team is likely to win; therefore, bookmakers, fans, and potential bidders are all interested in estimating the odds of the game in advance. So, provisional tactical analysis of field sports related to player formation in sports such as soccer, basketball, rugby, American football, and hockey, as well as pass prediction, shot prediction, and expectations of goals in a given game state or a possession, or more general game strategies, need to be analyzed in advance.

Tracking algorithms that are used in various sports cannot be compared on a common scale as experiments, requirements, situations, and infrastructure in every scenario differ. Determining the performance benchmark of algorithms quantitatively is quite difficult due to the unavailability of a comparable database with ground truths of different sports differing in many aspects. In addition to these, there are additional parameters such as different video capturing devices and their parameter variations which lead to difficulty in building an object tracking system in the sports field.