Automatic Detection of Camera Rotation Moments in Trans-Nasal Minimally Invasive Surgery Using Machine Learning Algorithm

Abstract

Background: Minimally invasive surgery (MIS) is an advanced surgical technique that relies on a camera to provide the surgeon with a visual field. When the camera rotates along its longitudinal axis, the horizon of the surgical view tilts, increasing the difficulty of the procedure and the cognitive load on the surgeon. To address this, we proposed training a convolutional neural network (CNN) to detect camera rotation, laying the groundwork for the automatic correction of this issue during MIS procedures. Methods: We collected trans-nasal MIS procedure videos from YouTube and labeled each frame as either “tilted” or “non-tilted”. The dataset consisted of 2116 video frames, with 497 frames labeled as “tilted” and 1619 frames as “non-tilted”. This dataset was randomly divided into three subsets: training (70%), validation (20%), and testing (10%) Results: The ResNet50 was trained on the dataset for 10 epochs, achieving an accuracy of 96.9% at epoch 6 with a validation loss of 0.0242 before validation accuracy began to decrease. On the test set, the model achieved an accuracy of 96% with an average loss of 0.0256. The final F1 score was 0.94, and the Matthews Correlation Coefficient was 0.9168, with no significant bias toward either class. The trained ResNet50 model demonstrated a high success rate in predicting significant camera rotation without favoring the more frequent class in the dataset. Conclusions: The trained CNN accurately detected camera rotation with high precision, establishing a foundation for developing an automatic correction system for camera rotation in MIS procedures.

1. Introduction

Minimally invasive surgery (MIS) is a surgical technique involving a camera to visualize the surgical field instead of opening up the patient [1]. Paralleled with rapid technological advances, the minimally invasive techniques are now a part of almost every surgical sub-specialty [2]. When compared to open surgery, minimally invasive techniques offer significant benefits such as reductions in bleeding, post-operative pain, recovery time, and length of hospital stay [3]. However, the advantages to the patient comes with a high cognitive load on the surgeon [4,5]. The high workload in MIS can significantly impact both surgical performance and jeopardize patient safety. While substantial efforts have been made to assess and evaluate surgeons’ workload, there remains limited evidence on how to effectively provide cognitive support [6,7,8,9]. The recent advancements in machine learning techniques have made it increasingly possible to bridge this gap.

As previously mentioned, the surgeon has to rely on a camera to visualize the surgical site; any disruptions to the camera will affect the visual orientation of operators, creating an environment for increased workload and operating errors [10]. Since MIS is predominantly a team-based task, the operator manipulates the surgical tools while the assistant holds the camera. It is common for the assistant to fall out of sync with the operator, leading to visual distortion [11,12]. One common type of distortions is horizontal line disruption caused by camera rotations; horizontal line disruptions cause confusion due to the mismatch between the orientation of the actual versus the perceived operative field [13,14]. Often, horizontal line disruptions are difficult to catch; unfortunately, they are typically recognized too late wherein the direction of the surgeon’s intended motion differs from that of the actual motion, causing damage to the internal organ structure [15].

Machine learning is a series of computational algorithms that takes raw input data and produces the desired output by mimicking the human cognitive process [16]. Convolutional neural networks (CNNs) are a type of machine learning algorithm that excels at image recognition. CNNs were designed to process visual inputs such as images and videos through various layers, including a convolutional layer, pooling layers, and fully connected layers. These layers allow for their excellent ability to detect and learn spatial hierarchies and edges [17]. In performance analysis, CNN significantly outperformed other deep learning models in the realm of image classification [17]. Other neural networks, like Long-Short Term Memory (LSTM), also have applications in image processing and healthcare. LSTM integrates time as a variable, allowing it to have a “memory”, thus allowing the sequential analysis of past data to predict the future [18]. However, because of the added variables, LSTM struggles to control the impact of noise in the data and may increase the amount of false predictions in complex environments [18]. Generative adversarial networks (GANs) have also been used in image classification and data synthesis, and it has two neural networks running in concert, consisting of a generator to generate synthetic data and a discriminator to classify data [19]. Despite the diverse applications, GAN suffers from the mode collapse problem where it is unable to analyze the data distribution as a whole but rather focuses on a small segment. Additionally, its complex systems slow down the processing speed, making it difficult to integrate to a fast pace surgical environment [20].

Currently, there is already some evidence showing the implementations of machine learning in MIS. Hajj et al. used CNN for instrument classification in cholecystectomy and achieved 97.9% accuracy on differentiating between different surgical tools [21]. Similarly, Laina et al. used a type of CNN in the binary segmentation of surgical data from colorectal surgery to classify surgical instrument, achieving an accuracy of 88.9% [22]. Other than tool recognition, machine learning algorithms can also recognize the phase of the surgery. By feeding laparoscopic cholecystectomy data to a CNN, the machine could accurately predict the phase of the surgery 93% of the time, which may help navigate novice surgeons during training [23]. In vitro experiments with a CNN model predicting tool trajectory using both image and kinematics data were able to achieve 70.6% accuracy, which could be used as a tool for precautionary warning before an intra-operative mistake [24]. Outside of image recognition, positional trackers placed on surgical camera can be used to determine its orientation [15]. Outside of image recognition, machine learning algorithms can also predict flow status, which has seen its application in blood flow rate during transfusions in the operating room [25]. Generalized additive models have been implemented to calculate post-operative risks like sepsis, acute kidney injury blot clots, etc., for at least 12 surgical specialties [26]. Linear Regression, Regression Trees, Support Vector Regression, and Bagging Regression Trees have all been used to estimate surgical time [27]. LSTM has also been used to predict correlation between length and resistance [28]. However, there is a current lack of comprehensive studies analyzing the ability of CNN for recognizing camera orientation during MIS.

We plan to construct a dataset comprising images from MIS with distinguishable features to train a CNN model capable of accurately identifying when the surgical camera is tilted. Based on previous literature on the application of CNNs in MIS, we anticipate that a trained CNN model will detect camera distortions with high accuracy in specific surgical contexts.

2. Materials and Methods

2.1. Environment

This study was conducted at the Surgical Simulation Research Lab at the University of Alberta. Ethics approval was obtained from the Health Research Ethics Board of the University of Alberta prior to the commencement of this study.

2.2. Data Selection and Image Processing

Twenty full-length, public, and minimally invasive nasal surgical videos totaling up to 1000 min were exported from YouTube (San Bruno, CA, USA). Nasal surgical videos were selected due to their consistent background and reference object within the nasal cavity. Selected videos were all above 480 p in quality with a frame rate above 30 frames per second and without major visual obstructions during the procedure.

The frames within the videos were then normalized to the same dimensions of 1080 px by 1080 px with a focus on the endoscope content using Da Vinci Resolve (Coral Spring, FL, USA). To determine at what degree of camera rotation is both noticeable and difficult for the operation, an expertise surgeon (YW) was shown multiple frames randomly sampled from the videos and determined that an angle deviation from the vertical line above 30° was considered noticeable and caused disorientation; additionally, according to Shepard et al., increasing the angle of rotation, in increments of 20°, results in an increase in reaction time due to the additional workload of mental rotation; an rotation of 30° yields a nearly 2 s reaction time, significantly affecting the surgical workflow [29]. Thus, angles of rotation above 30° were considered “tilted”, and frames below this threshold were considered “non-tilted”, with edge cases removed (Figure 1).

Subsequently, the videos were manually analyzed using binary classification, categorizing the camera angle in each frame as either “tilted” or “non-tilted”. Irrelevant segments of the videos, such as when the endoscope was temporarily removed from the nostril or when the camera was obstructed, were excluded. Each video was divided into segments containing only “tilted” or “non-tilted” frames, and frames of the same type were merged into two separate videos: one consisting entirely of “tilted” frames and the other of “non-tilted” frames (Figure 2).

To reduce redundancy from frames captured in close temporal proximity, the videos were processed at a frame rate of 25 frames per second, and images were sampled and labeled once per second. Each image was converted to the RGB format and augmented through random horizontal, vertical, or combined flips. The images were then resized to 224 × 224 pixels for standardization.

In total, 2116 samples were collected, with 497 frames labeled as “tilted” and 1619 frames labeled as “non-tilted”. This sufficiently large dataset ensures the robust training of the machine learning model and minimizes the risk of overfitting.

2.3. Model Selection and Training

For the purpose of this study on primarily classifying images, we used Residual Network 50 (ResNet50) as the CNN model for training; ResNet50 contains residual layers that eliminate the vanishing gradient problems, as it offers “skipping” between layers when performance does not improve and more direct backpropagation, allowing adjustments to the algorithm. Additionally, ResNet50 is a pre-trained model, which means that it already has features and basic image recognition abilities, setting the foundation for more fine-tune training. For those reasons, we believe that ResNet50 will comparatively boost the learning rate and image classification accuracy.

We split the dataset into 3 sets, each with a 1:4 ratio between tilted and non-tilted angles: training, validation, and test. A total of 1482 samples (70%) were in the training set, 424 (20%) were in the validation set, and 210 (10%) were in the test set (

Table 1. Number of video frames used for train set (70%); validation set (20%); test set (10%).

	Non-Tilt, 0	Tilt, 1
Train Set	1146	336
Validation Set	319	105
Test Set	154	56

). The samples were randomly selected to each set while maintaining the desired ratio of both classification groups. The model was trained with ADAM as the optimizer with a static learning rate of 0.001 and CrossEntropyLoss as the loss function.

3. Results

3.1. Validation Accuracy, Validation Loss, and Statistical Analysis

Validation accuracy and loss were monitored and recorded after each epoch. After the first epoch of training, the training accuracy was around 84.0%, the validation accuracy was around 88.7%, and the validation loss was around 0.0704. We continued running the training for 10 epochs and monitored the training accuracy, validation accuracy, and validation loss. After every epoch, the training accuracy increased as expected; however, the validation accuracy began to plateau at around epochs 5 and 6, reaching an accuracy of 96.9% at epoch 6 with a validation loss 0.0242 before it began to decrease in the following epochs along with an increase in validation loss (Figure 3). The increase in validation loss signifies that the model is overfitting, and future training should be avoided. Overall, a net 8.25% improvement in validation accuracy was observed before a decline in performance. A one-sample t-test using the validation accuracy of 10 epochs showed that the validation accuracy is significantly different from random chance (50%) (p < 0.0001). The 95% confidence interval for the difference between the prediction accuracy and random chance is [0.364969, 0.447271].

3.2. Accuracy Parameters and Confusion Matrix

After training, the model accuracy on the test set is 96%, with an average loss of 0.0256. The positive predicted value (PPV) was 0.92, the true negative rate (TNR) was 0.97, and the true positive rate (TPR) was 0.96. The final F1 score was 0.94. There was no observed imbalance between predicting false positive (2.4%) and false negative (1.0%) (Figure 4). To ameliorate the effects of an imbalanced dataset, the Matthews Correlation Coefficient (MCC) was calculated to be 0.9168.

$$PPV={\displaystyle \frac{TP}{TP+FP}}$$

$$TNR={\displaystyle \frac{TN}{FP+TN}}$$

$$TPR={\displaystyle \frac{TP}{TP+FN}}$$

$$F1Score={\displaystyle \frac{2\times TP}{2\times TP+FP+FN}}$$

$$MCC={\displaystyle \frac{TP\times TN-FP\times FN}{\surd (TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$

where TP = true positive, TN = true negative, FP = false positive, FN = false negative.

4. Discussion

The ability for a trained algorithm to analyze and classify intraoperative camera rotation provides an additional avenue for CNN applications within surgery and adds a layer of security during the procedure. In this study, we explored the reliability of ResNet50 to capture high-level features in trans-nasal MIS and to accurately distinguish frames of camera tilt. The accuracy of the post-training algorithm validated our hypothesis that a CNN can recognize camera tilt in specific surgical settings.

4.1. Dataset

The dataset used in the experiment was generated to minimize several dataset imbalance problems, and several factors were considered, including class distribution, data density, and dataset shift [30]. In binary classification, experimental data have shown that a 1:10 ratio between the training dataset of the two classes is alarming and significantly detrimental to feature detection, and a ratio of 1:35 is inadequate for building a model [30]. In cases where the samples between classes are imbalanced, the algorithm will learn to only predict the high frequency class as it yields the highest accuracy, thus losing the purpose of a classification algorithm. In this study, we collected enough samples for both classes with a 1:4 ratio, ensuring the validity of the training data. Additionally, to minimize the effect of the class imbalance, we generated a large enough dataset (N > 1000) to allow for rare features to be detected during training; experimentally, as the size of the training set increases, the error rate from the sample imbalance decreases [31]. Dataset shift is when the proportion of samples in each class differs in the training versus test set, which will cause minor features to not be learned [32]. To minimize the shift problem, we used the same proportions of each set of class sample when dividing the data into the training, validation, and test sets.

4.2. Training, Validation, and Test

During training, we tested for both training and validation accuracy, as well as validation loss. The validation parameters were analyzed to monitor model training status, primarily, to avoid overfitting. Overfitting occurs when the algorithm picks up information that is irrelevant to the classification and negatively influence its performance on unseen data [33]. In our study, we suspected the model to be overfitted after the sixth epoch, where the validation loss reaches its minima.

The accuracy during the classification of the test set was 96% overall. To analyze the ability of the model in predicting each class, we analyzed the true positive rate and true negative rate and found no significant difference between the two values, indicating that there was no bias in the predictions. The PPV, or precision, was measured to be 92%, indicating that, out of all the “tilted” guesses, 92% were true positives and 8% were false positives. The TPR, or recall, was 96%, which means that, out of 56 total true positives, the algorithm was able to correctly identify 54 of them. Combining both precision and recall, we can see that, for each “tilted” guess the algorithm provides, it has a 92% chance of being correct, and it was able to catch 96% of all the “tilted” moments. The TNR, or specificity, was measured as 97%, which means that, out of all the non-tilted moments, the algorithm was able to catch 97% of them. To prevent the algorithm from “cheating” and only guessing the high frequency class, the F1 score and MCC was measured to summarize the performance of the algorithm and assess for biases. The F1 score was measured as 0.94, and the MCC was 0.9168, which indicates that the model’s performance is well balanced despite dataset imbalance. When combining AI with medicine, the primary focus has always been to avoid false negatives, as misdiagnosis can lead to delayed treatment [34]. However, in the surgical environment, any sound or alert can also cause the deflection of attention from the surgeon, leading to poorer outcomes [35]. Thus, we believe that, in the case of MIS, having a high false positive rate is also detrimental, highlighting the importance of a balanced false positive rate and false negative rate. Trans-nasal procedures usually have a relatively stable surgical environment under the camera; however, there are some preoperative and intraoperative variability. For example, depending on the type of procedure, location of the surgical site, and the surgeon’s preference, the operation could be at different nostrils. In trans-sphenoidal pituitary tumor surgery, the infra-nasal portion of the procedure will depend on the location of the tumor [36]. However, we expect that our model will have a stable performance in either nostril as the training set contains normalized images vertically flipped to ensure that there are representations for both the left and the right nostril to eliminate bias.

4.3. Limitations

This study has a few limitations. In the data collection process, we used open-source videos from YouTube, which has certain biases for promotion or demonstration purposes. Thus, less camera tilt will be shown within the video, contributing to dataset imbalance. Secondly, during the training, the algorithm only learned 10 epochs. Increasing the number of epochs could yield a better model, although it would risk overfitting. Lastly, due to the limitation of existing pre-trained algorithms, a stable, recognizable background has to be maintained for learning and classification to occur; thus, the algorithm can only recognize camera tilt in the trans-nasal setting, reducing its generalizability to all types of surgeries. Additionally, this model lacks training for minor rotations; thus, it would be less sensitive to minor camera rotations. However, while camera rotations are dangerous in MIS, smaller rotations are generally less disruptive to the surgical workflow. Our model aims to identify critical moments during the procedure when unintended camera rotations cause disorientation, making it more sensitive to angle tilts above 30°. Increasing its sensitivity to minor tilts could introduce a level of distraction that might negatively affect the surgical flow more than the rotations themselves.

4.4. Future Directions

This study provides a solid foundation for a more comprehensive algorithm that can be generalized to all types of MIS; moreover, future algorithms can be integrated within the operating room to alert surgeons when the horizontal line is disrupted, preventing mistakes and leading to better patient outcomes. Additionally, with the automatic detection of camera tilt, the automatic correction algorithms can be developed to allow seamless surgical flow without alerting the surgeon.

5. Conclusions

Machine learning offers a valuable approach to supporting surgeons. In trans-nasal surgery, camera positioning plays a critical role in proprioception and orientation within the surgical field. In this study, we utilized open-source surgical videos to create a dataset and trained a ResNet50 model to classify moments of camera rotation. Our results demonstrate that the ResNet50 model is effective for the automated detection of camera rotation in trans-nasal procedures. Further refinement of this method will be essential to minimize false positives and prevent unnecessary distractions during surgery, ultimately enhancing its clinical utility.