On the Use of Muscle Activation Patterns and Artificial Intelligence Methods for the Assessment of the Surgical Skills of Clinicians ^†

Abstract

The ranking and evaluation of a surgeon’s surgical skills is an important factor in order to be able to appropriately assign patient cases according to the necessary level of surgeon competence in addition to helping us in the process of pinpointing the specific clinicians within the surgical cohort who require further developmental training. One of the more frequent means of surgical skills evaluation is through a qualitative assessment of a surgeon’s portfolio alongside other supporting pieces of information, a process which is rather subjective. The contribution presented as part of this paper involves the use of a set of Delsys Trigno EMG wearable sensors, which track and record the muscular activation patterns of a surgeon during a surgical procedure, alongside computationally driven artificial intelligence (AI) methods towards the differentiation and ranking of the surgical skills of a clinician in a quantitative fashion. The participants in the research involved novice-level surgeons, intermediate-level surgeons and expert-level surgeons in various simulated surgical cases. A comparison of different signal processing approaches has shown that the proposed approach can prove beneficial in monitoring and differentiating the skillsets of various surgeons for various kinds of surgical cases. The presented method could also be used to track the evolution of the surgical competencies of various trainee surgeons at various stages during their training.

1. Introduction

The ability to robustly assess and estimate a surgeon’s skillset is a core component of surgical training and education and aids towards the identification of the competence level of a particular surgeon [1,2]. The literature suggests that current means used for the assessment of these surgical competence levels mostly involve the review of tapes, which are ultimately interpreted and assessed by a peer reviewer, thereby opening the process up to factors such as bias and subjectivity in addition to being costly [3]. This has given rise to the application of alternate means of skill assessments primarily based around the use of kinematic and virtual reality measures alongside artificial intelligence methods for the classification of surgical competence levels, with the aid of objective and quantitative prediction machines [4,5,6,7,8,9].

Recent work has also seen the use of wearable sensors—attached to the surgeon’s upper limb—as part of the skill identification process in an attempt to identify movement and contraction patterns using electromyography (EMG) and accelerometers (ACCs). This was notably conducted in the work of Soangra et al. [3] who, as part of their studies, applied both sets of wearable sensors in various anatomical locations to a group comprising novice, intermediate and expert participants in order to distinguish skill and competence levels across a number of simulated surgical tasks. From this, they were also able to identify a number of anatomical “hotspots” whose movements encoded key information regarding the levels of competence during surgical tasks [3,10]. As part of the postprocessing of the acquired signals from the wearable sensors, it can be seen that a concise list of features was extracted from the signals, which happened to be mostly nonlinear features [3,10,11]. Although effective in the characterization of signals, these could be expanded upon and concatenated with other linear features in order to boost the overall modelling accuracy [3,10,11]. In addition to this, the related literature is yet to explore the use of multiresolution and signal decomposition algorithms such as deep wavelet scattering (DWS), linear series decomposition learner (LSDL) and empirical mode decomposition (EMD), to name a few [12,13,14].

The contributions presented in this work present a first-stage investigation on the use of varied signal processing approaches towards the classification of surgical expertise based on the signals obtained from the EMG sensors in particular. In this paper, expanded signal processing approaches are used to differentiate between surgical skillset and expertise for a specific task based solely on the acquired signals from areas deemed to be anatomical hotspots (as determined by a previous study [3]), which span the deltoid, biceps and extensor carpi ulnaris (ECU). From this, it is immediately hypothesized that the assembled model can form a basis for the evaluation of surgical skills with a much more robust approach, which can be used to rank surgeons based on appropriate levels of expertise for different kinds of surgeries ranging from basic all the way towards minimally invasive and robotic surgeries.

2. Materials and Methods

2.1. Dataset

The original dataset was acquired from a broad list of subjects of varied surgical expertise at the Department of Urology at the University of California, Irvine, from which all subjects provided written consent to take part in the study [3]. The subjects comprised three expertise classes as follows: novice surgeons who were individuals without surgical experience; intermediate surgeons who were primarily urology residents; and expert surgeons who were urology doctors with over five years’ worth of experience [3]. For the work carried out in this paper, one participant was taken from each class as part of the pilot exercise. Various surgical tasks were performed and conducted, while the pegboard transfer task was the surgical task used as part of the accompanying signal processing work carried out in this paper. The EMG electrode used was the DELSYS Trigno Wireless, Boston, MA, USA (for which data were sampled at 2 KHz), which was attached on a number of anatomical locations determined by the surgical ergonomics identified from prior studies [3]. Specifically, the deltoid was also chosen as a site of interest due to it being an area where laparoscopic surgeons report musculoskeletal pain [3]. Figure 1 is an image of one of the subjects performing the pegboard transfer task.

2.2. DWS

DWS is an unsupervised feature extraction approach that is capable of extracting features which are robust, continuous and a factor of a fused ensemble between the wavelet decomposition and the convolutional neural network (CNN) [15]. For the DWS, both the wavelets and filters are set at fixed values to prevent any form of iterative computations of these values, and it is able to work well with a small set of samples [15]. As mentioned, the mathematical formalism of the method can be seen in the published work of Andén and Mallat [15]. As part of the computational implementation, the DWS works with a CNN, which works in an iterative sense whilst performing convolutions through the wavelets and nonlinear modules with an average scaling function [15].

The implementation of the CNN in this work involved the use of the Gabor wavelet as the mother wavelet with a scale invariance of 1 s, with the filter banks of eight wavelets per octave in the first filter bank being followed by one wavelet per octave in the second set of filter banks.

2.3. Feature Extraction and Machine Learning Models

Prior to feature extraction, the EMG signals were windowed using a series of windows of 10,000 samples each, of which 10 windowed segments among these were used. The following features were extracted from the EMG signals which comprised a concatenation of both linear and nonlinear features: mean, waveform length, slope sign change, root mean squared, cepstrum, maximum fractal length, median frequency, simple square integral, variance, 4th order autoregressive coefficient, Higuchi fractal dimension, detrended fluctuation analysis, peak frequency, and sum of peaks [10,11].

The following machine learning models were used as part of this paper: decision tree (DT), linear discriminant analysis (LDA), linear support vector machine (LSVM), quadratic support vector machine (QSVM), cubic support vector machine (CSVM), fine Gaussian support vector machine (FGSVM) and K-nearest neighbors (KNN). The K-fold cross-validation approach was utilized for the validation of all models, where K was chosen as 10.

3. Results

Table 1. Classification accuracies of the various models for the raw signal and DWS.

Model	Raw Signal/Handcrafted Features (%)	DWS (%)
DT	87	92
LDA	83	86
LSVM	76	90
QSVM	90	97
CSVM	93	99
FGSVM	95	92
KNN	95	99

shows the results for the various scenarios investigated using the raw signal as well the DWS, with various models using different configurations. For the case of the raw signal, it can be seen that the results benefitted from a more complex model with a nonlinear architecture. It can be seen that the models with linear decision boundaries produced a dampened classification accuracy, which was seen to be improved upon by being trained with models with nonlinear decision boundaries. The DWS produced an improved classification accuracy across the majority of the models when compared with the raw signal results. The classification accuracy is seen to be improved through the use of the decomposition algorithm which provides unsupervised features and therein shows that the concept of decomposing the signal is beneficial in this case study. The machine learning models with nonlinear decision boundaries were also seen to be the best performing in this case.

This shows that the proposed model and methods in this paper could serve towards enhancing the recognition accuracy of the use of wearable sensors for the classification and assessment of surgical skills expertise.

4. Conclusions and Future Work

The use of wearable sensors has gained momentum for the characterization of muscular activation patterns as a means towards differentiating between the skillsets of various surgeons for competency purposes. In this paper, we have attempted to use an expanded feature extraction method, alongside the DWS, towards further analysis of the EMG signal from a group of subjects, in order to investigate the extent to which these methods aid towards differentiating various surgical skillsets. This exercise was conducted for the pegboard transfer task and for three subjects, i.e., one from each skill class. The results show that the DWS is capable of differentiating between the various classes to a greater degree than the raw signal.

Subsequent work in this area would involve the use of a broader sample set comprising more subject participants and a variety of surgical tasks, including tasks involving the use surgical robots, along with data from accelerometers to serve as a basis of comparison with the EMG. In addition, preprocessing of the data through the use of the LSDL signal decomposition algorithm, which has been seen to help boost the predictive performance of machine learning algorithms, would also be performed [16,17].

To conclude, these interim results suggest that the use of wearable sensors does indeed carry appeal for non-subjective interpretations of the skillsets and competencies of clinical surgeons.