Shoulder Implant Manufacturer Detection by Using Deep Learning: Proposed Channel Selection Layer

Abstract

Total Shoulder Arthroplasty (TSA) is the process of replacing the damaged ball and socket joint in the shoulder with a prosthesis made with polyethylene and metal components. After this procedure, intervention may be required as a result of damage to the prosthesis, except for the need for an examination regarding the prosthesis at certain periods. If the patient does not have information about the model and manufacturer of the prosthesis, the treatment process is delayed. Artificial intelligence-assisted systems can speed up the treatment process by classifying the manufacturer and model of the prosthesis. In this study, artificial intelligence methods were applied to classify shoulder implants using X-ray images. The model and manufacturer of the prosthesis is detected by using the proposed deep learning method. Besides, the most commonly used machine learning classifiers were applied for the same problem to compare the results and show the effectiveness of the proposed method. In addition, the accuracy and precision analysis and measurements of the processing times for the applied methods were performed to reveal the performance, accuracy, and efficiency of the study. In order to measure the performance of the proposed method, it was compared with studies on the same problem in the literature. As a result of the comparison, it was found that the proposed method, with an accuracy rate of 97.2%, performed better than the other studies. In the study, the implant manufacturer and model are classified in order to carry out the implant surgery process in the best way with the proposed deep learning model. With the success of the proposed system, the applicability of this model in similar prosthesis classifications has been shown. Differently from the studies in the literature, the channel selection formula is presented in the proposed deep learning method recommended for selecting the most distinctive feature filters.

1. Introduction

Total Shoulder Arthroplasty is a type of surgery performed on a damaged joint [1,2]. In particular, calcification and rheumatoid arthritis can cause shoulder damage. As a result of the trauma caused by the damage, surgical procedure may be required on the patient. With the total shoulder prosthesis procedure, the joint that has lost its function due to damage and is not fully functional is removed and a prosthetic joint is placed in its place [3,4]. In addition, total shoulder prosthesis is also performed due to damage to the cartilage tissue and the surrounding bones. The shoulder joint is one of the largest joints in the body [5]. In addition, it forms a complex structure with the surrounding muscle, fiber, nerve and ligament groups. It is also one of the most commonly used joints because it combines the body and the arms. For this reason, the shoulder joints are one of the most valuable structures for human life. X-ray images are used to decide the fit of the prosthesis before total shoulder prosthesis surgery. It also benefits from the X-ray images for evaluating the correct placement of the prosthesis after surgery. There are many manufacturers that produce prostheses for TSA. These manufacturers develop different prosthesis models compatible with cases and patients. After the prosthesis is applied to the patient by surgical procedure, it may be necessary to repair the prosthesis for certain periods. In addition, the prosthesis may be damaged as a result of some situations, such as an accident, and may need to be replaced. If the prosthesis needs to be maintained or replaced, the manufacturer and the model information of the prosthesis should be known by the patient or the doctor. Otherwise, the process will not progress, and treatment will be delayed without this information. In such a case, the X-ray images of the implant are examined in detail and compared with the implant images which are available. This study aims to determine the shoulder implant manufacturer and model from X-ray images by using the deep learning method. A computer-aided shoulder implant determining system is used to help to the scientist to identify the implant′s manufacturer and the model information. In this way, even if there is no information about implant′s manufacturer and its model, there will be no loss of time and effort in the treatment and surgical process. Urban et al. [6] have classified shoulder implants according to manufacturer using the CNN-based deep learning method. The study aimed to classify implants by manufacturers only. In addition, the pretrained models were also used for modelling. The classic CNN architecture was applied in the model. The Shoulder Implant X-ray dataset [7], which is available in the UCI open access repository, was utilized in their study. In this study, the dataset which was used by Urban et al. [6] has been applied. It has been considered that a better classification result would be obtained by proposing a novel deep learning architecture. In addition, it has been studied to classify shoulder implants by manufacturer and model in the study. In our study, the shoulder implants were classified by both manufacturer and model for extend to extend Urban et al. [6]’s study.

In this study, a deep learning model will be proposed for shoulder implant classification. Apart from that, the most commonly used convolutional neural network methods and traditional machine learning methods will be applied to the same problem for to compare the performances of the methods. A computer-aided shoulder implant determining system is used to help to the scientist to identify the implant’s manufacturer and the model information.

A new layer is implemented that uses a channel selection formula to generate the filter features in the proposed deep learning method. The proposed method is based on the Convolutional Neural Network (CNN) architecture. The first contribution of the study is to propose a new channel selection method that can be applied in deep learning methods. The second contribution is to detect implant manufacturer and model by using deep learning architectures and machine learning algorithms. The last contribution is to show the applicability of the proposed model based on its high performance.

The paper consists of four chapters. In the first chapter, the literature survey is presented. In the second part, information about the dataset and the proposed CNN architecture and other applied methods are presented. The third section consists of the experimental study, and the last sections contain the discussion and conclusions.

Related Works

Urban et al. [6], in 2020, classified the shoulder implants’ manufacturers from the X-ray images. In the study, machine learning algorithms were applied to the problem along with deep learning architectures such as NasNet, DenseNet, VGG, ResNet and Custom CNN. Their study presents the results obtained for different classifiers via 10-fold cross-validation. Among the methods applied, the highest accuracy rate was obtained by using the NASNet deep learning architecture with pretraining on ImageNet, at 80.4%, with no data augmentation. The DenseNet-201 architecture achieved the highest accuracy rate, at 78.9%, when the data augmentation was used.

Stark [8], in 2018, provided automatic detection and segmentation of TSA implants in X-ray images. The Hough transform method was applied to the images for circles to locate the implant. After this process a simple seeded region growing method was used for segmentation. The best accuracy rate was obtained from the applied algorithms is 94%.

Lindsey et al. [9], in 2018, developed a deep neural network to detect and localize fractures on radiographs. While the sensitivity of the clinician was 80.8% without system support, this increased to 91.5% with the help of the developed system.

Bredow et al. [10], in 2014, matched X-ray images with 3D knee prostheses. In the study, the methodology is separated into three parts. These are template image generation, X-ray image segmentation and template matching. As a result, an accuracy rate of 70% for lateral views and 90% for frontal views was obtained.

Nachimuthu et al. [11], in 2012, developed a multiple classification system for fracture detection in human bone X-ray images. In the study, Back Propagation Neural Network, K-Nearest Neighbor, and Support Vector Machine classification algorithms were applied on texture and shape features.

Takahashi et al. [12], in 2020, detected dental implants with deep learning methods. In the study, an object detection algorithm (Yolov3) was applied with deep learning. As a result, an accuracy rate of 72% was obtained.

Kim et al. [13], in 2020, used the deep learning methods for implant fixture system classification with periapical radiographs. Among the deep learning methods applied in the study, MobileNet-v2 achieved the highest accuracy with 97%.

Sukegawa et al. [14], in 2020, used the deep neural networks method for dental implant classification. Five CNN models were applied for implant classification. Among the five models, the finely tuned VGG16 model had the highest accuracy rate (90.7%).

Sharma et al. [15], in 2019, developed a convolutional neural network model to control a human arm prosthesis with sEMG signals. A classification accuracy of 96.2% was achieved in the CNN model.

Yi et al. [16], in 2020, used deep learning for the automatic detection and classification of shoulder arthroplasty models. In the study, ResNet DCNN-based binary classifiers were used for detection and the accuracy rate was 97%.

Gowd et al. [17], in 2019, predicted the rate of short-term postoperative complications following total shoulder arthroplasty using machine learning algorithms. In the study, the highest accuracy rate was found to be 77%.

Borjali et al. [18], in 2020, used deep convolutional neural networks to determine the design of a total hip prosthesis. The performance of the CNN model was compared with the responses of orthopedic surgeons. While the CNN model gets faster results, its accuracy rate is on par with the orthopedic surgeons.

2. Materials and Methods

2.1. Dataset

The dataset used in the study is the Shoulder Implant X-ray Manufacturer Classification dataset available in the UCI open access repository [7]. The dataset includes 597 X-ray images. These images consist of 4 different implant manufacturers (Tornier, Cofield, Zimmer, Depuy) and 16 different models. The dataset includes 71 X-ray images from Tornier (Bloomington, IN, USA), 83 X-ray images from Cofield (Aarau, Switzerland), 149 X-ray images from Zimmer (Warsaw, IN, USA), and 294 X-ray images from Depuy (Warsaw, IN, USA), (Figure 1). In the study, manufacturer and model information of implants were obtained from the University of Washington Shoulder website [19]. Commonly, shoulder implants are classified into different generations. First-generation implants like Cofield are monoblock and nonmodular. Depuy is classified in the second-generation implant class with its modular form. Third-generation implants like Tornier and Zimmer are inclined and modular with adaptable variables. Three implant models by manufactured Depuy were used in the study. There is no space between the head and body of the HRP model implant, which is monoblock in design with a dorsal three holed fin. The Global model is modular. In this implant model a space between upper and rim of stem and head is designed. The Global Advantage implant model is also modular. The difference of the global model from Global Advantage implant model is that it has a more pronounced front edge with three holes. The Tornier manufacturer’s implants are distinguished by two holes and a distinctive triangular dorsal fin with a medialized eccentric head. Cofield implants have a pointed distal stem and more space under the head. It differs from other manufacturers with these features. The Zimmer implant model is designed as monoblock with dorsal and ventral fins. The Zimmer implant’s stem tip is rounded. Furthermore, it lacks a ridge under the head [19]. The number of data are sufficient for modelling deep learning architectures owing to model label support and data augmentation applied.

Some of the X-ray images have low resolution or low contrast. Apart from that, the aspect ratios of the images are different. X-ray images have been preprocessed due to the low resolution and low contrast found in some of the images (Figure 2).

Besides, data augmentation was used to increase the number of X-ray images used in the study. For this, minor changes have been made to the images in the current dataset, such as flips or translations or rotations. In this way, the model perceived these images differently during learning. The training of the model was strengthened with the synthetic data created in this way.

2.2. Deep Learning

The artificial neural networks method is an artificial intelligence method that simulates the neuron structure in the brain [20]. With this modeling, the learning and adaptation ability is gained. Deep learning includes the artificial neural network architecture on each layer [21,22,23]. Unlike machine learning algorithms, the deep learning method makes feature selections by itself. For feature extraction, the most insignificant features are filtered in the first layers, while the most important feature is extracted in the last layer. In addition, the strength of the deep learning method compared to other artificial intelligence methods is that it can produce fast solutions for complex problems due to the large number of processing units it contains. Due to the high hardware power requirement of these processes, this support is provided through the computer′s GPU (Graphics Processing Unit). With the GPU’s support, the training process of the model is completed in a shorter time even for complex problems [24,25,26]. In the study, the proposed CNN-based deep learning model and the most frequently used ResNet-50, DarkNet-53, AlexNet, VGG-19, Inseption-ResNet deep learning networks were applied to the shoulder implant detection problem.

2.3. Proposed Channel Selection Formula

In this section, the channel selection formula is introduced in the proposed model for selecting the most discriminatory feature filters. Yılmaz [27], diagnosed COVID-19 from lung X-ray images using a different deep learning network architecture with a feature selection layer which was successfully applied. In addition to implementing the 3-channel architecture and using different activation functions, the scores on all feature maps were summed to calculate the probability of each class, as distinct from this study. The feature selection formula was used with a different deep learning architecture to classify shoulder implant manufacturer and model in this study. Firstly, the probability of each class is calculated using the convolution operator in the proposed method. In Equation (1), the final score(F_S) is summed over all channels of the feature maps.

$$F_{S_k}={\displaystyle \sum }_{i=1,j=1}^p{x}_{i,j}\ast w_{i,j}$$

(1)

In Equation (1), x is the output feature of network, w represents the convolution’s kernel and P is the set of pixel positions. The standard deviation value gives the distribution of all available scores. The main thing is to calculate how many times below or above the standard deviation from the average score by using the standard deviation value and scores. As a result of this thought, the relative position of each score to all scores can be expressed mathematically. Following, these scores can be compared with each other. The absolute grade of the final score is a function of the absolute grade average of all final scores and a standard deviation. The function gives the success of this final score (Equation (2)). In these equations, K is the number of channels and $\overline{F_S}$ is the success of final score.

$$Avg=\frac{{{\displaystyle \sum }}_{k=1}^N{F}_{S_k}}{N}, Std=\sqrt{\frac{1}{N}{\displaystyle \sum }_{k=1}^N{\left(F_{S_k}-Avg\right)}^2}, S_{F_{S_k}}=\frac{F_{S_k}-Avg}{Std}$$

(2)

A specific example of prediction probability calculation with the $\overline{F_S}$ formulas is shown in Figure 3.

2.4. Experimental Studies

The computer which was used in the study has an i7-3.50GHz processor and 16 GB of memory. All methods have been applied on Jupyter Notebook application with Python programming language for the detection of the shoulder implant manufacturer and model (Figure 4).

The proposed multichannel CNN model has a three-channel convolution layer. After the images are preprocessed, the model starts with the global convolution layer. It is designed as three channels in total, with this layer feeding two different convolution layers. There are three convolution layers, two pooling layers, an ReLU linear unit and fully connected layer in these layers. While it has a 64 × 64 × 3 convolution core in the first convolution layer in the global channel; it has 32 × 32 convolution cores in the second convolution layer and 16 × 16 convolution cores in the last convolution layer. In the second convolution channel in the architecture, 16 × 16 convolution cores are in the first convolution layer, 8 × 8 convolution cores are in the second convolution layer, and 4 × 4 convolution core is in the last convolution layer. There are 32 × 32 and 16 × 16 convolution cores in the first two convolution layers of the third convolution channel, and 8 × 8 convolution cores in the last convolution layer. The fully connected layer of the global convolution channel produces 500 size output, while the full connected layer of the other two convolution channels produces 100 size output. Outputs from all channels are calculated over the channel selection layer in order to produce the best result of the proposed multichannel CNN architecture with the formulas given in Section 2.3. After these feature maps were linked to fully connected layers, Softmax and Bounding Box activation was performed to determine the classification possibilities. The proposed model is shown in Figure 5.

The total number of parameters in the model is 12,447,924. Of these, 8,228,220 are in the global convolution channel and 4,219,704 are in other convolution channels. The training of the models took approximately 45,000 s and 500 iterations. In the training set, while the error value decreased until the 500th iteration, no decrease was observed in the training process after the 500th iteration.

Data augmentation was applied to the existing X-ray images before the training set was presented to the model. In the study, the size of dataset was increased by a factor of 2 with rotating the X-ray images 180 degrees. Because of this, the size of the images was preserved. Then, the images in the dataset were brought to equal size (128 × 128) by preprocessing. In the study, K-Folds Cross Validation was used to determine the training and test set. In K-Folds Cross Validation, the data is divided into k different subsets. While the k-1 subset is used for training, the last subset is used for testing. The average error value obtained as a result of k experiments stated the validity of the model. Firstly, the K value was tested as 10 and 15. With these values, the accuracy rate did not increase significantly. However, the training period increased considerably. Therefore, the K value was determined as 5 in the study. After the learning processes of the models were completed, the models were validated with the test set (Figure 6).

The most commonly used deep learning methods were applied to the problem along with the proposed model for shoulder implant detection. In addition, the logistic regression, support vector machine, naive bayes, random forest and XGBoost machine learning methods were also used on the same problem for comparison.

3. Results

In this study, the proposed model with a novel channel selection layer is presented which was used for shoulder implant manufacturer and model detection. Besides, the deep learning model with a novel channel selection layer was proposed and the formulas are presented. In the study, the “Shoulder Implant X-ray Manufacturer Classification” dataset was used which is in the UCI open access repository. We took 597 images from the dataset in order to detect shoulder implant brand and modal. Since the quantity of available data was insufficient for training, the number of X-ray images was increased by using data augmentation to train the model better. After the images were rotated 180 degrees, a total of 1194 X-ray images were obtained. Fivefold cross validation was used to determine the training and test set.

In addition to the proposed model, ResNet-50, DarkNet-53, AlexNet, VGG-19, Inseption-ResNet deep learning networks and machine learning algorithms have been applied to the same problem to compare their performance and evaluated. Besides, it was compared with previous studies in the literature that made shoulder implant classifications using the same dataset. Statistical information about the deep learning methods which were applied for shoulder implant detection is given in

Table 1. Comparison of applied deep learning models’ performances with no data augmentation.

Criteria	Proposed Model	ResNet-50	Dark-Net-53	AlexNet	VGG-19	Inseption-ResNet
Accuracy	97.20	92.73	93.85	94.41	91.06	95.53
Sensitivity	0.98	0.95	0.965	0.959	0.95	0.965
Specificity	0.019	0.05	0.034	0.041	0.048	0.034
Precision	0.986	0.956	0.958	0.972	0.937	0.979
RMSE	1.32	3.93	3.09	2.44	4.87	1.93
F1	0.983	0.953	0.962	0.965	0.944	0.972
Training Time (s)	46,723	45,752	45,250	44,213	45,182	45,923
Testing Time (s)	17,412	16,823	16,532	15,883	16,351	16,938

and

Table 2. Comparison of applied deep learning models’ performances with data augmentation.

Criteria	Proposed Model	ResNet-50	Dark-Net-53	AlexNet	VGG-19	Inseption-ResNet
Accuracy	96.31	88.69	90.95	93.88	86.51	93.13
Sensitivity	0.97	0.92	0.95	0.96	0.93	0.95
Specificity	0.021	0.07	0.047	0.036	0.066	0.044
Precision	0.974	0.923	0.927	0.955	0.891	0.957
RMSE	1.46	6.12	5.07	3.02	8.47	3.46
F1	0.976	0.926	0.939	0.959	0.91	0.956
Training Time (s)	81,792	80,103	79,275	77,432	79,112	80,392
Testing Time (s)	26,215	25,291	24,821	23,947	24,672	25,493

; the accuracy performance chart is shown in Figure 7. The logistic regression (LR), support vector machine (SVM), naive bayes (NB), random forest (RF) and XGBoost algorithms were also applied for supporting the applicability of the proposed model. Statistical accuracy analysis of these machine learning methods is shown in

Table 3. Comparison of applied machine learning algorithms′ performances with data augmentation.

Criteria	LR	SVM	NB	RF	XGBoost
Accuracy	70.43	77.38	74.62	81.74	83.91
Sensitivity	0.79	0.87	0.85	0.91	0.92
Specificity	0.191	0.114	0.128	0.08	0.07
Precision	0.75	0.804	0.75	0.823	0.876
F1	0.772	0.838	0.798	0.865	0.898

. The values which were shown in Table 1, Table 2 and Table 3 are the averages of five-fold cross validation.

The accuracies of the models are, respectively: 97.2−96.31% for proposed model, 92.73−88.69% for ResNet-50, 93.85−90.95% for Dark-Net-53, 94.41−93.88% for AlexNet, 91.06−86.51% for VGG-19 and 95.53−93.13% for Inseption-ResNet (Table 1 and Table 2).

These statistical results show that the proposed model is more successful than the other applied modals. The reason that the results of the proposed model are better than the other models applied is that the channel selection layer determines the most prominent feature vector. The performance of the proposed method for two situations in the study was compared with the studies on similar problems in the literature. It is shown in

Table 4. Comparison of results.

Reference Study	Classifier	Problem	Accuracy (Best%)
Urban et al. [6]	NasNET-CNN	Shoulder Implants	80.4
Stark [8]	Hough Transform	Shoulder Implants	94
Lindsey et al. [9]	CNN	Fractures	90.1
Bredow et al. [10]	CNN	Knee Prosthesis	90
Takahashi et al. [12]	Yolov3-CNN	Dental Implants	72
Kim et al. [13]	CNN	Implant Fixture	97
Sukegawa et al. [14]	CNN	Dental Implants	90.7
Yi et al. [16]	ResNET-CNN	Shoulder Arthroplasty	97
Proposed Model	CNN	Shoulder Implants	97.20
Proposed Model (with data augmentation)	CNN	Shoulder Implants	96.31

that the performance of the proposed model is higher than the other studies in the literature.

4. Discussion

For each applied situation, the proposed model achieved more successful results in terms of all performance evaluation criteria compared to both the commonly used deep learning models and the machine learning algorithms. The success of a machine learning algorithm depends on the choice of appropriate features. Various preprocesses, size reduction, feature selection etc. are performed to reveal these features. It is necessary to get rid of the dependence on features to increase the accuracy and reduce the transaction cost of these processes. In this study, the deep learning architectures have a higher accuracy rate and a more efficient system owing to their more successful, easier and lower cost extraction of the valuable features from the data. The results of the performance criteria and the processing times of the applied models, which were shown in Table 1, Table 2, Table 3 and Table 4, show that the proposed model performs much better than the other studies in the literature.

In addition, it should be considered that classification may be desired for a manufacturer or model X-ray image not included in the training set. To solve this situation, the dataset should be increased for more manufacturers or another manufacturer/model implant may be recommended to a patient by providing the closest similarity.

5. Conclusions

In this study, a proposed model is presented with the novel channel selection formula which selects the most prominent feature filters, unlike in the literature. The average score and standard deviation value of the channels are important for the channel selection formula. The current score for each feature is evaluated again with these values. In this way, it is possible to calculate which feature will be the most prominent and dominant among the features. This is the reason why the accuracy rate increases with a multichannel network model. The accuracy rate was increased significantly by using the most effective feature selection, which was applied among the channels for each image. The process of determining an appropriate weight for each feature extracted in the different stages of the proposed model is shown by this formula. The proposed model was applied for shoulder implant classification. Then, other deep learning and machine learning methods were applied to the same problem. Thus, the performance comparison of the applied methods was made. When the results are examined, it is shown that the proposed method is applicable in terms of both performance criteria and time efficiency. In addition, the predictability of the shoulder implant manufacturer and model with the deep learning method is presented in the study when there is an urgent need for surgical operation.