Diagnosis of Histopathological Images to Distinguish Types of Malignant Lymphomas Using Hybrid Techniques Based on Fusion Features

Abstract

Malignant lymphoma is one of the types of malignant tumors that can lead to death. The diagnostic method for identifying malignant lymphoma is a histopathological analysis of lymphoma tissue images. Because of the similar morphological characteristics of the lymphoma types, it is difficult for doctors and specialists to manually distinguish the types of lymphomas. Therefore, deep and automated learning techniques aim to solve this problem and help clinicians reconsider their diagnostic decisions. Because of the similarity of the morphological characteristics between lymphoma types, this study aimed to extract features using various algorithms and deep learning models and combine them together into feature vectors. Two datasets have been applied, each with two different systems for the reliable diagnosis of malignant lymphoma. The first system was a hybrid system between DenseNet-121 and ResNet-50 to extract deep features and reduce their dimensions by the principal component analysis (PCA) method, using the support vector machine (SVM) algorithm for classifying low-dimensional deep features. The second system was based on extracting the features using DenseNet-121 and ResNet-50 and combining them with the hand-crafted features extracted by gray level co-occurrence matrix (GLCM), fuzzy color histogram (FCH), discrete wavelet transform (DWT), and local binary pattern (LBP) algorithms and classifying them using a feed-forward neural network (FFNN) classifier. All systems achieved superior results in diagnosing the two datasets of malignant lymphomas. An FFNN classifier with features of ResNet-50 and hand-crafted features reached an accuracy of 99.5%, specificity of 100%, sensitivity of 99.33%, and AUC of 99.86% for the first dataset. In contrast, the same technique reached 100% for all measures to diagnose the second dataset.

1. Introduction

Malignant lymphoma is considered to be one of the most dangerous and complex cancers that lead to death. The exposure of lymphocytes to malignant lesions leads to the formation of malignant tumors [1]. Lymphoma accounts for 50% of malignant blood diseases [2]. According to a World Health Organization (WHO) report, there are approximately 100 types of malignant lymphoma. The most common types of malignant lymphoma are mantle cell lymphoma (MCL), follicular lymphoma (FL) and chronic lymphocytic leukemia (CLL) [3]. The most common symptom of the FL type is the emergence of a mass or many accumulated masses. This is due to the accumulated lymphoma cells leading to the emergence of a tumor in the neck area or above the collar bones and can develop in the armpits and thighs [4]. The MCL type develops in the mantle that is part of the lymph nodes and affects B cells. B cells begin to grow in an abnormal way, forming a mass of lymph nodes. These nodes appear in the area where they arise and cause problems within the lymphatic system. The MCL is a non-Hodgkin lymphoma of the B-cell. This type begins with the enlargement of the lymph nodes and spreads to the bone marrow, liver, and digestive system. The causes of MCL are unknown and have an abnormal genetic mutation; scientists do not know the cause of this mutation [5]. The CLL type is a leukemia that begins in the white blood cells; it is due to a disorder of the bone marrow and an inability to appropriately produce white blood cells. CLL slowly grows, and the affected person may not have any symptoms for years. However, the CLL cells begin to appear in other parts of the body, such as in the liver, lymph nodes, and spleen [6]. Due to the many types of malignant lymphomas, it is difficult for pathologists to determine the type of lymphoma. Microscopic images of biopsies taken from the patient are considered to be the gold standard for determining the type of lymphoma and for determining the appropriate treatment [7]. The histopathological diagnosis of malignant lymphoma is made to infer the type of malignant lymphoma and is performed using a slide stained with hematoxylin and eosin (H&E). A specific immunohistochemical (IHC) set is selected to identify the type of malignant lymphoma. Finally, a diagnosis of the malignant lymphoma is made by selecting the items stained by IHC. An examination of the characteristics and patterns of the malignant lymphoma subtype can then be conducted [8]. However, manual diagnosis has drawbacks. First, the long-term observation of microscopic histological images causes visual fatigue, which can cause a misdiagnosis. Secondly, there is a shortage of skilled doctors who have a good ability to analyze and efficiently diagnose images. Third, the differing opinions of doctors lead to inconsistent diagnoses. Therefore, there is an urgent need to achieve diagnoses using automated techniques to address the above challenges. Recently, deep learning techniques have been found superior in extracting deep features from histopathological images and classifying them with high efficiency. Deep learning requires multiple layers for processing specific tasks; it has been applied to perform specific tasks in many fields, including in medical image diagnoses [9]. This study aimed to develop systems that could predict the type of malignant lymphoma from histopathological samples stained with H&E. The generalizing power of a system depends on the training dataset, and the generalizability of a system often deteriorates when the dataset is small. We aimed to improve the ability of the system so that it can be generalized when it is trained on a small dataset by applying the data augmentation technique. The augmentation works to augment the training dataset. In this research, many different systems were developed to diagnose histopathological images of malignant lymphomas. This study aimed to extract features using deep learning and classify them using the SVM algorithm. A system has also been herein developed to diagnose malignant lymphoma datasets using the FFNN classifier according to mixed features, which were extracted using the CNN method with hand-crafted features.

The main contributions of this study are as follows:

• The enhancement of histopathological images of lymphomas and increase of the contrast of affected areas.
• The implementation of a hybrid technology that consists of using deep learning models to extract deep features, deleting the classification layers, and replacing them with the SVM algorithm for feature classification.
• A satisfactory and reliable diagnosis using an FFNN classifier according to mixed features of deep learning combined with hand-crafted features.
• A reduction of the high-dimensional features and determination of the most important features using the PCA algorithm

The rest of the work is organized as follows:

Section 2 summarizes some previous studies. Developing methodologies for analyzing and diagnosing histopathological images of lymphomas in Section 3. Summarizing the performance of the systems in Section 4. Discussing and comparing the systems’ performance for diagnosing two sets of malignant lymphoma data in Section 5. Concluding the work in Section 6.

2. Related Work

This section discusses some of the previous studies that are relevant to the diagnosis of malignant lymphoma. Each researcher aimed to achieve satisfactory results using different methodologies.

Miyoshi et al. proposed a deep learning method for the diagnosis of images of malignant lymphomas. The dataset was randomly divided with magnification factors of ×5, ×20, and ×40. The system achieved a 94% accuracy with ×5 images, 93% with ×20 images, and 92% with ×40 images [10]. Zhang et al. conducted a comparison of the performance of a BP network and optimized neural network with the GA-BP genetic method. The dataset was optimized before being fed into the systems, and the GA-BP network performed better than the BP network when classifying histopathological images of malignant lymphoma [11]. Hashimoto et al. applied a deep learning model according to multi-instance learning to diagnose malignant lymphoma subtypes. They measured the probability density ratio of IHC by determining the character of the H&E-stained slides and concluded that histopathological images in typical cases could be diagnosed with better accuracy than atypical ones [12]. Stefancu et al. discussed the ability of the surface-enhanced Raman scattering (SERS) method in diagnosing three subtypes of malignant lymphomas. The method distinguished species based on a specific signal for the DNA extracted from the lymph node biopsy sample. The method achieved an AUC of 70% when distinguishing between normal and malignant DNA, as well as an overall accuracy of 94.7% [13]. Sheng et al. presented the R-CNN model to diagnose the blood cell dataset. The R-CNN model was used to detect and differentiate malignant lymphoma from blast cells [14]. Gaidano et al. proposed four systems of machine learning to classify B-NHL lymphomas from more than nine clinical entities. All algorithms achieved an average accuracy of 92.68%, an average specificity of 98.77%, and an average sensitivity of 88.54% [15]. Steinbuss et al. presented the EfficientNet model for the classification of node and B-cell lymphomas. The network achieved an accuracy of 95.56% when testing the system with another dataset [16]. Ganguly et al. applied the pre-trained ResNet-50 model by adding layers to the model. The authors changed improvement options to diagnose the pathogenic tissue dataset for malignant lymphomas. The diagnostic accuracy of the CNN model depends on the structure and options for improvement of the CNN structure [17]. Li et al. presented multiple CNN models to diagnose a collection of textile photos of the B-cell lymphatics. B-cell lymphoma images were analyzed from three hospitals by artificial intelligence models. The technical contrast of pathological tissue slices reduced the system performance when testing new samples [18]. Syrykh et al. developed the Bayesian Neural Networks model to analyze stained slices and hematoxylin slices to achieve a good accuracy; the network has achieved good results in the diagnosis of small lymphomas [19]. Lisson et al. conducted a comparison of deep and machine learning to detect a type of malignant lymphoma, MCL. Three deep learning and two machine learning have been used. The improved CNN 3D network has had the best performance in the diagnosis of pathological textile images of MCL, achieving a resolution of 70% [20].

Our study developed new systems for hybrid feature extraction using deep learning, combining them with hand-crafted features to form feature vectors, where we then fed them into the FFNN network and achieved promising results.

3. Materials and Methods

The analysis and interpretation of the images of the two datasets of malignant lymphomas were performed to distinguish between their types. The images were enhanced, and then the images of the two datasets were inputted into two different systems. The first system was a hybrid system for extracting features using the CNN method, then classifying it using the SVM algorithm. The second system was to extract the hand-crafted features, merging them with the deep features of deep learning models and then sending them to the FFNN classifier to classify them with high efficiency, as shown in Figure 1. The PCA has been executed to reduce the high features.

3.1. Dataset

The systems evaluated histopathological images of the two datasets of malignant lymphomas.

3.1.1. First Dataset

The systems proposed in this study evaluated histopathological images of malignant lymphomas; for the first dataset that were taken as biopsies and stained with H&E. The dataset was obtained from Kaggle, which consisted of 15,000 images equally divided into three classes of malignant lymphoma: CLL, FL, and MCL. All dataset images were in the JPEG file format with a size of 512 × 512 pixels. Figure 2a shows samples of all classes of the first dataset [21].

3.1.2. Second Dataset

The systems proposed in this study evaluated histopathological images of malignant lymphomas; for the second dataset, the data used were H&E-stained biopsies. The dataset was obtained from various sites by several pathologists. The second dataset consisted of 374 histopathological images divided into three types of lymphomas, 113 histology images of CLL, 139 histology images of FL, and 122 histology images of MCL. Figure 2b displays a set of images for all classes of the second dataset [22].

3.2. Enhancing Histopathological Images

Histopathological images contain artifacts and low contrast between the affected and adjacent macular areas; It also contains dark spots and is stained with H&E and other chemical solutions. Thus, there is a resultant deterioration in the effectiveness of CNN models; CNN models require clear, well-formatted images. In this study, the average RGB color space and the scale of each image were adjusted to augment its consistency [23]. Then, the contrast of the spots of interest was increased by the contrast limited adaptive histogram equalization (CLAHE) algorithm. The CLAHE algorithm increases the brightness of dark spots through a fair distribution of illuminated pixels, which greatly improves the visibility of the edges of spots of interest and increases the local contrast of the histopathological images of malignant lymphomas. The mechanism of CLAHE is performed by taking the aim pixel and replacing it with neighboring pixels according to the derivative transformation of the CLAHE method [24]. When the value of the aim pixel is greater than the neighboring pixels’ values, the algorithm works to reduce the contrast. In comparison, when the value of the central pixel is less than the value of the neighboring pixels, the algorithm increases the contrast. Thus, histopathological images are improved, and they become ready to be fed into CNN models. Figure 3 shows samples from the two malignant lymphoma datasets after we performed the optimization. We would like to make it clear that the Figure 2 images are the same as in Figure 3 after enhancing them.

3.3. Approach to Integrating CNN with Machine Learning

Deep learning models require a huge dataset to train them to achieve high diagnostic efficiency. There are several reasons for using the hybrid technique, which are as follows: first, deep neural networks consume much time to train a dataset; second, deep neural networks require high specifications and expensive computers to classify a dataset; third, deep neural networks do not reach satisfactory results for diagnosing histopathological images of a malignant lymphoma dataset. Therefore, the hybrid approach solves these challenges [25]. The basic idea of the hybrid approach is to remove the last layers (classification layers) from the CNN method and replace them with the SVM algorithm. Thus, this approach possesses the high potential for a diagnosis within a short time. The hybrid approach consists of deep learning models for feature extraction and an SVM algorithm for receiving feature vectors from the PCA algorithm, classifying them with high accuracy. The PCA algorithm was applied after the deep learning models to reduce the dimensionality of features generated by deep learning models [26].

3.3.1. Deep Feature Extraction

Deep learning networks are used in many fields, including in health care, to diagnose medical images. Deep learning models are highly capable of extracting features without human intervention and contain many layers that are interconnected through neurons. Each layer performs a specific computational function to perform a special task that extracts deep features. Each neuron in the input layer receives data and passes them to the hidden layers. The hidden layers extract features through linear functions [27]. The essential hidden layers in deep learning networks for extracting features are the convolutional, auxiliary, and pooling layers, which will be briefly explained.

Convolutional layers: The convolutional layer is the basic building block of CNN models, and it bears the largest burden in network computations. In this layer, two matrices are multiplied: one of them is the filter f (t), which is one of the basic parameters in the convolutional layer; the second matrix is in the part from the image x (t), which is the same size as the filter as in Equation (1). The size of the filter determines how much it wraps around the image [28]. The filter slides around the height and width of the image and produces a representation of a specific part of an image. The filter slides around the image in certain steps, marked S, which are one of the basic parameters of the convolutional layer. The zero padding P is one of the convolutional layer’s basic parameters that maintains the original image’s size. Equation (2) describes the determination of the output size for each neuron [29]. We assume that the input image size is L × W × D, the filter size is F, the step amount is S, and the padding is P.

$$y\left(t\right)=\left(x\ast f\right)\left(t\right)=\int x\left(a\right)f\left(t-a\right) da$$

(1)

x (t) denotes input, f (t) means the filter, y (t) means output, and $a$ represents the time that the filter is wrapped around the image.

$$Output Neurons=\frac{W-K+2P}{S}+1$$

(2)

Pooling layer: The pooling layer is used to reduce the dimensionality of the features generated by the convolutional layers. Therefore, this layer reduces the computational cost and weight and speeds up the training process of the dataset. There are two main methods for this layer, Max and Average Pooling, each with a certain working method. First, in Max-Pooling, the maximum value is selected from a group of specified values and replaced by the maximum value, as seen in Equation (3). Second, for the Average-Pooling method, the average of a specific part of the image is computed and then replaced by the average value, as seen in Equation (4). Thus, the size of the spatial image is reduced.

$$z\left(i, j\right)=ma{x}_{m,n=1\dots .k} f\left[\left(i-1\right)p+m; \left(j-1\right)p+n\right]$$

(3)

$$z\left(i, j\right)=\frac{1}{k^2}{\displaystyle \sum }_{m,n=1\dots .k}f\left[\left(i-1\right)p+m; \left(j-1\right)p+n\right]$$

(4)

where f means the size of filter, p symbolizes the number of steps, m and n defines the location of the matrix, and k symbolizes the size of the matrix.

Auxiliary layers work on further improvements, such as the RLU layer. It passes on positive features and converts negative features to zero. To reduce the computational cost, CNN models provide a dropout layer by passing a certain amount of features onward in each iteration. In this paper, the layer was tuned by 50%; this means that the layer passed 50% of the parameters and prevented 50% on each repeat, which doubles the time needed for the training of the dataset [30].

Thus, the image features from the two malignant lymphoma datasets were extracted using DenseNet-121 and ResNet-50, and the PCA algorithm reduced the high-dimensional features.

3.3.2. SVM Algorithm

This technique removes the classification layers from the DenseNet-121 and ResNet-50 architectures and replaces them with the SVM algorithm.

SVM is a linear model used for solving linear and nonlinear regression and classification problems. The key idea of the SVM algorithm is to produce a hyperplane or line to separate the dataset into classes. The algorithm receives the dataset’s features and plots each feature value as a point in the N-dimensional space, where N is the size of the features. The algorithm generates many hyperplanes with different margins between classes, meaning that the SVM will choose the hyperplane (line) with the max-margin between the classes. The best hyperplane is determined by the locations of the support vectors that are near the hyperplane [31].

Figure 4 displays the architecture of the hybrid approach to the histopathological diagnosis of the two lymphoma datasets. DenseNet-121 and ResNet-50 models receive optimized images and extract features with high dimensions. Therefore, the PCA method was used to reduce the features and send them to the SVM algorithm for classification.

3.4. FFNN with Merging the Features of CNN and Hand-Crafted Features

This section discusses the methodology for classifying images of malignant lymphomas by the FFNN classifier based on combining features extracted from DenseNet-121 and ResNet-50 separately using hand-crafted features (extracted using DWT, GLCM, FCH, and LBP methods).

When fed to the FFNN network with deep features extracted by the CNN models, the FFNN classifier does not reach satisfactory accuracy. Even when fed to the FFNN network with hand-crafted features extracted by DWT, GLCM, FCH, and LBP methods, the FFNN classifier does not reach satisfactory accuracy. Thus, to achieve satisfactory accuracy, the features of the CNN models were combined with hand-crafted features in the same feature vectors, and then fed to the FFNN network to classify them with high accuracy.

The basic steps of the methodology were as follows:

First, we optimized the histopathological images of the two datasets, and then fed the images of the two datasets into the DenseNet-121 and ResNet-50 models, which extract deep features using convolution layers and reduce feature dimensions using pooling layers. Both the DenseNet-121 and ResNet-50 models produce 2048 deep features for each histological image and then store it in vectors of feature; thus, the size in our study became 15,000 × 2048 for the first dataset and 374 × 2048 for the second dataset.

Second, The PCA algorithm was executed to reduce the high features. Thus, the size of the two datasets after dimensional reduction was 15,000 × 512 for the first dataset and 374 × 512 for the second dataset.

Third, we applied GLCM, FCH, DWT, and LBP algorithms to extract the color, shape, and texture features.

The GLCM algorithm converts the region of interest (ROI) of the histopathological image into a gray-level matrix and then extracts texture features. The algorithm works by collecting spatial information of the area of interest. The algorithm works to check the spatial relation between the target pixel and neighboring pixels according to the directions θ and the distance d between the pixel and its neighbors. The ROI has coarse and smooth regions, where the coarse regions contain pixel values of divergent values. In contrast, the smooth region has pixels of close (almost similar) values [32]. Thus, the algorithm can find the coarse texture areas and distinguish them from the soft areas. The GLCM algorithm produced 13 statistical scales according to the spatial relationships between pixels. Each scale represents a characteristic of ROI.

The color features are one of the most important features that distinguish which class each pathological histological image belongs to. Therefore, the FCH method was used to obtain the color features. The method’s main idea is that it represents the color in the histogram bin and adopts a fuzzy representation of the colors instead of a binary representation. Thus, the method distributes each color in a certain histogram bin, then distributes all the colors contained in the ROI in varied histogram bins. Therefore, the colors that are similar will be placed into one histogram bin; when two or more colors that are similar have a different histogram bin, the two colors are classified as different [33]. Finally, the algorithm compares the membership value to find out the similarity of colors and distributes them in the histogram bin. This algorithm extracted 16 features for each image of the two malignant lymphoma datasets.

The DWT algorithm was used to analyze the histopathological images by high- and low-pass filters. The high-pass filters produce detailed parameters for the input image in the vertical, diagonal, and horizontal directions; each direction extracts three statistical features: the mean, standard deviation, and variance. Therefore, the high-pass filter produces nine statistical features. At the same time, approximate parameters are produced through a low-pass filter, which extracts three statistical features, namely the mean, standard deviation, and variance. The algorithm extracted 12 statistical features for each histological image of the two datasets of malignant lymphomas.

The LBP algorithm converted the image into a matrix form to extract the texture features by measuring the local pixels of the binary texture surfaces with their contrast. The algorithm size was set to 7 × 7 pixels; the algorithm targeted one pixel in each iteration and replaced it with a value of 48 close pixels, as shown in Equation (5) of the LBP. The LBP algorithm compares the gray value of the target pixel ($g_c$) with the gray value of 48 neighboring pixels ($g_p$). Thus, the process continues until all values of the histological image have been replaced [34].

$$LB{P}_{R,P}={\displaystyle \sum }_{p=0}^{P-1}s\left(g_p-g_c\right)2^p$$

(5)

where $g_c$ means the grey weight of the aim pixel, R denotes the radius of adjoining, $g_p$ means the grey weight of adjoining pixels, and P is the number of adjoining pixels.

Fourth, the hand-crafted features that were extracted using GLCM, DWT, FCH, and LBP methods were combined into feature vectors so that the size of the two datasets became 15,000 × 244 for the first dataset and 374 × 244 for the second dataset.

Fifth, we combined the low-dimensional features extracted by DenseNet-121 [35] and ResNet-50 [36] models with the features extracted in the fourth step. Thus, the feature size of the two datasets became 15,000 × 756 for the first dataset and 374 × 756 for the second dataset.

Sixth, the features of the two datasets were fed into the FFNN classifier to classify the images of the two datasets with high accuracy and efficiency.

Figure 5 describes the methodology of the feature fusion approach from several algorithms and their classification using the FFNN network.

4. Experimental Results

4.1. Splitting of the Datasets

This study aimed to achieve promising results through the hybrid techniques with fused features for the histological diagnosis of the two datasets of malignant lymphomas. The first dataset contained 15,000 histological images of three types of malignant lymphomas; we obtained 5000 histological images for the CLL type, 5000 histological images for the FL type, and 5000 histological images for the MCL type. In contrast, the second dataset contained 374 histological images of three types of malignant lymphomas, of which 113 were histological images of the CLL type, 139 were histological images of the FL type, and 122 were histological images of the MCL type. When implementing the proposed systems, the two datasets were randomly divided into 80% during the training and validation of the dataset (80:20%) and 20% for testing the accuracy and efficiency of the systems in making a correct diagnosis.

Table 1. Distribution of the two histological datasets of malignant lymphomas.

Datasets	The First Dataset			The Second Dataset
Phase	Training and Validation		Testing (20%)	Training and Validation		Testing (20%)
Classes	Training (80%)	Validation (20%)		Training (80%)	Validation (20%)
CLL	3800	800	1000	72	18	23
FL	3800	800	1000	89	22	28
MCL	3800	800	1000	78	20	24

shows the split of histological images of the datasets for all implementation stages.

4.2. Augmentation of Data

This section discusses our attempts to resolve the limitations of both the paucity of the second dataset for malignant lymphomas and the unbalanced dataset. The systems have been evaluated by two datasets, where the first dataset was balanced and also had a suitable number of images to train [37]. The second dataset on the other hand had 374 histological images unevenly distributed among the three types of malignant lymphomas. Therefore, the second dataset needed to process the insufficient and unbalanced dataset images with the assistance of a technique that augmented the data. In this paper, the augmentation of data technique was only applied to the second dataset. Data augmentation technology artificially increases histological images from the same dataset through image rotation, shift, flipping, etc.

Table 2. The method for augmenting the histological images and balancing the second dataset for malignant lymphomas.

Phase	Training Phase
Classes	CLL	FC	MCL
Before_aug	72	89	78
After_aug	3600	3649	3666

describes the improvement of the second dataset images through the training to solve the problem of overfitting. Note that the balancing of the second dataset by increasing the images differs from one class to the other.

4.3. Proposed Methods Evaluation Metrics

The systems were evaluated for the diagnosis of two datasets of malignant lymphomas by measuring the accuracy, specificity, precision, AUC, and sensitivity described in Equation (6)–(10). The variables noted in the equations have been obtained from the confusion matrix. Each proposed system produces a confusion matrix, which contains properly categorized samples and improperly categorized samples [38].

$$\mathrm{Accuracy}=\frac{\mathrm{TN}+\mathrm{TP}}{\mathrm{TN}+\mathrm{TP}+\mathrm{FN}+\mathrm{FP}}\times \mathrm{100}\%$$

(6)

$$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}\times 100$$

(7)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\times 100\%$$

(8)

$$\mathrm{AUC}=\frac{\mathrm{True}\mathrm{Positive}\mathrm{Rate}}{\mathrm{False}\mathrm{Positive}\mathrm{Rate}}$$

(9)

$$\mathrm{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\times 100\%$$

(10)

where TP means histological images of malignant lymphoma that were strictly classified as malignant lymphoma. TN means a normal histological image that was strictly categorized as normal. FP means a normal histological, but was misclassified as a malignant lymphoma. FN means histological images of malignant lymphoma that were incorrectly categorized as normal.

4.4. Experimental Results of Hybrid Technique

This section discusses the results of the hybrid systems of the deep learning and SVM methods. After enhancing the histopathological images, they were fed into the two models, DenseNet-121 and ResNet-50, to extract deep features. Then, the PCA algorithm was executed to reduce the high features. We removed the classification layers from the DenseNet-121 and ResNet-50 models and replaced them with the SVM algorithm to classify deep features after their dimension reduction. This technique is characterized by the high accuracy and a faster training of the datasets and can be implemented on medium-cost computers. The performance of the two-hybrid approaches was evaluated using the two datasets of images of malignant lymphomas.

Table 3. Results for the evaluation of the hybrid approach in diagnosing the two datasets of malignant lymphomas.

Datasets	First Dataset		Second Dataset
Metrics	DenseNet-121 + SVM	ResNet-50 + SVM	DenseNet-121 + SVM	ResNet-50 + SVM
Accuracy%	97.7	98.8	96	97.3
Specificity %	99.11	99.33	98	98.66
Precision %	97.66	98.66	96.33	97.33
Sensitivity %	97.71	98.7	95.97	97.12
AUC %	98.14	98.54	97.57	97.79

shows the results of the systems for image diagnosis of the two datasets of malignant lymphomas. The ResNet-50 + SVM network performed better than the DenseNet-121 + SVM network for both malignant lymphoma datasets. The results of the two-hybrid systems of the two datasets were as follows: First, their performance with the first dataset was that the DenseNet-121 + SVM network yielded an accuracy of 97.7%, specificity of 99.11%, precision of 97.66%, AUC of 98.14% and sensitivity of 97.71%. The ResNet-50 + SVM network yielded an accuracy of 98.8%, specificity of 99.33%, precision of 98.66%, sensitivity of 98.7%, and AUC of 98.54%. The performance of the two hybrid systems with the second dataset was that the DenseNet-121 + SVM network achieved an accuracy of 96%, specificity of 98%, precision of 96.33%, sensitivity of 95.97%, and AUC of 97.57. In contrast, the ResNet-50 + SVM network achieved an accuracy of 97.3%, specificity of 98.66%, precision of 97.33%, sensitivity of 97.12%, and AUC of 97.79%.

Figure 6 illustrates the performance of the two systems in diagnosing the malignant lymphoma datasets.

Figure 7 summarizes the confusion matrix for the DenseNet-121 + SVM and ResNet-50 + SVM systems in diagnosing the three types of malignant lymphoma of the first dataset. The DenseNet-121 + SVM achieved an accuracy of 97.7%, 98.4%, and 97.1% in the classification of CLL, FL, and MCL (malignant tumors), respectively. In contrast, ResNet-50 + SVM yielded an accuracy of 98.3%, 99.2%, and 98.8% in the detection of the CLL, FL, and MCL types, respectively.

Figure 8 describes the confusion matrix for DenseNet-121 + SVM and ResNet-50 + SVM systems in diagnosing the three types of malignant lymphoma of the second dataset. DenseNet-121 + SVM achieved an accuracy of 95.7%, 96.4%, and 95.8% in the classification of the CLL, FL, and MCL types (malignant tumors), respectively. In contrast, ResNet-50+SVM yielded an accuracy of 91.3%, 100%, and 100% in the detection of the CLL, FL, and MCL types, respectively.

4.5. Results of FFNN Network in Merging the Features of CNN with Hand-Crafted Features

This section discusses the performance of the FFNN classifier in diagnosing the histopathological images of two malignant lymphoma datasets according to the features extracted, separately using the DenseNet-121 and ResNet-50 models and combining them with hand-crafted features. Fusing features that were extracted using deep learning and hand-crafted features is a novel technique. The 2048 features were extracted from each of the DenseNet-121 and ResNet-50 models, and it is worth noting that the features were high dimensional. Therefore the PCA algorithm was used to reduce the high dimension features. Thus, the features extracted from both DenseNet-121 and ResNet-50 were 512 deep features. There were 244 hand-crafted features. Finally, all of the features were combined into the feature vectors, which formed 756 features for each histological image. The 15,000 × 756 feature vectors for the first dataset and 374 × 756 for the second dataset were fed into the FFNN classifier for classification. The following section discusses some tools for evaluating the FFNN of the two malignant lymphoma datasets.

4.5.1. Error Histogram

In this work, the performance of the FFNN classifier for the diagnosis of two lymphoma datasets was evaluated using the error histogram tool. This tool measures the error rate generated between the output and target values during the training, testing, and validation phases over many iterations. The classifier produces many colors that represent the stages of the split of the dataset, where the training phase is represented by blue; red is during the testing of new samples and green is used during the validation stage when adjusting the weights and parameters. Figure 9 shows the classifier’s performance by error histogram in the diagnosis of the first dataset of malignant lymphomas. When classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation at 20 bins that were represented between the values of −0.9386 and 0.9388. The FFNN classifier also reached its best evaluation for the hybrid of ResNet-50 and hand-crafted features at 20 bins, which were represented between the values of −0.9479 and 0.9479.

In contrast, listed here is the performance of the FFNN classifier when diagnosing the second dataset of malignant lymphomas. When classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation at 20 bins, which were represented between values of −0.9488 and 0.9488. The FFNN classifier also reached its best evaluation when classifying the hybrid of ResNet-50 and hand-crafted features at 20 bins, which were represented between −0.06809 and 0.06809.

4.5.2. Validation and Gradient

In this work, the performance of the FFNN classifier for the diagnosis of the two lymphoma datasets was evaluated using the validation and gradient algorithm. The tool measures the error between the expected and actual values across the training epochs of the dataset. Figure 10 shows the classifier’s performance, analyzed using the validation and gradient algorithm, for the diagnosis of the first dataset of malignant lymphomas. When classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation with a validation of 34 at epoch six and gradient value of 0.0015631. The FFNN classifier also reached its best evaluation when classifying the hybrid of ResNet-50 and hand-crafted features with a validation of 305 at epoch six and gradient value of 0.0074258.

In contrast, listed here are the results for the diagnosis of the second dataset of malignant lymphomas; when classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation with a validation of 0 at epoch 35 and a gradient value of 5.4382 × 10⁻⁷. The FFNN classifier also reached its best evaluation when classifying the hybrid of ResNet-50 and hand-crafted features at a validation of 3 at epoch 34 and a gradient value 9.5633 × 10⁻⁷.

4.5.3. Best Performance of Validation

In this study, the performance of the FFNN classifier in the diagnosis of the two sets of lymphoma data was evaluated using mean squared error, or cross-entropy. This tool measures the error rate generated between the predicted and actual output during the training, testing, and validation phases across multiple epochs. The classifier produces several colors representing the dataset’s stages, where blue represents the training phase, red represents the testing of new samples, and green represents the validation phase, when weights and parameters are being adjusted. Figure 11 shows the classifier’s performance by mean squared error for the diagnosis of the first dataset of malignant lymphomas. When classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation when it reached a minimum error of 0.0077946 at epoch 28. The FFNN classifier also reached its best evaluation when classifying the hybrid of ResNet-50 and hand-crafted features when it reached a minimum error of 0.00041545 at epoch 299.

In contrast, listed here is the performance of the FFNN classifier for the diagnosis of the second dataset of lymphoma malignancies. When classifying the hybrid of DenseNet-121 and hand-crafted features, the FFNN classifier reached its best evaluation when it reached a lowest error of 4.6673 × 10⁻⁷ at epoch 35. The FFNN classifier reached its best evaluation when classifying the hybrid of ResNet-50 and hand-crafted features when it reached a lowest error value of −05-76183e at epoch 31.

4.5.4. Confusion Matrix

The confusion matrix is a vital technique for evaluating a network’s performance. This section summarizes the results of the FFNN classifier through its production of a confusion matrix. It is the same quaternary matrix that contains all histological image samples for one of the malignant lymphoma datasets. Correctly diagnosed histological images are represented on the primary diameter, while incorrectly diagnosed histological images are above and below the primary diameter. Class 1 represents the CLL lymphoma type, Class 2 represents the FL lymphoma type, and Class 3 represents the MCL lymphoma type. Figure 12 summarizes the results of the FFNN classifier for the diagnosis of the histological images of the first dataset of malignant lymphomas. The FFNN classifier with the hybrid of DenseNet-121 and hand-crafted features yielded an accuracy of 99.3%. When fed with the hybrid of ResNet-50 and hand-crafted features, the classifier’s accuracy was 99.5%.

Figure 13 summarizes the results of the FFNN classifier in the diagnosis of the histological images of the second dataset of malignant lymphomas. The FFNN classifier with the hybrid of DenseNet-121 and hand-crafted features reached an accuracy of 98.7%. When fed with the hybrid of ResNet-50 and hand-crafted features, the classifier yielded an accuracy of 100%.

Table 4. FFNN classifier performance results with the hybrid features of the two datasets of malignant lymphomas.

Datasets	First Dataset		Second Dataset
Hybrid Features	DenseNet-121 and Hand-Crafted	ResNet-50 and Hand-Crafted	DenseNet-121 and Hand-Crafted	ResNet-50 and Hand-Crafted
Accuracy %	99.3	99.5	98.7	100
Specificity %	99.67	100	99.31	100
Precision %	99.43	99.65	98.65	100
Sensitivity %	99.1	99.33	98.74	100
AUC %	99.74	99.86	99.12	100

summarizes the results of the FFNN classifier with the fused features of several algorithms with the deep learning models. This technique achieved superior results in terms of an accurate and reliable diagnosis of the malignant lymphoma types. The FFNN filter achieved the following results: First, the performance results of the FFNN classifier with the first dataset: when extracting the hybrid of DenseNet-121 and hand-crafted features, the system yielded an accuracy of 99.3%, specificity of 99.67%, precision of 99.43%, AUC of 99.74% and sensitivity of 99.1%. When the classifier was fed the hybrid of ResNet-50 and hand-crafted features, the system reached an accuracy of 99.5%, specificity of 100%, precision of 99.65%, AUC of 99.86%, and sensitivity of 99.33%.

Second, the performance results of the FFNN classifier with the second dataset: when feeding the classifier with the hybrid of DenseNet-121 and hand-crafted features, the system reached an accuracy of 98.7%, specificity of 99.31%, precision of 98.65%, AUC of 99.12%, and sensitivity of 98.74%. When the classifier was fed the hybrid of ResNet-50 and hand-crafted features, the system yielded an accuracy of 100%, specificity of 100%, precision of 100%, AUC of 100%, and sensitivity of 100%.

Figure 14 displays the results of the FFNN classifier when fed with the mixed features extracted from DenseNet-121 and ResNet-50 models with the features of shape, texture, and color.

5. Discussion and Comparison of Performance

This study dealt with modern systems and was highly efficient in the diagnosis of histopathological images to distinguish between the types of malignant lymphomas. There were some challenges that we faced, which were: the lack of histopathological images of the dataset and the imbalance of the classes within the dataset. We overcame these challenges by applying the data augmentation method. Adding hemoxylin and eosin (H&E) to tissue biopsy presents a challenge to artificial intelligence-based techniques. This challenge was solved by using image optimization methods. There is also a similarity between the histopathological images and some types of malignant lymphoma, which represents a challenge to artificial intelligence techniques. This challenge was solved by extracting the features in several methods and fusing them into the same feature vectors.

The histological images contain some artifacts resulting from extracting biopsies, and therefore, it led to a breakdown in the performance of the proposed systems. Therefore, all histopathological images had to be optimized. The technique of data augmentation was also used in the second dataset due to the lack of histological images and the imbalance of the dataset classes. This study consisted of two proposed approaches, each with two systems, as follows: First, the development of a hybrid approach consisting of two blocks, namely the deep learning method and the SVM algorithm. Each block had a specific task where the first block (DenseNet-121 and ResNet-50) extracted deep features with high dimensions, so that the PCA algorithm could be used to reduce the high dimensions and store them into feature vectors. The second block (SVM) was fed with low-dimensional feature vectors to classify the images with high accuracy. These approaches yielded good results, where the DenseNet-121 + SVM and ResNet-50 + SVM models yielded accuracies of 97.7% and 98.8% for the first dataset and overall accuracies of 96% and 97.3% for the second dataset, respectively. The second approach was based on the hybrid features between DenseNet-121 and ResNet-50 models and was developed and fused with hand-crafted features, then stored in feature vectors to be fed into the FFNN classifier for a diagnosis. With the first dataset, the FFNN classifier reached an accuracy of 99.3% with the hybrid of DenseNet-121 and hand-crafted features and an accuracy of 99.5% with the hybrid of ResNet-50 and hand-crafted features. With the second dataset, the FFNN classifier reached an accuracy of 98.7% with the hybrid of DenseNet-121 and hand-crafted features and an accuracy of 100% with the hybrid of the ResNet-50 and hand-crafted features.

Table 5. The results of the proposed systems in the diagnosis of the types of malignant lymphomas.

Datasets			First Dataset				Second Dataset
Systems			CLL %	FL %	MCL %	Overall Accuracy %	CLL %	FL %	MCL %	Overall Accuracy %
Hybrid System		DenseNet-121 + SVM	97.7	98.4	97.1	97.7	95.7	96.4	95.8	96
		ResNet-50 + SVM	98.3	99.2	98.8	98.8	91.3	100	100	97.3
Fusion Features	FFNN Classifier	DenseNet-121, GLCM, FCH, DWT, and LBP	99.3	99.4	99.1	99.3	95.7	100	100	98.7
		ResNet-50, GLCM, FCH, DWT, and LBP	99.4	99.7	99.5	99.5	100	100	100	100

summarizes the diagnostic results of the proposed systems in the diagnosis of the histopathological images of the two datasets of malignant lymphomas. The evaluation of the systems on the first dataset was as follows: The FFNN with the hybrid of ResNet-50 and hand-crafted features yielded the best accuracy for diagnosing the CLL lymphoma type at 99.4%, FL lymphoma type at 99.7% and MCL lymphoma type at 99.5%. Thus, the hybrid features of ResNet-50 and hand-crafted performed better than the other methods in correctly representing the histological images. The evaluation of the systems on the second dataset was as follows: The FFNN with the hybrid of ResNet-50 and hand-crafted features yielded the best accuracy for diagnosing the CLL lymphoma type at 100%; for the FL lymphoma type, the ResNet-50 + SVM and FFNN classifier achieved a 100% accuracy; and for the MCL lymphoma type, the ResNet-50 + SVM and FFNN classifier yielded an accuracy of 100%.

Figure 15 shows the performance of all systems implemented, the overall diagnostic accuracy, and the accuracy in diagnosing each type of malignant lymphoma.

Table 6. A comparison of the performance of the proposed systems with the performance of the systems from previous relevant studies.

Previous Studies	Accuracy (%)	Sensitivity (%)	Precision %	Specificity (%)	AUC %
Miyoshi et al. [10]	87	91	89		-
Zhang et al. [11]	96	-	-	-	-
Hashimoto et al. [12]	68.3	-	-	-	-
Stefancu et al. [13]	94.7	-	-	-	70
Gaidano et al. [15]	94.28	84.09	95.67	-	-
Lippi et al. [39]		97	-	-	-
Proposed model	99.5	99.33	99.65	100	99.86

describes the performance results of the proposed system together with the performance results of the previous systems that are relevant. It can be noted from the table that the proposed system was evaluated using all of the metrics, while previous studies were only evaluated by some of the metrics. Previous studies achieved an accuracy ranging between 68.3% and 94.7%, while the proposed system reached 99.5%. Previous studies achieved a sensitivity ranging between 84.09% and 97%, while the proposed system reached 99.33%. Previous studies achieved a precision ranging between 89% and 95.67%, while the proposed system reached 99.65%.

6. Conclusions

Because of the diversity of malignant lymphomas and the similarity of their features, manual diagnosis has had limitations due to the similarity of features in the types of lymphomas. A great amount of experience and time are needed to analyze tissue biopsies. In this study, several proposed systems were developed that serve as a reference for doctors and experts to distinguish between the types of lymphomas. This study aimed to extract advantages out of the many algorithms and deep learning models and apply some hybrid techniques. The histopathological images were optimized to remove artifacts that degraded the system performance. The histological images of the two lymphoma datasets were fed to two proposed approaches, each of which had two systems. The first approach was to apply a hybrid system between the DenseNet-121 and ResNet-50 models and the SVM algorithm. As DenseNet-121 and ResNet-50 extract the deep features with high dimensions, the PCA algorithm was applied to reduce the high dimensions. The SVM algorithm worked to receive low-dimensional features and classify them with high efficiency. The hybrid techniques also achieved promising results in distinguishing the types of malignant tumors. The second approach was based on extracting the features from the DenseNet-121 and ResNet-50 models separately and then combining them with the hybrid features to feed them into the FFNN classifier and classify them with promising accuracy. This method yielded fantastic results in the histological diagnosis of the two datasets of malignant lymphomas. The FFNN classifier with the hybrid of ResNet-50 and hand-crafted features reached an accuracy of 99.5%, specificity of 100%, sensitivity of 99.33%, and AUC of 99.86% for the first dataset. In contrast, the same FFNN classifier when fed the hybrid of ResNet-50 and hand-crafted features yielded values of 100% for all measures in the second dataset.