Computer-Assisted Fine-Needle Aspiration Cytology of Thyroid Using Two-Stage Refined Convolutional Neural Network

Abstract

Fine-needle aspiration cytology (FNAC) is regarded as one of the most important preoperative diagnostic tests for thyroid nodules. However, the traditional diagnostic process of FNAC is time-consuming, and its accuracy is highly related to the experience of the cytopathologist. Computer-aided diagnostic (CAD) systems are rapidly evolving to provide objective diagnostic recommendations. So far, most studies have used fixed-size patches and usually hand-select patches for model training. In this study, we develop a CAD system to address these challenges. In order to be consistent with the diagnostic working mode of cytopathologists, the system is mainly composed of two task modules: the detecting module that is responsible for detecting the regions of interest (ROIs) from the whole slide image of the FNAC, and the classification module that identifies ROIs having positive lesions. The system can then output the top-k ROIs with the highest positive probabilities for the cytopathologists to review. In order to obtain the overall good performance of the system, we compared different object detection and classification models, and used a combination of the YOLOV4 and EfficientNet networks in our system.

1. Introduction

With the popularization of ultrasound technology and the improvement in people’s health awareness, an increasing number of thyroid nodules have been found in health examinations, and the incidence of thyroid cancer is also increasing worldwide [1]. In 2020, about 586,000 new cases of thyroid cancer were diagnosed, and more than 43,000 people died [2]. Fine-needle aspiration biopsy (FNAB), which is a minimally invasive examination guided by ultrasound, is regarded as one of the most important preoperative evaluations in the diagnosis of thyroid cancer [3]. The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) outlines the criteria for the cytological classification of thyroid FNAB specimens [4].

Traditionally, after sample preparation using conventional smear methods or liquid-based cytology, cytopathologists examine the specimen under an optical microscope to determine the risk of malignancy according to the morphology of thyroid follicular cell groups. With the development of high-performance scanners and digital pathology, it has become common practice to perform a diagnosis on a whole-slide image (WSI) [5]. However, the manual screening process is time-consuming and laborious. Such diagnostic methods are also highly subjective and are related to cytopathologists’ experience. Thus, computer-aided diagnostic (CAD) systems have been developing rapidly to accelerate the diagnostic process and provide objective diagnostic suggestions [6].

With the advancement of artificial intelligence (AI) and the advent of digital pathology, AI solutions in thyroid cytopathology have been a research hot spot for over a decade. Hand-crafted feature extraction has worked great in many fields [7,8]. Previous CAD systems generally adopted conventional machine-learning algorithms that required complex image-preprocessing and feature-extraction steps. In 2006, Cochand-Priollet et al. [9] extrapolated morphometric nuclear data from May Grunvald–Giemsa-stained smears and used statistical classifiers to categorize them into either benign or malignant. Gopinath et al. [10] developed an automatic CAD system to classify thyroid nodules into benign or malignant by extracting statistical features on the basis of the textural characteristics of the thyroid cells, and integrated four different classifiers’ results using majority-voting and linear-combination rules. Chain et al. [6] revealed that differences in nuclear area and elongation could objectively distinguish papillary thyroid carcinoma (PTC) from benign nodules and noninvasive follicular thyroid neoplasm with papillar features (NIFTP).

In recent years, deep learning has achieved great breakthroughs in the field of computer vision and image processing [11,12]. End-to-end image recognition methods have been widely applied in various tasks in medical image analysis [13]. Deep learning brings a promising solution for automatic thyroid cancer diagnosis, and an increasing number of scholars have studied deep-learning-based diagnostic methods for thyroid cancer in the past five years [14]. More recent studies focused on CNN-based methods for thyroid cancer screening in FNAB cytological images. Sanyal et al. [15] developed a simple five-layer network to differentiate between PTCs and non-PTCs on 186 microphotographs from Romanowsky/Pap-stained smears of papillary carcinoma and 184 microphotographs from the smears of other thyroid lesions (at ×10 and ×40 magnification). A subsequent study [16] used the VGG-16 [17] and Inception-v3 models [18] to differentiate PTCs from benign thyroid nodules using 279 cytological images. An improved two-stage multiple instance learning (MIL) algorithm proposed in [19] achieved thyroid malignancy prediction from cytopathology WSIs. Duc et al. [20] utilized stain normalization and ensemble deep learning methods to improve performance for the automatic classification of malignant PTC cell clusters from FNAB images. We summarize the works mentioned above and list them in

Table 1. General summary of recent works.

Reference	Dataset	Method
Cochand-Priollet et al. [9]	157 cases of thyroid FNA	Nuclear features extraction + parametric classifiers
Gopinath et al. [10]	35 cases of thyroid FNA	Textural feature extraction + four traditional classifiers
Chain et al. [6]	35 cases of thyroid FNA	Calculating nuclear area and elongation as classification criteria
Sanyal et al. [15]	370 cases of PTC and non-PTC	Simple five-layer network
Guan et al. [16]	279 cases of PTCs and non-PTCs	VGG-16 [17] and Inception-v3 model [18]
Dov et al. [19]	908 WSIs form 659 patients	Improved two-stage multiple instance learning (MIL) algorithm
Duc et al. [20]	367 hematoxylin–eosin (H&E)-stained images	Utilizing stain normalization and ensemble deep learning methods

.

However, these studies have several limitations. First, most of these studies classified fixed-size patches rather than using WSI-level data. Second, patch-level cytological images used for model training are often manually selected rather than automatically generated. Since the discriminative features that are helpful in diagnosis occupy only a small part of the entire WSI, it is crucial in clinical practice to find the regions of interest (ROIs) among much useless background information. These issues greatly impede the application of automatic end-to-end CAD systems in clinical practice.

Thus, to address the above challenges, this paper proposes a two-stage refined convolutional neural network (CNN) to realize automatic screening for thyroid cancer in FNAB cytology. To meet actual clinical application needs, we collected 360 samples from multiple medical institutions to construct the thyroid cytopathology dataset. Inspired by the diagnostic workflow of cytopathologists, namely, locating ROIs and then carefully screening, we built our system so that it (1) detected ROIs with object detection framework YOLOV4 [21] and (2) further refined those ROIs into benign and malignant using the EfficientNet [22] model. The main contributions of this paper are summarized as follows:

• To better migrate and apply it to clinical programs, we built a thyroid cytopathology dataset generated from 360 FNAB specimens in practical medical institutions. This dataset is well-annotated for both detection and classification tasks.
• Mimicking the diagnostic experiences of pathologists, we utilized an object detection algorithm to search suspected target areas in WSI and then leverage another network to refine the classification result. To the best of our knowledge, this is the first work to apply an object detection algorithm for the automatic discovery of ROIs in thyroid cytology screening.
• Extensive experiments with promising results on the built thyroid cytopathology dataset validated the effectiveness of the proposed method for thyroid cancer diagnosis. The proposed refined two-stage network provides a novel solution for automatic CAD systems in practical thyroid cancer screening.

The remainder of this paper is organized as follows: Section 2 describes the collection process of the thyroid cytology dataset in detail. Section 3 introduces the whole two-stage diagnostic procedure of thyroid cancer, including the detection of suspicious areas and further precise identification. In Section 4, experimental results are presented and analyzed to evaluate the performance of the proposed approach. Lastly, this paper is summarized in Section 5.

2. Dataset Generation

2.1. Overview

Clinical image data are key to constructing a deep-learning-based diagnostic system. A sufficient amount of high-quality data can contribute to the success of a study. However, there are few public thyroid cytological datasets due to the difficulty of data collection and the high cost involved to some extent. Therefore, at the beginning of this work, we built a real-world clinical dataset of thyroid FNAB cytology. We collected 360 WSIs, and both WSI-level labels and the malignant thyroid regions of each WSI were carefully annotated by expert pathologists.

As our system consisted of two main parts, (1) an object detection network and (2) a classification network, two different datasets for the two tasks were generated. The process is illustrated in Figure 1. First, we collected thyroid FNAB samples from the participants. Then, using liquid-based cytologic preparation, we obtained 360 WSIs, removing the ones that did not meet the requirements. Next, we developed one dataset for detection and another for classification.

This study abides by the ethical standards of the institutional research committee and the tenets of the Helsinki Declaration. Since this work is anonymous and retrospective, the need for informed consent has been waived.

2.2. WSI Collection

We conducted a retrospective study and collected thyroid cell samples from participants aged 25–64 years from multiple healthcare institutions in 2020 and 2021.

Table 2. The distribution of WSIs.

Class	TBSRTC Diagnostic Category	WSI Image Count	Type
0	Nondiagnostic or unsatisfactory	-	-
1	Benign	222	negative
2	Atypia of undetermined significance or follicular lesion of undetermined significance	10	positive
3	Follicular neoplasm or suspicious for a follicular	2	positive
4	Suspicious for malignancy	36	positive
5	Malignant	90	positive

shows the distribution of the 360 WSIs, each of which corresponds to a participant, with liquid-based cytologic preparation. The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC), a standardized system used for the diagnosis of thyroid FNAB, comprises 6 diagnostic categories: nondiagnostic, benign, atypia of undetermined significance or follicular lesion of undetermined significance, follicular neoplasm, suspicious for malignancy, and malignant [4]. The nondiagnostic category was obviously not considered, and the other five categories were classified as positive or negative for the present study.

2.3. Image Labeling

We built two datasets from the existing WSIs, one for the object detection module and one for the classification module. Expert pathologists provided the ground-truth annotations of the malignant thyroid regions for the detection model, and Figure 2a shows an annotated image. Images in the dataset of the classification model were labeled as negative or positive. Figure 2b,c give examples of each of the two categories. We obtained a total of 29,780 images and 63,356 images for the two datasets, and divided them for training and testing, respectively. The training set was for fitting and optimizing the parameters of the model, while the independent test set was for evaluating the performance of the model.

Table 3. Number of images in the developed datasets.

Dataset for the Detection Model
Set of boxing images		positive
Training set		26,802
Testing set		2978
total		29,780
Dataset for the Classification Model
Set of labeled images	Positive	Negative	total
Training set	30,726	31,085	61,811
Testing set	840	705	1545
Total	31,566	31,790	63,356

lists the outline.

3. System Architecture

As shown in Figure 3, our CAD system was designed on the basis of two artificial intelligence models. The input of the system was a WSI that was later split evenly into patches.

Then, the detection model detects the suspected positive areas and sends these areas to the classification model, which predicts the final pathology. After lesion detection and classification, the system displays the 10 images with the highest probability of positivity to the doctor.

3.1. Suspicious Area Detection

The YOLO series [21,23,24,25,26] is one of the most classical and effective one-stage object detection frameworks. It can handle both classification and location tasks at the same time. Considering the efficiency of clinical screening, we used the YOLO V4 framework in this work, which has fast detection speed while maintaining high accuracy, to realize the real-time detection of suspicious thyroid cancer areas. YOLO V4 is a fourth-generation derivative network of the YOLO network aiming for the optimal balance between the number of convolutional layers and the number of parameters. As illustrated in Figure 4, the overall structure of YOLO V4 consists of three parts: a backbone used for extracting base features, a detection neck for better feature fusion, and a detection head for final target classification and location regression.

Backbone. To reduce the amount of computation while maintaining high-accuracy performance, YOLO V4 utilizes CSPDarkNet-53 as the backbone, which adopts the network modeling strategy in a cross-stage partial network (CSPNet) [11]. CSPNet achieves better gradient interaction and combination via cross-stage hierarchal feature partitioning and concatenation. Due to the efficient division of feature maps, CSPNet becomes more lightweight and boosts the detection speed of YOLO V4.

Neck. High-level features in deep layers strongly respond to entire objects and are thereby more discriminative for a classification task. In contrast, low-level features in shallow layers are more likely to be activated by local patterns and edges, and are helpful in location tasks. Thus, the appropriate fusion of low-level and high-level features is critical for integrating contextual information and realizing comprehensive detection. FPN [27] augments a top–down path and horizontal connections to propagate semantically strong features and effectively fuse multilevel features. Inspired by FPN, YOLO V4 has adopted PANet [28] to further leverage multilevel features by introducing top–down and bottom–up paths. Moreover, YOLO V4 utilizes spatial pyramid pooling (SPP) [29] to enhance the discriminative capability of multiscale features in which different kernel sizes of pooling operations are implemented to generate different receptive fields.

Head. YOLO V4 is an anchor-based network that utilizes a single path to complete the construction of the detection head. Specifically, via cascade 3 × 3 and 1 × 1 convolutional layers, YOLO V4 transforms input feature maps generated from the PANet into a tensor $U\in {\mathbb{R}}^{H\times W\times L}$, where $L=\mathrm{anchors}\times (k+4+1)$ represents the values for k category classification, 4 coordinate regression, and 1 object probabilistic output. There are also 3 preset anchors for each feature map.

After the construction of the network, a CIoU-loss and a NMS are applied to refine the object localization performance and filter those predicted boxes with less confidence. On the whole, YOLO V4 realizes real-time detection with high accuracy and is ideal for the detection of thyroid lesions.

Anchor cluster analysis. The anchors of the original YOLO V4 were designed for the detection of natural images, which are significantly different from cytological images.Thus, we adjusted the size of anchors to allow them to fit better in our thyroid cytological dataset. Specifically, we first counted the size of all labeled boxes, and presented the width and height distribution of these boxes as shown in Figure 5. Then we used the k-means algorithm to cluster 9 points that could be assigned to the corresponding anchors. Through the above adjustment, the anchor size is more suitable for our dataset and is helpful for performance.

3.2. Lesion Classification

The classification model was trained on the dataset mentioned above to avoid missing genuinely negative cases that were similar to the positive ones. As shown in Table 3, 63,356 images were labeled as positive or negative by expert pathologists and divided into a training set and a testing set. The EfficientNet model provides higher accuracy values than those of other models due to the use of scaling at depth, width, and resolution. EfficientNet uses mobile inverted bottleneck MBConv [30,31] as its backbone, and the squeeze-and-excitation method [32] to optimize the network structure. It scales from B0 to B7 with different parameters from 5.3 to 66 M. We used EfficientNet-B0 as our lesion classifier, as it is suitable for the proposed CAD system. Its framework is shown in Figure 6. Each MBConv is followed by a 1 or a 6, where 1 or 6 is the multiplicity factor, i.e., the first 1 × 1 convolutional layer in the MBConv expands the channels of the input feature matrix by a factor of n, where k3 × 3 or k5 × 5 indicates the size of the convolutional kernel used in Depthwise Conv in the MBConv.

4. Experiment

4.1. Experimental Setup and Evaluation Measures

All models were trained, validated, and tested using four NVIDIA GeForce GTX 3090Ti GPUs. For lesion detection, the model experiment was completed on the MMDetection platform [33] of the deep-learning object-detection toolbox based on PyTorch; we adopted the 1× training schedule of MMDetection. For a fair comparison, we adopted the same training and testing strategies as those of the detection models, and they were not pretrained.

For lesion classification, we built all the models on the basis of timm 0.5.4 [34] and PyTorch. We used the same hyperparameters for all of the models for fairness, i.e., all experiments followed most of the settings in Deit [35]. In detail, we trained all the classification models in 200 epochs with a batch size of 512 (128 images per GPU). The initial learning rate was set to be 5 × 10${}^{-4}$, and for the first 5 epochs, we used a linear warm-up with a learning rate starting from 1 × 10${}^{-6}$. For model optimization, we used the AdaW [36] optimizer with a cosine decay learning scheduler. We set the weight decay to 0.05. To enlarge the dataset, we also used most of the augmentation strategies in Deit, including Cutmix [37], Mixup [38], and RandAugment [39].

In this work, we adopted $recall$, mean average accuracy (mAP), $accuracy$, $precision$, and $F1-score$ to evaluate the performance of the models, and the scores were calculated according to the following formulas:

$$\begin{array}{cc}\hfill recall& =\frac{TP}{TP+FN},\hfill \end{array}$$

$$\begin{array}{cc}\hfill accuracy& =\frac{TP+TN}{TP+FN+FP+TN},\hfill \end{array}$$

$$\begin{array}{cc}\hfill precision& =\frac{TP}{TP+FP},\hfill \end{array}$$

$$\begin{array}{cc}\hfill F1-score& =2\frac{precision\times recall}{precision+recall}\hfill \\ & =\frac{2TP}{2TP+FP+FN},\hfill \end{array}$$

where $TP$ (true positive) denotes the number of correctly detected positive samples, $FN$ (false negative) is the number of positive samples that were incorrectly identified as negative, $TN$ (true negative) denotes the number of correctly detected negative samples, and $FP$ (false positive) is the number of negative samples that were incorrectly identified as positive.

4.2. Lesion Detection Performance

Taking the application and deployment of our CAD system into consideration, we narrowed down the selection of models to one-stage object detection models, as one-stage models are faster than two-stage models. Used as the lesion detection model to localize suspicious areas, the YOLO V4 framework has fast detection speed while maintaining high accuracy. Figure 7 gives some examples of the detection results on the testing set by YOLO V4. YOLOX [26] and Single Shot MultiBox Detector (SSD) [40] are both fast one-shot object detectors. We evaluated the chosen detector YOLO V4 with these two. $mAP$ and $recall$ were the main evaluation indicators. Utilizing CSPDarkNet-53 as the backbone, YOLO V4 adopts PANet [28] and SSP [29] to better leverage and discriminate multilevel features. Owing to these improvements, YOLO V4 demonstrated greater ability to improve detector accuracy. Figure 8, where SSD300 means the input size of SSD is 300 × 300, lists the final results. Compared with YOLOX [26] and SSD [40], YOLO V4 performed better in both mAP and recall.

4.3. Lesion Classification Performance

The lesion classification model, EfficientNet, was trained and evaluated on the dataset mentioned in Section 2.

As shown in Figure 9, the $accuracy$ and $loss$ curves were plotted with the number of training iterations on the X axis, and the corresponding training accuracy and loss on the Y axis. Moreover, we compared EfficientNet with GhostNet [41], MobileNet V3 [42], Edge-Vit [43], and Light-Vit [12].

A confusion matrix was used to see how the model performed on each class and which class was not easily distinguishable. Figure 10 shows the confusion matrices of the final results, and $accuracy$, $recall$, $precision$, and $F1-score$ values are listed in

Table 4. Classification performance comparison results (%).

Model	Accuracy	F1-Score	Precision	Recall
Ghost50	90.16	90.04	90.30	89.88
Ghost130	91.59	91.50	91.62	91.41
Mobile50	93.07	93.04	92.97	93.21
Mobile100	93.40	93.34	93.37	93.32
EdgeVit_xxs	97.74	97.71	97.86	97.60
EdgeVit_s	97.02	96.98	97.31	96.78
LightVit	94.76	94.72	94.71	94.72
EfficientNet	97.93	97.92	97.83	98.08

.

On the basis of these results, the performance of the dataset that we built of different classifiers could be visually evaluated. According to the confusion matrices, EfficientNet was more likely to avoid identifying a negative as a positive. Reaching 97.93% in $accuracy$, 97.92% in $F1-score$, 98.08% in $recall$, and 97.83% in $precision$, EfficientNet also achieved more balanced performance compared to the other classifiers due to the use of the highly effective compound scaling method, which balances network width, depth, and resolution.

4.4. Two-Stage Network Performance

Lastly, we designed experiments to demonstrate the validity of the proposed two-stage refined network. Specifically, we used the detection results of the previously mentioned trained YOLOv4 object detection network on the test set as input to the EfficientNet classification network. The experimental results show that our proposed system, with the introduction of the classification module (81.84%), achieved a 3.16% improvement in $precision$ compared to the single detection network (78.68%). $Precision$ describes how many of the predicted positive cases are true-positive cases from the perspective of the predicted outcome. This result shows our proposed network’s capability of avoiding missing genuinely negative cases that are similar to positive ones.

5. Conclusions

In this paper, we proposed a two-stage CAD system for thyroid pathology detection and classification using AI technology. We established a dataset of 360 FNAB WSIs, of which 222 were negative and 138 were positive. On the basis of the dataset, we generated two datasets for the training and evaluation of our object detector and classifier separately. When given a WSI, our system detects the suspected positive areas and send them to the classifier. After classification, the system displays the 10 images with the highest probability of positivity to the doctor. The proposed two-stage refined network provides a novel solution for automatic CAD systems in practical thyroid cancer screening. The experimental results indicate that the proposed method can assist in the diagnosis of thyroid cancer and possesses a certain clinical value. For further study, we aim to focus on the classification of multiple categories, such as TBSRTC diagnosis and the segmentation of suspicious areas at the pixel level to render our system more practical in the clinic.