Next Article in Journal
Addressing Challenges in Diagnosis, Differential Diagnosis, and Treatment of Pemphigus: A Case Series
Next Article in Special Issue
Automatization of CT Annotation: Combining AI Efficiency with Expert Precision
Previous Article in Journal
Automated Machine Learning to Develop Predictive Models of Metabolic Syndrome in Patients with Periodontal Disease
Previous Article in Special Issue
A Review of the Clinical Applications of Artificial Intelligence in Abdominal Imaging
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Machine Learning Radiomics Signature for Differentiating Lymphoma versus Benign Splenomegaly on CT

1
Department of Medical Imaging and Intervention, Chang Gung Memorial Hospital at Linkou, 5 Fuhsing St., Guishan, Taoyuan 333, Taiwan
2
Department of Medical Imaging and Radiological Sciences, Chang Gung University, Taoyuan 333, Taiwan
3
Clinical Metabolomics Core and Imaging Core Laboratory, Institute for Radiological Research, Chang Gung Memorial Hospital at Linkou and Chang Gung University, 5 Fuhsing St., Guishan, Taoyuan 333, Taiwan
4
Department of Pathology, Chang Gung Memorial Hospital at Linkou, 5 Fuhsing St., Guishan, Taoyuan 333, Taiwan
5
Clinical Trial Center, Chang Gung Memorial Hospital at Linkou and Chang Gung University, 5 Fuhsing St., Guishan, Taoyuan 333, Taiwan
*
Author to whom correspondence should be addressed.
Diagnostics 2023, 13(24), 3632; https://doi.org/10.3390/diagnostics13243632
Submission received: 10 November 2023 / Revised: 1 December 2023 / Accepted: 6 December 2023 / Published: 8 December 2023
(This article belongs to the Special Issue Artificial Intelligence in Radiology 2.0)

Abstract

:
Background: We aimed to develop and validate a preoperative CT-based radiomics signature for differentiating lymphoma versus benign splenomegaly. Methods: We retrospectively analyzed CT studies from 139 patients (age range 26–93 years, 43% female) between 2011 and 2019 with histopathological diagnosis of the spleen (19 lymphoma, 120 benign) and divided them into developing (n = 79) and testing (n = 60) datasets. The volumetric radiomic features were extracted from manual segmentation of the whole spleen on venous-phase CT imaging using PyRadiomics package. LASSO regression was applied for feature selection and development of the radiomic signature, which was interrogated with the complete blood cell count and differential count. All p values < 0.05 were considered to be significant. Results: Seven features were selected for constructing the radiomic signature after feature selection, including first-order statistics (10th percentile and Robust Mean Absolute Deviation), shape-based (Surface Area), and texture features (Correlation, MCC, Small Area Low Gray-level Emphasis and Low Gray-level Zone Emphasis). The radiomic signature achieved an excellent diagnostic accuracy of 97%, sensitivity of 89%, and specificity of 98%, distinguishing lymphoma versus benign splenomegaly in the testing dataset. The radiomic signature significantly correlated with the platelet and segmented neutrophil percentage. Conclusions: CT-based radiomics signature can be useful in distinguishing lymphoma versus benign splenomegaly and can reflect the changes in underlying blood profiles.

1. Introduction

The spleen is the largest lymphatic organ and plays an important role in hematologic and immune homeostasis. Splenomegaly can be due to various hematological malignancies or benign conditions, such as liver disease, splenic congestions, infection, and infiltrative disorder [1]. In clinical practice, the nature of splenomegaly must be understood for effective management. Imaging can help delineate the architecture of the spleen and, thus, distinguish focal and diffuse splenic involvement [2]. Diffuse infiltrative processes are the most challenging to identify because they may be caused by the accumulation of normal and abnormal cells or a mere congestive process leading to blood pooling [3]. Among the many possible causes of splenomegaly, lymphoma is the most vital to detect because the additional information gleaned from such detection can aid in establishing the proper tumor staging, treatment protocols, and prognosis [4]. Pathological confirmation of the lesion can be performed through percutaneous biopsy of the spleen. However, this approach has not been widely used because of the complications associated with current practices and the low rate of adequate sampling [5,6].
The limitations of spleen biopsy have led to the development and advancement of non-invasive methods to evaluate splenic involvement in patients with lymphoma [4,7]. Currently, 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET)–CT is the most widely used imaging for staging, treatment response assessment, and follow-up of lymphoma [8]. Even though FDG PET/CT is more accurate than other conventional imaging for initial staging and treatment response assessment in most types of lymphomas, there are still some drawbacks in FDG PET/CT. This is because the 18F-FDG PET tracer is not specific for malignant lymphoma but is also increased in inflammation, other tumors, sarcoidosis, or even in physiological biodistributions [9]. Therefore, there is a pressing need for a cost-effective alternative method that can deliver diagnostic performance comparable to PET/CT.
This study aims to develop and validate a preoperative CT-based radiomics signature in order to provide clinicians with a non-invasive means to accurately differentiate lymphoma from benign splenomegaly, potentially influencing treatment decisions and patient management strategies. The main contribution and novelty of this study were as follows: (1) Establishing a CT-based radiomics approach as a valuable tool for preoperative assessment, by analyzing preoperative CT studies from patients with histopathologically confirmed cases, utilizing the PyRadiomics package to extract key features through the manual segmentation of spleens and the utilization of LASSO regression aided in selecting seven essential radiomic features encompassing statistics, shape, and texture characteristics. (2) Investigating the correlation between the derived radiomic signature and specific hematological parameters, including platelet count and segmented neutrophil percentage, to find a potential association between radiomics and underlying physiological changes, underscoring the clinical relevance of the radiomic signature beyond its role in imaging differentiation.

2. Related Work

Radiomics, as a modern approach, facilitates quantitative assessments of medical images that extend beyond mere morphological characteristics [10]. The utilization of radiomics has expanded significantly over the years, particularly within the domain of cancer research, such as in the liver [11], prostate [12], breast [13], lung [14], and bone metastasis [15]. The use of CT-based radiomics—a quantitative approach for tissue characterization based on histogram analysis and textural features—has increased [16,17,18]. Many reports have demonstrated the potential of radiomics-based techniques for analyzing malignancies [16,17,18]. One recent study reported a feasible method for detecting diffuse splenic infiltration in lymphoma through the use of spleen-to-liver attenuation ratio in addition to splenic volume [7]. Emerging studies have led to increased interest in the application of CT radiomics to the spleen, often regarded as the “forgotten organ.” While existing studies have predominantly explored the spleen’s radiomic features in conjunction with other organs [19,20,21,22,23,24,25,26,27,28,29], only two studies, to our knowledge, have specifically investigated spleen-related radiomics [30,31]. As a result, the correlation between histopathological and radiological findings in splenomegaly has not yet been investigated. Our study aims to address this gap in the current scientific understanding.

3. Materials and Methods

3.1. Study Patients

This multi-institutional retrospective study was approved by the institutional review board, and the requirement for informed consent was waived. We analyzed patients with splenectomy from January 2011 to December 2019 from Taipei, Linkou, and Taoyuan branches of Chang Gung Memorial hospitals in Taiwan. Their clinical data, pretreatment CT images, and pathological reports were retrieved and analyzed. Figure 1 demonstrates a flow diagram of the study population.
The inclusion criteria were patients that had accessible CT images before splenectomy. Exclusion criteria were: (1) according to pathological results, focal splenic lesions such as abscess, cyst, lymphangioma, hemangioma, hamartoma, trauma, focal splenic lymphoma, other types of cancer (primary or metastases), adhesion, acute and chronic inflammation, necrosis, sclerosing angiomatoid nodular transformation, sarcoidosis, lymphoproliferative disorder, or pseudoaneurysm, Epstein–Barr virus-associated lymphoproliferative diseases, Castleman disease, or X-linked lymphoproliferative disease; (2) patients lacking accessible CT images or inappropriate CT protocols before splenectomy were also excluded; (3) patients with inaccessible data collection like DICOM file conversion error and no accessible complete blood count (CBC) profile data. Finally, we included patients with splenic lymphoma presenting as diffuse infiltration, patients with splenic congestion, and patients with no splenic involvement of underling extra-splenic cancer according to the pathological diagnosis. All included patients were divided into the lymphoma splenomegaly versus benign splenomegaly (spleen with congestions or spleen free of underlying cancer involvement) groups according to their pathological diagnosis. Splenomegaly was confirmed as a craniocaudal length exceeding 9.5 cm and a width greater than 10.6 cm on CT scan [32]. A total of 139 patients entered the final analysis and were randomly divided into the training (n = 79) and testing (n = 60) datasets.

3.2. CT Image Acquisition

The latest CT images taken within 30 days before splenectomy were obtained for analysis. CT studies were routinely performed using multislice CT systems (Somatom Sensation 4 or 16, Siemens Medical Systems, Erlangen, Germany, Aquilion 64; Toshiba Medical Systems, Otawara, Japan) with the following parameters: automatic tube current modulation 120 kVp; and image reconstruction to 5 mm thickness and at 5 mm intervals. An intravenous contrast medium of 100 mL of iohexol (350 mg of iodine per milliliter, Omnipaque 350; GE Healthcare, Princeton, NJ, USA) was administered using a power injector with an injection rate of 2–3 mL/s. CT studies were conducted with contiguous axial sections in the craniocaudal direction between the lower chest and the pelvis. Images were obtained at 60–70 s for the portal venous phase after intravenous contrast medium injection. Daily, weekly, monthly, and yearly quality assurance procedures testing all CT scanners were in compliance with Taiwanese national regulations conducted by qualified medical physicists according to the American College of Radiology (ACR) guideline [33].

3.3. Radiomics Feature Extraction

The process of splenomegaly radiomics analysis begins with segmentation followed by feature extraction, feature selection, construction of radiomic signature, model building, and validation (Figure 2). Anonymous data were exported offline, and the first reader (J.A.C., a radiology residence with 3 years’ experience) manually delineated the volume of interest (VOI) on each axial slice to include the entire spleen parenchyma using an in-house-developed graphical user interface in MATLAB (Mathworks, Natick, MA, USA). The second reader (G.L., a board-certificated radiologist with 16 years’ experience) independently verified the VOIs. Both readers were blind to the clinical outcomes. Care was taken to avoid each VOI contaminating the adjacent splenic hilum or vascular structures and contamination by areas of fluid or cyst.
The radiomic features were extracted from the whole spleen on venous-phase CT imaging by using PyRadiomics, a python package for the extraction of radiomics features from medical imaging [34]. The extracted radiomic features included the following classes: first-order statistics, shape-based 3D features, Gray-level Co-occurrence Matrix (GLCM) features, Gray-Level Size Zone Matrix (GLSZM) features, Gray-Level Run Length Matrix (GLRLM) features, Gray-Level Dependence Matrix (GLDM) features, and Neighborhood Gray-Level Tone Different Matrix (NGTDM) features. To eliminate the redundant features and avoid the possibility of overfitting, the least absolute shrinkage and selection operator (LASSO) logistic regression model was used to identify the most predictable and effective by calculating the correlation coefficient between the features based on the training dataset, using the “glmnet” package in R statistics, with 10-fold cross-validation for calibration [35]. Features with nonzero coefficients were chosen and used to build the radiomic signature, which was tested in an independent dataset. The constructed radiomic signature was interrogated with the complete blood cell count and differential count.

3.4. Splenectomy Cause

We reviewed and analyzed the electronic medical records of the included patients for the causes of splenectomy. Splenectomy was conducted in 86 patients for therapeutic use, i.e., symptomatic splenomegaly or hemolytic anemia, and in the other 53 patients to make a diagnosis for unexplained splenomegaly for suspicious tumor involvement. In the lymphoma group, all 19 patients received therapeutic splenectomy.

3.5. Histopathology

All selected patients received splenectomy with a definite pathological diagnosis as reference standard. We analyzed the pretreatment imaging and histopathological data of the included patients. Histological subtyping was performed according to the World Health Organization (WHO) classification system version 5, updated in 2022 [36]. The selected patients were divided into two groups and were categorized based on the histological criteria of lymphoma versus benign groups. Representative examples of the CT images and pathological photomicrographs are shown in Figure 3 and Figure 4. Radiomics results were not available to the assessors of the reference standard.

3.6. Complete Blood Cell Profile

Data for blood-based measurements consisting of white blood cell (WBC), red blood cell (RBC), hemoglobin (Hb), hematocrit (Hct), mean corpuscular volume (MCV), mean corpuscular hemoglobin (MCH), corpuscular hemoglobin concentration (MCHC), red cell distribution width (RDW), platelet, segmented neutrophil percentage (Seg), lymphocyte (Lymph), and monocyte (Mono) were collected. The time interval between blood data collection and the examination of CT images was within 7 days. The correlation of basic hemogram and radiomic signature was further analyzed with Pearson correlation.

3.7. Statistical Analysis

Data were analyzed using standard statistical methods on MedCalc for Windows, Version 20.008 (MedCalc Software, Ostend, Belgium). The Shapiro–Wilk test was employed to check the data normal distribution. Differences in radiomic features between the lymphoma versus benign groups were examined using the nonparametric Mann–Whitney test. Cut-off value was determined by the 50% LASSO probability in the training dataset. The sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were calculated by the number of patients assigned. Correlations between constructed radiomic signature and selected blood-based measurements were evaluated using Pearson’s correlation analysis. The level of statistical significance was set at p < 0.05.

4. Results

4.1. Patient Characteristics

We screened 516 patients who had undergone splenectomy between 2011 and 2019 from Linkou Chang Gung Memorial Hospital (Figure 1). Further, 149 patients were excluded as the final spleen pathology did not match the inclusion/exclusion criteria. The excluded pathologies consist of 88 malignant focal tumors, 78 traumas, 53 benign focal tumors, 24 infections/inflammation, and 4 granulomatous diseases. The remaining 269 patients underwent data quality checks before final analysis was performed; 81 patients were excluded because there was no accessible CT prior to surgery, 34 had inappropriate CT protocol, 8 has DICOM file conversion errors, and 7 had no accessible CBC profiles. Finally, 139 patients were recruited for final analysis. A total of 120 patients were categorized into the benign group, comprising 67 cases of splenic congestion and 53 cases with other solid organ malignancy without splenic involvement. In the lymphoma group, 19 patients were identified, representing various lymphoma subtypes (Figure 1). Within the cohort of 19 lymphoma patients, 13 manifested classical symptoms, while 6 were incidentally identified during health checkups. Among the six asymptomatic patients, two presented abnormal CBC data at the initial assessment, whereas the remaining four exhibited unremarkable CBC results. All lymphoma patients in this study received their initial diagnosis as first-line cases, with no prior local or systemic treatments administered. The mean age of the study cohort is 54 years old (standard deviation 13.5), with 57% being male. The clinical characteristics of these patients, including their age, sex, and pathological diagnoses, are presented in Table 1. Among the 19 lymphoma patients, only 4 underwent pre-splenectomy PET-CT scans, with the spleen in two cases being FDG avid (2 diffuse large B cell lymphoma) and in two cases not FDG avid (2 splenic marginal zone lymphoma).

4.2. Radiomics Feature Comparison

A total of 105 radiomic features were extracted from CT images. The Mann–Whitney U test was used to compare the radiomic features of the lymphoma and benign groups. The radiomic analysis revealed that nine first-order statistics, six GLCM features, three GLDM features, one GLRLM features, three GLSCM features, one NGTDM feature, and 15 shape-based features significantly differed between the two groups. All the significant radiomics features of differentiating lymphoma and benign groups and their values are listed in Supplementary Table S1.

4.3. Radiomics Feature Selection, Predictive Model Development, and Discriminative Ability Evaluation

The least absolute shrinkage and selection operator (LASSO) algorithm was applied for selecting potential predictors of lymphoma and benign splenomegaly. Seven remaining features with non-zero coefficients were obtained: first-order statistics (10th percentile and Robust Mean Absolute Deviation), shape-based (Surface Area), and texture features (Correlation, MCC, Small Area Low Gray-level Emphasis, and Low Gray-level Zone Emphasis). Subsequently, a radiomic signature calculated based on the abovementioned seven radiomic features was used to build a prediction model for distinguishing the benign and lymphoma groups in the training cohort (n = 79), which comprised 10 patients with lymphoma and 69 benign patients. To validate the discriminative ability of the model, and to prevent the small training samples leading to overfitting, an independent test cohort comprising 60 patients (9 with lymphoma and 51 benign patients) was used. The radiomic signature achieved an excellent diagnostic accuracy of 97% (95% confidence interval (CI) 89–100%), sensitivity of 89% (95% CI, 52–100%), and specificity of 98% (95% CI, 90–100%), in differentiating lymphoma from benign patients in the test cohort (Table 2).

4.4. Correlation between Radiomic Signature and CBC/DC Profile

The correlation between our constructed radiomic signature and blood-based measurements (e.g., WBC, RBC, Hb, Hct, MCV, MCH, MCHC, RDW, platelet, Seg, Lymph and Mono) was calculated. There was a significant association with the platelet and segmented neutrophil levels (p < 0.05) but with a weak linear correlation (when p < 0.05, r < 0.4), as shown in Table 3.

5. Discussion

Our study included 139 post-splenectomy patients with definite pathological diagnosis. Using these data, we established a predictive model based on a radiomic signature for differentiating benign and lymphoma splenomegaly; the model achieved 89% sensitivity and 98% specificity in independent testing without retraining and adjustment of the cutoff value. The strength of the present study is the basis of definite pathological diagnoses and whole-spleen volumetric radiomics analysis as well as its validation using training and test cohorts. We established a radiomic signature for our predictive model by using LASSO regression to select features from routine CT images belonging to four classes (first-order, GLCM, GLSZM, and shape-based features); these features exhibited excellent accuracy in distinguishing lymphoma from benign splenomegaly. In addition, our radiomic signature was also found to significantly correlate with the platelet and segmented neutrophil levels. To our best knowledge, no previous studies have investigated the correlation between radiomics and laboratory data. Early detection of splenic involvement in patients with lymphoma is crucial since the information influences the disease staging and management [37]. Our proof-of-concept research necessitates a subsequent extensive study to validate its clinical relevance.
According to the Lugano classification, splenic involvement in lymphoma suggests an advanced disease stage [38]. The most common tool used to assess splenic involvement is through positron emission tomography/CT with homogeneous splenomegaly (>13 cm), diffuse infiltration with miliary lesions, focal nodular lesions, or a large solitary mass [37,38]. Homogeneous splenomegaly remained the most argumentative and difficult finding to decide because of a lacking objective standard for diagnosis. No known sensitive nor specific size criterion has been widely accepted [37]. Imaging-guided biopsy or fine needle aspiration is not generally recommended because of the increased risk of bleeding [5,6]. Clinically, splenectomy is not primarily a treatment for lymphoma; instead, it serves as a step toward achieving a definitive diagnosis [39]. The emergence of systemic therapies has demonstrated improved outcomes for lymphoma patients in recent years, subsequently reducing the necessity for therapeutic splenectomy. The imperative for novel diagnostic methods necessitates their alignment with the most current developments in clinical therapeutics. Our CT-based radiomics signature holds potential utility as an opportunistic screening tool, aiding in the improved differentiation of various splenomegaly conditions. The result from our study could be an alternative quantitative imaging approach and an effective way for discrimination.
In line with our study, Reinert et al. reported that 2D splenic CT texture analysis values were significantly different between patients with untreated lymphoma and healthy controls, including mean intensity, mean average (noise independent voxel intensity), and entropy of deviation [18]. Moreover, the mean intensity and mean average of patients with untreated lymphoma and patients with liver cirrhosis differed significantly [18]. Furthermore, the mean intensity in the lymphoma group was significantly lower than that in the healthy control or congestion group. These findings could be attributed to pathological differences, leading to alterations in microcirculation [18]. A normal spleen is typically homogenous in unenhanced images, with attenuation values ranging between 40 and 60 Hounsfield units (HU). This sequence mainly serves for detecting splenic calcifications. Contrast enhancement spleen CT plays a crucial role in differentiating various splenic diseases, particularly during the middle to late portal phases of CT scans. In these phases, the normal spleen displays homogeneous enhancement, while arterial phases usually reveal various mottled enhancing patterns [40]. Hence, for our study, we opted for late venous portal phases for more in-depth analysis. Splenic involvement in lymphoma is characterized by progressive infiltration of hematopoietic and lymphoid tumors of the spleen or destruction of the red and white pulp [41]. The density of the spleen in the portal-venous phase of CT images mainly represents contrast-enhanced blood in the vascular channel system of the red pulp [18]. Therefore, tumor cell infiltration of the white or red pulp reduces spleen attenuation (mean voxel intensity and average) [7,18,42].
In addition to engaging first- and second-order statistics, other studies also further investigated shape-based features. Enke et al. reported that shape-based radiomic features of the spleen have a high predictive value for lymphoma [30]. Among such shape-based features, sphericity is a more effective predictor than the craniocaudal diameter or volume of the spleen for discriminating between malignant lymphoma and non-lymphoma [30]. Our results also demonstrate the value of shape-based features for distinguishing between benign splenomegaly and lymphoma.
We sought to interrogate the biological insights of radiomic signatures by correlating the CBC/differential count (DC) data. The spleen plays an important part in both the hematological and lymphatic systems. Previous studies conducted on different splenic disease groups on the correlation between spleen volume with hematological parameters [43,44,45] and the strongest correlation between the changes in spleen size was the platelet level [44,45], which was also noted in our study. Our results have proved the attractive trait of both radiomics and blood test biomarkers. Ortega et al. also demonstrated that the integration of radiomic features of the cross-sectional images with clinical parameters elevates the value of both outcome predictions in Hodgkin lymphoma patients [46]. Because of the high performance of our radiomic signature alone, we did not intend to integrate the CBC profile with radiomic data in diagnosing splenic lymphoma in the current study. However, the efficacy of such combinations in monitoring outcomes or treatment response in splenic lymphoma might warrant investigations in future studies.
The metadata for each patient did not combine with features extracted by feature extraction algorithms. The strength of the present study is that the CT-only radiomics signature, without using the metadata, can be useful in distinguishing lymphoma versus benign splenomegaly and can reflect the changes in underlying blood profiles.

6. Study Limitations and Future Work

We acknowledge the study’s limitations, notably its retrospective nature covering the years 2011–2019, resulting in only four PET data before splenectomy for all 19 patients; hence, no further analysis was performed. Despite the number of patients in this study being low, contributing to the retrospective nature of data collection, all samples were obtained through splenectomy with histopathologic confirmation. One might argue that the results from our retrospective collection could suffer from overfitting [47,48]. Nonetheless, the multi-institutional nature of this study comprising a wide variety of CT vendors might alleviate this potential bias. More multicenter studies and prospective studies would be welcome to increase the reproducibility and robustness of our initial radiomic findings. Secondly, the segmentation manually performed by two radiologists might hinder its wide clinical usage. Segmentation is a critical step in radiomics research, and inter-observer variability could be crucial [49]. Fully automated segmentation methods for further validation would be attractive in the future [50,51]. Another future work could be exploring the use of radiomics in monitoring treatment responses of splenic lesions and identifying possible characteristics that could predict the treatment outcomes. Future prospective studies with external validation involving large-scale multicenter datasets comprising long-term outcomes would provide a better translation of radiomics to daily clinical practice.

7. Conclusions

Our study’s findings highlighted the potential of this CT-based radiomics approach as a valuable tool for preoperative assessment, drawing insights from histologically proven cases. The comprehensive radiomics analysis identified seven essential features, encompassing statistics, shape, and texture characteristics. The resulting radiomic signature exhibited exceptional accuracy (97%), sensitivity (89%), and specificity (98%) in differentiating lymphoma from benign splenomegaly, offering clinicians a non-invasive means, potentially influencing treatment decisions and patient management strategies. Furthermore, our results also reflected the correlation between radiomics and underlying changes in blood profiles. The application of the radiomic signature, thus, emerges as a promising alternative, non-invasive tool in the diagnosis of lymphoma.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics13243632/s1, Table S1: Analysis of radiomics features with significant difference on Mann-Whitney test and individual values of features.

Author Contributions

Conceptualization, G.L.; methodology, Y.-C.L. (Yu-Chun Lin) and G.L.; software, Y.-C.L. (Yu-Chun Lin); validation, J.-A.C. and G.L.; formal analysis, H.-Y.L. and L.-Y.Y.; investigation, Y.-C.L. (Ying-Chieh Lai); resources, G.L.; data curation, Y.L., H.-J.C., Y.-H.J. and Y.-C.L. (Ying-Chieh Lai); writing—original draft preparation, J.-A.C. and G.L.; writing—review and editing, All authors.; visualization, J.-A.C., R.-C.W. and Y.-C.L. (Yu-Chun Lin); supervision, G.L.; project administration, H.-J.C.; funding acquisition, G.L. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the Ministry of Science and Technology, Taiwan (grant no.: MOST 110-2628-B-182A-018) and Chang Gung Medical Foundation grant CLRPG3K0024, CMRPG3M0732, SMRPG3K0054.

Institutional Review Board Statement

(Chang Gung Medical Foundation IRB: 202200754B0) approval was obtained on 23 May 2022.

Informed Consent Statement

Written informed consent for publication was waived by the Institutional Review Board.

Data Availability Statement

The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Acknowledgments

The authors acknowledge the assistance provided by the Cancer Center and Clinical Trial Center, Chang Gung Memorial Hospital, Linkou, Taiwan, which was founded by the Ministry of Health and Welfare of Taiwan MOHW 109-TDU-B-212-114005.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Sjoberg, B.P.; Menias, C.O.; Lubner, M.G.; Mellnick, V.M.; Pickhardt, P.J. Splenomegaly: A Combined Clinical and Radiologic Approach to the Differential Diagnosis. Gastroenterol. Clin. N. Am. 2018, 47, 643–666. [Google Scholar] [CrossRef] [PubMed]
  2. Saboo, S.S.; Krajewski, K.M.; O’Regan, K.N.; Giardino, A.; Brown, J.R.; Ramaiya, N.; Jagannathan, J.P. Spleen in haematological malignancies: Spectrum of imaging findings. Br. J. Radiol. 2012, 85, 81–92. [Google Scholar] [CrossRef] [PubMed]
  3. McKenzie, C.V.; Colonne, C.K.; Yeo, J.H.; Fraser, S.T. Splenomegaly: Pathophysiological bases and therapeutic options. Int. J. Biochem. Cell Biol. 2018, 94, 40–43. [Google Scholar] [CrossRef] [PubMed]
  4. Lee, J.E.; Cho, J.S.; Shin, K.S.; Kim, S.S.; You, S.K.; Park, J.W.; Shin, H.S.; Yoon, Y.C. Diffuse Infiltrative Splenic Lymphoma: Diagnostic Efficacy of Arterial-Phase CT. Korean J. Radiol. 2016, 17, 734–741. [Google Scholar] [CrossRef] [PubMed]
  5. McInnes, M.D.; Kielar, A.Z.; Macdonald, D.B. Percutaneous image-guided biopsy of the spleen: Systematic review and meta-analysis of the complication rate and diagnostic accuracy. Radiology 2011, 260, 699–708. [Google Scholar] [CrossRef]
  6. Singh, A.K.; Shankar, S.; Gervais, D.A.; Hahn, P.F.; Mueller, P.R. Image-guided percutaneous splenic interventions. Radiographics 2012, 32, 523–534. [Google Scholar] [CrossRef]
  7. Reinert, C.P.; Hinterleitner, C.; Fritz, J.; Nikolaou, K.; Horger, M. Diagnosis of diffuse spleen involvement in haematological malignancies using a spleen-to-liver attenuation ratio on contrast-enhanced CT images. Eur. Radiol. 2019, 29, 450–457. [Google Scholar] [CrossRef]
  8. Schaefer, N.G.; Hany, T.F.; Taverna, C.; Seifert, B.; Stumpe, K.D.; von Schulthess, G.K.; Goerres, G.W. Non-Hodgkin lymphoma and Hodgkin disease: Coregistered FDG PET and CT at staging and restaging--do we need contrast-enhanced CT? Radiology 2004, 232, 823–829. [Google Scholar] [CrossRef]
  9. Rao, L.; Wang, X.; Zong, Z.; Chen, Z.; Shi, X.; Yi, C.; Zhang, X.; Yang, Z. PET-CT for Evaluation of Spleen and Liver 18F-FDG Diffuse Uptake without Lymph Node Enlargement in Lymphoma. Medicine 2016, 95, e3750. [Google Scholar] [CrossRef] [PubMed]
  10. Jensen, L.J.; Rogasch, J.M.M.; Kim, D.; Rießelmann, J.; Furth, C.; Amthauer, H.; Hamm, B.; Steffen, I.G.; Elgeti, T.; Nagel, S.N. CT radiomics to predict Deauville score 4 positive and negative Hodgkin lymphoma manifestations. Sci. Rep. 2022, 12, 20008. [Google Scholar] [CrossRef]
  11. Starmans, M.P.A.; Miclea, R.L.; Vilgrain, V.; Ronot, M.; Purcell, Y.; Verbeek, J.; Niessen, W.J.; Ijzermans, J.N.M.; de Man, R.A.; Doukas, M.; et al. Automated Assessment of T2-Weighted MRI to Differentiate Malignant and Benign Primary Solid Liver Lesions in Noncirrhotic Livers Using Radiomics. Acad. Radiol. 2023. [Google Scholar] [CrossRef]
  12. Nieri, A.; Manco, L.; Bauckneht, M.; Urso, L.; Caracciolo, M.; Donegani, M.I.; Borgia, F.; Vega, K.; Colella, A.; Ippolito, C.; et al. [18F]FDG PET-TC radiomics and machine learning in the evaluation of prostate incidental uptake. Expert. Rev. Med. Devices 2023, 20, 1183–1191. [Google Scholar] [CrossRef]
  13. Liu, Y.; Jia, X.; Zhao, J.; Peng, Y.; Yao, X.; Hu, X.; Cui, J.; Chen, H.; Chen, X.; Wu, J.; et al. A Machine Learning-Based Unenhanced Radiomics Approach to Distinguishing between Benign and Malignant Breast Lesions Using T2-Weighted and Diffusion-Weighted MRI. J. Magn. Reson. Imaging 2023. [Google Scholar] [CrossRef] [PubMed]
  14. Selvam, M.; Chandrasekharan, A.; Sadanandan, A.; Anand, V.K.; Murali, A.; Krishnamurthi, G. Radiomics as a non-invasive adjunct to Chest CT in distinguishing benign and malignant lung nodules. Sci. Rep. 2023, 13, 19062. [Google Scholar] [CrossRef] [PubMed]
  15. Wang, H.; Chen, Y.; Qiu, J.; Xie, J.; Lu, W.; Ma, J.; Jia, M. Machine learning based on SPECT/CT to differentiate bone metastasis and benign bone lesions in lung malignancy patients. Med. Phys. 2023. [Google Scholar] [CrossRef] [PubMed]
  16. Lewis, S.; Hectors, S.; Taouli, B. Radiomics of hepatocellular carcinoma. Abdom. Radiol. 2021, 46, 111–123. [Google Scholar] [CrossRef]
  17. Bhandari, A.; Ibrahim, M.; Sharma, C.; Liong, R.; Gustafson, S.; Prior, M. CT-based radiomics for differentiating renal tumours: A systematic review. Abdom. Radiol. 2021, 46, 2052–2063. [Google Scholar] [CrossRef]
  18. Reinert, C.P.; Kloth, C.; Fritz, J.; Nikolaou, K.; Horger, M. Discriminatory CT-textural features in splenic infiltration of lymphoma versus splenomegaly in liver cirrhosis versus normal spleens in controls and evaluation of their role for longitudinal lymphoma monitoring. Eur. J. Radiol. 2018, 104, 129–135. [Google Scholar] [CrossRef] [PubMed]
  19. Wang, X.; Sun, J.; Zhang, W.; Yang, X.; Zhu, C.; Pan, B.; Zeng, Y.; Xu, J.; Chen, X.; Shen, X. Use of radiomics to extract splenic features to predict prognosis of patients with gastric cancer. Eur. J. Surg. Oncol. 2020, 46, 1932–1940. [Google Scholar] [CrossRef]
  20. Yin, Y.; Yakar, D.; Dierckx, R.A.J.O.; Mouridsen, K.B.; Kwee, T.C.; de Haas, R.J. Combining Hepatic and Splenic CT Radiomic Features Improves Radiomic Analysis Performance for Liver Fibrosis Staging. Diagnostics 2022, 12, 550. [Google Scholar] [CrossRef]
  21. Yan, Y.; Li, Y.; Fan, C.; Zhang, Y.; Zhang, S.; Wang, Z.; Huang, T.; Ding, Z.; Hu, K.; Li, L.; et al. A novel machine learning-based radiomic model for diagnosing high bleeding risk esophageal varices in cirrhotic patients. Hepatol. Int. 2022, 16, 423–432. [Google Scholar] [CrossRef] [PubMed]
  22. Lyu, D.; Liang, P.; Huang, C.; Chen, X.; Cheng, M.; Zhu, B.; Liu, M.; Yue, S.; Gao, J. Are radiomic spleen features useful for assessing the differentiation status of advanced gastric cancer? Front. Oncol. 2023, 13, 1167602. [Google Scholar] [CrossRef]
  23. Tseng, Y.; Ma, L.; Li, S.; Luo, T.; Luo, J.; Zhang, W.; Wang, J.; Chen, S. Application of CT-based radiomics in predicting portal pressure and patient outcome in portal hypertension. Eur. J. Radiol. 2020, 126, 108927. [Google Scholar] [CrossRef] [PubMed]
  24. Pan, B.; Zhang, W.; Chen, W.; Zheng, J.; Yang, X.; Sun, J.; Sun, X.; Chen, X.; Shen, X. Establishment of the Radiologic Tumor Invasion Index Based on Radiomics Splenic Features and Clinical Factors to Predict Serous Invasion of Gastric Cancer. Front. Oncol. 2021, 11, 682456. [Google Scholar] [CrossRef] [PubMed]
  25. Meng, D.; Wei, Y.; Feng, X.; Kang, B.; Wang, X.; Qi, J.; Zhao, X.; Zhu, Q. CT-Based Radiomics Score Can Accurately Predict Esophageal Variceal Rebleeding in Cirrhotic Patients. Front. Med. 2021, 8, 745931. [Google Scholar] [CrossRef] [PubMed]
  26. Luo, R.; Gao, J.; Gan, W.; Xie, W.B. Clinical-radiomics nomogram for predicting esophagogastric variceal bleeding risk noninvasively in patients with cirrhosis. World J. Gastroenterol. 2023, 29, 1076–1089. [Google Scholar] [CrossRef]
  27. Li, P.; Wu, L.; Li, Z.; Li, J.; Ye, W.; Shi, Z.; Xu, Z.; Zhu, C.; Ye, H.; Liu, Z.; et al. Spleen Radiomics Signature: A Potential Biomarker for Prediction of Early and Late Recurrences of Hepatocellular Carcinoma After Resection. Front. Oncol. 2021, 11, 716849. [Google Scholar] [CrossRef] [PubMed]
  28. Li, L.; Lin, Y.; Yu, D.; Liu, Z.; Gao, Y.; Qiao, J. A Multi-Organ Fusion and LightGBM Based Radiomics Algorithm for High-Risk Esophageal Varices Prediction in Cirrhotic Patients. IEEE Access 2021, 9, 15041–15052. [Google Scholar] [CrossRef]
  29. Batur, A.; Kılınçer, A.; Ateş, F.; Aktuğ Demir, N.; Ergün, R. Evaluation of systemic involvement of Coronavirus disease 2019 through spleen; size and texture analysis. Turk. J. Med. Sci. 2021, 51, 972–980. [Google Scholar] [CrossRef]
  30. Enke, J.S.; Moltz, J.H.; D’Anastasi, M.; Kunz, W.G.; Schmidt, C.; Maurus, S.; Mühlberg, A.; Katzmann, A.; Sühling, M.; Hahn, H.; et al. Radiomics Features of the Spleen as Surrogates for CT-Based Lymphoma Diagnosis and Subtype Differentiation. Cancers 2022, 14, 713. [Google Scholar] [CrossRef]
  31. Yang, X.; Liu, J.; Lu, X.; Kan, Y.; Wang, W.; Zhang, S.; Liu, L.; Zhang, H.; Li, J.; Yang, J. Development and Validation of a Nomogram Based on 18F-FDG PET/CT Radiomics to Predict the Overall Survival in Adult Hemophagocytic Lymphohistiocytosis. Front. Med. 2021, 8, 792677. [Google Scholar] [CrossRef]
  32. Indiran, V.; Vinod Singh, N.; Ramachandra Prasad, T.; Maduraimuthu, P. Does coronal oblique length of spleen on CT reflect splenic index? Abdom. Radiol. 2017, 42, 1444–1448. [Google Scholar] [CrossRef] [PubMed]
  33. Dillon, C.; Clements, J.; Cody, D.; Gress, D.; Kanal, K.; Kofler, J.; McNitt-Gray, M.F.; Norweck, J.; Pfeiffer, D.; Ruckdeschel, T.G.; et al. Computed Tomography—Quality Control Manual; American Colloges of Radiology: Reston, VA, USA, 2017. [Google Scholar]
  34. van Griethuysen, J.J.M.; Fedorov, A.; Parmar, C.; Hosny, A.; Aucoin, N.; Narayan, V.; Beets-Tan, R.G.H.; Fillion-Robin, J.C.; Pieper, S.; Aerts, H. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017, 77, e104–e107. [Google Scholar] [CrossRef]
  35. Friedman, J.H.; Hastie, T.; Tibshirani, R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Softw. 2010, 33, 1–22. [Google Scholar] [CrossRef]
  36. WHO. Classification of Tumours Editorial Board. Hematolymphoid Tumors [Internet; Beta Version Ahead of Print. (In Progress)]. Lyon. (France): International Agency for Research on Cancer; [Cited 2022 Aug 29], 5th ed.; WHO Classification of Tumors Series; WHO: Geneva, Switzerland, 2022.
  37. Johnson, S.A.; Kumar, A.; Matasar, M.J.; Schöder, H.; Rademaker, J. Imaging for Staging and Response Assessment in Lymphoma. Radiology 2015, 276, 323–338. [Google Scholar] [CrossRef] [PubMed]
  38. Cheson, B.D.; Fisher, R.I.; Barrington, S.F.; Cavalli, F.; Schwartz, L.H.; Zucca, E.; Lister, T.A. Recommendations for initial evaluation, staging, and response assessment of Hodgkin and non-Hodgkin lymphoma: The Lugano classification. J. Clin. Oncol. 2014, 32, 3059–3068. [Google Scholar] [CrossRef] [PubMed]
  39. Fallah, J.; Olszewski, A.J. Diagnostic and therapeutic splenectomy for splenic lymphomas: Analysis of the National Cancer Data Base. Hematology 2019, 24, 378–386. [Google Scholar] [CrossRef]
  40. Vancauwenberghe, T.; Snoeckx, A.; Vanbeckevoort, D.; Dymarkowski, S.; Vanhoenacker, F.M. Imaging of the spleen: What the clinician needs to know. Singap. Med. J. 2015, 56, 133–144. [Google Scholar] [CrossRef]
  41. Chen, D.B.; Shen, D.H.; Yang, S.M.; Fang, X.Z. Clinicopathological features of splenic tumours of lymphoid tissue. Pathol. Res. Pr. 2018, 214, 1952–1958. [Google Scholar] [CrossRef]
  42. Li, M.; Zhang, L.; Wu, N.; Huang, W.; Lv, N. Imaging findings of primary splenic lymphoma: A review of 17 cases in which diagnosis was made at splenectomy. PLoS ONE 2013, 8, e80264. [Google Scholar] [CrossRef]
  43. Leite, L.A.; Domingues, A.L.; Lopes, E.P.; Ferreira, R.D.; Pimenta Filho, A.D.; Fonseca, C.S.; Santos, B.S.; Lima, V.L. Relationship between splenomegaly and hematologic findings in patients with hepatosplenic schistosomiasis. Rev. Bras. Hematol. Hemoter. 2013, 35, 332–336. [Google Scholar] [CrossRef]
  44. Ishigami, M.; Ishizu, Y.; Onishi, Y.; Kamei, H.; Kiuchi, T.; Itoh, A.; Hirooka, Y.; Katano, Y.; Goto, H. Long-term dynamics of hematological data and spleen volume in cirrhotic patients after liver transplantation-various dynamics depending on etiology. Springerplus 2013, 2, 374. [Google Scholar] [CrossRef] [PubMed]
  45. Medeiros, T.B.; Domingues, A.L.; Luna, C.F.; Lopes, E.P. Correlation between platelet count and both liver fibrosis and spleen diameter in patients with schistosomiasis mansoni. Arq. Gastroenterol. 2014, 51, 34–38. [Google Scholar] [CrossRef]
  46. Ortega, C.; Eshet, Y.; Prica, A.; Anconina, R.; Johnson, S.; Constantini, D.; Keshavarzi, S.; Kulanthaivelu, R.; Metser, U.; Veit-Haibach, P. Combination of FDG PET/CT Radiomics and Clinical Parameters for Outcome Prediction in Patients with Hodgkin’s Lymphoma. Cancers 2023, 15, 2056. [Google Scholar] [CrossRef]
  47. Yan, J.; Chu-Shern, J.L.; Loi, H.Y.; Khor, L.K.; Sinha, A.K.; Quek, S.T.; Tham, I.W.; Townsend, D. Impact of Image Reconstruction Settings on Texture Features in 18F-FDG PET. J. Nucl. Med. 2015, 56, 1667–1673. [Google Scholar] [CrossRef] [PubMed]
  48. Bae, K.T. Intravenous contrast medium administration and scan timing at CT: Considerations and approaches. Radiology 2010, 256, 32–61. [Google Scholar] [CrossRef] [PubMed]
  49. Rizzo, S.; Botta, F.; Raimondi, S.; Origgi, D.; Fanciullo, C.; Morganti, A.G.; Bellomi, M. Radiomics: The facts and the challenges of image analysis. Eur. Radiol. Exp. 2018, 2, 36. [Google Scholar] [CrossRef]
  50. Ahn, Y.; Yoon, J.S.; Lee, S.S.; Suk, H.I.; Son, J.H.; Sung, Y.S.; Lee, Y.; Kang, B.K.; Kim, H.S. Deep Learning Algorithm for Automated Segmentation and Volume Measurement of the Liver and Spleen Using Portal Venous Phase Computed Tomography Images. Korean J. Radiol. 2020, 21, 987–997. [Google Scholar] [CrossRef]
  51. Humpire-Mamani, G.E.; Bukala, J.; Scholten, E.T.; Prokop, M.; van Ginneken, B.; Jacobs, C. Fully Automatic Volume Measurement of the Spleen at CT Using Deep Learning. Radiol. Artif. Intell. 2020, 2, e190102. [Google Scholar] [CrossRef]
Figure 1. Flowchart of study population.
Figure 1. Flowchart of study population.
Diagnostics 13 03632 g001
Figure 2. Workflow of the radiomics analysis. Region of interest (ROI) was manually segmented on all CT slices for extracting radiomics features which can be subdivided into different classes such as first-order, shape and texture features. After radiomics analysis, a prediction model was built based on the training dataset and validated by testing dataset.
Figure 2. Workflow of the radiomics analysis. Region of interest (ROI) was manually segmented on all CT slices for extracting radiomics features which can be subdivided into different classes such as first-order, shape and texture features. After radiomics analysis, a prediction model was built based on the training dataset and validated by testing dataset.
Diagnostics 13 03632 g002
Figure 3. Lymphomatous splenomegaly in a 61-year-old man. (A) Axial contrast-enhanced CT reveals diffuse homogeneous splenomegaly. (B) Hematoxylin–eosin (H-E) photomicrograph shows diffuse red pulp infiltrates of monomorphic medium-sized lymphoid cells. It is morphologically compatible with splenic diffuse red pulp small B-cell lymphoma.
Figure 3. Lymphomatous splenomegaly in a 61-year-old man. (A) Axial contrast-enhanced CT reveals diffuse homogeneous splenomegaly. (B) Hematoxylin–eosin (H-E) photomicrograph shows diffuse red pulp infiltrates of monomorphic medium-sized lymphoid cells. It is morphologically compatible with splenic diffuse red pulp small B-cell lymphoma.
Diagnostics 13 03632 g003
Figure 4. Congestive splenomegaly in a 36-year-old woman. (A) Axial contrast-enhanced CT reveals diffuse homogeneous splenomegaly and portosystemic varices at the anterior abdominal wall (arrow). (B) Hematoxylin–eosin (H-E) photomicrograph shows splenic tissue with congestion.
Figure 4. Congestive splenomegaly in a 36-year-old woman. (A) Axial contrast-enhanced CT reveals diffuse homogeneous splenomegaly and portosystemic varices at the anterior abdominal wall (arrow). (B) Hematoxylin–eosin (H-E) photomicrograph shows splenic tissue with congestion.
Diagnostics 13 03632 g004
Table 1. Demographics and characteristics of 139 study patients.
Table 1. Demographics and characteristics of 139 study patients.
Characteristics Datum
Age (y) *54.3 ± 13.5
Sex
Male79 (57)
Female60 (43)
Pathological Diagnosis
Lymphoma
Splenic marginal zone lymphoma8 (5.8)
Diffuse large B cell lymphoma5 (3.6)
Follicular lymphoma2 (1.5)
Diffuse red pulp small B cell lymphoma1 (0.7)
Hepatosplenic T cell lymphoma1 (0.7)
Classical Hodgkin lymphoma1 (0.7)
Hairy cell leukemia variant 1 (0.7)
Benign
Congestion67 (48.2)
No cancer involvement53 (38.1)
Note. Unless otherwise indicated, data are numbers of patients with percentages in parentheses. * Data are mean ± standard deviation.
Table 2. Diagnostic accuracy of the radiomic signature in the training and testing cohorts with 95% confidence interval.
Table 2. Diagnostic accuracy of the radiomic signature in the training and testing cohorts with 95% confidence interval.
(%)Accuracy SensitivitySpecificityPPV NPV
Training cohort98 (93–100)90 (56–100)100 (95–100)100 (66–100)98 (92–100)
Testing cohort97 (89–100)89 (52–100)98 (90–100)89 (52–100)90 (90–100)
PPV, positive predictive value; NPV, negative predictive value.
Table 3. Correlation of radiomic signature and blood-based biomarkers.
Table 3. Correlation of radiomic signature and blood-based biomarkers.
Radiomic Signature
p Valuer Value
WBC0.0970.141
RBC0.5880.046
Hemoglobin0.757−0.027
Hematocrit0.899−0.011
MCV0.178−0.115
MCH0.168−0.118
MCHC0.308−0.087
RDW0.6780.036
Platelets0.021 *−0.196
Seg0.016 *−0.212
Lymph0.8910.012
Mono0.1380.131
* p < 0.05. WBC, white blood cell; RBC, red blood cell; MCV, mean corpuscular volume; MCH, mean corpuscular hemoglobin; MCHC, mean corpuscular hemoglobin concentration; RDW, red cell distribution width; Seg, segmented neutrophil percentage; Lymph, lymphocyte percentage; Mono, monocyte percentage.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Cheng, J.-A.; Lin, Y.-C.; Lin, Y.; Wu, R.-C.; Lu, H.-Y.; Yang, L.-Y.; Chiang, H.-J.; Juan, Y.-H.; Lai, Y.-C.; Lin, G. Machine Learning Radiomics Signature for Differentiating Lymphoma versus Benign Splenomegaly on CT. Diagnostics 2023, 13, 3632. https://doi.org/10.3390/diagnostics13243632

AMA Style

Cheng J-A, Lin Y-C, Lin Y, Wu R-C, Lu H-Y, Yang L-Y, Chiang H-J, Juan Y-H, Lai Y-C, Lin G. Machine Learning Radiomics Signature for Differentiating Lymphoma versus Benign Splenomegaly on CT. Diagnostics. 2023; 13(24):3632. https://doi.org/10.3390/diagnostics13243632

Chicago/Turabian Style

Cheng, Jih-An, Yu-Chun Lin, Yenpo Lin, Ren-Chin Wu, Hsin-Ying Lu, Lan-Yan Yang, Hsin-Ju Chiang, Yu-Hsiang Juan, Ying-Chieh Lai, and Gigin Lin. 2023. "Machine Learning Radiomics Signature for Differentiating Lymphoma versus Benign Splenomegaly on CT" Diagnostics 13, no. 24: 3632. https://doi.org/10.3390/diagnostics13243632

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop