1. Introduction
Immune checkpoint inhibitors have significantly changed the therapeutic landscape of advanced non-small cell lung cancer (NSCLC) patients. Compared to conventional chemotherapy drugs, the integration of immunotherapy into treatment protocols has led to improved patient outcomes [
1]. Despite their success, clinical benefit remains limited to only a subset of patients, and efforts are underway to build better immunotherapy biomarkers. In this regard, there has been an unprecedented increase in the availability of medical images and digitized pathological slides for cancer patients. Thus, these orthogonal data modalities provide a significant opportunity to aggregate and analyze them for the development of a new class of data-driven biomarkers. In the context of immunotherapy, their roles extend beyond diagnostics, monitoring of treatment responses, and early detection of immune-related adverse events [
2,
3]. Recent research efforts have been directed towards establishing radiomics and pathomics methodologies for developing more objective measurements and approaches to the clinical management of lung cancer [
4,
5,
6].
Radiomics aims to focus on the development of novel biomarkers derived from radiological images [
7,
8,
9]. The radiological features from the medical images contain information about biological processes that are extracted in a high-throughput manner. Radiomics can be quantified using well-defined mathematical equations to obtain the underlying characteristics of the disease through image intensity, shape, or texture, thereby overcoming the subjective nature of image interpretation [
10]. Many studies, including our recent works, have investigated and reported on the added clinical value of radiomics features to predict various clinical and biological outcomes in immunotherapy-treated patients, such as overall survival, progression-free survival, tumor histology, and genetic profiling, among other endpoints [
9,
11,
12,
13]. On the other hand, the rapidly expanding field of pathomics aims to investigate the micro-scale patterns from the digitized histopathology slides, or whole slide imaging (WSI), in a high-throughput manner [
4,
14]. This is carried out to define many phenotypic aspects of cancer and patient risk in a variety of organs, including the lung [
4,
15,
16]. Advances in scanning technology and the availability of large datasets of digital images have made it possible to explore the potential of computational image analysis for biomarker discovery [
17,
18,
19]. The pathomics features encompass many characteristics of the tumor, such as morphology, intensity, texture, higher-order features, and spatial information. Several open-source libraries facilitate the development of computational pathology workflows. One such platform is Histolab, which is designed to handle WSIs, automatically detect the tissue, and enable testing alternative tiling strategies, including random extraction of tiles according to tissue detection score thresholds [
20,
21,
22,
23]. In recent years, many studies have utilized WSIs to develop novel biomarkers for patient survival and response [
16,
24,
25].
The basic premise of integrating multimodal data types is that these modalities complement each other by augmenting the biological signals from individual data modalities to improve the inference of patients’ outcomes. The multimodal data integration from these two distinct imaging scales will definitely offer new research avenues to develop therapeutic biomarkers for immunotherapy-treated lung cancer patients. Furthermore, these integrative models will also help obtain insights into the biological features of tumors, such as intra-tumor heterogeneity. With the availability of medical imaging data (both radiological and WSI) in the context of immunotherapy, there are many challenges that are inherent to the multimodal integration of clinical cancer data, and some of them are more methodological [
26]. For example, what is the inherent relationship between features generated from these multimodal datasets? Can the developed multimodal-derived models improve clinical outcomes for patients? Do they provide novel insights into the clinical and biological features that are complementary in nature? [
4,
5,
27]. There are several works in the literature that show the ability of radiomics and pathomics to predict disease outcomes among NSCLC patients treated with immunotherapy; however, there is a paucity of studies on the combined role of radiomics and pathomics in inferring a patient’s response and survival. To achieve this, it is crucial to understand the underlying functional relationships between radiomics and pathomics features.
To date, few studies have explored the cross-scale association between radiomics and pathomics features in glioblastoma multiforme (GBM) and lung cancer. Recently, in a study led by Alvarez-Jimenez et al., the authors investigated the radiopathomics association between cellular density and image heterogeneity across radiology and histopathology datasets in lung cancer. This work demonstrated that the spatial patterns of tumor heterogeneity between these two approaches may be similar [
4]. In another study [
5], the authors determined the association between radiomic features extracted from preoperative ADC maps and post-contrast T1 images and pathomics features arising from H&E digitized pathology images in GBM patients. However, there has been a lack of studies investigating the relationship between radiomics and pathomics features in immunotherapy cohorts. With this premise, this work presents preliminary findings of the radiopathomics association in NSCLC patients treated with immunotherapy. Through this, we can identify pathomics features that may reflect the tissue composition basis of CT radiomics descriptors. Furthermore, we also explored the association of imaging-based features with clinical and biological endpoints—overall survival, progression-free survival, and CD8 immune cell counts.
2. Materials & Methods
2.1. Description of Cohorts
This retrospective study was performed on cohorts from two lung cancer centers: the Institut Universitaire de Cardiologie et de Pneumologie de Québec (Quebec Heart and Lung Institute, IUCPQ-Université Laval) and the Centre Hospitalier Universitaire de Montréal (CHUM). The corresponding institutional ethics committees have approved this project (MP-10-2020-3397). All patients who were diagnosed with advanced non-small-cell lung cancer and have received immune checkpoint inhibitor treatment have been included in this study. Accordingly, in these patients, we compiled survival endpoints and cell counts of the immunohistochemistry (IHC) marker, CD8 lymphocyte T phenotype. The progression of the disease was qualitatively evaluated by the radiologist. All the samples used in this study were part of the Quebec Respiratory Health Network Tissue Bank (
https://rsr-qc.ca/biobanque/ accessed on 15 November 2020) at the IUCPQ. Informed consent was obtained from all the participants. In these cohorts, at least one pre-treatment CT scan (2 months before the administration of ICIs) and one post-treatment CT scan were available to assess disease progression following the treatment. Patients were treated with one of the three ICI compounds—Atezolizumab, Nivolumab, or Pembrolizumab. H&E slides were scanned on a NanoZoomer 2.0-HT slide scanner at 20X objective. A total of 24 and 12 samples from the CHUM and IUCPQ cohorts had both radiological data and H&E tissue slides. Overall, the collective dataset from these two centers includes a total of 36 patients for the downstream analyses.
2.2. Clinical and Biological Endpoints
Progression-free survival (PFS) represents the duration from the start of treatment until either the time when disease progression is observed, the date of death due to any reason, or the last follow-up date (censored). The overall survival (OS) rate is computed from the date of the patient’s diagnosis to either the date of death from any cause or the date of censoring, whichever occurs first [
28]. Imaging techniques like CT scans, PET-CT scans, and MRIs are commonly employed to confirm disease advancement [
11,
12].
CD8 (cluster of differentiation 8) is a transmembrane glycoprotein that functions as a co-receptor for the T-cell receptor (TCR). The CD8 co-receptor serves a role in T cell signaling and facilitates interactions between cytotoxic T cells and antigens. CD8 T cells were evaluated to determine the status of immune infiltration in tumor samples. The CaloPix image analysis software (Tribun Health) was used to score IHC slides, with the results expressed as % of positive cells’ surface per square micrometer of tumor (μm2).
2.3. Radiomics Features Analysis
To extract features from the radiological scans, it is crucial to normalize the slice thicknesses of all CT images, which was done by interpolating them to a voxel size of 1 × 1 × 1
through the B-spline interpolation method available in the SimpleITK (Simple Insightful Toolkit) library. Normalizing the slice thicknesses was essential to ensure consistency and standardized representation, achieve voxel alignment, and improve the accuracy of feature calculations. These measures contribute to reliable and comparable results, enabling meaningful comparisons across different patients and imaging studies. We used the Pyradiomics version 3.0.1 computational platform in Python [
29] to extract the features from the pre-treatment CT scans. Pyradiomics is an open-source software library that allows for the extraction of radiomics features from medical imaging data in any format readable by ITK compliant with the image biomarker standardization initiative (IBSI
https://arxiv.org/abs/1612.07003 accessed on 20 October 2020) [
30]. This platform also provides several built-in features for pre-processing, such as normalization and resampling.
Radiomics features are generally categorized into three main groups based on their properties and calculation methods: (i) morphological or shape-based features—these features pertain to the shape and structure of the image, calculated in both 2D and 3D views. This category includes a subset of attributes describing the image’s shape, like volume, surface area, compactness, and sphericity; (ii) first-order or intensity-based features—these features quantify the intensity characteristics of the tumor based on the statistical properties of tumor intensity distribution within the region of interest (ROI). They include metrics like mean, median, and mode, along with indicators of distribution symmetry and diversity such as percentiles, skewness, kurtosis, and entropy; and (iii) second-order or texture-based features—these features consider the statistical dependencies and spatial relationships between adjacent voxels. Several methods are used to figure them out, including the gray-level co-occurrence matrix (GLCM), the gray-level run-length matrix (GLRLM), the gray-level size zone matrix (GLSZM), the gray-level distance zone matrix (GLDZM), and the neighborhood gray-tone difference matrix (NGTDM). For instance, GLCM quantifies the occurrence of pairs of pixel intensities at different spatial distances and orientations within the ROI, presenting them in matrix form. In addition, filter-based (e.g., wavelet, Laplacian of Gaussian, logarithm, exponential) radiomics features can be extracted from derived images using filters that can be applied to both texture- and intensity-based features.
2.4. Whole Slide Image (WSI) Pre-Processing
Pre-processing of WSIs is a crucial preliminary procedure before patch extraction, and various anomalies may occur during the process of digitizing WSIs. These anomalies observed in WSIs encompass various artifacts, such as noise and background interference. Prior to patch extraction, a thorough pre-processing of the WSIs is essential to methodically identify regions of interest. The ‘histolab’ module in Python was employed to carry out the WSI pre-processing methods. In order to clearly define the tissue region while eliminating artifacts and background interference, a series of procedures were implemented. These procedures consist of the following steps: conversion of the image to grayscale, application of Otsu thresholding, binary dilation, elimination of small holes, removal of small objects, etc. By using the aforementioned procedure, we obtained WSIs (without noise and artifacts) that exhibit a minimum of 80% tissue content. This procedure ensured that only the relevant part of the WSIs, known as the region of interest, was preserved for the downstream analyses. As the goal of this study was to investigate the relationship between radiomics features, pathomics features, and clinical endpoints, it was critical to consider only those areas of the WSI that have a significant number of nuclei.
This focus on regions with a substantial number of nuclei is driven by the need to ensure that the extracted patches are representative of the tissue’s cellular structure and pathology. Therefore, the pre-processing steps were designed to enhance the visibility and contrast of the nuclei within the tissue, facilitating their accurate identification and analysis. Moreover, this meticulous pre-processing approach aids in minimizing the potential biases that could arise from non-tissue elements or artifacts present in the WSIs. By concentrating on areas rich in nuclei, the study aims to establish a more robust and direct correlation between the extracted features and the clinical endpoints, thereby providing more reliable and clinically relevant insights. This careful selection of regions for analysis is not only crucial for the accuracy of the study but also enhances the reliability of the conclusions drawn regarding the interplay of radiomics and pathomics features in the context of clinical outcomes.
2.5. WSI Segmentation and Patching
The extracted patches from the WSI measure 512 × 512 pixels with a 20× magnification. All patches of this size containing a nucleus percentage of 15% or higher were extracted [
31]. The extraction of these patches was accomplished using the ScoreTiler class from histolab (available online:
https://histolab.readthedocs.io/en/latest/readme.html, accessed on 20 October 2020). The ScoreTiler class computes a score for each patch extracted from the WSI based on the detected nuclei count. It incorporates multiple techniques like threshold-based techniques, color space conversion [
32,
33], watershed transformation [
34,
35], and morphological operations [
36,
37] to facilitate nucleus detection. Patches were subsequently divided into bins of different scales (16 × 16 px, 32 × 32 px, and 64 × 64 px). The estimation of cell density, which measures the spatial arrangement of cells within a tissue, is conducted for individual patches at a specific scale, as previously described by Alvarez-Jimenez et al. [
4]. The cell density was determined by the segmentation of nuclei and the allocation of a certain gray level to each patch, which was determined by the estimated number of nuclei contained within that region [
4,
5]. The resulting output consists of a map that displays the density of cells at different scales, providing a visual representation of the distribution of cell density across different areas within the patch. The approach utilized for nucleus segmentation in this study was the Watershed algorithm [
35,
38]. Prior to using the algorithm, several preprocessing stages were conducted on the image patches. We removed small white noises present in the image, as well as small holes in the object (nuclei), to ensure that the region near the center of the object was considered as the foreground region, while those farther away from the object were regarded as the background. Subsequently, the area that represents the nuclei was identified by using the distance transform with an appropriate threshold. We then isolated the area that was not the nuclei by applying a dilation function, which shifts the object’s boundary towards the background [
39]. Following the isolation process, the Watershed algorithm was employed. This algorithm functions by discerning the high and low points in the grayscale image, thus effectively outlining each nucleus, even within regions where they are densely packed. Such segmentation is pivotal for accurately estimating cell density, as it facilitates the precise identification and enumeration of individual nuclei in each patch. Post-segmentation, the nuclei were categorized according to their size and shape attributes, enhancing the analysis of cell density. This categorization is crucial for differentiating among various cell types and conditions, which is essential for a deeper comprehension of the biological activities in the tissue. Employing this combination of techniques allows for an exhaustive and nuanced examination of the tissue microenvironment, yielding critical insights beneficial for both diagnostic and research objectives.
2.6. Extraction of Pathomics Features
Haralick features constitute a collection of statistical descriptors employed to characterize the texture or patterns discernible within an image. These features were derived from the gray-level co-occurrence matrix (GLCM), which is a matrix recording the frequency of occurrences of distinct pairs of pixel intensity values at predefined relative positions within an image. The utilization of Haralick features in the analysis of cell density maps enables a quantitative depiction of the spatial distribution and patterns of cell density. These features were obtained from each patch for each scale. From each matrix of cell density values, four co-occurrence matrices were computed for each scale (horizontal, vertical, minor diagonal, and main diagonal).
2.7. Radiopathomics Analysis
To investigate a possible association between radiomics and pathomics features in immunotherapy-treated NSCLC patients, an integrative study design was developed as presented in
Figure 1. The initial row delineates the steps carried out on the pre-treatment CT scans for radiomics analysis, while the subsequent row outlines the steps used for pathomics analysis. After performing radiomics and pathomics feature extraction, a comprehensive analysis was conducted to explore the relationship between radiomics and pathomics, radiomics and clinical outcomes (overall survival, progression-free survival, and CD8), and pathomics and clinical outcomes (overall survival, progression-free survival, and CD8). The association between imaging-derived features was computed using the Spearman correlation. In order to evaluate the variations in correlation distributions between radiomics and pathomics features and clinical outcomes, the Kolmogorov-Smirnov (KS) test was utilized. This statistical measure, obtained from the scipy.stats package in Python, is employed to quantify the similarity between two distributions.
2.8. Feature Reduction Analysis
Initially, data normalization was done by subtracting the mean and dividing by the standard deviation. This type of data normalization, referred to as z-score normalization, is vital as it ensures that all data sets operate on the same scale, which is crucial for correlation studies. A correlation filter was then constructed to reduce feature redundancy. This correlation filter is based on Spearman’s correlation coefficient, which is known for its ability to detect monotonic relationships and its robustness against outliers [
4]. A threshold of |r| = 0.9 was established, and all pairs of correlations between pathomics and radiomics features with an absolute correlation value greater than 0.9 were examined individually [
5,
40]. In summary, when two features exhibit the absolute value of the correlation coefficient (r) exceeding 0.9, the algorithm assesses the mean absolute correlation of each variable, and the variable with the highest mean absolute correlation is subsequently eliminated. Spearman’s correlation was also used to compute the correlation between radiomics features and clinical endpoints (OS, PFS, and CD8), as well as between pathomics features and clinical endpoints (OS, PFS, and CD8). To control for the false positives, we performed multiple hypothesis testing using the False Discovery Rate (FDR) method.
2.9. Hierarchical Clustering of Features
Unsupervised clustering using the hierarchical agglomerative clustering (HAC) methodology was employed to better understand the similarities and differences in terms of patient stratification for immune checkpoint inhibitors. HAC is a clustering technique that structures data based on their similarities by progressively building a clustering tree or dendrogram. This method was applied to the pathomics and radiomics features to group patients into clusters based on their respective data profiles. The algorithm was configured to divide the patients into two distinct clusters, resulting in the formation of two separate groups of patients. The division into two clusters was a crucial step in analyzing the clinical data, as it will facilitate the identification of significant distinctions between the groups.
4. Discussion
In the clinical management of cancer, a compendium of tests across different modalities is carried out by the patients. Through these tests, a variety of data sets are generated along the diagnostic and therapeutic trajectory of patients. A promising opportunity thus emerges for us to leverage these distinct datasets, analyze them, and build data-driven multimodal biomarkers to improve cancer care. For example, OMICS-based assay panels are being used in clinical care. And imaging modalities such as CT and pathological slides are being used significantly to build AI models for improving patient management in terms of diagnosis and interventions. This compendium of complementary datasets across larger patient cohorts opens up new research avenues to enhance clinical decision-making.
The fundamental idea of integrating disparate multimodal datasets is that they augment the biological information of a tumor and complement each other. This would improve the performance of the developed predictive models. In this context, radiological images and WSI’s describe the tumor characteristics at two different scales—micro and macro—thus inferring the underlying biological mechanisms of the phenotype. Therefore, a new class of multimodal biomarkers is required, harnessing information from various sources with refined predictive models that can improve the therapeutic response. There have been efforts to lead the development of radiomics and pathomics biomarkers to predict immunotherapy response and survival independently. In lung cancer, there are a few studies that focus on exploring the correlation between pathomics and radiomics features. In the study by Charlems et al. [
4], the authors examined the correlation between Haralick-type features and radiological features, which were later used to classify different cancer subtypes. Wan et al. successfully predicted a favorable pathological response after neoadjuvant radiotherapy for locally advanced rectal cancer patients using a model that integrates radiomics and pathomics features [
41]. Furthermore, Lu et al. provided a concise synthesis of the integration of pathomics, radiomics, and genomics, as well as an overview of research studies that have combined radiomics and pathomics, especially in the context of cancer prognosis [
6,
42]. However, there is a paucity of studies to build multimodal biomarkers integrating both radiomics and pathomics workflows in the context of immunotherapy. To achieve this, it is essential to understand the cross-sale association between radiomics and pathomics features among immunotherapy-treated NSCLC patients.
With this premise, we explored the association between radiomics and pathomics features (at different scales). The first part of our study focused on the analysis of the correlation between radiomics and pathomics features. Pathomics features were extracted from cell density maps at three levels of resolution. We focused exclusively on Haralick-type features for the analysis of histological images, following the approach employed by Alvarez et al. in their work [
4]. Radiomics features were extracted from the pre-treatment CT scans, which are categorized into morphological, intensity-based, and texture-based features. Nevertheless, it is important to note that not all retrieved features were incorporated into the radiopathomics study. A correlation-based filtering method was employed to remove any redundant radiomics and pathomics features. In order to prevent multicollinearity and reduce the loss of information, we employed a threshold of 0.9, as suggested by Peeters et al. in their research on predictive analysis using radiomics data [
40]. This filter allowed us to retain the most relevant variables for analysis. The correlation values obtained across the three different resolutions ranged from −0.64 to 0.58. Filter-based features, particularly wavelet features, proved to be the most influential in this analysis. On the pathomics side, the variables most correlated with wavelet features included sum variance, sum average, Information Measure of Correlation 1 and 2, and correlation.
Furthermore, we also investigated the association between radiomics and clinical endpoints (OS, PFS, and CD8) and between pathomics features and clinical endpoints (OS, PFS, and CD8). The results showed that the strongest correlations in absolute value between pathomics features and PFS, OS, and CD8 were 0.34, 0.33, and 0.35, respectively. For PFS and OS, positively associated correlation values were higher than negative correlations, while for CD8, the number of positive and negative correlations was balanced. Regarding radiomics features, the highest correlations in absolute value for PFS, OS, and CD8 were 0.27, 0.37, and 0.41, respectively. In addition, the KS test was utilized to assess the differences in correlation distributions between radiomics and pathomics features and clinical endpoints. In the case of the distribution of correlations with PFS and CD8, the KS statistics of 0.202 (p-value = 0.02) and 0.196 (p-value = 0.02) indicate distinct associations with these clinical endpoints, respectively. However, the analysis of the distribution of the correlations with OS showed no strong evidence of a significant difference in the distributions with KS statistics of 0.137 (p-value = 0.217). Furthermore, while both radiomics and pathomics features exhibit peaks at zero, the correlation distribution of pathomics features with PFS displays a lower maximum frequency at zero and extended tails. This suggests the existence of significant exceptions, where stronger correlations may be present compared to the radiomics features. These exceptions should be further explored with a larger sample size.
We have observed a more pronounced correlation in absolute value between wavelet features and certain Haralick-type features, such as “sum variance, sum average, correlation, and Information Measure of Correlation 1 and 2”. This can be attributed to the nature of these features, which inherently capture textural aspects of images. Wavelet features are particularly sensitive to abrupt changes in an image, such as edges or texture transitions. On the other hand, Haralick features, like “correlation and cumulative mean”, also describe texture and intensity relationships between adjacent pixels. Cell density maps can also identify histological or cytological changes at the microscopic level that are visible in radiological images. The correlations between radiomics, pathomics features, and clinical endpoints result from a complex interplay of several factors. Tumors are inherently heterogeneous, meaning different regions within a tumor exhibit distinct radiomics and pathomics characteristics and may impact the underlying relationship between radiomics and pathomics features. The hierarchical clustering allowed us to gain insight into patient stratification and its implications for immunotherapy treatment. By dividing patients into two clusters based on their pathomics and radiomics profiles, we aimed to uncover potential patterns in treatment response and disease progression. Irrespective of the imaging features, patients with better survival (PFS and OS) and CD8 counts were grouped into one cluster.
Despite promising findings in this work, we do acknowledge the limitations, which are: (i) potential cohort-specific biases in design; (ii) small sample size (n = 36); (iii) heterogeneity of CT scanners and staining variations across centers, along with the impact of gray level discretization on the radiomics feature extraction. These limitations can be partially addressed by the incorporation of harmonization approaches to combat variations induced by the scanner and staining protocols. While our work reveals interesting cross-scale relationships between radiology and pathology in immunotherapy-treated NSCLC patients, the findings warrant careful interpretation due to the small sample size. Altogether, discovering correlations between these multimodal features can enhance the use of medical imaging as an “in silico biopsy” to better manage immunotherapeutic interventions. This could potentially pave the way for personalized medicine using exclusively imaging data, which is more accessible than invasive biopsies.