Next Article in Journal
Integrin αE(CD103)β7 in Epithelial Cancer
Next Article in Special Issue
Prediction of Neoadjuvant Chemoradiotherapy Response in Rectal Cancer with Metric Learning Using Pretreatment 18F-Fluorodeoxyglucose Positron Emission Tomography
Previous Article in Journal
The Role of Breast Cancer Stem Cells in Chemoresistance and Metastasis in Triple-Negative Breast Cancer
Previous Article in Special Issue
Development and Validation of an Interpretable Artificial Intelligence Model to Predict 10-Year Prostate Cancer Mortality
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Cancers
Volume 13
Issue 24
10.3390/cancers13246210
cancers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Machine Learning for Prediction of Immunotherapy Efficacy in Non-Small Cell Lung Cancer from Simple Clinical and Biological Data

by
Sébastien Benzekry
1,*,†,
Mathieu Grangeon
2,†,
Mélanie Karlsen
1,
Maria Alexa
1,
Isabella Bicalho-Frazeto
1,
Solène Chaleat
2,
Pascale Tomasini
2,3,
Dominique Barbolosi
1,
Fabrice Barlesi
3,4 and
Laurent Greillier
1,2
1
Computational Pharmacology and Clinical Oncology (COMPO) Unit, Inria Sophia Antipolis–Méditerranée, Cancer Research Center of Marseille, Inserm UMR1068, CNRS UMR7258, Aix Marseille University UM105, 13385 Marseille, France
2
Multidisciplinary Oncology and Therapeutic Innovations Department, Assistance Publique—Hôpitaux de Marseille, Aix Marseille University, 13005 Marseille, France
3
Thoracic Oncology Department, Aix Marseille University, CNRS, INSERM, CRCM, 13385 Marseille, France
4
International Center of Thoracic Cancers, Gustave Roussy Cancer Campus, Université Paris-Saclay, 94805 Villejuif, France
*
Author to whom correspondence should be addressed.
†
These authors contributed equally.
Cancers 2021, 13(24), 6210; https://doi.org/10.3390/cancers13246210
Submission received: 8 November 2021 / Revised: 30 November 2021 / Accepted: 3 December 2021 / Published: 9 December 2021
(This article belongs to the Collection Artificial Intelligence and Machine Learning in Cancer Research)
Browse Figures
Review Reports Versions Notes

Abstract

:

Simple Summary

We studied determinants of response to immune-checkpoint inhibition in advanced non-small cell lung cancer patients. Specifically, we evaluated the association with response of multiple simple pre-treatment blood markers available from routine examination. We first used classical statistical tools and then developed a machine learning algorithm for individual predictions. We obtained a 69% accuracy. Hemoglobin levels and performance status were the strongest predictors. Neutrophil-to-lymphocyte ratio was also associated with outcome. A benchmark of 8 machine learning models also evidenced that the best model performed almost equally well than a logistic regression (basic statistical learning model).

Abstract

Background: Immune checkpoint inhibitors (ICIs) are now a therapeutic standard in advanced non-small cell lung cancer (NSCLC), but strong predictive markers for ICIs efficacy are still lacking. We evaluated machine learning models built on simple clinical and biological data to individually predict response to ICIs. Methods: Patients with metastatic NSCLC who received ICI in second line or later were included. We collected clinical and hematological data and studied the association of this data with disease control rate (DCR), progression free survival (PFS) and overall survival (OS). Multiple machine learning (ML) algorithms were assessed for their ability to predict response. Results: Overall, 298 patients were enrolled. The overall response rate and DCR were 15.3% and 53%, respectively. Median PFS and OS were 3.3 and 11.4 months, respectively. In multivariable analysis, DCR was significantly associated with performance status (PS) and hemoglobin level (OR 0.58, p < 0.0001; OR 1.8, p < 0.001). These variables were also associated with PFS and OS and ranked top in random forest-based feature importance. Neutrophil-to-lymphocyte ratio was also associated with DCR, PFS and OS. The best ML algorithm was a random forest. It could predict DCR with satisfactory efficacy based on these three variables. Ten-fold cross-validated performances were: accuracy 0.68 ± 0.04, sensitivity 0.58 ± 0.08; specificity 0.78 ± 0.06; positive predictive value 0.70 ± 0.08; negative predictive value 0.68 ± 0.06; AUC 0.74 ± 0.03. Conclusion: Combination of simple clinical and biological data could accurately predict disease control rate at the individual level.

1. Introduction

Immune checkpoint inhibitors (ICIs) are now a therapeutic standard in several advanced cancers, particularly in stage IV non-small cell lung cancer (NSCLC) without genetic alteration [1,2]. The development of ICIs is leading to treat an increasing number of patients with these expensive drugs. Even if the overall response rate is higher with ICIs than chemotherapy, it is equal to about 20% for ICIs in monotherapy [2]. Consequently, there are still 4 patients out of 5 with no response to single agent ICI. Thus, identification of predictive markers for ICIs efficacy is an important unmet medical need.
Biologically, ICIs mechanism of action rely on the immune system and tumor micro-environment. Tumor-infiltrating lymphocytes (TILs) are known to have different effects on survival [3]. Blood counts may be a surrogate marker of these TILs and reflect inflammation and adaptive immune response in lung cancer [4]. In this respect, analysis of blood counts before the start of ICIs showed some interesting correlation with response. In a meta-analysis [5], particularly in melanomas treated with Ipilimumab, a higher lymphocyte count and relative lymphocyte count predicted better overall survival (OS), as for a higher eosinophil count and lower neutrophil count. Neutrophil to lymphocyte ratio (NLR) at baseline has also been investigated, and its decrease was associated with better OS, progression-free survival (PFS) and response [6]. The derived NLR (dNLR = absolute neutrophils count/[white blood count—absolute neutrophils count]) is another ratio which has already been an alternative to NLR in melanomas [7,8] and metastatic colorectal cancer [9]. Furthermore, for lung cancer, in a Chinese meta-analysis published in 2016 [10], high levels of platelet to lymphocytes ratio (PLR) at baseline were associated with poor OS and PFS, but in all types of treatment. Specifically for NSCLC treated by ICIs, a study showed that a score combining dNLR greater than 3 with elevated LDH was correlated with worse outcome for ICIs [11]. Furthermore, an Italian study in NSCLC patients treated by ICIs [12] showed that low NLR and low PLR at baseline were associated with development of immune related adverse events (IRAEs), and low NLR was associated with better outcomes (OS, PFS). However, a comprehensive analysis of all classical blood markers for prediction of efficacy in a large number of patients is still lacking [13].
In the era of precision medicine, machine learning (ML) has recently developed as an alternative to classical statistical analysis [14]. The main difference is that statistical analysis focuses on inference and association between variable(s) and outcome(s), while ML puts emphasis on predictive performances only [15]. In oncology, ML has demonstrated great success for prediction from large-dimensional ‘big’ data, such as genomics or imaging [14]. Nevertheless, such data science methods also have relevance to establish predictive models from smaller sets of variables [16]. In addition, successes have mostly been limited to diagnosis and prognosis but seldom for predictive applications in a clinical oncology context (i.e., for therapeutic decision).
We hypothesized that ML could be useful to accurately predict the efficacy of ICIs in NSCLC patients. The present study aimed to develop a ML model for the selection of patients which could benefit from treatment with ICIs, from simple clinical and biological data.

2. Materials and Methods

2.1. Patients

In this observational monocentric retrospective study, we analyzed data from all patients older than 18 years of age who were diagnosed with advanced NSCLC and who received at least 1 cycle of ICI (anti-PD-L1, anti PD-1 or anti-CTLA-4), alone or in combination (anti-PD(L)1 and anti-CTLA4) following a first cycle of non-ICI systemic therapy from 2 April 2013 to 14 February 2017. Patients were treated according to available guidelines.
The study protocol and retrospective data collection were approved by the Institutional Review Board of the French Society of Respiratory Diseases (Société de Pneumologie de Langue Française—SPLF), under reference number: CEPRO 2019-007 and patients have signed an informed consent.

2.2. Data

Data were retrieved from electronic patient records. Clinical and epidemiological data (age, gender, tobacco status, asbestos exposure, performance status, body mass index), disease characteristics (histology, mutation status, TNM stage), treatment data (type, treatment line, toxicity), biological data (last blood count before first infusion of ICI) and outcome data (response and survival) were collected. From the pre-treatment blood counts, we calculated the PLR, NLR and derived NLR (dNLR) = absolute neutrophils count/(white blood count − absolute neutrophils count). Performance status was dichotomized into <2 or ≥2.
Tumor response was assessed once every 2 months through computed tomography scans, according to the Response Evaluation Criteria in Solid Tumors, version 1.1 [17]. Definition of response here is the best response observed. Overall response rate (ORR) included patients with complete or partial responses. Disease control (DCR) included patients with complete, partial responses or stable disease. Overall survival was defined as the time from start of immunotherapy to death from any cause, censored at the date of last follow-up. Progression-free survival was defined as the time from start of immunotherapy to documented disease progression or death from any cause, censored at the date of last follow-up period.

2.3. Statistical Analysis

In exploratory data analysis, two-tailed Student’s t-tests were used for continuous variables and chi-squared tests for categorical variables. Association of clinical and biological data response was assessed using univariable and multivariable logistic regression, as implemented in the glm function of the R software (version 3.6) [18]. Survival analyses of OS and PFS were performed using univariable and multivariable proportional hazard Cox’s regression models [19,20]. Continuous variables were centered and scaled before these analyses.

2.4. Machine Learning

Feature selection was performed using feature importance given by a random forest classification algorithm applied on the entire data set (randomForest R package, no tuning, 1000 trees). Once sorted by mean decrease accuracy, incremental logistic regression models were built with increasing number of variables. The selected optimal set of features was the maximal one before observing a decrease in 10-fold cross-validated accuracy. Predictive machine learning (ML) models were further built and assessed using repeated nested k-fold cross-validation with 5 repeats of 3 outer loops (to assess generalizability) containing each 5 repeats of 3 inner loops (to tune the models). Thus, 15 train and test sets were built to test the predictive performances. The models were implemented under the tidymodels framework in R version 4.0.4 [21]. They included: logistic regression (glm), random forest (ranger, 1000 trees), single layer neural network (2600 maximum number of weights), naïve Bayes, k-nearest neighbors (kknn, rectangular kernel function) and support vector machines (with linear, polynomial or radial basis kernel function). For each outer fold, models were tuned and trained on the train set and predictions were assessed on the test set. The decision tree was built using the rpart engine and tuned for hyper-parameters of tree depth, minimum number of data points required for further splitting (min_n) and complexity parameter (cost_complexity), using a grid search and a 10-fold cross-validation. The tree was then trained on the entire set.

3. Results

3.1. Patients and Disease Characteristics

Overall, 298 patients treated with ICIs for stage IV or relapsed NSCLC were retrieved from our database and analyzed. Patient and disease characteristics are summarized in Table 1. Regarding the treatments with ICIs, 89% (n = 266) of patients received an anti-PD-1 antibody, with 96.7% (n = 286) patients being pretreated with chemotherapy prior to ICIs. The report of blood counts values at start of treatment with ICIs are reported in Table S1.

3.2. Statistical Analysis

3.2.1. Response

The DCR was 53.4% and ORR was 15.4%. Exploratory data analysis was conducted for association of the considered variables with outcome (Figure 1). Significant associations were found for NLR (p < 0.001), derived NLR (p < 0.001), hemoglobin (p < 0.0001), leukocytes (p < 0.01) and neutrophils (p < 0.001). These results were confirmed by logistic regression analysis, with additional significance of (Table 2). However, in multivariable analysis, only hemoglobin and PS remained significant. They also remained significant in multivariable analysis controlling for possible additional confounding factors: sex, immunotherapy type and smoking status.

3.2.2. Survival Analysis

Progression-free and overall survival are reported in Figure S1. The median PFS was 3.27 months (95% CI: 2.63–4.07) and median OS was 11.4 months (95% CI: 8.8–15.5). Proportional hazard Cox regression confirmed association of hemoglobin and performance status with response. They were significantly correlated with PFS and OS, in univariable and multivariable analysis (Tables S2 and S3). They also remained significant in multivariable analysis controlling for possible additional confounding factors: sex, immunotherapy type and smoking status.

3.3. Machine Learning

The ML analysis was conducted for prediction of DCR (classification task) and comprised two steps. First, feature selection and then ML classification. The first step was conducted using random forest-based mean decrease in accuracy (Figure 2A) followed by selection of an optimal number of predictors (Figure 2B). The former revealed hemoglobin level as the strongest predictor of DCR. The second strongest predictor was performance status, followed by NLR. Adding further predictors resulted in a decrease in the cross-validated accuracy of logistic regression models (Figure 2B). Thus, these three variables were selected for further inclusion in ML models.
Multiple machine learning models using this set of variables were then assessed for their predictive abilities of DCR (Figure 3 and Table 3). First, learning curves—which evaluate the predictive abilities of the models with increasing number of patients—demonstrated that convergence to the optimal predictive power had been reached, for each model (Figure S2). Receiver-operator curves were similarly discriminant across the algorithms (Figure 3A), apart from k-nearest neighbors (knn) and the polynomial support vector machine (SVM) with poorer performances. Aside knn, mean areas under the curve ranged 0.72–0.74 (Table 3). Similarly, precision (sensitivity)–recall (positive predictive value) curves were comparable (Figure 3B). Best accuracy was 68%, achieved by the random forest model. Sensitivity was generally low (max 0.58, random forest) while specificity was high (0.73–0.94, Table 3). Best positive (naive Bayes, 0.72) and negative (random forest, 0.68) predictive values suggested good predictive power (Table 3). Altogether, the ML algorithms performances suggested the random forest algorithm as the most adequate, achieving highest score in the largest number of them (accuracy, area under the ROC curve, sensitivity and negative predictive value, Table 3) and exhibiting the smallest inter-score variability (Figure 3C).
A random forest algorithm being hard to interpret, we also trained a decision tree algorithm (Figure 3D). This confirmed performance status (<2 versus ≥2) and hemoglobin level (optimal threshold 13 g/dL) as the most important predictive variable. Then NLR, consistently with our random forest-based importance analysis. This tree could be useful for clinical decision at bedside. For instance, patients with performance status <2 and hemoglobin ≥13 g/dL are predicted to have an 83% chance of disease control.

4. Discussion

The selection of patients who will benefit from ICIs therapies is crucial in the era of precision medicine, in order to develop new strategies for those patients who are not likely to respond from current strategies. Clinical examination and blood counts are easily acquired, but their predictive power in combination remains to be determined. While classical statistical methods are appropriate and have been employed for determination of association with outcome [11,22], ML techniques are more adapted for prediction tasks. Therefore, we decided to analyze our data with the help of such methods.
Although well developed in several areas of science and industry, especially for dealing with ‘big data’, the use of ML for clinical oncology has remained limited to date [14,23]. In particular, very few studies have investigated ML for prediction of response to immune-checkpoint blockade, and none has focused on the predictive value of blood counts [16,24,25,26]. In addition, the main limitation of such studies is the small sample size, despite being a critical determinant to ensure the robustness and generalizability of the results [27]. For instance, Wiesweg et al. had 55 patients in the training set and 36 in the test set [25].
In comparison, we analyzed the data from 298 patients, allowing to have higher statistical power. Training and test sets were thus composed of about 200 and 100 patients in training and tests, respectively. This large number of patients ensured that the models had enough information to learn and predict (Figure S2). Our results even show that 200 patients are sufficient to reach the optimal accuracy, for models with six variables. Importantly, our data was collected from clinical practice (i.e., real-world data), which implies larger heterogeneity but is also more reflective of the reality at bedside [22]. The random forest algorithm emerged as the algorithm with best trade-off over all the metrics considered, resulting in a 68% mean accuracy and 0.74 mean area under the ROC curve, on test sets. Nevertheless, logistic regression, single layer neural network and naïve Bayes models performed almost equally well, suggesting that similar predictive power can be achieved with a standard, linear statistical learning algorithm such as logistic regression. This is in line with a review that suggested no benefit of ML over logistic regression for clinical prediction models [28].
Associations between blood count and efficacy were consistent with previous studies, in particular correlations with hemoglobin, NLR, dNLR and PLR, were the same than in previous studies [5,6,10,11,29]. This consistency comforts the interest of pre-treatment blood counts for prediction of ICIs efficacy. Neutrophils and leukocytes levels were also associated and predictive of response, a finding from our study that has not been reported elsewhere, to our knowledge. We also demonstrated that an ECOG performance status ≥2 was significantly associated with response, a result that remained to be fully established [30,31]. High PS and a high NLR were correlated with worse outcomes, consistently with the physio-pathological immune hypothesis of peripheral lymphocytes stimulation with ICIs, which could lead to redirect TILs to destruct tumor cells through an activation cascade.
The present study focused on the predictive power of blood biomarkers, but the model could include additional variables, even if their predictive values are not perfect individually. PD-L1 expression [32], because of its discordant results, its heterogeneity among histological components [33] and its poor accuracy when assessed in peripheral blood [34], is not an ideal predictive biomarker but would probably increase the performance of our model. Some characteristics based on genomic alterations could also be added. Tumor mutational burden (TMB), defined as the total number of nonsynonymous mutations in the tumor exome [35] and evaluated by next-generation-sequencing (NGS), seems to be a potential biomarker [36]. Even if its measure has some limitation such as high cost, large input DNA amount needed and time of analysis, lung cancer is among tumors with the highest TMB [37], which correlates with ICI efficacy in NSCLC [38]. Importantly, our results compare favorably with the latter study, where predictive power of (durable) response from PD-L1 expression (AUC = 0.646) or TMB (AUC = 0.601) was inferior. K-RAS mutation is a frequent genetic alteration with contrasting implications for ICIs efficacy. It was initially not linked with poor response to ICIs [39], but a better response when mutated was shown in 2019 [40], whereas the co-occurring genomic alteration of K-RAS and STK11/LKB1 was associated with a primary resistance to PD1 axis blockade [41]. Other recent work using metabolomics data obtained from blood sampling demonstrated impressive predictive power of response, although with small sample sizes [42,43,44]. The integration of such markers in our model might improve its efficacy. Based on the combination of quantitative imaging and machine learning, radiomics study have also started to emerge for prediction of ICIs efficacy from radiological images [24,45]. Such studies should nevertheless be carefully evaluated for their added predictive power in comparison to simpler biomarkers as the ones used here [46].
The first limitation of our study is the retrospective design and the limited number of centers involved (2 centers). Another limitation is the lack of tumor PD-L1 assessment. This testing was not systematically performed from 2013 to 2017 as nivolumab in second-line or more did not require the PD-L1 status to be prescribed to NSCLC patients. This status could have helped our algorithm using a quantitative variable, freeing ourselves from a qualitative threshold as currently used: 1% for the refund of Durvalumab in France for the maintenance after a radio-chemotherapy or pembrolizumab in second-line therapy [47] and 50% for the use of Pembrolizumab in first-line therapy [2]. Nevertheless, the PD-L1 determination methods were not the same as nowadays, involving either a different or biased analysis or a new determination of PD-L1 status.
Certain confounding factors such as bacterial and viral infection, demographic variables such as race, recent chemotherapy, or the use of corticosteroids before treatment could have modified the blood counts analyzed in this study. These data were not collected and not analyzed in our study because we wanted a simple tool to help clinical practice, which could benefit to the majority of patients. Furthermore, blood counts were not necessarily sampled the day of first ICIs infusion, but sometimes one or some days before. This period of time could also have modified the values. On the other hand, our study reflects clinical practice in real-life.
Eventually, the majority of studies to date, including ours, have focused on static pre-treatment predictive markers. However, dynamic markers that integrate on-treatment data start to emerge, with promising predictive value [48]. In such context, the use of statistical Bayesian modeling is particularly appropriate and holds great promise [49]. For ICI, a mathematical model-derived kinetic parameters of tumor kinetics regrowth during relapse has been shown to be the best predictor of overall survival in multivariable analysis including baseline clinical markers [50]. More mechanistic mathematical models for tumor-immune-ICI-radiotherapy dynamics have also been proposed [51]. In light of our results, such models should be applied to include dynamics of blood counts.

5. Conclusions

Blood counts prior to ICIs (elevation of hemoglobin, decrease of NLR, leukocytes or neutrophils) and clinical status (good PS) were significantly associated with better DCR in multivariable analysis. The practical application of these associations using machine learning algorithms was able to predict individual response to treatment. This could be improved further by increasing the number of variables in the model and should be further validated in an independent cohort.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/cancers13246210/s1, Figure S1: Progression-free and overall survival, Figure S2: Learning curves, Table S1: Blood counts at ICI treatment start, Table S2: Cox regression: progression-free survival, Table S3: Cox regression: overall survival.

Author Contributions

Conceptualization, S.B., D.B., F.B. and L.G.; methodology, S.B.; software, S.B., M.A., M.K. and I.B.-F.; validation, S.B., F.B. and L.G.; formal analysis, S.B.; investigation, S.B. and M.G.; resources, M.G., S.C., P.T., L.G.; data curation, M.G., S.C. and P.T.; writing—original draft preparation, S.B. and M.G.; writing—review and editing, S.B., F.B. and L.G.; visualization, S.B., M.A., M.K. and I.B.-F.; supervision, S.B. and L.G.; project administration, S.B., F.B. and L.G.; funding acquisition, S.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research and APC were funded by Inserm grant number 19CM148-00 as part of the QUANTIC project.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board the French Society of Respiratory Diseases (Société de Pneumologie de Langue Française—SPLF), under reference number: CEPRO 2019-007.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to ethical and privacy restrictions.

Acknowledgments

The authors would like to thank Pauline Fleury for valuable input on the work.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Planchard, D.; Popat, S.; Kerr, K.; Novello, S.; Smit, E.F.; Faivre-Finn, C.; Mok, T.S.; Reck, M.; Van Schil, P.E.; Hellmann, M.D.; et al. Metastatic non-small cell lung cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann. Oncol. 2018, 29 (Suppl. 4), iv192–iv237. [Google Scholar] [CrossRef] [PubMed]
  2. Reck, M.; Rodríguez-Abreu, D.; Robinson, A.G.; Hui, R.; Csőszi, T.; Fülöp, A.; Gottfried, M.; Peled, N.; Tafreshi, A.; Cuffe, S.; et al. Pembrolizumab versus Chemotherapy for PD-L1-Positive Non-Small-Cell Lung Cancer. N. Engl. J. Med. 2016, 375, 1823–1833. [Google Scholar] [CrossRef] [Green Version]
  3. Gooden, M.J.M.; de Bock, G.H.; Leffers, N.; Daemen, T.; Nijman, H.W. The prognostic influence of tumour-infiltrating lymphocytes in cancer: A systematic review with meta-analysis. Br. J. Cancer 2011, 105, 93–103. [Google Scholar] [CrossRef] [Green Version]
  4. Balkwill, F.; Mantovani, A. Inflammation and cancer: Back to Virchow? Lancet 2001, 357, 539–545. [Google Scholar] [CrossRef]
  5. Hopkins, A.M.; Rowland, A.; Kichenadasse, G.; Wiese, M.D.; Gurney, H.; McKinnon, R.A.; Karapetis, C.S.; Sorich, M.J. Predicting response and toxicity to immune checkpoint inhibitors using routinely available blood and clinical markers. Br. J. Cancer 2017, 117, 913–920. [Google Scholar] [CrossRef]
  6. Diem, S.; Schmid, S.; Krapf, M.; Flatz, L.; Born, D.; Jochum, W.; Templeton, A.J.; Früha, M. Neutrophil-to-Lymphocyte ratio (NLR) and Platelet-to-Lymphocyte ratio (PLR) as prognostic markers in patients with non-small cell lung cancer (NSCLC) treated with nivolumab. Lung Cancer 2017, 111, 176–181. [Google Scholar] [CrossRef] [PubMed]
  7. Proctor, M.J.; McMillan, D.C.; Morrison, D.S.; Fletcher, C.D.; Horgan, P.G.; Clarke, S.J. A derived neutrophil to lymphocyte ratio predicts survival in patients with cancer. Br. J. Cancer 2012, 107, 695–699. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Ferrucci, P.F.; Ascierto, P.A.; Pigozzo, J.; Del Vecchio, M.; Antonini Cappellini, G.C.; Guidoboni, M.; Queirolo, P.; Savoia, P.; Mandala, M.; Simeone, E.; et al. Baseline neutrophils and derived neutrophil-to-lymphocyte ratio: Prognostic relevance in metastatic melanoma patients receiving ipilimumab. Ann. Oncol. 2016, 27, 732–738. [Google Scholar] [CrossRef] [PubMed]
  9. Diakos, C.I.; Tu, D.; Gebski, V.; Yip, S.; Wilson, K.; Karapetis, C.S.; O’Callaghan, C.J.; Shapiro, J.; Tebbutt, N.; Jonker, D.J.; et al. Is the Derived Neutrophil to Lymphocyte Ratio (Dnlr) an Independent Prognostic Marker in Patients with Metastatic Colorectal Cancer (Mcrc)? Analysis of the CO.17 And CO.20 Studies. Ann. Oncol. 2016, 27, vi196. [Google Scholar] [CrossRef]
  10. Gu, X.; Sun, S.; Gao, X.-S.; Xiong, W.; Qin, S.; Qi, X.; Ma, M.; Li, X.; Zhou, D.; Wang, W.; et al. Prognostic Value of Platelet to Lymphocyte Ratio in Non-Small Cell Lung Cancer: Evidence from 3430 Patients. Sci. Rep. 2016, 6, 23893. [Google Scholar] [CrossRef] [PubMed]
  11. Mezquita, L.; Auclin, E.; Ferrara, R.; Charrier, M.; Remon, J.; Planchard, D.; Ponce, S.; Paz Ares, L.; Leroy, L.; Audigier-Valette, C.; et al. Association of the Lung Immune Prognostic Index with Immune Checkpoint Inhibitor Outcomes in Patients with Advanced Non–Small Cell Lung Cancer. JAMA Oncol. 2018, 4, 351–357. [Google Scholar] [CrossRef] [PubMed]
  12. Pavan, A.; Calvetti, L.; Dal Maso, A.; Attili, I.; Del Bianco, P.; Pasello, G.; Guarneri, V.; Aprile, G.; Conte, P.F.; Bonanno, L. Peripheral Blood Markers Identify Risk of Immune-Related Toxicity in Advanced Non-Small Cell Lung Cancer Treated with Immune-Checkpoint Inhibitors. Oncologist 2019, 24, 1128–1136. [Google Scholar] [CrossRef] [Green Version]
  13. Barlesi, F.; Greillier, L.; Monville, F.; Foa, C.; le Treut, J.; Audigier-Valette, C.; Vély, F.; Garcia, S.; Sabatier, F.; Ciccolini, J.; et al. LBA53 Precision immuno-oncology for advanced non-small cell lung cancer (NSCLC) patients (pts) treated with PD1/L1 immune checkpoint inhibitors (ICIs): A first analysis of the PIONeeR study. Ann. Oncol. 2020, 31, S1183. [Google Scholar] [CrossRef]
  14. Benzekry, S. Artificial intelligence and mechanistic modeling for clinical decision making in oncology. Clin. Pharmacol. Ther. 2020, 108, 471–486. [Google Scholar] [CrossRef] [PubMed]
  15. Breiman, L. Statistical modeling: The two cultures. Stat. Sci. 2001, 16, 199–231. [Google Scholar] [CrossRef]
  16. Ahn, B.-C.; So, J.-W.; Synn, C.-B.; Kim, T.H.; Kim, J.H.; Byeon, Y.; Kim, Y.S.; Heo, S.G.; Yang, S.-D.; Yun, M.R.; et al. Clinical decision support algorithm based on machine learning to assess the clinical response to anti-programmed death-1 therapy in patients with non-small-cell lung cancer. Eur. J. Cancer 2021, 153, 179–189. [Google Scholar] [CrossRef]
  17. Eisenhauer, E.A.; Therasse, P.; Bogaerts, J.; Schwartz, L.H.; Sargent, D.; Ford, R.; Dancey, J.; Arbuck, S.; Gwyther, S.; Mooney, M.; et al. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). Eur. J. Cancer 2009, 45, 228–247. [Google Scholar] [CrossRef] [PubMed]
  18. R: The R Project for Statistical Computing. Available online: https://www.r-project.org/ (accessed on 2 October 2019).
  19. Cox, D.R. Regression Models and Life-Tables. J. R. Stat. Soc. Ser. B Methodol. 1972, 34, 187–202. [Google Scholar] [CrossRef]
  20. Therneau, T.M. A Package for Survival Analysis in R. Available online: https://CRAN.R-project.org/package=survival (accessed on 8 December 2020).
  21. Kuhn, M.; Wickham, H. Tidymodels: A Collection of Packages for Modeling and Machine Learning Using Tidyverse Principles. Available online: https://www.tidymodels.org (accessed on 8 December 2020).
  22. Becker, T.; Weberpals, J.; Jegg, A.M.; So, W.V.; Fischer, A.; Weisser, M.; Schmich, F.; Rüttinger, D.; Bauer-Mehren, A. An enhanced prognostic score for overall survival of patients with cancer derived from a large real-world cohort. Ann. Oncol. 2020, 31, 1561–1568. [Google Scholar] [CrossRef] [PubMed]
  23. Nicolò, C.; Périer, C.; Prague, M.; Bellera, C.; MacGrogan, G.; Saut, O.; Benzakry, S. Machine Learning and Mechanistic Modeling for Prediction of Metastatic Relapse in Early-Stage Breast Cancer. JCO Clin. Cancer Inform. 2020, 4, 259–274. [Google Scholar] [CrossRef]
  24. Trebeschi, S.; Drago, S.G.; Birkbak, N.J.; Kurilova, I.; Călin, A.M.; Delli Pizzi, A.; Lalezari, F.; Lambregts, D.M.J.; Rohaan, M.W.; Parmar, C.; et al. Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers. Ann. Oncol. 2019, 30, 998–1004. [Google Scholar] [CrossRef] [Green Version]
  25. Wiesweg, M.; Mairinger, F.; Reis, H.; Goetz, M.; Walter, R.F.H.; Hager, T.; Metzenmacher, M.; Eberhardt, W.E.E.; McCutcheon, A.; Aigner, C.; et al. Machine learning-based predictors for immune checkpoint inhibitor therapy of non-small-cell lung cancer. Ann. Oncol. 2019, 30, 655–657. [Google Scholar] [CrossRef] [PubMed]
  26. Wiesweg, M.; Mairinger, F.; Reis, H.; Goetz, M.; Walter, R.F.H.; Hager, T.; Metzenmacher, M.; Eberhardt, W.E.E.; McCutcheonet, A.; Köster, J.; et al. Machine learning reveals a PD-L1–independent prediction of response to immunotherapy of non-small cell lung cancer by gene expression context. Eur. J. Cancer 2020, 140, 76–85. [Google Scholar] [CrossRef]
  27. Riley, R.D.; Ensor, J.; Snell, K.I.E.; Harrell, F.E., Jr.; Martin, G.P.; Reitsma, J.B.; Moons, K.G.M.; Collins, G.; van Smeden, M. Calculating the sample size required for developing a clinical prediction model. BMJ 2020, 368, m441. [Google Scholar] [CrossRef] [Green Version]
  28. Christodoulou, E.; Ma, J.; Collins, G.S.; Steyerberg, E.W.; Verbakel, J.Y.; Van Calster, B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J. Clin. Epidemiol. 2019, 110, 12–22. [Google Scholar] [CrossRef] [PubMed]
  29. Zhang, Z.; Zhang, F.; Yuan, F.; Li, Y.; Ma, J.; Ou, Q.; Liu, Z.; Yang, B.; Wang, L.; Tao, H.; et al. Pretreatment hemoglobin level as a predictor to evaluate the efficacy of immune checkpoint inhibitors in patients with advanced non-small cell lung cancer. Ther. Adv. Med. Oncol. 2020, 12, 1758835920970049. [Google Scholar] [CrossRef] [PubMed]
  30. Dall’Olio, F.G.; Maggio, I.; Massucci, M.; Mollica, V.; Fragomeno, B.; Ardizzoni, A. ECOG performance status ≥2 as a prognostic factor in patients with advanced non small cell lung cancer treated with immune checkpoint inhibitors—A systematic review and meta-analysis of real world data. Lung Cancer 2020, 145, 95–104. [Google Scholar] [CrossRef] [PubMed]
  31. Krishnan, M.; Kasinath, P.; High, R.; Yu, F.; Teply, B.A. Impact of Performance Status on Response and Survival Among Patients Receiving Checkpoint Inhibitors for Advanced Solid Tumors. JCO Oncol. Pract. 2021. [Google Scholar] [CrossRef]
  32. Melosky, B.; Chu, Q.; Juergens, R.A.; Leighl, N.; Ionescu, D.; Tsao, M.-S.; McLeod, D.; Hirsh, V. Breaking the biomarker code: PD-L1 expression and checkpoint inhibition in advanced NSCLC. Cancer Treat. Rev. 2018, 65, 65–77. [Google Scholar] [CrossRef] [PubMed]
  33. Lium, Y.; Dong, Z.; Jiang, T.; Hou, L.; Wu, F.; Gao, G.; He, Y.; Zhao, J.; Li, X.; Zhao, C.; et al. Heterogeneity of PD-L1 Expression Among the Different Histological Components and Metastatic Lymph Nodes in Patients with Resected Lung Adenosquamous Carcinoma. Clin. Lung Cancer 2018, 19, e421–e430. [Google Scholar]
  34. Boffa, D.J.; Graf, R.P.; Salazar, M.C.; Hoag, J.; Lu, D.; Krupa, R.; Louw, J.; Dugan, L.; Wang, Y.; Landers, M.; et al. Cellular Expression of PD-L1 in the Peripheral Blood of Lung Cancer Patients is Associated with Worse Survival. Cancer Epidemiol. Prev. Biomark. 2017, 26, 1139–1145. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Alexandrov, L.B.; Nik-Zainal, S.; Wedge, D.C.; Aparicio, S.A.J.R.; Behjati, S.; Biankin, A.V.; Bignell, G.R.; Bolli, N.; Borg, A.; Børresen-Dale, A.-L.; et al. Signatures of mutational processes in human cancer. Nature 2013, 500, 415–421. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Berland, L.; Heeke, S.; Humbert, O.; Macocco, A.; Long-Mira, E.; Lassalle, S.; Lespinet-Fabre, V.; Lalvée, S.; Bordone, O.; Cohenet, C.; et al. Current views on tumor mutational burden in patients with nonsmall cell lung cancer treated by immune checkpoint inhibitors. J. Thorac. Dis. 2018, 11, S71–S80. [Google Scholar] [CrossRef] [PubMed]
  37. Lawrence, M.S.; Stojanov, P.; Polak, P.; Kryukov, G.V.; Cibulskis, K.; Sivachenko, A.; Carter, S.L.; Stewart, C.; Mermel, C.H.; Roberts, S.A.; et al. Mutational heterogeneity in cancer and the search for new cancer genes. Nature 2013, 499, 214–218. [Google Scholar] [CrossRef] [PubMed]
  38. Rizvi, H.; Sanchez-Vega, F.; La, K.; Chatila, W.; Jonsson, P.; Halpenny, D.; Plodkowski, A.; Long, N.; Sauter, J.L.; Rekhtman, N.; et al. Molecular Determinants of Response to Anti–Programmed Cell Death (PD)-1 and Anti–Programmed Death-Ligand 1 (PD-L1) Blockade in Patients with Non–Small-Cell Lung Cancer Profiled with Targeted Next-Generation Sequencing. J. Clin. Oncol. 2018, 36, 633–641. [Google Scholar] [CrossRef]
  39. Jeanson, A.; Tomasini, P.; Souquet-Bressand, M.; Brandone, N.; Boucekine, M.; Grangeon, M.; Chaleat, S.; Khobta, N.; Milia, J.; Mhanna, L.; et al. Efficacy of Immune Checkpoint Inhibitors in KRAS-Mutant Non-Small Cell Lung Cancer (NSCLC). J. Thorac. Oncol. 2019, 14, 1095–1101. [Google Scholar] [CrossRef]
  40. Mazieres, J.; Drilon, A.; Lusque, A.; Mhanna, L.; Cortot, A.B.; Mezquita, L.; Thai, A.A.; Mascaux, C.; Couraud, S.; Veillon, R.; et al. Immune checkpoint inhibitors for patients with advanced lung cancer and oncogenic driver alterations: Results from the IMMUNOTARGET registry. Ann. Oncol. 2019, 30, 1321–1328. [Google Scholar] [CrossRef] [PubMed]
  41. Skoulidis, F.; Goldberg, M.E.; Greenawalt, D.M.; Hellmann, M.D.; Awad, M.M.; Gainor, J.F.; Schrock, A.B.; Hartmaier, R.J.; Trabucco, S.E.; Gay, L.; et al. STK11/LKB1 Mutations and PD-1 Inhibitor Resistance in KRAS-Mutant Lung Adenocarcinoma. Cancer Discov. 2018, 8, 822–835. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Li, H.; Bullock, K.; Gurjao, C.; David Braun, D.; Shukla, S.A.; Bossé, D.; Lalani, A.-K.A.; Gopal, S.; Jin, C.; Horak, C.; et al. Metabolomic adaptations and correlates of survival to immune checkpoint blockade. Nat. Commun. 2019, 10, 4346. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  43. Hatae, R.; Chamoto, K.; Kim, Y.H.; Sonomura, K.; Taneishi, K.; Kawaguchi, S.; Yoshida, H.; Ozasa, H.; Sakamori, Y.; Akrami, M.; et al. Combination of host immune metabolic biomarkers for the PD-1 blockade cancer immunotherapy. JCI Insight 2020, 5, e133501. [Google Scholar] [CrossRef] [Green Version]
  44. Ghini, V.; Laera, L.; Fantechi, B.; del Monte, F.; Benelli, M.; McCartney, A.; Tenori, L.; Luchinat, C.; Pozzessere, D. Metabolomics to Assess Response to Immune Checkpoint Inhibitors in Patients with Non-Small-Cell Lung Cancer. Cancers 2020, 12, 3574. [Google Scholar] [CrossRef] [PubMed]
  45. Sun, R.; Limkin, E.J.; Vakalopoulou, M.; Dercle, L.; Champiat, S.; Han, S.R.; Verlingue, L.; Brandao, D.; Lancia, A.; Ammari, S.; et al. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: An imaging biomarker, retrospective multicohort study. Lancet Oncol. 2018, 19, 1180–1191. [Google Scholar] [CrossRef]
  46. Welch, M.L.; McIntosh, C.; Haibe-Kains, B.; Milosevic, M.F.; Dekker, L.W.A.; Huan, S.H.; Purdie, T.G.; O’Sullivan, B.; Aerts, H.J.W.L.; Jaffray, D.A. Vulnerabilities of radiomic signature development: The need for safeguards. Radiother. Oncol. J. Eur. Soc. Ther. Radiol. Oncol. 2019, 130, 2–9. [Google Scholar] [CrossRef] [Green Version]
  47. Herbst, R.S.; Baas, P.; Kim, D.-W.; Felip, E.; Pérez-Gracia, J.L.; Han, J.-Y.; Molina, J.; Kim, L.-H.; Dubos Arvis, C.; Ahn, M.-J.; et al. Pembrolizumab versus docetaxel for previously treated, PD-L1-positive, advanced non-small-cell lung cancer (KEYNOTE-010): A randomised controlled trial. Lancet 2016, 387, 1540–1550. [Google Scholar] [CrossRef]
  48. Mezquita, L.; Preeshagul, I.; Auclin, E.; Saravia, D.; Hendriks, L.; Rizvi, H.; Park, H.R.W.; Nadal, E.; Martin-Romano, P.; Ruffinelli, J.C.; et al. Predicting immunotherapy outcomes under therapy in patients with advanced NSCLC using dNLR and its early dynamics. Eur. J. Cancer 2021, 151, 211–220. [Google Scholar] [CrossRef]
  49. Kurtz, D.M.; Esfahani, M.S.; Scherer, F.; Soo, J.; Jin, M.C.; Liu, C.L.; Newman, A.M.; Dührsen, U.; Hüttmann, A.; Casasnovas, O.; et al. Dynamic Risk Profiling Using Serial Tumor Biomarkers for Personalized Outcome Prediction. Cell 2019, 178, 699–713. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  50. Claret, L.; Jin, J.Y.; Ferté, C.; Winter, H.; Girish, S.; Stroh, M.; He, P.; Ballinger, M.; Sandler, A.; Joshi, A.; et al. A Model of Overall Survival Predicts Treatment Outcomes with Atezolizumab versus Chemotherapy in Non-Small Cell Lung Cancer Based on Early Tumor Kinetics. Clin. Cancer Res. 2018, 24, 3292–3298. [Google Scholar] [CrossRef] [Green Version]
  51. Serre, R.; Benzekry, S.; Padovani, L.; Christophe Meille, C.; André, N.; Ciccolini, J.; Barlesi, F.; Muracciole, X.; Barbolosi, D. Mathematical Modeling of Cancer Immunotherapy and Its Synergy with Radiotherapy. Cancer Res. 2016, 76, 4931–4940. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Cancers 13 06210 g001 550
Figure 1. Exploratory data analysis. (A) Boxplots of continuous variables. (B) Barplots of categorical variables. BMI = body mass index, NLR = neutrophil-to-lymphocyte ratio, PLR = platelets-to-lymphocytes ratio, CR = complete response, PR = partial response, SD = stable disease and PD = progressive disease. Stars indicate statistical significance: **: p < 0.01, ***: p < 0.001, ****: p < 0.0001, n.s. = non-significant.
Figure 1. Exploratory data analysis. (A) Boxplots of continuous variables. (B) Barplots of categorical variables. BMI = body mass index, NLR = neutrophil-to-lymphocyte ratio, PLR = platelets-to-lymphocytes ratio, CR = complete response, PR = partial response, SD = stable disease and PD = progressive disease. Stars indicate statistical significance: **: p < 0.01, ***: p < 0.001, ****: p < 0.0001, n.s. = non-significant.
Cancers 13 06210 g001
Cancers 13 06210 g002 550
Figure 2. Variable selection. (A) Feature importance based on random forest classification and mean decrease in accuracy. (B) Accuracy score of incremental logistic regression models built on an increasing number of predictors (i.e., the first one contains only hemoglobin, the second hemoglobin and NLR, etc.). NLR = neutrophil-to-lymphocyte ratio. PLR = platelet-to-lymphocyte ratio. BMI = body mass index.
Figure 2. Variable selection. (A) Feature importance based on random forest classification and mean decrease in accuracy. (B) Accuracy score of incremental logistic regression models built on an increasing number of predictors (i.e., the first one contains only hemoglobin, the second hemoglobin and NLR, etc.). NLR = neutrophil-to-lymphocyte ratio. PLR = platelet-to-lymphocyte ratio. BMI = body mass index.
Cancers 13 06210 g002
Cancers 13 06210 g003 550
Figure 3. Machine learning algorithms predictive performances. (A) Receiver-operator curves for prediction on test sets from each fold of the outer cross-validation loop, for each model. AUC = area under the curve. (B) Precision (positive-predictive value)–recall (sensitivity) curves. (C) Main performance metrics for each algorithm. (D) Decision tree obtained after tuning and training. Each node shows: the predicted class (0 = PD, 1 = CR + PR + SD), the predicted probability of response and the percentage of total observations in the node.
Figure 3. Machine learning algorithms predictive performances. (A) Receiver-operator curves for prediction on test sets from each fold of the outer cross-validation loop, for each model. AUC = area under the curve. (B) Precision (positive-predictive value)–recall (sensitivity) curves. (C) Main performance metrics for each algorithm. (D) Decision tree obtained after tuning and training. Each node shows: the predicted class (0 = PD, 1 = CR + PR + SD), the predicted probability of response and the percentage of total observations in the node.
Cancers 13 06210 g003
Table
Table 1. Patients and disease characteristics.
Table 1. Patients and disease characteristics.
VariableN = 298 1
Age62 (55, 69)
Sex
Female99 (33%)
Male199 (67%)
Tobacco status
Former Smoker140 (47%)
Non smoker36 (12%)
Smoker122 (41%)
Brain metastases72 (29%)
Performance status
≥226 (8.9%)
0–1265 (91%)
Mutation profile
ALK1 (0.8%)
BRAF9 (7.4%)
EGFR14 (11%)
KRAS87 (71%)
Other mutation5 (4.1%)
ROS12 (1.6%)
Wild type4 (3.3%)
Immunotherapy type
anti-CTLA-43 (1.0%)
anti-PD-1266 (89%)
anti-PD-L126 (8.7%)
Combination3 (1.0%)
Previous treatment
Chemotherapy281 (95%)
Chemotherapy + immunotherapy5 (1.7%)
Targeted therapy11 (3.7%)
Response
Complete response2 (0.7%)
Partial response44 (15%)
Progressive disease131 (45%)
Stable disease113 (39%)
1 Median (inter-quartile range); n (%).
Table
Table 2. Logistic regression analysis for disease control.
Table 2. Logistic regression analysis for disease control.
Univariable Logistic RegressionMultivariable Logistic Regression
VariableOdds Ratio [95% CI]pSignifOdds Ratio [95% CI]pSignif
Lymphocytes1.1 [0.83, 1.4]0.678 0.98 [0.15, 5.2]0.984
NLR0.49 [0.31, 0.73]0.000879***0.68 [0.098, 1.9]0.651
Platelets1 [0.82, 1.3]0.762 1.3 [0.72, 2.4]0.404
PLR0.84 [0.64, 1.1]0.156 1.1 [0.5, 2.4]0.788
Leukocytes0.68 [0.5, 0.89]0.00791**0.6 [0.0022, 3 × 102]0.847
Hemoglobin1.9 [1.5, 2.5]9.26 × 10−7***1.8 [1.3, 2.4]0.000122***
dNLR0.63 [0.47, 0.83]0.00155**0.8 [0.33, 2.7]0.689
Neutrophils0.62 [0.45, 0.83]0.00232**1.5 [0.0047, 2.7 × 102]0.863
Monocytes0.87 [0.69, 1.1]0.226 0.86 [0.5, 1.4]0.545
Eosinophils1.3 [0.97, 1.9]0.139 1.1 [0.75, 1.8]0.582
Basophils1.2 [0.95, 1.8]0.177 1.2 [0.89, 1.8]0.321
BMI1.2 [0.95, 1.5]0.123 1 [0.76, 1.3]0.997
Performance status0.5 [0.39, 0.64]6.21 × 10−8***0.58 [0.44, 0.75]7.79 × 10−5***
Stars indicate statistical significance: ** : p < 0.01, *** : p < 0.001. CI = confidence interval. signif = significant.
Table
Table 3. Summary of machine learning algorithms predictive performances (mean ± standard deviation, bold entries are maximum values).
Table 3. Summary of machine learning algorithms predictive performances (mean ± standard deviation, bold entries are maximum values).
ModelAccuracyROC AUCPPVNPVSensitivitySpecificity
Random Forest0.68 ± 0.040.74 ± 0.030.70 ± 0.080.68 ± 0.060.58 ± 0.080.78 ± 0.06
Logistic Regression0.67 ± 0.040.73 ± 0.030.69 ± 0.080.67 ± 0.060.57 ± 0.090.77 ± 0.07
Naive Bayes0.67 ± 0.040.73 ± 0.030.72 ± 0.070.65 ± 0.060.49 ± 0.070.83 ± 0.05
Single Layer Neural Network0.66 ± 0.030.72 ± 0.030.69 ± 0.090.66 ± 0.060.54 ± 0.090.78 ± 0.07
k-Nearest Neighbour0.66 ± 0.040.69 ± 0.040.65 ± 0.070.66 ± 0.060.58 ± 0.070.73 ± 0.07
Linear SVM0.58 ± 0.090.73 ± 0.030.72 ± 0.090.58 ± 0.100.19 ± 0.250.94 ± 0.09
Polynomial SVM0.55 ± 0.080.73 ± 0.030.61 ± 0.130.58 ± 0.130.19 ± 0.290.89 ± 0.23
Radial basis SVM0.55 ± 0.080.73 ± 0.030.67 ± 0.170.56 ± 0.060.20 ± 0.280.88 ± 0.25
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Benzekry, S.; Grangeon, M.; Karlsen, M.; Alexa, M.; Bicalho-Frazeto, I.; Chaleat, S.; Tomasini, P.; Barbolosi, D.; Barlesi, F.; Greillier, L. Machine Learning for Prediction of Immunotherapy Efficacy in Non-Small Cell Lung Cancer from Simple Clinical and Biological Data. Cancers 2021, 13, 6210. https://doi.org/10.3390/cancers13246210

AMA Style

Benzekry S, Grangeon M, Karlsen M, Alexa M, Bicalho-Frazeto I, Chaleat S, Tomasini P, Barbolosi D, Barlesi F, Greillier L. Machine Learning for Prediction of Immunotherapy Efficacy in Non-Small Cell Lung Cancer from Simple Clinical and Biological Data. Cancers. 2021; 13(24):6210. https://doi.org/10.3390/cancers13246210

Chicago/Turabian Style

Benzekry, Sébastien, Mathieu Grangeon, Mélanie Karlsen, Maria Alexa, Isabella Bicalho-Frazeto, Solène Chaleat, Pascale Tomasini, Dominique Barbolosi, Fabrice Barlesi, and Laurent Greillier. 2021. "Machine Learning for Prediction of Immunotherapy Efficacy in Non-Small Cell Lung Cancer from Simple Clinical and Biological Data" Cancers 13, no. 24: 6210. https://doi.org/10.3390/cancers13246210

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop