*3.5. Potential Impact Simulation*

A potential impact study compared the risk of KRT estimated by nephrologists with those calculated by PROGRES-CKD-24 and investigated the potential incremental efficiency of using PROGRES-CKD compared to physicians' assessments to inform referral to an intensified multidisciplinary prevention program to delay progression to ESKD.

Table 4 reports ratings of CKD progression risks provided by either physicians or the prediction model. In the evaluation sample, 25 patients required KRT within 2 years, while 53 patients did not reach the study endpoint. PROGRES-CKD-24 had excellent discrimination within this dataset (AUC = 0.96), while experts' ratings demonstrated good discrimination (average AUC = 0.79), with average sensitivity = 0.64 and average specificity = 0.85 at the optimal cut-off point (score > 6). Therefore, experts were less discriminative of endpoint occurrence compared to PROGRES-CKD-24 (ΔM-E = 0.17, *p* = 0.005). The correlation of physicians' ratings with PROGRES-CKD-24 ratings was moderate (r = 0.50, *p* < 0.01); furthermore, experts showed different abilities to discriminate patients' risk. (Table 4).

**Table 4.** PROGRES-CKD-24 and Experts' ratings of CKD progression risk.


Figure 4 shows the results of our impact simulation. Based on the experts' ratings (PPV = 17%; FOR = 2%), *n* = 1725 (17.3%) patients would be assigned to the high-risk category, while *n* = 8275 (82.8%) would be recommended to the standard care program (Figure 4, panel A). Based on the assumptions set for the simulation exercise (i.e., ESKD overall incidence without intervention: 2.3 events/100 patient-years; ESKD risk is reduced by 50% in the intensified intervention group) there would be 362 ESKD events overall.

Therefore, in this scenario, physicians' referral to the intensified program would delay 98 ESKD cases (i.e., an Overall Program Effect Size of 1.27). The number of patients needed to treat would be NNT = 18 (Figure 4, panel D). Conversely, risk stratification by PROGRES-CKD-24 (PPV = 48%; FOR = 1.2%) leads to referral of *n* = 732 (0.73%) patients to intensified intervention (Figure 4, panel B). In this case, 117 ESRD events would be prevented, i.e., an Overall Program Effect Size of 1.36. The number needed to treat would be NNT = 6 (Figure 4, panel D). Finally, under a hypothetical risk averse policy that would refer all stage 3 CKD patients to the intensified program, 153 ESRD events would be prevented with NNT = 65 (Figure 4, panel C).

**Figure 4.** Potential impact simulation of PROGRES-CKD-24 implementation in a hypothetical CKD cohort. Flowcharts showing patients' referral to intensified intervention programs based on (**A**) experts' ratings, and (**B**) PROGRES-CKD scores; (**C**) Number of ESKD events within 24 months: both observed and saved cases are shown; D) Number of patients needed to treat to save 1 patient; "all-in strategy" involves referral of all stage 3 patients to the intensified healthcare program. Abbreviations: ESKD, end-stage kidney disease; NNT, Number needed to treat.

#### **4. Discussion**

The present study reports the derivation and validation of the PROGRES-CKD algorithm in two independent cohorts of non-dialysis dependent CKD patients. Discrimination accuracy of PROGRES-CKD was excellent for both the short-term prediction horizon (6 months) and the long-term prediction horizon (24 months).

Of note is the fact that PROGRES-CKD-6 and PROGRES-CKD-24 had reproducible discrimination accuracy in both validation studies. The FMC NephroCare cohort included real-world clinical data of stage 3–5 CKD patients from 15 countries (Europe, South-America, Africa), while the GCKD study is a prospective CKD cohort study recruiting a wider range of NDD-CKD patients with moderate GFR impairment in Germany [21]. Given the substantial differences between the two cohorts in geographical area of recruitment (international vs. national), inclusion/exclusion criteria, and data collection strategies (real-world vs. pre-specified protocol), the observed consistency in discrimination and calibration corroborates the generalizability of PROGRES-CKD across different CKD subpopulations and clinical settings.

To further characterize PROGRES-CKD accuracy, we compared its discrimination performance against KFREs which were extensively validated in different CKD patient populations [11,17,18] and are routinely used in clinical practice. PROGRES-CKD was as accurate as KFREs for 24-month prediction in both validation cohorts and more accurate for 6-month forecasting in the GCKD study. Even though the two algorithms showed comparable performance in long-term prediction, the KFRE risk score could not be computed in a vast share of patients of the FMC NephroCare cohort because of missing information of key input variables (Figure 5). Conversely, PROGRES-CKD was available for all patients due to accurate handling of missing variables inherent to naïve Bayes classifiers (Figure 5) [36]. In fact, PROGRES-CKD potentially incorporates input from as many as 32 clinical parameters, yet its prediction can be computed with any subset of information. Therefore, PROGRES-CKD performance remained stable even for patients with many missing parameters representative of a real-world clinical practice setting. Furthermore, by assessment of VOI metrics, PROGRES-CKD allows the graphical representation of the uncertainty around prediction due to missing data. Given that VOI metrics are calculated for each missing clinical parameter within the patient's health records, they can be used to rank the potential prognostic benefit of additional diagnostic testing or biomarker assays for patients with incomplete medical data. These peculiar features of PROGRES-CKD significantly increase its clinical usability in that they enable to address the problem of missing predictors in real-world data [17] by exploiting the full wealth of information collected in routine clinical practice.

**Figure 5.** Discrimination ability of PROGRES-CKD and KFREs and percentage of computed scores by each prediction tool. Only cases with complete medical information were included in this analysis. (**A**) RRT prediction within 6 months; (**B**) RRT prediction within 24 months. Bars denote AUC (left yaxis), while dots denote the percentage of computed scores on the total number of recruited patients in each cohort (right y-axis). Abbreviations: P-CKD6, PROGRES-CKD-6; P-CKD24, PROGRES-CKD-24; 4VAR, KFRE 4 variables; 6VAR, KFRE 6 variables.

One additional advantage of NBCs such as PROGRES-CKD over traditional equationbased prediction tools rest in their ability to generate personalized, patient-specific impact metrics representing the relative contribution of each predictor to a patient's risk. Impact metrics can be used to estimate the potential impact of interventions addressing modifiable risk factors. This has important implications for patient care, since there can be considerable heterogeneity in underlying diseases, demographics, co-morbidities, and risk for progression among CKD patients and, consequently, optimal intervention strategies might deviate between patients with the same overall risk estimate depending on their individual high impact risk parameters. Therefore, both VOI and impact metrics could help physicians within their decision-making processes in tailoring interventions according to each individual patient's needs and characteristics [37]. Adoption of a more personalized clinical approach would lead not only to improved CKD clinical management (targeted diagnostic and treatment investigations with minimum adverse events and maximum efficacy, and consequently increased adherence to treatment), but it could also contribute towards optimizing the utilization of healthcare resources. In fact, ranking clinical parameters by their impact on risk score computation helps physicians' reasoning on priority and enables strategic and rational formulation of therapeutic plans considering both patient/disease-related factors and resource availability.

One specification of PROGRES-CKD allows the identification of patients whose kidney function is more likely to deteriorate within 6 months, a feature enabling timely referral to vascular access creation services and transition management [38,39]. The potential advantages of accurate short-term progression are two-fold. Patients starting on chronic dialysis with an arteriovenous fistula (AVF) rather than catheter have improved clinical outcomes in terms of survival, hospitalization, and complications [40]. On the other hand, inappropriate AVF creation in stage 4 and 5 patients who do not rapidly progress to KF is associated with complications and premature loss of patency [38].

Accurate risk prediction is a challenging task for physicians in real-world clinical practice, due to a number of disease, clinician, and organization related factors, including: inherent heterogeneity and variability in CKD progression rates [41,42], incomplete information, unrecognized case ambiguity, overconfidence leading to reduced analytical scrutiny, wrong perception of average population risk, over-generalization, fatigue, working overload, aging, altered affect impairing executive memory, switch of analytic scrutiny, and inexperience [43–48]. Therefore, readily available risk scores which prove to be accurate, generalizable to a wide array of CKD subpopulations and settings, and robust to missing data patterns observed in real-life applications may considerably assist clinical decision making, particularly when providing the opportunity to simulate the impact of interventions to individual patient cases.

In order to estimate the potential impact of improved prognostication around CKD progression on process outcomes, clinical outcomes, and costs [38,49], we conducted a simplified simulation using PROGRES-CKD as a patient stratification system for referral to intensified prevention programs for non-dialysis dependent (NDD)-CKD patients. In our simulation, risk estimates provided by either PROGRES-CKD or nephrology experts were used to stratify CKD patients. Subjects assigned to the "high-risk" category are referred to an intensified healthcare program aimed at reducing the risk of CKD progression. Our analysis suggested that PROGRES-CKD-driven referral to the intensified program would be more effective and largely more efficient than referral patterns determined by both healthcare expert risk assessment and an "all-in strategy" (i.e., all patients are referred to the intensified healthcare program when they reach stage 3 CKD). Therefore, personalized, risk-based referral may improve the efficiency of healthcare systems by enhancing the appropriateness of resource allocation in terms of direct expenditures and staff utilization. Personalized referral, however, is not just a matter of mere efficiency. In fact, inappropriate referral to the intensified intervention would involve unnecessary medicalization with greater risks of adverse events, impoverishment of quality of life even in people with a very low risk of progression, increased rate of therapeutic fatigue, and reduced adherence. Conversely, accurate and reliable patient stratification helps physicians and healthcare providers balance individual patient needs with overall resource utilization, ultimately leading to more effective care for both the individual patient and the population [50].

#### **5. Limitations**

Validation of risk score should be considered a continuous process of generalization tests rather than a single experiment. While the performance of PROGRES-CKD was stable in both well-conducted longitudinal cohort studies (i.e., GCKD) and historical cohorts of real-life practice (i.e., FMC NephroCare), evidence concerning PROGRES-CKD robustness with real-world-representing clinical practices outside FMC NephroCare is still missing. For this reason, PROGRES-CKD undergoes a periodical process of performance monitoring while external cohorts for validation exercises are actively sought for.

#### **6. Conclusions**

The Prognostic Reasoning System for CKD patients (PROGRES-CKD) demonstrated excellent discrimination accuracy in two independent cohorts of NDD-CKD patients. The underlying models provide accurate prediction for both 24 and 6 months KRT risk. Contrary to traditional equation-based algorithms which cannot be applied to a large proportion of patients with incomplete data, PROGRES-CKD extends to all patients and allows explicit assessment of prediction robustness in case of missing values for key risk factors. Furthermore, PROGRES-CKD enhances prognostic reasoning by providing patientspecific impact metrics representing the relative contribution of each predictor to a patient's risk and can be used to estimate the potential impact of tailored interventions in addressing individual and modifiable risk factors. While PROGRES-CKD-24 may contribute to efficient and effective referral to intensified prevention programs for NDD-CKD patients, prediction of short-term outcomes (PROGRES-CKD-6) can be a key enabler of timely AVF creation and transition management. Given these results, both PROGRES-CKD algorithms reported here have the potential to advance current standards in routine CKD risk estimation, patient stratification, and individualizing interventions.

**Supplementary Materials:** The following supplements are available online at https://www.mdpi. com/article/10.3390/ijerph182312649/s1, Supplementary Table S1. List of ICD10 codes used to abstract comorbidity variables; Supplementary Table S2. Urin protein Conversion table; Supplementary results. Case study; Supplementary Figure S1. Graphical output of PROGRES-CKD.

**Author Contributions:** F.B. contributed to study concept, design, statistical analysis, interpretation of results, manuscript drafting, and approved the final version of the manuscript; C.L., performed literature search, contributed to interpretation of results, drafted the first version of the manuscript, and approved the final version of the manuscript; J.I.T. contributed to interpretation of results and drafted the first version of the manuscript and approved the final version of the manuscript; J.N. contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; H.M. contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; M.S. (Matthias Schmid) contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; B.B. contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; U.T. performed literature search, project conceptualization and project administration, and reviewed and approved the final version of the manuscript; M.S. (Markus Schneider), contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; U.T.S. contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; C.B. contributed to study concept, interpretation of results, and approved the final version of the manuscript; C.M. contributed to interpretation of results, and approved the final version of the manuscript; S.S. (Sonja Steppan) contributed to interpretation of results, and approved the final version of the manuscript; K.-U.E. contributed to data acquisition, interpretation of results, and reviewed and approved the final version of the manuscript; S.S. (Stefano Stuard) contributed to interpretation of results, and reviewed and approved the final version of the manuscript; L.N., contributed to study concept, design, statistical analysis, interpretation of results, manuscript drafting, and approved the final version

of the manuscript. All authors have read and agreed to the published version of the manuscript. Authors confirm that they had full access to all the data in the study and accept responsibility of submission for publication.

**Funding:** This research was funded by Fresenius Medical Care Deutschland GmbH.

**Institutional Review Board Statement:** The study was approved by the ethics committees of all participating institutions and registered in the national registry for clinical studies (DRKS 00003971).

**Informed Consent Statement:** No patients' personal information has been used for the present study since all input data for modeling were aggregated statistics.

**Data Availability Statement:** We are unable to share the raw clinical data of the FMC NephroCare because data sharing would violate the terms and conditions under which Fresenius Medical Care acquired the data. Data from the GCKD study are not publicly available. External collaborators with a specific research proposal can access deidentified participant data only after review and approval of their proposal by the steering committee.

**Acknowledgments:** The GCKD study was supported by the German Ministry of Education and Research (Bundesministerium für Bildung und Forschung, FKZ 01ER 0804, 01ER 0818, 01ER 0819, 01ER 0820, and 01ER 0821), KfH Foundation for Preventive Medicine, Innovative Medicines Initiative 2 Joint Undertaking (BEAt-DKD, grant number 115974), and corporate sponsors (www.gckd.org).

**Conflicts of Interest:** The results presented in this paper have not been published previously in whole or part, except in abstract format. L.N., J.I.T., F.B., S.S. (Sonja Steppan), S.S. (Stefano Stuard), C.M., C.B., U.T. are full time employees at Fresenius Medical Care. C.L. provided medical writing services on behalf of Fresenius Medical Care. H.M. reports grants from KfH Foundation of Preventive Medicine, and grants from German ministry of Education and Research. M.S. (Matthias Schmid) reports grants from Fresenius Medical Care during the conduct of the study. B.B. reports grants from the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung (www.bmbf.de), FKZ 01ER 0804, 01ER 0818, 01ER 0819, 01ER 0820 und 01ER 0821), and grants from Foundation for Preventive Medicine of the KfH (Kuratorium für Heimdialyse und Nierentransplantation e.V.– Stiftung Präventivmedizin; www.kfh-stiftung-praeventivmedizin.de). MSchneider reports grants from Fresenius Medical Care outside the submitted work. K.-U.E. reports grants from: Astra Zeneca, Bayer, Fresenius Medical Care, Vifor, and Amgen during the conduct of the study; personal fees from Akebia, Astellas, Astra Zeneca, Bayer, and Boehringer Ingelheim; and grants from Genzyme, Shire, and Vifor outside the submitted work. J.N. has no conflicts of interest to disclose. U.T.S. has no conflicts of interest to disclose.
