1. Introduction
Hematuria (or blood in the urine) is a common condition that often has a benign cause; however, it is one of the main symptoms of urothelial carcinoma (UC) [
1]. Although the American Urological Association guidelines recommend that patients with hematuria undergo further clinical evaluation [
2], the rates of confirmed UC in patients with hematuria are low (3.3% of those with asymptomatic microhematuria and ~17% of those with gross hematuria) [
3]. Further, among patients with hematuria referred for further evaluation, ~18% may have atypical cytology or equivocal cystoscopy [
4], thereby returning inconclusive results that require further invasive procedures (e.g., biopsy). In addition, as the recurrence rates for non-muscle-invasive bladder cancer are high, with 38–61% of patients having tumor recurrence after transurethral resection [
5], ongoing guideline-directed surveillance is recommended in patients with a history of UC [
6].
Cystoscopy is the gold standard for UC diagnosis [
2,
6], but both white- and blue-light cystoscopy give variable results and sensitivities of less than 100% (73–86% and 95–96%, respectively) [
7], with false-positive rates of 12–45% and 15–37%, respectively [
8]. Historically, the main non-invasive methods used for evaluating patients for UC (either initially or for recurrence) were urine cytology (which has high specificity) and fluorescence in situ hybridization; however, both of these methods have poor sensitivity [
9,
10]. This leads to high false-negative rates, meaning that neither method can safely rule out UC. Therefore, there is a need for accurate, non-invasive urinary biomarker tests that can rule out patients with hematuria who do not have UC and identify those with the highest risk of UC, including those at risk of recurrent tumors.
The Cxbladder
® assays are reverse transcription quantitative-PCR (RT-qPCR)-based diagnostic urinary biomarker tests that can be used for (a) risk stratification of patients presenting for evaluation of hematuria (Cxbladder Detect and Cxbladder Triage) and (b) the surveillance of patients with previously treated UC (Cxbladder Monitor) [
11,
12,
13]. Cxbladder assays quantify the messenger RNA (mRNA) expression of five biomarkers, including four genes that are known to be correlated with UC (cyclin-dependent kinase 1 [
CDK1], midkine [
MDK], insulin-like growth factor binding protein 5 [
IGFBP5], and Homeobox A13 [
HOXA13]) [
14], plus a known marker of inflammation (C-X-C motif chemokine receptor 2 [
CXCR2]) [
11]. The
CXCR2 biomarker, which is highly expressed in neutrophils, is included in the Cxbladder assays to reduce the noise from background inflammation to help discriminate between individuals with nonmalignant disease and those with UC [
11]. In previous studies in patients with microhematuria or gross hematuria, Cxbladder Detect demonstrated moderate sensitivity and high specificity to enable risk stratification for the presence or absence of UC [
4,
15,
16], and Cxbladder Triage had high sensitivity and a high negative predictive value (NPV), providing a strong value proposition as a rule-out test (test-negative tool) [
4,
15,
16,
17]. These findings indicate that Cxbladder Detect and Triage may provide a more robust non-invasive method for detection of UC than urine cytology in patients with hematuria. In real-world clinical practice, the Cxbladder Detect assay can be used as the first step for evaluating patients with microhematuria to exclude those with negative results from further evaluation; urine cytology could then be conducted in patients with positive results to further improve the positive predictive value of the non-invasive evaluation for those patients. In patients undergoing surveillance for recurrent UC, Cxbladder Monitor also had high sensitivity and NPV [
4], thereby allowing the physician to safely rule out recurrent UC in patients who may benefit from a reduced intensity of invasive procedures.
For biomarker tests to be incorporated into routine clinical use, they must demonstrate strong analytical validity, clinical validity, and clinical utility [
18]. Prior to clinical validation, analytical validation studies of the Cxbladder assays were conducted to ensure that the analytical component of the assays were well controlled and provided reliable results with high precision and accuracy. The fundamental aspects of the assays (sample and reagent stability, RNA extraction quality, linearity of RT-qPCR, analytical sensitivity and specificity, and pre-analytical and analytical measurement of the five biomarker mRNA concentrations) are identical for all three assays and were assessed using Cxbladder Detect. The accuracy, precision, and reproducibility were then confirmed for each Cxbladder assay (Detect, Triage, and Monitor). Here, we report the results of this analytical validation study.
2. Materials and Methods
2.1. Sample Collection
For clinical use of Cxbladder assays, voided mid-stream urine samples (4.5 mL) are required. The analytical validation analysis used mid-stream urine samples collected from consenting healthy volunteers. For the clinical validation analysis of Cxbladder described here, voided mid-stream urine samples were collected from consenting patients presenting with hematuria (for Cxbladder Detect or Triage) or those undergoing surveillance for recurrence of UC (for Cxbladder Monitor). Clinical trial sample acquisition has been described in full previously for Cxbladder Detect [
11], Triage [
12], and Monitor [
13].
All clinical trials were registered on
ClinicalTrials.gov or the Australia New Zealand Clinical Trials Registry (ANZCTR) and adhered to the ethical principles of the Declaration of Helsinki of 2013. All sample collection was performed with written informed consent obtained from the participants of the study and was performed under local ethical approvals.
2.2. Sample and Reagent Stability
Urine samples were aspirated into a Cxbladder Urine Sampling System (USS) that contained Cxbladder stabilizing reagent (4.5 mL; a proprietary nucleic acid-stabilizing solution) and transported from the clinic to the Pacific Edge Diagnostics laboratories (Dunedin, New Zealand [NZ] or Hummelstown, PA, USA). To assess the stability of the sample and stabilizing reagent during transportation, 62 high- and low-extraction controls (HEC and LEC) with RNA-equivalent concentrations were added to Cxbladder stabilizing reagent, aliquoted, and stored at 2–8 °C, 18–22 °C (laboratory controlled ambient temperature), and 30 °C. To establish the baseline timepoint (T0), eight replicates were stored at each temperature for 2 h prior to being processed through the RNA extraction procedure described below. Aliquots stored at 2–8 °C, 18–22 °C, and 30 °C were assessed on Days 2, 5, 7, 9, 12, and 14 (in triplicate) and compared against the T0 samples.
The freeze/thaw stability of the Cxbladder stabilizing reagent was assessed by comparing two aliquots of reagents: aliquots that were frozen (at −80 °C) and thawed five times versus non-frozen aliquots.
The freeze/thaw stability of the RNA was assessed using reference RNA. The reference RNA was a plasmid-derived RNA transcript comprising the five biomarker targets measured in all Cxbladder assays (described in detail below). Four aliquots of RNA were stored at −80 °C (optimal condition) and an additional four aliquots underwent three −80 °C freeze/thaw cycles (a >12-h freeze step and a 1-h thaw step). Each reference RNA aliquot was then diluted to create a four-point standard curve in an RT-qPCR plate, with each serially diluted point plated in triplicate. Assessment of the sample stability of the extractions was performed using previously published methods [
14].
2.3. RNA Extraction Quality
Analytical validation of RNA extractions was conducted using an automated, silica matrix-coated, magnetic bead-based system. Briefly, 2.9 mL aliquots of stabilized sample were added to 1.1 mL of absolute ethanol containing 25 µL of MagMAX beads (Thermo Fisher Scientific Inc., Waltham, MA, USA), 3.625 µg of poly-A carrier RNA (Roche, Basel, Switzerland) and 17,260 copies of an exogenous control RNA (Cells-to-CTTM Control Kit; Thermo Fisher Scientific Inc., Waltham, MA, USA). Samples were extracted on a KingFisherTM Apex instrument (Thermo Fisher Scientific Inc., Waltham, MA, USA). The beads were washed twice with MagMAX cell-free DNA wash solution, and once with 80% ethanol. The washed beads were dried before elution in 64 µL of ribonuclease (RNase)-free water.
Quantification of RNA extracted from urine samples was not performed by spectrophotometry because the extraction process involves the addition of two synthetic RNA molecules to each extraction, thereby making spectrophotometric results inaccurate. These two RNA molecules are poly-A RNA, which acts as a carrier molecule to increase the efficiency of extraction for low RNA concentrations, and an internal control (IC) RNA, which is used to evaluate and control for extraction quality and RT-qPCR efficiency. Instead, quantification of RNA was performed using RT-qPCR, and extraction quality was determined using the spiked IC RNA.
2.4. Assay Controls
Each Cxbladder assay involved the detection of five target genes and the control gene, with the assay run in duplex (three duplex assays per Cxbladder assay). Each plate included the following controls with each duplex RT-qPCR: (1) no template control in triplicate for each duplex reaction (RNase-free water; a negative control for contamination); (2) three control samples containing either a high or low concentration of the five genes of interest, or no RNA for the genes of interest (HEC, LEC, and negative extraction control [NEC];
Table S1); and (3) an individual sample quality control (QC), where each sample was spiked with IC RNA, a synthetic RNA transcript containing a unique sequence that lacks homology to known biological sequences prior to the extraction step control. If the RNA concentrations for the HEC or LEC samples fell outside the QC ranges (below or above the expected lower and upper limits, respectively), or the NEC sample showed contamination levels equal to the limit of quantification (LOQ), or the IC RNA concentrations were outside the expected range (log
101.55–2.90 copies/μL extracted RNA), then the PCR plate failed the QC check.
2.5. Reverse Transcription-Quantitative PCR
The Cxbladder assays used RT-qPCR to assess the mRNA expression of five biomarker genes (CDK1, MDK, IGFBP5, HOXA13, and CXCR2). Each assay was run as three duplex reactions (two genes per well: well 1 = MDK + IGFBP5; well 2 = CDK1 + HOXA13; well 3 = CXCR2 + IC RNA), with each duplex reaction performed in duplicate. Fluorescence color compensation was conducted using the LightCyclerTM 480-II (Roche, Basel, Switzerland) to account for reporter dye interference.
Individual biomarker raw data were initially defined as a quantitative cycle (Cq) value (the cycle of PCR where the fluorescence signal significantly exceeds background fluorescence noise). The standard curve of reference RNA was used to convert the Cq value for each biomarker to an mRNA concentration in copies/μL. The parameters for the standard curve of the reference RNA and the confirmation of the RNA concentrations of the known RT-qPCR controls (using the standard curve) were required to be within the defined quality control parameters (as shown in
Table S2) or the PCR plate failed QC.
2.6. Cxbladder Score Calculation
The Cxbladder Detect assay used a logistic regression classifier, based on the fluorescence data from the RT-qPCR analysis, to generate a score (scale 0.00–1.00) that indicates the probability of the patient having UC. The reportable range was determined during creation of the classifier scores, based on a training set population of 317 patients (274 were UC-free according to cystoscopy) [
11]. Cxbladder Detect scores of ≥0 to <0.12 are categorized as normal gene expression (low risk of UC), scores of ≥0.12 to <0.23 as elevated gene expression (intermediate risk of UC), and scores of ≥0.23 to ≤1.0 as high gene expression (high risk of UC). Standard thresholds were applied to the classifier score to assist the clinician in making a decision about the likelihood of cancer based on the test population data.
The Cxbladder Triage algorithm included four clinical variables (age, sex, hematuria status, and smoking history; classified as the P-index) [
12], as well as the five biomarker mRNA concentrations (G-index). The creation of the classifier score range (scale 0.0–10.0) was based on samples from 587 patients, as previously described [
12]. The Cxbladder Triage score categorizes patients with hematuria as either “negative” (score < 4.0; low risk of UC) or “further investigation”/“not negative” (score ≥ 4.0; standard clinical workup is warranted).
The Cxbladder Monitor algorithm additionally included the status of the patient’s previous tumor (primary diagnosis or recurrent tumor) and the time since the previous tumor (in years) in addition to the five biomarker mRNA concentrations to provide the test score (scale 0.0–10.0). The Cxbladder Monitor score range was defined using a development set of 763 patients, as previously described [
13]. Cxbladder Monitor provides a binary outcome for patients with previously diagnosed UC, whereby patients with a score of <3.5 are categorized as “rule out” (low risk of recurrent UC) and samples with a score of ≥3.5 are categorized as “further investigation” (high risk of recurrent UC).
2.7. Linearity
To assess the linearity of the individual biomarker mRNA concentrations used to generate the Cxbladder score, 13 RNA samples of different known concentrations that covered the expected range in samples (from beyond the LOQ to the limit of detection [LOD]) were analyzed to determine their measured concentrations using the reference RNA standard curve. Each standard curve was run in triplicate over two plates and the measured concentrations were plotted against the known concentrations.
Simple linear regression models were fitted for each gene. The null hypothesis was that the linear regression was valid for the whole diluted range. The null hypothesis was rejected if the R-squared value was <0.99. Linear and quadratic models were used to test for curvature in the reported range.
2.8. Limit of Quantification
Using the linearity assay above, LOQ was defined as 2 × the lowest linear concentration in attograms (ag; 10−18 g)/μL (expressed as RNA copies/mL urine, where one copy equated to 0.44456 ag) on the reference RNA standard curve for all five biomarkers (10.71 ag/μL, 24 copies/μL [log10 1.385 copies/μL]), and was chosen to ensure near linearity of the standard curve.
2.9. Analytical Sensitivity
Analytical sensitivity (or LOD) was defined as the lowest RNA concentration that could be used to detect the biomarkers in 95% of assays performed. The numbers of total and amplified wells were counted for each gene. Assuming the number of wells had Poisson distribution (with rate = lambda [λ] × concentration), then the LOD was −log(0.05)/λ, and the maximum likelihood estimator for λ could be computed. The reference RNA was diluted to 3.2, 0.8, and 0.2 ag/µL (corresponding to 7.20, 1.80, and 0.45 RNA copies/mL, respectively). Reference RNA at 3.2 ag/μL was added to 24 wells, while RNA at 0.8 and 0.2 ag/µL was added to 28 wells each per duplex RT-qPCR, totaling 80 wells that spanned the LOD range. Based on the number of wells with “no signal”, the mRNA concentration could be computed, and the LOD corresponded to a 77.6% probability of detection in each of two replicate wells (i.e., [1 − 0.776]2 = 0.05).
2.10. Analytical Specificity
To determine the analytical specificity of the extracted RNA from biological interference, high-concentration samples were mixed individually with selected amounts of the following potentially interfering substances: red blood cells (RBC; 1 × 104–1 × 109 cells/mL); protein (0.01–10.00 mg/mL bovine serum albumin [BSA]); glucose (0.50 and 0.25 mg/mL); urea salts (75.0 and 37.5 mg/mL); bacteria (1000, 5000, and 10,000 cells/mL Escherichia coli); and yeast (10,000, 50,000, and 100,000 colony-forming units [cfu]/mL). Samples within the expected level of gene variance and where the Cxbladder score was within the 95% confidence interval (CI) for the control were considered acceptable.
To determine the analytical specificity of the RT-qPCR from process-related interference, the following potentially interfering substances were evaluated: absolute ethanol (0.230–3.750%); Cxbladder stabilizing reagent (0.012–3.000%); acetone (0.234–3.750%); MagMAX wash buffer (0.230–3.750%); and MagMAX bead solution (1.56–25.00%). For each potentially interfering substance, the starting concentration of inhibitor was combined, as a proportion of the final RT-qPCR volume, with reference RNA included at high and low concentrations.
2.11. Analytical Accuracy
The accuracy of the Cxbladder assays was determined by assessing the differences in the measured RNA concentrations in the controls versus the known concentrations using the extraction efficiency determined by the spiked extraction IC RNA. Data were collected from 20 RT-qPCRs run over ≥20 working days, consisting of 165 HEC and 65 LEC replicates.
To assess accuracy, a 1013 base pair (bp) reference RNA transcript, containing ~200 bp of each target sequence, was synthetically constructed and purified before its concentration was determined according to the Beer–Lambert law and spectrophotometry peak absorbance at 260 nm. From this concentration and the known nucleotide sequence, the actual transcript copy number was calculated utilizing the known molecular weight of each nucleotide. This copy number was used to determine the expected concentration, and the purified reference RNA was added to Cxbladder stabilizing reagent to create the HEC and LEC samples. A bias between the measured and known concentrations was expected, due to the RNA extraction process being unable to achieve 100% extraction efficiency. Acceptable inaccuracy was <15% for the linear region of the assay (>LOQ) and <20% for RNA concentrations that were <2 × LOQ.
The diagnostic sensitivity, specificity, positive predictive value (PPV), and NPV were calculated for Cxbladder Detect, Triage, and Monitor by comparison of clinical sample results against the cystoscopy-based diagnosis (see
Table S3 for equations). For the accuracy validation analyses, 485 clinical samples were used for Cxbladder Detect (66 [13.6%] positive and 419 [86%] negative for UC by cystoscopy) [
11], 587 samples for Triage (73 [12%] positive, 514 [88%] negative) [
12], and 1036 samples for Monitor (156 [15%] positive, 880 [85%] negative) [
13].
2.12. Analytical Precision
For the precision analysis, intra-assay and inter-assay variability of the genes in the Cxbladder Detect, Triage, and Monitor assays were assessed. For all genes, the acceptance criterion for precision was to have a total co-efficient of variation (CV%) of <10%.
Inter-assay variability was determined from 165 HECs and 65 LECs run over 20 plates. Intra-assay variability was determined by testing at least two HECs and at least two LECs run over ≥20 days by more than three operators. The data consisted of HEC samples from 14 plates of two replicates, >21 HEC samples over six plates, 19 plates containing two LEC samples, and one plate containing 23 LEC samples. Linear random effects models were fitted to estimate the intra-assay and inter-assay variability using R package LME4. Precision was calculated in the form of standard deviation (SD), and the bootstrapped SD was used to obtain the 95% CI of the estimation.
Lot-to-lot extraction reagent variation was evaluated by testing two sets of unqualified reagents against qualified reagents, using HEC and LEC samples in triplicate for each set. RT-qPCR reagent lot-to-lot variation was determined by testing two sets of unqualified versus existing qualified RT-qPCR reagents, using standard curves and RNA controls (1000, 120, and 30 ag RNA, giving Cxbladder Detect scores of 0.890, 0.332, and 0.055, respectively). Lot-to-lot variation was considered acceptable if the score variation was within the limits identified for an individual PCR plate.
The precision of the Cxbladder assays was further assessed by determining how well the assays performed in two laboratories (Pacific Edge Diagnostics, USA, Ltd. [PEDUSA], Hummelstown, PA, USA and Pacific Edge Diagnostics NZ [PEDNZ], Dunedin, NZ). Total variability was assessed at the gene concentration using HEC and LEC samples, and information was compared between laboratories (acceptable total CV% ≤ 10%).
2.13. Reproducibility
The inter-laboratory comparison between the NZ laboratory (PEDNZ) and the US laboratory (PEDUSA) was assessed for each Cxbladder assay. A random sample set from the PEDNZ validation was used to confirm the reproducibility of Cxbladder Detect (n = 153), Triage (n = 81), and Monitor (n = 41) at PEDUSA. Acceptable variability was defined as achieving ≥ 80% concordance for all clinical sample results.
2.14. Sample Stability
A mixed effects analysis of variance was used to assess trends over time for HEC and LEC RNA reference samples stored at 4 °C (2–8 °C), ambient room temperature (RT = 18–22 °C), or 30 °C. Statistical comparisons of sample scores were conducted using the t-test, where p-values of <0.05 were considered significant.
2.15. Statistical Analysis
Statistical analyses were conducted using R, version 4.3.1.
4. Discussion
This analytical validation study demonstrated that the Cxbladder assays can accurately and reproducibly quantify the mRNA expression of five biomarker genes, with an analytical sensitivity or LOD of 12.5–31.1 RNA copies/mL urine for the four biomarkers correlated with UC (CDK1, MDK, IGFBP5, and HOXA13), and 68.9 copies/μL for the inflammatory biomarker CXCR2. All pre-specified analytical criteria, including linearity, analytical sensitivity, specificity, accuracy, and precision, were met.
The Cxbladder assays are multi-gene RT-qPCR assays that are designed to aid clinicians in the risk stratification of patients presenting with hematuria and those with previously treated UC who are undergoing surveillance. Cxbladder Detect is intended to provide moderate sensitivity and high specificity when screening for the presence or absence of UC in patients undergoing evaluation of hematuria, while Cxbladder Triage and Monitor are designed for high sensitivity and NPV to safely rule out UC in patients presenting with hematuria or a history of UC, respectively.
The accuracy of RNA concentration measurement showed an accuracy of >89% for all genes in both HEC and LEC samples (i.e., high and low RNA concentrations), with inaccuracy ranging from −2.23% to +10.63%.
The clinical accuracy of each Cxbladder assay was validated by comparison against the known cystoscopy-based diagnosis [
11,
12,
13]. Cxbladder Detect scores for high (≥0.23) and elevated (≥0.12 to <0.23) gene expression samples had a sensitivity of 77% and 82%, respectively, as well as high specificity (94% and 83%) and NPVs (96% for both). Cxbladder Triage showed high sensitivity (95%) and NPV (98%), but lower specificity (46%). The sensitivity of Cxbladder Triage is higher than that of Detect due to the incorporation of four clinical risk factors into the algorithm that calculates the Triage test score [
12]. These accuracy findings are consistent with those reported in a 2023 clinical validation study, in which the sensitivity, NPV, and specificity were 74%, 97%, and 82%, respectively, for Cxbladder Detect, and 89%, 99%, and 63%, respectively, for Cxbladder Triage [
15]. In a 2021 systematic review of novel urinary biomarker tests, Cxbladder Detect and Triage had a pooled sensitivity of 97% for initial diagnosis of high-grade tumors [
19]. In the current analysis, Cxbladder Monitor demonstrated high sensitivity (91%) and NPV (96%), albeit with low specificity (39%), similar to the internal validation of Monitor, in which the assay had a sensitivity, NPV, and specificity of 93%, 97%, and 34%, respectively [
13]. The sensitivity and specificity of Cxbladder Monitor in the current study were also in line with the pooled sensitivity and NPV reported in the 2021 systematic review (91% and 98%, respectively) [
19].
The analytical validity of the Cxbladder assays was confirmed at a second laboratory (PEDUSA), with maximum intra- and inter-assay variability of 4.66% and 6.85%, respectively, and a total maximum CV% of 8.29%. Concordance between the two laboratories (PEDNZ and PEDUSA) remained high (>85%) for all three assays based on the 95% CIs of the output results.
One of the limitations of this study was the use of RNA reference-spiked normal samples (rather than clinical samples) for the stability, linearity, analytical sensitivity, and specificity analyses; however, this limitation was mitigated by the use of blinded clinical samples for the validation of the clinical accuracy of each Cxbladder assay. Another limitation was that the Cxbladder assays did not include detection of other known genetic markers of UC, such as fibroblast growth factor receptor 3 (
FGFR3) and telomerase reverse transcriptase (
TERT) [
20]. Ongoing studies have since led to the development of enhanced Cxbladder Detect and Triage assays [
15], in which six DNA single nucleotide polymorphisms from
FGFR3 and
TERT have been integrated to further improve risk stratification in patients with hematuria. Analytical and clinical validation of the enhanced Cxbladder Detect assay is currently ongoing, but a previous study found that this assay had a sensitivity, specificity, and NPV of 97%, 90%, and 99.7%, respectively [
15]. Lastly, our study may have also been limited by its analytical validation design, meaning that it did not provide clinical validation or describe the clinical utility for these assays. However, previous studies have demonstrated the clinical validity of Cxbladder Detect, Triage, and Monitor [
11,
12,
13], and the clinical utility of Cxbladder Triage and Monitor [
21,
22].