*2.3. Assessment of Canine Performance*

To measure the performance of canines individually and as a group, a binary classification test was performed on the data from each of the studies below and standard diagnostic accuracy statistics (latent-class metrics) were calculated:

*True Positive* (TP) correct canine alert on Xcc-positive target *True Negative* (TN) correct rejection, no canine alert on Xcc-negative target *False Positive* (FP) incorrect canine alert on Xcc-negative target, Type I error *False Negative* (FN) incorrect canine rejection of Xcc-positive target, Type II error *Sensitivity* (SEN) or true positive rate,

$$\text{SEN} = TP \% TP + FN \% \%$$

*Specificity* (SPE) or True Negative Rate

$$SPE = TN\%FP + TN\%$$

*Precision or Positive Predictive Value* (PPV)

$$PPV = TP \beta (TP + FP)$$

*Negative Predictive Value* (NPV)

*NPV* = *TN*/*(TN* + *FN)*

*False Positive Rate* (FPR)

*FPR* = *FP*/*(FP* + *TN)*


where *n* = total number of samples assessed in each trial.

The above metrics are commonly used to evaluate the performance of diagnostic tests in medicine and other endeavors, including the evaluation of canine detectors [26,29–32]. No single performance metric can capture all aspects of canine detection accuracy. Throughout this study, we report the metrics listed above, with a particular focus on sensitivity, specificity, and accuracy. In addition to those standard metrics for test evaluation, we also examined the diagnostic capability of the canines using three graphical approaches that either depend directly on information quantities or which have direct connections to information theoretic concepts.

To illustrate the use of these methods, we focus on a single set of performance statistics derived from assessment of the impact of disease prevalence in samples on the detection performance of the canines (see Section 3.5 below). Using the summary statistics for the overall performance for canines Bady and Maxi in this set of experiments, we constructed likelihood ratio graphs [33,34], a leaf plot based on the PPV and NPV for each canine, and a "loop" plot illustrating the expected mutual information for the positive and negative diagnoses for each animal [34].

#### *2.4. Detection of Xcc in Simulated New Plantings*

Ruby Red Grapefruit seedlings were inoculated using a needless syringe tightly appressed to the abaxial laminar surface of 1/2 to 2/3 expanded leaves and forcing inoculum (≈10<sup>5</sup> cfu/mL) into the lamina via pressure injection infiltration; then, they were allowed to develop citrus canker symptoms for 4 weeks prior to canine interrogation [35]. Non-inoculated trees were arrayed in a 100 tree, 10 × 10 grid, with ≈3 m between trees within row and between rows, and with 1 to 10 Xcc-infected trees randomly placed within the grid. Canines interrogated each tree in the grid in sequence via a serpentine pattern up and down the rows. Three detector canines associated with two collaborating commercial canine training companies assessed the randomized arrays of grapefruit trees.

For the first collaborating company, a single canine, a Labrador retriever "Juice", interrogated the simulated orchard with 2, 4, 5, 7, or 10 Xcc-infected trees randomly placed within the 100-tree grid. Each of the incidence levels was replicated ten times. Tree placement was re-randomized between each replication. The experiment was repeated during four separate months, July, September, December, and May to examine the effect of seasonality. Due to commercial handler availability, not all incidence levels were examined each month (Figure 2A–C). For the second collaborating company, two canines, German shepherds, "Bady" and "Maxi", each interrogated the simulated orchard with 1 to 6 Xcc-infected trees randomly placed within the 100-tree grid (Figure 2D,E). Tree placement was re-randomized between each replication, and care was taken to ensure that only technicians setting up each grid replicate knew the positions of Xcc-infected and non-infected trees, i.e., both handlers and canines were unaware of true positive and true negative target positions. This same "blind" test methodology was implemented through all subsequent trials. However, when a canine alerted correctly on a true positive, the technician confirmed the correct detection to the handler so the handler could appropriately reward the canine. The experiment was conducted over a two-month duration, depending upon commercial handler availability. Each canine interrogated a grid of each Xcc-incidence level at least twice. Canine alerts were recorded, and latent class metrics were calculated to assess canine performance.

**Figure 2.** Canine detection of *Xanthomonas citri* pv. *citri* (Xcc) in simulated and commercial orchards. (**A**) Xcc-infected, potted red grapefruit inserted into ground—Xcc lesions indicated by red arrows. Detector canine "Juice"—(**B**) interrogating, and (**C**) alerting on Xcc-infected trees. Detector canine "Bady"—(**D**) interrogating, and (**E**) alerting on Xcc-infected trees. (**F**) "Juice" alerting on Xcc-infected grapefruit tree in commercial orchard. (**G**) Sample of three Xcc-infected grapefruit leaves from commercial orchard identified by "Juice"—note multiple small brown Xcc lesions surrounded by chlorotic halos.

#### *2.5. Detection of Xcc Lesions of Various Ages*

Ruby Red Grapefruit seedlings were inoculated using the methods described in Section 2.4 above. Inoculations were conducted over time, such that on the day of canine interrogation, a temporal array of Xcc-infected seedlings were 1, 3, 6, 9, and 12 weeks post-inoculation. Two canines, Bady and Maxi, each assessed a 50-tree grid (5 rows of 10 seedlings per row) five times (replications). For each replication, two seedlings with Xcc infections of the same age were randomly placed within

a population of 48 additional non-infected seedlings of the same age. The experiment was repeated 7 days later using the same temporal array of seedlings, whose Xcc infections were now 1 week older. All tests were blinded as described in Section 2.4 above. Since there was no significant difference between the repeated experimental results, performance metrics were calculated for the combined data such that Xcc-infection age ranged 1–2, 3–4, 6–7, 9–10, and 12–13 weeks post-infection.

## *2.6. The E*ff*ect of Incidence of Xcc Lesions on Detection*

Early observations with the canine "Juice" indicated the potential that he alerted differentially to trees infected with few versus prevalent lesions. Subsequently, canines were trained to alert to Xcc infections regardless of infection prevalence. To assess the effect of lesion incidence, Duncan grapefruit seedling trees were inoculated by pin-prick inoculation as described above to establish trees with 1, 5, 50 and 500 lesions each. Two canines, Bady and Maxi, each interrogated grids of 50 trees with one Xcc-infected tree of each lesion incidence and 46 non-infected trees. Each Xcc incidence level was interrogated twice by each canine with the location of the Xcc-infected trees re-randomized (using a random number generator) between replicates. All tests were blinded as described in Section 2.4 above. The experiment was repeated two months later with the same set of trees. Canine alerts were recorded, and latent class metrics were calculated to assess canine performance. For each lesion incidence, performance metrics were calculated considering one infected tree of the specific lesion incidence being evaluated in a population of 46 non-infected trees, ignoring the canine response of the other three Xcc-infected trees of other lesion incidence.

## *2.7. Detection of Xcc Infections in Decaying Foliage*

Barring adverse environmental conditions or disease, the lifespan of citrus leaves is 1–3 years prior to abscission, after which they senesce and decay. However, Xcc-infected leaves experience foliar accumulation of elevated ethylene and often abscise early. To determine the duration of canine detection of Xcc-infections in abscised leaves, Duncan grapefruit leaves, *in planta*, were infected via a needless syringe inoculation described above. Non-infected leaves were physically abscised from the trees, and 20–30 leaves were placed in wire cages as non-infected controls. True positive targets were composed of cages with 20–30 non-infected leaves, with the addition of 2–6 leaves with 30-day Xcc infections. True positive (TP) and true negative (TN) cages were randomized in an open grassy field (Figure 3A,B). Two canines, Bady and Maxi, interrogated the leaf cages on 0, 1, 2, 5, 13, and 27 days post-abscission as the leaves decayed. Each canine interrogated the leaf cages in turn and then repeated the interrogation for two to five replications on each assay date; the number of replications/day/canine depended upon weather and handler availability (Figure 3C,D). All tests were blinded as described in Section 2.4 above. Canine alerts were recorded, data were combined across replications and canines, and latent class metrics were calculated to assess canine performance.

## *2.8. Detection of Xcc in Citrus Packinghouse Environments*

To examine canine performance for the detection of Xcc in a commercial packinghouse, commercially packed cardboard boxes, each containing approximately 50 red grapefruit fruits were arrayed in a 10 × 10 grid, with ≈1.5 m between rows and boxes within a row (Figure 3E,F). Within target boxes, 2 Xcc-infected fruits, each with 2–20 lesions were randomly placed in the center within each TP target carton. The target boxes containing Xcc-positive fruit were randomly arrayed on the concrete packinghouse floor and re-randomized between replications. A single canine, "Juice", interrogated each box in the grid in sequence via a serpentine search pattern up and down the rows. The experiment was repeated twice 5 days apart. On the first day, the canine interrogated nine randomized arrays (900 cartons) with Xcc-incidence ranging from 1–4%, over a ≈3-h period, whereupon the study was halted due to excessive temperatures in an effort to preserve canine health. On the second day, the ambient conditions were more favorable, and the canine interrogated 19 randomized arrays (1900 cartons) with Xcc-incidence ranging from 1–10% over a ≈5-h period. All tests were

blinded as described in Section 2.4 above. Human inefficiency of detection of packing-house inspected fruit was indicated by 100 commercial cartons selected at random post packinghouse processing, 2 of which contained 1 and 2 infected fruit with small lesions after passing multiple packing line inspections stations with trained fruit grader/inspectors and post packinghouse inspection by pathology technicians trained to detect Xcc symptoms. Canine detection of unknown true positives elucidated the human error, i.e., false negatives. These true positives were subsequently added into the grid data and considered true positives incorporated into the grid designs. Canine alerts were recorded, the results from the two days were combined, and latent class metrics were calculated to assess canine performance.

**Figure 3.** Canine detection of abscised Xcc-infected grapefruit leaves over time. (**A**) Mixed Xcc-infected and non-infected leaves in wire mesh cage, (**B**) close up of leaves in wire mesh decaying. Canine "Bady"—(**C**) interrogating, and (**D**) alerting on wire mesh cages with decaying Xcc-infected leaves. (**E**) Commercially packed grapefruit in cardboard carton with top layer of fruit removed to show Xcc-infected fruits—red arrows indicate infected fruit with Xcc lesions. (**F**) Grid of 100 cartons of commercial packed red grapefruit arrayed on packinghouse floor for canine interrogation; 1–6 cartons contain Xcc-infected fruits—positions of Xcc-infected cartons randomized between trials.

#### *2.9. Assessment of Xcc Detection in Commercial Citrus Orchards*

One canine, "Juice", surveyed two commercial red grapefruit orchard blocks in Indian River County, Florida with endemic low incidence Xcc infection (Figure 2F,G). Prior to canine assessment, the blocks were surveyed visually by human assessors to determine and map the location of Xcc-infected trees. Two assessors independently examined trees visually requiring ≈5 min/tree. If a tree had unusual symptoms or was difficult to assess, one to two additional assessors examined it as well. Subsequent to canine assessment, trees on which canines alerted were visually reassessed in an attempt to determine if human assessors could confirm the canine detections. However due to the previously documented inefficiency of human visual assessment ([11,36]; T. Riley, unpublished results), it is likely that many infected trees were missed and/or could not be confirmed. Canine alerts were recorded,

and the results and latent class metrics were calculated to assess canine performance against putative human visual assessment.

#### *2.10. Spatial Heterogeneity of Xcc Detection Errors*

It is not uncommon for detector canines to acquire a target scent at some distance from the true target, occasionally alerting on a negative target within the scent plume [37,38]. To address the concern of FN and FP canine alerts on grids of Xcc-infected and non-infected targets, respectively, we analyzed the cumulative randomized placement of Xcc-infected trees and compared it to correct Xcc true positive (TP) and true negative (TN) tree positions, calculating the distances between TP and FP and locations. We conducted this spatial assessment for (1) disease incidence of lesions in simulated orchards, (2) lesion age in simulated orchards, and (3) incidence of cartons with Xcc-infected fruit in the packinghouse.

#### *2.11. Direct Detection of Xcc Bacteria*

Initially, it was assumed that canines trained to detect Xcc-infected trees were alerting to a complex scent signature composed of VOCs from the bacteria plus unique plant-based VOCs produced in response to Xcc infection. We questioned if trained canines would alert directly on VOCs from Xcc bacteria without the background citrus host odor and/or unique VOCs produced by the bacteria/host interaction. To answer this question, we grew Xcc in axenic culture on nutrient agar for 1 week, harvested the bacteria, suspended them in sterile phosphate-buffered saline (PBS; 0.14 M NaCl, 1.5 mM KH2PO4, 6.5 mM Na2HPO4, 2.6 mM KCl (pH7.4)), and adjusted the suspension spectrophotometrically by diluting with sterile PBS to approximately 106 cfu/mL. Subsequently, 400 μL of the suspension (containing <sup>≈</sup><sup>2</sup> <sup>×</sup> <sup>10</sup><sup>5</sup> cfu/mL) or 400 <sup>μ</sup>L of sterile PBS was pipetted onto sterile cotton filter discs.

Prior to this experiment, one canine (Mi) was trained exclusively to detect Xcc *in planta* from Xcc-infected plants and a second canine (Ti) was trained to detect Xcc in vitro from axenic culture. Both canines interrogated a row of 10 metal paint cans ≈2 m apart. Into one can, a sterile cotton filter disc infused with 400 μL of Xcc dilution in PBS was placed, and into the nine remaining cans, a sterile cotton filter disc infused with 400 μL sterile PBS was placed. The canines repeatedly interrogated the line of 10 cans 10 times, the cans were re-randomized between each replication, and the study was repeated once. All tests were blinded as described in Section 2.4 above. Canine alerts were recorded, and latent class metrics were calculated to assess canine performance.

The reciprocal experiment was also conducted, wherein we questioned if canines trained exclusively on cultured bacteria could detect the bacteria *in planta*, i.e., in Xcc-infected plants. Both canines repeatedly interrogated a line of 10 trees. The line was composed of one Xcc-infected and 9 non-infected Duncan grapefruit seedlings. The plants were re-randomized between each replication, and the study was repeated once. Canine alerts were recorded and latent class metrics were calculated to assess canine performance.

#### *2.12. Estimation of Bacterial Detection Threshold*

Having determined that canines trained on either Xcc-infected plants or on Xcc cultures were able to detect the bacteria directly in vitro as well as *in planta*, we wanted to determine the sensitivity, i.e., lower limit of bacteria needed for canine detection. To answer this question, we grew Xcc in axenic culture on nutrient agar for 1 week, harvested the bacteria, suspended them in sterile PBS, and adjusted the suspension spectrophotometrically by diluting with sterile PBS to approximately 104 cfu/mL as described above. Then, a dilution series (4 <sup>×</sup> <sup>10</sup>4, 4 <sup>×</sup> 102, 0 <sup>×</sup> <sup>10</sup>0, 0 <sup>×</sup> <sup>10</sup><sup>−</sup>1, and 0 <sup>×</sup> <sup>10</sup><sup>−</sup>2) was prepared using sterile PBS as the diluent, and 400 μL of each bacterial dilution was pipetted onto individual sterile cotton filter discs. Immediately following the canine detection trials (described below), the individual dilutions were plated on nutrient agar, incubated for 2 days, enumerated, and the results were used to calculate the number of bacteria on the cotton filter discs (approximately 26.4, 3.60, 0.27, 0, or 0 cfu) as presented to the canines at the time of the test.

The canine (Mi) previously trained to detect Xcc-infected plants and the canine (Ti) trained to detect Xcc from culture, each interrogated rows of 10 paint cans ≈2 m apart. A sterile cotton filter disc infused with a specified dilution of Xcc in PBS was placed into one can, and into the nine remaining cans a sterile cotton filter disc infused with PBS was placed. All dilutions were interrogated by both canines and the performance was assessed. A known TP consisting of a filter pad with 400 μL of <sup>≈</sup><sup>4</sup> <sup>×</sup> <sup>10</sup><sup>2</sup> <sup>=</sup> 25.2 cfu Xcc culture was randomly placed in each line of cans as a TP control. Additionally, a TN control was assessed consisting of a single location of *Bacillus megaterium* to ensure that the canines were performing as expected. The original dilution was determined by subsequent culture to be 143 cfu/mL with ≈57 cfu/400 μL pipetted onto the sterile cotton filter disc.

To examine if canines were able to detect Xcc subcellular components, Xcc bacteria were collected from petri plate culture, suspended in PBS, and adjusted spectrophotmetrically to <sup>≈</sup>10<sup>2</sup> cfu/mL (25.2 cfu) by dilution with PBS. Then, the resulting suspension was passed through a 0.2 μm microbiological filter to remove bacterial cells and 400 μL of the filtrate was pipetted onto a sterile cotton filter disc. Filtrate was subsequently cultured and resulted in 0 cfu growth.

This "cell-free" suspension was placed in a single can and interrogated by both canines Mi and Ti in a line of 9 other cans into which were placed sterile cotton filter discs with 400 μL of PBS and one can with a sterile cotton filter disc infused with 400 <sup>μ</sup>L of Xcc culture adjusted to <sup>≈</sup>10<sup>2</sup> cfu/mL <sup>=</sup> 25.2 cfu as a positive control. All tests were blinded as described in Section 2.4 above. Canine alerts were recorded and performance was assessed.

#### **3. Results**

#### *3.1. Initial Training of Canines for Xcc Scent Signature Recognition*

The Florida researchers were not involved in the initial training and evaluation of the first Xcc-detector canines in California. However, the collaborating trainer/handlers indicated that after a few days of repeated training using STU collected scent pads, the experienced criminology detector canines quickly learned and imprinted on the Xcc scent signature, alerting on scent pads from Xcc-infected trees with 95–98% accuracy (data not shown). This was a similar level of accuracy to that which trainers normally expect and achieve with criminology target scent signatures. This also led us to believe that training canines for Xcc detection was possible and thus could be viably explored further.

We established that longer sample collection times were better for recognition/training, presumably because the scent deposition on the pads was stronger, i.e., the concentration of the unique Xcc VOCs was higher. However, once trained, dogs were capable of detecting the Xcc signature on samples collected over exposure durations as low as 1 min, i.e., lower scent concentration. We also determined that the Xcc scent signature was not long lived. Once the heat-sealed, vapor-proof plastic bags were removed from refrigeration and opened to the ambient environment, the trained canines would alert on them for 2–3 weeks before the Xcc scent signature diminished to unreliable levels. Scent pad viability could be lengthened to some extent by reheat-sealing and refrigerating the pads between uses, but due to scent degradation concerns, we limited scent pad use to 2 weeks from time of collection. Therefore, the VOC composition of Xcc scent does not appear to be stable over time, unlike some scents such as human or animal scents that are very stable and can remain on clothing and other objects for long periods up to years. However, repeated experiments with a canine trained to recognize Xcc resulted in >95% correct recognition and differentiation from non-infected citrus scents, from properly handled scent pads.

#### *3.2. Training Canines for Detection of Xcc-Infected Plant Material*

During initial proof-of-concept studies, the 11 September 2001 terrorist attacks occurred, and many of the detector canines throughout the country were diverted to security-related tasks and became unavailable for research and our studies were curtailed. Over the next three years, Xcc had become more widely spread in Florida, reducing regulatory concerns for the movement of infected plant

materials within state. Therefore, in 2004–2005, we were able to resume studies on canine detection of Xcc in collaboration with the NDDTC. However, the NDDTC's mission is to train and deploy agricultural contraband detector canines to US ports of entry and overseas ports of departure for US travelers and freight. Our research project utilized these same canines, who upon completion of their training were shipped out to their intended assignments, as were some of the staff. Thus, we did not have consistency of canines or staff, and we rotated through canines and trainers. Even so, trainers reported >95% detection accuracy after a few days training with plants in boxes, indicating good imprinting on Xcc-infected plant materials. When deployed on a single occasion to an infected orchard, the box-trained Labrador retriever also accurately detected Xcc-infected versus non-infected leaves from the orchard. Thus, we were able to discern that canines could be trained to detect Xcc using the differential of infected versus non-infected plant materials. Due to the high turnover rate of canines and handlers, the project was again curtailed.

## *3.3. Detection of Xcc in Simulated New Plantings*

Canine detection studies resumed in 2006 and continued through 2020 with three collaborative commercial canine detection companies. The performance of the first canine "Juice" indicated excellent scent signature recognition (Video S1, see Supplementary Materials). Sensitivity, specificity, precision, and accuracy ranged from 0.7333–0.9167, 0.9943–1.0, 0.8759–1.0, and 0.9733–0.9967, respectively, indicating that canine false negative (FN) alerts were slightly more prolific than false positive (FP) alerts, especially at higher incidence levels. Performance metrics indicated detection was superior for lower (<5%) Xcc-incidence, eroding slightly when incidence was ≥7%. Even so, overall accuracy was 0.9842 (Table 2).


**Table 2.** Latent class metrics for canine Xcc-infected tree detection in simulated 100-tree Duncan grapefruit citrus orchard by tree incidence accumulated over all the months tested.

<sup>a</sup> Performance metrics as described in Section 2.3 above. Orchard grid consisted of 100 trees with the indicated Xcc-infected tree incidence. The grid was re-randomized between each replication, at each incidence level and interrogated by a single canine—"Juice". Months tested: J = July, S = September, D = December, M = May.

Seasonality (i.e., July, summer; September, fall; December, winter; and May, spring) did not have a perceptible effect on canine detection performance when assay results were accumulated across incidence levels (Table 3). Canines were proficient at Xcc detection prior to assessing performance metrics; however, a training effect was noted. The canines improved notably in sensitivity and slightly in overall accuracy as they became more comfortable with the "game" of detection over time (Figure 4).


**Table 3.** Latent class metrics for canine Xcc-infected tree detection in simulated 100-tree Duncan grapefruit citrus orchard for each month tested accumulated over all tree incidence levels tested.

<sup>a</sup> Performance metrics as described in Section 2.3 above. Orchard grid consisted of 100 trees with the indicated Xcc-infected tree incidence. The grid was re-randomized between each replication, at each incidence level and interrogated by a single canine—"Juice".

**Figure 4.** Latent class metrics for the effect of incidence of Xcc lesions on canine detection. The data demonstrate a training effect where canine detection of Xcc-infected trees (sensitivity) significantly improves between the first and second tests, which also improves slightly the overall accuracy metric. In essence, the canines learn the "game" of detecting Xcc-infected trees when presented with a grid imposed by the experimental design and become more proficient at detection over time.

Since seasonality effects were not significant in the first study, performance assessment of the second two canines (Bady and Maxi) was compressed to two separate days separated by 31 days during the second study. During this second study, randomized Xcc-infected tree incidence ranged from 1 to 6% to reduce the probability of scent acquisition from nearby Xcc-infected trees when using higher incidence levels. The sensitivity, specificity, precision, and accuracy performance metrics for canine Bady ranges were 1.0–1.0, 0.9894–1.0, 0.6667–1.0 and 0.9900–1.0, respectively; whereas the same performance metrics for canine Maxi ranged slightly higher: 0.9167–1.0, 1.0–1.0, 1.0–1.0, and 0.995–1.0, respectively. The overall combined performance for the same metrics for the two canines were 0.9856, 0.9974, 0.9257, and 0.9970, respectively (Table 4). Canine Bady alerted to 11 FP and 0 FN over 2200 interrogations, whereas Maxi alerted on 0 FP and 2 FN over 2300 interrogations, suggesting that Maxi was slightly more accurate overall and both of these newer trained canines were slightly more accurate compared with Juice, the earlier trained canine. The 11 FP alerts by Bady were distributed across Xcc-infected tree incidence levels.

#### *3.4. Detection of Xcc Infections of Increasing Age*

Latent class performance metrics for detecting Xcc lesions indicated no effect of increasing lesion age. For canine Bady sensitivity, specificity, precision = positive predicted value, and the accuracy performance ranges were 0.9–1.0, 1.0–1.0, 1.0–1.0 and 0.9960–1.0, respectively, whereas the same performance metrics for canine Maxi ranged slightly higher: 0.95–1.0, 1.0–1.0, 1.0–1.0, and 0.9980–1.0, respectively. The overall combined performance for the same metrics for the two canines were 0.9450, 1.0, 1.0, and 0.9978, respectively (Table 5). Neither canine had any FP alerts, and Bady and Maxi had eight and three FN alerts, respectively, which were distributed across Xcc-infection age groups with no apparent effect of age (Figure 5).

**Figure 5.** Latent class metrics for canine detection of Xcc-infections of increasing age. There was no relationship of lesion age on canine detection of Xcc-infected trees. However, the data demonstrate a training effect for canine detection of Xcc-infected trees (sensitivity) which significantly improves between the first (1) and second (2) tests as canines learn the "game" imposed by the experimental design and become more proficient at detection. Acc= Accuracy.


**Table 4.** Latent class metrics for canine Xcc-infected tree detection in simulated 100-tree Duncan grapefruit citrus orchard by tree incidence accumulated for eachcanine

Performance metrics as described in Section 2.3 above. Orchard grid consisted of 100 trees with the indicated Xcc-infected tree incidence. The grid was re-randomized replication, at each incidence level and interrogated by two canines—"Bady" and "Maxi".



a population of 48 non-infected

 trees. The grid was

re-randomized

 between each replication, at each incidence level and interrogated by two

canines—"Bady"

 and "Maxi".

 a

*Entropy* **2020** , *22*, 1269

a

#### *3.5. The E*ff*ect of Incidence of Xcc Lesions on Detection*

Across all lesion incidence levels, both canines had a higher prevalence of FN than FP alerts (FDR = 0.1159, FPR = 0.0022, overall). Canine Bady had fewer FP than Maxi, but this effect was not consistent across the range of lesion incidence (Figure 6). The sensitivity of both canines increased with lesion incidence (0.6750 to 0.8750), whereas specificity remained high (0.9978) as did overall accuracy (0.9910–0.9952) across all incidence levels (Table 6).

**Figure 6.** Effect of lesion incidence on false negative canine detections. The data demonstrate a general erosion of canine detection of Xcc-infected trees (sensitivity) as the incidence of infection within individual trees increases. As the scent signature becomes stronger due to heavy infection in some trees, canines begin to false alert on nearby trees because they acquire the scent farther away from the true source.


**Table 6.** Latent class metrics for the effect of incidence of Xcc lesions on canine detection.

<sup>a</sup> Performance metrics as described in Section 2.3 above. The simulated orchard grid consisted of 100 trees with one Xcc-infected tree of each incidence level (1, 5, 50, and 500 lesions/tree) randomly placed within a population of 96 non-infected trees. The grid was re-randomized between each replication and interrogated by two canines—"Bady" and "Maxi".

Both PPV (0.8710–0.8974) and NPV (0.9930–0.9973) remained relatively constant across Xcc incidence, indicating that canines were superior in predicting actual Xcc-non-infected trees (NPV) and slightly less predictive of actual Xcc-infected trees (PPV), although both canines had low false positive rates (FPR). These data also demonstrated the advantage of repetitive training. This training effect was seen as a general improvement in the sensitivity of Xcc detection with accumulated experience over an increasing number of trials. Conversely, accuracy remained high and improved only slightly, and specificity remained high and stable throughout (Figure 7).

**Figure 7.** Effect of Xcc lesion incidence on overall accuracy, sensitivity, and specificity of canine detections. The data demonstrate a general improvement in the sensitivity of canine detection as training experience for both canines was accumulated over an increasing number of trials, whereas overall accuracy was high throughout and improved only slightly over accumulated trials and specificity remained high and stable. We use the overall results for each animal displayed in Table 6 to illustrate the diagnostic performance of the canines Bady and Maxi in an information theoretic framework. Figure 8 shows the results of this exercise.

The sequence of plots illustrates the diagnostic performance of the canine starting with the graphical summary provided by the likelihood ratio plot, which summarizes diagnostic performance using metrics that are independent of disease prevalence. The positive likelihood ratio (LR+ = TPP/FPP) is the slope of the solid line segment: for each animal, the values are Bady=701.5, Maxi=530.4. The slope of the dashed line segment is the negative likelihood ratio for each animal (LR− = 1 − TPP/(1 − FPP) = FNP/TNP). The values for each animal are Bady = 0.24, Maxi = 0.13. Generally, for effective diagnostic performance, LR+ >> 1 and LR− << 1 are required. Both canines achieved effective positive and negative diagnostic performance, with positive performance (i.e., confirmation of pathogen presence) being particularly effective.

**Figure 8.** (**A**) Likelihood ratio plot for canines Bady and Maxi, based on average performance over a range of pathogen prevalence values in citrus canker detection trials. The axes are the same as those for a receiver operator characteristic (ROC) curve, each canine being represented by a single point. In general, the closer the point for the animals to the upper left corner with TPP = 1, FPP = 0, the better the overall diagnostic performance. The gradient of the solid line section is the positive likelihood ratio for cases for each canine. The gradient of the dashed section is the negative likelihood ratio. (**B**) A predictive value leaf plot for Bady and Maxi based on the likelihood ratio values displayed panel (**A**). The plot displays the relationship between possible disease prevalence (or prior in a Bayesian framework) and the possible post-diagnostic probability of disease given either positive or negative diagnostic outcomes. The canines have very similar positive alert performance, but they differ in the information they provide in negative alerts. In general, a negative alert by Maxi provides more information than one by Bady. For both canines, positive alerts result in a high post-test probability of disease even at low prior disease values. (**C**) A relative entropy "loop" plot for each animal based on the same likelihood ratios. For each animal, disease prevalence increases clockwise around the loop which shows the expected information supplied (in bits) for a positive vs. negative alert at each possible disease prevalence between 0 and 1 in steps of 0.0001. In effect, the loop plot shows the information gain from alerts corresponding to the change in probable disease prevalence following diagnosis displayed in the leaf plot in panel (**B**).

Likelihood ratios derived from TPP, FPP, TNP, and FNP allow Bayesian updating of disease prevalence to give disease prevalence conditional on diagnostic outcomes; i.e., they can be used to produce predictive values. Figure 8B shows a leaf plot for Bady and Maxi, based on the likelihood ratios displayed in Figure 8A. The leaf plot shows values for PPV and (1-NPV) as functions of initial disease prevalence over the range of prevalence (0,1). To interpret the plot, select a value for prior Xcc prevalence. Locate the value on the diagonal of the plot. To read off the post-diagnostic probability of Xcc presence following either positive or negative diagnosis, trace up or down (respectively) from the point on the diagonal until intersecting with the curves for either canine. The PPV and 1-NPV values can be read off the vertical scale from reading across from the upper and lower curves (respectively); the 1-NPV value being converted to NPV by simple arithmetic thereafter. We draw attention to the high posterior (i.e., PPV) probability of Xcc presence generated by both Bady and Maxi even at low prior disease prevalence. Figure 8B also illustrates that, in general, Bady and Maxi provided more information in positive than negative diagnostic outcomes.

Following Hughes et al. [34], we calculated the expected mutual information (i.e., relative entropy) for positive and negative diagnostic outcomes (I+ and I−, respectively) for Bady and Maxi corresponding to the likelihood ratios in Figure 8A and the predictive values in Figure 8B. The resulting "loop" plot is shown in Figure 8C. The black curve, descending from left to right across the plot, shows the theoretical maximum information (in bits) that could be obtained from an error-free definitive diagnosis of disease status for an unknown sample. Disease prevalence increases from 0 to 1 sequentially along the curve from left to right, indicating that at low disease prevalence, a positive

diagnostic outcome contains more information than a negative one, with the opposite being true when disease prevalence is high. The short diagonal line intersecting the curve shows values where I+ = I−. Note that this diagonal intersects the curve at a value of I+ = I− = 1 bit, which occurs at a disease prevalence of 0.5; in effect, this corresponds to seeing the result of a coin flip.

Corresponding to the results in Figure 8B, the information loop plot shows clearly that Bady and Maxi provided more information about Xcc disease status in positive than in negative diagnostic outcomes. The maximum information supplied in a positive diagnosis for either canine was in the order of 4.5 bits, whereas the maximum value for negative diagnoses was less than 1 bit.

#### *3.6. Detection of Xcc Infections in Decaying Foliage*

On the day the leaves were abscised from infected trees (day 0) canines correctly identified all positive and negative leaf piles without error. There were very few FP errors throughout the study (3/448). However, as time post-abscission increased FN errors increased from 1 to 20 for days 1 and 27 post-abscission, respectively. On day 27, neither canine detected any of the decaying 20 Xcc-infected leaf piles after repeated attempts, i.e., FN error = 100% (Table 7).

**Table 7.** Latent class metrics for canine detection of Xcc-infections in leaves decaying over time post abscission.


<sup>a</sup> Performance metrics as described in Section 2.3 above. Two canines "Bady" and "Maxi", interrogated piles of Xcc-infected and non-infected decaying leaves at various assessment times post-leaf abscission. Leaf piles were continuously exposed to ambient conditions over time. Data were accumulated and combined over canines and replications. NA—metric could not be calculated due to a complete lack of canine detection by both canines, resulting in a calculated TP + FP = 0 value in the denominator.

#### *3.7. Detection of Xcc in Citrus Packinghouse Environments*

Canine Juice detected just a few Xcc lesions on 1–2 grapefruit fruits when commercially packed into cartons of ≈50 fruit per carton, especially at low incidence. FN errors generally exceeded FP errors throughout the packinghouse study. However, the total number of FN + FP errors began to increase when ≥4% Xcc-incidence of cartons were arrayed in the grid of 100 cartons on the packinghouse floor (Table 8, Figure 9 and Video S2, see Supplementary Materials).

We conducted higher Xcc-incidence interrogations later in the day, which corresponded to a general rise in temperature in the packinghouse environment, resulting in fatigue of the canine. It was observed that the canine often acquired the Xcc scent signature from cartons farther away depending on the direction of the airflow. This was evidenced by the canine's desire to disperse with the required serpentine search pattern through the grid in favor of going directly to a detected target at some distance. Additionally, the spatial proximity to other Xcc-infected cartons as well as the prior locations of Xcc-infected cartons with residual odor led to some FP results (see spatial analyses below).


**Table 8.** Latent class metrics for canine detection of Xcc-infections of various incidence in commercially packed boxes of red grapefruit.

<sup>a</sup> Performance metrics as described in Section 2.3 above. Incidence (1–10) indicates the number cartons containing Xcc-infected fruit placed in a grid of 100 commercially packed boxes each containing ≈50 red grapefruit.

**Figure 9.** Effect of incidence (proportion) of commercial fruit boxes containing Xcc-infected grapefruit on false negative canine detections. The data demonstrate a general erosion of canine detection of Xcc-positive boxes (sensitivity) as the incidence of infected boxes increases within the grid in the packinghouse. As the scent signature becomes more prevalent within the test grid commensurate with the number of boxes containing Xcc-infected fruit, canines begin to false alert on nearby boxes in close proximity to boxes with actual infected fruit because the canines acquire the scent farther away for the true source.

#### *3.8. Assessment of Xcc Detection in Commercial Citrus Orchards*

For the canine Juice, sensitivity, specificity, precision = positive predicted value, negative predictive value and accuracy performance metrics were 1.0, 0.9842, 0.2222, 1.0, and 0.9843 for Orchard 1, and 0.8667, 0.9848, 0.7222, 0.9939 and 0.9797 for Orchard 2, respectively (Table 9). When we progress from a known infection status of potted trees in a simulated orchard (composed of confirmed Xcc-infected and non-infected trees, held in an isolation greenhouse to ensure no additional infections — see simulated new planting results above) to a commercial orchard (Figure 10) (in which disease status is unknown and subject to human visual confirmation and errors), detection precision metrics declined. This decline relates to our inability to determine the infection status unequivocally via

human visual inspection. As previously demonstrated, in mature trees it is difficult-to-impossible to detect all incipient and low incidence infections by human visual inspection ([11,39]; T. Riley, unpublished results).


**Table 9.** Latent class metrics for canine detection of Xcc-infections in commercial red grapefruit orchards in Indian River County, Florida.

<sup>a</sup> Performance metrics as described in Section 2.3 above. Orchard 1—10 rows of trees, 38 trees per row, 8 missing trees. Orchard 2—17 rows of trees, 29 trees per row, 47 missing trees. Orchard 2 (theoretical results for discussion) considers that the 5 FP detections were Xcc-infected but not detected by human visual survey.

**Figure 10.** Canine versus human visual detection of Xcc-infection in commercial citrus orchards in Indian River County, Florida. (**A**) Orchard 1—Mature 42-year-old red grapefruit on sour orange rootstock, (**B**) Orchard 2—7-year-old red grapefruit.

#### *3.9. Analyses for Spatial Heterogeneity of Xcc Detection Errors*

Sufficient data were available for the spatial analysis of detection errors for three portions of the overall Xcc detection study for the following trials: (1) Disease incidence of lesions in simulated orchards, (2) Lesion age in simulated orchards, and (3) Incidence of cartons with Xcc-infected fruit in the packinghouse. In all three trials, there were more FN than FP errors. Errors were omnidirectional and more prevalent at shorter distances, especially <6.5 m from a true positive (TP) (Figure 11).

**Figure 11.** Spatial heterogeneity analysis of canine detection errors for trials of (**A**) lesion incidence, (**B**) lesion age, (**C**) packinghouse, and (**D**) combined data A through C. In all three trials there was a greater number of false negative (FN) than false positive (FP) errors. All trials had a greater prevalence of FN errors and errors were more prevalent at shorter distances from a true positive (TP). Distance is presented as multiples of the distance between plants or cartons (packinghouse) in the grid, i.e., 3.048 m (10 ft) within and between rows.

#### *3.10. Direct Detection of Xcc Bacteria*

The canine Mi trained on *in planta* Xcc-infected trees was competent at detecting Xcc bacteria harvested from in vitro axenic cultures (Videos S3 and S4, see Supplementary Materials). The reciprocal was also true, in that the canine Ti trained to detect Xcc from in vitro axenic cultures was able to detect the pathogen *in planta* from Xcc-infected trees (Videos S5 and S6, see Supplementary Materials). Both canines detected the reciprocal target without any additional training. There was a slight trend in that canine Mi was slightly superior at detecting Xcc-infected plants and canine Ti was slightly better at detecting Xcc from culture (Table 10).


**Table 10.** Latent class metrics for canine detection of Xcc-infected plants versus Xcc culture.

<sup>a</sup> Performance metrics as described in Section 2.3 above. Two canines, Mi and Ti, were trained to detect Xcc-infected plants or Xcc cultures, respectively. Both canines interrogated both Xcc-infected plants or Xcc cultures to determine the effect of target used for training. Data represent the combination of two replications.

## *3.11. Estimation of Bacterial Detection Threshold*

The result that canines trained on either Xcc-infected plants or on Xcc axenic cultures can detect the bacteria directly in vitro as well as *in planta*, with near equivalent accuracy, implied that canines may be detecting the bacteria directly in plants and not volatiles generated by the host plants in response to infection. Therefore, we sought to determine the detection limits, i.e., the lowest concentration of bacteria required by the canines for scent detection. Thus, both canines were challenged with the task of detecting Xcc harvested from axenic culture, suspended in PBS, over a range of dilutions from <sup>≈</sup><sup>4</sup> <sup>×</sup> 102 to 0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> cfu as described above, with final bacteria on sterile cotton filter discs estimated to be 26.4, 3.60, 0.27, 0, and 0 cfu. Both canines detected Xcc throughout the dilution range, indicating that a sufficient concentration of the Xcc scent signature was present throughout the range tested (Table 11 and Videos S7 and S8, see Supplementary Materials). Neither canine reacted to the *B. megaterium* isolate at the estimated 57 cfu on the scent pad, indicating that the scent signature was apparently specific to Xcc bacteria and not a ubiquitous bacteria scent. Neither canine reacted to a <sup>1</sup> <sup>×</sup> <sup>10</sup><sup>2</sup> cfu/mL concentration of 0.2 μm bacterial culture filtrate, indicating that the subcellular component(s) composing the scent signature was not diluted out at any concentration tested but instead was filtered out. This indicates that the canines are detecting Xcc at below the cellular population level, i.e., below a single bacterial cell.




**Table 11.** *Cont.*


a pipetted onto filter pads and presented to canines as targets, i.e., 26.4, 3.60, 0.27, 0, and 0 cfu, respectively. c **Xcc** 

culture through 0.2 μm filter to eliminate all bacterial cells. Filtrate was cultured and resulted in 0 cfu growth. d *B. Meg* = *Bacillius megaterium*, additional bacterium used as a negative

control to ensure that canines were not reacting to random bacteria. Original dilution was 143 cfu/mL with ≈57 cfu/400 μL pipetted onto a sterile cotton filter disc.

**Filtrate**—bacterial

 culture filtrate prepared by filtration of ≈102 cfu/mL

#### **4. Discussion**

#### *4.1. Canine Detection of Xcc in Simulated Plantings and Information Theoretic Analyses*

Through the multiple trials of this study, we were able to demonstrate unequivocally that canines can discriminate and detect Xcc *in planta* (in infected foliage and fruit) by the use of canine olfactory surveillance in simulated and commercial orchards and packinghouses.

We found that canines alerted to Xcc infections over a range of lesion ages from 1 to 12 weeks post-infection with equal accuracy. These results indicate that the scent signature of Xcc-infection as perceived by canine detectors does not change significantly, if at all, with age of infection. At one week post-infection, the lesions are not yet erumpent and only visible with 10× magnification. At the non-erumpent stage, the numbers of bacteria in such lesions are low and do not directly expose the bacteria to the environment (i.e., lesions have not broken through the intact epidermis and cuticle), yet a sufficient concentration of scent signature appears to emanate from leaves for canine recognition.

Canines were tested in simulated plantings with 2% to 10% (canine Juice initial test) and 1% to 6% (canines Bady and Maxi) incidence of infected trees. In the initial test with the canine Juice, sensitivity increased with lesion incidence, whereas the specificity and overall accuracy remained static across all incidence levels. In a more expansive test with two more extensively trained canines, sensitivity for canine Bady appeared to be unaffected by incidence of infected trees, whereas the sensitivity of canine Maxi seemed to erode slightly with increased incidence of infected trees. Bady had a number of FN alerts, eroding specificity and precision somewhat that were not related to infected tree incidence, whereas Maxi had none and experienced no such erosion. Overall, canines were superior at predicting Xcc-non-infected trees (NPV) and slightly less predictive of actual Xcc-infected trees (PPV). As a field deployable early detection tool, a slight trend toward false negatives is usually accepted by growers, as they are willing to tolerate an assay that misses a few infections rather than misidentifying disease-free trees as diseased, resulting in tree removal and commensurate loss of production.

Xcc lesion populations can range from a single lesion to thousands per tree depending on inoculum prevalence and the susceptibility of tissues following an inoculum dispersal event. Across the range of lesion incidence levels assayed, both canines had a higher prevalence of FN than FP alerts. The sensitivity of both canines increased with lesion incidence, whereas the specificity and overall accuracy was >99% across all incidence levels with low false positive rates (FPR).

Information theoretic analyses of the overall diagnostic performance of both canines showed very clearly that the information provided by positive diagnostic outcomes was far in excess of that provided by negative outcomes, which is in relation to the impact on the probable presence of Xcc. We used a graphical approach recently developed by Hughes et al. [34] to illustrate the diagnostic capacity of the animals. One noteworthy aspect of canine performance is their very high positive likelihood ratios for cases. It has been noted by several authors that disease diagnosis as a tool in decision making at low disease prevalence is problematic [40,41] because, with the diagnostic capability of many commonly used approaches, the post-test probability of disease is still relatively low, even after a positive test outcome, and the sampling error is pervasive (see Section 4.5 below). The positive likelihood ratios achieved by the canines assessed here are in the region of two orders of magnitude higher than values reported in the literature for many plant and human diseases [38,42,43]. As illustrated in Figure 8B this diagnostic capacity allows for effective disease screening even when the background disease prevalence is low.

#### *4.2. Detection of Xcc Infections in Decaying Foliage*

Lesions of Xcc cause the surrounding citrus tissues to produce ethylene, which can cause abscission if lesions occur near the junction where a leaf petiole joins a stem. When Xcc-infected leaves fall to the ground near citrus trees, rain or irrigation splash can cause inoculum dispersal and re-infection. However, Xcc survival in lesions of decaying leaves decreases exponentially over time [44,45]. Canines were able to detect Xcc in lesions when leaves were newly abscised, but canine detection eroded over

time. As time post-abscission increased, FN errors increased, and canines were unable to detect Xcc in leaves 27 days post-abscission. The results indicate a rapid decline in the emanation of Xcc scent signature over time as Xcc-infected foliage decays live Xcc population concurrently decline, resulting in an increase in FN errors through time post-abscission and that abscised infected leaves may not be a reliable indicator of a tree's infection status.

## *4.3. Detection of Xcc in Citrus Packinghouse Environments*

Canine detection of Xcc lesions in boxed citrus fruit in the packinghouse was exceptional considering the noisy, hot, and highly distracting environment. Total FN + FP errors increased when the incidence of Xcc-incidence of cartons was ≥4%. We conducted higher Xcc-incidence interrogations later in the day, which corresponded to a general rise in temperature in the packinghouse environment, causing fatigue of the canine. Additionally, the spatial proximity to adjacent or proximal Xcc-infected cartons may have led to FP alerts. Furthermore, FP alerts occasionally occurred due to persistent residual odor when Xcc-free cartons were relocated (during re-randomization between replications) to where a prior Xcc-infected carton had previously resided. This issue has been noted often by canine trainers across an array of target odor types. Therefore, we re-examined the data for error effects due to spatial proximity to Xcc-infected cartons and residual scent signature odor temporarily permeating the concrete packinghouse floor (see Section 4.4 below).

In an initial study prior to the full experiment, we noted that the canine alerted on two cartons that had recently passed through the packing line. These cartons were visually inspected thoroughly at multiple points in the packing line by commercial packinghouse inspectors and were determined to be Xcc-negative. At first, we thought these were FP detections, but when the grid was re-randomized and the cartons were relocated to different positions, the canine continued to alert on these same two cartons. Upon careful re-inspection, we determined that these two cartons contained a single fruit each with one and two small lesions (≈1 mm dia.), respectively. These observations demonstrate the keen sensitivity of canines to detect even a trace amount of odor and that human visual inspection has limitations for the detection of low-incidence infections especially with small lesions. Canine detection in the packinghouse is superior at low incidence levels. Current detection and elimination/discard of infected fruit were conducted by multiple skilled inspectors observing fruit as it passes through the multiple designated visual examination stations of the packing line. As expected, small Xcc lesions are the most difficult for inspectors to detect and differentiate from other small blemishes. However, Xcc lesions that go undetected can cause a rejection of fruit shipments when inspected at ports of destination. Thus, a highly sensitive detection tool such as canines that does not depend on visual detection could greatly enhance the sanitation of packed fruit shipments and diminish the proportion of rejections at destination markets.

#### *4.4. Spatial Heterogeneity of Xcc Detection Errors*

A spatial assessment of the detection data allowed us to determine if the proximity of Xcc-positive trees positioned immediately adjacent to, at oblique angles to, or upwind from canine FNs or FPs were related to the cause of the error. In the trials of this study, the majority of errors, both FN and FP, were within 1 to 2 plant or carton (packinghouse) spaces (within row, across row, or at oblique angles) of a TP. Errors also became more prevalent when the incidence of TP targets increased within the trial grids. However, there was no prominent directionality to errors. False positive alerts can be caused by the dissemination of the odor plume around an odor source. Jezierski states, "*Ideally, dogs should alert as close as possible to the site where the odorous material is hidden by comparing the di*ff*erences in odor concentration inside the odor plume. It is common for a dog to enter, then exit and reenter the scent cone during odor detection, which may account for the number of times a canine passed a hide as demonstrated in the data. The role of the distribution of the odor plume was evident in our experiment when comparing the percentage of false alerts in particular searching sites. When searching outdoors, the distribution of the odor plume may often enable a more easily directional scenting and localization of odor source, which thus takes less* *time with more correct and fewer false alerts*" [46]. Craven found that the odor plume of a drug moves and disperses, depending on air currents, humidity, temperature, or features in the terrain, which may also influence the detection performance [47]. The fluid dynamics of odor transport during canine scent detection is highly complex and has not been examined extensively. Angle stated that, *"Much more research needs to be conducted in order to understand the movement of biological VOC within the thermal plume (e.g., microcurrents) and in the aerodynamic wake*/*wind currents in order to develop search patterns to optimize biomedical detection"* [39].

Although errors were relatively few overall, FN alerts were more common with Xcc detector canines than FP alerts. In an orchard environment, airflow is highly channeled between hedge rows of trees, dynamic, and convoluted with ubiquitous eddies. The complexity of using canines as detectors in a highly heterogeneous open-air commercial orchard environment, or indoor in a harsh noise and distraction rich packinghouse, presents a myriad of potential causes for error, not all of which do we understand or recognize. Our data indicate that the relatively few FN errors increases slightly with an increase in the incidence of TP targets in the environment, especially when these TP targets are in close proximity. One explanation could be that when a canine correctly locates a TP (TP1), infrequently, it can be confused by another TP (TP2) in close proximity. In this example, the canine may misinterpret that the scent it acquired near TP2 is originating from TP1, disregarding TP2 as an additional scent source. Conversely, if the scent emanating from TP1 is of significantly greater concentration than the scent emanating from TP2, the canine might track to the "maximum scent concentration" of TP1, thus ignoring the "minor" scent signature of TP2.

It is well demonstrated that canine detectors can pick up a target scent at some distance. In our experience, training canines for Xcc detection, and additionally for CLas [26], plum pox virus, and vegetable virus detection (unpublished), we have often observed that canines acquire a scent signature at considerable distance from the known target. This is consistently experienced when training canines in spatial grids of predominantly non-infected plants with a low incidence of pathogen-infected plants randomly placed in the grid. Normally, we urge the canine to interrogate each plant in the grid in a serpentine pattern up and down each row so that we can ensure that all plants are assessed equally and collect performance data. However, if the canine is allowed to interrogate the grid off leash, it will often acquire a target scent plume and divert its trajectory obliquely across multiple rows directly to an infected plant: then, it will alert. In other words, the canine has already acquired a viable target at a considerable distance, and it is likely that by odor strength and gradient characteristics, the canine developed a mental picture of the target's estimated location that the canine refines as it hones in on the source.

#### *4.5. Assessment of Xcc Detection in Commercial Citrus Orchards*

As intimated in the results above, when we examined canine performance in grids for simulated orchards, lesion age, and lesion incidence, we had unequivocal knowledge of the Xcc infection status of all individual trees. This is because we performed the inoculations, enumerated the Xcc infections on each tree, and all trees were maintained in an isolation greenhouse prior to use where disease spread was highly unlikely. However, in a commercial orchard environment, precise mapping of disease is immensely more difficult. In orchards, we were hindered by the difficulty of Xcc detection and our reliance on human visual survey, especially when trying to accurately map low incidence infections. Tree canopies can be large, composed of >100,000 leaves per tree, and some leaves are in difficult-to-view positions — high in a tree or visually occluded by surrounding branches and foliage. Angle and quality of light, cloudy or bright sun can enhance or diminish visual acuity. For example, when visually surveying trees during the citrus canker epidemic for Xcc infections during the Citrus Canker Eradication Campaign in Miami, Florida, it required an average of 106 days (range 30–270 da) post-infection to visually detect Xcc infections in trees on residential properties. Thus, incipient and low incidence infections were rarely found [11,48]. In a study conducted by USDA, APHIS to determine the efficacy of visual detection of Xcc, 18 two-person teams of trained inspectors surveyed a 12.1 ha (30 ac)

block of commercial citrus with known Xcc infections. No two teams detected the same Xcc-infected trees, no teams detected all Xcc-infected trees, and each team found Xcc infections and infected trees previously unrecorded. [Riley, unpublished results].

In the current study, in Orchard 2, the canine alerted on five trees previously unconfirmed by human visual assessment (Figure 10). In an attempt to confirm or refute the canine alert on one of these trees, two highly trained technicians required over two hours on the ground and climbing in the 7-year-old tree, ≈3 m tall canopy to find a single leaf lesion that was obscured by sooty mold. Other lesions may have existed in this tree as well but eluded visual inspection. Visual inspection was limited due to the large number of trees that needed to be visually scrutinized for low incidence lesions and because of the immense amount of time needed to fully assess each tree. Therefore, the infection statuses of these five trees were categorized as FP alerts by the canine (Table 9, Orchard 2). However, if we give the benefit of the doubt to the canine and assume that these trees escaped human visual detection, and were thus TP trees that were correctly identified by the canine, the precision metrics improve considerably, with sensitivity, specificity, precision, negative predictive value, and accuracy performance metrics rising to 0.9, 1.0, 1.0, 0.9939, and 0.9942 (Table 9, Orchard 2—Theoretical). Thus, the precision = positive predicted value (PPV) increases from 0.7222 to 1.0 and overall accuracy increases from 0.9797 to 0.9942, both theoretically.

Humans have great difficulty detecting Xcc in tree canopies with low Xcc-infected leaf incidence. Whereas the canine was a highly sensitive detector (i.e., discovered one lesion within a large canopy) that very rapidly detected the infection and alerted within 1–2 s. The differential between canine and human sensitivity and speed, canine-2 s vs. human team-2 h, becomes apparent and exemplifies the significant differences in probability of detection. These differences translate directly into cost of survey, leaning heavily toward the superiority of canine detection. Additionally, from our experience, it was inappropriate to use a less sensitive detection method (human visual) to validate a more sensitive method (canines). This is demonstrated by the example of commercial Orchard 2 (Table 9, Figure 10), for which five TN trees (as determined by human visual survey) were probably incorrect although identified by the trained canine detector as alerts, and therefore diminished the true estimates of canine detection performance.

#### *4.6. PPV Versus NPV, the Choice between Risk-Aversion and Risk-Acceptance*

Growers and regulatory agencies need to ask themselves what is more important depending upon their detection requirements. Is it important to detect all Xcc-positive plants (PPV = 1.0), even if it means a few FP plants (NPV < 1.0) will be indicated as well (i.e., risk averse: willing to cull or treat some Xcc-negative plants in an attempt to best control/mitigate a disease epidemic)? Conversely, does a grower prefer a diagnostic that never falsely implicates a plant as Xcc-positive (NPV = 1.0), and is willing to accept a few FN indications (PPV < 1.0) (i.e., risk accepting: does not want to cull or treat any non-Xcc-positive plants, for example attempting to avoid an adversely harsh regulatory response during an eradication campaign)? Ideally, we want a diagnostic with the highest PPV and NPV possible. However, PPV and NPV are influenced by disease prevalence [30,31]. For example, if we hold sensitivity and specificity constant, the lower the disease prevalence, the higher the PPV. In contrast, as the incidence of the disease increases in a population, PPV improves. Therefore, when evaluating the PPV and NPV metrics for canines or any other diagnostics, we need to consider both the disease incidence within the population and the population size tested.

In general, near-perfect diagnostics are rare in both medicine and plant pathology for a wide array of reasons. Even near-perfect diagnostics are often plagued by sampling error. For example, PCR detection of another citrus bacterial pathogen, CLas, the causal agent of citrus Huanglongbing, is near 100% accurate when testing infected tissue, but when used to assay field trees, it often gives FN results with accuracy of ≈20% due to sampling error, due to the scarcity of infected cells even in systemically infected trees [26]. In mature trees with >100,000 leaves, selecting a leaf with CLas even from a systemically infected tree or even selecting tissue with CLas from an incompletely infected leaf can be

very improbable. In contrast, the canines are interrogating the tree holistically; that is, the CLas scent signature can be acquired regardless of where the bacteria are located in the tree, which circumvents both the potential paucity of CLas-infected tissue and the sampling problem. Canine detection of Xcc exhibited many of these same CLas hallmarks, although Xcc and CLas are very different types of phytobacteria. As noted above, in one case, the canine detected Xcc in a 7-year-old field tree with a single lesion obscured by sooty mold that required two trained technicians over 2 h to locate, and in another case, the canine detected single small lesions in fruit cartons missed by trained inspectors. Whereas human inspectors must spend several minutes examining each tree, canines trot along a row of orchard trees interrogating at a rate of ≈1 tree/2 s, continuously drawing in air parcels multiple times per second, thus efficiently surveilling large orchard areas quickly. Molecular or serological detection methods require collecting multiple samples per tree, returning to the lab to process and assay, and the use of moderate to extensive consumables; in addition, assay results are delayed depending upon laboratory backlog and are expensive (sampling and assay costs can exceed \$40 US per sample at the time of this writing). Conversely, canines are rapid (1–2 s/tree), results are essentially instantaneous, more cost effective (≈\$4.50/tree depending on orchard size, conditions, access, etc.), and more accurate due to the lack of sampling issues when using canines versus other detection methodologies [26].

Additionally, Xcc detector canines were successfully deployed by a commercial canine detection company to assess a commercial citrus nursery and successfully detected multiple Xcc-infected nursery plants unknown to the nurseryman, although no quantification of detection was documented. Such early detection can ensure that Xcc-infected plants are identified, eliminated in the nursery, and thus not transplanted to orchards. Avoiding the introduction of initial inoculum when establishing new plantings is an obvious advantage to mitigating an otherwise potential epidemic. Our collective results from this study imply that canines can be trained as viable early detectors of the pathogen in various agricultural environments, including citrus orchards, packinghouses, and nurseries.

## *4.7. Direct Detection of Xcc Bacteria and Estimation of Bacterial Detection Threshold*

We also trained canines exclusively to two targets: Xcc-infected trees and Xcc axenic cultures. We found that canines trained to either target inherently detected the heterologous targets, although both canines were superior at detecting the homologous target they were trained upon as opposed to the heterologous target. This implies that the scent signature does not need to be augmented with background citrus host VOCs and/or unique VOCs produced by the bacteria/host interaction. Therefore, Xcc cultures might be a sufficient training target if Xcc-infected plants are not available such as in a quarantine situation. In a recent study of five and four respiratory viruses and bacteria, respectively, researchers detected 12 and six VOC that were associated with bacterial and viral growth, respectively, and they identified two VOCs that differentiated bacterial and viral infection [49]. Angle and colleagues discuss the opportunity and complexity of discriminating VOC biomarker detection from diseased individuals [50]. Thus, the use of canines to detect and discriminate a phytobacterial pathogen is not unfounded. The precise and discriminating biomarker VOCs detected from Xcc bacterial cultures needs further examination. Such examination should include multiple canines and perhaps determination of the optimum Xcc-culture concentration on which to train canines to achieve optimized Xcc-infected plant detection, which is beyond the scope of this study.

In a deeper examination of direct in vitro detection, canines trained to either target were able to detect highly dilute Xcc culture solutions as low as 0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> cfu, i.e., 100-fold less than a single bacterial cell. This implies that the scent signature is composed of bacterial cell VOC constituents or exudates that occur at concentrations many-fold that of the bacterial cells. To explore this further, we had the canines interrogate a bacterial culture filtrate and found that the canines did not react to the filtrate without bacterial cells present. When considered concomitantly with the canines reacting to culture dilutions as low as 10−<sup>2</sup> cfu/mL, this led us to suspect that the canines are reacting to a subcellular component that is larger than the 0.2 μm filter pore size. Axenic bacterial cultures are composed of both intact cells and fragments of older dead and decomposing cells. One possibility is that bacterial cell fragments may play a role in the Xcc scent signature. The chemical fractionation and canine testing of such of filtrates and eventual identification of the single or multiple VOCs that compose the Xcc scent signature is beyond this study, as is their individual concentrations necessary for canine detection. However, the realization of a scent signature composition that is at least in part sub-cellular opens a clear and exciting path for future explorations.

## **5. Conclusions**

For millennia, humans have benefitted from application of the acute canine sense of smell to hunt, track, and find targets of importance. In this study, we demonstrated that canines can detect the Xcc phytobacterial pathogen of Asiatic citrus canker in simulated orchards, commercial orchards, and in a commercial packinghouse with high sensitivity, specificity, accuracy, and precision. Canines detected Xcc within 1–2 s of target interrogation time. Canines also alerted across a range of 1–12-week-old infections as well as across a range of pathogen prevalence with equal accuracy. Information theoretic analyses illustrate the diagnostic capacity of canines via their very high positive likelihood ratios for cases across pathogen prevalence at two orders of magnitude higher than values reported for other plant and human diseases. When trained to either Xcc-infected trees or Xcc axenic cultures, canines inherently detected the homologous and heterologous targets, suggesting they can detect Xcc directly rather than only volatiles produced by the host following infection. Canines were also able to detect the Xcc scent signature across a range of axenically cultured Xcc concentrations (104 to 100 = single cell) and even <1 bacterial cell, which implies that the scent signature is composed of bacterial cell volatile organic compound constituents or exudates that occur at concentrations many fold that of the bacterial cells. These findings indicate that Xcc cultures are a valuable surrogate targeting tool in the absence of infected plants. Results imply that canines can be trained and deployed as viable early detectors of Xcc across a diversity of environments and outperform the prevailing detection method, i.e., human visual detection.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/1099-4300/22/11/1269/s1, Video S1: Canine "Juice" surveilling simulated new planting of 100 young grapefruit trees. Trees with Xcc infection were randomized within rows. Note that the canine hesitates when it acquires the Xcc scent signature and the handler continues to pull on the leash in an attempt to dissuade the canine and thereby confirm the detection. Canine cannot be dissuaded to leave the TP Xcc-infected tree and eventually alerts by sitting next to the infected tree. Canines utilized in future trials were trained to alert (sit) immediately upon detection. Video S2: Canine "Juice" surveilling for 1–2 fruit with Xcc infections hidden within cardboard cartons of ≈50 commercially packed red grapefruit in a Florida packinghouse. Note canine hesitates when it acquires the Xcc scent signature and handler continues to pull on leash in an attempt to dissuade the canine and thereby confirm the detection. Canine cannot be dissuaded to leave the carton and eventually alerts by sitting next to a carton containing TP fruit with Xcc infections. Canines utilized in future trials were trained to alert (sit) immediately upon detection. Video S3: Canine "Mi" training to detect TP Xcc-infected tree randomized within a line of nine other TN trees. Canine alerts by sitting next to the TP Xcc-infected tree. Video S4: Canine "Mi" previously trained to detect Xcc-infected plants, correctly detects (alerts by sitting) on a metal can containing a TP cotton pad infused with axenic Xcc-culture suspension in PBS buffer in a line of cans with one TP and nine TN cans (with cotton pads infused with PBS buffer only) without prior training on Xcc culture. Video S5: Canine 'Ti' previously trained to detect TP cotton pads infused with axenic Xcc-culture suspension in PBS buffer, correctly detects (alerts by sitting) TP Xcc-infected tree in a line with one TP and nine TN trees without prior training on trees. Video S6: Canine "Ti" previously trained to detect TP cotton pads infused with axenic Xcc-culture suspension in PBS buffer, correctly detects (alerts by sitting) on a metal can containing a TP cotton pad infused with a 100 cfu/mL concentration of Xcc bacteria (0.27 bacteria/pad), in a line of nine TN pads in cans. Indicates that canine may be acquiring the Xcc-scent signature and alerting on subcellular bacterial components. Note canine surveilled line of cans off leash. Video S7: Canine "Mi" interrogates a row of 11 suspect cans. The canine alerts by sitting next to TP can #6 containing a cotton pad infused with a 400 μL Xcc bacterial suspension in PBS buffer of 10−<sup>2</sup> cfu/mL (0 bacteria/pad, subsequent determined via culturing). Remaining 10 cans contained TN cotton pads infused with 400 μL PBS buffer only. Handler rewards canine for correct detection with a few moments of play with a hard rubber 'kong' toy. Detection of bacterial concentration containing less than one bacteria indicates canine recognizes a scent signature composed of subcellular bacterial components. Video S8: Canine 'Ti' interrogates a row of 11 suspect cans. The canine alerts by sitting next to TP can #6 containing a cotton pad infused with a 400 μL Xcc bacterial suspension in PBS buffer of 10−<sup>2</sup> cfu/mL (0 bacteria/pad, subsequent determined via culturing). Remaining 10 cans contained TN cotton pads infused with 400 μL PBS buffer only. Handler rewards canine for correct detection with a few moments

of play with a hard rubber "kong" toy. Detection of bacterial concentration containing less than one bacteria indicates that the canine recognizes a scent signature composed of subcellular bacterial components.

**Author Contributions:** Conceptualization, T.G. and G.P.; methodology, T.G., and G.P.; software, W.L, D.P., N.M.; validation, T.G., G.P., W.S., E.T. and N.M.; formal analysis, T.G., W.L., D.P., and N.M.; investigation, T.G., G.P., E.T., W.S., and N.M.; resources, T.G. and W.S.; data curation, T.G., G.P.; writing—original draft preparation, T.G.; writing—review and editing, G.P., E.T., W.L., D.P., S.A., W.S., and N.M.; visualization, T.G., W.L., D.P. and N.M.; supervision, T.G., W.S. and G.P.; project administration, T.G.; funding acquisition, T.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was funded by USDA, APHIS Farm Bill grant 13-8130-0313-CA, and HLB MAC Grant 15-8130-0475. Mention of a trademark, warranty, proprietary product, or vendor does not constitute a guarantee by the USDA and does not imply its approval to the exclusion of other products or vendors that may also be suitable.

**Acknowledgments:** We thank Alan Hardison (deceased) for bringing canine detection to our attention many years ago. We thank Jerry Bishop, Bryan Brice, Tyler Meck, and William Moraitis F1-K9, Palm Coast FL for their deep collaboration and Peggy and Bill Heiser Coast to Coast K9 Teams, New Smyrna, FL specifically, on this project as well as K-9 Security and Detection International, Orange Co., CA, and Pepe Peruyero J&K Canine Academy, Inc., Alachua, FL, and the USDA, Animal and Plant Health Inspection Service (APHIS), National Detector Dog Training Center (NDDTC), Orlando, FL for their collaborations and assistance. We also thank Daniel Scott Citrus, Vero Beach, Florida, Dan Richie Riverfront Citrus Inc., Vero Beach, Florida and Michel Sallin, IMG Citrus, Vero Beach, Florida for research access to commercial orchards and packinghouse. We also thank Phil Berger and Laurene Levy (deceased), USDA, APHIS for valuable consultation and support. We thank Greg Para, USDA, APHIS, for administrative oversight and support. Finally we thank Len Therrien, Greg Brock, Joann Hodge, and Leigh Sitler for technical assistance.

**Conflicts of Interest:** The authors declare no conflict of interest. In addition, the funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

## **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
