Next Article in Journal
Comparative Animal Models of Human Viral Infections
Next Article in Special Issue
Echinococcus multilocularis and Other Taeniid Metacestodes of Muskrats in Luxembourg: Prevalence, Risk Factors, Parasite Reproduction, and Genetic Diversity
Previous Article in Journal
The Prevalence and Control of Lungworms of Pastoral Ruminants in Iran
Previous Article in Special Issue
Infection with a Recently Discovered Gammaherpesvirus Variant in European Badgers, Meles meles, is Associated with Higher Relative Viral Loads in Blood
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Pathogens
Volume 11
Issue 12
10.3390/pathogens11121393
pathogens-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

High-Specificity Test Algorithm for Bovine Tuberculosis Diagnosis in African Buffalo (Syncerus caffer) Herds

by
Charlene Clarke
1,
Netanya Bernitz
1,2,
Wynand J. Goosen
1,* and
Michele A. Miller
1
1
DSI-NRF Centre of Excellence for Biomedical Tuberculosis Research, SAMRC for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town 7505, South Africa
2
Faculty of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1 7HT, UK
*
Author to whom correspondence should be addressed.
Pathogens 2022, 11(12), 1393; https://doi.org/10.3390/pathogens11121393
Submission received: 21 October 2022 / Revised: 11 November 2022 / Accepted: 21 November 2022 / Published: 22 November 2022
(This article belongs to the Special Issue Emerging and Endemic Infections in Wildlife: Epidemiology, Ecology, & Management in a Changing World)
Browse Figure
Review Reports Versions Notes

Abstract

:
Ante-mortem bovine tuberculosis (bTB) tests for buffaloes include the single comparative intradermal tuberculin test (SCITT), interferon-gamma (IFN-γ) release assay (IGRA) and IFN-γ-inducible protein 10 release assay (IPRA). Although parallel test interpretation increases the detection of Mycobacterium bovis (M. bovis)-infected buffaloes, these algorithms may not be suitable for screening buffaloes in historically bTB-free herds. In this study, the specificities of three assays were determined using M. bovis-unexposed herds, historically negative, and a high-specificity diagnostic algorithm was developed. Serial test interpretation (positive on both) using the IGRA and IPRA showed significantly greater specificity (98.3%) than individual (90.4% and 80.9%, respectively) tests or parallel testing (73%). When the SCITT was added, the algorithm had 100% specificity. Since the cytokine assays had imperfect specificity, potential cross-reactivity with nontuberculous mycobacteria (NTM) was investigated. No association was found between NTM presence (in oronasal swab cultures) and positive cytokine assay results. As a proof-of-principle, serial testing was applied to buffaloes (n = 153) in a historically bTB-free herd. Buffaloes positive on a single test (n = 28) were regarded as test-negative. Four buffaloes were positive on IGRA and IPRA, and M. bovis infection was confirmed by culture. These results demonstrate the value of using IGRA and IPRA in series to screen buffalo herds with no previous history of M. bovis infection.

1. Introduction

African buffaloes (Syncerus caffer) are important maintenance hosts of bovine tuberculosis (bTB) in South Africa [1]. The primary causative agent of bTB in buffaloes is Mycobacterium bovis (M. bovis), a pathogenic member of the Mycobacterium tuberculosis complex (MTBC) [2]. However, other members of the MTBC have also been shown to cause TB in these bovids, including M. orygis [3] and M. tuberculosis [4]. Control measures in South Africa require mandatory screening for M. bovis infection prior to the movement of buffaloes [1]. Since test-positive animals are usually culled, and the farm placed under quarantine until evidence can be provided that M. bovis has been eradicated from the herd [1], it is crucial to use accurate diagnostic tests to identify all infected animals without the unnecessary culling of false-positive buffaloes [5]. A herd is considered eradicated from disease if all animals in the herd have two consecutive negative tuberculin skin test (TST) results [6,7].
To understand the diagnostic accuracy of a test, it is important to assess sensitivity and specificity, i.e., the ability to correctly distinguish truly infected or uninfected animals [8]. Ideally, these parameters should be close to 100% for optimal diagnostic performance, but with an inverse relationship between sensitivity and specificity for an imperfect test, increasing test sensitivity results in compromised specificity, and vice versa [8]. Various factors influence bTB test sensitivity and specificity when using methods based on immune responses, including the test antigens selected for the assay, exposure of the host to environmental mycobacteria, and the specific host biomarkers measured in cytokine release assays (CRA) [2,8].
Ante-mortem diagnosis of M. bovis infection in buffaloes relies on the measurement of antigen-specific cell-mediated host-immune responses [2,9]. These include the single comparative intradermal tuberculin test (SCITT), QuantiFERON®-TB Gold Plus (QFT) or Bovigam® interferon-gamma (IFN-γ) release assays (IGRA) and QFT IFN-γ inducible protein 10 release assay (IPRA) [10]. A number of QFT IGRAs have been described for bTB diagnosis in African buffaloes, including an in-house bovine IFN-γ enzyme-linked immunosorbent assay (ELISA) [11], the commercially available Mabtech bovine IFN-γ ELISA PRO [12] and Cattletype® IFN-γ ELISA (recently discontinued by manufacturer) [13]. Serological assays, which measure humoral immune responses to M. bovis antigens, have been shown to have low sensitivity for the early immunological detection of M. bovis infection in buffaloes [10,14]. In bovids, these assays are more likely to be positive in animals with disease; therefore, bTB detection in buffaloes relies on primarily CMI assays for early detection of infection [10,14].
The TST and Bovigam® IGRA measure host responses to purified protein derivatives (PPD), which are antigens consisting of a crude mixture of proteins, carbohydrates and lipids obtained from M. bovis (bovine PPD), M. avium (avian PPD), and in some cases from M. fortuitum (fortuitum PPD) [2,8,15]. Assays using PPDs typically have high sensitivity due to the greater number of antigens used to stimulate a detectable immune response, but often have compromised specificity due to cross-reactive responses to environmental mycobacteria that share antigens with MTBC [2,9,16]. The single intradermal test measures the host response to bovine PPD. However, the inclusion of avian PPD in the SCITT has led to increased specificity by accounting for sensitisation to M. avium and closely related nontuberculous mycobacterial (NTM) species [17]. In this test, the host immune response to bovine PPD is measured as a change in skin-fold thickness at 72 hours. This is compared to the change in skin-fold thickness in response to avian PPD. The difference in these results is then used to determine the test result of the buffalo [16]. Similarly, the inclusion of fortuitum PPD has also been shown to increase test specificity in PPD-based IGRAs [15]. Specificity can be further improved by using MTBC specific antigens, early secretory antigen 6 kDa (ESAT-6) and culture filtrate protein 10 (CFP-10), in the stimulation platform for CRAs [9,13]. This, however, comes at the expense of test sensitivity [13]. The inclusion of additional biomarkers, such as IP-10, which is released in significantly greater concentrations than IFN-γ, has been shown to improve sensitivity [13]. However, genes encoding for ESAT-6 and CFP-10 antigens are shared with some NTM species, leading to cross-reactive immunological responses and false-positive IGRA and IPRA results [17,18]. Therefore, test interpretation requires knowledge of potential confounders, such as NTM exposure of the host.
Although current guidelines for testing buffaloes in South Africa rely on the SCITT, the use of parallel testing (i.e., test-positive if positive on any bTB test included in the test algorithm) has been shown to improve the detection of infected animals [8,13,19]. With increased sensitivity, there is a trade-off in specificity, resulting in the culling of some false-positive buffaloes, which is of lesser concern when attempting to eradicate bTB in a herd [13]. However, this approach is less suitable for bTB testing of historically bTB-free African buffalo herds, especially those with high economic or genetic value since each false-positive buffalo that is culled leads to significant loss. In these scenarios, a testing algorithm with high specificity is needed to minimise unnecessary culling. However, this fit-for-purpose high-specificity algorithm has not been evaluated in African buffaloes. Previous studies, to determine bTB test specificity values, used buffaloes that were M. bovis culture-negative, but from exposed herds (bTB endemic herds); which may result in an inaccurate estimate of specificity [19]. Therefore, this study aimed to (1) determine individual specificities of the QFT Mabtech IGRA, QFT IPRA, and SCITT in known M. bovis unexposed African buffaloes, (2) evaluate test algorithms to optimise specificity in these M. bovis unexposed buffalo herds, (3) determine if there is an association between false-positive bTB test results in M. bovis unexposed herds and presence of NTMs, and (4) apply the high-specificity testing algorithm to an independent historically bTB-free herd as a proof of principle.

2. Materials and Methods

2.1. Animals and Sample Collection

Samples were collected between 2017 and 2021 from African buffaloes (n = 268) at historically bTB-free private game farms in South Africa (Limpopo, northwest and northern Cape provinces). These farms were regarded as historically bTB-free based on negative TST results during periodic testing. Any TST-positive buffaloes were culled, and mycobacterial culture and speciation performed on tissue samples. However, M. bovis has not been detected to date in these animals. Following the capture and immobilisation of buffaloes, whole blood was collected by jugular venipuncture into lithium heparin tubes, as previously described [11]. The SCITT was performed after blood collection. Within eight hours of sample collection, whole blood stimulation was performed using the QuantiFERON®-TB Gold Plus (QFT; Qiagen, Hilden, Germany) platform. Briefly, 1 mL heparinised whole blood was aliquoted into each of the QFT tubes, which included the antigen tube containing cluster of differentiation (CD) 4 and CD8 T-cells stimulating ESAT-6 and CFP-10 antigens (QFT TB2 tube), the negative control containing saline (QFT Nil), and a positive control (QFT mitogen) containing phytohaemagglutinin (PHA), which was additionally spiked with 10 µg/mL pokeweed mitogen (Sigma Aldrich, St. Louis, MO, USA) to ensure stimulation. The fourth tube (QFT TB1) only contains CD4 stimulating antigens. Following 100% agreement in test outcomes from TB1 and TB2 tubes in a previous study [19], the TB1 tube was not included for these buffaloes. After inverting all tubes 10 times, tubes were incubated at 37 °C for 24 h, prior to harvesting plasma fractions by centrifugation at 3000 rpm for 15 min.
Cohort 1 consisted of 115 buffaloes from three historically bTB-free farms tested in 2021. The QFT Mabtech IGRA and QFT IPRA were performed on all buffaloes, and SCITTs were performed on a subset (n = 69). Oronasal swabs were collected in saline from all buffaloes for mycobacterial culture. This cohort was used to calculate assay specificity.
Cohort 2 consisted of 153 buffaloes from one historically bTB-free farm (different location to Cohort 1). A single SCITT reactor buffalo bull was detected during bTB testing for movement for sale, according to regulations by the South African Department of Agriculture, Land Reform and Rural Development (DALRRD). Following this, the entire herd was tested in 2017, using QFT Cattletype® IGRA and QFT IPRA. The owner consented to have the buffaloes that had positive results on both assays culled, and tissue samples were collected during post-mortem examination for mycobacterial culture.
The Cattletype® IFN-γ ELISA was discontinued before Cohort 1 buffaloes were tested, and therefore the Mabtech IFN-γ ELISA was used. A comparison of test performance between these assays showed no significant differences [12]. Therefore, the assay specificities calculated using results from Cohort 1 buffaloes were used to retrospectively evaluate the high-specificity test algorithm using results from Cohort 2 buffaloes.

2.2. Single Comparative Intradermal Tuberculin Tests

Single comparative intradermal tuberculin tests (SCITT) were performed following immobilisation on a subset of Cohort 1 buffaloes (n = 69), and 1 buffalo from Cohort 2, as previously described [11]. A buffalo-specific cut-off value of 3 mm was used to identify SCITT-positive buffaloes [16].

2.3. Cytokine Release Assays

Cytokine concentrations in plasma from QFT-stimulated whole blood samples were measured with the bovine IP-10 ELISA (Kingfisher Biotech, St Paul, MN, USA) in all buffaloes, as previously described [20] and with commercially available bovine IFN-γ ELISAs (IGRA); the bovine IFN-γ ELISA Pro assay (Mabtech, Stockholm, Sweden) was used for Cohort 1 buffaloes, whereas the Cattletype® Ruminant IFN-γ ELISA (Indical Bioscience, Leipzig, Germany) was used with Cohort 2 animals, as previously described [12]. Mycobacterial antigen-specific cytokine concentrations were determined by subtracting QFT Nil cytokine concentrations from QFT TB2 cytokine concentrations. Buffalo-specific assay cut-off values have previously been described; buffaloes were regarded as IPRA-positive if the antigen-specific IPRA result was ≥ 1486 pg/ml [20], Mabtech IGRA-positive if the concentration was ≥8 pg/ml and Cattletype® IGRA-positive if the relative signal to positive ratio (S/P%) was ≥5% [12].

2.4. Bovine TB Test Interpretation

The parallel interpretation of the IGRA and IPRA was considered test-positive if the result was positive for either IGRA or IPRA (negative if both IGRA and IPRA negative), whereas a positive serial test result was assigned if both IGRA and IPRA were positive, or positive if all three tests were positive (based on IGRA, IPRA and SCITT).

2.5. Mycobacterial Culture and Speciation

Oronasal swab samples (Cohort 1) and tissue samples from culled buffaloes (Cohort 2) were processed for mycobacterial culture in the biosafety level 3 (BSL-3) laboratory at Stellenbosch University. A 500 µL aliquot of each oronasal swab saline media, or approximately 1 cm3 of homogenised tissue (head, peripheral, abdominal and thoracic lymph nodes, lung) was cultured using the BACTECTM MGITTM 960 TB System (BD, Franklin Lakes, NJ, USA), as previously described [21,22]. Samples were regarded as culture-negative if no growth was detected after 56 days of incubation; no downstream analysis was performed on these samples. Culture-positive samples were subjected to inactivation at 99 °C for 30 min before removal from the BSL-3 facility for speciation.
Mycobacterial speciation was based on polymerase chain reaction (PCR) amplicon sequences. Oronasal swab cultures from buffaloes in Cohort 1 were screened for highly conserved RNA polymerase beta subunit (rpoB) and heat-shock protein 65 kDa (hsp65) gene targets, using previously described primers for amplification [23,24] in a VeritiTM 96-Well Thermal Cycler (Applied Biosystems, Waltham, MA, USA). The preparation of PCRs, cycling conditions, and gel electrophoresis were performed as previously described [22]. All PCR amplicons that showed bands on gel electrophoresis were sent to the Central Analytical Facility (CAF), Stellenbosch University, for Sanger sequencing (Illumina, Inc., San Diego, CA, USA). BioEdit Biological Sequence Alignment Editor (BioEdit (RRID:SCR_007361) was used to generate sequence contigs, which were blasted on NCBI Basic Local Alignment Search Tool for nucleotides (BLASTn) to find sequence matches in the NCBI database. The presence of mycobacterial species was confirmed if sequences from either hsp65 or rpoB matched with mycobacterial species in the NCBI database, with an identity match ≥90%. The NTM species with the highest percentage match was selected as the result for each buffalo. Mycobacterium avium complex (MAC) classification was followed as described by To et al. [25]. Mycobacterial cultures from tissue samples collected from culled Cohort 2 buffaloes during post-mortem examination were speciated by region of difference (RD) PCR, as described by Warren et al. [26].

2.6. Data Analyses

Buffaloes were considered positive if positive on either IGRA or IPRA, using the parallel test interpretation, whereas serial test interpretation regarded buffaloes as positive if positive on IGRA, IPRA and SCITT (if included). The specificities for individual tests and test combinations of the Mabtech IGRA, IPRA, and SCITT (in series and in parallel) were calculated based on the percentage of M. bovis unexposed buffaloes from historically bTB-free herds (Cohort 1) that were test-negative on each test, using an online tool (MedCalc Software Ltd 2022; Ostend, Belgium). Since buffaloes originated from historically bTB-free herds, animals were regarded as true-negative if they were test-negative (on IGRA, IPRA or SCITT), and false-positive if test-positive (on IGRA, IPRA or SCITT). Proportions of test-positive and -negative buffaloes were compared using the exact binomial McNemar’s test with Bonferroni correction for multiple comparisons (GraphPad Software, Inc., 2022; online calculator; https://www.graphpad.com/quickcalcs/mcNemar2/; San Diego, CA, USA). Contingency tables were used to determine the association between NTM presence or absence in oronasal swab cultures and the Mabtech IGRA, IPRA and SCITT test results (Cohort 1), using the Fisher’s exact test. Differences in cytokine responses between animals with NTMs present or absent were compared using the nonparametric Mann-Whitney U test, following the D’Agostino–Pearson normality test. Results were considered statistically significant if p < 0.05. All statistical tests were performed on GraphPad Prism version 7 software (GraphPad Software Inc., San Diego, CA, USA).

3. Results

Specificity values were calculated for three assays (Mabtech IGRA, IPRA and SCITT), individually and applied in parallel and in series (Table 1), using blood test results from African buffaloes on historically bTB-free farms (Cohort 1). The SCITT (individually and in series with IGRA and IPRA) identified all 69 buffaloes as uninfected, with a calculated specificity of 100%. The specificities of the individual CRAs (Mabtech IGRA and IPRA) were 90.4% and 80.9%, respectively, although the difference was not statistically significant (p > 0.06). Parallel interpretation of the Mabtech IGRA and IPRA resulted in a decrease in specificity, to 73.0%. When a serial interpretation was applied to Mabtech IGRA and IPRA results, there was high specificity, at 98.3%, which was significantly greater than the specificities of the individual (p < 0.008) or parallel interpretations (p < 0.0001) of the cytokine release assays.
Mycobacterial culture results of oronasal swabs from buffaloes in Cohort 1 (n = 115) were used to assess an association between Mabtech IGRA and IPRA results and the presence of mycobacteria. No MTBC organisms were detected; however, NTMs were identified in 69 buffalo oronasal swabs, with 34 identified to species level based on hsp65 and rpoB amplicon Sanger sequencing and the other 35 samples identified as belonging to the Mycobacterium genus. There were no significant associations between NTM presence or absence and Mabtech IGRA-positive or -negative results (p = 0.5), or IPRA-positive or -negative results (p = 0.15), using the Fisher’s exact test (Table 2). There were also no significant differences in antigen-specific IFN-γ (p = 0.51) or IP-10 (p = 0.32) concentrations in buffaloes with NTMs present compared to those animals which did not have NTMs detected (Figure 1).
Since NTM species could be identified in 34 buffalo oronasal swabs, these were grouped according to whether the species had been shown to contain the RD1 region [17,27]. The presence of RD1-containing NTM species was not associated with a positive Mabtech IGRA or IPRA result in these buffaloes (p > 0.05; data not shown). The largest number of NTM species matches were Mycobacterium avium complex (MAC; 17 matches). However, there was also no association between the presence of MAC and a positive IPRA or IGRA result (p > 0.05; data not shown).
Based on the high specificity of the combined Mabtech IGRA and IPRA when used in series, this algorithm was selected for retrospective evaluation of results from an unrelated historically bTB-free buffalo herd (n = 153; Cohort 2). Although the Cattletype® IGRA was used for Cohort 2 buffaloes, Cattletype® and Mabtech IGRAs had similar test performances [12]. A total of 28 of 153 buffaloes (18%) had positive IPRA results but were negative on Cattletype® IGRA. Using the serial interpretation, these animals were consdered test-negative. Four buffaloes (3%) were positive for both Cattletype® IGRA and IPRA and considered test-positive. These four individuals had been culled due to suspicion of being infected, and post-mortem examinations were performed. All four buffaloes were confirmed to be infected with M. bovis based on mycobacterial culture of tissues and speciation by RD PCR. The remaining 124 buffaloes had negative results for both Cattletype® IGRA and IPRA.

4. Discussion

In this study, serial testing with Mabtech IGRA and IPRA (with or without SCITT included) showed high specificity compared to individual test specificity or parallel interpretation of test results when used in M. bovis unexposed African buffaloes. As a proof of principle, this high-specificity testing algorithm, i.e., serial interpretation of IGRA and IPRA results, was applied to an independent historically bTB-free buffalo herd. The four buffaloes that were test-positive (positive on both Cattletype® IGRA and IPRA) were confirmed M. bovis-infected based on mycobacterial culture and speciation, demonstrating that CRA serial test interpretation may have promise for screening buffaloes with no previous history of bTB, since no false test-positive buffaloes were culled. Following the detection of CRA reactor animals in Cohort 1, the presence of mycobacteria and its association with positive CRA results were investigated. Although only NTMs were detected in oronasal secretions, their presence showed no clear association with a positive CRA result.
The diagnostic accuracy of an assay is primarily defined by its specificity and sensitivity [8]. These parameters are often considered constant across different populations [8]. However, sensitivity and specificity may be influenced by various factors, including infection stage and disease severity, and exposure to environmental mycobacteria [2,8]. Furthermore, the infection status of the herd is an important consideration for the optimal application of a diagnostic assay or algorithms. Buffalo herds that are endemic for bTB usually require tests with high sensitivity, since the aim is to identify as many M. bovis-infected buffaloes as possible, at the expense of culling some false-positive animals [13]. A herd that is historically bTB-free or has a low bTB prevalence requires a test with high specificity, which will correctly identify M. bovis-uninfected buffaloes while minimising the removal of false-positive animals [8,28].
In this study, we aimed to determine which individual or combination of tests would have the highest specificity for screening buffaloes from historically bTB-free farms. The true specificity of bTB tests in buffaloes is often not accurately calculated due to a lack of samples from truly M. bovis-uninfected, -unexposed buffalo herds, especially based on the gold standard of mycobacterial culture of tissue [12]. Therefore, specificity is sometimes determined using M. bovis-uninfected, but -exposed animals [13]. However, mycobacterial culture is an imperfect gold-standard test, since some M. bovis-infected buffaloes may have negative culture results, thereby resulting in a decreased assay specificity [12,13].
The specificity values of the Mabtech IGRA, IPRA, and SCITT were calculated in a cohort of buffaloes that had no history of exposure to M. bovis, based on previous herd test results. Buffaloes from these herds had undergone sporadic culling and no M. bovis had been detected by mycobacterial culture of tissue samples [22]. Although ante-mortem culture has limited sensitivity, no MTBC members were detected in the oronasal secretions of the unexposed buffaloes used to calculate specificity. In these buffaloes, the Mabtech IGRA had a higher (but not significantly) specificity (90.4%) than IPRA (80.9%). This result was expected since IP-10 is known to be a more sensitive cytokine biomarker compared to IFN-γ [13]. The specificity for IPRA in the current study was higher than that reported by Bernitz et al. [19] at 67%. However, Bernitz et al. [19] determined specificity in M. bovis culture-negative buffaloes from a bTB-endemic herd. In a small group of M. bovis-unexposed buffaloes, IPRA specificity was calculated as 100% [13], and Mabtech IGRA and Cattletype® IGRA as 94% and 97%, respectively [12]. Surprisingly, in the current study, SCITT had the highest specificity at 100%. Previous studies have shown that the SCITT has decreased specificity when compared to assays using ESAT-6 and CFP-10 antigens [2,16]. Parallel interpretation of the Mabtech IGRA and IPRA had a lower specificity (73%) compared to specificities of the individual tests, although it was higher than that previously reported for buffaloes (63% using the Cattletype® IGRA and IPRA), which calculated specificity based on results from culture-negative buffaloes in a bTB endemic herd [19]. As expected, this value was lower compared to specificity previously calculated from unexposed herds (Cattletype® IGRA and IPRA) at 100% [13]. Serial testing with Mabtech IGRA and IPRA showed significantly improved specificity at 98.3% (or 100% when SCITT was included) over individual CRAs and parallel testing and was therefore selected as the optimal high-specificity algorithm. Although SCITT had the highest specificity at 100%, whereas serial testing with only CRAs was 98.3%, it was decided not to include this test in further algorithm evaluations for various reasons. The CRAs provide a cost-and-welfare advantage over SCITT since a single immobilisation is required for blood collection, whereas SCITT requires that buffaloes are immobilised twice and are usually held captive for the three days in between immobilisation events [10,13]. In addition, CRAs allow the retesting of animals without a waiting period between bTB testing, whereas SCITT requires a waiting period between tests to avoid in vivo sensitisation [2,13]. Finally, SCITTs may be subjective to operator bias [8,13].
Some Mabtech IGRA or IPRA reactors were detected in the buffaloes used for the specificity calculations (Cohort 1), which resulted in decreased assay specificity. This decreased test specificity has been hypothesised to be due to cross-reactivity with environmental mycobacteria [17]. In the current study, oronasal swab culture results were used as a proxy for NTM exposure, based on mycobacterial culture and hsp65 or rpoB amplicon Sanger sequencing [22]. Members of the MTBC were not detected in any buffalo oronasal swab cultures. However, NTMs were detected in 60% of the buffaloes. Various studies have found that NTMs may impede diagnostic performance through cross-reactive immune responses to antigens shared between MTBC members and NTMs [9,17]. However, no association was found between the presence of NTMs and positive Mabtech IGRA or IPRA results. Nontuberculous mycobacteria that were previously described as containing the RD1 region, a genetic region containing virulent factors esat-6 and cfp-10 and associated with virulent MTBC members as well as some NTM species [9,29], showed no association with CRA results. The presence of Mycobacterium avium complex (MAC) members also did not appear to influence CRA results. Positive CRA results could also have been an indication of M. bovis infection, although MTBC was not detected in oronasal swab cultures. However, since oronasal swab cultures indicate the presence of NTMs in the oronasal cavities of the animals, it may not represent NTM infection. Other studies have also reported possible cross-reactivity with organisms other than NTMs, such as Nocardia, Rhodococcus and Corynebacterium, which are closely related to the Mycobacterium genus [2,30,31]. Therefore, the cause of false-positive CRA results should be investigated in future studies.
As a proof-of-principle, the high-specificity test algorithm with CRAs showed good diagnostic performance when applied to a historically bTB-free buffalo herd (Cohort 2). Although the specificity calculation for QFT IGRA was performed using the Mabtech bovine IFN-γ ELISA Pro assay, the buffaloes in the test group used the Cattletype® Ruminant IGRA. However, no significant differences in test performance between these assays have been reported [12]. The Cattletype® IGRA was recently discontinued by the manufacturer, and therefore Mabtech IGRA was performed for Cohort 1 buffaloes (Cohort 2 buffaloes were tested before Cohort 1 animals). Even though a strict interpretation was used to define bTB test-positive individuals, four positive buffaloes were identified, which were confirmed to have M. bovis infection. Although some buffaloes were positive on IPRA, but not Cattletype® IGRA, it is possible that these animals had early infection, since IP-10 is released at greater concentrations than IFN-γ [13,20]. However, infection could not be confirmed since these animals were not culled, due to the absence of a compensation program, and the herd was lost for follow-up. In this scenario, it is recommended that these buffaloes be quarantined and retested since it is expected that truly infected animals will eventually become IGRA-positive. This example demonstrates the importance of having a herd history for M. bovis infection to determine whether a high specificity or high sensitivity testing approach should be used [8,13].
Limitations of this study included the absence of gold-standard confirmation of the infection status of the buffaloes tested, using mycobacterial culture from tissue samples, especially those with a positive CRA result. Additionally, the use of oronasal swab cultures could not distinguish between NTM infection or contamination, which could impact whether the buffaloes developed immune responses to NTMs [32]. Due to availability, the bovine IFN-γ ELISAs differed between the buffalo herds used to calculate specificity (Mabtech IGRA) and the test group (Cattletype® IGRA).
In summary, this study calculated the specificities of Mabtech IGRA, IPRA, and SCITT as individual tests, interpreted in parallel and in series in known M. bovis unexposed buffaloes, which facilitated the development of a high-specificity test algorithm for use in bTB-free herds. Even in the presence of NTMs, the use of the IGRA and IPRA (with or without SCITT) in series yielded a high-specificity value, and accurately identified true M. bovis-infected buffaloes without culling high-value uninfected animals. These findings demonstrate that a serial test algorithm may have promise for screening buffalo herds with no previous evidence of bTB and should be assessed on a larger scale.

Author Contributions

C.C., M.A.M., N.B. and W.J.G. conceived the experiments. C.C. and N.B. conducted the experiments. C.C., W.J.G. and M.A.M. performed sample collection and processing. M.A.M. and W.J.G. provided funding. C.C., M.A.M., W.J.G. and N.B. were involved in data analyses and reviewed the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by (1) the Harry Crossley Foundation, (2) South African government through the South African Medical Research Council and the National Research Foundation South African Research Chair Initiative (Grant #86949), (3) Wellcome Foundation (Grant # 222941/Z/21/Z) and (4) Stellenbosch University Postgraduate Scholarship Programme. The content is the sole responsibility of the authors and does not necessarily represent the official views of the funders.

Institutional Review Board Statement

Ethical approval for sample collection was granted by Stellenbosch University Animal Care and Use Committee (ACU-2019-9081, ACU-2019-9086 and SU-ACUM13-00016). Permission to perform animal research in terms of Section 20 of the Animal Diseases Act was granted by DALRRD, formerly the Department of Agriculture, Forestry and Fisheries (DAFF), South Africa (12/11/1/7/2). All buffaloes were handled by private wildlife veterinarians and game capture teams according to their guidelines. ARRIVE guidelines for reporting animal research have been followed as much as possible (https://arriveguidelines.org/).

Data Availability Statement

Correspondence and requests for materials should be addressed to Dr. Wynand Goosen.

Acknowledgments

The authors would like to thank Li-Ann Grobler, Angela Brüns, Peter Rogers, Manie du Plessis, Peter Buss, Guy Hausler, and the game capture teams for the buffalo immobilisation, sample collection and sample processing.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Arnot, L.F.; Michel, A. Challenges for Controlling Bovine Tuberculosis in South Africa. Onderstepoort J. Vet. Res. 2020, 87, e1–e8. [Google Scholar] [CrossRef] [PubMed]
  2. Borham, M.; Oreiby, A.; El-Gedawy, A.; Hegazy, Y.; Khalifa, H.O.; Al-Gaabary, M.; Matsumoto, T. Review on Bovine Tuberculosis: An Emerging Disease Associated with Multidrug-Resistant Mycobacterium Species. Pathogens 2022, 11, 715. [Google Scholar] [CrossRef] [PubMed]
  3. Van Pittius, N.C.G.; Perrett, K.D.; Michel, A.L.; Keet, D.F.; Hlokwe, T.; Streicher, E.M.; Warren, R.M.; van Helden, P.D. Infection of African Buffalo (Syncerus caffer) by Oryx Bacillus, a Rare Member of the Antelope Clade of the Mycobacterium tuberculosis Complex. J. Wildl. Dis. 2012, 48, 849–857. [Google Scholar] [CrossRef] [PubMed]
  4. Borham, M.; Oreiby, A.; El-Gedawy, A.; Hegazy, Y.; Hemedan, A.; Al-Gaabary, M. Abattoir Survey of Bovine Tuberculosis in Tanta, Centre of the Nile Delta, with in Silico Analysis of Gene Mutations and Protein–Protein Interactions of the Involved Mycobacteria. Transbound. Emerg. Dis. 2022, 69, 434–450. [Google Scholar] [CrossRef]
  5. De Vos, V.; Bengis, R.G.; Kriek, N.P.J.; Michel, A.; Keet, D.F.; Raath, J.P.; Huchzermeyer, H.F.K.A. The Epidemiology of Tuberculosis in Free-Ranging African Buffalo (Syncerus caffer) in the Kruger National Park, South Africa. Onderstepoort J. Vet. Res. 2001, 68, 119–130. [Google Scholar]
  6. Department of Agriculture, Forestry and Fisheries (DAFF). Bovine Tuberculosis Manual 2016. Available online: https://www.nda.agric.za/vetweb/pamphlets&Information/Policy/Tuberculosis%20in%20Cattle%20Interim%20Manual%20for%20the%20Veterinarian%20&%20AHT%20-%20Sept2....pdf (accessed on 7 October 2022).
  7. Department of Agriculture, Land Reform and Rural Development (DALRRD). Veterinary Procedudaral Notice for Buffalo Disease Risk Management in South Africa 2017. Available online: https://www.dalrrd.gov.za/vetweb/pamphlets&Information/Policy/Buffalo%20Disease%20Risk%20Management%20VPN_Signed%202017-02-17.pdf (accessed on 5 October 2022).
  8. de la Rua-Domenech, R.; Goodchild, A.T.; Vordermeier, H.M.; Hewinson, R.G.; Christiansen, K.H.; Clifton-Hadley, R.S. Ante Mortem Diagnosis of Tuberculosis in Cattle: A Review of the Tuberculin Tests, γ-Interferon Assay and Other Ancillary Diagnostic Techniques. Res. Vet. Sci. 2006, 81, 190–210. [Google Scholar] [CrossRef]
  9. Vordermeier, H.M.; Brown, J.; Cockle, P.J.; Franken, W.P.J.; Arend, S.M.; Ottenhoff, T.H.M.; Jahans, K.; Hewinson, R.G. Assessment of Cross-Reactivity between Mycobacterium bovis and M. kansasii ESAT-6 and CFP-10 at the T-Cell Epitope Level. Clin. Vaccine Immunol. 2007, 14, 1203–1209. [Google Scholar] [CrossRef] [Green Version]
  10. Bernitz, N.; Kerr, T.; Goosen, W.; Chileshe, J.; Higgit, R.; Roos, E.; Meiring, C.; Gumbo, R.; de Waal, C.; Clarke, C.; et al. Review of Diagnostic Tests for Detection of Mycobacterium bovis Infection in South African Wildlife. Front. Vet. Sci. 2021, 8, 588697. [Google Scholar] [CrossRef]
  11. Parsons, S.D.C.; Cooper, D.; McCall, A.J.; McCall, W.A.; Streicher, E.M.; le Maitre, N.C.; Müller, A.; Gey van Pittius, N.C.; Warren, R.M.; van Helden, P.D. Modification of the QuantiFERON-TB Gold (In-Tube) Assay for the Diagnosis of Mycobacterium bovis Infection in African Buffaloes (Syncerus caffer). Vet. Immunol. Immunopathol. 2011, 142, 113–118. [Google Scholar] [CrossRef]
  12. Smith, K.; Bernitz, N.; Goldswain, S.; Cooper, D.V.; Warren, R.M.; Goosen, W.J.; Miller, M.A. Optimized Interferon-Gamma Release Assays for Detection of Mycobacterium bovis Infection in African Buffaloes (Syncerus caffer). Vet. Immunol. Immunopathol. 2021, 231, 110163. [Google Scholar] [CrossRef]
  13. Bernitz, N.; Kerr, T.J.; Goosen, W.J.; Clarke, C.; Higgitt, R.; Roos, E.O.; Cooper, D.V.; Warren, R.M.; van Helden, P.D.; Parsons, S.D.C.; et al. Parallel Measurement of IFN-γ and IP-10 in QuantiFERON®-TB Gold (QFT) Plasma Improves the Detection of Mycobacterium bovis Infection in African Buffaloes (Syncerus caffer). Prev. Vet. Med. 2019, 169, 104700. [Google Scholar] [CrossRef] [PubMed]
  14. Lyashchenko, K.P.; Sridhara, A.A.; Johnathan-Lee, A.; Sikar-Gang, A.; Lambotte, P.; Esfandiari, J.; Bernitz, N.; Kerr, T.J.; Miller, M.A.; Waters, W.R. Differential Antigen Recognition by Serum Antibodies from Three Bovid Hosts of Mycobacterium bovis Infection. Comp. Immunol. Microbiol. Infect. Dis. 2020, 69, 101424. [Google Scholar] [CrossRef] [PubMed]
  15. Michel, A.L.; Cooper, D.; Jooste, J.; de Klerk, L.M.; Jolles, A. Approaches Towards Optimising the Gamma Interferon Assay for Diagnosing Mycobacterium bovis Infection in African Buffalo (Syncerus caffer). Prev. Vet. Med. 2011, 98, 142–151. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Smith, K.; Bernitz, N.; Cooper, D.; Kerr, T.J.; de Waal, C.R.; Clarke, C.; Goldswain, S.; McCall, W.; McCall, A.; Cooke, D.; et al. Optimisation of the Tuberculin Skin Test for Detection of Mycobacterium bovis in African Buffaloes (Syncerus caffer). Prev. Vet. Med. 2021, 188, 105254. [Google Scholar] [CrossRef] [PubMed]
  17. Jenkins, A.O.; Gormley, E.; Gcebe, N.; Fosgate, G.T.; Conan, A.; Aagaard, C.; Michel, A.L.; Rutten, V.P.M.G. Cross Reactive Immune Responses in Cattle Arising from Exposure to Mycobacterium bovis and Non-Tuberculous Mycobacteria. Prev. Vet. Med. 2018, 152, 16–22. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Varela-Castro, L.; Barral, M.; Arnal, M.C.; Fernández de Luco, D.; Gortázar, C.; Garrido, J.M.; Sevilla, I.A. Beyond Tuberculosis: Diversity and Implications of Non-tuberculous Mycobacteria at the Wildlife-Livestock Interface. Transbound. Emerg. Dis. 2022, 69, e2978–e2993. [Google Scholar] [CrossRef] [PubMed]
  19. Bernitz, N.; Kerr, T.J.; de Waal, C.; Cooper, D.V.; Warren, R.M.; Van Helden, P.D.; Parsons, S.D.C.; Miller, M.A. Test Characteristics of Assays to Detect Mycobacterium bovis Infection in High-Prevalence African Buffalo (Syncerus caffer) Herds. J. Wildl. Dis. 2020, 56, 462–465. [Google Scholar] [CrossRef]
  20. Goosen, W.J.; Cooper, D.; Miller, M.A.; Van Helden, P.D.; Parsons, S.D.C. IP-10 Is a Sensitive Biomarker of Antigen Recognition in Whole-Blood Stimulation Assays Used for the Diagnosis of Mycobacterium bovis Infection in African Buffaloes (Syncerus caffer). Clin. Vaccine Immunol. 2015, 22, 974–978. [Google Scholar] [CrossRef] [Green Version]
  21. Goosen, W.J.; Miller, M.A.; Chegou, N.N.; Cooper, D.; Warren, R.M.; van Helden, P.D.; Parsons, S.D.C. Agreement between Assays of Cell-Mediated Immunity Utilizing Mycobacterium bovis-Specific Antigens for the Diagnosis of Tuberculosis in African Buffaloes (Syncerus caffer). Vet. Immunol. Immunopathol. 2014, 160, 133–138. [Google Scholar] [CrossRef]
  22. Clarke, C.; Kerr, T.J.; Warren, R.M.; Kleynhans, L.; Miller, M.A.; Goosen, W.J. Identification and Characterisation of Nontuberculous Mycobacteria in African Buffaloes (Syncerus caffer), South Africa. Microorganisms 2022, 10, 1861. [Google Scholar] [CrossRef]
  23. Adékambi, T.; Colson, P.; Drancourt, M. RpoB-Based Identification of Nonpigmented and Late-Pigmenting Rapidly Growing Mycobacteria. J. Clin. Microbiol. 2003, 41, 5699–5708. [Google Scholar] [CrossRef] [PubMed]
  24. Telenti, A.; Marchesi, F.; Balz, M.; Bally, F.; Bottger, E.C.; Bodmer, T. Rapid Identification of Mycobacteria to the Species Level by Polymerase Chain Reaction and Restriction Enzyme Analysis. J. Clin. Microbiol. 1993, 31, 175–178. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. To, K.; Cao, R.; Yegiazaryan, A.; Owens, J.; Venketaraman, V. General Overview of Nontuberculous Mycobacteria Opportunistic Pathogens: Mycobacterium avium and Mycobacterium abscessus. J. Clin. Med. 2020, 9, 2541. [Google Scholar] [CrossRef] [PubMed]
  26. Warren, R.M.; Gey Van Pittius, N.C.; Barnard, M.; Hesseling, A.; Engelke, E.; De Kock, M.; Gutierrez, M.C.; Chege, G.K.; Victor, T.C.; Hoal, E.G.; et al. Differentiation of Mycobacterium tuberculosis Complex by PCR Amplification of Genomic Regions of Difference. Int. J. Tuberc. Lung Dis. 2006, 10, 818–822. [Google Scholar]
  27. Scherrer, S.; Landolt, P.; Friedel, U.; Stephan, R. Distribution and Expression of esat-6 and cfp-10 in Non-Tuberculous Mycobacteria Isolated from Lymph Nodes of Slaughtered Cattle in Switzerland. J. Vet. Diagn. Investig. 2019, 31, 217–221. [Google Scholar] [CrossRef] [Green Version]
  28. Bernitz, N.; Goosen, W.J.; Clarke, C.; Kerr, T.J.; Higgitt, R.; Roos, E.O.; Cooper, D.V.; Warren, R.M.; van Helden, P.D.; Parsons, S.D.C.; et al. Parallel Testing Increases Detection of Mycobacterium bovis-Infected African Buffaloes (Syncerus caffer). Vet. Immunol. Immunopathol. 2018, 204, 40–43. [Google Scholar] [CrossRef]
  29. Van Ingen, J.; De Zwaan, R.; Dekhuijzen, R.; Boeree, M.; Van Soolingen, D. Region of Difference 1 in Nontuberculous Mycobacterium Species Adds a Phylogenetic and Taxonomical Character. J. Bacteriol. 2009, 191, 5865–5867. [Google Scholar] [CrossRef] [Green Version]
  30. Lin, M.Y.; Reddy, T.B.K.; Arend, S.M.; Friggen, A.H.; Franken, K.L.M.C.; van Meijgaarden, K.E.; Verduyn, M.J.C.; Schoolnik, G.K.; Klein, M.R.; Ottenhoff, T.H.M. Cross-Reactive Immunity to Mycobacterium tuberculosis DosR Regulon-Encoded Antigens in Individuals Infected with Environmental, Nontuberculous Mycobacteria. Infect. Immun. 2009, 77, 5071–5079. [Google Scholar] [CrossRef] [Green Version]
  31. Malama, S.; Munyeme, M.; Mwanza, S.; Muma, J.B. Isolation and Characterization of Nontuberculous Mycobacteria from Humans and Animals in Namwala District of Zambia. BMC Res. Notes 2014, 7, 622. [Google Scholar] [CrossRef] [Green Version]
  32. Hermansen, T.S.; Thomsen, V.Ø.; Lillebaek, T.; Ravn, P. Non-Tuberculous Mycobacteria and the Performance of Interferon Gamma Release Assays in Denmark. PLoS ONE 2014, 9, e93986. [Google Scholar] [CrossRef]
Pathogens 11 01393 g001 550
Figure 1. Concentrations of QuantiFERON®-TB Gold Plus (QFT) antigen-specific Mabtech interferon-gamma (IFN-γ, indicated by black squares) and interferon gamma inducible protein 10 (IP-10, indicated by black circles) in African buffalo samples from historically bovine tuberculosis-free farms (Cohort 1). Results are categorised based on whether nontuberculous mycobacteria (NTM) were present or absent in oronasal swab cultures, as determined by hsp65 or rpoB PCR amplicon Sanger sequencing. P-values for comparison of cytokine concentrations based on NTM presence or absence are indicated. Buffalo-specific cut-off values for Mabtech IGRA and IPRA are indicated as a dotted line or dashed line, respectively.
Figure 1. Concentrations of QuantiFERON®-TB Gold Plus (QFT) antigen-specific Mabtech interferon-gamma (IFN-γ, indicated by black squares) and interferon gamma inducible protein 10 (IP-10, indicated by black circles) in African buffalo samples from historically bovine tuberculosis-free farms (Cohort 1). Results are categorised based on whether nontuberculous mycobacteria (NTM) were present or absent in oronasal swab cultures, as determined by hsp65 or rpoB PCR amplicon Sanger sequencing. P-values for comparison of cytokine concentrations based on NTM presence or absence are indicated. Buffalo-specific cut-off values for Mabtech IGRA and IPRA are indicated as a dotted line or dashed line, respectively.
Pathogens 11 01393 g001
Table
Table 1. Calculated specificities of QuantiFERON®-TB Gold Plus interferon-gamma release assay (IGRA), interferon-gamma inducible protein 10 release assay (IPRA) and single comparative intradermal tuberculin test (SCITT) using African buffaloes from historically bovine tuberculosis-free farms (Cohort 1), based on individual, parallel and serial test interpretations. The numbers (and percentages) of false-positive and true-negative results for each test were based on previously determined buffalo-specific assay cut-off values.
Table 1. Calculated specificities of QuantiFERON®-TB Gold Plus interferon-gamma release assay (IGRA), interferon-gamma inducible protein 10 release assay (IPRA) and single comparative intradermal tuberculin test (SCITT) using African buffaloes from historically bovine tuberculosis-free farms (Cohort 1), based on individual, parallel and serial test interpretations. The numbers (and percentages) of false-positive and true-negative results for each test were based on previously determined buffalo-specific assay cut-off values.
TestFalse
Positive		True
Negative		Specificity (95% Confidence Interval)
IGRA a (n = 115)11 (9.6%)104 (90.4%)90.4% (83.5–95.1%)
IPRA b (n = 115)22 (19.1%)93 (80.9%)80.9% (72.5–87.6%)
SCITT (n = 69)0 (0%)69 (100%)100% (94.8–100%)
Parallel testing (IGRA a, IPRA b)31 (27%)84 (73%)73.0% (64.0–80.9%)
Serial testing (IGRA a, IPRA b)2 (1.7%)113 (98.3%)98.3% (93.9–99.8%)
Serial testing (IGRA a, IPRA b, SCITT)0 (0%)69 (100%)100% (94.8–100%)
a QuantiFERON®-TB Gold Plus stimulation platform with Mabtech bovine IFN-γ ELISA. b QuantiFERON®-TB Gold Plus stimulation platform with Kingfisher bovine IP-10 ELISA
Table
Table 2. The proportions (and percentages) of African buffaloes from historically bovine tuberculosis-free herds (Cohort 1) that had positive or negative test results using the QuantiFERON®-TB Gold Plus stimulation platform with cytokine release assays (interferon-gamma release assay (IGRA), interferon gamma inducible protein 10 release assay (IPRA)), categorised based on those that had nontuberculous mycobacteria (NTM) present or absent in oronasal swab cultures.
Table 2. The proportions (and percentages) of African buffaloes from historically bovine tuberculosis-free herds (Cohort 1) that had positive or negative test results using the QuantiFERON®-TB Gold Plus stimulation platform with cytokine release assays (interferon-gamma release assay (IGRA), interferon gamma inducible protein 10 release assay (IPRA)), categorised based on those that had nontuberculous mycobacteria (NTM) present or absent in oronasal swab cultures.
NTM PresentNTM Absentp-Value (Fisher’s
Exact Test)
IGRA a test-positive8/11 (73%)3/11 (27%)0.5
IGRA a test-negative61/104 (59%)43/104 (41%)
IPRA b test-positive10/22 (45%)12/22 (55%)0.15
IPRA b test-negative59/93 (63%)34/93 (37%)
a QuantiFERON®-TB Gold Plus stimulation platform with Mabtech bovine IFN-γ ELISA. b QuantiFERON®-TB Gold Plus stimulation platform with Kingfisher bovine IP-10 ELISA
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Clarke, C.; Bernitz, N.; Goosen, W.J.; Miller, M.A. High-Specificity Test Algorithm for Bovine Tuberculosis Diagnosis in African Buffalo (Syncerus caffer) Herds. Pathogens 2022, 11, 1393. https://doi.org/10.3390/pathogens11121393

AMA Style

Clarke C, Bernitz N, Goosen WJ, Miller MA. High-Specificity Test Algorithm for Bovine Tuberculosis Diagnosis in African Buffalo (Syncerus caffer) Herds. Pathogens. 2022; 11(12):1393. https://doi.org/10.3390/pathogens11121393

Chicago/Turabian Style

Clarke, Charlene, Netanya Bernitz, Wynand J. Goosen, and Michele A. Miller. 2022. "High-Specificity Test Algorithm for Bovine Tuberculosis Diagnosis in African Buffalo (Syncerus caffer) Herds" Pathogens 11, no. 12: 1393. https://doi.org/10.3390/pathogens11121393

APA Style

Clarke, C., Bernitz, N., Goosen, W. J., & Miller, M. A. (2022). High-Specificity Test Algorithm for Bovine Tuberculosis Diagnosis in African Buffalo (Syncerus caffer) Herds. Pathogens, 11(12), 1393. https://doi.org/10.3390/pathogens11121393

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop