Next Article in Journal / Special Issue
Impact of Intrinsic and Extrinsic Factors on the Pharmacokinetics of Peptides: When Is the Assessment of Certain Factors Warranted?
Previous Article in Journal / Special Issue
Allergen-Specific IgE and IgG4 as Biomarkers for Immunologic Changes during Subcutaneous Allergen Immunotherapy
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Commentary

Disease Prevalence Matters: Challenge for SARS-CoV-2 Testing

1
Department of Laboratory Medicine, Changi General Hospital, Singapore 529889, Singapore
2
Department of Medicine, National University of Singapore, Singapore 119077, Singapore
3
Academic Pathology Program, Duke-NUS Medical School, Singapore 169857, Singapore
*
Author to whom correspondence should be addressed.
Antibodies 2021, 10(4), 50; https://doi.org/10.3390/antib10040050
Submission received: 29 October 2021 / Revised: 23 November 2021 / Accepted: 13 December 2021 / Published: 17 December 2021
(This article belongs to the Special Issue Reviews on Antibodies and Antigens)

Abstract

:
While sensitivity and specificity are important characteristics for any diagnostic test, the influence of prevalence is equally, if not more, important when such tests are used in community screening. We review the concepts of positive/negative predictive values (PPV/NPV) and how disease prevalence affects false positive/negative rates. In low-prevalence situations, the PPV decreases drastically. We demonstrate how using two tests in an orthogonal fashion can be especially beneficial in low-prevalence settings and greatly improve the PPV of the diagnostic test results.

1. Introduction

The need for robust screening policies is essential in the current pandemic. This has yet to be achieved: one study [1] of 936 centers showed that universal COVID-19 testing for surgical patients was only recommended in 18.4% of centers, with not all centers advocating the routine testing of hospital staff. It is thus essential for clinicians to understand how best to implement COVID-19 screening programs. For example, in our country, since May 2021, all healthcare workers are routinely screened twice a week with rapid antigen tests [2]. Several factors are important to consider in the community screening of COVID-19: the turnaround time of test results, the sensitivity of the assay, the frequency of testing, and the prevalence of the disease in the population.
Many rapid SARS-CoV-2 tests (antigen, RT-PCR, and antibody tests) are now available to assist in the screening and management of COVID-19. Although rapid nucleic acid tests are available and desirable due to their greater analytical sensitivity, supply of these reagents may be rate limiting, and they are also more expensive than antigen testing [3]. Although some studies have questioned the use of point-of-care tests (POCTs) in screening for COVID-19 [4,5], they did not consider an increased testing frequency in their assessments. For COVID-19, several studies have already demonstrated that repeated population screening, even with a low-sensitivity test, can reduce the average infectiousness of individuals and limit viral spread [6,7,8]. In a recent report [7], antigen lateral flow immunoassays (LFIAs) (Sofia SARS antigen assay) used every 3 days had sensitivities similar to centralized RT-PCR testing in patients with COVID-19 (sensitivities remained >0.98 over a 14-day period). This concurs with the latest Centers for Disease Control and Prevention guidelines [9] that recommend confirmatory PCR testing may not be necessary if serial antigen testing is performed every three days/twice per week. Antibody POCTs have also demonstrated good performance. In our own evaluation [10], we showed that assay sensitivities of the Abbott/Roche antibody POCTs can be 78.1% one week after the first positive RT-PCR.
However, in addition to testing frequency, it is important to understand how prevalence influences the performance of these tests, especially in low-prevalence situations. In previous articles that explore diagnostic methods of COVID-19 testing [11,12,13,14,15,16,17], this important concept is not elaborated upon, or only explained in brief. Thus, in this article, we explain the concepts of PPV/NPV and the influence of disease prevalence. We also emphasize how using two tests in an orthogonal fashion can be especially beneficial in improving the PPV of the individual tests in such circumstances.

2. Sensitivity, Specificity, PPV, and NPV

There are several articles [18,19] that provide very good explanations for the concepts of assay sensitivity and specificity. Sensitivity represents how many diseased patients have a positive result. Specificity is the percentage of non-diseased patients with a negative test result. PPV is the probability of true disease when the result is positive. NPV is the probability of no disease when the result is negative (see Figure 1).

The Effect of Disease Prevalence

The classic monograph [20] provides a clear explanation of the influence of prevalence on PPV/NPV. From the most recent Cochrane review [21], the SD Biosensor STANDARD Q COVID-19 rapid antigen test had a sensitivity/specificity of 69.2%/99.1%, with a PPV/NPV of 93.7%/93.3%, while a rapid molecular test (Cepheid Xpert Xpress) had a sensitivity/specificity of 88.1%/97.2%, with a PPV/NPV of 97.9%/98.4% (see Table 1). The performance of rapid antibody tests has improved greatly, with a recent study [22] stating that good lateral flow antibody assays with acceptable overall evaluations can have specificities of 98.0–99.5%, and sensitivities of 87.7–98.5%.
Applying these values to the readily available calculator from the Food and Drug Administration (FDA) [23], using a range of disease prevalence from 0.1 to 1.0%, we can obtain the corresponding PPV/NPV (see Table 2). Thus, when used alone, all three modalities will achieve a PPV of <10% when disease prevalence is 0.1%.
This means that in a population with a low disease prevalence, a positive result is confounded by a greater percentage of false positive cases. It is only when disease prevalence is at least 1.4%/3.1%/2.3% for the antigen/molecular/antibody POCTs that PPVs exceed 50%. This is especially important when testing the general population who are largely asymptomatic and when disease prevalence is low.

3. Orthogonal Testing

In low-prevalence settings, the high false positive results can be reduced by using two tests in an orthogonal fashion. In this situation, a second test is used to confirm the positive results from the first test. Only when both tests are positive is the final result considered truly positive. Calculations can be easily made on the FDA calculator [23]. In this way, when antigen testing is combined with either molecular or antibody testing in an orthogonal fashion at 0.1% prevalence, the PPV improves by 10- to 20-fold (see Table 2).

4. Discussion

Disease prevalence must be considered when interpreting test results. In a low-disease prevalence population, using one POCT on its own will generate more false positive results than when used in a higher-prevalence situation. This is especially challenging as the disease prevalence of COVID-19 varies greatly between and within populations. One recent meta-analysis [24] showed that the SARS-CoV-2 seroprevalence varied markedly among geographic regions, from 1.45% (South America) to 5.27% (Northern Europe). The difference in prevalence can occur even between different locations within the same country, as one UK study [25] showed that healthcare workers had an 11 times higher hazard ratio for COVID-19 than the general community. Tracking the prevalence of COVID-19 is also complicated by asymptomatic cases of COVID-19. In a survey of 936 centers across 71 countries [26], 27.5% of centers had experienced preoperatively asymptomatic patients testing positive only after surgery. Careful consideration of disease prevalence would be required before embarking on community testing, and screening programs may have to be more extensive in order to detect asymptomatic and pauci-symptomatic cases of COVID-19. When embarking on screening programs, medical centers would benefit from taking into account disease prevalence and orthogonal testing strategies.
However, in populations with a low disease prevalence, using two POCTs in an orthogonal fashion markedly improves the PPV of the results. The performance of orthogonal testing algorithms has already been reported. In a community COVID-19 seroprevalence study (n = 4333) [27], second-line testing (SARS-CoV-2 IgG spike antibodies, sensitivity/specificity 100%/98.4%) confirmed 78/98 (80%) initially positive test results (SARS-CoV-2 IgG nucleocapsid antibodies, sensitivity/specificity 96.4%/99.0%). The initial 2.3% seroprevalence was then revised downwards to 1.8%, proving that orthogonal testing can help reduce false positive rates. Another study [28] tested a population of seronegative Japanese healthcare workers (n = 1000) with a SARS-CoV-2 antibody POCT (Instant-view IgG/IgM Antibody COVID-19 LFIA) and had a positive rate of 3.3%. After applying a second automated antibody test (Roche Elecsys Anti-SARS-CoV-2 RUO assay) in an orthogonal fashion, all initially positive tests were considered false positives.
In summary, we wish to re-emphasize that population prevalence must always be considered when planning to implement SARS-CoV-2 population screening. In low-prevalence settings, the false positive rate may be unacceptably high, with disastrous consequences such as loss of confidence in the screening program, or over-use of resources from misclassification of cases. If community screening is planned, the use of two tests in an orthogonal fashion for cases with a positive first test is an important means to improve the PPV of COVID-19 screening: only cases with a positive first test are evaluated with a second test, and only when both tests are positive is the patient considered to have tested positive. As the second test is only used in those with a positive first test, this would be more cost efficient than performing two tests for all subjects.

Author Contributions

Conceptualization, T.-C.A. and C.-S.L.; resources, T.-C.A. and C.-S.L.; writing—original draft preparation, C.-S.L.; writing—review and editing, T.-C.A.; supervision, T.-C.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

FDA: Food and Drug Administration; LFIA: lateral flow immunoassay; NPV: negative predictive value; POCTs: point-of-care tests; PPV: positive predictive value.

References

  1. Bellato, V.; Konishi, T.; Pellino, G.; An, Y.; Piciocchi, A.; Sensi, B.; Siragusa, L.; Khanna, K.; Pirozzi, B.M.; Franceschilli, M.; et al. Screening policies, preventive measures and in-hospital infection of COVID-19 in global surgical practices. J. Glob. Health 2020, 10, 020507. [Google Scholar] [CrossRef] [PubMed]
  2. Lau, C.S.; Aw, T.C. Frequent severe acute respiratory syndrome coronavirus 2 antigen testing in a disease-free population. J. Infect. Dis. 2021, 224, 529. [Google Scholar] [CrossRef] [PubMed]
  3. Mo, X.; Wang, X.; Zhu, Z.; Yu, Y.; Chang, D.; Zhang, X.; Li, D.; Sun, F.; Zhou, L.; Xu, J.; et al. Quality management for point-of-care testing of pathogen nucleic acids: Chinese expert consensus. Front. Cell. Infect. Microbiol. 2021, 11, 755508. [Google Scholar] [CrossRef] [PubMed]
  4. Mulchandani, R.; Jones, H.E.; Taylor-Phillips, S.; Shute, J.; Perry, K.; Jamarani, S.; Brooks, T.; Charlett, A.; Hickman, M.; Oliver, I.; et al. Accuracy of UK rapid test consortium (UK-RTC) "AbC-19 Rapid Test" for detection of previous SARS-CoV-2 infection in key workers: Test accuracy study. BMJ 2020, 371, m4262. [Google Scholar]
  5. Garcia-Finana, M.; Hughes, D.M.; Cheyne, C.P.; Burnside, G.; Stockbridge, M.; Fowler, T.A.; Fowler, V.L.; Wilcox, M.H.; Semple, M.G.; Buchan, I. Performance of the innova SARS-CoV-2 antigen rapid lateral flow test in the Liverpool asymptomatic testing pilot: Population based cohort study. BMJ 2021, 374, n1637. [Google Scholar] [CrossRef]
  6. Larremore, D.B.; Wilder, B.; Lester, E.; Shehata, S.; Burke, J.M.; Hay, J.A.; Tambe, M.; Mina, M.J.; Parker, R. Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening. Sci. Adv. 2021, 7, eabd5393. [Google Scholar] [CrossRef]
  7. Smith, R.L.; Gibson, L.L.; Martinez, P.P.; Ke, R.; Mirza, A.; Conte, M.; Gallagher, N.; Conte, A.; Wang, L.; Fredrickson, R.; et al. Longitudinal assessment of diagnostic test performance over the course of acute SARS-CoV-2 infection. J. Infect. Dis. 2021, 224, 976–982. [Google Scholar] [CrossRef]
  8. Mina, M.J.; Parker, R.; Larremore, D.B. Rethinking Covid-19 test sensitivity—A strategy for containment. N. Engl. J. Med. 2020, 383, e120. [Google Scholar] [CrossRef]
  9. Centers for Disease Control and Prevention. Interim Guidance for Antigen Testing for SARS-CoV-2, Updated 9 September 2021. Available online: https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antigen-tests-guidelines.html#table1 (accessed on 22 October 2021).
  10. Lau, C.; Hoo, S.; Liang, Y.; Phua, S.; Aw, T.C. Performance of two rapid point of care SARS-CoV-2 antibody assays against laboratory-based automated chemiluminescent immunoassays for SARS-CoV-2 IG-G, IG-M and total antibodies. Pr. Lab. Med. 2021, 24, e00201. [Google Scholar] [CrossRef]
  11. Harahwa, T.A.; Yau, T.H.L.; Lim-Cooke, M.-S.; Al-Haddi, S.; Zeinah, M.; Harky, A. The optimal diagnostic methods for COVID-19. Diagnosis 2020, 7, 349–356. [Google Scholar] [CrossRef]
  12. Ghaffari, A.; Meurant, R.; Ardakani, A. COVID-19 serological tests: How well do they actually perform? Diagnostics 2020, 10, 453. [Google Scholar] [CrossRef]
  13. da Silva, S.J.R.; da Silva, C.T.A.; Guarines, K.M.; Medes, R.P.G.; Pardee, K.; Kohl, A.; Pena, L. Clinical and laboratory diagnosis of SARS-CoV-2, the virus causing COVID-19. ACS Infect. Dis. 2020, 6, 2319–2336. [Google Scholar] [CrossRef]
  14. Caeseele, P.V.; Bailey, D.; Forgie, S.E.; Dingle, T.C.; Krajden, M. SARS-CoV-2 (COVID-19) serology: Implications for clinical practice, laboratory medicine and public health. CMAJ 2020, 192, E973–E979. [Google Scholar] [CrossRef]
  15. Premraj, A.; Aleyas, A.G.; Nautiyal, B.; Rasool, T.J. Nucleic acid and immunological diagnostics for SARS-CoV-2: Processes, platforms and pitfalls. Diagnostics 2020, 10, 866. [Google Scholar] [CrossRef]
  16. Gulholm, T.; Basile, K.; Kok, J.; Chen, S.-A.; Rawlinson, W. Laboratory diagnosis of severe acute respiratory syndrome coronavirus 2. Pathology 2020, 52, 745–753. [Google Scholar] [CrossRef]
  17. Lagerqvist, N.; Maleki, K.T.; Verner-Carlsson, J.; Olausson, M.; Dillner, J.; Bystrom, J.W.; Monsen, T.; Forsell, M.; Eriksson, J.; Bogdanovic, G.; et al. Evaluation of 11 SARS-CoV-2 antibody tests by using samples from patients with defined IgG antibody titers. Sci. Rep. 2021, 11, 7614. [Google Scholar] [CrossRef]
  18. Loong, T.-W. Understanding sensitivity and specificity with the right side of the brain. BMJ 2003, 327, 716–719. [Google Scholar] [CrossRef]
  19. Kankam, H.K.; Hourston, G.J.; Ravindran, P.; Azhar, B.; Pope, C. COVID-19 antibody tests: Statistical implications. Br. J. Hosp. Med. 2020, 81, 1–5. [Google Scholar] [CrossRef]
  20. Galen, R.S.; Gambino, S.R. Beyond Normality, The Predictive Value and Efficiency of Medical Diagnoses; Wiley-Biomedical, John Wiley & Sons, Inc.: New York, NY, USA, 1975; pp. 9–13. [Google Scholar]
  21. Dinnes, J.; Deeks, J.J.; Berhane, S.; Taylor, M.; Adriano, A.; Davenport, C.; Dittrich, S.; Emperador, D.; Takwoingi, Y.; Cunningham, J.; et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst. Rev. 2021, 3, CD013705. [Google Scholar]
  22. Tollanes, M.C.; Jenum, P.A.; Kierkegaard, H.; Abildsnes, E.; Baevre-Jensen, R.M.; Breivik, A.C.; Sandberg, S. Evaluation of 32 rapid tests for detection of antibodies against SARS-CoV-2. Clin. Chim. Acta. 2021, 519, 133–139. [Google Scholar] [CrossRef]
  23. Food and Drug Administration. Calculator for Positive Predictive Value (PPV) and Negative Predictive Value (NPV) for individual tests and combined. Available online: https://www.fda.gov/media/137612/download (accessed on 22 October 2021).
  24. Rostami, A.; Sepidarkish, M.; Leeflang, M.M.G.; Riahi, S.M.; Shiadeh, M.N.; Esfandyari, S.; Mokdad, A.H.; Hotez, P.J.; Gasser, R.B. SARS-CoV-2 seroprevalence worldwide: A systematic review and meta-analysis. Clin. Microbiol. Infect. 2021, 27, 331–340. [Google Scholar] [CrossRef]
  25. Nguyen, L.H.; Drew, D.A.; Graham, M.S.; Joshi, A.D.; Guo, C.-G.; Ma, W.; Mehta, R.S.; Warner, E.T.; Sikavi, D.R.; Lo, C.-H.; et al. Risk of COVID-19 among front-line health-care workers and the general community: A prospective cohort study. Lancet Public Health 2020, 5, e475–e483. [Google Scholar] [CrossRef]
  26. Bellato, V.; Konishi, T.; Pellino, G.; An, Y.; Piciocchi, A.; Sensi, B.; Siragusa, L.; Khanna, K.; Pirozzi, B.M.; Franceschilli, M.; et al. Impact of asymptomatic COVID-19 patients in global surgical practice during the COVID-19 pandemic. Br. J. Surg. 2020, 107, e364–e365. [Google Scholar] [CrossRef]
  27. Xu, G.; Emanuel, A.J.; Nadig, S.; Mehrotra, S.; Caddell, B.A.; Curry, S.R.; Nolte, F.S.; Babic, N. Evaluation of orthogonal testing algorithm for detection of SARS-CoV-2 IgG antibodies. Clin. Chem. 2020, 66, 1531–1537. [Google Scholar] [CrossRef]
  28. Nakamura, A.; Sato, R.; Ando, S.; Oana, N.; Nozaki, E.; Endo, H.; Miyate, Y.; Soma, J.; Miyata, G. Orthogonal antibody testing for COVID-19 among healthcare workers in a non-epidemic place and time:Japan’s Iwate Prefecture, May 18-31, 2020. Fukushima J. Med. Sci. 2020, 67, 27–32. [Google Scholar] [CrossRef]
Figure 1. Overview of sensitivity, specificity, and positive and negative predictive values. Abbreviations: TP: true positive, FN: false negative, FP: false positive, TN: true negative, PPV: positive predictive value, NPV: negative predictive value.
Figure 1. Overview of sensitivity, specificity, and positive and negative predictive values. Abbreviations: TP: true positive, FN: false negative, FP: false positive, TN: true negative, PPV: positive predictive value, NPV: negative predictive value.
Antibodies 10 00050 g001
Table 1. Sensitivity, specificity, and predictive values of rapid tests.
Table 1. Sensitivity, specificity, and predictive values of rapid tests.
AssaySensitivitySpecificityPPVNPV
Antigen
(SD Biosensor)
69.2%99.1%93.7%93.3%
Molecular
(Cepheid)
88.1%97.2%97.9%98.4%
Antibody *87.7%98.0%--
* Values are the lower end of sensitivity and specificity from reference [22].
Table 2. Effect of prevalence and orthogonal testing on predictive values.
Table 2. Effect of prevalence and orthogonal testing on predictive values.
ParameterPrevalence 0.1%Prevalence 0.5%Prevalence 1.0%
PPVNPVPPVNPVPPVNPV
Single test
Antigen7.110027.999.843.799.7
Molecular3.110013.799.924.199.9
Antibody4.210018.199.930.799.9
Orthogonal Testing
Antigen → molecular70.899.192.495.596.191.3
Antigen → Antibody77.199.094.495.497.191.1
Abbreviations: PPV: positive predictive value, NPV: negative predictive value.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Lau, C.-S.; Aw, T.-C. Disease Prevalence Matters: Challenge for SARS-CoV-2 Testing. Antibodies 2021, 10, 50. https://doi.org/10.3390/antib10040050

AMA Style

Lau C-S, Aw T-C. Disease Prevalence Matters: Challenge for SARS-CoV-2 Testing. Antibodies. 2021; 10(4):50. https://doi.org/10.3390/antib10040050

Chicago/Turabian Style

Lau, Chin-Shern, and Tar-Choon Aw. 2021. "Disease Prevalence Matters: Challenge for SARS-CoV-2 Testing" Antibodies 10, no. 4: 50. https://doi.org/10.3390/antib10040050

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop